I want to chain routes in NodeJs with values after sending API response to end-ser,
WHY: > The uploaded files would be somewhat large (5-50mb each) and require some processing, can not make my API user wait/timeout while my NodeJS code is working.. so need, 1: Upload files and send success immediately to user, Process files (few promises) and return/log success/failure for notification system.
My individual code blocks are done and working fine (i.e. upload service and file processing service both are good under tests and work nicely when tested individually.)
now with the API to upload in place, I've added following code:
router.post('/upload', upload.array('upload_data', multerMaxFiles), (req, res, next) => {
////some uploading and processing stuff - works nicely
res.json({ 'message': 'File uploaded successfully.' });// shown to API client nicely
console.log("what next? " + utilz.inspect(uploaded_file_paths)) //prints file names on console
next();
});
PROBLEM:
app.use('/api', uploadRoute); //The above code route
//want some processing to be done
app.use(function(req, res, next) {
**want those uploaded file names here**
tried with few response object options but stabs with error
});
OR
use something like ....
app.use(someFunction(uploaded_file_names)); **want those uploaded file names as params**
PS:
Any promise after the file upload success would result in 'Error: Can't set headers after they are sent.', so not helpful writing anything there.
Any suggestions folks.
--
N Baua
Once you've sent a response back to the browser (to keep it from timing out during your long processing time), that http request is done. You cannot send any more data on it and trying to do so will trigger a server-side error. You cannot "chain routes" the way you were asking as you seem to want to do because you simply can't send more data over that http request once you've sent the first response.
There are two common ways to deal with this issue.
As part of your initial response, send back a transaction ID and then have the client poll back every few seconds with an Ajax call asking what the final status is of that transaction. The server can return "in progress" until it is finally done and then it can return the final status.
You can connect a webSocket or socket.io connection from client to server. As part of your initial response to the upload, send back a transaction ID. Then, when the transaction is done server-side, it sends a notification on the webSocket or socket.io connection for that particular client with the transactionID with the final status. The client can then respond accordingly to that final status. You can either keep the webSocket/socket.io connection open for use with other requests or you can then close that connection.
Using either technique, you could also return/send a progress value (like percent complete) that the client could use to display completion progress. This is generally very helpful on the client-side to keep an impatient user from giving up or refreshing the page. If they can see that the processing is proceeding, they won't give up thinking that maybe it stopped working.
This should work with res.write(). But it does depend on your clients cache i think.
I tried this, but it does not work in my firefox.
app.get('/test', function(req, res) {
var count = 0;
var interval = setInterval(function() {
if (count++ === 100) {
clearInterval(interval);
res.end();
}
res.write('This is line #' + count + '\n');
}, 100);
});
After I increased frequency and number of writes it seems to work.
So try:
router.post('/upload', upload.array('upload_data', multerMaxFiles), (req, res, next) => {
////some uploading and processing stuff - works nicely
res.write(JSON.stringify({ 'message': 'File uploaded successfully.' }));// shown to API client nicely
console.log("what next? " + utilz.inspect(uploaded_file_paths)) //prints file names on console
next();
});
//STEP (1)
//Example simulate upload multiple files with chained api calls. In this example the parameters "idFile" and "arrayidFileExample" are helpful.
//You should create and send this data from view.
{
"idFile": "04fe640f6e4w", //id first file
"arrayidFileExample": ["04fe640f6e4w","03g5er4g65erg","g0er1g654er65g4er","0g4er4g654reg654re"] //simulate idFiles array from view
}
//STEP (2)
//your upload files api
app.post('/upload', function(req, res) {
//Express headers, no important in this code
res.header("Access-Control-Allow-Origin", "*");
res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
let arrayIdFiles = req.body.arrayidFileExample; //create arrayIdFiles
let isBlock = false; //flag to block call loop to api
let countIndex = 0;
let currentIndex = 0;
//STEP (3)
//If "arrayIdFiles" index is not exist, set isBlock to true. Remeber flag to block call loop to api
for(let count in arrayIdFiles) {
if(req.body.idFile == arrayIdFiles[count]) {
console.log("current index --> ", countIndex)
countIndex++;
console.log("next index --> ", countIndex)
if(arrayIdFiles[countIndex] == undefined) {
isBlock = true;
}
break;
}
countIndex++;
currentIndex++;
}
//STEP (4)
//If isBlock is equal false, call get upload api with next idFile. this is simulate "recursive api loop"
if(isBlock == false) {
postUploadFile(
'http://localhost:3500/upload',
{
"idFile":arrayIdFiles[currentIndex + 1], //send next idFile from arrayIdFiles
"arrayidFileExample": arrayIdFiles //initial arrayIdFiles
});
}
//STEP (6)
//response json example
const json = JSON.stringify({
error:false,
statusCode: 200,
body:{
message:'current id file '+req.body.idFile,
}
});
res.write(json);
return res.end();
});
//STEP (5)
//call "/upload" api post
const postUploadFile = (url = '', body = {}) => {
return new Promise((resolve, reject)=>{
axios.post(url, body).then(response => {
return resolve(response.data);
}).catch(error => {});
});
};
//server listen instance
server.listen(3500,() => {
});
Related
I'm working on a SPA website with node.js, jQuery, mongoose and MongoDB for a shopping website.
The ajax requests and responses work perfectly when starting from the index.html file. So for example begining on http://localhost:3000 and someone clicks on a link called 'products' I send an ajax request to the the server and the server sends the necessary product information back asynchronously which lead to http://localhost:3000/products. But the problem is that if someone types http://localhost:3000/products directly in the search bar it will show the json representation of the products.
This is my code:
script.js
function redirect (link) {
$.ajax({
type: 'GET',
url: 'http://localhost:3000/' + link,
contentType: 'application/json',
data: {
link
},
success: function (res) {
let container = $('#contentToSwap');
container.html('');
res.products.forEach(function (products_) {
...
});
}
});
}
app.js
var Product = require('./models/product');
var mongoose = require('mongoose');
var bodyParser = require('body-parser');
var path = require('path');
var express = require('express');
var app = express();
mongoose.connect('mongodb://localhost:27017/shopping');
var PORT = process.env.PORT || 3000;
app.use(express.static(path.join(__dirname, 'public')));
app.use(bodyParser.json());
app.get('*', function(req, res) {
Product.find(function(err, docs) {
let productChunks = [];
let chunksize = 4;
let display = [];
for (var i = 0; i < docs.length; i++) {
if (docs[i].productType == req.query.link) display.push(docs[i]);
}
for (var i = 0; i < display.length; i += chunksize) {
productChunks.push(display.slice(i, i + chunksize));
}
res.send({ products: productChunks });
});
});
app.listen(PORT, function () {
console.log('Listening on port ' + PORT);
});
So I need some sort of frontend routing if the user doesn't start at the index.html file. I know that I could write my own router to route the urls correctly and that I could route all requests back to the index.html like
app.get('*', function(req, res) {
res.sendFile(__dirname + '/public/index.html');
});
But then I cannot load all the necessary product information from the server when someone clicks a link. So I'm a little bit confused on hwo to tackle this issue. Any help is appreciated
This is usually achieved by separating api routes from normal ones by adding specific url prefixes such as /api for all routes that return json data. What you can do is to specify /api/whatever-you-want, make it the target for your ajax call and place it above app.get('*' ....
Since routes and middleware functions are resolved top to bottom, it will be matched by your ajax call only, leaving the /products unaffected.
answer to question -- Is it possible to redirect user from /api/products to /product if the request wasn't generated by ajax?
Yes, it is possible by adding request query parameter to ajax call which will not be present on normal call and then check those on the server side and decided what to do if it (that specific query parameter) is missing or not.
Let's assume some client side JS that generates ajax call.
fetch('/api/products?api=true')
.then((data) => data.json())
.then((json) => console.log(json));
Notice the request url - /api/products?api=true
Now assume a normal call from html file.
products
These two calls differ in that api query parameter (ajax call has it, the other one doesn't).
And for the server side part of the task -- request query parameters object can be accessed via query property on request object (req.query).
app.get('/api/products', (req, res) => {
if (!req.query.api) {
// if get request doesn't contain api param. then
// handle it accordingly, e.g. do redirect
return res.redirect('/products');
}
// request comming from ajax call, send JSON data back
res.json({ randomStuff: 'abcd' });
});
I am new to both node and express so I figure I am doing something stupid.
Complete source code can be found at:
https://github.com/wa1gon/aclogGate/tree/master/server
logRouter.get("/loggate/v1/listall", function(req, res) {
let countStr = req.param('count');
let count: number;
if (!countStr) {
count = null;
} else {
count = Number.parseInt(countStr);
if (count == NaN) count = null;
}
acConn.listAllDatabase(count, (err: string, result: Array<LogGateResp>) => {
console.log("got list all data resp")
return res.json(result).end();
});
}
);
app.use('/', logRouter);
It works the first time though, but blows up the second.
listallDatabase connects to a network socket which gets XML database back, parses it and calls back with an JS object. Which in turn calls res.json.
Suggestions?
Remove the end() after res.json().
res.josn() send the response to frontend and end() try to send the response again.
That why you are getting the error. Because node.js don't allow the API to send response twice. Either use res.end() or res.json().
My former server.js is like:
After running the server I could see my index.html
var connect = require('connect');
var serveStatic = require('serve-static');
connect().use(serveStatic(__dirname)).listen(5000, '192.168.xx.xx', function(){
console.log('Server running on 5000');
});
I want to create http login and password to secure the website, so I found online the information of http module: if I put right login and password, I could see congratulations message:
var http = require('http');
var server = http.createServer(function(req, res) {
// console.log(req); // debug dump the request
// If they pass in a basic auth credential it'll be in a header called "Authorization" (note NodeJS lowercases the names of headers in its request object)
var auth = req.headers['authorization']; // auth is in base64(username:password) so we need to decode the base64
console.log("Authorization Header is: ", auth);
if(!auth) { // No Authorization header was passed in so it's the first time the browser hit us
// Sending a 401 will require authentication, we need to send the 'WWW-Authenticate' to tell them the sort of authentication to use
// Basic auth is quite literally the easiest and least secure, it simply gives back base64( username + ":" + password ) from the browser
res.statusCode = 401;
res.setHeader('WWW-Authenticate', 'Basic realm="Secure Area"');
res.end('<html><body>Need authorization</body></html>');
}
else if(auth) { // The Authorization was passed in so now we validate it
var tmp = auth.split(' '); // Split on a space, the original auth looks like "Basic Y2hhcmxlczoxMjM0NQ==" and we need the 2nd part
var buf = new Buffer(tmp[1], 'base64'); // create a buffer and tell it the data coming in is base64
var plain_auth = buf.toString(); // read it back out as a string
console.log("Decoded Authorization ", plain_auth);
// At this point plain_auth = "username:password"
var creds = plain_auth.split(':'); // split on a ':'
var username = creds[0];
var password = creds[1];
if((username == 'admin') && (password == 'admin')) { // Is the username/password correct?
res.statusCode = 200; // OK
res.end('<html><body>Congratulations, feel free to explre!</body></html>');
}
else {
res.statusCode = 401; // Force them to retry authentication
res.setHeader('WWW-Authenticate', 'Basic realm="Secure Area"');
// res.statusCode = 403; // or alternatively just reject them altogether with a 403 Forbidden
res.end('<html><body>You shall not pass</body></html>');
}
}
});
server.listen(5000, function() { console.log("Server Listening on http://localhost:5000/"); });
I am new to nodejs, I want to know how to combine this 2 js? In order to realize my function of adding authorization to my web.
Could I do something to show my index instead of showing congratulation message after putting the login and password?
Thanks a lot.
In order to show HTML page instead of congratulation message, you can follow these steps:
Get request path by req.url, such as / or /introduction.html.
According to the above path, read the corresponding HTML file in server disk, using fs.readFile().
Return HTML file content to browser if the read is successful. Otherwise, return 404 error page.
Here is some example code for above steps:
if((username == 'admin') && (password == 'admin')) { // Is the username/password correct?
res.statusCode = 200; // OK
// res.end('<html><body>Congratulations, feel free to explre!</body></html>');
var requestURL = req.url; // e.g. / or /a or /a.html
var requestFilePath = getFilePathFromRequestURL(requestURL); // you need to implement this logic yourself, such as "/" mapping to "./index.html"
fs.readFile(requestFilePath, function(error, data) {
if (error) {
res.statusCode = 404;
res.write('File not found.');
} else {
res.statusCode = 200;
res.write(data);
}
res.end();
});
}
However, unless you want to write some low-level node.js code to better understand this language, I highly recommend using node.js web framework such as Express. Serve HTTP request using low-level node.js would be tedious, especially in production code.
Also, please note that using WWW-Authenticate Basic for authentication is neither secure nor user-friendly. You need some other way to implement authentication, such as JSON Web Tokens
There are two examples in between these pages 16 and 18.
Example 1.3 is a server app.
Example 1.4 is a client app doing GET requests to the server.
When I run the two examples (at the same time) I notice some quite weird behavior
in the client. All requests are executed (i.e. the for loop in the client completes)
but the callbacks of only 5 of them get called. The client doesn't exit and also
doesn't error out. And just no more callbacks are called.
Any ideas what might be happening or how I can troubleshoot this further?
Note: I am running Node.js v0.10.20 on Windows 7.
Server:
var http = require('http');
var fs = require('fs');
// write out numbers
function writeNumbers(res) {
var counter = 0;
// increment, write to client
for (var i = 0; i<100; i++) {
counter++;
res.write(counter.toString() + '\n');
}
}
// create http server
http.createServer(function (req, res) {
var query = require('url').parse(req.url).query;
var app = require('querystring').parse(query).file;
// content header
res.writeHead(200, {'Content-Type': 'text/plain'});
if (!app){
res.end();
console.log('No file argument found in query string.');
return;
}else{
app = app + ".txt";
}
// write out numbers
writeNumbers(res);
// timer/timeout to open file and read contents
setTimeout(function() {
console.log('Opening file: ' + app + '.');
// open and read in file contents
fs.readFile(app, 'utf8', function(err, data) {
res.write('\r\n');
if (err)
res.write('Could not find or open file ' + app + ' for reading.\r\n');
else {
res.write(data);
}
// response is done
res.end();
});
},2000);
}).listen(8124);
console.log('Server running at 8124');
Client:
var http = require('http');
var N = 200;
// The URL we want, plus the path and options we need
var options = {
host: 'localhost',
port: 8124,
path: '/?file=automatic',
method: 'GET'
};
var callback_function = function(response) {
// finished? ok, write the data to a file
console.log('got response back');
};
for (var i = 1; i <= N; i++) {
// make the request, and then end it, to close the connection
http.request(options, callback_function).end();
console.log('done with call # ' + i);
}
--- Experiment Done ---
If I lower N to 10 and also if I do a
global "var i = 1" and then do this thing:
function schedule(){
http.request(options, callback_function).end();
console.log('done with call ' + i);
i++;
if (i<=N){
setTimeout(function(){
schedule();
}, 1000);
}
}
schedule();
instead of the loop in the client, I get similar behavior.
I guess that's what Milimetric meant by "sleep" i.e. just
to make sure I don't hit the server too quickly with too
many simultaneous requests.
But the behavior is not fully identical, it takes several mins
to print 'got response back' on the second set of 5 requests
and then another maybe 5-6 mins for the client to exit.
Still, all that does look weird to me.
C:\PERSONAL\NODE_TEST>node test004.js
done with call 1
got response back
done with call 2
got response back
done with call 3
got response back
done with call 4
got response back
done with call 5
got response back
done with call 6
done with call 7
done with call 8
done with call 9
done with call 10
got response back
got response back
got response back
got response back
got response back
C:\PERSONAL\NODE_TEST>
The problem is that the client doesn't consume the response body sent by the server, so the connection remains (half) open and the http agent only allows 5 concurrent requests per client by default, causing it to hang after 5 requests. The connection will eventually timeout, causing the next 5 requests to be processed.
node.js http.get hangs after 5 requests to remote site
Change your callback function to consume any data sent down the response.
var callback_function = function(response) {
// finished? ok, write the data to a file
console.log('got response back');
response.on('data', function () {});
};
I have a page with a route GET /team which is loading a list of teams, and DEL /team which is deleting a team from /team/:key. So you navigate to a team's profile page and delete them from there, on deletion it should redirect you to the /team page. I have put logs into the console and it is successfully deleting the team and wierdly, it says it is loading /team but the browser does not load this. I have put my code below, any ideas?
Routes:
app.get('/team'/*, lim("Must be logged in to see teams")*/, getAllTeams, function(req, res){
util.log('Serving request for url [GET] ' + req.route.path);
// Pass it the list of all Teams
res.render('team', {'teamsList' : req.teamsList} );
});
app.get('/team/:key', function(req, res) {
util.log('Serving request for url [GET] ' + req.route.path);
Team.findByKey(req.params.key, function(err, teamData){
if(!err && teamData){
teamData = teamData;
res.render('teamDetails', { 'teamData' : teamData } );
} else {
util.log('Error in fetching Team by key : ' + req.params.key);
res.json({
'retStatus' : 'failure',
'msg' : 'Error in fetching Team by key ' + req.params.key
});
}
});
});
/**
* DEL /team/:key
* Delete Team by key
*/
app.del('/team/:key', getAllTeams, function(req, res) {
util.log('Serving request for url [DEL] ' + req.route.path);
Team.remove({key : req.params.key}, function(err){
var message = '';
var retStatus = '';
if (!err) {
util.log('Successfully deleting Team with key: ' + req.params.key);
message = 'Successfully deleting Team with key: ' + req.params.key;
retStatus = 'Success';
res.redirect('/team');
} else {
util.log('Error deleting Team with key: ' + req.params.key + 'Error: ' + util.inspect(err));
res.json({
'retStatus' : 'failure',
'msg' : 'Error in fetching Team with key ' + req.params.key
});
}
});
});
JavaScript + HTML template:
button#teamDelete.btn.btn-danger.btn-mini(type="submit", value="Delete Team") Delete
script(type='text/javascript')
$('#teamDelete').live('click',function(){
var teamId = #{teamData.key};
$.post('/team/' + teamId, { _method : 'delete' }, function(response) {
console.log(response);
if(response.retStatus === 'Success') {
if('/team' && '/team' !== "") {
window.location = '/team';
}
}
});
});
console logs:
10 Mar 11:52:01 - Serving request for url [GET] /team
10 Mar 11:52:02 - Serving request for url [GET] /team/:key
10 Mar 11:52:03 - Serving request for url [DEL] /team/:key
10 Mar 11:52:03 - Successfully deleting Team with key: 1362855941128
10 Mar 11:52:03 - Serving request for url [GET] /team
getAllTeams:
var getAllTeams = function(req, res, next){
Team.getAll(function(err, teamsList){
if(!err && teamsList){
req.teamsList = teamsList;
return next();
}
});
};
Team.getAll (Team schema)
Team.statics.getAll = function(cb){
var query = this.find({});
query.sort({key : -1});
return query.exec(cb);
};
Your request is POST ($.post) and you route is app.del, so it never gets to res.redirect inside app.del route.
Why don't you use app.post?
Updated:
Assuming $.post sends HTTP DEL request here what is happening: server sends 302 response with no data but browser never sends another request to GET route as server instructs it (or does jQuery handle redirects too? Not sure). res.redirect() is actual HTTP response not some internal server-side instruction to re-route the request to another route like you can do in ASP.NET (and which is wrong actually)... Route is designed to receive request, reply with the response and forget about it. You need to separate routes from actual functions processing them, then you will be able to call that function instead of sending redirect.
Code suggestions
In app.del('/team/:key' ...
...
retStatus = 'Success';
// res.redirect('/team');
res.send({
retStatus : retStatus,
redirectTo: '/team',
msg : 'Just go there please' // this should help
});
...
Client-side in $.post('/team/' ...
...
$.post('/team/' + teamId, { _method : 'delete' }, function(response) {
console.log(response);
if(response.retStatus === 'Success') {
// not sure what did you mean by ('/team' && '/team' !== "")
// if('/team' && '/team' !== "") {
if (response.redirectTo && response.msg == 'Just go there please') {
window.location = response.redirectTo;
}
}
});
...
Not sure it will work though because I don't understand what your getAllTeams does and why you store teamList in req. If you want to store in session, than assuming the middleware is correctly configured you need to use req.session. If you need to store it only within request and your getAllTeams prepares this list of teams it is better to store in res.locals (like res.locals.teamList).
And make sure your getAllTeams calls next. So basically your getAllTeams should look like this:
function getAllTeams (req, res, next) {
res.locals.teamList = [/* whatever */];
next();
}
And then you can use res.locals.teamList in your route handler instead of req.teamList.
res.render('team', {teamsList : res.locals.teamsList} );
And 'team' template also can have a problem...
Express advice :)
Also the way you use Express makes it very difficult to extend/manage application. I don't remember where exactly, but somewhere in docs they write that Express is supposed to be used as the foundation for your application framework, not as a complete framework like most PHP frameworks are. It gives you a lot of power and flexibility, but also makes it necessary to think about your application architecture well in advance.
The most powerful feature of express is that you can have any route handled by many route-specific handlers/middlewares passing control to each other via next(). I have a static table that defines which handlers are used on each route allowing to see the whole application with 30 or so routes on one page. And this table is used to assemble the routing dynamically when the application starts. It leads to a lot of flexibility, manageability (I can move/copy-paste handlers from route to route - each handler is represented as a single word) and code re-use. I also use the same hash of routes definition in the client for client-side routing.
For a quick workaround, just add the redirect url to the response and on the client side do:
if (redirectUrl && redirectUrl !== "")
window.location = redirectUrl;