How could I make the 'request' module in Node.js load things in a synchronous fashion? The best advice I've seen is to somehow use the callback to get the function to not return itself until it is done. I am trying to use the 'request' function inline in code (things need to be processed based on that data that can't be placed in callbacks).
So how could I use the callback of the 'request' module to keep it from returning itself until it is finished with loading the resource?
What I'm doing is running a loop that downloads two values from an API, and then has to do some math based on those values. While the math could be done in callbacks... the loop would advance without the values it needs to perform the next operation. (So stopping the loop from advancing until the data is ready would solve the issue)
/* loop */ {
/* URL Generation */
request( {url: base + u_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
ucomp = split2[1];
});
request( {url: base + v_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
vcomp = split2[1];
});
/* math which needs to be after functions get variables and before loop advances */
}
The short answer is: don't. (...) You really can't. And that's a good thing
I'd like to set the record straight regarding this:
NodeJS does support Synchronous Requests. It wasn't designed to support them out of the box, but there are a few workarounds if you are keen enough, here is an example:
var request = require('sync-request'),
res1, res2, ucomp, vcomp;
try {
res1 = request('GET', base + u_ext);
res2 = request('GET', base + v_ext);
ucomp = res1.split('\n')[1].split(', ')[1];
vcomp = res2.split('\n')[1].split(', ')[1];
doSomething(ucomp, vcomp);
} catch (e) {}
When you pop the hood open on the 'sync-request' library you can see that this runs a synchronous child process in the background. And as is explained in the sync-request README it should be used very judiciously. This approach locks the main thread, and that is bad for performance.
However, in some cases there is little or no advantage to be gained by writing an asynchronous solution (compared to the certain harm you are doing by writing code that is harder to read).
This is the default assumption held by many of the HTTP request libraries in other languages (Python, Java, C# etc), and that philosophy can also be carried to JavaScript. A language is a tool for solving problems after all, and sometimes you may not want to use callbacks if the benefits outweigh the disadvantages.
For JavaScript purists this may rankle of heresy, but I'm a pragmatist so I can clearly see that the simplicity of using synchronous requests helps if you find yourself in some of the following scenarios:
Test Automation (tests are usually synchronous by nature).
Quick API mash-ups (ie hackathon, proof of concept works etc).
Simple examples to help beginners (before and after).
Be warned that the code above should not be used for production. If you are going to run a proper API then use callbacks, use promises, use async/await, or whatever, but avoid synchronous code unless you want to incur a significant cost for wasted CPU time on your server.
In 2018, you can program the "usual" style using async and await in Node.js.
Below is an example, that wraps request callback in a promise and then uses await to get the resolved value.
const request = require('request');
// wrap a request in an promise
function downloadPage(url) {
return new Promise((resolve, reject) => {
request(url, (error, response, body) => {
if (error) reject(error);
if (response.statusCode != 200) {
reject('Invalid status code <' + response.statusCode + '>');
}
resolve(body);
});
});
}
// now to program the "usual" way
// all you need to do is use async functions and await
// for functions returning promises
async function myBackEndLogic() {
try {
const html = await downloadPage('https://microsoft.com')
console.log('SHOULD WORK:');
console.log(html);
// try downloading an invalid url
await downloadPage('http:// .com')
} catch (error) {
console.error('ERROR:');
console.error(error);
}
}
// run your async function
myBackEndLogic();
Though asynchronous style may be the nature of node.js and generally you should not do this, there are some times you want to do this.
I'm writing a handy script to check an API and want not to mess it up with callbacks.
Javascript cannot execute synchronous requests, but C libraries can.
https://github.com/dhruvbird/http-sync
Aredridels answer is relatively good (upvoted it), but I think it lacks the loop equivalent. This should help you:
Sync code equivalent:
while (condition) {
var data = request(url);
<math here>
}
return result;
Async code for serial execution:
function continueLoop() {
if (!condition) return cb(result);
request(url, function(err, res, body) {
<math here>
continueLoop()
})
}
continueLoop()
You should take a look at library called async
and try to use async.series call for your problem.
See sync-request: https://github.com/ForbesLindesay/sync-request
Example:
var request = require('sync-request');
var res = request('GET', 'http://example.com');
console.log(res.getBody());
You can use retus to make cross-platform synchronous HTTP requests. It's a library based on sync-request but with a few comfort features I added:
const retus = require("retus");
const { body } = retus("https://google.com");
//=> "<!doctype html>..."
That's it!
These days if you want to do things sequentially, you'd do something like:
for (const task of tasks) {
const response = await request(...);
}
This code (above) runs the requests in order, but is not synchronous (meaning it does not block the main thread). If you really need synchronous, then most of these libraries boil down to something like:
const child = require('child_process');
const buffer = child.execSync(`curl https://example.com/foo.json`);
const data = JSON.parse(buffer.toString('utf8'));
You can do something exactly similar with the request library, but this is sync using const https = require('https'); or const http = require('http');, which should come with node.
Here is an example,
const https = require('https');
const http_get1 = {
host : 'www.googleapis.com',
port : '443',
path : '/youtube/v3/search?arg=1',
method : 'GET',
headers : {
'Content-Type' : 'application/json'
}
};
const http_get2 = {
host : 'www.googleapis.com',
port : '443',
path : '/youtube/v3/search?arg=2',
method : 'GET',
headers : {
'Content-Type' : 'application/json'
}
};
let data1 = '';
let data2 = '';
function master() {
if(!data1)
return;
if(!data2)
return;
console.log(data1);
console.log(data2);
}
const req1 = https.request(http_get1, (res) => {
console.log(res.headers);
res.on('data', (chunk) => {
data1 += chunk;
});
res.on('end', () => {
console.log('done');
master();
});
});
const req2 = https.request(http_get2, (res) => {
console.log(res.headers);
res.on('data', (chunk) => {
data2 += chunk;
});
res.on('end', () => {
console.log('done');
master();
});
});
req1.end();
req2.end();
The easiest solution I came up for myself was to use the node's native "child_request" and simply call exec with a simple curl command inside.... all dependencies and asynchronity hassle me way too much for the simple rare cases in which I simply want to "save the http response to a variable in node"
As of this writing all answers are either:
Sync but with a control flow (such as using async library)
Sync, but blocking [which means that all other threads on Node are stopped, which is bad performance wise], such as retus or sync-request.
Out of date, such as http-request.
Check out the non-blocking sync-request, based on deasync.
Or see some of the answers in this thread, such as this, which use deasync directly.
The short answer is: don't. If you want code that reads linearly, use a library like seq. But just don't expect synchronous. You really can't. And that's a good thing.
There's little or nothing that can't be put in a callback. If they depend on common variables, create a closure to contain them. What's the actual task at hand?
You'd want to have a counter, and only call the callback when the data is there:
var waiting = 2;
request( {url: base + u_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
ucomp = split2[1];
if(--waiting == 0) callback();
});
request( {url: base + v_ext}, function( err, res, body ) {
var split1 = body.split("\n");
var split2 = split1[1].split(", ");
vcomp = split2[1];
if(--waiting == 0) callback();
});
function callback() {
// do math here.
}
Update 2018: node.js supports async/await keywords in recent editions, and with libraries that represent asynchronous processes as promises, you can await them. You get linear, sequential flow through your program, and other work can progress while you await. It's pretty well built and worth a try.
Related
I am trying to make a http request with node module http on an azure function with javascript and for some reason http.request is not receiving data(no error printed its like the requested is blocked). There is anything wrong with azure configuration or the code?Am i missing something really obvious?
Code is running fine on local js file
context.log('JavaScript timer trigger function ran!', timeStamp);
printed as expected but context.log("SUCCESS", JSON.parse(data)); not printed at all.
Also tryed different libraries (request, axios) with no success
module.exports = async function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const http = require('http');
http.get('http://myAPIurl', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
I rewrote your code using Axios. It has support for async/await out of the box, and simplifies a lot of the code to function as your would expect since it makes asynchronous code perform like synchronous code.
I think the main issue you may have been running into it that everything in JavaScript is async. As a result, the Azure Function runtime was exiting before your async function finished since it didn't block like synchronous code would when making the HTTP request. If you want to use callbacks, you need to call context.done() inside your callback function so the Azure Function doesn't exit before the callback is completed. By using async/await, you can guarantee that your code will block on the HTTP request until it receives a response or timeout. In the example below, Axios will return a response object that includes data as an element. I'm extracting data with a deconstruction operation, which allows me to log data.
const axios = require('axios');
const url = 'https://google.com';
module.exports = async function (context, myTimer) {
try {
const { data } = await axios.get(url);
context.log(data);
// do something with the data
return data;
} catch (err) {
context.log(err);
// do something with the error
}
context.done();
}
To Install a Package on an Azure Function
Go to the Azure Portal; select your Azure Function App
Select Platform Features. Then Advanced Tools (Kudu) under Development Tools
Using the file explorer, navigate to site/wwwroot/. You should see a package.json file in there, and several folders each with the name of your functions.
From that directory, run npm install --save axios
To confirm it worked, edit your package.json with the pencil icon. axios should be listed under dependencies json element
I've kept your code callback based.
I removed the async moniker from the definition and added a call to context.done (this signals the functions host when your function has ended) in your resp.end handler
module.exports = function (context, myTimer) {
var timeStamp = new Date().toISOString();
if (myTimer.IsPastDue)
{
context.log('JavaScript is running late!');
}
context.log('JavaScript timer trigger function ran!', timeStamp);
const https = require('https');
https.get('https://raw.githubusercontent.com/LearnWebCode/json example/master/animals-1.json', (resp) => {
let data = '';
// A chunk of data has been recieved.
resp.on('data', (chunk) => {
data += chunk;
});
// The whole response has been received. Print out the result.
resp.on('end', () => {
context.log("SUCCESS", JSON.parse(data));
context.done(null, data);
});
}).on("error", (err) => {
context.log("ERROR: " + err.message);
});
}
Another option would be to keep the function as async but you'd need to replace the callbacks with promise based calls. In some scenarios this can be achieved by wrapping them using util.promisify and then calling them with the await keyword
I know there are several posts about this, but according to those I've found, this should work correctly.
I want to make an http request in a loop and I do not want the loop to iterate until the request callback has been fired. I'm using the async library like so:
const async = require("async");
const request = require("request");
let data = [
"Larry",
"Curly",
"Moe"
];
async.forEachOf(data, (result, idx, callback) => {
console.log("Loop iterated", idx);
let fullUri = "https://jsonplaceholder.typicode.com/posts";
request({
url: fullUri
},
(err, res, body) => {
console.log("Request callback fired...");
if (err || res.statusCode !== 200) return callback(err);
console.log(result);
callback();
});
});
What I see is:
Loop iterated 0
Loop iterated 1
Loop iterated 2
Request callback fired...
Curly
Request callback fired...
Larry
Request callback fired...
Moe
What I need to see is:
Loop iterated 0
Request callback fired...
Curly
Loop iterated 1
Request callback fired...
Larry
Loop iterated 2
Request callback fired...
Moe
Also, if there's a built-in way to do the same thing (async/await? Promise?) and the async library could be removed, that'd be even better.
I've seen some examples of recursion out there that are clever, but when I put this to use in a much more complex situation (e.g. multiple request calls per-loop, etc.) I feel like that approach is hard to follow, and isn't as readable.
You can ditch async altogether and go for async/await quite easily.
Promisify your request and use async/await
Just turn request into a Promise so you can await on it.
Better yet just use request-promise-native that already wraps request using native Promises.
Serial example
From then on it's a slam dunk with async/await:
const rp = require('request-promise-native')
const users = [1, 2, 3, 4]
const results = []
for (const idUser of users) {
const result = await rp('http://foo.com/users/' + idUser)
results.push(result)
}
Parallel example
Now, the problem with the above solution is that it's slow - the requests run serially. That's not ideal most of the time.
If you don't need the result of the previous request for the next request, just go ahead and do a Promise.all to fire parallel requests.
const users = [1, 2, 3, 4]
const pendingPromises = []
for (const idUser of users) {
// Here we won't `await` on *each and every* request.
// We'll just prepare it and push it into an Array
pendingPromises.push(rp('http://foo.com/users/' + idUser))
}
// Then we `await` on a a `Promise.all` of those requests
// which will fire all the prepared promises *simultaneously*,
// and resolve when all have been completed
const results = await Promise.all(pendingPromises)
Error handling
Error handling in async/await is provided by plain-old try..catch blocks, which I've omitted for brevity.
If you have many (thousands) of urls to process it's best to define a batch size and recursively call the process function to process one batch.
It's also best to limit amount of active connections, you can use this to throttle active connections or connections within a certain time (only 5 per second).
Last but not least; if you use Promise.all you want to make sure not all successes are lost when one promise rejects. You can catch rejected requests and return a Fail type object so it'll then resolve with this Fail type.
The code would look something like this:
const async = require("async");
//lib comes from: https://github.com/amsterdamharu/lib/blob/master/src/index.js
const lib = require("lib");
const request = require("request");
const Fail = function(reason){this.reason=reason;};
const isFail = o=>(o&&o.constructor)===Fail;
const requestAsPromise = fullUri =>
new Promise(
(resolve,reject)=>
request({
url: fullUri
},
(err, res, body) => {
console.log("Request callback fired...");
if (err || res.statusCode !== 200) reject(err);
console.log("Success:",fullUri);
resolve([res,body]);
})
)
const process =
handleBatchResult =>
batchSize =>
maxFunction =>
urls =>
Promise.all(
urls.slice(0,batchSize)
.map(
url=>
maxFunction(requestAsPromise)(url)
.catch(err=>new Fail([err,url]))//catch reject and resolve with fail object
)
)
.then(handleBatch)
.catch(panic=>console.error(panic))
.then(//recursively call itself with next batch
_=>
process(handleBatchResult)(batchSize)(maxFunction)(urls.slice(batchSize))
);
const handleBatch = results =>{//this will handle results of a batch
//maybe write successes to file but certainly write failed
// you can retry later
const successes = results.filter(result=>!isFail(result));
//failed are the requests that failed
const failed = results.filter(isFail);
//To get the failed urls you can do
const failedUrls = failed.map(([error,url])=>url);
};
const per_batch_1000_max_10_active =
process (handleBatch) (1000) (lib.throttle(10));
//start the process
per_batch_1000_max_10_active(largeArrayOfUrls)
.then(
result=>console.log("Process done")
,err=>console.error("This should not happen:".err)
);
In your handleBatchResult you can store failed requests to a file to try later const [error,uri] = failedResultItem; you should give up if a high amount of requests are failing.
After handleBatchResult there is a .catch, that is your panic mode, it should not fail there so I'd advice to pipe errors to a file (linux).
Assume we have such a program:
// imagine the string1 to string1000 are very long strings, which will take a while to be written to file system
var arr = ["string1",...,"string1000"];
for (let i = 1; i < 1000; i++) {
fs.write("./same/path/file.txt", arr[i], {flag: "a"}});
}
My question is, will string1 to string1000 be gurantted to append to the same file in order?
Since fs.write is async function, I am not sure how each call to fs.write() is really executed. I assume the call to the function for each string should be put somewhere in another thread (like a callstack?) and once the previous call is done the next call can be executed.
I'm not really sure if my understanding is accurate.
Edit 1
As in comments and answers, I see fs.write is not safe for multiple write to same file without waiting for callback. But what about writestream?
If I use the following code, would it guarantee the order of writing?
// imagine the string1 to string1000 are very long strings, which will take a while to be written to file system
var arr = ["string1",...,"string1000"];
var fileStream = fs.createWriteFileStream("./same/path/file.txt", { "flags": "a+" });
for (let i = 1; i < 1000; i++) {
fileStream.write(arr[i]);
}
fileStream.on("error", () => {// do something});
fileStream.on("finish", () => {// do something});
fileStream.end();
Any comments or corrections will be helpful! Thanks!
The docs say that
Note that it is unsafe to use fs.write multiple times on the same file without waiting for the callback. For this scenario, fs.createWriteStream is strongly recommended.
Using a stream works because streams inherently guarantee that the order of strings being written to them is the same order that is read out of them.
var stream = fs.createWriteStream("./same/path/file.txt");
stream.on('error', console.error);
arr.forEach((str) => {
stream.write(str + '\n');
});
stream.end();
Another way to still use fs.write but also make sure things happen in order is to use promises to maintain the sequential logic.
function writeToFilePromise(str) {
return new Promise((resolve, reject) => {
fs.write("./same/path/file.txt", str, {flag: "a"}}, (err) => {
if (err) return reject(err);
resolve();
});
});
}
// for every string,
// write it to the file,
// then write the next one once that one is finished and so on
arr.reduce((chain, str) => {
return chain
.then(() => writeToFilePromise(str));
}, Promise.resolve());
You can synchronize the access to the file using the read/write locking for node, please see the following example an you could read the documentation
var ReadWriteLock = require('rwlock');
var lock = new ReadWriteLock();
lock.writeLock(function (release) {
fs.appendFile(fileName, addToFile, function(err, data) {
if(err)
console.log("write error"); //logging error message
else
console.log("write ok");
release(); // unlock
});
});
I had the same problem and wrote an NPM package to solve it for my project. It works by buffering the data in an array, and waiting until the event loop turns over, to concatenate and write the data in a single call to fs.appendFile:
const SeqAppend = require('seqappend');
const writeLog = SeqAppend('log1.txt');
writeLog('Several...');
writeLog('...logged...');
writeLog('.......events');
i am taking my first steps with node.js and i came across this issue with passing variable in asynchronous way. I have this piece of code im using to create Facebook user:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
req.tmpPassport.me = me;
console.log(req.tmpPassport.me) // works
});
}
console.log(req.tmpPassport.me) // not working -> undefined var
i tried to figure out why the second log isn't working and i ended up reading about synchronous and asynchronous functions, so in attempt to implement what i read i tried coming up with a solution using callbacks, but no success.
my last attempt was this:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
req.tmpPassport.me = me;
});
fb.my.events(function (err, events) {
//console.log(events);
req.tmpPassport.events = events;
});
fb.my.friends(function (err, result) {
req.tmpPassport.results = result;
});
}
function passUser(){
console.log(req.tmpPassport);
return req.tmpPassport;
}
cp.exec(initUser, passUser);
but its not working...
what i am actually trying to achieve its to render this object with my express router var which looks like this:
router.get('/welcome', securePages, function(req, res, next){
res.render('welcome', {title:'Welcome to aDating', user:req.tmpPassport});
})
but i cant figure out how to pass this object only after created...any help please?
A method of chaining function calls when certain async tasks are done is one way to deal with this.
Looking at the first snippet of code, it would be rewritten as follows:
req.tmpPassport = {};
var fb = new fbgraph.Facebook(accessToken, 'v2.2');
function initUser() {
fb.me(function (err, me) {
console.log(req.tmpPassport.me) // works
req.tmpPassport.me = me;
// triggers execution of the next step
post_populating_passport();
});
}
function post_populating_passport() {
// this function is only executed after the callback from the async call
console.log(req.tmpPassport.me);
}
Node.js is an asynchronous programming language, a fundamental concept in its core. You either need to use function callbacks (not a good practice) or available npms utilities for async flow.
I'm using the Node.JS driver for MongoDB, and I'd like to perform a synchronous query, like such:
function getAThing()
{
var db = new mongo.Db("mydatabase", server, {});
db.open(function(err, db)
{
db.authenticate("myuser", "mypassword", function(err, success)
{
if (success)
{
db.collection("Things", function(err, collection)
{
collection.findOne({ name : "bob"}, function(err, thing)
{
return thing;
});
});
}
});
});
}
The problem is, db.open is an asychronous call (it doesn't block), so the getAThing returns "undefined" and I want it to return the results of the query. I'm sure I could some sort of blocking mechanism, but I'd like to know the right way to do something like this.
ES 6 (Node 8+)
You can utilize async/await
await operator pauses the execution of asynchronous function until the Promise is resolved and returns the value.
This way your code will work in synchronous way:
const query = MySchema.findOne({ name: /tester/gi });
const userData = await query.exec();
console.log(userData)
Older Solution - June 2013 ;)
Now the Mongo Sync is available, this is the right way to make a synchronous MongoDB query in Node.js.
I am using this for the same. You can just write sync method like below:
var Server = require("mongo-sync").Server;
var server = new Server('127.0.0.1');
var result = server.db("testdb").getCollection("testCollection").find().toArray();
console.log(result);
Note: Its dependent on the node-fiber and some issues are there with it on windows 8.
Happy coding :)
There's no way to make this synchronous w/o some sort of terrible hack. The right way is to have getAThing accept a callback function as a parameter and then call that function once thing is available.
function getAThing(callback)
{
var db = new mongo.Db("mydatabase", server, {});
db.open(function(err, db)
{
db.authenticate("myuser", "mypassword", function(err, success)
{
if (success)
{
db.collection("Things", function(err, collection)
{
collection.findOne({ name : "bob"}, function(err, thing)
{
db.close();
callback(err, thing);
});
});
}
});
});
}
Node 7.6+ Update
async/await now provides a way of coding in a synchronous style when using asynchronous APIs that return promises (like the native MongoDB driver does).
Using this approach, the above method can be written as:
async function getAThing() {
let db = await mongodb.MongoClient.connect('mongodb://server/mydatabase');
if (await db.authenticate("myuser", "mypassword")) {
let thing = await db.collection("Things").findOne({ name: "bob" });
await db.close();
return thing;
}
}
Which you can then call from another async function as let thing = await getAThing();.
However, it's worth noting that MongoClient provides a connection pool, so you shouldn't be opening and closing it within this method. Instead, call MongoClient.connect during your app startup and then simplify your method to:
async function getAThing() {
return db.collection("Things").findOne({ name: "bob" });
}
Note that we don't call await within the method, instead directly returning the promise that's returned by findOne.
While it's not strictly synchronous, a pattern I've repeatedly adopted and found very useful is to use co and promisify yield on asynchronous functions. For mongo, you could rewrite the above:
var query = co( function* () {
var db = new mongo.Db("mydatabase", server, {});
db = promisify.object( db );
db = yield db.open();
yield db.authenticate("myuser", "mypassword");
var collection = yield db.collection("Things");
return yield collection.findOne( { name : "bob"} );
});
query.then( result => {
} ).catch( err => {
} );
This means:
You can write "synchronous"-like code with any asynchronous library
Errors are thrown from the callbacks, meaning you don't need the success check
You can pass the result as a promise to any other piece of code