I have a large number of images that I want to download. I'm using the request-promise package to download the images from their URL(s). Since there were a large number of images, the server was getting overloaded and the downloads would hang or get corrupted, so I decided to use the promise-limit library to set the concurrency limit. Here is my code :
const fs = require('fs');
const request = require('request-promise');
const promiseLimit = require('promise-limit');
var limit = promiseLimit(30);
async function download(data) {
console.log(data);
return Promise.all(data.map( (obj) => {
return limit(() => downloadRequest(obj))
})).then(() => {
console.log("All images downloaded.")
})
function downloadRequest(obj) {
var img = obj.dest;
var url = obj.url;
var req = request(url);
req.pipe(fs.createWriteStream(img));
}
I replicated the sample exactly as is given in the github page, but the method returns without ever having the Promise.all() fulfilled. I do not understand what I am doing wrong, I thought that Promise.all() will definitely wait till it resolves all promises.
In the backend, this call is used as
...
.then((data) => {
return downloader.download(data);
})
.then(() => {
var filename = "Sample.pdf";
// Pages.json is written early in the chain, contains URL and corresponding image paths
return generator.generate('./Pages.json', filename);
});
Does this mean NodeJS is already trying to generate the file out of pages.json? How can I make this part of the code synchronous to download?
Your function downloadRequest() does not return a promise. It must return a promise that is tied to the asynchronous operation in contains such that the promise is resolved when that asynchronous operation is complete or rejected when that asynchronous operation has an error. Only when it does that can the limit() package properly do its job.
Since you're using a stream and piping it in downloadRequest(), you will have to manually construct a promise and then monitor the various events in the stream to know when it's done or has an error so you can resolve or reject that promise.
Here's an idea how to make downloadRequest() properly return a promise:
function downloadRequest(obj) {
return new Promise((resolve, reject) => {
const img = obj.dest;
const url = obj.url;
const req = request(url);
req.on('error', reject);
const ws = fs.createWriteStream(img);
ws.on('error', reject);
req.pipe(ws).on('finish', resolve);
});
}
And, it is now recommended to use the pipeline() function instead of .pipe() because it does more complete cleanup in error conditions and there is also a promise version of that built-in:
const { pipeline } = require('stream/promises');
function downloadRequest(obj) {
return pipeline(request(obj.url), fs.createWriteStream(obj.dest));
}
P.S. In case you didn't know, the request() library has been deprecated and it is recommended that you not use it in new projects any more. There is a list of alternative libraries to choose from here, all of which also have built-in promise support. I've looked at the various choices, tried several and decided I'm using got() in my work.
Related
I'm trying to return an SVG file located in the same folder as the Javascript file when a function is executed.
The function is called returnSVG, the SVG is called image.
import image from './image.svg'
returnSVG= () => {
return image;
}
But when I call it I get this: /static/media/image.f85cba53.svg
An SVG file contains a textual representation of a graphic, encoded in an XML-like syntax.
So, you cannot simply import thing from 'filepath' because the file contents are not a JavaScript module, a loadable binary extension, or a JSON-encoded string.
Rather than importing what cannot be imported, the correct approach would be to explicitly read the contents from the file:
const { readFileSync } = require('fs')
const returnSvg = (path = './image.svg') => return readFileSync(path)
Note that this example uses the synchronous version of the fs function to read a file. This will pause (block) all processing in NodeJS until the contents have been read.
The better solution would be to use Promises via async/await as in:
const { readFile } = require('fs')
const { promisify } = require('util')
const asyncReadFile = promisify(readFile)
const returnSvg = async (path = './image.svg') => {
const data = await asyncReadFile(path)
// since fs.readFile returns a buffer, we should probably convert it to a string.
return data.toString()
}
Using Promises and async/await requires that all of your processing occur within the chain of promises, so you'd need to restructure your code to:
returnSvg()
.then(data => { /* do something with the file contents here. */ })
.catch(err => console.error(`failed to read svg file: ${err}`))
Remember, in NodeJS, operations can be either asynchronous (via callbacks or Promises which async/await uses "under the covers"), or synchronous (via special "blocking" calls).
One option, if you choose to use the synchronous version would be to load all your files into your server before you call app.listen(8080, () => console.log('listing on 8080') but this implies you know what files will be required before you run your server and would not work for dynamically loaded files.
You might imagine doing something like:
const data = readSvg()
console.log(data)
or perhaps:
let data
readSvg().then(res => data = res)
console.log(data)
but neither will work because an asynchronous function (one defined with the async keyword) can only return a Promise.
Both attempts will not print any usable value to the console because at the time console.log() is called, NodeJS has no idea what the value of data will be.
The rule of thumb here is:
You cannot return a value or interact with any higher context from within a Promise.
Any manipulation of the data generated within a Promise chain, can ONLY be accessed from within its own chain.
I am using node and axios (with TS, but that's not too important) to query an API. I have a suite of scripts that make calls to different endpoints and log the data (sometimes filtering it.) These scripts are used for debugging purposes. I am trying to make these scripts "better" by adding a delay between requests so that I don't "blow up" the API, especially when I have a large array I'm trying to pass. So basically I want it to make a GET request and pause for a certain amount of time before making the next request.
I have played with trying setTimeout() functions, but I'm only putting them in places where they add the delay after the requests have executed; everywhere I have inserted the function has had this result. I understand why I am getting this result, I just had to try everything I could to at least increase my understanding of how things are working.
I have though about trying to set up a queue or trying to use interceptors, but I think I might be "straying far" from a simpler solution with those ideas.
Additionally, I have another "base script" that I wrote on the fly (sorta the birth point for this batch of scripts) that I constructed with a for loop instead of the map() function and promise.all. I have played with trying to set the delay in that script as well, but I didn't get anywhere helpful.
var axios = require('axios');
var fs = require('fs');
const Ids = [arrayOfIds];
try {
// Promise All takes an array of promises
Promise.all(Ids.map(id => {
// Return each request as its individual promise
return axios
.get(URL + 'endPoint/' + id, config)
}))
.then((vals) =>{
// Vals is the array of data from the resolved promise all
fs.appendFileSync(`${__dirname}/*responseOutput.txt`,
vals.map((v) => {
return `${JSON.stringify(v.data)} \n \r`
}).toString())
}).catch((e) => console.log)
} catch (err) {
console.log(err);
}
No errors with the above code; just can't figure out how to put the delay in correctly.
You could try Promise.map from bluebird
It has the option of setting concurrency
var axios = require('axios');
var fs = require('fs');
var Promise = require('bluebird');
const Ids = [arrayOfIds];
let concurrency = 3; // only maximum 3 HTTP request will run concurrently
try {
Promise.map(Ids, id => {
console.log(`starting request`, id);
return axios.get(URL + 'endPoint/' + id, config)
}, { concurrency })
.then(vals => {
console.log({vals});
})
;
} catch (err) {
console.log(err);
}
This question already has answers here:
How do I return the response from an asynchronous call?
(41 answers)
Closed 5 years ago.
I have question regarding asynchronous functions and how to send something after a function has returned it's result. This is what I am trying to accomplish:
Within the handling of a GET request in Node I read the contents of a folder, returning the files in that folder. Next I want to loop over the stats of each file in that folder, loading only the files created within a certain period and lastly send the data in those files as a response to the request. It looks something like this:
array = []
fs.readdir(path, function(err, items) {
items.forEach(function(item) {
fs.stat(path, function(err, stats) {
if (period check) {
array.push(data)
}
})
})
}
res.send(array)
This approach ends up sending an empty array, and I've looked into Promises which seem the solution here but I can't get them to work in this scenario. Using fs.statSync instead of fs.stat does work but this greatly reduces the performance, and it feels like it should be doable with Promises but I just don't know how.
Is there anyone with a solution for this?
EDIT: Regarding to the question marked as duplicate, I tried to solve my problem with the answer there first but didn't succeed. My problem has some nested functions and loops and is more complex than the examples given there.
Use this if you prefer a Promise-based approach:
var path = require('path')
fs.readdir(myPath, function(err, items) {
var array = [];
Promise.all(items.map(function(item) {
return new Promise(function(resolve, reject) {
fs.stat(path.resolve(myPath, item), function(err, stats) {
if (err) {
return reject(err)
}
if (/* period check */) {
array.push(data)
}
resolve()
})
})
})).then(function() {
res.send(array)
}).catch(function(error) {
// error handling
res.sendStatus(500)
})
}
Here is what I would suggest.
// This is a new API and you might need to use the util.promisify
// npm package if you are using old node versions.
const promisify = require('util').promisify;
const fs = require('fs');
// promisify transforms a callback-based API into a promise-based one.
const readdir = promisify(fs.readdir);
const stat = promisify(fs.stat);
const dataProm = readdir(path)
.then((items) => {
// Map each items to a promise on its stat.
const statProms = items.map(path => fs.stat(path);
// Wait for all these promises to resolve.
return Promise.all(statProms);
})
// Remove undesirable file stats based on the result
// of period check.
.then(stats => stats.filter(stat => periodCheck(stat)));
// dataProm will resolve with your data. You might as well return it
// as is. But if you need to use `res.send`, you can do:
dataProm.then((data) => {
res.send(data);
}, (err) => {
// If you go away from the promise chain, you need to handle
// errors now or you are silently swallowing them.
res.sendError(err);
});
Here is a link toward the util.promisify package I am referring to. If you are using node v8+, you do not need it. If you do, do not forget to replace require('util').promisify; by require('util.promisify');.
I am working on a simple TCP client for a server and am using the latest node 7.6 because of the async/await functions. I'm new to node and asynchronous coding, so I apologize if this is stupidly easy.
I want to run a function that calls the callServer() function with specific parameters, wait until it finishes getting the data, and return the data as a variable.
Here is my code:
'use strict'
const net = require('net')
var formattedJson = funcRequestToJson("Server.GetStatus", false)
doThings()
async function doThings() {
var info = await callServer()
}
async function callServer() {
var client = new net.Socket()
client.connect(1705, '192.168.28.16', () => {
console.log('connected to server')
client.write(formattedJson)
})
client.on('data', (data) => {
client.destroy()
//return await data
console.log(data.toString())
})
client.on('close', () => {
})
}
// method and paramBool are always required
// macAddress, paramKey, paramValue are required for if paramBool is true
function funcRequestToJson(method, paramBool, macAddress, paramKey, paramValue) {
var objectRequest = {}
objectRequest[ "jsonrpc" ] = "2.0"
objectRequest[ "method" ] = method
objectRequest[ "id" ] = 0
if (paramBool == true) {
objectRequest[ "params" ] = {
"client": macAddress,
[paramKey]: paramValue
}
}
var json = (JSON.stringify(objectRequest) + '\r\n')
return json
}
So I didn't declare objectRequest() as async because it's not waiting on the server, but I think callServer() should be async, right? I know this can be done with promises, but I wanted to use async/await and this seems to be right.
Now, I want to return the data that comes from inside callServer() and client.on('data', (data) but I can't seem to figure out how to do it asynchronously. I would think there'd be a way to make an async function and call it with await like I tried (await return data) but it never works right.
I'm sorry if this is terribly convoluted, but I've been poring over async node tutorials for the past week and am still stuck.
Thanks!
Async/Await relies on code that uses promises for async operations. So you need to return a promise from any async operation in order to use it with async/await.
So, callServer() needs to return a promise that is resolved when the async operation inside it is done. In fact, you can only await an async operation in a function if that function returns a promise. await saves you from having to write .then() handlers on promises, but async/await does not magically know when async operations are done. You still have to wrap them in promises.
Here's an example of how you could make callServer() return a promise:
async function callServer(formattedJson) {
return new Promise((resolve, reject) => {
let client = new net.Socket()
client.connect(1705, '192.168.28.16', () => {
console.log('connected to server')
client.write(formattedJson)
})
client.on('data', (data) => {
resolve(data);
client.destroy()
})
client.on('close', () => {
})
client.on('error', reject);
});
}
Sample Usage:
try {
var info = await callServer(funcRequestToJson("Server.GetStatus", false));
} catch(e) {
// error here
}
Just to show you what await is doing, in ES5 (without async/await), the sample usage would be this:
callServer(funcRequestToJson("Server.GetStatus", false)).then(info => {
// info is available here
}).catch(err => {
// error here
});
So, await is just letting you avoid writing the .then() handler. Internally in the execution of the code, there's just a promise that the interpreter sets up a .then() handler for so it will know when that promise is resolved. This is syntactical sugar, not really any magic with async operations. You still have to use promises for all your async operations. You can use await instead of .then() and your code will appear more sequential even though it's really the same under the covers.
People think that await allows one to write asynchronous code as if it was synchronous, but that isn't really the case, especially if you handle errors and understand concurrency issues. It will look more synchronous, but isn't actually any more synchronous than it was with .then() in ES5.
And, watch out for error handling. In people's quest to write synchronous looking code, people seem to completely forget to handle rejected promises in their async operations (which are handled with try/catch when using await). I'm personally not yet convinced that ES6 is a step forward when it comes to error handling as the early indications are that ES6 seems to encourage people to just forget about error handling or get lazy and not do it, whereas it's easier when using .then() to just know that there should be a .catch() somewhere for it to be solid code. Maybe that's just a learning process, but it seems to be an early issue when people use await.
I have a file where I'm writing things:
var stream = fs.createWriteStream("my_file.txt");
stream.once('open', function(fd) {
names.forEach(function(name){
doSomething(name);
});
stream.end();
});
This is working ok and I'm able to write to the file.
The problem is that the doSomething() function has some parts that are asynchronous. An example can be given with the dnsLookup function. Somewhere in my doSomething() I have:
dns.lookup(domain, (err, addresses, family) => {
if(err){
stream.write("Error:", err);
}else{
stream.write(addresses);
}
});
Now, my problem is, since the DNS check is asynchronous, the code keeps executing closing the stream. When the DNS response finally comes it cannot write to anywhere.
I already tried to use the async module but it didn't work. Probably I did something wrong.
Any idea?
Now that NodeJS is mostly up to speed with ES2015 features (and I notice you're using at least one arrow function), you can use the native promises in JavaScript (previously you could use a library):
var stream = fs.createWriteStream("my_file.txt");
stream.once('open', function(fd) {
Promise.all(names.map(name => doSomething(name)))
.then(() => {
// success handling
stream.end();
})
.catch(() => {
// error handling
stream.end();
});
});
(The line Promise.all(names.map(name => doSomething(name))) can be simply Promise.all(names.map(doSomething)) if you know doSomething ignores extra arguments and only uses the first.)
Promise.all (spec | MDN) accepts an iterable and returns a promise that is settled when all of the promises in the iterable are settled (non-promise values are treated as resolved promises using the value as the resolution).
Where doSomething becomes:
function doSomething(name) {
return new Promise((resolve, reject) => {
dns.lookup(domain, (err, addresses, family) => {
if(!err){ // <== You meant `if (err)` here, right?
stream.write("Error:", err);
reject(/*...reason...*/);
}else{
stream.write(addresses);
resolve(/*...possibly include addresses*/);
});
});
});
There are various libs that will "promise-ify" Node-style callbacks for you so using promises is less clunky than the mix above; in that case, you could use the promise from a promise-ified dns.lookup directly rather than creating your own extra one.