We Keep Coding

nodejs download image from url synchronously

I am trying to build a small node app it calls an api which returns an array of urls which point to image blob png files.
I am then trying to loop over the array and download the files using a utility function. I need this to work synchronously. Once the downloads are complete I then want to fire an additional function.
I started off using some asynchronous code which I took from here: https://sabe.io/blog/node-download-image
The async code in my utils file looked like this:
import { promises as fs } from "fs";
import fetch from "node-fetch";
const downloadFile = async (url, path, cb) => {
const response = await fetch(url);
const blob = await response.blob();
const arrayBuffer = await blob.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
fs.writeFile(path, buffer);
cb();
}
export { downloadFile };
I have tried to convert it to be purely synchronous using this code:
import fs from "fs";
import fetch from "node-fetch";
const downloadFile = (url, path, cb) => {
const response = fetch(url);
const blob = response.blob();
const arrayBuffer = await blob.arrayBuffer();
const buffer = Buffer.from(arrayBuffer);
fs.writeFileSync(path, buffer);
cb();
}
export { downloadFile };
Then in my index.js file I am using it like so:
import { downloadFile } from './utils/downloadFiles.js';
let imagesArray = [];
let newImageNames = [];
imagesArray.forEach((item, index) => {
const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
const filePath = path.join('src', 'images');
newImageNames.push(fileName);
downloadFile(item, filePath, fileDownloadCallback);
});
processDataCallback(); // This is the function which is being fired before the previous downloadFile functions have finished processing.
const fileDownloadCallback = () => {
console.log(`File download callback`);
}
My images array is being populated and looks like this as an example:
data: [
{
url: 'https://someurl.com/HrwNAzC8YW/A%3D'
},
{
url: 'https://someurl.com/rGL7UeTeWTfhAuLWPg%3D'
},
{
url: 'https://someurl.com/xSKR36gCdOI3/tofbQrR8YTlN6W89DI%3D'
},
{
url: 'https://someurl.com/2y9cgRWkH9Ff0%3D'
}
]
When I try and use the synchronous method I get this error TypeError: response.blob is not a function. This function does work when using it asynchronously, but then it is firing my next function before the image downloads have finished.
I have tried several iterations, first off using createWriteStream and createWriteStreamSync (which I believe are deprecated). So switched to fileWrite. I also tried using a synchronous fileWriteSync inside the async function, but still no dice. The other issue is that fetch works asynchronously, so I still don't know how to wire this up to only work synchronously. I was also wondering If I could chain a then onto the end of my fileDownload util function.
All of my code is in github, so I can share a url if required. Or please ask for more explanation if needed.
Is there something equivalent to jsfiddle for Node? If so I am more than happy to try and make a demo.
Any help greatly appreciated.

We can leave the original async downloadFile util alone (though there's a little room for improvement there).
In the index file...
import { downloadFile } from './utils/downloadFiles.js';
let imagesArray = [];
let newImageNames = [];
// I'm a little confused about how we get anything out of iterating an empty array
// but presuming it get's filled with URLs somehow...
const promises = imagesArray.map((item, index) => {
const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
const filePath = path.join('src', 'images');
newImageNames.push(fileName);
// we can use the callback for progress, but we care about finishing all
return downloadFile(item, filePath, () => {
console.log('just wrote', filePath);
});
});
Promise.all(promises).then(() => {
console.log('all done')
})

Array.forEach doesnt work with async code.
Convert your code into a for...of or for code and it will works.
Also. You don't use callbacks with async/await code.
Your code will look like this:
let index = 0;
for(const item of imageArray) => {
const fileName = `${prompt}__${index}_${uuid.v4()}.png`;
const filePath = path.join('src', 'images');
newImageNames.push(fileName);
downloadFile(item, filePath);
fileDownloadCallback();
index++;
});

How do I make a function wait for the completion of another function before running

The problem I'm facing is that I want to create a temporary work folder for a certain function to hold its assets and work on them.
So:
await function someFunc() {
createFolder()
...
makeSomeFiles()
doOtherStuff()
...
deleteFolder()
}
But the functions that I am using, in node.js, are all async. Creating a folder is fs.mkdir() and deleting a folder is fs.rmdir() and downloading images and saving them is also an async procedure of some kind.
The problem is such: the folder gets created, and deleted, before any of the code in the middle executes. So I get errors from the middle section code that the folder doesn't exist, because it gets deleted prematurely. How do i make fs.rmdir(), at the end, wait for all the middle code to run first, before deleting the folder.
The specific code is this:
async function run() {
//GENERATE SESSION ID AND FOLDER
const sessionID = str.random(50);
fs.mkdir('images/'+sessionID, (err) => {
if (err) return err;
});
//DOWNLOAD IMAGE
https.get('https://en.wikipedia.org/wiki/Main_Page#/media/File:RE_Kaja_Kallas.jpg', (file) => {
file.pipe(fs.createWriteStream('images/'+sessionID+'/image.jpeg'));
});
//CLEANUP
fs.rmdir('images/'+sessionID, { recursive: true }, (err) => {
if (err) return err;
});
}

I would use promise-based versions of functions that do these operations and then use async/await with those promises:
const stream = require('stream');
const {promisify} = require('util');
const fs = require('fs');
const fsp = fs.promises;
const got = require('got');
const pipeline = promisify(stream.pipeline);
async function run() {
const sessionID = str.random(50);
const dir = 'images/'+sessionID;
await fsp.mkdir(dir);
await pipeline(
got.stream('https://en.wikipedia.org/wiki/Main_Page#/media/File:RE_Kaja_Kallas.jpg'),
fs.createWriteStream(dir + '/image.jpeg')
);
// not sure why you're trying to remove a directory that you just
// put a file in so it's not empty
await fsp.rmdir(dir, { recursive: true })
}
run().then(() => {
console.log("all done");
}).catch(err => {
console.log(err);
});
But, this function isn't making a lot of sense to me because you're creating a directory, downloading a file to it and then trying to remove a non-empty directory.
This uses the library got() for downloading the file because it's my goto library for http requests since it has both stream and promise interfaces.

Accessing Parsed CSV data , CSV-Parser Node.js

Im using csv-parser npm package and doing a sample csv parse. My only confusion is accessing the parsed array after running these functions. I understand im pushing the data in .on('data') , then doing a console.log(results); statement in .on('end'); to show what's being stored. Why do I get undefined when i try to access results after running those functions. Doesn't results get the information stored?
const csv = require('csv-parser');
const fs = require('fs');
const results = [];
fs.createReadStream('demo.csv')
.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
console.log(results);
});

I came here to find the solution to the same issue.
Since this is an async operation, what works here is to call that function that acts on your parsed data once the end handler is called. Something like this should work in this situation:
const csv = require('csv-parser');
const fs = require('fs');
const results = [];
fs.createReadStream('demo.csv')
.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
console.log(results);
csvData(results);
});
const csvData = ((csvInfo) => {
console.log(csvInfo);
console.log(csvInfo.length);
})

I can get results in .on('end', () => { console.log(results);}); , but
if I put a console.log() after the createReadStream , results is
undefined, does that make sense? – Videoaddict101
Your stream acts asynchronously, that means your data and your end handler will be called later, meanwhile your javascript continue to be executed. So accessing your array just after fs.createReadStream instruction will result of an empty array.
Understanding async is very important using javascript, even more for nodejs.
Please have a look on differents resources for handling async like Promise, Async/Await ...

You should you neat-csv which is the endorsed wrapper for csv-parser that gives you a promise interface.
That said, you can create a promise and resolve it in the on("end", callback)
import fs from "fs";
import csv from "csv-parser";
function getCsv(filename) {
return new Promise((resolve, reject) => {
const data = [];
fs.createReadStream(filename)
.pipe(csv())
.on("error", (error) => reject(error))
.on("data", (row) => data.push(row))
.on("end", () => resolve(data));
});
}
console.log(await getCsv("../assets/logfile0.csv"));

Putting s3 object data into an array - node

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.
const dataFromS3 = async (bucket, file) => {
let lines = [];
const options = {
Bucket: bucket,
Key: file
};
s3.getObject(options, (err, data) => {
if (err) {
console.log(err);
} else {
let objectData = data.Body.toString('utf-8');
lines.push(objectData);
console.log(lines);
return lines;
}
});
};
Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:
app.get('/', async (req, res, next) => {
try {
let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
res.send(apolloKey);
} catch (err) {
console.log('Error: ', err);
}
});
It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...
Thank you

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.
const dataFromS3 = async (bucket, file) => {
const lines = [];
const options = {
"Bucket": bucket,
"Key": file
};
const data = await s3.getObject(options).promise();
const objectData = data.Body.toString("utf-8");
lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
console.log(lines);
return lines;
};

How to attach multiple files from the same directory with supertest package in Node?

I want to attach around 100 files into a POST request using supertest, but I haven't found any way to do this.
const supertest = require('supertest);
const fs = require('fs');
const { promisify } = require('utils');
const request = supertest('http://localhost:3000');
const readdir = promisify(fs.readdir);
(async () => {
try {
const files = await readdir(path.resolve(__dirname, '/path/to');
request
.post('/upload')
.attach('file', files)
.end((response => {
console.log(response);
});
} catch(err) => {
console.error(err);
}
)();
Given the following piece of code, the request needs to chain the attach method every time a fileN is wanted to be send in the request, so I want how to attach recursively every file inside the same folder using the fs node core module

supertest extends superagent with testing functionality and works the same way. request.post('/upload') creates request instance and is chainable. Instance methods return an instance itself for chaining:
const requestInstance = request.post('/upload');
requestInstance === requestInstance.attach('file', ...);
supertest and superagent are thenable, it's preferable to chain the result as a promise for correct control flow when it's used with async..await:
try {
const files = await readdir(path.resolve(__dirname, '/path/to');
let requestInstance = request.post('/upload');
for (const file of files) {
// no need to reassign requestInstance because it's same instance
// requestInstance = requestInstance.attach('file', file);
requestInstance.attach('file', file);
}
const response = await requestInstance;
} catch (err) {
console.error(err);
}

You might just want to make several individual requests with supertest and Promise.all. You can use globby to get a list of files eg. without async might even be more straightforward, unless you need the result somewhere else.
const files = glob(path.resolve(__dirname, '/path/to'));
const requests = files.map((file) => {
return request
.post('/upload')
.attach('file', file)
.end((response => {
console.log(response);
});
});
Promise.all(requests).then(console.log('done')).catch(console.error);