I am trying to parse data from a .csv file, and save it to an array for later use.
I understand the concept of promises, but I have no idea what am I missing in my code that I cannot resolve the Promise and get the value (the string in the .csv file). It while I can view all the data inside the promise (.on('data')) from debugging mode, I just can't save it in order to use it later in my 'try&catch'.
const fs = require("fs");
const csv = require("csv-parser");
const { resolve } = require("path");
async function readCSV(filepath) {
return new Promise(async (resolve, reject) => {
await fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const results = [];
const csvFilePath =
"/languages.csv";
try {
const languages = readCSV(csvFilePath).then((res) => {
return res;
});
console.log(languages);
} catch (e) {
console.log(e);
}
and the output on the console is:
>Promise {<pending>}
No debugger available, can not send 'variables'
** That's from the debugging mode when I pause inside the promise:
https://i.stack.imgur.com/H9nHi.png
You can't try catch a returned promise without the await keyword in an async function.
If you're returning a promise, you need to use the .catch method on the promise.
Also, when you're logging languages you're doing so before the promise resolves because you're not using the await keyword.
I'm sure the promise resolves. Instead, log res inside the .then method.
const fs = require("fs");
const csv = require("csv-parser");
const results = [];
function readCSV(filepath) {
return new Promise((resolve, reject) => {
fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const csvFilePath = "./languages.csv";
(async () => {
const output = await readCSV(csvFilePath);
console.log(output)
})();
Related
I'm working on a microcontroller that would either take docx files or html strings in input and would transform it into a singular pdf file and return its link as an ouput.
My code looks like this so far:
// 'files' is an array of uploaded docx files.
const uploaded = files.map((file) => {
return new Promise((resolve, reject) => {
pump(
file.toBuffer(),
fs.createWriteStream(join(__dirname, 'files', file.filename))
.on('finish', resolve)
)
})
})
Promise.all(uploaded)
// Is triggered
.then(async () => await convertFiles())
// Is not triggered
.then(async () => {
// concatStoreFiles() is an external function because I need it somewhere else too
test = await concatStoreFiles(join(__dirname, 'files'))
console.log({test})
res.send(test)
})
const convertFiles = () => {
return new Promise((resolve, reject) => {
const cmd = `soffice --headless --convert-to pdf --outdir ${join(__dirname, 'files')} ${join(__dirname, 'files', '*.*')}`
exec(cmd, (error, stdout, stderror) => {
if (error) console.warn(error)
resolve(stdout ?? stderror)
})
})
}
concatStoreFile.js
module.exports = async function concatFiles (dirPath, outPath) {
return new Promise ((resolve, reject) => {
const existingFiles = []
fs.readdir(dirPath, (e, files) => {
files.forEach((file) => {
// is added to the files list only if finishing with ".pdf"
if (/[\d\w_-]+.pdf/.matches(file)) {
existingFiles.push(file)
}
});
resolve(existingFiles)
})
})
}
I'm working with Insomnia for my development / test process, and it tells me that I get an empty response. However, I'm supposed to get an array of pdf files existing in a specific directory. I'm not even getting console.log({test}), so I don't think my second then() is triggered.
I'm really rusty with async / await and Promise syntaxes, what should I do in this situation?
Thank you in advance
The #fastify/multipart's toBuffer() API returns a Promise, not a buffer. Checkout this article
So you need to write something like:
const uploaded = files.map(processFile)
async function processFile (file) {
const buffer = await file.toBuffer()
const storedFileName = join(__dirname, 'files', file.filename)
const writeStream = fs.createWriteStream(storedFileName)
return new Promise((resolve, reject) => {
pump(buffer, writeStream, (err) => {
if(err) { return reject(err) }
resolve(storedFileName)
})
}
}
Moreover, to improve the code, I returned the storedFileName instead of recalculating it.
You can convert this:
.then(async () => await convertFiles())
to this:
.then(() => convertFiles())
Mixing async/await and promise then/catch leads to hidden bugs hard to find
I am new to jest and trying to get some understanding of how mocks work. I am not able to figure out how I resolve an actual promise from a mocked module function.
I have a simple data interface module as shown below
const fs = require('fs');
const path = require('path');
function loadData(fileName){
let promise = new Promise((resolve, reject) => {
fs.readFile(path.join(__dirname, `/${fileName}`), (err, data) => {
if (err) {
console.error(err)
reject(err);
}
data = data.toString();
try{
data = JSON.parse(data);
}
catch(err){
console.error(err)
reject(new Error('Error while parsing json'))
}
resolve(data);
})
})
return promise;
}
The test file for the module looks like
jest.mock('fs');
const fs = require('fs');
const DI = require('./../data/data_interface');
describe('data loading functionality',() => {
test('load data opens file system',async () => {
try {
expect.assertions(1);
let fileName = 'expenses.json';
await DI.loadData(fileName);
expect(fs.readFile.mock.calls.length).toBe(1);
}
catch(err){
throw(err);
}
})
})
When I run the above test I am getting a timeout error which seems because the mock is not able to resolve the promise.
How do I resolve a promise from the provided mock?
You should remove the async await in your test
I am attempting to load some CSV data in my API such that I can manipulate it and pass through to my front end, however I am having a few issues returning the data.
I am using fast-csv to do the parsing here.
service.js
const fs = require('fs');
const csv = require('fast-csv');
module.exports.getFileContents = (filepath) => {
let data = [];
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => console.error(error))
.on('data', row => data.push(row))
.on('end', () => {
console.log(data) // This will print the full CSV file fine
return data;
});
};
routes.js
router.get('/data/:filename', (req, res) => {
const file = FS.getFileContents(testUrl + '/' + req.params.filename + '.csv');
console.log(file); // This prints 'undefined'
res.send(file);
});
I can print out the CSV contents fine from the service, but I just get 'undefined' from the actual routes. Can somebody please point out what I'm missing?
This is a common problem with JavaScript code, in the following.
.on('end', () => {
console.log(data);
return data;
});
Your on-end handler is an anonymous callback function (because of () =>), so when you return data, you are returning data out of your on-end handler callback function. You are not returning data out of your enclosing getFileContents() function.
Here's a typical way to write this kind of code:
const getFileContents = async (filepath) => {
const data = [];
return new Promise(function(resolve, reject) {
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => reject(error))
.on('data', row => data.push(row))
.on('end', () => {
console.log(data);
resolve(data);
});
});
}
And then, call it as follows, though this must be within an async function:
const data = await getFileContents('games.csv');
What's happened here is as follows:
your getFileContents is now async and returns a promise
the CSV data will be available when resolve(data) is executed
the caller can await the fulfillment/resolution of this promise to get the data
You could just create a Promise in the service and return it. Once the job is done, resolve it. The returned Promise will wait until it is resolved.
service.js
const fs = require('fs');
const csv = require('fast-csv');
module.exports.getFileContents = (filepath) => {
let data = [];
return new Promise((resolve) => {
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => console.error(error))
.on('data', row => data.push(row))
.on('end', () => {
resolve(data);
});
}
};
routes.js
router.get('/data/:filename', (req, res) => {
const file = await FS.getFileContents(testUrl + '/' + req.params.filename + '.csv');
console.log(file); // This prints only after it is resolved
res.send(file);
});
I am using a package called Okrabyte to extract words from each image file in a folder. The result should be a new array containing the extracted text that I can use in other functions.
When I run this:
var fs = require("fs");
var okrabyte = require("okrabyte");
fs.readdir("imgs/", function(err, files){
files.map((file)=>{
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
console.log(word);
})
})
})
the console logs each word. To return an array with those words I've tried the following:
async function words() {
await fs.readdir("imgs/", function (err, files) {
return files.map(async (file) => {
await okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), async (err, data) => {
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
return word
})
})
})
}
var testing = await words();
console.log(testing);
This gives undefined I've tried turning everything into a promise, I've tried async-await, I've tried pushing each word into a new array and returning that array in closure but nothing works - what am I doing wrong??
If your map function is async then it's returning a promise, so your mapped array is in fact an array of promises. But you can then use a Promise.all to get the resolved values of that array.
Additionally, you're trying to await the call to fs.readdir and okrabyte.decodeBuffer, which both accept a callback and do not return a promise. So if you want to use a promise there you'll have to wrap them in a promise constructor manually.
Here's how I would do it:
async function words() {
// Wrap `fs` call into a promise, so we can await it:
const files = await new Promise((resolve, reject) => {
fs.readdir("imgs/", (err, files) => { err ? reject(err) : resolve(files); });
});
// Since map function returns a promise, we wrap into a Promise.all:
const mapped = await Promise.all(files.map((file) => {
// Wrap okrabyte.decodeBuffer into promise, and return it:
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/" + file), (err, data) => {
if (err) return reject(err);
const splitWords = data.split(" ");
const word = splitWords[0].substr(1);
resolve(word);
})
})
}))
// Mapped is now an array containing each "word".
return mapped;
}
var testing = await words();
// Should now log your array of words correctly.
console.log(testing);
You should not be using async-await that way. That should be used when you are dealing with promises. The library okrabyte uses the concept of callbacks.
I suggest you follow this approach:
(1) Enclose the okrabyte.decodeBuffer part in a function that returns a promise that resolves in the callback.
(2) Use files.map to generate an array of promises calling the function you defined in (1)
(3) Use Promise.all to wait for all promises to execute and finish before moving on to dealing with all the words.
Walkthrough:
Part 1
const processWord = (file) => {
return new Promise((resolve, reject) => {
okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), (err, data)=>{
if (err) {
reject(err); // <--- reject the promise if there was an error
return;
}
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
resolve(word); // <--- resolve the promise with the word
})
});
}
You make a function that wraps the decoding part into a promise that eventually resolves with the word (or is rejected with an error).
Part 2
const promises = files.map((file)=>{
return processWord(file);
})
The above will generate an array of promises.
Part 3
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
// responses holds a list of words
// You will get each word accessing responses[0], responses[1], responses[2], ...
console.log(responses);
})
.catch(error => {
console.log(error); // Deal with the error in some way
});
})
The above uses Promise.all to wait for all promises to resolve before going to the then() block, assuming no errors occurred.
You can further isolate the construct above in a method that will return a promise with a list of all the words, much in the same fashion that was done in the processWord function from Part 1. That way, you can finally use async-await if you wish, instead of handling things in the then() block:
const processEverything = () => {
return new Promise((resolve, reject) => {
fs.readdir("imgs/", function(err, files){
const promises = files.map((file)=>{
return processWord(file);
})
Promise.all(promises)
.then(responses => {
resolve(responses);
})
.catch(error => {
reject(error);
});
})
});
};
const words = await processEverything();
console.log(words);
You're returning word value to the enclosed function but not map() function.
Hope this code help you.
async function words() {
global.words = [];
await fs.readdir("imgs/", function(err, files){
return files.map( async(file)=>{
await okrabyte.decodeBuffer(fs.readFileSync("imgs/"+ file), async (err, data)=>{
let splitWords = data.split(" ");
let word = splitWords[0].substr(1);
global.words.push(word);
})
})
})
}
var testing = await words();
testing = global.words;
console.log(testing);
let arr = [];
function getData(fileName, type) {
return fs.readFile(fileName,'utf8', (err, data) => {
if (err) throw err;
return new Promise(function(resolve, reject) {
for (let i = 0; i < data.length; i++) {
arr.push(data[i]);
}
resolve();
});
});
}
getData('./file.txt', 'sample').then((data) => {
console.log(data);
});
When I use above code and run it in command line using nodejs I get following error.
getData('./file.txt', 'sample').then((data) => {
^
TypeError: Cannot read property 'then' of undefined
How can I solve this?
You'll want to wrap the entire fs.readFile invocation inside a new Promise, and then reject or resolve the promise depending on the callback result:
function getData(fileName, type) {
return new Promise(function(resolve, reject){
fs.readFile(fileName, type, (err, data) => {
err ? reject(err) : resolve(data);
});
});
}
[UPDATE] As of Node.js v10, you can optionally use the built-in Promise implementations of the fs module by using fs.promises.<API>. In the case of our readFile example, we would update our solution to use fs.promises like this:
function getData(fileName, type) {
return fs.promises.readFile(fileName, {encoding: type});
}
Nobody told about util.promisify so I'm going to post, however old the question is.
Why are you having this message?
getData('./file.txt', 'sample').then((data) => {
^
TypeError: Cannot read property 'then' of undefined
getData is a wrapper for fs.readFile file here. fs.readfile is not a thenable (it does not implement a then function). It is built on the other pattern, the callback pattern. The most well-known thenable are Promises, and that's what you want to get from readFile I believe. A little reminder: Mozilla - Promises
So what you can do is either implement it yourself as did #hackerrdave or I would suggest using promisify: this function is a built-in function of Node.js which was implemented to transform the callback-based function into promised based. You will find it here: Node.js Documentation for util.promisfy
It basically does the same as #hackerrdave but it's more robust and built-in node util.
Here's how to use it:
const util = require('util');
const fs = require('fs');
const readFile = util.promisify(fs.readFile)
readFile("path/to/myfile").then(file => console.log(file))
Here is a one-liner as of node 10.2.0:
(async () => console.log(String(await require('fs').promises.readFile('./file.txt'))))();
Yes, it is now out of the box.
As of Node 12+, you can use the fs.promises API.
See an example below:
const { readFile } = require('fs').promises
readFile('./file.txt', { encoding: 'utf8' })
.then((data) => console.log(data))
.catch((error) => console.error(error));
Using async/await
const { readFile } = require('fs').promises
async function readFile(filePath) {
try {
const data = await readFile(filePath, { encoding: 'utf8' })
console.log(data)
} catch (error) {
console.error(error.message)
}
}
readFile('./file.txt')
const getData = (fileName, type) =>
new Promise((resolve, reject) =>
fs.readFile(fileName, type, (err, data) => {
//if has error reject, otherwise resolve
return err ? reject(err) : resolve(data);
})
);
getData('./file.txt', 'utf8')
.then(data => console.log('Data: ', data))
.catch(error => console.log('Error: ', error));
Update for current node As of node 10.0.0 you can now use fs.promises:
const fs = require('fs')
(async function(){
var fileContents = await fs.promises.readFile(FILENAME)
var data = JSON.parse(fileContents)
})()