My code is nearly working well my problem is that only the first CSV data is logged and the lambda function ends.
I guess I need someway to wait for all stream pipes to end.
myCsvList.forEach((myElem) => {
const data = [];
// setup params
const csvFile = s3.getObject(params).createReadStream(); // works fine
csvFile
.pipe(csv())
.on('data', function(entry) {
data.push(entry);
})
.on('end', () => {
console.log(data); // after the log of the first element on myCsvList, the code finishes. It should log all csvFiles from myCsvList
});
});
I guess I need a promises or something?
you can change every item to Promise, then use Promise.all():
const getItem = (myElem) => {
const data = [];
// setup params
const csvFile = s3.getObject(params).createReadStream(); // works fine
return new Promise((resolve) => {
csvFile
.pipe(csv())
.on('data', function (entry) {
data.push(entry);
})
.on('end', () => {
resolve(data); // after the log of the first element on myCsvList, the code finishes. It should log all csvFiles from myCsvList
});
});
};
const start = () => {
// all promises have been finished
return Promise.all(myCsvList.map(myElem => getItem(myElem)));
}
myCsvList.map(myElem => getItem(myElem)) can get a promise list, when all promises have been resolved, it will trigger Promise.all().
Related
I am trying to parse data from a .csv file, and save it to an array for later use.
I understand the concept of promises, but I have no idea what am I missing in my code that I cannot resolve the Promise and get the value (the string in the .csv file). It while I can view all the data inside the promise (.on('data')) from debugging mode, I just can't save it in order to use it later in my 'try&catch'.
const fs = require("fs");
const csv = require("csv-parser");
const { resolve } = require("path");
async function readCSV(filepath) {
return new Promise(async (resolve, reject) => {
await fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const results = [];
const csvFilePath =
"/languages.csv";
try {
const languages = readCSV(csvFilePath).then((res) => {
return res;
});
console.log(languages);
} catch (e) {
console.log(e);
}
and the output on the console is:
>Promise {<pending>}
No debugger available, can not send 'variables'
** That's from the debugging mode when I pause inside the promise:
https://i.stack.imgur.com/H9nHi.png
You can't try catch a returned promise without the await keyword in an async function.
If you're returning a promise, you need to use the .catch method on the promise.
Also, when you're logging languages you're doing so before the promise resolves because you're not using the await keyword.
I'm sure the promise resolves. Instead, log res inside the .then method.
const fs = require("fs");
const csv = require("csv-parser");
const results = [];
function readCSV(filepath) {
return new Promise((resolve, reject) => {
fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const csvFilePath = "./languages.csv";
(async () => {
const output = await readCSV(csvFilePath);
console.log(output)
})();
I am confused because I cannot seem to extract a value from an asynchronous operation. Basically I have to parse a csv file and the operation is asynchronous as shown below.
const csv = require('csv-parser')
const fs = require('fs')
const results = [];
fs.createReadStream('courses.csv')
.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
console.log(results);
});
I am unable to completly extract and isolate the results variable from that stream. I have tried doing this by wrapping it with a promise but it shows pending. Here is what I am doing.
const getData = () => {
const prom = new Promise((res, rej) => {
fs.createReadStream('courses.csv')
.pipe(csv())
.on('data', (data) => results.push(data))
.on('end', () => {
res(results);
});
})
return prom;
}
async function GIVEMEMYVALUE() {
var result = await getData();
return result;
};
let data = GIVEMEMYVALUE();
console.log(data);
I have read other questions relating to promises but I still don't see what I am doing wrong. I can do whatever I want with the results variable inside the 'end' callback but cannot seem to extract it(for whatever reason I want to extract it.)
Is it wrong to want to extract that value outside the scope of the 'end' callback ?
Can everything I possibly want to do with the results be done inside the callback ?
I have already gone through How do I return the response from an asynchronous call? but don't quite get it as it doesn't mention anything about pending promises.
GIVEMEMYVALUE returns also an promise. However you could shorten your processes alot:
const getData = () =>
new Promise((res, rej) => {
fs.createReadStream("courses.csv")
.pipe(csv())
.on("data", data => results.push(data))
.on("end", () => {
res(results);
});
});
getData().then(data => {
console.log(data);
});
async/ await does not make your code work synchronous. As soon as you put async infront of your function, your function automatically returns an promise and acts like an promise.
From my understanding createReadStream is asynchronous and there is no equivalent synchronous method. This leads us to my problem I'm having today. As you can see in my code, I have two createReadStream, and the second one depends on the first one and it absolutely needs to be run in order so that stationIds gets filled with the desired values. My problem here is that stationIds is empty and so the second createReadStream doesn't do anything at all, how can I run these two createReadStream in the correct way I want?
// Read content of files
csvDataFiles.forEach(file => {
const data = []
fs.createReadStream(directory + file)
.pipe(csv())
.on('data', function (row) {
data.push(row)
})
.on('end', function() {
const object = {}
let stationId = parseInt(file.substr(0, file.indexOf('.')))
stationIds.push(stationId)
object[stationId] = {}
object[stationId]['length'] = data.length
object[stationId]['data'] = data
stationsData.push(object)
})
})
// Read stations
fs.createReadStream(directory + 'asfdsfsd.csv')
.pipe(csv({
skipLines: 3
}))
.on('data', function (row) {
if (stationIds.includes(parseInt(row['Station ID']))) {
// console.log(row)
stations.push(row)
}
})
.on('end', function () {
// console.log(stations)
})
You could wrap the processing of the streams in a promise and wait until all data is read. After that you can process the final file. This still needs some error handling, but should give you something to start with:
// Read content of files
async function processStationsCsv(directory, csvDataFiles) {
let stationsIds = [];
let stationsData = [];
for (const csvFile of csvDataFiles) {
await readStationDataFromFile(directory + csvFile, stationsIds, stationData);
}
let stations = [];
await new Promise(resolve => {
// Read stations
fs.createReadStream(directory + 'asfdsfsd.csv')
.pipe(csv({
skipLines: 3
}))
.on('data', row => {
if (stationsIds.includes(parseInt(row['Station ID']))) {
stations.push(row)
}
})
.on('end', () => {
resolve();
})
})
console.log(stations);
}
function readStationDataFromFile(filePath, stationIds, stationsData) {
return new Promise(resolve => {
fs.createReadStream(filePath)
.pipe(csv())
.on('data', function (row) {
data.push(row)
})
.on('end', function () {
const object = {}
let stationId = parseInt(file.substr(0, file.indexOf('.')))
stationIds.push(stationId)
object[stationId] = {}
object[stationId]['length'] = data.length
object[stationId]['data'] = data
stationsData.push(object)
resolve();
})
});
}
// call it with the directory path and the array of csvDataFiles-paths
processStationsCsv(directory, csvDataFiles);
I am attempting to load some CSV data in my API such that I can manipulate it and pass through to my front end, however I am having a few issues returning the data.
I am using fast-csv to do the parsing here.
service.js
const fs = require('fs');
const csv = require('fast-csv');
module.exports.getFileContents = (filepath) => {
let data = [];
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => console.error(error))
.on('data', row => data.push(row))
.on('end', () => {
console.log(data) // This will print the full CSV file fine
return data;
});
};
routes.js
router.get('/data/:filename', (req, res) => {
const file = FS.getFileContents(testUrl + '/' + req.params.filename + '.csv');
console.log(file); // This prints 'undefined'
res.send(file);
});
I can print out the CSV contents fine from the service, but I just get 'undefined' from the actual routes. Can somebody please point out what I'm missing?
This is a common problem with JavaScript code, in the following.
.on('end', () => {
console.log(data);
return data;
});
Your on-end handler is an anonymous callback function (because of () =>), so when you return data, you are returning data out of your on-end handler callback function. You are not returning data out of your enclosing getFileContents() function.
Here's a typical way to write this kind of code:
const getFileContents = async (filepath) => {
const data = [];
return new Promise(function(resolve, reject) {
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => reject(error))
.on('data', row => data.push(row))
.on('end', () => {
console.log(data);
resolve(data);
});
});
}
And then, call it as follows, though this must be within an async function:
const data = await getFileContents('games.csv');
What's happened here is as follows:
your getFileContents is now async and returns a promise
the CSV data will be available when resolve(data) is executed
the caller can await the fulfillment/resolution of this promise to get the data
You could just create a Promise in the service and return it. Once the job is done, resolve it. The returned Promise will wait until it is resolved.
service.js
const fs = require('fs');
const csv = require('fast-csv');
module.exports.getFileContents = (filepath) => {
let data = [];
return new Promise((resolve) => {
fs.createReadStream(filepath)
.pipe(csv.parse({ headers: true }))
.on('error', error => console.error(error))
.on('data', row => data.push(row))
.on('end', () => {
resolve(data);
});
}
};
routes.js
router.get('/data/:filename', (req, res) => {
const file = await FS.getFileContents(testUrl + '/' + req.params.filename + '.csv');
console.log(file); // This prints only after it is resolved
res.send(file);
});
I have this function
function getJsonObjectFromURL(url, onData) {
let chunks = [];
return require('https').get(url, res => {
res.setEncoding('utf8')
.on('data', (chunk) => {
chunks.push(chunk);
})
.on('end', () => {
onData(JSON.parse(chunks.join('')));
});
}).on('error', function(e) {
console.log("Got an error: ", e);
});
}
Also I have this script that converts url's data to json array.
url = https://pu.vk.com/c824502/upload.php?act=do_add&mid=213468131&aid=-14&gid=156603484&hash=7ab9a7e723425f4a6ca08709cbd5ebd0&rhash=ba8f0ec6580a6eafce38349b12ed3789&swfupload=1&api=1&wallphoto=1
getJsonObjectFromURL(url, data => {
console.log(data.server, data.photo, data.hash);
});
It goes well when console.log. But when I want to make from this script variable, it gives me huge collection
var xx = getJsonObjectFromURL(url, data => {
return data.server;
});
console.log(xx);
Your function getJsonObjectFromURL() doesn't return the object returned by the URL. It returns the object responsible for the https request code, which is something you don't want.
I see that you are using ES6, so the best solution for you is to probably create an async function that returns a promise, which will give you great flexibility. Here is an improved version of your code:
const https = require('https');
async function getJsonObjectFromURL(url) {
return new Promise((resolve, reject) => {
const chunks = [];
try {
https.get(url, res => {
res.setEncoding('utf8')
.on('data', (chunk) => {
chunks.push(chunk);
})
.on('end', () => {
resolve(JSON.parse(chunks.join('')));
});
}).on('error', e => reject(e));
} catch (err) {
reject(err);
}
});
};
This code allows you to retrieve the remote contents of the HTTPS url synchronously or asynchronously.
Asynchronous Call
As you have already done in your code, you can use a lambda callback that handles the response when it is ready.
const url = 'https://pu.vk.com/c824502/upload.php?act=do_add&mid=213468131&aid=-14&gid=156603484&hash=7ab9a7e723425f4a6ca08709cbd5ebd0&rhash=ba8f0ec6580a6eafce38349b12ed3789&swfupload=1&api=1&wallphoto=1';
// here we use a lambda callback that handles the response
getJsonObjectFromURL(url)
.then(data => {
console.log(data.server, data.photo, data.hash);
})
.catch(err => console.error(err));
Synchronous Call
The synchronous call forces the function to wait for the result. This is how you can do it:
async function getSync() {
try {
// wait for the result
const data = await getJsonObjectFromURL(url);
console.log(data.server);
} catch(err) {
console.error(err);
}
}
getSync();
Please note that we can only use the await keyword when we are inside an async function. This is why I had to wrap the synchronous call with a function.