Putting s3 object data into an array - node - javascript

I'm trying to fetch data from an S3 object and put it into an array. I plan to map through this array and display the data on a React front end in grid/list whatever. I'm struggling with nested functions though, so I'd appreciate some help.
const dataFromS3 = async (bucket, file) => {
let lines = [];
const options = {
Bucket: bucket,
Key: file
};
s3.getObject(options, (err, data) => {
if (err) {
console.log(err);
} else {
let objectData = data.Body.toString('utf-8');
lines.push(objectData);
console.log(lines);
return lines;
}
});
};
Formatting is a bit weird but this is my function to get data from s3. I want to take the output of this function in the form of an array and pass it to my '/' route which I'm testing:
app.get('/', async (req, res, next) => {
try {
let apolloKey = await dataFromS3(s3Bucket, apolloKeywords);
res.send(apolloKey);
} catch (err) {
console.log('Error: ', err);
}
});
It seems that the return value of lines in the s3.getObject function needs to be returned within the first function so that I can access it in app.get but I can't seem to do it after some attempts. The value in lines turns into an empty array if I return it at the end of datafromS3() and I can't find a way to return it. I've tried using promises also using a method found here - How to get response from S3 getObject in Node.js? but I get a TypeError: Converting Circular Structure to JSON...
Thank you

You need to make your dataFromS3 func like htis. You were not returning anything from that. AWS also provided promise based function.
const dataFromS3 = async (bucket, file) => {
const lines = [];
const options = {
"Bucket": bucket,
"Key": file
};
const data = await s3.getObject(options).promise();
const objectData = data.Body.toString("utf-8");
lines.push(objectData); // You might need to conversion here using JSON.parse(objectData);
console.log(lines);
return lines;
};

Related

Iterating the creation of objects with asynchonous javascript map method data [duplicate]

This question already has answers here:
Using async/await with a forEach loop
(33 answers)
Use async await with Array.map
(9 answers)
Closed 27 days ago.
In an async IIFE at the bottom of this javascript, you'll see that I'm trying to: 1) read a JSON file, 2) get multiple RSS feed URLs from that data, 3) pull and parse the data from those feeds, and create an object with that data, so I can 4) write that pulled RSS data object to a JSON file. Everything for #1 and #2 is fine. I'm able to pull data from multiple RSS feeds in #3 (and log it to console), and I'm comfortable handling #4 when I get to that point later.
My problem is that, at the end of step #3, within the const parseFeed function, I am trying to create and push an object for that iteration of rssJSONValsArr.map() in the IIFE and it's not working. The rssFeedDataArr result is empty. Even though I am able to console.log those values, I can't create and push the new object I need in order to reach step #4. My creating of a similar object in #2 works fine, so I think it's the map I have to use within const parseFeed to pull the RSS data (using the rss-parser npm package) which is making object creation not work in step #3. How do I get rssFeedOject to work with the map data?
import fs from 'fs';
import path from 'path';
import { fileURLToPath } from 'url';
import Parser from 'rss-parser';
const parser = new Parser();
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const feedsJSON = path.join(__dirname, 'rss-feeds-test.json');
const rssJSONValsArr = [];
const rssFeedDataArr = [];
const pullValues = (feedObject, i) => {
const url = feedObject.feed.url;
const jsonValsObject = {
url: url,
};
rssJSONValsArr.push(jsonValsObject);
};
const parseFeed = async (url) => {
try {
const feed = await parser.parseURL(url);
feed.items.forEach((item) => {
console.log(`title: ${item.title}`); // correct
});
const rssFeedOject = {
title: item.title,
};
rssFeedDataArr.push(rssFeedOject);
} catch (err) {
console.log(`parseFeed() ERROR 💥: ${err}`);
}
};
(async () => {
try {
console.log('1: read feeds JSON file');
const feedsFileArr = await fs.promises.readFile(feedsJSON, {
encoding: 'utf-8',
});
const jsonObj = JSON.parse(feedsFileArr);
console.log('2: get feed URLs');
jsonObj.slice(0, 30).map(async (feedObject, i) => {
await pullValues(feedObject, i);
});
console.log('rssJSONValsArr: ', rssJSONValsArr); // correct
console.log('3: pull data from rss feeds');
rssJSONValsArr.map(async (feedItem, i) => {
await parseFeed(feedItem.url, i);
});
console.log('rssFeedDataArr: ', rssFeedDataArr); // empty !!!
// console.log('4: write rss data to JSON file');
// await fs.promises.writeFile(
// `${__dirname}/rss-bulk.json`,
// JSON.stringify(rssFeedDataArr)
// );
console.log('5: Done!');
} catch (err) {
console.log(`IIFE CATCH ERROR 💥: ${err}`);
}
})();
Example JSON file with two RSS feed URLs:
[
{
"feed": {
"details": {
"name": "nodejs"
},
"url": "https://news.google.com/rss/search?q=nodejs"
}
},
{
"feed": {
"details": {
"name": "rss-parser"
},
"url": "https://news.google.com/rss/search?q=rss-parser"
}
}
]
Any and all help appreciated. Thanks
The problem is you are printing rssFeedDataArr right after the .map call, which, like stated on the comments, is being incorrectly used, since you are not using the returned value, forEach would be the way to go here. For every value in rssJSONValsArr you are calling an anonymous and async function which in turn awaits for parseFeed, so you are basically creating a Promise in each iteration, but obviously those promises are resolved after your print statement is executed. You need to wait for all of those promises to be resolved before printing rssFeedDataArr. One way to do that, since you are creating a bunch of promises which can be run in parallel is to use Promise.all, like this:
await Promise.all(
rssJSONValsArr.map(async (feedItem, i) => {
await parseFeed(feedItem.url, i);
});
)
and you we can simplify it even more and return the promise created by parseFeed directly:
await Promise.all(
rssJSONValsArr.map((feedItem, i) => parseFeed(feedItem.url, i))
)
And in this case the right method is map and not forEach
In the case of rssJSONValsArr it works because the call to pullValues is being resolved instantly, it doesnt run asynchronously, even when its declared as async, there is not await inside the function definition.

Res.send sends an empty object, even though console.log shows it isn't empty?

I am trying to use the google-sheets api with express and don't have much experience with javascript. I'm attempting to use pass a json object from express to react, but it seems that whenever I finally send the object, it just renders as empty on the frontend?
I've tried using res.body/res.data, but the object doesn't seem to have either. I've also tried to put as many awaits as I can everywhere to make sure the object is loaded in before sending, but nothing seems to do the trick. If I use res.json or res.send with just the response object, I get a circular structure converting to JSON error. Here is the code I'm working with.
async function docShit() {
// Initialize the sheet - doc ID is the long id in the sheets URL
const doc = new GoogleSpreadsheet(
"--SPREADSHEET ID--"
);
// Initialize Auth - see https://theoephraim.github.io/node-google-spreadsheet/#/getting-started/authentication
await doc.useServiceAccountAuth({
// env var values are copied from service account credentials generated by google
// see "Authentication" section in docs for more info
client_email: process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL,
private_key: process.env.GOOGLE_PRIVATE_KEY,
});
await doc.loadInfo(); // loads document properties and worksheets
const sheet = doc.sheetsByTitle[--WORKSHEET TITLE--];
const rows = await sheet.getRows(); // can pass in { limit, offset }
return rows;
}
app.get("/home", async (req, res) => {
try {
await docShit()
.then((response) => {
res.send(Promise.resolve(response)); //console log shows the object, but res.send just sends nothing??
})
.catch((err) => console.log(err));
} catch (err) {
console.error(err.message);
}
});
There is no res.send at all in your code. Also, you use await and .then together, but I consider them alternatives. Try the following:
app.get("/home", async (req, res, next) => {
try {
var response = await docShit();
console.log(response);
/* If response is circular, decide which parts of it you want to send.
The following is just an example. */
res.json(response.map(function(row) {
return {id: row.id, cells: row.cells.map(function(cell) {
return {id: cell.id, value: cell.value};
};
})};
} catch (err) {
console.error(err.message);
next(err);
}
});

Using the response of a Axios/NodeJS API call function as a parameter to make another API call

General concept: I am accessing two API endpoints. One responds with general information, and the other with specific information.
I have called the first endpoint and with the response, I return an object in the format that the second endpoint requires as a parameter.
Problem: I keep getting "Data after transformation must be a string, an ArrayBuffer, a Buffer, or a Stream"
I want to learn to use functions across controllers, so as not having to REPEAT myself.
Code:
This returns the object that I need to use in the second API call:
const axios = require("axios");
const getSpecials = async (req, res) => {
try {
const response = await axios.get(`${apiURL}/categories`);
let specials = [];
let skus = [];
response.data.forEach((element, index) => {
if (response.data[index].is_special == true) {
specials.push({
name: response.data[index].name,
product_skus: response.data[index].product_skus,
_id: response.data[index]._id,
});
skus.push(response.data[index].product_skus);
}
});
return { product_skus: skus.flat() };
} catch (error) {
console.log(error.message);
}
};
module.exports = {getSpecials}
This returns the product details if I post the return from the first API call as a parameter:
const axios = require("axios");
const { getSpecials } = require("./categoriesControllers");
const specialProducts = async (req, res) => {
try {
const response = await axios.post(`${apiURL}/products`, getSpecials);
res.status(200).json(response.data);
} catch (error) {
console.log(error.message);
}
};
Any suggestions on how best to do this would be great. I want learn to let the backend to do all the heavy lifting and I want to serve the data as cleanly as possible to the front end.
If I understand correctly, you just need to change
const response = await axios.post(`${apiURL}/products`, getSpecials);
to this
const response = await axios.post(`${apiURL}/products`, await getSpecials());
because right now you're passing a function declaration to second argument to axios post function, not a return value from getSpecials

How to merge multiple collection in mongodb using nodejs

Trying to merge multiple collection using nodejs and mongoose but not working.Anyone can find solution for this.
Getting this error:
Error [ERR_HTTP_HEADERS_SENT]: Cannot set headers after they are sent to the client
product.model.js:
module.exports = mongoose.model('Test1', userSchemaTest1, 'test1');
module.exports = mongoose.model('Test2', userSchemaTest2, 'test2');
module.exports = mongoose.model('Test3', userSchemaTest3, 'test3');
module.exports = mongoose.model('Test4', userSchemaTest4, 'test4');
module.exports = mongoose.model('Test5', userSchemaTest5, 'test5');
product.controller.js:
module.exports.getAllProducts = (req, res, next) => {
let collection = req.query.collection;
var newVal = collection.split(',');
var allP = [];
var getAllp;
allP.push(...newVal);
allP.forEach((element) => {
let tabledatas = mongoose.model(element);
tabledatas.find({}, function(err, docs) {
if (err) {
console.log('ss' + err);
return
}
getAllp = [...getAllp, res.json(docs)];
})
})
return getAllp;
}
api call
http://localhost:3000/api/getAllProducts?collection=Test1,Test2,Test3,Test4,Test5
There are several issues with your code, I'll focus only on how to combine results of multiple queries, which is actually how to deal with async function results in general.
But you should take notes of other issues mentioned here as well
Calling res.json() in a loop, you'll certainly get "Headers already sent" errors
Putting the result of res.json() in to an array; res.json() returns ServerResponse, which has nothing to do with what we want.
Accepting user inputs (collection names) and pass them directly to the database operations is dangerous.
Handling async functions:
// models would be a list of model names
const models = collection.split(',')
// map array of model names to array of Promises (result of async operations)
const resultPromises = models.map(modelName => {
return mongoose.model(modelName).find({}).exec() // use Promise result instead of passing callback
})
// wait on all Promises to be fulfilled and resolved to actual results.
const combinedResults = await Promise.all(resultPromises)
// now you will get a result of this form [[item1, item2, ...], [itemX, itemY, ...], ...]
// but what you want is [item1, item2, ..., itemX, itemY, ...], we can use array.flat()
cont results = combinedResults.flat()
Note that you can only use await in an async function, so you'll have to modify your function like this
module.exports.getAllProducts = async (req, res, next) => { // add async keyword
// ...
}
more on array.flat()

Why isn't my cloud function returning my map correctly?

I have the following function, an I am trying to return an array of maps to my flutter app
export const usersFromContacts = functions.region('europe-west1').https.onCall(async (data, context) => {
Authentication.authenticate(context);
const phonenumbers = Validator.validateArray<string>('phonenumbers', data['phonenumbers']);
const privateDataPromises = [];
for (let i = 0; i < phonenumbers.length; i++)
privateDataPromises.push(...(await firestore.collectionGroup('userPrivateData')
.where('phonenumber', '==', phonenumbers[i]).get()).docs);
const userSnapshots = await Promise.all(privateDataPromises);
const userPromises = [];
for (let i = 0; i < userSnapshots.length; i++) {
const userID = privateDataPromises[i].id;
userPromises.push(userCollection.doc(userID).get());
}
const returnValue = (await Promise.all(userPromises)).map((document) => UserModel.fromFirestoreDocumentData(document).toMap());
console.log(returnValue);
return returnValue;
});
If I log the test field to the cloud functions console I get the correct array of maps with the correct data in the maps.
[ Map {
'id' => 'ID',
'displayName' => 'Name',
'phoneNumber' => 'Phonenumber',
'photoUrl' => 'PhotoUrl' } ]
Is what I get back in the firebase functions console. Where I replaced the values ofc.
Then inside flutter I do
Future<HttpsCallableResult> _callCloudFunction(String functionName, {Map<String, dynamic> data}) async {
final cloudFunction = api.getHttpsCallable(functionName: functionName);
try {
final httpCallable = await cloudFunction.call(data).timeout(Duration(seconds: timeout));
print(httpCallable.data); // this prints [{}] an empty array of maps?
return httpCallable;
} on CloudFunctionsException catch (error) {
throw new UserFriendlyException(error.code, error.message);
} on TimeoutException catch (error) {
throw new UserFriendlyException('Operation Failed', error.message);
} catch (error) {
throw new UserFriendlyException('Operation Failed', 'There was a problem with with executing the operation');
}
}
And I get an empty array of maps? What am I doing wrong?
I'm pretty sure the call is correct because if I return the documents directly I do get them in my front-end so am I returning the values in the wrong way from the function?
You are trying to serialize an ES6 Map object from the function to the client. That's not going to work the way you want.
Cloud Functions callable function only support serializing plain JavaScript objects, arrays, and primitive values. Instead of passing a Map, pass a normal JS object. This is the code you will have to change:
UserModel.fromFirestoreDocumentData(document).toMap()
You will have to get inside that toMap function and change it, or convert its output to normal JS objects for serialization.

Categories