I'm wanting to get/read two documents with a Promise.all then insert some fields into one response which I got from the other, .then set to a final document.
I'm trying to do below and it doesn't error/fail but the data don't get transferred. I'm assuming I must "unpack" the responses, i.e., create a new object and append all the properties then hand that object off for the .set? ...Issue is that these responses can be full of stuff so I was hoping to not have to handle all that.
var promises = [getUserInfoFromFirestore(),getOrder(order,"orders")];
Promise.all(promises).then(function (res) {
//move some user fields to order fields
res[1].data().soldToEmail = res[0].email;
finalRef.set(res[1].data()).then(function() {
deleteOrder(order).then(function() {
toast("Order Submitted");
});
});
res[1].data().soldToFirstName = res[0].firstName;
}).catch(function (error) {
console.log("Error fetching order:", error);
});
DocumentSnapshot objects are immutable. You will have to remember the results of the first call to data(), as it creates a new object each time. Modify that object instead, and use it in the call to set().
Alternatively, it's easier and more straightforward to use update() if you just want to modify the contents of a single field in a document. You don't even have to read the document you want to update.
Related
I have a callback, where I need to get the array out of my callback.
I am trying to return the awaited array into the predefined one.
let documents = [];
socket.on('echo', async function(documents, data){
documents = await data;
console.log(documents); // shows me, what I want to see
});
console.log(documents); // empty Array
I need the result in my predefined Array documents
I have read several Tuts, but I dont get it. I know on stackoverflow it is sked several times. But all threads seem to be more complex then my situation. So I hope to get it cleared out with an more incomplex one.
You need to understand something first. When this runs what is inside the callback doesn't run unit the server will emit that event, in your case 'echo'.
What I think you want to do is use documents outside the callback. You can create a function and call it when the event is emitted.
Something like this:
const manageDocuments = (documents) => {
// do what you want with documents array
console.log(documents);
}
socket.on('echo', async function(documents, data){
documents = await data;
manageDocuments(documents);
});
Of course you can also get rid of the async/await
let documents = await socket.on('echo', async function(documents, data){
console.log(documents);
return data;
});
console.log(documents);
The problem is that the code outside the socket function executes with the empty array because is only executed once at runtime.
If you want to have access to the documents inside the socket function you have to make then persist, or use the socket.on inside of another loop.
I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?
In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.
I'm building a small API that fetches data and perform tasks on it (using async), stores some of this data in an array using push, and then shows it to a client with Hapi's reply().
I'm looking to empty my array (for example using arrayname.length = 0) right after the server has sent an answer to the client.
Current code follows this logic:
let data = []
server.route({
method: 'GET',
path: '/api/{data}',
handler: function (request, reply) {
async.parallel([
function(callback) {
// fetch some data, work with it and then add it to our array using push
data.push ({ // example data
bla:'blabla',
number:'10'
});
callback();
},
function(callback) { // another time (...)
data.push ({
bla:'blabla',
number:'2'
});
callback();
}
],
function(err) { // This is the final callback, sort data and send it using reply
data.sort(function (a, b) {
return a.number - b.number
})
reply(data)
console.log(request.params.data)
})
}
});
server.start((err) => {
if (err) {
throw err;
}
console.log(`Server running at: ${server.info.uri}`);
});
With the current code if I refresh my page, data gets added to the already existing data array.
I tried for several hours and find no way to empty this array right after reply(data) has been called.
Is there any way to do that, or would I have to nest my parallel async functions with an async serie and perform the emptying that way? (Haven's been able to succeed either + sounds overkill).
Is there a way to simply execute something once "reply" has been called or when there is a new "get" so that each client can have the only the data generated by it's request and not also the data that was in the array before?
Thanks!
I'm in agreement with all of the answers suggesting to put the array within the handler. With the array situated out side of the request handler, all request coming through will be writing to the same array. Even if you clear this array at the end of your handler logic, it is not guaranteed to be empty for the next request context since you can have many request context in play.
Following #piotrbienias's advice I created the array inside handler rather than at the beginning of my file and it's cleared each time I do a new request on my API, so I don't need to empty it once the reply is sent.
Thanks, #piotrbienias!
Disclaimer: I'm new to ES6 and promises in general so its possible my approach is fundamentally wrong.
The Problem
I have an api that returns paged data. That is it returns a certain number of objects for a collection and if there's more data it returns a Next property in the response body with the url to get the next set of data. The data will eventually feed the autocomplete on a text input.
To be specific, if my collection endpoint is called /Tickets the response would look like:
{
Next: "/Tickets?skip=100&take=100"
Items: [ ... an array of the first 100 items ... ]
}
My current solution to get all the ticket data would be to
Make a new promise for the returning the whole combined set of data
"Loop" the ajax calls by chaining dones until there is no more Next value
Resolve the promise
getTickets(filterValue) {
let fullSetPromise = new Promise();
let fullSet = [];
// This gets called recursively
let append = function(previousResult) {
fullSet.concat(previousResult.Items);
// Loop!
if(previousResult.Next) {
$.ajax({
url: previousResult.Next
})
.done(append)
.catch(reason => fullSetPromise.reject(reason));
}
else {
fullSetPromise.resolve(fullSet);
}
}
// We set things off by making the request for the first 100
$.ajax({
url: `/Tickets?skip=0&take=100&filter=${filterValue}`
})
.done(append)
.catch(reason => fullSetPromise.reject(reason));
return fullSetPromise;
}
Eventually the promise is used by the frontend for autocomplete on a text input. Ideally I'd like to be able to abort the previous call when new input comes in.
inputChanged(e) {
this.oldTicketPromise.abort();
this.oldTicketPromise =
Api.GetTickets(e.target.value).then(updateListWithResults);
}
I am aware of debouncing. But that just means the problem happens every n seconds instead of on every key press.
I know the jqxhr object has an abort() property on it and I'd like that to be available to the caller somehow. But because there are multiple jqXHR objects used in GetTickets I'm not sure how to handle this.
So my main 2 questions are:
What is the appropriate way to consume paged data from an api while returning a promise.
How can the returned promise be made abortable?
Side question:
I feel like if I don't catch the errors then my "wrapper" promise will swallow any thrown errors. Is that a correct assumption?
Note the javascript code might have errors. It's mostly demonstrative for the logic.
Edit: Solved
I have solved this by combining this answer https://stackoverflow.com/a/30235261/730326 with an array of xhrs as suggested in the comments. I will add a proper answer with code when I find the time.
Every Parse Installation object instance in my Parse database has a pointer to a specific user. I have a background job that runs for every user, and what I want to do in this part of the background job is to set the respective user Installation's channel property to ["yesPush"], for push notification targeting purposes.
The way I figured I would do it is by querying for the specific Parse.Installation instance, and then setting its channels property. This doesn't seem to be working however. I'm trying to follow the guidelines in the Parse Cloud Code Docs, but it's either not the correct use case, or I'm not following it correctly.
Code:
var installationQuery = new Parse.Query(Parse.Installation);
installationQuery.equalTo('userId', parentUser);
installationQuery.find().then(function(results) {
return Parse.Object.set('channels', ["yesPush"]);
});
The way I would do it is as follows:
// for every User
var userQuery = new Parse.Query(Parse.User);
userQuery.each(function(user) {
var installationQuery = new Parse.Query(Parse.Installation);
installationQuery.equalTo('userId', user);
return installationQuery.first().then(function(installation) {
installation.set('channels', ['yesPush']);
return installation.save();
});
}).then(function() {
status.success();
}, function(error) {
status.error(error.message);
});
The each() function is how you process large sets of data in a job. If you are performing other async tasks inside it you need to return their promise.
The first() function is much faster and easier if we only expect one record.
I am calling set() on the actual installation object returned by first().
I am returning the save() promise to allow promise chaining.