Every Parse Installation object instance in my Parse database has a pointer to a specific user. I have a background job that runs for every user, and what I want to do in this part of the background job is to set the respective user Installation's channel property to ["yesPush"], for push notification targeting purposes.
The way I figured I would do it is by querying for the specific Parse.Installation instance, and then setting its channels property. This doesn't seem to be working however. I'm trying to follow the guidelines in the Parse Cloud Code Docs, but it's either not the correct use case, or I'm not following it correctly.
Code:
var installationQuery = new Parse.Query(Parse.Installation);
installationQuery.equalTo('userId', parentUser);
installationQuery.find().then(function(results) {
return Parse.Object.set('channels', ["yesPush"]);
});
The way I would do it is as follows:
// for every User
var userQuery = new Parse.Query(Parse.User);
userQuery.each(function(user) {
var installationQuery = new Parse.Query(Parse.Installation);
installationQuery.equalTo('userId', user);
return installationQuery.first().then(function(installation) {
installation.set('channels', ['yesPush']);
return installation.save();
});
}).then(function() {
status.success();
}, function(error) {
status.error(error.message);
});
The each() function is how you process large sets of data in a job. If you are performing other async tasks inside it you need to return their promise.
The first() function is much faster and easier if we only expect one record.
I am calling set() on the actual installation object returned by first().
I am returning the save() promise to allow promise chaining.
Related
I'm wanting to get/read two documents with a Promise.all then insert some fields into one response which I got from the other, .then set to a final document.
I'm trying to do below and it doesn't error/fail but the data don't get transferred. I'm assuming I must "unpack" the responses, i.e., create a new object and append all the properties then hand that object off for the .set? ...Issue is that these responses can be full of stuff so I was hoping to not have to handle all that.
var promises = [getUserInfoFromFirestore(),getOrder(order,"orders")];
Promise.all(promises).then(function (res) {
//move some user fields to order fields
res[1].data().soldToEmail = res[0].email;
finalRef.set(res[1].data()).then(function() {
deleteOrder(order).then(function() {
toast("Order Submitted");
});
});
res[1].data().soldToFirstName = res[0].firstName;
}).catch(function (error) {
console.log("Error fetching order:", error);
});
DocumentSnapshot objects are immutable. You will have to remember the results of the first call to data(), as it creates a new object each time. Modify that object instead, and use it in the call to set().
Alternatively, it's easier and more straightforward to use update() if you just want to modify the contents of a single field in a document. You don't even have to read the document you want to update.
I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?
In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.
I have a situation where I am building a data layer based on ES6 JS Promises that fetch data from the network. I am caching all Promises internally by the url.
Everything seems to be working fine except one thing. I want to ensure that the data coming out of the network layer is a copy/clone of the data retrieved from the network and I obviously do not want to do that everywhere in the client code that implements Promise's then handlers.
I would like to set this up so then handler automatically gets a copy of the cached data.
To add a twist to this, I would like this to be configurable on a url basis inside the data layer so that some Promises do the extra post-processing copy while others return just the raw result.
Can anyone suggest a proper implementation to accomplish this? I should mention that I would like to get a new copy of the original raw result each time a new client asks for it.
The current simplified pseudo implementation looks like this
getCachedData(url){
if (cache[url]) {
return cache[url];
} else {
var promise = new Promise(function(resolve,reject){
var data = ...ajax get...;
resolve(data);
});
cache[url] = promise;
}
getCachedData(url).then(result=>{
here I want the result to be a copy of data I resolved the original promise with.
});
Structure it like this:
function retrieveCopiedData () {
// getDataFromServer is your original Promise
return getDataFromServer().then(function (value) {
// use a library of your choice for copying the object.
return copy(value);
})}
}
This means that all consumers of retrieveCopiedData will receive the value returned from retrieveCopiedData's then() handler.
retrieveCopiedData().then(function (value) {
// value is the copy returned from retrieveCopiedData's then handler
})
You can add conditional logic to retrieveCopiedData as you see fit.
It seems like you just want to incorporate the cloning process right in your data layer:
getCachedData(url){
if (!cache[url]) {
cache[url] = new Promise(function(resolve,reject){
var data = ...ajax get...;
resolve(data);
});
}
if (requiresPostProcessing(url))
return cache[url].then(clone);
else
return cache[url];
}
Notice that it might be a good idea not to clone the data each time it is retrieved, but to simply freeze the object that your promise is resolved with.
if I have a code like this
if (request.params.friends != null)
{
_.each(request.params.friends, function(friend) {
// create news
var News = Parse.Object.extend("News");
var news = new News();
news.set("type", "ask");
news.save();
});
response.success();
}
and the length of request.params.friends is 2, does the second news get saved for certain? If not, how to make sure it gets saved? I looked at Parse.Promise documentation and in all the examples, the loop is inside a query or a save. Do I need to save the first news first and then create the Promise? I still don't get how "asynchronous" works.. Does the response.success() work like a return or break?
The loop does get executed twice.
response.success() acts like a return.
The asynchronous magic is in the "save" method. When "save" is called, the Parse.com says, "ok, you want me to save it. I'll save it, but not now. For now, here is a promise that I'll save it later." The save method returns an Promise object and the promise will be fulfilled when the object is actually saved.
So what happens is a little like
First time through the loop: create friend #1.
Ask Parse to save friend #1.
Second time through the loop: create friend #2.
Ask Parse to save friend #2.
Return successful response.
Parse actually saves friend #1 and friend #2
It's been a while since I've used Parse, but I'm not sure usually both the friend objects would actually get saves. Calling response.success() could kill work-in-progress. Here is an alternative implementation:
var objectsToSave = _.collect(request.params.friends, function(friend) {
var news = new News();
news.set({type : "ask"});
return news;
});
Parse.Object.saveAll(objectsToSave, {
success: function(list) {
// All the objects were saved.
response.success();
},
error: function(error) {
// An error occurred while saving one of the objects.
},
});
The saveAll function saves all the objects at once. It's usually faster than saving objects one-at-a-time. In addition to providing saveAll with the objects to save, we provide it an object with a success function and an error function. Parse.com promises only to call the functions AFTER the save is complete (or it experienced an error).
There are a few other things going on. The Parse.Object.extend statement belongs in a different place in your code. Also, the set function doesn't take a list of strings. It takes a JavaScript object.
I'd like to use the MongoDB native JS driver with bluebird promises. How can I use Promise.promisifyAll() on this library?
The 2.0 branch documentation contains a better promisification guide https://github.com/petkaantonov/bluebird/blob/master/API.md#promisification
It actually has mongodb example which is much simpler:
var Promise = require("bluebird");
var MongoDB = require("mongodb");
Promise.promisifyAll(MongoDB);
When using Promise.promisifyAll(), it helps to identify a target prototype if your target object must be instantiated. In case of the MongoDB JS driver, the standard pattern is:
Get a Db object, using either MongoClient static method or the Db constructor
Call Db#collection() to get a Collection object.
So, borrowing from https://stackoverflow.com/a/21733446/741970, you can:
var Promise = require('bluebird');
var mongodb = require('mongodb');
var MongoClient = mongodb.MongoClient;
var Collection = mongodb.Collection;
Promise.promisifyAll(Collection.prototype);
Promise.promisifyAll(MongoClient);
Now you can:
var client = MongoClient.connectAsync('mongodb://localhost:27017/test')
.then(function(db) {
return db.collection("myCollection").findOneAsync({ id: 'someId' })
})
.then(function(item) {
// Use `item`
})
.catch(function(err) {
// An error occurred
});
This gets you pretty far, except it'll also help to make sure the Cursor objects returned by Collection#find() are also promisified. In the MongoDB JS driver, the cursor returned by Collection#find() is not built from a prototype. So, you can wrap the method and promisify the cursor each time. This isn't necessary if you don't use cursors, or don't want to incur the overhead. Here's one approach:
Collection.prototype._find = Collection.prototype.find;
Collection.prototype.find = function() {
var cursor = this._find.apply(this, arguments);
cursor.toArrayAsync = Promise.promisify(cursor.toArray, cursor);
cursor.countAsync = Promise.promisify(cursor.count, cursor);
return cursor;
}
I know this has been answered several times, but I wanted to add in a little more information regarding this topic. Per Bluebird's own documentation, you should use the 'using' for cleaning up connections and prevent memory leaks.
Resource Management in Bluebird
I looked all over the place for how to do this correctly and information was scarce so I thought I'd share what I found after much trial and error. The data I used below (restaurants) came from the MongoDB sample data. You can get that here: MongoDB Import Data
// Using dotenv for environment / connection information
require('dotenv').load();
var Promise = require('bluebird'),
mongodb = Promise.promisifyAll(require('mongodb'))
using = Promise.using;
function getConnectionAsync(){
// process.env.MongoDbUrl stored in my .env file using the require above
return mongodb.MongoClient.connectAsync(process.env.MongoDbUrl)
// .disposer is what handles cleaning up the connection
.disposer(function(connection){
connection.close();
});
}
// The two methods below retrieve the same data and output the same data
// but the difference is the first one does as much as it can asynchronously
// while the 2nd one uses the blocking versions of each
// NOTE: using limitAsync seems to go away to never-never land and never come back!
// Everything is done asynchronously here with promises
using(
getConnectionAsync(),
function(connection) {
// Because we used promisifyAll(), most (if not all) of the
// methods in what was promisified now have an Async sibling
// collection : collectionAsync
// find : findAsync
// etc.
return connection.collectionAsync('restaurants')
.then(function(collection){
return collection.findAsync()
})
.then(function(data){
return data.limit(10).toArrayAsync();
});
}
// Before this ".then" is called, the using statement will now call the
// .dispose() that was set up in the getConnectionAsync method
).then(
function(data){
console.log("end data", data);
}
);
// Here, only the connection is asynchronous - the rest are blocking processes
using(
getConnectionAsync(),
function(connection) {
// Here because I'm not using any of the Async functions, these should
// all be blocking requests unlike the promisified versions above
return connection.collection('restaurants').find().limit(10).toArray();
}
).then(
function(data){
console.log("end data", data);
}
);
I hope this helps someone else out who wanted to do things by the Bluebird book.
Version 1.4.9 of mongodb should now be easily promisifiable as such:
Promise.promisifyAll(mongo.Cursor.prototype);
See https://github.com/mongodb/node-mongodb-native/pull/1201 for more details.
We have been using the following driver in production for a while now. Its essentially a promise wrapper over the native node.js driver. It also adds some additional helper functions.
poseidon-mongo - https://github.com/playlyfe/poseidon-mongo