After going through simple examples on cloud functions that worked, I want to test something more complex. I have created a table (BTDevices) table in the Firebase realtime database, populated with few data:
Now I want my cloud function to use the MAC addresses for retrieving the "userId" fields. This is a snippet of my JavaScript cloud function:
var getUserIdPromisesArray = [];
for (var i = 0; i < nearbyMACsArray.length; i++) {
var tmpNearbyMAC = nearbyMACsArray[i];
var promise = admin.database().ref(`${DB_BT_DEVICES}/${tmpNearbyMAC}/userId`).once('value');
getUserIdPromisesArray.push(promise);
}
return Promise.all(getUserIdPromisesArray).then(results => {
results.forEach(result => {
var userId = result.val();
console.log('result.val() = ', result.val());
});
});
From other posts like this, this, and this, I know that I need to first create all my promises, and when they are done, then process the fetched data. So the for-loop iterates the nearbyMACsArray that contains the MAC addresses, creates a promise, and pushes it to the getUserIdPromisesArray.
Then I wait for all promises to finish and iterate the "results" array to print the read value, but the cloud function log shows a null value in one of the outputs:
Any ideas on why this is happening? Am I missing something with the promises?
Thanks.
According to the documentation, val() returns null when there is no data at the requested location. This is most likely what's going on. It's not likely a problem with your promises.
Related
I am trying to get the count for how many documents I have in my collection. I have this, but it is not returning what I need it to, it is returning a whole bunch of unnecessary info I don't need for this simple task:
var estimatedDocumentCount = ServicesModel.countDocuments({});
console.log(estimatedDocumentCount)
It is returning the entire query, plus all its embedded parameters it seems like. how do I do this properly?
async function countDocuments() {
const count = await ServicesModel.countDocuments({});
return count;
};
const count = countDocuments();
It's probably because countDocuments is an asynchronous call and you are executing it synchronously.
Follow the syntax mentioned in Mongoose Docs which uses a callback function to get the count.
ServicesModel.countDocuments({}, function (err, count) {
console.log(count);
});
I am trying to resolve the array of promises together. Not sure how to do it. Let me share the pseudo code for it.
async function sendNotification(user, notificationInfo) {
const options = {
method: 'POST',
url: 'http://xx.xx.xx:3000/notification/send',
headers:
{ 'Content-Type': 'application/json' },
body:
{ notificationInfo, user },
json: true,
};
console.log('sent');
return rp(options);
}
I have wrapped the sendNotification method in another method which returns the promise of rp(request-promise) module.
Next i am pushing this sendNotification method in array of promise , something like this
const notificationWorker = [];
for (const key3 in notificationObject) {
if(notificationObject[key3].users.length > 0) {
notificationWorker.push(sendNotification(notificationObject[key3].users, notificationObject[key3].payload)); // problem is notification are going as soon as i am pushing in notificationWorker array.
}
}
// task 1 - send all notifications
const result = await Promise.all(notificationWorker); // resolving all notification promises together
// task 2 - update values in db , after sending all notifications
const result2 = await Promise.all(updateWorker); // update some values in db
In above code , my problem is notifications are going as soon as i am pushing it in notificationWorker array. I want all notifications to go together, when i run await Promise.all(notificationWorker)
Not sure , how to achieve what i am trying?
I understood the question partially , but then i feel this is difference between nodejs working concurrently and we trying to achieve parallelism , isn't that so.
Nodejs just switching between the tasks by , and not actually parallely doing it.Child Process might help you in that case.
So for eg. if you go through a snippet
function done(i){
try{
return new Promise((resolve,reject)=>{
console.log(i);
resolve("resolved " + i + "th promise");
})
}catch(e){
return null;
}
}
let promises = [];
for(let i=0;i < 100000; i++){
promises.push(done(i));
}
So console starts even when you dont call Promise.all right ? this was your question but infact Promise.all should not suffice your thing , should go by spwaning child processes to achieve parallelism to some extent.
The point i am trying to make it you are potraying the question to do something like first generate array of promises and start all of them once when Promise.all is called but in my opinion Promise.all also will be running concurrently not giving you what you want to achieve.
Something like this - https://htayyar.medium.com/multi-threading-in-javascript-with-paralleljs-10e1f7a1cf32 || How to create threads in nodejs
Though most of these cases show up when we need to do a cpu intensive task etc but here we can achieve something called map reduce to distribute you array of users in parts and start that array to loop and send notifications.
All of the solutions, i am presenting is to achieve some kind of parallelism but i dont think sending array of huge amount of users would ever be done easily (with less resources - instance config etc) at same instant
I'm working with Cloud Functions for Firebase, and I get a timeout with some of my functions. I'm pretty new with JavaScript. It looks like I need to put a for inside a promise, and I get some problems. The code actually get off from for too early, and I think he make this in a long time. Do you have some way to improve this code and make the code faster?
exports.firebaseFunctions = functions.database.ref("mess/{pushId}").onUpdate(event => {
//first i get event val and a object inside a firebase
const original = event.data.val();
const users = original.uids; // THIS ITS ALL USERS UIDS!!
// so fist i get all users uids and put inside a array
let usersUids = [];
for (let key in users) {
usersUids.push(users[key]);
}
// so now i gonna make a promise for use all this uids and get token's device
//and save them inside a another child in firebase!!
return new Promise((resolve) => {
let userTokens = [];
usersUids.forEach(element => {
admin.database().ref('users/' + element).child('token').once('value', snapShot => {
if (snapShot.val()) { // if token exist put him inside a array
userTokens.push(snapShot.val());
}
})
})
resolve({
userTokens
})
}) // now i make then here, from get userTokens and save in another child inside a firebase database
.then((res) => {
return admin.database().ref("USERS/TOKENS").push({
userTokens: res,
})
})
})
You are making network requests with firebase, so maybe that's why it's slow. You are making one request per user, so if you have 100 ids there, it might as well take a while.
But there's another problem that I notice, that is: you are just resolving to an empty list. To wait for several promises, create an array of promises, and then use Promise.all to create a promise that waits for all of them in parallel.
When you call resolve, you have already done the forEach, and you have started every promise, but they have not been added to the list yet. To make it better, chance it to a map and collect all the returned promises, and then return Promise.all.
I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?
In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.
I know there are tons of questions and answers on using Javascript promises to load returned Firebase objects, but I haven't succeeded in capturing the data in an array.
The issue: I've set up an array (productArray below) to iteratively take in values of a Firebase snapshot, but the array remains empty after exiting the forEach loop.
database.js:
getProductList = function(){
let productArray =[];
loadFBData = function(){
firebase.initializeApp(config);
return firebase.database().ref().once("value", function(snapshot){
return snapshot.val();
});
}
Promise.all([loadFBData()]).then(function(snapshot) {
snapshot.forEach(function(product) {
productArray.push(product.val());
});
});
}
Question: I think I'm using Promise.all wrong, but can't figure out how (also am new asynchronous functions). Thanks for any hints on how to get productArray to successfully load.
Edit: To elaborate, it seems snapshot is loading with the correct Firebase data, and the forEach is correctly looping through each item in snapshot but productArray becomes (or remains) empty outside of the Promise.all statement.
Update I implemented all the suggestions below and yet, the program will not stop long enough to build an array from the snapshot object. Last year I used Angularfire's $loaded function which was so effective in getting the program to wait so I don't understand why an ordinary JS promise won't work here. Does it make sense to resort to timeout? I even tried to combine alexmac's and Pointy's suggestions, but no cigar:
getProductList = function(){
let productArray =[];
loadFBData = function(){
firebase.initializeApp(config);
return new Promise(resolve => {
firebase.database().ref().once('value')
.then(function(snapshot){
return resolve(function(snapshot){
productArray = Object.keys(snapshot).map((key, prod) => prod);
});
});
});
}
loadFBData();
}
Background
I think you're missing an understanding of both promises and the Firebase SDK.
First, note that all Firebase SDK functions return promises, so there's no need to create your own. Second, Promise.all is for combining multiple promises. A single promises can be chained off of with a simple .then().
I have a video on saving and querying data here: Save and Query Firebase Data
I have another video on promises here: Promises
Loading Firebase
You should be initializing firebase at the very top of your page. It only needs to be initialized once, so don't initialize it within a function.
You can call firebase.initializeApp(config) just after loading the Firebase SDK on the page. Or you can reference Firebase Hosting's init.js as shown below.
Example
The following example loads the Firebase SDK, uses my testing db's init.js file to initialize firebase, then queries, manipulates and returns an array.
Note that Firebase doesn't support zero-indexed arrays, so everything you get back from Firebase will be an unsorted object. This example shows a few different ways of returning the data :)
<html>
<head>
<script src="https://www.gstatic.com/firebasejs/4.3.0/firebase.js"></script>
<script src="https://quiver-two.firebaseapp.com/__/firebase/init.js"></script>
<script>
function loadFirebaseArray(path) {
return firebase.database().ref(path).once('value').then(snap => snap.val());
}
function toArray(items) {
return Object.keys(items).reduce((array, key) => {
const value = items[key];
array.push({key, value});
return array;
}, []).sort();
}
loadFirebaseArray('stackoverflow').then(items => {
console.log('items', items);
const asArray = toArray(items);
alert(JSON.stringify(asArray));
const justTheValues = asArray.map(x => x.value).sort();
alert(justTheValues);
});
</script>
</head>
</html>
A working example might look like this:
getProductList = () => {
let loadFBData = () => {
firebase.initializeApp(config);
return new Promise(resolve => {
firebase.database().ref().once('value', resolve);
});
}
let productArray =[];
loadFBData()
.then(snapshot => {
productArray = Object.keys(snapshot).map((key, prod) => prod);
}
Here, I use promisification to create a promise for firebase query. So loadFBData returns a promise, I can create a promise chain or use ES7 async/await to wait until promise will be fulfilled to use result.