We Keep Coding

Firebase response is too slow

Hey Guys I'm using Firebase Realtime Database to fetch some data for my React.js Web App.
There is a useState called Corr_User that should store the value of correct username.
I have a function to fetch the username from Firebase : -
function Login(){
const path = firebase.database().ref("users").child(username);
path.once("value")
.then(snapShot => {
setCorr_User(snapShot.child("username").val());
})
When I do this console.log(Corr_User) it prints a empty string indicating that the useState is not updated.
For confirmation, I also logged the snapshot.val(). This printed the appropriate value.
I could understand from this that Firebase is too slow in returning response hence my useState is not getting updated. Is my understanding correct? Is there any way to make my Login() function wait for the response?
Please Help !! Thanks in Advance.
EDIT -
Ok there was a bit of confusion I guess. The code goes likes this
function Login(){
....
.once("value")
.then(snapShot => { // After fetching the data
setCorr_User(snapShot.child("username").val()); // Assigning the value
console.log(snapShot.val()); // This should be logged first
}) // End of function snapShot()
// This is still inside Login()
console.log(Corr_User) // But this is logged first
} // End of Login()
It is from this, I was able to guess that Firebase is too slow in responding and useState is not getting updated accordingly as snapShot.val() is still empty.

Data is loaded from Firebase asynchronously, and the then is called when that data is available. What this means in practice is that your console.log(Corr_User) is called before console.log(snapShot.val()). For this reason, any code that needs data from the database must be inside the then callback, or be called from there.
This is an incredibly common stumbling block for developers new to asynchronous APIs, so I recommend studying some of these links:
Why Does Firebase Lose Reference outside the once() Function?
Best way to retrieve Firebase data and return it, or an alternative way
Firebase Query inside function returns null
Passing variable in parent scope to callback function

Assigining Data from firestore get to a variable

Im trying to assign variables to their respected value from the firestore database using the get doc function, I've noticed it does not assign or update the values what so ever.
I've tried to work with async and awaits but cannot seem to make it work.
getFromDatabase(nameOfCollection,nameOfDocument){
const db = firebase.firestore();
var docRef = db.collection(nameOfCollection).doc(nameOfDocument);
docRef.get().then(function(doc) {
if (doc.exists) {
outvariable = doc.data().anyfield; // THIS IS WHAT I WANT
console.log(" Document data:", doc.data());
} else {
console.log("No such document!");
}
}).catch(function(error) {
console.log("Error getting document:", error);
});
}
im expecting outvariable = doc.data().anyfield

Most likely you're confused by the fact that data is loaded from Firestore asynchronously. It's not so much that the data isn't assigned to the values, because it really is. It just happens at a different time than you expect.
It's easiest to see this by adding some simple logging statements around the code that loads data:
const db = firebase.firestore();
var docRef = db.collection(nameOfCollection).doc(nameOfDocument);
console.log("Before starting to load data");
docRef.get().then(function(doc) {
console.log("Got data";
});
console.log("After starting to load data");
When you run this code, the output is:
Before starting to load data
After starting to load data
Got data
This is probably not what you expected, but it's actually completely correct. The data is loaded from Firestore asynchronously (since it may take some time), and instead of waiting, the main code continues. Then when the data is available, your callback function is called with that data.
This means that any code that requires the data from the database must be inside the callback, or be called from there. So the console.log(" Document data:", doc.data()) in your original code should work fine. But a similar console.log outside of the callback won't work, because it runs before the data is available.
This is an extremely common source of confusion for developers new to this type of API. But since most modern web/cloud APIs, and many other APIs, are asynchronous, it's best to learn how to work with them quickly. For that, I recommend reading:
get asynchronous value from firebase firestore reference
Doug's blog post on why Firebase APIs are synchronous
Firestore query in function return
NodeJS, Firestore get field
How do I return the response from an asynchronous call?

The data can be extracted with .data() or .get() to get a specific field.
For example: doc.get(anyfield);
More info can be found on the official documentation.

Not able to iterate over array of users pulled from firebase [duplicate]

This question already has an answer here:
How to get data from firestore DB in outside of onSnapshot
(1 answer)
Closed 3 years ago.
I am using Firebase's Cloud Firestore for a web page I'm working on. I have it currently setup to create a new document in the "Users" collection when a new user is added/joined. The issue is when I try to pull the list of users down to iterate over them, I'm not able to.
I have tried iterating over it wither different kinds of loops. The loops don't seem to run as the length of the object when console logging it is 0.
let temp = [];
db.collection("Users").onSnapshot(res => {
const changes = res.docChanges();
changes.forEach(change => {
if (change.type === "added") {
temp.push({
id: change.doc.id,
email: change.doc.data().email
});
}
});
});
console.log(temp);
console.log(temp.length);
I expected the 2nd console log to be 2 but it outputs 0. The weird thing is when I look at the object from the console log above, it shows it has a length of 2 and shows the current data in it:

Data is loaded from Firestore asynchronously. Since this may take some time, your main code will continue to run while the data is loading. Then when the data is loaded, your callback functions is called.
You can easily see this in practice by adding some logging:
console.log("Before starting onSnapshot");
db.collection("Users").onSnapshot(res => {
console.log("Got data");
});
console.log("After starting onSnapshot");
When you run this code, it logs:
Before starting onSnapshot
After starting onSnapshot
Got data
This is probably not the order you expected, but it completely explains why the log statements don't work as you expected. By the time you log the array length, the data hasn't been loaded from the database yet.
The reason logging the array does seem to work, is that Chrome updates the log output after the data has loaded. If you change it to console.log(JSON.stringify(temp));, you'll see that it logs an empty array.
This means that all code that needs access to the data, must be inside the callback, or be called from there:
let temp = [];
db.collection("Users").onSnapshot(res => {
const changes = res.docChanges();
changes.forEach(change => {
if (change.type === "added") {
temp.push({
id: change.doc.id,
email: change.doc.data().email
});
}
});
console.log(temp);
console.log(temp.length);
});
As you can probably imagine, many developers who are new to asynchronous APIs run into this problem. I recommend studying some of the previous questions on this topic, such as:
How do I return the response from an asynchronous call?
THEN ARRAY EMPTY PROMISSE
How to get data from firestore DB in outside of onSnapshot

Firestore set fails on first try but works on second

I have a simple firestore set operation like so
var docRef = firebase.firestore().collection('sessions').doc(this.state.userID);
docRef.set({
Name: seshName,
Type: theType,
}).then(()=> {
resolve(true);
});
but for some reason, it always fails on the first try after app install, but then subsequent attempts are fast and smooth.
Anyone encounter similar experiences? Any clues?

Thanks for trying to help. I found the cause. Adding .catch does not help as it just hangs as usual without any output.
It seems that Firestore only allows 1 connection at a time? This issue only happens if I do a DB action right before this .set operation, it seems that the previous DB action isn't complete yet.

Duplicate Array Data Web Scraping

I can't seem to get the article duplicates out of my web scraper results, this is my code:
app.get("/scrape", function (req, res) {
request("https://www.nytimes.com/", function (error, response, html) {
// Load the HTML into cheerio and save it to a variable
// '$' becomes a shorthand for cheerio's selector commands, much like jQuery's '$'
var $ = cheerio.load(html);
var uniqueResults = [];
// With cheerio, find each p-tag with the "title" class
// (i: iterator. element: the current element)
$("div.collection").each(function (i, element) {
// An empty array to save the data that we'll scrape
var results = [];
// store scraped data in appropriate variables
results.link = $(element).find("a").attr("href");
results.title = $(element).find("a").text();
results.summary = $(element).find("p.summary").text().trim();
// Log the results once you've looped through each of the elements found with cheerio
db.Article.create(results)
.then(function (dbArticle) {
res.json(dbArticle);
}).catch(function (err) {
return res.json(err);
});
});
res.send("You scraped the data successfully.");
});
});
// Route for getting all Articles from the db
app.get("/articles", function (req, res) {
// Grab every document in the Articles collection
db.Article.find()
.then(function (dbArticle) {
res.json(dbArticle);
})
.catch(function (err) {
res.json(err);
});
});
Right now I am getting five copies of each article sent to the user. I have tried db.Article.distinct and various versions of this to filter the results down to only unique articles. Any tips?

In Short:
Switching the var results = [] from an Array to an Object var results = {} did the trick for me. Still haven't figured out the exact reason for the duplicate insertion of documents in database, will update as soon I find out.
Long Story:
You have multiple mistakes and points of improvement there in your code. I will try pointing them out:
Let's follow them first to make your code error free.
Mistakes
1. Although mongoose's model.create, new mongoose() does seem to work fine with Arrays but I haven't seen such a use before and it does not even look appropriate.
If you intend to create documents one after another then represent your documents using an object instead of an Array. Using an array is more mainstream when you intend to create multiple documents at once.
So switch -
var results = [];
to
var results = {};
2. Sending response headers after they are already sent will create for you an error. I don't know if you have already noticed it or not but its pretty much clear upfront as once the error is popped up the remaining documents won't get stored because of PromiseRejection Error if you haven't setup a try/catch block.
The block inside $("div.collection").each(function (i, element) runs asynchronously so your process control won't wait for each document to get processed, instead it would immediately execute res.send("You scraped the data successfully.");.
This will effectively terminate the Http connection between the client and the server and any further issue of response termination calls like res.json(dbArticle) or res.json(err) will throw an error.
So, just comment the res.json statements inside the .create's then and catch methods. This will although terminate the response even before the whole articles are saved in the DB but you need not to worry as your code would still work behind the scene saving articles in database for you (asynchronously).
If you want your response to be terminated only after you have successfully saved the data then change your middleware implementation to -
request('https://www.nytimes.com', (err, response, html) => {
var $ = cheerio.load(html);
var results = [];
$("div.collection").each(function (i, element) {
var ob = {};
ob.link = $(element).find("a").attr("href");
ob.title = $(element).find("a").text();
ob.summary = $(element).find("p.summary").text().trim();
results.push(ob);
});
db.Article.create(results)
.then(function (dbArticles) {
res.json(dbArticles);
}).catch(function (err) {
return res.json(err);
});
});
After making above changes and even after the first one, my version of your code ran fine. So if you want you can continue on with your current version, or you may try reading some points of improvement.
Points of Improvements
1. Era of callbacks is long gone:
Convert your implementation to utilise Promises as they are more maintainable and easier to reason about. Here are the things you can do -
Change request library from request to axios or any one which supports Promises by default.
2. Make effective use of mongoose methods for insertion. You can perform bulk inserts of multiple statements in just one query. You may find docs on creating documents in mongodb quite helpful.
3. Start using some frontend task automation library such as puppeteer or nightmare.js for data scraping related task. Trust me, they make life a hell lot easier than using cheerio or any other library for the same. Their docs are really good and well maintained so you won't have have hard time picking these up.