Batch get DocumentReferences? - javascript

I'm trying to improve a firestore get function, I have something like:
return admin.firestore().collection("submissions").get().then(
async (x) => {
var toRet: any = [];
for (var i = 0; i < 10; i++) {
try {
var hasMedia = x.docs[i].data()['mediaRef'];
if (hasMedia != null) {
var docData = (await x.docs[i].data()) as MediaSubmission;
let submission: MediaSubmission = new MediaSubmission();
submission.author = x.docs[i].data()['author'];
submission.description = x.docs[i].data()['description'];
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
submission.media = mediaRef.data() as MediaData;
toRet.push(submission);
}
}
catch (e) {
console.log("ERROR GETTIGN MEDIA: " + e);
}
}
return res.status(200).send(toRet);
});
The first get is fine but the performance is worst on the line:
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
I think this is because the call is not batched.
Would it be possible to do a batch get on an array of mediaRefs to improve performance?
Essentially I have a collection of documents which have foreign references stored by a string pointing to the path in a separate collection and getting those references has been proven to be slow.

What about this? I did some refactoring to use more await/async code, hopefully my comments are helpful.
The main idea is to use Promise.all and await all the mediaRefs retrieval
async function test(req, res) {
// get all docs
const { docs } = await admin
.firestore()
.collection('submissions')
.get();
// get data property only of docs with mediaRef
const datas = await Promise.all(
docs.map(doc => doc.data()).filter(data => data.mediaRef),
);
// get all media in one batch - this is the important change
const mediaRefs = await Promise.all(
datas.map(({ mediaRef }) =>
admin
.firestore()
.doc(mediaRef)
.get(),
),
);
// create return object
const toRet = datas.map((data: MediaSubmission, i) => {
const submission = new MediaSubmission();
submission.author = data.author;
submission.description = data.description;
submission.media = mediaRefs[i].data() as MediaData;
return submission;
});
return res.status(200).send(toRet);
}

Related

How to access individual object in array using Javascript

Hi I have exported using data (hawkers collection) using getDocs() from Firebase.
After that I put each hawker data as an object in an array called allStall as shown in the screenshot of the console log below.
Question 1 - How do I access each individual object in my allStall array. I try to use .map() to access each of it, but i am getting nothing.
Do note that I already have data inside my allStall array, see screenshot above.
[Update] map doesn't work in code below because field is stallname not stallName. However, it needs to be async + await if using/call in/from other function.
Question 2 - Why is there [[Prototype]]: Array(0) in my allStall array
export /*Soln add async*/function getAllStall(){
var allStall = [];
try
{
/*Soln add await */getDocs(collection(db, "hawkers")).then((querySnapshot) =>
{
querySnapshot.forEach((doc) =>
{
var stall = doc.data();
var name = stall.stallname;
var category = stall.category;
var description = stall.description;
var stallData = {
stallName:name,
stallCategory:category,
stallDescription:description
};
allStall.push(stallData);
});});
console.log(allStall);
//Unable to access individual object in Array of objects
allStall.map(stall =>{console.log(stall.stallName);});}
catch (e) {console.error("Error get all document: ", e);}
return allStall;
}
In my main js file, i did the following:
useEffect(/*Soln add await*/() =>
{
getAllStall();
/*Soln:replace the statement above with the code below
const allStall = await getAllStall();
allStall.map((stall)=>console.log(stall.stallname));
*/
}
);
You are getting nothing because allStall is empty since you are not waiting for the promise to be fullfilled
try this
export const getAllStall = () => getDocs(collection(db, "hawkers"))
.then((querySnapshot) =>
querySnapshot.map((doc) =>
{
const {stallName, category, description} = doc.data();
return {
stallName:name,
stallCategory:category,
stallDescription:description
};
});
)
try to change use effect like this
useEffect(async () =>
{
const allStats = await getAllStall();
console.log(allStats)
allStats.forEach(console.log)
}
);
A very big thanks to R4ncid, you have been an inspiration!
And thank you all who commented below!
I managed to get it done with async and await. Latest update, I figure out what's wrong with my previous code too. I commented the solution in my question, which is adding the async to the function and await to getDocs.
Also map doesn't work in code above because field is stallname not stallName. However, it needs to be async + await if using in/calling from other function.
Helper function
export async function getAllStall(){
const querySnapshot = await getDocs(collection(db, "hawkers"));
var allStall = [];
querySnapshot.forEach(doc =>
{
var stall = doc.data();
var name = stall.stallname;
var category = stall.category;
var description = stall.description;
var stallData = {
stallName:name,
stallCategory:category,
stallDescription:description
};
allStall.push(stall);
}
);
return allStall;
}
Main JS file
useEffect(async () =>
{
const allStall = await getAllStall();
allStall.map((stall)=>console.log(stall.stallname));
}
);
Hurray

JavaScript: How can I insert an Object into an Array of Objects?

I have this script that takes data from a JSON with almost 100 data, then uses this data to bring the weather from an API and after that, inserts this data into an object (using a for for creating my 100 objects), I would like to add the objects that have a temperature > 99 in one array and the ones that have a temperature < 99 into another I have tried this way but doesn't seem to work, sorry if it's a super fool mistake that I can't see, thanks for your help!
This is my script:
async function calcWeather(){
const info = await fetch('../json/data.json')
.then(function(response) {
return response.json()
});
for (var i in info) {
const _idOficina = info[i][0].IdOficina
const _nombreOficina = info[i][0].NombreOficinaSN
const _zona = info[i][0].Zona
const _estado = info[i][0].NombreEstado
const lat = info[i][0].latjson
const long = info[i][0].lonjson
const base = `https://api.openweathermap.org/data/2.5/weather?lat=${lat}&lon=${long}&appid=${api_key}&units=metric&lang=sp`
fetch(base)
.then((responses) => {
return responses.json()
})
.then((data) => {
// console.log(data)
var myObject = {
Id_Oficina: _idOficina,
Latitud: data.coord.lat,
Longitud: data.coord.lon,
Ciudad: data.name,
Estado: _estado,
Zona: _zona,
Nombre_Oficina: _nombreOficina,
Temperatura: data.main.temp,
Descripcion: data.weather[0].description
};
// validation
if (myObject.Temperatura < 99){
var lstValid = [];
function pushValid(){
lstValid.push(myObject[i]);
}
pushValid();
console.log(pushValid())
}
});
}
};
Your array is local, so for every object you create new lstValid array with no previous data. The solution is to create the array before fetching the data or before the loop:
async function calcWeather(){
var lstValid = []; // HERE
const info = await fetch('../json/data.json')
.then(function(response) {
return response.json()
});
var lstValid = []; // OR HERE (ONLY ONE OF THEM)
for (...) {
...
}
You'll probably be best served by creating the array outside of that call since you're clearing it every run. Then simply add your object. Like Trincot's comment, i'm not sure what exactly you're indexing.
async function calcWeather(){
var lstValid = [];
....
if (myObject.Temperatura < 99){
lstValid[someindex] = myObject;
}
else{
lstNotValid[someOtherIndex] = myObject;
}
}

How to make a chained fetch and show results as each step becomes available?

I'm fetching article list data from API and use/fetch Unsplash API to get relative images according to each fetched article title.
This is my code:
let url = 'http://127.0.0.1:8000/api';
async function getData(url) {
const res = await fetch(url);
const objects = await res.json();
await Promise.all(objects.map(async (object) => {
const res = await fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title);
const image = await res.json();
object.image_url = image.results[0].urls.small
object.image_alt = image.results[0].alt_description
}));
}
let articles_1 = getData(url + '/articles/index/1/');
let articles_2 = getData(url + '/articles/index/2/');
let articles_3 = getData(url + '/articles/index/3/');
I am showing three different categories at once on the same page. That's why I call that function three times.
Question:
When this function kicks, results are shown after both article data and images are fetched. But I want to show article data first when it's been fetched and then images when they get fetched in order to shorten the user waiting time. How can I achieve it wether with Svelte reactive declaration or plain Javascript?
You would want to seperate the two functions, so they can be called in sequence.
const endpoint = 'http://127.0.0.1:8000/api';
const getArticles = async (url) => {
return fetch(url).res.json();
};
const renderArticles = async (articles) => {
// render article set and return it as a DOM object
return articlesDOM;
};
const getImageForArticle = async (articleNode) => {
const res = await = fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title);
const img = new Image();
img.src = res.results[0].urls.small;
img.alt = res.results[0].alt_description;
return {img, articleNode};
};
const renderImage = async (stuff) => {
const {img, articleNode} = stuff;
// inject your img into your article
};
// now call in sequence
getArticles(endpoint+'/articles/index/1/').then(renderArticles).then(articleNodes => {
const promises = articleNodes.map(articleNode => {
return getImageForArticle(articleNode).then(renderImage);
});
return Promise.all(promises);
});
While I'm not completely sure what you're trying to do (I don't have a minimal working example), here's my best attempt at it:
var url = 'http://127.0.0.1:8000/api';
async function getData(url) {
var data = fetch(url)
.then(data => data.json())
await data.then(data => ArticleFunc(data))
await data.then(function(data) {
data.map(function(object) {
fetch('https://api.unsplash.com/search/photos?client_id=XXX&content_filter=high&per_page=1&query=' + object.title)
.then(data => data.json())
object.image_url = image.results[0].urls.small
object.image_alt = image.results[0].alt_description
ImageFunc(object)
})
})
}
function ArticleFunc(data){
//display article
}
function ImageFunc(data){
//display image
}
getData(url + '/articles/index/1/');
getData(url + '/articles/index/2/');
getData(url + '/articles/index/3/');
Note that this is to be treated as pseudocode, as again, it is untested due to the absense of a minimal working example.

Firebase async scheduler issue

I'm new to both firebase and async javascript. I'm trying to create a scheduled task in firebase to fetch a bunch of rss URLs from a collection, parse it and store it inside another collection but I'm getting Error: 4 DEADLINE_EXCEEDED: Deadline exceeded at Object.callErrorFromStatus upon its execution.
const refreshRSS = functions.pubsub.schedule('every 30 mins').onRun(async context => {
let newRSS = addRSS();
return await newPodcasts;
});
addRSS = async () => {
const newRSSFeed = new Array();
let rssURLs = await db.collection('rssURLs').get();
rssURLs.forEach(async rssURLsObject=>{
rss = rssURLsObject.data();
let rssData = await parser.parseURL(rss.url);
newRSSFeed.push(db.collection('rss').doc(encodeURIComponent(rss.url))
.set(podcast));
})
return newRSSFeeds;
}
I also tried returning Promise.all(newRSS) inside refreshRSS but it throws another error stating: is not iterable.
I'm not sure what exactly is firebase expectingas return parameter.
When you have to resolve promise in loop, try to use map. This will work:
const refreshRSS = functions.pubsub.schedule('every 30 mins').onRun(async context => {
let newRSS = await addRSS();
return await newPodcasts;
});
addRSS = async () => {
const newRSSFeed = new Array();
let rssURLs = await db.collection('rssURLs').get();
let promise = rssURLs.map(async rssURLsObject=>{
rss = rssURLsObject.data();
let rssData = await parser.parseURL(rss.url);
let con = await db.collection('rss').doc(encodeURIComponent(rss.url)
newRSSFeed.push(con)
.set(podcast));
})
await Promise.all(promise)
return newRSSFeeds;
}

Code snippet executes after function is finished in Firebase Cloud Functions

I have a bit of a problem using firebase cloud functions. The code bellow is a function that writes to the Firestore DB an object that contains 2 arrays. After deploying the function, the idf_words array is populated, and the idf_weight is empty.
I tried placing a few log messages in the for loop and found the the query.get() executes after the function ends. Is there a way to make firestore wait until the query.get() finishes?
export const updateCakeAndPastriesIDF = functions.firestore.document("TF/tf/Cake_and_Pastries/{itemCategory}")
.onUpdate((change, context) => {
const itemBefore = change.before.data();
const itemAfter = change.after.data();
if (itemAfter['tf_tf_score'] === itemBefore['tf_tf_score']){
console.log('This TF score of the words in this item has not changed');
return null;
} else {
console.log('This TF score of the words in this item has changed');
const tfWords:string[] = itemAfter['tf_unique_words'];
const tfItemUid:string = itemAfter['tf_item_uid'];
const idfWords:string[] = [];
const idfWeight: number[] = [];
const db = admin.firestore().collection('TF').doc('tf').collection('Cake_and_Pastries');
tfWords.forEach(function (tfword) {
idfWords.push(tfword);
const query = db.where("tf_unique_words", "array-contains", tfword);
query.get().then(function (itemDoc) {
if (!itemDoc.empty){
const numberOfDocs = itemDoc.size;
console.log("For item: "+tfItemUid+", there are "+numberOfDocs+"Documents");
admin.firestore().collection('Number_of_Items')
.doc('Cake_and_Pastries')
.get()
.then(function (numberDoc){
const numberOfCakesAndPastries = numberDoc.data()['number_of_items_in_category'];
const idfOfWord = (Math.log(numberOfDocs/numberOfCakesAndPastries)+1);
idfWeight.push(idfOfWord);
console.log("Word IDF: "+idfOfWord);
console.log(idfWeight);
})
}else {
console.log("No such document!");
}
})
});
console.log("IDF weight array outside of loop: "+idfWeight);
admin.firestore()
.collection('IDF')
.doc('idf')
.collection('Cake_and_Pastries')
.doc(tfItemUid).set({
idf_item_uid: tfItemUid,
idf_words: idfWords,
idf_weight: idfWeight
});
}
});

Categories