We Keep Coding

Why does firebase cloud-function javascript promise run more than the number of loop invocations?

I have a cloud function that is triggered when a sale/purchase is committed into firestore. This function's purpose is to update the inventory level centrally.
The function works just fine if I'm updating an item's inventory at only 1 warehouse, but doing so for multiple warehouses has unexpected behavior. I'm looping through all the warehouses that are affected to calculate the total inventory level changes, and every iteration kicks-off a javascript promise.
The problem seems to occur with the way the promises are invoked. E.g: if I want to update 3 warehouses and loop 3 times, somehow 5 promises are being kicked-off. This is visible through the logs. I've researched similar questions here, but the solutions were suggested while firestore was still in beta and might not be the right way forward. (Firestore transactions getting triggered multiple times resulting in wrong data)
Here is the code
export const onTransactionCreate = functions.firestore
.document('/companies/{companyId}/sub_transactions/{transId}')
.onCreate(async (snapshot, context) => {
const transId = context.params.transId
const stock_transaction: IStockTransaction = <IStockTransaction>snapshot.data()
const trans_type: TRANS_TYPE = stock_transaction.trans_type
const promises: any[] = []
stock_transaction.lineItems.forEach((element, index) => {
const ITEM_GUID = element.item_guid
const is_increasing = isIncreasingTransaction(element.line_trans_type)
const delta_stock = element.qty_transaction * (is_increasing ? 1 : -1)
const TARGET_BRANCH_ID = element.target_branch_guid
const itemRef = db.collection(FIRESTORE_PATHS.COL_COMPANIES).doc(companyId).
collection(FIRESTORE_PATHS.SUB_COMPANIES_ITEMS).
doc("" + ITEM_GUID)
const item_promise = db.runTransaction(async t => {
try {
const item_doc = await t.get(itemRef)
const item_branch_quantities: IBranchQuantity[] = (item_doc.data()!.branch_quantities || new Array())
const item_branch_ids: string[] = (item_doc.data()!.available_branch_ids || new Array())
const branch_index = item_branch_ids.indexOf(TARGET_BRANCH_ID)
console.log(`${transId} Line Item ${index}, after document.get(), search branch index: ${branch_index}`)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty = prev_qty.quantity + delta_stock
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
console.log(`${transId} Line Item ${index} Updating qty # item ${delta_stock}, prev qty ${prev_qty.quantity}`)
} else {
item_branch_ids.push(TARGET_BRANCH_ID)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: TARGET_BRANCH_ID,
quantity: delta_stock
})
console.log(`${transId} Line Item ${index} Adding qty # item ${delta_stock}`)
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
available_branch_ids: item_branch_ids
})
} catch (err) {
throw new Error(err)
}
})
promises.push(item_promise)
});
return Promise.all(promises)
})

we have found the solution by reading this article.
A transaction consists of any number of get() operations followed by any number of write operations such as set(), update(), or delete(). In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
lineItems.forEach(element => {
const delta_transaction = element.qty * (isLineTransIncrease(element.line_trans_type) ? 1 : -1)
const itemRef = db.collection('companies').doc(companyId).collection('sub_items').doc("" + element.item_guid)
const p = db.runTransaction(t => {
return t.get(itemRef)
.then(doc => {
let item_branch_quantities: IBranchQuantity[] = doc.data()!.branch_quantities
let item_branch_ids: string[] = doc.data()!.available_branch_ids
if (!item_branch_quantities)
item_branch_quantities = new Array()
if (!item_branch_ids)
item_branch_ids = new Array()
const branch_index = item_branch_ids.indexOf(current_branch_id)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty: number = prev_qty.quantity + delta_transaction
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
} else {
item_branch_ids.push(current_branch_id)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: current_branch_id,
quantity: delta_transaction
})
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
branch_ids: item_branch_ids
})
})
})
item_update_transactions.push(p)
});
return Promise.all(item_update_transactions)
})
function isLineTransIncrease(line_trans: number): boolean {
return (line_trans === 1) || (line_trans === 2)
}

correct output of async await promise?

I'm using the firstore, and I get information from the first.
Here, I want to retrieve data from the firstore and print it out, but there is a problem with asynchronous mode.
The output is output after undefined.
I want to know how to print it correctly.
When data is received, the function receives data, and the method of receiving data after searching the firestore.
in react component
index = (date) =>{
let date_total = UserAction.getIndex(date)
return date_total
}
render(){
const {user_list} = this.props;
console.log(this.state)
const date = "2020-02-24"
console.log("data",date)
let index = this.index(date) < this
console.log("index", index)
return(...)
and useraction function
export function getIndex(data){
let time_now = moment(data).utc().format()
const user_history_start = moment(time_now).startOf("d").utc().format();
const user_history_end = moment(time_now).endOf("d").utc().format();
let db = loadFB().firestore();
let query = db.collection('users').where('create_date','>=',user_history_start).where('create_date','<=',user_history_end);
let number ;
return query.get().then( docs=>{
number = docs.size
return number
})
}
i want output
data 2020-02-24
index 11 < (firestore given data)
but output
data 2020-02-24
index promise{<pending>} < (firestore given data)
Give me a good solution.

I guess you can't print a promised value in render, set the state and print it instead?
constructor() {
super();
this.state = {
index: null
}
}
getIndex = (date) => {
// --------- update ----------
UserAction.getIndex(date).then(date_total => this.setState({ index: date_total }))
}
componentDidMount() {
const date = "2020-02-24"
// --------- update ----------
this.getIndex(date)
}
render() {
console.log("index", this.state.index)
// will first print null,
// then print the index when the promise is done (re-rendered due to state change)
}
You may want to read this as a reference.

React Axios API call with array loop giving wrong order?

I was learning react and doing some axios api call with an array. I did a code on gathering data through coinmarketcap api to learn.
So, my intention was to get the prices from the api with a hardcoded array of cryptocurrency ids and push them into an array of prices. But I ran into a problem with the prices array, as the prices were all jumbled up. I was supposed to get an array in this order
[bitcoinprice, ethereumprice, stellarprice, rippleprice]
but when I ran it in the browser, the prices came randomly and not in this order, sometimes I got my order, sometimes it didn't. I used a button which onClick called the getPrice method. Does anyone know what went wrong with my code? Thanks!
constructor(){
super();
this.state = {
cryptos:["bitcoin","ethereum","stellar","ripple"],
prices:[]
};
this.getPrice = this.getPrice.bind(this);
}
getPrice(){
const cryptos = this.state.cryptos;
console.log(cryptos);
for (var i = 0; i < cryptos.length; i++){
const cryptoUrl = 'https://api.coinmarketcap.com/v1/ticker/' + cryptos[i];
axios.get(cryptoUrl)
.then((response) => {
const data = response.data[0];
console.log(data.price_usd);
this.state.prices.push(data.price_usd);
console.log(this.state.prices);
})
.catch((error) => {
console.log(error);
});
}
}

If you want to receive the data in the order of the asynchronous calls you make, you can use Promise.all, that waits until all the promises of an array get executed and are resolved, returning the values in the order they were executed.
const cryptos = ['bitcoin', 'ethereum', 'stellar', 'ripple'];
const arr = [];
for (var i = 0; i < cryptos.length; i++){
const cryptoUrl = 'https://api.coinmarketcap.com/v1/ticker/' + cryptos[i];
arr.push(axios.get(cryptoUrl));
}
Promise.all(arr).then((response) =>
response.map(res => console.log(res.data[0].name, res.data[0].price_usd))
).catch((err) => console.log(err));

You could use a closure in the for loop to capture the value of i and use it as the index once the data is returned rather than using push:
getPrice(){
const cryptos = this.state.cryptos;
console.log(cryptos);
for (var i = 0; i < cryptos.length; i++) {
const cryptoUrl = 'https://api.coinmarketcap.com/v1/ticker/' + cryptos[i];
(function (x) {
axios.get(cryptoUrl)
.then((response) => {
const data = response.data[0];
console.log(data.price_usd);
var newPrices = this.state.prices;
newPrices[x] = data.price_usd;
this.setState({prices: newPrices});
console.log(this.state.prices);
})
.catch((error) => {
console.log(error);
});
})(i);
}
}

Continue on Null Value of Result (Nodejs, Puppeteer)

I'm just starting to play around with Puppeteer (Headless Chrome) and Nodejs. I'm scraping some test sites, and things work great when all the values are present, but if the value is missing I get an error like:
Cannot read property 'src' of null (so in the code below, the first two passes might have all values, but the third pass, there is no picture, so it just errors out).
Before I was using if(!picture) continue; but I think it's not working now because of the for loop.
Any help would be greatly appreciated, thanks!
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
let title = document.querySelector('h1').innerText;
let article = document.querySelector('.c-entry-content').innerText;
let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
let source = "The Verge";
let categories = "Tech";
if (!picture)
continue; //throws error
return {
title,
article,
picture,
source,
categories
}
});
}

let picture = document.querySelector('.c-picture img').src;
if (!document.querySelector('.c-picture img').src) {
let picture = 'No Link'; } //throws error
If there is no picture, then document.querySelector() returns null, which does not have a src property. You need to check that your query found an element before trying to read the src property.
Moving the null-check to the top of the function has the added benefit of saving unnecessary calculations when you are just going to bail out anyway.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
//...Getting to correct page and scraping it three times
const result = await page.evaluate(() => {
const pictureElement = document.querySelector('.c-picture img');
if (!pictureElement) return null;
const picture = pictureElement.src;
const title = document.querySelector('h1').innerText;
const article = document.querySelector('.c-entry-content').innerText;
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
if (!result) continue;
// ... do stuff with result
}
Answering comment question: "Is there a way just to skip anything blank, and return the rest?"
Yes. You just need to check the existence of each element that could be missing before trying to read a property off of it. In this case we can omit the early return since you're always interested in all the results.
async function scrape3() {
// ...
for (let i = 1; i <= 3; i++) {
const result = await page.evaluate(() => {
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1');
const content = document.querySelector('.c-entry-content');
const picture = img ? img.src : '';
const title = h1 ? h1.innerText : '';
const article = content ? content.innerText : '';
const source = "The Verge";
const categories = "Tech";
return {
title,
article,
picture,
source,
categories
}
});
// ...
}
}
Further thoughts
Since I'm still on this question, let me take this one step further, and refactor it a bit with some higher level techniques you might be interested in. Not sure if this is exactly what you are after, but it should give you some ideas about writing more maintainable code.
// Generic reusable helper to return an object property
// if object exists and has property, else a default value
//
// This is a curried function accepting one argument at a
// time and capturing each parameter in a closure.
//
const maybeGetProp = default => key => object =>
(object && object.hasOwnProperty(key)) ? object.key : default
// Pass in empty string as the default value
//
const getPropOrEmptyString = maybeGetProp('')
// Apply the second parameter, the property name, making 2
// slightly different functions which have a default value
// and a property name pre-loaded. Both functions only need
// an object passed in to return either the property if it
// exists or an empty string.
//
const maybeText = getPropOrEmptyString('innerText')
const maybeSrc = getPropOrEmptyString('src')
async function scrape3() {
// ...
// The _ parameter name is acknowledging that we expect a
// an argument passed in but saying we plan to ignore it.
//
const evaluate = _ => page.evaluate(() => {
// Attempt to retrieve the desired elements
//
const img = document.querySelector('.c-picture img');
const h1 = document.querySelector('h1')
const content = document.querySelector('.c-entry-content')
// Return the results, with empty string in
// place of any missing properties.
//
return {
title: maybeText(h1),
article: maybeText(article),
picture: maybeSrc(img),
source: 'The Verge',
categories: 'Tech'
}
}))
// Start with an empty array of length 3
//
const evaluations = Array(3).fill()
// Then map over that array ignoring the undefined
// input and return a promise for a page evaluation
//
.map(evaluate)
// All 3 scrapes are occuring concurrently. We'll
// wait for all of them to finish.
//
const results = await Promise.all(evaluations)
// Now we have an array of results, so we can
// continue using array methods to iterate over them
// or otherwise manipulate or transform them
//
results
.filter(result => result.title && result.picture)
.forEach(result => {
//
// Do something with each result
//
})
}

Try-catch worked for me:
try {
if (await page.$eval('element')!==null) {
const name = await page.$eval('element')
}
}catch(error){
name = ''
}

JavaScript multi-level promises skipped .then()

I was having some problem with multi-level of promises. What I tried to do is first get list of receipt items under certain category, then for each receipt item, I get its detail & receipt ID, after I get the receipt ID, I search for the account ID. Then, I get the account details based on account ID. Here is my code:
var query = // get receipt items under certain category
var outerPromise = query.once('value').then(data => {
var promises = [];
var datasetarr = [];
data.forEach(snapshot => {
var itemData = // get receipt item unique push ID
var query = // get details of receipt items
var promise = query.once('value').then(data => {
var itemDetail = // get receipt item detail
if(type == subtype){
var receiptID = itemDetail.receiptID;
var query = // query receipts table by receiptID
return query.once('value').then(data => {
data.forEach(snapshot => {
snapshot.forEach(childSnapshot => {
if(childSnapshot.key == receiptID){
var accountKey = // get accountID
var query = // query accounts table
return query.once('value').then(data => {
var accountDetail = data.val();
var age = accountDetail.age;
var gender = accountDetail.gender;
console.log(age + ' ' + gender);
datasetarr.push({age: age, gender: gender});
});
}
});
});
});
}
});
promises.push(promise);
});
return Promise.all(promises).then(()=> datasetarr);
});
I managed to print out the result from the console.log above. However, when I tried to print out here which is when the promise is done:
outerPromise.then((arr) => {
console.log('promise done');
for(var i = 0; i < arr.length; i++){
console.log(arr[i].age + ' ' + arr[i].gender);
}
});
I get nothing here. The console now is showing 'promise done' first before any other results I printed out above.
How can I do this correctly?

I will provide a more detailed explanation in a couple of hours, I have a prior engagement which means I can't provide details now
First step to a "easy" solution is to make a function to make an array out of a firebase snapshot, so we can use map/concat/filter etc
const snapshotToArray = snapshot => {
const ret = [];
snapshot.forEach(childSnapshot => {
ret.push(childSnapshot);
});
return ret;
};
Now, the code can be written as follows
// get list of receipt items under category
var query // = // get receipt items under certain category
var outerPromise = query.once('value').then(data => {
return Promise.all(snapshotToArray(data).map(snapshot => {
var itemData // = // get receipt item unique push ID
var query // = // get details of receipt items
return query.once('value').then(data => {
var itemDetail // = // get receipt item detail
if(type == subtype){
var receiptID = itemDetail.receiptID;
var query //= // query receipts table by receiptID
return query.once('value').then(data => {
return Promise.all([].concat(...snapshotToArray(data).map(snapshot => {
return snapshotToArray(snapshot).map(childSnapshot => {
if(childSnapshot.key == receiptID){
var accountKey //= // get accountID
var query //= // query accounts table
return query.once('value').then(data => {
var accountDetail = data.val();
var age = accountDetail.age;
var gender = accountDetail.gender;
console.log(age + ' ' + gender);
return({age, gender});
});
}
}).filter(result => !!result);
}).filter(result => !!result)));
});
}
});
})).then([].concat(...results => results.filter(result => !!result)));
});
To explain questions in the comments
[].concat used to add the content of multiple arrays to a new array, i.e
[].concat([1,2,3],[4,5,6]) => [1,2,3,4,5,6]
...snapshotToArray(data).map(etc
... is the spread operator, used as an argument to a function, it takes the iterable and "spreads" it to multiple arguments
console.log(...[1,2,3]) == console.log(1,2,3)
In this case snapshotToArray(data).map returns an array of arrays, to give a console log example
console.log(...[[1,2],[3,4]]) == console.log([1,2], [3,4])
adding the concat
[].concat(...[[1,2],[3,4]]) == [].concat([1,2],[3,4]) == [1,2,3,4]
so it flattens a two level array to a single level, i.e.
console.log(...[[1,2],[3,4]]) == console.log(1,2,3,4)
So in summary, what that code fragment does is flatten a two level array
filter(result => !!result)
simply "filters" out any array elements that are "falsey". As you have this condition
if(childSnapshot.key == receiptID){
if that is false, the result will be undefined for that map - all other results will be an array, and even empty arrays are truthy - that's why the filtering is done so often! There's probably a better way to do all that, but unless you're dealing with literally millions of items, there's no real issue with filtering empty results like this
End result is a flat array with only the Promises returned from the code within