We Keep Coding

Slow firebase function response from background activity

The function shown below puzzles me for two reasons:
the function execution terminates before all output is given
the function execution takes more than 3 minutes; a very long time (so long, that it might not be because of the "cold starts" issue only).
When searching for bestpractices I found a hint, that background acitivities are slowed down after function execution is terminated (https://cloud.google.com/functions/docs/bestpractices/tips#do_not_start_background_activities).
How can I create a function, that terminates after all output is created and avoids background activity?
Is there any way how to speed up the get() processing?
screenshot of firebase functions dashboard
screensthot of firestore showing the document created to trigger the function
Please have a look on the code:
// The Cloud Functions for Firebase SDK to create Cloud Functions .
const functions = require("firebase-functions");
// The Firebase Admin SDK to access Firestore.
const admin = require("firebase-admin");
admin.initializeApp();
const db = admin.firestore();
exports.evaluateScore = functions
.region('europe-west1')
.firestore
.document('organizations/{orgId}/persons/{personId}')
.onWrite(async (snap, context) => {
const newDocument = snap.after.exists ? snap.after.data() : null;
const oldDocument = snap.before.exists ? snap.before.data() : null;
console.log(`lastName: '${newDocument.personLastName}'; id: '${snap.after.id}'`);
// if only newDocument exists
if (newDocument != null && oldDocument == null ) {
const arrayNameSplit = snap.after.ref.path.split('/');
var orgId = arrayNameSplit[arrayNameSplit.length -3];
var listOfProfiles = newDocument.listOfProfiles;
console.log(`listOfProfiles: `, JSON.stringify(listOfProfiles));
for (let i = 0; i < listOfProfiles.length; i++) {
db.collection('organizations').doc(orgId).collection('profiles').doc(listOfProfiles[i]).get()
.then(docRef => {
const profile = docRef.data();
console.log(i, ' profileTitle:', JSON.stringify(profile.profileTitle))
}).catch(e => {
console.error('something went wrong', e)
});
}
}
});

You have asynchronous calls in your code, but are not telling the Cloud Functions runtime about that (through the return value). It is very likely that your database get() calls don't even complete at this stage.
To fix that problem, you can use await inside the loop or Promise.all:
exports.evaluateScore = functions
.region('europe-west1')
.firestore
.document('organizations/{orgId}/persons/{personId}')
.onWrite(async (snap, context) => {
const newDocument = snap.after.exists ? snap.after.data() : null;
const oldDocument = snap.before.exists ? snap.before.data() : null;
console.log(`lastName: '${newDocument.personLastName}'; id: '${snap.after.id}'`);
// if only newDocument exists
if (newDocument != null && oldDocument == null ) {
const arrayNameSplit = snap.after.ref.path.split('/');
var orgId = arrayNameSplit[arrayNameSplit.length -3];
var listOfProfiles = newDocument.listOfProfiles;
console.log(`listOfProfiles: `, JSON.stringify(listOfProfiles));
for (let i = 0; i < listOfProfiles.length; i++) {
const docRef = await db.collection('organizations').doc(orgId).collection('profiles').doc(listOfProfiles[i]).get();
const profile = docRef.data();
console.log(i, ' profileTitle:', JSON.stringify(profile.profileTitle))
}
}
});
There may be more problems with your code, so I recommend reading the documentation on sync, async and promises, and how to create a minimal, complete, verifiable example for future questions.

Google Apps Script Working on backend but not on sheets

I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?

I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.

You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price

I don't understand how to get over Stripe's rate limit

I am trying to develop the backend of an ecommerce website using Stripe and NodeJS (Express precisely).
When the server starts, I am trying to fetch my products from Stripe. But after the first stripe.products.list call I get an error which says that I exceeded the api rate limit. This is not true because as it says in the Stripe doc the rate is limited to 25/sec in test mode whereas I am waiting 10 SECONDS before making my second call.
Please find below the function I use to make my calls. I simply use it in a loop with a sleep() function before each call.
async function fetchFromLastObj(last_obj){
const data = stripe.products.list({
active: true,
limit: maxRetrieve,
starting_after: last_obj,
})
.then((resp) => {
console.log(`Retrieved ${resp.data.length} products.`);
return resp.data;
})
.catch((e) => { });
return data;
}
The sleep function:
const { promisify } = require('util')
const sleep = promisify(setTimeout)
The loop in question:
var last_obj_seen = null;
var nb_iters = 0;
// fetching all products from stripe
while (true) {
console.log(`Iteration ${nb_iters+1}...`)
let fetchedList = [];
if (last_obj_seen == null) {
fetchedList = await fetchFirstBatch();
} else {
fetchedList = await fetchFromLastObj(last_obj_seen);
}
fetchedList = Array.from(fetchedList);
if (fetchedList.length == 0) { break; };
last_obj_seen = fetchedList.slice(-1)[0];
await sleep(10000);
fetchPrices((fetchedList))
.then((fetchedListWithPrices)=>{
saveList(fetchedListWithPrices);//not asynchronous
})
.catch((err) => { console.error("While fetching products from Stripe..."); console.error(err); });
nb_iters += 1;
if(nb_iters > 100){ throw Error("Infinite loop error"); }
if (nb_iters !== 0){
console.log("Waiting before request...");
await sleep(10000);
}
}
console.log("Done.");

Rather than handling pagination logic yourself you can use the auto-pagination feature of the official Stripe libraries.
Our libraries support auto-pagination. This feature easily handles fetching large lists of resources without having to manually paginate results and perform subsequent requests.
In Node 10+ you can do this, for example:
for await (const product of stripe.products.list()) {
// Do something with product
}
The Stripe Node library will handle pagination under the hood for you.

Need the second firebase query to start after the first ends javascript

I'm a beginner in javascript and recently I've been working on a project and in this project I'm trying to save initial score in an empty firebase database which works perfectly fine. However just as the score is saved I want to retrieve it and do some calculations on it. I've tried setTimeout but it didn't work. Btw if there are scores in the firebase already it's working fine.
This is my code and thanks in advance:
function resultOne() {
var firstScore = trim(newGroup[0]);
scores(firstScore);
setTimeout(function() {return true;}, 30000);
var firstguyScore = getScore(firstScore)
console.log(firstGuyScore);
}
This is a function to set the initial score of 1500 and set name....
function scores(firstGuy) {
// Firebase query to increment the chosen girl and her seen status by 1 and to initialize each score by 1500
let ref = firebase.database().ref("scores");
let query = ref.orderByChild("name").equalTo(firstGuy);
query.once("value").then((snapshot) => {
if (snapshot.exists()) {
snapshot.forEach((userSnapshot) => {
let userRef = userSnapshot.ref;
userRef.child("chosen").transaction((currentValue) => {
return currentValue + 1;
});
userRef.child("seen").transaction((currentValue) => {
return currentValue + 1;
});
});
}
else {
ref.push({
name: firstGuy,
chosen: 1,
seen: 1,
score: 1500
});
}
});
and this is a function to retreive the data
async function getScore(firstGuy) {
let ref = firebase.database().ref("scores");
let query = ref.orderByChild("name").equalTo(firstGuy);
const snapshot = await query.once("value")
if (snapshot.exists()) {
snapshot.forEach((userSnapshot) => {
var userData = userSnapshot.val();
score = userData.score;
console.log(score);
});
}
}

setTimeout() calls the function (callback) you provide after a certain time. It does not block and wait. Your call to getScores() is executed immediately, as you can see in the console.
You can change the code as follows:
function resultOne() {
const firstScore = trim(newGroup[0]);
scores(firstScore);
setTimeout(() => {
const firstguyScore = getScore(firstScore);
console.log(firstGuyScore);
}, 30000);
}
Using setTimeout() this way is okay for testing and debugging. You should not use it this way in your production-ready code.
Why not await on scores() as well?
async function scores(firstGuy) {
...
const snapshot = await query.once("value");
...
}
async function resultOne() {
const firstScore = trim(newGroup[0]);
await scores(firstScore);
const firstguyScore = await getScore(firstScore);
console.log(firstGuyScore);
}

Firebase function won't deleted child after certain time

I want to deleted a child after a certain time. I know that you need Firebase function to achief this. This is what I got so far:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const Ref = admin.database().ref('/Feed');
Ref.once('value', (snapshot) => {
snapshot.forEach((child) => {
if (1000*(Number(child.val()['timestamp']) + Number(child.val()['duration'])) >= timeNow) {
child.ref.set(null);
}
});
});
return res.status(200).end();
});
I want to deleted the child when the duration is over (the duration is in seconds). This is my structure:
Thanks!

You're sending a response to the caller at the end of the function, which will be executed before the data from the database is returned. And Cloud Functions will stop executing your code straight after that res.status(200).end(), so the database cleanup never happens.
To prevent this, only send a response to the caller after all data has been deleted from the database:
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const Ref = admin.database().ref('/Feed');
return Ref.once('value', (snapshot) => {
let updates = [];
snapshot.forEach((child) => {
if (1000*(child.val().timestamp + child.val().duration) >= timeNow) {
updates[child.key] = null;
}
});
return Ref.update(updates).then(() => {
return res.status(200).end();
});
});
});
I highly recommend storing an additional property in your child nodes though, with the precalculated value of timestamp + duration. By having such a property, you can run a query on the nodes that have expired, instead of having to read all child nodes and then filtering in code.
For an example of this, see my answer to Delete firebase data older than 2 hours, and the Cloud Functions example that was based on that.