We Keep Coding

Getting consistent data from API in my node server without breaking down [duplicate]

This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.

If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}

Firebase function won't deleted child after certain time

I want to deleted a child after a certain time. I know that you need Firebase function to achief this. This is what I got so far:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const Ref = admin.database().ref('/Feed');
Ref.once('value', (snapshot) => {
snapshot.forEach((child) => {
if (1000*(Number(child.val()['timestamp']) + Number(child.val()['duration'])) >= timeNow) {
child.ref.set(null);
}
});
});
return res.status(200).end();
});
I want to deleted the child when the duration is over (the duration is in seconds). This is my structure:
Thanks!

You're sending a response to the caller at the end of the function, which will be executed before the data from the database is returned. And Cloud Functions will stop executing your code straight after that res.status(200).end(), so the database cleanup never happens.
To prevent this, only send a response to the caller after all data has been deleted from the database:
exports.removeOldMessages = functions.https.onRequest((req, res) => {
const timeNow = Date.now();
const Ref = admin.database().ref('/Feed');
return Ref.once('value', (snapshot) => {
let updates = [];
snapshot.forEach((child) => {
if (1000*(child.val().timestamp + child.val().duration) >= timeNow) {
updates[child.key] = null;
}
});
return Ref.update(updates).then(() => {
return res.status(200).end();
});
});
});
I highly recommend storing an additional property in your child nodes though, with the precalculated value of timestamp + duration. By having such a property, you can run a query on the nodes that have expired, instead of having to read all child nodes and then filtering in code.
For an example of this, see my answer to Delete firebase data older than 2 hours, and the Cloud Functions example that was based on that.

Firebase Functions not firing for Cloud Firestore

I simply cannot see where I'm going wrong here. My Cloud Firestore is on "europe-west3", the functions are deployed to "europe-west1" (the docs tell me that this is the closest location to west3).
Structure is thus: I've got a bunch of "tickets" each of which can have a subcollection named "comments". The console thus looks like this:
The upload was successful:
The function code was taken from the official code samples
Github repo for Function samples
This is what my code looks like:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.countComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments/{commentsid}')
.onWrite(
async (change) => {
const ticketsRef = change.after.ref.parent;
const countRef = ticketsRef.parent.child('comments_count');
let increment;
if(change.after.exists() && !change.before.exists()) {
increment = 1;
} else if(!change.after.exists() && change.before.exists()) {
increment = -1;
} else {
return null;
}
await countRef.transaction((current) => {
return (current || 0) + increment;
});
console.log('Counter updated');
return null;
});
exports.recountComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments_count')
.onDelete(
async (snap) => {
const counterRef = snap.ref;
const collectionRef = counterRef.parent.child('comments');
const commentsData = await collectionRef.once('value');
return await counterRef.set(commentsData.numChildren());
}
)
Now, the problem is that these functions simply do not fire. I'm not seeing anything in the logs, regardless of whether I'm pushing changes through my clients (a Flutter app) or if I'm changing things directly in the Firebase console.
In my desperation I've also tried to simply listen to "/tickets" as any changes below that path should also trigger - but there's nothing.
So. What is the obvious thing I overlooked? And, yes, I had a look at the other questions/answers but nothing jumped at me...
edit:
This would be the corrected version, probably not optimal.
exports.countComments = functions.region('europe-west1').firestore.document('/tickets/{ticketId}/comments/{commentsId}')
.onWrite(
async (change, context) => {
const ticketId = context.params.ticketId;
const ticketRef = admin.firestore().collection('tickets').doc(ticketId);
let increment;
if(change.after.exists && !change.before.exists) {
increment = 1;
} else if(!change.after.exists && change.before.exists) {
increment = -1;
} else {
return null;
}
return transaction = admin.firestore().runTransaction(t => {
return t.get(ticketRef)
.then(doc => {
let count = (doc.data().comments_count || 0) + increment;
t.update(ticketRef, {comments_count: count});
});
}).then(res => {
console.log('Counter updated');
}).catch(err => {
console.log('Transaction error:', err);
});
});

Your database is Cloud Firestore, but you've written a Realtime Database trigger. They are two completely different databases. Follow the documentation for writing Cloud Firestore triggers instead.
Your function will start like this:
functions.region('europe-west1').firestore.document('...')
Note "firestore" instead of "database".

Firebase Realtime Database and Cloud Functions -- increment counter based on data nodes

I'm working on a web application that will visualize data from my Firebase database. But first, I want to be able to "count" the total number of users with a given data so that I can then use that count number in my graphs.
For reference, my database looks like this:
Because I expect separate totals for the required keys, I'm guessing that I'll need separate counters for each one. I've started writing a cloud function to keep track of when a new user is created:
import * as functions from 'firebase-functions'
export const onMessageCreate = functions.database
.ref('/students/{studentID}')
.onCreate((snapshot, context) => {
const userData = snapshot.val()
const afterGrad = userData.afterGrad
const gender = userData.gender
const gradDate = userData.gradDate
const program = userData.program
const race = userData.race
const timeToComplete = userData.timeToComplete
})
But now, I'm extremely lost at how I should go about creating counters. Would something like this suffice, with an individual counter for each constant?
import * as functions from 'firebase-functions'
var counterAfterGrad;
export const onMessageCreate = functions.database
.ref('/students/{studentID}')
.onCreate((snapshot, context) => {
const userData = snapshot.val()
const afterGrad = userData.afterGrad
var counterAfterGrad++
})
Or should I be thinking about using a transaction in this case? I'm really not sure of the best way, and would really appreciate some help.

Yes, you should use a transaction. See the documentation here: https://firebase.google.com/docs/database/web/read-and-write#save_data_as_transactions and https://firebase.google.com/docs/reference/js/firebase.database.Reference#transaction
For counting the overall number of users you could do as follows:
export const onMessageCreate = functions.database
.ref('/students/{studentID}')
.onCreate((snapshot, context) => {
const userData = snapshot.val()
const afterGrad = userData.afterGrad
const allUsersCounterRef = admin
.database()
.ref('allUsersCounter');
return allUsersCounterRef
.transaction(counter_value => {
return (counter_value || 0) + 1;
})
})
Note that you may have to take into consideration the deletion of a user.
You could very well have several counters, for example by "gender" (male/female) and by "program". You would then use an object in the transaction as follows:
exports.onMessageCreate = functions.database
.ref('/students/{studentID}')
.onCreate((snapshot, context) => {
const userData = snapshot.val();
const countersRef = admin.database().ref('counters');
return countersRef.transaction(currentData => {
currentData[userData.gender] = (currentData[userData.gender] || 0) + 1;
currentData[userData.program] = (currentData[userData.program] || 0) + 1;
return currentData;
});
});

Nodejs synchronous loops block execution

when I try to run a function in background it blocks every other requests until it is done...
For example if I execute that function and then try to make a get request to a route that returns some information from the database then the response will come only after that function execution is done and I don't understand why.
This is the basic structure of my function that runs in background (it finds the 3rd party requests from a page and then look for the initiator request for each of them):
const thirdPartyReq = [];
let allRequests = [];
const findInitiatorReq = async () => {
allRequests = allRequests.reverse();
for(const [_, request] of thirdPartyReq.entries()) {
if(!request["Initiator Request"]) {
const fullRequest = request['Request URL'];
const parseUrl = new URL(fullRequest);
let hostname = parseUrl.hostname || null;
const domain = await extractDomain(hostname);
let pathname = parseUrl.pathname || null;
hostname = hostname.replace(/www./g, '')
let checkUrl;
const domainIndex = hostname.indexOf(domain) - 1;
const subdomain = (hostname.substr(0, domainIndex));
const queryString = parseUrl.search || '';
const noProtocol = hostname + pathname + queryString;
const noQueryString = hostname + pathname;
const requestProcessing = [fullRequest, noProtocol, noQueryString, hostname];
const requestIndex = allRequests.findIndex((el) => {
return (el.url == request['Request URL'] && el.thirdParty);
});
for(const [_, query] of requestProcessing.entries()) {
for(const [index, checkRequest] of allRequests.entries()) {
if(index > requestIndex) {
if(checkRequest.content && checkRequest.content.body) {
const contentBody = checkRequest.content.body;
if(contentBody.includes(query)) {
request['Initiator Request'] = checkRequest.url;
}
}
}
}
}
}
}
}
for(const [pageIndex, page] of results.entries()) {
const pageUrl = page.url;
const requests = page.requests;
const savedRequestUrls = [];
let parseUrl = new URL(pageUrl);
let hostname = parseUrl.hostname;
let requestsCounter = 0;
const pageDomain = await extractDomain(hostname);
if(!urlList.includes(pageUrl)) {
crawledUrls.push(pageUrl);
}
for(const [_, request] of Object.entries(requests)) {
if(request.url.indexOf('data:') == -1) {
parseUrl = new URL(request.url);
hostname = parseUrl.hostname;
let requestDomain = await extractDomain(hostname);
const reqObj = await findThirdPartyReq(pageUrl, request, requestDomain);
if(reqObj != null) {
request.thirdParty = true;
savedRequestUrls.push(reqObj);
}
// Store all requests that have a domain
if(requestDomain) {
request.page = pageUrl;
allRequests.push(request);
requestsCounter++;
}
}
}
findInitiatorReq();
}
I noticed that everything will work well if I remove this part of code:
for(const [_, query] of requestProcessing.entries()) {
for(const [index, checkRequest] of allRequests.entries()) {
if(index > requestIndex) {
if(checkRequest.content && checkRequest.content.body) {
const contentBody = checkRequest.content.body;
if(contentBody.includes(query)) {
request['Initiator Request'] = checkRequest.url;
}
}
}
}
}
This is the route that calls the function:
router.get('/cookies',async (req, res) => {
res.status(200).send(true);
const cookies = await myFunc();
}
Can anyone please tell me why that function is blocking everything until it returns a response and how can I fix this?

The obvious answer here is to convert your function into an asynchronous one. There are already multiple answers here on StackOverflow about that topic.
The gist: use asynchronous functions when you're elaborating some heavy task. Bear in mind that NodeJS is single threaded, thus the fact that sync functions block execution of other functions, is somewhat expected.
The tool you need to use to achieve asynchronous functions are: async/await (included without libraries/transpiling in the latest NodeJS LTS) and Promises. Forget about callbacks, since they are a really bad design.
How to use async / await in js:
https://medium.com/#Abazhenov/using-async-await-in-express-with-node-8-b8af872c0016
https://medium.freecodecamp.org/how-to-write-beautiful-node-js-apis-using-async-await-and-the-firebase-database-befdf3a5ffee
How to use Promises and what they are:
Replacing callbacks with promises in Node.js
understanding javascript promise object

Well, obviously you have a synchronous loop, which, of course, blocks execution. It will eventually block it anyway, as it has to perform several heavy operations. The response to the client is sent, but you still continue to work on some stuff, so other requests will have to wait.
A probable solution could be something like triggering another node process and handling stuff out there (something similar to a WebWorker in the browser)
You can try this library: async, there is a eachSeries method in it, meant specifically for handling big chunks of data/arrays. See the documentation for further information