We Keep Coding

Getting consistent data from API in my node server without breaking down [duplicate]

This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.

If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}

Antispam for telegram bot

I am working on creating a telegram bot, I want to make an anti-spam system, that is, when a person presses a button too many times, the bot will freeze for him for a certain number of seconds, it is possible to write a message about blocking. I just started learning JavaScript.
I use node-telegram-bot-api.
import {
bot
} from '../token.js';
import {
keyboardMain
} from '../keyboards/keyboardsMain.js';
export function commands() {
bot.on('message', msg => {
const text = msg.text;
const chatId = msg.chat.id;
if (text === '/start') {
return bot.sendMessage(chatId, 'hello', keyboardMain);
}
return bot.sendMessage(chatId, 'error');
});
}

You can create a user throttler using Javascript Map
/*
* #param {number} waitTime Seconds to wait
*/
function throttler(waitTime) {
const users = new Map()
return (chatId) => {
const now = parseInt(Date.now()/1000)
const hitTime = users.get(chatId)
if (hitTime) {
const diff = now - hitTime
if (diff < waitTime) {
return false
}
users.set(chatId, now)
return true
}
users.set(chatId, now)
return true
}
}
How to use: You'll get the user's chatId from telegram api. You can use that id as an identifier and stop the user for given specific time.
For instance I'm gonna stop the user for 10seconds once the user requests.
// global 10 second throttler
const throttle = throttler(10) // 10 seconds
// in your code
const allowReply = throttle(chatId) // chatId obtained from telegram
if (allowReply) {
// reply to user
} else {
// dont reply
}

I don't understand how to get over Stripe's rate limit

I am trying to develop the backend of an ecommerce website using Stripe and NodeJS (Express precisely).
When the server starts, I am trying to fetch my products from Stripe. But after the first stripe.products.list call I get an error which says that I exceeded the api rate limit. This is not true because as it says in the Stripe doc the rate is limited to 25/sec in test mode whereas I am waiting 10 SECONDS before making my second call.
Please find below the function I use to make my calls. I simply use it in a loop with a sleep() function before each call.
async function fetchFromLastObj(last_obj){
const data = stripe.products.list({
active: true,
limit: maxRetrieve,
starting_after: last_obj,
})
.then((resp) => {
console.log(`Retrieved ${resp.data.length} products.`);
return resp.data;
})
.catch((e) => { });
return data;
}
The sleep function:
const { promisify } = require('util')
const sleep = promisify(setTimeout)
The loop in question:
var last_obj_seen = null;
var nb_iters = 0;
// fetching all products from stripe
while (true) {
console.log(`Iteration ${nb_iters+1}...`)
let fetchedList = [];
if (last_obj_seen == null) {
fetchedList = await fetchFirstBatch();
} else {
fetchedList = await fetchFromLastObj(last_obj_seen);
}
fetchedList = Array.from(fetchedList);
if (fetchedList.length == 0) { break; };
last_obj_seen = fetchedList.slice(-1)[0];
await sleep(10000);
fetchPrices((fetchedList))
.then((fetchedListWithPrices)=>{
saveList(fetchedListWithPrices);//not asynchronous
})
.catch((err) => { console.error("While fetching products from Stripe..."); console.error(err); });
nb_iters += 1;
if(nb_iters > 100){ throw Error("Infinite loop error"); }
if (nb_iters !== 0){
console.log("Waiting before request...");
await sleep(10000);
}
}
console.log("Done.");

Rather than handling pagination logic yourself you can use the auto-pagination feature of the official Stripe libraries.
Our libraries support auto-pagination. This feature easily handles fetching large lists of resources without having to manually paginate results and perform subsequent requests.
In Node 10+ you can do this, for example:
for await (const product of stripe.products.list()) {
// Do something with product
}
The Stripe Node library will handle pagination under the hood for you.

How do I wait until a cookie is set?

I am writing the acceptance tests for my application's login feature. At some point, I want to double-check the cookie's expiry time.
Upon clicking on the "Login" button, a graphql query is sent to my server which responds with a Jwt. Upon reception of the jwt, the application sets the cookie with
document.cookie = ...
In my Cypress test, I check the token in the following way:
Then("sa session s'ouvre pour {SessionDurationType}", expectedDuration => {
cy.get('#graphql').then(() => {
cy.wait(1000)
cy.getCookie('token').then(cookie => {
const tokenDuration = getTokenDuration(cookie.value)
expect(tokenDuration.asSeconds()).to.equal(expectedDuration.asSeconds())
})
})
})
With cy.get('#graphql'), I am waiting for the graphql query to return a response. The alias is defined like this:
cy.stub(win, 'fetch', fetch).as('graphql')
Upon reception, the application sets the cookie.
My problem is that I am not fond of the following call:
cy.wait(1000)
Without that call, I always get an undefined cookie.
Is there a way to get that cookie within some time that might be much less than 1000 ms? I tried many things without success...

You must write a recursive promise function, try the following
function checkCookie() {
// cy.getCookie returns a thenebale
return cy.getCookie('token').then(cookie => {
const tokenDuration = getTokenDuration(cookie.value);
// it checks the seconds right now, without unnecessary waitings
if(tokenDuration.asSeconds() !== expectedDuration.asSeconds()) {
// waits for a fixed milliseconds amount
cy.wait(100);
// returns the same function recursively, the next `.then()` will be the checkCookie function itself
return checkCookie();
}
// only when the condition passes returns a resolving promise
return Promise.resolve(tokenDuration.asSeconds());
})
}
Then("sa session s'ouvre pour {SessionDurationType}", expectedDuration => {
cy.get('#graphql').then(() => {
checkCookie()
.then(seconds => {
expect(seconds).to.equal(expectedDuration.asSeconds())
})
})
})
Note that the function must be improved because
I didn't parametrize the expectedDuration etc. (it's out of the scope of showing you how to do that)
it waits forever without a loop counter check
But it works (I checked in another context before replying to you) and if you have some more troubles please share a "working" GitHub repo so I can clone and check it with your own solution.
Let me know if it isn't enough clear 😉
UPDATE
We (me and Tommaso) have written a plugin to help you with this kind of checks, its name is cypress-wait-until.
Please thank the Open Source Saturday community for that, we developed it during one of them Saturdays 😊

I dont like the timeout in this i have to say for dom changes. I have come up with this solution based on #NoriSte Answer together with DomMutation Observers.
getFileUploadItem().get(".upload-item--state i")
.should("have.class", "ngx-fileupload-icon--start")
.then(item => {
const iconEl = item.get(0);
const states: string[] = [];
return new Promise((resolve, reject) => {
const observer = new MutationObserver((mutations: MutationRecord[]) => {
const mutationEl = mutations[0].target as HTMLElement;
const className = mutationEl.getAttribute("class");
states.push(className);
if (className === "ngx-fileupload-icon--uploaded") {
resolve(states);
}
});
observer.observe(iconEl, {
subtree: true,
attributes: true,
attributeFilter: ["class"]
});
});
})
.then((value) => expect(value).to.deep.equal(
["ngx-fileupload-icon--progress", "ngx-fileupload-icon--uploaded"])
);

Based on #NoriSte's answer, I came up with the following working code:
function awaitNonNullToken(elapsedTimeInMs = 0) {
let timeDeltaInMs = 10
if (elapsedTimeInMs > Cypress.env('timeoutInMs')) {
return Promise.reject(new Error('Awaiting token timeout'))
}
return getTokenCookie().then(cookie => {
if (cookie === null) {
cy.wait(timeDeltaInMs)
elapsedTimeInMs += timeDeltaInMs
return awaitNonNullToken(elapsedTimeInMs)
}
return Promise.resolve(cookie.value)
})
}
I transformed that into an ES6 class that I find a bit more elegant:
class TokenHandler {
constructor () {
this.TIME_DELTA_IN_MS = Cypress.env('timeDeltaInMs')
this.TIMEOUT_IN_MS = Cypress.env('timeoutInMs')
this.elapsedTimeInMs = 0
}
getToken () {
if (this.elapsedTimeInMs > this.TIMEOUT_IN_MS) {
return Promise.reject(new Error('Awaiting token timeout'))
}
return getTokenCookie().then(cookie => {
if (cookie === null) {
cy.wait(this.TIME_DELTA_IN_MS)
this.elapsedTimeInMs += this.TIME_DELTA_IN_MS
return this.getToken()
}
return Promise.resolve(cookie.value)
})
}
}
and reworked my step like this:
cy.get('#graphql').then(() => {
const handler = new TokenHandler
handler.getToken().then(token => {
const tokenDuration = getTokenDuration(token)
expect(tokenDuration.asSeconds()).to.equal(expectedDuration.asSeconds())
})
})
This is working perfectly, thanks.

Javascript 'then' not working as expected in Ripple-lib calls

I'm trying to create a simple example of payments over the XRPL using Ripple-lib. The idea is to send several payments to different accounts stored in an array. I've made it kind of work in a different way as it is expected, but when using the 'then' method (as the docs recommend) does not work at all.
I'm a total newbie to Javascript so I don't have a good grasp on the language nor asyncronous coding and promises. When using the 'then' paradigm, the code stops working and no output can be seen in the console. This is the code I'm currently using. In the comments inside the 'SendXRP' function I explain the problem. How can this be re-arranged? Between the two ways, what is the proper one to code it?
'use strict';
const RippleAPI = require('ripple-lib').RippleAPI;
const sender = 'r*********************************';
const secret = 's****************************';
const destinations = ['r*********************************',
'r*********************************',
'r*********************************'];
const amount = 5;
// Instantiate Ripple API
const api = new RippleAPI({
server: "wss://s.altnet.rippletest.net:51233"
});
run();
async function sendXRP(amount, fee, destination, memo) {
// Update amount
amount = (amount - fee).toString();
// Build payment
const payment = {
source: {
address: sender,
maxAmount: {
value: amount,
currency: 'XRP'
}
},
destination: {
address: destination,
amount: {
value: amount,
currency: 'XRP'
}
},
memos: [
{
data: memo
}
]
};
// Build instuctions
const instructions = {
maxLedgerVersionOffset: 5
};
console.log('Sending ' + amount + ' to ' + destination);
// THIS KIND OF WORKS FOR NOW
// Prepare the payment
const preparedTX = await api.preparePayment(sender, payment, instructions);
// Sign the payment
const signedTX = api.sign(preparedTX.txJSON, secret);
// Submit the payment
const result = await api.submit(signedTX['signedTransaction']);
// Return TX hash on successful TX
if ('resultCode' in result && result['resultCode'] == 'tesSUCCESS') {
return signedTX.id;
} else {
return null;
}
// THIS IS MORE SIMILAR TO HOW IT IS DONE IN THE DOCS! NOT WORKING!
// ALSO, HOW DO I RETURN THE RESULT OF API.SIGN TO THE MAIN FUNCTION?
// Prepare the payment
// api.preparePayment(sender, payment, instructions).then(preparedTX => {
// // Sign the payment
// api.sign(preparedTX.txJSON, secret).then(signedTX => {
// // Submit the payment
// api.submit(signedTX['signedTransaction']);
// })
// }).catch(console.error);
}
function run() {
// Connect to Ripple server
api.connect().then(() => {
return api.getFee();
}).then(async fee => {
for (var i in destinations) {
var hash = await sendXRP(amount, Number(fee), destinations[i], 'memotext');
console.log(hash);
}
}).then(() => {
return api.disconnect();
}).catch(console.error);
}

Could it be that some of the transactions failed to send? If it failed, the result variable from sendXRP should have the txresult, but since you returned null if the result code is not tesSUCCESS, it doesn't return the result information.
const result = await api.submit(signedTX['signedTransaction']);
if ('resultCode' in result && result['resultCode'] == 'tesSUCCESS') {
return signedTX.id;
} else {
return null;
}
Before, when I tried submitting transactions consecutively, it would fail and return error code tefPAST_SEQ.
"The sequence number of the transaction is lower than the current sequence number of the account sending the transaction." from https://developers.ripple.com/tef-codes.html
I recommend removing the if('resultCode' in result...) block and check the transaction result. If the transactions failed with tefPAST_SEQ error, my solution to this is set the account sequence in instructions manually or add setTimeOut after each submit.