We Keep Coding

Getting consistent data from API in my node server without breaking down [duplicate]

This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.

If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}

Google Apps Script Working on backend but not on sheets

I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?

I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.

You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price

How do I create Async code?

Hey so im making a leaderboard for a discord bot using discord.js And I want to display users by their names instead of their ID's so using discord.js I use the function .fetchUser(ID)
.fetchUser(ID) is a promise which can take a some of time depending on the bandwidth.
So because discord.js uses a promise my code is no longer Async, I thought that by putting the code in a promise it would run Async.
And I was wrong.
my code:
//This is ran inside a .prototype function so (this) is defined
return new Promise((resolve, reject) => {
this.list = [];
//users is an object with user's IDs as the key
//Currently it only has one key in it (mine)
for (let i in users) {
let pos = 0;
let score = this.getScore(users[i]);
if (score === 0) {
client.fetchUser(i).then((user)=> {
console.log(`pushed`);//logs way after the "finish" is logged
this.list.push([user.username.substring(0,13), score])
});
continue;
}
for (let h = 0; h < this.list.length; h++) {
if (score >= this.list[h][1]) {
pos = h;
break;
}
}
client.fetchUser(users[i].id).then((user) => {
this.list.splice(pos, 0, [user.username.substring(0,13), score])
})
}
console.log(`Finished: `+this.list.length);
resolve(this.list);
})

You have to chain off of Promises you receive. Client#fetchUser() returns a Promise which you are waiting on, but not enough. You have to propagate up Promises. If something in your function call chain is asynchronous, you should consider the whole chain async.
You fill this.list from within the fetchUser(...).then(...), which isn't necessarily bad, as long as you don't try to use list until after fetchUser's resolution chain is done. You aren't doing that; you immediately resolve(this.list).
Consider this abbreviated form of your original function:
return new Promise((resolve, reject) => {
this.list = [];
for (let i in users) {
// A promise is created right here
client.fetchUser(i).then((user) => {
// This will populate list AFTER the then callback
this.list.push([user.username.substring(0, 13), score])
});
}
// You aren't waiting until the promise created by fetchUser completes
resolve(this.list);
})
this.list can't be considered "complete" until all the users involved have had their profiles loaded and their scores retrieved. Considering that, we can use Promise.all() which takes an array of Promises and then resolves once all of the provided promises have resolved. So to wait that way, we would do something like this, which still isn't ideal, but waits correctly:
return new Promise((resolve, reject) => {
this.list = [];
// This is an array of Promises
const discordUsersPromise = users.map(user => client.fetchUser(user));
// Wait till all the fetchUser calls are done
const listIsPopulatedPromise = Promise.all(discordUsersPromise).then(dUsers => {
// This replaces your for (let i in users) {}
Object.entries(users).forEach((user, idx) => {
const score = this.getScore(user);
const discordUser = dUsers[idx];
this.list.push([discordUser.username.substring(0, 13), score])
});
});
// We still have to wait for the list to be completely populated
return listIsPopulatedPromise.then(() => this.list);
})
Consider this implementation. I have made some assumptions about your code since you use this.list but don't include what this is an instance of, but most of it should be the same:
/**
* Object to composite certain user properties
* #typedef {RealUser}
* #property {String} user The local string for the user
* #property {User} realUser The user that Discord gives us
* #property {Number} score The score this user has
*/
/**
* Class to encapsulate user and score and data
*/
class Game {
/**
* Constructs a game
*/
constructor() {
/**
* The users we are keeping score of
* #type {Object}
*/
this.users = {};
}
/**
* Get the score of a particular user
* #param {String} user User to get score of
* #returns {Number} User's score
*/
getScore(user) {
return this.users[user] || 0;
}
/**
* Get a composite of users and their status
* #param {String[]} users The users to put on our leaderboard
* #returns {Promise<RealUser[]>} Sorted list of users that we included in our leaderboard
*/
getLeaderBoard(users) {
// Map all the users that we are given to Promises returned bye fetchUser()
const allRealUsersPromise = Promise.all(users.map(user => client.fetchUser(user)
/*
* Create an object that will composite the string that we use
* to note the user locally, the Discord User Object, and the
* current score of the user that we are tracking locally
*/
.then(realUser => ({
user,
realUser,
score: this.getScore(user)
}))));
/*
* Once we have all the data we need to construct a leaderboard,
* we should sort the users by score, and hand back an array
* of RealUsers which should contain all the data we want to
* print a leaderboard
*/
return allRealUsersPromise
.then(scoredUsers => scoredUsers.sort((a, b) => a.score - b.score));
}
/**
* Prints out a leaderboard
* #param {String[]} users The users to include on our leaderboard
*/
printLeaderBoard(users) {
// Go get a leaderboard to print
this.getLeaderBoard(users).then(sortedScoredUsers => {
// Iterate our RealUsers
sortedScoredUsers.forEach((sortedScoredUser, idx) => {
const username = sortedScoredUser.realUser.username;
const score = sortedScoredUser.score;
// Print out their status
console.log(`${username.substring(0, 13)} is in position ${idx + 1} with ${score} points`);
});
});
}
}
const game = new Game();
game.users["bob"] = 5;
game.users["sue"] = 7;
game.users["tim"] = 3;
game.printLeaderBoard(Object.keys(game.users));

Javascript function does not exit the function wehn the condtion failed

I have following code which process a queue and I need to exist the function when there are no messages in the queue and there is no enough time to process more messages. My problem is, it doesn't jump out of the function upon failing the condition and I think it's due to that this a recursive function but I cannot figure it out.
/**
* Check if there is enough time to process more messages
*
* #param {} context
* #returns {boolean}
*/
async function enoughTimeToProcess(context) {
return context.getRemainingTimeInMillis() > 230000;
}
/**
* Consume the queue and increment usages
*
* #param context
*
* #returns {boolean}
*/
async function process(context) {
const messagesPerRequest = queueConst.messagesPerRequest;
const messagesToBeDeleted = [];
const queue = new queueClient();
const messages = await queue.getMessages(messagesPerRequest);
if (messages === undefined) {
if (await enoughTimeToProcess(context) === true) {
await process(context);
} else {
return false;
}
}
const responses = messages.map(async(messageItem) => {
const messageBody = JSON.parse(messageItem.Body);
const parsedMessage = JSON.parse(messageBody.Message);
const accountId = parsedMessage[0].context.accountId;
let code = parsedMessage[0].context.code;
// Our DB support only lowercase characters in the path
code = code.toLowerCase();
const service = parsedMessage[0].name;
const count = parsedMessage[0].increment;
const storageResponse = await incrementUsage(
{ storageClient: storage, code, accountId, service, count }
);
if (storageResponse) {
messagesToBeDeleted.push({
Id: messageItem.MessageId,
ReceiptHandle: messageItem.ReceiptHandle,
});
}
return 1;
});
const processedMessages = await Promise.all(responses);
const processedMessagesCount = processedMessages.length;
if (messagesToBeDeleted.length > 0) {
console.log(`${processedMessagesCount} messages processed.`);
await queue.deleteMessageBatch(messagesToBeDeleted);
}
if (await enoughTimeToProcess(context) === true) {
await process(context);
}
return true;
}

I think the problem can be when messages are undefined and there is still enough time, because the recursive function is going to be called infinite times, because it always accomplishes both conditions, and probably it exceeds the available resources.
Try to sleep some time before calling process function again, just to be sure it is the problem

how to clone the mongoose query object in javascript

I am facing the problem of clone of the mongoose query object .Javascript the copy the one object into another object by call-by-ref but in my project there is scenario i need to copy one object into another object by call-by-value.
var query=domain.User.find({
deleted: false,
role: role
})
var query1=query;
I have the scenario change in the query object is not reflected in query1. I google and try so many way to clone the object but it does't work.The query object is used in another function for pagination and query1 object is used for count query.
1.I used to Object.clone(query1) error Object.clone is not function
2.I used Object.assign(query1) but it does't works fine.
3.I used other so many ways can anybody help me to sort this problem

Alternative solution using merge method:
const query = domain.User.find({
deleted: false,
role: role
}).skip(10).limit(10)
const countQuery = query.model.find().merge(query).skip(0).limit(0)
const [users, count] = await Promise.all([query, countQuery.count()])

you are trying to clone a cursor, but it is not the right approach, you probably just need to create another
like this:
var buildQuery = function() {
return domain.User.find({
deleted: false,
role: role
});
};
var query = buildQuery();
var query1 = buildQuery();

This is work for me:
const qc = sourceQuery.toConstructor();
const clonedQuery = new qc();
This code work in pagination function where sourceQuery passed as parameter and i dont known what models used. Also it work with aggregations and complex queries.
public async paging(
query: mongoose.DocumentQuery<mongoose.Document[], mongoose.Document>,
params,
transformer: any = null
) {
let page = Number(params.page);
if (!page) page = 1;
let page_size = Number(params.count);
if (!page_size) page_size = 100;
const qc = query.toConstructor();
const cq = new qc();
return cq.countDocuments().exec()
.then(async (total) => {
const s = params.sort;
if (s) {
query.sort(s);
}
query.limit(page_size);
query.skip(page_size * (page - 1));
let results = await query.exec();
if (transformer) {
results = await Promise.all(results.map((i) => transformer(i)));
}
const r = new DtoCollection();
r.pages = Math.ceil(total / page_size);
r.total = total;
(r.results as any) = results;
return r;
});
}

Sergii Stotskyi's answer works just fine and is very elegant, except that count is deprecated.
countDocuments or estimatedDocumentCount should be used instead.
However, this causes the error the limit must be positive. We can walk around this by set limit to a large integer.
const query = domain.User.find({
deleted: false,
role: role
}).skip(10).limit(10)
const countQuery = query.model.find().merge(query).skip(0).limit(Number.MAX_SAFE_INTEGER)
const [users, count] = await Promise.all([query, countQuery.countDocuments()])

Since mongoose v6 you can use Query.prototype.clone
E.g. for your code snippet:
const query = domain.User.find({
deleted: false,
role: role
})
const query1 = query.clone();