I have a piece of code where I consume events from RabbitMQ and save the events(3 types of events A,B,C) into 2 different databases A,B. I can push into Database A without any problem but I need to wait for no of events to be at least 100 to push events of type B & C into database or until the code is trying to fill up the queue for last 5 minutes from the point saveToDb in invoked. I am not able to figure out how to wait for events for B and C and then save data in database.
Note that Event A will go into Database A and Event B,C will go into Database B.
I have written following piece of code.
import { Channel, ConsumeMessage } from 'amqplib';
const BATCH_SIZE = 100;
var eventBQueue = [];
var eventAQueue = [];
const shiftElements = (message) => {
if ( message.length >= BATCH_SIZE) {
const batch= message.splice(0, BATCH_SIZE);
return batch;
}
return message;
}
const saveToDb = async (messages, database) => {
const eventsA = filterEventsA(messages);
const eventsB = filterEventsB(messages);
const eventsC = filterEventsC(eventsB);
const promises = [];
promises.push(databaseAsync.publish(eventsC));
if (eventBQueue.length < BATCH_SIZE) {
eventBQueue.push.apply(eventBQueue, eventsB);
}
else {
var eventsBBatched = shiftElements(eventBQueue);
promises.push(database.publish(eventsBBatched, EVENTS_TABLE_A));
}
if (eventAQueue.length < BATCH_SIZE) {
eventAQueue.push.apply(eventAQueue, eventsA);
}
else {
var eventsABatched = shiftElements(eventAQueue);
promises.push(database.publish(eventsABatched, EVENTS_TABLE_B));
}
return new Promise((resolve, reject) => {
Promise.all(promises).then(resolve).catch(reject);
});
}
export const process = async (database,
rabbitmq): Promise<void> => {
return new Promise((resolve, _) => {
rabbitmq.consume(async (channel, message: ConsumeMessage) => {
const messages = somefunction(message);
await saveToDb(messages,database)
.then(_ => {
try {
channel.ack(message)
} catch (error) {
}
})
.catch((error) => {
try {
console.error('error');
channel.ack(message)
} catch (error) {
}
});
});
somefunction(resolve)
});
}
Now I want to add some condition in if where no of events < Batch_SIZE to wait for data from rabbitMQ and to save in database when eventAQueue and eventBQueue has adequate size or there is a time limit waiting for this data. But I am not sure how to add it.
Related
This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.
If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}
I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?
I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.
You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price
I'm currently implementing a WebSocket connection and I'm using a command pattern approach to emit some messages according to the command that users execute.
This is an abstraction of my implementation:
let socketInstance;
const globalName = 'ws'
const globalObject = window[globalName];
const commandsQueue = isArray(globalObject.q) ? globalObject.q : [];
globalObject.q = {
push: executeCommand
};
commandsQueue.forEach(command => {
executeCommand(command);
});
function executeCommand(params) {
const actions = {
create,
send
};
const [command, ...arg] = params;
if (actions[command]) {
actions[command](arg);
}
}
function send([message]) {
socketInstance.send(message);
}
function create([url]) {
socketInstance = new WebSocket(url);
}
In order to start sending messages, the user should be run:
window.ws.push('create', 'ws://url:port');
window.ws.push('send', 'This is a message');
The problem that I have is the connection is async, and I need to wait until the connection is done to continue to the next command. Is it a good idea to implement an async/await in commandsQueue.forEach or an iterator is a better approach? What other best approaches do you recommend?
The solution that I'm using right now is: I created an empty array of messages at the beginning and then every time that I call the send command I verify if the connection wasn't opened and I added to this array.
Something like that:
const messages = [];
let socketInstance;
let isConnectionOpen = false;
const globalName = "ws";
const globalObject = window[globalName];
const commandsQueue = isArray(globalObject.q) ? globalObject.q : [];
globalObject.q = {
push: executeCommand,
};
commandsQueue.forEach((command) => {
executeCommand(command);
});
function executeCommand(params) {
const actions = {
create,
send,
};
const [command, ...arg] = params;
if (actions[command]) {
actions[command](arg);
}
}
function send([message]) {
if (isConnectionOpen) {
socketInstance.send(message);
} else {
messages.push(message);
}
}
function onOpen() {
isConnectionOpen = true;
messages.forEach((m) => {
send([m]);
});
messages.length = 0;
}
function create([url]) {
socketInstance = new WebSocket(url);
socketInstance.onopen = onOpen;
}
Working on dialog component with angular js and now I find out that my function is subscribed and in if condition do not quit method, but continuously executing another function afterClosed() , here is example of code :
openCreateNewContentDialog(): void {
const oldData = this.dataSource.data;
const dialogConfig = AppConstants.matDialogConfig();
const dialog = this.dialog.open(LicenseDialogComponent, dialogConfig);
dialog.beforeClosed().subscribe(licenceDate => {
for (const datesToCheck of oldData) {
const newDateFrom = new Date(licenceDate.expirationDateFrom);
const oldDateTo = new Date(datesToCheck.expirationDateTo.toString());
if (newDateFrom <= oldDateTo) {
// console.log('return?');
return;
}
}
});
dialog.afterClosed().subscribe(licence => {
if (licence) {
this._value.push(licence);
this.dataSource.data = this.value;
this.change();
}
});
}
What is the best and optimized way to unsubscribe beforeClosed() function?
So from your description, I understand that you dont want a second subscription to happen if the condition in the first subscriber is true, right? But you subscription will happen anyway because you instantiated it in the method, the code in the subscribe() it's just a callback. So if you dont want a lot of rewriting I will suggest storing
subscriptions in variables, so you will have an access to them and can unsubscribe at any time.
openCreateNewContentDialog(): void {
const oldData = this.dataSource.data;
const dialogConfig = AppConstants.matDialogConfig();
const dialog = this.dialog.open(LicenseDialogComponent, dialogConfig);
const beforeClosed = dialog.beforeClosed().subscribe(licenceDate => {
for (const datesToCheck of oldData) {
const newDateFrom = new Date(licenceDate.expirationDateFrom);
const oldDateTo = new Date(datesToCheck.expirationDateTo.toString());
if (newDateFrom <= oldDateTo) {
// console.log('return?');
afterClosed.unsubscribe();
return;
}
}
});
const afterClosed = dialog.afterClosed().subscribe(licence => {
if (licence) {
this._value.push(licence);
this.dataSource.data = this.value;
this.change();
}
});
}
I hope it helps! Also you can try https://www.digitalocean.com/community/tutorials/angular-takeuntil-rxjs-unsubscribe if you have to handle multiple subscriptions.
I'm trying to improve a firestore get function, I have something like:
return admin.firestore().collection("submissions").get().then(
async (x) => {
var toRet: any = [];
for (var i = 0; i < 10; i++) {
try {
var hasMedia = x.docs[i].data()['mediaRef'];
if (hasMedia != null) {
var docData = (await x.docs[i].data()) as MediaSubmission;
let submission: MediaSubmission = new MediaSubmission();
submission.author = x.docs[i].data()['author'];
submission.description = x.docs[i].data()['description'];
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
submission.media = mediaRef.data() as MediaData;
toRet.push(submission);
}
}
catch (e) {
console.log("ERROR GETTIGN MEDIA: " + e);
}
}
return res.status(200).send(toRet);
});
The first get is fine but the performance is worst on the line:
var mediaRef = await admin.firestore().doc(docData.mediaRef).get();
I think this is because the call is not batched.
Would it be possible to do a batch get on an array of mediaRefs to improve performance?
Essentially I have a collection of documents which have foreign references stored by a string pointing to the path in a separate collection and getting those references has been proven to be slow.
What about this? I did some refactoring to use more await/async code, hopefully my comments are helpful.
The main idea is to use Promise.all and await all the mediaRefs retrieval
async function test(req, res) {
// get all docs
const { docs } = await admin
.firestore()
.collection('submissions')
.get();
// get data property only of docs with mediaRef
const datas = await Promise.all(
docs.map(doc => doc.data()).filter(data => data.mediaRef),
);
// get all media in one batch - this is the important change
const mediaRefs = await Promise.all(
datas.map(({ mediaRef }) =>
admin
.firestore()
.doc(mediaRef)
.get(),
),
);
// create return object
const toRet = datas.map((data: MediaSubmission, i) => {
const submission = new MediaSubmission();
submission.author = data.author;
submission.description = data.description;
submission.media = mediaRefs[i].data() as MediaData;
return submission;
});
return res.status(200).send(toRet);
}