We Keep Coding

Parallel processing in node js function

I have a function that has more than 1400+ crypto pairs and I have to send an API against each pair and store the trades. Now each pair takes 3-4 seconds hence the whole function takes a lot of time. I am getting the pairs from my DB and I am storing trade data in my DB as well. I need to process the pairs in parallel so the trades from the pair in the beginning don't miss because the function is not processing.
This is my current function:
const getTrades = async () => {
let page = 1;
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
const coinResult = await db.query("SELECT * FROM coins");
let coinsInfo = coinResult.rows;
coinsInfo = coinsInfo.flat();
for (const pair of pairs) {
let biggestTrade = [];
const response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of filtered) {
let priceUSD = 0;
let baseAssetIcon = "null";
for (const coin of coinsInfo) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image_url;
}
if (priceUSD > 0 && baseAssetIcon != "null") {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("TRADE ADDED");
}
}
}
console.log("PAIRS ARE OVER");
};
pairs has over 1400 entries and this is the one where are looping through.

depends on how many servers you are running this function on.
if it's one single machine, use worker_threads, basically run the same function in separate threads to achieve parallelization, but to be honest, 1400 pairs are a lot, each for 3-4 seconds, so total around 1-2hrs per run if in serial. Depending on your machines, if you have 8 cores, it might reduce the time by 8 folds but still leave you like around 10 minutes. and cloud service usually charge a lot more for instances that have more cpu cores.
if it's multiple machines, use a master and a queue to push new pairs to each worker machine and for each worker machine, you can also generate multiple threads for each machine, in that way you can scale horizontally, and it's possible to finish the run in seconds. in this situation, each machine you can get the cheap one from cloud providers.
so depends on your requirements, if you wanna super fast, you gotta add more machines.

botpress - increment vlaue

I am trying to get a custom action running to simply incrementing a value on passing a specific node on the flow.
My custom actions looks like this:
function action(bp: typeof sdk, event: sdk.IO.IncomingEvent, args: any, { user, temp, session } = event.state) {
/** Your code starts below */
let i = undefined
const p = new Promise((resolve, reject) => {
if (i === undefined) {
resolve((i = 0))
} else if (i >= 0) {
resolve(i + 1)
} else {
reject('i cannot be < 0')
}
})
const runCount = async () => {
try {
const counter = await p
i = counter
return (session.count = counter)
} catch (err) {
console.log(err)
}
}
return runCount()
/** Your code ends here */
}
When I runCount() variable i will be set to 0. But then, after in rerun runCount() it does not increment further.
What do I need to do to save the variable so it increments on every runCount() call.
Greetings
Lorenz

I just managed to solve the problem.
I had to declare i = session.count at the beginning.
Now it gets the value out of the session state and increments the state on every call.
Maybe someone gets some help out of this.
Lorenz

Why does firebase cloud-function javascript promise run more than the number of loop invocations?

I have a cloud function that is triggered when a sale/purchase is committed into firestore. This function's purpose is to update the inventory level centrally.
The function works just fine if I'm updating an item's inventory at only 1 warehouse, but doing so for multiple warehouses has unexpected behavior. I'm looping through all the warehouses that are affected to calculate the total inventory level changes, and every iteration kicks-off a javascript promise.
The problem seems to occur with the way the promises are invoked. E.g: if I want to update 3 warehouses and loop 3 times, somehow 5 promises are being kicked-off. This is visible through the logs. I've researched similar questions here, but the solutions were suggested while firestore was still in beta and might not be the right way forward. (Firestore transactions getting triggered multiple times resulting in wrong data)
Here is the code
export const onTransactionCreate = functions.firestore
.document('/companies/{companyId}/sub_transactions/{transId}')
.onCreate(async (snapshot, context) => {
const transId = context.params.transId
const stock_transaction: IStockTransaction = <IStockTransaction>snapshot.data()
const trans_type: TRANS_TYPE = stock_transaction.trans_type
const promises: any[] = []
stock_transaction.lineItems.forEach((element, index) => {
const ITEM_GUID = element.item_guid
const is_increasing = isIncreasingTransaction(element.line_trans_type)
const delta_stock = element.qty_transaction * (is_increasing ? 1 : -1)
const TARGET_BRANCH_ID = element.target_branch_guid
const itemRef = db.collection(FIRESTORE_PATHS.COL_COMPANIES).doc(companyId).
collection(FIRESTORE_PATHS.SUB_COMPANIES_ITEMS).
doc("" + ITEM_GUID)
const item_promise = db.runTransaction(async t => {
try {
const item_doc = await t.get(itemRef)
const item_branch_quantities: IBranchQuantity[] = (item_doc.data()!.branch_quantities || new Array())
const item_branch_ids: string[] = (item_doc.data()!.available_branch_ids || new Array())
const branch_index = item_branch_ids.indexOf(TARGET_BRANCH_ID)
console.log(`${transId} Line Item ${index}, after document.get(), search branch index: ${branch_index}`)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty = prev_qty.quantity + delta_stock
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
console.log(`${transId} Line Item ${index} Updating qty # item ${delta_stock}, prev qty ${prev_qty.quantity}`)
} else {
item_branch_ids.push(TARGET_BRANCH_ID)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: TARGET_BRANCH_ID,
quantity: delta_stock
})
console.log(`${transId} Line Item ${index} Adding qty # item ${delta_stock}`)
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
available_branch_ids: item_branch_ids
})
} catch (err) {
throw new Error(err)
}
})
promises.push(item_promise)
});
return Promise.all(promises)
})

we have found the solution by reading this article.
A transaction consists of any number of get() operations followed by any number of write operations such as set(), update(), or delete(). In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
lineItems.forEach(element => {
const delta_transaction = element.qty * (isLineTransIncrease(element.line_trans_type) ? 1 : -1)
const itemRef = db.collection('companies').doc(companyId).collection('sub_items').doc("" + element.item_guid)
const p = db.runTransaction(t => {
return t.get(itemRef)
.then(doc => {
let item_branch_quantities: IBranchQuantity[] = doc.data()!.branch_quantities
let item_branch_ids: string[] = doc.data()!.available_branch_ids
if (!item_branch_quantities)
item_branch_quantities = new Array()
if (!item_branch_ids)
item_branch_ids = new Array()
const branch_index = item_branch_ids.indexOf(current_branch_id)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty: number = prev_qty.quantity + delta_transaction
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
} else {
item_branch_ids.push(current_branch_id)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: current_branch_id,
quantity: delta_transaction
})
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
branch_ids: item_branch_ids
})
})
})
item_update_transactions.push(p)
});
return Promise.all(item_update_transactions)
})
function isLineTransIncrease(line_trans: number): boolean {
return (line_trans === 1) || (line_trans === 2)
}

Why does my forEach() loop only execute once?

I seem to have encountered a problem while looping through an array. The loop seems to only execute once, no matter the size of the array. I tried using different methods of looping and the error still persists.
As background information, I'm trying to make a bot with which users can award each other points. Everything else seemed alright. The only issue is that I wish to set up a maximum amount of points one user can give to another in a day, and I'm having problems looping through the array which stores this information.
These are the relevant parts of my code:
var timer = []; //Timer stores the values.
const getTimerSenderIdTable = (id) => {
let found = false;
timer.forEach(function(dat) { // This is the problematic loop.
if (dat.id === id) {
found = dat;
}
})
console.log("loop end, results: " + found);
return found;
};
const timerManager = (senderId, targetId, pointSurp) => { //All arguments are integers.
let d = new Date()
if (getTimerSenderIdTable("date") !== d.getDate()) {
timer = [];
timer.push({"id":"date", "time":d.getDate()});
if (getTimerSenderIdTable("date")) {
if (getTimerSenderIdTable(senderId)) {
console.log("path 1");
} else {
console.log("path 2");
timer.push({"id":senderId, [targetId]:pointSurp});
}
}
} else {
if (getTimerSenderIdTable("date")) {
if (getTimerSenderIdTable(senderId)) {
console.log("path 3");
} else {
console.log("path 4");
timer.push({"id":senderId, [targetId]:pointSurp});
}
}
}
console.log(timer)
};
*Edit:
Thank you for your comments. Here is an example:
Calling timerManager(123456, 654321, 3) will produce the following output:
loop end, results: false
loop end, results: [object Object]
loop end, results: false
path 2
[ { id: 'date', time: 28 }, { '654321': 3, id: 123456 } ]
(This is a repost from comments. My appologies.)

It seems because of this line
if (getTimerSenderIdTable("date") !== d.getDate()) {
timer = [];
This will empty the array and next lines of code will only push single element

as #mbojko has pointed out, you'll want to use the find method for returning the found obj inside getTimerSenderIdTable function, like this
const getTimerSenderIdTable = (id) => {
return timer.find(item => item.id === id});
};

Javascript code portion not running in the right order

Im creating a trading bot on Javascript (I got no prior experience with this language). The way the trailing stoploss function runs is:
Websocket receives current market price
"active" is a boolean variable, if true, run the code
If price rises a %, cancel old stoploss and add a new one higher.
The problem I am getting is the code doesn't run in the right order.
If you look at the picture, I don't understand why the blue box still executes if active is false. And because the program runs in the wrong order at times, the websocket stops or acts how it isn't supposed to.
This is my trailing stoploss websocket code:
function binanceTrailingSLOrder(symbol, orderId, quantity, oldPrice, percentage, active) {
const clean_trade = client.ws.trades([symbol], trade => { //run websocket
var livePrice = parseFloat(binance_symbols[symbol]["close"]); //set new price to live price
if (active == true) {
binanceCheckOrderStatus(symbol, orderId).then(r => {
switch (r.status) {
case "PENDING":
if (livePrice >= (oldPrice * ((100 + percentage) / 100)) && active == true) {
active = false;
binanceCancelOrder(symbol, orderId).then((r4) => { //Cancel previous SL
var newSL = livePrice * ((100 - percentage) / 100);
binanceStopOrder(symbol, 'SELL', r4.origQty, newSL, newSL).then((r5) => { //Set new SL
orderId = r5.orderId; quantity = r5.origQty; oldPrice = r5.price;
active = true;
}).catch((err) => {
console.log(err);
});
});
}
break;
default:
break;
}
});
}
});
}
Check order status function:
//Get specific order status
function binanceCheckOrderStatus(symbol, orderId) {
if(!orderId){
console.log("order Id not found");
return false;
} else {
var client = loadBinanceKeys2();
return client.getOrder({
symbol: symbol,
orderId: orderId,
recvWindow: 1000000
}).then((order) => {
return order;
}).catch((err) => {
console.log(err);
});
}
}

Javascript is asynchronous in nature. The function binanceCheckOrderStatus() returns a promise. The execution engine will call this function, and then move on to the next line. The code block after .then(r => only executes after the binanceCheckOrderStatus's getOrder() is completed. Now in this time period, the active may have become false in other .then() requests. It may be confusing for new developers. Since you are using lot of .then() in your code, you have to understand that the .then() part is only executed after the function before .then() completes the execution. So the function taking less time will execute it's .then() part before the others. So in short, you CANNOT control the order in this scenario unless you know how much time every function will take, which is probably impossible to confirm. For overcoming this problem, you have to use async/await. Or, you need to change your logic so it is less dependent on that deep level promises.
I am not very sure about what you are trying to achieve here, but here is the idea about how you can solve the ordering problem. It is just a reference code, I have not tested it. Just an idea on how you can hold your threads to make sure your code runs in an order using async/await.
async function binanceTrailingSLOrder(symbol, orderId, quantity, oldPrice, percentage, active) {
const clean_trade = client.ws.trades([symbol], async trade => { //run websocket
var livePrice = parseFloat(binance_symbols[symbol]["close"]); //set new price to live price
if (active == true) {
try {
const order = await binanceCheckOrderStatus(symbol, orderId);
if (!order) {
throw new Error('order not found')
}
switch (order.status) {
case "PENDING":
if (livePrice >= (oldPrice * ((100 + percentage) / 100)) && active == true) {
active = false;
const r4 = await binanceCancelOrder(symbol, orderId);
if (r4) {
var newSL = livePrice * ((100 - percentage) / 100);
var r5 = binanceStopOrder(symbol, 'SELL', r4.origQty, newSL, newSL);
if (r5) {
orderId = r5.orderId; quantity = r5.origQty; oldPrice = r5.price;
active = true;
}
}
}
break;
default:
break;
}
}
catch(error) {
console.log('error found: ', error);
}
}
});
}
async function binanceCheckOrderStatus(symbol, orderId) {
if(!orderId){
console.log("order Id not found");
return false;
} else {
var client = loadBinanceKeys2();
return new Promise((resolve, reject) => {
client.getOrder({
symbol: symbol,
orderId: orderId,
recvWindow: 1000000
}).then((order) => {
resolve(order);
}).catch((err) => {
reject(err);
});
});
}
}