We Keep Coding

Parallel processing in node js function

I have a function that has more than 1400+ crypto pairs and I have to send an API against each pair and store the trades. Now each pair takes 3-4 seconds hence the whole function takes a lot of time. I am getting the pairs from my DB and I am storing trade data in my DB as well. I need to process the pairs in parallel so the trades from the pair in the beginning don't miss because the function is not processing.
This is my current function:
const getTrades = async () => {
let page = 1;
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
const coinResult = await db.query("SELECT * FROM coins");
let coinsInfo = coinResult.rows;
coinsInfo = coinsInfo.flat();
for (const pair of pairs) {
let biggestTrade = [];
const response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of filtered) {
let priceUSD = 0;
let baseAssetIcon = "null";
for (const coin of coinsInfo) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image_url;
}
if (priceUSD > 0 && baseAssetIcon != "null") {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("TRADE ADDED");
}
}
}
console.log("PAIRS ARE OVER");
};
pairs has over 1400 entries and this is the one where are looping through.

depends on how many servers you are running this function on.
if it's one single machine, use worker_threads, basically run the same function in separate threads to achieve parallelization, but to be honest, 1400 pairs are a lot, each for 3-4 seconds, so total around 1-2hrs per run if in serial. Depending on your machines, if you have 8 cores, it might reduce the time by 8 folds but still leave you like around 10 minutes. and cloud service usually charge a lot more for instances that have more cpu cores.
if it's multiple machines, use a master and a queue to push new pairs to each worker machine and for each worker machine, you can also generate multiple threads for each machine, in that way you can scale horizontally, and it's possible to finish the run in seconds. in this situation, each machine you can get the cheap one from cloud providers.
so depends on your requirements, if you wanna super fast, you gotta add more machines.

NodeJS - Two same I2C address BME280 sensors connected through TCA9548A multiplexer (clashing results)

Good evening everyone!
For some reason there are a lot information on TCA9548A in Python, but not much on NodeJS.
I was trying to use this library from Github TCA9548A with this library BME280
Both BME280 sensors have address 0x76.
TCA9548A - 0x70.
The final code I was running was this:
const tca9548a_1 = new TCA9548A({addr: 0x70, bus: 1});
//as an example, if using two bme280 temp sensors (sensor1 and sensor2)
//that have the same address need to enable the specific
//multiplexer port each time you want to read from a certain device
//singlePortOn activates the port of the argument and
//disables all other ports
//argument has to be a number 0-7
//
//use a callback to ensure that the port is enabled
//before proceeding with other processing
//
//for example, sensor1 is attached to port 2 on the multiplexer
//
tca9548a_1.singlePortOn(2, doSomethingWithSensor());
//then read from sensor2 attached to port 6 on the multiplexer
//
tca9548a_1.singlePortOn(7, doSomethingWithSensor());
function doSomethingWithSensor () {
//process sensor data magic
const bme280 = require('bme280');
const format = number => (Math.round(number * 100) / 100).toFixed(2);
const delay = millis => new Promise(resolve => setTimeout(resolve, millis));
const reportContinuous = async _ => {
const sensor = await bme280.open({
i2cBusNumber: 1,
i2cAddress: 0x76,
humidityOversampling: bme280.OVERSAMPLE.X1,
pressureOversampling: bme280.OVERSAMPLE.X16,
temperatureOversampling: bme280.OVERSAMPLE.X2,
filterCoefficient: bme280.FILTER.F16
});
for (let i = 1; i <= 250; ++i) {
const reading = await sensor.read();
console.log(
`${i} ` +
`${format(reading.temperature)}°C, ` +
`${format(reading.pressure)} hPa, ` +
`${format(reading.humidity)}%`
);
await delay(1000); // 1 second
}
await sensor.close();
};
reportContinuous().catch(console.log);
}
Output and the problem: Only one sensor is being read and is outputting information that is doubled. This code is not reading information from them both, but just from one.
I am not very strong in JavaScript, but really want to learn.
Seems like I need to create a callback - from this comment in the code:
//use a callback to ensure that the port is enabled before proceeding with other processing
Will greatly appreciate any help on this subject.

Code that worked for me:
const TCA9548A = require('tca9548a');
const i2c = require('i2c-bus');
const tca9548a_1 = new TCA9548A({ addr: 0x70, bus: 1 });
function two(callback) {
tca9548a_1.singlePortOn(2);
console.log("Two")
const bme280 = require('bme280');
bme280.open().then(async sensor => {
console.log(await sensor.read());
await sensor.close();
}).catch(console.log);
callback()
}
function seven(callback) {
return setTimeout(() => {
tca9548a_1.singlePortOn(7);
console.log("Seven")
const bme280 = require('bme280');
bme280.open().then(async sensor => {
console.log(await sensor.read());
await sensor.close();
}).catch(console.log);
callback()
}, 1000);
}
function run() {
two(() => {
seven(() => { })
})
}
run();
OUTPUT:
Two
{
temperature: 28.303820903622544,
pressure: 991.7407636221005,
humidity: 39.50036416630162
}
Seven
{
temperature: 24.77324210727238,
pressure: 991.0048461479205,
humidity: 25.391542980733707
}

Google Apps Script Working on backend but not on sheets

I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?

I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.

You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price

GIthub Api get all commits per repo of user

I am trying to use Github API to get all commits of user. Github API doesn't explicitly provide this, so the only way is to loop through all the repos and find number of commits authored by the user for each repo.
The idea for getting this is
for each repo in repoList:
response = https://api.github.com/repos/{user}/{repo_name}/commits?per_page=100&page={page}
for each item in response:
if(item.commit.author===user)
totalCommmitsPerRepo += 1
update state as {repo : totalCommitsPerRepo}
(Max results per page = 100)
I am having problems when updating the page number. My code is given below
getCommitsInfo = async = () => {
let flag = 0
const repos = [...this.state.repos]
let totalCommits = 0
const requests = repos.map((repo) => {
do {
flag = 0;
axios.get("https://api.github.com/repos/" + this.state.user + '/' + repo + "/commits?per_page=100&page=" + page)
.then(res => {
totalCommits = 0
//CHECK IF RESULTS PER PAGE = MAXIMUM
if (res.data.length === 100) {
flag = 1
}
res.data.map(commit => {
let committer = commit.author.login.toLowerCase()
let user = this.state.user.toLowerCase()
if (user === committer) {
totalCommits += 1
}
})
let obj = {
name: repo,
commits: totalCommits
}
this.setState({
reposAndCommits: [...this.state.reposAndCommits, obj]
})
})
//UPDATE PAGE NUMBER
if (flag) {
page += 1
}
} while (flag);
})
await Promise.all(requests)
}
The do while loop doesn't seem to work as expected. I am not getting the page number updated. This is possibly because the do-while loop is asynchronous and I don't know how to make the loop wait until each response is fully complete.
Can someone please help regarding this, Thanks

Why does firebase cloud-function javascript promise run more than the number of loop invocations?

I have a cloud function that is triggered when a sale/purchase is committed into firestore. This function's purpose is to update the inventory level centrally.
The function works just fine if I'm updating an item's inventory at only 1 warehouse, but doing so for multiple warehouses has unexpected behavior. I'm looping through all the warehouses that are affected to calculate the total inventory level changes, and every iteration kicks-off a javascript promise.
The problem seems to occur with the way the promises are invoked. E.g: if I want to update 3 warehouses and loop 3 times, somehow 5 promises are being kicked-off. This is visible through the logs. I've researched similar questions here, but the solutions were suggested while firestore was still in beta and might not be the right way forward. (Firestore transactions getting triggered multiple times resulting in wrong data)
Here is the code
export const onTransactionCreate = functions.firestore
.document('/companies/{companyId}/sub_transactions/{transId}')
.onCreate(async (snapshot, context) => {
const transId = context.params.transId
const stock_transaction: IStockTransaction = <IStockTransaction>snapshot.data()
const trans_type: TRANS_TYPE = stock_transaction.trans_type
const promises: any[] = []
stock_transaction.lineItems.forEach((element, index) => {
const ITEM_GUID = element.item_guid
const is_increasing = isIncreasingTransaction(element.line_trans_type)
const delta_stock = element.qty_transaction * (is_increasing ? 1 : -1)
const TARGET_BRANCH_ID = element.target_branch_guid
const itemRef = db.collection(FIRESTORE_PATHS.COL_COMPANIES).doc(companyId).
collection(FIRESTORE_PATHS.SUB_COMPANIES_ITEMS).
doc("" + ITEM_GUID)
const item_promise = db.runTransaction(async t => {
try {
const item_doc = await t.get(itemRef)
const item_branch_quantities: IBranchQuantity[] = (item_doc.data()!.branch_quantities || new Array())
const item_branch_ids: string[] = (item_doc.data()!.available_branch_ids || new Array())
const branch_index = item_branch_ids.indexOf(TARGET_BRANCH_ID)
console.log(`${transId} Line Item ${index}, after document.get(), search branch index: ${branch_index}`)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty = prev_qty.quantity + delta_stock
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
console.log(`${transId} Line Item ${index} Updating qty # item ${delta_stock}, prev qty ${prev_qty.quantity}`)
} else {
item_branch_ids.push(TARGET_BRANCH_ID)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: TARGET_BRANCH_ID,
quantity: delta_stock
})
console.log(`${transId} Line Item ${index} Adding qty # item ${delta_stock}`)
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
available_branch_ids: item_branch_ids
})
} catch (err) {
throw new Error(err)
}
})
promises.push(item_promise)
});
return Promise.all(promises)
})

we have found the solution by reading this article.
A transaction consists of any number of get() operations followed by any number of write operations such as set(), update(), or delete(). In the case of a concurrent edit, Cloud Firestore runs the entire transaction again. For example, if a transaction reads documents and another client modifies any of those documents, Cloud Firestore retries the transaction. This feature ensures that the transaction runs on up-to-date and consistent data.
lineItems.forEach(element => {
const delta_transaction = element.qty * (isLineTransIncrease(element.line_trans_type) ? 1 : -1)
const itemRef = db.collection('companies').doc(companyId).collection('sub_items').doc("" + element.item_guid)
const p = db.runTransaction(t => {
return t.get(itemRef)
.then(doc => {
let item_branch_quantities: IBranchQuantity[] = doc.data()!.branch_quantities
let item_branch_ids: string[] = doc.data()!.available_branch_ids
if (!item_branch_quantities)
item_branch_quantities = new Array()
if (!item_branch_ids)
item_branch_ids = new Array()
const branch_index = item_branch_ids.indexOf(current_branch_id)
if (branch_index !== -1) {
const prev_qty = item_branch_quantities[branch_index]
const updated_qty: number = prev_qty.quantity + delta_transaction
item_branch_quantities[branch_index] = {
item_guid: prev_qty.item_guid,
branch_guid: prev_qty.branch_guid,
quantity: updated_qty
}
} else {
item_branch_ids.push(current_branch_id)
item_branch_quantities.push({
item_guid: element.item_guid,
branch_guid: current_branch_id,
quantity: delta_transaction
})
}
t.update(itemRef, {
branch_quantities: item_branch_quantities,
branch_ids: item_branch_ids
})
})
})
item_update_transactions.push(p)
});
return Promise.all(item_update_transactions)
})
function isLineTransIncrease(line_trans: number): boolean {
return (line_trans === 1) || (line_trans === 2)
}