I am trying to develop the backend of an ecommerce website using Stripe and NodeJS (Express precisely).
When the server starts, I am trying to fetch my products from Stripe. But after the first stripe.products.list call I get an error which says that I exceeded the api rate limit. This is not true because as it says in the Stripe doc the rate is limited to 25/sec in test mode whereas I am waiting 10 SECONDS before making my second call.
Please find below the function I use to make my calls. I simply use it in a loop with a sleep() function before each call.
async function fetchFromLastObj(last_obj){
const data = stripe.products.list({
active: true,
limit: maxRetrieve,
starting_after: last_obj,
})
.then((resp) => {
console.log(`Retrieved ${resp.data.length} products.`);
return resp.data;
})
.catch((e) => { });
return data;
}
The sleep function:
const { promisify } = require('util')
const sleep = promisify(setTimeout)
The loop in question:
var last_obj_seen = null;
var nb_iters = 0;
// fetching all products from stripe
while (true) {
console.log(`Iteration ${nb_iters+1}...`)
let fetchedList = [];
if (last_obj_seen == null) {
fetchedList = await fetchFirstBatch();
} else {
fetchedList = await fetchFromLastObj(last_obj_seen);
}
fetchedList = Array.from(fetchedList);
if (fetchedList.length == 0) { break; };
last_obj_seen = fetchedList.slice(-1)[0];
await sleep(10000);
fetchPrices((fetchedList))
.then((fetchedListWithPrices)=>{
saveList(fetchedListWithPrices);//not asynchronous
})
.catch((err) => { console.error("While fetching products from Stripe..."); console.error(err); });
nb_iters += 1;
if(nb_iters > 100){ throw Error("Infinite loop error"); }
if (nb_iters !== 0){
console.log("Waiting before request...");
await sleep(10000);
}
}
console.log("Done.");
Rather than handling pagination logic yourself you can use the auto-pagination feature of the official Stripe libraries.
Our libraries support auto-pagination. This feature easily handles fetching large lists of resources without having to manually paginate results and perform subsequent requests.
In Node 10+ you can do this, for example:
for await (const product of stripe.products.list()) {
// Do something with product
}
The Stripe Node library will handle pagination under the hood for you.
Related
I am new to Neo4j so I quite stuck with looping through some values.
I have a list of skill to skill strings
let data = [
'big_data, business_intelligence',
'big_data, data_collection',
'big_data, economic_growth',
'big_data, economy'
]
And I want to create or update the relation between left side with right side
for (let item of data) {
CreateSkillToSkillRelation(item);
}
const CreateSkillToSkillRelation = async (relation) => {
let mainSkill = relation.split(",")[0];
let secundarySkill = relation.split(",")[1];
try {
// Check if the relationship exists
let { records } = await session.run(
"MATCH(main:SKILL {name:$mainSkill}) -[relation:SKILL_TO_SKILL]-> (secundary:SKILL {name:$secundarySkill}) RETURN relation",
{ mainSkill, secundarySkill }
);
let count =
records[0]?._fields[0].properties.count > 0
? records[0]._fields[0].properties.count + 1
: 1;
// If count is greater then 1 then lets update the counter
if (count > 1) {
await session.run(
"MATCH(main:SKILL {name:$mainSkill}) -[relation:SKILL_TO_SKILL]-> (secundary:SKILL {name:$secundarySkill}) SET relation.count = $count RETURN main, secundary",
{
mainSkill,
secundarySkill,
count,
}
);
}
// Otherwise the skill relation is not created so lets create one
else {
await session.run(
"CREATE(main:SKILL {name:$mainSkill}) -[:SKILL_TO_SKILL {count:$count}]-> (secundary:SKILL {name:$secundarySkill}) RETURN main, secundary",
{
mainSkill,
secundarySkill,
count,
}
);
}
await session.close();
} catch (error) {
console.log(error);
}
};
But every time when I run this I get the following error Neo4jError: Queries cannot be run directly on a session with an open transaction; either run from within the transaction or use a different session.
Any idea how can I solve this?
for (let item of data) {
CreateSkillToSkillRelation(item);
}
Is not awaiting the promises you create and so you are basically trying to run all of these promises concurrently against a single session which only supports a single concurrent transaction.
You should create a session in each call of CreateSkillToSkillRelation or await each call to it using a single session.
Though note you close the session at the end of CreateSkillToSkillRelation but only on success, might I suggest moving await session.close(); into a finally block.
The answer of the colleague #just_another_dotnet_dev is absolutely correct: you run asynchronous functions in a loop, and close the session in one of them.
The Cipher language is very rich, and you can use it to do everything that you tried to do with a loop in Javascript. Something like this, using UNWIND and MERGE:
const CreateSkillToSkillRelations = async (data) => {
const session = driver.session();
try {
let { records } = await session.run(
`WITH split(row, ',') as rels
WITH trim(rels[0]) as mainSkill,
trim(rels[1]) as secundarySkill
MERGE (main:SKILL {name: mainSkill})
-[relation:SKILL_TO_SKILL]->
(secundary:SKILL {name: secundarySkill})
ON CREATE SET relation.count = 1
ON MATCH SET relation.count = relation.count + 1
RETURN main, relation, secundary`,
{ data }
);
} catch (error) {
console.log(error);
} finally {
await session.close()
}
};
This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.
If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}
I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?
I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.
You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price
I simply cannot see where I'm going wrong here. My Cloud Firestore is on "europe-west3", the functions are deployed to "europe-west1" (the docs tell me that this is the closest location to west3).
Structure is thus: I've got a bunch of "tickets" each of which can have a subcollection named "comments". The console thus looks like this:
The upload was successful:
The function code was taken from the official code samples
Github repo for Function samples
This is what my code looks like:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.countComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments/{commentsid}')
.onWrite(
async (change) => {
const ticketsRef = change.after.ref.parent;
const countRef = ticketsRef.parent.child('comments_count');
let increment;
if(change.after.exists() && !change.before.exists()) {
increment = 1;
} else if(!change.after.exists() && change.before.exists()) {
increment = -1;
} else {
return null;
}
await countRef.transaction((current) => {
return (current || 0) + increment;
});
console.log('Counter updated');
return null;
});
exports.recountComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments_count')
.onDelete(
async (snap) => {
const counterRef = snap.ref;
const collectionRef = counterRef.parent.child('comments');
const commentsData = await collectionRef.once('value');
return await counterRef.set(commentsData.numChildren());
}
)
Now, the problem is that these functions simply do not fire. I'm not seeing anything in the logs, regardless of whether I'm pushing changes through my clients (a Flutter app) or if I'm changing things directly in the Firebase console.
In my desperation I've also tried to simply listen to "/tickets" as any changes below that path should also trigger - but there's nothing.
So. What is the obvious thing I overlooked? And, yes, I had a look at the other questions/answers but nothing jumped at me...
edit:
This would be the corrected version, probably not optimal.
exports.countComments = functions.region('europe-west1').firestore.document('/tickets/{ticketId}/comments/{commentsId}')
.onWrite(
async (change, context) => {
const ticketId = context.params.ticketId;
const ticketRef = admin.firestore().collection('tickets').doc(ticketId);
let increment;
if(change.after.exists && !change.before.exists) {
increment = 1;
} else if(!change.after.exists && change.before.exists) {
increment = -1;
} else {
return null;
}
return transaction = admin.firestore().runTransaction(t => {
return t.get(ticketRef)
.then(doc => {
let count = (doc.data().comments_count || 0) + increment;
t.update(ticketRef, {comments_count: count});
});
}).then(res => {
console.log('Counter updated');
}).catch(err => {
console.log('Transaction error:', err);
});
});
Your database is Cloud Firestore, but you've written a Realtime Database trigger. They are two completely different databases. Follow the documentation for writing Cloud Firestore triggers instead.
Your function will start like this:
functions.region('europe-west1').firestore.document('...')
Note "firestore" instead of "database".
I'm trying to create a simple example of payments over the XRPL using Ripple-lib. The idea is to send several payments to different accounts stored in an array. I've made it kind of work in a different way as it is expected, but when using the 'then' method (as the docs recommend) does not work at all.
I'm a total newbie to Javascript so I don't have a good grasp on the language nor asyncronous coding and promises. When using the 'then' paradigm, the code stops working and no output can be seen in the console. This is the code I'm currently using. In the comments inside the 'SendXRP' function I explain the problem. How can this be re-arranged? Between the two ways, what is the proper one to code it?
'use strict';
const RippleAPI = require('ripple-lib').RippleAPI;
const sender = 'r*********************************';
const secret = 's****************************';
const destinations = ['r*********************************',
'r*********************************',
'r*********************************'];
const amount = 5;
// Instantiate Ripple API
const api = new RippleAPI({
server: "wss://s.altnet.rippletest.net:51233"
});
run();
async function sendXRP(amount, fee, destination, memo) {
// Update amount
amount = (amount - fee).toString();
// Build payment
const payment = {
source: {
address: sender,
maxAmount: {
value: amount,
currency: 'XRP'
}
},
destination: {
address: destination,
amount: {
value: amount,
currency: 'XRP'
}
},
memos: [
{
data: memo
}
]
};
// Build instuctions
const instructions = {
maxLedgerVersionOffset: 5
};
console.log('Sending ' + amount + ' to ' + destination);
// THIS KIND OF WORKS FOR NOW
// Prepare the payment
const preparedTX = await api.preparePayment(sender, payment, instructions);
// Sign the payment
const signedTX = api.sign(preparedTX.txJSON, secret);
// Submit the payment
const result = await api.submit(signedTX['signedTransaction']);
// Return TX hash on successful TX
if ('resultCode' in result && result['resultCode'] == 'tesSUCCESS') {
return signedTX.id;
} else {
return null;
}
// THIS IS MORE SIMILAR TO HOW IT IS DONE IN THE DOCS! NOT WORKING!
// ALSO, HOW DO I RETURN THE RESULT OF API.SIGN TO THE MAIN FUNCTION?
// Prepare the payment
// api.preparePayment(sender, payment, instructions).then(preparedTX => {
// // Sign the payment
// api.sign(preparedTX.txJSON, secret).then(signedTX => {
// // Submit the payment
// api.submit(signedTX['signedTransaction']);
// })
// }).catch(console.error);
}
function run() {
// Connect to Ripple server
api.connect().then(() => {
return api.getFee();
}).then(async fee => {
for (var i in destinations) {
var hash = await sendXRP(amount, Number(fee), destinations[i], 'memotext');
console.log(hash);
}
}).then(() => {
return api.disconnect();
}).catch(console.error);
}
Could it be that some of the transactions failed to send? If it failed, the result variable from sendXRP should have the txresult, but since you returned null if the result code is not tesSUCCESS, it doesn't return the result information.
const result = await api.submit(signedTX['signedTransaction']);
if ('resultCode' in result && result['resultCode'] == 'tesSUCCESS') {
return signedTX.id;
} else {
return null;
}
Before, when I tried submitting transactions consecutively, it would fail and return error code tefPAST_SEQ.
"The sequence number of the transaction is lower than the current sequence number of the account sending the transaction." from https://developers.ripple.com/tef-codes.html
I recommend removing the if('resultCode' in result...) block and check the transaction result. If the transactions failed with tefPAST_SEQ error, my solution to this is set the account sequence in instructions manually or add setTimeOut after each submit.