I am trying to use Github API to get all commits of user. Github API doesn't explicitly provide this, so the only way is to loop through all the repos and find number of commits authored by the user for each repo.
The idea for getting this is
for each repo in repoList:
response = https://api.github.com/repos/{user}/{repo_name}/commits?per_page=100&page={page}
for each item in response:
if(item.commit.author===user)
totalCommmitsPerRepo += 1
update state as {repo : totalCommitsPerRepo}
(Max results per page = 100)
I am having problems when updating the page number. My code is given below
getCommitsInfo = async = () => {
let flag = 0
const repos = [...this.state.repos]
let totalCommits = 0
const requests = repos.map((repo) => {
do {
flag = 0;
axios.get("https://api.github.com/repos/" + this.state.user + '/' + repo + "/commits?per_page=100&page=" + page)
.then(res => {
totalCommits = 0
//CHECK IF RESULTS PER PAGE = MAXIMUM
if (res.data.length === 100) {
flag = 1
}
res.data.map(commit => {
let committer = commit.author.login.toLowerCase()
let user = this.state.user.toLowerCase()
if (user === committer) {
totalCommits += 1
}
})
let obj = {
name: repo,
commits: totalCommits
}
this.setState({
reposAndCommits: [...this.state.reposAndCommits, obj]
})
})
//UPDATE PAGE NUMBER
if (flag) {
page += 1
}
} while (flag);
})
await Promise.all(requests)
}
The do while loop doesn't seem to work as expected. I am not getting the page number updated. This is possibly because the do-while loop is asynchronous and I don't know how to make the loop wait until each response is fully complete.
Can someone please help regarding this, Thanks
Related
This question already has answers here:
How do I debug error ECONNRESET in Node.js?
(18 answers)
Closed 11 months ago.
I am using a node server to get trades data from binance. There are over a thousand pairs against which trades need to be fetched. The function takes time to run completely. I need the function to restart whenever it is finished running so I keep getting new data while my server is live and running. However, after my server has been running for 10-15 minutes, the following error occurs:
I want the server to run permanently in the background and for this function to keep fetching trades from API and storing those trades in my DB. I have another GET method defined that then fetches the trades from my DB.
The function that I am trying to run permanently lies in my main server.js file:
const getTrades = async () => {
let page = 1;
let coins = [];
const results = await db.query("SELECT * FROM pairs;");
const pairs = results.rows;
const latest = await db.query("SELECT MAX(trade_time) FROM trades");
const latestTrade = latest.rows[0].max;
while (page < 55) {
gecko = await axios(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
coins.push(gecko.data);
page++;
}
console.log("Loop over");
coins = coins.flat();
for (const pair of pairs) {
let biggestTrade = [];
response = await axios.get(
`https://api.binance.com/api/v3/trades?symbol=${pair.pair}`
);
let filtered = response.data;
filtered = filtered.filter((trade) => trade.time > latestTrade);
let sells = filtered.filter((trade) => trade.isBuyerMaker === true);
let buys = filtered.filter((trade) => trade.isBuyerMaker === false);
if (sells.length > 0) {
biggestTrade.push(
sells.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
if (buys.length > 0) {
biggestTrade.push(
buys.reduce(function (prev, current) {
return prev.quoteQty > current.quoteQty ? prev : current;
})
);
}
biggestTrade = biggestTrade.flat();
for (const trade of biggestTrade) {
let priceUSD = 0;
let baseAssetIcon = "";
for (const coin of coins) {
if (coin.symbol.toUpperCase() === pair.quote_asset) {
priceUSD = coin.current_price;
}
if (coin.symbol.toUpperCase() === pair.base_asset) {
baseAssetIcon = coin.image;
}
if (priceUSD > 0 && baseAssetIcon.length > 0) {
break;
}
}
if (trade.quoteQty * priceUSD > 50000) {
const results = db.query(
"INSERT INTO trades (exchange_name, exchange_icon_url, trade_time, price_in_quote_asset,price_in_usd, trade_value, base_asset_icon, qty, quoteQty, is_buyer_maker, pair, base_asset_trade, quote_asset_trade) VALUES($1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12, $13)",
[
"Binance",
"https://assets.coingecko.com/markets/images/52/small/binance.jpg?1519353250",
trade.time,
trade.price,
priceUSD,
trade.quoteQty * priceUSD,
baseAssetIcon,
trade.qty,
trade.quoteQty,
trade.isBuyerMaker,
pair.pair,
pair.base_asset,
pair.quote_asset,
]
);
console.log("Trade Added");
}
}
}
console.log("PAIRS ARE OVER");
};
How can I make it so that the function runs repeatedly after a specified time period and the server does not break.
If you make continuous multiple calls to external third party API's without interval between calls, you are asking for being disconnected because API's have security policies that will prevents this kind of clients. Imagine if the entire world making 55 connections at once in a server. The server certainly will crash.
I see in your code you are making 55 calls at once. I recommend you put a delay between each call.
const delay = time => new Promise(res => setTimeout(res,time));
await delay(1000); // 1 second delay
There are other optimization that can prevent issues with connections in axios, like sharing httpAgent:
import http from "http"
import https from "https"
import axios from "axios"
const httpAgent = new http.Agent({ keepAlive: true })
const httpsAgent = new https.Agent({ keepAlive: true })
const api = axios.create({
baseURL: "http://google.com",
httpAgent,
httpsAgent,
})
//now you will reuse the axios instance:
while(page < 55) {
await delay(500);
gecko = await api(
`https://api.coingecko.com/api/v3/coins/markets?vs_currency=USD&order=market_cap_desc&per_page=250&page=${page}`
);
}
I am trying to create a script that pulls from the coin market cap API and displays the current price. The script is working fine on the back end when I assign the variable a value. However, when I try to run the function on sheets the returned value is null.
function marketview(ticker) {
var url = "https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?CMC_PRO_API_KEY=XXX&symbol=" + ticker;
var data = UrlFetchApp.fetch(url);
const jsondata = JSON.parse(data);
Logger.log(jsondata.data[ticker].quote['USD'].price)
}
My execution logs show that the scripts are running, but when when I use the function and try and quote ETH for example, the script is running for BTC.
When I do this on the backend and assign ETH the script works fine and returns the right quote. Any ideas on what I'm missing?
I did the same with coingecko API and add an issue having all my requests being rejected with quota exceeded error.
I understood that Google sheets servers IPs address were already spamming coingecko server. (I was obviously not the only one to try this).
This is why I used an external service like apify.com to pull the data and re-expose data over their API.
This is my AppScripts coingecko.gs:
/**
* get latest coingecko market prices dataset
*/
async function GET_COINGECKO_PRICES(key, actor) {
const coinGeckoUrl = `https://api.apify.com/v2/acts/${actor}/runs/last/dataset/items?token=${key}&status=SUCCEEDED`
return ImportJSON(coinGeckoUrl);
}
You need ImportJSON function, available here: https://github.com/bradjasper/ImportJSON/blob/master/ImportJSON.gs
Then in a cell I write: =GET_COINGECKO_PRICES(APIFY_API_KEY,APIFY_COINGECKO_MARKET_PRICES), you will have to create two field named APIFY_API_KEY and APIFY_COINGECKO_MARKET_PRICES in order for this to work.
Then register on apify.com, then you'll have to create an actor by forking apify-webscraper actor.
I set the StartURLs with https://api.coingecko.com/api/v3/coins/list, this will give me the total number of existing crypto (approx 11000 as of today), and number of page so I can run the request concurrently (rate limit is 10 concurrent requests on coingecko), then I just replace /list with /market and set the proper limit to get all the pages I need.
I use the following for the tasks page function:
async function pageFunction(context) {
let marketPrices = [];
const ENABLE_CONCURRENCY_BATCH = true;
const PRICE_CHANGE_PERCENTAGE = ['1h', '24h', '7d'];
const MAX_PAGE_TO_SCRAP = 10;
const MAX_PER_PAGE = 250;
const MAX_CONCURRENCY_BATCH_LIMIT = 10;
await context.WaitFor(5000);
const cryptoList = readJson();
const totalPage = Math.ceil(cryptoList.length / MAX_PER_PAGE);
context.log.info(`[Coingecko total cryptos count: ${cryptoList.length} (${totalPage} pages)]`)
function readJson() {
try {
const preEl = document.querySelector('body > pre');
return JSON.parse(preEl.innerText);
} catch (error) {
throw Error(`Failed to read JSON: ${error.message}`)
}
}
async function loadPage($page) {
try {
const params = {
vs_currency: 'usd',
page: $page,
per_page: MAX_PER_PAGE,
price_change_percentage: PRICE_CHANGE_PERCENTAGE.join(','),
sparkline: true,
}
let pageUrl = `${context.request.url.replace(/\/list$/, '/markets')}?`;
pageUrl += [
`vs_currency=${params.vs_currency}`,
`page=${params.page}`,
`per_page=${params.per_page}`,
`price_change_percentage=${params.price_change_percentage}`,
].join('&');
context.log.info(`GET page ${params.page} URL: ${pageUrl}`);
const page = await fetch(pageUrl).then((response) => response.json());
context.log.info(`Done GET page ${params.page} size ${page.length}`);
marketPrices = [...marketPrices, ...page];
return page
} catch (error) {
throw Error(`Fail to load page ${$page}: ${error.message}`)
}
}
try {
if (ENABLE_CONCURRENCY_BATCH) {
const fetchers = Array.from({ length: totalPage }).map((_, i) => {
const pageIndex = i + 1;
if (pageIndex > MAX_PAGE_TO_SCRAP) {
return null;
}
return () => loadPage(pageIndex);
}).filter(Boolean);
while (fetchers.length) {
await Promise.all(
fetchers.splice(0, MAX_CONCURRENCY_BATCH_LIMIT).map((f) => f())
);
}
} else {
let pageIndex = 1
let page = await loadPage(pageIndex)
while (page.length !== 0 && page <= MAX_PAGE_TO_SCRAP) {
pageIndex += 1
page = await loadPage(pageIndex)
}
}
} catch (error) {
context.log.info(`Fetchers failed: ${error.message}`);
}
context.log.info(`End: Updated ${marketPrices.length} prices for ${cryptoList.length} cryptos`);
const data = marketPrices.sort((a, b) => a.id.toLowerCase() > b.id.toLowerCase() ? 1 : -1);
context.log.info(JSON.stringify(data.find((item) => item.id.toLowerCase() === 'bitcoin')));
function sanitizer(item) {
item.symbol = item.symbol.toUpperCase()
return item;
}
return data.map(sanitizer)
}
I presume you are hiting the same issue I had with coinmarketcap, and that you could do the same with it.
You're not return ing anything to the sheet, but just logging it. Return it:
return jsondata.data[ticker].quote['USD'].price
I want to implement a function in JavaScript which calls a series of web service endpoint and checks for a value in the response of the API call.
I need to achieve it in a way that the first endpoint page is called first then the there would be a filter method to filter out the specific object from the response. If the object is found, this process should break and the object must be returned. However if the object is not found in the first endpoint, then the second endpoint must be called and the same process is repeated until the object is found.
The Web service endpoint that I am working on is:
https://jsonmock.hackerrank.com/api/countries?page=1
This API returns a list of country data. Here the value of page query varies from 1 to 25. I need to call the endpoint and check for a specific country from 1 to 25 until the country object is found.
I tried achieving this using JavaScript Promise and Fetch API and couldn't think of a way to call the APIs one after the other.
I am really looking forward for your answer. Thank you in advance.
You can use async and await for this:
async function findCountry(country) {
for (let page = 1; page < 26; page++) {
console.log("page = " + page); // for debugging only
let response = await fetch("https://jsonmock.hackerrank.com/api/countries?page=" + page);
let {data} = await response.json();
let obj = data.find(obj => obj.name == country);
if (obj) return obj;
}
}
let country = "Belgium";
findCountry(country).then(obj => {
if (obj) {
console.log("The capital of " + country + " is " + obj.capital);
} else {
console.log("Could not find " + country);
}
});
If you know that the data is sorted by country name, then you could reduce the average number of requests by using a binary search.
Here's a way that you can do it.
const url = 'https://jsonmock.hackerrank.com/api/countries'
const fetchFromApi = async (countryName, page) => {
const res = await fetch(`${url}?page=${page}`)
return await res.json()
}
const getCountryFromResults = (countryName, data) => {
const country = countryName.toLowerCase()
return data.find(({name}) => name.toLowerCase() === country)
}
const findCountry = async (countryName) => {
let page = 1;
let totalPages = 1;
while(page <= totalPages) {
const res = await fetchFromApi(countryName, page);
if(totalPages < res.total_pages) {
totalPages = res.total_pages
}
const country = getCountryFromResults(countryName, res.data)
if(country){
return country
}
page = page + 1
}
}
( async () => {
console.log(await findCountry("Afghanistan"))
console.log(await findCountry("Argentina"))
}
)()
I simply cannot see where I'm going wrong here. My Cloud Firestore is on "europe-west3", the functions are deployed to "europe-west1" (the docs tell me that this is the closest location to west3).
Structure is thus: I've got a bunch of "tickets" each of which can have a subcollection named "comments". The console thus looks like this:
The upload was successful:
The function code was taken from the official code samples
Github repo for Function samples
This is what my code looks like:
const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();
exports.countComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments/{commentsid}')
.onWrite(
async (change) => {
const ticketsRef = change.after.ref.parent;
const countRef = ticketsRef.parent.child('comments_count');
let increment;
if(change.after.exists() && !change.before.exists()) {
increment = 1;
} else if(!change.after.exists() && change.before.exists()) {
increment = -1;
} else {
return null;
}
await countRef.transaction((current) => {
return (current || 0) + increment;
});
console.log('Counter updated');
return null;
});
exports.recountComments = functions.region('europe-west1').database.ref('/tickets/{ticketId}/comments_count')
.onDelete(
async (snap) => {
const counterRef = snap.ref;
const collectionRef = counterRef.parent.child('comments');
const commentsData = await collectionRef.once('value');
return await counterRef.set(commentsData.numChildren());
}
)
Now, the problem is that these functions simply do not fire. I'm not seeing anything in the logs, regardless of whether I'm pushing changes through my clients (a Flutter app) or if I'm changing things directly in the Firebase console.
In my desperation I've also tried to simply listen to "/tickets" as any changes below that path should also trigger - but there's nothing.
So. What is the obvious thing I overlooked? And, yes, I had a look at the other questions/answers but nothing jumped at me...
edit:
This would be the corrected version, probably not optimal.
exports.countComments = functions.region('europe-west1').firestore.document('/tickets/{ticketId}/comments/{commentsId}')
.onWrite(
async (change, context) => {
const ticketId = context.params.ticketId;
const ticketRef = admin.firestore().collection('tickets').doc(ticketId);
let increment;
if(change.after.exists && !change.before.exists) {
increment = 1;
} else if(!change.after.exists && change.before.exists) {
increment = -1;
} else {
return null;
}
return transaction = admin.firestore().runTransaction(t => {
return t.get(ticketRef)
.then(doc => {
let count = (doc.data().comments_count || 0) + increment;
t.update(ticketRef, {comments_count: count});
});
}).then(res => {
console.log('Counter updated');
}).catch(err => {
console.log('Transaction error:', err);
});
});
Your database is Cloud Firestore, but you've written a Realtime Database trigger. They are two completely different databases. Follow the documentation for writing Cloud Firestore triggers instead.
Your function will start like this:
functions.region('europe-west1').firestore.document('...')
Note "firestore" instead of "database".
I think a firebase function updating a list that I have in the firebase database is being captured by a subscription that is subscribed to that list. From what the list output looks like on my phone (in the app)...and from what my console output looks like (the way it repeats) it seems like it is capturing the whole list and displaying it each time one is added. So (I looked this up)...I believe this equation represents what is happening:
(N(N + 1))/2
It is how you get the sum of all of the numbers from 1 to N. Doing the math in my case (N = 30 or so), I get around 465 entries...so you can see it is loading a ton, when I only want it to load the first 10.
To show what is happening with the output here is a pastebin https://pastebin.com/B7yitqvD.
In the output pay attention to the array that is above/before length - 1 load. You can see that it is rapidly returning an array with one more entry every time and adding it to the list. I did an extremely rough count of how many items are in my list too, and I got 440...so that roughly matches the 465 number.
The chain of events starts in a page that isn't the page with the list with this function - which initiates the sorting on the firebase functions side:
let a = this.http.get('https://us-central1-mane-4152c.cloudfunctions.net/sortDistance?text='+resp.coords.latitude+':'+resp.coords.longitude+':'+this.username);
this.subscription6 = a.subscribe(res => {
console.log(res + "response from firesbase functions");
loading.dismiss();
}, err => {
console.log(JSON.stringify(err))
loading.dismiss();
})
Here is the function on the page with the list that I think is capturing the entire sort for some reason. The subscription is being repeated as the firebase function sorts, I believe.
loadDistances() {
//return new Promise((resolve, reject) => {
let cacheKey = "distances"
let arr = [];
let mapped;
console.log("IN LOADDISTANCES #$$$$$$$$$$$$$$$$$$$$$");
console.log("IN geo get position #$$$$$$$5354554354$$$$$$$");
this.distancelist = this.af.list('distances/' + this.username, { query: {
orderByChild: 'distance',
limitToFirst: 10
}});
this.subscription6 = this.distancelist.subscribe(items => {
let x = 0;
console.log(JSON.stringify(items) + " length - 1 load");
items.forEach(item => {
let storageRef = firebase.storage().ref().child('/settings/' + item.username + '/profilepicture.png');
storageRef.getDownloadURL().then(url => {
console.log(url + "in download url !!!!!!!!!!!!!!!!!!!!!!!!");
item.picURL = url;
}).catch((e) => {
console.log("in caught url !!!!!!!$$$$$$$!!");
item.picURL = 'assets/blankprof.png';
});
this.distances.push(item);
if(x == items.length - 1) {
this.startAtKey4 = items[x].distance;
}
x++;
})
//this.subscription6.unsubscribe();
})
}
The subscription in loadDistances function works fine as long as I don't update the list from the other page - another indicator that it might be capturing the whole sort and listing it repeatedly as it sorts.
I have tried as as I could think of to unsubscribe from the list after I update...so then I could just load the list of 10 the next time the page with the list enters, instead of right after the update (over and over again). I know that firebase functions is in beta. Could this be a bug on their side? Here is my firebase functions code:
exports.sortDistance = functions.https.onRequest((req, res) => {
// Grab the text parameter.
var array = req.query.text.split(':');
// Push the new message into the Realtime Database using the Firebase Admin SDK.
// Get a database reference to our posts
var db = admin.database();
var ref = db.ref("profiles/stylists");
var promises = [];
// Attach an asynchronous callback to read the data at our posts reference
ref.on("value", function(snapshot) {
//console.log(snapshot.val());
var snap = snapshot.val();
for(const user in snap) {
promises.push(new Promise(function(resolve, reject) {
var snapadd = snap[user].address;
console.log(snapadd + " snap user address (((((((())))))))");
if(snapadd != null || typeof snapadd != undefined) {
googleMapsClient.geocode({
address: snapadd
}).asPromise()
.then(response => {
console.log(response.json.results[0].geometry.location.lat);
console.log(" +++ " + response.json.results[0].geometry.location.lat + ' ' + response.json.results[0].geometry.location.lng + ' ' + array[0] + ' ' + array[1]);
var distanceBetween = distance(response.json.results[0].geometry.location.lat, response.json.results[0].geometry.location.lng, array[0], array[1]);
console.log(distanceBetween + " distance between spots");
var refList = db.ref("distances/"+array[2]);
console.log(snap[user].username + " snap username");
refList.push({
username: snap[user].username,
distance: Math.round(distanceBetween * 100) / 100
})
resolve();
})
.catch(err => { console.log(err); resolve();})
}
else {
resolve();
}
}).catch(err => console.log('error from catch ' + err)));
//console.log(typeof user + 'type of');
}
var p = Promise.all(promises);
console.log(JSON.stringify(p) + " promises logged");
res.status(200).end();
}, function (errorObject) {
console.log("The read failed: " + errorObject.code);
});
});
What is weird is, when I check the firebase functions logs, all of this appears to only run once...but I still think the subscription could be capturing the whole sorting process in some weird way while rapidly returning it. To be as clear as possible with what I think is going on - I think each stage of the sort is being captured in an (N(N + 1))/2...starting at 1 and going to roughly 30...and the sum of the sorting ends up being the length of my list (with 1-10 items repeated over and over again).
I updated to angularfire2 5.0 and angular 5.0...which took a little while, but ended up solving the problem:
this.distanceList = this.af.list('/distances/' + this.username,
ref => ref.orderByChild("distance").limitToFirst(50)).valueChanges();
In my HTML I used an async pipe, which solved the sorting problem:
...
<ion-item *ngFor="let z of (distanceList|async)" no-padding>
...