I am reading data in realtime from a device through http requests, however the way I am currently doing it is, something like this:
setInterval(() => {
for (let request of requests) {
await request.GetData();
}
}, 1000);
However, sometimes there is lag in the network, and since there are 4-5 requests, sometimes they don't all finish within a second, so they start stacking up, until the device eventually starts to timeout, so I need to somehow get rid of the setInterval. Increasing the time is not an option.
Essentially, I want them to get stuck in an infinite loop and I can add an inner timer to let the requests run again when half a second or a second has passed since the last run, but how do I get them stuck in an infinite loop without blocking the rest of the application?
Or maybe a way to make setInterval wait for all requests to finish before starting to count the 1 second interval?
Try:
(async () => {
while (true) {
for (let request of requests) {
await request.GetData();
}
await new Promise((resolve) => setTimeout(resolve, 1000));
}
})();
This will only start waiting once all the requests are finished, preventing them from stacking up.
Alternatively, on the inside of the async function, use:
while (true) {
await Promise.all(requests.map(request => request.GetData()));
await new Promise((resolve) => setTimeout(resolve, 1000));
}
This is different because all the calls to request.GetData() will run concurrently, which may or may not be what you want.
Related
I want to constantly sync data from a website, however I only have 300 calls/15 minutes. Thus I thought i could put all my sync requests (around 1000) into an array and then resolve just 300 of them every 15 minutes till the requests array is empty and then start again. However when I do the following:
let requests = []
params = 'invoice-items?invoice_id='
let positions = null
for (const invoice of invoices) {
requests.push(new Promise(async (resolve, reject) => {
positions = await getBillomatData(params + invoice.id, null, 0, null)
await updateDatabase(positions, models.billomat.Position)
}))
}
console.log(requests[0])
await requests[0]
console.log(requests[0])
As soon as I wait for the request at requests[0] it executes all of the requests and I go over the limit of calls.
All async calls are executed at once because JavaScript executes ALL code as soon as each call is processed. Await awaits the result, it does not await execution.
You'll need to use a tool such as bottleneck to rate limit your requests.
Processing 300 requests of 1000 every 15 minutes will take an hour to complete. This is a long time to keep the node job running doing nothing.
A basic a basic limiter might keep track of all the requests you want to make in an external file or database and then use a cron job to execute your JavaScript code every 15 minutes to process another 300 requests. If there are no additional requests, your app could just terminate. However, it will wake up and run every 15 minutes as long as the cron job keeps running.
The most simplest approach (not optimised though) would be
make batches of 300 calls
execute one batch and wait for all of them to be resolved before proceeding to next batch
let batch = []
// imagine urls is a array of url of 900 items
urls.map(async (url)=>{
batch.push(somePromiseFuctionToDoTheApiCall(url))
if(batch.length >= 300){
await Promise.all(batch)
// sleep is a promisified settimeout function, ref: https://stackoverflow.com/a/56520579/3359432
await sleep(calculateTimeToWaitBeforeProceedingToNextBatch)
batch = []
}
})
// there might be some leftovers at the end of batch you should process them also
if you are ok with using libraries and stop reinventing the wheel then have a look at lodash.chunk, bluebird.map, bluebird.each, bluebird.delay
I'm stuck in Async programming in Javascript.
I know that Promise.all() will run parallel.
Could you pls tell me what I'm wrong with this code below?
It should take 100ms. But actually, it takes 200ms :(
// 1. Define function here
var getFruit = async (name) => {
const fruits = {
pineapple: ":pineapple:",
peach: ":peach:",
strawberry: ":strawberry:"
};
await fetch('https://jsonplaceholder.typicode.com/photos'); // abount 100ms
return fruits[name];
};
var makeSmoothie = async () => {
const a = getFruit('pineapple');
const b = getFruit('strawberry');
const smoothie = await Promise.all([a, b]);
return smoothie;
//return [a, b];
};
/// 2. Execute code here
var tick = Date.now();
var log = (v) => console.log(`${v} \n Elapsed: ${Date.now() - tick}`);
makeSmoothie().then(log);
Your logic is fine, it is running as parallel as possible from client side.
You can test this by waiting on setTimeout instead of the fetch:
await new Promise(resolve => setTimeout(resolve, 100));
It's possible the placeholder site is queuing connections on its side or something. Either way you should measure with something predictable, not network.
Edit:
Just to explain a bit more that won't fit in a comment.
Let's put my waiting trick into an actual function:
const wait = ms => new Promise(resolve => setTimeout(resolve, ms));
OK, now you would call this as await wait(100). Now to compare:
await wait(100)
await fetch(...)
In terms of your code setup, they are the same. They are perform some async task that takes around 100ms and returns. If Promise.all() were not running this in parallel, the wait(100) version would certainly take 200ms or longer.
fetch is not as reliable of a test as setTimeout because it is running over the network. There's a lot of things you can't control with this call, like:
Some browsers limit how many parallel connections are made to the same domain
Lag in DNS or the server itself, typical network hiccups
The domain itself might force 1 connection at a time from your IP
It's not clear exactly what is causing the apparent synchronous behavior here. Maybe you will find some answers in the network panel. But in terms of code, there is nothing else for you to do. It is provably parallelized with the setTimeout test, which is much more reliable as a test since it is just a local timer.
Disclaimer: I'm not experienced with programming or with networks in general so I might be missing something quite obvious.
So i'm making a function in node.js that should go over an array of image links from my database and check if they're still working. There's thousands of links to check so I can't just fire off several thousand fetch calls at once and wait for results, instead I'm staggering the requests, going 10 by 10 and doing head requests to minimize the bandwidth usage.
I have two issues.
The first one is that after fetching the first 10-20 links quickly, the other requests take quite a bit longer and 9 or 10 out of 10 of them will time out. This might be due to some sort of network mechanism that throttles my requests when there are many being fired at once, but I'm thinking it's likely due to my second issue.
The second issue is that the checking process slows down after a few iterations. Here's an outline of what I'm doing. I'm taking the string array of image links and slicing it 10 by 10 then I check those 10 posts in 10 promises: (ignore the i and j variables, they're there just to track the individual promises and timeouts for loging/debugging)
const partialResult = await Promise.all(postsToCheck.map(async (post, j) => await this.checkPostForBrokenLink(post, i + j)));
within checkPostForBrokenLink I have a race between the fetch and a timeout of 10 seconds because I don't want to have to wait for the connection to time out every time timing out is a problem, I give it 10 seconds and then flag it as having timed out and move on.
const timeoutPromise = index => {
let timeoutRef;
const promise = new Promise<null>((resolve, reject) => {
const start = new Date().getTime();
console.log('===TIMEOUT INIT===' + index);
timeoutRef = setTimeout(() => {
const end = new Date().getTime();
console.log('===TIMEOUT FIRE===' + index, end - start);
resolve(null);
}, 10 * 1000);
});
return { timeoutRef, promise, index };
};
const fetchAndCancelTimeout = timeout => {
return fetch(post.fileUrl, { method: 'HEAD' })
.then(result => {
return result;
})
.finally(() => {
console.log('===CLEAR===' + index); //index is from the parent function
clearTimeout(timeout);
});
};
const timeout = timeoutPromise(index);
const videoTest = await Promise.race([fetchAndCancelTimeout(timeout.timeoutRef), timeout.promise]);
if fetchAndCancelTimeout finishes before timeout.promise does, it will cancel that timeout, but if the timeout finishes first the promise is still "resolving" in the background, despite the code having moved on. I'm guessing this is why my code is slowing down. The later timeouts take 20-30 seconds from being set up to firing, despite being set to 10 seconds. As far as I know, this has to be because the main process is busy and doesn't have time to execute the event queue, though I don't really know what it could be doing except waiting for the promises to resolve.
So the question is, first off, am I doing something stupid here that I shouldn't be doing and that's causing everything to be slow? Secondly, if not, can I somehow manually stop the execution of the fetch promise if the timeout fires first so as not to waste resources on a pointless process? Lastly, is there a better way to check if a large number of links are valid that what I'm doing here?
I found the problem and it wasn't, at least not directly, related to promise buildup. The code shown was for checking video links but, for images, the fetch call was done by a plugin and that plugin was causing the slowdown. When I started using the same code for both videos and images, the process suddenly became orders of magnitude quicker. I didn't think to check the plugin at first because it was supposed to only do a head request and format the results which shouldn't be an issue.
For anyone looking at this trying to find a way to cancel a fetch, #some provided an idea that seems like it might work. Check out https://www.npmjs.com/package/node-fetch#request-cancellation-with-abortsignal
Something you might want to investigate here is the Bluebird Promise library.
There are two functions in particular that I believe could simplify your implementation regarding rate limiting your requests and handling timeouts.
Bluebird Promise.map has a concurrency option (link), which allows you to set the number of concurrent requests and it also has a Promise.timeout function (link) which will return a rejection of the promise if a certain timeout has occurred.
function sleep(ms) {
var start = new Date().getTime(),
expire = start + ms;
while (new Date().getTime() < expire) {}
return;
}
async function executeWithDelay(offers) {
return Promise.all(
offers.map((offer, i) =>
getDetailedInfo(offer).then(data => {
offer = data;
if (i % 5 === 0) {
console.log('executed but it delays now for 3 seconds');
sleep(3000);
}
})
)
).then(function(data) {
return offers;
});
}
Trying to achieve web-scraping with possible best solution available. I am combining cheerio and puppeteer together and i have some nice code to debug. The above code works fine if the offers data is less i.e 5-10 records. If it exceeds, the browser gets crashed, though i am running in headless version.
This is basically due to in-house bug from the package what i am using which is unable to handle the load. My experiment is to a little delay for every batch of records and complete the full records execution.
But with the below code, it's behaving as non-blocking code and first time it delays and rest is executed constantly.
Should i use for loop instead of map or is there any alternative to handle this situation?
You never want to "sleep" with a busy-wait loop like that. It means nothing else can be done on the main thread.
Once the operations the promises represent have been started, unless they provide some custom means of pausing them, you can't pause them. Remember, promises don't actually do anything; they provide a standard way to observe something being done.
Another problem with that code is that you're returning offers, not the results of calling getExtendedInfo.
If the problem is calling getExtendedInfo too often, you can insert a delay (using setTimeout, not a busy-loop). For instance, this does the first request to getExtendedInfo almost immediately, the next after one second, the next after two seconds, and so on:
function sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms));
}
function executeWithDelay(offers) {
return Promise.all(
offers.map((offer, i) =>
sleep(i * 1000).then(getDetailedInfo)
)
});
}
Obviously, you can adjust that delay as required.
(Note that executeWithDelay isn't declared async; no need, since it has to use Promise.all anyway.)
Is it possible to halt execution of javascript for a finite amount of time? await is the closest thing I came across, but still it does not stop current execution of javascript. I actually tried busy-waiting on a variable, but in that case it was not able to come out of busy-waiting loop. Also I don't even know whether halting/stopping is ever possible in javascript since it is single threaded.
Exact scenario in which I am trying :
inside a callback() {
<make 2 ajax requests, wait for response>
<stop for sometime until one of the response is received>
return <redirect page using response received>
}
Note that callback can be called only once, and it should return a redirect url, if no return is given the page actually redirects to default fallback . So I want to stop for sometime to actually wait for events inside this callback.
You can certainly achieve it in an async scenario (by simulating it). As you postpone your task to a certain time in the future, when the event loop will pick it up.
Async/Await Example:
const sleep = t => new Promise(res => setTimeout(res, t))
const main = async _ => {
/*do something*/
await sleep(3000) //3 seconds
/*rest of the stuff*/
}
Promise way:
const sleep = t => new Promise(res => setTimeout(res, t))
const main = _ => {
doSometasks().then(_ => sleep(3000)).then(someClosingTasks)
}
Note: doSometasks() must return a promise (someClosingTasks() may or may not return a promise)
I think you could create js generator and (re)schedule execution with setTimeout until generator completes.
You can use synchronous XHR requests to block execution without busy loop, but in any case condition will not have a chance be changed if js is blocked in your method, so you have to allow execution of microtasks/macrotasks depending on what you really need.