How to handle streaming data using fetch? - javascript

I have used async for with great success in handling output streams from processes with node.js, but I'm struggling to get something that I was hoping could "just work" with the browser fetch API.
This works great to async'ly handle chunks of output streaming from a process:
for await (const out of proc.child.stdout) {
...
}
(in an async function context here of course)
I tried to do something similar in a browser where I want to gain access to the data while it is being sent to me from the server.
for await (const chunk of (await fetch('/data.jsonl')).body) {
console.log('got', chunk);
}
This does not work in Chrome (Uncaught TypeError: (intermediate value).body is not async iterable).
For my use case this is not necessary, so I am simply using let data = await (await fetch(datapath)).text(); in my client code for now. This is analogous to the typical use of .json() instead of .text() on the awaited fetch, so no processing can begin until the entire response is received by the browser. This is not ideal for obvious reasons.
I was looking at Oboe.js (I think the relevant impl is somewhere near here) which pretty much deals with this but its internals are fairly ugly so it looks like that might be the only way to do this for now?
If async iteration isn't implemented (meaning async for cannot be used yet) isn't there another way to use the ReadableStream in a practical way?

Unfortunately async iterable support is not yet implemented, despite being in the spec. Instead you can manually iterate, as shown in this example from the spec. (I'll convert examples to async/await for you in this answer.)
const reader = response.body.getReader();
const { value, done } = await reader.read();
if (done) {
console.log("The stream was already closed!");
} else {
console.log(value);
}
You can use recursion or a loop to do this repeatedly, as in this other example:
async function readAllChunks(readableStream) {
const reader = readableStream.getReader();
const chunks = [];
let done, value;
while (!done) {
({ value, done } = await reader.read());
if (done) {
return chunks;
}
chunks.push(value);
}
}
console.log(await readAllChunks(response.body));

According to the spec, a ReadableStream such as the fetch-API's Response.body does have a getIterator method. For some reason it's not async-iterable itself, you explicitly have to call that method:
const response = await fetch('/data.json');
if (!response.ok)
throw new Error(await response.text());
for await (const chunk of response.body.getIterator()) {
console.log('got', chunk);
}

I believe the current state of affairs in mid 2020 is that async for does not work on the fetch body yet.
https://github.com/whatwg/streams/issues/778 This issue appears to have tracking bugs for browsers and none of them have the functionality implemented yet.
I don't currently know of another way to make use of the .body ReadableStream provided by fetch.
The standard way to do the task implicit in the question is to use a websocket.

Related

Blob to Uint8Array without callback

Is it possible to convert a Blob to Uint8Array without using a callback and just using JavaScript?
How to return the uint8array for the calling variable uint8data?
let str='content';
str2blob=(str,type='text/plain')=>new Blob([str],{type:type});
let blob=str2blob(str);
function blob2uint(blob){
new Response(blob).arrayBuffer().then(buffer=>{
uint=[...new Uint8Array(buffer)];
console.log("Uint8Array",uint);
return uint;
});
}
let uint8data=blob2uint(blob);
console.log(uint8data); //Show Uint8Array
Assuming this is somewhere inside a function, you can make it async and use await to wait for the Promise to complete once you return it from the blob2uint function:
async function myBlobToUIntDemo() {
let str='content';
str2blob=(str,type='text/plain')=>new Blob([str],{type:type});
let blob=str2blob(str);
function blob2uint(blob){
return new Response(blob).arrayBuffer().then(buffer=>{
uint=[...new Uint8Array(buffer)];
console.log("Uint8Array",uint);
return uint;
});
}
let uint8data = await blob2uint(blob);
console.log(uint8data); //Show Uint8Array
}
Let's spike this up with some more information:
Why do I need a callback in the first place?
Because you are doing asynchronous work by pulling the arrayBuffer from a response. JavaScript (as far as I know) only runs in a single thread. Waiting for a website in your code without using a callback would halt all other JavaScript execution on your page if you didn't put it in the background somehow; that's what callbacks and Promises do for you (from the outside at least).
Promises?
Functions doing asynchronous work typically return a Promise object, that promises to provide you with the value once the work completes (by calling the function passed in the then callback with it) or an error in case something goes wrong (by calling the catch callback with it).
async / await
async / await was introduced to make working with Promises a bit nicer and readable, reducing the need for callbacks a lot. You can read more about that in this MDN article.
The code block above only contains the bare minimum changes to get things working. Let's simplify that a bit more!
async function myBlobToUIntDemo() {
let str = 'content';
let str2blob = (str, type = 'text/plain') => new Blob([str], { type: type });
let blob = str2blob(str);
const buffer = await new Response(blob).arrayBuffer();
const uint = [...new Uint8Array(buffer)];
console.log(uint);
}
Note that this returns a Promise as well (this automatically happens if you mark your function as async) so make sure to remember that if you were to use this function anywhere in your code.

Multiple fetch request in a raw

For a project, I have to generate a PDF that lists the vehicles in a DB. To do this, I use the JSPDF library.
1st step: I get data from a DB (via asynchronous requests on an API) and images on a server that I store in an Array.
2nd step: I call a function that generates the PDF with JSPDF.
The problem is that I need to have all my data retrieved before calling my generatePDF function otherwise the fields and images are empty because they have not yet been retrieved from the DB or the server.
A solution I found is to use the setTimeout function to put a delay between each call. However, this makes the code very slow and inflexible because you have to change the timeout manually depending on the number of data and images to retrieve. Moreover, it is impossible to determine exactly how long it will take to retrieve the data, especially since this can vary depending on the state of the network, so you have to allow for a margin which is often unnecessary.
Another solution is to use callbacks or to interweave fetch / ajax calls with .then / .done calls, but this becomes very complicated when it comes to retrieving the images since they are retrieved one by one and there are more than a hundred of them.
What would be the easiest way to do this in a clean and flexible way?
Thanks for your help and sorry for the long text, I tried to be as clear as possible :)
To do a series of asynchronous things in order, you start the next operation in the fulfillment handler of the previous operation.
An async function is the easiest way:
async function buildPDF() {
const response = await fetch("/path/to/the/data");
if (!response.ok) {
throw new Error(`HTTP error ${response.status}`);
}
const data = await response.json(); // Or `.text()` or whatever
const pdf = await createPDF(data); // Assuming it returns a promise
// ...
}
If you can't use async functions in your environment and don't want to transpile, you can write your fulfullment handlers as callbacks:
function buildPDF() {
return fetch("/path/to/the/data")
.then(response => {
if (!response.ok) {
throw new Error(`HTTP error ${response.status}`);
}
return response.json(); // Or `.text()` or whatever
})
.then(data => createPDF(data))
.then(pdf => {
// ...
});
}
Note that I'm returning the result of the chain, so that the caller can handle errors.

data type of value returned from Promise or async / await

i know there's a bunch of related questions / posts regarding this question but they don't really answer my question, my question here is simple, if I have a promise and i wrapped it in an async function and await it til it's settled like this:
async function getUserData() {
return fetch('url'); // fetch returns a promise, wrapping the user data returned
}
async main() {
const user = await getUserData();
// what is the type of the user object here?
}
I'm asking this question because I'm using TypeScript, I usually try to type cast the return value to the expected type like this:
async function getUserData() {
return fetch('url') as UserInfo;
}
say UserInfo has a name attribute but if I try write in this way then user.name is undefined:
async function main() {
const user = await getUserData();
console.log(user.name); // undefined.
}
it makes me how should I 'unwrap' a promise with a value in it?
You can't really know at compile time what it is, because this is explicitly something that happens at runtime.
So usually the result of a JSON.parse() is any. A stricter version could be the unknown type.
If you want to just assume whatever you got back is going to be the right type, without validating (because you trust the server), I think I would do that as such:
async function getUserData(): Promise<UserInfo> {
return fetch('url'); // fetch returns a promise, wrapping the user data returned
}
I could be wrong, but I think the way that Async/Await works is that it wraps the function in a native promise.
I tested out your examples in the browser console and both functions return a Promise.
Promise {<pending>}
I am not sure about what that is cast to in Typescript as I don't use it. But if you drop into the browser console you can test all of this. It functions as a pretty good REPL.
As a function of design, the getUserData() function does not need to be async as you are not awaiting anything in it.
As for unwrapping it, you can use the fetch API since you now have a fetch result:
const data = getUserData();
data.then(response => response.json())

Using a async function in createConnection of promise-mysql does change anything at loading speed?

I need to know if it is better to use asynchronous createConnection or not
Does this change anything at loading speed?
I'm using express, ReactJS, promise-mysql
What should I use?
This:
async connect () {
try{
const conn = await db.createConnection(this.config);
this.conn = conn;
} catch(error){
console.log(error)
}
}
Or this
connect () {
return db.createConnection(this.config).then(conn => {
this.conn = conn
})
}
Well, the error handling is completely different in your two examples. In the first, you log the error and allow the returned promise to then be resolved. In the second, a connection error will reject the returned promise. So, that's a major structural difference.
If you changed the first one to this:
async connect () {
this.conn = await db.createConnection(this.config);
}
Then, that would be structurally the same as your second example:
connect () {
return db.createConnection(this.config).then(conn => {
this.conn = conn;
});
}
Now, if you compare these two they have the same outcomes (except in an edge case where db.createConnection() would throw synchronously which it hopefully doesn't do).
So, then if you reasked your question based on these two that have the same outcomes, the answer is that it doesn't really make a difference.
If there was a measurable difference in execution speed, it would be so small as to be very unlikely to be meaningful and whatever difference there was would be only dependent upon the particular JS engine implementation and likely not constant as the JS engine matures.
So, it's really just a matter of coding style and which you prefer. The await version is less typing, less lines of code (as it often is). I personally tend to not use async/await unless I have more than one asynchronous operation that I'm trying to sequence, but that's perhaps just some inertia from coding with .then() for awhile before await came along.

Vue.js async/await with then function not executing

I'm attempting to use async/await with axios.then() within a for of loop. The function fails to run or even attempt the axios call. I have a feeling using the then() function is part of the problem.
I need the for loop to wait until the then() function has run before continuing on to the next array item. Any ideas how I can alter my code to get the axios.then() functions to run asynchronously?
accounts = [{a},{b},{c}, ...] // Example of accounts array
async function get_user_data (accounts) {
// For of loop
for (let [index, user] of accounts.entries()) {
// Currently using await before axios call
await axios.get(url, headers)
.then(function(data) {
console.log(data)
})
.catch(function(error) {
console.log(error)
})
}
}
UPDATE:
Issue was ultimately being caused by the Vue frontend compiling my application. Resolved by following the stack overflow solution posted here. Note, that the code now functions as expected. The solution offered by Dacre Denny helped me decide the issue must be located somewhere else as his should have worked but did not until the problem with Vue was solved. Takeaways:
Use simple tests to confirm issue not with code
Check webpack, babel, and other compiler configs if above doesn't work
In general, it is considered an anti-pattern to mix the promise interface (ie .then()) with await/async semantics.
Seeing that the get_user_data function is defined async, consider a revised implementation based on a try/catch block for clearer program flow and greater predictability in the asynchronous behaviour of your loop:
async function get_user_data (accounts) {
for (let [index, user] of accounts.entries()) {
/* try/catch block allows async errors/exceptions to be caught
without then need for a .catch() handler */
try {
/* Using await, the response data can be obtained in this
way without the need for a .then() handler */
const data = await axios.get(url, headers)
console.log(data);
}
catch(error) {
/* If an error occurs in the async axios request an exception
is thrown and can be caught/handled here */
console.log(error)
}
}
}
Thanks to Dacre Denny for the quick help. His test code helped me determine the problem was not with the code.
The Issue was ultimately being caused by the Vue frontend compiling my application which does not at this date support async/await out of the box. Resolved by following the stack overflow solution posted here. Note, that the code now functions as expected. Takeways:
Use simple tests to confirm issue not with code
Check webpack, babel, or other compiler configs if above doesn't
work
A lack of errors when function fails to run may indicate a compilation error. Check configs.
async get_user_data(accounts) {
// For of loop
(async() => {
for (let [index, user] of accounts.entries()) {
// Currently using await before axios call
await axios.get(url, headers)
.then(function(data) {
console.log(data)
})
.catch(function(error) {
console.log(error)
})
}
})();
},

Categories