I want to fetch an URL and process the response in chunks of a defined size. I also don't want to block while waiting for the whole chunk to be available. Is something like this available in the Fetch API?
Example how it could look like:
const response = await fetch(url)
const reader = response.body.getReader()
const chunk = await reader.read(CHUNK_SIZE)
There is support in fetch() to be consumed like a stream. See MDN reference here. It appears that you need some boilerplate code for ReadableStream...
Code would like this:
const workOnChunk = (chunk) => { console.log("do-work")};
// Fetch your stuff
fetch(url)
// Retrieve its body as ReadableStream
.then(response => response.body)
// Boilerplate for the stream - refactor it out in a common utility.
.then(rs => {
const reader = rs.getReader();
return new ReadableStream({
async start(controller) {
while (true) {
const { done, value } = await reader.read();
// When no more data needs to be consumed, break the reading
if (done) {
break;
}
// Do your work: ¿¿ Checkout what value returns ¿¿
workOnChunk(value)
// Optionally append the value if you need the full blob later.
controller.enqueue(value);
}
// Close the stream
controller.close();
reader.releaseLock();
}
})
})
// Create a new response out of the stream (can be avoided?)
.then(rs => new Response(rs))
// Create an object URL for the response
.then(response => response.blob())
.then(blob => { console.log("Do something with full blob") }
.catch(console.error)
NOTE: The nodejs-fetch API is not exactly the same. If you are on nodejs, see nodeje-fetch's stream support.
Related
I am using the code given below to run multiple https GET request for Wikipedia API.
app.get("/data_results", (req, res) => {
const articlesData = names.map(nameObj => {
let name = nameObj.name;
let articleExtract = "";
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
// Getting article content
https.get(contentURL, response => {
response.on("data", async data => {
const wikiArticle = JSON.parse(data);
// Mapping the keys of the JSON data of query to its values.
articleExtract = await Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
nameObj.article = articleExtract.substring(0,350);
})
});
return nameObj;
});
res.send(articlesData);});
This is my names array
[
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
]
My aim :-
Run the HTTPS GET request for every value in the names array sequentially.
Add the article extract with its name and save in it in an objects array.
You could also suggest me better ways of doing this.
Try this solution. it creates an object like
{
Mahabharata: response
}
The code
const getValues = async () => {
const name_array = [
{name: 'Indus%20Valley%20Civilisation'},
{name: 'Ramayana'},
{name: 'Mahabharata'},
{name: 'Gautama%20Buddha'},
];
const namePromises = name_array.map(value =>
fetch('baseurl' + value.name).then(res => res.json()),
);
const response = await Promise.all(namePromises);
return response.reduce((obj, responseResults, index) => {
obj[name_array[index].name] = responseResults;
}, {});
};
I would suggest you use fetch and promise. all. Map over your array of names and create promises as input. and then return the array of resolved promises.
something like this
const namePromises = [
{ name: 'Indus%20Valley%20Civilisation' },
{ name: 'Ramayana' },
{ name: 'Mahabharata' },
{ name: 'Gautama%20Buddha' }
].map((value) => fetch(baseurl + value.name).then(res=>res.json()));
Promise.all(namePromises).then((response) => {
console.log(response);
});
Problems:
So, you have several things going on here.
http.get() is non-blocking and asynchronous. That means that the rest of your code continues to run while your multiple http.get() operations are running. That means you end up calling res.send(articlesData); before any of the http requests are complete.
.map() is not promise-aware or asynchronous-aware so it will not wait its loop for any asynchronous operation. As such, you're actually running ALL your http requests in parallel (they are all in-flight at the same time). The .map() loop starts them all and then they all finish some time later.
The data event for http.get() is not guaranteed to contain all your data or even a whole piece of data. It may just be a partial chunk of data. So, while your code may seem to get a complete response now, different network or server conditions could change all that and reading the result of your http requests may stop functioning correctly.
You don't show where names comes from in your request handler. If it's a statically declared array of objects declared in a higher-scoped variable, then this code has a problem that you're trying to modify the objects in that array in a request handler and multiple requests to this request handler can be conflicting with one another.
You don't show any error handling for errors in your http requests.
You're using await in a place it doesn't belong. await only does something useful when you are awaiting a promise.
Solutions:
Managing asynchronous operations in Javascript, particularly when you have more than one to coordinate) is a whole lot easier when using promises instead of the plain callback that http.get() uses. And, it's a whole lot easier to use promises when you use an http request interface that already supports promises. My goto library for built-in promise support for http requests in nodejs is got(). There are many good choices shown here.
You can use got() and promises to control the asynchronous flow and error handling like this:
const got = require('got');
app.get("/data_results", (req, res) => {
Promise.all(names.map(nameObj => {
let name = nameObj.name;
// create separate result object
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
return got(contentURL).json().then(wikiArticle => {
// Mapping the keys of the JSON data of query to its values.
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
return result;
});
})).then(articlesData => {
res.send(articlesData);
}).catch(err => {
console.log(err);
res.sendStatus(500);
});
});
This code attempts to fix all six of the above mentioned problems while still running all your http requests in parallel.
If your names array was large, running all these requests in parallel may consume too many resources, either locally or on the target server. Or, it may run into rate limiting on the target server. If that was an issue, you can run sequence the http requests to run them one at a time like this:
const got = require('got');
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await got(contentURL).json();
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
If you really want to stick with https.get(), then you can "promisify" it and use it in place of got():
function myGet(url, options = {}) {
return new Promise((resolve, reject) => {
https.get(url, options, (res) => {
res.setEncoding('utf8');
let rawData = '';
res.on('data', (chunk) => { rawData += chunk; });
res.on('end', () => {
try {
const parsedData = JSON.parse(rawData);
resolve(parsedData);
} catch (e) {
reject(e);
}
});
}).on('error', reject);
});
}
app.get("/data_results", async (req, res) => {
try {
let results = [];
for (let nameObj of names) {
let name = nameObj.name;
let result = { name };
let contentURL =
`https://en.wikipedia.org/w/api.php?
action=query&titles=${name}&prop=extracts&format=json&exintro=1&explaintext=false&origin=*`;
const wikiArticle = await myGet(contentURL);
let articleExtract = Object.keys(wikiArticle.query.pages).map(key => wikiArticle.query.pages[key])[0].extract;
result.article = articleExtract.substring(0, 350);
results.push(result);
}
res.send(results);
} catch (e) {
console.log(e);
res.sendStatus(500);
}
});
I need to change a parameters that defines what data should come from my requests, also this application needs to refresh on a regular time interval. If the user changes the parameter in the middle of an unfinished request things start to behave strange and some unexpected behavior occurs.
So my approach was to abort all previous requests before starting the new ones, but after using await controller.abort() it seems that the next requests are never triggered, Do I need to clear the signal or something like that?
const controller = new AbortController();
const async fetchData = (url, body = null) => {
let data;
const signal = controller.signal;
const headers = { ... };
response = await fetch(url, body ? {
method: "POST",
body: JSON.stringify(body),
signal,
headers
} : { headers, signal });;
data = await response.json()
return data
}
const firstData = await fetchData(url1, body1);
await controller.abort();
const secondData= await fetchData(url2, body2);
What happens is that secondData always is undefined, actually this second request never happens (looking on network traffic). If I stop source and try to run await fetchData(url2) after .abort() has executed it prompts an erros saying that Uncaught SyntaxError: await is only valid in async function or if I try to run it without await it returns a pending promise, but the actual request is nowhere to be seen in traffic tab.
Solved
Applying what was suggested on the ansewr I created wrapper on the function, to call new controllers everytime.
let controller = null;
let fetchData = null;
const initializeFetchData = () => {
const controller = new AbortController();
const async fetchData = (url, body = null) => {
let data;
const signal = controller.signal;
const headers = { ... };
response = await fetch(url, body ? {
method: "POST",
body: JSON.stringify(body),
signal,
headers
} : { headers, signal });;
data = await response.json()
return data
}
}
initializeFetchData();
const firstData = await fetchData(url1, body1);
controller.abort();
initializeFetchData();
const secondData= await fetchData(url2, body2);
You are using the sameAbortController for two different requests. After calling .abort() on theAbortController you have updated the state of it's AbortSignal which then renders the second request void.
You should use a separate AbortController for each request if you want this behavior. Of course, it is perfectly acceptable to reuse an AbortController for multiple fetch requests if you want to be able to abort all of them in one go.
A couple of other points...
.abort() is a synchronous method which returns void so you do not need the await prefix when calling .abort().
In your code example, the first request will never be aborted as you are awaiting the fetch request, which will complete before the .abort() is called.
This question already has answers here:
Why is my variable unaltered after I modify it inside of a function? - Asynchronous code reference
(7 answers)
Closed 2 years ago.
I am creating an array of objects from an endpoint with the fetch api. Each object in the response contains a new url and I scan this array for new objects. The purpose is to access the endpoint, obtain the urls and access these urls to store your objects and attributes. But when accessing the array with objects and from an index it returns undefined.
let url = [];
let pokemon = [];
function getPokemons(){
fetch("https://pokeapi.co/api/v2/type/11/")
.then(async response =>{
await response.json().then(data =>
data.pokemon.forEach(item =>
url.push(item.pokemon.url)
)
)
}).then( response => {
url.map(item =>
fetch(item).then(response =>{
response.json().then(data =>
pokemon.push(data)
)
})
)
})
}
getPokemons()
console.log(pokemon[1])
Solution for web browser :
The node-fetch module is not required, I just clean your code using async/await syntax.
// getPokemon is now an async function for cleaner syntax
async function getPokemon() {
const pokemon = [];
// Then we can use await
const response = await fetch("https://pokeapi.co/api/v2/type/11/"),
json = await response.json(),
urls = json.pokemon.map(item => item.pokemon.url);
for (const url of urls) {
const response = await fetch(url),
json = await response.json();
pokemon.push(json)
}
// We return the pokemon array instead of polluting global scope
return pokemon;
};
getPokemon().then(pokemon => {
console.log(pokemon)
});
Hope it help !
NodeJS solution
You must install node-fetch (see npmjs.org :
npm install -D node-fetch
Then you can use a fetch:
// Importing the node module.
// You can delete this line if your not using a node environment.
const fetch = require('node-fetch');
// getPokemon is now an async function for cleaner syntax
async function getPokemon() {
const pokemon = [];
// Then we can use await
const response = await fetch("https://pokeapi.co/api/v2/type/11/"),
json = await response.json(),
urls = json.pokemon.map(item => item.pokemon.url);
for (const url of urls) {
const response = await fetch(url),
json = await response.json();
pokemon.push(json)
}
// We return the pokemon array instead of polluting global scope
return pokemon;
};
getPokemon().then(pokemon => {
console.log(pokemon)
});
Explanation
fetch is not part of the Javascript language specification but is a Web API. Each browser may or not choose to implement it. Each Implementations may work differently under the hood but the Javascript API provided must match the standard (MDN Web Docs).
This is why you need a module to fetch the data.
EDIT : Adding solution for web browser environment
The problem is that the function getPokemons should be async.
You should await it before accessing:
getPokemons().then(()=> console.log(pokemon[1]))
// or if it inside an async function:
await getPokemons();
console.log(pokemon[1]);
But there's another reason also. You have internal promises outside the parent promise chain. I mean:
.then((response) => {
// this is array of promises
// you should return it to have ability await it on parent promise
url.map((item) =>
fetch(item).then((response) => {
response.json().then((data) => pokemon.push(data));
})
);
});
Your code might look like:
// if you really need global variable
let pokemon = [];
async function getPokemons() {
const result = await fetch("https://pokeapi.co/api/v2/type/11/")
.then((response) => response.json())
.then((data) =>
Promise.all(
data.pokemon.map((item) =>
fetch(item.pokemon.url).then((response) => response.json())
)
)
);
pokemon.push(...result);
}
getPokemons().then(() => {
console.log(pokemon[1]);
});
Or, the same result without global variables:
function getPokemons() {
return fetch("https://pokeapi.co/api/v2/type/11/")
.then(...)
.then(...);
}
getPokemons().then((pokemons) => {
console.log(pokemons[1]);
});
I was doing a pretty easy task of getting information from an API but then i got the response as ReadableStream and things start to turn dark as always. I am trying to use the cat-facts API.
URL : https://alexwohlbruck.github.io/cat-facts/docs/
and render the results with VueJS. What i found on the internet but this code just return some random numbers.
Here is the code:
created() {
this.getFacts();
},
methods: {
getFacts() {
let vm = this;
fetch('http://localhost:8080/facts')
.then(response => {
const reader = response.body.getReader();
return new ReadableStream({
start(controller) {
return pump();
function pump() {
return reader.read().then(({ done, value }) => {
// When no more data needs to be consumed, close the stream
if (done) {
controller.close();
return;
}
// Enqueue the next data chunk into our target stream
vm.facts = value;
controller.enqueue(value);
return pump();
});
}
}
})
})
.catch(err => console.error(err));
},
}
I am using vue.config.js to manage cors error:
module.exports = {
devServers = {
proxy: 'https://cat-fact.herokuapp.com/'
}
}
I made the request with POSTMAN and worked just well.
What happens if you do it like this:
async getFacts() {
let vm = this;
const response = await fetch('http://localhost:8080/facts');
const myJson = await response.json();
console.log(JSON.stringify(myJson));
}
Just wondring if you need to use pump at all.
EDIT
Without async/await
fetch('http://localhost:8080/facts')
.then(response => response.json())
.then(data => console.log(data));
I tried with ajax and a proxy made with php using guzzle to comunicate with the API and it worked! Just a simple line. But I am very confused about how fetch method works!
I'm trying to manipulate JSON data received from an API url (this is my first time handling this type of work)
The following function returns a promise of a 20 element array:
const articles = () => {
return fetch(url)
.then(res => res.json())
.then(post => post.articles);
};
Console view:
Now, I'd like to extract the elements from the array - I tried something like:
articles()[0].name
but this doesn't work and I'm not sure of an alternative way to go about this? Appreciate your help. Thanks
Your articles fucntion returns a promise. You have to consume the promise (more on MDN):
articles().then(articleArray => {
console.log(articleArray);
});
or within an async function:
const articleArray = await articles();
console.log(articleArray);
Side note: Your fetch code is missing a check for HTTP success (HTTP failure isn't a rejection). You're by far not the only person who misses out this check, so much so that I've written a post on my anemic blog about it. With the check:
const articles = () => {
return fetch(url)
.then(res => {
if (!res.ok) {
throw new Error("HTTP error " + res.status);
}
return res.json();
})
.then(post => post.articles);
};