I have a backend that returns large files as chunks of 206 partial content. I had logic:
fetch(url, query)
.then((resp)=>resp.json())
.then(...)
but surpricingly the code fails because the full json object does not get returned by the server.
Is there some commonly used library to solve this (monkeypatch fetch) or should I write a service-worker or a proxy for this? Why does the default browser fetch not support fetching the full object?
It's not surprising that JSON parsing fails on a partial object. fetch is just going to fetch what you ask it to (well, it follows redirects), so if the query has options for requesting a partial response, that's what you'll get.
You can build up the JSON string until you have the full response to parse, something along these lines:
async function getAll() {
// Start with a blank string
let json = "";
do {
// Get this chunk
const resp = await fetch(url, /*...query for next chunk...*/);
if (!resp.ok && resp.status !== 206) {
throw new Error(`HTTP error ${resp.status}`);
}
// Read this chunk as text and append it to the JSON
json += await resp.text();
} while (resp.status === 206);
return JSON.parse(json);
}
Obviously, that's a rough sketch, you'll need to handle the Range header, etc.
That code also requests each chunk in series, waiting for the chunk to arrive before requesting the next. If you know in advance what the chunk size is, you might be able to parallelize it, though it'll be more complicated with chunks possibly arriving out of sequence, etc.
It also assumes the server doesn't throw you a curveball, like giving you a wider range than you asked for that overlaps what you've already received. If that's a genuine concern, you'll need to add logic for it.
Related
I am struggling to understand when does the server return XML data vs HTML text data, and how come responseXML can return both? Is XML data returned only when making a POST request?
It's hard to actually test these, because I couldn't properly setup a PHP server, and making a POST request keeps returning 404 bad request, but when I make a GET request, I always get the HTML document in the responseText property, but when I try to use responseXML, I get null. So, if responseXML can return either HTML or XML, why does it not return the HTML document then?
Note: Before you accuse me of not doing any research. Let me tell you that I have been doing research for the past 3 days, and the book I'm reading just doesn't clarify these differences, and does not explain what exactly is XML in the first place. It says that XML data needs to be parsed to be displayed as a text, but doesn't explain why. It's all very ambiguous. So, I would appreciate if someone could clarify things for me.
POST requests return XML data, if the backend server is configured to return XML data. Completely depends on the server you're talking to, there's no way of predicting the behavior otherwise. Also, it's worth noting that every input can change the behavior of the server. E.g. if you provide a query with a specific value, the server could also return a CSS file, instead of an HTML or XML one.
This is an example of getting HTML from server response using fetch API.
fetch('https://someweb.com/api/list').then(function (response) {
// The API call was successful!
return response.text();
}).then(function (html) {
// Convert the HTML string into a document object
var parser = new DOMParser();
var doc = parser.parseFromString(html, 'text/html');
}).catch(function (err) {
// There was an error
console.warn('Something went wrong.', err);
});
[SOLVED] readable streams
Chunks (byte[]) are written (an nothing else) directly to the response outputStream , on client side this code inside an async function reads that response's chunks nicely as they arrive as uInt8Array
const response = await fetch(url);
const reader = response.body.getReader();
while (true) {
const { value, done } = await reader.read();
if (done) break;
console.log('Received', value);
}
console.log('Response fully received');
Code read at https://web.dev/fetch-upload-streaming/
[OLD]
I would like to do the same implementation as <img src="stream/video.mjpeg"> does when that url returns a response with contentType: multipart/x-mixed-replace. Somehow the browser reads "chucks" from an unfinished/on-going response.
Why? I want to stream .h264 format, with jMuxer (/example/h264.html), here WebSocket are used, but "feels wrong" because sockets are for bidirectional communication (I suposse this is true).
So, the response which <img> consumes succesfully and it is generated in server side(Java) looks like:
for(int[] bytes: listWithFrames){
responseOs.write((
"--BoundaryString\r\n" +
"Content-type: image/jpeg\r\n" +
"Content-Length: " +bytes.length +
"\r\n\r\n").getBytes());
responseOs.write(bytes);
responseOs.write("\r\n\r\n".getBytes());
responseOs.flush();}
First I've tried writting just bytes on the response ussing XMLHttpRequest like XMLHttpRequest/Sending_and_Receiving_Binary_Data, but oReq.response is set when response ends and oReq.responseText is not null only when oReq.responseType = "text";
Encoding in Base64 each bytes to make it visible on oReq.responseText and then decoding it can't be also the way because it adds a lot of size and oReq.responseText grows forever (could be avoided by doing multiple request and a buffer to keep feeding jMuxer)
Do I really need Node? making-http-request-and-receiving-multipart-x-mixed-replace-response-in-node-js I am looking for some kind of fetch that give me access to each recived binary (arraybuffer/uInt8array) part as they arrive.
I am sending a fetch request with node-fetch to the following url: http://fantasy.premierleague.com/api/bootstrap-static/ in order to get back some JSON-data. Accessing the URL in the browser, or sending a get-request with postman both returns the expected JSON data.
However, when i send the request from node, I get back an object that I do not know how to extract the data from (pics below).
I am not very experienced with node but I have made successful API calls before. Usually parsing the response with response.json() or JSON.parse(response) or response.body or response.toString() or some combinations of those have worked for me. I am half familiar with buffers and streams, but not confident and the solution might be related to those, however I cannot seem to figure it out.
I get som different errors and objects depending on what I try. I have tried using fetch and just plain http requests from node.
This call:
Returns this:
If I do JSON.parse(response) i get the following error:
Response.body looks like this:
Fetch returns a response stream as mentioned here in the answer to a similar question
You can read data in chunks and add the chunk to array and then do whatever you need to do with that data. A simpler approach would be to use npm request package. Here's an example.
const request = require('request');
let options = {json: true};
const url = 'http://fantasy.premierleague.com/api/bootstrap-static/'
request(url, options, (error, res, body) => {
if (error) {
return console.log(error)
};
if (!error && res.statusCode == 200) {
console.log(body);
// do something with JSON, using the 'body' variable
};
});
I'm creating a firebase application which uses firebase-cloud-functions.
index.js
exports.auth = functions.https.onRequest((request, response) => {
response.status(200).send({
status : "Some Status"
});
}
This is very simple functions. I want to make a POST request on the endpoint with some payload. When I tested the API using Firebase Cloud Function Emulator and POSTman with bad json
{
"phoneNumber: "9632725300"
}
The server just crashed! My question is how to handle the bad request in firebase functions like these.
with this error
The server did not crash. You have sent it a bad request (malformed JSON) and it responded perfectly with a status code 400 which is "Bad Request".
You'd rather correct your JSON...
EDIT:
If you really wanted to be able to send invalid JSON, you could do so by circumventing the JSON body parser. To do so, you could either change your request to have a content-type header set to "text/plain". This content-type will use the text body parser, which will not parse any JSON.
Note that doing so will require you to handle the JSON parsing yourself, but will permit to handle to error yourself using a try-catch.
let json;
try {
json = JSON.parse(json);
} catch (e) {
// Handle JSON error.
}
Taken from https://firebase.google.com/docs/functions/http-events
What you're experiencing is not actually a server crash. In fact, technically, by using Cloud Functions, you don't have a server to crash. (For this reason they're called "Serverless Infrastructure") Each request / operation you perform on Cloud Functions is kind of like a brand new server. Which is actually what's fantastic about Cloud Functions in general. (This is an overly simplified explanation, I'd suggest reading up a bit more about it for a better in depth explanation)
That being said, from what I understand you're trying to figure out if the JSON you got is invalid (bad) or not. Occasionally, when I have to hook up a bunch of external services, rarely, but sometimes, they return a bad JSON that my Cloud Functions can't parse, therefore throws an error.
The solution is to put your JSON.parse in to a separate function and a try / catch block like this:
function safelyParseJSON (json) {
var parsed;
try {
parsed = JSON.parse(json);
} catch (e) {
// BAD JSON, DO SOMETHING ABOUT THIS HERE.
}
return parsed; // will be undefined if it's a bad json!
}
function doSomethingAwesome () {
var parsedJSON = safelyParseJSON(data);
// Now if parsedJSON is undefined you know it was a bad one,
// And if it's defined you know it's a good one.
}
With this helper function, if you have to deal with a lot of external JSON resources, you can easily determine if the JSON you're trying to parse is good, and if not, you can at least handle the error your way.
Hope this helps :)
{\n\t"phoneNumber: "9632725300"\n}
From the screenshot, I see that the JSON is invalid or malformed. It contains newline (\n) and tab space (\t) characters. Also, the key "phoneNumber" is not wrapped in double quotes, which again invalidates the JSON.
Here's a valid format of the JSON that the server should receive
{
"phoneNumber": "9632725300"
}
I am working on a simple AWS Lambda function in Javascript (Node 6.x) which should 'proxy' an RSS of an italian news provider.
This is the code of the function:
var http = require("http")
exports.handler = (event, context, callback) => {
http.get("http://www.milanotoday.it/rss/", (response) => {
response.setEncoding("utf8")
let xml = ""
response.on("data", (chunk) => { xml += chunk })
response.on("end", () => { callback(null, xml) })
})
}
It works, or at least it loads the response inside the xml variable.
I can't get why the string is something like this:
"<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<rss version=\"2.0\" xmlns:content=\"http://purl.org/rss/1.0/modules/content/\"...
It seems to be sort of 'escaped'.
Can somebody help me?
Thank you in advance...
http.get is probably working just fine. If you run your code outside of the AWS ecosystem, you'll see that you are getting unescaped XML from http.get. The problem is that Lambda wants you to return JSON so when you pass the string to the callback it gets escaped.
If you are running this function through API Gateway you can do a transform in the integration response. The way to do this is to return an object from your lambda:
callback(null, {myXML: xml}))
Then in the API Gateway go to the integration response area under your GET (or POST) resource and click the arrow next to the 200 response. This should reveal an area for body mappings. You want to add a mapping for application/xml and then add something like:
#set($inputRoot = $input.path('$'))
$inputRoot.myXML
This should get you nice clean XML. It's a little hard to describe so I'll post a screen shot that might help:
Double quotes have to be escaped inside a double quoted string (Makes sense, right?). Now you just have to parse the XML, using for instance xml2js