Im trying to send a large amount of data using a GET request which is naturally leading to the HTTP 414 URI too long error. The RESTful server I am communicating with is not under my control and I can therefor not send it it via POST as the server will not allow this.
Is there any way around this such that I can still send the same amount of data over in a GET request?
Kieran, the answer is no. GET method, as per HTTP standard, does not allow a body as part of the request. You only have the URI to pass data. You're going to need to break your request into a series of smaller requests and group your results afterwards. You'll need to approach this problem with it's asynchronous nature in mind - you might want to look into promises to help you ensure that you've gathered all the results from a series of requests.
Related
So, I am in the process of learning JavaScript and was wondering whether I could use a custom header field of a GET request to pass certain parameters to tell the Node-Server how to handle the data passed in the URL.
As I understand, this is not standard practice, but are there actually technical reasons that would prohibit such a use case?
It is possible to pass the actions via Headers to the Server.
Though it is not in practice because it brings security risks to the application. Complete control of operation is with the Client and Clients can be compromised to do some malicious activity.
The whole point of having a Back End in between your app data and your client is to mitigate the security risks by having more control over the operation.
One thumb rule that you can continue with: The Clint (The Website, Apps etc) are dumb and its the responsibility of Server to take preventive measures.
That's why passing operations via headers could be dangerous.
But if you are asking operations like Add, Delete, Update etc; you can use REST API specs. This gives you different type of HTTP request for different kind of operations.
GET - Read
POST - Create
PUT - Update
DELETE - Delete
And response can be a good readable message and/or HTTP Codes like
200 - Ok
404 - Not Found
403 - Unauthorized
204 - Ok but no content
More descriptive list: https://httpstatuses.com/
I would like to retrieve the size of the request body to filter out the large HTTP post request.
So, How can I get the length of the request?
As per its documentation, Cloud Functions has a hard limit of 10 MB for HTTP request and response payloads. For more detail, you can check documentation.
and also, you cannot extend the limit as well(I have not found yet).
Just to check the length of your request, you can use req.get("content-length").
Actually, I´ve create a Batch HTTP API that receives a JSON array with many different requests to our backend server. The Batch API just call all of these requests to a load balancer, wait for the return of all of them and return a new JSON to the client.
The client receives a huge JSON array response with its indices in the same position as the request, so it is easy to know what response is addressed for what request.
The motivation for this API was to solve the 5 browser simultaneous connections and improve performance as the Batch API has a much more direct access to the server (we do not have a reverse proxy or a SSL server between then).
The service is running pretty well, but now I have some new requirements as it is gaining more use. First, the service can use a lot of memory as it has a buffer for each request that will only be flushed when all responses are ready (I am using an ordered JSON Array). More, since it can take some time to all requests be delivered, the client will need to wait everything be processed before receiving a single byte.
I am planning change the service to return each response as soon it is available (and solve both issues above). And would like to share and validate my ideas with you:
I will change the response from a JSON response to a multipart response.
The server will include, for every part, the index of the response
The server will flush the response once its available
The client XHR will need to understand multipart content type response and be able to process each part as soon as it is available.
I will create a PoC to validate every step, but at this moment I would like to validate the idea and hear some thoughts about it. Here some doubts I have about the solution:
From what I read, I am in doubbt of that content-type is right for the response. multipart/mixed? multipart/digest?
Can I use an accept request header to identify if the client is able to handle the new service implementation? If so, what is the right accept header for this? My plan is to use the same endpoint but very accept header.
How can I develop a XHR client that is able to process many parts of a single response as soon as they are available? I found some ideias on the Web but I am not entirely confident with then.
I will change the response from a JSON response to a multipart
response.
The server will include, for every part, the index of the
response
The server will flush the response once its available
The
client XHR will need to understand multipart content type response and
be able to process each part as soon as it is available.
The XHR protocol will not support this work flow through a single request from the client. Since XHR relies heavily on the HTTP protocol for communications, XHR follows the HTTP connection rules. The first and most important rule: HTTP connections are always initiated by the client. Another rule: XHR returns the entire content-body or fails.
The implications for your workflow is that each part of the multipart response must be requested individually by the client.
From what I read, I am in doubbt of that content-type is right for the
response. multipart/mixed? multipart/digest?
You should be in doubt as there is no provision in the specfication to do this. The response-type attribute is limited to the empty string (default), "arraybuffer", "blob", "document", "json", and "text". it is possible to set the override MIME type header, but that does not change the response type. Event given that case, the XHR spec is very clear about what it will send back. It is one of the types listed above as documented here.
Can I use an accept
request header to identify if the client is able to handle the new
service implementation? If so, what is the right accept header for
this? My plan is to use the same endpoint but very accept header.
Custom HTTP headers are designed to assist us in telling the server what our capabilities are on the client. This is easily done. it doesn't necessarily have to be in the accept header (as that also is a defined list of MIME types).
How
can I develop a XHR client that is able to process many parts of a
single response as soon as they are available? I found some ideias on
the Web but I am not entirely confident with then.
XHR is processed natively by the client and cannot be overridden for all sorts of security reasons. So this is unlikely to be available as a solution for this reason.
Note: ordinarily one might suggest the use of a custom version of Chromium, but your constraints do not allow for that.
I've a JS (Angular) client that makes a PUT request (REST API) to server and server sends back a large payload that I'm not using in the client currently.
Is there a way to just fire the request and ignore any response that comes back? The main need here is to avoid the data cost incurred by receiving that payload. I've looked at closing the connection once the request is fired, but am not sure if that's the best way to handle this.
If able, I think the only way to change this would be to change the api endpoint to not include a payload from the put request.
I'm assuming you are using angular's http class and using Observables. But even if you aren't, your angular client is going to need to read the response status sent back from the server to determine whether or not the put request was successful or not. In order to read the status, you'll need to response, and unfortunately the full response sent from the server.
You could close the connection right after the request, but as I've mentioned you'll have no way of knowing whether or not the request was successful.
To ignore the request just don't do anything if the request is successful.
If you don't want the request to exist at all then do it on the backend.
I can't seem to figure out a way to ignore the for(;;); in the response body of my cross domain JSONP requests. I am doing this on my own servers, nothing else going on here. I am trying to include that for(;;); inside the response body of my callback as such:
_callbacks_.callback(for(;;);[jsondata....]);
but how can I remove it from the response body before the JS code gets parsed? I am using the Google Closure Library btw.
Ok I think I figured it out.
The reason why the for(;;); is there is to prevent cross-domain data requests of certain information. So basically if you have information you are trying to protect you go through a normal Ajax JSON channel and if you are storing data on multiple servers you deal with it on server level.
JSONP requests are actually a remote script inclusion, which means whatever the server outputs is actual Javascript code, so if you have a for(;;); before your _callbacks_.callback(); the code will be executed on the origin domain on request success. If it's an infinite for loop, it will obviously jam the page.
So the normal implementation method is the following:
Send a normal Ajax request to a file located on the same server.
Perform the server level stuff and send requests to external servers via encrypted CURL.
Add security to the server response(a for(;;); or while(1); or throw(1); followed by a <prevent eval statements> string.
Get the response as a text string.
Remove your security implementations from the string.
Convert the string(which is now a "JSON string") to a JS Object/Array etc with a standard JSON parser.
Do whatever you want to do with the data.
Just thought I should put this out here in case someone else will Google it in the future, as I didn't find proper information by Google-ing. This should help prevent cross domain request forgery.