We Keep Coding

How can I get the raw HTTP message body using the request library in Node.js?

The npm-request library allows me to construct HTTP requests using a nice JSON-style syntax, like this.
request.post(
{
url: 'https://my.own.service/api/route',
formData: {
firstName: 'John',
lastName: 'Smith'
}
},
(err, response, body) => {
console.log(body)
}
);
But for troubleshooting, I really need to see the HTTP message body of the request as it would appear on the wire. Ideally I'm looking for a raw bytes representation with a Node.js Buffer object. It seems easy to get this for the response, but not the request. I'm particularly interested in multipart/form-data.
I've looked through the documentation and GitHub issues and can't figure it out.

Simplest way to do this is to start a netcat server on any port:
$ nc -l -p 8080
and change the URL to localhost in your code:
https://localhost:8080/v1beta1/text:synthesize?key=API_KEY
Now, any requests made will print the entire, raw HTTP message sent to the localhost server.
Obviously, you won't be able to see the response, but the entire raw request data will be available for you to inspect in the terminal you have netcat running

I figured out how to dump the HTTP message body with Request. In both cases, I'm just copying the same approach that request uses internally.
Multipart Form Uploads
req._form.pipe(process.stdout);
URL-encoded Forms
console.log(req.body);

You could try #jfriend00 suggestion an use a network sniffer like wireshark but as you're fetching an https URL this might not be the easiest route as it requires some setup to intercept TLS connections.
So maybe it would be enough turning on debug mode for the request module itself, you can do that by simply setting require('request').debug = true. As a third option you could go with the dedicated debug module for request here which allows you to view request and response headers and bodies.

I can think of a number of ways to see the bytes of the request:
Turn on debugging in the request module. There are multiple ways to do that documented here including setting NODE_DEBUG=request or require('request').debug = true or using the request-debug module.
Use a network sniffer to see what's actually being sent over the socket, independent of your node.js code.
Create your own dummy http server that does nothing but log the exact incoming request and send your same request to that dummy server so it can log it for you.
Create or use a proxy (like nginx) that can dump the exact incoming request before forwarding it on to its final destination and send the request to the proxy.
Step through the sending of the request in the debugger to see exactly what it is writing to the socket (this may be time consuming, particularly with async callbacks, but will eventually work).

you could use a nodejs server capable of logging the raw request/response string , then direct your request to that server
i gave an example using both http and https server - no dependencies
nodejs getting raw https request and response

Create a Batch HTTP API With multipart response

Actually, I´ve create a Batch HTTP API that receives a JSON array with many different requests to our backend server. The Batch API just call all of these requests to a load balancer, wait for the return of all of them and return a new JSON to the client.
The client receives a huge JSON array response with its indices in the same position as the request, so it is easy to know what response is addressed for what request.
The motivation for this API was to solve the 5 browser simultaneous connections and improve performance as the Batch API has a much more direct access to the server (we do not have a reverse proxy or a SSL server between then).
The service is running pretty well, but now I have some new requirements as it is gaining more use. First, the service can use a lot of memory as it has a buffer for each request that will only be flushed when all responses are ready (I am using an ordered JSON Array). More, since it can take some time to all requests be delivered, the client will need to wait everything be processed before receiving a single byte.
I am planning change the service to return each response as soon it is available (and solve both issues above). And would like to share and validate my ideas with you:
I will change the response from a JSON response to a multipart response.
The server will include, for every part, the index of the response
The server will flush the response once its available
The client XHR will need to understand multipart content type response and be able to process each part as soon as it is available.
I will create a PoC to validate every step, but at this moment I would like to validate the idea and hear some thoughts about it. Here some doubts I have about the solution:
From what I read, I am in doubbt of that content-type is right for the response. multipart/mixed? multipart/digest?
Can I use an accept request header to identify if the client is able to handle the new service implementation? If so, what is the right accept header for this? My plan is to use the same endpoint but very accept header.
How can I develop a XHR client that is able to process many parts of a single response as soon as they are available? I found some ideias on the Web but I am not entirely confident with then.

I will change the response from a JSON response to a multipart
response.
The server will include, for every part, the index of the
response
The server will flush the response once its available
The
client XHR will need to understand multipart content type response and
be able to process each part as soon as it is available.
The XHR protocol will not support this work flow through a single request from the client. Since XHR relies heavily on the HTTP protocol for communications, XHR follows the HTTP connection rules. The first and most important rule: HTTP connections are always initiated by the client. Another rule: XHR returns the entire content-body or fails.
The implications for your workflow is that each part of the multipart response must be requested individually by the client.
From what I read, I am in doubbt of that content-type is right for the
response. multipart/mixed? multipart/digest?
You should be in doubt as there is no provision in the specfication to do this. The response-type attribute is limited to the empty string (default), "arraybuffer", "blob", "document", "json", and "text". it is possible to set the override MIME type header, but that does not change the response type. Event given that case, the XHR spec is very clear about what it will send back. It is one of the types listed above as documented here.
Can I use an accept
request header to identify if the client is able to handle the new
service implementation? If so, what is the right accept header for
this? My plan is to use the same endpoint but very accept header.
Custom HTTP headers are designed to assist us in telling the server what our capabilities are on the client. This is easily done. it doesn't necessarily have to be in the accept header (as that also is a defined list of MIME types).
How
can I develop a XHR client that is able to process many parts of a
single response as soon as they are available? I found some ideias on
the Web but I am not entirely confident with then.
XHR is processed natively by the client and cannot be overridden for all sorts of security reasons. So this is unlikely to be available as a solution for this reason.
Note: ordinarily one might suggest the use of a custom version of Chromium, but your constraints do not allow for that.

How to process a request with empty content-type and content-length header

A software that i'm using(https://camlytics.com/) sends a request to a webhook whenever a particular event occurs, now i want to process that request but the problem is that the request is sent with the following headers as empty
content-length
content-type
Due to this reason my node code completely ignores the request. I have verified that the request is actually being sent via creating a webhook # webhook.site.
I fail to understand if webhook.site can show and process that request, why cant node do it? the code easily processes all other get requests.
Would appreciate if someone could either
help me process the request as in make it accessible via the code
if somone with experience on camlytics help me configure it in such a way that i can configure the headers of the request.
I have tried this on serverless azure function which is supposed to trigger for all HTTP requests but event that doesnt trigger, neither does it trigger on my local NODE server.
This is the request details that webhook.site shows me

Camlytics webhooks seem to work OK if you add Content-Type application/json to the Headers properties box for the HTTP Response node.

Sending a 'simple' POST request with jQuery, but getting CORS anyway

The CORS specification states that if a HTTP request is considered 'simple', no CORS and/or preflight is needed.
I'm trying to do a HTTP request that appears to have these conditions:
I'm not setting custom HTTP headers.
I'm using a POST method.
I'm using application/x-www-form-urlencoded.
Code sample:
$.ajax({
type: 'POST',
url: 'http://example.org/',
data: {foo: 'bar'}
});
However, when running this, the request is still preflighted with OPTIONS (which fails). Is there something obvious I'm missing?
A few references to simple requests:
https://w3c.github.io/webappsec-cors-for-developers/#cross-origin-send-permissions-simple-safelisted-request
https://developer.mozilla.org/en-US/docs/Web/HTTP/Access_control_CORS#Simple_requests

CORS restrictions affect all requests going from one domain to another. example: localhost -> example.com. I end up just going to my example.com server-side code and make sure I enable requests from myotherexample.com where I am making calls from. Do this using the CORS header while developing locally
Access-Control-Allow-Origin: *
Another example when you are ready for production
Access-Control-Allow-Origin: https://myotherexample.com

I realized my mistake when re-reading the documentation.
What I am doing is indeed a simple request.
The request was actually being sent to the server without an OPTIONS request and succeeded!
However, I was not allowed to read the response when it came back. So the true difference between simple and non-simple CORS requests is:
For simple requests a preflight is not needed, but the server still needs to respond with CORS headers.
So my options are as follows:
I ignore the error. The request succeeded after all, I just can't read the response.
I implement CORS server-side anyway. In my case I can't, because I don't control the target server.
I use a html form to submit the data, call .submit() on it and target a hidden iFrame.
I proxy the request through a server that I do control.
Future:
I think, but I'm not sure, that the new Fetch API also allows a mode where you can make HTTP requests cross-domain, opt-out of CORS and simply be denied access to the HTTP response. If this is correct, then this would be the ideal way to do this (to me). But I don't know 100% certain if this is indeed how this works.

jquery default caching is not working :(

I have set cache:true in my jquery get service request. But this is not working. When i run the fiddler parallel, everytime my url is being hit rather than response fetched from cache. How do I test my data is fetched from cache. I believe using fiddler is the correct approach to test? Any pointers will be really helpful to me. Also I have noted down in my http headers it always send the below two parameters though I have set as cache:true
Cache-Control:no-cache
Pragma:no-cache
But my http response headers which is comes from server is not having these above two http attributes.
CodePen : http://codepen.io/selvaonline/pen/gpyLRz