We Keep Coding

Send some data to server without REST in JS

As far as i understand all what REST do is standartize a data sended to server by adding some headers. For example REST request can generate a line of bytes like so: POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive and finish it with some user input.
Now im just playing with writing my own JS server and here is my question: is there a way in JS to send some data(bytes) without this REST addings like headers/method and will it work for browsers and HTTP protocol itself?
For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
Iv tried to google and end up that XMLhttpRequest and fetch both require some CRUD method to be specified and adding some headers in request anyway.

HTTP requests:
Need to specify the method
Need to specify the Host as a header (in HTTP 1.1. and newer)
Will include some other request headers automatically when make using JS from a browser
This has nothing to do with REST. It's just how HTTP works.
A non-HTTP protocol could avoid having that. JavaScript in a browser has no mechanisms that allow making non HTTP requests.
You might want to research WebSocket which allows two way communication over a single connection … but that is a bootstrapped by HTTP so doesn't really fulfil your requirement.

For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
I suspect you're misunderstanding what a request is, on a fundamental level. Without POST (the method), /qwe (the path), HTTP/1.1 (the protocol) and 127.0.0.1 (the address) there is no way for your computer to know where and how to send the data. These are necessary if you want to communicate with a server, and removing them will mean your code no longer works.
You're working with very low-level data here, which is probably not what you actually want to be doing. There are some packages which will let you ignore the how and what of the request, and focus on just the data inside it. Express might be a good place to start. You can set up a simple express server to handle requests on specific paths, and reply with data that your frontend can then use.
A REST API is a high-level concept and largely unrelated to what you're asking about.

Getting JSON from the Arduino server. Problems with CORS

I have an Arduino server that runs on IP 192.168.0.177. If you go to this address, it returns data in JSON format. For example, in a React application, I get this JSON something like this:
const [data, setData] = useState([])
axios.get('http://192.168.0.177')
.then((res) => setData(res.data))
.catch((err) => console.log(err))
In the Arduino sketch itself part of the code is like this:
client.println("HTTP/1.1 200 OK");
client.println("Access-Control-Allow-Origin: *");
client.println("Access-Control-Allow-Methods: GET");
client.println("Content-Type: application/json");
client.println("Connection: close");
client.println();
client.println("[");
client.println("{ \"id\": 1, \"name\": \"John\" }, ");
client.println("{ \"id\": 2, \"name\": \"Paul\" } ");
client.println("]");
If you run the React app on a local network, everything works fine. Localhost has http, not https, and it sends a request to the http server, so everything works.
The problem already appeared when I launched my React app on the global network. In other words, the address is something like this: https://app.com. Now if I send a request to the Arduino via https, the console writes an error that you can't send a request from https to http. Then I changed the query in the React code, or rather added only the 's'symbol:
axios.get('https://192.168.0.177')
^
Then there was another problem related to CORS. The console displays this error:
Request from an external source is blocked: the single source Policy
prohibits reading a remote resource on https://192.168.0.177/.
(Reason: the CORS request failed).
I read all about CORS and everywhere they write that you need to add the header "Access-Control-Allow-Origin: *"to the server. But it was in my Arduino code all the time.
I'm sorry, I'm not very good at this, so I'm asking for help. Where did I make a mistake?

As far as I understand if you don't have the "W5500 Ethernet Shield S", which is a security enhanced version, you will not be able to deliver proper https secured website answers.
And because of the new security rules, that every modern browser enforces this will be a problem.
If you only need this in a develop environment, you could use special startup flags for your browser, to make it ignore those limitations. Or something like this for Chrome for example:
How to get Chrome to allow mixed content?
If you need this in production environment you could put a proxy server in between, that calls http on the arduino and forwards the request over https. Such a server could be build with a raspberry pi.
But then again you could probably build the whole appliance just with a raspberry. That would probably even be cheaper then the Arduino plus W5500 shield and give you way more options. I personally build everything that needs network with raspberries theses days, because it gives me way more options.

How can I get the raw HTTP message body using the request library in Node.js?

The npm-request library allows me to construct HTTP requests using a nice JSON-style syntax, like this.
request.post(
{
url: 'https://my.own.service/api/route',
formData: {
firstName: 'John',
lastName: 'Smith'
}
},
(err, response, body) => {
console.log(body)
}
);
But for troubleshooting, I really need to see the HTTP message body of the request as it would appear on the wire. Ideally I'm looking for a raw bytes representation with a Node.js Buffer object. It seems easy to get this for the response, but not the request. I'm particularly interested in multipart/form-data.
I've looked through the documentation and GitHub issues and can't figure it out.

Simplest way to do this is to start a netcat server on any port:
$ nc -l -p 8080
and change the URL to localhost in your code:
https://localhost:8080/v1beta1/text:synthesize?key=API_KEY
Now, any requests made will print the entire, raw HTTP message sent to the localhost server.
Obviously, you won't be able to see the response, but the entire raw request data will be available for you to inspect in the terminal you have netcat running

I figured out how to dump the HTTP message body with Request. In both cases, I'm just copying the same approach that request uses internally.
Multipart Form Uploads
req._form.pipe(process.stdout);
URL-encoded Forms
console.log(req.body);

You could try #jfriend00 suggestion an use a network sniffer like wireshark but as you're fetching an https URL this might not be the easiest route as it requires some setup to intercept TLS connections.
So maybe it would be enough turning on debug mode for the request module itself, you can do that by simply setting require('request').debug = true. As a third option you could go with the dedicated debug module for request here which allows you to view request and response headers and bodies.

I can think of a number of ways to see the bytes of the request:
Turn on debugging in the request module. There are multiple ways to do that documented here including setting NODE_DEBUG=request or require('request').debug = true or using the request-debug module.
Use a network sniffer to see what's actually being sent over the socket, independent of your node.js code.
Create your own dummy http server that does nothing but log the exact incoming request and send your same request to that dummy server so it can log it for you.
Create or use a proxy (like nginx) that can dump the exact incoming request before forwarding it on to its final destination and send the request to the proxy.
Step through the sending of the request in the debugger to see exactly what it is writing to the socket (this may be time consuming, particularly with async callbacks, but will eventually work).

you could use a nodejs server capable of logging the raw request/response string , then direct your request to that server
i gave an example using both http and https server - no dependencies
nodejs getting raw https request and response

Why are my server sent events arriving as a batch?

I have a Java 8 / Spring4-based web application that is reporting the progress of a long-running process using Server Sent Events (SSEs) to a browser-based client running some Javascript and updating a progress bar. In my development environment and on our development server, the SSEs arrive in near-real-time at the client. I can see them arriving (along with their timestamps) using Chrome dev tools and the progress bar updates smoothly.
However, when I deploy to our production environment, I observe different behaviour. The events do not arrive at the browser until the long-running process completes. Then they all arrive in a burst (the events all have the timestamps within a few hundred milliseconds of each other according to dev tools). The progress bar is stuck at 0% for the duration and then skips to 100% really quickly. Meanwhile, my server logs tell me the events were generated and sent at regular intervals.
Here's the relevant server side code:
public class LongRunningProcess extends Thread {
private SseEmitter emitter;
public LongRunningProcess(SseEmitter emitter) {
this.emitter = emitter;
}
public void run() {
...
// Sample event, representing 10% progress
SseEventBuilder event = SseEmitter.event();
event.name("progress");
event.data("{ \"progress\": 10 }"); // Hand-coded JSON
emitter.send(event);
...
}
}
#RestController
public class UploadController {
#GetMapping("/start")
public SseEmitter start() {
SseEmitter emitter = new SseEmitter();
LongRunningProcess process = new LongRunningProcess(emitter);
process.start();
return emitter;
}
}
Here's the relevant client-side Javascript:
EventSource src = new EventSource("https://www.example.com/app/start");
src.addEventListener('progress', function(event) {
// Process event.data and update progress bar accordingly
});
I believe my code is fairly typical and it works just fine in DEV. However if anyone can see an issue let me know.
The issue could be related to the configuration of our production servers. DEV and PROD are all running the same version of Tomcat. However, some of them are accessed via a load balancer (F5 in out case). Almost all of them are behind a CDN (Akamai in our case). Could there be some part of this setup that causes the SSEs to be buffered (or queued or cached) that might produce what I'm seeing?
Following up on the infrastructure configuration idea, I've observed the following in the response headers. In the development environment, my browser receives:
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: Keep-Alive
Content-Type: text/event-stream;charset=UTF-8
Keep-Alive: timeout=15, max=99
Pragma: no-cache
Server: Apache
Transfer-Encoding: chunked
Via: 1.1 example.com
This is what I'd expect for an event stream. A chunked response of an unknown content length. In the production environment, my browser receives something different:
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Connection: keep-alive
Content-Type: text/event-stream;charset=UTF-8
Content-Encoding: gzip
Content-Length: 318
Pragma: no-cache
Vary: Accept-Encoding
Here the returned content has a known length and is compressed. I don't think this should happen for an event stream. It would appear that something is converting my event stream into single file. Any thoughts on how I can figure out what's doing this?

It took a significant amount of investigation to determine that the cause of the issue was the elements in our network path. So the code above is correct and safe to use. If you find SSE buffering you will most likely want to check the configuration of key networking elements.
In my case, it was Akamai as our CDN and the use of an F5 device as a load balancer. Indeed it was the fact that both can introduce buffering that made it quite difficult to diagnose the issue.
Akamai Edge servers buffer event streams by default. This can be disabled through the use of Akamai's advanced metadata and controlled via custom behaviours. At this time, this cannot be controlled directly through Amakai's portal, so you will need to get their engineers to do some of the work for you.
F5 devices appear to default to buffering response data as well. Fortunately, this is quite simple to change and can be done yourself via the device's configuration portal. For the virtual device in question, go to Profile : Services : HTTP and change the configuration of Response Chunking to Preserve (in our case it had defaulted to Selective).
Once I made these changes, I began to receive SSEs in near real-time from our PROD servers (and not just our DEV servers).

Have you tried alternative browsers? I'm trying to debug a similar problem in which SSE works on an iPhone client but not on MacOS/Safari or Firefox.
There may be a work-around for your issue - if the server sends "Connection: close" instead of keep-alive, or even closes the connection itself, the client should re-connect in a few seconds and the server will send the current progress bar event.
I'm guessing that closing the connection will flush whatever buffer is causing the problem.

This is not a solution to this question exactly, but related to SSE, Spring and use of compression.
In my case I had ziplet CompressionFilter configured in my Spring application and it was closing the Http Response and causing SSE to fail. This seems to be related to an open issue in the ziplet project. I disabled the filter and enabled Tomcat compression in application.properties (server.compression.enabled=true) and it solved the SSE issue.
Note that I did not change the default compressionMinSize setting, which may have something to do with SSE traffic not getting compressed and passing through.

The webpack dev server also buffers server sent events when using the proxy setting.

Firefox combining 'Connection: keep-alive, Upgrade' conflicts with mobile operator proxy

I have a WebSocketServer running on a server box, with a website attempting to connect to it and send back and forth information.
I have noticed that on WiFi it works perfectly on all the browsers I have tested, however over Mobile Data Firefox. I intercepted and edited headers and managed to reproduce the problem. Firefox is sending a combined header Connection: keep-alive, Upgrade in the request. Chrome in comparison is just sending Connection: Upgrade. My theory is that when the request passes through the mobile data provider's proxy, as well as adding their own identifying headers, it re-parses all of the other headers, and does not understand a combined header. This is confirmed by the fact that at the server end, the request is received (from Firefox) but the Connection header is truncated to Connection: keep-alive. If I manually remove the keep-alive from the Connection header using the interception program, the problem is solved.
I don't need the keep-alive part of the request (in fact if anything I would prefer it not to be enabled) so I'm asking if there is a way to stop Firefox sending it without using about:config etc (e.g. in JS or HTML), as I would like for this to work for the general end-user.
Many thanks,
Richard

I had a similar problem, henceforth resolved.
In my case, the problem was that my hosting provider had a proxy which was not dealing correctly with the Connection and/or Upgrade headers. Indeed, these headers are hop-by-hop and as such:
Hop-by-hop headers
are meaningful only for a single transport-level connection and must not be retransmitted by proxies or cached. Such headers are: Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, TE, Trailer, Transfer-Encoding and Upgrade. Note that only hop-by-hop headers may be set using the Connection general header.
Souce: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers
Shortly, these headers are not retransmitted but somehow interpreted before being passed to your server. When these headers are sent by Firefox, this phase of interpretation becomes critical since the value associated to the Connection header is more "complicated" than that sent by other browsers, i.e.
Firefox sends Connection: keep-alive, Upgrade
Chrome/Edge/... sends Connection: Upgrade
Solution : I simply told my hosting provider that only Connection: keep alive arrives to my server when one sends Upgrade: <my_protocole> AND Connection: keep alive, Upgrade (and he had the possibility to correct the issu within 72 hours).