I have an Arduino server that runs on IP 192.168.0.177. If you go to this address, it returns data in JSON format. For example, in a React application, I get this JSON something like this:
const [data, setData] = useState([])
axios.get('http://192.168.0.177')
.then((res) => setData(res.data))
.catch((err) => console.log(err))
In the Arduino sketch itself part of the code is like this:
client.println("HTTP/1.1 200 OK");
client.println("Access-Control-Allow-Origin: *");
client.println("Access-Control-Allow-Methods: GET");
client.println("Content-Type: application/json");
client.println("Connection: close");
client.println();
client.println("[");
client.println("{ \"id\": 1, \"name\": \"John\" }, ");
client.println("{ \"id\": 2, \"name\": \"Paul\" } ");
client.println("]");
If you run the React app on a local network, everything works fine. Localhost has http, not https, and it sends a request to the http server, so everything works.
The problem already appeared when I launched my React app on the global network. In other words, the address is something like this: https://app.com. Now if I send a request to the Arduino via https, the console writes an error that you can't send a request from https to http. Then I changed the query in the React code, or rather added only the 's'symbol:
axios.get('https://192.168.0.177')
^
Then there was another problem related to CORS. The console displays this error:
Request from an external source is blocked: the single source Policy
prohibits reading a remote resource on https://192.168.0.177/.
(Reason: the CORS request failed).
I read all about CORS and everywhere they write that you need to add the header "Access-Control-Allow-Origin: *"to the server. But it was in my Arduino code all the time.
I'm sorry, I'm not very good at this, so I'm asking for help. Where did I make a mistake?
As far as I understand if you don't have the "W5500 Ethernet Shield S", which is a security enhanced version, you will not be able to deliver proper https secured website answers.
And because of the new security rules, that every modern browser enforces this will be a problem.
If you only need this in a develop environment, you could use special startup flags for your browser, to make it ignore those limitations. Or something like this for Chrome for example:
How to get Chrome to allow mixed content?
If you need this in production environment you could put a proxy server in between, that calls http on the arduino and forwards the request over https. Such a server could be build with a raspberry pi.
But then again you could probably build the whole appliance just with a raspberry. That would probably even be cheaper then the Arduino plus W5500 shield and give you way more options. I personally build everything that needs network with raspberries theses days, because it gives me way more options.
Related
I need to make a request to this other website and I need to send a cookie alongside it. I have no control over this other website so I cannot modify its cors policy or allow credentials in any meaningful way.
So, I decided to convert my website to a react app that will run contained inside an electron web browser. I thought that as the electron has full control over the computer, I would be able to send a normal request without having to deal with cors, but it seems, that cors and all sorts of other protections are still enabled.
To clarify my current setup, I have:
A react website localhost:3000
A electron app showing the react website.
There is no other communication happening between the two.
All requests are made solely by the react website.
The react website needs to communicate with foreignwebsite.com and send a cookie to it.
Normally this in any server-side type of application, including electron itself, would be of no problem, but react cannot make direct requests.
One way I could do this was to potentially create a middle server that communicates with the foreigner API and friendly gives me the data I need for my front-end app. But, I don't want to host a server, and if I create a server on the client machine I think the app would be too heavy.
I could use the electron process as a server and make node.js requests with it, but to communicate between react and electron seems weirdly complicated, it feels like it would be a mess of events.
I found an answer online that tries to kind of deactivate the cors of the electron browser, it partially did work, or at least it seems that it worked, it shows no error but the cookie simply isn't transferred, maybe it just tricks the browser but it doesn't work behind the scenes, idk...
const filter = {
urls: ['http://foreignwebsite.com/*']
}
mainWindow.webContents.session.webRequest.onBeforeSendHeaders(
filter,
(details, callback) => {
details.requestHeaders.Origin = `http://foreignwebsite.com/`
details.requestHeaders.Referer = `http://foreignwebsite.com/`
callback({ requestHeaders: details.requestHeaders })
}
)
mainWindow.webContents.session.webRequest.onHeadersReceived(
filter,
(details, callback) => {
details.responseHeaders['access-control-allow-origin'] = ['http://localhost:3000']
details.responseHeaders['access-control-allow-credentials'] = 'true'
callback({ responseHeaders: details.responseHeaders })
}
)
So above it sets the AccessControlAllowOrigin and the AllowCredentials of the response accordingly and it changes the origin and referer of the requests.
And when I go make requests on the react website I do it like this:
document.cookie = 'OmegaCookie=' + cookieVal + '; path=/; SameSite=None; Secure';
let response = await fetch('http://foreignwebsite.com', {credentials: 'include'});
But the cookie is not sent. Do you know what could be going wrong or how to handle it better in this situation?
Ive been trying to implement the WebSocket protocol from scratch in nodejs, doing so i have a question thats since been unawnsered. What exactly are subprotocols in regards to websockets? The second parameter of the WebSocket constructor is where you specify "subprotocols" -
let socket = new WebSocket("ws://localhost:3000",["http",...]);
Can anybody give me a clear awnser to what purpose they have?
Websockets just define a mechanism to exchange arbitrary messages. What those messages mean, what kind of messages a client can expect at any particular point in time or what messages they are allowed to send is entirely up to the implementing application. So you need an agreement between the server and client about these things. You might say… you need a protocol specification. The subprotocol parameter simply lets clients formally exchange this information. You can just make up any name for any protocol you want. The server can simply check that the client appears to adhere to that protocol during the handshake. You can also use it to request different kinds of protocols from the server, or use it for versioning (e.g. when you introduce my-protocol-v2, but still need to support clients only understanding my-protocol-v1).
Explained on MDN here
Think of a subprotocol as a custom XML schema or doctype declaration.
You're still using XML and its syntax, but you're additionally
restricted by a structure you agreed on. WebSocket subprotocols are
just like that. They do not introduce anything fancy, they just
establish structure. Like a doctype or schema, both parties must agree
on the subprotocol; unlike a doctype or schema, the subprotocol is
implemented on the server and cannot be externally referred to by the
client.
Subprotocols are explained in sections 1.9, 4.2, 11.3.4, and 11.5 of the spec.
A client has to ask for a specific subprotocol. To do so, it will send
something like this as part of the original handshake:
http GET /chat HTTP/1.1 ... Sec-WebSocket-Protocol: soap, wamp
or, equivalently:
... Sec-WebSocket-Protocol: soap Sec-WebSocket-Protocol: wamp
Now the server must pick one of the protocols that the client
suggested and it supports. If there is more than one, send the first
one the client sent. Imagine our server can use both soap and wamp.
Then, in the response handshake, it sends:
Sec-WebSocket-Protocol: soap
The server can't send more than one Sec-Websocket-Protocol header. If
the server doesn't want to use any subprotocol, it shouldn't send any
Sec-WebSocket-Protocol header. Sending a blank header is incorrect.
The client may close the connection if it doesn't get the subprotocol
it wants.
If you want your server to obey certain subprotocols, then naturally
you'll need extra code on the server. Let's imagine we're using a
subprotocol json. In this subprotocol, all data is passed as JSON. If
the client solicits this protocol and the server wants to use it, the
server needs to have a JSON parser. Practically speaking, this will be
part of a library, but the server needs to pass the data around.
Some sample code, copy from https://hpbn.co/websocket/#subprotocol-negotiation, to make it clear.
The client can advertise which protocols it supports to the server as
part of its initial connection handshake:
var ws = new WebSocket('wss://example.com/socket',
['appProtocol', 'appProtocol-v2']);
ws.onopen = function () {
if (ws.protocol == 'appProtocol-v2') {
...
} else {
...
}
}
The npm-request library allows me to construct HTTP requests using a nice JSON-style syntax, like this.
request.post(
{
url: 'https://my.own.service/api/route',
formData: {
firstName: 'John',
lastName: 'Smith'
}
},
(err, response, body) => {
console.log(body)
}
);
But for troubleshooting, I really need to see the HTTP message body of the request as it would appear on the wire. Ideally I'm looking for a raw bytes representation with a Node.js Buffer object. It seems easy to get this for the response, but not the request. I'm particularly interested in multipart/form-data.
I've looked through the documentation and GitHub issues and can't figure it out.
Simplest way to do this is to start a netcat server on any port:
$ nc -l -p 8080
and change the URL to localhost in your code:
https://localhost:8080/v1beta1/text:synthesize?key=API_KEY
Now, any requests made will print the entire, raw HTTP message sent to the localhost server.
Obviously, you won't be able to see the response, but the entire raw request data will be available for you to inspect in the terminal you have netcat running
I figured out how to dump the HTTP message body with Request. In both cases, I'm just copying the same approach that request uses internally.
Multipart Form Uploads
req._form.pipe(process.stdout);
URL-encoded Forms
console.log(req.body);
You could try #jfriend00 suggestion an use a network sniffer like wireshark but as you're fetching an https URL this might not be the easiest route as it requires some setup to intercept TLS connections.
So maybe it would be enough turning on debug mode for the request module itself, you can do that by simply setting require('request').debug = true. As a third option you could go with the dedicated debug module for request here which allows you to view request and response headers and bodies.
I can think of a number of ways to see the bytes of the request:
Turn on debugging in the request module. There are multiple ways to do that documented here including setting NODE_DEBUG=request or require('request').debug = true or using the request-debug module.
Use a network sniffer to see what's actually being sent over the socket, independent of your node.js code.
Create your own dummy http server that does nothing but log the exact incoming request and send your same request to that dummy server so it can log it for you.
Create or use a proxy (like nginx) that can dump the exact incoming request before forwarding it on to its final destination and send the request to the proxy.
Step through the sending of the request in the debugger to see exactly what it is writing to the socket (this may be time consuming, particularly with async callbacks, but will eventually work).
you could use a nodejs server capable of logging the raw request/response string , then direct your request to that server
i gave an example using both http and https server - no dependencies
nodejs getting raw https request and response
We have been encountering inconsistent client errors with a single-page JavaScript application making fetch requests. Of note, they are all same-origin requests.
let request = new Request(url, options);
...
window.fetch(request)
.then(response => response.json())
.then(data => ...)
.catch(error => ...)
Around 5% of the promises are rejecting with the following error despite the server and the browser receiving a 200 OK response:
TypeError: Failed to fetch
I'm stumped... All of my searches lead to discussions about CORS errors. That doesn't seem to apply given these are all same-origin requests. What is causing the fetch to throw the TypeError?
I can confirm using the Network tab in Chrome DevTools that the fetch request completes with a 200 OK response and valid JSON. I can also confirm that the URLs are same-origin. I can also confirm that there are no CORS pre-flight requests. I have reproduced this issue on Chrome 66 and Safari 11.1. However, we've received a stream of error reports from a mix of Chrome and Safari versions, both desktop and mobile.
EDIT:
This does not appear to be a duplicate of the linked question as we are not sending CORS requests, not setting mode: "no-cors", and not setting the Access-Control-Allow-Origin header.
Additionally, I re-ran tests with the mode: 'same-origin' option set explicitly. The requests are (still) successful; however, we (still) receive the intermittent TypeError.
I know that this is an old issue, but after searching the entire evening I want to share my findings so you can spend your time better.
My web app also worked well for most users but from time to time visitors received the error mentioned in the question. I'm not using any complicated infrastructure (reverse proxy etc.) setup nor do I communicate with services on a different domain/protocol/port. I'm just sending a POST request to a PHP-File on the same server where the React app is served from.
The short answer: My problem was that I've sent the request to the backend by using an absolute URL, like https://my-fancy-domain.com/funky_service.php. After changing this to a relative path like /funky-service.php the issue was gone.
My explanation: Most users come to the site without www in the URL, but some users actually do type this part in their address bars (www.my-fancy...). It turned out that the www is part of the origin, so when these users submit the form and send post requests to https://my-fancy... it's technically another origin. This is why the browser expects CORS headers and sometimes even sends an OPTIONS preflight request. When you use a relative path in your JavaScript-Code the post request will also include the www-part (uses the origin from the address bar) -> same-origin -> no CORS hassle. As it only affects visitors that come with the www to your site it also explains the fact that it worked for most users even with the absolute URL.
Also important to know: The request fails in the browser/ JavaScript-Code but is actually sent to the backend (very ugly!).
Let me know if you need more information. Actually, it is very simple but hard to explain (and to find)
The issue could be with the response you are receiving from back-end. If it was working fine on the server then the problem could be with the response headers. Check the Access-Control-Allow-Origin (ACAO) in the response headers. Usually react's fetch API will throw fail to fetch even after receiving response when the response headers' ACAO and the origin of request won't match.
Ref: Getting "TypeError: failed to fetch" when the request hasn't actually failed
I've started to write a HTML file which displays data with JavaScript. Since it shall be done as easy as possible I don't want to run nodejs oder any other local http server. I've just opened the HTML file in a browser (url is file:///home/visu/index.htm).
Everything is fine, till a jquery ajax request to a online API is done in the index.htm. The browser blocks the request with the message:
Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://x.x.x.x. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing)."
How can I get rid of the problem without starting a local http server?
A possible solution is to start the browser with some "no security flags" or disable CORS with plugins, but this I've to do manually all the time so I don't like it.
When your browser will perform an AJAX request to a different server than the one hosting the current page, it first sends an OPTIONS HTTP message. In that message it sends the following header:
origin: http://my-web-server.com
And the backend server will respond with:
access-control-allow-origin: http://my-web-server.com
But, when you don't have a webserver, there is no address for your browser to put in that origin header. That's why your browser disallows you to do any AJAX request from a local file (maybe you can disable the browser's CORS security as someone mentioned in the comments, but that can put you at risk of malicious sites).
Another option
You can tell your browser to allow to connect from localhost to a backend if you change your backend to return the following header:
access-control-allow-origin: https://localhost:8888
And, you also need to tell your localhost server to serve your page in HTTPS instead of HTTP. Once both conditions are met, CORS validations won't fail.
Notice that to enable HTTPS you'll need to have a SSL cert and key, you can generate them with the following command:
openssl req -x509 -out localhost.crt -keyout localhost.key \
-newkey rsa:2048 -nodes -sha256 \
-subj '/CN=localhost' -extensions EXT -config <( \
printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")
The source of that command and more information are found in this page from Let's Encrypt.
On Firefox, you can install this addon: https://addons.mozilla.org/en-US/firefox/addon/cors-everywhere/ to disable CORS for the respective tab. Then, any request will also work on file:/// URIs. Be careful though!
Either mock Ajax calls, or start web server with reverse proxy and HTTP rewriting configured, since I'm sure you don't want, or have not access to configure API server CORS headers.
If you don't want to mock ajax calls, then use either:
node-http-proxy
nginx - if you don't have nodejs and you don't want to install it.
Not Possible By Design
CORS are always blocked when attempted from a file on disk (web pages using the file:// protocol). There is nothing you can do to make it work from a file. It is simply impossible.
The reasoning for this is that files on disk have no real "origin" to allow the backend server to determine the validity of the request. You can have a file for an issue tracking html on the same disk as a file for a blog html. The server cannot know which html requested the data (you can even have someone else's file shared via Dropbox with embedded javascript that may attempt to access your server's data when you open it - nobody expects a hacking attempt when they simply open a plain html file!!).
This is why no browser vendor will allow you do make CORS requests from a file.
You Need a Server
To make it work you will need a HTTP server. There are lots of options for this from installing Apache/Nginx on your machine to running dev servers like webpack-dev-server or local-web-server. As long as the protocol is http:// or https:// you are allowed to make CORS requests.
Once you have a server serving your html file you can configure CORS on your backend as usual.
If you can not set it up access-control-allow-origin, you can try this.
Use "callback" function if your data is not in same domain.
And wrap your data "jsonCallback(" ... data ... ") as my example: http://www.ceducation.cz/akce-plnytext.json?short=1&callback=?
function jsonCallback(json) {
$.each(json, function(key, val) {
// your data is here
console.log('date: ' + val.date);
console.log('address: ' + val.address);
console.log('---');
});
}
$(document).ready(function() {
$.getJSON("http://www.ceducation.cz/akce-plnytext.json?short=1&callback=?", function(data) {
});
});
Working example