I'm messing around with the Darksky API and under one of the query parameters it states:
extend=hourly optional
When present, return hour-by-hour data for the next 168 hours, instead
of the next 48. When using this option, we strongly recommend enabling
HTTP compression.
I'm using Express as a node proxy which hits the Darksky api (i.e. localhost:3000/api/forecast/LATITUDE, LONGITUDE).
What does "HTTP compression" mean and how would I go about enabling it?
Here compression means the gzip compression on the express server. You can use the compression middleware to add easy gzip compression to your server.
Read more about how you can install that middleware on here.
https://github.com/expressjs/compression
An example implementation should be look like this.
var compression = require('compression')
var express = require('express')
var app = express()
// compress all responses
app.use(compression())
// add all routes
To quote from https://darksky.net/dev/docs
The Forecast Data API supports HTTP compression. We heartily recommend using it, as it will make responses much smaller over the wire. To enable it, simply add an Accept-Encoding: gzip header to your request. (Most HTTP client libraries wrap this functionality for you, please consult your library’s documentation for details.)
I'm not familiar with the Dark Sky API but I would imagine it returns a large amount of highly redundant data, which is ideal for compression. HTTP requests have a compression mechanism built in via Accept-Encoding, as mentioned above.
In your case that data will be travelling across the wire twice, once from Dark Sky to your server and then again from your server to your end user. You could compress just one of these two transmissions or both, it's up to you but it's likely you'd want both unless the end user is on the same local network as your server.
There are various SO questions about making compressed requests, such as:
node.js - easy http requests with gzip/deflate compression
The key decision for you is whether you want to decompress and recompress the data in your proxy or just stream it through. If you don't need a decompressed copy of the data in the server then it would be more efficient to skip the extra steps. You'd need to be careful to ensure all the headers are set correctly but if you just pass on the relevant headers that you receive (in both directions) it should be relatively simple to pipe through the response from Dark Sky.
Related
As far as i understand all what REST do is standartize a data sended to server by adding some headers. For example REST request can generate a line of bytes like so: POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive and finish it with some user input.
Now im just playing with writing my own JS server and here is my question: is there a way in JS to send some data(bytes) without this REST addings like headers/method and will it work for browsers and HTTP protocol itself?
For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
Iv tried to google and end up that XMLhttpRequest and fetch both require some CRUD method to be specified and adding some headers in request anyway.
HTTP requests:
Need to specify the method
Need to specify the Host as a header (in HTTP 1.1. and newer)
Will include some other request headers automatically when make using JS from a browser
This has nothing to do with REST. It's just how HTTP works.
A non-HTTP protocol could avoid having that. JavaScript in a browser has no mechanisms that allow making non HTTP requests.
You might want to research WebSocket which allows two way communication over a single connection … but that is a bootstrapped by HTTP so doesn't really fulfil your requirement.
For example instead of sending POST /qwe HTTP/1.1 Host: 127.0.0.1 Connection: keep-alive MY DATA OVER THERE!!! just send MY DATA OVER THERE!!! so my server can read only user data without everything else.
I suspect you're misunderstanding what a request is, on a fundamental level. Without POST (the method), /qwe (the path), HTTP/1.1 (the protocol) and 127.0.0.1 (the address) there is no way for your computer to know where and how to send the data. These are necessary if you want to communicate with a server, and removing them will mean your code no longer works.
You're working with very low-level data here, which is probably not what you actually want to be doing. There are some packages which will let you ignore the how and what of the request, and focus on just the data inside it. Express might be a good place to start. You can set up a simple express server to handle requests on specific paths, and reply with data that your frontend can then use.
A REST API is a high-level concept and largely unrelated to what you're asking about.
Ive been trying to implement the WebSocket protocol from scratch in nodejs, doing so i have a question thats since been unawnsered. What exactly are subprotocols in regards to websockets? The second parameter of the WebSocket constructor is where you specify "subprotocols" -
let socket = new WebSocket("ws://localhost:3000",["http",...]);
Can anybody give me a clear awnser to what purpose they have?
Websockets just define a mechanism to exchange arbitrary messages. What those messages mean, what kind of messages a client can expect at any particular point in time or what messages they are allowed to send is entirely up to the implementing application. So you need an agreement between the server and client about these things. You might say… you need a protocol specification. The subprotocol parameter simply lets clients formally exchange this information. You can just make up any name for any protocol you want. The server can simply check that the client appears to adhere to that protocol during the handshake. You can also use it to request different kinds of protocols from the server, or use it for versioning (e.g. when you introduce my-protocol-v2, but still need to support clients only understanding my-protocol-v1).
Explained on MDN here
Think of a subprotocol as a custom XML schema or doctype declaration.
You're still using XML and its syntax, but you're additionally
restricted by a structure you agreed on. WebSocket subprotocols are
just like that. They do not introduce anything fancy, they just
establish structure. Like a doctype or schema, both parties must agree
on the subprotocol; unlike a doctype or schema, the subprotocol is
implemented on the server and cannot be externally referred to by the
client.
Subprotocols are explained in sections 1.9, 4.2, 11.3.4, and 11.5 of the spec.
A client has to ask for a specific subprotocol. To do so, it will send
something like this as part of the original handshake:
http GET /chat HTTP/1.1 ... Sec-WebSocket-Protocol: soap, wamp
or, equivalently:
... Sec-WebSocket-Protocol: soap Sec-WebSocket-Protocol: wamp
Now the server must pick one of the protocols that the client
suggested and it supports. If there is more than one, send the first
one the client sent. Imagine our server can use both soap and wamp.
Then, in the response handshake, it sends:
Sec-WebSocket-Protocol: soap
The server can't send more than one Sec-Websocket-Protocol header. If
the server doesn't want to use any subprotocol, it shouldn't send any
Sec-WebSocket-Protocol header. Sending a blank header is incorrect.
The client may close the connection if it doesn't get the subprotocol
it wants.
If you want your server to obey certain subprotocols, then naturally
you'll need extra code on the server. Let's imagine we're using a
subprotocol json. In this subprotocol, all data is passed as JSON. If
the client solicits this protocol and the server wants to use it, the
server needs to have a JSON parser. Practically speaking, this will be
part of a library, but the server needs to pass the data around.
Some sample code, copy from https://hpbn.co/websocket/#subprotocol-negotiation, to make it clear.
The client can advertise which protocols it supports to the server as
part of its initial connection handshake:
var ws = new WebSocket('wss://example.com/socket',
['appProtocol', 'appProtocol-v2']);
ws.onopen = function () {
if (ws.protocol == 'appProtocol-v2') {
...
} else {
...
}
}
I have a node.js express proxy server, which I would like to debug, So I want to capture the traffic through fiddler:
const express = require('express');
const expressApp = express();
const proxyTarget = 'https://my-proxy.azurewebsites.net/proxy';
//attempting to configure fiddler proxy
process.env.https_proxy= 'http://127.0.0.1:8888'
process.env.http_proxy= 'http://127.0.0.1:8888'
process.env.NODE_TLS_REJECT_UNAUTHORIZED=0;
expressApp.use('/api', proxy({
target: proxyTarget,
secure: false, // don't verify https certificates
logLevel: 'debug'
}));
I've tried to capture traffic using fiddler but fiddler doesn't capture the request to Azure, only the wrapped proxy request on localhost.
How can I configure my proxy settings so that first I pass through fiddler?
Unfortunately, Node.js doesn't listen to the http_proxy/https_proxy environment variables (see https://github.com/nodejs/node/issues/8381 for the full debate).
That means this is difficult with Fiddler - you need to change every place where HTTP requests are made to use the proxy settings, or to replace Node's default agent to do so (and then ensure no requests set a custom agent). There are some node modules that can help here, like https://www.npmjs.com/package/global-agent, but you'll also need to handle HTTPS trust separately (or you'll just see unreadable CONNECT requests, as mentioned by the commenter above).
I have hit this same issue myself though, so I've been building an open-source alternative to Fiddler to fix it! HTTP Toolkit can do what you're looking for: you can open a terminal from there (or enable it in an existing terminal) and any node processes started there will automatically have all HTTP & HTTPS requests intercepted, so you can examine and/or rewrite them, just like Fiddler. It handles all the cert trust stuff too.
Under the hood, this is just wrapping node on the command line to do the same thing you'd do manually to do that, reconfiguring settings & defaults to ensure everything plays nicely. If you do want to do that manually, the full source to do so is here: https://github.com/httptoolkit/httptoolkit-server/blob/master/overrides/js/prepend-node.js
I am questioning whether it is required to validate fields like req.ip or req.path server-side.
It boils down to the question: Is it possible for the client to set something like .set('Remote-Addr', <Malicious JavaScript>) and it successfully being propagated to my Node or Express middleware?
Thanks for helping!
There is no way to validate source IP, particular when proxies are involved. In the proxy case, a chain of IP addresses is supposed to be put in http headers, but that can certainly be faked so what express thinks is the original IP cannot be trusted. It is likely accurate, but not guaranteed accurate.
req.path is entirely local and does not involve any client headers and is not subject to any client spoofing. It just comes from the actual HTTP request URL that arrives at your server. The only way it wouldn't be the same as the actual request URL is if you were using a mount point for routers in which case the mount point part of the path will have been removed by express. Or perhaps if your own middleware attempted to mess with it.
I have a file that creates an express server. Let's call it foo_serever.coffee
# foo_server.coffee
module.exports = express.createServer()
And now I want to require that and send it requests directly in javascript. (Note, sendRequest is not a real method I just want one like it)
app = require './foo_server'
app.sendRequest 'get', '/some/url.html', (status, body, headers) ->
console.log 'yay' if status is 200
Is there a way to do this? Or do I need to spawn it as another process and send it requests over http which seems like much more of a pain.
Short answer: No. Express only works as a real HTTP server, not a simulated one. Use a library like request for testing. You don't have to spawn a separate process to send HTTP requests.
As Trevor said you can use Mikeal's request library for testing or Learnboost's Tobi if you want to do 'Expressive server-side functional testing with jQuery'.
I know this isn't as elegant as doing it through the express frame work but I know that the Nodejs http module allows for making http requests to any site, and allows you to supply the headers and host ect.
See the node http documentation for details and examples.
http://nodejs.org/docs/v0.6.5/api/http.html#http.get