I have a node server that will serve more than 10,000 users at the same time with big data, I already tried lz-string https://www.npmjs.org/package/lz-string but it's not good module because it's blocking the node thread.
Please answer these questions:
is it better to compress the data in server and then decompress in client instead of send plain/json data?
what is the best and fastest way to compress/decompress the data?
If you are sending large chunks of text data over the internet using HTTP protocol, then there are already some technologies in place to help you.
One is called HTTP Compression. HTTP protocol specifications allow few compression algorithms to perform on data being sent, but that requires the server and client to be properly configured for compression. Standard Node.js server will not compress the data without modifying code.
For bare Node.js without any frameworks and 3rd party modules there is zlib module made specially for HTTP compression, both server and client.
Using Express? Then there is a compression middleware.
It might also be worth looking using nginx as a proxy server to your node.js applications. Then you can easily flip the switch ON for compression in nginx, without needing to do anything in your Node.js application at all:
server {
gzip on;
...
}
It really depends on the stack you are using, but the idea is same: compress the HTTP stream itself, as it is supported by the protocol.
Related
How a streaming (e.g. a sound or video) HTTP server should report failure in the DB from which it streams (when it already streams, 503 header cannot be generated).
The client should be able to differentiate if it received complete file or its beginning only.
I am especially interested how to return such errors in Rust frameworks.
The HTTP server library you are using should be taking care of this for you.
If you are writing your own server for some reason, you should use the chunked encoding method, which has a clear way to specify that the stream is over (a zero-lengthed chunk).
The situation:
I have a Node.JS server. The NGINX is set to limit the size of request to be 5MB.
I need to upload large files (~15MB) from the client running in a browser to the server.
There are 3 instances of the server running WITHOUT shared memory/file systems.
What I have done:
I used some libraries to break down the files into chunks (< 5MB), sending them to the server. After successfully sending the last chunk to the server, the client called to a server endpoint to signal the completion and then the merging of chunks happened. This worked when I had one instance of the server running. Due to the load balancing, etc., each request of sending a chunk might be handled by a different instance of the server. Therefore, the chunks may not be merged correctly.
Solutions I have thought of:
The ultimate solution (in my opinion) would be how to stream the chunks to the server in just one request, which is handled by just one server instance.
The Stream API is still experimental. I would like to try but have not found a good example to follow. I heard that the Stream API on client side was different than the stream on Node.JS and some more things needed to be done.
I did some research on Transfer-Encoding: chunked of the HTTP header. Someone said it was good to send large files but I haven't found a good working example how to achieve this.
I also thought of WebSocket (or even Socket.io) to establish a connection with a server instance and send the chunks over. However, some reading told me that WebSocket was not a good way to send large files.
The question:
How can I achieve the goal of sending large files to one server instance in the most efficient way?
To use socket.io on the client side, usually we start a node.js server and go like this:
<script src="/socket.io/socket.io.js"></script>
or with specific port:
<script src="http://localhost:3700/socket.io/socket.io.js"></script>
Question is:
is it necessary to use node.js server to serve socket.io.js ?
...or is it possible to
make a local copy of socket.io.js instead of goes to server every single time we need socket.io?
like, we go to view source and copy everything we got from the source of script tag,
paste and save it as socket.io-local.js so that next time we use:
<script src="socket.io-local.js"></script>
will that work ?
Updates
Thanks for everyone's great response,
I'm asking this because in the case I'm involved, I don't actually have access to the server:
I am writing the client-side to connect to other developer's Socket Sever which is written in Java.
Therefore I'll have to think a way to work around the fact that I don't have a server there for me.
from what I've been testing,
this way seems to work but I really don't know what's happening behind the scene.
You obviously can host the socket.io client library anywhere and pull it in to a page. However, it will almost certainly not work with your Java-based server.
To understand why, you need to understand what socket.io is really doing behind the scenes; the client library is only a small part of it.
Socket.io actually defines and implements its own protocol for realtime communication between a browser and a server. It does so in a way that supports multiple transports: if—for example—a user's browser or proxy doesn't support WebSockets, it can fall back to long polling.
What the socket.io client actually does is:
Makes a XHR GET request for /socket.io/1. The server responds with a session ID, configured timeouts, and supported transports.
The client chooses the best transport that the user browser supports. In modern browsers, it will use WebSockets.
If WebSockets are supported, it creates a new WebSocket to initiate a WebSocket connection (HTTP GET with Upgrade: websocket header) to a special URL – /socket.io/1/websocket/<session id>.
If WebSockets aren't supported by the browser or fail to connect (there are lots of intermediaries in the wild like proxies, filters, network security devices, and so forth that don't support WebSocket requests), the library falls back to XHR long polling, and makes a XHR request to /socket.io/1/xhr-polling/<sesion id>. The server does not respond to the request until a new message is available or a timeout is reached, at which point the client repeats the XHR request.
Socket.io's server component handles the other end of that mess. It handles all the URLs under /socket.io/, setting up sessions, parsing WebSocket upgrades, actually sending messages, and a bunch of other bookkeeping.
Without all of the services provided by the socket.io server, the client library is pretty useless. It will just make a XHR request to a URL that doesn't exist on your server.
My guess is that your Java-based server just implements the WebSockets protocol. You can connect directly to it using the browser-provided WebSocket APIs.
It is possible that your server does implement the socket.io protocol – there are a few abandoned Java projects to do that – but that's unlikely. Talk with the developer of your server to find out exactly how he's implemented a "socket server."
A standalone build of socket.io-client is exposed automatically by the socket.io server as /socket.io/socket.io.js. Alternatively you can serve the file socket.io-client.js found at the root of this repository.
https://github.com/LearnBoost/socket.io-client
I have a module called shotgun-client that actually wraps socket.io. I needed to serve a custom client script as well as the socket.io client script, but I didn't want every user of my module to have to include multiple script references on their pages.
I found that, when installed, you can serve the generated client script from socket.io by reading the file /node_modules/socket.io/node_modules/socket.io-client/dist/socket.io.js. So my module adds a listener for its own URL and when it serves my custom client script it also serves the socket.io client script with it. Viola! Only a single script reference for the users of my module :)
While this is technically possible, I don't see why you'd need to do that. If you're concerned about reducing the data that goes over the wire, this change won't actually do much beyond the few characters saved in the shorter src tag. Simply changing the location of the JS file on the server won't actually improve performance - the JS has to be sent.
Proper caching (which Socket.IO has) will return a 304 Not Modified (and not re-send the JS file every time you load a page).
I'm working on a small personal project (to learn node) and I'm wondering what the best way to send chunks of a video data from a client to the server is? Obviously, I don't want to use Node's http module, but I've never used anything other than http. I know there is a net module -- is this better?
An API returns large resultsets and I was wishing that I could gzip it in PHP but I wouldn't know how to gunzip it in Javascript. Is there some sort of library that is capable of this in Javascript? I was searching the net and found some stuff but couldn't quite figure out how to make use of it. So, if anybody has ever dealt with this before, advise is highly appreciated.
(The API response is worth about 1MB)
Usually compression of HTTP responses is done by either a proxy or a web server. You should be able to configure Apache to do this for you. See the documentation for mod_deflate for more information.
In terms of unzipping in JavaScript, this is a non-issue. Provided that the HTTP response contains the correct header information. (Content-Encoding: gzip) the browser will handle the unzipping for you.
Have the web server do it. Set the HTTP header to accept gzip when you send your request.
The web server and browser can usually handle the compression transparently, without needing php and javascript support. For example, a google search turned up: http://forums.digitalpoint.com/showthread.php?t=43