I know how to handle chunked downloads in javascript, using the XmlHttpRequest object. Is there any way to perform a chunked upload using javascript, opening a connection but only uploading blobs of data bit by bit?
I know chunked uploads should be possible with Http 1.1 servers, and have found a lot of references to making chunked uploads using various other platforms (C# java etc.) but have not found any references to doing so in the browser with javascript.
EDIT: The use case is to stream data up to the server, and not to upload a large file, kind of mirroring the use of a chunked response to stream data down to the client. This is as an alternative to making individual ajax requests, since the chunks of data that's going up from client to server are pretty frequent (< 0.5s interval).
As of today (November 2021), I believe support for UPLOADS using HTTP chunked data transfer is still largely missing in browsers.
If you look at the "Send ReadableStream in request body" column of the browser support matrix for the Request (Fetch API), you can see that it is currently "No" for all browsers except "Deno". You will however notice the "Experimental" flag beside the column. So it is available experimentally in some browsers such as Chrome. I wouldn't hold your breath about it becoming mainstream anytime soon though.
HTTP chunked data transfer encoding is not technically necessary for sending data a few chunks at a time, I believe regular HTTP data transfer also only sends data a few chunks at a time but the "chunking" is done at the TCP level instead (please correct me if I'm wrong here). Consequently, both protocols can be used to stream a file upload. WebSockets are of course another option as well. The main difference in which protocol to choose is based on whether or not you know the final length of the stream in advance or not.
If you need to stream upload data for which you DON'T know the length in advance (such as live video, video conference calls, remote desktop sessions, chats, etc.) then your best bet is perhaps the WebSocket API (or something built on top of it).
If you need to stream upload data for which you DO know the length in advance (files, images, videos, etc.) then I believe your best bet is probably a normal POST or PUT using the Fetch API or even the old XmlHttpRequest API.
You can use the FileReader API and the slice method.
with slice you can get block of data that you can upload, then you need to reassemble them server side.
here is a good intro on how to handle files in javascript
http://www.html5rocks.com/en/tutorials/file/dndfiles/
you can have a look at http://caniuse.com/#feat=filereader for browser support
Related
How a streaming (e.g. a sound or video) HTTP server should report failure in the DB from which it streams (when it already streams, 503 header cannot be generated).
The client should be able to differentiate if it received complete file or its beginning only.
I am especially interested how to return such errors in Rust frameworks.
The HTTP server library you are using should be taking care of this for you.
If you are writing your own server for some reason, you should use the chunked encoding method, which has a clear way to specify that the stream is over (a zero-lengthed chunk).
We have an REST API endpoint that will stream multiple GB of data as a response. Currently using xhr and responseType: blob, then In our web interface, we would like to stream that response do a file instead of storing the entire response in memory, and the trying to save it to a file. {oked around the fetch API. Still can't quite figure out how to get something like that to work. What are we missing?
The canonical HTTP way of doing this is to use a HTTP chunked response. That means that instead of using XHR one points the browser at the file URI and the server should respond with a response that indicates it supports chunking.
You're essentially trying to reinvent the wheel (of downloading files) with XHR and JavaScript.
Point the browser at the proper URI and it will be streamed directly to client's disk. Supports resuming broken downloads out-of-the-box!
The situation:
I have a Node.JS server. The NGINX is set to limit the size of request to be 5MB.
I need to upload large files (~15MB) from the client running in a browser to the server.
There are 3 instances of the server running WITHOUT shared memory/file systems.
What I have done:
I used some libraries to break down the files into chunks (< 5MB), sending them to the server. After successfully sending the last chunk to the server, the client called to a server endpoint to signal the completion and then the merging of chunks happened. This worked when I had one instance of the server running. Due to the load balancing, etc., each request of sending a chunk might be handled by a different instance of the server. Therefore, the chunks may not be merged correctly.
Solutions I have thought of:
The ultimate solution (in my opinion) would be how to stream the chunks to the server in just one request, which is handled by just one server instance.
The Stream API is still experimental. I would like to try but have not found a good example to follow. I heard that the Stream API on client side was different than the stream on Node.JS and some more things needed to be done.
I did some research on Transfer-Encoding: chunked of the HTTP header. Someone said it was good to send large files but I haven't found a good working example how to achieve this.
I also thought of WebSocket (or even Socket.io) to establish a connection with a server instance and send the chunks over. However, some reading told me that WebSocket was not a good way to send large files.
The question:
How can I achieve the goal of sending large files to one server instance in the most efficient way?
My App involves heavy client side image manipulation. Since the files are modified so frequently during a users session we do not physically upload them to the server until the user is finished and chooses to save the image(s).
The images are read from the client using the html5 file API and stored in memory as base64 strings where the user proceeds to perform his/her manipulations quickly and efficiently.
When the user chooses to save we transfer the string or strings to the server via a standard ajax POST and then build the image into a physical file server-side.
What I would like to do is provide a progress bar for this final upload stage.
Now, I am well aware that the html5 spec includes support for file upload progress reporting. However we are not using the standard upload methods, instead, as mentioned above the images are sent as strings.
So my question really boils down to the following:
A) Is there anyway to actual measure the actual bytes sent during a simple post (PHP serverside). If so this would work as I already have the image size in bytes from the HTML5 filereader api.
B) If not is there anyway I can convert the base64 data into actual files on the fly and send them via the standard html5 upload method, reporting the progress as we go.
In either case... If so, how?
Many thanks in advance.
It's easier when you send the Base64 image via ajax. Ajax works with a callback function, so just use an indicator of some sort to let the user know that ajax has been initiated (i.e. the file download has started ). On completion the callback function will run, now you can turn off the indicator whatever it might be.
As far as measuring actual bytes, via ajax I'm not sure.
This method
showing progressbar progress with ajax request
suggest estimating it. I would take it a step further and maybe time how long it take the callback over say 10 tries( use this average to estimate your progress bar and also use the 90% trick mentioned above ).
Or if you want to encode/decode your Base64 text.
How can you encode a string to Base64 in JavaScript?
Depending on the server technology you are using, there are several options.
Use Comet or Javascript long polling to get the progress of the upload from the server, and update the progress bar from the client side
If you are using .Net technologies in the backend, explore the SignalR library, which will provide all the plumbing to provide real time communication between the server and the client.
I'm prototyping a realtime notification mechanism using http over port 80. The aim of the project is to allow a Flash application to respond to an event on a remote server quickly (specifically an inbound phone call being connected to a phone next to the computer.) Polling is one approach, but is too slow. Currently I use a socket connection to get low latency notification of the events on the server, which works well but isn't firewall friendly. I don't want to install anything except Flash, or Silverlight on the client. Cross compatibility of browsers isn't a concern - in this application I can specify what browser the client uses but IE is preferred.
I've made a server HttpHandler in .NET which never closes the connection and sends the "events" to the client by writing out bytes to the http response stream (ConnectedClientContext.Response.OutputStream.Write etc) and I have a .NET client application which can read these messages okay.
My Question:
Can I receive the bytes from the server over HTTP as they arrive using JavaScript, Flash or Silverlight? So far I can only find a way to get notified of the "download progress" and don't get the actual bytes until the response is closed - I want them as they arrive.
Best Regards,
Daniel
I don't know about Flash but in Javascript (by which you mean in browser) and Silverlight you are limited pretty much to the http protocol.
You can use the AJAX Http Streaming pattern. The basic ideas which is different from what you are trying is that as soon as data is available outstanding request ends and a new is immediately initiated asychronously, mean while your client process the newly arrived data.
Silverlight gives you more options since is HTTP stack is purely asynchronous but you can get your hands on the stream to you as soon as data starts to arive by setting the HttpWebRequest.AllowReadStreamBuffering to false. (Unlike XmlHttpRequest which always buffers).
it's very easy to use the Comet ideas for notifications. you don't even have to use a comet-specific framework. simply do an ajax request with a callback on answer, wrap this on a loop and you have an event loop, just like a GUI app. on the server side, don't bother answering the request until there's either an event, or a timeout (which is just a 'null' event).
Flex and Flash have several AMF/XML remoting libraries available that support data pushing. I would certainly look into that.
http://raghuonflex.wordpress.com/2008/04/17/data-push-in-flex-with-backend/
These libraries use a Comet - like implementation to achieve this.