We Keep Coding

What is the Correct Way to Download Blob in Chrome? [duplicate]

Update
Since asking the question below and arriving at a more fundamental question after finding the error in the code, I found some more information such as in the MDN web docs for the downloads API method downloads.download() it states that a revoke of an object url should be performed only after the file/url has been downloaded. So, I spent some time trying to understand whether or not a web extension makes the downloads API onChanged event 'available' to javascript of a web page and don't think it does. I don't understand why the downloads API is available to extensions only, especailly when there are quite a few questions concerning this same memory-usage/object-url-revocation issue. For example, Wait for user to finish downloading a blob in Javascript.
If you know, would you please explain? Thank you.
Starting with Firefox browser closed, and right clicking on a local html file to open in Firefox, it opens with five firefox.exe processes as viewed in Windows Task Manager. Four of the processes start with between 20,000k and 25,000k of memory and one with about 115,000k.
This html page has an indexedDB database with 50 object stores each containing 50 objects. Each object is extracted from its object store and converted to string using JSON.stringify, and written to a two-dimensional array. Afterward, all elements of the array are concatenated into one large string, converted to a blob and written to the hard disk through a URL object which is revoked immediately afterward. The final file is about 190MB.
If the code is stopped just before the conversion to blob, one of the firefox.exe process's memory usage increases to around 425,000k and then falls back to 25,000k in about 5-10 seconds after the elements of the array have been concatenated into a single string.
If the code is run to completion, the memory usage of that same firefox.exe process grows to about 1,000,000k and then drops to about 225,000k. The firefox.exe process that started at 115,000k also increases at the blob stage of the code to about 325,000k and never decreases.
After the blob is written to disk as a text file, these two firefox.exe processes never release the approximate 2 x 200,000k increase in memory.
I have set every variable used in each function to null and the memory is never freed unless the page is refreshed. Also, this process is initiated by a button click event; and if it is run again without an intermediate refresh, each of these two firefox.exe processes grab an additional 200,000k of memory with each run.
I haven't been able to figure out how to free the memory?
The two functions are quite simple. json[i][j] holds the string version of the jth object from the ith object store in the database. os_data[] is an array of small objects { "name" : objectStoreName, "count" : n }, where n is the number of objects in the store. The build_text fuction appears to release the memory if write_to_disk is not invoked. So, the issue appears to be related to the blob or the url.
I'm probably overlooking something obvious. Thank you for any direction you can provide.
EDIT:
I see from JavaScript: Create and save file that I have a mistake in the revokeObjectURL(blob) statment. It can't revoke blob, the createObjectURL(blob) needed to be saved to a variable like url and then revoke url, not blob.
That worked for the most part and the memory is released from both of the firefox.exe processes mentioned above, in most cases. This leaves me with one small question about the timing of the revoke of the url.
If the revoke is what allows for the release of memory, should the url be revoked only after the file has been successfully downloaded? If the revoke takes place before the user clicks ok to download the file, what happens? Suppose I click the button to prepare the file from the database and after it's ready the browser brings up the window for downloading, but I wait a little while thinking about what to name the file or where to save it, won't the revoke statment be run already but the url is still 'held' by the browser since it is what will be downloaded? I know I can still download the file, but does the revoke still release the memory? From my small amount of experimenting with this one example, it appears that it does not get released in this scenario.
If there was an event that fires when the file has either successfully or unsuccessfully been downloaded to the client, is not that the time when the url should be revoked? Would it be better to set a timeout of a few minutes before revoking the url, since I'm pretty sure there is not an event indicating download to client has ended.
I'm probably not understanding something basic about this. Thanks.
function build_text() {
var i, j, l, txt = "";
for ( i = 1; i <=50; i++ ) {
l = os_data[i-1].count;
for ( j = 1; j <= l; j++ ) {
txt += json[i][j] + '\n';
}; // next j
}; // next i
write_to_disk('indexedDB portfolio', txt);
txt = json = null;
} // close build_text
function write_to_disk( fileName, data ) {
fileName = fileName.replace(".","");
var blob = new Blob( [data], { type: 'text/csv' } ), elem;
if ( window.navigator.msSaveOrOpenBlob ) {
window.navigator.msSaveBlob(blob, fileName);
} else {
elem = window.document.createElement('a');
elem.href = window.URL.createObjectURL(blob);
elem.download = fileName;
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);
window.URL.revokeObjectURL(blob);
}; // end if
data = blob = elem = fileName = null;
} // close write_to_disk

I am a bit lost as to what is the question here...
But let's try to answer, at least part of it:
For a starter let's explain what URL.createObjectURL(blob) roughly does:
It creates a blob URI, which is an URI pointing to the Blob blob in memory just like if it was in an reachable place (like a server).
This blob URI will mark blob as being un-collectable by the Garbage Collector (GC) for as long as it has not been revoked, so that you don't have to maintain a live reference to blob in your script, but that you can still use/load it.
URL.revokeObjectURL will then break the link between the blob URI and the Blob in memory. It will not free up the memory occupied by blob directly, it will just remove its own protection regarding the GC, [and won't point to anywhere anymore].
So if you have multiple blob URI pointing to the same Blob object, revoking only one won't break the other blob URIs.
Now, the memory will be freed only when the GC will kick in, and this in only decided by the browser internals, when it thinks it is the best time, or when it sees it has no other options (generally when it misses memroy space).
So it is quite normal that you don't see your memory being freed up instantly, and by experience, I would say that FF doesn't care about using a lot of memory, when it is available, making GC kick not so often, whihc is good for user-experience (GCing often results in lags).
For your download question, indeed, web APIs don't provide a way to know if a download has been successful or failed, nor even if it has just ended.
For the revoking part, it really depends on when you do it.
If you do it directly in the click handler, then the browser won't have done the pre-fetch request yet, so when the default action of the click (the download) will happen, there won't be anything linked by the URI anymore.
Now, if you do revoke the blob URI after the "save" prompt, the browser will have done a pre-fetch request, and thus might be able to mark by itself that the Blob resource should not be cleared. But I don't think this behavior is tied by any specs, and it might be better to wait at least for the window's focus event, at which point the downloading of the resource should already have started.
const blob = new Blob(['bar']);
const uri = URL.createObjectURL(blob);
anchor.href = uri;
anchor.onclick = e => {
window.addEventListener('focus', e=>{
URL.revokeObjectURL(uri);
console.log("Blob URI revoked, you won't be able to download it anymore");
}, {once: true});
};
<a id="anchor" download="foo.txt">download</a>

Detecting if the user drops the same file twice on a browser window

I want to allow users to drag images from their desktop onto a browser window and then upload those images to a server. I want to upload each file only once, even if it is dropped on the window several times. For security reasons, the information from File object that is accessible to JavaScript is limited. According to msdn.microsoft.com, only the following properties can be read:
name
lastModifiedDate
(Safari also exposes size and type).
The user can drop two images with the same name and last modified date from different folders onto the browser window. There is a very small but finite chance that these two images are in fact different.
I've created a script that reads in the raw dataURL of each image file, and compares it to files that were previously dropped on the window. One advantage of this is that it can detect identical files with different names.
This works, but it seems overkill. It also requires a huge amount of data to be stored. I could improve this (and add to the overkill) by making a hash of the dataURL, and storing that instead.
I'm hoping that there may be a more elegant way of achieving my goal. What can you suggest?
<!DOCTYPE html>
<html>
<head>
<title>Detect duplicate drops</title>
<style>
html, body {
width: 100%;
height: 100%;
margin: 0;
background: #000;
}
</style>
<script>
var body
var imageData = []
document.addEventListener('DOMContentLoaded', function ready() {
body = document.getElementsByTagName("body")[0]
body.addEventListener("dragover", swallowEvent, false)
body.addEventListener("drop", treatDrop, false)
}, false)
function swallowEvent(event) {
// Prevent browser from loading the dropped image in an empty page
event.preventDefault()
event.stopPropagation()
}
function treatDrop(event) {
swallowEvent(event)
for (var ii=0, file; file = event.dataTransfer.files[ii]; ii++) {
importImage(file)
}
}
function importImage(file) {
var reader = new FileReader()
reader.onload = function fileImported(event) {
var dataURL = event.target.result
var index = imageData.indexOf(dataURL)
var img, message
if (index < 0) {
index = imageData.length
console.log(dataURL)
imageData.push(dataURL, file.name)
message = "Image "+file.name+" imported"
} else {
message = "Image "+file.name+" imported as "+imageData[index+1]
}
img = document.createElement("img")
img.src = imageData[index] // copy or reference?
body.appendChild(img)
console.log(message)
}
reader.readAsDataURL(file)
}
</script>
</head>
<body>
</body>
</html>

Here is a suggestion (that I haven't seen being mentioned in your question):
Create a Blob URL for each file-object in the FileList-object to be stored in the browsers URL Store, saving their URL-String.
Then you pass that URL-string to a webworker (separate thread) which uses the FileReader to read each file (accessed via the Blob URL string) in chunked sections, re-using one fixed-size buffer (almost like a circular buffer), to calculates the file's hash (there are simple/fast carry-able hashes like crc32 which can often be simply combined with a vertical and horizontal checksum in the same loop (also carry-able over chunks)).
You might speed up the process by reading in 32 bit (unsigned) values instead of 8 bit values using an appropriate 'bufferview' (that's 4 times faster). System endianness is not important, don't waste resources on this!
Upon completion the webworker then passes back the file's hash to the main-thread/app which then simply performs your matrix comparison of [[fname, fsize, blobUrl, fhash] /* , etc /*].
Pro
The re-used fixed buffer significantly brings down your memory usage (to any level you specify), the webworker brings up performance by using the extra thread (which doesn't block your main browser's thread).
Con
You'd still need serverside fall-back for browsers with javascript disabled (you might add a hidden field to the form and set it's value using javascript as means of a javascript-enabled check, as to lower server-side load). However.. even then.. you'd still need server-side fallback to safeguard against malicious input.
Usefulness
So.. no net gain? Well.. if the chance is reasonable that the user might upload duplicate files (or just uses them in a web-based app) than you have saved on waisted bandwith just to perform the check. That is quite a (ecological/financial) win in my book.
Extra
Hashes are prone to collision, period. To lower the (realistic) chance of collision you'd select a more advanced hash-algo (most are easily carry-able in chunked mode). Obvious trade-off for more advanced hashes is larger code-size and lower speed (higher CPU usage).

Use FileAPI to download big generated data file

The JavaScript process generates a lot of data (200-300MB). I would like to save this data for further analysis but the best I found so far is saving using this example http://jsfiddle.net/c2U2T/ which is not an option for me, because it looks like it requires all the data being available before starting the downloading. But what I need is something like
var saver = new Saver();
saver.save(); // The Save As ... dialog appears
saver.onaccepted = function () { // user accepted saving
for (var i = 0; i < 1000000; i++) {
saver.write(Math.random());
}
};
Of course, instead of the Math.random() will be some meaningful construction.

#dader - I would build upon dader's example.
Use HTML5 FileSystem API - but instead of writing to the file each and every line (more IO than it is worth), you can batch some of the lines in memory in a javascript object/array/string, and only write it to the file when they reach a certain threshold. You are thus appending to a local file as the process chugs (makes it easy to pause/restart/stop etc)
Of note is the following, which is an example of how you can spawn the dialoge to request the amount of data that you would need (it sounds large). Tested in chrome.:
navigator.persistentStorage.queryUsageAndQuota(
function (usage, quota) {
var availableSpace = quota - usage;
var requestingQuota = args.size + usage;
if (availableSpace >= args.size) {
window.requestFileSystem(PERSISTENT, availableSpace, persistentStorageGranted, persistentStorageDenied);
} else {
navigator.persistentStorage.requestQuota(
requestingQuota, function (grantedQuota) {
window.requestFileSystem(PERSISTENT, grantedQuota - usage, persistentStorageGranted, persistentStorageDenied);
}, errorCb
);
}
}, errorCb);
When you are done you can use Javascript to open a new window with the url of that blob object that you saved which you can retrieve via: fileEntry.toURL()
OR - when it is done crunching you can just display that URL in an html link and then they could right click on it and do whatever Save Link As that they want.
But this is something that is new and cool that you can do entirely in the browser without needing to involve a server in any way at all. Side note, 200-300MB of data generated by a Javascript Process sounds absolutely huge... that would be a concern for whether you are storing the "right" data...

What you actually are trying to do is a kind of streaming. I mean FileAPI is not suited for the task. Instead, I could suggest two options :
The first, using XHR facility, ie ajax, by splitting your data into several chunks which will sequencially be sent to the server, each chunk in its own request along with an id ( for identifying the stream ) and a position index ( for identifying the chunk position ). I won't recommend that, since it adds work to break up and reassemble data, and since there's a better solution.
The second way of achieving this is to use Websocket API. It allows you to send data sequentially to the server as it is generated. Following a usual stream API. I think you definitely need this.
This page may be a good place to start at : http://binaryjs.com/
That's all folks !
EDIT considering your comment :
I'm not sure to perfectly get your point though but, what about HTML5's FileSystem API ?
There are a couple examples here : http://www.html5rocks.com/en/tutorials/file/filesystem/ among which this sample that allows you to append data to an existant file. You can also create a new file, etc. :
function onInitFs(fs) {
fs.root.getFile('log.txt', {create: false}, function(fileEntry) {
// Create a FileWriter object for our FileEntry (log.txt).
fileEntry.createWriter(function(fileWriter) {
fileWriter.seek(fileWriter.length); // Start write position at EOF.
// Create a new Blob and write it to log.txt.
var blob = new Blob(['Hello World'], {type: 'text/plain'});
fileWriter.write(blob);
}, errorHandler);
}, errorHandler);
}
EDIT 2 :
What you're trying to do is not possible using javascript as said on SO here. Tha author nonetheless suggest to use Java Applet to achieve needed behaviour.
To put it in a nutshell, HTML5 Filesystem API only provides a sandboxed filesystem, ie located in some hidden directory of the browser. So if you want to access the true filesystem, using java would be just fine considering your use case. I guess there is an interface between java and javascript here.
But if you want to make your data only available from the browser ( constrained by same origin policy ), use FileSystem API.

How long does a Blob persist?

I'm trying to write a fail-safe program that uses the canvas to draw very large images (60 MB is probably the upper range, while 10 MB is the lower range). I have discovered long ago that calling the canvas's synchronous function toDataURL usually causes the page to crash in the browser, so I have adapted the program to use the toBlob method using a filler for cross-browser compatibility. My question is this: How long do Blob URLs using the URL.createObjectURL(blob) method last?
I would like to know if there's a way to cache the Blob URL that will allow it to last beyond the browser session in case somebody wants to render part of the image at one point, close the browser, and come back and finish it later by reading the Blob URL into the canvas again and resuming from the point at which it left off. I noticed that this optional autoRevoke argument may be what I'm looking for, but I'd like a confirmation that what I'm trying to do is actually possible. No code example is needed in your answer unless it involves a different solution, all I need is a yes or no on if it's possible to make a Blob URL last beyond sessions using this method or otherwise. (This would also be handy if for some reason the page crashes and it acts like a "restore session" option too.)
I was thinking of something like this:
function saveCache() {
var canvas = $("#canvas")[0];
canvas.toBlob(function (blob) {
/*if I understand correctly, this prevents it from unloading
automatically after one asynchronous callback*/
var blobURL = URL.createObjectURL(blob, {autoRevoke: false});
localStorage.setItem("cache", blobURL);
});
}
//assume that this might be a new browser session
function loadCache() {
var url = localStorage.getItem("cache");
if(typeof url=="string") {
var img = new Image();
img.onload = function () {
$("#canvas")[0].getContext("2d").drawImage(img, 0, 0);
//clear cache since it takes up a LOT unused of memory
URL.revokeObjectURL(url);
//remove reference to deleted cache
localStorage.removeItem("cache");
init(true); //cache successfully loaded, resume where it left off
};
img.onprogress = function (e) {
//update progress bar
};
img.onerror = loadFailed; //notify user of failure
img.src = url;
} else {
init(false); //nothing was cached, so start normally
}
}
Note that I am not certain this will work the way I intend, so any confirmation would be awesome.
EDIT just realized that sessionStorage is not the same thing as localStorage :P

Blob URL can last across sessions? Not the way you want it to.
The URL is a reference represented as a string, which you can save in localStorage just like any string. The location that URL points to is what you really want, and that won't persist across sessions.
When using URL.toObjectUrl() in conjuction with the autoRevoke argument, the URL will persist until you call revokeObjectUrl or "till the unloading document cleanup steps are executed." (steps outlined here: http://www.w3.org/TR/html51/browsers.html#unloading-document-cleanup-steps)
My guess is that those steps are being executed when the browser session expires, which is why the target of your blobURL can't be accessed in subsequent sessions.
Some other discourse on this: How to save the window.URL.createObjectURL() result for future use?
The above leads to a recommendation to use the FileSystem API to save the blob representation of your canvas element. When requesting the file system the first time, you'll need to request PERSISTENT storage, and the user will have to agree to let you store data on their machine permanently.
http://www.html5rocks.com/en/tutorials/file/filesystem/ has a good primer everything you'll need.

Large file upload with WebSocket

I'm trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can't figure out how to write "send this slice of the file, release the resources used then repeat". I was hoping I could avoid using something like Flash/Silverlight for this.
Currently, I'm working with something along the lines of:
function FileSlicer(file) {
// randomly picked 1MB slices,
// I don't think this size is important for this experiment
this.sliceSize = 1024*1024;
this.slices = Math.ceil(file.size / this.sliceSize);
this.currentSlice = 0;
this.getNextSlice = function() {
var start = this.currentSlice * this.sliceSize;
var end = Math.min((this.currentSlice+1) * this.sliceSize, file.size);
++this.currentSlice;
return file.slice(start, end);
}
}
Then, I would upload using:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
for(var i = 0; i < fs.slices; ++i) {
socket.send(fs.getNextSlice()); // see below
}
}
}
Basically this returns immediately, bufferedAmount is unchanged (0) and it keeps iterating and adding all the slices to the queue before attempting to send it; there's no socket.afterSend to allow me to queue it properly, which is where I'm stuck.

Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice().
This article helps you to handle large files in workers. change XHR send to Websocket in main thread.
//Messages from worker
function onmessage(blobOrFile) {
ws.send(blobOrFile);
}
//construct file on server side based on blob or chunk information.

I believe the send() method is asynchronous which is why it will return immediately. To make it queue, you'd need the server to send a message back to the client after each slice is uploaded; the client can then decide whether it needs to send the next slice or a "upload complete" message back to the server.
This sort of thing would probably be easier using XMLHttpRequest(2); it has callback support built-in and is also more widely supported than the WebSocket API.

In order to serialize this operation you need the server to send you a signal every time a slice is received & written (or an error occurs), this way you could send the next slice in response to the onmessage event, pretty much like this:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
socket.send(fs.getNextSlice());
}
socket.onmessage = function(ms){
if(ms.data=="ok"){
fs.slices--;
if(fs.slices>0) socket.send(fs.getNextSlice());
}else{
// handle the error code here.
}
}
}

You could use https://github.com/binaryjs/binaryjs or https://github.com/liamks/Delivery.js if you can run node.js on the server.

EDIT : The web world, browsers, firewalls, proxies, changed a lot since this answer was made. Right now, sending files using websockets
can be done efficiently, especially on local area networks.
Websockets are very efficient for bidirectional communication, especially when you're interested in pushing information (preferably small) from the server. They act as bidirectional sockets (hence their name).
Websockets don't look like the right technology to use in this situation. Especially given that using them adds incompatibilities with some proxies, browsers (IE) or even firewalls.
On the other end, uploading a file is simply sending a POST request to a server with the file in the body. Browsers are very good at that and the overhead for a big file is really near nothing. Don't use websockets for that task.

I think this socket.io project has a lot of potential:
https://github.com/sffc/socketio-file-upload
It supports chunked upload, progress tracking and seems fairly easy to use.