The JavaScript process generates a lot of data (200-300MB). I would like to save this data for further analysis but the best I found so far is saving using this example http://jsfiddle.net/c2U2T/ which is not an option for me, because it looks like it requires all the data being available before starting the downloading. But what I need is something like
var saver = new Saver();
saver.save(); // The Save As ... dialog appears
saver.onaccepted = function () { // user accepted saving
for (var i = 0; i < 1000000; i++) {
saver.write(Math.random());
}
};
Of course, instead of the Math.random() will be some meaningful construction.
#dader - I would build upon dader's example.
Use HTML5 FileSystem API - but instead of writing to the file each and every line (more IO than it is worth), you can batch some of the lines in memory in a javascript object/array/string, and only write it to the file when they reach a certain threshold. You are thus appending to a local file as the process chugs (makes it easy to pause/restart/stop etc)
Of note is the following, which is an example of how you can spawn the dialoge to request the amount of data that you would need (it sounds large). Tested in chrome.:
navigator.persistentStorage.queryUsageAndQuota(
function (usage, quota) {
var availableSpace = quota - usage;
var requestingQuota = args.size + usage;
if (availableSpace >= args.size) {
window.requestFileSystem(PERSISTENT, availableSpace, persistentStorageGranted, persistentStorageDenied);
} else {
navigator.persistentStorage.requestQuota(
requestingQuota, function (grantedQuota) {
window.requestFileSystem(PERSISTENT, grantedQuota - usage, persistentStorageGranted, persistentStorageDenied);
}, errorCb
);
}
}, errorCb);
When you are done you can use Javascript to open a new window with the url of that blob object that you saved which you can retrieve via: fileEntry.toURL()
OR - when it is done crunching you can just display that URL in an html link and then they could right click on it and do whatever Save Link As that they want.
But this is something that is new and cool that you can do entirely in the browser without needing to involve a server in any way at all. Side note, 200-300MB of data generated by a Javascript Process sounds absolutely huge... that would be a concern for whether you are storing the "right" data...
What you actually are trying to do is a kind of streaming. I mean FileAPI is not suited for the task. Instead, I could suggest two options :
The first, using XHR facility, ie ajax, by splitting your data into several chunks which will sequencially be sent to the server, each chunk in its own request along with an id ( for identifying the stream ) and a position index ( for identifying the chunk position ). I won't recommend that, since it adds work to break up and reassemble data, and since there's a better solution.
The second way of achieving this is to use Websocket API. It allows you to send data sequentially to the server as it is generated. Following a usual stream API. I think you definitely need this.
This page may be a good place to start at : http://binaryjs.com/
That's all folks !
EDIT considering your comment :
I'm not sure to perfectly get your point though but, what about HTML5's FileSystem API ?
There are a couple examples here : http://www.html5rocks.com/en/tutorials/file/filesystem/ among which this sample that allows you to append data to an existant file. You can also create a new file, etc. :
function onInitFs(fs) {
fs.root.getFile('log.txt', {create: false}, function(fileEntry) {
// Create a FileWriter object for our FileEntry (log.txt).
fileEntry.createWriter(function(fileWriter) {
fileWriter.seek(fileWriter.length); // Start write position at EOF.
// Create a new Blob and write it to log.txt.
var blob = new Blob(['Hello World'], {type: 'text/plain'});
fileWriter.write(blob);
}, errorHandler);
}, errorHandler);
}
EDIT 2 :
What you're trying to do is not possible using javascript as said on SO here. Tha author nonetheless suggest to use Java Applet to achieve needed behaviour.
To put it in a nutshell, HTML5 Filesystem API only provides a sandboxed filesystem, ie located in some hidden directory of the browser. So if you want to access the true filesystem, using java would be just fine considering your use case. I guess there is an interface between java and javascript here.
But if you want to make your data only available from the browser ( constrained by same origin policy ), use FileSystem API.
Related
I'm using jquery.fileupload() to upload files from a browser to a Node.js server, which parses the files with the "multiparty" npm. I need to enforce a size limit on each file, and also on total size of all the files being uploaded in one request.
The "multiparty" npm allows me to do the latter, but not the former. And even for the latter, the limit isn't enforced until the browser uploads enough data to hit the limit. So the user can wait a long time only to get an error message.
I'd like to enforce the limit on the client-side. I've searched the Internet for solutions, but none of them seem to work. They may have worked in the past, but not with the newest version of Chrome.
I've found that I can determine that the files are too big by watching for a "change" event on the file-input element, like this:
$('#file-input-id').on('change', function() {
console.log(this.files);
});
When this event triggers, this.files contains an array of selected files, including the name and size of each. So I can determine that the caps have been exceeded, and I can alert the user. But I don't know how to stop the files from uploading anyway. Various source on the Internet suggest that I can do this by returning false or manipulating this.files. But none of this seems to work.
I'm testing this against the latest version of Chrome (66.0.3359.139), but I'd like a solution that works with any modern browser.
The file object that exists on the element has a size property which you can use to compare and make validations on the client. I wrote an example in javascript. I know you want it in JQuery but, that was kind of already answered here
Anyways, this is what I came up with ...
var inputElement = document.getElementById("file")
inputElement.addEventListener('change', function(){
var fileLimit = 100; // could be whatever you want
var files = inputElement.files;
var fileSize = files[0].size; //inputElement.files is always an array
var fileSizeInKB = (fileSize/1024); // this would be in kilobytes defaults to bytes
if(fileSizeInKB < fileLimit){
console.log("file go for launch")
// add file to server here
} else {
console.log("file too big")
// do not pass go, do not add to server. Pass error to user
document.getElementById("error").innerHTML = "your file is over 100 KB "
}
})
(CodePen https://codepen.io/HappinessFactory/pen/yjggbq)
Hope that answers your question. Good luck!
Thanks! I'm sure your answer would work if I weren't using jquery.fileupload(), but jquery.fileupload() starts the upload automatically. So there's no "add file to server" logic to perform/skip.
But your answer sent me off in the right direction. For anyone else stuck on this: The trick is to use the "start" or "submit" properties of the "options" object passed into jquery.fileupload(). Both of these are functions, and if either one returns false, the upload is cancelled.
I’d like to use javascript to create and save a large .csv file client-side. This has been asked and answered before (see 1, 2, 3, or 4, for example), and this comes down to two approaches (among others):
Use FileSaver.js to implement w3c’s saveAs():
var lines = ["line 1,cell 11,cell 12\n","line 2,cell 21,cell 22\n"];
var type = "text/csv;charset=utf-8";
var blob = new Blob(lines,{"type":type});
saveAs(blob,"download.csv");
Use a[download] and a data uri:
var lines = ["line 1,cell 11,cell 12\n","line 2,cell 21,cell 22\n"];
var type = "text/csv;charset=utf-8";
var downloader = $('<a download href="data:'+type+','+escape(lines.join(''))+'"></a>')
.appendTo("body")
downloader[0].click();
downloader.remove();
My problem is that my file can be gigantic: 18 million lines and 2.5 Gb (and I’d like to be able to handle more). Creating lines uses too much memory and crashes the site. But there’s no reason to store lines in the browser’s memory just to save it to the hard drive. Is there a way to create a progressive download using javascript (in other words, start the download but keep appending lines as I calculate them)? Or would my only option be to download the file as separate chunks that the user must then join together?
If you are using HTML5, you can try using the FileSytem API. See here for information on the subject, and here for a more practical example on how to create and access files fron the client-side.
So I ran into an issue where I was writing contents to a file via the HTML5 File-system api. The issue occurs when new content is shorter than the previous content, the old content is written-over as expected, but the tail of the old contents remain at the end of the file. The data I am writing is meta data for a given web-app and tends to change periodically, but not very often, generally increasing in size but occasionally the meta data is smaller in size.
Example, original content of file 0000000000, new content 11123 and after writing to the file, the contents become 1112300000
To get around this, I have been removing the file and passing a callback to write the new information in on every call. (cnDAO.filesystem is the filesystem object obtained when requesting persistent memory and has been initialized appropriately)
function writeToFile(fPath,data,callback){
rmFile(fPath,function(){
cnDAO.fileSystem.root.getFile(fPath, {
create: true
}, function(fileEntry) {
fileEntry.createWriter(function(writer) {
writer.onwriteend = function(e) {
callback();
};
writer.onerror = function(e3) { };
var blob = new Blob([data]);
writer.write(blob);
}, errorHandler);
}, errorHandler);
});
}
function rmFile(fPath,callback){
cnDAO.fileSystem.root.getFile(fPath, {
create: true
}, function(fileEntry) {
fileEntry.remove(callback);
}, errorHandler);
}
So, I was wondering if there was a better way to do what I am doing. truncate appeared in the following while I was searching for a solution (this post). As pointed out in the previous post truncate can only be called immediately after opening a file - Is truncate a better approach? Is what I'm doing better practice? Is there a quicker and easier way that I do not know about?
I would like to just start-fresh on every write to file- if that is plausible and/or good practice.
I'm trying to upload large files (at least 500MB, preferably up to a few GB) using the WebSocket API. The problem is that I can't figure out how to write "send this slice of the file, release the resources used then repeat". I was hoping I could avoid using something like Flash/Silverlight for this.
Currently, I'm working with something along the lines of:
function FileSlicer(file) {
// randomly picked 1MB slices,
// I don't think this size is important for this experiment
this.sliceSize = 1024*1024;
this.slices = Math.ceil(file.size / this.sliceSize);
this.currentSlice = 0;
this.getNextSlice = function() {
var start = this.currentSlice * this.sliceSize;
var end = Math.min((this.currentSlice+1) * this.sliceSize, file.size);
++this.currentSlice;
return file.slice(start, end);
}
}
Then, I would upload using:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
for(var i = 0; i < fs.slices; ++i) {
socket.send(fs.getNextSlice()); // see below
}
}
}
Basically this returns immediately, bufferedAmount is unchanged (0) and it keeps iterating and adding all the slices to the queue before attempting to send it; there's no socket.afterSend to allow me to queue it properly, which is where I'm stuck.
Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice().
This article helps you to handle large files in workers. change XHR send to Websocket in main thread.
//Messages from worker
function onmessage(blobOrFile) {
ws.send(blobOrFile);
}
//construct file on server side based on blob or chunk information.
I believe the send() method is asynchronous which is why it will return immediately. To make it queue, you'd need the server to send a message back to the client after each slice is uploaded; the client can then decide whether it needs to send the next slice or a "upload complete" message back to the server.
This sort of thing would probably be easier using XMLHttpRequest(2); it has callback support built-in and is also more widely supported than the WebSocket API.
In order to serialize this operation you need the server to send you a signal every time a slice is received & written (or an error occurs), this way you could send the next slice in response to the onmessage event, pretty much like this:
function Uploader(url, file) {
var fs = new FileSlicer(file);
var socket = new WebSocket(url);
socket.onopen = function() {
socket.send(fs.getNextSlice());
}
socket.onmessage = function(ms){
if(ms.data=="ok"){
fs.slices--;
if(fs.slices>0) socket.send(fs.getNextSlice());
}else{
// handle the error code here.
}
}
}
You could use https://github.com/binaryjs/binaryjs or https://github.com/liamks/Delivery.js if you can run node.js on the server.
EDIT : The web world, browsers, firewalls, proxies, changed a lot since this answer was made. Right now, sending files using websockets
can be done efficiently, especially on local area networks.
Websockets are very efficient for bidirectional communication, especially when you're interested in pushing information (preferably small) from the server. They act as bidirectional sockets (hence their name).
Websockets don't look like the right technology to use in this situation. Especially given that using them adds incompatibilities with some proxies, browsers (IE) or even firewalls.
On the other end, uploading a file is simply sending a POST request to a server with the file in the body. Browsers are very good at that and the overhead for a big file is really near nothing. Don't use websockets for that task.
I think this socket.io project has a lot of potential:
https://github.com/sffc/socketio-file-upload
It supports chunked upload, progress tracking and seems fairly easy to use.
I'm working on a PhoneGap application that captures images using the camera and, later, uploads them. There are two modes of operation for camera in PhoneGap: raw base64 encoded data or a file URI.
The docs themselves say:
Note: The image quality of pictures taken using the camera on newer
devices is quite good. Encoding such images using Base64 has caused
memory issues on some of these devices (iPhone 4, BlackBerry Torch
9800). Therefore, using FILE_URI as the 'Camera.destinationType' is
highly recommended.
So I'm keen to use FILE_URI option. This works great and you can even show the images in IMG tags. The URL looks like this:
file://localhost/var/mobile/Applications/4FE4642B-944C-449BB-9BD6-1E442E47C7CE/tmp/photo_047.jpg
However, at some point later I want to read the contents of the file to upload to a server. I was going to do this using the FileReader type. This doesn't work and I think it's because I can't access the file at the URL above.
The error code I get back from readDataUrl is 1 > FileError.NOT_FOUND_ERR = 1;
Any ideas how I can get to the file? I tried just accessing the last part of the path (photo_047.jpg) based on another sample I saw but no luck.
I'm just getting started with PhoneGap, and given the age of this question you may have found an answer already, but I'll give it a try anyway.
First, would you be able to use the built-in FileTransfer object? It takes a file: URI as an argument.
If FileTransfer won't work for you, and you need to read the file data yourself, you'll need the PhoneGap File objects, like FileReader , as you said. But most of those expect a plain pathname -- not a URI -- to specify the file to work with. The reason you're getting NOT_FOUND_ERR is because it's trying to open a file named file:/localhost/var....
Here's a quick one-liner to extract the path part from your URI:
var path = /file:\/\/.*?(\/.*)/.exec(fileuri)[1];
Hope this helps!
The answer from jgarbers was of help to me but it did not solve the problem. I realized the camera stores photos in Temp folder instead of Document folder. Setting my local file system to temporary allowed it to find the correct location for the camera images.
window.requestFileSystem(LocalFileSystem.TEMPORARY, 0, ...
...
window.requestFileSystem(LocalFileSystem.TEMPORARY, 0, ...
...
var path = /file://.?(/.)/.exec(fileuri)[1];
Ref. above jgarbers and Rik answers (solution has been tested successfully on iOs 7)
you can user the file transfer plugin for uploading any file to the server.
//// pass your file uri to the mediafie param
function uploadFile(mediaFile) {
var ft = new FileTransfer();
path = mediaFile.fullPath;
name = mediaFile.name;
////your service method url
var objUrl = http://example.com;
ft.upload(path,
objUrl,
function (result) {
alert("Success");
},
function (error) {
alert('Error uploading file ' + path + ': ' + error.code);
},
{ fileName: name });
}