I've got a WebGL application which requires me to load a lot of x,y vertex data, but to minimise bandwidth usage I want to also compress the data (on a one off basis) using gzip.
Below is the code I will use to load in the data. I want to retrieve data from a server and pass it straight into a Float32Array.
var xhr = new XMLHttpRequest();
xhr.open('GET', 'array.gz', true);
xhr.responseType = 'arraybuffer';
xhr.onload = function(data) {
console.log("loaded");
var dataArray = new Float32Array(this.response);
xhr.onprogress = function(e) {
};
xhr.onerror = function(error) {
console.log('error!');
};
xhr.send();
Now my problem isn't linked to the code directly, but to the file format supported. In what format (i.e. csv, json, xml) does the data need to be in, before being gzipped so that this method can properly consume it?
I've played around to find that if I JSON.stringify a Float32Array and place the content in a file, then load it in, it works fine. However, to load in all my uncompressed data into a JavaScript array, before copying all its contents back into a new file to be compressed isn't very feasible. So I'm really looking for an alternative way to this (assuming this file format is the only one supported).
Related
There are two ways I can upload files using Ajax (XHR2). First, I can read the file content as array buffer or binary string and then simply stream using XHR send method. For example, as shown here:
function uploadFile(img, file) {
const reader = new FileReader();
const xhr = new XMLHttpRequest();
xhr.upload.addEventListener("progress", function(e) {
if (e.lengthComputable) {
const percentage = Math.round((e.loaded * 100) / e.total);
// Do something with percentage
}
});
xhr.upload.addEventListener("load", (e) => console.log('Do something more'));
xhr.open("POST", "some-url");
xhr.overrideMimeType('text/plain; charset=x-user-defined-binary');
reader.onload = function(evt) {
xhr.send(evt.target.result);
};
reader.readAsBinaryString(file);
}
Second, I can use FormData to upload my file as shown here:
var formData = new FormData();
// HTML file input, chosen by user
formData.append("userfile", fileInputElement.files[0]);
var request = new XMLHttpRequest();
request.open("POST", "some-url");
request.send(formData);
Are the two methods equivalent? Is there any advantage of using FileReader instead of FormData? Is one more performant than the other?
First, there is a third option you omitted which is to send the File directly through xhr.send(file) just like you did with the ArrayBuffer.
That being said, there doesn't exist any possible advantage to first reading the file in memory through FileReader.
When doing a file upload from a File on disk, the browser doesn't load the full file in memory but streams it through the request. This is how you can upload gigs of data even though it wouldn't fit in memory. This also is more friendly with the HDD since it allows for other processes to access it between each chunk instead of locking it.
When reading the File through a FileReader you are asking the browser to read the full file to memory, and then when you send it through XHR the data from memory is being used. You are thus limited by the memory available, bloating it for no good reasons, and even asking the CPU to work here while the data could have gone from the disk to the network card almost directly.
As to what's the difference between formdata.append(file); xhr.send(formdata); and xhr.send(file), basically only request headers. The former will wrap the request as a multipart/form-data enctype request, while the latter will send it as is.
So you'd handle both requests differently on the receiving end.
I am working on a JavaScript library that is a rich text editor. Currently, we can not copy + paste from a Word Document due to the clipboard being a local file and most browsers block the ability to read local files. Our work around to this issue is to convert the local image file to a base-64, and using our local API, store the image on our servers so the img has a proper, hosted source.
The problem is this has to be handled entirely in JavaScript, which still is run in the browser and I can not see local files to be able to convert them:
This is currently the code I am trying to use, but can't seem to figure out a way to handle this entirely in JavaScript without using some time of File Loader from HTML.
function convertToBase64(){
var xhr = new XMLHttpRequest();
// Set up our request
xhr.open('POST', 'http://uploadFileHere');
//convert image to base64
xhr.onload = function() {
var reader = new FileReader();
reader.onloadend = function() {
callback(reader.result);
}
reader.readAsDataURL(xhr.response);
};
//handle the response from Image API
xhr.onreadystatechange = function() {
if(xhr.readyState == XMLHttpRequest.DONE && xhr.status == 200) {
return xhr.responseText;
}
}
xhr.send(base64String);
}
Is it possible to work around this issue or does my team and I have to come up with another solution?
EDIT: Here is another rich text editor, where it IS possible to copy and paste images from Word, Froala. If this is not possible, how are they doing it?
is it possible to load an image directly into a canvas control using a generic handler without using the image element?
This is my handler:
public void ProcessRequest(HttpContext context)
{
context.Response.AddHeader("Access-Control-Allow-Origin", "*");
context.Response.ContentType = "image/jpeg";
var camIndex = Convert.ToInt16(context.Request.QueryString["camIndex"]);
context.Response.BinaryWrite( Shared.Feeder[camIndex].JpegData);
}
My JavaScript:
var xhr = new XMLHttpRequest();
xhr.open('GET', 'http://127.0.0.1/Media/FrameHandler.ashx?camIndex=' + camIndex, true);
xhr.responseType = 'arraybuffer';
xhr.onload = function (e) {
var uInt8Array = new Uint8ClampedArray(this.response);
imageData[camIndex].data.set(uInt8Array);
ctxLiveViews[camIndex].putImageData(imageData[camIndex], 0, 0);
};
xhr.send();
which gives me this image (which is obviously wrong)
Is it possible to load an image directly into a canvas control using a generic handler without using the image element?
It is, but you are in for a hand-full as you will need to manually parse the image file format yourselves (with all the various combinations of image format, color model etc. and error checks and so forth). It's doable (been there done that), but chances are, unless you need to access some special data or bitmap formats, that the browser will do the job at least faster than a manual approach in JavaScript.
Using an Image element is easy and does all these steps for you in compiled code.
This line:
var uInt8Array = new Uint8ClampedArray(this.response);
will only hold the original file bytes, uncompressed, undecoded including header, chunks and metadata. While putImageData() require a raw bitmap in the format RGBA per pixel. This is why you see the noise as the data being fed to putImageData is the file itself, not a bitmap.
I have a webworker that is producing a CSV for downloading, and to conserve memory I only have it return the URL it produces from teh blob..
My worker code looks something like::
var blob = new Blob([ResultOfSomeWork()],{type:'text/csv'});
URL.createObjectURL(blob);
self.postMessage({url:blob.url});
My goal is to just be able to download it in firefox and chrome this is very easy as I can just set up an invisible <a> and have it be clicked to download it.
For IE10 I want to use msSaveBlob but I need a blob which I don't want to transfer.
How can I download a object dataurl in IE10?
So I found a solution that works. Apparently, I can XHR and read the content back in my main thread.
worker.onmessage = function(event){
var url = event.data.url;
var xhr = new XMLHttpRequest();
xhr.open('GET', url, true);
xhr.responseType='blob';
xhr.onload = function(){
msSaveBlob(xhr.response, fileName +'.csv');
};
xhr.send();
}
While this feels extremely complicated it works well in practice and is really fast.
I've searched related questions but wasn't able to find any relevant info.
I'm trying to get the Web Audio API to play an mp3 file which is encoded in another file container, so what I'm doing so far is parsing said container, and feeding the result binary data (arraybuffer) to the audioContext.decodeAudioData method, which supposedly accepts any kind of arraybuffer containing audio data. However, it always throws the error callback.
I only have a faint grasp of what I'm doing so probably the whole approach is wrong. Or maybe it's just not possible.
Has any of you tried something like this before? Any help is appreciated!
Here's some of the code to try to illustrate this better. The following just stores the arraybuffer:
newFile: function(filename){
var that=this;
var oReq = new XMLHttpRequest();
oReq.open("GET", filename, true);
oReq.responseType = "arraybuffer";
oReq.onload = function (oEvent) {
var arrayBuffer = oReq.response; //
if (arrayBuffer) {
that.arrayBuffer=arrayBuffer;
that.parsed=true;
}
};
oReq.send(null);
And this is what I'm doing in the decoding part:
newTrack: function(tracknumber){
var that=this;
var arraybuffer=Parser.arrayBuffer;
that.audioContext.decodeAudioData(arraybuffer,function(buffer){
var track={};
track.trackBuffer=buffer;
track.isLoaded=true;
track.trackSource=null;
track.gainNode=that.audioContext.createGainNode();
that.tracklist.push(track);
},alert('error'));
Where Parser is an object literal that I've used to parse and store the arraybuffer (which has the newFile function)
So, to sum up, I don't know if I'm doing something wrong or it simply cannot be done.
Without the container, I'm not sure how decodeAudioData would know that it's an MP3. Or what the bitrate is. Or how many channels it has. Or a lot of other pretty important information. Basically, you need to tell decodeAudioData how to interpret that ArrayBuffer.
The only thing I could think of on the client side is trying to use a Blob. You'd basically have to write the header yourself, and then readAsArrayBuffer before passing it in to decodeAudioData.
If you're interested in trying that out, here's a spec:
http://www.mpgedit.org/mpgedit/mpeg_format/mpeghdr.htm
And here's RecorderJS, which would show you how to create the Blob (although it writes RIFF/WAV headers instead of MP3):
https://github.com/mattdiamond/Recorderjs/blob/master/recorderWorker.js
You'd want to look at the encodeWAV method.
Anyway, I would strongly recommend getting this sorted out on the server instead, if you can.