I try to create code to upload files using XHR2 and web workers.
I thought I should use web workers , so if a file is big, web page will not freeze.
This is not working for two reasons, I never used web workers before, and I want to post to the server the file and vars at the same time, with the same xhr. When I say vars I mean the name of the file, and an int.
Heres is what I got
Client side
//create worker
var worker = new Worker('fileupload.js');
worker.onmessage = function(e) {
alert('worker says '+e.data);
}
//handle workers error
worker.onerror =werror;
function werror(e) {
console.log('ERROR: Line ', e.lineno, ' in ', e.filename, ': ', e.message);
}
//send stuff to the worker
worker.postMessage({
'files' : files, //img or video
'name' : nameofthepic, //text
'id':imageinsertid //number
});
Inside the worker (fileupload.js file)
onmessage = function (e) {var name=e.data.name; var id=e.data.id ; var file=e.data.files;
//create a var to catch the anser of the server
var datax;
var xhr = new XMLHttpRequest();
xhr.onload = function() {
if (xhr.status == 200) {datax=xhr.response;}
else { datax=525;}//actually, whatever, just give a value
};
xhr.open('POST', 'upload.php');
xhr.send(file,name,id);
//i also tried xhr.send('file=file&name=name&id=id'); and still nothing
//i also tried just the text/int xhr.send('name=name&id=id'); and still nothing
I am confused. I cannot send anything to the server. I get no feedback from the worker. I dont even know if the data are send to the fileupload.js. Server side does not INSERT.
Is that possible, sending files and text at the same time? What am I missing?
I need to pass text and int along with the file, so server side not only will upload the file, but also will INSERT to the database the int and the text, if the file is uploaded succesfully. This was easy just with formData and xhr, but, putting web workers in the middle, I cant get it right.
Also, can I use Transferable Objects to speed things up? Are Transferable Objects supported in all major browsers?
Thanks in advance
Related
I have a JS library that is responsible to perform the download of JPEG images for the client. All of this is done asynchronously. In some cases, the count of images is really large... Around 5000 images. In this case, the Chrome browser issues the "ERR_INSUFFICIENT_RESOURCES" error for the ajax request.
Each request must be done individually, there is no option to pack the images on the server-side.
What are my options here? How can I find a workaround for this problem? The download works fine in Firefox...
Attached code of the actual download:
function loadFileAndDecrypt(fileId, key, type, length, callback, obj) {
var step = 100 / length;
eventBus.$emit('updateProgressText', "downloadingFiles");
var req = new dh.crypto.HttpRequest();
req.setAesKey(key);
let dataUrl;
if (type == "study") {
dataUrl = "/v1/images/";
}else {
dataUrl = "/v1/dicoms/";
}
var url = axios.defaults.baseURL + dataUrl + fileId;
req.open("GET", url, true);
req.setRequestHeader("Authorization", authHeader().Authorization+"")
req.setRequestHeader("Accept", "application/octet-stream, application/json, text/plain, */*");
req.responseType = "arraybuffer";
req.onload = function() {
console.log(downloadStep);
downloadStep += step;
eventBus.$emit('updatePb', Math.ceil(downloadStep));
var data = req.response;
obj.push(data);
counter ++;
//last one
if (counter == length) {
callback(obj);
}
};
req.send();
}
The error means your code is overloading your memory (most likely, or the quota of pending requests was exhausted). Instead of sending all the data from the backend, make your frontend request for 5000 individual images instead and control the requests flow. regardless, downloading 5000 images is bad. You should pack them up for downloading. If you mean fetching the images, then loading images from the frontend through static or dynamic links is much more logical ;)
Create a class:
Which accepts the file-Id (image that needs to be downloaded) as an argument
Which can perform the HTTP API request
Which can store the result of the request
Create an array of objects from this class using how many ever file-Ids that needs to be downloaded.
Store the array in a RequestManager which can start and manage the downloads:
can batch the downloads, say it fires 5 requests from the array and waits for them to finish before starting the next batch
can stop the downloads on multiple failures
manipulate batch size depending on the available bandwidth
stops download on auth expiry and resumes on auth refresh
offers to retry the previously failed downloads
I am using a node sever to send a table from a sqlite db to the browser. This table contains filename and path of a pdf file that I want to render on the browser. Until now I was using hard coded paths for the the pdf file and rendering. But now i have setup a get route and a controller in node such that whenever '/content' is hit in browser , the server queries the database and and sends the data to the client. To the send the data I am using
res.render('content/index',{data:queryData});
Now, how do I access this data using client side javascript so that I can pass the path of the pdf file to the function that renders the pdf? I have done research and the nearest answer I got was using XMLHttpRequest. I tried this method
var xhr = new XMLHttpRequest();
const path = "http://localhost:3000/content";
xhr.onreadystatechange = function () {
if (xhr.readyState == 4 && xhr.status == 200)
{
var myResponseText = xhr.responseText;
console.log(myResponseText);
}
};
xhr.open('get', path, true);
xhr.send();
When I do this I get the entire html code for the view. Not the data I expected. How do I solve this issue. I have done some more reading while writing this post and I suppose. I have set a header somewhere? But the documentation says
app.render(view, [locals], callback)
which means res.render can take local variables, shouldn't be setting the headers?
You should return json instead of render template:
app.get('content/index', (req, res) => {
res.json({data: queryData});
});
I am using pdf.js
PDF.js needs the PDF file, e.g.:
pdfjsLib.getDocument('helloworld.pdf')
I'm assuming your queryData goes something like this:
{ filename: 'file.pdf', path: './path/to/file.pdf' }
I'm not sure what's in your content/index or what path this is on, but you obviously need to find a way to make your PDF file ('./path/to/file.pdf') available (as a download). See Express's built-in static server or res.download() to do that.
Once you have the PDF file available as a download, plug that path into PDF.js's .getDocument('/content/file.pdf') and do the rest to render the PDF onto the canvas or whatever.
Hope that helps.
I'm making a JavaScript script that is going to essentially save an old game development sandbox website before the owners scrap it (and lose all of the games). I've created a script that downloads each game via AJAX, and would like to somehow upload it straight away, also using AJAX. How do I upload the downloaded file (that's stored in responseText, presumably) to a PHP page on another domain (that has cross origin headers enabled)?
I assume there must be a way of uploading the data from the first AJAX request, without transferring the responseText to another AJAX request (used to upload the file)? I've tried transferring the data, but as expected, it causes huge lag (and can crash the browser), as the files can be quite large.
Is there a way that an AJAX request can somehow upload individual packets as soon as they're recieved?
Thanks,
Dan.
You could use Firefox' moz-chunked-text and moz-chunked-arraybuffer response types. On the JavaScript side you can do something like this:
function downloadUpload() {
var downloadUrl = "server.com/largeFile.ext";
var uploadUrl = "receiver.net/upload.php";
var dataOffset = 0;
xhrDownload = new XMLHttpRequest();
xhrDownload.open("GET", downloadUrl, true);
xhrDownload.responseType = "moz-chunked-text"; // <- only works in Firefox
xhrDownload.onprogress = uploadData;
xhrDownload.send();
function uploadData() {
var data = {
file: downloadUrl.substring(downloadUrl.lastIndexOf('/') + 1),
offset: dataOffset,
chunk: xhrDownload.responseText
};
xhrUpload = new XMLHttpRequest();
xhrUpload.open("POST", uploadUrl, true);
xhrUpload.setRequestHeader('Content-Type', 'application/json; charset=UTF-8');
xhrUpload.send(JSON.stringify(data));
dataOffset += xhrDownload.responseText.length;
};
}
On the PHP side you need something like this:
$in = fopen("php://input", "r");
$postContent = stream_get_contents($in);
fclose($in);
$o = json_decode($postContent);
file_put_contents($o->file . '-' . $o->offset . '.txt', $o->chunk);
These snippets will just give you the basic idea, you'll need to optimize the code yourself.
I have firefox addon (made using the firefox addon sdk) which when instructed takes a screenshot of the currently active tab and sends it in an ajax request to a remote server.
The following code in addon-script main.js is responsible for taking getting the thumbnail
var tab_image_data=tabs.activeTab.getThumbnail();
var tab_image_data = base64.encode(tab_image_data);
panel.port.emit("screenshot",{image_data:tab_image_data});
The image data generated by the function getThumbnail() is sent to a content script file belonging to a panel.
In the content script the following code is responsible for sending the image data to the server
var tab_image_data=addonmessage.image_data;
var myBlob = new Blob([tab_image_data], { "type" : "text/base64data"} );
var formData = new FormData();
formData.append('img',myBlob);
var xhr = new XMLHttpRequest();
xhr.open("POST", "http://xyz.abc.com/imageshot/index.php", true);
xhr.onload = function() {
if (xhr.status == 200) {
console.log('all done: sq');
} else {
console.log('Nope');
}
};
xhr.send(formData);
Everything works fine and the image file is created on server in jpeg format. But when I try to view that image, the windows photo viewer gives a not supported file format error.
I tried to base64 encode the data before sending it but that did not work as well. Suprisingly I have a chrome extension that performs a similar function and that works flawlessly. In both cases the same php script is being used on the server side.
Any help regarding this matter will be highly appreciated
Thanks in advance.
I need to create a small browser-based application that helps users download/save, and possibly print to the default printer, a large number of files from a webserver we have no control over (but we have all the URIs beforehand).
These files are hidden behind a "single sign-on" (SSO) that can only be performed via a browser and requires a user. Hence, it must be a browser-based solution, where we piggyback on to the session established by the SSO.
The users' platform is Windows 7.
The point is to save the users from going through a lot of clicks per file (download, where to save, etc.) when they need to perform this operation (daily).
At this point all the files are PDF, but that might change in the future.
A browser-agnostic solution is preferred (and I assume more robust when it comes to future browser updates).
But we can base it on a particular browser if needed.
How would you do this from Javascript?
As the comments to my question says, this isn't really allowed by the browsers for security reasons.
My workaround for now (only tested using IE11) is to manually change the security settings of the users browser, and then download the files as a blob into a javascript variable using AJAX, followed by upload of same blob to my own server again using AJAX.
"My own server" is a Django site created for this purpose, that also knows which files to download for the day, and provide the javascript needed. The user goes to this site to initiate the daily download after performing the SSO in a separate browser tab.
On the server I can then perform whatever operations needed for said files.
Many thanks to this post https://stackoverflow.com/a/13887220/833320 for the handling of binary data in AJAX.
1) In IE, add the involved sites to the "Local Intranet Zone", and enable "Access data sources across domains" for this zone to overcome the CORS protection otherwise preventing this.
Of course, consider the security consequences involved in this...
2) In javascript (browser), download the file as a blob and POST the resulting data to my own server:
var x = new XMLHttpRequest();
x.onload = function() {
// Create a form
var fd = new FormData();
fd.append('csrfmiddlewaretoken', '{{ csrf_token }}'); // Needed by Django
fd.append('file', x.response); // x.response is a Blob object
// Upload to your server
var y = new XMLHttpRequest();
y.onload = function() {
alert('File uploaded!');
};
y.open('POST', '/django/upload/');
y.send(fd);
};
x.open('GET', 'https://external.url', true);
x.responseType = 'blob'; // <-- This is necessary!
x.send();
3) Finally (in Django view for '/django/upload/'), receive the uploaded data and save as file - or whatever...
filedata = request.FILES['file'].read()
with open('filename', 'wb') as f:
f.write(filedata)
Thanks all, for your comments.
And yes, the real solution would be to overcome the SSO (that requieres the user), so it all could be done by the server itself.
But at least I learned a little about getting/posting binary data using modern XMLHttpRequests. :)
Actually, I had a problem like it, I wanted to download a binary file(an image) and store it and then use it when I need it, So I decided to download it with Fetch API Get call:
const imageAddress = 'an-address-to-my-image.jpg'; // sample address
fetch(imageAddress)
.then(res => res.blob) // <-- This is necessary!
.then(blobFileToBase64)
.then(base64FinalAnswer => console.log(base64FinalAnswer))
The blobFileToBase64 is a helper function that converts blob binary file to a base64 data string:
const blobToBase64 = blob => {
const reader = new FileReader();
reader.readAsDataURL(blob);
return new Promise(resolve => {
reader.onloadend = () => {
resolve(reader.result);
};
});
};
In the end, I have the base64FinalAnswer and I can do anything with it.