I send multiple files chunked into Blob's over XHR2 to a Node.js/Express server.
How can I receive them on the server while making sure they are put together correctly? In their right order and to the right file when multiple files are uploaded "at once".
Following is the code (both front- and backend) I have so far but doesn't account for multiple uploads yet.
Frontend:
// 'files' is of type FileList, directly from file input.
for (var i = 0, length = files.length; i < length; i++) {
var file = files[i];
var bytes = 51200; // 50 KB
var size = file.size;
var start = 0;
var end = bytes;
while (start < size) {
sendBlob(file.slice(start, end), file.name, file.type);
start = end;
end = start + bytes;
}
}
// sendBlob()
var sendBlob: function (data, filename, filetype) {
var xhr = new XMLHttpRequest();
xhr.open('POST', this.url, false);
xhr.setRequestHeader('X_FILENAME', filename);
xhr.setRequestHeader('Content-Type', filetype);
xhr.send(data);
};
Backend:
app.post('/', function (req, res) {
var body = '';
req.on('data', function (data) {
body += data;
});
req.on('end', function () {
var filename = req.headers['x_filename'];
var newPath = __dirname + '/upload/' + filename;
fs.writeFile(newPath, body, function (err) {
res.send({
filename: filename
});
});
});
});
Very small text files are stored correctly but images seem to always get messed up and end up with a bigger file size. Bigger text files are written correctly but there the first chunk seems to be missing.
Your upload logic is naive. Here are some things you should do to ensure correctness :
You have to maintain and communicate the chunk id/number between client and server so that order can be maintained.
var sendBlob: function (data, filename, filetype, chunkid)
//set chunkid in header or in data.
In your server you are accepting any post request and appending it to the body. You should maintain variables for filename and filetype and match it with incoming request before appending it.
Files[Name] = { //Create a new Entry in The Files Variable for each new file
Filetype : "",
FileSize: 0,//size of Data in buffer
Data: "", //buffer for storing data
Downloaded: //chunks recieved
}
Append to Data only when you check it. (Extra file size could be due to this)
In your fs.writeFile you should set encoding as binary, image and video files are binary encoded and writing them into default utf-8 encoding may corrupt them.
fs.writeFile(newPath, body, 'binary', function (err){...});
(optional) For each chunk received by server it should send an acknowledgement back to client so that it knows which chunk is dropped and must be sent.
Related
I'm using the gcloud API on a Nodejs web server to upload files. I'd prefer the files not be uploaded on the client side and instead uploaded on the server. Currently, I am producing a blob on the client side, then converting it to text and passing that to the server through a POST request. All of the information gets successfully passed from the client to the server as expected. This data is also uploaded to gcloud, however, Gcloud does not recognize this as a valid file nor does my computer when I download it.
What is the best way to get the contents of the file to gcloud from the server side? I've tried using dataURIs and reading the orignal file by text and both produce similiar issues. I've also explored piping a readFileStream from the blob on the server end but blobs are not natively supported by node so I have not done so yet.
Client Side
function readSingleFile(e, func, func2){
var file = e.target.files[0];
if(!file){
return; // Add error msg_here
}
var reader = new FileReader();
reader.onload = function(e){
let contents = e.target.result;
let img = document.createElement('img')
let cvs = document.createElement('canvas');
img.onload = ()=>{
cvs.width = img.width;
cvs.height= img.height;
let ctx = cvs.getContext('2d');
ctx.drawImage(img,0,0);
cvs.toBlob((res)=>{res.text().then((text)=>{func2(text)})}, "image/jpeg", 0.92);
}
img.src=contents;
func(contents);
}
reader.readAsDataURL(file);
}
Server Side
function publishPrintjob(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpg');
gcloudFile.save(dataObj.sockImageFile, function(err) {
if (!err) {
Console.log("File Uploaded!")
}
});
var data = {
date: dataObj.Date,
email: dataObj.email,
design: dataObj.Design,
author: dataObj.Author,
address: dataObj.address,
imageKey: newElemKey,
}
admin.database().ref('queue/' + newElemKey).set(data);
} catch(err){
console.log(err)
}
}
Note: func simply shows the image on the client side, func2 just adds the contents to the POST object.
Uploading a file directly from the computer would be easiest using the storage.bucket(bucketName).upload() function from the cloud storage library. However, this uses location of a file locally and thus will not work unless a file is transferred to the server and saved first. This could be achieved using multi-part form data. Using multipart or uploading locally are better methods for uploading to google storage.
Instead, I solve this by first converting the image to a dataURI, sending the data URI to the server via the body of a GET request, and then converting it to a buffer with a readable stream that can be piped to google storage.
Client
let formData = getFormData('myForm');
var xhttp = new XMLHttpRequest();
xhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {
// Typical action to be performed when the document is ready:
}
};
xhttp.open("POST", "dashboard", true);
xhttp.setRequestHeader('Content-Type', 'application/json');
xhttp.send(JSON.stringify(formData));
xhttp.onload = ()=> {
console.log(JSON.parse(xhttp.response))
// Handle server response here
};
}
Server
// DataObject is the body of the GET request, the property imageFile is the URI from readFileAsURI
function uploadImageOnServer(dataObj){
try{
var newElemKey = database.ref().child('queue').push().key; // Get random Key to use as filename
// Create a new blob in the bucket and upload the file data.
const gcloudFile = storage.file('images/' + newElemKey + '.jpeg');
var fs = require('fs'); // Should be required at the top of the file
var string = dataObj.ImageFile;
var regex = /^data:.+\/(.+);base64,(.*)$/;
var matches = string.match(regex);
var ext = matches[1];
var data = matches[2];
var buffer = Buffer.from(data, 'base64');
// Create the readstream
const readableInstanceStream = new Readable({
read() {
this.push(buffer);
this.push(null);
}
});
readableInstanceStream.pipe(gcloudFile.createWriteStream()) // link to gcloud storage api
.on('error', function(err) {
console.log('error')
})
.on('finish', function() {
console.log('upload complete')
});
} catch(err){
console.log(err)
}
}
I have an express server running with the following route:
exports.getUserFile = function (req, resp) {
let filePath = path.join(__dirname, 'storage', req.params.fileName);
resp.download(filePath);
});
}
In my web app i'm calling this route and trying to save the file locally using file-saver:
let req = request.get('/users/' + userId + '/files/' + file.name);
req.set('Authorization', 'Bearer ' + this.state.jsonWebToken);
req.end((err, resp) => {
let f = new File([resp.text], file.name, {type: resp.type});
fileSaver.saveAs(f);
});
If the file is plain text then it works ok, but for other file types like images i'm not able to open the file (it's 'corrupt').
This is what the response looks like:
Do I need to decode the data in some way first? What is the correct way to save the content of the file?
If you're using superagent to perform the requests, you can explicitly set the response type to "blob", which would prevent any attempts to decode the response data. The binary data will end up in resp.body:
req.responseType('blob').end((err, resp) => {
saveAs(resp.body, file.name);
});
I haven't used express for a long time ago and I'm typing from mobile, it's seems a encoding issue, so it's seems that you're a sending raw image, you will need to encode it in base64 try something like:
//Here your saved file needs to be encoded to base 64.
var img = new Buffer(data, 'base64');
res.writeHead(200, {
'Content-Type': 'image/png',
'Content-Length': img.length
});
res.end(img);
Where data is your saved image, If you can render the image you just add the headers for download or just chain method download.
If you want to download the image as attachment in the page you can use res
exports.getUserFile = function (req, resp) {
let filePath = path.join(__dirname, 'storage', req.params.fileName);
var check = fs.readFileSync(__dirname+req.params.fileName);
resp.attachment(req.params.fileName); // The name of the file to be saved as. Eg Picture.jpg
res.resp(check) // Image buffer read from the path.
});
}
Reference:
http://expressjs.com/en/api.html#res.attachment
http://expressjs.com/en/api.html#res.end
Hope this helps.
In node.js how can I download specific amount of file, like only first 10 kilobyte of a file.
in my project, I need to extract mp3 duration and bitrate from remote file and the only way I think is download just a few bytes to the entire file.
As some guy said here If you download (at least) the first 32kB of the file you should be OK for most MP3 files.
In Node.js you know when a chunk of data is available. If you do a http request you are given a response object which contains the headers. This response object is also a stream with events. The one you are searching for is "data". With this event you get a buffer filled with the data received (only the new data, the previously received chunks are not present). You just have to have a buffer and append to it each time you get a chunk of data. You can also get the length of the buffer. You can choose to stop downloading the data when you have enough data using the method destroy method. Here is an example :
var http = require("http");
var buff = new Buffer(0);
http.get("http://epfl.ch", function(res) {
res.on('data', function(chunk) {
buff = Buffer.concat([buff, chunk]);
if (buff.length > 10240) {
res.destroy();
console.log(buff);
}
});
})
this code will wait to fetch 10kb bytes then end the request. Then you can dou whatever you want with the data (buff)
If you want to save the data to file while downloading you can do this instead :
var http = require("http");
var buff = new Buffer(0);
var fs = require("fs");
var file = fs.createWriteStream("file.mp3");
http.get("http://epfl.ch", function(res) {
res.pipe(file);
res.on('data', function(chunk) {
buff = Buffer.concat([buff, chunk]);
if (buff.length > 10240) {
res.destroy();
file.close();
console.log(buff);
}
});
})
This code will create an input stream for a file and pipe the body of the request to this file (ie. each chunk of data received will be append to the file).
If you don't want to do anything with your buffer you don't need to keep it you can just count the number of bytes received and stop when you need.
var http = require("http");
var bytesRecieved = 0;
var fs = require("fs");
var file = fs.createWriteStream("file.mp3");
http.get("http://epfl.ch", function(res) {
res.pipe(file);
res.on('data', function(chunk) {
bytesRecieved += chunk.length;
if (bytesRecieved > 10240) {
res.destroy();
file.close();
}
});
})
Best regards,
So I'm trying to write a basic file server in Node.js, and all the images I've tried uploading and storing on it are coming back as corrupted. The problem seems to have something to do with the way that Node Buffers handle being converted to UTF-8 and back again (which I have to do in order to get the POST body headers out and away from the binary data).
Here's a simple Node server that shows my current approach and the problems I've been having:
var http = require('http');
var server = http.createServer(function(request, response) {
if (request.method === "GET") {
// on GET request, output a simple web page with a file upload form
var mypage = '<!doctype html><html><head><meta charset="utf-8">' +
'<title>Submit POST Form</title></head>\r\n<body>' +
'<form action="http://127.0.0.1:8008" method="POST" ' +
'enctype="multipart/form-data"> <input name="upload" ' +
'type="file"><p><button type="submit">Submit</button>' +
'</p></form></body></html>\r\n';
response.writeHead(200, {
"Content-Type": "text/html",
"Content-Length": mypage.length
});
response.end(mypage);
} else if (request.method === "POST") {
// if we have a return post request, let's capture it
var upload = new Buffer([]);
// get the data
request.on('data', function(chunk) {
// copy post data
upload = Buffer.concat([upload, chunk]);
});
// when we have all the data
request.on('end', function() {
// convert to UTF8 so we can pull out the post headers
var str = upload.toString('utf8');
// get post headers with a regular expression
var re = /(\S+)\r\nContent-Disposition:\s*form-data;\s*name="\w+";\s*filename="[^"]*"\r\nContent-Type: (\S+)\r\n\r\n/i,
reMatch = str.match(re);
var lengthOfHeaders = reMatch[0].length,
boundary = reMatch[1],
mimeType = reMatch[2];
// slice headers off top of post body
str = str.slice(lengthOfHeaders);
// remove the end boundary
str = str.replace("\r\n" + boundary + "--\r\n", '');
// convert back to buffer
var rawdata = new Buffer(str, 'utf8');
// echo back to client
response.writeHead(200, {
"Content-Type": mimeType
});
response.end(rawdata);
});
}
});
server.listen(8008);
console.log("server running on port 8008");
To test it, run the script in node and go to 127.0.0.1:8008 in your browser. Try uploading an image and submitting the form. The image comes back as corrupt every time -- even though the script should just be directly echoing the image data back to the browser.
So does anyone know what I'm doing wrong here? Is there a better way to handle POST body headers in Node that I haven't figured out yet? (And before anyone says anything, no, I don't want to use Express. I want to figure out and understand this problem.)
The problem seems to have something to do with the way that Node Buffers handle being converted to UTF-8 and back again
I guess you are right about that, convert to UTF-8 is a bad idea, but can do it just to work with the file and get the headers and boundaries positions, but keep the buffer file untouched, and when you have all the positions to get the header and boundary out of the file just copy the buffer to a new buffer like that
originalBuffer.copy(newBuffer,0, positionHeader, positionEndBoundary)
var http = require('http');
var fs = require('fs');
var connections = 0;
var server = http.createServer(function (req, res) {
connections++;
console.log(req.url,"connections: "+connections);
if(req.url == '/'){
res.writeHead(200, { 'content-type': 'text/html' });
res.end(
'<form action="/upload" enctype="multipart/form-data" method="post">' +
'<input type="file" name="upload" multiple="multiple"><br>' +
'<input type="submit" value="Upload">' +
'</form>'
);
}
var body = new Buffer([]);
if (req.url == '/upload') {
req.on('data', function (foo) {
//f.write(foo);
body = Buffer.concat([body,foo]);
if(isImage(body.toString())){
console.log("é imagem do tipo "+isImage(body.toString()));
}
else{
console.log("Não é imagem");
res.end("Não é imagem");
}
console.log(body.length, body.toString().length);
});
req.on('end', function () {
// console.log(req.headers);
//I converted the buffer to "utf 8" but i kept the original buffer
var str = body.toString();
console.log(str.length);
imageType = isImage(body.toString());
//get the index of the last header character
//I'm just using the string to find the postions to cut the headers and boundaries
var index = str.indexOf(imageType)+(imageType+"\r\n\r\n").length;
// var headers= str.slice(0,index).split(';');
// console.log(headers);
//Here comes the trick
/*
*I have to cut the last boundaries, so i use the lastIndexOf to cut the second boundary
* And maybe that is the corruption issues, because, I'm not sure, but I guess
* the UTF-8 format only use 7bits to represent all characters, and the buffer can use 8bits, or two hex,
*So, i need to take the difference here (body.length-str.length)
*/
var indexBoundayToBuffer = str.lastIndexOf('------WebKitFormBoundary')+(body.length-str.length);
console.log(index, indexBoundayToBuffer);
//maybe you can change this to use less memory, whatever
var newBuffer = Buffer.alloc(body.length);
/*
*And now use the index, and the indexBoudayToBuffer and you will have only the binary
*/
body.copy(newBuffer,0,index,indexBoundayToBuffer);
// f.end();
//file type
var type = imageType.substr("image/".length);
console.log("END");
fs.writeFile("nameFile."+type,newBuffer,function(err,ok){
if(err){
console.log(err);
return false;
}
res.end();
});
});
}
});
function isImage(str){
if(str.indexOf('image/png')!=-1) return 'image/png';
else if(str.indexOf('image/jpeg')!=-1) return 'image/jpeg';
else if(str.indexOf('image/bmp'!=-1)) return 'image/bmp';
else if(str.indexOf('image/gif'!=-1)) return 'image/gif';
else false;
}
var port = process.env.PORT || 8080;
server.listen(port, function () {
console.log('Recording connections on port %s', port);
});
You really shouldn't use regular expressions like that to parse multipart payloads as it can easily make trying to parse your image data very unreliable. There are modules on npm that parse forms for you such as busboy, multiparty, or formidable. None of them use regular expressions and they don't require Express.
I have to create an image uploader for a future project (No flash, IE10+, FF7+ etc.) that does image resizing/converting/cropping on the clientside and not on the server.
So I made a javascript interface where the user can 'upload' their files and get resized/cropped in the browser directly, without ever contacting the server. The performance is OK, not that good, but it works.
The endresult is an array of canvas elements. The user can edit/crop the images after they got resized, so I keep them as canvas instead of converting them to jpeg. (Which would worsen the initial performance)
Now this works fine, but I don't know what's the best way to actually upload the finished canvas elements to the server now. (Using a asp.net 4 generic handler on the server)
I have tried creating a json object from all elements containing the dataurl of each canvas.
The problem is, when I got 10-40 pictures, the browser starts freezing when creating the dataurls, especially for images that are larger than 2 megabyte.
//images = array of UploadImage
for (var i = 0; i < images.length; i++) {
var data = document.getElementById('cv_' + i).toDataURL('image/jpg');
images[i].data = data.substr(data.indexOf('base64') + 7);
}
Also converting them to a json object (I am using json2.js) usually crashes my browser. (FF7)
My object
var UploadImage = function (pFileName, pName, pDescription) {
this.FileName = pFileName;
this.Name = pName;
this.Description = pDescription;
this.data = null;
}
The upload routine
//images = array of UploadImage
for (var i = 0; i < images.length; i++) {
var data = document.getElementById('cv_' + i).toDataURL('image/jpg');
images[i].data = data.substr(data.indexOf('base64') + 7);
}
var xhr, provider;
xhr = jQuery.ajaxSettings.xhr();
if (xhr.upload) {
xhr.upload.addEventListener('progress', function (e) {
console.log(Math.round((e.loaded * 100) / e.total) + '% done');
}, false);
}
provider = function () {
return xhr;
};
var ddd = JSON.stringify(images); //usually crash here
$.ajax({
type: 'POST',
url: 'upload.ashx',
xhr: provider,
dataType: 'json',
success: function (data) {
alert('ajax success: data = ' + data);
},
error: function () {
alert('ajax error');
},
data: ddd
});
What would be the best way to send the canvas elements to the server?
Should I send them all at once or one by one?
Uploading files one by one is better. Requires less memory and as soon as one file ready to upload, the upload can be started instead of waiting while all files will be prepared.
Use FormData to send files. Allows to upload files in binary format instead of base64 encoded.
var formData = new FormData;
If Firefox use canvas.mozGetAsFile('image.jpg') instead of canvas.toDataUrl(). Allow to avoid unnecessary conversion from base64 to binary.
var file = canvas.mozGetAsFile('image.jpg');
formData.append(file);
In Chrome use BlobBuilder to convert base64 into blob (see dataURItoBlob function
accepted
After playing around with a few things, I managed to figure this out myself.
First of all, this will convert a dataURI to a Blob:
//added for quick reference
function dataURItoBlob(dataURI) {
// convert base64/URLEncoded data component to raw binary data held in a string
var byteString;
if (dataURI.split(',')[0].indexOf('base64') >= 0)
byteString = atob(dataURI.split(',')[1]);
else
byteString = unescape(dataURI.split(',')[1]);
// separate out the mime component
var mimeString = dataURI.split(',')[0].split(':')[1].split(';')[0];
// write the bytes of the string to a typed array
var ia = new Uint8Array(byteString.length);
for (var i = 0; i < byteString.length; i++) {
ia[i] = byteString.charCodeAt(i);
}
return new Blob([ia], {type:mimeString});
}
From this question):
var blob = dataURItoBlob(canvas.toDataURL('image/jpg'));
formData.append(blob);
And then send the formData object. I'm not sure how to do it in jQuery, but with plain xhr object it like so:
var xhr = new XMLHttpRequest;
xhr.open('POST', 'upload.ashx', false);
xhr.send(formData);
On server you can get files from Files collection:
context.Request.Files[0].SaveAs(...);