We Keep Coding

Multer-s3 Uploading empty files on mobile

I'm using angular and multer-s3 to upload files from an angular app to a node server. Everything works well on the desktop but for some reason when trying to upload the photo via my iPhone 7 the uploaded file is corrupt. I'm using the same image and running through the same flow on both devices but getting different results so I'm assuming its because of mobile?
Here's the alert I get when trying to open the S3 file on the mobile
The file “1519398514215-test.png” could not be opened because it is empty.
Here's my code
var aws = require('aws-sdk');
var path = require('path');
var path3 = path.join(__dirname, "../config/config-aws.json");
var multer = require('multer');
var multerS3 = require('multer-s3');
var request = require('request');
aws.config.loadFromPath(path3);
var s3 = new aws.S3();
var fileName = '';
var uploadM = multer({
storage: multerS3({
s3: s3,
bucket: 'XXXX',
acl: 'public-read',
metadata: function (req, file, cb) {
cb(null, {fieldName: file.fieldname + '.png'});
},
key: function (req, file, cb) {
fileName = Date.now().toString() + "-" + file.originalname + '.png' ;
cb(null, fileName)
}
})
});
router.post('/', uploadM.array('photos', 3), function(req,res) {
if (res.error) {
console.log(error.stack);
return res.status(400).json({
message: "Error",
error: res.error
});
}
const url = 'https://s3-us-west-2.amazonaws.com/XXXX/' + fileName;
return res.status(200).json({
fileName: url
});
});
And here's my client-side
sendImage() {
const formData: FormData = new FormData();
this.removeObjectFromCanvas('polygon');
if (!fabric.Canvas.supports('toDataURL')) {
alert('This browser doesn\'t provide means to serialize canvas to an image');
} else {
// window.open(this.canvas.toDataURL('png'));
const image = new Image();
image.src = this.canvas.toDataURL('png');
const blob = this.dataURItoBlob(image.src);
const file = new File([blob], 'test.png');
formData.append('photos', file, 'test');
this.postFile(formData);
}
}
postFile(file) {
this.fileService.post(file)
.subscribe(data => {
}, error => {
console.log(error);
});
}
UPDATE **********
So found out you can debug on mobile. It looks like the buffer I am sending has data in it. My first thought was the buffer was not sending.
**** Update
Still can't figure this out. I've done some research and its possible it has something to do with formData and append? But as you can see by the image above both seem to be fine. Will continue to research ...
***** UPDATE
Definitely uploading empty files. But its only on mobile?
Also, I checked the formData prior to sending to the node server, seems to have the correct data in it.
*** UPDATE
Ok, even weirder experience. It seems multer-s3 is uploading empty files but when I take the file on the server-side and return it to the client-side, then read that file and display it, the image is displayed perfectly. So the formData is not the issue, it's something with multer-s3 I'm assuming?
****UPDATE
I forgot to mention I am using fabricjs and getting the image from the canvas. I read in some places there may be an issue there but like I said above when I send the file to the server and send it back to the client, after reading the file it displays the image perfectly.
****Update
I tried adding contentType to the multer method and now I'm receiving a 503 service unavailable error when running on mobile only. For desktop it is fine.
aws.config.loadFromPath(path3);
var file1;
var s3 = new aws.S3();
var fileName = '';
var uploadM = multer({
storage: multerS3({
s3: s3,
bucket: 'rent-z',
acl: 'public-read',
contentType: function (req, file, cb) {
cb(null, 'image/png');
},
metadata: function (req, file, cb) {
console.log(file);
cb(null, {fieldName: file.fieldname});
},
key: function (req, file, cb) {
fileName = Date.now().toString() + "-" + file.originalname;
file1 = file;
cb(null, fileName)
}
})
}).array('photos', 1);
router.post('/', function(req,res) {
uploadM(req, res, function (err) {
if (err) {
console.log(err);
return res.status(400).json({
message: "Error uploading to multer",
error: err
});
}
console.log('worked');
if (res.error) {
console.log(error.stack);
return res.status(400).json({
message: "Error",
error: res.error
});
}
// fs.readFile(req.body, function (err, data) {
const url = 'https://s3-us-west-2.amazonaws.com/rent-z/' + fileName;
return res.status(200).json({
fileName: url
});
// });
})
});
I even tried running multer-s3's automatic find mime-type function and that did give the same result
**** Day 4
It's been 96 hours since I started debugging this problem. No progress has been made. Still trying to figure out why its working on desktop and not mobile. For anyone looking for a quick summary of behavior:
User uploads image on the desktop
User places image on canvas
Use scales image
User presses sendImage
This converts the image to dataUri then blob
This blob is added to a file which is appended to formData
This formData is sent to a nodejs server where multer-s3 middleware
uploads the file to s3 successfully
User tries on mobile
Fails at step 7. The file is uploaded but is empty.
Let me know if anyone has any ideas on how to continue.

I'll make this an "official" answer since this may work for your needs. Anytime I have an intricate issue like this, my first thought is often "I wonder if there is an API/SaaS/service out there that can abstract this for me." As you've found, file uploads are tricky, particularly when you start throwing in the myriad devices we have to deal with these days.
I won't mention any particular services, but googling "file upload saas" will generally get you the top industry players. For $25 - $50/month you can abstract file uploads to a very simple api call. Not only do you get time savings now, but (assuming you choose a solid provider) you get no more headaches regarding file uploads in the future. It's the SaaS's job to make sure file uploads work on a million different devices; it's the SaaS's job to make sure S3 integration works, even when S3's api changes; it's the SaaS's job to make sure the user sees a nice friendly message if their upload fails for some reason, etc. I get to spend my time building features for our app instead of worrying about whether or not file uploads work on the iPhone 47.
"But then I'm tied to a SaaS, and live at the whim of their prices and feature set" Ah, but you can minimize that problem. For many services we use, I like to make a wrapper/interface/whatever you'd like to call it. In the case of file uploads, I made an ES6 module: fileUploads.js
In this module, I have a method upload. What does this method do? It simply implements and abstracts the API of [fileupload SaaS X]. If in the future we want, or need, to change from SaaS X to SaaS Y, I only have to change one thing in our entire app: my fileUpload.js module.

express node server return a value to async client call

Sorry, I tend to be a bad writer when I have not fully woken up, let me revise.
I am using expressjs with passportjs (local strategy) to manage my server and using connect-busboy to manage file uploading. I do not think passport will play a role in this.
Here is the server code for managing file uploads:
app.post('/upload', isLoggedIn, (req, res) => {
if(req.busboy){
req.pipe(req.busboy);
req.busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
if(mimetype.match(/^image\//)){
var root = path.join(__dirname, "../public/images/");
if(fs.existsSync(path.join(root, filename))){
var name = getUnique(path.join(root, filename));
} else {
var name = filename;
}
var ws = fs.createWriteStream(path.join(root, name), { flags: "a" });
file.pipe(ws);
}
});
}
});
As for my client page, it is used to change a JSON object which will get re-uploaded to the server as a configuration tool. When I upload a new image asynchronously I need to get the filename to update this JSON object while working on it. For uploading from the clients end I am using dropzonejs, which did not require any configuration on my part to work.
So, in summary I upload a number of images via dropzone asynchronously, busboy and fs on my server save the file, and I would like to get the filename returned to my javascript to modify the existing JSON object.
Edit solution:
Thanks to Elliot Blackburn for pointing me in the right direction.
By calling:
ws.on('close', () => {
res.send({filename: name});
});
after file.pipe(ws); to send the response back to the client. On the client side modify dropzone to handle the response like so:
dropzone.on('success', (file, res) => {
console.log(res);
});

Just send it in the normal http response. It'll depend what library you're using but most will allow you to trigger a normal req, res, next express call. From that you can access the file object, and return anything you want.
Something like:
req.send({filename: name}); // name is the filename var set earlier in the code.
Once you've finished editing the file and such, you can get the name and put it into that returned object and your client will receive that as object as the response which you can act upon.

Use .json extension to force JSON output from an Express.js endpoint

Is it possible to set up endpoints in Express.js in such a way that the user can signal they'd like only JSON output by appending .json to the URL?
For example, if I have the endpoint example.com/data/:data_id, is there a function someFunction such that if I hit the endpoint example.json/data/:data_id with the following code I would be able to get JSON output out?
router.get('/data/:data_id', function(req, res, next) {
var extension = req.someFunction();
if (extension === 'json') {
res.json({...});
} else {
res.render('view', {...});
};
});
I know there is probably a way to do that with content headers, but I've seen it recommended that content type be specified by an extension so that APIs can be browsed by a browser.

I think you could just add a format parameter in the query url
router.get('/data/:data_id', function(req, res, next) {
if (req.query.format && req.query.format === 'json') {
res.json({...});
} else {
res.render('view', {...});
};
});
something like this !

Express js: How to download a file using POST request

When I use GET, everything works fine. However, I struggle to use POST to achieve the same effect. Here are the code I have tried:
1.
app.post("/download", function (req, res) {
res.download("./path");
});
2.
app.post("/download", function (req, res) {
res.attachment("./path");
res.send("ok");
});
3.
app.post("/download", function (req, res) {
res.sendFile("./path");
});
None of them work. What is the correct way to do this?
EDIT:
I submit a POST request through a HTML form to /download. ./path is a static file. When I use code in method 1, I can see the correct response header and response body in the developer tool. But the browser does not prompt a download.

This might not be exactly what you want, but I have been having the same trouble.
This is what I did in the end:
Client - See EDIT below for updated client code
$http.post('/download', /**your data**/ ).
success(function(data, status, headers, config) {
$window.open('/download'); //does the download
}).
error(function(data, status, headers, config) {
console.log('ERROR: could not download file');
});
Server
// Receive data from the client to write to a file
app.post("/download", function (req, res) {
// Do whatever with the data
// Write it to a file etc...
});
// Return the generated file for download
app.get("/download", function (req, res) {
// Resolve the file path etc...
res.download("./path");
});
Alternatively, have you just tried calling $window.open(/download); from the HTML? This was the main reason why my download did not start. It returned in the XHR and I could see the data, but also did not prompt a download.
*EDIT:
The client code was not accurate, after some more testing it turned out that I only needed to do the following on the client:
// NOTE: Ensure that the data to be downloaded has
// already been packaged/created and is available
$window.open('/download'); //does the download

Meteor.js - Serve image directly from MongoDB?

Using Meteor.js, how can I serve an arbitrary HTTP response, eg. an image or PDF?
Example 1 - I need to generate PDF reports, which I cannot store in public/ or on a third-party server. Or, the report may be generated live in response to a HTTP GET.
Example 2 - If I have a url like:
/images/myimage.png
I would like to detect that request on the server, read the image from MongoDB, and serve it with the correct headers, so it is available to use with img tags, ie.
<img src="/images/myimage.png">
I do not want to store the images in the /public/ directory, so that I can have more control over exactly what is served and how it is permissioned.
Edit I was also able to get a basic example working using Iron Router.
ImageController = RouteController.extend({
run: function() {
var f = fs.readFileSync("/path/to/image.png");
var res = this.response;
res.writeHead(200, { "content-type": "image/png" });
res.write(f);
res.end();
}
});
Router.map(function() {
Router.route("images", {
path: "/images/image.png",
where: "server",
controller: ImageController // Note - cannot use string here - Iron Router has a dependency on window
});
});

You may write the response code as in any node app, using the middleware:
WebApp.connectHandlers.stack.splice (0, 0, {
route: '/path/to/the/file',
handle: function(req, res, next) {
res.writeHead(200, {
'Content-Type': ...ITEM TYPE... ,
});
res.write( ...ITEM DATA... );
res.end();
},
});

You can use filepicker. In filepicker the upload images is save in the bucket(cloud) and returns the url of that image. You can save the url in your mongo database. and when you want to use that image just use <img src="{{saveurl}}" > .
For more help see the documentation https://developers.inkfilepicker.com/docs/web/