We Keep Coding

How i can handle multipart form data without using any node modules

I am trying to submit a image file using a POST Request, to server, where in the body of the front end fetch request i added body as FormData like this
let formdata = new FormData(form)
async function finalFetch(formdata){
let postReq = await fetch('/api/fileupload', {method : 'POST', body : formdata})
let result = await postReq.json()
return result.url;
}
Now on submit of this form in the backend i am handling the data like this
req.on('data', (chunk)=>{
console.log(chunk);
})
req.on('end', ()=>{
// pseudo code, this will get replaced by something else
res.write(JSON.stringify({
msg : 'File Upload done',
url : '/232'
}));
res.end();
})
and the above implementation gives me a Buffer, i have no idea how to write file in the server form this Buffer
I know there are lots of modules out there to handle the forms like multer, express, formidable, but i don't want to use any of them i am trying to understand how these packages work. I am just using core node js.

There are tons of third party packages why not use them, one of the famous one is https://www.npmjs.com/package/multer, by default node dosnt give you an option to write files.

After the buffer is fully received you should use core node module 'fs' to write buf to file.
const fs = require('fs');
fs.writeFile('file.name', buffer);
see docs fs.writeFile

Download PDF resource from PHP to Javascript via Ajax

So I will explain the problem:
Steps:
1) client (browser javascript) sends an Ajax request to the server that hits a controller method called download.
2) the controller's method creates a PDF resource(without saving on the filesystem), and returns a response with the PDF binary stream back to the client.
3) the client receives the PDF binary stream and download it on the client's computer. Is that possible?
Code:
Things I have already tried -
Client-side:
<script>
(function($) {
var button; // some random DOM button
button.on('click', function(e) {
e.preventDefault();
$.ajax({
url: "/download/:userId"
method: "POST",
dataType: "json"
success: function(response) {
var reader = new FileReader;
var file = new Blob([response.pdf_stream], 'application/pdf');
// create a generic download link
var a = $('<a/>', {
href: file,
download: response.filename
});
// trigger click event on that generic link.
a.get(0).click();
}
});
}
})(jQuery);
</script>
On the server-side:
class Controller
{
public function download($userId)
{
// fetching the user from the database
$user = User::find($userId);
// creating a pdf file using barry pdfdom package
// this will actually parse an HTML view and give us the PDF blob.
$pdf = PDF::loadView('pdf.view')->output();
// using Laravel helper function
return response()->json([
'pdf_stream' => utf8_encode($pdf),
'filename' => 'blahblah.pdf"
]);
// Or if you will in native PHP, just in case you don't use laravel.
echo json_encode([
'pdf_stream' => utf8_encode($pdf),
'filename' => 'blahblah.pdf"
]);
}
}
Any Idea what am I doing wrong here? How could I download that PDF file without saving it to the system (security and space concerns).
Any help would be appreciated.
Eden

If you want download pdf on client side, just open this pdf in new window. Use GET request for that things, like in RESTfull application (e.g. download/user/:id or somehow like that).
Could be useful:
Download and open pdf file using Ajax

The main problem is the returned response from controller. Try this:
public function download($userId)
{
// fetching the user from the database
$user = User::find($userId);
// creating a pdf file using barry pdfdom package
// this will actually parse an HTML view and give us the PDF blob.
$pdf = PDF::loadView('pdf.view')->output();
return response($pdf, 200,
[
'Content-Type' => 'application/pdf',
'Content-Length' => strlen($pdf),
'Cache-Control' => 'private, max-age=0, must-revalidate',
'Pragma' => 'public'
]
);
About calling the route which executes download($userid) method:
You do not have to use Ajax. Easy way:
Click view PDF

Export a file to Node server and then upload to S3

I'm generating HTML webpage as PDF, and then exporting it locally. How can I save this file to my node server and upload to S3
Please find the attached psuedo code
const convertDataToPdf = (exportFlag,cb)=>{ //set to switch between export and save
const doc = new jsPDF();
//... adding metadata and styling the pdf
if(exportFlag) {
doc.save('sample.pdf') //export PDF locally
} else {
cb(doc.output()) //this converts the PDF to raw to send to server
}
}
Based on a this answer, I'm appending the raw PDF data to a new FormData object, and then an ajax call to post the raw data to my nodejs server
convertDataToPdf(false, pdfData => {
let formData = new FormData();
formData.append(`file-1`, pdfData)
$.ajax({
url: '/file-upload',
data: formData,
processData: false,
contentType: false,
type: 'POST',
}).then(data => {
console.log('PDF upload to s3 successful!', data)
}).catch(err => {
console.log('Error! PDF Upload to S3 failed', err)
})
});
});
Now, how can I parse the raw PDF data on the server and upload it?
As an alternative, is it possible to save my file locally and then upload the file to s3?

First question - you can use on Node server multer https://www.npmjs.com/package/multer . This way you don't have to decode pdf. You just handle request and pass file to S3 (via S3 node API). You can use mimetype to be sure someone is sending you pdf.
For sure if you've got application server such as Nginx, you can limit transfer file.
For example in Nginx client_max_body_size 10M;. It's more secure to check limit on server, because naughty users can always cheat your web validations. Multer also has size validation if you would like to return specific exception from your backend.

Multer-s3 Uploading empty files on mobile

I'm using angular and multer-s3 to upload files from an angular app to a node server. Everything works well on the desktop but for some reason when trying to upload the photo via my iPhone 7 the uploaded file is corrupt. I'm using the same image and running through the same flow on both devices but getting different results so I'm assuming its because of mobile?
Here's the alert I get when trying to open the S3 file on the mobile
The file “1519398514215-test.png” could not be opened because it is empty.
Here's my code
var aws = require('aws-sdk');
var path = require('path');
var path3 = path.join(__dirname, "../config/config-aws.json");
var multer = require('multer');
var multerS3 = require('multer-s3');
var request = require('request');
aws.config.loadFromPath(path3);
var s3 = new aws.S3();
var fileName = '';
var uploadM = multer({
storage: multerS3({
s3: s3,
bucket: 'XXXX',
acl: 'public-read',
metadata: function (req, file, cb) {
cb(null, {fieldName: file.fieldname + '.png'});
},
key: function (req, file, cb) {
fileName = Date.now().toString() + "-" + file.originalname + '.png' ;
cb(null, fileName)
}
})
});
router.post('/', uploadM.array('photos', 3), function(req,res) {
if (res.error) {
console.log(error.stack);
return res.status(400).json({
message: "Error",
error: res.error
});
}
const url = 'https://s3-us-west-2.amazonaws.com/XXXX/' + fileName;
return res.status(200).json({
fileName: url
});
});
And here's my client-side
sendImage() {
const formData: FormData = new FormData();
this.removeObjectFromCanvas('polygon');
if (!fabric.Canvas.supports('toDataURL')) {
alert('This browser doesn\'t provide means to serialize canvas to an image');
} else {
// window.open(this.canvas.toDataURL('png'));
const image = new Image();
image.src = this.canvas.toDataURL('png');
const blob = this.dataURItoBlob(image.src);
const file = new File([blob], 'test.png');
formData.append('photos', file, 'test');
this.postFile(formData);
}
}
postFile(file) {
this.fileService.post(file)
.subscribe(data => {
}, error => {
console.log(error);
});
}
UPDATE **********
So found out you can debug on mobile. It looks like the buffer I am sending has data in it. My first thought was the buffer was not sending.
**** Update
Still can't figure this out. I've done some research and its possible it has something to do with formData and append? But as you can see by the image above both seem to be fine. Will continue to research ...
***** UPDATE
Definitely uploading empty files. But its only on mobile?
Also, I checked the formData prior to sending to the node server, seems to have the correct data in it.
*** UPDATE
Ok, even weirder experience. It seems multer-s3 is uploading empty files but when I take the file on the server-side and return it to the client-side, then read that file and display it, the image is displayed perfectly. So the formData is not the issue, it's something with multer-s3 I'm assuming?
****UPDATE
I forgot to mention I am using fabricjs and getting the image from the canvas. I read in some places there may be an issue there but like I said above when I send the file to the server and send it back to the client, after reading the file it displays the image perfectly.
****Update
I tried adding contentType to the multer method and now I'm receiving a 503 service unavailable error when running on mobile only. For desktop it is fine.
aws.config.loadFromPath(path3);
var file1;
var s3 = new aws.S3();
var fileName = '';
var uploadM = multer({
storage: multerS3({
s3: s3,
bucket: 'rent-z',
acl: 'public-read',
contentType: function (req, file, cb) {
cb(null, 'image/png');
},
metadata: function (req, file, cb) {
console.log(file);
cb(null, {fieldName: file.fieldname});
},
key: function (req, file, cb) {
fileName = Date.now().toString() + "-" + file.originalname;
file1 = file;
cb(null, fileName)
}
})
}).array('photos', 1);
router.post('/', function(req,res) {
uploadM(req, res, function (err) {
if (err) {
console.log(err);
return res.status(400).json({
message: "Error uploading to multer",
error: err
});
}
console.log('worked');
if (res.error) {
console.log(error.stack);
return res.status(400).json({
message: "Error",
error: res.error
});
}
// fs.readFile(req.body, function (err, data) {
const url = 'https://s3-us-west-2.amazonaws.com/rent-z/' + fileName;
return res.status(200).json({
fileName: url
});
// });
})
});
I even tried running multer-s3's automatic find mime-type function and that did give the same result
**** Day 4
It's been 96 hours since I started debugging this problem. No progress has been made. Still trying to figure out why its working on desktop and not mobile. For anyone looking for a quick summary of behavior:
User uploads image on the desktop
User places image on canvas
Use scales image
User presses sendImage
This converts the image to dataUri then blob
This blob is added to a file which is appended to formData
This formData is sent to a nodejs server where multer-s3 middleware
uploads the file to s3 successfully
User tries on mobile
Fails at step 7. The file is uploaded but is empty.
Let me know if anyone has any ideas on how to continue.

I'll make this an "official" answer since this may work for your needs. Anytime I have an intricate issue like this, my first thought is often "I wonder if there is an API/SaaS/service out there that can abstract this for me." As you've found, file uploads are tricky, particularly when you start throwing in the myriad devices we have to deal with these days.
I won't mention any particular services, but googling "file upload saas" will generally get you the top industry players. For $25 - $50/month you can abstract file uploads to a very simple api call. Not only do you get time savings now, but (assuming you choose a solid provider) you get no more headaches regarding file uploads in the future. It's the SaaS's job to make sure file uploads work on a million different devices; it's the SaaS's job to make sure S3 integration works, even when S3's api changes; it's the SaaS's job to make sure the user sees a nice friendly message if their upload fails for some reason, etc. I get to spend my time building features for our app instead of worrying about whether or not file uploads work on the iPhone 47.
"But then I'm tied to a SaaS, and live at the whim of their prices and feature set" Ah, but you can minimize that problem. For many services we use, I like to make a wrapper/interface/whatever you'd like to call it. In the case of file uploads, I made an ES6 module: fileUploads.js
In this module, I have a method upload. What does this method do? It simply implements and abstracts the API of [fileupload SaaS X]. If in the future we want, or need, to change from SaaS X to SaaS Y, I only have to change one thing in our entire app: my fileUpload.js module.

Download Remote File With Angular Using $http.get() - OAuth Authentication

My users have private files that need to be downloaded by an authenticated users. My server first downloads a file from S3 using it's own S3 app_id/secret_token credentials. The downloaded file is then constructed and sent to the client using Rails' send_data method.
Ruby (on Rails):
# documents_controller.rb
def download
some_file = SomeFile.find(params[:id])
# download file from AWS S3 to server
data = open(some_file.document.url)
# construct and send downloaded file to client
send_data data.read, filename: some_file.document_identifier, disposition: 'inline', stream: 'true'
end
Originally, I wanted to do trigger the download directly from the HTML template.
HTML:
<!-- download-template.html -->
<a target="_self" ng-href="{{ document.download_url }}" download="{{document.file_name}}">Download</a>
Looks simple enough but the problem is that Angular's $http interceptor doesn't catch this type of external link click and therefore the appropriate headers are not appended for server-side authentication. The result is a 401 Unauthorized Error.
Instead, I need to trigger the download using ng-click and then performing an $http.get() request from the angular controller.
HTML:
<!-- download-template.html -->
<div ng-controller="DocumentCtrl">
<a ng-click="download(document)">Download</a>
</div>
Javascript:
// DocumentCtrl.js
module.controller( "DocumentCtrl",
[ "$http", "$scope", "FileSaver", "Blob",
function( $http, $scope, FileSaver, Blob ) {
$scope.download = function( document ) {
$http.get(document.download_url, {}, { responseType: "arraybuffer" } )
.success( function( data ) {
var blob = new Blob([data], { type: "application/vnd.openxmlformats-officedocument.wordprocessingml.document" });
FileSaver.saveAs(blob, document.file_name);
});
};
}]);
FileSaver is a simple library to save files using Blobs (on the client, obviously).
This gets me passed by my authentication problem but results in the file being saved/download to the client in an unreadable/unusable format.
Why is the file being downloaded in an unusable format?
Thanks in advance.

Angular's $http method needs to be configured to accept a binary data response.
Rails' send_data documentation:
Sends the given binary data to the browser. This method is similar to
render plain: data, but also allows you to specify whether the browser
should display the response as a file attachment (i.e. in a download
dialog) or as inline data. You may also set the content type, the
apparent file name, and other things.
Angular's $http documentation is very poor regarding $http's configuration of responseType. Essentially, $http needs to be told to expect a binary data response by setting responseType to "arraybuffer" (see below).
$scope.download = function( document ) {
console.log("download: ", document);
$http({
url: document.download_url,
method: "GET",
headers: {
"Content-type": "application/json"
},
responseType: "arraybuffer" // expect to handle binary data response
}).success( function( data, status, headers ) {
var type = headers('Content-Type');
var blob = new Blob([data], { type: type });
FileSaver.saveAs(blob, document.file_name);
});
};
Angular's $http documentation could be a little more descriptive than:
Usage
$http(config);
Arguments
config
responseType - {string} - see XMLHttpRequest.responseType.

Hi I have an example of how I download a file from my server with angular:
I call the file with GET request:
file download html(client side):
<a ng-href="/api/downloadFile/{{download.id}}" type="submit" class="btn btn-primary col-lg-12 btn-modal-costume" >download</a>
file download java(server side):
public static Result download(String id) {
String content = null;
for (controllers.file file : files) {
if (file.getId().equals(id)){
content = file.getContent();
}
}
return ok(new java.io.File("/temp/" + id+ "file" + content)).as("application/force-download");
}
If you like you can see the all code in my github project

I think you were on the right track with the javascript solution, but just had a typo. In the $http.get call you pass an empty object as the second parameter. This is where the options argument with {responseType: arraybuffer} should have gone. See docs for $http.get here:
https://docs.angularjs.org/api/ng/service/$http#get