transfer files between two node.js servers over http - javascript

I have two node.js/express servers that communicate with each other with http. server A is also communicates with the browser and can handle file upload requests. When file is uploaded to server A i want to transfer it as is to server B for further processing. what is the best way to do so? preferably with request-promise module which is what i'm using for communication between the two servers.
This is what i got so far, but i am unable to transfer the file between the servers, the file is uploaded successfully to server A, and there is no Error while sending it to server B, but server B is doesn't recognise the request as file. what am i missing here?
Server A Routes:
'use strict';
// Routes
const express = require('express');
const router = express.Router();
const multer = require('multer');
const upload = multer();
const uploadController = require('./controllers/file/upload');
router.post('/upload',upload.single('file'),uploadController);
module.exports = router;
Server A uploadController:
'use strict';
const RP = require('request-promise');
module.exports = (req, res) => {
const body = req.body;
if(req.file) {
const file = req.file;
RP('http://serverB.net/process', {
method: 'POST',
formData: {file_buffer: file.buffer},
body: body
})
.then((response) => {
return res.status(200).send(response);
})
.catch((e) => {
return res.status(500).send(e.message);
})
}
else {
return res.status(500).send('unable to upload file');
}
};
Server B Routes:
'use strict';
// Routes
const express = require('express');
const router = express.Router();
const multer = require('multer');
const process = multer();
const processFile = require('./controllers/file/processFile');
router.post('/process', process.single('file'), processFile);
module.exports = router;
Server B processFile:
here i want to process the file from Server A but req.file is undefined
'use strict';
module.exports = (req, res) => {
if(req.file) {
// here i want to do something with the file.
}
};

router.post('/process', process.single('file'), processFile);
This line, specifically process.single('file') tells multer to look for a field file which will contain the actual file data. However, in your request you never specify a value for file. Change the name file_buffer to just file in your request. Or change your process.single() to process.single('file_buffer')
RP('http://serverB.net/process', {
method: 'POST',
formData: {
file: file.buffer,
body: body
}
})
.then((response) => {
return res.status(200).send(response);
})
.catch((e) => {
return res.status(500).send(e.message);
})
Now on Server B, inside processFile you should see a req.body with a field body that contains your entire body passed in your request and you should have a req.file now that contains your multer file object.

The encoding that you are using so send the file may not be multipart/form-data. Try setting that header in your file send request. Express req.files will only be populated with a multipart/form-data Express req.files will only be populated with a multipart/form-data . You're already using multer so I'm guessing the sending encoding is not quite right.
RP({
method: "POST",
uri: "serverB.net",
form: {
file: file.buffer
},
headers: {
"Content-Type": "multipart/form-data"
}
});

Related

Unexpected end of form error when using Multer

I'm trying to upload an image (jpg/jpeg/png) from the browser to NodeJS. I have read through several tutorials and many posts on forums but very few seem to have this specific issue.
I've made sure to match the name provided to multer (upload.single('upload')) with the formData key (formData.append('upload', selectedFile, selectedFile.name))
I tried using headers originally, but later read that I should exclude them.
I tried submitting through a <form action="/upload" method="post" enctype="multipart/form-data"> but still got the same error.
I have found this similar question with only one answer which isn't clear
Multer gives unexpetcted end of form error and this question Unexpected end of form at Multipart._final which has no answers.
Every other question seems to be about an 'Unexpected field' or 'Unexpected end of multipart data' error which - judging from the solutions - is irrelevant here.
Below is my code...
Browser:
<body>
<input type="file" id="file_uploader" name="upload" />
<button onclick="uploadImage()" class="btn-default">SUBMIT</button>
<!-- OTHER STUFF -->
</body>
<script>
let selectedFile;
let uploadData = new FormData();
const fileInput = document.getElementById('file_uploader');
fileInput.onchange = () => {
selectedFile = fileInput.files[0];
uploadData.append('upload', selectedFile, selectedFile.name);
}
function uploadImage(){
fetch('/upload', {
method: 'POST',
body: uploadData
})
.then((response) => {
console.log(response);
})
.catch((error) => {
console.error('Error: ', error);
});
}
</script>
NodeJS
let express = require('express');
const multer = require('multer');
//multer options
const upload = multer({
dest: './upload/',
limits: {
fileSize: 1000000,
}
})
const app = express();
app.post('/upload', upload.single('upload'), (req, res) => {
res.send();
}, (error, req, res, next) => {
console.log(error.message);
})
exports.app = functions.https.onRequest(app);
...And here is the error log, if it helps:
Error: Unexpected end of form
> at Multipart._final (C:\Users\p\Downloads\MyInvestmentHub\functions\node_modules\busboy\lib\types\multipart.js:588:17)
> at callFinal (node:internal/streams/writable:694:27)
> at prefinish (node:internal/streams/writable:723:7)
> at finishMaybe (node:internal/streams/writable:733:5)
> at Multipart.Writable.end (node:internal/streams/writable:631:5)
> at onend (node:internal/streams/readable:693:10)
> at processTicksAndRejections (node:internal/process/task_queues:78:11)
I haven't posted many questions as of yet, so I apologise if I'm missing something or the format is off. Let me know and I will make appropriate edits.
Thanks.
I also got the exact same error.
Before using multer I had installed express-fileupload. When I unistalled it using the command npm uninstall express-fileupload I could get rid of the error.
And if it is the same case with you don't forget to delete the commands you already added for express-fileupload module. (like requiring fileupload)
Hi there I ran into the same issue for me was the lack of a bodyParser middleware that allows our requests files to parsed into Buffers.
I was able to resolve the problem like so in express:
var bodyParser = require('body-parser')
bodyParser.json([options])
I had this problem using multer with next js api. What worked for me is, I exported an a config that sets bodyParser to false like so;
export const config = {
api: {
bodyParser: false
}
}
In my case, the cause was other middleware. Check for other middleware running before multer. For me, the issue was express-openapi-validator middleware. Once I removed that middleware, it worked as expected.
Using body-parser package worked for me:
const bodyParser = require('body-parser')
// ...
app.use(bodyParser()) // support encoded bodies
My upload single file route:
const multer = require('multer')
const express = require('express')
const router = express()
const path = require('path') // node built-in path package
// needs "app.use(bodyParser())" middleware to work
const storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, process.cwd() + '/public/')
},
filename: function (req, file, cb) {
// generate the public name, removing problematic characters
const originalName = encodeURIComponent(path.parse(file.originalname).name).replace(/[^a-zA-Z0-9]/g, '')
const timestamp = Date.now()
const extension = path.extname(file.originalname).toLowerCase()
cb(null, originalName + '_' + timestamp + extension)
}
})
const upload = multer({
storage: storage,
limits: { fileSize: 1 * 1024 * 1024 }, // 1 Mb
fileFilter: (req, file, callback) => {
const acceptableExtensions = ['png', 'jpg', 'jpeg', 'jpg']
if (!(acceptableExtensions.some(extension =>
path.extname(file.originalname).toLowerCase() === `.${extension}`)
)) {
return callback(new Error(`Extension not allowed, accepted extensions are ${acceptableExtensions.join(',')}`))
}
callback(null, true)
}
})
router.post('/api/upload/single', upload.single('file'), (req, res) => {
res.status(200).json(req.file)
})
module.exports = {
uploadRouter: router
}
I think this is may causes by the responsed end,so in your continuous Middleware,you can do upload file at last.
i do this resolve problems.
const upload = multer({
dest: "./uploads",
});
app.use(upload.any());
app.post(
"/upload",
(req, res, next) => {
res.end("文件上传成功");
},
upload.single("fileKey")
);
try using these it work
const express = require('express')
const app = express()
const path = require('path')
const multer = require('multer')
var filestorageEngine = multer.diskStorage({
destination: (req, file, cb) => {
cb(null,'./uploads')
},
filename:(req,file, cb) => {
cb(null,"[maues]-" + file.originalname)
}
})
var upload = multer({
storage:filestorageEngine
})
app.post('/file', upload.array('file', 3),(req, res) => {
console.log(req.file)
res.send("file uploaded successfully")
})
app.listen(5000, ()=> {
console.log("server running")
})
in my frontend or client-side removing the headers in my request. And make sure your inputs are as a formData.
For example:
let formData = new FormData();
formData.append("fileName", file);
const res = await fetch("/api/create-card", {
method: "POST",
body: formData,
})
This worked for me.
I think, the problem is in the express and body-parser module, I just eliminated it
app.use(bodyParser.text({ type: '/' }));
and it works!
Try downgrading Multer to 1.4.3. It worked for me.
See https://github.com/expressjs/multer/issues/1144

NodeJS API response fails after fs.writefile

I've created an API which I want to create a file, and after the file was written, request a log API and after its response, response relatively to the user.
I've simplified the code like this:
const express = require('express');
const router = express.Router();
const fetch = require("node-fetch");
const util = require('util');
const fs = require("fs-extra")
router.get('/works/', (req, res) => {
logData(res)
})
router.get('/fails/', (req, res) => {
let t = Date.now();
const writeFile = util.promisify(fs.writeFile)
writeFile(`./data/${t}.json`, 'test').then(function(){
logData(res)
})
})
function logData(res) {
return new Promise(resolve => {
fetch('https://webhook.site/44dad1a5-47f6-467b-9088-346e7222d7be')
.then(response => response.text())
.then(x => res.send('x'));
});
}
module.exports = router
The /works/ API works fine,
but the /fails/ API fails with Error: read ECONNRESET
OP clarified in the comments that he uses nodemon to run this code.
The problem is that nodemon watches .json files too and restarts the server. So the request that changes a JSON file fails with Error: read ECONNRESET.
To prevent nodemon from restarting server when you change .json files see this.
For example, you can add nodemon.json configuration file to ignore ./data directory (make sure to restart nodemon after this file is added):
{
"ignore": ["./data"]
}

Node / Express Post request using a function to send data

I am using a function that prevents the default submit of a form and want to use a second function that posts this so i can work / modify the data in the fist function before submitting.
the first function
const Test = document.querySelector('.test')
Test.addEventListener('submit', (e) => {
e.preventDefault()
const username = CreateUser.querySelector('.username').value
const password = CreateUser.querySelector('.password').value
post('/about', { username, password })
})
i found the following the function that submits the Post request. It works fine when the destination is another function without leaving the actual page.
function post (path, data) {
return window.fetch(path, {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify(data)
})
}
I use the following routing in my index.js
const express = require('express')
const bodyParser = require('body-parser')
const store = require('./store')
const app = express()
app.use(express.static(__dirname + '/public'))
app.use(bodyParser.json())
var path = require('path')
app.post('/createUser', (req, res) => {
store
.createUser({
username: req.body.username,
password: req.body.password
})
.then(() => res.sendStatus(200))
})
app.get('/about',(req, res) => {
res.sendFile(path.join(__dirname, './public', 'about.html'));
})
app.post('/about',(req, res) => {
res.sendFile(path.join(__dirname, './public', 'about.html'));
})
app.listen(7555, () => {
console.log('Server running on http://localhost:7555')
})
When i make a post to /createUser i works fine and i can insert the data to a mysql table using a function.
I now want to make a post to /about using a function and eventually pass the data.
Why does it not work? I dont get any error.
The about.html, index.html and the js file with my functions are all in the public folder.
thanks for helping
Your route for the post function
app.post('/about',(req, res) => {
res.sendFile(path.join(__dirname, './public', 'about.html'));
})
is just returning the about.html page from your public folder. So there wouldn't be an error, you should just be getting back that HTML after posting to that endpoint with how it is currently configured.
The problem is that you'll only be getting this back as the body of your fetch() request. If you're wanting to see the about.html page, you'll want to actually redirect to http://localhost:7555/about.html. If you want to see the result of your fetch() request, you should be able to see the payload in the Networks tab of your DevTools (or your browser of choice's equivalent).

Sending html2pdf generated pdf back to the server

I have to send a PDF generated on the client side using the html2pdf to a server. I have managed to convert the generated PDF to base64 and want to send it back using axios. Here is my client side code:
function myFunction(){
var element = document.getElementById('element-to-print');
html2pdf().from(element).outputPdf().then(function(pdf) {
//Convert to base 64
const newpdf=btoa(pdf);
console.log(newpdf)
var formData = new FormData();
formData.append("uploadedFile", newpdf);
axios.post('/upload',formData).then(res => { console.log(res) }).catch(error => {
console.log(error.response)
})
});
Here is my server side code:
app.post('/upload', fileUpload(), function(req, res) {
const sampleFile = req.files.uploadedFile;
// do something with file
res.send('File uploaded');
})
I think the problem is in client side as I get the base64 version of my converted pdf on my console but after that I get the error:
POST http://localhost:3000/upload 500 (Internal Server Error)
How do I solve this problem? Thanks.
For the FormData in express you should use a middleware such Multer
and you code will be like this :
var express = require('express')
var multer = require('multer')
var upload = multer({ dest: 'uploads/' })
var app = express()
app.post('/upload', upload.single('uploadedFile'), function (req, res, next) {
// req.file is the `avatar` file
// req.body will hold the text fields, if there were any
})

can not get response return from node api into angular

I wrote a file upload snippet with multer in nodejs. It is working and files have been successfully uploaded to the server and I got the response(tested using postman),But when I call that API from client side(angular),I'm NOT getting the returned response from node.but file was uploaded. I'm using promise to handle responses.
const express = require('express');
const router = express.Router();
const upload = require('../startup/config');
router.post('/', upload.single('invoice'), (req, res) => {
return res.status(201).send({ status: 'success'});
});
module.exports = router;
client side
async onUpload() {
const uploadData = new FormData();
uploadData.append('invoice', this.selectedInvoice, this.selectedInvoice.name);
const result = await this.orderService.uploadInvoice(uploadData);
console.log('result', result); // <= not execute this line
}
order service
uploadInvoice(formData) {
return this.http.post('/api/invoices', formData).toPromise();
}
I need to run the line mentioned here, Did i miss something here? Need your help.
Thanks!

Categories