We Keep Coding

File content missing when i download from s3

I am using node.js aws sdk for s3 related methods. I have a method to download the file from s3 bucket.
I am downloading the file using the below code.
const downloadFileBase64 = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
const response = await s3
.getObject(params, (err) => {
if (err) {
return err;
}
})
.promise();
return {
data: response.Body.toString('base64'),
fileName: payload.fileName
};
} catch (error) {
return Boom.badRequest(error.message);
}
};
Once i get the base64 content i am sending it over an email using sendgrid.
Issue: When i download small files everything is working fine. But when i download large files, some part of the file is missing in multiple pages. I just copy pasted the base64 in few online websites and downloaded the file from there, it's the same issue in those websites also. With this i concluded that there is some issue while returning the response from s3 itself. When i go to s3 and check it in the folder, it's showing proper file.
If you see the above screenshot, its the pdf which is having some random grey background in few pages and some text is also missing from the pdf.
I tried to use another method which just download buffer excluding the base64 conversion as shown below.
const downloadFileBuffer = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
const response = await s3
.getObject(params, (err) => {
if (err) {
return err;
}
})
.promise();
return {
data: response.Body,
fileName: payload.fileName
};
} catch (error) {
return Boom.badRequest(error.message);
}
};
And once i get the file content in this above response, i am storing temporarily in a folder on server and then reading again and sending over email. But i am still having the same issue.
const fileContent = await docs.downloadFileBuffer({ payload: req.payload.action.dire });
await fs.writeFileSync(`${temp}testinggg.pdf`, fileContent?.data);
const fileData = await fs.readFileSync(`${temp}testinggg.pdf`, { encoding: 'base64' });
Any help on this issue is really appreciated.

After days of research and trying different ways, I found the issue. The issue was with .promise() used in s3.getObject(params, (err) => {}).promise();. Instead of that, I used callback using Promise as shown below. Now the file is properly showing the full content without missing any data.
const downloadFileBuffer = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
return new Promise((resolve, reject) => {
s3.getObject(params, (err, response) => {
if (err) {
reject(err);
}
resolve({
data: response.Body,
fileName: payload.fileName
});
});
});
} catch (error) {
return Boom.badRequest(error.message);
}
};

How do I get uploaded image in next js and save it?

How do I get uploaded image in next.js API route and save it on public folder? I have front end ready. I'm uploading images to an endpoint using plain JavaScript.
here is the onSubmit function for uploading images. Suggest me if I'm doing it wrong here. The main question is how do I retrieve it?
const onSubmit=async(e)=>{
e.preventDefault();
const fd=new FormData()
fd.append('myfile',image.name)
let res=await fetch(`http://localhost:3000/api/upload`,{
method: 'POST',
headers: {
"Content-Type": "image/jpeg",
},
body: fd,
})
let response=await res.json();
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.

This is the endpoint code I used for uploading image in nextjs, it requires some additional packages I will list them bellow also.
next-connect
multer
uuid
import nextConnect from "next-connect";
import multer from "multer";
import { v4 as uuidv4 } from "uuid";
let filename = uuidv4() + "-" + new Date().getTime();
const upload = multer({
storage: multer.diskStorage({
destination: "./public/uploads/profiles", // destination folder
filename: (req, file, cb) => cb(null, getFileName(file)),
}),
});
const getFileName = (file) => {
filename +=
"." +
file.originalname.substring(
file.originalname.lastIndexOf(".") + 1,
file.originalname.length
);
return filename;
};
const apiRoute = nextConnect({
onError(error, req, res) {
res
.status(501)
.json({ error: `Sorry something Happened! ${error.message}` });
},
onNoMatch(req, res) {
res.status(405).json({ error: `Method '${req.method}' Not Allowed` });
},
});
apiRoute.use(upload.array("file")); // attribute name you are sending the file by
apiRoute.post((req, res) => {
res.status(200).json({ data: `/uploads/profiles/${filename}` }); // response
});
export default apiRoute;
export const config = {
api: {
bodyParser: false, // Disallow body parsing, consume as stream
},
};

no Need to use any packages to handle file uploading you can use base64 to convert file to string and return it back to file by using "fs" module
why This way is beterr then using formData ?
because you duleing with normal post request where you can send any kind of data with it and use body parsere .
converting
const toBase64 = (file: File) => new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => resolve(reader.result);
reader.onerror = error => reject(error);
}
send a post request to server
const base64: string = await toBase64(file) as string;
const fileData = { base64, fileName: file.name };
const result = await api.post("/foo", fileData, name: "Salih", massage: "Hello World"})
converting base64 to file in server
function base64ToFile(file: { base64: string, fileName: string }) {
const fileContents = file.base64.replace(/^data:image\/png;base64,/, "");
fs.mkdirSync("./public/uploads", { recursive: true });
const fileName = `./public/uploads/${Date.now().toString() + file.fileName}`
fs.writeFile(fileName, fileContents, 'base64', function (err) { console.log(err) });
}

I suggest the popular and lightweight formidable library:
# install
yarn add formidable#v3 #types/formidable
// pages/api/file-upload.ts
import fs from "fs";
import path from "path";
import { File } from "formidable";
// Important for NextJS!
export const config = {
api: {
bodyParser: false,
},
};
export default async function handler(
req: NextApiRequest,
res: NextApiResponse<string>
) {
try {
// Parse request with formidable
const { fields, files } = await parseFormAsync(req);
// Files are always arrays (formidable v3+)
const myfile = (files["myfile"] as any as File[])[0];
// Save file in the public folder
saveFile(myfile, "./public/uploads");
// Return success
res.status(200).json("success!");
} catch (e) {
return res.status(500).json(e);
}
}
function saveFile(file: File, publicFolder: string): void {
const fileExt = path.extname(file.originalFilename || "");
fs.renameSync(file.filepath, `${publicFolder}/${file.newFilename}${fileExt}`);
}
// ./helpers/formidable.ts
import type { NextApiRequest } from "next";
import formidable from "formidable";
export type FormidableParseReturn = {
fields: formidable.Fields;
files: formidable.Files;
};
export async function parseFormAsync(
req: NextApiRequest,
formidableOptions?: formidable.Options
): Promise<FormidableParseReturn> {
const form = formidable(formidableOptions);
return await new Promise<FormidableParseReturn>((resolve, reject) => {
form.parse(req, async (err, fields, files) => {
if (err) {
reject(err);
}
resolve({ fields, files });
});
});
}
Bonus question
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.
S3 and other cloud services
You can save on cloud services with Formidable.
See the official examples: https://github.com/node-formidable/formidable/blob/master/examples/store-files-on-s3.js
But you don't need to use cloud storage to protect private uploads. You can store them locally.
Working with private uploads locally
Saving:
Store the uploads in a non-public folder;
Ex. /private-uploads/{logged_user_id}/;
Reading:
Create an API page to fetch the file
Ex. https://.../uploads/{filename}
Fail if the file doesn't belong to the authenticated user;
Send the file as the response;
Security:
With the above folder scheme, hackers can use .. and similar on the filename to obtain unauthorized access;
Sanitize the filename having this in mind (ex. only allow alphanumeric characters);
Alternatively, use a database table to control ownership instead of a folder scheme;

AWS S3 file upload with Node.js: Unsupported body payload error

I am trying to get my node.js backend to upload a file to AWS S3, which it got in a post request from my front-end. This is what my function looks like:
async function uploadFile(file){
var uploadParams = {Bucket: '<bucket-name>', Key: file.name, Body: file};
s3.upload (uploadParams, function (err, data) {
if (err) {
console.log("Error", err);
} if (data) {
console.log("Upload Success", data.Location);
}
});
}
When I try uploading the file this way, I get an Unsupported Body Payload Error...
I used fileStream.createReadStream() in the past to upload files saves in a directory on the server, but creating a fileStream did not work for me, since there is no path parameter to pass here.
EDIT:
The file object is created in the angular frontend of my web application. This it the relevant html code where the file is uploaded by a user:
<div class="form-group">
<label for="file">Choose File</label>
<input type="file" id="file"(change)="handleFileInput($event.target.files)">
</div>
If the event occurs, the handleFileInput(files: FileList) method in the corresponding component is called:
handleFileInput(files: FileList) {
// should result in array in case multiple files are uploaded
this.fileToUpload = files.item(0);
// actually upload the file
this.uploadFileToActivity();
// used to check whether we really received the file
console.log(this.fileToUpload);
console.log(typeof this.fileToUpload)
}
uploadFileToActivity() {
this.fileUploadService.postFile(this.fileToUpload).subscribe(data => {
// do something, if upload success
}, error => {
console.log(error);
});
}
the postFile(fileToUpload: File) method of the file-upload service is used to make the post request:
postFile(fileToUpload: File): Observable<Boolean> {
console.log(fileToUpload.name);
const endpoint = '/api/fileupload/single';
const formData: FormData = new FormData();
formData.append('fileKey', fileToUpload, fileToUpload.name);
return this.httpClient
.post(endpoint, formData/*, { headers: yourHeadersConfig }*/)
.pipe(
map(() => { return true; }),
catchError((e) => this.handleError(e)),
);
}
Here is the the server-side code that receives the file and then calls the uploadFile(file) function:
app.post('/api/fileupload/single', async (req, res) => {
try {
if(!req.files) {
res.send({
status: false,
message: 'No file uploaded'
});
} else {
let file = req.files.fileKey;
uploadFile(file);
//send response
res.send({
status: true,
message: 'File is uploaded',
data: {
name: file.name,
mimetype: file.mimetype,
size: file.size
}
});
}
} catch (err) {
res.status(500).send(err);
}
});
Thank you very much for your help in solving this!
Best regards, Samuel

Best way is stream the file. Assuming you are. reading it from disk. You could do this
const fs = require("fs");
const aws = require("aws-sdk");
const s3Client = new aws.S3();
const Bucket = 'somebucket';
const stream = fs.createReadStream("file.pdf");
const Key = stream.path;
const response = await s3Client.upload({Bucket, Key, Body: stream}).promise();
console.log(response);

TypeError: res.status is not a function

I'm making a function that permits me to upload a picture to imgur in my express api (nodejs),
i'm encoutering an error when calling a function returning a promise:
TypeError: res.status is not a function
at uploadpicture.then
This is my code:
Where error is raised:
router.post('/upload', (req, res, next)=> {
var busboy = new Busboy({headers: req.headers});
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
if(fieldname == 'image') {
// the buffer
file.fileRead = [];
file.on('data', function(data) {
// add to the buffer as data comes in
this.fileRead.push(data);
});
file.on('end', function() {
// create a new stream with our buffered data
var finalBuffer = Buffer.concat(this.fileRead);
upload = uploadpicture(finalBuffer).then((res)=>{ //success request
console.log(res);
res.status(200).json({success: true, message: "Successfully uploaded !", url: res.data.link});
},(err)=>{ //error
res.status(500).json({success: false, message: "Error happenned while uploading !"});
}).catch((error)=>{
console.log(error);
res.status(500).json({success: false, message: "Error happenned while uploading !"});
});
})
}
});
busboy.on('finish', function() {
//busboy finished
});
req.pipe(busboy);
});
And the function :
function uploadpicture(stream){ //get picture stream
return new Promise((resolve, reject)=>{
var options = {
uri: 'https://api.imgur.com/3/image',
method: 'POST',
headers: {
//'Authorization': 'Client-ID ' + config.client_id_imgur // put client id here
},
formData: {
image: stream,
type: 'file'
},
auth: {
bearer: config.access_token_imgur,
}
};
request(options)
.then((parsedBody)=> {
resolve(parsedBody);
})
.catch((err)=> {
console.log(err);
reject(err.toString())
});
});
}
The code works perfectly, but i don't know why suddendly this error happened,
i tried to :
change arrow functions to function(){}
Add next to the route parameters
Nothing worked, Thanks for your help

The accepted answer directly addresses the OP's problem, but I post another solution since you can also encounter this error in other places.
When you have:
api.use((error: ErrorRequestHandler, request: ExpressRequest, response: ExpressResponse) => {
response.status(500).end() // response.status is not a function
})
Because the error handling route must accept 4 arguments for express to identify it as an error middleware.
api.use((error: ErrorRequestHandler, request: ExpressRequest, response: ExpressResponse, next: NextFunction) => {
response.status(500).end()
})
Just adding the next function (or whatever argument you're missing) will fix it.
https://github.com/visionmedia/supertest/issues/416#issuecomment-514508137

At this point:
upload = uploadpicture(finalBuffer).then((res)=>{ //success request
the resis the result of promise uploadpicture function (that is the parsedBody), not the res from the express route. So indeed, it has no status function. Try change the then callback name like:
upload = uploadpicture(finalBuffer).then((otherName)=>{ //success request

You are getting this error:
TypeError: res.status is not a function
Because the order should be (err, res, req, next) not (req, res, err, next),
example below
const errorHandler = (err, req, res, next) => {
const statusCode = res.statusCode === 200 ? 500 : res.statusCode;
res.status(statusCode)
res.json({
message : err.message,
stack :process.env.NODE_ENV === 'production' ? null : err.stack,
})
}

Order of parameters really matters i had error in below code
const getImagesByBrand = async (res) => {
try {
const images = await Image.find();
res.status(200).json(images);
} catch (error) {
res.status(500).json(error);
}
};
I was not giving req as parameter and that was the reason for error i just add req,res and it worked

If you are using the async/await method:
const notifications = await notifications.aggregate({...})
if(notifications){
return res.status(200).json({ data: notifications })
}else{
return res.status(404).json({ message: 'No notifications found'})
}
Make sure that you are including your return statements. Not including a return statement will cause this. Something else that I was doing is I had JSON instead of json, which will most definitely throw an error.

returning status once the file is written to local from s3 bucket

trying to fetch a file from s3 bucket and storing it on the local, once its written to the local reading the file from the local and converting the data to json format and sending it.
i need to check whether the file is downloaded and written to local, once the file exist only read and convert it to json else send an error message.
once the file is on open i am writing the file and making end. So after end i can't send a return value. So how i can solve this one and use try catch to send proper error message.
const fetchFileDownloadAndWriteIt = () => {
let Bucket = "DataBucket";
let filename = "sample_data.csv";
let s3 = new AWS.S3();
const params = {
Bucket: Bucket,
Key: filename
};
return s3.getObject(params)
.promise()
.then(data => {
const file = fs.createWriteStream('./localdata/' + filename);
file.on("open", () => {
file.write(data.Body);
file.end();
})
.on("error", err => {
console.log("Error Occured while writing", err.message)
})
})
.catch(err => {
console.log("unable to fetch file from s3 Bucket", err.message)
})
}
exports.fetchData = async (req,res) => {
let fileDownloadAndWrite = await fetchFileAndDownloadWriteIt();
// need to check file is downloaded and written properly
const path = "./localdata/sample_data.csv";
const json = await csv().fromFile(path);
res.send({data: json})
}

You can return a new Promise instead of the one instead of the one you get by calling the SDK's API.
return new Promise((res, rej) => {
s3.getObject(params)
.promise()
.then(data => {
const file = fs.createWriteStream('./localdata/' + filename);
file
.on("open", () => {
file.write(data.Body);
file.end();
//success
res();
})
.on("error", err => {
rej(err);
})
})
.catch(err => {
rej(err);
})
});
This will resolve to undefined and rejected with the proper error occured, like while writing file, etc.
How to Call it in your handler?
Something like this would be fine.
exports.fetchData = async (req, res, next) => {
try {
await fetchFileDownloadAndWriteIt();
// need to check file is downloaded and written properly - here the file is actually downloaded and written properly.
const path = "./localdata/sample_data.csv";
const json = await csv().fromFile(path);
res.send({ data: json })
}
catch (err) {
return next(err);
}
}