I am using node.js aws sdk for s3 related methods. I have a method to download the file from s3 bucket.
I am downloading the file using the below code.
const downloadFileBase64 = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
const response = await s3
.getObject(params, (err) => {
if (err) {
return err;
}
})
.promise();
return {
data: response.Body.toString('base64'),
fileName: payload.fileName
};
} catch (error) {
return Boom.badRequest(error.message);
}
};
Once i get the base64 content i am sending it over an email using sendgrid.
Issue: When i download small files everything is working fine. But when i download large files, some part of the file is missing in multiple pages. I just copy pasted the base64 in few online websites and downloaded the file from there, it's the same issue in those websites also. With this i concluded that there is some issue while returning the response from s3 itself. When i go to s3 and check it in the folder, it's showing proper file.
If you see the above screenshot, its the pdf which is having some random grey background in few pages and some text is also missing from the pdf.
I tried to use another method which just download buffer excluding the base64 conversion as shown below.
const downloadFileBuffer = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
const response = await s3
.getObject(params, (err) => {
if (err) {
return err;
}
})
.promise();
return {
data: response.Body,
fileName: payload.fileName
};
} catch (error) {
return Boom.badRequest(error.message);
}
};
And once i get the file content in this above response, i am storing temporarily in a folder on server and then reading again and sending over email. But i am still having the same issue.
const fileContent = await docs.downloadFileBuffer({ payload: req.payload.action.dire });
await fs.writeFileSync(`${temp}testinggg.pdf`, fileContent?.data);
const fileData = await fs.readFileSync(`${temp}testinggg.pdf`, { encoding: 'base64' });
Any help on this issue is really appreciated.
After days of research and trying different ways, I found the issue. The issue was with .promise() used in s3.getObject(params, (err) => {}).promise();. Instead of that, I used callback using Promise as shown below. Now the file is properly showing the full content without missing any data.
const downloadFileBuffer = async (payload) => {
let params = { Bucket: s3BucketName, Key: `${payload.folderName}/${payload.fileName}` };
try {
return new Promise((resolve, reject) => {
s3.getObject(params, (err, response) => {
if (err) {
reject(err);
}
resolve({
data: response.Body,
fileName: payload.fileName
});
});
});
} catch (error) {
return Boom.badRequest(error.message);
}
};
Related
I am trying to get my node.js backend to upload a file to AWS S3, which it got in a post request from my front-end. This is what my function looks like:
async function uploadFile(file){
var uploadParams = {Bucket: '<bucket-name>', Key: file.name, Body: file};
s3.upload (uploadParams, function (err, data) {
if (err) {
console.log("Error", err);
} if (data) {
console.log("Upload Success", data.Location);
}
});
}
When I try uploading the file this way, I get an Unsupported Body Payload Error...
I used fileStream.createReadStream() in the past to upload files saves in a directory on the server, but creating a fileStream did not work for me, since there is no path parameter to pass here.
EDIT:
The file object is created in the angular frontend of my web application. This it the relevant html code where the file is uploaded by a user:
<div class="form-group">
<label for="file">Choose File</label>
<input type="file" id="file"(change)="handleFileInput($event.target.files)">
</div>
If the event occurs, the handleFileInput(files: FileList) method in the corresponding component is called:
handleFileInput(files: FileList) {
// should result in array in case multiple files are uploaded
this.fileToUpload = files.item(0);
// actually upload the file
this.uploadFileToActivity();
// used to check whether we really received the file
console.log(this.fileToUpload);
console.log(typeof this.fileToUpload)
}
uploadFileToActivity() {
this.fileUploadService.postFile(this.fileToUpload).subscribe(data => {
// do something, if upload success
}, error => {
console.log(error);
});
}
the postFile(fileToUpload: File) method of the file-upload service is used to make the post request:
postFile(fileToUpload: File): Observable<Boolean> {
console.log(fileToUpload.name);
const endpoint = '/api/fileupload/single';
const formData: FormData = new FormData();
formData.append('fileKey', fileToUpload, fileToUpload.name);
return this.httpClient
.post(endpoint, formData/*, { headers: yourHeadersConfig }*/)
.pipe(
map(() => { return true; }),
catchError((e) => this.handleError(e)),
);
}
Here is the the server-side code that receives the file and then calls the uploadFile(file) function:
app.post('/api/fileupload/single', async (req, res) => {
try {
if(!req.files) {
res.send({
status: false,
message: 'No file uploaded'
});
} else {
let file = req.files.fileKey;
uploadFile(file);
//send response
res.send({
status: true,
message: 'File is uploaded',
data: {
name: file.name,
mimetype: file.mimetype,
size: file.size
}
});
}
} catch (err) {
res.status(500).send(err);
}
});
Thank you very much for your help in solving this!
Best regards, Samuel
Best way is stream the file. Assuming you are. reading it from disk. You could do this
const fs = require("fs");
const aws = require("aws-sdk");
const s3Client = new aws.S3();
const Bucket = 'somebucket';
const stream = fs.createReadStream("file.pdf");
const Key = stream.path;
const response = await s3Client.upload({Bucket, Key, Body: stream}).promise();
console.log(response);
I am trying to upload a file from mobile to google bucket using ionic 4. Although a file can upload into the could. I am struggling to get the file properties out of file object.
Here is my method,
async selectAFile() {
const uploadFileDetails = {
name: '',
contentLength: '',
size: '',
type: '',
path: '',
};
this.fileChooser.open().then(uri => {
this.file.resolveLocalFilesystemUrl(uri).then(newUrl => {
let dirPath = newUrl.nativeURL;
const dirPathSegments = dirPath.split('/');
dirPathSegments.pop();
dirPath = dirPathSegments.join('/');
(<any>window).resolveLocalFileSystemURL(
newUrl.nativeURL,
function(fileEntry) {
uploadFileDetails.path = newUrl.nativeURL;
const file: any = getFileFromFileEntry(fileEntry);
//log 01
console.log({ file });
uploadFileDetails.size = file.size;
uploadFileDetails.name = `${newUrl.name
.split(':')
.pop()}.${file.type.split('/').pop()}`;
uploadFileDetails.type = file.type;
async function getFileFromFileEntry(fileEntry) {
try {
return await new Promise((resolve, reject) =>
fileEntry.file(resolve, reject)
);
} catch (err) {
console.log(err);
}
}
},
function(e) {
console.error(e);
}
);
});
});
// here uploadFileDetails is simller to what I declared at the top ;)
// I wan't this to be populated with file properties
// console.log(uploadFileDetails.name) --> //''
const uploadUrl = await this.getUploadUrl(uploadFileDetails);
const response: any = this.uploadFile(
uploadFileDetails,
uploadUrl
);
response
.then(function(success) {
console.log({ success });
this.presentToast('File uploaded successfully.');
this.loadFiles();
})
.catch(function(error) {
console.log({ error });
});
}
even though I can console.log the file in log 01. I am unable to get file properties like, size, name, type out of the resolveLocalFileSystemURL function. basically, I am unable to populate uploadFileDetails object. What am I doing wrong? Thank you in advance.
you actually need 4 Ionic Cordova plugins to upload a file after getting all the metadata of a file.
FileChooser
Opens the file picker on Android for the user to select a file, returns a file URI.
FilePath
This plugin allows you to resolve the native filesystem path for Android content URIs and is based on code in the aFileChooser library.
File
This plugin implements a File API allowing read/write access to files residing on the device.
File Trnafer
This plugin allows you to upload and download files.
getting the file's metadata.
file.resolveLocalFilesystemUrl with fileEntry.file give you all the metadata you need, except the file name. There is a property called name in the metadata but it always contains value content.
To get the human readable file name you need filePath. But remember you can't use returning file path to retrieve metadata. For that, you need the original url from fileChooser.
filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1) is used to get only file name from filePath.
You need nativeURL of the file in order to upload it. Using file path returning from filePath is not going to work.
getFileInfo(): Promise<any> {
return this.fileChooser.open().then(fileURI => {
return this.filePath.resolveNativePath(fileURI).then(filePathUrl => {
return this.file
.resolveLocalFilesystemUrl(fileURI)
.then((fileEntry: any) => {
return new Promise((resolve, reject) => {
fileEntry.file(
meta =>
resolve({
nativeURL: fileEntry.nativeURL,
fileNameFromPath: filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1),
...meta,
}),
error => reject(error)
);
});
});
});
});
}
select a file from the file system of the mobile.
async selectAFile() {
this.getFileInfo()
.then(async fileMeta => {
//get the upload
const uploadUrl = await this.getUploadUrl(fileMeta);
const response: Promise < any > = this.uploadFile(
fileMeta,
uploadUrl
);
response
.then(function(success) {
//upload success message
})
.catch(function(error) {
//upload error message
});
})
.catch(error => {
//something wrong with getting file infomation
});
}
uploading selected file.
This depends on your backend implementation. This is how to use File Transfer to upload a file.
uploadFile(fileMeta, uploadUrl) {
const options: FileUploadOptions = {
fileKey: 'file',
fileName: fileMeta.fileNameFromPath,
headers: {
'Content-Length': fileMeta.size,
'Content-Type': fileMeta.type,
},
httpMethod: 'PUT',
mimeType: fileMeta.type,
};
const fileTransfer: FileTransferObject = this.transfer.create();
return fileTransfer.upload(file.path, uploadUrl, options);
}
hope it helps. :)
I have a camera component which I clicks a picture. I store the clicked picture using expo's FileSystem in the local cacheDirectory. Looks some thing like this:
onPictureSaved = async photo => {
await FileSystem.moveAsync({
from: photo.uri,
to: `${FileSystem.cacheDirectory}test.jpg`
});}
My next step is to my next stop is converting the image in the local cacheDirectory into a blob and upload the image into S3 via the aws-sdk:
var params = {
Bucket: "my-bucket",
Key: 'test.jpg',
Body: blob
};
s3.upload(params, function(err, data) {
if (err) {
console.log(err);
} // an error occurred
else {
console.log(data);
} // successful response
}
However, any methods or modules I install in order to accomplish this tiny step in the process hasn't been working at all. I can't use RNFS, react-native-fetch-blob or any other modules that require linking thanks to the expo client. I don't want to detach expo just for one thing. Is there any other way to accomplish this?
Take a look at https://github.com/expo/image-upload-example/issues/3#issuecomment-387263080. The latest expo release supports blobs, so then you can do something like the following:
uploadToS3 = async (fileUri, s3Bucket, s3Key) => {
const response = await fetch(fileUri);
const blob = await response.blob();
return new Promise((resolve, reject) => {
const params = {
Bucket: s3Bucket,
Key: s3Key,
Body: blob,
};
s3.upload(params, function(err, data) {
if (err) {
console.log('Something went wrong');
console.log(err);
reject(err);
} else {
console.log('Successfully uploaded image');
resolve(data);
}
});
});
};
Hope this helps!
trying to fetch a file from s3 bucket and storing it on the local, once its written to the local reading the file from the local and converting the data to json format and sending it.
i need to check whether the file is downloaded and written to local, once the file exist only read and convert it to json else send an error message.
once the file is on open i am writing the file and making end. So after end i can't send a return value. So how i can solve this one and use try catch to send proper error message.
const fetchFileDownloadAndWriteIt = () => {
let Bucket = "DataBucket";
let filename = "sample_data.csv";
let s3 = new AWS.S3();
const params = {
Bucket: Bucket,
Key: filename
};
return s3.getObject(params)
.promise()
.then(data => {
const file = fs.createWriteStream('./localdata/' + filename);
file.on("open", () => {
file.write(data.Body);
file.end();
})
.on("error", err => {
console.log("Error Occured while writing", err.message)
})
})
.catch(err => {
console.log("unable to fetch file from s3 Bucket", err.message)
})
}
exports.fetchData = async (req,res) => {
let fileDownloadAndWrite = await fetchFileAndDownloadWriteIt();
// need to check file is downloaded and written properly
const path = "./localdata/sample_data.csv";
const json = await csv().fromFile(path);
res.send({data: json})
}
You can return a new Promise instead of the one instead of the one you get by calling the SDK's API.
return new Promise((res, rej) => {
s3.getObject(params)
.promise()
.then(data => {
const file = fs.createWriteStream('./localdata/' + filename);
file
.on("open", () => {
file.write(data.Body);
file.end();
//success
res();
})
.on("error", err => {
rej(err);
})
})
.catch(err => {
rej(err);
})
});
This will resolve to undefined and rejected with the proper error occured, like while writing file, etc.
How to Call it in your handler?
Something like this would be fine.
exports.fetchData = async (req, res, next) => {
try {
await fetchFileDownloadAndWriteIt();
// need to check file is downloaded and written properly - here the file is actually downloaded and written properly.
const path = "./localdata/sample_data.csv";
const json = await csv().fromFile(path);
res.send({ data: json })
}
catch (err) {
return next(err);
}
}
I wrote a Hapi.js route to receive an uploaded file and have called it successfully using Postman. Now I want to save the file.
How do I
get the file extension?
save the file to disk?
Here's my route:
{
method: 'POST',
path: this.config.apiPrefix + 'uploadprofilephoto',
config: { payload: { maxBytes: 10485760, /* 10 MB */ output: 'stream', parse: true } },
handler: (request: hapi.Request, reply: hapi.IReply) => {
const result = new Promise<string>( async (resolve, reject) => {
try {
this.profilePhotoRouteHelper.savePhotoAndUploadToAws(jwtData.userId, request.payload['image']);
resolve(responseHelper.getSuccessResponse<string>(null, newJwt));
}
catch (error) {
log.error(error);
resolve(responseHelper.getErrorResponse(ResponseErrorCode.unknownError));
}
});
reply(result);
}
and an idea of how to save:
fs.writeFile(filename, data, [encoding], () => { } );
but I'd rather use promises and await if possible.
Here's the uploaded file:
I found fs-promise, which works well.
const photoId = uuid.v4();
await fsp.writeFile(photoId + '__' + image.hapi.filename, image._data, 'utf8');
And here's how to get file extensions: Node.js get file extension