Uploaded gz file in Google Cloud is damaged - javascript

I need help in uploading the gzip file into gcs. I am using nest js and #google-cloud/storage package.
#Post('/uploadFile')
async uploadFile(#Req() req: Request, #Res() res: Response) {
req.setEncoding('utf-8');
req.on('data', (chunk) => {
const storage = new Storage({ keyFilename: 'datastore.json' });
const bucket = storage.bucket('bucket_name');
let mimeType = req.headers.filetype;
let file = req.headers.filename;
const blob = bucket.file(file + '');
let strOut = "";
for (var i = 0; i < chunk.length; i++) {
strOut += chunk[i].charCodeAt(0).toString(2);
}
blob.save(strOut, {
metadata: {
contentType: mimeType,
contentEncoding: 'gzip'
},
resumable: false,
gzip:true
});
});
I am facing an issue while downloading the data file got downloaded but shows damaged/corrupted may be i am uploading the .gz file wrong. I am streaming the .gz file like this
fs.createReadStream("compressedData/" + names[i]).pipe(
request.post(
{
headers: config.headers,
url: "http://localhost:3000/uploadFile",
},
(err, res) => {
if (err) throw err;
console.log(res.body, config);
}
)
);
here is my config from client side
config = {
headers: {
filetype: "application/gzip",
filename: "sitemap/test-upload/" + names[i],
token: "sadjfbkjbkjvkjvkvkcvkjxsdnkn",
},
};

Related

How to read files after sending a post request in the req.body

I am running into an issue where when I want to upload an image to s3 bucket nothing goes through.
Basically the only message I get is
API resolved without sending a response for /api/upload/uploadPhoto, this may result in stalled requests.
In the front end, I have an input which can take multiple files ( mainly images ) and then those are stored in event.target.files.
I have a function that stores each file in a state array, and with the button submit it sends a post request to my next.js API.
Here's the logic on the front end:
This function handles the photos, so whenever I add a photo it will automatically add it to the listingPhotos state:
const handleListingPhotos = async (e: any) => {
setMessage(null);
let file = e.target.files;
console.log("hello", file);
for (let i = 0; i < file.length; i++) {
const fileType = file[i]["type"];
const validImageTypes = ["image/jpeg", "image/png"];
if (validImageTypes.includes(fileType)) {
setListingPhotos((prev: any) => {
return [...prev, file[i]];
});
} else {
setMessage("Only images are accepted");
}
}
};
Once the photos are stored in the state, I am able to see the data of the files in the browsers console.log. I run the onSubmit to call the POST API:
const handleSubmit = async (e: any) => {
e.preventDefault();
const formData = new FormData();
formData.append("files[]", listingPhotos);
await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
headers: { "Content-Type": "multipart/form-data" },
body: formData,
}).then((res) => res.json());
};
console.log("listingphotos:", listingPhotos);
Which then uses this logic to upload to the S3 Bucket, but the issue is that when I log req.body I am getting this type of information:
req.body ------WebKitFormBoundarydsKofVokaJRIbco1
Content-Disposition: form-data; name="files[]"
[object File][object File][object File][object File]
------WebKitFormBoundarydsKofVokaJRIbco1--
api/upload/UploadPhoto logic:
import { NextApiRequest, NextApiResponse } from "next";
const AWS = require("aws-sdk");
const access = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID as string,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY as string,
};
// creates an S3 Client
const s3 = new AWS.S3({ region: "region", credentials: access });
export default async function uploadPhoto(
req: NextApiRequest,
res: NextApiResponse
) {
// take info from parent page
// console.log("req.body: ", req.body);
if (req.method === "POST") {
console.log("req.body", req.body);
let body = req.body;
let headers = req.headers;
let contentType = headers["Content-Type"] || headers["content-type"];
// check for correct content-type
if (!contentType.startsWith("multipart/form-data")) {
return { statusCode: 400, body: "Invalid content type" };
}
let boundary = contentType.replace("multipart/form-data; boundary=", "");
let parts = body.split(boundary);
for (let part of parts) {
if (part.startsWith("Content-Disposition")) {
let [fileData] = part.split("\r\n\r\n");
fileData = fileData.slice(0, -2);
let [fileName] = part.split("filename=");
fileName = fileName.slice(1, -1);
let params = {
Bucket: "RANDOM BUCKET NAME",
Key: fileName,
Body: fileData,
ContentType: { "image/png": "image/jpg" },
};
// Need to set the PARAMS for the upload
await s3.putObject(params);
console.log(
"Successfully uploaded object: " + params.Bucket + "/" + params.Key
);
}
}
return {
statusCode: 200,
body: "File uploaded",
};
// Uploads the files to S3
}
}
I was able to find a way to read if the files were correctly displayed.
req.body {
fileName: 'b699417375e46286e5a30fc252b9b5eb.png',
fileType: 'image/png'
}
POST request code was changed to the followng:
const s3Promises = Array.from(listingPhotos).map(async (file) => {
const signedUrlRes = await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
body: JSON.stringify({
fileName: file.name,
fileType: file.type,
}),
headers: { "Content-Type": "application/json" },
});
Obviously, this is not the solution but it's part of it. The only problem I am running into right now is handling CORS in order to see if the files are sent to the bucket.

NodeJs Access [object FormData]

I am trying to post an array of multiple objects (2 strings, 1 date & 1 file) from Vue.3/Axios to A Nodejs server using Multer. However in the Nodejs Server when I console.log req.files and req.body they return
files - []
&
body - [Object: null prototype] { formData: '[object FormData]' }
How do I access my data from that object FormData.
Axios Post
async post(images) {
var formData = new FormData();
for (let i = 0; i < images.length; i++) {
let file = Array.from(images[i].file[0]);
formData.append("file", file);
formData.append("title", images[i].title);
formData.append("project", images[i].project);
formData.append("date", images[i].date);
}
var isValid = true;
if (isValid) {
console.log(
await axios
.post(
`http://localhost:3080/api/image/post`,
{
formData,
},
{
headers: {
"Content-Type": "multipart/form-data",
},
}
)
.then((res) => console.log(res))
.catch((err) => ("Error Occured", err))
);
}
}
Where images is a proxy containing an array with image objects holding:
date: "2022-06-10"
file: FileList {0: File, length: 1}
project: "Project Name"
title: "Test"
Multer is setup like so
const multer = require('multer')
var storage = multer.diskStorage({
destination: function (request, file, callback) {
callback(null, './uploads/');
},
filename: function (request, file, callback) {
callback(null, file.originalname)
}
});
var upload = multer({ storage: storage });
app.post('/api/image/post', upload.array('file'), function (req, res, next)){
console.log(req.body)
}
remove the object property shorthand, use just formData:
await axios
.post(
`http://localhost:3080/api/image/post`,
formData,
{
headers: {
"Content-Type": "multipart/form-data",
},
})

Filename is 'Untitled' when uploading a file using Google Drive API Node js

I want to upload a file to Google Drive but the file is uploaded without a filename. It upload as a 'Untitled' file. Please give me a solution if it works then I accept your answer Anyone here with a solution for this? Thanks in Advance. Here is my code.
userController.uploadToDrive = function(req, res){
token = req.body.token;
console.log(token);
var formData = new FormData();
console.log(token);
var fileMetadata = {
'name': 'all.vcf'
};
formData.append("data",fs.createReadStream('./all.vcf'), "all.vcf");
request({
headers: {
'Authorization': token,
'Content-Type' :'text/x-vcard',
},
resource: fileMetadata,
uri: 'https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart',
body: formData,
filename:'all.vcf',
method: 'POST'
}, function (err, resp, body) {
if(err){
console.log(err);
}else{
console.log('resp',body);
res.status(200).send()
fs.readdir('./contacts', function (err, files) {
var removefiles = function (file) {
fs.unlinkSync('./contacts/' + file)
}
files.forEach(function (file) {
removefiles(file)
})
})
}
});
}
It response like this:
resp {
"kind": "drive#file",
"id": "1tXu9Fc4sdi-yk8QGGvMJqSgxLXhuXNhQ",
"name": "Untitled",
"mimeType": "text/x-vcard"
}
I believe your goal and situation as follows.
You want to upload a file to Google Drive using multipart/form-data with Drive API.
Your access token can be used for uploading the file to Google Drive.
I think that in your case, the metadata and file content cannot be uploaded as multipart/form-data. By this, the file metadata cannot be reflected to the uploaded file. So in order to achieve this, I would like to propose the following modification.
Pattern 1:
In this pattern, const request = require("request") is used.
Modified script:
const fs = require("fs");
const request = require("request");
token = req.body.token;
fs.readFile("./all.vcf", function (err, content) {
if (err) {
console.error(err);
}
const metadata = {
name: "all.vcf",
mimeType: "text/x-vcard"
};
const boundary = "xxxxxxxxxx";
let data = "--" + boundary + "\r\n";
data += 'Content-Disposition: form-data; name="metadata"\r\n';
data += "Content-Type: application/json; charset=UTF-8\r\n\r\n";
data += JSON.stringify(metadata) + "\r\n";
data += "--" + boundary + "\r\n";
data += 'Content-Disposition: form-data; name="file"\r\n\r\n';
const payload = Buffer.concat([
Buffer.from(data, "utf8"),
Buffer.from(content, "binary"),
Buffer.from("\r\n--" + boundary + "--\r\n", "utf8"),
]);
request(
{
method: "POST",
url:
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers: {
Authorization: token,
"Content-Type": "multipart/form-data; boundary=" + boundary,
},
body: payload,
},
function (err, resp, body) {
if(err){
console.log(err);
}else{
console.log('resp',body);
res.status(200).send()
fs.readdir('./contacts', function (err, files) {
var removefiles = function (file) {
fs.unlinkSync('./contacts/' + file)
}
files.forEach(function (file) {
removefiles(file)
})
})
}
}
);
});
Pattern 2:
In this pattern, node fetch is used. In your script, new FormData() is used. So I thought that this pattern might be the direction you expect.
Modified script:
const FormData = require("form-data");
const fetch = require("node-fetch");
const fs = require("fs");
token = req.body.token;
var formData = new FormData();
var fileMetadata = {
name: "all.vcf",
mimeType: "text/x-vcard",
};
formData.append("metadata", JSON.stringify(fileMetadata), {
contentType: "application/json",
});
formData.append("data", fs.createReadStream("./all.vcf"), {
filename: "all.vcf",
});
fetch(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
{ method: "POST", body: formData, headers: { Authorization: token } }
)
.then((res) => res.json())
.then((json) => console.log(json));
Note:
In the case of uploadType=multipart, the maximum file size is 5 MB. Please be careful this. When you want to upload more large size, please use the resumable upload. Ref
References:
Upload file data
node-fetch

Sending image via ajax to multer

I'm using Multer to upload an image file into my node server, and it always comes back to giving me undefined for using ajax to send the image.
Ajax :
image = $("#input-file-img").val()
const data = new FormData();
data.append("image", image);
$.ajax({
url: '/uploadfile/' + userName,
method: 'POST',
async: false,
processData: false ,
contentType: false,
data: data
})
Upload.js
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, 'uploads')
},
filename: function (req, file, cb) {
cb(null, file.originalname)
}
})
var upload = multer({ storage: storage })
router.post('/uploadfile/:userName', upload.single('image'), async (req, res, next) => {
let user = await User.findOne({ userName: req.params.userName })
let file = req.file
let fileName = file.filename
let path = variables.kepler + 'uploads/' + fileName
user.image = path
await user.save()
if (!path) {
const error = new Error('Please upload a file ...')
error.httpStatusCode = 400
return next(error)
}
if (path) {
return res.status(200).send({
status: '200',
message: 'Operation completed successfully ...',
data: user,
path
})
}
})
I checked the image value with console and it shows C:\fakepath\Capture d’écran de 2019-09-19 11-33-59.png'
Would appreciate any help.
I think your server side code is fine, if I modify the client side code as below, everything works nicely, we end up with images in the /uploads folder:
function base64toBlob(base64, mimeType) {
const bytes = atob(base64.split(',')[1]);
const arrayBuffer = new ArrayBuffer(bytes.length);
const uintArray = new Uint8Array(arrayBuffer);
for (let i = 0; i < bytes.length; i++) {
uintArray[i] = bytes.charCodeAt(i);
}
return new Blob([ arrayBuffer ], { type: mimeType });
}
function submitForm() {
const imgRegEx = /^data:(image\/(gif|png|jpg|jpeg))/;
const imageData = $('#input-file-img').attr('src');
const mimeType = imgRegEx.exec(imageData)[1];
const blob = base64toBlob(imageData, mimeType);
const fileExt = mimeType.replace("image/", "");
const fileName = "test-image." + fileExt; // Change as appropriate..
const data = new FormData();
data.append("image", blob, fileName);
$.ajax({
url: '/uploadfile/' + userName,
method: 'POST',
async: false,
processData: false ,
contentType: false,
data: data
})
}
Solved!!!
getting value by
image = $("#input-file-img").val()
that means I was sending a type String as a file
so I had to change it to
image = $('#input-file-img')[0].files[0]
and everything works really well

Angular 2 Download a Zip format file

I have a Node JS Rest get service which send the zip folder created on my server in binary encoded format. But unable to download this on client. I am unable to open the zip.
Server controller
botsController.exportBot = (req,res) =>{
let botId = req.params.botId;
zipFolder("...pathtoZip/", "pathToFolder/my.zip", function(err) {
if(err) {
res.statusCode = 500;
return res.send({
success: false,
message: "Something went wrong while fetching the bot data",
err_details: err
});
} else {
let filetext;
let zipFolder= "pathToFolder/my.zip";
if (fs.existsSync(zipFolder)) {
filetext = fs.readFileSync(zipFolder, "utf-8");//tried encoding binary
}
var headers = {
'Content-Type': 'application/octet-stream',//tried application/zip
'Content-Disposition': "attachment; filename=" + botId + '.zip'
};
res.writeHead(200, headers);
return res.end(filetext,"binary");
}
});
And on angular js Service I have fetch the data. And on component I have downloaded it.But download zip is corrupted it gives error unable to open the zip.
Angular 2 Service
exportBot() {
let token = localStorage.token;
let headerObj = {
'Authorization': token,
responseType: ResponseContentType.ArrayBuffer
};
let headers = new Headers(headerObj);
return this.http.get("export/botId", { headers: headers })
.map((res) => new Blob([res['_body']], { type: 'application/zip' }))
.catch(this.errorHandler);
}
And on component end
exportBot() {
this.loader = false;
this.botdataservice.exportBot()
.subscribe(res => {
console.log(`excel data: ${res}`);
window.open(window.URL.createObjectURL(res));
},
err => {
console.log(err);
})
}
In your package.json add the dependency :
"file-saver": "^1.3.3",
And you can use the file saver on your get request as below :
public getBlob(url: string): Observable<Blob> {
this.showLoader();
return this.http.get(url, {responseType: 'blob'})
.catch(this.onCatch)
.do(res => {
this.onSuccess(res);
}, (error: any) => {
this.onError(error);
})
.finally(() => {
this.onEnd();
});
}
and the method to download :
downloadFile(fileid: string) {
return this._http.getBlob(this._http.backendURL.downloadFile.replace(':fileid', fileid))
}
And the you call the Filesaver when subscribing data as this :
downloadFile() {
this._downloadFileService.downloadFile(this.model.fileid)
.subscribe(
data => {
FileSaver.saveAs(data, this.form.getRawValue().title);
}
);
}
I hope this can help.
You can convert your file to base64 by using btoa(unescape(encodeURIComponent(binarydata))) or in case of array buffer try btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
Either can even send base64 file from your node server
filetext = fs.readFileSync(zipFolder);
return res.end(new Buffer(filetext ).toString('base64'));
in that case change your headers
and use following code to download it
var base64Str = "UEsDBAoAAgAAAMWqM0z4bm0cBQAAAAUAAAAJAAAAdmlub2QudHh0dmlub2RQSwECFAAKAAIAAADFqjNM+G5tHAUAAAAFAAAACQAkAAAAAAABACAAAAAAAAAAdmlub2QudHh0CgAgAAAAAAABABgAUCMddj2R0wFA/Bx2PZHTAWCGE3Y9kdMBUEsFBgAAAAABAAEAWwAAACwAAAAAAA==";
var elem = window.document.createElement('a');
elem.href = "data:application/zip;base64,"+base64Str
elem.download = "my.zip";
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);

Categories