I need help in uploading the gzip file into gcs. I am using nest js and #google-cloud/storage package.
#Post('/uploadFile')
async uploadFile(#Req() req: Request, #Res() res: Response) {
req.setEncoding('utf-8');
req.on('data', (chunk) => {
const storage = new Storage({ keyFilename: 'datastore.json' });
const bucket = storage.bucket('bucket_name');
let mimeType = req.headers.filetype;
let file = req.headers.filename;
const blob = bucket.file(file + '');
let strOut = "";
for (var i = 0; i < chunk.length; i++) {
strOut += chunk[i].charCodeAt(0).toString(2);
}
blob.save(strOut, {
metadata: {
contentType: mimeType,
contentEncoding: 'gzip'
},
resumable: false,
gzip:true
});
});
I am facing an issue while downloading the data file got downloaded but shows damaged/corrupted may be i am uploading the .gz file wrong. I am streaming the .gz file like this
fs.createReadStream("compressedData/" + names[i]).pipe(
request.post(
{
headers: config.headers,
url: "http://localhost:3000/uploadFile",
},
(err, res) => {
if (err) throw err;
console.log(res.body, config);
}
)
);
here is my config from client side
config = {
headers: {
filetype: "application/gzip",
filename: "sitemap/test-upload/" + names[i],
token: "sadjfbkjbkjvkjvkvkcvkjxsdnkn",
},
};
Related
I am running into an issue where when I want to upload an image to s3 bucket nothing goes through.
Basically the only message I get is
API resolved without sending a response for /api/upload/uploadPhoto, this may result in stalled requests.
In the front end, I have an input which can take multiple files ( mainly images ) and then those are stored in event.target.files.
I have a function that stores each file in a state array, and with the button submit it sends a post request to my next.js API.
Here's the logic on the front end:
This function handles the photos, so whenever I add a photo it will automatically add it to the listingPhotos state:
const handleListingPhotos = async (e: any) => {
setMessage(null);
let file = e.target.files;
console.log("hello", file);
for (let i = 0; i < file.length; i++) {
const fileType = file[i]["type"];
const validImageTypes = ["image/jpeg", "image/png"];
if (validImageTypes.includes(fileType)) {
setListingPhotos((prev: any) => {
return [...prev, file[i]];
});
} else {
setMessage("Only images are accepted");
}
}
};
Once the photos are stored in the state, I am able to see the data of the files in the browsers console.log. I run the onSubmit to call the POST API:
const handleSubmit = async (e: any) => {
e.preventDefault();
const formData = new FormData();
formData.append("files[]", listingPhotos);
await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
headers: { "Content-Type": "multipart/form-data" },
body: formData,
}).then((res) => res.json());
};
console.log("listingphotos:", listingPhotos);
Which then uses this logic to upload to the S3 Bucket, but the issue is that when I log req.body I am getting this type of information:
req.body ------WebKitFormBoundarydsKofVokaJRIbco1
Content-Disposition: form-data; name="files[]"
[object File][object File][object File][object File]
------WebKitFormBoundarydsKofVokaJRIbco1--
api/upload/UploadPhoto logic:
import { NextApiRequest, NextApiResponse } from "next";
const AWS = require("aws-sdk");
const access = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID as string,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY as string,
};
// creates an S3 Client
const s3 = new AWS.S3({ region: "region", credentials: access });
export default async function uploadPhoto(
req: NextApiRequest,
res: NextApiResponse
) {
// take info from parent page
// console.log("req.body: ", req.body);
if (req.method === "POST") {
console.log("req.body", req.body);
let body = req.body;
let headers = req.headers;
let contentType = headers["Content-Type"] || headers["content-type"];
// check for correct content-type
if (!contentType.startsWith("multipart/form-data")) {
return { statusCode: 400, body: "Invalid content type" };
}
let boundary = contentType.replace("multipart/form-data; boundary=", "");
let parts = body.split(boundary);
for (let part of parts) {
if (part.startsWith("Content-Disposition")) {
let [fileData] = part.split("\r\n\r\n");
fileData = fileData.slice(0, -2);
let [fileName] = part.split("filename=");
fileName = fileName.slice(1, -1);
let params = {
Bucket: "RANDOM BUCKET NAME",
Key: fileName,
Body: fileData,
ContentType: { "image/png": "image/jpg" },
};
// Need to set the PARAMS for the upload
await s3.putObject(params);
console.log(
"Successfully uploaded object: " + params.Bucket + "/" + params.Key
);
}
}
return {
statusCode: 200,
body: "File uploaded",
};
// Uploads the files to S3
}
}
I was able to find a way to read if the files were correctly displayed.
req.body {
fileName: 'b699417375e46286e5a30fc252b9b5eb.png',
fileType: 'image/png'
}
POST request code was changed to the followng:
const s3Promises = Array.from(listingPhotos).map(async (file) => {
const signedUrlRes = await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
body: JSON.stringify({
fileName: file.name,
fileType: file.type,
}),
headers: { "Content-Type": "application/json" },
});
Obviously, this is not the solution but it's part of it. The only problem I am running into right now is handling CORS in order to see if the files are sent to the bucket.
I am trying to post an array of multiple objects (2 strings, 1 date & 1 file) from Vue.3/Axios to A Nodejs server using Multer. However in the Nodejs Server when I console.log req.files and req.body they return
files - []
&
body - [Object: null prototype] { formData: '[object FormData]' }
How do I access my data from that object FormData.
Axios Post
async post(images) {
var formData = new FormData();
for (let i = 0; i < images.length; i++) {
let file = Array.from(images[i].file[0]);
formData.append("file", file);
formData.append("title", images[i].title);
formData.append("project", images[i].project);
formData.append("date", images[i].date);
}
var isValid = true;
if (isValid) {
console.log(
await axios
.post(
`http://localhost:3080/api/image/post`,
{
formData,
},
{
headers: {
"Content-Type": "multipart/form-data",
},
}
)
.then((res) => console.log(res))
.catch((err) => ("Error Occured", err))
);
}
}
Where images is a proxy containing an array with image objects holding:
date: "2022-06-10"
file: FileList {0: File, length: 1}
project: "Project Name"
title: "Test"
Multer is setup like so
const multer = require('multer')
var storage = multer.diskStorage({
destination: function (request, file, callback) {
callback(null, './uploads/');
},
filename: function (request, file, callback) {
callback(null, file.originalname)
}
});
var upload = multer({ storage: storage });
app.post('/api/image/post', upload.array('file'), function (req, res, next)){
console.log(req.body)
}
remove the object property shorthand, use just formData:
await axios
.post(
`http://localhost:3080/api/image/post`,
formData,
{
headers: {
"Content-Type": "multipart/form-data",
},
})
I want to upload a file to Google Drive but the file is uploaded without a filename. It upload as a 'Untitled' file. Please give me a solution if it works then I accept your answer Anyone here with a solution for this? Thanks in Advance. Here is my code.
userController.uploadToDrive = function(req, res){
token = req.body.token;
console.log(token);
var formData = new FormData();
console.log(token);
var fileMetadata = {
'name': 'all.vcf'
};
formData.append("data",fs.createReadStream('./all.vcf'), "all.vcf");
request({
headers: {
'Authorization': token,
'Content-Type' :'text/x-vcard',
},
resource: fileMetadata,
uri: 'https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart',
body: formData,
filename:'all.vcf',
method: 'POST'
}, function (err, resp, body) {
if(err){
console.log(err);
}else{
console.log('resp',body);
res.status(200).send()
fs.readdir('./contacts', function (err, files) {
var removefiles = function (file) {
fs.unlinkSync('./contacts/' + file)
}
files.forEach(function (file) {
removefiles(file)
})
})
}
});
}
It response like this:
resp {
"kind": "drive#file",
"id": "1tXu9Fc4sdi-yk8QGGvMJqSgxLXhuXNhQ",
"name": "Untitled",
"mimeType": "text/x-vcard"
}
I believe your goal and situation as follows.
You want to upload a file to Google Drive using multipart/form-data with Drive API.
Your access token can be used for uploading the file to Google Drive.
I think that in your case, the metadata and file content cannot be uploaded as multipart/form-data. By this, the file metadata cannot be reflected to the uploaded file. So in order to achieve this, I would like to propose the following modification.
Pattern 1:
In this pattern, const request = require("request") is used.
Modified script:
const fs = require("fs");
const request = require("request");
token = req.body.token;
fs.readFile("./all.vcf", function (err, content) {
if (err) {
console.error(err);
}
const metadata = {
name: "all.vcf",
mimeType: "text/x-vcard"
};
const boundary = "xxxxxxxxxx";
let data = "--" + boundary + "\r\n";
data += 'Content-Disposition: form-data; name="metadata"\r\n';
data += "Content-Type: application/json; charset=UTF-8\r\n\r\n";
data += JSON.stringify(metadata) + "\r\n";
data += "--" + boundary + "\r\n";
data += 'Content-Disposition: form-data; name="file"\r\n\r\n';
const payload = Buffer.concat([
Buffer.from(data, "utf8"),
Buffer.from(content, "binary"),
Buffer.from("\r\n--" + boundary + "--\r\n", "utf8"),
]);
request(
{
method: "POST",
url:
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers: {
Authorization: token,
"Content-Type": "multipart/form-data; boundary=" + boundary,
},
body: payload,
},
function (err, resp, body) {
if(err){
console.log(err);
}else{
console.log('resp',body);
res.status(200).send()
fs.readdir('./contacts', function (err, files) {
var removefiles = function (file) {
fs.unlinkSync('./contacts/' + file)
}
files.forEach(function (file) {
removefiles(file)
})
})
}
}
);
});
Pattern 2:
In this pattern, node fetch is used. In your script, new FormData() is used. So I thought that this pattern might be the direction you expect.
Modified script:
const FormData = require("form-data");
const fetch = require("node-fetch");
const fs = require("fs");
token = req.body.token;
var formData = new FormData();
var fileMetadata = {
name: "all.vcf",
mimeType: "text/x-vcard",
};
formData.append("metadata", JSON.stringify(fileMetadata), {
contentType: "application/json",
});
formData.append("data", fs.createReadStream("./all.vcf"), {
filename: "all.vcf",
});
fetch(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
{ method: "POST", body: formData, headers: { Authorization: token } }
)
.then((res) => res.json())
.then((json) => console.log(json));
Note:
In the case of uploadType=multipart, the maximum file size is 5 MB. Please be careful this. When you want to upload more large size, please use the resumable upload. Ref
References:
Upload file data
node-fetch
I'm using Multer to upload an image file into my node server, and it always comes back to giving me undefined for using ajax to send the image.
Ajax :
image = $("#input-file-img").val()
const data = new FormData();
data.append("image", image);
$.ajax({
url: '/uploadfile/' + userName,
method: 'POST',
async: false,
processData: false ,
contentType: false,
data: data
})
Upload.js
var storage = multer.diskStorage({
destination: function (req, file, cb) {
cb(null, 'uploads')
},
filename: function (req, file, cb) {
cb(null, file.originalname)
}
})
var upload = multer({ storage: storage })
router.post('/uploadfile/:userName', upload.single('image'), async (req, res, next) => {
let user = await User.findOne({ userName: req.params.userName })
let file = req.file
let fileName = file.filename
let path = variables.kepler + 'uploads/' + fileName
user.image = path
await user.save()
if (!path) {
const error = new Error('Please upload a file ...')
error.httpStatusCode = 400
return next(error)
}
if (path) {
return res.status(200).send({
status: '200',
message: 'Operation completed successfully ...',
data: user,
path
})
}
})
I checked the image value with console and it shows C:\fakepath\Capture d’écran de 2019-09-19 11-33-59.png'
Would appreciate any help.
I think your server side code is fine, if I modify the client side code as below, everything works nicely, we end up with images in the /uploads folder:
function base64toBlob(base64, mimeType) {
const bytes = atob(base64.split(',')[1]);
const arrayBuffer = new ArrayBuffer(bytes.length);
const uintArray = new Uint8Array(arrayBuffer);
for (let i = 0; i < bytes.length; i++) {
uintArray[i] = bytes.charCodeAt(i);
}
return new Blob([ arrayBuffer ], { type: mimeType });
}
function submitForm() {
const imgRegEx = /^data:(image\/(gif|png|jpg|jpeg))/;
const imageData = $('#input-file-img').attr('src');
const mimeType = imgRegEx.exec(imageData)[1];
const blob = base64toBlob(imageData, mimeType);
const fileExt = mimeType.replace("image/", "");
const fileName = "test-image." + fileExt; // Change as appropriate..
const data = new FormData();
data.append("image", blob, fileName);
$.ajax({
url: '/uploadfile/' + userName,
method: 'POST',
async: false,
processData: false ,
contentType: false,
data: data
})
}
Solved!!!
getting value by
image = $("#input-file-img").val()
that means I was sending a type String as a file
so I had to change it to
image = $('#input-file-img')[0].files[0]
and everything works really well
I have a Node JS Rest get service which send the zip folder created on my server in binary encoded format. But unable to download this on client. I am unable to open the zip.
Server controller
botsController.exportBot = (req,res) =>{
let botId = req.params.botId;
zipFolder("...pathtoZip/", "pathToFolder/my.zip", function(err) {
if(err) {
res.statusCode = 500;
return res.send({
success: false,
message: "Something went wrong while fetching the bot data",
err_details: err
});
} else {
let filetext;
let zipFolder= "pathToFolder/my.zip";
if (fs.existsSync(zipFolder)) {
filetext = fs.readFileSync(zipFolder, "utf-8");//tried encoding binary
}
var headers = {
'Content-Type': 'application/octet-stream',//tried application/zip
'Content-Disposition': "attachment; filename=" + botId + '.zip'
};
res.writeHead(200, headers);
return res.end(filetext,"binary");
}
});
And on angular js Service I have fetch the data. And on component I have downloaded it.But download zip is corrupted it gives error unable to open the zip.
Angular 2 Service
exportBot() {
let token = localStorage.token;
let headerObj = {
'Authorization': token,
responseType: ResponseContentType.ArrayBuffer
};
let headers = new Headers(headerObj);
return this.http.get("export/botId", { headers: headers })
.map((res) => new Blob([res['_body']], { type: 'application/zip' }))
.catch(this.errorHandler);
}
And on component end
exportBot() {
this.loader = false;
this.botdataservice.exportBot()
.subscribe(res => {
console.log(`excel data: ${res}`);
window.open(window.URL.createObjectURL(res));
},
err => {
console.log(err);
})
}
In your package.json add the dependency :
"file-saver": "^1.3.3",
And you can use the file saver on your get request as below :
public getBlob(url: string): Observable<Blob> {
this.showLoader();
return this.http.get(url, {responseType: 'blob'})
.catch(this.onCatch)
.do(res => {
this.onSuccess(res);
}, (error: any) => {
this.onError(error);
})
.finally(() => {
this.onEnd();
});
}
and the method to download :
downloadFile(fileid: string) {
return this._http.getBlob(this._http.backendURL.downloadFile.replace(':fileid', fileid))
}
And the you call the Filesaver when subscribing data as this :
downloadFile() {
this._downloadFileService.downloadFile(this.model.fileid)
.subscribe(
data => {
FileSaver.saveAs(data, this.form.getRawValue().title);
}
);
}
I hope this can help.
You can convert your file to base64 by using btoa(unescape(encodeURIComponent(binarydata))) or in case of array buffer try btoa(String.fromCharCode.apply(null, new Uint8Array(arrayBuffer)));
Either can even send base64 file from your node server
filetext = fs.readFileSync(zipFolder);
return res.end(new Buffer(filetext ).toString('base64'));
in that case change your headers
and use following code to download it
var base64Str = "UEsDBAoAAgAAAMWqM0z4bm0cBQAAAAUAAAAJAAAAdmlub2QudHh0dmlub2RQSwECFAAKAAIAAADFqjNM+G5tHAUAAAAFAAAACQAkAAAAAAABACAAAAAAAAAAdmlub2QudHh0CgAgAAAAAAABABgAUCMddj2R0wFA/Bx2PZHTAWCGE3Y9kdMBUEsFBgAAAAABAAEAWwAAACwAAAAAAA==";
var elem = window.document.createElement('a');
elem.href = "data:application/zip;base64,"+base64Str
elem.download = "my.zip";
document.body.appendChild(elem);
elem.click();
document.body.removeChild(elem);