We Keep Coding

How to read files after sending a post request in the req.body

I am running into an issue where when I want to upload an image to s3 bucket nothing goes through.
Basically the only message I get is
API resolved without sending a response for /api/upload/uploadPhoto, this may result in stalled requests.
In the front end, I have an input which can take multiple files ( mainly images ) and then those are stored in event.target.files.
I have a function that stores each file in a state array, and with the button submit it sends a post request to my next.js API.
Here's the logic on the front end:
This function handles the photos, so whenever I add a photo it will automatically add it to the listingPhotos state:
const handleListingPhotos = async (e: any) => {
setMessage(null);
let file = e.target.files;
console.log("hello", file);
for (let i = 0; i < file.length; i++) {
const fileType = file[i]["type"];
const validImageTypes = ["image/jpeg", "image/png"];
if (validImageTypes.includes(fileType)) {
setListingPhotos((prev: any) => {
return [...prev, file[i]];
});
} else {
setMessage("Only images are accepted");
}
}
};
Once the photos are stored in the state, I am able to see the data of the files in the browsers console.log. I run the onSubmit to call the POST API:
const handleSubmit = async (e: any) => {
e.preventDefault();
const formData = new FormData();
formData.append("files[]", listingPhotos);
await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
headers: { "Content-Type": "multipart/form-data" },
body: formData,
}).then((res) => res.json());
};
console.log("listingphotos:", listingPhotos);
Which then uses this logic to upload to the S3 Bucket, but the issue is that when I log req.body I am getting this type of information:
req.body ------WebKitFormBoundarydsKofVokaJRIbco1
Content-Disposition: form-data; name="files[]"
[object File][object File][object File][object File]
------WebKitFormBoundarydsKofVokaJRIbco1--
api/upload/UploadPhoto logic:
import { NextApiRequest, NextApiResponse } from "next";
const AWS = require("aws-sdk");
const access = {
accessKeyId: process.env.AWS_ACCESS_KEY_ID as string,
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY as string,
};
// creates an S3 Client
const s3 = new AWS.S3({ region: "region", credentials: access });
export default async function uploadPhoto(
req: NextApiRequest,
res: NextApiResponse
) {
// take info from parent page
// console.log("req.body: ", req.body);
if (req.method === "POST") {
console.log("req.body", req.body);
let body = req.body;
let headers = req.headers;
let contentType = headers["Content-Type"] || headers["content-type"];
// check for correct content-type
if (!contentType.startsWith("multipart/form-data")) {
return { statusCode: 400, body: "Invalid content type" };
}
let boundary = contentType.replace("multipart/form-data; boundary=", "");
let parts = body.split(boundary);
for (let part of parts) {
if (part.startsWith("Content-Disposition")) {
let [fileData] = part.split("\r\n\r\n");
fileData = fileData.slice(0, -2);
let [fileName] = part.split("filename=");
fileName = fileName.slice(1, -1);
let params = {
Bucket: "RANDOM BUCKET NAME",
Key: fileName,
Body: fileData,
ContentType: { "image/png": "image/jpg" },
};
// Need to set the PARAMS for the upload
await s3.putObject(params);
console.log(
"Successfully uploaded object: " + params.Bucket + "/" + params.Key
);
}
}
return {
statusCode: 200,
body: "File uploaded",
};
// Uploads the files to S3
}
}

I was able to find a way to read if the files were correctly displayed.
req.body {
fileName: 'b699417375e46286e5a30fc252b9b5eb.png',
fileType: 'image/png'
}
POST request code was changed to the followng:
const s3Promises = Array.from(listingPhotos).map(async (file) => {
const signedUrlRes = await fetch(`/api/upload/uploadPhoto`, {
method: "POST",
body: JSON.stringify({
fileName: file.name,
fileType: file.type,
}),
headers: { "Content-Type": "application/json" },
});
Obviously, this is not the solution but it's part of it. The only problem I am running into right now is handling CORS in order to see if the files are sent to the bucket.

Uploaded gz file in Google Cloud is damaged

I need help in uploading the gzip file into gcs. I am using nest js and #google-cloud/storage package.
#Post('/uploadFile')
async uploadFile(#Req() req: Request, #Res() res: Response) {
req.setEncoding('utf-8');
req.on('data', (chunk) => {
const storage = new Storage({ keyFilename: 'datastore.json' });
const bucket = storage.bucket('bucket_name');
let mimeType = req.headers.filetype;
let file = req.headers.filename;
const blob = bucket.file(file + '');
let strOut = "";
for (var i = 0; i < chunk.length; i++) {
strOut += chunk[i].charCodeAt(0).toString(2);
}
blob.save(strOut, {
metadata: {
contentType: mimeType,
contentEncoding: 'gzip'
},
resumable: false,
gzip:true
});
});
I am facing an issue while downloading the data file got downloaded but shows damaged/corrupted may be i am uploading the .gz file wrong. I am streaming the .gz file like this
fs.createReadStream("compressedData/" + names[i]).pipe(
request.post(
{
headers: config.headers,
url: "http://localhost:3000/uploadFile",
},
(err, res) => {
if (err) throw err;
console.log(res.body, config);
}
)
);
here is my config from client side
config = {
headers: {
filetype: "application/gzip",
filename: "sitemap/test-upload/" + names[i],
token: "sadjfbkjbkjvkjvkvkcvkjxsdnkn",
},
};

sending blob from expo to express server

I am getting the blob of an image from a URI to send it to my server to store it. I fetch the image using a get request and everything is correct, however, the post request has something wrong. The blob which is received by the server always have the size 0. I am sure that i get the blob of the image correct since i could display it in the frontend after converting the blob to base64.
Here is the code.
const url = "https://lh3.googleusercontent.com/a-/AOh14Gj1THuWiRu7Vpn85YETJN-aMui7NE8bpnNWOzdi"
var x, base64data;
const scope = this
var formData = new FormData();
var request = new XMLHttpRequest();
request.responseType = "blob";
request.onload = function () {
const blob = request.response
formData.append("file", blob.data);
console.log(blob.data)
//Converting the image to base64 so i could display it,
//and make sure that the blob received is not corrupted
const fileReaderInstance = new FileReader();
fileReaderInstance.readAsDataURL(blob);
fileReaderInstance.onload = () => {
base64data = fileReaderInstance.result;
scope.setState({base64data: base64data})
}
//Sending the blob to the server
x = new XMLHttpRequest();
x.open("POST",`${link}images/upload/${data.id}`,true);
x.setRequestHeader("Content-type", "image/jpeg");
x.setRequestHeader("Content-Length", formData.length);
x.send(formData);
}
request.open("GET", url);
request.send();
The console.log(blob.data) shows
Object {
"blobId": "69B8ACFE-5E31-4F16-84D8-002B17399F7E",
"name": "unnamed.jpg",
"offset": 0,
"size": 74054,
"type": "image/jpeg",
}
Ther Server-side code.
router.post('/upload/:id', upload.single('file'), async (req, res) => {
console.log(req.file)
const imgId = req.file.id
const id = req.params.id;
if (!id) return res.send({ error: "Missing playerID" });
if (!idValidator.isMongoId(id))
return res.send({ error: "Invalid Id" })
const player = await Player.findById(id)
if (!player)
return res.send({ error: "Player does not exists" })
const oldImg = player.photo;
if(oldImg && oldImg.toString() !== "5d3ae42c3f2b2c379c361652"){
await Images.findByIdAndDelete(oldImg)
}
player.photo = imgId;
const updatedPlayer = await Player.findByIdAndUpdate(id, player,{new:true})
return res.send({player: updatedPlayer,msg:"Photo Updated!"})
});
The console.log(req.file) shows the following.
{ fieldname: 'file',
originalname: 'unnamed.jpg',
encoding: '7bit',
mimetype: 'image/jpeg',
id: 5e780359fbea1452b8409a3e,
filename: '9cdeec500a17eccbb302d8571058da59.jpg',
metadata: null,
bucketName: 'images',
chunkSize: 261120,
size: 0,
md5: 'd41d8cd98f00b204e9800998ecf8427e',
uploadDate: 2020-03-23T00:31:21.640Z,
contentType: 'image/jpeg' }

After a long search i found this and it worked finalllllyyy!!!!
postPicture(data) {
const apiUrl = `${link}images/upload/${data.id}`;
const uri = "https://lh3.googleusercontent.com/a-/AOh14Gj1THuWiRu7Vpn85YETJN-aMui7NE8bpnNWOzdi";
const fileType = "jpeg";
const formData = new FormData();
formData.append('file', {
uri,
name: `photo.${fileType}`,
type: `image/${fileType}`,
});
const options = {
method: 'POST',
body: formData,
headers: {
Accept: 'application/json',
'Content-Type': 'multipart/form-data',
},
};
return fetch(apiUrl, options);
}

Upload file to a server just by relative path /assets/image/img.png - By JavaScript

I'm building an ionic app, I've some images in the assets directory. I'm trying to send one of each by getting its path and casting to a File object as you can see in my post and sending it to a server.
Here how I can implement in this way
const img = "/assets/img/E88MIfBTVCjmB10U1GLF_elderlies.jpg";
var imgage = new File(["foo"], img, {
type: "image/jpg"
});
var headers = new HttpHeaders();
headers.append("Content-Type", "application/x-www-form-urlencoded");
let formData = new FormData();
formData.append("file", image);
return this.http.post(`${this.baseUrl}`, formData, {
headers: headers
});

Here is the solution.
async uploadFile() {
const img = "/assets/img/iKtafmNpSGyj57ZRckVt_earing.jpg";
this.convertRelativeUriToFile(img, 'a',null, (file) => {
console.log(file)
this.uploadCare.uploadImage(file).subscribe(console.log, console.log);
})
}
async convertRelativeUriToFile (filePath, fileName, mimeType, cb) {
mimeType = mimeType || `image/${filePath.split('.')[filePath.split('.').length - 1]}`;
const imageUrl = await fetch(filePath);
const buffer = await imageUrl.arrayBuffer();
cb(new File([buffer], fileName, {type:mimeType}));
}

Node/Request Error: "Processing POST Request: No Content-Type"

I have a front end Canvas that I transform into a png file that I need to POST to a third party vendor's api. It passes back to node as a base64 file and I decode it, but when I attempt the upload, it gives me the following error:
Problem processing POST request: no Content-Type specified
However, I am clearly specifying the content type in my POST call. My end goal is to upload the file to my vendor's API.
Here are the key front end aspects:
var canvasImage = document.getElementById("c");
var img = canvas.toDataURL({
multiplier: canvasMultiplier
});
var fileTime = Date.now();
var myFileName = $scope.productCode + fileTime;
$scope.filenameForVendor = myFileName;
var filename = $scope.filenameForVendor;
$http.post('/postVendor', { filename: filename, file: img }).success(function (data) {
console.log("Uploaded to Vendor");
Here is the backend POST:
app.post('/postVendor', function (req, res, next) {
var filename = req.body.filename;
var file = req.body.file;
fileBuffer = decodeBase64Image(file);
request({
url: "http://myvendorapi/ws/endpoint",
method: "POST",
headers: {
'contentType': fileBuffer.type
},
body: fileBuffer.data
}, function (error, response, body) {
console.log(response);
});
})
// Decode file for upload
function decodeBase64Image(dataString) {
var matches = dataString.match(/^data:([A-Za-z-+\/]+);base64,(.+)$/),
response = {};
if (matches.length !== 3) {
return new Error('Invalid input string');
}
response.type = matches[1];
response.data = new Buffer(matches[2], 'base64');
return response;
}
I can POST using AJAX on the front end, but because of CORS and the vendor blocking all but server side calls to the endpoints (and they don't have JSONP), I can't use this. They are allowing my IP through for testing purposes so only I can make this work from my machine:
var send = function (blob) {
var fileTime = Date.now();
var myFileName = $scope.productCode + fileTime;
$scope.filenameForVendor = myFileName;
var filename = $scope.filenameForVendor;
var formdata = new FormData();
formdata.append('File1', blob, filename);
$.ajax({
url: 'http://myvendorapi/ws/endpoint',
type: "POST",
data: formdata,
mimeType: "multipart/form-data",
processData: false,
contentType: false,
crossDomain: true,
success: function (result) {
console.log("Upload to Vendor complete!");
// rest of code here/including error close out
}
var bytes = atob(dataURL.split(',')[1])
var arr = new Uint8Array(bytes.length);
for (var i = 0; i < bytes.length; i++) {
arr[i] = bytes.charCodeAt(i);
}
send(new Blob([arr], { type: 'image/png' }));
Update:
I realized that contentType should be 'content-type'. When I did this, it creates an error of no boundary specified as I am trying multipart-form data (which I did all wrong). How can I pass formData to Node for uploading?
Update 2:
Per the advice offered, I tried using multer but am getting an ReferenceError: XMLHttpRequest is not defined.
Client side:
var fileTime = Date.now();
var myFileName = $scope.productCode + fileTime;
$scope.filenameForVendor = myFileName;
var filename = $scope.filenameForVendor;
var formdata = new FormData();
formdata.append('File1', blob, filename);
$http.post('/postVendor', formdata, { transformRequest: angular.identity, headers: { 'Content-Type': undefined } }).success(function (data) {
Server side:
app.post('/postVendor', function (req, res, next) {
var request = new XMLHttpRequest();
request.open("POST", "http://myvendorapi.net/ws/endpoint");
request.send(formData);
})

Why do you base64 encode the file?
You can upload raw file to your Node using FormData and you will not have to decode anything.
Front end
...
var request = new XMLHttpRequest();
request.open('POST', 'http://node.js/method');
request.send(formData); // vanilla
--- or ---
...
$http.post('http://node.js/method', formData, {
transformRequest: angular.identity,
headers: {'Content-Type': undefined}
}); // angular
Back end
Just install request.
...
var request = require('request');
app.post('/method', function (req, res, next) {
// if you just want to push request you don't need to parse anything
req.pipe(request('http://vendor.net')).pipe(res);
}) // express