We Keep Coding

How do I get uploaded image in next js and save it?

How do I get uploaded image in next.js API route and save it on public folder? I have front end ready. I'm uploading images to an endpoint using plain JavaScript.
here is the onSubmit function for uploading images. Suggest me if I'm doing it wrong here. The main question is how do I retrieve it?
const onSubmit=async(e)=>{
e.preventDefault();
const fd=new FormData()
fd.append('myfile',image.name)
let res=await fetch(`http://localhost:3000/api/upload`,{
method: 'POST',
headers: {
"Content-Type": "image/jpeg",
},
body: fd,
})
let response=await res.json();
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.

This is the endpoint code I used for uploading image in nextjs, it requires some additional packages I will list them bellow also.
next-connect
multer
uuid
import nextConnect from "next-connect";
import multer from "multer";
import { v4 as uuidv4 } from "uuid";
let filename = uuidv4() + "-" + new Date().getTime();
const upload = multer({
storage: multer.diskStorage({
destination: "./public/uploads/profiles", // destination folder
filename: (req, file, cb) => cb(null, getFileName(file)),
}),
});
const getFileName = (file) => {
filename +=
"." +
file.originalname.substring(
file.originalname.lastIndexOf(".") + 1,
file.originalname.length
);
return filename;
};
const apiRoute = nextConnect({
onError(error, req, res) {
res
.status(501)
.json({ error: `Sorry something Happened! ${error.message}` });
},
onNoMatch(req, res) {
res.status(405).json({ error: `Method '${req.method}' Not Allowed` });
},
});
apiRoute.use(upload.array("file")); // attribute name you are sending the file by
apiRoute.post((req, res) => {
res.status(200).json({ data: `/uploads/profiles/${filename}` }); // response
});
export default apiRoute;
export const config = {
api: {
bodyParser: false, // Disallow body parsing, consume as stream
},
};

no Need to use any packages to handle file uploading you can use base64 to convert file to string and return it back to file by using "fs" module
why This way is beterr then using formData ?
because you duleing with normal post request where you can send any kind of data with it and use body parsere .
converting
const toBase64 = (file: File) => new Promise((resolve, reject) => {
const reader = new FileReader();
reader.readAsDataURL(file);
reader.onload = () => resolve(reader.result);
reader.onerror = error => reject(error);
}
send a post request to server
const base64: string = await toBase64(file) as string;
const fileData = { base64, fileName: file.name };
const result = await api.post("/foo", fileData, name: "Salih", massage: "Hello World"})
converting base64 to file in server
function base64ToFile(file: { base64: string, fileName: string }) {
const fileContents = file.base64.replace(/^data:image\/png;base64,/, "");
fs.mkdirSync("./public/uploads", { recursive: true });
const fileName = `./public/uploads/${Date.now().toString() + file.fileName}`
fs.writeFile(fileName, fileContents, 'base64', function (err) { console.log(err) });
}

I suggest the popular and lightweight formidable library:
# install
yarn add formidable#v3 #types/formidable
// pages/api/file-upload.ts
import fs from "fs";
import path from "path";
import { File } from "formidable";
// Important for NextJS!
export const config = {
api: {
bodyParser: false,
},
};
export default async function handler(
req: NextApiRequest,
res: NextApiResponse<string>
) {
try {
// Parse request with formidable
const { fields, files } = await parseFormAsync(req);
// Files are always arrays (formidable v3+)
const myfile = (files["myfile"] as any as File[])[0];
// Save file in the public folder
saveFile(myfile, "./public/uploads");
// Return success
res.status(200).json("success!");
} catch (e) {
return res.status(500).json(e);
}
}
function saveFile(file: File, publicFolder: string): void {
const fileExt = path.extname(file.originalFilename || "");
fs.renameSync(file.filepath, `${publicFolder}/${file.newFilename}${fileExt}`);
}
// ./helpers/formidable.ts
import type { NextApiRequest } from "next";
import formidable from "formidable";
export type FormidableParseReturn = {
fields: formidable.Fields;
files: formidable.Files;
};
export async function parseFormAsync(
req: NextApiRequest,
formidableOptions?: formidable.Options
): Promise<FormidableParseReturn> {
const form = formidable(formidableOptions);
return await new Promise<FormidableParseReturn>((resolve, reject) => {
form.parse(req, async (err, fields, files) => {
if (err) {
reject(err);
}
resolve({ fields, files });
});
});
}
Bonus question
one more bonus question, it's surely not a good idea to save the uploaded images on public folder. I have save it somewhere on the cloud.
S3 and other cloud services
You can save on cloud services with Formidable.
See the official examples: https://github.com/node-formidable/formidable/blob/master/examples/store-files-on-s3.js
But you don't need to use cloud storage to protect private uploads. You can store them locally.
Working with private uploads locally
Saving:
Store the uploads in a non-public folder;
Ex. /private-uploads/{logged_user_id}/;
Reading:
Create an API page to fetch the file
Ex. https://.../uploads/{filename}
Fail if the file doesn't belong to the authenticated user;
Send the file as the response;
Security:
With the above folder scheme, hackers can use .. and similar on the filename to obtain unauthorized access;
Sanitize the filename having this in mind (ex. only allow alphanumeric characters);
Alternatively, use a database table to control ownership instead of a folder scheme;

AWS S3 file upload with Node.js: Unsupported body payload error

I am trying to get my node.js backend to upload a file to AWS S3, which it got in a post request from my front-end. This is what my function looks like:
async function uploadFile(file){
var uploadParams = {Bucket: '<bucket-name>', Key: file.name, Body: file};
s3.upload (uploadParams, function (err, data) {
if (err) {
console.log("Error", err);
} if (data) {
console.log("Upload Success", data.Location);
}
});
}
When I try uploading the file this way, I get an Unsupported Body Payload Error...
I used fileStream.createReadStream() in the past to upload files saves in a directory on the server, but creating a fileStream did not work for me, since there is no path parameter to pass here.
EDIT:
The file object is created in the angular frontend of my web application. This it the relevant html code where the file is uploaded by a user:
<div class="form-group">
<label for="file">Choose File</label>
<input type="file" id="file"(change)="handleFileInput($event.target.files)">
</div>
If the event occurs, the handleFileInput(files: FileList) method in the corresponding component is called:
handleFileInput(files: FileList) {
// should result in array in case multiple files are uploaded
this.fileToUpload = files.item(0);
// actually upload the file
this.uploadFileToActivity();
// used to check whether we really received the file
console.log(this.fileToUpload);
console.log(typeof this.fileToUpload)
}
uploadFileToActivity() {
this.fileUploadService.postFile(this.fileToUpload).subscribe(data => {
// do something, if upload success
}, error => {
console.log(error);
});
}
the postFile(fileToUpload: File) method of the file-upload service is used to make the post request:
postFile(fileToUpload: File): Observable<Boolean> {
console.log(fileToUpload.name);
const endpoint = '/api/fileupload/single';
const formData: FormData = new FormData();
formData.append('fileKey', fileToUpload, fileToUpload.name);
return this.httpClient
.post(endpoint, formData/*, { headers: yourHeadersConfig }*/)
.pipe(
map(() => { return true; }),
catchError((e) => this.handleError(e)),
);
}
Here is the the server-side code that receives the file and then calls the uploadFile(file) function:
app.post('/api/fileupload/single', async (req, res) => {
try {
if(!req.files) {
res.send({
status: false,
message: 'No file uploaded'
});
} else {
let file = req.files.fileKey;
uploadFile(file);
//send response
res.send({
status: true,
message: 'File is uploaded',
data: {
name: file.name,
mimetype: file.mimetype,
size: file.size
}
});
}
} catch (err) {
res.status(500).send(err);
}
});
Thank you very much for your help in solving this!
Best regards, Samuel

Best way is stream the file. Assuming you are. reading it from disk. You could do this
const fs = require("fs");
const aws = require("aws-sdk");
const s3Client = new aws.S3();
const Bucket = 'somebucket';
const stream = fs.createReadStream("file.pdf");
const Key = stream.path;
const response = await s3Client.upload({Bucket, Key, Body: stream}).promise();
console.log(response);

How to get file properties and upload a file from ionic 4?

I am trying to upload a file from mobile to google bucket using ionic 4. Although a file can upload into the could. I am struggling to get the file properties out of file object.
Here is my method,
async selectAFile() {
const uploadFileDetails = {
name: '',
contentLength: '',
size: '',
type: '',
path: '',
};
this.fileChooser.open().then(uri => {
this.file.resolveLocalFilesystemUrl(uri).then(newUrl => {
let dirPath = newUrl.nativeURL;
const dirPathSegments = dirPath.split('/');
dirPathSegments.pop();
dirPath = dirPathSegments.join('/');
(<any>window).resolveLocalFileSystemURL(
newUrl.nativeURL,
function(fileEntry) {
uploadFileDetails.path = newUrl.nativeURL;
const file: any = getFileFromFileEntry(fileEntry);
//log 01
console.log({ file });
uploadFileDetails.size = file.size;
uploadFileDetails.name = `${newUrl.name
.split(':')
.pop()}.${file.type.split('/').pop()}`;
uploadFileDetails.type = file.type;
async function getFileFromFileEntry(fileEntry) {
try {
return await new Promise((resolve, reject) =>
fileEntry.file(resolve, reject)
);
} catch (err) {
console.log(err);
}
}
},
function(e) {
console.error(e);
}
);
});
});
// here uploadFileDetails is simller to what I declared at the top ;)
// I wan't this to be populated with file properties
// console.log(uploadFileDetails.name) --> //''
const uploadUrl = await this.getUploadUrl(uploadFileDetails);
const response: any = this.uploadFile(
uploadFileDetails,
uploadUrl
);
response
.then(function(success) {
console.log({ success });
this.presentToast('File uploaded successfully.');
this.loadFiles();
})
.catch(function(error) {
console.log({ error });
});
}
even though I can console.log the file in log 01. I am unable to get file properties like, size, name, type out of the resolveLocalFileSystemURL function. basically, I am unable to populate uploadFileDetails object. What am I doing wrong? Thank you in advance.

you actually need 4 Ionic Cordova plugins to upload a file after getting all the metadata of a file.
FileChooser
Opens the file picker on Android for the user to select a file, returns a file URI.
FilePath
This plugin allows you to resolve the native filesystem path for Android content URIs and is based on code in the aFileChooser library.
File
This plugin implements a File API allowing read/write access to files residing on the device.
File Trnafer
This plugin allows you to upload and download files.
getting the file's metadata.
file.resolveLocalFilesystemUrl with fileEntry.file give you all the metadata you need, except the file name. There is a property called name in the metadata but it always contains value content.
To get the human readable file name you need filePath. But remember you can't use returning file path to retrieve metadata. For that, you need the original url from fileChooser.
filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1) is used to get only file name from filePath.
You need nativeURL of the file in order to upload it. Using file path returning from filePath is not going to work.
getFileInfo(): Promise<any> {
return this.fileChooser.open().then(fileURI => {
return this.filePath.resolveNativePath(fileURI).then(filePathUrl => {
return this.file
.resolveLocalFilesystemUrl(fileURI)
.then((fileEntry: any) => {
return new Promise((resolve, reject) => {
fileEntry.file(
meta =>
resolve({
nativeURL: fileEntry.nativeURL,
fileNameFromPath: filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1),
...meta,
}),
error => reject(error)
);
});
});
});
});
}
select a file from the file system of the mobile.
async selectAFile() {
this.getFileInfo()
.then(async fileMeta => {
//get the upload
const uploadUrl = await this.getUploadUrl(fileMeta);
const response: Promise < any > = this.uploadFile(
fileMeta,
uploadUrl
);
response
.then(function(success) {
//upload success message
})
.catch(function(error) {
//upload error message
});
})
.catch(error => {
//something wrong with getting file infomation
});
}
uploading selected file.
This depends on your backend implementation. This is how to use File Transfer to upload a file.
uploadFile(fileMeta, uploadUrl) {
const options: FileUploadOptions = {
fileKey: 'file',
fileName: fileMeta.fileNameFromPath,
headers: {
'Content-Length': fileMeta.size,
'Content-Type': fileMeta.type,
},
httpMethod: 'PUT',
mimeType: fileMeta.type,
};
const fileTransfer: FileTransferObject = this.transfer.create();
return fileTransfer.upload(file.path, uploadUrl, options);
}
hope it helps. :)

Stream file upload to s3 via express server

I have an express endpoint where i currently handle uploading of files. Large files are taking lots of memory b/c i was using bodyParser which buffers the entire file in memory before calling my handler function.
I removed the bodyParser middleware from this endpoint and i'm strugging to properly use streams to basically stream the file upload -> express -> s3.
This is the docs on the s3 method and it accepts a buffer or a stream.
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/S3.html#upload-property
route
router.put('/files/:filename', putHandler({ s3Client: s3Client }))
I tried this which streams the file to my handler method, but it doesn't seem to be streaming it to the s3.upload method (no surprise really)
function put ({ s3Client }) {
return (req, res) => {
...
let whenFileUploaded = new Promise((resolve, reject) => {
// const { Readable } = require('stream')
// const inStream = new Readable({
// read() {}
// })
let data = ''
req.on('data', function (chunk) {
req.log.debug('in chunk')
data += chunk
// inStream.push(chunk)
})
req.on('end', function () {
req.log.debug('in end')
})
s3Client.upload(
{
Key: filepath,
Body: data,
SSECustomerAlgorithm: 'AES256',
SSECustomerKey: sseKey.id.split('-').join('')
},
{
partSize: 16 * 1024 * 1024, // 16mb
queuSize: 1
},
(err, data) => err ? reject(err) : resolve(data)
)
})
My guess is that I need to create a stream and pipe req.on('data... to my stream and then set Body: inStream which you can see i attempted with the commented out stuff, but that didn't seem to work either.
Help?

Turns out the answer is actually very simple. All I had to do was pass the req object.
function put ({ s3Client }) {
return (req, res) => {
...
let whenFileUploaded = new Promise((resolve, reject) => {
s3Client.upload(
{
Key: filepath,
Body: req, // <-- NOTE THIS LINE
SSECustomerAlgorithm: 'AES256',
SSECustomerKey: sseKey.id.split('-').join('')
},
{
partSize: 16 * 1024 * 1024, // 16mb
queuSize: 1
},
(err, data) => err ? reject(err) : resolve(data)
)
})
The way i found this out is b/c I looked at the express source code for what a req object is and I see that it is a http.IncomingMessage object - https://github.com/expressjs/express/blob/master/lib/request.js#L31
Then i looked at the Node docs and I see that http.IncomingMessage implements the Readable Stream interface
It implements the Readable Stream interface, as well as the following
additional events, methods, and properties.
https://nodejs.org/docs/latest-v9.x/api/http.html#http_class_http_incomingmessage

Graphql Apollo Server + Vue => Image Upload

Im using Vue with vue-apollo in the frontend and graphql stand-alone Apollo Server 2 with mongodb through mongoose in the backend. I have a simple blog application in which posts also have an Image. Everything works fine except uploading Images. I want the images to be uploaded to my local filesystem in a folder on my backend and only the path to the image saved in my mongodb document.
the mutation:
async createPost(parent, args, context, info) {
//...
const {stream, filename} = await args.img
const img_path = await upload({stream, filename})
const post = await Post.save({
//img is a string in my mongo model
img: img_path,
author_name: args.user.username,
author_email: args.user.email
});
}
the upload method that should return the path and save the image to local:
const upload = ({ stream, filename }) => {
const id = shortid.generate()
const path = `${UPLOAD_DIR}/${filename}-${id}`
new Promise((resolve, reject) =>
stream
.pipe(fs.createWriteStream(filename))
.on("finish", () => resolve(path))
.on("error", reject(Error))
);
}
The error im getting is that stream and filename are undefined when calling upload() but args.img is an object if i log it. And uploading them to my local folder doesnt work neither. Any help is appreciated and marked as accepted answer

It would be nice to share your graphql Schema so that we can see the types you're returning. However, Here's how i have been handling file uploads in most of my apps.
graphql-schema
type File {
id: ID!
filename: String!
mimetype: String!
path: String!
}
mongoose schema
import { Schema, model } from "mongoose";
const fileSchema = new Schema({
filename: String,
mimetype: String,
path: String,
});
export default model("File", fileSchema);
Function to store uploads:
const storeUpload = async ({ stream, filename, mimetype }) => {
const id = shortid.generate();
const path = `images/${id}-${filename}`;
// (createWriteStream) writes our file to the images directory
return new Promise((resolve, reject) =>
stream
.pipe(createWriteStream(path))
.on("finish", () => resolve({ id, path, filename, mimetype }))
.on("error", reject)
);
};
To process the uploads
const processUpload = async (upload) => {
const { createReadStream, filename, mimetype } = await upload;
const stream = createReadStream();
const file = await storeUpload({ stream, filename, mimetype });
return file;
};
Mutation
export default {
Mutation: {
uploadFile: async (_, { file }) => {
mkdir("images", { recursive: true }, (err) => {
if (err) throw err;
});
const upload = await processUpload(file);
// save our file to the mongodb
await File.create(upload);
return upload;
},
},
};
Here you can find an article i wrote on how to handle file uploads