We Keep Coding

How to make CSV to Array in Next.js API from Postman

I want to read the CSV file from Next.js API and it's sent from POSTMAN like the below image
But how do I read that CSV file?
Like this below code, I don't read the file anyhow.
export default async function handler(req, res) {
console.log(req.body);
// console.log(req.file); // Undefined
// console.log(req.files['file']); // Undefined
}
My goal is to read files and convert them to an array of data.
Thanks

I recommend you to use the formidable package to read the files server side. You also need to disable bodyParser. A minimum example would be this code:
export default async function handler(
req: NextApiRequest,
res: NextApiResponse
) {
if (req.method === "POST") {
const data: {fields, files} = await new Promise((resolve, reject) => {
const form = new IncomingForm();
form.parse(req, (err, fields, files) => {
if (err) return reject(err);
resolve({ fields, files });
});
});
const file = data.files.file;
let fileData = await fs.promises.readFile(file.filepath);
// now you need to parse the csv file (contents is in the fileData buffer) or do what ever you want
res.status(200).end("Uploaded");
return;
}
res.status(400).end("Method not allowed");
}
export const config = {
api: {
bodyParser: false,
}
};
A package for parsing csv files can be found here

AWS S3 file upload with Node.js: Unsupported body payload error

I am trying to get my node.js backend to upload a file to AWS S3, which it got in a post request from my front-end. This is what my function looks like:
async function uploadFile(file){
var uploadParams = {Bucket: '<bucket-name>', Key: file.name, Body: file};
s3.upload (uploadParams, function (err, data) {
if (err) {
console.log("Error", err);
} if (data) {
console.log("Upload Success", data.Location);
}
});
}
When I try uploading the file this way, I get an Unsupported Body Payload Error...
I used fileStream.createReadStream() in the past to upload files saves in a directory on the server, but creating a fileStream did not work for me, since there is no path parameter to pass here.
EDIT:
The file object is created in the angular frontend of my web application. This it the relevant html code where the file is uploaded by a user:
<div class="form-group">
<label for="file">Choose File</label>
<input type="file" id="file"(change)="handleFileInput($event.target.files)">
</div>
If the event occurs, the handleFileInput(files: FileList) method in the corresponding component is called:
handleFileInput(files: FileList) {
// should result in array in case multiple files are uploaded
this.fileToUpload = files.item(0);
// actually upload the file
this.uploadFileToActivity();
// used to check whether we really received the file
console.log(this.fileToUpload);
console.log(typeof this.fileToUpload)
}
uploadFileToActivity() {
this.fileUploadService.postFile(this.fileToUpload).subscribe(data => {
// do something, if upload success
}, error => {
console.log(error);
});
}
the postFile(fileToUpload: File) method of the file-upload service is used to make the post request:
postFile(fileToUpload: File): Observable<Boolean> {
console.log(fileToUpload.name);
const endpoint = '/api/fileupload/single';
const formData: FormData = new FormData();
formData.append('fileKey', fileToUpload, fileToUpload.name);
return this.httpClient
.post(endpoint, formData/*, { headers: yourHeadersConfig }*/)
.pipe(
map(() => { return true; }),
catchError((e) => this.handleError(e)),
);
}
Here is the the server-side code that receives the file and then calls the uploadFile(file) function:
app.post('/api/fileupload/single', async (req, res) => {
try {
if(!req.files) {
res.send({
status: false,
message: 'No file uploaded'
});
} else {
let file = req.files.fileKey;
uploadFile(file);
//send response
res.send({
status: true,
message: 'File is uploaded',
data: {
name: file.name,
mimetype: file.mimetype,
size: file.size
}
});
}
} catch (err) {
res.status(500).send(err);
}
});
Thank you very much for your help in solving this!
Best regards, Samuel

Best way is stream the file. Assuming you are. reading it from disk. You could do this
const fs = require("fs");
const aws = require("aws-sdk");
const s3Client = new aws.S3();
const Bucket = 'somebucket';
const stream = fs.createReadStream("file.pdf");
const Key = stream.path;
const response = await s3Client.upload({Bucket, Key, Body: stream}).promise();
console.log(response);

How to get file properties and upload a file from ionic 4?

I am trying to upload a file from mobile to google bucket using ionic 4. Although a file can upload into the could. I am struggling to get the file properties out of file object.
Here is my method,
async selectAFile() {
const uploadFileDetails = {
name: '',
contentLength: '',
size: '',
type: '',
path: '',
};
this.fileChooser.open().then(uri => {
this.file.resolveLocalFilesystemUrl(uri).then(newUrl => {
let dirPath = newUrl.nativeURL;
const dirPathSegments = dirPath.split('/');
dirPathSegments.pop();
dirPath = dirPathSegments.join('/');
(<any>window).resolveLocalFileSystemURL(
newUrl.nativeURL,
function(fileEntry) {
uploadFileDetails.path = newUrl.nativeURL;
const file: any = getFileFromFileEntry(fileEntry);
//log 01
console.log({ file });
uploadFileDetails.size = file.size;
uploadFileDetails.name = `${newUrl.name
.split(':')
.pop()}.${file.type.split('/').pop()}`;
uploadFileDetails.type = file.type;
async function getFileFromFileEntry(fileEntry) {
try {
return await new Promise((resolve, reject) =>
fileEntry.file(resolve, reject)
);
} catch (err) {
console.log(err);
}
}
},
function(e) {
console.error(e);
}
);
});
});
// here uploadFileDetails is simller to what I declared at the top ;)
// I wan't this to be populated with file properties
// console.log(uploadFileDetails.name) --> //''
const uploadUrl = await this.getUploadUrl(uploadFileDetails);
const response: any = this.uploadFile(
uploadFileDetails,
uploadUrl
);
response
.then(function(success) {
console.log({ success });
this.presentToast('File uploaded successfully.');
this.loadFiles();
})
.catch(function(error) {
console.log({ error });
});
}
even though I can console.log the file in log 01. I am unable to get file properties like, size, name, type out of the resolveLocalFileSystemURL function. basically, I am unable to populate uploadFileDetails object. What am I doing wrong? Thank you in advance.

you actually need 4 Ionic Cordova plugins to upload a file after getting all the metadata of a file.
FileChooser
Opens the file picker on Android for the user to select a file, returns a file URI.
FilePath
This plugin allows you to resolve the native filesystem path for Android content URIs and is based on code in the aFileChooser library.
File
This plugin implements a File API allowing read/write access to files residing on the device.
File Trnafer
This plugin allows you to upload and download files.
getting the file's metadata.
file.resolveLocalFilesystemUrl with fileEntry.file give you all the metadata you need, except the file name. There is a property called name in the metadata but it always contains value content.
To get the human readable file name you need filePath. But remember you can't use returning file path to retrieve metadata. For that, you need the original url from fileChooser.
filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1) is used to get only file name from filePath.
You need nativeURL of the file in order to upload it. Using file path returning from filePath is not going to work.
getFileInfo(): Promise<any> {
return this.fileChooser.open().then(fileURI => {
return this.filePath.resolveNativePath(fileURI).then(filePathUrl => {
return this.file
.resolveLocalFilesystemUrl(fileURI)
.then((fileEntry: any) => {
return new Promise((resolve, reject) => {
fileEntry.file(
meta =>
resolve({
nativeURL: fileEntry.nativeURL,
fileNameFromPath: filePathUrl.substring(filePathUrl.lastIndexOf('/') + 1),
...meta,
}),
error => reject(error)
);
});
});
});
});
}
select a file from the file system of the mobile.
async selectAFile() {
this.getFileInfo()
.then(async fileMeta => {
//get the upload
const uploadUrl = await this.getUploadUrl(fileMeta);
const response: Promise < any > = this.uploadFile(
fileMeta,
uploadUrl
);
response
.then(function(success) {
//upload success message
})
.catch(function(error) {
//upload error message
});
})
.catch(error => {
//something wrong with getting file infomation
});
}
uploading selected file.
This depends on your backend implementation. This is how to use File Transfer to upload a file.
uploadFile(fileMeta, uploadUrl) {
const options: FileUploadOptions = {
fileKey: 'file',
fileName: fileMeta.fileNameFromPath,
headers: {
'Content-Length': fileMeta.size,
'Content-Type': fileMeta.type,
},
httpMethod: 'PUT',
mimeType: fileMeta.type,
};
const fileTransfer: FileTransferObject = this.transfer.create();
return fileTransfer.upload(file.path, uploadUrl, options);
}
hope it helps. :)

How to return PDF file from controller

I am trying to return a PDF file from a Controller Endpoint using NestJs. When not setting the Content-type header, the data returned by getDocumentFile gets returned to the user just fine. When I add the header however, the return I get seems to be some strange form of a GUID, the response always looks like this: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx where x is a lowercase hexadecimal character. It also seems to be totally unrelated to the actual return value of the handler function, as I even get this strange GUID-thing when not returning anything at all.
When not setting Content-type: application/pdf, the function returns the data of the buffer just fine, however I need to set the header in order to get the browser to recognize the response as a PDF file which is important for my use case.
The controller looks like this:
#Controller('documents')
export class DocumentsController {
constructor(private documentsService: DocumentsService) {}
#Get(':id/file')
#Header('Content-type', 'application/pdf')
async getDocumentFile(#Param('id') id: string): Promise<Buffer> {
const document = await this.documentsService.byId(id)
const pdf = await this.documentsService.getFile(document)
// using ReadableStreamBuffer as suggested by contributor
const stream = new ReadableStreamBuffer({
frequency: 10,
chunkSize: 2048,
})
stream.put(pdf)
return stream
}
}
and my DocumentsService like this:
#Injectable()
export class DocumentsService {
async getAll(): Promise<Array<DocumentDocument>> {
return DocumentModel.find({})
}
async byId(id: string): Promise<DocumentDocument> {
return DocumentModel.findOne({ _id: id })
}
async getFile(document: DocumentDocument): Promise<Buffer> {
const filename = document.filename
const filepath = path.join(__dirname, '..', '..', '..', '..', '..', 'pdf-generator', 'dist', filename)
const pdf = await new Promise<Buffer>((resolve, reject) => {
fs.readFile(filepath, {}, (err, data) => {
if (err) reject(err)
else resolve(data)
})
})
return pdf
}
}
I originally just returned the buffer (return pdf), but that brought the same result as the attempt above. On the repository of NestJs a user suggested to use the above method, which obviously does not work for me either. See the GitHub thread here.

Update 2021:
From now on in Nest Version 8 you can use the class StreamableFile:
import { Controller, Get, StreamableFile } from '#nestjs/common';
import { createReadStream } from 'fs';
import { join } from 'path';
#Controller('file')
export class FileController {
#Get()
getFile(): StreamableFile {
const file = createReadStream(join(process.cwd(), 'package.json'));
return new StreamableFile(file);
}
}
More info in the offical Nest Docs: https://docs.nestjs.com/techniques/streaming-files

You can just use ready decorator #Res this is my working solution:
Controller(NestJs):
async getNewsPdfById(#Param() getNewsParams: GetNewsPdfParams, #Req() request: Request, #Res() response: Response): Promise<void> {
const stream = await this.newsService.getNewsPdfById(getNewsParams.newsId, request.user.ownerId);
response.set({
'Content-Type': 'image/pdf',
});
stream.pipe(response);
}
In my case stream variable is just ready stream created by html-pdf library because i create pdf by html https://www.npmjs.com/package/html-pdf but it doesnt matter how you create your stream. The thing is that you should use #Res decorator and pipe it because its native NestJs solution.
Also here is code how to claim file on client side:
https://gist.github.com/javilobo8/097c30a233786be52070986d8cdb1743
Anyway lets try this one in your case:
#Controller('documents')
export class DocumentsController {
constructor(private documentsService: DocumentsService) {}
#Get(':id/file')
async getDocumentFile(#Param('id') id: string, #Res res: Response): Promise<Buffer> {
const document = await this.documentsService.byId(id)
const pdf = await this.documentsService.getFile(document)
const stream = new ReadableStreamBuffer({
frequency: 10,
chunkSize: 2048,
})
res.set({
'Content-Type': 'image/pdf',
});
stream.pipe(res);
}
}

It works for me.
#Get('pdf')
#HttpCode(HttpStatus.OK)
#Header('Content-Type', 'application/pdf')
#Header('Content-Disposition', 'attachment; filename=test.pdf')
pdf() {
return createReadStream('./nodejs.pdf');
}
BTW, I think it should be better to use Stream instead of readFile. Because it loads all contents of the file into RAM.

I know this old thread. But it might help someone.
similar to #Victor
#Get('pdf')
#HttpCode(201)
#Header('Content-Type', 'image/pdf')
#Header('Content-Disposition', 'attachment; filename=test.pdf')
public pdf() {
return fs.createReadStream('./test.pdf');
}

Updated in 2021 too:
I prefer this way as I don't need the post-controller interceptor logic.
We can control the name of the file and make it be inline or downloading the file.
#Get()
download(#Res() res) {
const filename = '123.pdf';
// make it to be inline other than downloading
// res.setHeader('Content-disposition', 'inline; filename=' + filename);
res.setHeader('Content-disposition', 'attachment; filename=' + filename);
const filestream = createReadStream('files/' + filename);
filestream.pipe(res);
}
More info in the offical Nest Docs: https://docs.nestjs.com/techniques/streaming-files

Using the StreamableFile API:
#Get('openPdf/:filename')
async openPdf(
#Param('filename') filename: string,
#Response({ passthrough: true }) res: Res,
):Promise<StreamableFile> {
try {
//if your pdf or file is other directory add this process.cwd()+'/parentfolder/childfolder' or else leave at it is like below
const readableStream = fs.createReadStream(join(process.cwd(),`${filename}.pdf` ));
//set application type as you need json,pdf etc
res.set({
'Content-Type': 'application/pdf',
'Content-Disposition': `attachment; filename=${filename}.pdf`
})
// return readableStream.pipe(res)
const streamdata= new StreamableFile(readableStream)
return streamdata
} catch (error) {
return error.message
}
}

Unfortunately, "StreamableFile" can be used to v8 and upper versions. Every time I look at NestJS it doesn't match the needs and version of NestJS. This is a terrible thing...

Graphql Apollo Server + Vue => Image Upload

Im using Vue with vue-apollo in the frontend and graphql stand-alone Apollo Server 2 with mongodb through mongoose in the backend. I have a simple blog application in which posts also have an Image. Everything works fine except uploading Images. I want the images to be uploaded to my local filesystem in a folder on my backend and only the path to the image saved in my mongodb document.
the mutation:
async createPost(parent, args, context, info) {
//...
const {stream, filename} = await args.img
const img_path = await upload({stream, filename})
const post = await Post.save({
//img is a string in my mongo model
img: img_path,
author_name: args.user.username,
author_email: args.user.email
});
}
the upload method that should return the path and save the image to local:
const upload = ({ stream, filename }) => {
const id = shortid.generate()
const path = `${UPLOAD_DIR}/${filename}-${id}`
new Promise((resolve, reject) =>
stream
.pipe(fs.createWriteStream(filename))
.on("finish", () => resolve(path))
.on("error", reject(Error))
);
}
The error im getting is that stream and filename are undefined when calling upload() but args.img is an object if i log it. And uploading them to my local folder doesnt work neither. Any help is appreciated and marked as accepted answer

It would be nice to share your graphql Schema so that we can see the types you're returning. However, Here's how i have been handling file uploads in most of my apps.
graphql-schema
type File {
id: ID!
filename: String!
mimetype: String!
path: String!
}
mongoose schema
import { Schema, model } from "mongoose";
const fileSchema = new Schema({
filename: String,
mimetype: String,
path: String,
});
export default model("File", fileSchema);
Function to store uploads:
const storeUpload = async ({ stream, filename, mimetype }) => {
const id = shortid.generate();
const path = `images/${id}-${filename}`;
// (createWriteStream) writes our file to the images directory
return new Promise((resolve, reject) =>
stream
.pipe(createWriteStream(path))
.on("finish", () => resolve({ id, path, filename, mimetype }))
.on("error", reject)
);
};
To process the uploads
const processUpload = async (upload) => {
const { createReadStream, filename, mimetype } = await upload;
const stream = createReadStream();
const file = await storeUpload({ stream, filename, mimetype });
return file;
};
Mutation
export default {
Mutation: {
uploadFile: async (_, { file }) => {
mkdir("images", { recursive: true }, (err) => {
if (err) throw err;
});
const upload = await processUpload(file);
// save our file to the mongodb
await File.create(upload);
return upload;
},
},
};
Here you can find an article i wrote on how to handle file uploads