I have excel file stored in S3 bucket fro where I am storing the file url inside mongodb.I
want to read data from this excel file and transform into JSON.So for that I am XLSX package but it is not reading the file as I am passing the file url inside the method,
Below is my code:
try {
const data = await productSchema.findOne({ report_id: reportId });
if (data) {
console.log(data.me_url); //I am getting the file url
const file = xlsx.readFile(data.me_url);
const sheetNames = file.SheetNames;
const totalSheets = sheetNames.length;
console.log(sheetNames);
console.log(totalSheets);
}
}
catch (err) {
return err;
}
Any idea why it is not reading the file or anyhthing I have missed.
Related
I have file hosted on S3 bucket from where I am fetching the Excel file using an Axios request. After that I am writing that Excel file in the local filesystem. After that I am converting that Excel file data into JSON and sending it as a response; for that I am using XLSX library.
After sending the Excel file data as a JSON response I want to delete that Excel file from local filesystem.
When I am trying to delete the file it's failing to delete as well as failed to read the file. But when file is already available then it's reading the file and sending the response also.
Below is my code:
const response = await axios.get(data.me_url, { responseType: 'arraybuffer' });
if (response) {
const ostream = await fs.createWriteStream(`./${filename}`, 'binary');
ostream.write(response.data);
ostream.end();
const file = xlsx.readFile(`./${filename}`);
const sheetNames = file.SheetNames;
const totalSheets = sheetNames.length;
let parsedData = [];
for (let i = 0; i < totalSheets; i++) {
const tempData = xlsx.utils.sheet_to_json(file.Sheets[sheetNames[i]]);
tempData.shift();
parsedData.push(...tempData);
}
if (parsedData.length > 0) {
console.log('if filename',filename);
try {
fs.unlinkSync(`./${filename}`);
console.log("Delete File successfully.");
return res.status(200).send({ 'data': parsedData, 'message': 'Success', 'code': 200 });
} catch (error) {
console.log(error);
}
}
}
What am I doing wrong in the above code?
You don't need to create file in the local machine, send the file Array Buffer directly to xlsx directly as mentioned in the link here
So I created my first big project: https://rate-n-write.herokuapp.com/
In brief, this is a blog app where the user can write reviews and publish them along with pictures.
I have used firebase as the database to store the articles. The app is working fine on localhost. Whenever I am trying to upload an image on Heroku, I get this error
The error is showing up in line number 8 of the following code (editor.js):
uploadInput.addEventListener('change', () => {
uploadImage(uploadInput, "image");
})
const uploadImage = (uploadFile, uploadType) => {
const [file] = uploadFile.files;
if(file && file.type.includes("image")){
const formdata = new FormData();
formdata.append('image', file);
//Error shows up here in the fetch line
fetch('/upload', {
method: 'post',
body: formdata
}).then(res => res.json())
.then(data => {
if(uploadType == "image"){
addImage(data, file.name);
} else{
bannerPath = `${location.origin}/${data}`;
banner.style.backgroundImage = `url("${bannerPath}")`;
}
})
const change_text = document.getElementById("uploadban");
change_text.innerHTML = " ";
} else{
alert("upload Image only");
}
}
This is just a snippet of the whole editor.js file.
Is it because I am trying to upload the file to the project directory? (server.js snippet below):
app.post('/upload', (req, res) => {
let file = req.files.image;
let date = new Date();
// image name
let imagename = date.getDate() + date.getTime() + file.name;
// image upload path
let path = 'public/uploads/' + imagename;
// create upload
file.mv(path, (err, result) => {
if(err){
throw err;
} else{
// our image upload path
res.json(`uploads/${imagename}`)
}
})
})
Do I need to use an online storage service like AWS S3?
Heroku is not suitable for persistent storage of data, the uploaded pictures would be deleted after a while (when the dyno is restarted) read this.
I would suggest using 3rd party object Storage services like
cloudinary or AWS S3
How do I download the document I receive in return in react?
Here is the my node.js app. fetchContracts is a function which getting data from mongodb then ganere a excel file by using json2xls npm package.
Its returns as like this:
const xls = json2xls(contracts);
return xls;
If tying to write file fs.writeFileSync(path.join(__dirname, filename), xls, 'binary'); generating successfully xlsx file in the server.
But I need to send the file to the server without writing file. For this, I made some experiments that you can see below.
export const EXPORT_EXCEL: SessionedAsyncControllerType = async (req: SessionedRequest, res: Response) => {
const fileName = 'hello_world.xlsx'
const fileType = 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'
const xls = await fetchContracts({}, "fileName.xlsx")
const fileData = xls;
res.writeHead(200, {
'Content-Disposition': `attachment; filename="${fileName}"`,
'Content-Type': fileType,
})
const download = Buffer.from(fileData, 'base64')
res.end(download)
}
I getting response like this.
But i don't know how can i download the response file in react?
In react side:
return api.get(`api/excel`).then((response: any) => {
console.log(response);
})
I just log into console. How can i download directly file which coming node response in react.js?
Try this
return api.get(`api/excel`).then((response: any) => {
const outputFilename = `${Date.now()}.xlsx`;
// If you want to download file automatically using link attribute.
const url = URL.createObjectURL(new Blob([response.data]));
const link = document.createElement('a');
link.href = url;
link.setAttribute('download', outputFilename);
link.click();
})
I'have currently a problem to create and to send an archive from a node.js server.
In fact, i succeeded create an archive with 5 csv files on the archive but when i send the archive to my React Application and when i unzip this archive i'have got just one file ....
I have checked (local) the archive created by the server and there are 5 files instead of one when i unzip the folder send to the React Application.
My http response :
import { Parser } from "json2csv";
import archiver from "archiver";
import fs from "fs";
import path from "path";
const output = fs.createWriteStream(`csv/${campaign.title}.zip`, 'utf-8');
const archive = archiver("zip", {
zlib: { level: 9 }, // Sets the compression level.
});
output.on("close", function () {
console.log(archive.pointer() + " total bytes");
console.log("archiver has been finalized and the output file descriptor has closed.");
});
output.on("end", function () {
console.log("Data has been drained");
});
archive.on("warning", function (err) {
if (err.code === "ENOENT") {
// log warning
} else {
// throw error
throw err;
}});
archive.on("error", function (err) {
throw err;
});
archive.pipe(output);
d.map(items => {
let file = items;
archive.append(file, { name: file });
});
archive.pipe(res)
archive.finalize();
res.setHeader('application/zip')
res.setHeader("Content-Disposition", `attachment; filename=${campaign.title}.zip`);
res.status(200).send(archive);
What is content of mystical d? You use archiver incorrectly I think:
your code:
d.map(items => {
let file = items;
archive.append(file, { name: file });
});
but documentation say:
https://www.archiverjs.com/docs/archiver#append
append(source, data) → {this}
Appends an input source (text string, buffer, or stream) to the instance.
When the instance has received, processed, and emitted the input, the entry event is fired.
Parameters
source - Buffer | Stream | String - The input source.
data - Object - The entry data.
Check content and use of d - you need data and filename for use in append
EDIT:
Don't use Array.map like Array.forEach, it's Anti-pattern: Is performing a mapping operation without using returned value an antipattern?
I am trying to upload file to s3, before that I am altering the name of the file. Now I am accepting 2 files from request form-data object, renaming the filename, and uploading the file to s3. And end of the task I need to return the renamed file list which is uploaded successfully.
I am using S3.upload() function. But the problem is, the variable which is assigned as empty array initially, that will contain the renamed file list. But the array is returning empty response. The s3.upload() is taking much time. is there any probable solution where I can store the file name if upload is successful and return those names in response.
Please help me to fix this. The code looks like this,
if (formObject.files.document && formObject.files.document.length > 0) {
const circleCode = formObject.fields.circleCode[0];
let collectedKeysFromAwsResponse = [];
formObject.files.document.forEach(e => {
const extractFileExtension = ".pdf";
if (_.has(FILE_EXTENSIONS_INCLUDED, _.lowerCase(extractFileExtension))) {
console.log(e);
//change the filename
const originalFileNameCleaned = "cleaning name logic";
const _id = mongoose.Types.ObjectId();
const s3FileName = "s3-filename-convension;
console.log(e.path, "", s3FileName);
const awsResponse = new File().uploadFileOnS3(e.path, s3FileName);
if(e.hasOwnProperty('ETag')) {
collectedKeysFromAwsResponse.push(awsResponse.key.split("/")[1])
}
}
});
};
use await s3.upload(params).promise(); is the solution.
Use the latest code - which is AWS SDK for JavaScript V3. Here is the code you should be using
// Import required AWS SDK clients and commands for Node.js.
import { PutObjectCommand } from "#aws-sdk/client-s3";
import { s3Client } from "./libs/s3Client.js"; // Helper function that creates Amazon S3 service client module.
import {path} from "path";
import {fs} from "fs";
const file = "OBJECT_PATH_AND_NAME"; // Path to and name of object. For example '../myFiles/index.js'.
const fileStream = fs.createReadStream(file);
// Set the parameters
export const uploadParams = {
Bucket: "BUCKET_NAME",
// Add the required 'Key' parameter using the 'path' module.
Key: path.basename(file),
// Add the required 'Body' parameter
Body: fileStream,
};
// Upload file to specified bucket.
export const run = async () => {
try {
const data = await s3Client.send(new PutObjectCommand(uploadParams));
console.log("Success", data);
return data; // For unit tests.
} catch (err) {
console.log("Error", err);
}
};
run();
More details can be found in the AWS JavaScript V3 DEV Guide.