I'm working on a microcontroller that would either take docx files or html strings in input and would transform it into a singular pdf file and return its link as an ouput.
My code looks like this so far:
// 'files' is an array of uploaded docx files.
const uploaded = files.map((file) => {
return new Promise((resolve, reject) => {
pump(
file.toBuffer(),
fs.createWriteStream(join(__dirname, 'files', file.filename))
.on('finish', resolve)
)
})
})
Promise.all(uploaded)
// Is triggered
.then(async () => await convertFiles())
// Is not triggered
.then(async () => {
// concatStoreFiles() is an external function because I need it somewhere else too
test = await concatStoreFiles(join(__dirname, 'files'))
console.log({test})
res.send(test)
})
const convertFiles = () => {
return new Promise((resolve, reject) => {
const cmd = `soffice --headless --convert-to pdf --outdir ${join(__dirname, 'files')} ${join(__dirname, 'files', '*.*')}`
exec(cmd, (error, stdout, stderror) => {
if (error) console.warn(error)
resolve(stdout ?? stderror)
})
})
}
concatStoreFile.js
module.exports = async function concatFiles (dirPath, outPath) {
return new Promise ((resolve, reject) => {
const existingFiles = []
fs.readdir(dirPath, (e, files) => {
files.forEach((file) => {
// is added to the files list only if finishing with ".pdf"
if (/[\d\w_-]+.pdf/.matches(file)) {
existingFiles.push(file)
}
});
resolve(existingFiles)
})
})
}
I'm working with Insomnia for my development / test process, and it tells me that I get an empty response. However, I'm supposed to get an array of pdf files existing in a specific directory. I'm not even getting console.log({test}), so I don't think my second then() is triggered.
I'm really rusty with async / await and Promise syntaxes, what should I do in this situation?
Thank you in advance
The #fastify/multipart's toBuffer() API returns a Promise, not a buffer. Checkout this article
So you need to write something like:
const uploaded = files.map(processFile)
async function processFile (file) {
const buffer = await file.toBuffer()
const storedFileName = join(__dirname, 'files', file.filename)
const writeStream = fs.createWriteStream(storedFileName)
return new Promise((resolve, reject) => {
pump(buffer, writeStream, (err) => {
if(err) { return reject(err) }
resolve(storedFileName)
})
}
}
Moreover, to improve the code, I returned the storedFileName instead of recalculating it.
You can convert this:
.then(async () => await convertFiles())
to this:
.then(() => convertFiles())
Mixing async/await and promise then/catch leads to hidden bugs hard to find
Related
I am trying to parse data from a .csv file, and save it to an array for later use.
I understand the concept of promises, but I have no idea what am I missing in my code that I cannot resolve the Promise and get the value (the string in the .csv file). It while I can view all the data inside the promise (.on('data')) from debugging mode, I just can't save it in order to use it later in my 'try&catch'.
const fs = require("fs");
const csv = require("csv-parser");
const { resolve } = require("path");
async function readCSV(filepath) {
return new Promise(async (resolve, reject) => {
await fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const results = [];
const csvFilePath =
"/languages.csv";
try {
const languages = readCSV(csvFilePath).then((res) => {
return res;
});
console.log(languages);
} catch (e) {
console.log(e);
}
and the output on the console is:
>Promise {<pending>}
No debugger available, can not send 'variables'
** That's from the debugging mode when I pause inside the promise:
https://i.stack.imgur.com/H9nHi.png
You can't try catch a returned promise without the await keyword in an async function.
If you're returning a promise, you need to use the .catch method on the promise.
Also, when you're logging languages you're doing so before the promise resolves because you're not using the await keyword.
I'm sure the promise resolves. Instead, log res inside the .then method.
const fs = require("fs");
const csv = require("csv-parser");
const results = [];
function readCSV(filepath) {
return new Promise((resolve, reject) => {
fs
.createReadStream(filepath)
.pipe(csv())
.on("data", (data) => {
results.push(data);
})
.on("error", (error) => reject(results))
.on("end", () => {
resolve(results);
});
});
}
const csvFilePath = "./languages.csv";
(async () => {
const output = await readCSV(csvFilePath);
console.log(output)
})();
I have an async function that creates a folder and I want to be able to do a computation with this folder and then have the same function remove it.
What I have tried:
const t = async () => {
let createdFolderPath = ''
return new Promise(async (resolve, reject) => {
createdFolderPath = await createAFolder()
console.log('Folder Created')
resolve(createdFolderPath)
}).finally(async () => {
await deleteCreatedFolder(createdFolderPath)
console.log('Deleted Folder')
})
}
t().then(async (folderpath) => {
await doSomethingWithFolderThatIsAsync(folderPath)
console.log('Computation Done')
})
What I want to happen is:
Folder Created
Computation Done
Deleted Folder
What actually happens is:
Folder Created
Deleted Folder
Computation Done
I want the t function to also remove the folder so the person using it doesn't have to worry about removing the folder.
I'd pass the .then callback into t instead of using the returned Promise from t. You should also avoid the explicit Promise construction antipattern:
const t = async (thenCallback) => {
let createdFolderPath = ''
return createAFolder()
.then(thenCallback)
.finally(async () => {
await deleteCreatedFolder(createdFolderPath)
console.log('Deleted Folder')
})
}
t(async (folderpath) => {
await doSomethingWithFolderThatIsAsync(folderPath)
console.log('Computation Done')
})
.then(() => {
// everything finished
});
I am trying to retrieve multiple files from S3 using a readstream, and insert them into a single file locally.
Below, the 'output' variable is the single writestream I wish to append to using the downloaded S3 file data.
I am looping through days where the nextDay variable is used for the S3 key. The fileservice.s3Handler.getS3Obj returns an S3 object which allows a readstream for a single file and appending to the output file.
However, no other files are being read and are not showing the console either using the on('data', ()) method.
I tried to wrap the readstream in a promise to try to wait until the read was finished but it is running the same error.
More recently I keep get this error: "ERR_STREAM_WRITE_AFTER_END"
Not sure what is going wrong here.
async fetchCSV(req, res) {
const output = fs.createWriteStream(outputPathWithFile, {
'flags': 'a'});
let nextDay = startDate;
while (nextDay !== endDate) {
const s3path = path.join(`${req.params.stationId}`, `${nextDay}.csv`);
const file = await this.fileService.s3Handler.getS3Obj(s3path);
await this.completePipe(file, output);
nextDay = await getTomorrow(nextDay);
}
}
completePipe(file, output) {
return new Promise((resolve) => {
file.createReadStream().on('finish', () => {
resolve();
}).on('error', (err) => {
resolve();
}).on('data', (data) => {
console.log(data.toString());
}).pipe(output);
})
}
}
getS3Obj(file) {
return new Promise(async (resolve) => {
const getParams = {
Bucket: this.bucket,
Key: file
};
resolve(this.s3.getObject(getParams, (err) => {
if (err) {
console.log('Error in getS3 object')
}
}));
})
}
Please help me?
Solved it.
Did a couple things:
Added a tag to the pipe method.
stream.pipe(output, {end: false})
Instead of creating a new function for the promise I just put this code in instead:
await new Promise((resolve) => {
stream.once('finish', () => {
resolve();
});
});
But the tag was what made it work, the promise was just a tidy up.
Yay.
I try to process a Video file (stored in Google Firebase storage) through a Google Cloud Function. I have working code that download the entire video files into the NodeJS Google cloud function: await bucket.file(filePath).download({ destination: tempFile }).
But the goal is only to read the framerate, therefore the headers of the videofile would suffice. But createReadStream gives me an empty tempFile. Any advise much appreciated!
exports.checkFramerate = functions.region('europe-west1').storage.object().onFinalize(async (object, context) => {
const bucket = admin.storage().bucket(object.bucket); // Bucket class
const filePath = object.name; // videos/xbEXdMNFb1Blbd9r2E8m/comp_test.mp4
const fileName = filePath.split('/').pop(); // comp_test.mp4
const bucketDir = path.dirname(filePath); // videos/xbEXdMNFb1Blbd9r2E8m
const tempFile = path.join(os.tmpdir(), 'temp.mp4')
fs.closeSync(fs.openSync(tempFile, 'w'))
console.log("tempFile size1", fs.statSync(tempFile).size)
// await bucket.file(filePath).download({ destination: tempFile }); // this works: tempFile size2 = 3180152
await bucket.file(filePath).createReadStream({ // this does not work: tempFile size2 = 0
start: 10000,
end: 20000
})
.on('error', function(err) {console.log(err)})
.pipe(fs.createWriteStream(tempFile));
console.log("tempFile size2", fs.statSync(tempFile).size)
mi(tempFile).then(data => {
console.log("frameRate", data[0].general.frame_rate[0])
return data[0].general.frame_rate[0];
}).catch(e => {console.error(e)});
});
I tried implementing even the example of https://googleapis.dev/nodejs/storage/latest/File.html#createReadStream but to no avail. remoteFile.download works beautifully but remoteFile.createReadStream gives me empty files...
const remoteFile = bucket.file(filePath);
const localFilename = tempFile;
remoteFile.createReadStream()
.on('error', function(err) {})
.on('response', function(response) {})
.on('end', function() {})
.pipe(fs.createWriteStream(localFilename));
fs.stat(localFilename, (err, stats) => {
if (err) {console.log(err)}
return console.log("stats async",stats.size)
})
as mentioned, promise should be used
reading json file example
let buf = '';
const loadData = async () => {
return await new Promise((resolve, reject) => {
storage.bucket('bucket-name').file('test-config.json')
.createReadStream()
.on('error', reject)
.on('data', function(d) {
buf += d;
}).on('end', function() {
resolve(buf)
});
})
}
const data = await loadData()
Your problem is that the stream API isn't promisifed. So, the await does nothing, and your function continues before the stream is piped, and the file is still zero-length when you stat it the second time.
The download method works just fine because it returns a Promise.
This answer outlines the general approach you need to take. In summary though, you basically want the section of your code that does the piping to read like this:
const stream = bucket.file(filePath).createReadStream({
start: 10000,
end: 20000
})
.pipe(fs.createWriteStream(tempFile));
await new Promise((resolve, reject) => {
stream.on('finish', resolve);
stream.on('error', reject);
});
console.log("tempFile size2", fs.statSync(tempFile).size)
Your function will then wait until the finish event occurs when the piping is complete and the stream is closed. Obviously you probably want to do something more clever with the error handler too, but this is the general form of what you need.
I know that there are better ways to do this and tools such as serve-index already exist as an npm package to list a directory structure, but I wanted to test this out on my as I am still rather new to using Node.js and JavaScript.
Currently, my code is as follows, and I am confused as to why my array is empty.
let directoryTree = [];
let directoryList = new Promise((resolve, reject) => {
resolve(fs.readdir(path.join(__dirname, '..'), (err, fileList) => {
for (file of fileList) {
console.log(file);
directoryTree.push(file);
}
}));
}).then(() => console.log(directoryTree))
.catch((err) => console.log(error));
What gets displayed is
[]
app
files
node_modules
package-lock.json
package.json
This indicates to me that the promise is being resolved before readdir finishes executing, otherwise the empty array wouldn't be logged first. I believe that something may be incorrect in my usage of promises then, but I might be wrong. I appreciate any advice!
You can call resolve() to resolve the promise when the callback is done fetching the data asynchronously.
Then when the data is fetched and resolved you would get the list of files in the then() callback:
new Promise((resolve, reject) => {
fs.readdir(path.join(__dirname, '..'), (err, fileList) => {
err ? reject(err) : resolve(fileList); //resolve with list of files
});
})
.then((list) => console.log(list))
.catch((err) => console.log(err));
You're using Promise incorrectly.
The syntax is (from MDN):
const myFirstPromise = new Promise((resolve, reject) => {
// do something asynchronous which eventually calls either:
//
resolve(someValue); // fulfilled
// or
reject("failure reason"); // rejected
});
Refactoring:
let directoryList = new Promise((resolve, reject) => {
fs.readdir(path.join(__dirname, '..'), (err, fileList) => {
if (err) reject(err)
else resolve(fileList)
});
})
// directoryList is a Promise that could reject or resolve
directoryList.then(() => console.log(directoryTree))
.catch((err) => console.log(error));