I'm creating a process that converts multiple markdown files into a single pdf. It creates a pdf file for each .md file found in the source directory. Then it merges the individual pdf files into one pdf. It is this last step that is failing saying the individual pdf files do not exist.
const markdownpdf = require('markdown-pdf')
const path = require('path')
const PDFMerge = require('pdf-merge')
const fse = require('fs-extra')
const srcDir = '../manuscript'
const outDir = 'out'
const main = () => {
fse.pathExists(outDir)
.then(() => {
fse.remove(outDir).then(() => {
fse.ensureDir(outDir)
}).then(() => {
return fse.readdir(srcDir)
}).then((srcDirFiles) => {
console.log('source directory file count = ', srcDirFiles.length)
return srcDirFiles.filter(f => path.extname(f) === '.md')
}).then((mdFiles) => {
console.log('number of md files', mdFiles.length);
return mdFiles.map(file => {
const outFileName = `${path.basename(file, '.md')}.pdf`
fse.createReadStream(`${srcDir}/${file}`)
.pipe(markdownpdf())
.pipe(fse.createWriteStream(`${outDir}/${outFileName}`))
return `${outDir}/${outFileName}`
})
}).then(outFiles => {
console.log('number of pdf files created =', outFiles.length)
PDFMerge(outFiles, { output: `${__dirname}/3.pdf` })
})
})
}
main()
If I wrap the PDFMerge() line in setTimeout() it does work
setTimeout(() => {
PDFMerge(outFiles, { output: `${__dirname}/3.pdf` })
}, 1000)
I'm wondering why the setTimeout() is needed and what needs to be changed so it isn't.
I also wrote an async/await version that had the same problem and also worked with setTimeOut()
Edit
In response to Zach Holt's suggestion, here is the async/await version:
const markdownpdf = require('markdown-pdf')
const path = require('path')
const PDFMerge = require('pdf-merge')
const fse = require('fs-extra')
const srcDir = '../manuscript'
const outDir = 'out'
const createPdf = async (file) => {
try {
const outFileName = `${path.basename(file, '.md')}.pdf`
await fse.createReadStream(`${srcDir}/${file}`)
.pipe(markdownpdf())
.pipe(await fse.createWriteStream(`${outDir}/${outFileName}`))
}
catch (e) {
console.log(e)
}
}
const makePdfFiles = (files) => {
files.forEach(file => {
if (path.extname(file) === '.md') {
createPdf(file)
}
})
}
const mergeFiles = async (files) => {
try {
await PDFMerge(files, {output: `${__dirname}/3.pdf`})
}
catch (e) {
console.log(e)
}
}
const addPathToPdfFiles = (files) => {
return files.map(file => {
return `${outDir}/${file}`
})
}
const main = async () => {
try {
const exists = await fse.pathExists(outDir)
if (exists) {
await fse.remove(outDir)
}
await fse.ensureDir(outDir)
const mdFiles = await fse.readdir(srcDir)
const filesMade = await makePdfFiles(mdFiles)
const pdfFiles = await fse.readdir(outDir)
const pdfFilesWithPath = addPathToPdfFiles(pdfFiles)
mergeFiles(pdfFilesWithPath)
// setTimeout(() => {
// mergeFiles(pdfFilesWithPath)
// }, 1000)
} catch (e) {
console.log(e)
}
}
It has the same problem.
I also tried:
const makePdfFiles = files => {
return new Promise((resolve, reject) => {
try {
files.forEach(file => {
if (path.extname(file) === '.md') {
createPdf(file)
}
})
resolve(true)
} catch (e) {
reject(false)
console.log('makePdfFiles ERROR', e)
}
})
}
But it made no difference.
You need to return the promise from ensureDir() to make it wait for it.
I think the issue might be that you're creating a read stream for each of the .md files, but not waiting for the reads to finish before trying to merge outFiles.
You could likely wait until the outFiles length is the same as the number of md files found before merging.
Also, you should stick with async/await for this. It'll keep the code much clearer
Let me over-simplify your code to illustrate the problem:
p1.then(() => {
p2.then().then().then()
}).then(/* ??? */)
which is the same as:
p1.then(() => {
p2.then().then().then()
return undefined
}).then(/* undefined */)
What you need for chaining is to return the inner Promise:
p1.then(() => // no {code block} here, just return value
p2.then().then().then()
).then(/* ??? */)
which is the same as:
p1.then(() => {
p3 = p2.then()
p4 = p3.then()
p5 = p4.then()
return p5
}).then(/* p5 */)
As far as I can tell the original problem was the approach and not the obvious errors correctly pointed out by others. I found a much simpler solution to the overall goal of producing a single pdf from multiple md files.
const markdownpdf = require('markdown-pdf')
const path = require('path')
const fse = require('fs-extra')
const srcDir = '../manuscript'
const filterAndAddPath = (files) => {
try {
const mdFiles = files
.filter(f => path.extname(f) === '.md')
.map(f => `${srcDir}/${f}`)
return mdFiles
}
catch (e) {
console.log('filterAndAddPath', e)
}
}
const main4 = async () => {
const allFiles = await fse.readdir(srcDir)
const mdFiles = filterAndAddPath(allFiles)
const bookPath = 'book.pdf'
markdownpdf()
.concat.from(mdFiles)
.to(bookPath, function() {
console.log('Created', bookPath)
})
}
main4()
Related
Here is a functon that requests images files. How to return a promise when all files are loaded?
function request() {
for (const [src, nodes] of this.icons.entries()) {
fetch(`${this.baseHref}assets/images/${src}`)
.then((res) => res.text())
.then((content: string) => {
nodes.forEach((node: LayerNode) => {
const { icon } = node;
const { color } = icon;
const replaceIcon = getReplaceIcon(this.defPointIcon, color);
const defIcon = getDefIcon(content, color);
this.cachedIcons.set(node, { defIcon, replaceIcon });
});
});
}}
I have tried this:
const promises = [];
for (const [src, nodes] of this.icons.entries()) {
promises.push(fetch(`${this.baseHref}assets/images/${src}`));
}
Then
Promise.all(promises)
.then((res) => res.text())
.then((content: string) => {
nodes.forEach((node: LayerNode) => {
const { icon } = node;
const { color } = icon;
const replaceIcon = getReplaceIcon(this.defPointIcon, color);
const defIcon = getDefIcon(content, color);
this.cachedIcons.set(node, { defIcon, replaceIcon });
});
});
Problem is I lost the nodes.
If your problem is that you lost the reference to nodes, you can make your promise to include it in the result through a closure:
const promises = [];
for (const [src, nodes] of this.icons.entries()) {
promises.push(fetch(`${this.baseHref}assets/images/${src}`).then(result => [result, nodes]));
}
Then you should change the way you read the result:
.then(([res, nodes]) => res.text().then(content => [content, nodes]))
.then(([content, nodes]: [string, any[]]) => {
I'm writing a little proof of concept thing, which downloads all my HTML assets via fetch(). Currently, I query all the tags with a compatible asset and run it through a for loop for each asset type. What I need to do is run a callback function after all the requests in the loops have finished. I tried await but it downloads each asset one by one. How can I do this?
const scripts = document.querySelectorAll("script");
const links = document.querySelectorAll("link");
const images = document.querySelectorAll("img");
for (const script of scripts) {
(async (s) => {
const doIgnore = s.getAttribute("data-ignoreload");
if (doIgnore) return;
const src = s.getAttribute("data-src");
if (!src) {
console.error("Script does not have a data-src prop.");
return;
}
fetch(src)
.then(x => x.text())
.then(r => {
const newScript = document.createElement("script");
newScript.innerHTML = r;
document.body.appendChild(newScript);
})
.catch(err => {
console.error(`Error loading script with src ${src}: ${err}`);
});
})(script);
}
for (const link of links) {
(async (l) => {
const doIgnore = l.getAttribute("data-ignoreload");
if (doIgnore) return;
const rel = l.getAttribute("rel");
if (!(rel == "stylesheet")) {
return;
}
const href = l.getAttribute("data-href");
if (!href) {
console.error("Stylesheet does not have a data-href prop.");
return;
}
fetch(href)
.then(x => x.text())
.then(r => {
const newStyle = document.createElement("style");
newStyle.innerHTML = r;
document.head.append(newStyle);
})
.catch(err => {
console.error(`Error loading stylesheet with href ${href}: ${err}`);
});
})(link);
}
for (const image of images) {
(async (i) => {
const doIgnore = i.getAttribute("data-ignoreload");
if (doIgnore) return;
const src = i.getAttribute("data-src");
if (!src) {
console.error("Image does not have a data-src prop.");
return;
}
fetch(src)
.then(x => x.blob())
.then(r => {
const url = URL.createObjectURL(r);
i.setAttribute("src", url);
})
.catch(err => {
console.error(`Error loading image ${src}: ${err}`);
});
})(image);
}
Push all of your promises into an array, and then use Promise.allSettled(promises).then((results) => {})
Documentation: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/allSettled
Example:
const promises = images.map(async (image) => {
// do some async work
return fetch(whatever); // don't .catch this
})
Promise.allSettled(promises).then((results) => {
// results is a result of errors or successes
})
I would like to know to read the files and search for keyword sample in nodejs.
If keyword found, display the path
const allfiles = [
'C:\\Users\\public',
'C:\\Users\\public\\images',
'C:\\Users\\public\\javascripts\\index1.js',
'C:\\Users\\public\\javascripts\\index2.js'
]
const readFile = (path, opts = 'utf8') =>
new Promise((resolve, reject) => {
try{
let result=[];
fs.readFile(path, opts, (err, data) => {
if (err) reject(err)
else {
if(data.indexOf("sample")>=0){
result.push(data);
resolve(result);
}
}
})
}
catch (e) {
console.log("e", e);
}
})
const run = async () => {
allfiles.forEach(e=>{
const s = await readFile(e);
console.log(s);
})
}
run();
Expected Output
[
'C:\\Users\\public\\javascripts\\index1.js',
'C:\\Users\\public\\javascripts\\index2.js'
]
Some tips:
What happens when "sample" isn't found in readFile?
You're currently pushing the data into result instead of the path.
Think about what you're trying to accomplish with readFile. To me, what you want to do is see if that file has the word "sample", and return true if so and if not return false. So I'd name the function checkIfFileHasSample and have it return a boolean. Then in your run function, in the forEach you have the path, so that is where I'd add the path to a list of results.
Maybe you already realized this, but run is never actually called in your code sample. Ie. run() doesn't happen.
Solution:
You had some syntax errors and a tricky gotcha with async-await with run. For the syntax errors, it'll come with experience, but I'd also recommend using ESLint to help you catch them, as well as making sure your code is always properly indented.
const fs = require("fs");
const allfiles = [
"C:\\Users\\public",
"C:\\Users\\public\\images",
"C:\\Users\\public\\javascripts\\index1.js",
"C:\\Users\\public\\javascripts\\index2.js",
];
const checkIfFileHasSample = (path, opts = "utf8") =>
new Promise((resolve, reject) => {
fs.readFile(path, opts, (err, data) => {
if (err) {
reject(err);
} else {
if (data.includes("sample")) {
resolve(true);
} else {
resolve(false);
}
}
});
});
const run = async () => {
const results = [];
for (let i = 0; i < allFiles.length; i++) {
try {
const file = allFiles[i];
const hasSample = await checkIfFileHasSample(file);
if (hasSample) {
results.push(file);
}
} catch (e) {
console.log(e);
}
}
console.log(results);
};
run();
I'm writing an recursive function, which creates an object tree of selected file directory. My code works, but in the wrong order that I expected. I can't see the output of my code. Here is the code:
const fs = require("fs");
const basePath = process.argv[2];
const result = {};
const isDirectory = path => {
return new Promise((resolve, reject) => {
fs.lstat(path, (err, stats) => {
if (err) reject("No such file or Directory");
resolve(stats.isDirectory());
});
});
};
const createTree = (path, target) => {
return new Promise((reject, resolve) => {
fs.readdir(path, (err, list) => {
for (const item of list) {
const currentLocation = `${path}/${item}`;
isDirectory(currentLocation).then(isDir => {
console.log(result); //I CAN SEE THE RESULT HERE
if (!isDir) {
target[item] = true;
} else {
target[item] = {};
resolve(createTree(currentLocation, target[item]));
}
});
}
});
reject("Somthing went wrong while getting the list of files");
});
};
createTree(basePath, result)
.then(() => console.log("result --->", result)) //BUT NOT HERE
.catch(err => console.log("Consume Error ==>", err));
I also done it with async await, but I'm curious why it doesn't work with promises.
Here is the fully working exaple with async await:
const fs = require("fs");
const basePath = process.argv[2]; //Getting the path
const result = {};
//Function to check my path is exist and it's a directory
const isDirectory = async path => {
try {
const stats = await fs.promises.lstat(path); //Used istat to get access to the "isDirectory()" method
return stats.isDirectory();
} catch (error) {
throw new Error("No such file or Directory");
}
};
//Recursive function that should create the object tree of the file system
const createTree = async (path, target) => {
try {
const list = await fs.promises.readdir(path);
for (const item of list) {
const currentLocation = `${path}/${item}`;
const isDir = await isDirectory(currentLocation);
//If it's a file, then assign it to true
//Otherwise create object of that directory and do the same for it
if (!isDir) {
target[item] = true;
} else {
target[item] = {};
await createTree(currentLocation, target[item]);
}
}
} catch (err) {
console.log("Somthing went wrong while getting the list of files");
}
};
//Consuming the createTree function
(async () => {
try {
await createTree(basePath, result);
console.log(result);
} catch (error) {
console.log(error.message);
}
})();
I'm just curious is it possible to do the same but only with promises.
async and await are simply syntactic sugar that makes it easier to work with Promise-based programs. Any program depending on those keywords can be rewritten not to use them -
// main.js
import { readdir } from "fs/promises"
import { join } from "path"
function createTree (init = ".")
{ const one = path => p =>
p.isDirectory()
? many(join(path, p.name)).then(r => ({ [p.name]: r }))
: { [p.name]: true }
const many = path =>
readdir(path, { withFileTypes: true })
.then(r => Promise.all(r.map(one(path))))
.then(r => Object.assign(...r))
return many(init)
}
createTree(".")
.then(v => console.log(JSON.stringify(v, null, 2)))
.catch(console.error)
Now let's add some sample files so we can see our program working correctly -
$ yard add immutable # (any example package)
$ node main.js
Output -
{
"main.js": true,
"node_modules": {
".yarn-integrity": true,
"immutable": {
"LICENSE": true,
"README.md": true,
"contrib": {
"cursor": {
"README.md": true,
"__tests__": {
"Cursor.ts.skip": true
},
"index.d.ts": true,
"index.js": true
}
},
"dist": {
"immutable-nonambient.d.ts": true,
"immutable.d.ts": true,
"immutable.es.js": true,
"immutable.js": true,
"immutable.js.flow": true,
"immutable.min.js": true
},
"package.json": true
}
},
"package.json": true,
"yarn.lock": true
}
If you would like the init path to be included in the tree, only a small modification is necessary -
// main.js
import { readdir } from "fs/promises"
import { join, basename } from "path" // !
function createTree (init = ".")
{ const one = path => p =>
p.isDirectory()
? many(join(path, p.name)).then(r => ({ [p.name]: r })) // !
: { [p.name]: true }
const many = path =>
readdir(path, { withFileTypes: true })
.then(r => Promise.all(r.map(one(path))))
.then(r => Object.assign(...r)) // !
.then(r => ({ [basename(path)]: Object.assign(...r) })) // !
return many(init)
}
Now the tree contains our initial path -
createTree(".")
.then(v => console.log(JSON.stringify(v, null, 2)))
.catch(console.error)
{ ".": // <- starting path
{ ... }
}
To see how to write this program using async generators, please see the original Q&A.
After calling fetchNotes from the addNote function it shows me undefined as push method is not defined in the addNote function
const fs = require('fs');
const fetchNotes = ()=>{
fs.readFile('data.json',(err,notes)=>{
if(err){
// return empty array if data.json not found
return [];
}else{
// return Object from data found data.json file
return JSON.parse(notes)
}
});
}
const saveNotes = (notes) =>{
fs.writeFile('data.json',JSON.stringify(notes),()=>{
console.log('Notes is successfully saved');
});
}
const addNote = (title, body)=>{
const note = {
title,
body
}
const notes = fetchNotes();
//Push method not defined
notes.push(note);
saveNotes(notes);
return note;
}
module.exports.addNote = addNote;
It returns undefined because when you are returning in the callback you are not exactly returning from the fetchNotes function itself.
Maybe you can use the readFileSync and don't use callback or maybe you can make it a promise and use async/await
const fetchNotes = () => {
return new Promise((res, rej) => {
fs.readFile('data.json', (err, notes) => {
if (err) {
// return empty array if data.json not found
res([]);
} else {
// return Object from data found data.json file
res(JSON.parse(notes));
}
});
});
}
const addNote = async (title, body) => {
const note = {
title,
body
}
const notes = await fetchNotes();
//Push method not defined
notes.push(note);
saveNotes(notes);
return note;
}
Alternatively, you can use utils.promisify
return JSON.parse(notes) does not store this value inside fetchNotes because it is asynchronous, so you get the content of the file later in time.
To do it asynchronously, you can use async/await :
const fetchNotes = () => {
return new Promise((resolve, reject) => {
fs.readFile('data.json', (err,notes) => resolve(JSON.parse(notes)));
})
}
const addNote = async (title, body) => {
// ...
const notes = await fetchNotes();
notes.push(note);
saveNotes(notes);
return note;
}
You can also do it synchronously :
const fetchNotes = () => JSON.parse( fs.readFileSync('data.json') );
const notes = fetchNotes();