How to zip a directory with node.js [duplicate] - javascript

I need to zip an entire directory using Node.js. I'm currently using node-zip and each time the process runs it generates an invalid ZIP file (as you can see from this Github issue).
Is there another, better, Node.js option that will allow me to ZIP up a directory?
EDIT: I ended up using archiver
writeZip = function(dir,name) {
var zip = new JSZip(),
code = zip.folder(dir),
output = zip.generate(),
filename = ['jsd-',name,'.zip'].join('');
fs.writeFileSync(baseDir + filename, output);
console.log('creating ' + filename);
};
sample value for parameters:
dir = /tmp/jsd-<randomstring>/
name = <randomstring>
UPDATE: For those asking about the implementation I used, here's a link to my downloader:

I ended up using archiver lib. Works great.
Example
var file_system = require('fs');
var archiver = require('archiver');
var output = file_system.createWriteStream('target.zip');
var archive = archiver('zip');
output.on('close', function () {
console.log(archive.pointer() + ' total bytes');
console.log('archiver has been finalized and the output file descriptor has closed.');
});
archive.on('error', function(err){
throw err;
});
archive.pipe(output);
// append files from a sub-directory, putting its contents at the root of archive
archive.directory(source_dir, false);
// append files from a sub-directory and naming it `new-subdir` within the archive
archive.directory('subdir/', 'new-subdir');
archive.finalize();

I'm not going to show something new, just wanted to summarise the solutions above for those who like Promises as much as I do 😉.
const archiver = require('archiver');
/**
* #param {String} sourceDir: /some/folder/to/compress
* #param {String} outPath: /path/to/created.zip
* #returns {Promise}
*/
function zipDirectory(sourceDir, outPath) {
const archive = archiver('zip', { zlib: { level: 9 }});
const stream = fs.createWriteStream(outPath);
return new Promise((resolve, reject) => {
archive
.directory(sourceDir, false)
.on('error', err => reject(err))
.pipe(stream)
;
stream.on('close', () => resolve());
archive.finalize();
});
}
Hope it will help someone 🤞

Use Node's native child_process api to accomplish this.
No need for third party libs. Two lines of code.
const child_process = require("child_process");
child_process.execSync(`zip -r <DESIRED_NAME_OF_ZIP_FILE_HERE> *`, {
cwd: <PATH_TO_FOLDER_YOU_WANT_ZIPPED_HERE>
});
The example above showcases the synchronous API. You can also use child_process.exec(path, options, callback) if you want async behavior. There are a lot more options you can specify other than cwd to further fine-tune your request.
If you don't have the ZIP utility:
This question is specifically asks about the zip utility for archiving/compression purposes. Therefore, this example assumes you have the zip utility installed on your system. For completeness sakes, some operating systems may not have utility installed by default. In that case you have at least three options:
Work with the archiving/compression utility that is native to your platform
Replace the shell command in the above Node.js code with code from your system. For example, linux distros usually come with tar/gzip utilities:
tar -cfz <DESIRED_NAME_OF_ZIP_FILE_HERE> <PATH_TO_FOLDER_YOU_WANT_ZIPPED_HERE>.
This is a nice option as you don't need to install anything new onto your operating system or manage another dependency (kind of the whole point for this answer).
Obtain the zip binary for your OS/distribution.
For example on Ubuntu: apt install zip.
The ZIP utility is tried and tested for decades, it's fairly ubiquitous and it's a safe choice. Do a quick google search or go to the creator, Info-ZIP's, website for downloadable binaries.
Use a third party library/module (of which there are plenty on NPM).
I don't prefer this option. However, if you don't really care to understand the native methods and introducing a new dependency is a non-issue, this is also a valid option.

This is another library which zips the folder in one line :
zip-local
var zipper = require('zip-local');
zipper.sync.zip("./hello/world/").compress().save("pack.zip");

Archive.bulk is now deprecated, the new method to be used for this is glob:
var fileName = 'zipOutput.zip'
var fileOutput = fs.createWriteStream(fileName);
fileOutput.on('close', function () {
console.log(archive.pointer() + ' total bytes');
console.log('archiver has been finalized and the output file descriptor has closed.');
});
archive.pipe(fileOutput);
archive.glob("../dist/**/*"); //some glob pattern here
archive.glob("../dist/.htaccess"); //another glob pattern
// add as many as you like
archive.on('error', function(err){
throw err;
});
archive.finalize();

To include all files and directories:
archive.bulk([
{
expand: true,
cwd: "temp/freewheel-bvi-120",
src: ["**/*"],
dot: true
}
]);
It uses node-glob(https://github.com/isaacs/node-glob) underneath, so any matching expression compatible with that will work.

To pipe the result to the response object (scenarios where there is a need to download the zip rather than store locally)
archive.pipe(res);
Sam's hints for accessing the content of the directory worked for me.
src: ["**/*"]

I have found this small library that encapsulates what you need.
npm install zip-a-folder
const zip-a-folder = require('zip-a-folder');
await zip-a-folder.zip('/path/to/the/folder', '/path/to/archive.zip');
https://www.npmjs.com/package/zip-a-folder

Adm-zip has problems just compressing an existing archive https://github.com/cthackers/adm-zip/issues/64 as well as corruption with compressing binary files.
I've also ran into compression corruption issues with node-zip https://github.com/daraosn/node-zip/issues/4
node-archiver is the only one that seems to work well to compress but it doesn't have any uncompress functionality.

Since archiver is not compatible with the new version of webpack for a long time, I recommend using zip-lib.
var zl = require("zip-lib");
zl.archiveFolder("path/to/folder", "path/to/target.zip").then(function () {
console.log("done");
}, function (err) {
console.log(err);
});

As today, I'm using AdmZip and works great:
import AdmZip = require('adm-zip');
export async function archiveFile() {
try {
const zip = new AdmZip();
const outputDir = "/output_file_dir.zip";
zip.addLocalFolder("./yourFolder")
zip.writeZip(outputDir);
} catch (e) {
console.log(`Something went wrong ${e}`);
}
}

import ... from answer based on https://stackoverflow.com/a/51518100
To zip single directory
import archiver from 'archiver';
import fs from 'fs';
export default zipDirectory;
/**
* From: https://stackoverflow.com/a/51518100
* #param {String} sourceDir: /some/folder/to/compress
* #param {String} outPath: /path/to/created.zip
* #returns {Promise}
*/
function zipDirectory(sourceDir, outPath) {
const archive = archiver('zip', { zlib: { level: 9 }});
const stream = fs.createWriteStream(outPath);
return new Promise((resolve, reject) => {
archive
.directory(sourceDir, false)
.on('error', err => reject(err))
.pipe(stream)
;
stream.on('close', () => resolve());
archive.finalize();
});
}
To zip multiple directories:
import archiver from 'archiver';
import fs from 'fs';
export default zipDirectories;
/**
* Adapted from: https://stackoverflow.com/a/51518100
* #param {String} sourceDir: /some/folder/to/compress
* #param {String} outPath: /path/to/created.zip
* #returns {Promise}
*/
function zipDirectories(sourceDirs, outPath) {
const archive = archiver('zip', { zlib: { level: 9 }});
const stream = fs.createWriteStream(outPath);
return new Promise((resolve, reject) => {
var result = archive;
sourceDirs.forEach(sourceDir => {
result = result.directory(sourceDir, false);
});
result
.on('error', err => reject(err))
.pipe(stream)
;
stream.on('close', () => resolve());
archive.finalize();
});
}

You can try in a simple way:
Install zip-dir :
npm install zip-dir
and use it
var zipdir = require('zip-dir');
let foldername = src_path.split('/').pop()
zipdir(<<src_path>>, { saveTo: 'demo.zip' }, function (err, buffer) {
});

I ended up wrapping archiver to emulate JSZip, as refactoring through my project woult take too much effort. I understand Archiver might not be the best choice, but here you go.
// USAGE:
const zip=JSZipStream.to(myFileLocation)
.onDone(()=>{})
.onError(()=>{});
zip.file('something.txt','My content');
zip.folder('myfolder').file('something-inFolder.txt','My content');
zip.finalize();
// NodeJS file content:
var fs = require('fs');
var path = require('path');
var archiver = require('archiver');
function zipper(archive, settings) {
return {
output: null,
streamToFile(dir) {
const output = fs.createWriteStream(dir);
this.output = output;
archive.pipe(output);
return this;
},
file(location, content) {
if (settings.location) {
location = path.join(settings.location, location);
}
archive.append(content, { name: location });
return this;
},
folder(location) {
if (settings.location) {
location = path.join(settings.location, location);
}
return zipper(archive, { location: location });
},
finalize() {
archive.finalize();
return this;
},
onDone(method) {
this.output.on('close', method);
return this;
},
onError(method) {
this.output.on('error', method);
return this;
}
};
}
exports.JSzipStream = {
to(destination) {
console.log('stream to',destination)
const archive = archiver('zip', {
zlib: { level: 9 } // Sets the compression level.
});
return zipper(archive, {}).streamToFile(destination);
}
};

Related

How do I display all images in a folder in html? [duplicate]

I'm trying to get a list of the names of all the files present in a directory using Node.js. I want output that is an array of filenames. How can I do this?
You can use the fs.readdir or fs.readdirSync methods. fs is included in Node.js core, so there's no need to install anything.
fs.readdir
const testFolder = './tests/';
const fs = require('fs');
fs.readdir(testFolder, (err, files) => {
files.forEach(file => {
console.log(file);
});
});
fs.readdirSync
const testFolder = './tests/';
const fs = require('fs');
fs.readdirSync(testFolder).forEach(file => {
console.log(file);
});
The difference between the two methods, is that the first one is asynchronous, so you have to provide a callback function that will be executed when the read process ends.
The second is synchronous, it will return the file name array, but it will stop any further execution of your code until the read process ends.
IMO the most convenient way to do such tasks is to use a glob tool. Here's a glob package for node.js. Install with
npm install glob
Then use wild card to match filenames (example taken from package's website)
var glob = require("glob")
// options is optional
glob("**/*.js", options, function (er, files) {
// files is an array of filenames.
// If the `nonull` option is set, and nothing
// was found, then files is ["**/*.js"]
// er is an error object or null.
})
If you are planning on using globby here is an example to look for any xml files that are under current folder
var globby = require('globby');
const paths = await globby("**/*.xml");
The answer above does not perform a recursive search into the directory though. Here's what I did for a recursive search (using node-walk: npm install walk)
var walk = require('walk');
var files = [];
// Walker options
var walker = walk.walk('./test', { followLinks: false });
walker.on('file', function(root, stat, next) {
// Add this file to the list of files
files.push(root + '/' + stat.name);
next();
});
walker.on('end', function() {
console.log(files);
});
As of Node v10.10.0, it is possible to use the new withFileTypes option for fs.readdir and fs.readdirSync in combination with the dirent.isDirectory() function to filter for filenames in a directory. That looks like this:
fs.readdirSync('./dirpath', {withFileTypes: true})
.filter(item => !item.isDirectory())
.map(item => item.name)
The returned array is in the form:
['file1.txt', 'file2.txt', 'file3.txt']
Get files in all subdirs
const fs=require('fs');
function getFiles (dir, files_){
files_ = files_ || [];
var files = fs.readdirSync(dir);
for (var i in files){
var name = dir + '/' + files[i];
if (fs.statSync(name).isDirectory()){
getFiles(name, files_);
} else {
files_.push(name);
}
}
return files_;
}
console.log(getFiles('path/to/dir'))
Here's a simple solution using only the native fs and path modules:
// sync version
function walkSync(currentDirPath, callback) {
var fs = require('fs'),
path = require('path');
fs.readdirSync(currentDirPath).forEach(function (name) {
var filePath = path.join(currentDirPath, name);
var stat = fs.statSync(filePath);
if (stat.isFile()) {
callback(filePath, stat);
} else if (stat.isDirectory()) {
walkSync(filePath, callback);
}
});
}
or async version (uses fs.readdir instead):
// async version with basic error handling
function walk(currentDirPath, callback) {
var fs = require('fs'),
path = require('path');
fs.readdir(currentDirPath, function (err, files) {
if (err) {
throw new Error(err);
}
files.forEach(function (name) {
var filePath = path.join(currentDirPath, name);
var stat = fs.statSync(filePath);
if (stat.isFile()) {
callback(filePath, stat);
} else if (stat.isDirectory()) {
walk(filePath, callback);
}
});
});
}
Then you just call (for sync version):
walkSync('path/to/root/dir', function(filePath, stat) {
// do something with "filePath"...
});
or async version:
walk('path/to/root/dir', function(filePath, stat) {
// do something with "filePath"...
});
The difference is in how node blocks while performing the IO. Given that the API above is the same, you could just use the async version to ensure maximum performance.
However there is one advantage to using the synchronous version. It is easier to execute some code as soon as the walk is done, as in the next statement after the walk. With the async version, you would need some extra way of knowing when you are done. Perhaps creating a map of all paths first, then enumerating them. For simple build/util scripts (vs high performance web servers) you could use the sync version without causing any damage.
Using Promises with ES7
Asynchronous use with mz/fs
The mz module provides promisified versions of the core node library. Using them is simple. First install the library...
npm install mz
Then...
const fs = require('mz/fs');
fs.readdir('./myDir').then(listing => console.log(listing))
.catch(err => console.error(err));
Alternatively you can write them in asynchronous functions in ES7:
async function myReaddir () {
try {
const file = await fs.readdir('./myDir/');
}
catch (err) { console.error( err ) }
};
Update for recursive listing
Some of the users have specified a desire to see a recursive listing (though not in the question)... Use fs-promise. It's a thin wrapper around mz.
npm install fs-promise;
then...
const fs = require('fs-promise');
fs.walk('./myDir').then(
listing => listing.forEach(file => console.log(file.path))
).catch(err => console.error(err));
non-recursive version
You don't say you want to do it recursively so I assume you only need direct children of the directory.
Sample code:
const fs = require('fs');
const path = require('path');
fs.readdirSync('your-directory-path')
.filter((file) => fs.lstatSync(path.join(folder, file)).isFile());
Dependencies.
var fs = require('fs');
var path = require('path');
Definition.
// String -> [String]
function fileList(dir) {
return fs.readdirSync(dir).reduce(function(list, file) {
var name = path.join(dir, file);
var isDir = fs.statSync(name).isDirectory();
return list.concat(isDir ? fileList(name) : [name]);
}, []);
}
Usage.
var DIR = '/usr/local/bin';
// 1. List all files in DIR
fileList(DIR);
// => ['/usr/local/bin/babel', '/usr/local/bin/bower', ...]
// 2. List all file names in DIR
fileList(DIR).map((file) => file.split(path.sep).slice(-1)[0]);
// => ['babel', 'bower', ...]
Please note that fileList is way too optimistic. For anything serious, add some error handling.
I'm assuming from your question that you don't want directories names, just files.
Directory Structure Example
animals
├── all.jpg
├── mammals
│ └── cat.jpg
│ └── dog.jpg
└── insects
└── bee.jpg
Walk function
Credits go to Justin Maier in this gist
If you want just an array of the files paths use return_object: false:
const fs = require('fs').promises;
const path = require('path');
async function walk(dir) {
let files = await fs.readdir(dir);
files = await Promise.all(files.map(async file => {
const filePath = path.join(dir, file);
const stats = await fs.stat(filePath);
if (stats.isDirectory()) return walk(filePath);
else if(stats.isFile()) return filePath;
}));
return files.reduce((all, folderContents) => all.concat(folderContents), []);
}
Usage
async function main() {
console.log(await walk('animals'))
}
Output
[
"/animals/all.jpg",
"/animals/mammals/cat.jpg",
"/animals/mammals/dog.jpg",
"/animals/insects/bee.jpg"
];
if someone still search for this, i do this:
import fs from 'fs';
import path from 'path';
const getAllFiles = dir =>
fs.readdirSync(dir).reduce((files, file) => {
const name = path.join(dir, file);
const isDirectory = fs.statSync(name).isDirectory();
return isDirectory ? [...files, ...getAllFiles(name)] : [...files, name];
}, []);
and its work very good for me
Load fs:
const fs = require('fs');
Read files async:
fs.readdir('./dir', function (err, files) {
// "files" is an Array with files names
});
Read files sync:
var files = fs.readdirSync('./dir');
My one liner code:
const fs = require("fs")
const path = 'somePath/'
const filesArray = fs.readdirSync(path).filter(file => fs.lstatSync(path+file).isFile())
its just 2 lines of code:
fs=require('fs')
fs.readdir("./img/", (err,filename)=>console.log(filename))
Image:
Get sorted filenames. You can filter results based on a specific extension such as '.txt', '.jpg' and so on.
import * as fs from 'fs';
import * as Path from 'path';
function getFilenames(path, extension) {
return fs
.readdirSync(path)
.filter(
item =>
fs.statSync(Path.join(path, item)).isFile() &&
(extension === undefined || Path.extname(item) === extension)
)
.sort();
}
My 2 cents if someone:
Just want to list file names (excluding directories) from a local sub-folder on their project
✅ No additional dependencies
✅ 1 function
✅ Normalize path (Unix vs. Windows)
const fs = require("fs");
const path = require("path");
/**
* #param {string} relativeName "resources/foo/goo"
* #return {string[]}
*/
const listFileNames = (relativeName) => {
try {
const folderPath = path.join(process.cwd(), ...relativeName.split("/"));
return fs
.readdirSync(folderPath, { withFileTypes: true })
.filter((dirent) => dirent.isFile())
.map((dirent) => dirent.name.split(".")[0]);
} catch (err) {
// ...
}
};
README.md
package.json
resources
|-- countries
|-- usa.yaml
|-- japan.yaml
|-- gb.yaml
|-- provinces
|-- .........
listFileNames("resources/countries") #=> ["usa", "japan", "gb"]
Try this, it works for me
import fs from "fs/promises";
const path = "path/to/folder";
export const readDir = async function readDir(path) {
const files = await fs.readdir(path);
// array of file names
console.log(files);
}
This is a TypeScript, optionally recursive, optionally error logging and asynchronous solution. You can specify a regular expression for the file names you want to find.
I used fs-extra, because its an easy super set improvement on fs.
import * as FsExtra from 'fs-extra'
/**
* Finds files in the folder that match filePattern, optionally passing back errors .
* If folderDepth isn't specified, only the first level is searched. Otherwise anything up
* to Infinity is supported.
*
* #static
* #param {string} folder The folder to start in.
* #param {string} [filePattern='.*'] A regular expression of the files you want to find.
* #param {(Error[] | undefined)} [errors=undefined]
* #param {number} [folderDepth=0]
* #returns {Promise<string[]>}
* #memberof FileHelper
*/
public static async findFiles(
folder: string,
filePattern: string = '.*',
errors: Error[] | undefined = undefined,
folderDepth: number = 0
): Promise<string[]> {
const results: string[] = []
// Get all files from the folder
let items = await FsExtra.readdir(folder).catch(error => {
if (errors) {
errors.push(error) // Save errors if we wish (e.g. folder perms issues)
}
return results
})
// Go through to the required depth and no further
folderDepth = folderDepth - 1
// Loop through the results, possibly recurse
for (const item of items) {
try {
const fullPath = Path.join(folder, item)
if (
FsExtra.statSync(fullPath).isDirectory() &&
folderDepth > -1)
) {
// Its a folder, recursively get the child folders' files
results.push(
...(await FileHelper.findFiles(fullPath, filePattern, errors, folderDepth))
)
} else {
// Filter by the file name pattern, if there is one
if (filePattern === '.*' || item.search(new RegExp(filePattern, 'i')) > -1) {
results.push(fullPath)
}
}
} catch (error) {
if (errors) {
errors.push(error) // Save errors if we wish
}
}
}
return results
}
Out of the box
In case you want an object with the directory structure out-of-the-box I highly reccomend you to check directory-tree.
Lets say you have this structure:
photos
│ june
│ └── windsurf.jpg
└── january
├── ski.png
└── snowboard.jpg
const dirTree = require("directory-tree");
const tree = dirTree("/path/to/photos");
Will return:
{
path: "photos",
name: "photos",
size: 600,
type: "directory",
children: [
{
path: "photos/june",
name: "june",
size: 400,
type: "directory",
children: [
{
path: "photos/june/windsurf.jpg",
name: "windsurf.jpg",
size: 400,
type: "file",
extension: ".jpg"
}
]
},
{
path: "photos/january",
name: "january",
size: 200,
type: "directory",
children: [
{
path: "photos/january/ski.png",
name: "ski.png",
size: 100,
type: "file",
extension: ".png"
},
{
path: "photos/january/snowboard.jpg",
name: "snowboard.jpg",
size: 100,
type: "file",
extension: ".jpg"
}
]
}
]
}
Custom Object
Otherwise if you want to create an directory tree object with your custom settings have a look at the following snippet. A live example is visible on this codesandbox.
// my-script.js
const fs = require("fs");
const path = require("path");
const isDirectory = filePath => fs.statSync(filePath).isDirectory();
const isFile = filePath => fs.statSync(filePath).isFile();
const getDirectoryDetails = filePath => {
const dirs = fs.readdirSync(filePath);
return {
dirs: dirs.filter(name => isDirectory(path.join(filePath, name))),
files: dirs.filter(name => isFile(path.join(filePath, name)))
};
};
const getFilesRecursively = (parentPath, currentFolder) => {
const currentFolderPath = path.join(parentPath, currentFolder);
let currentDirectoryDetails = getDirectoryDetails(currentFolderPath);
const final = {
current_dir: currentFolder,
dirs: currentDirectoryDetails.dirs.map(dir =>
getFilesRecursively(currentFolderPath, dir)
),
files: currentDirectoryDetails.files
};
return final;
};
const getAllFiles = relativePath => {
const fullPath = path.join(__dirname, relativePath);
const parentDirectoryPath = path.dirname(fullPath);
const leafDirectory = path.basename(fullPath);
const allFiles = getFilesRecursively(parentDirectoryPath, leafDirectory);
return allFiles;
};
module.exports = { getAllFiles };
Then you can simply do:
// another-file.js
const { getAllFiles } = require("path/to/my-script");
const allFiles = getAllFiles("/path/to/my-directory");
Here's an asynchronous recursive version.
function ( path, callback){
// the callback gets ( err, files) where files is an array of file names
if( typeof callback !== 'function' ) return
var
result = []
, files = [ path.replace( /\/\s*$/, '' ) ]
function traverseFiles (){
if( files.length ) {
var name = files.shift()
fs.stat(name, function( err, stats){
if( err ){
if( err.errno == 34 ) traverseFiles()
// in case there's broken symbolic links or a bad path
// skip file instead of sending error
else callback(err)
}
else if ( stats.isDirectory() ) fs.readdir( name, function( err, files2 ){
if( err ) callback(err)
else {
files = files2
.map( function( file ){ return name + '/' + file } )
.concat( files )
traverseFiles()
}
})
else{
result.push(name)
traverseFiles()
}
})
}
else callback( null, result )
}
traverseFiles()
}
Took the general approach of #Hunan-Rostomyan, made it a litle more concise and added excludeDirs argument. It'd be trivial to extend with includeDirs, just follow same pattern:
import * as fs from 'fs';
import * as path from 'path';
function fileList(dir, excludeDirs?) {
return fs.readdirSync(dir).reduce(function (list, file) {
const name = path.join(dir, file);
if (fs.statSync(name).isDirectory()) {
if (excludeDirs && excludeDirs.length) {
excludeDirs = excludeDirs.map(d => path.normalize(d));
const idx = name.indexOf(path.sep);
const directory = name.slice(0, idx === -1 ? name.length : idx);
if (excludeDirs.indexOf(directory) !== -1)
return list;
}
return list.concat(fileList(name, excludeDirs));
}
return list.concat([name]);
}, []);
}
Example usage:
console.log(fileList('.', ['node_modules', 'typings', 'bower_components']));
Using flatMap:
function getFiles(dir) {
return fs.readdirSync(dir).flatMap((item) => {
const path = `${dir}/${item}`;
if (fs.statSync(path).isDirectory()) {
return getFiles(path);
}
return path;
});
}
Given the following directory:
dist
├── 404.html
├── app-AHOLRMYQ.js
├── img
│ ├── demo.gif
│ └── start.png
├── index.html
└── sw.js
Usage:
getFiles("dist")
Output:
[
'dist/404.html',
'dist/app-AHOLRMYQ.js',
'dist/img/demo.gif',
'dist/img/start.png',
'dist/index.html'
]
I usually use: FS-Extra.
const fileNameArray = Fse.readdir('/some/path');
Result:
[
"b7c8a93c-45b3-4de8-b9b5-a0bf28fb986e.jpg",
"daeb1c5b-809f-4434-8fd9-410140789933.jpg"
]
Just a heads up: if you're planning to perform operations on each file in a directory, try vinyl-fs (which is used by gulp, the streaming build system).
This will work and store the result in test.txt file which will be present in the same directory
fs.readdirSync(__dirname).forEach(file => {
fs.appendFileSync("test.txt", file+"\n", function(err){
})
})
I've recently built a tool for this that does just this... It fetches a directory asynchronously and returns a list of items. You can either get directories, files or both, with folders being first. You can also paginate the data in case where you don't want to fetch the entire folder.
https://www.npmjs.com/package/fs-browser
This is the link, hope it helps someone!
I made a node module to automate this task: mddir
Usage
node mddir "../relative/path/"
To install: npm install mddir -g
To generate markdown for current directory: mddir
To generate for any absolute path: mddir /absolute/path
To generate for a relative path: mddir ~/Documents/whatever.
The md file gets generated in your working directory.
Currently ignores node_modules, and .git folders.
Troubleshooting
If you receive the error 'node\r: No such file or directory', the issue is that your operating system uses different line endings and mddir can't parse them without you explicitly setting the line ending style to Unix. This usually affects Windows, but also some versions of Linux. Setting line endings to Unix style has to be performed within the mddir npm global bin folder.
Line endings fix
Get npm bin folder path with:
npm config get prefix
Cd into that folder
brew install dos2unix
dos2unix lib/node_modules/mddir/src/mddir.js
This converts line endings to Unix instead of Dos
Then run as normal with: node mddir "../relative/path/".
Example generated markdown file structure 'directoryList.md'
|-- .bowerrc
|-- .jshintrc
|-- .jshintrc2
|-- Gruntfile.js
|-- README.md
|-- bower.json
|-- karma.conf.js
|-- package.json
|-- app
|-- app.js
|-- db.js
|-- directoryList.md
|-- index.html
|-- mddir.js
|-- routing.js
|-- server.js
|-- _api
|-- api.groups.js
|-- api.posts.js
|-- api.users.js
|-- api.widgets.js
|-- _components
|-- directives
|-- directives.module.js
|-- vendor
|-- directive.draganddrop.js
|-- helpers
|-- helpers.module.js
|-- proprietary
|-- factory.actionDispatcher.js
|-- services
|-- services.cardTemplates.js
|-- services.cards.js
|-- services.groups.js
|-- services.posts.js
|-- services.users.js
|-- services.widgets.js
|-- _mocks
|-- mocks.groups.js
|-- mocks.posts.js
|-- mocks.users.js
|-- mocks.widgets.js
Use npm list-contents module. It reads the contents and sub-contents of the given directory and returns the list of files' and folders' paths.
const list = require('list-contents');
list("./dist",(o)=>{
if(o.error) throw o.error;
console.log('Folders: ', o.dirs);
console.log('Files: ', o.files);
});
If many of the above options seem too complex or not what you are looking for here is another approach using node-dir - https://github.com/fshost/node-dir
npm install node-dir
Here is a somple function to list all .xml files searching in subdirectories
import * as nDir from 'node-dir' ;
listXMLs(rootFolderPath) {
let xmlFiles ;
nDir.files(rootFolderPath, function(err, items) {
xmlFiles = items.filter(i => {
return path.extname(i) === '.xml' ;
}) ;
console.log(xmlFiles) ;
});
}
function getFilesRecursiveSync(dir, fileList, optionalFilterFunction) {
if (!fileList) {
grunt.log.error("Variable 'fileList' is undefined or NULL.");
return;
}
var files = fs.readdirSync(dir);
for (var i in files) {
if (!files.hasOwnProperty(i)) continue;
var name = dir + '/' + files[i];
if (fs.statSync(name).isDirectory()) {
getFilesRecursiveSync(name, fileList, optionalFilterFunction);
} else {
if (optionalFilterFunction && optionalFilterFunction(name) !== true)
continue;
fileList.push(name);
}
}
}

How to handle situation with node.js streams when one stream is dependant on another one?

I am developing a functionality for bulk uploading and I came with this issue.
I want to archive files and that will be uploaded to my server. Also the archive will contain a manifest file - which will describe each file with various properties / meta data / etc.
The issue occurs when I want to send back the response. The stream which is reading the manifest file is closed which leads to immediate callback execution. Bellow I will show the examples.
const csv = require("fast-csv");
const fs = require("fs");
const path = require("path");
async function proccesUpload() {
const manifestReadStream = fs.createReadStream(
path.join(__dirname, "manifest.txt")
);
manifestReadStream
.pipe(
csv.parse({
delimiter: ";",
})
)
.on("data", async (row) => {
// do processing for each file described in manifest file
const hash = crypto.createHash("sha1");
const rs = fs.createReadStream(targetFile, {
flags: "r",
autoClose: true,
});
rs.on("data", (data) => hash.update(data, "utf-8"));
rs.on("close", function onReadStreamClose() {
// do proccessing for file
});
})
.on("end", async () => {
// return response when all formating was performed
});
}
By using nest read stream, the on "end" is executed before all the files are processed.
How can I solve this?
I recommend using async iterators will make the code easier and callback free
async function proccesUpload() {
const manifestReadStream = fs.createReadStream(
path.join(__dirname, "manifest.txt")
);
const parserStream = manifestReadStream.pipe(
csv.parse({
delimiter: ";",
})
);
for await (const row of parserStream) {
// do processing for each file described in manifest file
const hash = crypto.createHash("sha1");
const rs = fs.createReadStream(targetFile, {
flags: "r",
autoClose: true,
});
for await (const data of rs) {
hash.update(data, "utf-8");
}
// DONE PROCESSING THE ROW
}
// DONE PROCESSING ALL FILES
// return response when all formating was performed
}

Outputting file details using ffprobe in ffmpeg AWS Lambda layer

I am trying to output the details of an audio file with ffmpeg using the ffprobe option. But it is just returning 'null' at the moment? I have added the ffmpeg layer in Lambda. can anyone spot why this is not working?
const { spawnSync } = require("child_process");
const { readFileSync, writeFileSync, unlinkSync } = require("fs");
const util = require('util');
var fs = require('fs');
let path = require("path");
exports.handler = (event, context, callback) => {
spawnSync(
"/opt/bin/ffprobe",
[
`var/task/myaudio.flac`
],
{ stdio: "inherit" }
);
};
This is the official AWS Lambda layer I am using, it is a great prooject but a little lacking in documentation.
https://github.com/serverlesspub/ffmpeg-aws-lambda-layer
First of all, I would recommend using NodeJS 8.10 over NodeJs 6.10 (it will be soon EOL, although AWS is unclear on how long it will be supported)
Also, I would not use the old style handler with a callback.
A working example below - since it downloads a file from the internet (couldn't be bothered to create a package to deploy on lambda with the file uploaded) give it a bit more time to work.
const { spawnSync } = require('child_process');
const util = require('util');
var fs = require('fs');
let path = require('path');
const https = require('https');
exports.handler = async (event) => {
const source_url = 'https://upload.wikimedia.org/wikipedia/commons/b/b2/Bell-ring.flac'
const target_path = '/tmp/test.flac'
async function downloadFile() {
return new Promise((resolve, reject) => {
const file = fs.createWriteStream(target_path);
const request = https.get(source_url, function(response) {
const stream = response.pipe(file)
stream.on('finish', () => {resolve()})
});
});
}
await downloadFile()
const test = spawnSync('/opt/bin/ffprobe',[
target_path
]);
console.log(test.output.toString('utf8'))
const response = {
statusCode: 200,
body: JSON.stringify([test.output.toString('utf8')]),
};
return response;
}
NB! In production be sure to generate a unique temporary file as instances that the Lambda function run on are often shared from invocation to invocation, you don't want multiple invocations stepping on each others files! When done, delete the temporary file, otherwise you might run out of free space on the instance executing your functions. The /tmp folder can hold 512MB, so it can run out fast if you work with many large flac files
I'm not fully familiar with this layer, however from looking at the git repo of the thumbnail-builder it looks like child_process is a promise, so you should be waiting for it's result using .then(), otherwise it is returning null because it doesn't wait for the result.
So try something like:
return spawnSync(
"/opt/bin/ffprobe",
[
`var/task/myaudio.flac`
],
{ stdio: "inherit" }
).then(result => {
return result;
})
.catch(error => {
//handle error
});

Unable to pass a dynamically generated array of cacheable links to Service Worker

How do I pass a programmatically populated array of links to a service worker script for caching?
I am generating the array in cachelist.js like this:
const fs = require('fs');
const path = require('path');
require('dotenv').config();
var cachedItems = ['/'];
function walkSync(currentDirPath, callback) {
fs.readdirSync(currentDirPath).forEach(function (name) {
var filePath = path.join(currentDirPath, name);
var stat = fs.statSync(filePath);
if (stat.isFile()) {
callback(filePath, stat);
} else if (stat.isDirectory()) {
walkSync(filePath, callback);
}
});
}
walkSync('./pages/', function(filePath, stat) {
cachedItem = filePath.substr(5);
if(cachedItem.indexOf('_') == -1) {
cachedItems.push(cachedItem);
}
});
module.exports = { cachedItems };
And then I'm trying to use this cachedItems array in /offline/serviceWorker.js as follows:
const URLSTOCACHE = require("../cachelist.js");
const CACHE_NAME = "version-0.0.46";
// Call install event
self.addEventListener("install", e => {
e.waitUntil(
caches
.open(CACHE_NAME)
.then(cache => cache.addAll(URLSTOCACHE))
.then(() => self.skipWaiting())
);
});
// Call fetch event
self.addEventListener("fetch", e => {
e.respondWith(
fetch(e.request).catch(() => caches.match(e.request))
)
});
However, this fails with an "Uncaught ReferenceError: require is not defined." Any workaround?
require isn't a built-in browser-side utility. There are various libraries (like RequireJS) and bundlers (which rewrite the require call), but unless you're using one of those, you can't use require browser-side.
If your goal is to read that file in the service worker and add the URLs in it to the cache, use fetch to load it, and then use thme in addAll.

Replace a string in a file with nodejs

I use the md5 grunt task to generate MD5 filenames. Now I want to rename the sources in the HTML file with the new filename in the callback of the task. I wonder what's the easiest way to do this.
You could use simple regex:
var result = fileAsString.replace(/string to be replaced/g, 'replacement');
So...
var fs = require('fs')
fs.readFile(someFile, 'utf8', function (err,data) {
if (err) {
return console.log(err);
}
var result = data.replace(/string to be replaced/g, 'replacement');
fs.writeFile(someFile, result, 'utf8', function (err) {
if (err) return console.log(err);
});
});
Since replace wasn't working for me, I've created a simple npm package replace-in-file to quickly replace text in one or more files. It's partially based on #asgoth's answer.
Edit (3 October 2016): The package now supports promises and globs, and the usage instructions have been updated to reflect this.
Edit (16 March 2018): The package has amassed over 100k monthly downloads now and has been extended with additional features as well as a CLI tool.
Install:
npm install replace-in-file
Require module
const replace = require('replace-in-file');
Specify replacement options
const options = {
//Single file
files: 'path/to/file',
//Multiple files
files: [
'path/to/file',
'path/to/other/file',
],
//Glob(s)
files: [
'path/to/files/*.html',
'another/**/*.path',
],
//Replacement to make (string or regex)
from: /Find me/g,
to: 'Replacement',
};
Asynchronous replacement with promises:
replace(options)
.then(changedFiles => {
console.log('Modified files:', changedFiles.join(', '));
})
.catch(error => {
console.error('Error occurred:', error);
});
Asynchronous replacement with callback:
replace(options, (error, changedFiles) => {
if (error) {
return console.error('Error occurred:', error);
}
console.log('Modified files:', changedFiles.join(', '));
});
Synchronous replacement:
try {
let changedFiles = replace.sync(options);
console.log('Modified files:', changedFiles.join(', '));
}
catch (error) {
console.error('Error occurred:', error);
}
Perhaps the "replace" module (www.npmjs.org/package/replace) also would work for you. It would not require you to read and then write the file.
Adapted from the documentation:
// install:
npm install replace
// require:
var replace = require("replace");
// use:
replace({
regex: "string to be replaced",
replacement: "replacement string",
paths: ['path/to/your/file'],
recursive: true,
silent: true,
});
You can also use the 'sed' function that's part of ShellJS ...
$ npm install [-g] shelljs
require('shelljs/global');
sed('-i', 'search_pattern', 'replace_pattern', file);
Full documentation ...
ShellJS - sed()
ShellJS
If someone wants to use promise based 'fs' module for the task.
const fs = require('fs').promises;
// Below statements must be wrapped inside the 'async' function:
const data = await fs.readFile(someFile, 'utf8');
const result = data.replace(/string to be replaced/g, 'replacement');
await fs.writeFile(someFile, result,'utf8');
You could process the file while being read by using streams. It's just like using buffers but with a more convenient API.
var fs = require('fs');
function searchReplaceFile(regexpFind, replace, cssFileName) {
var file = fs.createReadStream(cssFileName, 'utf8');
var newCss = '';
file.on('data', function (chunk) {
newCss += chunk.toString().replace(regexpFind, replace);
});
file.on('end', function () {
fs.writeFile(cssFileName, newCss, function(err) {
if (err) {
return console.log(err);
} else {
console.log('Updated!');
}
});
});
searchReplaceFile(/foo/g, 'bar', 'file.txt');
On Linux or Mac, keep is simple and just use sed with the shell. No external libraries required. The following code works on Linux.
const shell = require('child_process').execSync
shell(`sed -i "s!oldString!newString!g" ./yourFile.js`)
The sed syntax is a little different on Mac. I can't test it right now, but I believe you just need to add an empty string after the "-i":
const shell = require('child_process').execSync
shell(`sed -i "" "s!oldString!newString!g" ./yourFile.js`)
The "g" after the final "!" makes sed replace all instances on a line. Remove it, and only the first occurrence per line will be replaced.
Expanding on #Sanbor's answer, the most efficient way to do this is to read the original file as a stream, and then also stream each chunk into a new file, and then lastly replace the original file with the new file.
async function findAndReplaceFile(regexFindPattern, replaceValue, originalFile) {
const updatedFile = `${originalFile}.updated`;
return new Promise((resolve, reject) => {
const readStream = fs.createReadStream(originalFile, { encoding: 'utf8', autoClose: true });
const writeStream = fs.createWriteStream(updatedFile, { encoding: 'utf8', autoClose: true });
// For each chunk, do the find & replace, and write it to the new file stream
readStream.on('data', (chunk) => {
chunk = chunk.toString().replace(regexFindPattern, replaceValue);
writeStream.write(chunk);
});
// Once we've finished reading the original file...
readStream.on('end', () => {
writeStream.end(); // emits 'finish' event, executes below statement
});
// Replace the original file with the updated file
writeStream.on('finish', async () => {
try {
await _renameFile(originalFile, updatedFile);
resolve();
} catch (error) {
reject(`Error: Error renaming ${originalFile} to ${updatedFile} => ${error.message}`);
}
});
readStream.on('error', (error) => reject(`Error: Error reading ${originalFile} => ${error.message}`));
writeStream.on('error', (error) => reject(`Error: Error writing to ${updatedFile} => ${error.message}`));
});
}
async function _renameFile(oldPath, newPath) {
return new Promise((resolve, reject) => {
fs.rename(oldPath, newPath, (error) => {
if (error) {
reject(error);
} else {
resolve();
}
});
});
}
// Testing it...
(async () => {
try {
await findAndReplaceFile(/"some regex"/g, "someReplaceValue", "someFilePath");
} catch(error) {
console.log(error);
}
})()
I ran into issues when replacing a small placeholder with a large string of code.
I was doing:
var replaced = original.replace('PLACEHOLDER', largeStringVar);
I figured out the problem was JavaScript's special replacement patterns, described here. Since the code I was using as the replacing string had some $ in it, it was messing up the output.
My solution was to use the function replacement option, which DOES NOT do any special replacement:
var replaced = original.replace('PLACEHOLDER', function() {
return largeStringVar;
});
ES2017/8 for Node 7.6+ with a temporary write file for atomic replacement.
const Promise = require('bluebird')
const fs = Promise.promisifyAll(require('fs'))
async function replaceRegexInFile(file, search, replace){
let contents = await fs.readFileAsync(file, 'utf8')
let replaced_contents = contents.replace(search, replace)
let tmpfile = `${file}.jstmpreplace`
await fs.writeFileAsync(tmpfile, replaced_contents, 'utf8')
await fs.renameAsync(tmpfile, file)
return true
}
Note, only for smallish files as they will be read into memory.
This may help someone:
This is a little different than just a global replace
from the terminal we run
node replace.js
replace.js:
function processFile(inputFile, repString = "../") {
var fs = require('fs'),
readline = require('readline'),
instream = fs.createReadStream(inputFile),
outstream = new (require('stream'))(),
rl = readline.createInterface(instream, outstream);
formatted = '';
const regex = /<xsl:include href="([^"]*)" \/>$/gm;
rl.on('line', function (line) {
let url = '';
let m;
while ((m = regex.exec(line)) !== null) {
// This is necessary to avoid infinite loops with zero-width matches
if (m.index === regex.lastIndex) {
regex.lastIndex++;
}
url = m[1];
}
let re = new RegExp('^.* <xsl:include href="(.*?)" \/>.*$', 'gm');
formatted += line.replace(re, `\t<xsl:include href="${repString}${url}" />`);
formatted += "\n";
});
rl.on('close', function (line) {
fs.writeFile(inputFile, formatted, 'utf8', function (err) {
if (err) return console.log(err);
});
});
}
// path is relative to where your running the command from
processFile('build/some.xslt');
This is what this does.
We have several file that have xml:includes
However in development we need the path to move down a level.
From this
<xsl:include href="common/some.xslt" />
to this
<xsl:include href="../common/some.xslt" />
So we end up running two regx patterns one to get the href and the other to write
there is probably a better way to do this but it work for now.
Thanks
Nomaly, I use tiny-replace-files to replace texts in file or files. This pkg is smaller and lighter...
https://github.com/Rabbitzzc/tiny-replace-files
import { replaceStringInFilesSync } from 'tiny-replace-files'
const options = {
files: 'src/targets/index.js',
from: 'test-plugin',
to: 'self-name',
}
# await
const result = replaceStringInFilesSync(options)
console.info(result)
I would use a duplex stream instead. like documented here nodejs doc duplex streams
A Transform stream is a Duplex stream where the output is computed in
some way from the input.
<p>Please click in the following {{link}} to verify the account</p>
function renderHTML(templatePath: string, object) {
const template = fileSystem.readFileSync(path.join(Application.staticDirectory, templatePath + '.html'), 'utf8');
return template.match(/\{{(.*?)\}}/ig).reduce((acc, binding) => {
const property = binding.substring(2, binding.length - 2);
return `${acc}${template.replace(/\{{(.*?)\}}/, object[property])}`;
}, '');
}
renderHTML(templateName, { link: 'SomeLink' })
for sure you can improve the reading template function to read as stream and compose the bytes by line to make it more efficient

Categories