We Keep Coding

Clarify instructions for learnyounode exercise 6 - MAKE IT MODULAR

I have a hard time following along with the provided instructions. I'm unclear on references made to which file is the module, and which file is the "original program file," which is "your program" and similar.
I can find solutions to this exercise, and I vaguely understand how/why they are solutions, but when I try to build from scratch I have a hard time following along with the provided instructions. I'm unclear on references made to which file is the module, and which file is the "original program file."
My question is: will someone please clarify these instructions for me? Secondly, I'd like to know if it is possible to solve this problem with all of the module import statements in one file -- maybe the "main.js" file? (As opposed to having the fs and path imports within the MAKE IT MODULAR module.) Also, is it possible to console.log from the module file, so that in the main.js file you only have to call the module function with parameters in order to log the filtered results?
I can see scope rules requiring file-specific imports, but am interested to learn what is required or possible.
This lesson begins by referencing the solution to the previous exercise, which is a js file which imports fs and path and defines a function which does a file sort by file extension within a specified directory. The directory name and file extension string are taken as arguments.
MAKE IT MODULAR asks to expand upon the previous answer by putting the sort function into a module and using that module to deliver the same sort results.
Here is my solution to the previous exercise. I understand that I will have to move the "." concat to another function per this exercise's requirements. I also understand I could have used a .forEach method, as was used in the official solution, included below.
var fs = require('fs');
var path = require('path');
var fileType = "." + process.argv[3]
fs.readdir(process.argv[2], function (err, list) {
for (i = 0; i < list.length; i++) {
if (path.extname(list[i]) === fileType) {
console.log(list[i]);
}
}
});
Official:
var folder = process.argv[2]
var ext = '.' + process.argv[3]
fs.readdir(folder, function (err, files) {
if (err) return console.error(err)
files.forEach(function (file) {
if (path.extname(file) === ext) {
console.log(file)
}
})
})
Here is the exercise:
This problem is the same as the previous but introduces the concept of modules. You will need to create two files to solve this.
Create a program that prints a list of files in a given directory, filtered by the extension of the files. The first argument is the directory name and the second argument is the extension filter. Print the list of files (one file per line) to the console. You must use asynchronous I/O.
This "program" is the file from the previous exercise, with two arguments, correct?
You must write a module file to do most of the work. The module must export a single function that takes three arguments: the directory name, the filename extension string and a callback function, in that order. The filename extension argument must be the same as what was passed to your program. Don't turn it into a RegExp or prefix with "." or do anything except pass it to your module where you can do what you need to make your filter work.
This "module file" is the same as referenced in the previous paragraph ("program"), correct? The module file exports the function, which is the file which is imported to something like main.js?
The filename extension argument must be the same as what was passed to your program.
But if the "original program" is the one with the function with two arguments how can the same file be the one with the function with three arguments? And a reference to "that was passed to your program" makes it sound like the "other" file is the program file?
The callback function must be called using the idiomatic node(err, data) convention. This convention stipulates that unless there's an error, the first argument passed to the callback will be null, and the second will be your data. In this exercise, the data will be your filtered list of files, as an Array. If you receive an error, e.g. from your call to fs.readdir(), the callback must be called with the error, and only the error, as the first argument.
My understanding is that both files have callback functions, so I think I'll have to error handle both files.
You must not print directly to the console from your module file, only from your original program.
In the case of an error bubbling up to your original program file, simply check for it and print an informative message to the console.
OK, now I'm really confused... The "original program" IS the "module file", isn't it? As in the "module file" is the one that exports, and is then imported into something like main.js, or a main "program" file, but not the "Original" program, the one that was created in the previous exercise?
And lastly:
These four things are the contract that your module must follow.
Export a single function that takes exactly the arguments described.
Call the callback exactly once with an error or some data as described.
Don't change anything else, like global variables or stdout.
Handle all the errors that may occur and pass them to the callback.
The benefit of having a contract is that your module can be used by anyone who expects this contract. So your module could be used by anyone else who does learnyounode, or the verifier, and just work.
I expect the "module" file to be the "original program" file, with a function with 2 arguments exported, which is imported to a "main program" file with a function with 3 arguments, which prints to the console.
Please help me reconcile the terminology I have in my mental model with the terminology specified in the wording of this lesson. Thank you.

This "program" is the file from the previous exercise, with two arguments, correct?
Yes, you can keep the file from the previous exercise, but edit it. It's still a program that will take two command line arguments, yes. Let's call this file main.js.
This "module file" is the same as referenced in the previous paragraph ("program"), correct?
No. It's a new, separate file. Let's call it filesHelper.js. It's a module that will be imported by the main module ("the program").
The new file will contain much of the code that the program from the previous exercise did contain, yes. You should move it from the main.js into the filesHelper.js, inside that function that the module will export.
The module file exports the function, which is the file which is imported to something like main.js?
Yes. The filesHelper.js module file will declare and export the function, the main.js file will import and call the function. So the code you previously had will now be distributed to the files as following:
// main.js
… // require the filesHelper
… process.argv[2], process.argv[3] … // a function call with a callback
… // error handling
for (… i = 0; i < ….length; i++) { // don't forget to declare the index variable
console.log(list[i]);
}
});
// filesHelper.js
var fs = require('fs');
var path = require('path');
… // a function declaration with three parameters
fs.readdir(process.argv[2], function (err, list) {
… // error handling (delegate to the callback)
…
for (… i = 0; i < list.length; i++) {
if (path.extname(list[i]) === fileType) {
… // build the result array
}
}
… // call the callback with the result
});
… // export the function
Fill in the blanks :-)
I'd like to know if it is possible to solve this problem with all of the module import statements in one file -- maybe the main.js file? (As opposed to having the fs and path imports within the filesHelper.js module.)
Yes, it's kinda possible, but it becomes absolutely unwieldy quickly. This is called "dependency injection", and you'd do it by giving your exported function 5 parameters not only 3, so that you can pass in the fs and path as extra arguments, and don't need to import them in the filesHelper any more.
The downside of this approach is that usual modules have more than just two imports, and having to pass dozens of them to every function from another module that you call is not maintainable. Also, your main.js would need to do all the imports, would need to know about the dependency details of every single module you wrote, and quickly become a god module.
So unless you actually have an application use case to use the same module multiple times with different dependencies, go for the normal declarative module import style.
Is it possible to console.log from the module file, so that in the main.js file you only have to call the module function with parameters in order to log the filtered results?
Yes, it's possible for sure, but then it is no longer modular. The key point to modularity - apart from separation of concerns - is reusability.
Say you had another program, main2, that wanted to read all files with a certain extension from a certain directory, and then send their contents to a printer? You couldn't use the filesHelper for this if you had hardcoded the file name logging, but you can when you have a generic callback that is called with an array of results.

After a few days of research, and using Bergi's clarifying answer above, I've re-authored the question in a way that is perhaps more clear. I still don't think I would have been able to answer this within the context of the learnyounode tutorial, as this exercise introduces numerous concepts that weren't yet made clear in previous exercises, namely the division of logic between the callback placeholder variable name parameter in the module and the callback function definition within the actual parameter being specified as an argument when the function is called.
This problem is the same as the previous but introduces the concept of modules. You will need to create two files to solve this: main.js and filesHelper.js.
The high-level requirements of this lesson are the same as the previous: create a program that prints a list of files in a given directory, filtered by the extension of the files. The first command line argument is the directory name and the second argument is the extension filter. Print the list of files (one file per line) to the console. You must use asynchronous I/O.
You must write a fileHelper.js file to do most of the work. This file is the module file, and will contain much of the code from your solution to the previous exercise. The main difference is the module file must export a single function that takes three arguments: the directory name, the filename extension string and a callback function, in that order. The filename extension argument must be the same as what will be passed to your program via the command line arguments. Don't turn it into a RegExp or prefix with "." or do anything except pass it to your module where you can do what you need to make your filter function work.
To define a single function export, you assign your function to the module.exports object within filesHelper.js, thus overwriting any value which is already there:  
module.exports = function (args) { /* ... */ } 
    Or you can use a named function and assign the name.
function filterFn(args) { /* ... */ }
module.exports = filterFn
To use your new module within main.js, use the require() call in the same way that you require('fs') to load the fs module. The only difference is that for local modules must be prefixed with './'. So, if your file is named mymodule.js then:  
var mymodule = require('./mymodule.js')
The '.js' is optional here and you will often see it omitted. You now have the module.exports object in your module assigned to the mymodule variable within main.js. Since you are exporting a single function, mymodule is a function you can call!  
The third argument of the module (the callback function) will not have a function definition within the exported module function, but rather will have to be defined when provided as an actual parameter when the module function is being called from the main.js program file. (i.e. when calling the function from main.js, the 3 arguments will be file directory, file type filter, and "function (err, data) {/* … */}".) However, instructions for how errors or data should be handled by the callback function will be required within the exported function.
In other words, the details of the "callback" argument/function are shared between the main function and the module function. Instructions within the exported function are responsible for returning either an error message, or the filtered list of files by specifying how callback will be called, either with only one argument in case of an error (callback(err)), or with the first argument null and the second containing the filtered file list as an array (callback(null, list)), while the actual function definition is to be specified when entering actual arguments to the program via main.js.
Example:
function bar (callback) {  
foo(function (err, data) {  
if (err)  
return callback(err) // early return
// ... no error, continue doing cool things with `data`  
// all went well, call callback with `null` for the error argument  
callback(null, data)
})
}  
Furthermore, the callback function, when defined as an actual parameter in main.js, must be called using the idiomatic node(err, data) convention, and in the case of an error bubbling up to your main.js file, simply check for it and print an informative message to the console.
You must not print directly to the console from your module file, only from main.js.  
These four things are the contract that your module must follow.
Export a single function that takes exactly the arguments described.
Call the callback exactly once with either an error or some data as described.
Don't change anything else, like global variables or stdout.  
Handle all the errors that may occur and pass them to the callback.  
The benefit of having a contract is that your module can be used by anyone who expects this contract. So your module could be used by anyone else who does learnyounode, or the verifier, and just work.
Here is my code which passed the tests:
//filesHelper.js
var path = require('path')
var fs = require('fs')
module.exports = function (directory, filterString, callbackFunction) {
fs.readdir(directory, function (err, list) {
if (err) {
return callbackFunction(err)
} else {
var filteredList = []
for (let i = 0; i < list.length; i++) {
if (path.extname(list[i]) === "." + filterString) {
filteredList.push(list[i])
}
}
}
callbackFunction(null, filteredList)
})
}
...
//main.js
var myModule = require('./filesHelper.js')
var directory = process.argv[2]
var filterString = process.argv[3]
myModule(directory, filterString, function (err, data) {
if (err) {
console.log('There was an error: ', err)
} else {
for (const file of data) console.log(file)
}
})

Require multiple times

File my_script.js:
(function() {
console.log("IMPORTED");
})();
Calling this file (run_me.js) should cause IMPORTED to print twice:
require("./my_script");
require("./my_script");
However it only prints once.
How can I change run_me.js so that IMPORTED is printed to the console twice.
Assume for this question, no changes can be made to my_script.js

require() caches its results. So, the first time a module is required, then its initialization code runs. After that, the cache just returns the value of module.exports without running the initialization code again. This is a very desirable feature of node.js modules.
If you want code to be run each time, then you should export a function that you can call after you require it like this:
Your module:
module.exports = function() {
console.log("IMPORTED");
}
Requiring it and running the code each time
require("./my_script")();
require("./my_script")();
Also, please note that there is no reason to use an IIFE in a module. The node.js module is automatically wrapped in a private function already so you don't need to do it again.
As you now say in a comment (but your question does not directly say), if you don't want to edit my_script at all (which is simply the wrong way to solve this issue), then you have to delete the module from the node.js cache before requiring it again which can be done like this:
delete require.cache[require.resolve('./my_script')];
I would not recommend this as a solution. It's not the proper way to code in node.js. It's a hack work-around. And, it is not compatible with ESM modules.

If you use jest and want code to be run each time for testing, you can use jest.isolateModules:
jest.isolateModules(() => {
require("./my_script");
});
jest.isolateModules(() => {
require("./my_script");
});

I don't think it is possible without modifying the myscript.js file. Especially since as you show it, it doesn't export anything.
It will execute the first time you require it (which is why you see "Imported" once), but then nothing will happen on future calls to require because the "cached" value (ie. module.exports) which is returned is empty.
See below for an example of what I think you want (except that myscript.js has been modified). The biggest difference is that in your original myscript.js file the function was actually executed, while in the example below it is just defined, and then actually executed in the require call in the run_me.js file.
File myscript.js:
module.exports = () => console.log("Imported");
File run_me.js:
require('myscript.js')(); // note the () at the end which actually calls the function
require('myscript.js')(); // note the () at the end which actually calls the function

You can use this package, it is an npm module that will clear the cache and load a module from source fresh each time.
https://www.npmjs.com/package/require-uncached
const requireUncached = require('require-uncached');
require('./foo')();
//=> 1
require('./foo')();
//=> 2
requireUncached('./foo')();
//=> 1
requireUncached('./foo')();
//=> 1

Very unusual scope behaviour when I call modernizr.build in Node JS

I'm trying to build several jsons using Modernizr at once, but it appears to break the scope of my function.
It's very hard to explain so have a look at this example, give it a go if you don't believe me:
[1,2,3,4,5].forEach(function(i){
require("modernizr").build({}, function (result) {
console.log(i);
});
})
outputs:
5
5
5
5
5
Instead of the expected 1, 2, 3, 4, 5, as would any similar function.
I have not come across this behaviour before in all my years of coding in ECMAScript like languages, and have built my project (and previous projects) around the idea that you cannot break a function's scope like that.
It breaks any system based on promises or even just simple callbacks.
It's baffled me all day, and I can't find an appropriate fix for it.
I'm have a very hard time even conceptualizing what it is that's causing this to happen.
Please help.
EDIT:
OK, it appears you're all hung up on the forEach...
Here's another example that will make it a little clearer:
function asd(i){
require("modernizr").build({}, function (result) {
console.log(i);
});
}
asd(1);
asd(2);
asd(3);
asd(4);
outputs
4
4
4
4
What on earth is happening?

The issue specific to Modernizr had to to with a global variable being clobbered.
the build command is basically a large requirejs configuration function, all powered by a large config object. There is some basic things that are true, always, that are established at the top of the function
{
optimize: 'none',
generateSourceMaps: false,
optimizeCss: 'none',
useStrict: true,
include: ['modernizr-init'],
fileExclusionRegExp: /^(.git|node_modules|modulizr|media|test)$/,
wrap: {
start: '\n;(function(window, document, undefined){',
end: '})(window, document);'
}
}
Then, since Modernizr works in both the browser and in node without changes, there needs to be a way for it to know if it should be loading its dependencies via the filesystem or via http. So we add some more options like basePath inside of a environment check
if (inBrowser) {
baseRequireConfig.baseUrl = '/i/js/modernizr-git/src';
} else {
baseRequireConfig.baseUrl = __dirname + '/../src';
}
At this point, the config object gets passed into requirejs.config, which wires up require and allows us to start calling build.
Finally, after all of that has been created, we have a build function that also ends up modifying the config object yet again for build specific settings (the actual detects in your build, regex to strip out some AMD crud, etc).
So here is a super simplified pseudocode version of what is ended up happening
var config = {
name: 'modernizr'
}
if (inBrowser) {
config.env = 'browser';
} else {
config.env = 'node';
}
requirejs.config(config);
module.exports = function(config, callback) {
config.out = function (output) {
//code to strip out AMD ceremony, add classPrefix, version, etc
callback(output)
}
requirejs.optimize(config)
}
spot the problem?
Since we are touching the .out method of the config object (whose scope is the entire module, and therefore its context is saved between build() calls) right before we run the asynchronous require.optimize function, the callback you were passing was rewriting the .out method every time build is called.
This should be fixed in a couple hours in Modernizr

The function block is called asynchronously, so this behavior is expected because this call is much slower than the walk of your foreach, so when you reach the function (result) {} block iis already five
Quite the same problem as described in Node.JS: How to pass variables to asynchronous callbacks? here and you should be able to use the same solution
[1,2,3,4,5].forEach(function(i){
(function(i) {
require("modernizr").build({}, function (result) {
console.log(i);
});
})(i);
})
untested but somethign like that should work

wait for an async operation before requiring other modules

In my main.js, I am reading a file asynchronously. Once my file is loaded, I set some objects in GLOBAL namespace and use them in my required modules (using GLOBAL namespace or not is a different story, I am using it anyway).
My required module immediately expects that variable to exist at the time of loading. So how do I make it wait till my file reading is complete in the main.js? Do I simply require module in the callback of readFile? Or there's a better way to do it?
example:
fs.readFile('./file', function (data) {
// do something
GLOBAL.obj = data;
});
require('./module.js');
module.js
obj.someFunction();

Your gut feeling of disliking that solution is understandable. (Your stomach is right). The proper way of cleaning this up (and you should take the time – future-you will thank you for it):
go through every one of your ten modules
for each one go through all the functions it exports
for each function figure out, what globals they actually depend on.
add those as arguments to the function.
if they take a lot of arguments now, consider grouping them into objects, creating useful models.
if a bunch of functions all depend on the same set of variables, you can also consider creating a factory function
create a function that takes the formerly global variables as arguments and wrap all of the module's code into that function.
make that function the single export of your module. It serves as a factory function and creates the context for all the other functions in that module. It should return whatever the module exported before.
Example
// DB used to be a global
module.exports = function(DB) {
function getUser(user, cb) {
DB.get('user', db);
}
return {getUser: getUser};
};
You can then use it like this:
var module = require('module')(DB);
module.getUser(myUser, function(){}};

Yes, just follow the rule #1 of async programming. Stuff that depends on callback happening must be executed in that callback. Since your require depends on the variable set in async, you can only use your require inside it:
fs.readFile('./file', function (data) {
// do something
GLOBAL.obj = data;
require('./module.js');
});

How do I test 'normal' (non-Node specific) JavaScript functions with Mocha?

This seems like it should be extremely simple; however, after two hours of reading and trial-and-error without success, I'm admitting defeat and asking you guys!
I'm trying to use Mocha with Should.js to test some JavaScript functions, but I'm running into scoping issues. I've simplified it down to the most basic of test cases, but I cannot get it working.
I have a file named functions.js, which just contains the following:
function testFunction() {
return 1;
}
And my tests.js (located in the same folder) contents:
require('./functions.js')
describe('tests', function(){
describe('testFunction', function(){
it('should return 1', function(){
testFunction().should.equal(1);
})
})
})
This test fails with a ReferenceError: testFunction is not defined.
I can see why, because most of the examples I've found either attach objects and functions to the Node global object or export them using module.exports—but using either of these approaches means my function code would throw errors in a standard browser situation, where those objects don't exist.
So how can I access standalone functions which are declared in a separate script file from my tests, without using Node-specific syntax?

Thanks to the other answers here, I've got things working.
One thing which wasn't mentioned though—perhaps because it's common knowledge among Noders—was that you need to assign the result of the require call to a variable, so that you can refer to it when calling your exported functions from within the test suite.
Here's my complete code, for future reference:
functions.js:
function testFunction () {
return 1;
}
// If we're running under Node
if (typeof exports !== 'undefined') {
exports.testFunction = testFunction;
}
tests.js:
var myCode = require('./functions')
describe('tests', function(){
describe('testFunction', function(){
it('should return 1', function(){
// Call the exported function from the module
myCode.testFunction().should.equal(1);
})
})
})

require('./functions.js')
That doesn't do anything since you're not exporting anything. What you're expecting is that testFunction is globally available, essentially the same as
global.testFunction = function() {
return 1;
}
You just can't bypass the export/globals mechanism. It's the way node has been designed. There is no implicit global shared context (like window on a browser). Every "global" variable in a module is trapped in it's context.
You should use module.exports. If you intend to share that file with a browser environments, there are ways to make it compatible. For a quick hack just do window.module = {}; jQuery.extend(window, module.exports) in the browser, or if (typeof exports !== 'undefined'){ exports.testFunction = testFunction } for node.

If you want to make any module available through require you should use
module.exports
as you know ;)
there is a solution if you want to use a module in Node and in browser by doing this
function testFunction() { /* code */ }
if (typeof exports !== 'undefined') {
exports.testFunction = testFunction
}
by doing this you will be able to use the file in browser and in node environment