We Keep Coding

how to bust #providesModule cache

I'm building a REST API that will live in AWS Lambda. I'm using apex to deploy it, and since I'm implementing it in javascript, I'm using webpack and babel to get my ES6+ goodies. I've discovered a problem with respect to #providesModule names, and I haven't been able to work around it.
I just created a new lambda function, and when I compile it, I can see from the output that one module couldn't be found -- but the other one can. Here's a sketch of the filesystem:
functions
├─ database.js #providesModule Database -- this works
├─ configuration.js #providesModule MyConfig -- this does not work
├── entity-post
├── index.js imports both 'Database' & 'MyConfig'
When I compile the lambda, the output indicates that the MyConfig dependency couldn't be resolved, but that Database could be. I've tried replacing the content of configuration.js with dead-simple code, in case resolution is failing on a basic parse error; no luck. I should add that I've got 5 other lambda functions in the same project that use similar imports, and they all work great.
I assume this is some kind of caching problem: the module resolution system hasn't noticed the #providesModule inside configuration.js.
How do I clear that cache? For that matter, which piece of software is even handling these resolutions? I've tried looking around online, but it's not clear whether recognition of #providesModule is a webpack thing, or a babel thing, or a webpack-babel-loader thing, or maybe just a node v5 thing.
I've done a lot of digging, both online and within my project's codebase. Here's what I've learned:
Almost all of the links you'll see online about #providesModule are about react or react-native projects. That's because Facebook's tooling -- specifically HasteMap (I think), part of the react-native packager suite -- explicitly adds support for #providesModule. My project is neither react nor react-native, and it doesn't use any Facebook tools in the build process. My project uses webpack v1.14 and babel v6.22 to build the source.
I've found the error message within webpack's source, at ./node_modules/webpack/lib/dependencies/WebpackMissingModule.js:
exports.moduleCode = function(request) {
return "var e = new Error(" + JSON.stringify("Cannot find module \"" + request + "\"") + "); " +
"e.code = 'MODULE_NOT_FOUND'; " +
"throw e;";
};
Webpack seems to have a term for scripts that declare a module name in this way: 'Labeled Modules'. There are several scripts within webpack's source that deal with labeled modules, but I haven't been able to pin down the code that recognizes a label within a module. My assumption is that the problem exists in the part of the system that populates the module registry, and the labeled module code I've found all seems to be related to accessing those modules within the (already populated) registry.
It's hard to trace this much further within webpack's source code, which is understandably very meta, and distributed across a bajillion files.
Any help troubleshooting this, or links to relevant documentation or framework code that shows how #providesModule is handled, is appreciated. I want to understand why my new module isn't resolving, so I can fix that problem. If my caching theory is off-base, fine.

This may not come as a surprise to most, but webpack doesn't (currently) honor #providesModule directives.
The reason it appeared as though it did is that webpack seems to flatten case when doing filename comparisons, and my filenames generally match the labels that I give them:
database.js == #providesModule Database
assemble.js == #providesModule Assemble
The reason I ran into trouble is that configuration.js != #providesModule MyConfig.
This also means the question I asked originally was off-base, in that I assumed resolution was failing because labels were being cached, when in fact labels were being ignored. Facebook's HasteMap may indeed use a cache, but it wasn't the culprit here.

Flow type checker errors in node_modules/*

I've initialized flow project with flow init in fresh https://github.com/davezuko/react-redux-starter-kit project.
When Flow does it check, it finds several errors in node_modules.
Errors are happening in /* flow */ annotated library files.
It looks like this:
node_modules/editions/source/index.js:33
33: const {name, editions} = require(packagePath)
^^^^^^^^^^^^^^^^^^^^ The parameter passed to require() must be a literal string.
node_modules/fbjs/lib/Deferred.js.flow:60
60: Promise.prototype.done.apply(this._promise, arguments);
^^^^ property `done`. Property not found in
474: declare class Promise<+R> {
^ Promise. See lib: /private/tmp/flow/flowlib_d34ebcf/core.js:474
node_modules/fbjs/lib/shallowEqual.js.flow:29
29: return x !== 0 || 1 / (x: $FlowIssue) === 1 / (y: $FlowIssue);
^^^^^^^^^^ identifier `$FlowIssue`. Could not resolve name
Should I make Flow ignoring those files? I assume it may affect type checking correctness.

Both fbjs and editions are written using Flow. They each have .flowconfig files with various configurations. All the errors that you are seeing are due to your .flowconfig being configured slightly differently.
The easiest fix is to modify your .flowconfig to support the things that edition and fbjs are using.
Adding module.ignore_non_literal_requires=true to the [options] section should fix the first error. By default Flow will error if you pass a variable to require(), since Flow wants to understand the dependency graph. This option relaxes this requirement.
Adding ./node_modules/fbjs/flow/lib to the [libs] section should fix the second error. fbjs uses a non-standard version of Promise, but it does ship with a library definition for that version of Promise.
Adding suppress_type=$FlowIssue to the [options] section should fix the third error. This option just aliases the any type to $FlowIssue. It makes it more clear when you're using any to suppress an error.
In the future, the Flow team imagines that Flow users will choose to ignore node_modules/ altogether and instead rely on library definitions from https://github.com/flowtype/flow-typed/, so we're investing in definitions and tooling around flow-typed. This will avoid the sort of situation that you're running into.

I personally like to ignore everything under node_modules by doing this.
[ignore]
.*/node_modules/.*
I then use flow-typed to install or stub all imports
https://github.com/flowtype/flow-typed

You can ignore node_modules by adding below line in .flowconfig in the section [declarations]
<PROJECT_ROOT>/node_modules/.*
Refer below screenshot as well
Below is link of document reference
https://flow.org/en/docs/config/declarations/
It also says
Often third-party libraries have broken type definitions or have type
definitions only compatible with a certain version of Flow. In those
cases it may be useful to use type information from the third-party
libraries without typechecking their contents.

How do you define node_modules as externs in Closure Compiler?

I have a Node.js project that I want to compile with Closure Compiler. I do not want it to run in the browser/use browserify. I mainly want the utility of type checking. I originally got the compiler to work correctly using the following:
java -jar compiler.jar -W VERBOSE
--language_in ECMASCRIPT5_STRICT
--externs closure-externs.js
--js="lib/**.js"
Where closure-externs.js manually defined variables and functions which I was using from Node.js in a rather crude way:
// closure-externs.js
/** #constructor */function Buffer(something){}
function require(path){}
var process = {};
[...]
It turns out that this worked only through sheer luck. There is no dependency tracking between files, so you can have cases where you return a type {Foo} and the compiler will complain that it doesn't exist (depending on the machine, depending on the compile order). I then found out I was doing it all wrong and should be using --process_common_js_modules so the compiler will do dependency tracking where I require("foo"). I am currently invoking the compiler like so:
java -jar compiler.jar -W VERBOSE
--language_in ECMASCRIPT5_STRICT
--externs externs/fs.js
--js="lib/**.js"
--process_common_js_modules
--common_js_entry_module app.js
But this is failing with:
ERROR - required entry point "module$crypto" never provided
ERROR - required entry point "module$dgram" never provided
ERROR - required entry point "module$extend" never provided
ERROR - required entry point "module$fs" never provided
ERROR - required entry point "module$net" never provided
ERROR - required entry point "module$q" never provided
Some of these modules are native to Node.js (e.g. fs) whereas others are contained in node_modules like q. I don't want to run these external modules through the compiler, so I know I need to set up externs file(s) for them. I know there is https://github.com/dcodeIO/node.js-closure-compiler-externs for common Node.js externs, and I know how to invoke them on the compiler, but for some reason when I do something like --externs externs/fs.js the error for module$fs remains. What am I doing wrong?
I know there's other flags like --module and --common_js_module_path_prefix but I'm not sure if I need to use them to get this to work or not. My Google-fu has failed to come up with any answers on the correct incantation here. :(

The issue is that you wish for the compiler to somehow recognize that certain require calls are internal, namely that the required module should be processed by the compiler as source, and others are external so should be left alone. There isn't a good way to handle this situation currently.
Workarounds
Use Post-processing to Add External Require Statements
In this scenario you would completely omit any require statements to external modules. The compiler would only process code with internal require statements and modules. After compilation, you would prepend the external require statements:
Header JS To Be Prepended
var crypto = require('crypto');
Source To Be Compiled
console.log(crypto);
Because crypto is declared in an extern, the compiler will correctly recognize the type and symbol name.
Alias Require Calls
When the --process_common_js_modules is specified, the compiler recognizes require statements and expands them in a similar fashion to the way macros work in other languages. By aliasing the require statements that should remain external, the compiler will not recognize them and thus not expand them.
Source To Be Compiled
var externalRequire = require;
/** #suppress {duplicate} this is already defined in externs */
var crypto = externalRequire('crypto');
console.log(crypto)

If you're using the Closure Compiler only for type-checking—i.e., with the --checks-only option—there is another workaround which has the advantage (over the ones mentioned in Chad's answer) of working correctly with unmodified third-party NPM modules which in turn import built-in modules.
Using Stubs
The trick is to create stub NPM modules to stand in for the built-in ones. These can be minimal; they only need to declare the parts of the API you're actually using.
Here's an example for the path built-in module.
In externs/path/path.js I have the "externs" declarations (not actually externs, so e.g. you can't use #nosideeffects) for the part of path that I need:
/** #const */
var path = {};
/**
* #param {string} path
* #return {string}
*/
path.dirname = function(path) {};
/**
* #param {string} path
* #return {string}
*/
path.extname = function(path) {};
/**
* #param {...string} var_args
* #return {string}
*/
path.join = function(var_args) {};
module.exports = path;
In externs/path/package.json, I have a minimal NPM package config:
{
"description": "Fake package.json for require('path')",
"main": "path.js",
"name": "path",
}
Then create a symlink from node_modules/path to externs/path, and add the following to my compiler flags:
node_modules/path/package.json
node_modules/path/path.js
(You could put the stub implementation directly in to node_modules but I prefer to keep my stubs separate from the real modules managed by npm. I just need to remember to manually add the symlinks to my Git repo, because it's otherwise configured to ignore node_modules.)

How to introduce typescript on a big nodeJS code basis step by step?

I do own quite a big code basis implemented in JavaScript based on NodeJS. I do want to give typescript a try and want to implement all new modules in typescript. Just to see how it performs and how I like the idea. I do need to be able to revert back at any time so I do not want to put to much effort into the migration.
I started by using WebStorm and Gulp to support typescript. I created a new module and used the import / require combination.
import settings = require("./settings");
import _ = require("lodash-node");
By doing so I receive a couple of errors.
Error:(22, 27) TS2307: Cannot find external module './settings'.
Error:(23, 20) TS2307: Cannot find external module 'lodash-node'.
Those modules have no typescript definition file and are plain javascript modules (lodash is obviously a external library). I do have a couple of dependencies and I do not want to create definitions manually even if they are just empty stubs. As mentioned I do want to keep the integration as simple as possible.
Is there a general flag or something like that I can set?

The easiest way to proceed (if you don't want type information for a module) is to write:
declare module "lodash-node" {
var notTyped: any; // naming is unimportant here
export = notTyped;
}
Put that in a separate .d.ts file and /// <reference> it. Then _ will be of type any in your implementation file and the compiler won't complain about not know what module you're talking about.

You can use webpack's typescript-loader and start from there. Make sure you put target: 'node' on your webpack config so you don't have all the browser junk.

Webstorm 7 cannot recognize node API methods

I just installed WebStorm. I'm working on a small Node.js app.
I've attached the Node.js source code, and when I click on the Node.js settings, I can see that it can recognize my various node modules, etc.
I'm having two issues:
Unresolved variable or type: WebStorm doesn't seem to recognize simple API methods (exports, require).
No code insight for…: If I call require('winston'), it tells me that it has no code insight. (Is there a way I can add the source code?)

For 2018 and later versions of WebStorm:
In Settings -> Languages & Frameworks -> Node.js and NPM, check Coding assistance for Node.js:
In older Webstorm versions, this was called Enable Node.js Core library.
If you still see unrecognized Node symbols even with that option enabled, unckeck it, restart WebStorm, then right click on the warning and choose Enable Node.js coding assistance or just check the option again. Watch for WebStorm to show it's Indexing files. (Just had this happen today - looks like a WebStorm bug, and what I just wrote fixed the situation.)

For WebStorm 7 thru 10 (on OSX)…
WebStorm → Preferences → Languages & Frameworks → Javascript → Libraries
Select "Node.js Globals" and "Node.js vXXX Core Modules".

I use WebStorm 2020 and I had everything enabled but WebStill though showed that module.exports is unknown function. Then I turned off NodeJS.core library and NodeJS code assistance, applied and then turned them on again. And suddenly it started to work.

As I've answered on the WebStorm says console is an unresolved variable question, to solve these problems on the new Webstorm versions, you need to enable the Coding assistance for Node.js.
To do this, go on the Settings > Languages & Frameworks > Node.js and NPM and click on the Coding assistance for Node.js option, and then click OK to save:
This will all Node.js unresolved variables and functions.
Update
On the new Webstorm versions, just going above error and clicking in More Actions... (or ALT+ENTER) and selecting
Enable Node.js coding assistance will solve this.

Working with WebStorm and Node.js these two code-segments regularly gave me "false positive" warnings:
FALSE POSITIVE WARNING CASE 1:
module.exports = ...
That gave me the warning "Element is not exported". I was able to get rid of the warning caused by that by putting this above the "module.exports = ..."
/** #namespace module.exports **/
FALSE POSITIVE WARNING CASE 2:
let something = global.something ;
That gave me the warning "Unresolved variable or type global". I was able to eliminate the warning caused by that by putting this above it:
/** #namespace global **/
I now put the following on top of my .js- or .mjs-files to be executed by Node.js, and get rid of warnings referring to these variables or properties:
/** #namespace global **/
/** #namespace console **/
/** #namespace process **/
/** #namespace Buffer **/
/** #namespace process.stdin **/
/** #namespace module.exports **/
I am using WebStorm 2022.2.3

Updating to Webstorm 8 or higher will fix your require methods problem. As posted earlier by checking if Settings > JavaScript > Libraries > Node.js are all checked will fix your problem