We Keep Coding

Read environment variables and then replace them in client-side JS when using gulp for building prod or dev code

So lets say I have some code in js
const myApiKey = 'id_0001'
But instead of harcoding it I want to put it in some bash script with other env vars and read from it and then replace it in the JS
So lets say for prod I would read from prod-env.sh or for dev I would read them from dev-env.sh and then gulp or some other tool does the magic and replaces MY_API_KEY based on whatever is established inside of prod-env.sh or dev-env.sh.
const myApiKey = MY_API_KEY
Update: I want to add I only care about unix OS, not concerned about windows. In golang there is way to read for example envVars.get('MY_API_KEY'), I'm looking for something similar but for JS in the client side.

If you're using gulp, it sounds like you could use any gulp string replacer, like gulp-replace.
As for writing the gulp task(s). If you are willing to import the environment into your shell first, before running node, you can access the environment via process.env
gulp.task('build', function(){
gulp.src(['example.js'])
.pipe(replace('MY_API_KEY', process.env.MY_API_KEY))
.pipe(gulp.dest('build/'));
});
If you don't want to import the environment files before running node, you can use a library like env2 to read shell environment files.
Another option would be to use js/json to define those environment files, and load them with require.
prod-env.js
{
"MY_API_KEY": "api_key"
}
gulpfile.js
const myEnv = require('./prod-env')
gulp.task('build', function(){
gulp.src(['example.js'])
.pipe(replace('MY_API_KEY', myEnv.MY_API_KEY))
.pipe(gulp.dest('build/'));
});
Also, for a more generic, loopy version of the replace you can do:
gulp.task('build', function () {
stream = gulp.src(['example.js']);
for (const key in process.env) {
stream.pipe('${' + key + '}', process.env[key]);
}
stream.pipe(gulp.dest('build/'));
});
In that last example I added ${} around the environment variable name to make it less prone to accidents. So the source file becomes:
const myApiKey = ${MY_API_KEY}

This answer is an easy way to do this for someone who doesn't want to touch the code they are managing. For example you are on the ops team but not the dev team and need to do what you are describing.
The environment variable NODE_OPTIONS can control many things about the node.js runtime - see https://nodejs.org/api/cli.html#cli_node_options_options
One such option we can set is --require which allows us to run code before anything else is even loaded.
So using this you can create a overwrite.js file to perform this replacement on any non-node_modules script files:
const fs = require('fs');
const original = fs.readFileSync;
// set some custom env variables
// API_KEY_ENV_VAR - the value to set
// API_KEY_TEMPLATE_TOKEN - the token to replace with the value
if (!process.env.API_KEY_TEMPLATE_TOKEN) {
console.error('Please set API_KEY_TEMPLATE_TOKEN');
process.exit(1);
}
if (!process.env.API_KEY_ENV_VAR) {
console.error('Please set API_KEY_ENV_VAR');
process.exit(1);
}
fs.readFileSync = (file, ...args) => {
if (file.includes('node_modules')) {
return original(file, ...args);
}
const fileContents = original(file, ...args).toString(
/* set encoding here, or let it default to utf-8 */
);
return fileContents
.split(process.env.API_KEY_TEMPLATE_TOKEN)
.join(process.env.API_KEY_ENV_VAR);
};
Then use it with a command like this:
export API_KEY_ENV_VAR=123;
export API_KEY_TEMPLATE_TOKEN=TOKEN;
NODE_OPTIONS="--require ./overwrite.js" node target.js
Supposing you had a script target.js
console.log('TOKEN');
It would log 123. You can use this pretty much universally with node, so it should work fine with gulp, grunt, or any others.

Using RequireJS with node to optimize creating single output file does not include all the required files

I use the FayeJS and the latest version has been modified to use RequireJS, so there is no longer a single file to link into the browser. Instead the structure is as follows:
/adapters
/engines
/mixins
/protocol
/transport
/util
faye_browser.js
I am using the following nodejs build script to try and end up with all the above minified into a single file:
var fs = require('fs-extra'),
requirejs = require('requirejs');
var config = {
baseUrl: 'htdocs/js/dev/faye/'
,name: 'faye_browser'
, out: 'htdocs/js/dev/faye/dist/faye.min.js'
, paths: {
dist: "empty:"
}
,findNestedDependencies: true
};
requirejs.optimize(config, function (buildResponse) {
//buildResponse is just a text output of the modules
//included. Load the built file for the contents.
//Use config.out to get the optimized file contents.
var contents = fs.readFileSync(config.out, 'utf8');
}, function (err) {
//optimization err callback
console.log(err);
});
The content of faye_browser.js is:
'use strict';
var constants = require('./util/constants'),
Logging = require('./mixins/logging');
var Faye = {
VERSION: constants.VERSION,
Client: require('./protocol/client'),
Scheduler: require('./protocol/scheduler')
};
Logging.wrapper = Faye;
module.exports = Faye;
As I under stand it the optimizer should pull in the required files, and then if those files have required files, it should pull in those etc..., and and output a single minified faye.min.js that contains the whole lot, refactored so no additional serverside calls are necessary.
What happens is faye.min.js gets created, but it only contains the content of faye_browser.js, none of the other required files are included.
I have searched all over the web, and looked at a heap of different examples and none of them work for me.
What am I doing wrong here?

For anyone else trying to do this, I mist that on the download page it says:
The Node.js version is available through npm. This package contains a
copy of the browser client, which is served up by the Faye server when
running.
So to get it you have to pull down the code via NPM and then go into the NPM install dir and it is in the "client" dir...

npm global packages: Reference content files from package

I'm in the process of building an npm package which will be installed globally. Is it possible to have non-code files installed alongside code files that can be referenced from code files?
For example, if my package includes someTextFile.txt and a module.js file (and my package.json includes "bin": {"someCommand":"./module.js"}) can I read the contents of someTextFile.txt into memory in module.js? How would I do that?

The following is an example of a module that loads the contents of a file (string) into the global scope.
core.js : the main module file (entry point of package.json)
//:Understanding: module.exports
module.exports = {
reload:(cb)=>{ console.log("[>] Magick reloading to memory"); ReadSpellBook(cb)}
}
//:Understanding: global object
//the following function is only accesible by the magick module
const ReadSpellBook=(cb)=>{
require('fs').readFile(__dirname+"/spellBook.txt","utf8",(e,theSpells)=>{
if(e){ console.log("[!] The Spell Book is MISSING!\n"); cb(e)}
else{
console.log("[*] Reading Spell Book")
//since we want to make the contents of .txt accesible :
global.SpellBook = theSpells // global.SpellBook is now shared accross all the code (global scope)
cb()//callBack
}
})
}
//·: Initialize :.
console.log("[+] Time for some Magick!")
ReadSpellBook((e)=>e?console.log(e):console.log(SpellBook))
spellBook.txt
ᚠ ᚡ ᚢ ᚣ ᚤ ᚥ ᚦ ᚧ ᚨ ᚩ ᚪ ᚫ ᚬ ᚭ ᚮ ᚯ
ᚰ ᚱ ᚲ ᚳ ᚴ ᚵ ᚶ ᚷ ᚸ ᚹ ᚺ ᚻ ᚼ ᚽ ᚾ ᚿ
ᛀ ᛁ ᛂ ᛃ ᛄ ᛅ ᛆ ᛇ ᛈ ᛉ ᛊ ᛋ ᛌ ᛍ ᛎ ᛏ
ᛐ ᛑ ᛒ ᛓ ᛔ ᛕ ᛖ ᛗ ᛘ ᛙ ᛚ ᛛ ᛜ ᛝ ᛞ ᛟ
ᛠ ᛡ ᛢ ᛣ ᛤ ᛥ ᛦ ᛧ ᛨ ᛩ ᛪ ᛫ ᛬ ᛭ ᛮ ᛯ
If you require it from another piece of code, you will see how it prints to the console and initializes by itself.
If you want to achieve a manual initalization, simply remove the 3 last lines (·: Initialize :.) and use reload() :
const magick = require("core.js")
magick.reload((error)=>{ if(error){throw error}else{
//now you know the SpellBook is loaded
console.log(SpellBook.length)
})

I have built some CLIs which were distributed privately, so I believe I can illuminate a bit here.
Let's say your global modules are installed at a directory called $PATH. When your package will be installed on any machine, it will essentially be extracted at that directory.
When you'll fire up someCommand from any terminal, the module.js will be invoked which was kept at $PATH. If you initially kept the template file in the same directory as your package, then it will be present at that location which is local to module.js.
Assuming you edit the template as a string and then want to write it locally to where the user wished / pwd, you just have to use process.cwd() to get the path to that directory. This totally depends on how you code it out.
In case you want to explicitly include the files only in the npm package, then use files attribute of package.json.
As to particularly answer "how can my code file in the npm package locate the path to the globally installed npm folder in which it is located in a way that is guaranteed to work across OSes and is future proof?", that is very very different from the template thingy you were trying to achieve. Anyway, what you're simply asking here is the global path of npm modules. As a fail safe option, use the path returned by require.main.filename within your code to keep that as a reference.

When you npm publish, it packages everything in the folder, excluding things noted in .npmignore. (If you don't have an .npmignore file, it'll dig into .gitignore. See https://docs.npmjs.com/misc/developers#keeping-files-out-of-your-package) So in short, yes, you can package the text file into your module. Installing the module (locally or globally) will get the text file into place in a way you expect.
How do you find the text file once it's installed? __dirname gives you the path of the current file ... if you ask early enough. See https://nodejs.org/docs/latest/api/globals.html#globals_dirname (If you use __dirname inside a closure, it may be the path of the enclosing function.) For the near-term of "future", this doesn't look like it'll change, and will work as expected in all conditions -- whether the module is installed locally or globally, and whether others depend on the module or it's a direct install.
So let's assume the text file is in the same directory as the currently running script:
var fs = require('fs');
var path = require('path');
var dir = __dirname;
function runIt(cb) {
var fullPath = path.combine(__dirname, 'myfile.txt');
fs.readFile(fullPath, 'utf8' , function (e,content) {
if (e) {
return cb(e);
}
// content now has the contents of the file
cb(content);
}
}
module.exports = runIt;
Sweet!

Is there a way to replace some javascript variables (at build time?) with some DEV or RELEASE config data?

I have a really simple website (ASP.NET core) that is a single .html static page and 6 .js files.
In one of the js files are some data that is based on my configuration:
localhost
dev
production
right now, it's hardcoded for my localhost.
Is there way that I can build/package the simple app so that if i say dev or prod in some command line arg, it replaces those values with something from somewhere else?
eg.
in main.js:
var environment = "localhost";
var rooturl = "https://localhost:43210";
and lets imagine i wish to build to my dev server...
var environment = "dev";
var rooturl = "https://pewpew.azurewebsites.com";
Is this possible? To keep things simple, assume I know nothing of JS tools and processes. (it's actually the truth, but lets not tell anyone that).
Update (further clarifications):
with 1x static html file and 6x static JS files, I have a static website. So i'm hoping to generate the js files as static files (still) but with the environment data already compiled in it.

you can use some build tools like grunt. where you can define build task which takes the environment parameter and change the variables to the desired values.
another (more simple) way is to dynamicaly create main.js (with dependency on the environment) file with your backend and the frontend will load it when it starts. src of the script tag can be the asp script, where the output is javascript

This is a snippet from a project in which I do just that. I replace various place holders with values stored in the environment variables.
This example is based on a linux environment, so I used sed to modify the file in-place, however you could just as easily read the file into memory, do the replace and write it back to disk.
grunt.task.registerTask('secretkeys', 'Replace various keys', function() {
var oauth;
try{
oauth = JSON.parse(process.env.oauthKeys).oauth;
}
catch(e){
oauth = {google:{}};
}
var replaces = {
'==GOOGLECLIENTID==':oauth.google.client_id || '{**GOOGLECLIENTID**}',
'==GOOGLESECRETKEY==':oauth.google.client_secret || '{**GOOGLESECRETKEY**}',
'==SECRETKEY==':oauth.secret || '{**SECRETKEY**}',
'==LOCALAUTH==':oauth.login,
};
const child = require('child_process');
grunt.file.expand('bin/**/*.json').forEach(function(file) {
grunt.log.write(`${file} \n`);
for(var key in replaces){
var cmd = 'sed -i s~{{orig}}~{{new}}~g {{file}}'
.replace(/{{file}}/g,file)
.replace(/{{orig}}/g,key.replace(/~/g,'\\~'))
.replace(/{{new}}/g,replaces[key].replace(/~/g,'\\~'))
;
grunt.log.write(` - ${key} \n`);
//grunt.log.write(` ${cmd} \n`);
child.execSync(cmd);
}
});
});
Hopefully you can modify to your purposes.
EDIT : I am reconsidering my answer, you are modifying javascript on a windows environment. You are likely better using PowerShell
(gc script.js) `
.replace("==GOOGLECLIENTID==",$Env:GoogleClientId) `
.replace("==SECRETKEY==",$Env:SecretKey) `
> script-build.js

So after re-reading your question, I realize there is a better solution that I have used in the past. My other answer is still relevant, so I'll leave it.
It may be simplest to just create a config file in the same folder.
<html>
<head>
<script type="text/javascript" src="config.js" ></script>
<script type="text/javascript" src="myscript.js" ></script>
</head>
<body>
ask me your questions, bridgekeeper
</body>
</html>
config.js
var config = {
'colour': 'yellow'
};
myscript.js
var user = prompt("What is your favourite colour?", "");
if(user !== config.colour){
alert("No BLUE! Ahhh....");
}
else{
alert("You may pass");
}
This is the technique I use when developing simple HTA apps for use around the office.

Check out envify. You can run it from the command line. https://github.com/hughsk/envify
sudo npm install -g envify
Say you have
var myVar = process.env.MYVAR;
Run from the command line
MYVAR=somevalue envify input.js > output.js
and the output js file should have
var myVar = 'somevalue';

Using MozMill for testing standalone XUL applications

As a follow-up of this question, I am trying MozMill for testing standalone XUL applications (not a firefox extension). However, I did not "get it" yet - more specifically, how to test a XULRunner-based application.
Consider this app, for example. For now, I can run it from command line, more or less this way:
$ /path/to/xulrunner /full/path/to/mozmillexample/application.ini
I would like to write Mozmill scripts to test it. For example, I would like to write a test such as this one, which has as "taste" of unit testing:
Components.utils.import("chrome://mozmillexample/content/example.js", example);
var setupModule = function(module) {
module.controller = mozmill.getBrowserController(); // Or what?
}
var testEquals = function() {
jumlib.assertEqual(example.exHello(), "Hello, World!", "Should be equal");
}
I would like to write some functional tests, too:
Components.utils.import("chrome://mozmillexample/content/example.js", example);
var setupModule = function(module) {
module.controller = mozmill.getBrowserController(); // Or what?
}
var testAlerts = function() {
var button = findElement.Elem(document.getElementById('button'));
button.click();
controller.window.assertAlert(); // I bet it does not exist but it gives the idea...
}
Unfortunately, however, I did not find any documentation about testing standalone apps, at least none explaining the basic steps. So I ask: is it possible to write tests like these ones? How could I do it, if it is possible?

I got it to work with the MozMill extension; unpack the extension and edit the following files:
add this to install.rdf at the right place:
<em:targetApplication>
<Description>
<em:id>mozmill#example.com</em:id>
<em:minVersion>0.9</em:minVersion>
<em:maxVersion>1.1</em:maxVersion>
</Description>
</em:targetApplication>`
create a folder "extensions" in the application's root (where application.ini and the "chrome" and the "defaults" folder are); paste the unpacked mozmill extension there.
Enable the extension manager as described in MDC
embed the MozMill Extension code in your XULRunner app: <script src="chrome://mozmill/content/overlay.js"/>
Enable the extension by adding or modifying %appdata%\Adam Brandizzi\MozMill Example\Profiles\123455.default\extensions.ini:
[ExtensionDirs]
Extension0=C:\Users\John\Desktop\myXULApp\extensions\mozmill
Use the following JS code to open MozMill: MozMill.onMenuItemCommand(event);