Problem Statement
I wrote a JavaScript code to pull data from Uniswap V3 SDK (https://www.npmjs.com/package/#uniswap/v3-sdk) and I would like to open-source this code on GitHub because currently this isn't possible (for specific use cases). I am not an amazing system designer/programmer so please bare with me (I can only code in Python and barely understand JavaScript).
Essentially, I have a JavaScript code that pulls data and I wanted to wrap it in a Python code, so I can then take the data and do other things with the data (create alerts, flask app, etc.)
I am using Ubuntu, JS6, npm v8.19.2 and Node v16.18.1.
In my research, I found two packages that could help enable me wrapping this JS code in Python:
py2js - https://github.com/PiotrDabkowski/Js2Py
stpyv8 - https://github.com/cloudflare/stpyv8
Coding Attempt
py2js attempt
This package only works for JS5 and below, not JS6. Here is a quote from the GitHub that helps explain the workaround for this issue (which uses babel) https://github.com/PiotrDabkowski/Js2Py:
JavaScript 6 support was achieved by using Js2Py to translate javascript library called Babel. Babel translates JS 6 to JS 5 and afterwards Js2Py translates JS 5 to Python.
While trying this solution out, I am getting the following error:
/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/es6/__init__.py:10: UserWarning:
Importing babel.py for the first time - this can take some time.
Please note that currently Javascript 6 in Js2Py is unstable and slow. Use only for tiny scripts!
warnings.warn(
/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/base.py:2854: FutureWarning: Possible nested set at position 5
self.pat = re.compile(
Initialised babel!
Traceback (most recent call last):
File "/home/bobby/uni_balances/main.py", line 104, in <module>
result = js2py.eval_js6(js) # executing JavaScript and converting the result to python string
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/evaljs.py", line 120, in eval_js6
return eval_js(js6_to_js5(js))
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/evaljs.py", line 115, in eval_js
return e.eval(js)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/evaljs.py", line 204, in eval
self.execute(code, use_compilation_plan=use_compilation_plan)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/evaljs.py", line 199, in execute
exec (compiled, self._context)
File "<EvalJS snippet>", line 4, in <module>
File "<EvalJS snippet>", line 3, in PyJs_LONG_0_
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/base.py", line 949, in __call__
return self.call(self.GlobalObject, args)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/base.py", line 1464, in call
return Js(self.code(*args))
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/host/jseval.py", line 17, in Eval
py_code = translate_js(code.to_string().value, '')
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/translators/translator.py", line 70, in translate_js
parsed = parse_fn(js)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/js2py/translators/translator.py", line 62, in pyjsparser_parse_fn
return parser.parse(code)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 3008, in parse
program = self.parseProgram()
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 2974, in parseProgram
body = self.parseScriptBody()
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 2963, in parseScriptBody
statement = self.parseStatementListItem()
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 2110, in parseStatementListItem
return self.parseStatement()
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 2718, in parseStatement
self.consumeSemicolon()
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 1120, in consumeSemicolon
self.throwUnexpectedToken(self.lookahead)
File "/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pyjsparser/parser.py", line 1046, in throwUnexpectedToken
raise self.unexpectedTokenError(token, message)
js2py.internals.simplex.JsException: SyntaxError: Line 34: Unexpected token function
What have I done to try to fix this?
I have updated node and even downgraded to version 5 as some suggested, but nothing worked.
JavaScript Code
Here is the JavaScript code (note that you would need an Alchemy API ID to run this script):
import { JSBI } from "#uniswap/sdk";
import { ethers } from 'ethers';
import * as fs from 'fs';
// ERC20 json abi file
let ERC20Abi = fs.readFileSync('Erc20.json');
const ERC20 = JSON.parse(ERC20Abi);
// V3 pool abi json file
let pool = fs.readFileSync('V3PairAbi.json');
const IUniswapV3PoolABI = JSON.parse(pool);
// V3 factory abi json
let facto = fs.readFileSync('V3factory.json');
const IUniswapV3FactoryABI = JSON.parse(facto);
let NFT = fs.readFileSync('UniV3NFT.json');
const IUniswapV3NFTmanagerABI = JSON.parse(NFT);
const provider = new ethers.providers.JsonRpcProvider(ALCHEMY_API_ID)
// V3 standard addresses (different for celo)
const factory = "0x1F98431c8aD98523631AE4a59f267346ea31F984";
const NFTmanager = "0xC36442b4a4522E871399CD717aBDD847Ab11FE88";
async function getData(tokenID){
let FactoryContract = new ethers.Contract(factory, IUniswapV3FactoryABI, provider);
let NFTContract = new ethers.Contract(NFTmanager, IUniswapV3NFTmanagerABI, provider);
let position = await NFTContract.positions(tokenID);
let token0contract = new ethers.Contract(position.token0, ERC20, provider);
let token1contract = new ethers.Contract(position.token1, ERC20, provider);
let token0Decimal = await token0contract.decimals();
let token1Decimal = await token1contract.decimals();
let token0sym = await token0contract.symbol();
let token1sym = await token1contract.symbol();
let V3pool = await FactoryContract.getPool(position.token0, position.token1, position.fee);
let poolContract = new ethers.Contract(V3pool, IUniswapV3PoolABI, provider);
let slot0 = await poolContract.slot0();
let pairName = token0sym +"/"+ token1sym;
let dict = {"SqrtX96" : slot0.sqrtPriceX96.toString(), "Pair": pairName, "T0d": token0Decimal, "T1d": token1Decimal, "tickLow": position.tickLower, "tickHigh": position.tickUpper, "liquidity": position.liquidity.toString()}
return dict
}
const Q96 = JSBI.exponentiate(JSBI.BigInt(2), JSBI.BigInt(96));
const MIN_TICK = -887272;
const MAX_TICK = 887272;
function getTickAtSqrtRatio(sqrtPriceX96){
let tick = Math.floor(Math.log((sqrtPriceX96/Q96)**2)/Math.log(1.0001));
return tick;
}
async function getTokenAmounts(liquidity,sqrtPriceX96,tickLow,tickHigh,token0Decimal,token1Decimal){
let sqrtRatioA = Math.sqrt(1.0001**tickLow).toFixed(18);
let sqrtRatioB = Math.sqrt(1.0001**tickHigh).toFixed(18);
let currentTick = getTickAtSqrtRatio(sqrtPriceX96);
let sqrtPrice = sqrtPriceX96 / Q96;
let amount0wei = 0;
let amount1wei = 0;
if(currentTick <= tickLow){
amount0wei = Math.floor(liquidity*((sqrtRatioB-sqrtRatioA)/(sqrtRatioA*sqrtRatioB)));
}
if(currentTick > tickHigh){
amount1wei = Math.floor(liquidity*(sqrtRatioB-sqrtRatioA));
}
if(currentTick >= tickLow && currentTick < tickHigh){
amount0wei = Math.floor(liquidity*((sqrtRatioB-sqrtPrice)/(sqrtPrice*sqrtRatioB)));
amount1wei = Math.floor(liquidity*(sqrtPrice-sqrtRatioA));
}
let amount0Human = (amount0wei/(10**token0Decimal)).toFixed(token0Decimal);
let amount1Human = (amount1wei/(10**token1Decimal)).toFixed(token1Decimal);
console.log("Amount Token0 wei: "+amount0wei);
console.log("Amount Token1 wei: "+amount1wei);
console.log("Amount Token0 : "+amount0Human);
console.log("Amount Token1 : "+amount1Human);
return [amount0wei, amount1wei]
}
async function start(positionID){
let data = await getData(positionID);
let tokens = await getTokenAmounts(data.liquidity, data.SqrtX96, data.tickLow, data.tickHigh, data.T0d, data.T1d);
}
start(273381)
// Also it can be used without the position data if you pull the data it will work for any range
getTokenAmounts(12558033400096537032, 20259533801624375790673555415)
Python Code
Here is the python code I wrote using the js2py package. To save space, I removed the JS Code in the js variable below, but you can copy it from above and place it in the triple quotes:
import js2py
js = """
<<< JS CODE COPIED HERE >>>
""".replace("document.write", "return ")
result = js2py.eval_js6(js) # executing JavaScript and converting the result to python string
One other note, you will need to install babel as well to make it work. Babel will help convert the code from JS6 to JS5.
stpyv8 attempt
This package is maintained by Cloudflare, which makes this a much more attractive approach, but I am having difficulty installing the pyv8 package on my machine.
I was able to successfully run:
sudo apt install python3 python3-dev build-essential libboost-dev libboost-system-dev libboost-python-dev libboost-iostreams-dev
The probem is that I now need to run setup.py, which I can't find. I also below, if you look at the pyv8 installation I tried below, it does it automatically, but it throws an error. Seems like I am getting stuck at trying to run setup.py
pyv8 attempt (alternative to stpyv8 I believe)
https://code.google.com/archive/p/pyv8/
When I run pip3 install -v pyv8, I get the following error:
Using pip 22.0.2 from /home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting pyv8
Using cached PyV8-0.5.zip (22 kB)
Running command python setup.py egg_info
running egg_info
creating /tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info
writing /tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/PKG-INFO
writing dependency_links to /tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/dependency_links.txt
writing top-level names to /tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/top_level.txt
writing manifest file '/tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/SOURCES.txt'
reading manifest file '/tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/SOURCES.txt'
writing manifest file '/tmp/pip-pip-egg-info-g__o7shf/PyV8.egg-info/SOURCES.txt'
Preparing metadata (setup.py) ... done
Using legacy 'setup.py install' for pyv8, since package 'wheel' is not installed.
Installing collected packages: pyv8
Running command Running setup.py install for pyv8
running install
/home/bobby/uni_balances/uni_venv/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.10
copying PyV8.py -> build/lib.linux-x86_64-3.10
running build_ext
building '_PyV8' extension
creating build/temp.linux-x86_64-3.10
creating build/temp.linux-x86_64-3.10/src
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DBOOST_PYTHON_STATIC_LIB -Ilib/python/inc -Ilib/boost/inc -Ilib/v8/inc -I/home/bobby/uni_balances/uni_venv/include -I/usr/include/python3.10 -c src/Context.cpp -o build/temp.linux-x86_64-3.10/src/Context.o
In file included from src/Wrapper.h:8,
from src/Context.h:7,
from src/Context.cpp:1:
src/Exception.h:6:10: fatal error: v8.h: No such file or directory
6 | #include <v8.h>
| ^~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
error: subprocess-exited-with-error
× Running setup.py install for pyv8 did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /home/bobby/uni_balances/uni_venv/bin/python3 -u -c '
exec(compile('"'"''"'"''"'"'
# This is <pip-setuptools-caller> -- a caller that pip uses to run setup.py
#
# - It imports setuptools before invoking setup.py, to enable projects that directly
# import from `distutils.core` to work with newer packaging standards.
# - It provides a clear error message when setuptools is not installed.
# - It sets `sys.argv[0]` to the underlying `setup.py`, when invoking `setup.py` so
# setuptools doesn'"'"'t think the script is `-c`. This avoids the following warning:
# manifest_maker: standard file '"'"'-c'"'"' not found".
# - It generates a shim setup.py, for handling setup.cfg-only projects.
import os, sys, tokenize
try:
import setuptools
except ImportError as error:
print(
"ERROR: Can not execute `setup.py` since setuptools is not available in "
"the build environment.",
file=sys.stderr,
)
sys.exit(1)
__file__ = %r
sys.argv[0] = __file__
if os.path.exists(__file__):
filename = __file__
with tokenize.open(__file__) as f:
setup_py_code = f.read()
else:
filename = "<auto-generated setuptools caller>"
setup_py_code = "from setuptools import setup; setup()"
exec(compile(setup_py_code, filename, "exec"))
'"'"''"'"''"'"' % ('"'"'/tmp/pip-install-b9d0zoul/pyv8_c9dfd2e4ec0e4c66a2bff47f44972004/setup.py'"'"',), "<pip-setuptools-caller>", "exec"))' install --record /tmp/pip-record-pel12p8h/install-record.txt --single-version-externally-managed --compile --install-headers /home/bobby/uni_balances/uni_venv/include/site/python3.10/pyv8
cwd: /tmp/pip-install-b9d0zoul/pyv8_c9dfd2e4ec0e4c66a2bff47f44972004/
Running setup.py install for pyv8 ... error
error: legacy-install-failure
× Encountered error while trying to install package.
╰─> pyv8
note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
Conclusion
I am not married to using the two options I mentioned above, but I would like to pull the data into Python from the JS code.
Also this is the output from the JS code that I need to port over into Python:
I'm trying to get JavaScript to read/write to a PostgreSQL database. I found this project on GitHub. I was able to get the following sample code to run in Node.
var pg = require('pg'); //native libpq bindings = `var pg = require('pg').native`
var conString = "tcp://postgres:1234#localhost/postgres";
var client = new pg.Client(conString);
client.connect();
//queries are queued and executed one after another once the connection becomes available
client.query("CREATE TEMP TABLE beatles(name varchar(10), height integer, birthday timestamptz)");
client.query("INSERT INTO beatles(name, height, birthday) values($1, $2, $3)", ['Ringo', 67, new Date(1945, 11, 2)]);
client.query("INSERT INTO beatles(name, height, birthday) values($1, $2, $3)", ['John', 68, new Date(1944, 10, 13)]);
//queries can be executed either via text/parameter values passed as individual arguments
//or by passing an options object containing text, (optional) parameter values, and (optional) query name
client.query({
name: 'insert beatle',
text: "INSERT INTO beatles(name, height, birthday) values($1, $2, $3)",
values: ['George', 70, new Date(1946, 02, 14)]
});
//subsequent queries with the same name will be executed without re-parsing the query plan by postgres
client.query({
name: 'insert beatle',
values: ['Paul', 63, new Date(1945, 04, 03)]
});
var query = client.query("SELECT * FROM beatles WHERE name = $1", ['John']);
//can stream row results back 1 at a time
query.on('row', function(row) {
console.log(row);
console.log("Beatle name: %s", row.name); //Beatle name: John
console.log("Beatle birth year: %d", row.birthday.getYear()); //dates are returned as javascript dates
console.log("Beatle height: %d' %d\"", Math.floor(row.height/12), row.height%12); //integers are returned as javascript ints
});
//fired after last row is emitted
query.on('end', function() {
client.end();
});
Next I tried to make it run on a webpage, but nothing seemed to happen. I checked on the JavaScript console and it just says "require not defined".
So what is this "require"? Why does it work in Node but not in a webpage?
Also, before I got it to work in Node, I had to do npm install pg. What's that about? I looked in the directory and didn't find a file pg. Where did it put it, and how does JavaScript find it?
So what is this "require?"
require() is not part of the standard JavaScript API. But in Node.js, it's a built-in function with a special purpose: to load modules.
Modules are a way to split an application into separate files instead of having all of your application in one file. This concept is also present in other languages with minor differences in syntax and behavior, like C's include, Python's import, and so on.
One big difference between Node.js modules and browser JavaScript is how one script's code is accessed from another script's code.
In browser JavaScript, scripts are added via the <script> element. When they execute, they all have direct access to the global scope, a "shared space" among all scripts. Any script can freely define/modify/remove/call anything on the global scope.
In Node.js, each module has its own scope. A module cannot directly access things defined in another module unless it chooses to expose them. To expose things from a module, they must be assigned to exports or module.exports. For a module to access another module's exports or module.exports, it must use require().
In your code, var pg = require('pg'); loads the pg module, a PostgreSQL client for Node.js. This allows your code to access functionality of the PostgreSQL client's APIs via the pg variable.
Why does it work in node but not in a webpage?
require(), module.exports and exports are APIs of a module system that is specific to Node.js. Browsers do not implement this module system.
Also, before I got it to work in node, I had to do npm install pg. What's that about?
NPM is a package repository service that hosts published JavaScript modules. npm install is a command that lets you download packages from their repository.
Where did it put it, and how does Javascript find it?
The npm cli puts all the downloaded modules in a node_modules directory where you ran npm install. Node.js has very detailed documentation on how modules find other modules which includes finding a node_modules directory.
Alright, so let's first start with making the distinction between Javascript in a web browser, and Javascript on a server (CommonJS and Node).
Javascript is a language traditionally confined to a web browser with a limited global context defined mostly by what came to be known as the Document Object Model (DOM) level 0 (the Netscape Navigator Javascript API).
Server-side Javascript eliminates that restriction and allows Javascript to call into various pieces of native code (like the Postgres library) and open sockets.
Now require() is a special function call defined as part of the CommonJS spec. In node, it resolves libraries and modules in the Node search path, now usually defined as node_modules in the same directory (or the directory of the invoked javascript file) or the system-wide search path.
To try to answer the rest of your question, we need to use a proxy between the code running in the the browser and the database server.
Since we are discussing Node and you are already familiar with how to run a query from there, it would make sense to use Node as that proxy.
As a simple example, we're going to make a URL that returns a few facts about a Beatle, given a name, as JSON.
/* your connection code */
var express = require('express');
var app = express.createServer();
app.get('/beatles/:name', function(req, res) {
var name = req.params.name || '';
name = name.replace(/[^a-zA_Z]/, '');
if (!name.length) {
res.send({});
} else {
var query = client.query('SELECT * FROM BEATLES WHERE name =\''+name+'\' LIMIT 1');
var data = {};
query.on('row', function(row) {
data = row;
res.send(data);
});
};
});
app.listen(80, '127.0.0.1');
I noticed that whilst the other answers explained what require is and that it is used to load modules in Node they did not give a full reply on how to load node modules when working in the Browser.
It is quite simple to do. Install your module using npm as you describe, and the module itself will be located in a folder usually called node_modules.
Now the simplest way to load it into your app is to reference it from your html with a script tag which points at this directory. i.e if your node_modules directory is in the root of the project at the same level as your index.html you would write this in your index.html:
<script src="node_modules/ng"></script>
That whole script will now be loaded into the page - so you can access its variables and methods directly.
There are other approaches which are more widely used in larger projects, such as a module loader like require.js. Of the two, I have not used Require myself, but I think it is considered by many people the way to go.
It's used to load modules. Let's use a simple example.
In file circle_object.js:
var Circle = function (radius) {
this.radius = radius
}
Circle.PI = 3.14
Circle.prototype = {
area: function () {
return Circle.PI * this.radius * this.radius;
}
}
We can use this via require, like:
node> require('circle_object')
{}
node> Circle
{ [Function] PI: 3.14 }
node> var c = new Circle(3)
{ radius: 3 }
node> c.area()
The require() method is used to load and cache JavaScript modules. So, if you want to load a local, relative JavaScript module into a Node.js application, you can simply use the require() method.
Example:
var yourModule = require( "your_module_name" ); //.js file extension is optional
Necromancing.
IMHO, the existing answers leave much to be desired.
At first, it's very confusing.
You have a (nowhere defined) function "require", which is used to get modules.
And in said (CommonJS) modules, you can use require, exports and module, WITHOUT THEM EVER BEING DEFINED.
Not that it would be new that you could use undefined variables in JS, but you couldn't use an undefined function.
So it looks a little like magic at first.
But all magic is based on deception.
When you dig a little deeper, it turns out it is really quite simple:
Require is simply a (non-standard) function defined at global scope.
(global scope = window-object in browser, global-object in NodeJS).
Note that by default, the "require function" is only implemented in NodeJS, not in the browser.
Also, note that to add to the confusion, for the browser, there is RequireJS, which, despite the name containing the characters "require", RequireJS absolutely does NOT implement require/CommonJS - instead RequireJS implements AMD, which is something similar, but not the same (aka incompatible).
That last one is just one important thing you have to realize on your way to understanding require.
Now, as such, to answer the question "what is require", we "simply" need to know what this function does.
This is perhaps best explained with code.
Here's a simple implementation by Michele Nasti, the code you can find on his github page.
Let's call our minimalisc implementation of the require function "myRequire":
function myRequire(name)
{
console.log(`Evaluating file ${name}`);
if (!(name in myRequire.cache)) {
console.log(`${name} is not in cache; reading from disk`);
let code = fs.readFileSync(name, 'utf8');
let module = { exports: {} };
myRequire.cache[name] = module;
let wrapper = Function("require, exports, module", code);
wrapper(myRequire, module.exports, module);
}
console.log(`${name} is in cache. Returning it...`);
return myRequire.cache[name].exports;
}
myRequire.cache = Object.create(null);
window.require = myRequire;
const stuff = window.require('./main.js');
console.log(stuff);
Now you notice, the object "fs" is used here.
For simplicity's sake, Michele just imported the NodeJS fs module:
const fs = require('fs');
Which wouldn't be necessary.
So in the browser, you could make a simple implementation of require with a SYNCHRONOUS XmlHttpRequest:
const fs = {
file: `
// module.exports = \"Hello World\";
module.exports = function(){ return 5*3;};
`
, getFile(fileName: string, encoding: string): string
{
// https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Synchronous_and_Asynchronous_Requests
let client = new XMLHttpRequest();
// client.setRequestHeader("Content-Type", "text/plain;charset=UTF-8");
// open(method, url, async)
client.open("GET", fileName, false);
client.send();
if (client.status === 200)
return client.responseText;
return null;
}
, readFileSync: function (fileName: string, encoding: string): string
{
// this.getFile(fileName, encoding);
return this.file; // Example, getFile would fetch this file
}
};
Basically, what require thus does, is it downloads a JavaScript-file, evals it in an anonymous namespace (aka Function), with the parameters "require", "exports" and "module", and returns the exports, meaning an object's public functions and properties.
Note that this evaluation is recursive: you require files, which themselfs can require files.
This way, all "global" variables used in your module are variables in the require-wrapper-function namespace, and don't pollute the global scope with unwanted variables.
Also, this way, you can reuse code without depending on namespaces, so you get "modularity" in JavaScript. "modularity" in quotes, because this is not exactly true, though, because you can still write window.bla/global.bla, and hence still pollute the global scope... Also, this establishes a separation between private and public functions, the public functions being the exports.
Now instead of saying
module.exports = function(){ return 5*3;};
You can also say:
function privateSomething()
{
return 42:
}
function privateSomething2()
{
return 21:
}
module.exports = {
getRandomNumber: privateSomething
,getHalfRandomNumber: privateSomething2
};
and return an object.
Also, because your modules get evaluated in a function with parameters
"require", "exports" and "module", your modules can use the undeclared variables "require", "exports" and "module", which might be startling at first. The require parameter there is of course a pointer to the require function saved into a variable.
Cool, right ?
Seen this way, require looses its magic, and becomes simple.
Now, the real require-function will do a few more checks and quirks, of course, but this is the essence of what that boils down to.
Also, in 2020, you should use the ECMA implementations instead of require:
import defaultExport from "module-name";
import * as name from "module-name";
import { export1 } from "module-name";
import { export1 as alias1 } from "module-name";
import { export1 , export2 } from "module-name";
import { foo , bar } from "module-name/path/to/specific/un-exported/file";
import { export1 , export2 as alias2 , [...] } from "module-name";
import defaultExport, { export1 [ , [...] ] } from "module-name";
import defaultExport, * as name from "module-name";
import "module-name";
And if you need a dynamic non-static import (e.g. load a polyfill based on browser-type), there is the ECMA-import function/keyword:
var promise = import("module-name");
note that import is not synchronous like require.
Instead, import is a promise, so
var something = require("something");
becomes
var something = await import("something");
because import returns a promise (asynchronous).
So basically, unlike require, import replaces fs.readFileSync with fs.readFileAsync.
async readFileAsync(fileName, encoding)
{
const textDecoder = new TextDecoder(encoding);
// textDecoder.ignoreBOM = true;
const response = await fetch(fileName);
console.log(response.ok);
console.log(response.status);
console.log(response.statusText);
// let json = await response.json();
// let txt = await response.text();
// let blo:Blob = response.blob();
// let ab:ArrayBuffer = await response.arrayBuffer();
// let fd = await response.formData()
// Read file almost by line
// https://developer.mozilla.org/en-US/docs/Web/API/ReadableStreamDefaultReader/read#Example_2_-_handling_text_line_by_line
let buffer = await response.arrayBuffer();
let file = textDecoder.decode(buffer);
return file;
} // End Function readFileAsync
This of course requires the import-function to be async as well.
"use strict";
async function myRequireAsync(name) {
console.log(`Evaluating file ${name}`);
if (!(name in myRequireAsync.cache)) {
console.log(`${name} is not in cache; reading from disk`);
let code = await fs.readFileAsync(name, 'utf8');
let module = { exports: {} };
myRequireAsync.cache[name] = module;
let wrapper = Function("asyncRequire, exports, module", code);
await wrapper(myRequireAsync, module.exports, module);
}
console.log(`${name} is in cache. Returning it...`);
return myRequireAsync.cache[name].exports;
}
myRequireAsync.cache = Object.create(null);
window.asyncRequire = myRequireAsync;
async () => {
const asyncStuff = await window.asyncRequire('./main.js');
console.log(asyncStuff);
};
Even better, right ?
Well yea, except that there is no ECMA-way to dynamically import synchronously (without promise).
Now, to understand the repercussions, you absolutely might want to read up on promises/async-await here, if you don't know what that is.
But very simply put, if a function returns a promise, it can be "awaited":
"use strict";
function sleep(interval)
{
return new Promise(
function (resolve, reject)
{
let wait = setTimeout(function () {
clearTimeout(wait);
//reject(new Error(`Promise timed out ! (timeout = ${timeout})`));
resolve();
}, interval);
});
}
The promise would then normally be used like this:
function testSleep()
{
sleep(3000).then(function ()
{
console.log("Waited for 3 seconds");
});
}
But when you return a promise, you can also use await, which means we get rid of the callback (sort of - actually, it is being replaced with a state-machine in the compiler/interpreter).
This way, we make asynchronous code feel like synchronous, so we now can use try-catch for error-handling.
Note that if you want to use await in a function, that function must be declared async (hence async-await).
async function testSleep()
{
await sleep(5000);
console.log("i waited 5 seconds");
}
And also please note that in JavaScript, there is no way to call an async function (blockingly) from a synchronous one (the ones you know). So if you want to use await (aka ECMA-import), all your code needs to be async, which most likely is a problem, if everything isn't already async...
An example of where this simplified implementation of require fails, is when you require a file that is not valid JavaScript, e.g. when you require css, html, txt, svg and images or other binary files.
And it's easy to see why:
If you e.g. put HTML into a JavaScript function body, you of course rightfully get
SyntaxError: Unexpected token '<'
because of Function("bla", "<doctype...")
Now, if you wanted to extend this to for example include non-modules, you could just check the downloaded file-contents for code.indexOf("module.exports") == -1, and then e.g. eval("jquery content") instead of Func (which works fine as long as you're in the browser). Since downloads with Fetch/XmlHttpRequests are subject to the same-origin-policy, and integrity is ensured by SSL/TLS, the use of eval here is rather harmless, provided you checked the JS files before you added them to your site, but that much should be standard-operating-procedure.
Note that there are several implementations of require-like functionality:
the CommonJS (CJS) format, used in Node.js, uses a require function and module.exports to define dependencies and modules. The npm ecosystem is built upon this format. (this is what is implemented above)
the Asynchronous Module Definition (AMD) format, used in browsers, uses a define function to define modules. (basically, this is overcomplicated archaic crap that you wouldn't ever want to use). Also, AMD is the format that is implemented by RequireJS (note that despite the name containing the characters "require", AMD absolutely is NOT CommonJS).
the ES Module (ESM) format. As of ES6 (ES2015), JavaScript supports a native module format. It uses an export keyword to export a module’s public API and an import keyword to import it. This is the one you should use if you don't give a flying f*ck about archaic browsers, such as Safari and IE/EdgeHTML.
the System.register format, designed to support ES6 modules within ES5. (the one you should use, if you need support for older browsers (Safari & IE & old versions of Chrome on mobile phones/tablets), because it can load all formats [for some, plugins are required], can handle cyclic-dependencies, and CSS and HTML - don't define your modules as system.register, though - the format is rather complicated, and remember, it can read the other easier formats)
the Universal Module Definition (UMD) format, compatible to all the above mentioned formats (except ECMA), used both in the browser and in Node.js. It’s especially useful if you write modules that can be used in both NodeJS and the browser. It's somewhat flawed, as it doesn't support the latest ECMA modules, though (maybe this will get fixed) - use System.register instead.
Important sidenote on the function argument "exports":
JavaScript uses call-by-value-sharing - meaning objects are passed as a pointer, but the pointer-value itselfs is passed BY VALUE, not by reference. So you can't override exports by assigning it a new object. Instead, if you want to override exports, you need to assign the new object to module.exports - because hey, module is the pointer passed by value, but exports in module.exports is the reference to the original exports pointer.
Important sidenote on module-Scope:
Modules are evaluated ONCE, and then cached by require.
That means all your modules have a Singleton scope.
If you want a non-singleton scope, you have to do something like:
var x = require("foo.js").createInstance();
or simply
var x = require("foo.js")();
with appropriate code returned by your module.
If you need CommonJS-support for the browser (IE5+, Chrome, Firefox),
check out my code in my comment on Michele Nasti's project
You know how when you are running JavaScript in the browser, you have access to variables like "window" or Math? You do not have to declare these variables, they have been written for you to use whenever you want.
Well, when you are running a file in the Node.js environment, there is a variable that you can use. It is called "module" It is an object. It has a property called "exports." And it works like this:
In a file that we will name example.js, you write:
example.js
module.exports = "some code";
Now, you want this string "some code" in another file.
We will name the other file otherFile.js
In this file, you write:
otherFile.js
let str = require('./example.js')
That require() statement goes to the file that you put inside of it, finds whatever data is stored on the module.exports property. The let str = ... part of your code means that whatever that require statement returns is stored to the str variable.
So, in this example, the end-result is that in otherFile.js you now have this:
let string = "some code";
or -
let str = ('./example.js').module.exports
Note:
the file-name that is written inside of the require statement: If it is a local file, it should be the file-path to example.js. Also, the .js extension is added by default, so I didn't have to write it.
You do something similar when requiring node.js libraries, such as Express. In the express.js file, there is an object named 'module', with a property named 'exports'.
So, it looks something like along these lines, under the hood (I am somewhat of a beginner so some of these details might not be exact, but it's to show the concept:
express.js
module.exports = function() {
//It returns an object with all of the server methods
return {
listen: function(port){},
get: function(route, function(req, res){}){}
}
}
If you are requiring a module, it looks like this:
const moduleName = require("module-name");
If you are requiring a local file, it looks like this:
const localFile = require("./path/to/local-file");
(notice the ./ at the beginning of the file name)
Also note that by default, the export is an object .. eg module.exports = {} So, you can write module.exports.myfunction = () => {} before assigning a value to the module.exports. But you can also replace the object by writing module.exports = "I am not an object anymore."
Two flavours of module.exports / require:
(see here)
Flavour 1
export file (misc.js):
var x = 5;
var addX = function(value) {
return value + x;
};
module.exports.x = x;
module.exports.addX = addX;
other file:
var misc = require('./misc');
console.log("Adding %d to 10 gives us %d", misc.x, misc.addX(10));
Flavour 2
export file (user.js):
var User = function(name, email) {
this.name = name;
this.email = email;
};
module.exports = User;
other file:
var user = require('./user');
var u = new user();
class Test
def initialize
end
def crash
print x
end
end
Test.new.crash
Clearly this snippet will crash at line 8. If you parse this with Opal, you will get this compiled code:
/* Generated by Opal 0.8.0.beta1 */
(function(Opal) {
Opal.dynamic_require_severity = "error";
var self = Opal.top, $scope = Opal, nil = Opal.nil, $breaker = Opal.breaker, $slice = Opal.slice, $klass = Opal.klass;
Opal.add_stubs(['$print', '$x', '$crash', '$new']);
(function($base, $super) {
function $Test(){};
var self = $Test = $klass($base, $super, 'Test', $Test);
var def = self.$$proto, $scope = self.$$scope;
def.$initialize = function() {
var self = this;
return nil;
};
return (def.$crash = function() {
var self = this;
return self.$print(self.$x());
}, nil) && 'crash';
})(self, null);
return $scope.get('Test').$new().$crash();
})(Opal);
And of course it will throw the same error.
However, is there a way to determine the Ruby line where this error comes from?
I can see this question: Is there a way to show the Ruby line numbers in javascript generated by Opal, but I don't understand the answer: it leads me to https://github.com/opal/opal/tree/0-6-stable/examples/rack and I'm not sure what am I supposed to be looking at or doing.
When I run my javascript, I have an index.html file that loads opal.min.js and opal-parser.min.js, then finally I have my compiled Ruby-Javascript code in a <script> tag.
Opal has source map support, to facilitate this kind of source level of debugging. I will not go into details about sourcemaps, but HTML5Rocks has a great article that covers the topic in depth.
Here is the minimal boilerplate to set that up with Opal:
Let index.rb be our source file:
class Test
def initialize
end
def crash
print x
end
end
Test.new.crash
Since you would rather not use a lot of extraneous utilties, let us directly use the Opal API. Create a file builder.rb which will compile the file above:
require 'opal'
Opal::Processor.source_map_enabled = true
Opal.append_path "."
builder = Opal::Builder.new.build('index')
# Write the output file containing a referece to sourcemap
# which we generate below : this will help the browser locate the
# sourcemap. Note that we are generating sourcemap for only code and not
# the entire Opal corelib.
#
File.binwrite "build.js", "#{builder.to_s}\n//# sourceMappingURL=build.js.map"
File.binwrite "build.js.map", builder.source_map.to_s
File.binwrite "opal_lib.js", Opal::Builder.build('opal_lib')
Also create an opal_lib.rb file containing only:
require 'opal'
Finally create an index.html which will allow us to run the script in browser.
<!DOCTYPE html>
<html>
<head>
<script src="opal_lib.js"></script>
<script src="build.js"></script>
</head>
<body>
</body>
</html>
Now to actually compile your file, run:
ruby builder.rb
This will generate compiled javascript files opal_lib.js and build.js which are referenced by our index.html file. Now just open index.html in your browser. You will get a full call-stack and source view:
The line numbers of the source file are available:
As an alternative to using the browser, you can also use Node.js for the same purpose. This requires you have Node.js and npm installed. You will also need to install npm module source-map-support
npm install source-map-support
Now you can open the node repl and enter the following:
require('source-map-support').install();
require('./opal_lib');
require('./build');
You will get a stack trace with correct source line numbers :
/home/gaurav/Workspace/opal-playground/opal_lib.js:4436
Error.captureStackTrace(err);
^
NoMethodError: undefined method `x' for #<Test:0x102>
at OpalClass.$new (/home/gaurav/Workspace/opal-playground/opal_lib.js:4436:15)
at OpalClass.$exception (/home/gaurav/Workspace/opal-playground/opal_lib.js:4454:31)
at $Test.$raise (/home/gaurav/Workspace/opal-playground/opal_lib.js:4204:31)
at $Test.Opal.defn.TMP_1 (/home/gaurav/Workspace/opal-playground/opal_lib.js:3032:19)
at $Test.method_missing_stub [as $x] (/home/gaurav/Workspace/opal-playground/opal_lib.js:886:35)
at $Test.$crash (/home/gaurav/Workspace/opal-playground/index.rb:8:11)
at /home/gaurav/Workspace/opal-playground/index.rb:13:10
at Object.<anonymous> (/home/gaurav/Workspace/opal-playground/index.rb:13:10)
at Module._compile (module.js:435:26)
at Object.Module._extensions..js (module.js:442:10)
I recommend that you use bundler for gem management. Here is the Gemfile for fetching Opal master:
source 'http://production.cf.rubygems.org'
gem 'opal', github: 'opal/opal'
To compile you will have to run:
bundle install
bundle exec ruby builder.rb
Sprockets integration / rack integration that others have mentioned use the same API underneath, abstracting away the plumbing.
Update:
Since we have the correct line numbers in stack, it is fairly to programatically parse the stack and extract this line number into a variable:
require('./opal_lib');
require('source-map-support').install();
var $e = null;
try {
require('./build');
} catch (e) {
$e = e;
}
var lines = e.split('\n').map(function(line){ return line.match(/^.*\((\S+):(\d+):(\d+)\)/) })
var first_source_line;
for (var i = 0; i < lines.length ; i++) {
var match = lines[i];
if (match == null) continue;
if (match[1].match(/index.rb$/) {
first_source_line = match;
break;
}
}
var line_number;
if (first_source_line) line_number = first_source_line[2] // ==> 8
And of course you can do it ruby as well (but if you are running this in browser you will have to include source-map-support here as well):
class Test
def initialize
end
def crash
print x
end
end
line_num = nil
begin
Test.new.crash
rescue => e
if line = e.backtrace.map{|line| line.match(/^.*\((\S+):(\d+):(\d+)\)/) }.compact.find{|match| match[1] =~ /index.rb$/ }
line_num = line[2]
end
end
puts "line_num => #{line_num}" # ==> 8
It is stated here that Babel can extract gettext messages for Python and Javascript files.
Babel comes with a few builtin extractors: python (which extracts
messages from Python source files), javascript, and ignore (which
extracts nothing).
The command line extractor is documented here - but with no examples on usage.
Also in the same pointer above, there is some mention of a config file to be used with extraction, but not much expanded on.
When I run the basic command for the extractor on a dir with js files, I get only the .PO header generated but no messages.
$ pybabel extract /path/to/js-dir
# Translations template for PROJECT.
# Copyright (C) 2012 ORGANIZATION
# This file is distributed under the same license as the PROJECT project.
# FIRST AUTHOR <EMAIL#ADDRESS>, 2012.
#
#, fuzzy
msgid ""
msgstr ""
"Project-Id-Version: PROJECT VERSION\n"
"Report-Msgid-Bugs-To: EMAIL#ADDRESS\n"
"POT-Creation-Date: 2012-04-22 19:39+1000\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL#ADDRESS>\n"
"Language-Team: LANGUAGE <LL#li.org>\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=utf-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 0.9.6\n"
$
Here is a sample segment from a js file I'm trying to extract messages for:
else if(data.status == "1"){
var follow_html = gettext('Follow');
object.attr("class", 'button follow');
object.html(follow_html);
var fav = getFavoriteNumber();
fav.removeClass("my-favorite-number");
if(data.count === 0){
data.count = '';
fav.text('');
}else{
var fmts = ngettext('%s follower', '%s followers', data.count);
fav.text(interpolate(fmts, [data.count]));
}
}
I would appreciate it if someone can provide exact CLI options and config settings to make the extraction work, or a pointer to such.
Create a file (babel.cfg) with the following content:
[javascript:*.js]
encoding = utf-8
Then, do:
pybabel extract -F babel.cfg /path/to/js-dir
That should be enough for you to have some message strings.
BTW, you can consult the help for the extract command by doing:
pybabel extract --help
I had a similar issue and was able to get around it by disabling default keywords with babel.
pybabel extract -k __ -F babel.cfg --no-default-keywords /path/to/js-dir
You must specify at least one keyword in the command when you disable the defaults (-k [keyword]). I chose -k __ because "__" was a pattern I was looking for.
Just use this command and replace the "__" after -k with one from your babel.cfg file.
Edit: this allows you to use your own keywords rather than gettext()
You can create an object in as flask global and translate it with gettext
g.i18n = {
'Casa' : lazy_gettext('Home'),
'Auto' : lazy_gettext('Car'),
'Persona' : lazy_gettext('Person')
}
Then add it as a variable
<script>
var i18n = {{ g.i18n | tojson }}
</script>
and use it in JS:
var labelTranslate = {
Casa: i18n.Casa,
Persona: i18n.Persona,
Auto: i18n.Auto
};
You can actually use gettext directly in Javascript.
See: jsgettext. It allows you to use the standard *gettext functions, including the one using contexts and/or plural forms.
It can read PO/MO files or you can import custom made JSON files instead.
See this file of this project for a complete example.