I am trying to use string similarity inside Google App Script, however it is not entirely clear to me how to get it working inside App Script, I get multiple errors, such as "require" is not defined, as another note, the module itself appears to have a dependency.
My final goal is to use this script to match string score between one array with strings full of typos to one with correct strings all within App Script. This is my failed code.
function matchEmails() { var doc =
SpreadsheetApp.openById(("ID"));
var ts = doc.getSheetByName("dir"); var source =
doc.getSheetByName("data"); var names =
source.getDataRange().getValues(); var data =
ts.getDataRange().getValues(); var arr = []; var arr2 = []; var
arr3 = []; var arr4 = [];
for (i=0; i<names.length; i++) {
for (j=0; j<data.length; j++) {
stringSimilarity.compareTwoStrings(names[i][0], data[j][0]); Logger.log(stringSimilarity);
/*
var n = data[j][0].split();
for (N=0; N<n.length; N++) {
arr2.push(n[N].charAt(0));
}
var string2 = arr2.join("");
var arr2 = [];
if (names[i][0] === data[j][0]) {
arr.push([data[j][1]]);
} */ //I want to replace this blanked out code with String >similarity.
if (string === string2) {
arr.push([data[j][1]]);
arr3.push([i+1]);
arr4.push([data[j][0]]);
}
} } for (i = 0; i<arr.length; i++) {
source.getRange(arr3[i],6).setValue(arr[i]);
source.getRange(arr3[i],7).setValue(arr4[i]); } }
Your GAS project is not a Node.js app, so the above will not work. While both Google Apps Script and Node use JavaScript, they provide different runtime environments for executing JS code. In GAS, the environment is a closed ecosystem on Google Servers that end users don't know anything about.
In Node, the runtime consists of V8 (JS engine) and C++ add-ons that expose low-level APIs (access to the file system, etc.). The library you referenced is an NPM package created for Node.js. Installing the package via NPM will make it available for Node projects, but don't think it will magically appear on Google servers as well.
You must either use GAS-specific versions of these dependencies, or, if they don't exist, refactor the source code to make it compatible with GAS (Node and GAS use different ECMAScript versions, so some latest features like arrow functions will break your GAS code).
For example, here's lodash for GAS
https://github.com/contributorpw/lodashgs
Using libraries in GAS
https://developers.google.com/apps-script/guides/libraries
P.S. In GAS, all ".gs" files share the same namespace, so calling 'require' function is redundant. If you want to mimic this behavior, you'll still need to write your own require function.
I wrote up some guidance on packaging up an npm module for usage in Apps Script in this article.
tl;dr is to create an index.js with the following in a new directory:
import {compareTwoStrings, findBestMatch} from 'string-similarity';
export {compareTwoStrings, findBestMatch};
and then run the following in that directory:
npm init -y
npm install --save-dev string-similarity
npx esbuild index.js --bundle --global-name=stringSimilarity --outfile=StringSimilarity.js
You can then copy the contents of StringSimiliarity.js into a new .gs file in the Apps Script editor. The exported functions will be usable from your own Code.gs as stringSimilarity. compareTwoStrings() and stringSimilarity.findBestMatch().
(In case you'd rather not bundle it yourself, the output of the esbuild process for string-similarity can be found in this gist. But the general steps should apply to most npm modules that don't require a Node or browser-specific runtime environment.)
Google Apps Script files by default have access to Google APIs, such as the Spreadsheet Service in the G Suite Services:
https://developers.google.com/apps-script/reference/spreadsheet/
A GAS project (so scripts in that project) doesn't by default have access to Node.js or any other frameworks. However, a GAS project can include another GAS project by reference as a library. In the GAS Script Editor, use menu Resources > Libraries... to add an external GAS project as a library by setting the source project's project key (and some other source project properties).
So if you have a Javascript that depends on external resources (like Node.js) by require then if you can find an external GAS project that provides the same services with the same API you can provide it as a library to the script. You omit the require statements from the original script as they're replaced by the GAS library dependency configuration I mentioned.
If your original script and its dependencies are all open source you could create GAS projects for each level of dependencies in the originals. You can already find some popular JS frameworks scripts available as GAS libraries.
GAS also allows packaging external resources as Web apps and other package formats. They can be used with corresponding techniques, and some of them found already available from other developers.
Building upon the answer of Jeff Posnick, note we can use https://esbuild.github.io in the same simple way to bring a complete library into Google Apps Script. For example with Ramda :
npm init -y
npm install --save-dev ramda
cp ./node_modules/ramda/dist/ramda.js .
npx esbuild ramda.js --minify --bundle --global-name=R --outfile=R.js
Then copy the contents of R.js into a new .gs file in the Apps Script editor.
Now you can use ramda in your code, for example:
const seventeen = R.add(7)(10);
An easy workaround is to use JavaScript's Eval function to call an external library hosted online
i. e. eval(UrlFetchApp.fetch('http://path.to/external/javascript.js').getContentText());
This won't work if the library itself depends on other modules which aren't available in the GAS runtime, but for certain libraries it can be a simple solution.
Related
I have a website where the jQuery.LiveAddress plugin is implemented, but it was deprecated and then totally removed by SmartyStreets in 2014.
https://www.smartystreets.com/archive/jquery-plugin/website/configure
<script src="//ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<script src="//d79i1fxsrar4t.cloudfront.net/jquery.liveaddress/5.2/jquery.liveaddress.min.js"></script>
<script>var liveaddress = $.LiveAddress({
key: "YOUR_WEBSITE_KEY_HERE",
debug: true,
target: "US|INTERNATIONAL",
addresses: [{
address1: '#street-address',
locality: '#city',
administrative_area: '#state',
postal_code: '#zip',
country: '#country'
}]
});
</script>
There is now a JavaScript SDK: https://www.smartystreets.com/docs/sdk/javascript
const SmartyStreetsSDK = require("smartystreets-javascript-sdk");
const SmartyStreetsCore = SmartyStreetsSDK.core;
const Lookup = SmartyStreetsSDK.usStreet.Lookup;
// for Server-to-server requests, use this code:
// let authId = process.env.SMARTY_AUTH_ID;
// let authToken = process.env.SMARTY_AUTH_TOKEN;
// const credentials = new SmartyStreetsCore.StaticCredentials(authId, authToken);
// for client-side requests (browser/mobile), use this code:
let key = process.env.SMARTY_WEBSITE_KEY;
const credentials = new SmartyStreetsCore.SharedCredentials(key);
It says "for client-side requests (browser/mobile), use this code", but am I correct in assuming that it is not possible to get this code working as browser JavaScript? Wouldn't I need to be running node as a service on my server to be using this code?
The client-side code example can be used in the browser, not just in node. You will need to store your website key in a different way since the browser doesn't have access to process.env. You can store the website key in plaintext here since it will be tied to your hostname.
You will need to be using some kind of library or JS bundler to process the require statement you see on line 1. You can also import the SDK if that's what works better for your setup, for example
import SmartyStreetsSDK from "smartystreets-javascript-sdk"
Something to keep in mind with using the SDK vs the jQuery plugin is that the SDK does not provide and UI elements on the page. It's less of a drop in solution than the jQuery plugin is. You will need to design and create your own UI elements for the user to interact with.
disclaimer: I work for Smarty (formerly SmartyStreets)
I came across this same issue and unfortunately smarty streets decided to make it more difficult for users to implement their product.
Upon my search I came across this fiddle, it doesn't have all of the features the jquery plugin did but it is a good place to start that doesn't require building your own suggestion UI.
<script async src="//jsfiddle.net/smartystreets/uptgh7c8/embed/"></script>
jsfiddle source
The key file that I was missing was the browserify.js file in the SmartyStreets JavaScript SDK.
Run npm install smartystreets-javascript-sdk --save-dev in your project's root directory
Change directory to node_modules/smartystreets-javascript-sdk
Run npm install in this directory
Run rm -rf dist && node browserify.js (script will fail if the dist folder exists)
Copy dist/smartystreets-sdk-1...min.js to a dependencies folder for your main project.
I tried to write a grunt task to take the SmartyStreets SDK and automatically update it, browserify it, and deploy it along with the rest of our website, but it turned out to be too much of a hassle with the time constraints given. If anyone else writes a grunt task for this please share!
Im working with project that is written in Node CommonJS modules. My point was to make this project accessible from a browser. I decided to use Rollup.js, so when the bundle is created you can include script in browser and use functions from library (thanks to iife format). I needed to install plugins for Rollup to convert CommonJS modules into ES6 modules, so browser can understand it.
Everything is fine, except that this project uses bson library from npm. This bson library is required in one of my modules which uses some of it's functions. After I create a bundle and include it into index.html an error appears in console which says: "require is not defined". When I look inside the created bundle there are some requires.
var Map = require('./map'),
Long = require('./long'),
Double = require('./double'),
Timestamp = require('./timestamp'),
ObjectID = require('./objectid'),
BSONRegExp = require('./regexp'),
Symbol$1 = require('./symbol'),
Int32 = require('./int_32'),
Code = require('./code'),
Decimal128 = require('./decimal128'),
MinKey = require('./min_key'),
MaxKey = require('./max_key'),
DBRef = require('./db_ref'),
Binary = require('./binary');
I have created simple code in Plunker to illustrate you my config and simplified structure.
https://plnkr.co/edit/YuiVJxhwhjUQ0Flw0Mg3?p=preview
In this plunker there are two simple modules, which one requires bson library, and second requires this first module. There is also Rollup config file, where I use plugins (if there is no globals plugin there is an error: Uncaught ReferenceError: Buffer is not defined).
I'm really confused. Am I misunderstanding something? Why isn't it converted into ES6 modules just like other of my code?
Here is link to bson library: https://www.npmjs.com/package/bson
Let's see:
In order for rollup to bundle your application you need to transform commonjs modules into es modules first, this is done with rollup-plugin-commonjs (https://github.com/rollup/rollup-plugin-commonjs) which you are already using, maybe you could explicitly set in the options to include the bson package. I usually use it like this, just in case:
commonjs({
include: ['node_modules/**']
})
If some library uses node global modules you will need to include them in the browser, thats why you need rollup-plugin-node-globals.
Finally, if you take a look at bson github repository there is a folder called browser_build, which contains the UMD definition of the library, so if you require 'bson/browser_build' instead of 'bson' it should work, and you may not need to use globals plugin.
Take a look at js module formats (cjs, umd, iife, es, amd) it is worth it.
I am trying to get some of the NPM functionality into my Node.js programs. In particular, I would like to be able to analyze the available node modules on my system. "Module" here means "module identifier", thus either an identifier like "fd", or a file path; in other words, anything that can be put into a require() call and does load a module.
This question is split into three sub-problems:
1) Get a list of all core modules
2) Get a list of all loaded modules
3) Get a list of all installed and available modules.
Question 1 is answered by zeke's node-core-module-names list. Loading another module to find the core modules list is not elegant and it may outdate itself, but it is an option and does work. The (ordered) list is thus ['assert', 'buffer', 'child_process', ..., 'zlib'].
The 2. question can be answered by a call of Object.keys(require.cache), which returns a list of file paths.
What I cannot elegantly solve by now is the 3. question. There is the shell command npm ls which returns a tree graph. But is there anything usable and better?
Thank's for listening!
Tom
Here is something I have found tinkering around and I think this should be valid.
The V8 code has a standard set of bindings which you have seen in Node. They include but are not limited to:
fs
path
http
etc.
Also, there is a global variable by name of process. This exposes process level information and functionality, but also lets you get your hands on some V8 code through a function inside of the process variable called bindings.
The bindings(...) functions allows you to interface into exposed C++ libraries created by Node or you can create your own NodeJS modules by following the V8 developer guide (beyond the scope of this answer, read more here).
A funny little line I saw in the Node.cc file included a static check for checking for bindings for a keyword natives. This returns, it seems, a list of system level modules that you are looking for, and then some.
So that being said, I went into Node REPL and plugged in two lines (which I am sure can be shortened in a more elegant, expressive manner). Also note that I am pruning out anything starting with an underscore (_) so to preserve private functions or bindings:
var natives = process.binding('natives');
for (var key in natives) {if (key.indexOf('_') !== 0) {console.log(key);}}
npm list has various output options/flags, including json and parseable (which outputs a list of paths)
Try this:
var exec = require('child_process').exec;
var cmd = 'npm ls --json';
exec(cmd, function(error, stdout, stderr) {
var treeObject = JSON.parse(stdout);
});
The above requires no external packages but might need more code to work around the buffer limit: https://github.com/nodejs/node/issues/4236
Alternatively npm can also be used programmatically, perhaps through global-npm:
var npm = require('global-npm');
npm.load({}, function (err) {
npm.commands.list(null, function(err, treeObject) {
var firstLevelDependenciesArray = Object.keys(treeObject.dependencies);
});
});
On our production server, I have minified javascript published and I'm not including a map file with it, because I don't want the user to be able to understand what's happening based on the error.
I have a logging service I've written to forward the angular exceptions (caught by $exceptionHandler) to myself via email. However, this stack trace is near unreadable:
n is not defined
at o (http://localhost:9000/build/app.min.js:1:3284)
at new NameController (http://localhost:9000/build/app.min.js:1:3412)
at e (http://localhost:9000/build/bower.min.js:44:193)
at Object.g.instantiate (http://localhost:9000/build/bower.min.js:44:310)
at b.$get (http://localhost:9000/build/bower.min.js:85:313)
at d.compile (http://localhost:9000/build/bower.min.js:321:23333)
at aa (http://localhost:9000/build/bower.min.js:78:90)
at K (http://localhost:9000/build/bower.min.js:67:39)
at g (http://localhost:9000/build/bower.min.js:59:410)
at http://localhost:9000/build/bower.min.js:58:480 <ui-view class="ng-scope">
What I'm wondering is: Is there a program where I can analyze this stack trace against the actual non-minified source code via map file (or not via map file if there's another way)
What you want to do is parse the source maps. This has nothing to do with web browsers. All you need to do is translate the minified reference into the unminified resource.
If you have any experience with NodeJS there is already a package that does this for you.
https://github.com/mozilla/source-map/
To install the library
npm install -g source-map
or
yarn global add source-map
Create a file named "issue.js"
fs = require('fs');
var sourceMap = require('source-map');
var smc = new sourceMap.SourceMapConsumer(fs.readFileSync("./app.min.js.map","utf8"));
console.log(smc.originalPositionFor({line: 1, column: 3284}));
Run the file with node
node issue.js
It should output the location in the original file to the console for first line from the stack trace.
Note: I tell you install source-map globally for ease of use, but you could create a node project that does what you need and installs it locally.
I figured there was no super simple tool for converting a minified stack trace into a readable one using a source map (without having to use a web service), so I created a tool for it:
https://github.com/mifi/stacktracify
Install and use it as follows:
npm install -g stacktracify
Now copy a minified stacktrace to your clipboard - then run:
stacktracify /path/to/js.map
Adding to #Reactgular's answer, the below snippet will work with the latest version of source-map
const rawSourceMap = fs.readFileSync("./app.min.js.map","utf8");
const whatever = sourceMap.SourceMapConsumer.with(rawSourceMap, null, consumer => {
console.log(consumer.originalPositionFor({
line: 1,
column: 3284
}));
});
And to add to the discussion on the thread a simple regex like /\/(\w*[-\.]?\w*).js:\d*:\d*/g
Below is a very simple regex to find all line numbers in a stacktrace.
//regex for patterns like utils.js, utils123.js, utils-name.js, utils.version.js
var patt = /\/(\w*[-\.]?\w*).js:\d*:\d*/g;
// returns matches like ['/app.min.js:1:3284', '/bower.min.js:44:193', ..]
var errorPositions = line.match(patt);
console.log(errorPositions);
if(!errorPositions || errorPositions.length === 0) {
console.log("No error line numbers detected in the file. Ensure your stack trace file is proper");
return;
}
errorPositions.forEach(function(error) {
findInSourceMap(error);
});
});
If you had access to the source map file externally and could get the same file structure you could work it out I guess, but I'm not aware of any tools outside the browser that will help you with that.
The added advantage of having the data in a running browser will allow checking of locals which you won't get even with a source map.
You might want to consider a tool such as rollbar to do error reporting. This will report all the locals in each frame to help debugging. It has support for sourcemaps outside the browser to address your security concerns.
Append comment directive for the JS running in the page.
//# sourceMappingURL=/path/to/your/sourcemap.map
In firefox (not sure about chrome) to tell the Debugger to use source maps if they are available, click the "Debugger settings" button and select "Show original sources" from the list of settings that pops up:
I'm writing a javascript library that contains a core module and several
optional submodules which extend the core module. My target is the browser
environment (using Browserify), where I expect a user of my module will only
want to use some of my optional submodules and not have to download the rest to
the client--much like custom builds work in lodash.
The way I imagine this working:
// Require the core library
var Tasks = require('mymodule');
// We need yaks
require('mymodule/yaks');
// We need razors
require('mymodule/razors');
var tasks = new Tasks(); // Core mymodule functionality
var yak = tasks.find_yak(); // Provided by mymodule/yaks
tasks.shave(yak); // Provided by mymodule/razors
Now, imagine that the mymodule/* namespace has tens of these submodules. The
user of the mymodule library only needs to incur the bandwidth cost of the
submodules that she uses, but there's no need for an offline build process like
lodash uses: a tool like Browserify solves the dependency graph for us and
only includes the required code.
Is it possible to package something this way using Node/npm? Am I delusional?
Update: An answer over here seems to suggest that this is possible, but I can't figure out from the npm documentation how to actually structure the files and package.json.
Say that I have these files:
./lib/mymodule.js
./lib/yaks.js
./lib/razors.js
./lib/sharks.js
./lib/jets.js
In my package.json, I'll have:
"main": "./lib/mymodule.js"
But how will node know about the other files under ./lib/?
It's simpler than it seems -- when you require a package by it's name, it gets the "main" file. So require('mymodule') returns "./lib/mymodule.js" (per your package.json "main" prop). To require optional submodules directly, simply require them via their file path.
So to get the yaks submodule: require('mymodule/lib/yaks'). If you wanted to do require('mymodule/yaks') you would need to either change your file structure to match that (move yaks.js to the root folder) or do something tricky where there's a yaks.js at the root and it just does something like: module.exports = require('./lib/yaks');.
Good luck with this yak lib. Sounds hairy :)