Is there an existing API or library that can be used to load a JSON file in both the browser and Node?
I'm working on a module that I intend to run both from the command-line in NodeJS, and through the browser. I'm using the latest language features common to both (and don't need to support older browsers), including class keywords and the ES6 import syntax. The class in question needs to load a series of JSON files (where the first file identifies others that need to be loaded), and my preference is to access them as-is (they are externally defined and shared with other tools).
The "import" command looks like it might work for the first JSON file, except that I don't see a way of using that to load a series of files (determined by the first file) into variables.
One option is to pass in a helper function to the class for loading files, which the root script would populate as appropriate for NodeJS or the browser.
Alternatively, my current leading idea, but still not ideal in my mind, is to define a separate module with a "async function loadFile(fn)" function that can be imported, and set the paths such that a different version of that file loads for browser vs NodeJS.
This seems like something that should have a native option, or that somebody else would have already written a module for, but I've yet to find either.
For node, install the node-fetch module from npm.
Note that browser fetch can't talk directly to your filesystem -- it requires an HTTP server on the other side of the request. Node can talk to your filesystem, as well as making HTTP calls to servers.
It sounds like as of now, there is no perfect solution here. The 'fetch' API is the most promising, but only if Node implements it some day.
In the meantime I've settled for a simple solution that works seamlessly with minimal dependencies, requiring only a little magic with my ExpressJS server paths to point the served web instance to a different version of utils.js.
Note: To use the ES-style import syntax for includes in NodeJS (v14+) you must set "type":"module" in your package.json. See https://nodejs.org/api/esm.html#esm_package_json_type_field for details. This is necessary for true shared code bases.
Module Using it (NodeJS + Browser running the same file):
import * as utils from "../utils.js";
...
var data = await utils.loadJSON(filename);
...
utils.js for browser:
async function loadJSON(fn) {
return $.getJSON(fn); // Only because I'm using another JQuery-dependent lib
/* Or natively something like
let response = await fetch(fn);
return response.json();
*/
}
export { loadJSON };
utils.js for nodeJS
import * as fs from 'fs';
async function loadJSON(fn) {
return JSON.parse(await fs.promises.readFile(fn));
}
export { loadJSON };
Related
A quick problem I faced while developing a web-app using SvelteKit (and by extension, Vite):
Inside the <script> tag of my +page.svelte file I tried defining an empty placeholder File object the following way:
let formObject: FormCreationData = {
fileToUpload: new File([], ''),
anotherField: "",
...
};
While it should work in normal JS/TS (and Svelte if you aren't using SvelteKit), it now throws the following error:
ReferenceError: File is not defined
at +page.svelte:13:14
Why is this the case?
Since SvelteKit implements Server-Side Rendering (SSR) - The code that is on the +page.svelte file has to run both on client browsers and the Vite server.
The File class is only available in browsers, so it won't be able to fulfill this requirement (You might know that Node.js offers the fs module in order to allow for file operations instead).
This means that there are two ways to possibly fix this problem:
Disable SSR using the variable ssr in the +page.ts/js file:
export const ssr = false;
Find a way to define the File object at a point where the code runs on the browser (This can be done by checking the browser variable under the $app/environment module, or inside of one of the supported Svelte hooks, such as onMount()).
I cannot find an accurate answer to this anywhere. I understand the browser doesn't recognise commonJS syntax hence why if require() is loaded into the browser a reference error occurs like this
let isEven = require('is-even') // ReferenceError: Can't find variable: require
console.log(isEven(6)); // this doesn't run
So I try to use ES6 modules like this:
import isEven from "is-even";
console.log(isEven(6));
I then get this error in the console:
TypeError: Module specifier, 'is-even' does not start with "/", "./", or "../".
My fix at the moment is to run a module bundler like Webpack which allows me to use both of these syntax's effectively and I have no problems. However I would rather find an alternative because I find it harder to fix bugs when using a module bundler as it can be quite difficult to locate the code causing the error. Is there an alternative other than using a module bundler to transform to older Javascript modules before served to the browser ?
import in a browser does not allow importing from a plain filename. You must at least construct some kind of path starting with "/", "./", or "../" or even a full URL as import in the browser does not have a "default" place to load a plain filename from. You must be explicit about what the path is.
And, keep in mind that import from a browser will make a request to your server and your server must be configured to provide that file to the browser when requested.
So, depending upon how you have your server configured, you may want something like this:
import isEven from "/is-even.js";
console.log(isEven(6));
Or, if your server handles scripts separately, you might even have this:
import isEven from "/scripts/is-even.js";
console.log(isEven(6));
And, then it is the web server's responsibility to know how to receive one of these requests, know where that actual script file is in the server's file system, get it and send it back to the browser.
Keep in mind that the main reason client-side code uses bundlers is that it is not efficient to have a web page that loads tons of script modules, each loaded separately because each script load is an http round-trip request to your server. Loading can benefit some from client-side caching, but you still don't want your web-site's home page to have to load 50 separate modules the first time a user hits your home page just to get all your scripts loaded into the browser.
So, it would generally not be efficient in client-side code to import a script just for one function like isEven(). This is why bundlers exist so that you can write the code in nice, modular, separately testable modules and then have the bundler go grab all the code that is needed for a particular client-side operation and collect it into one common script file that can then be loaded more efficiently into the client-side environment.
try this
export { isEven }; // module exports a function:
module imports the function from is-even.mjs
import { isEven } from './is-even.mjs';
console.log(is-even(6));
I'm using Node.js (v16) dynamic imports in a project to load plugins using a function loadJsPlugin shown here:
import { pathToFileURL } from 'url';
async function loadJsPlugin(pluginPath) {
const pluginURL = pathToFileURL(pluginPath).toString();
const result = await import(pluginURL);
return result.default;
}
My main program provides absolute paths to the loadJsPlugin function, such as /home/sparky/example/plugins/plugin1.js (Linux) or C:\Users\sparky\example\plugins\plugin1.js (Windows). The pathToFileURL function then converts these absolute paths to URLs like file:///home/sparky/example/plugins/plugin1.js (Linux) or file:///C:/Users/sparky/example/plugins/plugin1.js (Windows).
Loading the plugins this way works fine when the loadJsPlugin function is in the same package as the main program, like this:
import { loadJsPlugin } from './plugin-loader.js';
async function doSomething() {
const plugin = await loadJsPlugin('...'); // works
// use plugin
}
However, if I try to move loadJsPlugin to a separate library and use it from there, it fails with Error: Cannot find module '<url here>'
import { loadJsPlugin } from '#example/plugin-loader';
async function doSomething() {
const plugin = await loadJsPlugin('...'); // error
// use plugin
}
NOTE: the dependency name here is not on NPM, it's on a private repository and there's no problem loading the dependency itself. Also, static ES6 imports in general are working fine in this system.
I looked through Node.js documentation, MDN documentation, and other StackOverflow questions for information about what is allowed or not, or whether dynamic import works differently when in the same package or a dependency, and didn't find anything about this. As far as I can tell, if a relative path or file URL is provided, and the file is found, it should work.
Ruling out file not found:
I can switch back and forth between the two import lines to load the loadJsPlugin function from either ./plugin-loader.js or #example/plugin-loader, give it the same input, and the one in the same package works while the one from the dependency doesn't.
When I test in VS Code, I can hover the mouse over the URL in the Error: Cannot find module 'file:///...' message and the file opens just fine
I can also copy the 'file:///...' URL to a curl command (Linux) or paste it into the address bar of Windows Explorer and it works.
If I try a path that actually doesn't exist, I get a slightly different message Error [ERR_MODULE_NOT_FOUND]: Cannot find module '<path here>', and it shows the absolute path to the file that wasn't found instead of the file URL I provided.
Checking different file locations:
I tried loading plugins that are located in a directory outside the program (the paths shown above like /home/sparky/example/plugins/...); got the results described above
I tried loading plugins that are located in the same directory (or subdirectory) as the main program; same result
I tried loading plugins that are packaged with the dependency in node_modules/#example/plugin-loader; same result (obviously this is not a useful set up but I just wanted to check it)
I'd like to put the plugin loader in a separate library instead of having the same code in every project, but it seems that dynamic import only works from the main package and not from its dependencies.
I'm hoping someone here can explain what is going on, or give me a pointer to what might make this work.
I am making a small NPM package which would essentially serve as a convenient data import in a React app. My package currently has one exported method, getSystems(), which returns an array of objects. The module code reads the data file using fs/promises.readFileSync which obviously is not available in a browser environment.
How can I bundle the data in my package so that it can be used in a React app?
Here is what the module is doing:
import { decode } from "#msgpack/msgpack";
import { readFileSync } from "fs";
import path from "path";
const file = readFileSync(path.join(__dirname, "../assets/systems.dat"));
const systems = decode(file);
export const getSystems = () => systems;
This works in node. I suppose what I need is a way to configure the build in such a way that the data is included in the output JS files.
Full code here.
Disclaimer'ish: I understand this is usually done via an API instead of bloating the application code with data (which is what I'm doing here, I suppose). This is kind of a learning and testing thing. And also I got sick of copying data files around in Dockerfiles.. :)
Turns out if I use JSON files as source and just import them, the module works in browsers. While the webpack bundle gets pretty big, gzip appears to do a great job compressing the embedded data, so I guess this is the way to go.
I am trying to call a function in a python file from a js file, I got this to work through my console, but I am now trying to implement it in a mobile app using expo.
The way I had set this up is, I have the JS file for a certain screen in my app, this then calls a function in a separate JS file, which then calls the function in the python file.
I am using the child_process module to talk to python from JS.
And as I said, this was working before I tried to export the JS function to my screen file.
index.js
export function foo(process, sentence){
const spawn = require("child_process").spawn;
const process = spawn("python3", ["./python.py", sentence]);
...
}
screen.js
*other imports
import { foo } from "./filepath..."
...
But when I run npm start I get the following error:
Failed building JavaScript bundle.
While trying to resolve module `child_process` from file `/Users/mee/Documents/GitHub/project/app/screens/screen.js`, the package `/Users/mee/Documents/GitHub/project/node_modules/child_process/package.json` was successfully found. However, this package itself specifies a `main` module field that could not be resolved (`/Users/me/Documents/GitHub/project/node_modules/child_process/app/screens/screen.js`. Indeed, none of these files exist:
How can I fix this?
It won't work for few reasons
child_process is part of the node standard library, it's not available in other environments like react-native or browser
even if above was not true, there is no python3 executable on your phone
python.py file from your local directory wouldn't be even uploaded to the phone because bundler is only uploading one big js file with entire js code combined + assets, python.py is neither of those.
Only solution that make sense it to rewrite that code to javascript.
Technically it's not impossible, there might be a way to do that, by compiling python interpreter for mobile platform, or using some tool that translates python code into js, but it's not something that you should consider.