I am making a small NPM package which would essentially serve as a convenient data import in a React app. My package currently has one exported method, getSystems(), which returns an array of objects. The module code reads the data file using fs/promises.readFileSync which obviously is not available in a browser environment.
How can I bundle the data in my package so that it can be used in a React app?
Here is what the module is doing:
import { decode } from "#msgpack/msgpack";
import { readFileSync } from "fs";
import path from "path";
const file = readFileSync(path.join(__dirname, "../assets/systems.dat"));
const systems = decode(file);
export const getSystems = () => systems;
This works in node. I suppose what I need is a way to configure the build in such a way that the data is included in the output JS files.
Full code here.
Disclaimer'ish: I understand this is usually done via an API instead of bloating the application code with data (which is what I'm doing here, I suppose). This is kind of a learning and testing thing. And also I got sick of copying data files around in Dockerfiles.. :)
Turns out if I use JSON files as source and just import them, the module works in browsers. While the webpack bundle gets pretty big, gzip appears to do a great job compressing the embedded data, so I guess this is the way to go.
Related
I am building a jsFiddle-like application and I want the user to be able to enter the URL of a js module that I can then import in the runtime.
So somewhere in the code I do something like import(userModuleUrl).then(...)
I want to use webpack to bundle the app, but now webpack hijacks the original import ES6 function and is not very happy not knowing about the userModuleUrl at compile time.
I can’t use externals because I can’t know ahead of time all the urls the user may import.
I only wish there was a way for webpack to not interpret that line, or to give me a function that’s equivalent to the raw ES import
I ended up doing it this way:
const importFunc = new AsyncFunction('url','return import(url);');
const script = await importFunc(url);
you'll need to allow your app to run eval though... Which is fine for a jsFiddle-like app.
Is there an existing API or library that can be used to load a JSON file in both the browser and Node?
I'm working on a module that I intend to run both from the command-line in NodeJS, and through the browser. I'm using the latest language features common to both (and don't need to support older browsers), including class keywords and the ES6 import syntax. The class in question needs to load a series of JSON files (where the first file identifies others that need to be loaded), and my preference is to access them as-is (they are externally defined and shared with other tools).
The "import" command looks like it might work for the first JSON file, except that I don't see a way of using that to load a series of files (determined by the first file) into variables.
One option is to pass in a helper function to the class for loading files, which the root script would populate as appropriate for NodeJS or the browser.
Alternatively, my current leading idea, but still not ideal in my mind, is to define a separate module with a "async function loadFile(fn)" function that can be imported, and set the paths such that a different version of that file loads for browser vs NodeJS.
This seems like something that should have a native option, or that somebody else would have already written a module for, but I've yet to find either.
For node, install the node-fetch module from npm.
Note that browser fetch can't talk directly to your filesystem -- it requires an HTTP server on the other side of the request. Node can talk to your filesystem, as well as making HTTP calls to servers.
It sounds like as of now, there is no perfect solution here. The 'fetch' API is the most promising, but only if Node implements it some day.
In the meantime I've settled for a simple solution that works seamlessly with minimal dependencies, requiring only a little magic with my ExpressJS server paths to point the served web instance to a different version of utils.js.
Note: To use the ES-style import syntax for includes in NodeJS (v14+) you must set "type":"module" in your package.json. See https://nodejs.org/api/esm.html#esm_package_json_type_field for details. This is necessary for true shared code bases.
Module Using it (NodeJS + Browser running the same file):
import * as utils from "../utils.js";
...
var data = await utils.loadJSON(filename);
...
utils.js for browser:
async function loadJSON(fn) {
return $.getJSON(fn); // Only because I'm using another JQuery-dependent lib
/* Or natively something like
let response = await fetch(fn);
return response.json();
*/
}
export { loadJSON };
utils.js for nodeJS
import * as fs from 'fs';
async function loadJSON(fn) {
return JSON.parse(await fs.promises.readFile(fn));
}
export { loadJSON };
I am using in a web application react-router.
When using import 'lodash' I import the whole lodash lib in my project.
Code splitting is about using an async import using import().then() to dynamically load chunk while the application is running.
Read about code splitting in react-router/web.
For example: function atRuntime() { import('lodash').then(() => {});}
This will import the library at runtime with an ajax request so it is not bundled in the main.js.
I'd like to recycle my code between web and native and we use a lot of code splitting for each page change.
My app has two main parts and some user will only visit one part so they don't need all user authenticated part.
I expect to be able to use tree shaking during react-native, but it is missing in react-router/native documentation.
What's react-native opinion about code splitting?
if you want to use webpack, you can try with haul https://github.com/callstack/haul
but i highly recommend this implementation without webpack -> https://www.npmjs.com/package/react-native-bundle-splitter
I've been looking all over for a solution to this and I can't figure it out for the life of me...
The sitch: I have a lazy loaded React component which is supposed to parse a CSV file (with PapaParse) which is all built within the create-react-app framework. But for some reason, despite everything saying it should work, when I try to use PapaParse, I get this error:
Error: Script path cannot be determined automatically when Papa Parse is loaded asynchronously. You need to set Papa.SCRIPT_PATH manually.
But since this is bundled with Webpack I have no idea what this script path should be and I've tried setting the path to the PapaParse folder within the project folder structure (i.e. something like ../../node_modules/papaparse) to no avail. I actually got a different error when I put in a path:
papaparse?papaworker:1 Uncaught SyntaxError: Unexpected token <
For some more context, the component in question looks a little like this:
import Papa from 'papaparse';
class Dialog extends React.Component {
...
handleFileChange = () => {
...
Papa.parse(file, config);
...
}
...
}
I installed PapaParse via npm, so it should be the latest version, some things go back to 2014-15 where these problems existed, but it's said to have been updated...
Actually, I'm confused, because it works well on both of my projects, hence, React Web App and React Native iOS, Android App.
I imported it like below:
import Papa from 'papaparse/papaparse.min';
Surely, for each project, there are some configs that maybe some libraries work badly. but you maybe can use Node.js path for setting SCRIPT_PATH for Papa:
import path from 'path';
...
Papa.SCRIPT_PATH = path.resolve('/node_modules/papaparse');
...
Or if it has the issue too, then use this link answer for using papaparse directly in your code. that is not recommended.
I'm confused by how we import [Publish] functions and [Meteor Methods] that reside under /imports/server to the Blaze client (either in /imports/client/ui or even just under app-name/client).
The classic way, is to just use the publish or meteor method, and everything is consumed by the client without imports. But if everything resides under the /imports directory (including the Blaze templates) how does that work? Are there real examples out there?
Illustration:
// imports/server/publishing/test.js
Meteor.publish('publish.test', function() {
if (this.userId) return TestCollection.find({});
return self.ready();
});
// imports/client/ui/test.js
import { Template } from "meteor/templating";
import { ReactiveDict } from "meteor/reactive-dict";
import { Mongo } from 'meteor/mongo';
import TestCollection from '../../imports/collections/test.js';
import "./test.html";
Template.Test.onCreated(function() {
this.state = new ReactiveDict();
this.autorun(() => {
const subscription = this.subscribe('publish.test');
...
}
});
});
How do server side only stuff make its way to the client in the new imports syle of developing?
UPDATE1:
Reponding to Answer1, would something like this work? Also, does the client look okay?
// app-name/imports/server/trades-pubs.js
// This code only runs on the server
Meteor.publish('trades', function tradesPublication() {
return Trades.find({},{sort: {timestamp: -1}, limit: 1000});
});
// app-name/imports/server/trades-methods.js
Meteor.methods({
// Only on server
'trades.importFromFiles'() {
fs = require('fs');
const path = "/home/daemmon/trades_data/";
var files = fs.readdirSync(path);
...
}
});
// app-name/server/main.js
import '../imports/server/trades-methods.js';
import '../imports/server/trades-pubs.js';
Is this all that's needed to get a publish methods to the client and server side meteor methods avaialbe to the client?
// imports/client/ui/test.js
import { Template } from "meteor/templating";
import { ReactiveDict } from "meteor/reactive-dict";
import { Mongo } from 'meteor/mongo';
import TestCollection from '../../imports/collections/test.js';
import "./test.html";
Template.Test.onCreated(function() {
this.state = new ReactiveDict();
this.autorun(() => {
const subscription = this.subscribe('trades');
...
}
});
});
UPDATE2:
you might want to consider importing app-name/imports/server/trades-methods.js somewhere in your client code as well, i.e. in a file like app-name/client/main.js
I thought we could not import server code on the client? If I wanted to import trades-methods.js for example, I'd have to move it to app-name/imports/api or something outside /imports/server.
UPDATE3:
Reading the Meteor Guide, I'm confused by this paragraph:
.
To fully use the module system and ensure that our code only runs when we ask it to, we recommend that all of your application code should be placed inside the imports/ directory. This means that the Meteor build system will only bundle and include that file if it is referenced from another file using an import (also called “lazy evaluation or loading”).
Meteor will load all files outside of any directory named imports/ in the application using the default file load order rules (also called “eager evaluation or loading”). It is recommended that you create exactly two eagerly loaded files, client/main.js and server/main.js, in order to define explicit entry points for both the client and the server. Meteor ensures that any file in any directory named server/ will only be available on the server, and likewise for files in any directory named client/. This also precludes trying to import a file to be used on the server from any directory named client/ even if it is nested in an imports/ directory and vice versa for importing client files from server/.
These main.js files won’t do anything themselves, but they should import some startup modules which will run immediately, on client and server respectively, when the app loads. These modules should do any configuration necessary for the packages you are using in your app, and import the rest of your app’s code.
.
Doesn't this mean for example that if there is a file inside the [/app-name/imports/server] directory, this file can NOT be imported in the client here [/app-name/client/main.js]?
.
For example I could NOT do the following:
Module inside the imports /server directory:
/app-name/imports/server/server-test.js
Module inside the imports /client directory:
/app-name/imports/client/client-test.js
.
Entry point in Meteor client:
/app-name/client/main.js
// => Would NOT work?
import { ServerTest } from "../../imports/server/server-test.js";
// => Would work?
import { ClientTest } from "../../imports/client/client-test.js";
UPDATE4:
Your wording here on your Update2:
Within the /imports folder, there are no special folder names - so you
can import a file from /imports/server in your client side code.
... is incorrect according to the author of this portion of the Meteor Guide.
First of all, since publications and methods are referenced only by their string name, they don't need to be imported in the code that subscribes to them or calls the functions. When doing Meteor.subscribe('publication') Meteor will try to find a publication named publication in the server, and subscribe to it. Same works with Meteor methods.
However, when using the /imports folder, your publications and methods need to be imported somewhere in server code so that Meteor loads them at all. The best practice to do this is to place a file somewhere in the imports folder, like /imports/startup/server/index.js (as recommended by the Meteor guide), where you simply import all files that declare publications and methods, and then importing this single file in some file outside the imports folder. For more about this, see the Meteor Guide and its example app.
Note also that for Meteor methods, you may want to include them also somewhere in your client code, so that the client can run simulations of them for optimistic UI before the server call returns. To do this, you can do the same as above but with a file like /imports/startup/client/index.js, that you include in the client code. I also recommend checking out the mdg:validated-method package, it makes using methods cleaner.
Update regarding the update in the question:
Yes, that seems like it would work, I think you got the point :)
As a minor detail, as I said, you might want to consider importing app-name/imports/server/trades-methods.js somewhere in your client code as well, i.e. in a file like app-name/client/main.js. This would enable the client to run a simulation of the method and update the UI immediately for better user experience. Don't do this if you don't want to expose some super-secret server code to the client though.
Update 2 in the question
Within the /imports folder, there are no special folder names - so you can import a file from /imports/server in your client side code. However, I do recommend placing code that is only meant for the server, like publications, in a folder named server, and not placing code that is meant to be used from both sides to a folder named server or client. Thus you might want to move your trades-methods.js file outside your /imports/server folder anyway. However this is only for clarity, Meteor does not care about folder names inside /server!
I really, really recommend you to read the Meteor guide, particularly the chapter on application structure and checking out the structure of the related example app. You'll save so much time in the long run!