My project currently involves several components using either Javascript and Python
Node.js server
Python flask API server
Python data processing scripts (numpy, pandas, etc)
React.js web client
React Native mobile client
These components share several common config values such as a database connection URI string postgresql://admin:password#localhost:5432/foobar used by server-side components and the API server url https://api.example.com/v1 used by client-side components.
Question: Is there a best practice for sharing config files across components that uses different languages?
For example,
Should all components (server and client side) access the same config file? If so, which format should it be in, JSON, YAML, etc?
Should all components share a common config file (where only common config values are stored in), but also have a second config file specific to that component?
Should all server side components share the same config file, while client side components have their own individual config files?
Thank you for your advices :)
I’d recommend using TOML. It’s supported by all major languages and allows the use of comments which JSON does not.
In my mind there are two main approaches you'll see in the wild.
Config file. As nc. mentioned, you could make a TOML file that all of the different tools and applications can access. This is nice since there is only one source of truth and TOML is very human readable. The cons are having to find a TOML parser for your language (imo not that difficult), and making sure the file is there when a script/service/whatever expects it. I've gone this route and its not that bad, you can even put the file in "the cloud" (NAS, shared data store, folder with broad read permissions), and have it easily accessible if you need to spin up more than one instance of a service.
Environment variables. I've liked this approach lately as it is always the minimal amount of config you could possibly do and pretty much any language has facilities for accessing environment variables. Also, you don't have possibly sensitive information hanging around in a publicly accessible plain text file.
For python you would call os.environ[MY_CONFIG_NAME] and presto, you have the config info.
Both approaches are common and have plenty of support in almost every language.
As far as client, server separation. I tend to make my clients as "dumb" as possible and my servers as "smart" as possible.
Related
I want to let users export some pieces of data in XLS files. I have a front-end application on Angular 9 and back-end on ExpressJS. Now I consider 2 ways to implement this with SheetJS: either create some ExportService to do it directly in the browser or implement this on the server-side and provide an API endpoint for exporting. To be more objective, I'd like to know what the community thinks: what are the pros and cons of the both ways?
My special concern is whether there are any functional limitations on what can be generated in browser vs. server. Aren't there some special features that are only available with server-side implementation?
Here's what I can think of. Please feel free to supplement or correct this list to build a full picture.
Client-side implementation
Pros:
no need for an extra HTTP request (unless I need to fetch some extra data; in my case, I don't)
less load on the server: everything happens on the user's machine
it can be faster than server-side implementation when server is over-loaded or internet is slow
Cons:
significantly increased bundle size: the whole SheetJS library has to be bundled
it can be slower than server-side implementation when user's machine is slow
What am I missing?
So I have an isomorphic node based application running (with react). The page is rendered on server first and then the client JS takes over. I am having the data populate first on server (via an api endpoint) and then the same api is being accessed on client side. On the server side I am using "request" library and on the client-side I am using AJAX for fetching the data. I observe that there is duplication in the code I am writing for making api calls on server and client side. Is there a way I can unify this (via some library)?
Your best bet would just be to set up your build pipelines properly to allow you to share the code. You could then have a client, server, and common directory, and tell your build tools (Webpack, Rollup, Browserify, etc., whatever you may be using) to include everything from client and common for the front-end, and common and server for the backend.
There are lots of other ways to deal with it, but that's one of the cleanest. You might also want to take a look at Meteor, which does a lot of what you're talking about for you out-of-the-box (and allows you to use React as a front-end).
I'm developing an Android application, typically with a fairly small database (hundreds to a thousand or so main entries, plus similar order of magnitude in joined tables) populated by the user. There is also a file associated with each main entry, but those are kept separate, and not part of this question.
In my current prototype, the structured data is stored in a SQLite database in the usual way, and synced between devices by uploading the file to an Application Folder on Google Drive. This allows me to avoid having to provide any user storage, which is something I hope to stick to.
Storing files here also has the happy side effect that I can access them via JavaScript code in a web page using the Google Drive API. But JavaScript interacting with a remote SQLite file is an incredibly ugly thing to attempt. (Downloading the database server-side and handling it there is another option which sounds not great either.) I am wondering if anyone has any recommendations of the best practice approach here, given I would like to have a web-based version of the app accessing the same data.
The obvious most sensible idea seems to be
Exporting from SQLite to some other structured format XXX (like XML, JSON, CSV, ...) first, uploading that XXX file to Google Drive, so that JavaScript can straightforwardly interact with it, and then on sync reimporting this XXX data back into SQLite.
though I'm happy to hear any criticisms/improvements on this idea.
I'm mainly interested in best-practice recommendations of efficient ways to approach these problems. Question: What file format XXX would be good; and do you have any recommended methods for (a) exporting and importing in Java from SQLite to XXX, (b) interacting in preferably client-side JavaScript with a remote Google Drive file in format XXX? (If maintaining server-side code to interact with XXX is substantially better practice do say.)
It depends on how many clients your app will manage
Exporting from SQLite to some other structured format XXX (like XML, JSON, CSV, ...) first, uploading that XXX file to Google Drive, so that JavaScript can straightforwardly interact with it, and then on sync reimporting this XXX data back into SQLite.
IMHO, this is quite a bad idea for quite a few reasons:
If multiple users try to change data at the same time, you will have a tough problem managing all those data entries and possible conflicts
Providing direct access to your data layer is ALWAYS a bad idea since any client can wipe out all your data
If you have very few users or are making it as a personal project, you can stick with SQLite.
Kripken, the developer of Emscripten (the C/C++ to Javascript LLVM compiler) has compiled the SQLite Library to Javascript to make the sql.js library. You can find it on GitHub. There's also a demo page of sql.js
But if you have a lot of users, you might want to use Firebase by Google. It gives you a database and a REST API to use it. There are libraries to access Firebase from different programming languages too.
Cheers!
I'm thinking of switching from PHP to using Node.js for developing my website. However, after researching Node.js for a little while, I can't seem to find exactly how to write a webpage with Node. I see that you use response.write() in Node to write html to your webpage, but that seems like a tedious thing to do, having your entire webpage as a string literal in your node file. How does web development work in Node as opposed to PHP's method of embedding the script into the HTML file?
You don't necessarily need to use response.write for each line of the view, you can use template engines as well. Search for "node.js template engines". At first impression it could seem tedious, but a similar approach prevents you from writing bad code.
PHP is a scripting language, node is a platform built on javascript.
To start web development using node.js, at first you have to understand what makes node different. Node gives you a way to make async calls to your database (which is a very simplified explanation), which you can then wrap in nice html and send (route) it to the browser. Alternatively, you can use something like angular.js in the frontend and use node.js to make db requests and response which is picked up by angular.js which updates the front html. If you like the idea of Single page app with async calls to fetch data, use node with angular. The tutorial that I like is https://scotch.io/tutorials/creating-a-single-page-todo-app-with-node-and-angular Hope this helps!
As others have answered, there exist templating engines for Node. With the current trends in web development, most modern web frameworks encourage the separation of code from the view (or the HTML you deliver to the client). For instance, Ruby's ERB templates, Jinja2 in Python, Handlebars/Jade for Node, and now a lot of modern PHP frameworks support this as well (Zend/Slim).
Another main difference is in how they work and how the languages are designed. PHP is an object oriented language supporting classes, inheritance, member visibility, interfaces, etc. Node.js is Javascript, so using prototypical inheritance.
The communities and ecosystems are different as well. Modern PHP tends to embrace the use of the Composer package manager, and that came after PEAR. However, npm is the official node package manager and it is deeply integrated with the platform. It is trivial to search for new packages and then use them in your projects.
The main architectural difference is that Node is also asynchronous by design, meaning it runs in a single thread and can potentially handle much more connections than PHP on systems with limited memory. When a request comes in to a PHP application, all the services/controllers and everything you defined have to be reinstatiated, you define PHP files and let Apache/Nginx process them. In Node you have a node process which you can proxy outside requests to.
Node.js Provides so many modules to do these things there is framework called express for node.js http://expressjs.com/ You can use a templating engine and create views. some examples are like ejs or jade. It doesnt have to be a string.
PHP is very strongly orientated towards creating web pages from a template, while Node.js is lower-level and broader in scope. A very rough overview of the differences between PHP and Node.js:
In PHP, you'd start a web server (almost certainly Apache), and then put a PHP file in a directory where you want to serve something from. You might use some fancy .htaccess directives to make URLs nicer, etc.
In Node.js you create a script, in which you use the http module to start a web server, and then you supply a callback for whenever a request is made to your server. Deciding which page to respond to the request with, etc, is all your work to do.
In PHP, things like routing a request to a particular PHP file, compression, decoding POST and GET variables, are all done by using Apache - your PHP files are sort of just like templates which Apache runs whenever a request is received. In Node.js, everything, from starting the server to sending HTML, is all done within your Node.js script - you have to do everything.
HTML isn't the first class citizen in Node.js that it is PHP. Generally, in Node.js you are just sending strings to the client. There are plenty of third party templating tools for Node.js - but they will be dependencies, not builtin functions.
We are currently developing a few web applications and are letting our designers convert signed off paper prototypes into static web pages. In addition to having hyperlinks between pages the designers have started adding jquery calls to update elements on the pages by fetching data from static json files. Once the designers are finished and handoff the completed web pages, CSS, and JavaScript files; the server-side developers then edit the pages and replace the references to the local static json documents with references to live json urls that return the same json data structures.
My question:
What are efficient ways to decouple GUI design from serverside dev and reduce the integration time and effort? Some examples:
Do you have the developers manually change every json reference in the designers' prototype web pages?
Do you add a global variable somewhere to enable the designers' pages to be easily switched back and forth between using static and dynamic data?
Do you make the web pages self-aware of when they are running from a web server or just being served from a folder somewhere?
It depends on how much state information does your application need to manage. The examples you have given mention only read operations from the server. Are there any writes? Both read and write operations can potentially fail. Do your designers take care of those cases, or the server-side team jumps in and patches up the GUI later on?
I think it's best to provide a mock server-side implementation of your services for the designers. The server mock could simulate real life behavior as in throw errors and exceptions beyond just the happy path. It really is much less of a hassle to install a simple web server nowadays and a single script that acts as a mock service, rather than trying to create processes for later integration.
One of the things I have seen work fairly successfully for the whole "url/path" problem (dev vs test vs production, etc) is to create a subdomain for your project of "proto" (so proto.mycoolapp.company.int) and have that be the basis for all urls used anywhere in the app.
On top of that using whatever framework you are using set up global page handlers to identify section/page/function+[url data] for all of your data files (i.e. json calls etc).
This allows you to segregate your app into sections and build up functionality as you go and allows both the UI team and the dev team to work on hints for the other team.
For example say I have an app with three sections (login, account management, preview) I would have a login url service (data, files, whatever) at proto.mycoolapp.company.int/frontend/login/securelogin?user=GrayWizardX. The UI team can add these as they identify a need for data and your app dev team can see what functions are being requested (through server logs, etc) and make sure everything is matching up.
The good part is that when you move to production, the only change is to find "proto" and replace it. If that is in your global var, then its a quick and simple change.
It's not real clear what exactly your environment is (plain html only? or something like php?), so this is a bit of a shot in the dark...
If it's plain old HTML + javascript, you could probably use a javascript include on each page to get the right set of addresses. When the include file with all this environment specific information (i.e. use local or use server) remains in the same relative place, you don't need to worry about modifying the actual page. Just tell the GUI guys to always use this same set of variables for the address information and define the address information in that include file. The variable names don't change whether your getting locally or from the server, just the values stored in the variables.
I'm low on Vitamin Coffee, so hopefully that makes sense.