We Keep Coding

Organising a client & server SPA/Node.js application

I am writing an application with a Node.js backend and a single-page web app front-end.
I am keeping the client and server logic in the same project for simplicity and speed of development.
I am considering how best to organise the artifacts.
The Node.js part is straightforward because it doesn't need to go through a battery of pre-processors (transpilation, minification, concatenation etc).
The front-end needs to be transformed per the above, and I guess placed in a dist folder.
The current hierarchy of files is like so:
my-app
- src
- client
- server
Should I put the dist folder for the client artifacts under src/client?
Has anyone tried this and found problems with this approach?
I am using Heroku (a deployment system that uses git).
Committing the built artifacts for the client feels wrong, but if I want to deploy it by pushing to Heroku I think I need to commit them. Is this correct?

This question, as is, invites opinionated answers, so I'll start by saying this is by no means the only way to go, but in my opinion, it is the easiest to work with and makes the most sense.
The production client code, after pre-processing, should be located in my-app/dist or my-app/dst, which could either mean distribution or destination, depending on how you look at it. Either way, my recommendation is to commit this folder, as it saves you a lot of hassle debugging remotely.
For example, if your code works locally but not remotely, using something like the postinstall hook to generate your dist folder adds yet another suspect to check when trying to determine what the issue is with your program.
Another advantage of committing the dist folder is it allows you to specify all the packages you use for your build process as devDependencies rather than dependencies. This is a huge plus, and makes deployment a lot faster, as well as less memory usage on your heroku process.
That being said, I still recommend (as you already probably plan to do) using an automated watch task to build your dist folder for ease of development, even if you decide you don't want to use that same build process remotely and opt for committing the dist directory instead. You could add that as a custom npm command, e.g. npm run build and have that invoke your gulp task.
One last thing. For those of you using templating languages like pug or dust or ejs instead of a framework like react or angular, I recommend determining whether you can run any of your templates to build static HTML files that will be served in production.
If not, you should at least compile your templates (not to be confused with running them) by following the recommendations provided by your particular templating language. Typically, they'll suggest using their command line utility to generate the compiled templates, so that they don't have to be compiled every time they're invoked in production. This will make your node.js server respond faster to requests at the expense of using more memory to cache the compiled templates.

If you're planning to go this route, I would edit nodejs app.js/index.js to serve static file and point the directory to dist/.
Also, you would need to tell express to forward all non-api requests to the frontend.

Version control strategy for webpack-encore project

I'm learning to use webpack-encore and noticed it is installed only as a dev dependency. Does that mean I should compile my js and css files on development and push them to the repository, and then to production?
That seems to me what the docs are implying, but wouldn't that mean a merge-conflict hell? Compiled files would be impossible to merge.
Also wouldn't that be contrary to version control philosophy? As far as I know, you don't publish binaries in compiled languages (i.e. C/C++), you push the code and expect the server to compile them. I know this isn't the same type of "compilation" in javascript, but what is the expected behavior of the production server in this case? To receive the files ready to serve them, or to compile them at the time of release?
Thanks in advance

Does that mean I should compile my js and css files on development and push them to the repository, and then to production?
Not exactly - it depends on how you deploy.
When you deploy, you need to run ./node_modules/.bin/encore production to build your assets. Once you've done this, only your built assets (e.g. web/build) need to be transferred to production.
You could run this command locally (or on some "build" server) and the transfer all the files to production. Or, you could use a git pull on production, and then run this command on production (the downside being that you would need Node.js installed on production).
You shouldn't / don't need to commit your built files to your repository. But... if it simplifies your deploy (i.e. you want to do a git pull and be done), there's no real problem with that.
I just added a PR to answer these in the FAQ (http://symfony.com/doc/current/frontend/encore/faq.html) - here's the PR until it's deployed: https://github.com/symfony/symfony-docs/pull/8109
Cheers!

Solution 1:
Run yarn run encore production locally
Check out which files have been created / modified
Add them to VCS
Commit
Push / deploy
Solution 2:
Push / deploy
Run yarn run encore production remotely during deployment
To my eyes the 2nd solution is way better, because you don't need an extra human-checking before deployment, everything is automated.
But this has a strong drawback: building assets can be a slow process, and when I deploy, my production is down during 5 to 20 seconds until assets are built.
Here's the HTTP 500 error:
An exception has been thrown during the rendering of a template ("Asset manifest file "[...]/web/build/manifest.json" does not exist.").
It looks like the manifest.json file is deleted at the beginning of the process, and created from scratch later on.
Something that should be improved?

How do front end devs bundle and minify files?

What's the best practice for minifying and bundling js/css in a pure front end app, and how do the tools work?
I know how this can be done with server side apps like .NET/Java/LAMP/etc. But what about pure front end projects, SPA projects or backendless projects that are built with say, ember or angular these days? Say your entire project consists of HTML/css/js, which interfaces with a RESTful service elsewhere.
What kind of process or tool do you use to minify and bundle the resources for that?
I've seen grunt plugins that exist for this, but I find the documentation to be pretty magical and it's still unclear to me how they work.
Specifically, does the tool:
1) Replace src="/js/a.js",src="/js/b.js" with src="/js/bundle-a+b.min.js"? (and likewise with css?) in the source html files?
2) have different modes for dev and release, or is the tool only run when the project is released?
Or are the resource requests entirely managed by a js tool and js/css files have to be requested via a library function? Wouldn't the lag be noticeable in this case?
Thanks.

Through the use of Build tools front end devs can have minified javascript, css, or even images and html files automatically minified as they develop. The most common is grunt, with gulp close behind.
You configure grunt tasks, like grunt-contrib-uglify and grunt-contrib-copy, and put those tasks under a grunt-contrib-watch task. Have the grunt watch task watch the files you modify, and every time a change is detected those .min files are automatically generated.
These build tools have no impact on your application, they are run before the files are servered. You were correct to assume there was an easy way to do this. I suggest you look at grunt getting started, a sample gruntfile, or a project that uses grunt - here's mine, it does minification like you requested. Clone my repo, run sudo npm install, then sudo grunt. I don't have watch set up in my project but grunt is very well documented.

Deployment Strategy for Require JS Optimized/Concatenated Website Files

My question is partly technical and partly about deployment strategies and workflow. I built a project using Require JS. It includes a number of distinct js modules, and is built upon Kirby CMS. The directory structure of the project is something like this:
project
assets
styles
style.css
js
scripts
script1.js
script2.js
script3.js
vendor
app.js
images
fonts
content
...
kirby folders
....
The file app.js is called in the footer of my site's page like so:
<script data-main="/assets/js/app" src="/assets/js/vendor/require.js"></script>
It configures RequireJS by calling the requirejs.config() function and then calls the main script file that loads everything else using RequireJS's requirejs() function.
I've used RequireJS' s optimization tool to compile the project in such a way that the optimized files are all dumpted into a directory called dist (a name I just picked up from this tutorial). So in the end dist contains a replication of every directory and file under assets, only optimized, and the file app.js is a concatenated and optimized version of all the js modules that I have in the project. So far so good.
What I am unsure about, however, is how I'm the supposed to make use of this new secondary version of all the code. What for instance if I want to deploy a version of the site to the production server without all the source js files? Each time I deploy the site, I would need to go through my code and in every place that I referred to files under the assets directory, I would need to replace that with dist. I deploy using git and beanstalk. One way to do this would be to manage different branches for staging, production, and development, in which the production and perhaps staging branches have references to the files under dist, but this seems awkward.
So my question is given this kind of optimization set up, which if you look at the tutorial linked above is one way to do this, how then do you manage the switch to the optmized version of everything seemlessly, without having to go back into your code and change everything up? Is there some key part of the process that I'm missing here?

Each time I deploy the site, I would need to go through my code and in every place that I referred to files under the assets directory, I would need to replace that with dist.
I've looked at the tutorial you've linked to and do not see how it is true for the tutorial. The tutorial does not use absolute paths, so should be deployable from dist just as well as from the directory that contains the pre-optimization sources. If you cannot do this for your application, that's because you've done something different from the tutorial. Your script tag, for instance, shows absolute paths.
So the solution is to design your application to avoid absolute paths. This way, you won't have to change paths when you deploy from dist. I'm using this very method to deploy optimized and non-optimized versions of one of my apps.

Include external JS in Rails asset pipeline that gets bundled during deploys

I reviewed this question already - Including external libraries using the Rails 3.1 asset pipeline - which is basically what I'm looking for with one caveat: I'd like to include the external url in the asset pipeline so that when I deploy to production, the external url gets bundled into a file with the other local files that were required.
More details:
My use case here is that I have found a plugin on GitHub that I like, and being a good member of the open source community, I'd like to help contribute back. The most time efficient technique that I'm considering is to fork the repo, then point my local project that needs that file to the raw source of the required JS file during development, but have that file get bundled with the rest of my application when I deploy to staging / production. This would allow me to keep the plugin tied closely to the parent project, but keep things on my own track so that I can decide when to merge in updates from the base project as well as submit any fixes I have back to the parent with easy pull requests.
With that said, I have not been able to find any documentation about how to do this with the current rails JS asset pipeline or if it is possible at all. Some quick tests locally point to the idea that this works in a css file but not for js files. Can anyone confirm if this is possible? Thanks!

I would use something like this custom rake task. Basically what you do is write that rake task into the beginning of your deploy script, but modified to put the file in your vendor/assets folder (or where ever you want that is pulled into the asset pipeline.)
That way you pull the requested file on deploy, but it's there in your code when your asset pipeline bundles everything up.
Here is the code in-case the link rots:
namespace :remote_file do
desc "Get a file from a remote server"
task :fetch do
# based on http://snippets.dzone.com/posts/show/2469
# http://farm1.static.flickr.com/92/218926700_ecedc5fef7_o.jpg
Net::HTTP.start("farm1.static.flickr.com") do |http|
resp = http.get("/92/218926700_ecedc5fef7_o.jpg")
open("fun.jpg", "w") { |file| file.write(resp.body) }
end
end