We Keep Coding

dynamic javascript loading

I have some abstract question.
Lets say I have a website which uses 1000+ JS script files. (Not at once, but in the overall 1000+ JS scripts are needed if user goes through all existing flows).
I know that using RequireJS I could dynamically load whichever scripts I need, and that saves me a long time loading them all at once.
But!
In order for RequireJS to load the JS files, they need to be downloaded by the client.
So what happens is that the first time I open the web page, I download more than 1000 JS script files, which takes long time too!
I was wondering if there's any way for dynamic loading to also dynamic download them from remote.
I know that for this to work best, there must be some clever mechanism that will finish download them before something tries to use them, otherwise it'll slowdown things.
Anyway, I hope you understand my question, I want to avoid downloading 1000+ files on the first time I open the page. I'd like them to be downloaded on demand.
I can't make combine them to a single file like many web sites do.
Thanks for your help and discussion :)

My guess is that you're trying to achieve 'lazy loading'.
You can use this plugin for require:
https://github.com/nikospara/require-lazy
Read more here:
http://www.joezimjs.com/javascript/lazy-loading-javascript-with-requirejs/
The main idea here is that, instead of loading all the dependencies when the page loads, you provide a method for loading the necessary items, and you call that method when needed
For example: (load files after a user clicks a button)
$('#somebutton').on('click', loadFiles);
function loadFiles() {
require(['every', 'javascript', 'dependency', 'for', 'other', 'screen'],
function(ev, js, dep, fr, othr, scrn){
// Create the screen with your dependencies
});
}

Is there a danger to creating a js file that simply uses document.write() to bundle js and css files?

I am building a front-end UI framework for developers in my firm to use to build internal web apps. It consists of customized Bootstrap, jQuery, other open-source libraries, internal modules and stylesheets. The user environment is entirely IE9 and my server is .NET 3.5. I am hosting the shared files. Dev teams in the firm will place the links in their project pages and apply the framework to their pages.
I want to offer them the simplest method of implementing this which would be one line of code to paste that builds the library for them. Cutting and pasting 30 lines of code is stale the moment Ctrl + V is pressed, it leaves me no control and is simply inelegant.
Failed Experiments
I tried using Head.js and LazyLoad both of which use best practices for inserting scripts. But each of them has caused either content to display before styled or conditions where methods are called before scripts load. I am giving up on this approach.
It's too volatile.
A simple document.write() solution
Over the weekend, I thought: Why don't I just make a js file named "framework,js", add the script and link files in order with a stack of document.write() lines. Tell developers to put it in the head and that's it. Heck I could add the necessary metatags for IE9 and mobile too for that matter. It's so nasty and simple... but it just might work!
The user base is on an internal network and of limited size. Bandwidth is not a problem. I'll test performance before I choose this. I can direct the developer teams on where to place the link.
Knowing this and providing it actually works, is there any reason why I shouldn't do this?
My only other option to explore is bundling on the server. I am hoping not to have to resort to this since I don't own the server myself and I am not a .NET developer.

Your proposed approach is an excellent one for your situation. It has several advantages over the more sophisticated solutions, including utter simplicity, completely predictable order of execution, and full compatibility with scripts that may not lend themselves to asynchronous loading. I have used it many times in production applications, as have many other developers.
Of course the other solutions have their advantages too, but for what you're doing there is nothing wrong with good old document.write().
It sounds like you have a number of scripts and stylesheets and are probably loading them from a common directory (or a common root directory). To reduce repetition in your framework.js file, you might want to define two functions, one to write a <link> tag for CSS and another to write a <script> tag for JavaScript. So a skeleton framework.js file might look something like this:
(function() {
var scriptBase = 'js/';
var styleBase = 'css/';
function writeStyle( name ) {
document.write(
'<link rel="stylesheet" href="', styleBase, name, '">',
'</link>'
);
}
function writeScript( name ) {
document.write(
'<script src="', scriptBase, name, '">',
'</script>'
);
}
writeStyle( 'one.css' );
writeStyle( 'two.css' );
writeScript( 'one.js' );
writeScript( 'two.js' );
})();
Note that you don't have to do any special escaping of the </script> text as you may see in code that uses document.write(). That's only necessary when you're putting this code directly inside a <script> tag within the HTML file. The purpose of that escaping is to prevent the enclosing <script> tag from being closed by the </script> text inside the document.write() call. Since your code is in an external .js file this is not an issue: the presence of the </script> text inside this file won't terminate the .js file.
Another point to keep in mind is that all of the document.write() calls you make inside a .js file are inserted into the document after the .js file that writes them. So don't expect to be able to do a document.write() inside framework.js and then have other code inside framework.js that depends on the .js file you just wrote. All of those .js files (and .css) are loaded after framework.js, not interleaved with it.
One more consideration, of course, is load time. Your page could load faster if you combine and minify your CSS and JS files. But if these are internal web apps, page load time may be the least of your worries: reliability and maintainability may be more important. And in any case you can always use the document.write() solution to get up and running right now, and later optimize this only in the unlikely event that you need to.

Mastering external scripts loading order in Meteor (Google Maps)

I tried unsuccessfully to add a google map(externally loaded script) to a meteor app, and I noticed there were two kinds of problems:
If I do the simple thing and add the main API script to my <head></head>, then it gets rendered last.
When this happens, I am obliged to insert any scripts that depend on the API again in my template's <head> - after the main API script. (otherwise scripts complain they don't see the API blabla..)
Then the time for the actually function call comes - and now putting it inside <head> after the rest won't work. You need to use Template.MyTemplate.rendered.
Basically my question is:
What's the cleanest way to handle these kinds of things?
Is there some other variable/method I can use to make sure my Google main API file is called very first in my HTML?

I just released a package on atmosphere (https://atmosphere.meteor.com) that might help a bit. It's called session-extras, and it defines a couple functions that I've used to help with integrating external scripts. Code here: https://github.com/belisarius222/meteor-session-extras
The basic idea is to load a script asynchronously, and then in the callback when the script has finished loading, set a Session variable. I use the functions in the session-extras package to try to make this process a bit smoother. I have a few functions that have 3 or 4 different dependencies (scripts and subscriptions), so it was starting to get hairy...
I suppose I should add that you can then conditionally render templates based on whether all the dependencies are there. So if you have a facebook button, for example, with helpers that check the Session variables, you can give it a "disabled" css class and show "loading facebook..." until all the necessary scripts have loaded.
edit 03/14/2013
There is also an entirely different approach that is applicable in many cases: create your own package. This is currently possible with Meteorite (instructions), and the functionality should soon be available in Meteor itself. Some examples of this approach are:
jquery-rate-it: https://github.com/dandv/meteor-jquery-rateit
meteor-mixpanel: https://github.com/belisarius222/meteor-mixpanel
If you put a js file in a package, it loads before your app code, which is often a good way to include libraries. Another advantage of making a package is that packages can declare dependencies on each other, so if the script in question is, for example, a jQuery plugin, you can specify in the package's package.js file that the package depends on jQuery, and that will ensure the correct load order.
Sometimes it gets a little more interesting (in the Chinese curse sense), since many external services, including mixpanel and filepicker.io, have a 2-part loading process: 1) a JS snippet to be included at the end of the body, and 2) a bigger script loaded from a CDN asynchronously by that snippet. The js snippet generally (but not always!) makes some methods available for use before the bigger script loads, so that you can call its functions without having to set up more logic to determine its load status. Mixpanel does that, although it's important to remember that some of the JS snippets from external services expect you to set the API key at the end of the snippet, guaranteed to be before the bigger script loads; in some cases if the script loads before the API key is set, the library won't function correctly. See the meteor-mixpanel package for an example of an attempt at a workaround.
It's possible to simply download the bigger js file yourself from the CDN and stick it in your application; however, there are good reasons not to do this:
1) the hosted code might change, and unless you check it religiously, your code could get out of date and start to use an old version of the API
2) these libraries have usually been optimized to load the snippet quickly in a way that doesn't increase your page load time dramatically. If you include the bigger JS file in your application, then your server has to serve it, not a CDN, and it will serve it on initial page load.

It sounds like you're loading your Javascript files by linking it with HTML in your template. There's a more Meteor way of doing this:
From the Meteor Docs:
Meteor gathers all JavaScript files in your tree with the exception of
the server and public subdirectories for the client. It minifies this
bundle and serves it to each new client. You're free to use a single
JavaScript file for your entire application, or create a nested tree
of separate files, or anything in between.
So with that in mind, rather than link the gmaps.js into head, just download the un-minified version of gmaps and drop it in you application's tree.
Also from the Meteor Docs:
It is best to write your application in such a way that it is
insensitive to the order in which files are loaded, for example by
using Meteor.startup, or by moving load order sensitive code into
Smart Packages, which can explicitly control both the load order of
their contents and their load order with respect to other packages.
However sometimes load order dependencies in your application are
unavoidable. The JavaScript and CSS files in an application are loaded
according to these rules:
Files in the lib directory at the root of your application are loaded
first.
[emphasis added]
And if the sequence is still an issue, drop the js file into client/lib and it will load before all the Javascript you've written.

I've used meteor-external-file-loader and a bit of asynchronous looping to load some scripts, which will load javascript (or stylesheets) in the order you specify.
Make sure to have meteorite and add the package above >> mrt add external-file-loader
Here's the function I wrote to make use of this package:
var loadFiles = function(files, callback) {
if (!callback) callback = function() {};
(function step(files, timeout, callback) {
if (!files.length) return callback();
var loader;
var file = files.shift();
var extension = file.split(".").pop();
if (extension === "js")
loader = Meteor.Loader.loadJs(file, function() {}, timeout);
else if (extension === "css") {
Meteor.Loader.loadCss(file);
loader = $.Deferred().resolve();
}
else {
return step(files, timeout, callback);
}
loader.done(function() {
console.log("Loaded: " + file);
step(files, timeout, callback);
}).fail(function() {
console.error("Failed to load: " + file);
step(files, timeout, callback);
});
})(files, 5000, callback);
}
Then to use this, add to one of your created methods for a template like so:
Template.yourpage.created = function() {
var files = [
"//ajax.googleapis.com/ajax/libs/jqueryui/1.10.3/jquery-ui.min.js",
"javascripts/bootstrap.min.js"
];
loadFiles(files, function() {
console.log("Scripts loaded!");
});
}
Quick edit: Found it's just a good idea to place the functionality from the .created method in the /lib folder in Meteor.startup();
Meteor.startup(function() {
if (Meteor.isClient) {
// Load files here.
}
});
Caveat: Lots of javascript files = really long load time.... Not sure how that will be affected by the normal meteor javascript files, and the load order there. I would just make sure that there are no conflicts until users take action on the website (or that if there is, they are loaded first).

Page-level execution of JavaScript when serving concatenated files

Scenario:
A web site with x number of pages is being served with a single, concatenated JavaScript file. Some of the individual JavaScript files pertain to a page, others to plugins/extensions etc.
When a page is served, the entire set of JavaScript is executed (as execution is performed when loaded). Unfortunately, only a sub-section of the JavaScript pertains directly to the page. The rest is relevant to other pages on the site, and may have potential side-effects on the current page if written poorly.
Question:
What is the best strategy to only execute JavaScript that relates directly to the page, while maintaining a single concatenated file?
Current solution that doesn't feel right:
JavaScript related to a specific page is wrapped in a "namespaced" init function for that page. Each page is rendered with an inline script calling the init function for that page. It works hunky-dory, but I would rather not have any inline scripts.
Does anyone have any clever suggestions? Should I just use an inline script and be done with it? I'm surprised this isn't more of an issue for most developers out there.

Just use an inline script. If it's one or two lines to initialize the JavaScript you need that's fine. It's actually a good design practice because then it allows re-use of your JavaScript across multiple pages.

The advantages of a single (or at least few) concatenated js files are clear (less connections in the page mean lower loading time, you can minify it all at once, ...).
We use such a solution, but: we allow different pages to get different set of concatenated files - though I'm sure there exists different patterns.
In our case we have split javascript files in a few groups by functionality; each page can specify which ones they need. The framework will then deliver the concatenated file with consistent naming and versioning, so that caching works very well on the browser level.
We use django and a home-baked solution - but that's just because we started already a few years ago, when only django-compress was available, and django compress isn't available any more. The django-pipeline successor seems good, but you can find alternatives on djangopackages/asset-managers.
On different frameworks of course you'll find some equivalent packages. Without a framework, this solution is probably unachievable ;-)
By the way, using these patterns you can also compress your js files (statically, or even dynamically if you have a good caching policy)

I don't think your solution is that bad although it is a good thing that you distrust inline scripts. But you have to find out on what page you are somehow so calling the appropriate init function on each page makes sense. You can also call the init function based on some other factors:
The page URL
The page title
A class set in the document body
A parameter appended to your script URL and parsed by the global document ready function.

I simply call a bunch of init functions when the document is ready. Each checks to see if it's needed on the page, if not, simply RETURN.

You could do something as simple as:
var locationPath = window.location.pathname;
var locationPage = locationPath.substring(locationPath.lastIndexOf('/') + 1);
switch(locationPage) {
case 'index.html':
// do stuff
break;
case 'contact.html':
// do stuff
break;
}

I'm really confused exactly why it doesn't feel right to call javascript from the page? There is a connection between the page and the javascript, and making that explicit should make your code easier to understand, debug, and more organized. I'm sure you could try and use some auto wiring convention but I don't think it really would help you solve the problem. Just call the name spaced function from your page and be done with it..

jquery and script speed?

Quick question, I have some scripts that only need to be run on some pages and some only on a certain page, would it be best to include the script at the bottom of the actual page with script tags or do something like in my js inlcude;
var pageURL = window.location.href;
if (pageURL == 'http://example.com') {
// run code
}
Which would be better and faster?

The best is to include the script only on pages that need it. Also in terms of maintenance your script is more independant from the pages that are using it. Putting those ifs in your script makes it tightly coupled to the structure of your site and if you decide to rename some page it will no longer work.

I can recommend you to use an asynchrounous resource loader, LAB.js for example. Then you could build a dependencies list, for instance:
var MYAPP = MYAPP || {};
/*
* Bunches of scripts
* to load together
*/
MYAPP.bunches = {
defaults: ["libs/jquery-1.6.2.min.js"],
cart: ["plugins/jquery.tmpl.min.js",
"libs/knockout-1.2.1.min.js",
"scripts/shopping-cart.js"],
signup: ["libs/knockout-1.2.1.min.js",
"scripts/validator.js"]
/*
... etc
*/
};
/*
* Loading default libraries
*/
$LAB.script(MYAPP.defaults);
if (typeof MYAPP.require !== 'undefined') {
$LAB.script(MYAPP.dependencies[MYAPP.require]);
}
and in the end of your page you could write:
<script type="text/javascript">
var MYAPP = MYAPP || {};
MYAPP.require = "cart";
</script>
<script type="text/javascript" src='js/libs/LAB.min.js'></script>
<script type="text/javascript" src='js/dependencies.js'></script>
By the way, a question to everyone, is it a good idea to do so?

In so far as possible only include the scripts on the pages that requirement. That said, if you're delivering content via AJAX that can be hard to do, since the script might already be loaded and reloading could cause problems. Of course you can deliver code in a script block (as opposed to referencing an external js file), in code delivered via AJAX.
In cases where you need to load scripts (say via a master page) for all pages, but that only apply to certain pages, take advantage of the fact that jQuery understands and deals well with selectors that don't match any elements. You can also use live handlers along with very specific selectors to allow scripts loaded at page load time to work with elements added dynamically later.
Note: if you use scripts loaded via content distribution network, you'll find that they are often cached locally in the browser anyway and don't really hurt your page load time. The same is true with scripts on your own site, if they've already been loaded once.

You have two competing things to optimize for, page load time over the network and page initialization time.
You can minimize your page load time over the network by taking maximum advantage of browser caching so that JS files don't have to be loaded over the network. To do this, you want as much javascript code for your site in on or two larger and fully minimized JS files. To do this, you should put JS for multiple different pages in one common JS file. It will vary from site to site whether the JS for all pages should be ine one or two larger JS files or whether you group it into a small number of common JS files that are each targeted at part of your site. But, the general idea is that you want to combine the JS code from different pages into a common JS file that can be most effectively cached.
You can minimize your page initialization time by only calling initialization code that actually needs to execute on the particular page that is being displayed. There are several different ways to approach this. I agree with the other callers that you do not want to be looking at URLs to decide which code to execute because this ties your code to the URL structure which is better to avoid. If your code has a manageable number of different types of pages, then I'd recommend identifying each of those page types with a unique class name on the body tag. You can then have your initialization code look for the appropriate class on the body tag and branch to the appropriate initialization code based on that. I've even seen it done where you find a class name with a particular common prefix, parse out the non-common part of the name and call an initialization function by that name. This allows you to give a page a specific set of behaviors by only adding a classname to the body tag. The code remains very separate from the actual page.
the less general purpose way of doing this is to keep all the code in the one or two common JS files, but to add the appropriate initialization call to each specific page's HTML. So, the JS code that does the initialization code lives in the common JS files and thus is maximally cached, but the calling of the appropriate initialization code for that page is embedded inline in each specific page. This minimizes the execution time of the initialization, but still lets you use maximal caching. It's slightly less generic than the class name technique mentioned earlier, but some may like the more direct calling technique.

Include scripts at bottom of pages that need it only.

The YSlow add-on is the best solution to know why your website is slow.
There are many issues which could be the reason for slowness.
Combining many jQuery to one could help you increasing your performance.
Also you can put the script at the bottom of your page and CSS at top.

Its basically up to you and depends on what the code is.
Generally with small things I will slip it into the bottom of the page. (I'm talking minor ui things that relate only to that page).
If you're doing the location ref testing for more than a couple pages it probably means you're doing something wrong.
You might want to take a look at one of these:
http://en.wikipedia.org/wiki/Unobtrusive_JavaScript
http://2tbsp.com/node/91
And as for which is faster it's wildly negligible, pick what is easier for you to maintain.