My previous post on how to structure an Angular application was well received, and by and large the structure I discussed in that post served very well for the development of an enterprise level application that is still in development and use today.
But I moved jobs, and in my new role I’ve been tasked with creating, from the ground up, an Angular based solution to providing web client front ends for over 700 programs that are currently accessed via text based Unix command line interfaces.
“700 applications hey?” I asked with a raise of one eyebrow, “That’s a lot. We won’t be building all of those at once. And we won’t want a given user to have to download all of those either, especially as they’re only likely to use a couple in any given session.”
“Sounds like a job for lazy loading!” I cried while placing my hands on my hips Peter Pan style.
“OK, have at it.” They said, and I was off excitedly to explore how far Angular had moved on since I last really checked it’s progress, and find out how to get things loading in a lazy manner.
What we needed
The requirements were that there would be a main ‘control’ application that would handle login and session management and then a user could select one of the hundreds of applications to load up into the ‘program’ area of the application. This list of available programs will be growing for a long time, and the main application should have no coded knowledge of them, and on a per client basis, they will likely be given access to only a handful of the total set. So we need a solution that can dynamically load these sub applications into the main Angular application at runtime. And this is no small feat it would seem. Angular is not designed to have new pieces added to it after it starts up, it likes to know about all its modules, controllers, factories etc. right up front so it can organise its dependency injection.
So, we have to solve that, and also it’d be really great to bundle our resultant code into as few files as possible for production to reduce the overhead of http requests… so some form of optimization/build process is a must.
What’s out there?
I spent a long time researching the various solutions for this problem, and discovered that our set of requirements meant that a number of the popular solutions were not suitable
- WebPack: I got really excited about Webpack for a while. Taking modular bits of application, packing them neatly, and then loading on demand… awesome. And they get generated just by the way you write your code, It’ll work it out for you? Cool. And Instagram uses it for their website, which makes it seem production ready. This seems fantastic! It got me going down the RequireJS path, and loading my dependencies that way, so that was great. Except then when it came to making sure that the main application was defined as a standalone pack, and then each program was also one, and that what was in or out of those parts wasn’t fluid (so we could make sure if we were doing a release of program657, then it wouldn’t be affecting any other program) it started falling down. And being that our programs aren’t referenced in the code at all, but are loaded in dynamically at runtime, it couldn’t work things out quite right. And on top of that you have to have it building its bundles as you code, so the code you’re developing with is bundled, which I find really hard to debug… and so I reluctantly waved goodbye to Weback.
- OC LazyLoad: I found this early on, and thought it looked awesome, just what I needed, a way to load things on demand in Angular, it handles getting things registered properly when you add them. Great! Except then I started looking into using AMD style module definitions, and OCLazyload doesn’t really like working with RequireJS. (The author himself doesn’t recommend it). And then when I was working on breaking the application into the multiple, lazy loaded pieces and loading them on demand I discovered it didn’t like that. And someone had asked the question about how to do that, how to have different optimized files that get loaded on demand, and the author couldn’t understand why you’d want to do that… so I ruled it out.
- RequireJS: So, now that it’s entered my mind to actually organise Angular code using RequireJS, what about just using it? Well, yes, it works, you can define dependencies for any given file using it’s dependency system, which pushes these definitions down the chain and makes things more modular. Nice. What about lazy loading? Yeah, it handles on the fly requirements, but with Angular? Yeah, not so much, it doesn’t help with Angular’s need there at all.
- Browserify: Great for turning your code into ‘bundles’, but again, works via having things be built while you develop via a code watcher, and isn’t geared around dynamic loading at all.
- Angular’s newNgRouter: Ooh, a new router from the Angular team… from Angular 2, but backported to 1.x… features lazy loading… niiicee. Oh, it’s horrendously unstable and doesn’t really work and is constantly changing. OK then.
- AngularAMD: Ok, now this has the two things I’m interested in right there in the name of the library, Angular and AMD (Asynchronous Module Definition), this looks promising. You can run the code you write in development straight on your dev server, no need for building, so I like that immediately. And yet if we use the RequireJS optimizer, it looks like we can create nicely packaged code for our final production build.
Using AngularAMD
AngularAMD’s reason for existence is to facilitate using RequireJS with Angular, and lazy loading portions of your application. So this tied the two things I decided I wanted to use together. And now it looks like this is what I’m going to use going forward.
By way of example, let’s build an application like this (And yes, the code is available on GitHub):
main-app/ |--config/ |--controllers/ |--modules/ |--templates/ |--app.js |--init.js programs/ |--program1/ |--controllers/ |--templates/ |--main.js |--module.js index.html
And we only want the main-app part to load initially, we want what’s in program1 to load on demand, and the existence of that program can be passed in via a config JSON file. This will form the pattern by which we can add as many programs as we like under programs, and only load those into the browser that we need on a per session basis.
index.html
So, one of the cool things about using RequireJS to handle dependencies is that it unloads all those script includes from the index.html page (which I know you get rid of during a build, but it’s a pain to maintain during dev). You end up with code like:
<!DOCTYPE html> <html> <head> <title>Dynamic example</title> <script data-main="main-app/init.js" src="bower_components/requirejs/require.js"></script> </head> <body> <div class="appWrapper" ui-view="appWrapper"></div> <div class="appProgram" ui-view="program"></div> </body> </html>
and you don’t have to touch it when you add new modules, directives, factories, anything, because they all get added in as dependencies of whatever actual part of your app needs them, and nowhere else. It’s really pretty cool.
So, all that code is doing is loading require.js and telling it to load init.js from the main-app folder. And that’s where the fun begins.
init.js
require.config({
baseUrl: "",
// Paths for just the core application and its controllers/factories/services
paths: {
"angular": "bower_components/angular/angular",
"jquery": "bower_components/jquery/dist/jquery",
"angular-ui-router": "bower_components/angular-ui-router/release/angular-ui-router",
"angularAMD": "bower_components/angularAMD/angularAMD",
"app": "main-app/app",
"templateStorage":"main-app/modules/templates"
},
// Add angular modules that does not support AMD out of the box, put it in a shim
shim: {
//Tell requirejs to pipe in angular's return variable as "angular"
"angular": {exports: "angular"},
"angularAMD": ["angular"],
"angular-ui-router": ["angular"]
},
// Say we have a dep on App, so it gets loaded
deps: ["app"]
});
This is where we have some configuration of requireJS to give friendly names to things we want to include within our application, and also give the name ‘app’ to the application itself within the paths section. The shim block is where you list modules that are not AMD modules already, and have them wrapped up as if they were, and also allows you to specify dependencies for them, so you can say that angularAMD requires angular before it tries to load.
The last thing in the file is the deps array, which are the things that need to now be loaded, which in our case is the application.
app.js
This is our main application module. Now this is messy, there’s too much in here, it’s not how you really want a main Angular application file to be, you’d split functionality off into providers and factories and managers… but for the sake of keeping the total number of files down for this demo, and the flow more obvious, I give you:
define(["angularAMD","angular-ui-router","require","templateStorage"],function (angularAMD) {
//1. Define the application
var app = angular.module("app", ["ui.router","templateStorage"]);
//listing dependancy on controllers here. AFTER app is defined, as they require it
//but Before defining the config, as that requires them to be available
require(["./main-app/controllers/main"],function(includes){
app.config(["$stateProvider", "$locationProvider","$urlRouterProvider", function ($stateProvider, $locationProvider,$urlRouterProvider) {
$urlRouterProvider.deferIntercept(); //Stop path resolution until we've loaded them all.
//Set up persistant pointers to the state and urlRouter providers for dynamically adding things to them
app._stateProvider = $stateProvider;
app._urlRouterProvider = $urlRouterProvider;
$stateProvider
.state("app",{
url: "/",
authenticate: true,
views:{
"appWrapper":{
templateUrl:"main-app/templates/header.html",
controller: "appCtrl"
},
"program":{
templateUrl:"main-app/templates/home.html",
controller:"homeCtrl"
}
}
});
}]);
app.run(["$http","$rootScope","$location","$urlRouter","$state", function($http,$rootScope,$location,$urlRouter,$state){
$http.get('main-app/config/validComponents.json').then(function (resp) {
angular.forEach(resp.data, function (item) {
var prog = item.name;
var stateName = "app."+prog;
var state = {
url: prog,
authenticate: item.authenticate,
views: {
"program@":angularAMD.route({
templateUrl: "programs/" + prog + "/templates/" + prog + ".html",
controllerUrl: "programs/" + prog + "/main.js",
controller: prog+"Ctrl"
})
}
};
app._stateProvider.state(stateName, state);
});
app._urlRouterProvider.otherwise("/");
angularAMD.processQueue(); //Process things added to config/run blocks
$urlRouter.sync();
$urlRouter.listen();
});
}]);
angularAMD.bootstrap(app);
});
return app;
});
This shows how we now wrap all our files in define statements, these define what the file is to RequireJS. The first array tells RequireJS what other things that this module needs to run, its dependencies, and so RequireJS will make sure they are loaded before it runs the code within this file. This is the magic of RequireJS, and the feature which really cleans up the file management of your web application. Rather than having to load in every script you’re going to need to run the application up front, you just add things you need for a given module in its dependency array, and you’re done, RequireJS will handle the loading of those dependencies, and if they have their own, it’ll load those if they haven’t yet been, and so on.
So, you can do this within the file’s main define also. Here I create the application
var app = angular.module("app", ["ui.router","templateStorage"]);
And then, because the app is now defined, I can include in any controllers, factories, services, whatever I want to hang off the main app, and then move forward with the application config and run blocks knowing that my application object are ready:
Having a require statement of the form:
require(["SomeDependency","AnotherDependency"],function(dep1ret, dep2ret){
means that it’ll load whatever you’ve listed in your dependency array, and then run the function you define, passing it the returns from those dependencies, so, when we do it in this app:
require(["./main-app/controllers/main"],function(includes){
app.config(...
we are making sure we’ve loaded our app level controllers before we move on.
The biggest thing we now do in the config block is to save references to our state and url providers to we can then use the in our run block. There are some things you can only access from config blocks in Angular, and some you can only get to from the run block. And in order for us to load our states from a file (like in this demo), or from a server (like in the real world), you need to be able to do that within the run block, when the state provider is not injectable. So we save them for this very use.
Then, we configure our main application states, in the boring old, normal, static way. Here we’re using ui-router as it’s a lot more flexible than the standard Angular router.
In the run section, where we can actually load things, we fetch a JSON file that looks like:
[
{"name":"program1", "label":"Program 1","isState":true, "authenticate":true},
{"name":"common_lib", "label":"Common Library","isState":false}
]
and build states based on it. (I’ve removed the code regarding non state includes here for brevity, but it’s in the github code) And the last thing we do is “bootstrap” the application using AngularAMD:
And that’s the guts of how this process works. Save reference to the stateProvider, and add states as you see fit afterwards, then they can be navigated to.
But wait, there must be more?
Structure of a dynamic program
Sure there’s more. There’s a pattern to how you build a program. In the code you can see that I add the dynamic states via AngularAMD’s route function
"program@":angularAMD.route({
templateUrl: "programs/" + prog + "/templates/" + prog + ".html",
controllerUrl: "programs/" + prog + "/main.js",
controller: prog+"Ctrl"
})
What this does is to wrap the controllerUrl in a promise and put that into the ui-router’s resolve object. UI Router will make sure that anything within that resolve object is resolved before loading the controller, and so, if we make that file return our controller, and that controller in turn requires a number of other dependencies to load, then we can make sure that everything for the program is up and loaded before we load in the controller for our state.
So the convention is that any program will have a main.js file within its root directory. I took this convention from CommonJS, and it means that for libraries that we load dynamically (as in, we can have some collections of code that we want to load in via our mechanism that aren’t going to be exposed as states, but will be used by some of the dynamic programs), we can load them using RequireJS as a CommonJS package. The main.js file is of the form:
/**
* Entry point for Program1
*/
define(["./controllers/program1"], function(controller) {
console.log("Program 1 Main :");
return controller;
});
Which means that when it resolves, it will return the controller, and by doing so, that controller will have loaded everything else it requires, like so:
define(["../module"], function (program1_module) {
var program1 = program1_module.controller('program1Ctrl',['$scope','$state',function($scope,$state) {
console.log("Program 1 Controller");
}]);
return program1;
});
And so on down the chain. The program module itself can add nested states for its own program flow within its config block like a normal angular application, and as long as it calls angularAMD.processQueue(); after defining it’s run/config blocks, then everything will run and you will now have access to a dynamically loaded program within the main Angular application.
Optimization
I had a lot of trouble getting this working properly. I do it using Gulp, which in turn uses the RequireJS optimizer (r.js) to trace dependencies and pull them into single files, and it has tasks for optimizing the main application, and a process that scans the program directory and will optimize each program it finds in there. One of the biggest hurdles was working out how to make it have an optimized program file return anything. By default, an optimized program is basically just a long list of define statements. Nothing is returned from the file. And if nothing is returned, then when AngularAMD.route resolves the controllerUrl (which is pointing at the same URL of programs/program1/main.js) it gets nothing back, so doesn’t wait for the code resolution, and so tries to load the controller for the view before it exists, and the app grinds to a halt. Also, if you don’t have a require([The main program name here]) somewhere, then nothing within the optimized file will actually get run.
This took me ages to work out, but you can append pre and post strings to an optimized file via config params for the optimizer. So, a part of my gulp task that handles optimizing a program looks like this:
name: program,
exclude:["app"],
wrap: {
start: "define([],function() {",
end: "var defer = $.Deferred();"+
"require(['"+program+"'], function (ctrl) {"+
"defer.resolve(ctrl);});"+
"return defer;});"
}
Which means that what the whole file is treated as is one large define statement (containing lots of others) which returns a promise which will contain the controller when resolved… just like we have in the non optimized version of the code. I think that’s pretty cool. 🙂
Now, there’s also some funky stuff in there which handles injection of templates into the template cache for the built version, so they all end up in the optimized file, but that’s not the focus of this post.
Wrapup
That’s where I’m up to now. I have this running a prototype application within my work, it’s building to an optimized version, and I have the templates being injected into the code. It’s not perfect by a long shot, at the moment I can only have it successfully retrieve the injected templates within the built version of a program by returning a promise that waits a second before resolving for the templateProvider.
var state = {
url: prog,
auth: item.auth,
views: {
"program@":angularAMD.route({
//Normal way.... templateUrl: "programs/" + prog + "/templates/" + prog + ".html",
controllerUrl: "programs/" + prog + "/main.js",
controller: prog+"Ctrl",
templateProvider: function($http,$templateCache,$q){
return $q(function(resolve,reject){
setTimeout(function(){
if($templateCache.get("programs/" + prog + "/templates/" + prog + ".html"))
resolve($templateCache.get("programs/" + prog + "/templates/" + prog + ".html"))
else
resolve( $http.get("programs/" + prog + "/templates/" + prog + ".html").then(function(response){return response.data;}))
},1000); //=-=-=-=-=-DODGY! DODGY! Just waiting so things have resolved... DODGY!!!=-=-=-=-
});
}
})
}
};
That’s very, very hacky (It’s waiting for the module/controller code to load and resolve, but doing so purely by going “Eh, a second’s up, must be ready”), there has to be a proper way of doing that, and I didn’t put that in the github version. So there’s that, and other bits and bobs that need fixing and tweaking and streamlining.
But I wanted to share what I have worked out in the hopes that others get some use from it, and maybe can point out ways I could do things better.
tl;dr
The code is on Github.