Introducing RequireJS into Large Codebases. Delicately.
And this is how our deploys work. When we do a deploy, the Rails Asset Pipeline is configured to read in this file, concat each of these assets together, minify the result, and push it to our CDN. This isn't great. For one thing, the order of the entries in the build.json matters. Dependency changes in the code require re-shuffling of the JSON. Dead code still gets included. There's no way to get a slice of the included files without creating some new JSON structure or creating more build.json files. Lots of limitations here.The funny thing about this strategy though, is that it really is the simplest thing that will work. And is usually the case,you can get pretty far with it. When you add a new file to the system, so long as you maintain a certain level of discipline, and your code is modular-ish, it's not *that* difficult to figure out where it should go in the file.This was 'good enough' for a very long time.
This past fall, we decided to try a feature experiment -- popout chat. We believed that people wanted to the ability to pop out their chat windows. In order to do this, we would only need to load our chat-specific code right? Wrong. Chat depended on feeds and the construct of an application and models and soon, we had to pull in our base package of js which was 1.8mb, 1mb compressed. This didn't make ANY sense.
We realized this was the perfect opportunity to explore a loader. We spent the next two days trying to get requirejs in place. Dan Lee likes to refer this serendipitous moment as, 'programming out of anger’.
Fortunately for us, there is a single entry point into all the chat code. We spent some time to AMD the components and modules within this hierarchy.
Here’s an example of the ChatManager that manages all of the chat sessions. At the top, you can see the define() method that declares an array of all the file dependencies to pull in, followed by a function that aliases those dependencies for us.
But we also load chat on yammer.com…
We started using the global define() method, which is provided by requirejs. But on yammer.com, define() doesn’t exist which means we would get a runtime ReferenceError.Realistically, we couldn't change everything in one go – Our code would be running in two different environments. This is when we realized this needed to be a transition.
We had to make define() work in both environments. In popout chat, define used require’sdefine. Everywhere else, we shimmed in a noop function that immediately called the closure.This also mean that our dependencies wouldn’t get loaded properly in control....
This enabled a workflow that allowed us to continue to make changes to components without breaking production. This meant we could have short-lived branches, merge our changes constantly and not miss a beat with deploys. We also have 6k tests and this allowed all of those tests to still function properly.
At Yammer, we use feature flags, called experiments, where we gate code changes and measure the effectiveness of that feature. With this infrastructure already in place, we leveraged this to manage which files were served up. We could enable the requirejs experiment for ourselves and isolate breaking code without affecting all of our users and developers.
r.Js is the node requirejs optimizer that traverses all the dependencies and builds a single uglified and minified package.We could enable the requirejs experiment for ourselves and isolate breaking code without affecting all of our users.
[Screenshot of announcement]
Work-----Dev environment setupNginxCode proxyThis only affected 10 files out of hundreds… which meant we still have A LOT of work to do.Concessions----Global namespaces must live on (for now) – all of our components and tests new() up instances using global namespaces.Couldn’t rely on the dependency injectionTwo worlds - AMD and not[Screenshot of globals in praise_editor.js]There were points in time where developers would break the AMD world, which was okay because we were still protected by the experiment. However, after fixing enough of these issues, we realized we had to make the AMD world the default world.
A few months have passed since we completed this project, and we’ve had time to reflect on its lessons.
Isolation is probably the biggest takeaway here. If you are thinking about making a huge change to your app's structure, you should try to think about how you might 'try out' that change on a tiny slice of your app as we did with Popout chat. And in that tiny slice, go "all-the-way" with that change. You'll learn much more, much faster by doing steps A-Z on a small piece of your app than you will be doing step A on ALL of your code, then doing step B on ALL of your code, etc.I can't tell you how significant it was when we *deployed* Popout chat with just a couple weeks of work. Sure, there was still 90% of the codebase left to do, but since we took Popout chat "all-the-way", we knew we had a viable technique to do the rest of the code. This was a huge confidence builder for us personally, and the team rallied around the work.We also isolated our changes by using that Feature Flag. It allowed us experiment with dependency management with almost zero risk to production stability, which was obviously a *great* thing.
When you work on a large product feature, it's natural to carve it up into iterations, but we tend to ignore this strategy when doing infrastructure-ish changes. *Don't do that*.Try to identify the logical iteration points. For us, it was pop-out chat, then the account page, then the main homepage. And when we did the main homepage, we figured out a way to do that in pieces too. The flow should feel like feature development. Iterate, ship. Iterate, ship.
Making big changes is going to disrupt your team's work. You should really think about how to minimize this disruption.Ideally, the team can continue working as if you’re not even doing this big, scary thing. To give you an example, and your experience may differ than mine, but I've found that one thing team-mate's really hate is when they 'git pull' and now have a broken dev environment. :) Try *really* hard not to break the dev environment. At some point, you will disrupt the team. When this happens, over-communicate it. Warn that the change is coming. When the code lands, announce it. Be there to help team-mates get their stuff working. And for your team-mates offsite, bring somebody from that site into the loop early so that they are ready to help un-block the team.
In a codebase like ours, 'Rewrite the whole thing and merge' is just not possible. You either leave it alone and continue with the status quo, or figure out a *transition* plan. You have to come up with a new, preferred way of doing something and then gradually migrate your code over to the new, preferred style.Here's the thing about code in transition though...it's kinda ugly. Developers hate code in transition. While the transition is happening, you’re left with something we started calling 'two world syndrome'. Some of the code looks like *this*, and some of the code looks like *that*. Feature A is written *this* way, Feature B is written *that* way. Now developers have to learn both systems, and that just makes them dissatisfied. We hate this. Developers just have this *need* for cohesion.But here's the way we look at it. Code in transition is a temporary *concession*. Like just about everything in software, It’s a tradeoff. We'll concede that the code is going to have this imperfection for a while. We'll own that, and we'll also own the idea that it's everybody's responsibility to keep that transition moving.
And finally, we'll leave you with this thought. When you have a codebase like ours, large, lots of hand in it, constantly deployed, and you want to make a big, core, scary change, these are your two options.Either punt on the change and stagnate, or figure out that transition plan.Something we’ve learned over the last handful of years is that stagnation is not only bad for code, it’s also bad for morale. And *entrenched* stagnation is terrible. If your team has any pride it their work, this has potential to be downright demoralizing. We've found it infinitely better, for both codebase quality *and* team morale, for the codebase to be *transitioning* to something of a higher quality, warts and all,than to be stuck in the past. That’s it. Thank you.
Introducing RequireJS into Large Codebases. Delicately.