mini-cases: controlling the source

Collaboration tools are a big business.

When you’re working on something by yourself, life is simple. You don’t need to coordinate with anybody else’s schedule, and only one person needs to remember what you were working on last, and what needs to get done next—which can be done with a sticky note at worst, and your memory at best. If you put your tools down, they should be where you left them when you come back. The only communication that needs to happen occurs within your own head.

But add just one person to the mix, and the overhead costs suddenly increase dramatically. You have to talk to another person. If I’m painting the walls and my cohort is waxing the floor, we need to make sure we’re not expecting to do the same room at the same time. Worst case, I might actually get trapped in a side room behind a freshly-waxed floor, thus wreaking havoc with the schedule.

Now, add a whole crew. Was the room already painted by somebody else? Did the guy who does window trim make sure he did his work after the wall guy? Are multiple people now “gated” behind that nice, shiny floor?

In real life, we have a number of solutions for the problems created by multiple people working together on multiple intertwined paths towards the same goal. They’re not all high-tech, either: the simplest is called “conversation,” and the second-simplest is called “writing things down.”

Interestingly, despite the evolution of more advanced communication media like email (persistent, auditable, multi-participant written conversations), scheduling tools (reconciliation between task status, resource availability and dependencies) and multimedia systems (instant text messaging, voice communication, video imaging), we still depend on the basics. As complex and efficient as all these, they still are fundamentally just improvements upon writing and talking.

So, things change—not just in the development of project management technology, but within projects themselves. Ideally, you want to keep track of what changed, when, and who was responsible. And beyond that, if you discover that something broke because of a recent change, you want to (quickly) get back to a working state. This isn’t always possible in all situations—if a floor of a building gets built wrong, you can’t just click a button to “revert” to the version before floor 30 was done.

Fortunately, in information work like software or most of the analytical parts of business, “rolling back” isn’t impossible. In fact, it’s an essential safety feature for software development: the ability to “code, merge, test, and undo if needed” is simply not optional.

In the vernacular, this operation is called “source control” or “version control” or “versioning.” It’s all about making sure that you know exactly what “stuff” is in the software that your customers are using and what “stuff” is still being worked upon. It’s also about making sure that, once you fix something, the broken part doesn’t somehow make it back in front of the customer.

In this particular case, I arrived at a company that had no real source control or versioning system. Since there were really only a few developers, source control was based on the old poor-man’s method of “I’ll copy the code into a directory somewhere, and let’s just agree that nobody makes changes to the same pages or subdirectories at the same time.”

And if a change went into production and ended up breaking something, it could become a mad scramble to figure out what changed, and how to roll it back to the previously-working condition.

So we had two real problems to solve: how could we keep from shooting ourselves in the foot if a change went south in a hurry, and how would we, mechanically, keep our growing development staff from overwriting each-other’s work?

In this case, the solution was versioning software. We evaluated a number of options before settling on Subversion, which is a free, open-source version control system.

A quick aside: a detailed description of the “open-source” and/or “free software” movements would require its own article, given that their nuances and history rival those of contemporary religious schisms. Basically, open-source software is software where the “source code”—the structural, editable, human-readable DNA of the finished application—is made available to users for their own modifications. Examples include the Linux operating system, the Firefox browser, and even the software used to produce my blog: WordPress. It differs from “closed-source” software in that such programs cannot be modified by their users: MacOS, Microsoft Office, Photoshop, and countless others.

The business model between open-source and closed-source tends to focus on a sort of “the razor is free, but shaving lessons will cost you” model for open-source, versus a more traditional product-based model for closed source. Both have their advantages: closed-source tends to be a little more user-friendly, while open-source tends to be more flexible. And both have weaknesses: closed-source means you are beholden to the developer for any innovation or new features, while open-source requires expertise, patience, and the bandwidth to participate in a community-driven development process.

But, as it happens, Subversion is what we might call a “mature” open-source project: not just stable and functional, it has spawned a subset of support applications designed to make the core application useful beyond the default ambitions of the developers. It’s a solution more than a program; the system provides for a number of ways to accomplish the same objective, from obscure typed commands to pretty drag-and-drop graphical icons and streamlined menus that remind you of any Microsoft product.

So there was no controversy in adopting such a solution. It didn’t hurt that the company adopting it was a startup with a limited budget, but they were pleased to be using the same system as companies a hundred times their size.

The process of adoption had a few steps. First, we needed to narrow down the “products” that we offered, and compartmentalize them into their own “repositories,” in the Subversion nomenclature. Step two involved some relatively uninteresting and technical discussions about the relationships between, say, our development environment and our production environment, and where exactly we would store our new repository. Finally, we needed to synchronize our real live website with our versioned website—again, a fairly technical exercise.

This project was an absolute success: for 14 consecutive quarters, we were able to roll out software updates with absolutely no compatibility-related issues.

Well…better make that a qualified success. There were no issues with code changes being overwritten. We did have the occasional issue with merging our code together—as it turns out, source control systems are great for keeping track of changes, but when you’re comparing two different changes on the same line of code, you still need a human to oversee the process and pick the one you want.

And it took time for us to get used to the new system, which forced us to have to scale back the scope of our first few releases. In much the same way as your productivity might initially suffer when you hire a new employee and have to take extra time to teach them the ropes, we found that certain tasks took longer than they used to take.

Which leads to the first lesson learned: software is a tool, and tools only get you so far. Far more important—and almost always overlooked—is the challenge of doing proper process analysis.

In other words, you have to think about more than just the tool. You have to think about how you do things, in what order, with what interactions. What exactly is the root of the problem you’re trying to solve? What might actually get worse with your new system? Are there things that can’t change, no matter how much you want them to? You can’t just up and start doing things a different way without considering the impact: the impact doesn’t go away just because you ignored it.

I think that we struggled a little in the early days of using Subversion because we weren’t yet used to doing things, to thinking about things, the way the tool wanted us to do them. And you have to adapt to fit your tool, no matter how “seamless” a transition the sales guy tells you it’s going to be. It’s like this: a microwave oven is great for reheating food. But if you don’t adjust your technique for the specialized tool, you’re going to set the kitchen on fire. You have to take that slice of pizza out of the foil first, and that pot doesn’t even fit in the microwave.

Software can’t make two programmers talk to each other to ensure that they’re working together towards solving a problem. Software also can’t make decisions about what problems to solve in the first place, or what the solution might look like. All it can do is keep track of the changes along the way.
And we still had plenty of problems to solve long after we got source control working smoothly. But, I don’t think that we would have been able to move much further along without having solved the simple problem of versioning.

That’s the second lesson. Business, like anything in life, proceeds in steps, and you have to walk before you can run. If you want to build a restaurant chain, you need to start with good inventory control in your first location; if you want to be a wedding photographer to the stars, start by making sure your tripod is straight. If you want to build a business that relies on software, you’re going to need to keep track of changes.

Finally, remember not to reinvent the wheel. As I just said, source control is an essential part of every software business, so if you think you’re encountering a common problem, your first step should be looking for a common solution.

That said, open-source free-software Subversion wasn’t free—far from it. We had lower initial productivity as we implemented it, and the hours spent setting it up, training people to use it and administrating it cost real money. But it was cheaper than building a system from scratch, and cheaper than finding a more user-friendly system. You can’t always find something that does exactly what you need it to do for exactly the price you’re willing to pay, but in this case, we were close enough.