How to Rewrite Software Without Breaking Everything - Part 1

[apathy]

For going on 4 years now, the SWG:ANH team has been chipping away in some form or another on a significantly ambitious piece of software, a fully emulated massively multiplayer server for a game that doesn't technically exist anymore. This is a particularly difficult feat to pull off considering that at the project's inception there where many unknown factors, particularly in the area of how the server and the game clients' communicate with one another and the particulars of the data being sent. To hope to get things right and design a perfect server from that vantage point is a fools dream, but with careful planning software can be made to adapt to changes over time without resorting to a complete rewrite of everything.

Approximately a year ago, 3 years into the development of the server, there was a significant stirring among the team. At this particular juncture all of the packets, the bundles of information sent between the server and the game clients, were more or less figured out and the most of the mysteries of "how things are done" in terms of managing game data and processing input from players were solved. To put it another way, there existed at this time a wiki of some 14,000 pages of information that was not known at the beginning of the project. With all of this information it was easy to see where certain areas of the code needed significant attention in order to properly continue. Another particularly frustrating roadblock at the time was that everyone who was in charge of the "design" of the server had left the team and had left little in the way of the documentation of that design. This left remaining team members and new comers to fill in the blanks based historical data from forums and conversations with other developers or by spending lengthy amounts of time reviewing existing code. Needless to say things had to change, the question then became do we continue forward with what already exists or start over from scratch?

This question was debated off and on for a period of time however in the end there existed too many reasons not to start from scratch, the primary ones being: 1) There was an existing community of testers and developers who would be left with nothing to do for a significant and unknown period of time while a new server was designed and implemented and 2) by and large the existing code worked and already supported a significant amount of features. With the question of why we should continue forward understood it then became a matter of how? The answer to that has been an initiative spanning many months (and is still in progress as of the time of this writing) that required a solid plan of action and a change in the then current development workflows.

In order to achieve anything you have to first set a goal and then plan a course of action to achieve it. It's a simple enough concept yet how many times have you sat down to write a piece of software with no real planning, just a vague idea of what you want? (note: this hypothetical is for developers only for anyone else reading along out of curiosity, just go with it) This way of writing software leaves out one of the biggest parts of writing software... planning how you're going to do it!

For SWG:ANH the overall goal was to clean up the existing codebase while still allowing for new features and most importantly, while maintaining a functional server throughout the process.

"Well that's a particularly lofty goal," some may say. As a developer on the team since the start of these changes I can safely say that anyone who sat down at their PC and started typing out code would have failed miserably, if not for a solid plan of action. What follows is a general order of the plan of action to improve the way the team approaches development and an overview of each change.

Make It Simple To Build the Project

Compiling the source code for any project should be as simple as possible and a double-click is about as simple as it gets. That was the goal behind the BuildServer.bat file found in the root directory of the main MMOServer project. This script is responsible for downloading and compiling all external libraries as well as grabbing all the data files needed to run the server, all from a double-click. Being able to compile the entire project from a single command like this has a number of obvious benefits, but one of the most useful is the next item.

Perform Automated Builds on Every Commit

Every time source is checked in an automated process should grab the latest updates and compile it, making sure to run all code tests and report any failures. Doing this makes everyone responsible for the code they contribute and should any code be introduced that causes either the server not to compile, or any tests not to run, the issue is immediately brought to light and dealt with.

This goal was implemented shortly after the completion of the BuildServer.bat and the result can be seen at: http://pulse.swganh.org:8080/browse/projects/MMOServer/home/

Each time a new commit is made on the development branch this build server checks out the code and builds the server in debug and release mode and then emails the results to the project leads. Snap-shot's of the each build can be found here (this is where the "Builds" are obtained for our internal Test Center servers) as well as historical data on build times, status, and unit test results.

Switching to a VCS That Facilitates Branching Often

Once the project was easy to build and we had an automated way of keeping everyone updated on the status of each build the next step was to make it easier to work on changes without the worry of disrupting the work of others. At the time Subversion was being used and all code was committed on-top of existing code. What resulted is about what you'd expect, sometimes things would work great but often someone would check in code that was incomplete and not ready for testing.

This lead to the adoption of a "branch often" style of coding in which every new addition of code is created in it's own separate branch and then merged into the main development line once it's been completed. This allowed developers to work on features and bug fixes, or prototype ideas, without worrying about interfering with the work of others. It soon became apparent, however, that Subversion was not up to this task. Every action using Subversion results in remote calls over http and the simplest of commands were just taking too long to complete. Enter Git, a very powerful alternative VCS with a feature set that was up to the needs of the team. The switch was a little bumpy at first, primarily due to the switch to SSH for authentication, but after a short time the whole team was happily branching and merging. For more information on the development workflow see the wiki guide at: http://wiki.github.com/swganh/mmoserver/swganh-developer-workflow

Perform Peer Reviews of All Incoming Code

Moving to a "branch often" workflow provided us with the opportunity to easily implement the last major change to our way of development, performing peer reviews. Before any branch gets merged into the main line of development another developer on the team must take a look over the submitted code and provide feedback to the author. Once code has passed inspection it can be merged into the development line for QA to test.

These peer reviews often spark discussion on how and why an implementation was chosen which invariably leads to further improvements of the code or catch mistakes that the author may have missed before they make it to the development branch. The problem with trying to do these types of reviews with a team such as SWG:ANH is that programmers are on at varying times of the day and there isn't always someone around to immediately review code. This particular issue was a deciding factor in making the switch to Github for hosting the project source. Github provides many facilities such as commenting directly on source code pages which makes it much easier to hold discussions about changes in code.

This concludes the overview of the major changes to the way we do development around here. Each of the above topics deserves a a more detailed examination and will likely get it in an upcoming article. Next up we'll be looking at a few of the major code changes that have taken place to date as well as a glimpse at a few of the big changes in store for the near future.

Until next time… may the source be with you.

[Powerking]

Great Job apathy! Can't wait for part 2