Intro to git

Git is a revision control system originally written by Linus Torvalds to track changes to the Linux kernel, and now used very widely.  It is free and open source, trivial to set up (no server needed), lightning quick, and works well for both small private projects and very large distributed projects.  If you've never used git before, the easiest place to start is the git tutorial manpage.  For a more in-depth look at git, a good resource is the Pro Git book. 

The git feature that is most useful for us at Linerate is the top-notch support for distributed workflows. Developers perform all day-to-day operations locally in their private working directories, without interacting with any other developer or server repository.  We do all work in local branches that are derived from a shared lineage, but progress independently of the shared ancestral branch from which they sprung.  Git lets us easily share these feature branches between developers as needed for code review and for testing by our continuous integration system.  When we finish a feature branch, git provides easy and efficient methods for combining the feature branch with the main line of our product code (merge and rebase).

The layout of LineRate's software

LineRate's software is called LROS.  The LROS kernel is derived from FreeBSD, and we manage it as a local fork of the upstream code base.  I'll describe FreeBSD's somewhat unusual branching structure in the following section, then describe the repository structure we use internally for deriving our product from FreeBSD.

FreeBSD's use of git

The FreeBSD project has been around for a very long time.  There are over 200,000 commits in the main development branch.  Their main repository is currently in Subversion, but they helpfully maintain a git mirror at github.

The FreeBSD project uses the workflow that git calls "Graduation".  All bug fixes and new development goes in an experimental branch called CURRENT.  Bugfixes are tested in this bleeding-edge branch, and when the stable branch managers deem them safe and successful, the bugfix commits "graduate" to the stable (release) branches.

The LineRate method

We manage the source for the LROS kernel as a a set of patches against the FreeBSD kernel.

We originally started development by selecting a version of FreeBSD to base our work on: a particular commit in one of the stable branches in their git repo.  We created our internal master branch at that point.

Since the early days we've organized our in-house branches using git's Merging Upwards workflow with Topic Branches:

  • New development happens on topic branches, which get merged into the master branch when they're ready.
  • Bug fixes are committed to the oldest supported branch that require them.
  • Older branches are merged upwards into newer branches, all the way to the master branch.

Eventually our software matured and we made a local stable branch to prepare for our first release.  In keeping with the Merging Upwards workflow this branch only got bug fixes (which were then merged into master), while new features kept going into master.  Nowdays we have several simultaneous stable branches; one for each stable release series (2.6, 2.7, etc).

Keeping up with upstream

While we are working on our product, countless volunteers across the Internet are working on the upstream FreeBSD project.

Linerate originally started LROS development based on FreeBSD 7.3, all the LROS 0.x and 1.x releases are based on that version.  Occasionally we would run in to problems such as bugs or missing support for newer hardware.  We'd look at the upstream FreeBSD commit history after our branch point and see that the issue had already been fixed there.  In that situation git makes it easy to take the fix from upstream and apply it to our product branch using the cherry-pick operation (this is similar to the "Merge From CURRENT" technique that FreeBSD uses to back-propagate fixes).

As time went on and the upstream version that we were based on aged, we found ourselves spending more and more time back-porting drivers and fixes from upstream.  Eventually the FreeBSD 7.3 release reached its end-of-support, and we started having to backport fixes & drivers from newer, still-supported versions of FreeBSD, but changes in upstream since 7.3 made this onerous.

At this point we decided to pick a new stable upstream commit to internalize into our product.  We hoped for several benefits from doing this work:

  • Acquire the upstream bug fixes and drivers we had not yet bothered to backport to our product.
  • Make the future work of backporting even newer upstream commits to our product easier.

So once again we picked a commit in upstream's history, FreeBSD 9.1, a newish (at the time) stable branch a step or two behind the bleeding edge.

We then rebased our local master branch onto that point.  "Rebase" is a git operation where you apply each commit from one branch to another branch, effectively transplanting those changes to a new starting place.

We call this process "uplifting".  We uplifted LROS to FreeBSD 9.1.  This was a huge project which took several months of one of our best engineers, and extensive testing.  It was successful, and brought lots of bug fixes and drivers and new hardware support to our product line.  All our 2.x releases are based on this uplifted source base, and we are currently in the process of uplifting again, to FreeBSD 10.1, primarily to pick up the improved NUMA support in upstream.

This system of maintaining and keeping up to date our fork of FreeBSD is not without pain, but it's much better than the alternatives we've come up with so far.