Back to files

SourcePuller 0.1

Andrew Tridgell <tridge@samba.org>
February 2005

SourcePuller (also known as 'sp') is a free client for talking to BitKeeper(tm) source code management servers. It is written to offer enough functionality to enable reasonable access to source code repositories held in BitKeeper without attempting to actually replace BitKeeper as a source code management system.

BitKeeper is a powerful source code management system. It has been credited with greatly helping the Linux community by providing a system that fits in with the way many free software projects are developed, while being fast and scalable enough to handle the huge development load of the Linux kernel development community. BitKeeper was initially written by Larry McVoy and is now sold commercially by BitMover inc. See http://www.bitmover.com/ for more information on BitMover and how to obtain a BitKeeper license. BitKeeper is made available at no cost to free software developers. (see note at end for an update on this)

SourcePuller was written for two reasons. First, because the terms of the free BitKeeper license are not suitable for some members of the free software community. This can occasionally lead to frustrating situations where a free software developer wishes to access a BitKeeper repository, and is either unable to, or can only access it via a gateway that translates the repository into another format, possibly losing some information.

The second reason for writing SourcePuller was to provide a open library of routines that can talk to BitKeeper servers and manipulate local BitKeeper repositories. It is hoped that this library will be used by the authors of other source code management systems to allow them to interoperate with BitKeeper. Eventually this should result in an improvement in the quality of the various bk repository gateways.

SourcePuller is not intended to be a full replacement for BitKeeper. Instead, you should use SourcePuller as an interoperability tool for situations where you cannot use bk itself. SourcePuller is missing a large amount of core functionality from BitKeeper, and thus is not suitable as a full replacement.

Using SourcePuller

You may find that SourcePuller has a similar command syntax to bk, but you will probably also find that there are a large number of differences. As I don't use bk myself I have no way to determine the "correct" syntax for various commands except by reading the various howtos that are available. My apologies if you find the command syntax confusing.

To start using sp you need to either have a local repository from the BitKeeper client, or you need to perform a clone. In the examples below I will use the "ccache" repository hosted on ccache.bkbits.net as the test server. Please substitute your favourite BitKeeper server.

To perform a clone do this:

sp clone bk://ccache.bkbits.net/ccache myccache

that will copy the full ccache repository from bkbits.net and place it into a local directory called "myccache". If you look in this directory you will see it contains a number of SCCS files, one for each file in the ccache source tree, plus some control files.

At some stage in the future you may wish to update your local repository to pull in the latest changes from bkbits.net. To do that issue this command:

sp pull bk://ccache.bkbits.net/ccache myccache

If you are somewhere in the myccache directory then you can omit the "myccache" on the end, as sp will find the root of the repository by looking for the SCCS/s.ChangeSet file.

Next, you may want to checkout the files from your local repository. Issue this command:

sp co -r myccache myccache-test

That will do a recursive checkout of the myccache repository into a local directory "myccache-test". If you wanted an older release of ccache you could have done this:

sp co -r -R release-1-7 myccache myccache-1.7

Now you may wish to see what "tags" are in the repository so you know what releases are available. Run these commands:

cd myccache
sp tags

As with many sp commands, you can use the -v option to increase the verbosity of the output.

Now you may wish to see if there are any branches in your repository. Run this command:

cd myccache
sp branches

The ccache repository is pretty boring. If you try this on the Linux kernel you will see over 6000 branches.

You may wish to see what files would be in a checkout or a specific release. Try these commands:

cd myccache
sp inventory
sp -R release-1-4 inventory

Sometimes you wish to look at the changelog of a specific file. Try this:

cd myccache
sp prs SCCS/s.Makefile.in

or for the changelog of the main ChangeSet, with a bit of extra detail, try this:

cd myccache
sp prs -v SCCS/s.ChangeSet

You might want to checkout a specific file, you can use "sp get" for that, like the following examples:

cd myccache
sp get SCCS/s.Makefile.in
sp get SCCS/s.Makefile.in -R 1.3
sp get SCCS/s.Makefile.in -R 'tridge@samba.org|Makefile.in|20020403041055|58965' sp get SCCS/s.Makefile.in -R 7

In the second last example, a "commit id" is used to specify the revision to fetch. You can find the commits ids using "sp prs -vv".

In the last example above a "SCCS sequence number" is used. These are also shown in the output of "sp prs".

Next, you might wish to examine the differences between two revisions or release tags. For example:

cd myccache
sp diff -R release-1-7 -R release-1-8

as usual, you can use any combination of sequence numbers, release tags, commit IDs or revision strings with the -R option. The diff output will include the commit log (comments, dates, user etc) of all of the commits that are active in the second revision but not the first. If you omit the second -R option then the mainline head revision is used.

By default the external program "diff" is used to display differences, wiith prepended patch comments giving the commit logs. You can control what program to use and what options to pass it using the --diff and --diff-options flags.

You might also like to see an annotated view of a file, showing who wrote which line and when. Use the command "sp annotate" for that:

cd myccache
sp annotate ccache.c -R 1.7

Finally, you may wish to validate your repository to make sure it is not corrupt. To perform a basic check that all SCCS files have a correct overall checksum run this command:

cd myccache
sp check -r

To perform a more complete test, checksumming every revision of every SCCS file in the repository, run this:

cd myccache
sp checksum -r

You can also use sp to produce a complete project dump, which might be suitable to use as an import format for another SCM system.

sp dumper myccache > myccache.dump

See DUMP_FORMAT.txt for a description of the dump format. More sophisticated (and faster) imports can instead be done by directly linking to the sp library functions.

About the -R option

The -R option is used in many sp commands. It is used to specify a revision, tag, sequence number or commit id to apply the operation to. Rather than having separate options for each type of revision specifier, sp recognises what type you mean by its format.

For example, if you specify -R "1.307" then sp knows you mean a SCCS revision string (also known as a 'sid'). If you specify something like "-R v2.6.10" and that release tag exists, then sp will recognise it as a release tag.

Some examples include:

a SCCS sequence number: -R 37 a commit tag : -R v2.6.10
a SCCS revision string: -R 1.2049
a SCCS branch string : -R 1.337.2.10 a commit ID : -R 'tridge@samba.org|Makefile.in|20020403041055|58965' the tip of the tree : -R .

The libraries

SourcePuller is really just a very thin C wrapper around a set of library routines. The libraries come in two parts, the first is libsp which implements the wire protocol. The second is libsccs, which implements the SCCS file format, and also knows about the extra tags that are used manage information not stored in a traditional SCCS file.

If all you want to do is import from a bk repository to some other system, then you might find it easiest to just use 'sp dumper' and the dump format described in DUMP_FORMAT.txt. If you want closer interoperability with bk, such as being able to do incremental updates from a bk repository, then you will probably want to link to the sp libraries.

At the moment the only documentation on the library API is in the two header files (libsp/libsp.h and libsccs/libsccs.h), plus the example usage of the API in the sp main code. If you are the type of programmer that likes to work by example, then maybe this is sufficient for you, otherwise you will need to wait for someone to write a proper set of API docs. My apologies that I have not done this myself - I only have very limited time to spend on sp, and I'm a slow documentation writer.

One thing I would suggest though is that you read lib/talloc/talloc_guide.txt. That is a guide to the 'talloc' memory allocation library that I use in sp. Some of the code will be a bit obscure unless you understand talloc.

Update - April 2005

As you probably know, there has been quite a fuss lately about this code and the fact that BitMover has now withdrawn the free version of bk. First off, I would like to say that this result was not the intention when I wrote this code. I had hoped that an alternative open client would be able to coexist happily with the proprietary BitKeeeper client, as has happened with so many other protocols. An open client combined with the ability to accurately import into other source code management tools would have been a big step forward, and should have allowed BitMover to flourish in the commercial environment while still being used by the free software community.

I would also like to say that BitMover is well within its rights to license BitKeeper as it sees fit. I am of course disappointed at how BitMover has portrayed some of my actions, but please understand that they are under a lot of pressure. Under stress people sometimes say things that perhaps they shouldn't.

As I have stated previously, my code was written without using bk. Some people expressed some skepticism over that, perhaps because they haven't noticed that bk servers have online protocol help (just type 'help' into a telnet session). I don't think it is unreasonable to assume that this help was intended for people like myself who wished to implement new clients.

I would like to thank all the people who have supported me in the development of this tool by providing useful advice both before, during and after the development of the code. I tried to consult with a wide range of interested parties and the feedback I got was certainly appreciated.

Finally, I would like to point out the obvious fact that Linus was perfectly within his rights to choose bk for the kernel. I personally would not have chosen it, but it was his choice to make, not anyone elses. Linus is now in the unenviable position of changing source code management systems, which is a painful task, particularly when moving away from a system that worked as well as bk did. If you want to help, then help with code not commentary. There have been enough flames over this issue already.