On the topic of release quality

So how does PostgreSQL measure up in all this? Well, we aim to never release with known critical bugs in the software. But that all depends on the definition of "known". For example, PostgreSQL does not have a bug-tracker. For those following our lists, you will know that there is an endless debate started whenever somebody tries to discuss that, so I'm not going to discuss the topic in general. I am personally on the side that thinks we should have one. But I also feel that the MySQL bug-tracker is a clear example of the kind that we do not want to have. I don't have any exceedingly high thoughts about the bug-freeness of MySQL, but I have a hard time thinking that all the 41,000+ entries in their tracker are actually bugs. So it represents exactly the part that I really dislike about a lot of the bug-trackers around for open source projects - it's full of junk that aren't bugs, which makes it very hard to find what actually are bugs. (so thanks to Monty for outlining some for us )

That means we can't use "bugs in the bug-tracker" as a metric. PostgreSQL does keep an "open items list" before each release, which does track things that came up during the development. It used to just be a static file on Bruce Momjians machine - now it's in the wiki. It's a lot less structured, but it does give us a lot of help in the let's-not-forget-this-bug-before-release department. But it doesn't track bugs past release, and the history is not preserved across versions (that I'm aware of at least).

Another metric would be to look at what actually changes in the minor versions. If a full release is not reasonably bug-free, there will be a lot of turnaround after the release, once people put it in production. We are, as many open source projects, lucky in that we have a lot of people running our betas and release candidates in near-production environments, which means we can iron out a lot of things before release. But it's obvious that a lot more people will deploy a product once it's released (in fact, that has been quoted as one of the reasons why MySQL 5.1 hit RC so early - "to get more people testing it"). So I decided to pull some statistics from our CVS repository about the latest four major PostgreSQL releases (8.0 to 8.3), and the turnaround on their branches after commit.

Update: Since several people asked: the X axis is "minor version number"

This first graph shows the number of commits on the back-branches. This has been somewhat filtered - I have removed for example commits that are just updates of translations, and the commits that stamp the release itself. Or rather - I have tried to, with some simple regexp matching. So don't take these numbers as absolute, but they should be good enough to show trends. Also note that due to some problems with cvs not wanting to be nice to me, there are no commit statistics for 8.0.1, that's why the blue bar is missing in the first one.

A couple of things I think we can read from this graph. First of all, the total number of commits in the back-branches after a release are very low. Only very few of the branches have more than 50 commits going in. And this includes things that are just whitespace fixes or comment fixes, I didn't manually look through things to determine what were actual code fixes. It also "kind of" shows that 8.1 had more patches going in early on - which aligns with the general consensus that 8.1 was not one of our best releases. 8.3 had quite a lot of patches going into the first two releases, which can probably be attributed to several of the new features being very complex and thus obviously exposed to corner-case bugs. And there is (whew, lucky!) a steady decline in the number of fixes as the versions gets older - both because there are less people using them and finding bugs, and obviously because there are less bugs left in them.

So, moving on:

Update: Since several people asked: the X axis is "minor version number"

This second graph shows the number of lines added and removed. I used a simple diffstat output and probably ran diff in the wrong mode first, so it all came out as inserts + deletes. Thus, any changed row actually counts as two in the graph. The true number of modified rows is therefore much lower than the ones indicated in the graph - but not necessarily half, since there are still pure additions and removals happened.

The first thing that stands out from this graph, is 8.1.1 having almost 10,000 lines changed. The reason for this, after trolling the cvs logs for a while, is that pgindent was re-run after release. This is the tool we use to make sure indenting and code formatting is consistent across the source code. Obviously, this generates a lot of changes, all of which are whitespace only. We normally don't run this after release, because it tends to make it harder to track history of the files. But for some reason that I can't remember (I'm sure it can be found in the list archives if somebody cares enough to go digging), we did it for 8.1.

In the 8.0 series, 8.0.2 stands out as having a lot of changes. This was the release where we pulled out the ARC cache method (for several reasons) and replaced it with a completely new one.

8.3 overall shows quite a lot of changes in the early releases, with a large drop for 8.3.5. But it's too early to tell if that's a permanent drop yet. But even "quite a lot of changes" is only just over 2,000 lines of code, which is less than 0.3% of the code base (code base size according to Ohloh, and not counting the effect of the add/remove for change as mentioned above).

I don't know the MySQL code base enough to make similar graphs for them - but would be very interested in seeing them if somebody else did!

Finally, another metric:

This is the number of lines modified between the latest release candidate and the release. This is essentially the number of lines that go into the release without being tested outside the developers. Before you get all excited about how large this number is - looking through the actual changes, very much of it is actually changes to error messages, comments or whitespace.

The end!

Yeah, that turned out to be a very long post. If there are more statistics around the code base that you think would be interested, let me know! I have some of this stuff loaded in a database now, so I can do some more analysis without too much work as long as it fits within that model.