Debian OpenSSL Bug: Some Thoughts

Last week's big security news was the discovery that Debian has been shipping a busted version of OpenSSL for over a year. There's been tons written about it (e.g. coverage on Slashdot and Schneier's site). Everyone agrees it's a major screwup, and we'll probably never know how much data (if any) was compromised because of it. What can we learn from it, from a software engineering standpoint?

1) Modifying code just to appease a source analysis tool is a terrible idea. It's got to be one of the worst reasons to modify code. Tools like this will highlight potential problems, but they will generate false positives that must be ignored.

2) Private source branches are a bad idea. Debian maintains private versions of several packages they include. I don't know anything about the benefits of this practice, but the downside is obvious: changes don't make it back to the maintainers, who are the ones most qualified to vet any changes. If these patches had been submitted to the OpenSSL project, they would have been rejected and the problem averted.
I've never maintained my own Linux distribution, but I have had to make custom software builds for customers. It's sometimes tempting to keep a change local and not commit it -- Not a good idea.

3) Some problem domains are inherently difficult to write automated tests for. In this case, the PRNG would still have passed all statistical tests - it's just that it could generate only 32767 different number streams. That's an awfully hard failure to write a test for, and a test for it would be useless in catching a bug that's even slightly different.

4) "Never ascribe to malice, that which can be explained by incompetence." [I don't want to call anyone incompetent, but that's how the saying goes.] In all I've read about this fiasco, I don't remember anyone suggesting that this was anything more than an accident - even though it's a result Mallory could only dream of! It must be because of the relative transparency of the Debian process, but can you imagine the response if a similar problem turned up in Windows? The reaction to the discovery of the NSAKEY debugging symbol was pretty heated, and as far as I recall nothing was ever proved - talk of "back doors" never got beyond speculation.