Well, this topic is a little bit more technical than my usual postings. I recently had an e-mail exchange with someone on this topic and decided that it could make a pretty good post as well. Senior management can drive significant improvements in software quality by encouraging certain practices and behaviors in a development organization. Sometimes this means changing the culture of the organization to see problems as an opportunity to improve rather than an opportunity to initiate a witch hunt.
In a dysfunctional development culture software bugs are seen as failures that should not occur and should be hidden at all costs. Development teams and test teams are adversaries – fighting over the differences between designating a problem as just a “user error” vs an actual problem in the code. The worst instance of this that I have ever seen is a developer arguing that there wasn’t a problem – the code was working as designed. Nevermind that the design as it currently stood was something that would cause the customer untold amount of pain in order to work around. You’re supposed to test to the design right? No – actually you’re supposed to test based on the requirements.
First off, the team has to determine the best ways to find problems as early as possible in the development process – and with the least amount of manual effort. The concept of continuous integration is key here. Every time a programmer checks in a software change it should initiate an incremental product build automatically. This incremental build then should also kick off a base set of regression test cases to test the software functionality. BAM! The problem will be identified before a full build gets done at night *AND* you know exactly which check in to the source code repository caused it. This is awesome. It is easy to fix a problem if you know exactly where to look. If you wait a week or two to do a build you are looking through possibly hundreds of checkins to identify possible candidates that could have caused the failure. Now, granted sometimes this means that the change has to be backed out in order to do some serious refactoring. However, better to do this now rather than during system test. Full nightly builds are also crucial. Sometimes two independent changes can each work, but when you put them together all hell breaks lose. The full build should start from scratch – no reuse of any object files or libraries (unless they are third party code that hasn’t changed). It also should have a full set of “smoke” or regression tests associated with it that test more than just the basic functionality of the code. Another key point that a lot of people miss is that if software is checked in right before the nightly build starts it should NOT be included in the build if the incremental build testing it has not been yet run. It should be included the next day.
I think that a lot of good quality improvements also can be driven through metrics – you just have to pick the right ones. For example, I’ve never seen a lot of success with Lines of Code written. Some software is inherently much more complex. Frequently the engineers writing the least number of lines of code are doing the hardest work – and the most sensitive from a reliability / failure perspective. They also can generate the most number of bugs per lines of code because of this sensitivity.
Overall bug trends as different levels of smoke, unit, and system test are run have been very helpful however. The number of bugs should ramp up steeply when heavy development and testing commences. Over time the number of new bugs (and total open) will level off, and then should drop just as steeply when the product is ready to ship. Using that data to determine the overall health of the product is key. However, once management starts to use that data to find the organization or individual who is to “blame” for the code instability strange things happen. The culture needs to be setup in a way to encourage people to track all bugs. If individuals worry about being blamed they will stop formally tracking their bugs and will even try to negotiate with test groups to make bugs disappear. It is an odd phenomenon, but it can be easily mitigated by how the organization reacts to finding bugs. Finding bugs is a *good* thing. The earlier you find them, and the fact that the customer didn’t is most important!
This clearly leads into test driven design methods. Whenever you plan to add new functionality, you add the automated tests first to go along with it. When the code is done, the tests better pass. Whenever a new bug is discovered you add a new test that clearly shows the problem and can later prove the fix for that bug actually worked. By doing this a product develops a full set of test cases as it grows. There is no need for a big test case development push at the end. Quality is tested in from the start.
This is the fun part. Don’t think of this as stalking – think of this as market research. Keep in mind that the company probably has already done these types of searches before talking to you. This will help level the playing field. Google everyone on your interview list. See what you can learn about their industry involvement, where they’ve been quoted in the press, and maybe even what their personal hobbies are. Read their blogs if you can find them. Look them up on Facebook. A lot of people have public profiles.
