Go to Bug Lists for links to lists of bugs.
Psycho-Toxicity of Bugtrackers
Bugtrackers are psychotoxic. They contain defects and faults - black marks against the people involved. Everyone using a bugtracker should be aware of the hazard. Hazmat clothing is available via your union rep.
The toxic sludge in bugtrackers is likely to cause arguments, frustration and lethargy.
The bug tracker is a necessary evil because our natural inclination is only to accept a small amount of personal criticism. A bugtracker is chock-a-block full of implied criticism. The attitude to a bugtracker needs to be not as a blame-base but as a specialized collaborative to-do list that promotes improvement to the user's experience of the software.
Culture of Blame
Unfortunately, compounding the problem, the design of bugtrackers comes from corporate culture, often a culture of blame. The focus is on an audit trail. Middle managers need to have documented justification for why it took so long to ship the product. In commerce it is often more important to show that there was busy-work being done. When the heat gets too hot, it needs to be shown that the 'hot potatoes' had already been passed on to someone else. Commercial bugtrackers end up with hierarchies of software enforced privileges to compensate for an absence of collaboration culture.
In the open source volunteer world we need the opposite focus. "What is the current problem?" is our top question about any bug. We want to fix the bugs. Our next question is probably "Who knows the most about this?". We want to capture their input early, even if they don't fix the problem, to save time overall.
What to do about it?
- Don't reward the antipatterns of commercial bugtracker work.
- Do give major kudos to people who convert a moonphase bug into a reproducible bug. That is the hardest step we ever have in clearing most bugs.
- Have 'topic experts', drain busters for each area of expertise, and call them in early to at least have a look.
Rules for Bugtracker Use
A list of rules for using bugtrackers would be in huge danger of Instruction Creep. To try to avoid that, I'm using instead a more descriptive style.
Bug Titles should be Symptoms
A bug goes into the database with the intention and hope that it will be cleared. We all know of bug reports from users "Something go Wrong when I use Audio-city". If we are going to ever clear a bug we need to know what the symptoms are. The title should be a reminder of the symptoms. "Audacity is not using latest Portaudio" is a bad bug title. If the latest portaudio handles subsonic seismology USB microphones, then a better title would be "Crash when using subsonic seismology USB mics". The comments in the bug could then explain that this is a known bug in portaudio and that using Portaudio 11.7 is a possible fix. Provable symptoms in the title, not hypothesized root causes.
Steps-to-reproduce is a Vital Field
The steps to reproduce a bug in the bug report are there for a reason. Developers have to reproduce the bug. Once the bug has been dealt with, we want a way for QA to establish that it has actually been dealt with. If the steps still show the bug, then NO the bug has not been cleared. If a bug does not have steps to reproduce, we'll likely close it with INVALID before a developer even gets to try much with it. Paradoxically it's even more important for moonphase bugs, to have a good description of the kind of thing that is likely to exhibit the bug.
New Bugs from Old
I don't need to spell out here again the roles of PX, P1, P2, P3, P4 and P5. Oh. Perhaps I do.
PX to P5
- PX bugs often enter the system as PX.
- P1 bugs prevent a release.
- A P2 may prevent a release, if the current RM decides so.
- P3 bugs must be release noted.
- P4 and P5 bugs are not release noted.
The Inexact Science of Bug Priorities
Deciding on bug priority is not an exact science, and some of the natural 'rules' conflict with each other.
- Reproducible crashes and regressions (relative to 2.0.0) are generally P1.
- However, reproducible bugs which can't be progressed (e.g. because they are incompatibilities in a driver or operating system beyond our control) will usually be P3 or lower, so they don't needlessly prevent stable releases.
- Similarly where a bug lacks a ready way to reproduce it there may be pressures to demote it to P3, even if it is a frequently reported crash or regression. One solution to handling a bug demoted for this reason may be to flag or keyword it, or otherwise encourage testers to discover how to reproduce it.
Also a P1 bug in a new feature we can turn off will generally be a reason to turn that feature off so that we can get a release out. For ambitious features we aim to do them as plug-ins so that we don't have to turn the feature off.
Morphing Bugs and 120 Comment-long Bug Trails
When the headline issue of a P2 bug is cleared but there are remaining problems that are not P2, it is usually better to open a new bug with a new title for the residual issues. In a bug with 120 comments, it may take time to identify these residual issues. It's a priority to remove the P2, but not necessarily to document the residual P4/P5s. The morphing-bug problem is another reason to move discussions out of a bug tracker that can't cope with them.
Discussion in Bugtracker
Ideally the bugtracker comments are a record of work done to fix the problem, actual results of investigation and summaries of reports and updates of the problem in the field. As soon as we start discussing alternative possible fixes to a problem, we risk the volume of text in the bugtracker ballooning. That in itself is not a problem, but the fact that we can't edit it down in the light of clarifications and decision is. If a bugtracker discussion is about to balloon, we try and take it to the wiki. It's particularly valuable where we are looking at user interface changes - wording and the like.
There's a link field in bugzilla that we often use for a corresponding wiki page.
In many cases it makes sense to disable a broken feature rather than to put disproportionate work into fixing it. "Shipping is a feature". If it is a choice between getting Audacity onto users laptops now, or holding off until the seismology feature works, you know, we might just choose to ship now. We must be sure to demote to a lower priority bug to fix the feature, not just close the bug.
Moonphase, Heisenbug and Comet Returns
- A moonphase bug is a bug which there is no reasonably reliable strategy for reproducing on a developer's machine. A bug that happens 50% of the time is not automatically moonphase. A developer has the patience to try twice!
- Heisenbugs are technically bugs that disappear when you try to debug them. Most commonly a bug that occurs in a release build but not in a debug build. We incorrectly used the term for a while for moonphase bugs.
- A comet returns bug is a bug so rare that we have no hope of observing it and besides which it affects so few of our users, perhaps users with steam powered machines that are puffing along on their last tender load of coal, that it isn't a worry. We need to be supremely careful that we don't classify a moonphase bug as a comet returns bug just from wishful thinking. A feature of our recent moonphase bugs is that on some user's machines they can actually be near-repeatable. That should indicate caution about a "comet returns" classification.
Some Commercial Bug-tracker Anti-patterns
- Bug snow-drift: Company rewards developers for fixing bugs. Developers are incentivized to create lots of bugs and then fix them, rather than get the software right first time.
- No helicopters: 30 bugs are reported on a particular feature. Special case fixes are added to clear the bugs one by one. No one stands back and looks at the big picture and sees that the underlying cause is two competing incompatible designs.
- Schlemiel's bug Meeting: Much work has been done on a large number of longstanding bugs. Unfortunately in the bug meeting it is necessary to read the bug audit trail back to 1989, as there is no summary field. As the buglists get longer the work spent reviewing buglists increases with the square of the time passed since the bugs were first reported.
Cherry Picking and Dredging
- Cherry Picking: is searching bugzilla for easy quick to fix bugs - and fixing them. They'll usually be P4s. It's a great way to make inroads on the actual bug numbers. Its main benefit is on the bugtracking, since it's P2s and P3s that have most impact on users.
- Toxic Sludge Dredging: is a review meeting involving long time developers of the long outstanding and open bugs. The purpose is not so much to close the specific bugs discussed right now as to learn from the experience. It can lead to new policies, such as the policy to move to a plug-in API for new features. Finding ways to protect the core of Audacity from new bugs is part of the quality drive.