Go to Bug Lists for links to lists of bugs.
|This page is a rant about bugtracking, and how we try to use bugtracking. To make it really easy to submit a bug, we have reopened our GitHub issue tracker. Anyone can add any bug or any issue there.|
People who work with Audacity developers a lot may want to transition to our Bugzilla bug tracker, which is a bit more 'locked down'. Those people should read this page to understand how we use it.
- 1 Psycho-Toxicity of Bugtrackers
- 2 Rules for Bugtracker Use
- 3 New Bugs from Old
- 4 Severity vs Priority
- 5 Discussion in Bugtracker
- 6 Disabling Features
- 7 Moonphase, Heisenbug and Comet Returns
- 8 Some Commercial Bug-tracker Anti-patterns
- 9 Cherry Picking and Dredging
Psycho-Toxicity of Bugtrackers
Bugtrackers are psychotoxic. They contain defects and faults - black marks against the people involved. Everyone using a bugtracker should be aware of the hazard. Hazmat clothing is available via your union rep.
The toxic sludge in bugtrackers is likely to cause arguments, frustration and lethargy.
The bug tracker is a necessary evil because our natural inclination is only to accept a small amount of personal criticism. A bugtracker is chock-a-block full of implied criticism. The attitude to a bugtracker needs to be not as a blame-base but as a specialized collaborative to-do list that promotes improvement to the user's experience of the software.
Culture of Blame
Unfortunately, compounding the problem, the design of bugtrackers comes from corporate culture, often a culture of blame. The focus is on an audit trail. Middle managers need to have documented justification for why it took so long to ship the product. In commerce it is often more important to show that there was busy-work being done. When the heat gets too hot, it needs to be shown that the 'hot potatoes' had already been passed on to someone else. Commercial bugtrackers end up with hierarchies of software enforced privileges to compensate for an absence of collaboration culture.
In the open source world we need the opposite focus. "What is the current problem?" is our top question about any bug. We want to fix the bugs. Our next question is probably "Who knows the most about this?". We want to capture their input early, even if they don't fix the problem, to save time overall.
What to do about it?
- Don't reward the antipatterns of commercial bugtracker work.
- Do give major kudos to people who convert a moonphase bug into a reproducible bug. That is the hardest step we ever have in clearing most bugs.
- Have 'topic experts', drain busters for each area of expertise, and call them in early to at least have a look.
Rules for Bugtracker Use
A list of rules for using bugtrackers would be in huge danger of Instruction Creep. To try to avoid that, I'm using instead a more descriptive style.
Bug Titles should be Symptoms
A bug goes into the database with the intention and hope that it will be cleared. We all know of bug reports from users "Something go Wrong when I use Audio-city". If we are going to ever clear a bug we need to know what the symptoms are. The title should be a reminder of the symptoms. "Audacity is not using latest Portaudio" is a bad bug title. If the latest portaudio handles subsonic seismology USB microphones, then a better title would be "Crash when using subsonic seismology USB mics". The comments in the bug could then explain that this is a known bug in portaudio and that using Portaudio 11.7 is a possible fix. Provable symptoms in the title, not hypothesized root causes.
Steps-to-reproduce is a Vital Field
The steps to reproduce a bug in the bug report are there for a reason. Developers have to reproduce the bug. Once the bug has been dealt with, we want a way for QA to establish that it has actually been dealt with. If the steps still show the bug, then NO the bug has not been cleared. If a bug does not have steps to reproduce, we'll likely close it with INVALID before a developer even gets to try much with it. Paradoxically it's even more important for moonphase bugs, to have a good description of the kind of thing that is likely to exhibit the bug.
New Bugs from Old
I don't need to spell out here again the roles of PX, P1, P2, P3, P4 and P5. Oh. Perhaps I do.
PX to P5
- PX bugs often enter the system as PX.
- P1 bugs prevent a release.
- A P2 may prevent a release, if the current RM decides so.
- P3 bugs must be release noted.
- P4 and P5 bugs are not release noted.
The Inexact Science of Bug Priorities
Deciding on bug priority is not an exact science, and some of the natural 'rules' conflict with each other.
- Reproducible crashes and regressions (relative to 2.0.0) are generally P1.
- However, reproducible bugs which can't be progressed (e.g. because they are incompatibilities in a driver or operating system beyond our control) will usually be P3 or lower, so they don't needlessly prevent stable releases.
- Similarly where a bug lacks a ready way to reproduce it there may be pressures to demote it to P3, even if it is a frequently reported crash or regression. One solution to handling a bug demoted for this reason may be to flag or keyword it, or otherwise encourage testers to discover how to reproduce it.
Also a P1 bug in a new feature we can turn off will generally be a reason to turn that feature off so that we can get a release out. For ambitious features we aim to do them as plug-ins so that we don't have to turn the feature off.
Severity vs Priority
Here is a bug table from Bugzilla.
You can get the current version by clicking on the Audacity icon in bugzilla (or on the image above). Each of the numbers can be clicked to get a list of bugs of that kind.
This is what our Severity terms mean:
- Repeatable - A bug that can be reproduced easily. We split out the ones which are on all platforms to RepeatableAll.
- The logic of splitting these is partly psychological. A bug that is on all platforms is typically 'our fault'. A bug that is only on some is typically due to a platform quirk, and we are looking for a workaround for that quirk.
- Locale - A bug in localisation. These typically have little to no impact on English speaking users.
- You might wonder why that's a severity. The designation 'Locale' is an indication of whom the bug affects. Most of our users are English speaking.
- Moonphase / CometReturns / Heisenbug - see later.
- These give us an indication of how hard tracking down the cause is likely to be.
- Summary - Used to attempt to coordinate a number of related bugs. Often this ends up on a wiki page for discussion.
- Review - A bug topic where team discussion is essential. Often this ends up on a wiki page.
- Enhancement - used where arguably the report is not so much a bug as a request for a new feature, which needs design and agreement. Often this ends up on a wiki page for discussion. (see a pattern here?)
- Enhancements may not even be bugs in the traditional sense - though that's a 'hot topic'. When is something we want to improve a bug, and when is it a 'missing feature'?
Morphing Bugs and 120 Comment-long Bug Trails
When the headline issue of a P2 bug is cleared but there are remaining problems that are not P2, it is usually better to open a new bug with a new title for the residual issues. In a bug with 120 comments, it may take time to identify these residual issues. It's a priority to remove the P2, but not necessarily to document the residual P4/P5s. The morphing-bug problem is another reason to move discussions out of a bug tracker that can't cope with them. We add *** STEPS UPDATED *** where the steps to reproduce have been updated, making it easier for people to just read part of the bug trail.
Discussion in Bugtracker
Ideally the bugtracker comments are a record of work done to fix the problem, actual results of investigation and summaries of reports and updates of the problem in the field. As soon as we start discussing alternative possible fixes to a problem, we risk the volume of text in the bugtracker ballooning. That in itself is not a problem, but the fact that we can't edit it down in the light of clarifications and decision is. If a bugtracker discussion is about to balloon, we try and take it to the wiki. It's particularly valuable where we are looking at user interface changes - wording and the like.
There's a link field in bugzilla that we often use for a corresponding wiki page.
In many cases it makes sense to disable a broken feature rather than to put disproportionate work into fixing it. "Shipping is a feature". If it is a choice between getting Audacity onto users laptops now, or holding off until the seismology feature works, you know, we might just choose to ship now. We must be sure to demote to a lower priority bug to fix the feature, not just close the bug.
Moonphase, Heisenbug and Comet Returns
- A moonphase bug is a bug which there is no reasonably reliable strategy for reproducing on a developer's machine. A bug that happens 50% of the time is not automatically moonphase. A developer has the patience to try twice!
- Heisenbugs are technically bugs that disappear when you try to debug them. Most commonly a bug that occurs in a release build but not in a debug build. We incorrectly used the term for a while for moonphase bugs.
- A comet returns bug is a bug so rare that we have no hope of observing it and besides which it affects so few of our users, perhaps users with steam powered machines that are puffing along on their last tender load of coal, that it isn't a worry. We need to be supremely careful that we don't classify a moonphase bug as a comet returns bug just from wishful thinking. A feature of our recent moonphase bugs is that on some user's machines they can actually be near-repeatable. That should indicate caution about a "comet returns" classification. In rare cases an exceptional bug like bug 1714 is designated 'comet returns', when it has become quite predictable and reproducable, if you know when and where to wait for it - like a real comet.
Some Commercial Bug-tracker Anti-patterns
- Bug snow-drift: Company rewards developers for fixing bugs. Developers are incentivized to create lots of bugs and then fix them, rather than get the software right first time.
- No helicopters: 30 bugs are reported on a particular feature. Special case fixes are added to clear the bugs one by one. No one stands back and looks at the big picture and sees that the underlying cause is two competing incompatible designs.
- Schrödinger bugs No one really knows if the bug is alive or dead. Perhaps because no one has looked in a long time, and the people who did know have left. Perhaps because the explanations in the tracker about the bug are harder to grok than quantum physics.
- Schlemiel's bug Meeting: Much work has been done on a large number of longstanding bugs. Unfortunately in the bug meeting it is necessary to read the bug audit trail back to 1989, as there is no summary field. As the buglists get longer the work spent reviewing buglists increases with the square of the time passed since the bugs were first reported.
Cherry Picking and Dredging
- Cherry Picking: is searching bugzilla for easy quick to fix bugs - and fixing them. They'll usually be P4s. It's a great way to make inroads on the actual bug numbers. Its main benefit is on the bugtracking, since it's P2s and P3s that have most impact on users.
- Toxic Sludge Dredging: is a review meeting involving long time developers of the long outstanding and open bugs. The purpose is not so much to close the specific bugs discussed right now as to learn from the experience. It can lead to new policies, such as the policy to move to a plug-in API for new features. Finding ways to protect the core of Audacity from new bugs is part of the quality drive.