The Ideal Bugtracker
Discussion page started by User:James
|Highlights of the Design
Existing bug trackers:
- Bug trackers waste time: As the bug list gets longer, so do the bug reports - and so do the bug review meetings. From an efficiency perspective, that's a Sclemiel The Painter's Procedure. There is no summarisation. Work to do to review bugs grows worse than linearly. Eventually that weekly bug review meeting has to be abandoned. No one has the stamina to do it all in one sitting. It's an unsatisfying meeting too. It keeps degenerating into trivia - often the same 'push my button' topics. The bug trackers discourage changing of the bug title to focus on the current (remaining) problems, because doing so can seem to make a nonsense of the audit trail.
- Have a culture of blame. Who messed up? Have I passed the buck in time? That's counter productive for an open source project where everyone is a volunteer. The culture of blame is why the audit trail takes centre stage in conventional bug trackers.
- Current bug trackers are psychologically toxic. This is down to their design. A rule of thumb is that the maximum number of Defects most people can cope with before hitting the "I-don't-care/It's-not-my-responsibility" attitude is about 20. That limit makes bug trackers the wrong place for feature requests or identifying usability issues - but logically they belong there too. The top twenty limit is particularly problematic as over time bug trackers accumulate toxic sludge - these are long standing issues that no one really wants to look at, often for good reason - and these eat in to the useful 'top twenty'.
There is a clear opportunity for something better!
Antidotes to the issues
- Try and keep a light touch on special powers on the issue tracker - just as in wiki. Set the culture as on wikipedia where the people who actually do work get to decide. As in wiki make it easy to join in. It spreads the load and makes for more of a community environment. The special powers, where needed, would be the equivalent of a revert or a page lock - rather than most features being locked by default.
- Short punchy lists. Keep the 20-defect care-about limit in mind. In Audacity we are already below that limit on P1's and P2's. We are aiming to get below it on P3's (the must be released noted issues).
- Moonphase bugs (bugs which no one knows how to reproduce) could defeat that since they can take forever to clear. We'll have a separate list for Moonphase bugs, and we give big kudos to anyone who turns one of these from a Moonphase to a repeatable. It might well not be a developer.
- We have summary issues which coordinate several issues. Examples could be 'Label Linking Issues' and 'Theming Issues'. Summary issues give us helicopter views of our bugs. Individuals can have personalised summary issues, e.g User:James/MyCareAbouts. The developer care abouts might themselves be collected into Developer_Is_Interested_Issues.
- We are perfectly OK with having feature request lists overflowing with ideas, as we already do on wiki. They are seen as opportunities to mine for ideas, not black marks against the project or developers. We will still use the summary issues to organise them, but we don't need to work as hard to keep the component lists small.
- Distinguish between 'Important' and 'Urgent' (which is not only Important but can and must be done soon) when making lists. The urgent list should always flow quickly. Putting an important task that can't be done quickly into the urgent list is bad. It just gums up the works.
- Dredge the sludge and turn it into pages with valuable content so that it is no longer toxic. A wiki is better at facilitating this than normal bugtrackers. This is a cultural norm rather more than it is an issue tracker feature, but the wiki nature facilitates it. This is an important principle. Ideal issues are identified and fixed in a short space of time. The sludgy issues have problems, such as moonphasiness, disagreement about whether there is a problem at all, too many conflicting alternative proposed solutions, solutions we can't implement for patent/other reasons. Rather than being bogged down by these we want to turn the discussions into policy pages, gold that adds to how we do things in the project. Closed state not a black-hole: Use closed state issues as a resource that feed into other pages. Turn 'Won't fix' issues into policy page items on the project's goals.
See Also: Review of Issue Trackers
What we do in Audacity
We use a wiki
- We have a Feature Request Pipeline, and keep our active buglist on the Release Checklist page.
- We note (or used to note) MoonphaseBugs that is bugs that we can't reproduce. These unrerpoducible bugs are bugs which would otherwise clog up the issue tracking.
Why might we change it?
- GSoC 2009 brought a lot more bugfixing activity to Audacity and this is continuing into 2010. We want us and our students to avoid stepping on each others toes. The formal idea of claiming a bug is useful and something we have in only a weak form as initials on the Release Checklist
- We'd like to participate in GHoP some time, and for that there are additional requirements.
- We're not good at tracking and dealing with bugs that 'belong' to other projects that we use.
- We look a little ramshackle with our wiki page bug lists. Nicely formatted sortable tables might make us look more professional.
- Understanding our bugs better would lead us to better processes. For example 'Multi-Clip' and 'Linked-Labels' each brought in a whole slew of new bugs. We can change how we do things so that we identify bug breeding grounds early and change the dynamic.
These are items that there is general consensus on...
Wiki with scatter gather
- Summary issues where we create a new issue to coordinate other issues. This is not at all the same as merging issues as 'duplicates'. The original issues are subordinate to the summary issue. Closing one subordinate issue does not close the other. We may also want to turn an existing issue into one that coordinates several issues. A single report may be of multiple related issues. Similarly for feature requests, there are often several competing proposals to solve the same problem. They too should be linked to off a summary.
- Flexible merging and splitting bug reports often become feature requests that then become progressively more complex. We want to be able to work with several reports at once through merging so that there is no penalty for people working through the list for breaking issues up.
- We're fairly certain that wiki functionality is central. We're fairly certain that we don't want a dedicated hard-coded issue tracker. Rather we still want a wiki, but one that is flexible enough that it can be configured as a general issue tracker. A wiki that can extract information from multiple pages could work well for us. We want to retain the flexibility of using wiki.
- We're looking at the possibilities of a wiki-with-forms with scatter-gather functionality. What that means is that you can transclude other pages as in existing wiki, but more than that you can edit the compound document and have the server edit the component pieces. A really important aspect here is that one can work with the text of several related issues and commit a change to the reports on several related issues with one click. Current designs of issue tracker require that you work on the text of issues one issue at time.
Release notes integration
- We'd like to generate release notes directly off the bug tracker. So we'd see the issues cleared in this release - we'd also see the 'known problems' and the 'MoonphaseBugs' that we'd like help with tracking down. The key here is that we want high level text, not issue by issue. We want text phrased for release notes, not text phrased for the issue tracker. Examples of how release notes might look:
- '8 issues with label track linking fixed (#1234,#7823,#...)'
- 'Under Windows Vista wacky-soundz soundcards do not register properly. You can work around this by...'
- 'Ways to reliably reproduce the following problems would be appreciated:....'
- Release notes view shows us what the release notes would look like if we had to release right now. Open bugs of sufficient severity will need release note text in them to say how they are to be reported in release notes.
- Rather than thinking of the tracker as one big list, we want a coordinated set of lists and related information pages. The key is the state of the issue, the 'what needs to happen next'. The format of these pages will vary depending on the state.
- There are Defects which are not contentious. Everyone agrees.
- There are MoonPhaseBugs that we have difficulty making happen. You gain major kudos for converting a bug from a MoonPhaseBugs to one with a scenario. Within the MoonPhaseBugs we have finer classifications. Some are bugs which no one has been able to reproduce, that may be peculiar to what the user is doing. Until we realised that people were opening the same project twice many of the recovery bugs were unreproducible. Other MoonPhaseBugs are intermittent. Possibly they are race conditions that only very occasionally affect people. We can sometimes get these to happen more often by using smaller block sizes to stress Audacity.
- The Feature Requests become more organized, so that in particular how they relate to Use Cases is clearer. Use cases can be like summary issues. They act as an umbrella for multiple related feature requests. There is also some community building in the use cases. We build sub communities of people with special interests. This is a little like blueprints in Launchpad.
- Old Faithfuls are any bug that's over a year old. These may be bugs we can't solve because they are actually in driver code that we use. These should lead to explanatory pages where we tell you how to update drivers and hassle the driver maker. Old faithfuls may lead us to audacious plans, like a 'blue-screen-recovery' feature so that we can detect a driver issue and tell the user what really happened.
- Code temperature is a high level view of issue densities. This is a report based off the summary issues. Hot means more bugs reported against it. Generally we expect newer features to be hotter than older ones. Major kudos for introducing a new feature that is cool. Code temperature also applies to documentation. Some parts of the documentation may be causing more issues than others.
- Review chart tracks which bugs have been visited in a review, and which ones have been updated since the review started and not yet revisited. Any number of people can have review charts.
Value from issues in closed-states
- We don't lose information from issues in the closed state.
- As already noted a verified fix becomes or contributes to a Release Note.
- A won't-fix becomes a note on a policy page. We won't fix the 'bug' that WMA is not supported. It's not an open standard that we can support. We need to tell people this or it will just get reported again, and again, and again. We know. We've been there.
- A can't-reproduce becomes a MoonPhaseBugs on the watching brief page.
- A use-workaround becomes a Tips and Tricks and as well as that it becomes a potential future enhancement.
- A doc-fix gets worked into wiki documentation. The interesting part here is to identify where many people are experiencing the same issue because that may be a case not just for improved docs but for improved GUI.
Note that we are feeding the bug outcome into other online documentation such as FAQs.
- Triage: Deciding what needs to happen for ALL live issues in the tracker.
- Cherry Picking: Going through fixing and closing trivial issues (i.e. quick to fix issues) so that they do not clutter up the tracker. Often leads to trivial issues changing status (e.g. to NeedsInfo ).
- Dredging: Going through issues which are 6 months+ old, checking they are still valid, ensuring there is some policy in place so that they don't clog up the tracker. Check for example that old issues which are not live have milestone 'future'. 'Dredging' can also be used for learning from closed issues, for example a spate of related invalid issues may indicate a need for some documentation changes.
- Duplicate avoidance: How about a keyword search at the time of creating a bug entry... someone types in something about WYSIWYG, and the tracker asks when you enter the bug... "Have you looked at issues 1234: (WYSIWYG fails on..)?"
- Highlight NEW contributors to the issue list. This is about the people, not the issues. 'New' may at times be a euphemism for 'does not know how to write good issues'! We don't want to block the general user population from adding to the tracker. We do want to encourage people to follow our norms (good naming of issues, correctly understands difference between defect and feature request etc...). Someone helping with triage can look at the issues from new contributors and help get them into shape. We're recognising good contributors to the issue list. They quickly become established contributors and lose the 'new' tag.
- Wiki works well for us.
- Wiki reduces clicks: We can edit multiple 'bugs' in one POST - they are all on one page - and we can see the history of what has gone on, if we want to.
- Wiki shows current status: One could say that standard bugtrackers emphasise the 'audit trail' whilst using wiki emphasises the current status (with audit trail available if it is wanted).
- Collecting from BUG pages: At a telecoms company I worked for I modified a wiki so that some special syntax would give us the results of queries from 'all pages that start with the word Bug'. That actually worked very well for a project with several thousand issues. The biggest single advantage was the flexibility of what fields we could include in the bug report.
- Support for voting: The voting scheme that we currently use on Feature Requests is primitive. Moving of popular features to the top is not automatic. Currently we restrict people to ten votes. I'd like people to be able to vote on as many features as they like. I'd like to see who has voted for what (it matters to me). It can be done by looking at history, but not easily. I'd also like a star rating on proposals by the developers so that contributors of any kind can see what we developers consider a good feature proposal.
- Make bugfixing fun: Paul Graham makes the interesting point of how bugfixing became a game in viaweb. Developers interacted with the end users, and it became a challenge as to how quickly the issue could be identified and fixed. It was usually the most savvy users who were hitting the bugs, because they were using the most advanced features.
- Identify psychological toxicity in bug hunting and change it, by changing the roles. In my first job in Ireland I wrote software to validate a scientific satellite. I remember how the first few times when I came to the developers with bug reports of new bugs the very negative vibes I got. I quickly learned that coming to the developers with a bug report AND pinpointing in the code the required fix turned the bad vibes to positive ones. We can't expect that from most users, but we should do everything we can to encourage those bug reporters who can do so to progress the bug a bit further. Make it easy for them to link to a line and suggest 'is this where the problem is?'.
- Procedural changes: Many improvements are outside the issue tracker itself, but are suggested by the structure. For example, a 'dredging' meeting to turn the long standing issues into useful documents is suggested by the 'Old Faithfuls' category.