Bugzilla marking of tested bugs
- Add a keyword "test_single_OS" (not single platform) which will usually be added by the committer of the fix if appropriate. QM can remove that keyword if he sees fit (or add it).
- Add a "FIX_FULLY_TESTED" status. If the "test_single_OS" keyword is absent, then even if the bug occurs only on one platform, the status cannot move from DEVEL-FIX MADE to FIX_FULLY_TESTED until bug comments indicating successful test on all platforms are posted.
So for clarity and to bring this to a clear conclusion:
- There will be a new status: FIX-FULLY-TESTED. Just as DEVEL-FIX MADE really needs a GitHub commit id link, FIX-FULLY-TESTED needs a comment, or comments, stating what has been tested.
- The new status is optional. Gale can take a bug straight from DEVEL-FIX MADE to RESOLVED FIXED as we do now.
- There is a new keyword test_single_OS. It operates as described below, and is also optional. As now, Gale could mark a wording bug RESOLVED FIXED having tested on one platform, without the test_single_OS keyword being present.
- Anyone can add the test_single_OS keyword, but if Gale removes it, that 'sticks'. I.e. if a developer thinks a fix only needs testing on one platform and Gale disagrees, Gale's view holds. Generally if anyone removes it that should stick, because likely they know something we don't.
The reason I am explicitly making these new features optional is that means we only need to use them where they actually save us time and energy. We're not encumbered by having to make sure we have used these everywhere where we could have used them.
The choice of how to collaboratively operate Bugzilla and what statuses to have is down to Gale - though if Gale stops being in charge of QA that changes too. My statements above are, I believe, consistent with what Gale has set out below, and are just to clearly set out how we use the new state.
Update from Gale & James 08Feb16
- Gale: The only comment I have is that if Status moves to RESOLVED - FIXED after testing only on one OS I think test_single_OS should be added if it was absent. Otherwise if the bug is reviewed later it may not be clear why the bug was RESOLVED. So if test_single_OS was legitimate, it's mandatory and not optional.
- James: Totally fine with that. 'test_single_OS' mandatory if bug was marked RESOLVED - FIXED after only testing on one platform.
Side discussion 1
Gale mentions an important aspect of testing, that I think if paid more attention to will speed up the rate at which Gale can mark bugs off as resolved. It's around testing more than the steps to reproduce. It's about being smart about what a possible fix may have done.
For example, I fix a problem with writing junk on the ends of files. Tested properly, a tester will check that in fixing it I haven't inadvertently left off something that should be in the file and previously was. Testing of second guessing what could have gone wrong, if reported in Bugzilla, should give Gale more confidence that the fix is good.
I think this would make the most difference on P3 and lower, where it is less risky for Gale to just rely on the reported testing. Nope, we're not going to discuss statuses for "second-guessing tests" at this stage :-) It's just awareness. So if you do such a test, reporting briefly on it in Bugzilla is good.
Side discussion 2
I'm really pleased with the way Bugzilla comment trails have got shorter and more purposeful. This helps such a lot.