Difference between revisions of "QA Procedures overview"

From Audacity Wiki
Jump to: navigation, search
(→‎3. Bug finding: numbered list)
(→‎Testing fixed bugs)
 
(38 intermediate revisions by 2 users not shown)
Line 2: Line 2:
 
__TOC__
 
__TOC__
 
<div id="alpha"></div>
 
<div id="alpha"></div>
−
==1. Alpha testing==
+
==Alpha testing==
−
We continually test the Master branch alpha builds hosted here
+
We continually test the Master branch alpha builds hosted here on GitHub:
−
https://github.com/audacity/audacity/actions
+
* [https://github.com/audacity/audacity/actions Github Actions]
  
−
Referenced by the commit log
+
Referenced by the commit log on GitHub:
−
https://github.com/audacity/audacity/commits/master
+
* [https://github.com/audacity/audacity/commits/master Commit Log]
−
 
+
{{note|I, ''(Peter Sampson)'' personally, tend to be bold (live on the bleeding-edge) and use the alphas for my production work, as the things I do are mostly easily repeatable.}}
−
I, (Peter Sampson) personally, tend to be bold (live on the bleeding-edge) and use the alphas for my production work, as the things I do are mostly easily repeatable.
 
  
  
 
<div id="rc"></div>
 
<div id="rc"></div>
  
−
==2. Release Candidates==
+
==Release Candidates==
 
When the RM is ready – Release Candidates are built for external (and further internal) testing on all three platforms:
 
When the RM is ready – Release Candidates are built for external (and further internal) testing on all three platforms:
−
https://www.fosshub.com/Audacity-devel.html
+
* [https://www.fosshub.com/Audacity-devel.html Release Candidates] - ''hosted on FossHub''
  
 +
{{tip|QA and release planning work together.  QA is often reviewing recent commits to GitHub to understand recent changes. 
 +
*The whole process of rating bugs P1 to P5 is about sharing (or not sharing) responsibility. 
 +
*A P2 bug that is allowed to go through to release is a shared responsibility between QA and RM}}
  
 
<div id="find"></div>
 
<div id="find"></div>
−
==3. Bug finding==
+
 
 +
==Bug finding==
 
So how do we find bugs?   
 
So how do we find bugs?   
−
#Our own testing – reports via dev and quality email lists
+
#Our own testing – ''reports via dev and quality email lists''
−
#User reports on the Forum – review daily at least
+
#User reports on the Forum – ''review daily at least''
−
#User reports on the three Facebook sites – review daily at least
+
#User reports on the dev and quality email lists – ''review daily at least''
 +
#User reports on the three Facebook sites – ''review daily at least''
 +
# Github  [https://github.com/audacity/audacity/issues Audacity > Issues] -  ''mainly techy issues''
 +
 
 +
 
 +
<div id="requests"></div>
 +
 
 +
==Feature Requests==
 +
Feature Requests reside in various places:
 +
* [https://wiki.audacityteam.org/wiki/Feature_Requests Feature Requests]
 +
* [https://wiki.audacityteam.org/wiki/Pending_Feature_Requests Pending Feature Requests]
 +
* [https://forum.audacityteam.org/viewforum.php?f=20 Forum Feature Requests] - ''see also the Audacity Forum''
 +
* Some are logged in [https://bugzilla.audacityteam.org/report.cgi?x_axis_field=priority&y_axis_field=bug_severity&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0= Bugzilla] as ENH enhancements
 +
* [https://wiki.audacityteam.org/wiki/Proposals Wiki > Proposals]
  
  
 
<div id="log"></div>
 
<div id="log"></div>
  
−
==4. Logging Bugs==
+
==Logging Bugs==
 
All bugs are logged in Bugzilla bug-tracker
 
All bugs are logged in Bugzilla bug-tracker
 
https://bugzilla.audacityteam.org/
 
https://bugzilla.audacityteam.org/
  
−
*A key part of the bug entry is the “Steps to Reproduce” – mandatory, the bug will not log without it.  
+
*A key part of the bug entry is the “Steps to Reproduce” – ''mandatory, the bug will not log without it''.  
 
*Platforms: we need to ascertain and record if the bug is single or multi-platform
 
*Platforms: we need to ascertain and record if the bug is single or multi-platform
 
*Priority setting:  this is perhaps the trickiest part to decide.  We work with five levels  
 
*Priority setting:  this is perhaps the trickiest part to decide.  We work with five levels  
Line 40: Line 56:
 
*# P3: must have a Release Note and ideally a workaround
 
*# P3: must have a Release Note and ideally a workaround
 
*# P4 & P5: bugs are not release noted. Many of these may be "enhancement issues"
 
*# P4 & P5: bugs are not release noted. Many of these may be "enhancement issues"
 +
*# See:  [https://wiki.audacityteam.org/wiki/Bug_Lists#Bugs_by_Priority Bug Lists > Bugs by Priority]
  
−
See:  https://wiki.audacityteam.org/wiki/Bug_Lists#Bugs_by_Priority
 
  
 +
<div id="test"></div>
  
−
<div id="test"></div>
+
==Testing fixed bugs==
 +
When a bug is fixed by a developer it gets marked DEVEL FIX MADE
  
−
==5. Testing fixed bugs==
+
If subscribed (I’m subscribed to “all bugs”) you get an email on the status change.  
−
When a bug is fixed by a developer it gets marked DEVEL FIX Made
+
   
−
If subscribed (I’m subscribed to “all bugs”) you get an email on the status change.   
 
 
In theory this should be enough to track fixed bugs that need testing – but I found that hard to manage, so I created a workbook in the Wiki:
 
In theory this should be enough to track fixed bugs that need testing – but I found that hard to manage, so I created a workbook in the Wiki:
−
https://wiki.audacityteam.org/wiki/User:PeterSampson/Bug_testing
+
* [[Bug testing workbench]]
 +
 
 +
===Monitoring===
 
This enables me to:
 
This enables me to:
−
a) See it all in a glance
+
#See it all in a glance
−
b) monitor the progress of testing on the three platforms
+
#monitor the progress of testing on the three platforms
−
c) Keep a record of bugs fixed for the current alpha
+
#Keep a record of bugs fixed for the current alpha
  
−
At release time we copy over the key fixed bugs to the Manual’s What’s New… page:  https://alphamanual.audacityteam.org/man/New_features_in_this_release
+
===Documenting===
 +
At release time we copy over the key fixed bugs to the Manual’s What’s New… page:   
 +
* [https://alphamanual.audacityteam.org/man/New_features_in_this_release What's New ...]
  
−
See the 3.0.2 version:
+
See the 3.0.2 released version:
−
https://manual.audacityteam.org/man/new_features_in_this_release.html#bugs
+
* [https://manual.audacityteam.org/man/new_features_in_this_release.html#bugs Bugs fixed for 3.0.2]
  
  
 
<div id="monitor"></div>
 
<div id="monitor"></div>
−
==6. Monitoring outstanding bugs==
+
 
 +
==Monitoring outstanding bugs==
 
See the table in Bugzilla
 
See the table in Bugzilla
 
*[https://bugzilla.audacityteam.org/report.cgi?bug_severity=Repeatable&bug_severity=RepeatableAll&bug_severity=Locale&bug_severity=Accessibility&bug_severity=MoonPhase&bug_severity=HeisenBug&bug_severity=Summary&bug_severity=Review&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&x_axis_field=priority&y_axis_field=bug_severity&format=table&action=wrap Bug Table]
 
*[https://bugzilla.audacityteam.org/report.cgi?bug_severity=Repeatable&bug_severity=RepeatableAll&bug_severity=Locale&bug_severity=Accessibility&bug_severity=MoonPhase&bug_severity=HeisenBug&bug_severity=Summary&bug_severity=Review&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&x_axis_field=priority&y_axis_field=bug_severity&format=table&action=wrap Bug Table]
  
 
or with added ENH enhancements
 
or with added ENH enhancements
−
*[https://bugzilla.audacityteam.org/report.cgi?x_axis_field=priority&y_axis_field=bug_severity&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0= Bug Tab;e with Enhancements]
+
*[https://bugzilla.audacityteam.org/report.cgi?x_axis_field=priority&y_axis_field=bug_severity&z_axis_field=&query_format=report-table&short_desc_type=allwordssubstr&short_desc=&longdesc_type=allwordssubstr&longdesc=&bug_file_loc_type=allwordssubstr&bug_file_loc=&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&emailassigned_to1=1&emailtype1=substring&email1=&emailassigned_to2=1&emailreporter2=1&emailcc2=1&emailtype2=substring&email2=&bugidtype=include&bug_id=&chfieldfrom=&chfieldto=Now&chfieldvalue=&format=table&action=wrap&field0-0-0=noop&type0-0-0=noop&value0-0-0= Bug Table with Enhancements]
  
  
 
<div id="speed"></div>
 
<div id="speed"></div>
  
−
==7. Comparison Speed testing==
+
==Comparison Speed testing==
 
I like to do comparison speed testing between versions for:
 
I like to do comparison speed testing between versions for:
−
a) I/O project open and close
+
#I/O project open and close
−
b) Exports and Imports
+
#Exports and Imports
−
c) Effects (and Generators and Analyzers)
+
#Effects (and Generators and Analyzers)
−
I have developed Macros for this.
+
{{note|I have developed Macros for this.}}
  
  
 
<div id="manual"></div>
 
<div id="manual"></div>
−
==8. Manual==
+
 
 +
==Manual==
 
For me, the Manual is a key part of QA – plus it’s now integrated as a part of the app via the “?” help buttons in many error messages and warnings.
 
For me, the Manual is a key part of QA – plus it’s now integrated as a part of the app via the “?” help buttons in many error messages and warnings.
 
Given the lack of any specs the manual can often be a good guide as to how the app is supposed to work for the user.
 
Given the lack of any specs the manual can often be a good guide as to how the app is supposed to work for the user.
  
−
There are always two current manuals
+
There are always two current Manuals:
−
a) the released version this is immutable as it gets release with and as prt of the app: https://manual.audacityteam.org/
+
*the released version this is immutable as it gets release with and as part of the app   
−
b) the alpha version: https://alphamanual.audacityteam.org/man/Main_Page
+
** https://manual.audacityteam.org/
−
 
+
*the alpha version:  
−
I find it easiest to keep the alpha Manual in lock-step with changes to the app, new developments and bug fixes – playing catch-up later is too hard work.
+
**https://alphamanual.audacityteam.org/man/Main_Page
 +
{{note|I find it easiest to keep the alpha Manual in lock-step with changes to the app, new developments and bug fixes – playing catch-up later is too hard work.}}
 +
{{tip|The '''''big''''' strength of the Manual is that it saves support people's time.  When there are 'How do I?' queries, pointing people at the Manual is a huge time saver - and the Manual has been developed as a result of decades of such questions.
 +
*Plus there are now many links to the Manual from {{button|?}} help buttons in Effects, error messages and warnings. }}
  
  
 
<div id="connie"></div>
 
<div id="connie"></div>
−
==9. Connie==
 
−
An important couple of pages in the manual help us to maintain consistency in the Manual (and to some extent the app) – we anthropomorphise this and refer to this as “Connie”
 
−
https://alphamanual.audacityteam.org/man/Consistency
 
  
−
https://alphamanual.audacityteam.org/man/Consistency_in_wording_and_punctuation_in_Audacity
+
==Connie==
 +
An important couple of pages in the manual help us to maintain consistency in the Manual (and to some extent the app) – we anthropomorphize this and refer to this as “Connie”
 +
* [https://alphamanual.audacityteam.org/man/Consistency Consistency]
 +
* [https://alphamanual.audacityteam.org/man/Consistency_in_wording_and_punctuation_in_Audacity Consistency in wording and punctuation in Audacity]
  
  
 
<div id="wiki"></div>
 
<div id="wiki"></div>
−
==10.Wiki==
 
−
The Wikipedia page should not need checking for malefactors as accounts are strictly limited:
 
−
https://wiki.audacityteam.org/wiki/Audacity_Wiki_Home_Page
 
  
−
The Wiki has a shared dev/QA section
+
==Wiki==
−
https://wiki.audacityteam.org/wiki/For_Developers
+
The Wiki should not need checking for malefactors as accounts are strictly limited:
 +
* [https://wiki.audacityteam.org/wiki/Audacity_Wiki_Home_Page Wiki]
  
−
Pages of QA interest:
+
'''The Wiki has a shared dev/QA section:'''
−
a) https://wiki.audacityteam.org/wiki/Proposals
+
* [https://wiki.audacityteam.org/wiki/For_Developers Developers' Wiki]
−
b) https://wiki.audacityteam.org/wiki/Feature_Requests
+
 
−
c) https://wiki.audacityteam.org/wiki/Pending_Feature_Requests
+
'''Pages of QA interest:'''
−
d) https://wiki.audacityteam.org/wiki/Next_Release  -  RM’s workbench
+
* [https://wiki.audacityteam.org/wiki/Proposals Proposals]
−
e) https://wiki.audacityteam.org/wiki/Audacity_Versions
+
* [https://wiki.audacityteam.org/wiki/Next_Release Next Release] -  ''the RM’s workbench''
−
f) https://wiki.audacityteam.org/wiki/Wording
+
* [https://wiki.audacityteam.org/wiki/Audacity_Versions Audacity Versions]
 +
* [https://wiki.audacityteam.org/wiki/Wording Wording]
  
  
 
<div id="wikipedia"></div>
 
<div id="wikipedia"></div>
−
==11. Wikipedia==
+
 
 +
==Wikipedia==
 +
The Audacity Wikipedia page needs updating after each release.
 +
* [https://en.wikipedia.org/wiki/Audacity_(audio_editor) Audacity on Wikipedia]
 +
{{note|The page appears to have a "guardian-angel" in the shape of Adam Hunt.}}
 +
 
 
From time to time the Wikipedia page needs checking for malefactors who may corrupt the page:
 
From time to time the Wikipedia page needs checking for malefactors who may corrupt the page:
−
https://en.wikipedia.org/wiki/Audacity_(audio_editor)
 

Latest revision as of 13:24, 6 May 2021

This is a set of notes outlining Audacity QA procedures

Alpha testing

We continually test the Master branch alpha builds hosted here on GitHub:

Referenced by the commit log on GitHub:

I, (Peter Sampson) personally, tend to be bold (live on the bleeding-edge) and use the alphas for my production work, as the things I do are mostly easily repeatable.


Release Candidates

When the RM is ready – Release Candidates are built for external (and further internal) testing on all three platforms:

Bulb icon QA and release planning work together. QA is often reviewing recent commits to GitHub to understand recent changes.
  • The whole process of rating bugs P1 to P5 is about sharing (or not sharing) responsibility.
  • A P2 bug that is allowed to go through to release is a shared responsibility between QA and RM

Bug finding

So how do we find bugs?

  1. Our own testing – reports via dev and quality email lists
  2. User reports on the Forum – review daily at least
  3. User reports on the dev and quality email lists – review daily at least
  4. User reports on the three Facebook sites – review daily at least
  5. Github Audacity > Issues - mainly techy issues


Feature Requests

Feature Requests reside in various places:


Logging Bugs

All bugs are logged in Bugzilla bug-tracker https://bugzilla.audacityteam.org/

  • A key part of the bug entry is the “Steps to Reproduce” – mandatory, the bug will not log without it.
  • Platforms: we need to ascertain and record if the bug is single or multi-platform
  • Priority setting: this is perhaps the trickiest part to decide. We work with five levels
    1. P1: these are top priority and block releases
    2. P2: important but RM decides if they can be waved through into release
    3. P3: must have a Release Note and ideally a workaround
    4. P4 & P5: bugs are not release noted. Many of these may be "enhancement issues"
    5. See: Bug Lists > Bugs by Priority


Testing fixed bugs

When a bug is fixed by a developer it gets marked DEVEL FIX MADE

If subscribed (I’m subscribed to “all bugs”) you get an email on the status change.

In theory this should be enough to track fixed bugs that need testing – but I found that hard to manage, so I created a workbook in the Wiki:

Monitoring

This enables me to:

  1. See it all in a glance
  2. monitor the progress of testing on the three platforms
  3. Keep a record of bugs fixed for the current alpha

Documenting

At release time we copy over the key fixed bugs to the Manual’s What’s New… page:

See the 3.0.2 released version:


Monitoring outstanding bugs

See the table in Bugzilla

or with added ENH enhancements


Comparison Speed testing

I like to do comparison speed testing between versions for:

  1. I/O project open and close
  2. Exports and Imports
  3. Effects (and Generators and Analyzers)
I have developed Macros for this.


Manual

For me, the Manual is a key part of QA – plus it’s now integrated as a part of the app via the “?” help buttons in many error messages and warnings. Given the lack of any specs the manual can often be a good guide as to how the app is supposed to work for the user.

There are always two current Manuals:

I find it easiest to keep the alpha Manual in lock-step with changes to the app, new developments and bug fixes – playing catch-up later is too hard work.
Bulb icon The big strength of the Manual is that it saves support people's time. When there are 'How do I?' queries, pointing people at the Manual is a huge time saver - and the Manual has been developed as a result of decades of such questions.
  • Plus there are now many links to the Manual from ? help buttons in Effects, error messages and warnings.


Connie

An important couple of pages in the manual help us to maintain consistency in the Manual (and to some extent the app) – we anthropomorphize this and refer to this as “Connie”


Wiki

The Wiki should not need checking for malefactors as accounts are strictly limited:

The Wiki has a shared dev/QA section:

Pages of QA interest:


Wikipedia

The Audacity Wikipedia page needs updating after each release.

The page appears to have a "guardian-angel" in the shape of Adam Hunt.

From time to time the Wikipedia page needs checking for malefactors who may corrupt the page:

Retrieved from "https://wiki.audacityteam.org/w/index.php?title=QA_Procedures_overview&oldid=43815"