Talk:New Builds Team
- Use audacityteam.org blogs to coordinate this, e.g. the Calls and the tracking of user response?
- Want high payoff for testers - they get early access to features that interest them?
- What level of participation by testers makes what level of effort by us in setting this up worthwhile?
Lots to work out...
Gale 06Jul16: I would not like to track and co-ordinate in the WordPress interface we have now. And if we want any tester input there, it's ruled out because we can't give access. So really the blogs are only for calls and announcements, I guess. And I still think we should use the announcements e-mail system as well. Who can find the "blogs" on our current WordPress site?
James 06July16: Tracking is more easily doable than you think and similar in ease to tables in Wiki. We could even give a few alpha testers non publishing rights on Wordpress, I believe, but let's not plan to. Blogs can't currently be found because we don't have any written, other than announcements. Blogs can appear in sidebar and be very findable.
- Gale 16Feb17: Sidebar news/blog entries are currently hidden below the scroll (or at the bottom of the page if zoomed in or using mobile).
James' Ideas for Recruiting / Engaging
- Use a moderated google group for the 'early adopters'
- Use Q2A for feature-feedback. Always phrase a design decision as a question.
- Have blog posts with work-in-progress (that is ready to try out) and screenshots.
- Take the most photogenic screenshots and post them on Facebook.
- Connect discussion of themes with the Website Project so we have a clearer plan for graphic designers.
Safety in Numbers
One of the disincentives to trying experimental builds is the risk. For a user this is reduced if:
- There are significant numbers of users of the experimental build.
- All users subscribe to an email list where deprecated builds (and reasons) are announced.
- Users know to save their recordings as wav, before doing extensive work on them.
If we do make a build like 2.1.3 RC1 which has a serious problem (the problem with SaveAs with multiple-projects), the likelihood is that we will find out about it quickly and alert users to it. The risk to each early adopter is then very small - even when there are problems.
- Gale 12Mar17: E-mail lists are ruled out by Buanzo as I understand it, so why not just announce the deprecation to to the Google Group, special Forum or whatever the hangout place is?
- James 13Mar17: Google Groups are an e-mail list, just ones of a particular kind. Since this is new sign-ups with a very clear disjoin my understanding is that Buanzo is fine with Google Groups.
Different levels of engagement
Gale 16Feb17: Users interested in trying out new features in an experimental release probably have a little less engagement than those willing to test more broadly in a milestone alpha (a.k.a. KTT alpha). Both will have much less engagement than anyone willing to test nightly builds on a regular basis from the start of the development cycle, but we should try to encourage a few users into that too.
Peter 16Feb17: James wrote on the page: "This is for selected alphas, ones that could be suitable for production use."
Tradtionally, and customarily, such alphas that are considered "suitable for production use" are labelled as Betas.
Our Alpha nightlies are usually bleeding-edge works-in progress.
At risk of opening up an old Audacity discussion about "no Betas" I really think we should be considering forming not an "Alpha Test Team" but rather a "Beta Test Team" - it would appear more honest and more clear.
IIRC the reason we adopted the "no Betas" was because of the long run of the 1.3.x "Beta" series without getting anywhere near a formal release. This should not be the case if we adopt what I am suggesting here.
See this Wikipedia page on Software release life cycle
- Gale 16Feb17: I suppose it is possible that we could call an alpha at semi-freeze (which typically has no open P1's) a "Beta" however there is as you say the historical baggage of our Betas and the perceived instability that comes with them. I notice that many users call our alpha nightly builds "Betas" and clearly don't understand the difference between the two stages.
- Gale 17Feb17: My vote is for alpha-milestone-<number> for milestones and for topical alphas, <alpha-<feature>-<number> and not to split the testers initiative into two parts (at least, I think that right now).
- James 17Feb17: Fine by me for the file-name of the build on Audacity-devel.html, even if a build does happen to be of very high quality. We may not know for sure it is high quality until later.
- Gale 17Feb17: Do you or Peter think the semi-freeze build only should be deemed a beta? That seems closest to me to the idea of the version cycle. Thence onwards nightlies might stay as beta according to About Audacity. I'm not sure though given my points about user perception/understanding of "beta".
- James 17Feb17: Your observation that many users conflate alpha and beta, leads me to recommend we settle on just saying alpha. Perhaps in the context of Audacity-devel.html it is even enough to say 'build'. It is clear they are not final Audacity releases.
- (Peter 18Feb17: It seems to me that there are three types of testers that we need to recruit and maintain:
- A-testers: these are folk who test nightlies and in particular test the (often small) changes that are introduced in each nightly, for example bug-fixes.
- B-testers: these are folk who we would want to deploy when big changes are rolled into Audacity. A good upcoming example could be when we fold James' new menu structures into Audacity. A past example could have been scrubbing in its later implementation days. A good cadre of B-testers could prove useful then.
- C-testers: these are folk who would test an Audacity build in its entirety, typically using their normal workflows.
- In the software industry the A-testers typically worked with alpha releases, the B-testers were typically provided with Betas and the C-testers had Release Candidates or Kick-The-Tyres builds. Whether we have to ability to recruit and maintain such complexity is an issue for discussion I feel.
- James 18Feb17: I'm with Peter on this. I think our best way to attract testers is through tasty new auteur builds that give previews of possible new features. These attract C-Testers, people using it in their production work (with a little more care around the newest features). Some of those will get interested in a particular feature and be keen to try out iterations (B-Testers) and some will also learn of Audacity test approach from being a C-Tester and decide they want to be more involved by being an A-Testers subscribed to [email protected] C-Testers and B-Testers are in my view the responsibility of the auteur. It's in the auteur's interest to use this feedback and engage with these testers. That's not to say QA can't be involved. I would advocate a single board on the forum for C & B builds. Auteurs will maintain a page on wiki about their builds, and may update with news. Occasional blog posts with screenshots will be made by auteurs and reshared on facebook. To convert more C & B testers to A, auteurs and QA can do the occasional blog post about bugs in HEAD found and fixed via C and B testing.
- Gale 19Feb17: Some of us are opposed to "auteur" builds from outside the Audacity project. That is what the upcoming vote is about. We can't have an integrated Audacity(R) test program IMO with third-party projects involved - not until Team approves it anyhow.
IMO we're playing down new features in HEAD too much with the suggested approach - those features likely will be in the next release. New features in "auteur" builds will I assume mostly come on stream in the release after next (if at all). Otherwise, what is the essential difference between "auteur" builds and a well chosen alpha pre-release from HEAD, other than "auteur" branding and distancing from the real Audacity(R) project? Is the difference a very slightly earlier preview of what will be in the Audacity(R) alpha pre-release builds? If yes, we should be careful of providing too many similar builds to test.
- James 19Feb17: Team might decide that a fork by a team member that is interested in working with Audacity is automatically a friendly fork. The extra 'auteur collateral' (website/facebook) shows that that 'auteur' is going beyond the minimum to generate interest and bring in people to help advance their vision for Audacity. I suppose if the auteur said their fork was not actually GPL (i.e. in violation of the GPL) or displayed no interest at all in their changes moving upstream to Audacity or was trollishly rude to Audacity, those would be grounds for reversing the friendly designation and deciding not to work with that 'auteur' and their fork.
- James 19Feb17: Yes. Features in an Auteur build will often be one-release ahead. Often it will be being used by the auteur as a mechanism to convince team that their features actually are good ideas and ready. I had a bit more success with that than I could really cope with during 2.1.3, as Peter wanted three of my features expedited into 2.1.3, which I did. The reverse could happen - e.g. from team and user feedback I realise that my 'rearranged MUTE/SOLO buttons' need more thought. We do get there eventually, but it's a two-release-ahead feature, because what is actually picked into Audacity is a version with icons with tooltips rather than text (solving long translation into Russian issue a different way) and with a hover-over effect that is currently missing.
- James 19Feb17: I think the experimental designation of the builds does possibly too much lower the bar for the developer. DA 2.1.3x is more stable than most RCs we have produced. It had to be, or I'd lose credibility. I am thinking that with an easy channel to users on Audacity-devel, auteurs will be tempted to be a bit more slack.
Confused nomenclature talk moved from main page
Gale 17Feb17: Although James now thinks "beta" is the best term to use, some of this page refers to "alphas" so it is confusing.
Although "beta" is more correct, it is the probably the wrong term for us
- due to the bad experience of the 1.3 Beta series
- because most of our users don't understand the alpha / beta distinction - I have lost count how many times the nightly builds are referred to as betas.
- because a beta could/probably will be the same as the corresponding nightly build. If those nightly builds switch between alpha and beta in "About Audacity" it could get confusing (if we call them "beta" then About Audacity must say "beta").
Calling this page "Beta Test Team" means it is necessarily different from the just as important initiative to recruit people to test and report on the nightly builds. The "alpha test team" is likely to become depleted sooner rather than later and so it is essential new blood is attracted that will be fully rather than partially engaged testers. The Beta Test Team initiative if called that must therefore mention the Alpha Test Team initiative so we have the chance of some trickle down from beta testers to nightly build testers.
My vote is for alpha-milestone-<number> for milestones and for topical alphas, <alpha-<feature>-<number> and not to split the testers initiative into two parts (at least, I think that right now).