Proposal Development of spectral selection and manipulation

From Audacity Wiki
Jump to: navigation, search
Proposal pages help us get from feature requests into actual plans. This page is a proposal to ...
Proposal pages are used on an ongoing basis by the Audacity development team and are open to edits from visitors to the wiki. They are a good way to get community feedback on a proposal.


  • Note: Proposals for Google Summer of Code projects are significantly different in structure, are submitted via Google's web app and may or may not have a corresponding proposal page.


The Problem

I copied a slightly edited version of James' first comment in Bug #917 as it seemed to me to state fairly succinctly some of the problems.

There are interface problems with spectral selection that make the Audacity user interface considerably more confusing. [We actually need to be moving in the opposite direction, as Audacity is already too confusing for new users].

Selection on the spectrum is now always spectral selection (when in the Spectral Selection views). However for nearly all effects the spectral aspect of the selection does nothing. This is not at all obvious to the casual user. The snapping of the center line whilst valuable for people using the three spectral selection effects would look fairly random and incomprehensible to a typical new user.

The spectral selection feature as a whole violates a key principle of Audacity, that of discoverability. It is almost impossible to discover how to use it without having watched a video or read the manual.

Possible partial solutions are:

  1. making the spectral selection aspect an opt in feature,
  2. extending spectral selection to work with all effects,
  3. full solution would do both.

Peter's take on the problem

  • For me the basic problem is that Spectral selection, editing and manipulation is a fairly advanced piece of functionality like to be of interest and use to a smallish majority of advanced users. The nub of the problem here is that there is no way to turn spectral selection off, it is present whether used/usable or not (a topic we have been discussing since January 2015) - for the majority of users this would be useful and for those users the default should really be off. A simple switch for turning it on and off should also aid discoverability for the more adventurous users. This was raised and discussed in Bug #894.
  • The solution that James provided for 2.1.1 does have some benefits:
    • it restricts the spectral manipulation to the spectral views - but only sort of, as a spectral selection made in a spectral view persists in a non-spectral view even if it has no impact in that view
    • It enables users to use a spectrogram view without being bothered by a spectral selection (and there are plenty of uses of spectrogram views that do not need or benefit from a spectral selection).
    • It aids discoverabilty via the TDDM menu items for spectral views.
    • It does have a major downside imo and that is that it introduced two additional spectrogram views which would be unnecessary if we had a simple on/off switch (which would appear to be Gale's view too). And it appears to be Paul's view too (looks like a consensus may be developing here).
  • There is no visual cue in non-spectral views that a spectral selection may be extant (unless the user has turned on the Spectral Selection Toolbar - and note that this is not a pre-requisite for making spectral selections)
    • The Status Bar should indicate that the Spectral edit effects are required to edit the spectral selection (as Gale has proposed earlier see Bug #917 Comment#1.
  • I support the idea that in future releases spectral selections may operate on various effects:
    • but we would then need an indication of which effects did and did not work with spectral selections.
    • And I think that an effect that can operate with a spectral selection should probably have a visual cue in its dialog to show that a spectral selection is extant and will be operated on.
Updated 27Jul15 in an email thread:

Paul wrote:

>James, I think it is weird to have a per-track setting to enable spectral selection

Not at all, I don't think there is anything weird in wanting to make that setting per track for those do need and want to use spectral selection. It could be perfectly appropriate to have some tracks where spectral selection is available and others where it is definitely not required I ran this past my "mystery shopper" (Mrs S. a fairly naïve Audacity user) and even she could see the benefit of a per-track setting.

But I do agree with Paul that we also need a global on/off switch for Spectral Selection and have argued for such a switch since the 2.0.0 alphas, as there a vast majority of us users, including myself, who have no need of Spectral Selection. It is imo an advanced tool for the benefit of skilled and advanced users and is best left turned off for all the rest of us.

My vision for the UI is that

  1. The global switch (however implemented a button/preferences/whatever) would be set by default to be off
  2. There should be some visual cue to indicate that SS in "on" and available
  3. When SS is enabled then the TCP for each track will carry an additional dropdown menu item to enable/disable SS for that particular track (this could perhaps be the required visual cue)
  4. I can see that it might also be very useful to have some mechanism to enable/disable SS for all tracks or all selected tracks - in which case the "global switch" could be tri-polar:
    1. off (default setting),
    2. on and available for per track section from the TCP
    3. globally on for all tracks (but individual tracks could perhaps then have a mech. to turn SS off for that particular track).

Doing this would enable us to lose the two new Spectral Selection track views that were introduced in 2.1.1 and thus shorten further the TCP dropdown menu.

Paul's opinions

  • See this page for a project that bears on this issue, among other things. https://wiki.audacityteam.org/wiki/Proposal_Reorganizing_Preferences,_Track_control_menu,_and_View_Settings
  • It is not correct that the influence of spectral selection is limited to three effects. Advanced users can program Nyquist to use spectral selection information in new effects. 2.1.0 Noise Reduction almost shipped with awareness of spectral selection too. Mention of particular effect names in status messages should be avoided.
  • Remember that there were two independent "dimensions" of selection: a choice of a time interval, and a choice of a set of tracks. Time selection might be defined even when the set of selected tracks is empty. Shift-click in Track Control causes the shaded selection to appear and disappear. To keep spectral selection code simple, we just extend that notion: there is one choice of a frequency range, independent of the set of selected tracks. Frequency range selection may remain in effect even when no track is selected -- just as for time selection.
  • It was argued that user confusion should be avoided by making invisible spectral selections ineffective. But this should not be done in cases where track view is waveform and yet the spectral selection toolbar is visible. A sophisticated user who knows what he is doing, and prefers that means of entering frequency parameters to an effect, should not be prevented. Prevent spectral selection from influencing the effect only when none of the selected tracks shows spectrogram, AND the toobar is hidden.
  • Or instead, simply use spectral selection information always, as in my original design. Make spectral selection minimally visible whenever any track is selected, by drawing some new glyph in the Waveform views. (And what about Pitch views?)
  • The simple means to prevent spectral selection from having effect is to change Effect.cpp so the frequency selection is not communicated to the effect. A less general and more complicated solution was implemented instead, which I strongly dislike now and want retracted in 2.1.2.
    • The type of the view is communicated to the Lisp runtime in the 'VIEW property of the *TRACKS* global variable. The three spectral editing effects check this variable and report error messages instead.
    • This is wrong in principle as it allows effects to detect view type and change behavior in arbitrary ways. An effect should be oblivious to the view type.
    • This requires replication in any new spectral effects we might invent. Better to have one point in the code that simply prevents passage of spectral selection parameters. This would cover any built-in effects, too, which might use the information, such as Noise Reduction.
    • 'VIEW is used for one other thing in a few Nyquist effects: it is compared to NIL to detect when an effect is executing in preview, and in that case an optimization is done, trimming the input sound. Communicate only a boolean instead, not a string. Or even, again, find a general solution at a higher level instead so that replicating this logic in several Nyquist effects is not needed. Find a way to truncate the sound before it is passed to the Lisp runtime.
    • 'VIEW is documented here: https://wiki.audacityteam.org/wiki/Nyquist_Plug-ins_Reference#.2ATRACK.2A So it has been published. It should be retracted, or at least deprecated. Can we hope nobody is using it and just do that without notice? Do we define version 4.1?
    • The above documentation is outdated already! The set of view types has been changed again in 2.1.1 and I hope to change it yet again in 2.1.2. So long as this property remains, just link to another manual page that lists the types.
  • The extended list of View types, as I think we agree, was only a quick and not entirely satisfactory solution to the problem of limiting user confusion while also making spectral selection discoverable. Users who use spectrograms at all will now be required to make the choice to enable spectral selection in the drop-down menu.
    • Here's a minor problem I just thought of: the old preference value of "/GUI/DefaultViewMode" may be misinterpreted in 2.1.1 because the new enum values were not added at the end!
    • But "Spectrogram" and "Spectrogram log F" never really made sense as view "types," and they (and other things) lengthen the track control menu too much. See my proposal below. In a more elaborated version of the project, "Spectrogram" only will be a view type, contrasting with Waveform. Choice among linear and log (and mel and bark) is a subsidiary choice among "scales" in a new menu in View Settings, and choice whether to enable spectral selection or not will be a checkbox, orthogonal to the view choice. (Perhaps the choice of Waveform dB, too, is a choice of scale, not of type, and that belongs in another page of the View Settings dialog.)
    • The choice to enable spectral selection or not makes more sense as a global preference than as a per-track setting.

Other discussion

Steve 14Jun15: As has been mentioned on the mailing lists, "spectral" editing in Audacity is very much in its infancy. At this stage we are just scratching the surface of the possibilities that spectral editing opens up. Some spectral editing features that I would like to see developing over the next few years (in no particular order):

  • Multiple selections: Rather than restricting the selection to a single rectangle defined by one upper frequency, one lower frequency, one start time and one end time, allow an arbitrary number of selected "spectral" regions on which an effect will act.
  • Frequency selection performed in the common effect code: Rather than defining frequency values in the common effect code and handling the filtering in the effect, to handle the filtering in the common effect code. This would allow all processing effects, including third party plug-ins, to act on spectral selections.
  • Arbitrary shaped spectral selections: In real world audio, for a sound to have a static frequency is the exception. Most real world sounds vary in frequency over time. To be able to "draw" a selection that follows not only the start / end times of sounds but also follow changing frequency, would be a very powerful feature.
  • Simultaneous spectrogram and waveform views: All track views are conceptually "visualisations of sound" and we provide different track views because they reveal different information about the sound. Some types of detailed editing currently require frequent switching from one view to another. In such cases the work-flow would be greatly enhanced by the ability to display multiple views of the audio at the same time.
  • Select by intensity: Similar to a "magic wand" tool in a photo-editing program, clicking this selection tool on a bright region would select trace a selection region around a sound based on the intensity. As an example, a musical instrument note could be quickly selected by clicking on each of the visible harmonics of the note.
  • Editing functions: Cutting, pasting, duplicating and deleting spectral selections.

Developer/QA Backing

  • Gale:
    • +1 to removing the choice for Spectral Selection from the Track dropdown menu for 2.1.2, in favor of a Spectrograms Preference which is mirrored in some other easy-to-access control in the main project window.
    • +1 to Paul's original design "use spectral selection information always, make spectral selection minimally visible whenever any track is selected, by drawing some new glyph in the Waveform views".
    • -1 on compelling use of Selection Toolbar as the trigger for whether an effect recognises a spectral selection or not. The trigger should be having the Spectral Selection Preference on.
  • Peter: +1 to Gale's suggestion - this sounds like the "simple switch" I have been asking for

Use Cases

TBP


Details

  1. Remove the choice for Spectral Selection from the Track dropdown menu for 2.1.2
  2. Add a Spectrograms Preference which is mirrored in some other easy-to-access control in the main project window.


GUI Examples

TBP


Previous Feature Requests relating to this proposal


  • repair in spectral windows: can audacity or exist some vst plug-in for repair visual tool in spectral windows like audiodirector or audition http://www.youtube.com/watch?v=PZyjaShGzJA or http://www.youtube.com/watch?v=ZCb1xofQ18E
    • Steve wrote: No, Audacity does not have anything like that. I'll move this to the "feature request" board. I think this would be a terrific feature, though I suspect that it would be difficult to implement.


  • Spot Healing Brush: see http://www.youtube.com/watch?v=qKyuqnnguDk
    • Steve: I was watching that wondering if you'd linked to the right video, but then there it was, right at the end. Pretty cool feature - I guess that you'd want to do it in a similar way in Audacity using the track "Spectrum" view? I'd guess that would be a major project. I don't suppose you are both an audio and C+ programming expert are you?
    • OP: Ok for audio (reasonably) expert, unfortunately a newbie in C programming. By the way I believe it would be an extremely powerful feature.
    • Bruno: that spectral healing brush looks interesting