Talk:Completed Proposal: Improvements to Scrubbing - Phase-2
- 1 James' email of 29Sep15
- 2 Peter's email of 29Sep15 (an excerpt)
- 3 Steve's email of 29Sep - how other audio editors do scrubbing
- 4 Steve's email 02Oct15
- 5 Paul's email of 29Sep15
- 6 James (talk) 17:00, 2 October 2015 (EDT)
James' email of 29Sep15
James wrote in an email 29Sep15:
Play and record are indeed modes too, just ones users are much more familiar with. As we improve Audacity I/we would like to be able to do more things we currently can't do in those modes, but that is nice to have, not an urgent problem. The problem with scrubbing being a mode is more serious.
With play and record the cursor is moving and either sound is coming out or new waveform is being drawn. With scrubbing, if the cursor has caught up with the mouse, it isn't as clear that the user is in a mode. Sure, the play button is down, but that does look a bit as if play is active but for some reason stuck. Attempting to select a region won't work, whereas it does work in play. To a user not familiar with scrubbing it looks as if Audacity is broken.
Some random clicking around won't get the user out of the mode that they don't really know they are in.
A solution that would work for my use of scrubbing is that we go out of scrubbing as soon as we release the left mouse, (or release middle wheel - and middle click as well as ctrl-left-click activates scrubbing). That might defeat some of how you use scrubbing. It would though completely satisfy me that scrubbing was no longer 'a mode a user could be trapped in'.
Other alternative design features that could make this mode more OK:
- Change the green triangle on the cursor, so indicating that it is not recording, nor playing, but some other mode.
- Change the green triangle on the play button to indicate scrubbing-play (in both cases possibly a double headed triangle would work, indicating bi-directional). Combined these give a clue that clicking on transport buttons could help.
- Make selecting a region of more than 10 pixels take us out of scrubbing (selecting is probably one of the first things a stuck user would do)
- Make ESC take us out of scrubbing.
- Make clicking outside the track take us out of scrubbing.
These are not prescriptive, they are just possible approaches to the problem. Hopefully there is something in the suggestions that from your perspective does not damage scrubbing in any way, and that I and Steve would be happy with as solving the problems for the stuck-in-scrubbing users.
A clarification: I am not stating a requirement that selecting be possible whilst scrubbing! For example if scrubbing only happens with mouse down and dragging something, that is fine. Rather, with the currently implemented scrubbing mode, the mode-stuck user can think selection is broken. It is one of the first broken things they will find (and different to playing mode too). Making it throw them out of the mode and actually select is just one approach to get them out of the mode.
Peter 02Oct15: I have incorporated most of James' bulleted points in the proposal as points 1.1 through 1.4. The "Make selecting a region of more than 10 pixels ... " was not incorporated as you cannot make a selection while you are scrubbing. His suggestion that "we go out of scrubbing as soon as we release the left mouse" should be addressed later in Phase 2.1.
Peter's email of 29Sep15 (an excerpt)
Paul wrote: >If it is not then obvious that you can stop scrub play just as you stop other play, I don't know what would make it more obvious.
There are a couple of things that would make it "more obvious" and they are not mutually exclusive
1) You ask: "do we need some other interface for initiating scrub that you are less likely to do by accident?" My answer is "Oh yes we do" - and we on QA have been asking you for such for quite a while now.
As we already know from a posting on the Forum we have already had one poor benighted user who stumbled into scrubbing mode by mistake and then couldn't find his /her way out.
The problem is that the interface for moving into scrubbing is not GUI, i.e. it is not graphical, so there is nothing to see. What is needed is some form of graphical interface for moving into scrubbing.
This could either be an additional Tool in the Tools Toolbar for scrubbing which would place you in Scrubbing mode, Steve has suggested this approach I seem to recall. For me this is sub-optimal as I strongly think that scrubbing should be independent of the currently selected tool (just as Play and Record are). Also it would imply that to escape from scrubbing the user would need to choose and alternative tool from the Tools Toolbar. And fundamentally Scrubbing is not a tool, rather it is a form of playing. So I do not support this approach.
So what I would really like to see is an additional button on the Transport Toolbar for Scrub-play (scrubbing brush icon? maybe superimposed on a green Play icon?). This should make it clearer that the user should consider using the Stop button in the same Transport Toolbar to stop scrubbing. And btw you still need to fix the problem that the Transport Toolbar's Pause button and the Transport>Pause menu command are not grayed out as unavailable while scrubbing is active.
And if we do add such a button then we should probably also add a "Scrubbing" or "Scrub-Play" command to the Transport menu.
Apart from any other consideration I suspect that you will find more folk using scrubbing if there was a more visual method of entering scrub-play (and I'm sure that given all your work on scrubbing you would like to see that).
Steve's email of 29Sep - how other audio editors do scrubbing
In Protools, scrubbing is switched on by selecting the "Scrubbing" tool. It has a loudspeaker icon. Scrubbing is then performed by clicking on a track and dragging left (scrub forward) or right (scrub backward). Scrub play stops when the mouse button is released. When the scrubbing tool is selected, scrubbing may also be controlled with the "J" and "K" keys. "Jog wheel" control is supported when using an Avid control surface (nice, but expensive).
Soundforge uses a slightly different approach, but with similarities. In Soundforge, each audio track has its own window, and at the bottom of each window is a mini-transport toolbar. The mini-transport toolbar has an additional button as the final item, which is the "scrub play" button. When the scrub play button is pressed, scrubbing is performed by clicking on a track and dragging left (scrub forward) or right (scrub backward). Scrub play stops when the mouse button is released. When the scrubbing tool is selected, scrubbing may also be controlled with the "J" and "K" keys.
Sweep on Linux has another variation. Sweep has a scrub tool button. When selected, the cursor changes to the scrub icon when over the track panel. The cursor has the same design as the icon on the scrub tool button. Scrub play starts by clicking on a track. The Play button immediately changes to "button down". The difference between the Sweep behaviour and the other two examples above is that releasing the mouse button "lets go" and the track plays from the current position at normal speed. To stop playback, either click the stop button or press spacebar to pause, or the return key to stop. In each case the playback cursor stops at the current playback position. Sweep also has a "playback" menu in addition to the transport toolbar.
Ardour 3 has yet another variation. In Ardour 3 there are also many "advanced" options for scrubbing, but the basic operation is as follows:
Select the "scrub" tool to enter scrubbing mode. The cursor changes to a loudspeaker icon when over a track. Click and drag on the track to start scrub playback. A bar under the transport toolbar shows the current playback speed and direction. The direction and speed of playback may be controlled either by dragging in the required direction on the track, or by dragging the pointer in the bar under the transport toolbar, or from the keyboard. When dragging either on the track, or in the bar below the transport toolbar, playback stops when the mouse button is released.
Of all the methods described above, Ardour is my favourite. It is very clear what is happening, highly intuitive, provides immediate visual feedback and has "advanced" options for the more experienced user. Someone has clearly spent a lot of time and effort getting this right. It also supports "jog wheel" if the user has a compatible control surface. The only thing that I'm not so keen on, is that keyboard control is less easy than some of the alternatives.
Steve's email 02Oct15
As I understand it, the purpose of scrubbing is to enable the user to locate a specific point in time for editing. Perhaps I'm not using it the way it is intended, but for me, I find the exact spot that I want to edit, the playback cursor is exactly in the right place, but as soon as I stop scrub playback the cursor position is no longer there - it has jumped back to the previous playback position.
OK, so "I" know that there are some special key bindings available, but I've been using Audacity for over a decade.
Shouldn't the default behaviour be that when the user stops scrubbing, the playback cursor remains at its current position?
What should happen if there was a Selection when scrub mode was started?
Peter 02Oct15: To be addressed in Phase 2.2
Paul's email of 29Sep15
I am trying to assimilate the criticisms. I don't feel a consensus yet about what the better interface would be.
I) The non-immediacy of appearance change when you enter scrubbing is a problem but it is not the "stuck in a mode" problem. It is the easier thing to fix than the user interface redesign. II) James' "stuck in a mode" perception results from (a) selection change not working during scrub and (b) stopping when the play head catches up to the cursor.
How much of this perception results from each? I feel more concerned to fix (a) so that scrub play is no more restrictive of other actions, than is ordinary play. But (b) seems inherent to scrubbing, if the purpose of scrubbing is to find some precise place so that you can stop and put the cursor there. Unless you would rather have a play-head with "inertia" when you do not move the mouse. I don't know whether that will feel better in practice.
Meanwhile let's remember the scrubbing features we have. Binding each to some gesture crowds out the use of that gesture for some other purpose and that means tradeoffs in designing the user interface. Are the features all important enough to keep? Do we sacrifice any of them?
1) We have the scrolling version of display in which mouse position relative to the center affects the speed of scrubbing or the length of skips for seeking. You choose that variation or not when scrubbing begins. 2) We can switch between a stuttering seek and variable speed play during scrubbing. Variable speed simulates tape head scrubbing and can even go backwards. Seeking plays always at normal speed but with frequent skips. 3) If not seeking, then we can accelerate or decelerate the maximum speed during the scrub.
And to remember the user interface choices for the above.
1) ctrl-double-click in track panel to start scrolling scrub, ctrl-single-click for the ordinary scrub. 2) Hold left mouse button for seeking, release it for scrubbing. 3) Roll the mouse wheel.
Now let me remind you of tradeoffs that led to these decisions.
If mouse wheel is best for number 3, that is one reason why WE DO NOT WANT SCRUB TO BE A DRAG WITH A BUTTON DOWN. (An additional reason might be, we want to allow selection changes during scrub.) I think left button while rolling the wheel is just too awkward (two fingers at once, rather than just using the index finger for one or the other), and remember that when I used middle drag, Steve did not like rolling the wheel while also pressing it.
If seeking happens with left button down, that is all right because the speed determined by the wheel is irrelevant, so there is no conflict. But it is agreed that this interface is not yet the best. THIS IS ALSO WHY YOU CAN'T SELECT DURING SCRUBBING, part of what James says he means by "stuck in a mode," because certain familiar actions (selection changes) are excluded. Other alternatives were considered and rejected for their other problems. Making a shift or ctrl modifier key do it would interfere with the use of certain useful keyboard commands during scrubbing, like ctrl-m to drop a label or shift-a to stop and set cursor.
So please suggest an alternative to the left button. Perhaps we should require you to click and drag the green triangle in the time ruler (which is not part of the track panel proper). But then clicks and drags in the tracks would be free to change selection during scrubbing.
- James:. Yes, why not? What's wrong with doing that?
Another problem is that SCRUBBING WORKS ONLY IN SELECT TOOL. I am not pleased with that. It should work at least in multi tool too. Why not indeed in all tools? But the reason for this was choice 1. Scrubbing must be started by some mouse action. The choice of ctrl-click conflicts with things like time shifting in the multi tool.
What is a better choice for starting a scrub then? A click in the time ruler instead? There are already several mouse actions with meaning in the time ruler. Is anything available for scrubbing to use?
Finally Steve likes the idea of a button for a scrubbing mode. But James dislikes the multiplicity of tool buttons that we have already. He would rather have graphical handles to click on to access different functions. Also in another project, spectral selection, I deliberately avoided making a new button and a new mode. I still think it is better to accept scrub as "another kind of play" that is stopped in the same ways play is stopped, and you don't need finger pressure anywhere to continue in it. But during that play, as with ordinary play, certain other mouse actions and keyboard commands remain useful.
And don't forget ctrl-mousewheel to magnify -- I like it too that this remains available during scrubbing. But again it would be awkward if you had to keep the left button down.
Phase-1 is mostly about the 'mode' and not being stuck in it. Phase-2 suggestions here may improve discoverability and ease of use.
Possible Phase 2 changes.
- Scrubbing and seeking treated like play, loop-play, play-excluding are. The mode is initiated by toolbar button or bound keyboard shortcut.
- Ctrl, shift click etc become user bindable, in same way that keyboard shortcuts are. Possibly unbound by default.
- Scrubbing/Seeking could be started from a multi-button
- Radical change where:
- Play no longer resets cursor to start
- Scrubbing is done by dragging the play cursor. It only happens whilst the mouse button is down and dragging. Play cursor WILL lag behind desired drag target.
- Single scrub-seek mode rather than a scrub mode and a seek mode. In scrub-seek you scrub, if not too big a change, and seek if distance is above some threshold.
- Can scrub-seek when playing or when not. You could start playing at x1, then scrub-seek by dragging and continue playing at x1 after you stop dragging.
- Fixed-cursor scrolling-wave becomes an independent feature. One of the tick boxes in the multi-buttons.
The above Phase 2 ideas are NOT prescriptive and some are mutually exclusive.
If we are dragging something tangible rather than just moving the mouse the need for ctrl (or shift or double click) modifier goes away. Then the argument for restricting scrubbing to the selection-cursor mode, which Paul has done but is not keen on, goes away. The main remaining interface problem from Paul's perspective is then, I think, that he really wants to at least have the option to use the mouse wheel to set a speed. Since mouse-wheel and drag has been found to be troublesome when used together, that seems to require that scrubbing and seeking can operate with mouse up.
My picture is that the mouse x-y and the mouse wheel-roll are valuable resources. The tools-buttons are a clunky way of sharing that resource. Draggable tangible things are a better way of sharing. We need a way to share wheel-roll too. I don't have a precise proposal here. The outline idea is that mouse-wheel clicking on a slider binds the wheel-roll to that slider. ESC (or another mouse-wheel click) cancels the binding. So we could bind to the scrub-speed slider, the play-at-speed slider, or the pan slider, or volume slider, if we wanted to and vary those whilst playing.
If, as now, moving the mouse without any dragging can change the playback speed, then in my view we have a mode. It is possible to be stuck in that mode and not know what is going on. I thought we had agreed that we do not want scrubbing to be a new mode?
- Gale 08Oct15: I see in "2.1)" that "It should be possible to enter scrubbing mode via GUI." So if does not look to me it is agreed we do not want scrubbing to be a mode. There are benefits in reduced RSI in not having to drag and hold, but those benefits are IMO thrown away by the current implementation.
So I wrote:
"2.2) IF mouse position is used, THEN there should be some tangible widget to be dragged."
(note the CAPS). This has been changed to:
- Gale 08Oct15: James changed my "Should?" question to a statement that I could not understand out of its context. I did not want to be represented as making that statement.
"2.2) If mouse position is used to drag playback, there could be some tangible widget to be dragged. However this goes against 2.5) which seeks to make the drag less physical effort. Therefore an alternative could be to have the playhead widget not as the dragger but only as the up (off) /down (on) control to initiate scrubbing. If there is no drag widget then some option is needed to keep scrubbing alive when the playhead reaches the pointer."
Surely the alternative is then a mode? The only way it is better than the existing stuck-in-mode problem is that there is better signalling of the mode, which might give more clue to a new user as to how to turn it off again!
- Gale 08Oct15: I think being stuck by accident in scrub in much less issue than our showing how to scrub for those that want to use it, and IF we have a scrub mode, giving a clear indication that it is on.
- Peter 08Oct15: I think both are important - but I think if we fix the UI properly "showing how to scrub for those that want to use it" then that will of itself lead to fewer folk getting stuck in scrubbing without realizing it - and provide them with a clearer, more discoverable, method of escape. Those are the key challenges for this proposed redesign of the scrubbing UI: discoverability, usability, escapabilty.
The 2.5 interpretation seems to imply that moving the mouse without dragging can change the playback speed, and so if we accept 2.5 it seems we must have a mode.
- Gale 08Oct15: The standard single-click scrub (the only one the user is likely to discover) does not really change playback speed even when dragging. The scroll wheel usage to actually change playback speed doesn't even work when dragging. All drag does is to jump standard playback like holding down period or comma (optionally with SHIFT) do. Double-click scroll scrub does change playback speed when merely moving the mouse but that seems unintuitive to me.
The scroll wheel speed change is unexpected and undiscoverable, though *very* nice if you do discover it. It is more flexible than a dragged widget that changed playback speed would be. If we kept scroll wheel speed change, I would like to see at least a temporary mouse pointer when entering scrub mode that showed a scroll wheel with + above it and - below it.
Question: Is the Audacity audio engine good enough to really change playback speed when dragging a widget, so that playback speeded up and slowed down according to the speed of dragging the slider, as opposed to the on/off "stutter play" which drag does now?
I am rather strongly against modes of this kind and think avoiding having a mode is more important than the extra effort of holding a mouse button down to drag something tangible. IF we go for no mouse down THEN we need a very good way to signal how to get out of the mode. I'm -1 on the mode. If we did go for a mode a line from the mouse cursor to the widget that allows us to turn off the mode would help users get out of the mode.
- Gale 08Oct15: Again, users getting trapped in the mode is the lesser of scrub's issues. I thought no-one wanted a draggable widget except James, as per "2.1".
The draggable widget would be the most intuitive solution for users who don't use pro DAW's (i.e. I guess, most of our users). I would support a drag widget but we have lots of great, undiscoverable functionality in the current implementation. My ideal would be a simple, drag-hold, no-mode widget, but also add some other control on the play cursor to enable more advanced "mode" features that perhaps hide the drag widget. Perhaps that control might open a dropdown menu where I could choose a mode that does not stop when the playhead reaches the pointer, to replace and improve on Timeline Quick-Play.
All that said, should we be looking at what the big boy DAW's do? Is what we have now really like the big boys? What does their "jog scrub" do?
James' More detailed Proposal
Here is what I think my ideal would be:
- Scrubbing: We can drag the play cursor triangle (using left mouse) at any time, whether we are playing or not. If we are playing, Audacity continues playing when we release. I regard this as sufficiently modeless, in that users are only 'stuck-in-the-mode' whilst the mouse is down.
- Details: The cursor triangle changes to a green double-headed arrow, with a tooltip "Drag to Scrub" when we hover over it. The fact we can still drag whilst playing isn't very discoverable, but I think people who want to may actually try it. If the drag distance is small enough the speed changes (up to + or - x3). If the drag distance is too large we jump/stutter-play. There is some smoothing of the cursor speed, so the mouse movement can be a bit jerky but the playback is still smooth. That means we're dragging a 'ghost' cursor which the real cursor is trying to keep up with.
- Right-Only Scrubbing: We can drag the left edge of the selection. When it bumps the play cursor, this behaves very like dragging the cursor triangle does, except that the cursor only bumps left to right.
- Left-Only Scrubbing: We can drag the right edge of the selection. When it bumps the play cursor, this behaves very like dragging the cursor triangle does, except the cursor only moves right to left.
- Details: In loop-play mode we loop rather than bump, if dragging the right edge would bump. In normal play mode, if we catch up with a right selection edge that is being dragged, but not actually being moved, we don't exit play mode. That only happens when we release the mouse. We need to do this or Left-Only scrubbing would often unexpectedly end playback. There are also some tweaks needed to how the cursor behaves at other times. In current Audacity we reset the cursor to start of selection when making a new selection and at the end of play. I'd like to change that, if we support Left-Only Scrubbing.
- Vari-Speed Play: The variable speed slider now also supports negative speeds and is real time. Clicking on it binds the middle mouse wheel to it. ESC or clicking on some other slider, scroll bar or magnify-icon, ends that binding.
- Seeking: We can hop the play cursor to a new point, even whilst playing, by clicking the new position (on the timeline). If we're playing when we do this, we continue playing, even if we are now outside the selection region.
- Continuous Scrolling Wave: An option for 'play' in which the wave scrolls past continuously as it plays and (if not dragged) the cursor stays in a fixed position.
- Details: To be discoverable and avoid stuck-in-a-mode this requires multibuttons. To move the cursor and wave together we can scroll, as now. The cursor could also drift back to the screen position it was dragged from over time. When dragging the cursor there could/should be some gearing, so that as I drag the cursor right the wave also moves left so we can scrub more than a screen's width. More important at high zoom.
- Scrubbing is more discoverable, mainly due to the cursor change and tooltip.
- No stuck in mode. We exit scrubbing on mouse up. Mouse-wheel users could be stuck in a binding, but they will be out of it with just about the first thing they try.
- Functionality. What you can do as an expert user is at least as good as the existing scrubbing for an expert user. Partly this is because the features combine. You can now do scrubbing with/without play. Play (at vari-speed), scrubbing and seeking are now orthogonal features rather than mutually exclusive.
- RSI: The vari-speed mode is as RSI free as before.
- Gale 08Oct15:
- I was not able to understand what the right- and left-only scrubbing does. Perhaps this is because I missed AU14. This is not about left and right mouse button, is it?
- James: not about mouse buttons. Right-Only-Scrubbing is a bit like scrubbing without reverse play. Instead of dragging the cursor you drag the left edge of the selection (you need to be playing a selection). Only when the left edge catches up with the cursor is the cursor pushed right by it.
- Are we going to implement centered play cursor? If not, a possible benefit of scroll-scrub will be lost, and I don't see how we could grab the playhead if we are zoomed in such that the playhead is moving rapidly.
- James: Added. Grabbing the moving playhead at high zoom is indeed impractical. But in such a circumstance scrubbing would only make tiny differences to the play speed anyway.
- Dragging the playhead without modifying its appearance after pressing Play or SPACE may not be that discoverable. Can we also click and drag the green vertical line? Discoverability is further reduced if the playhead is moving fast. Are you sure we don't want a conventional drag bar whose handle is fairly static irrespective of the zoom level?
- James: As soon as you hover over the playhead it changes appearance, whether or not we are playing. (But not if recording). I was not going to have the green vertical line draggable, but could do. A conventional drag bar is more discoverable. If someone else makes a detailed proposal with a conventional drag bar, then I might be just as happy with their detailed proposal.
- Playhead dragging seems to imply Timeline Quick-Play "click to play" is removed, yes? Otherwise I can envisage lots of confusion when the smaller Timeline Quick-Play cursor appears when moving the mouse into the Timeline while playing.
- James: UI for Quick Play would need to change. Possibly 'click to play' is now the same thing as seeking.
- Does playback stop when I drag the playhead then stop dragging and still hold the mouse button down?
- James: If the playhead has caught up with dragging it stops moving (and you hear silence) until you release the mouse.
- I just noticed Steve's list of scrubbing in four DAW's. All seem to have something to press on to start scrub mode. I think Paul's concern about not being able to do anything else with the mouse while dragging have validity. This is why I suggested retaining advanced scrubbing "modes" (enabled by a clearly marked control with clear signage you are in that mode).
- James: IF mouse position is controlling scrubbing THEN it shouldn't be controlling something else at the same time! So if I understand the point correctly I disagree. There is no point having a non-dragging (i.e. non mouse down) way of having mouse position control scrubbing. Of course we can use keyboard scrubbing or mouse wheel vari-speed and have the mouse position free for other roles.
- Real Time Vari-Speed with the Play-at-Speed slider would be very nice on its own terms. Clicking it to bind scroll wheel to it seems like a hard to discover extra step, needed most times you start scrub, so not an improvement on that point. I would rather see some option on by default that entering any type of playback always bound scroll wheel to the play at speed slider, and stopping playback unbinds.
- James: Well, scroll wheel doing anything other than scroll is hard to discover too. You click on the varispeed slider to drag it, so if you start changing speed then you can continue doing so by mouse wheel. We should show that the slider has focus as an indication of the binding. We could also give it focus (and bind to it) when you click on play-at-speed.
- I was not able to understand what the right- and left-only scrubbing does. Perhaps this is because I missed AU14. This is not about left and right mouse button, is it?