Difference between revisions of "Proposal Fish Eye"

From Audacity Wiki
Jump to: navigation, search
(removed my backing for 2.2.0)
(Developer / QA backing: adding my support)
Line 10: Line 10:
* James Crook  
* James Crook  
* Paul L
* Paul L
* Peter Sampson
== Use Cases ==
== Use Cases ==

Revision as of 11:53, 12 December 2017

Proposal pages help us get from feature requests into actual plans. This proposal page is about adding a way to see detail at the same time as seeing context.
Proposal pages are used on an ongoing basis by the Audacity development team and are open to edits from visitors to the wiki. They are a good way to get community feedback on a proposal.

  • Note: Proposals for Google Summer of Code projects are significantly different in structure, are submitted via Google's web app and may or may not have a corresponding proposal page.

Proposed Feature

Some method, or maybe a choice of several, for simultaneously looking at the same waveform at different resolutions.

Developer / QA backing

  • James Crook
  • Paul L
  • Peter Sampson

Use Cases

Detailed work on a longer recording.


This example by Julian Dorn and Leon Schlechtriem shows an inset enlarged view that moves with the cursor:


Brett Park's focus plus context interface, similar in concept to the 'dock' from Mac OS X. The green area is enlarged, and moves as you move the scrollbar:



Julian and Leon's cutting magnifier is visible from 1:45 in their video [1]. The feature looks great in the demo, but some details need to be worked out for actual use. It is essential that the magnifier moves in steps that are a fraction of a pixel, otherwise it isn't actually giving any finer cutting precision than a normal cursor (even though you get to see exactly where you will cut).

  • One way to do this is to slow the gearing of the mouse movement so that big mouse movements make small magnifier movements. This, unfortunately, is against Apple's HCI guidelines. The speed of the cursor movement relative to mouse and 'warping' the cursor are disallowed. Also a choice is needed as to when to have a low gear and when high. An existing feature of mice is that fast movement has a multiplier and slow movement is made more precise. We'd need a lot more of that. We could also have low gearing when close to the center line of the audio and normal/high gearing when further away. A key to switch modes is getting rather close to a zoom key, and we are trying to avoid explicit mode switching.
  • Another way to do this is separate movement of the magnifier from movement of the cut line. In other words, the magnifier would only track the cursor when you dragged it. Then you could drag the cut line within the magnifier as a separate step.

Brett's focus plus context has a full video and explanation here: [2]. The design is demoed with relatively small difference in zoom between the fish eye and the actual wave. For precise cutting a difference of maybe 500x might be needed. In the demo the scrollbar is used to control the position of the focus, but the scrollbar is also needed to control the position of the view. The demo assumes that the full track fits easily on the screen at a normal resolution. That works for short tracks, but not for long ones, so some details of design need refinement.

The existing user experience is not as cumbersome as shown in the video, since scroll wheel on mouse can be used to zoom in and out quickly.

Current Status

The AU14 TrackPanel can show a copy of the same track at different zooms. Selections and play cursor are shown on both views. Navigation in one can cause scrolling of the other.


Paul's prototype

I am indebted to Brett Park's video for ideas -- but unable to find any pointers to source code, I made my own version, here: https://github.com/Paul-Licameli/audacity/commit/92f31c681a523fd8a3dae2f0f213e7167faaa671

Screenshot example -- a duplicated click track in Waveform dB and Spectrogram views, with an exponentially decaying envelope: Fisheye-1.jpg

I do not have a video demo yet. This summarizes the UI in the prototype, though I am not satisfied with all of the choices.

  • New keystrokes
    • Fisheye can be toggled on and off with the f key.
    • Fisheye can be played with the g key.
  • Modified commands
    • ctrl-mousewheel, zoom in, zoom out, zoom normal commands change fisheye magnification instead when pointer is in it.
    • Zooming with the pointer outside the fisheye zooms the background as usual, maintains the same ratio of fisheye to background magnification (except at extreme zoom levels), keeps the pixel width of the fisheye constant, and keeps the center of the fisheye at the same track time.
    • Scrubbing with fisheye enabled causes fisheye to stay centered at the play head.
  • Interface Preferences has a new section, illustrated in the screenshot.
    • style
      • a simple magnified inset, which causes portions before and after the fisheye to be hidden; or,
      • two choices for drawing transitional areas at reduced magnification so that no part of the audio is obscured.
    • width in pixels
    • default magnification ratio, which is restored by the zoom-normal (ctrl-2) command with pointer in the fisheye.
  • Mouse events, also summarized in Mouse Preferences dialog -- note that these work in any of the six cursor modes and are all dependent on pointer position.
    • Ctrl-mousewheel changes zoom, as already mentioned.
    • Ctrl-shift-mousewheel changes pixel width. (I no longer like this and want to make draggable handles of the fisheye edges.)
    • Right click in the fisheye, mouse movement, and right click again will un-pin and re-pin the fisheye, for coarse adjustment of position. Fisheye follows the mouse over an unmoving background.
    • Right-drag fine-adjusts the fisheye, moving it so that the foreground sound appears motionless while the "window" into it moves. This requires the background to move instead.
    • Right-double-click in the fisheye recenters it at the cursor.
    • Right-double-click-drag does something in this version, moving the window to follow the mouse but at a constrained speed, for very fine adjustment, but I now think it is redundant with scrubbing (which in fact I implemented later than fisheye, then I merged the projects). This should be removed.
  • Fisheye does not otherwise move; of course the main point of it is to make very fine selections inside it, so it must stay pinned as you mouse over it.

Code reorganization, and testing implications

Many parts of drawing code, and of mouse interaction code, needed reexamining. Many places assumed that ViewInfo::zoom describes a uniform zoom level that applies across the screen and replicated computations with it for converting between times and screen positions. I rewrote ViewInfo::zoom to be private, and instead cause all these places to call common functions that abstract the conversions. I believe I did a thorough job. I would like to commit at least this supporting reorganization early in 2.1.2 development even if there is no fisheye feature.

With the feature, a tester should verify correct display and mouse interaction in the presence of fisheye for:

  • Time ruler, including quick play and adjustment of play region ends
  • Time track, with its draggable control points
  • Label tracks, with text boxes and handles for moving and resizing
  • Wave tracks
    • Envelope points
    • Samples, with draw tool -- the fisheye can be zoomed enough to be editable while the background is not.
    • Zoom tool dragging
    • Time shift tool
    • Expanding and deleting cut lines (see the Tracks preferences)
    • Removing boundaries between touching clips
    • Sync lock tiles
    • Red clipping lines
    • Making and adjusting time and frequency selections
    • Yellow snapping lines when making selections

Did Brett Park also do them all? I can't tell from the video, though I do see that he got drawing of the time ruler done right.

Comparisons with Brett Park's version

In the video, Park's version does not appear to have transitions before and after the fisheye, nor to obscure any part of the waveform -- from which I conclude that part of the background must move when fisheye is toggled on or off, to make room for the portion with increased magnification. If that is so, I think it is undesirable and that my version improves on the idea.

Future ideas

  • Implement a toggle command that "maximizes" the fisheye -- that is, fills the entire screen at the fisheye magnification, while the sound at the fisheye center remains unmoved -- and "minimizes" again to the normal state. Perhaps the f key could cycle among more than two states.

Paul's proposal for 2.2.0

Since I wrote the above, I did submit the changes to internals of drawing and mouse interaction code, in 2.1.1. The Fisheye (or "Magnifier" as I would now rather call it) has waited a few releases as priorities were elsewhere.

The experience of implementing better scrubbing in 2.1.3 suggests that an alternative interface for fisheye could be based mostly on interactions with the time ruler instead of the track panel. This allows fisheye control to be independent of the choice of editing tool and the need to avoid conflicting meanings assigned to clicks and drags in the track panel by means of unnatural right mouse drags or double-click drags.

Fisheye adjustment can be made especially convenient for users of the Macbook touchpad's gestures, some of which map to control wheel actions with modifier keys for other platforms.

I now intend best effort for 2.2.0. I give an outline of details, in no particular order, and subject to change, which could serve as a starting point for documentation in the manual.

  • When the magnifier is visible, pinch-and-spread (or Ctrl+mousewheel) with the pointer in it changes its width.
    • Spread gesture in the ruler, with magnifier not visible, could make it visible and center it at the pointer?
    • Pinching to zero width could hide the magnifier?
  • As in the previous proposal: pinch and spread in the track area within the magnifier, and other zooming commands and keystrokes, cause a change in its magnification.
    • Outside the magnifier, do as before, now preserving the ratio of magnification inside and outside the fisheye except at extreme zoom-in.
    • Individual samples may be drawn inside the magnifier but not outside. Draw tool may work inside the magnifier but not outside.
  • Two-fingered horizontal swipe (or Shift+mousewheel) with the pointer in the tracks causes the magnifier to remain fixed on the screen while the waveform or other data move across it. (This was not in the original implementation.)
  • Two-fingered horizontal swipe in the ruler makes coarse adjustment of the magnifier position. That is, the unmagnified waves remain fixed while the magnifier slides across.
  • Two-fingered vertical swipe (or unmodified mousehweel) in the ruler makes fine adjustment of the magnifier. That is, the magnified wave remains fixed, but the edges of the magnifier move, and the unmagnified background must move.
  • Double-click in the ruler moves the center of the magnifier to the pointer.
  • An icon appears on the ruler above the center of the magnified area.
    • Should this icon simply be the pin, which might remain on screen even when not playing or recording? Could this give you a means to place the pin somewhere else than mid-screen if you want?
    • Left click on the icon toggles "zoomed" state, in which the magnified portion fills the screen. The icon does not move.
    • Dragging the icon causes the "very-fine" adjustment, in which the magnifier moves as for coarse adjustment, but lags the pointer movement.
      • I have implemented in a branch, and rather like, an exponential slowing. Speed of the movement of the magnifier decays as it approaches the pointer.
    • Right click on it opens a context menu. Contents to be determined, but at least this might include:
      • "Select", to select exactly the magnified part.
      • Hide.
      • Zoom toggle.
      • Checkmarks for whatever boolean preferences we decide to have.
      • Center.
  • Magnifier stays centered at the play/record head (or acts as the pin?), including for scrubbing, but excepting when the selection is a region and lies entirely within the magnifier.
    • This behavior might be controlled by a preference.
  • Appropriate status bar messages when the pointer is in the ruler.
  • A toggle in the context menu for the ruler to show and hide the magnifier.
  • A new toggle button, next to the pinned playhead button, showing an icon, perhaps a magnifying glass. This shows and hides the magnifier.
    • This button might not be necessary, if the magnifier could be shown and hidden just by pinch and spread or by context menu items, but it would greatly aid discoverability.
  • All means of moving the magnifier continuously might cause playback of the moving center. This might be scrubbing by another means. This might be a preference.
    • Would this make the scrub bar unnecessary? Perhaps not, because the scrub bar still gives you the means to seek, or to play only up to a chosen point, which is easier to do with a drag of the pointer than by scroll wheel or swipte.
    • Might we also make the two fingered swipe (or Shift+mousehweel) in the waves also cause a scrub? That could stand as an independent development without the magnifier.