Proposal Fish Eye

From Audacity Wiki
Revision as of 05:09, 2 July 2015 by Paul L (talk | contribs) (Added section for my prototype)
Jump to: navigation, search
Proposal pages help us get from feature requests into actual plans. This proposal page is about adding a way to see detail at the same time as seeing context.
Proposal pages are used on an ongoing basis by the Audacity development team and are open to edits from visitors to the wiki. They are a good way to get community feedback on a proposal.

  • Note: Proposals for Google Summer of Code projects are significantly different in structure, are submitted via Google's web app and may or may not have a corresponding proposal page.

Proposed Feature

Some method, or maybe a choice of several, for simultaneously looking at the same waveform at different resolutions.

Developer backing

  • James Crook
  • Paul L

Use Cases

Detailed work on a longer recording.


This example by Julian Dorn and Leon Schlechtriem shows an inset enlarged view that moves with the cursor:


Brett Park's focus plus context interface, similar in concept to the 'dock' from Mac OS X. The green area is enlarged, and moves as you move the scroll bar:



Julian and Leon's cutting magnifier is visible from 1:45 in their video [1]. The feature looks great in the demo, but some details need to be worked out for actual use. It is essential that the magnifier moves in steps that are a fraction of a pixel, otherwise it isn't actually giving any finer cutting precision than a normal cursor (even though you get to see exactly where you will cut).

  • One way to do this is to slow the gearing of the mouse movement so that big mouse movements make small magnifier movements. This, unfortunately, is against Apple's HCI guidelines. The speed of the cursor movement relative to mouse and 'warping' the cursor are disallowed. Also a choice is needed as to when to have a low gear and when high. An existing feature of mice is that fast movement has a multiplier and slow movement is made more precise. We'd need a lot more of that. We could also have low gearing when close to the center line of the audio and normal/high gearing when further away. A key to switch modes is getting rather close to a zoom key, and we are trying to avoid explicit mode switching.
  • Another way to do this is separate movement of the magnifier from movement of the cut line. In other words, the magnifier would only track the cursor when you dragged it. Then you could drag the cut line within the magnifier as a separate step.

Brett's focus plus context has a full video and explanation here: [2]. The design is demoed with relatively small difference in zoom between the fish eye and the actual wave. For precise cutting a difference of maybe 500x might be needed. In the demo the scroll bar is used to control the position of the focus, but the scrollbar is also needed to control the position of the view. The demo assumes that the full track fits easily on the screen at a normal resolution. That works for short tracks, but not for long ones, so some details of design need refinement.

The existing user experience is not as cumbersome as shown in the video, since scroll wheel on mouse can be used to zoom in and out quickly.

Current Status

The AU14 TrackPanel can show a copy of the same track at different zooms. Selections and play cursor are shown on both views. Navigation in one can cause scrolling of the other.


Paul's prototype

I am indebted to Brett Park's video for ideas -- but unable to find any pointers to source code, I made my own version, here:

Screenshot example -- a duplicated click track in Waveform dB and Spectrogram views, with an exponentially decaying envelope: Fisheye-1.jpg

I do not have a video demo yet. This summarizes the UI in the prototype, though I am not satisfied with all of the choices.

  • New keystrokes
    • Fisheye can be toggled on and off with the f key.
    • Fisheye can be played with the g key.
  • Modified commands
    • ctrl-mousewheel, zoom in, zoom out, zoom normal commands change fisheye magnification instead when pointer is in it.
    • Zooming with the pointer outside the fisheye zooms the background as usual, maintains the same ratio of fisheye to background magnification (except at extreme zoom levels), keeps the pixel width of the fisheye constant, and keeps the center of the fisheye at the same track time.
    • Scrubbing with fisheye enabled causes fisheye to stay centered at the play head.
  • Interface Preferences has a new section, illustrated in the screenshot.
    • style
      • a simple magnified inset, which causes portions before and after the fisheye to be hidden; or,
      • two choices for drawing transitional areas at reduced magnification so that no part of the audio is obscured.
    • width in pixels
    • default magnification ratio, which is restored by the zoom-normal (ctrl-2) command with pointer in the fisheye.
  • Mouse events, also summarized in Mouse Preferences dialog -- note that these work in any of the six cursor modes and are all dependent on pointer position.
    • Ctrl-mousewheel changes zoom, as already mentioned.
    • Ctrl-shift-mousewheel changes pixel width. (I no longer like this and want to make draggable handles of the fisheye edges.)
    • Right click in the fisheye, mouse movement, and right click again will un-pin and re-pin the fisheye, for coarse adjustment of position. Fisheye follows the mouse over an unmoving background.
    • Right-drag fine-adjusts the fisheye, moving it so that the foreground sound appears motionless while the "window" into it moves. This requires the background to move instead.
    • Right-double-click in the fisheye recenters it at the cursor.
    • Right-double-click-drag does something in this version, moving the window to follow the mouse but at a constrained speed, for very fine adjustment, but I now think it is redundant with scrubbing (which in fact I implemented later than fisheye, then I merged the projects). This should be removed.
  • Fisheye does not otherwise move; of course the main point of it is to make very fine selections inside it, so it must stay pinned as you mouse over it.

Code reorganization, and testing implications

Many parts of drawing code, and of mouse interaction code, needed reexamining. Many places assumed that ViewInfo::zoom describes a uniform zoom level that applies across the screen and replicated computations with it for converting between times and screen positions. I rewrote ViewInfo::zoom to be private, and instead cause all these places to call common functions that abstract the conversions. I believe I did a thorough job. I would like to commit at least this supporting reorganization early in 2.1.2 development even if there is no fisheye feature.

With the feature, a tester should verify correct display and mouse interaction in the presence of fisheye for:

  • Time ruler, including quick play and adjustment of play region ends
  • Time track, with its draggable control points
  • Label tracks, with text boxes and handles for moving and resizing
  • Wave tracks
    • Envelope points
    • Samples, with draw tool -- the fisheye can be zoomed enough to be editable while the background is not.
    • Zoom tool dragging
    • Time shift tool
    • Expanding and deleting cut lines (see the Tracks preferences)
    • Removing boundaries between touching clips
    • Sync lock tiles
    • Red clipping lines
    • Making and adjusting time and frequency selections
    • Yellow snapping lines when making selections

Did Brett Park also do them all? I can't tell from the video, though I do see that he got drawing of the time ruler done right.

Comparisons with Brett Park's version

In the video, Park's version does not appear to have transitions before and after the fisheye, nor to obscure any part of the waveform -- from which I conclude that part of the background must move when fisheye is toggled on or off, to make room for the portion with increased magnification. If that is so, I think it is undesirable and that my version improves on the idea.

Future ideas

  • Implement a toggle command that "maximizes" the fisheye -- that is, fills the entire screen at the fisheye magnification, while the sound at the fisheye center remains unmoved -- and "minimizes" again to the normal state. Perhaps the f key could cycle among more than two states.