Proposal TrackPanel Evolution
|Proposal pages help us get from feature requests into actual plans. This proposal page is about a radical change to the TrackPanel.|
Proposal pages are used on an ongoing basis by the Audacity development team and are open to edits from visitors to the wiki. They are a good way to get community feedback on a proposal.
- Note: Proposals for Google Summer of Code projects are significantly different in structure, are submitted via Google's web app and may or may not have a corresponding proposal page.
This proposal is a bit different to most proposals.
The intent is to find an evolutionary path to a major reworking of the TrackPanel. The proposal suggests a way to make changes safely and incrementally, without taking a step backwards in stability. Refactoring TrackPanel has been acting as a log-jam to GUI development and needs to be addressed. Some of the possible new behaviors that work on track panel will facilitate are shown in the experiments.
The approach is to make a completely new TrackPanel as a plug-in, and do all the experimenting in that. This new TrackPanel is an optional extra that can run alongside the existing one. Initially it will be used for features that the existing TrackPanel does not offer. Eventually it will have enough functionality that the 'old' TrackPanel can be retired. On the way, some important features will get folded into the old TrackPanel so that they are there even for those who choose not to use the optional plug-in.
- James Crook
- Vaughan Johnson
- Leland Lucius
Some Changes we Want
Lots and lots of upgrades to tracks, that would overload the current track panel.
- API: for registering a track type. This will allow new track types to arrive as plug-ins.
- Ruler tracks: enhanced version, and making it a track allows us to have more than one ruler/scale in a project.
- MIDI tracks: based initially on Roger's code, recast as a plug-in.
- Diff track: see Proposal Audio Diff, initially just the display side of this.
- Source separation Track: see Proposal Source Separation, initially just the display side of this and a Lo-pass/Hi-pass filter pair.
- Automation Tracks: used for example to control echo delay or a panning that varies with time.
- Computed Data: show a graph that is calculated in some way, rather than being stored.
- Multiple channels and modes: on the same track. For example show spectrum view and waveform view of the same mono audio in the one track - or show tracks for surround sound.
- Overlays: ability to show labels on top of a waveform, or add a draggable 'threshold' marker to a track.
- Themability: see Proposal New Skins.
- Discontinuous Selections: for Proposal Select Then Act.
- Fixed Cursor, Moving Wave: a mode, mostly for playback, where the cursor stays fixed, that can make editing easier.
- Antigrain/Cairo Rendering: sub-pixel resolution and antialiasing waveform repaint - more modern look.
Screenshots of Experiments
The four rows in the image below shows four states of the experimental new toolbar. The pause, stop, fast-forward, rewind buttons have all gone. Instead the record and play buttons are toggles.
The new buttons on the end are for preferences and information. 'Information' covers both help and general diagnostic information, such as size of file loaded, free disk space, amount of history information, so both experienced and inexperienced users are still likely to want this button.
The four versions of the toolbar shown below show:
- No recording or playing.
- Playing and not recording.
- Recording and not playing.
- Recording with play-through.
I'm regarding play-through as an 'advanced feature', so to get the fourth behaviour you have to ctrl-click to add recording to play (or to add play to recording). If you just click, the other button pops up, and you have to click again from a both buttons up state to record or play, so it behaves like stop until you've stopped the other action.
Playing is loop-play by default. I've not yet added an option to make it stop when it reaches the end of the play region, and I probably won't.
Implementation detail: The buttons are drawn using anti-grain and the icons on them are from a font (WebDings - which is free to download). This makes them perfectly scalable. (Update) I've reverted to using bitmaps made in 'The GIMP' (with WebDing font on them) for now. It allowed me to more finely customise the graphics, and saved me from having to ship the WebDing font with the plug-in. There is scaling down of the bitmaps on display, so at some point I'll build in bitmaps that are twice the size actually needed, so that users can once again choose to have a bigger version than shown here.
All six small mode 'tool' buttons (selection, envelope, draw, zoom, shift and multi-tool) are gone. We can do zooming with the ruler. We are always in 'multi-tool' mode, and the action depends on what item we are hovering over.
In the image below:
- Dragging on the black triangle moves it left or right. Here it is on 670. It is the zoom center.
- Dragging on the grey ruler zooms, by stretching/compressing the ruler. The black triangle stays where it is. Here the number 670 stays where it is as the scale expands or compresses.
- Dragging on the white part of the ruler pans the ruler left or right.
Below shows three stages of zooming in in the experimental new ruler.
- The zoom is continuous as we drag on the grey area. The cursor is over the '900' in all three shots. I clicked there and dragged right.
- We can pan the ruler by dragging on the white area - so we get zoom and pan on the one ruler widget.
- As can be seen from the images, the tick marks increase in height, and new labels and tick marks fade in gradually. There are also smart abbreviations on the ruler like '1.0K'. The code knows not to use 1.0K for 1,000 if 1,050 appears on the ruler.
The continuous rescaling of the ruler is possible because the ruler marks are being drawn to fractional pixel positions by antigrain. Overall speed of drawing for the whole screen is at least as fast as current Audacity, and is fast enough to support fixed cursor, moving wave. However experiments make it clear that at high zooms fixed-cursor moving-wave the wave looks fine, like an oscilloscope display, but the ruler is moving too fast to keep track of, and you can get the wagon-wheel effect where it appears to be moving the wrong way. So in the fast moving case we should either fade out the ruler and fade in a time-counter, or optically stabilise the ruler by choosing our time intervals. The dedicated counter expected to be the better option.
The smooth zooming of the ruler means we could drop the zoom-in and zoom-out buttons. This is more precise and flexible. I've put a magnify glass on the top right of the ruler, and a left-right arrow over the bottom right (not shown here), to make these actions discoverable. When I've added tooltips to those it will be enough of a visual cue as to what the ruler does. The triangle is the centre of the zoom and can be moved elsewhere by dragging. It needs a tooltip too.
One loss in the new ruler is that we can't use poor-man's scrubbing in the ruler, as the in-ruler dragging is being used for zoom and pan. However, other experiments (unexpectedly) have given a far superior way of 'poor man scrubbing' to replace it. Making the selection region boundaries support real time adjustment has been found to be a more elegant solution. By dragging the start of a selection right as audio is playing a user can 'chase' the play cursor, forcing it to move on.
Anyway, here are screenshots of the zoomable ruler, showing it zooming.
Drag on grey part of ruler to drag-zoom continuously in or out.
Black triangle (movable) is zoom center.
Here we see four tracks that have been overlaid. The envelope/automation track is deliberately translucent, so that the waveform shows through. We can overlay as many tracks as we want, so we can have multiple automation tracks on the same track.
Hit-testing works from the top layer down finding the first hit. This means that we are automatically in 'multi-tool-mode'. We can be non-modal in editing the tracks. What action we do depends on what object we are over, not on what tool we have selected.
These tracks all pan and zoom smoothly with the ruler. The wavetrack was the hardest to do this for because there is a switch over between methods for drawing when there are more/fewer pixel positions than samples being shown. Getting that to join up nicely required careful attention to exactly how lines are drawn.
Overlay tacks. Any track can be shown over another
Here we have an updated LabelTrack. The main addition is more styles for the labels. Using the track as an overlay we can superimpose labels over the waveform, if we want. These are very similar to the existing labels in Audacity in how they respond to panning, dragging and zooming, except it is smooth and continuous now as the ruler is dragged/panned. Some of the lines are coming out on top of the labels, and I need to change the order of drawing to fix this.
New styles of Labels
This is a track for musical notation. It's at a very early stage. You can drag notes about on it and it scrolls left and right, but that is about it. The intention is that it can export/import in MuseScore format, to interact with that very cool software. This track pans but does not zoom. It is on a different time-base/ruler to the other tracks.
Music score Track (you can drag notes up/down but little else so far).
Should be started this year (2011) (yes, first experiments were done then, then a big gap until December 2013). Time to complete years, if ever, since there are research projects in there. However, from start of coding to something that is actually useful to some people should be less than four months elapsed time. It's possible to break the log-jam that TrackPanel is causing in about one year of coding.
Steps to Get There
- Rather than risk hard-won stability of the existing code, create a new plug-in with an alternative TrackPanel. Initially it will only be invoked for special purposes, such as MIDI or diff.
- Progress: Screenshots from the alternative TrackPanel are shown above.
- General plug-in preparation, removing cleanspeech (or making it a plug-in), moving mixer panel to a plug-in, adding hooks into preferences to show available dlls and the plug-ins they contain. Reorganizing menu so that menu items are registered with a central menu manager, rather than all being coded in one place.
- Relevant cleanspeech features totally merged into Audacity. 'Batch Chains' and 'Truncate Silence' which come from cleanspeech are mainstream features now. The alternative mode with different menu items is gone. Simplified versions of menus can be created using the menu translation trick, if so desired.
- Preferences with EXPERIMENTAL_MODULE_PREFS now shows all plug-ins found, and allows you to enable/disable them, detects plug-ins that crash and doesn't load them again.
- Menu 'Registrar' created in plug-in and works fine there. Not used in Audacity yet.
- Use lightweight widgets and create lightweight sizers. Current TrackPanel layout is tangled due to using constants for positioning and putting too much of the painting/hit-testing code into one class - rather than a class for each widget.
- Progress: The implementation of a lightweight widget track class (that draws the background, if it is not in overlay mode) and lightweight sizers has now been done. These work similarly to how wxWidgets sizers work, in that there is a fixed minimum size plus an optional proportionality size.
- The modified display will need more complex caching and partial recalculation to avoid full repaints. The code to do this will use a dependency graph, and step through that graph to work out what to update.
- Progress: Implemented. Used in the example screenshots.
- Classes will be more fragmented.
- We will have caching classes, mostly using a random-drop cache which behaves well even in worst case situations.
- We will have separate classes for keeping track of data from those used to draw it. This will allow us to attach different rendering code to the same data without having two copies of the data.
- Code written rather like an extension of wxWidgets. I.e. the code for positioning tracks and info panels and so on is using a sizer based mechanism. It is oblivious of what is in the tracks and panels it positions.
- Progress: Implemented, and using dependency graph. This is already more flexible than wxWidgets sizers as it supports 'overlays' and 'choice' as sizer options. It also does fewer repaints because it is using dependencies (flow graph) with lazy evaluation rather than windows messages to decide when to.
- More information in effects.
- Rather than custom code for syncing labels, have a central mechanism that describes any time-warping by an effect. The label syncing is one client of this code.
These may or may not happen, they are just current thinking
- Dragging and stretching, of the time track, the labels, the clips, will be put 'on the same footing'. The intent is to use closely related code, and closely related visuals. Stretching a clip will offer you the option to use the change-tempo effect to lock-in the change.
- Causing clips to overlap will be corrected by Audacity automatically adding an extra track to take the overlapped clip.
- Sync-lock grouped tracks will be shown with no space between them.
Create a branch of Audacity.
- Initially an SVN branch, because main repository is SVN and we're better off all on just one type of version control.
- Plan to later move to a Mercurial branch, because eventually we can host that on Google code. (Mercurial is supported under Windows by TortoiseHg, whereas Git is aimed at Linux, so Hg in preference to Git).
- OR don't branch, and just be very careful that the new code is all in a plug-in 'mod-track-panel' with minimal safe #ifdeffed change to the core of Audacity to allow it to hook in.
- OR develop the new code in parallel, and occasionally fold chosen improvements back into Audacity.
Summary of Current State
updated June 2014
- SVN now has mod-null, mod-nyq-bench, mod-script-pipe and mod-track-panel projects under the lib-src subdirectory. The ability to patch into Audacity from a plug-in with minimal disturbance to the core Audacity code is established. However mod-track-panel isn't using any of the other new code. It's still the 'old' tracks there. It's just proving the feasibility of replacement track panel plug in.
- Progress (v1): Experiments have been done, but not checked in, with using antigrain to allow continuous zoom of the ruler rather than only zooming in steps.
- Progress (v2): The v1 version of this established that rulers drawn this way were fast and looked great. v1 has been completely superseded now by v2, and v2 is shown in the screenshots.
- Progress (v1): Experiments with a more generic drawing framework have highlighted a problem that comes with generic code. The generic code uses void* pointers, to avoid a lot of repetitive code - with the downside that the debugger then does not know what it is dealing with. That's too important an issue for maintainability. The critical issue is in an application created stack object. Whilst it leads to small efficiency loss, a stack of pointers to objects rather than a stack of objects could be used instead. The speed loss is minuscule if we make leaf routines process lists rather than individual items.
- Progress (v2): The updated experiments are using a different solution. All the objects are derived from Track, and we use the actual program stack rather than a custom homebrew drawing stack. The debugger plays well with this, and shows us the actual derived classes. An additional trick now being used is to use a python module, cogapp, which ia a code generator, to generate strongly-typed code that is rather repetitive that would otherwise need void pointers. This works better than using templates as the code is much more readable.