View source for GSoc 2008 - QuickLoad Progress

[[Image:GSoC.png|left|Summer of Code 2008 logo]]The Quickload Project progress can be reviewed and discussed here. There is also other discussion about the project going on in the audacity devel mailing list. <br clear="all" />

==Abstract==
* [http://code.google.com/soc/2008/audacity/appinfo.html?csaid=DEB0755F08511863 On Demand/Level of Detail File Loading]

==Progress==
===BlockFile Coloration Experiment (PRE-GSoC)===
[[Image:OnDemandLoadingTopblockcolor.jpg|thumb|Block-File Coloration Experiment]]

The drawing of the waveform when zoomed out to a summary (peak+rms) level of detail will draw whatever summary data has been completed already.  This means that the trackartist will need to be able to discern between blockfiles - currently it does not have access to that information within the drawing routine.  For Michael to familiarize himself with the drawing method, an experiment was done to draw each blockfile with a random color.  

The patch can be downloaded at http://eamusic.dartmouth.edu/~michael/temp/mchinenblockfilecoloring.patch


===ODPCMAliasBlockFile (5/28/08)===
[[Image:ODTopLevel.png|thumb|Partial ODPCMAliasBlockFile and Regular BlockFile Waveform, top level]]
A new class called ODPCMAliasBlockFile was added to the project.  It is a subclass of PCMAliasBlockFile that does not compute and write a summary file to the disk by default.  In the future this computing and writing will be done on a background thread, and then on-demand (changing the order of computation, whenever the user clicks on the waveform.)  

It is save and open-safe, because it implements its own XML writing and reading.  It supports playback and sample-level viewing.  Effects can also be applied to it; doing so will replace the instance with a different (non-OD) blockFile with a fully computed and viewable summary.  The picture above shows a waveform with both OD and non OD blockfiles.  

With this patch a different method of drawing has been implemented so that TrackArtists can be aware of which and what number of blockFile they are drawing, and whether or not summary data is available.  When drawing a BlockFile without summary data, diagonal stripes are displayed.  Dark and light are alternated with every blockFile to show the user where the load edges will be.

There are currently several "hacks" inside of ODPCMAliasBlockFile.  One of the most contestable is the way it returns min max rms data.  It was noticed that some effects, such as Amplify use the summary data of the blockfile, which OD blockFile does not have.  



The patch progress against CVS_HEAD from various dates can be seen at.  Because Michael does not have official CVS permission yet, the new files are also included in the zip.

====ODPatch 1 (5/29/08) ====
Class created, opens wavs instantly, breaks opening and saving
http://eamusic.dartmouth.edu/~michael/temp/ODPatch1AndNewFiles.zip

====ODPatch 2 (6/1/08) ====
Opening and Saving made safe
http://eamusic.dartmouth.edu/~michael/temp/ODPatch2AndNewFiles.zip

====ODPatch 3 (6/3/08)  ====
TrackArtist Drawing methods
http://eamusic.dartmouth.edu/~michael/temp/ODpatch3AndNewFiles.zip


===ODManager/ODTaskThread/ODTask/ODWaveTrackTaskQueue/ODComputeSummaryTask (6/10/08)===
These tasks effectively implement a thread-safe method for scheduling tasks for WaveTracks.  All of the references to WaveTracks and blockfiles are stored inside the OD classes, and not the other way around.  The only OD related things inserted into the original code are Mutexes and critical sections.  The idea is that the OD code will be a layer on top of the orignal code, a layer that can be peeled off easily if need be, because the references all point in one direction.

There are probably still some areas that need to have mutexes/critical sections that need handling.  Any place that deletes or modifies sequences, waveclips, or wavetracks needs to be handled carefully, and all I've done so far is handle the obvious cases.  Of course, once the file loads, this isn't a problem, because the Tasks and threads all are deleted.

===WaveCache (6/21/08)===
The WaveCache class in WaveClip has received a significant overhaul.  Before the only option for updating a portion of the screen was to delete the WaveCache, thus causing all pixels to be recomputed and loaded, which meant looking into each blockfile's actual file.  This is especially horrible for large files, which have many blockfiles on the screen at far zoom levels.  ODComputeSummaryTask is constantly updating blockfiles and requesting the screen to update, which is why the WaveCache has made a sudden appearance into the quickload project.  

I've added support for invalidating regions based on samples.  Before, where the programmer needed to delete the wave cache after a small blockfile update, he needs only now to add an InvalidRegion to the WaveCache.  The WaveCache is smart about these regions, if you give it two overlapping regions, it takes the union and stores it.  Then, the next time GetWaveDisplay in WaveClip is called, the WaveCache goes and fetches/computes the data needed for just that region, and preserves the rest of the cache for use on painting.

===On Demand (7/16/08)===
The on-demand function, which means that on top of the background computing of summary data, the user can dictate the order and location of the process by clicking on the not-yet loaded section of the waveform.   The exact behavior of this process has not been decided yet, since there is not a lot of prior examples that would otherwise provide user expectations. 

----

===Profiler (8/12/08)===
A very simple profiling class called Profiler, in Profiler.h/cpp has been added to the project.  I wrote and used it to test the speed of OD vs non OD.  

The main benefits of this class are: 
*can use it with writen code to profile a task that a normal function/method profiler might not be able to profile accurately.  (For example, a task that starts at the drag-drop of 5 waves into audacity and ends when their summaries are compelte.
*exists within just one cpp/h pair
*can automatically generate persistent text files.

Here are the results for OD vs non OD summarizing of the same files, showing that OD doesn't make things slower, and makes things faster for large files:
<pre>
Audacity Profiler Run, Ended at Tue Aug 12 19:25:45 2008
****************************************
Task: ON Demand Load 2hr stereo wav
(begins at line 207 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 4
Total run time (seconds): 259.310000
Average run time (seconds): 64.827500

****************************************

Audacity Profiler Run, Ended at Tue Aug 12 19:38:53 2008
****************************************
Task: Pre-GSOC (PCMAliasBlockFile) Load 2hr stereo wav
(begins at line 208 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 4
Total run time (seconds): 284.470000
Average run time (seconds): 71.117500

****************************************

Audacity Profiler Run, Ended at Tue Aug 12 21:37:31 2008
****************************************
Task: On Demand Drag and Drop 5 80 mb files into audacity, 5 wavs per task
(begins at line 214 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 4
Total run time (seconds): 99.300000
Average run time (seconds): 24.825000

****************************************

Audacity Profiler Run, Ended at Tue Aug 12 19:47:35 2008
****************************************
Task: Pre-GSOC (PCMAliasBlockFile) Drag and Drop 5 80 mb files into audacity (average per file)
(begins at line 208 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 20
Total run time (seconds): 90.940000
Average run time (seconds): 4.547000*5=22.735

****************************************

Audacity Profiler Run, Ended at Tue Aug 12 21:40:28 2008
****************************************
Task: On Demand open an 80 mb wav stereo file
(begins at line 215 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 4
Total run time (seconds): 17.340000
Average run time (seconds): 4.335000

****************************************

Audacity Profiler Run, Ended at Tue Aug 12 19:50:46 2008
****************************************
Task: Pre-GSOC (PCMAliasBlockFile) open an 80 mb wav stereo file
(begins at line 210 in /Users/apple/cvscheckouts/audacity/mac/../src/import/ImportPCM.cpp)

Number of times run: 4
Total run time (seconds): 17.190000
Average run time (seconds): 4.297500

****************************************

</pre>

----

===OD Guidlines and support for non-wav formats (8/15/08)===
One of the reasons the Quickload project was approved was because the OD framework will provide a method in which other tasks, such as loading non-wav formats, processing effects, and exporting, can be made multithreaded.  The current implementation of the OD framework is written generally so that this is possible, which means that future implementations of OD tasks will be done writing a minimum of code.  Taking advantage of polymorphism, this kind of thing should get easier and easier as more tasks are made to support OD.

Currently there exists a skeleton framework that assists in decoding non-wav files on the background in an on-demand fashion.  The classes ODDecodeBlockFile, ODFileDecoder, and ODDecodeTask are where the basic functionality is being implemented.  

To support a new type of file for OD, here is a set of steps that need to be taken for the example of mp3:
*Subclass ODDecodeTask and ODFileDecoder (for example, create subclass ODDecodeMP3Task and ODMP3Decoder)
**The ODMP3Decoder class will need to implement Decode(), which fills a buffer of arbitrary length, starting from an arbitrary sample number and ReadHeader(), which is called once before any Decode() have been called.  This will involve looking at whatever library is being used to import mp3s, and making sure the calls are thread-safe.  If they are not, then you will need to surround decode() or the lib calls with mutexes.
**The ODDecodeMP3Task will need to implement CreateSuitableFileDecoder, which should just return a ODMP3Decoder.  
*After the classes are implemented, go into ImportMP3, and instead of appending regular blockfiles to the newly imported wavetrack, use AppendODDecodeBlockFile (TODO), and after this, create a new ODDecodeMP3Task, add the wavetrack(s) to the task (ODTask::AddWaveTrack(WaveTrack*)), and then add the task to the ODManager (ODManager::Instance()->AddNewTask(ODTask*).  


Here is a more general breakdown of the important classes in the ODFramework:
*ODManager - A singleton class that fires off threads to dispatch tasks in a global queue that consists of the tasks at the heads of the WaveTrackTaskQueues (which correspond to per-track queues).  This class itself runs on its own thread.
*ODTask - An abstract class that performs a task, relaying information to the main thread about percent complete.  The method DoSomeInternal() is a pure virtual
*ODWaveTrackTaskQueue - A class that brings together n WaveTracks with n ODTasks.  This is an internal class not seen outside of the ODManager.
*ODLock - A mutex.
*ODTaskThrad - a thread on which ODTasks are run.  One of these instances generally do not complete the task, but just run it until a certain percent has been completed.  (This is so that we can juggle tasks around when we have high numbers of tasks queued up.)

===Summary of On-Demand files/code changes (9/5/08)===

Here is a summary of the files that were touched over the summer for On-Demand.

The following files are 99-100% my own code as of Aug 16, 2008.  The On-Demand (OD) framework is largely defined in these files:

*ODManager.cpp/h - The main public interface object for On-Demand Singleton.  Runs on its own thread.
*ODPCMAliasBlockFile.cpp/h - implements the PCMAliasBlockFile class for OD        
*ODComputeSummaryTask.cpp/h - Represents a background task that computes a summary for an audio file.       
*ODTask.cpp/h - An abstract class that represents a background task that can be queued/paused/modified
*ODDecodeBlockFile.cpp/h - implements the BlockFile class to be OD compatible for compressed audio files
*ODTaskThread.cpp/h  - represents a background thread.  ODLock, a mutex is also defined here.              
*ODDecodeFlacTask.cpp/h - Represents a background task that decodes a flac file.
*ODWaveTrackTaskQueue.cpp/h - Contains N WaveTracks that have M ODTasks to be run on them
*ODDecodeTask.cpp/h - Represents a background task that decodes an arbitrary compressed file type.  Abstract.
*Profiler.cpp/h - A performance testing class that profiles point to point code runs.       

The following files have some code inserted by me.  These files are generally previously existing c++ that I've attached the OD framework to.  These files are several thousands lines long, so I recommend looking for my changes by searching for the string "ODManager", which is the main public interface for the framework.
*AudacityApp.cpp/h 
*ImportPCM.cpp/h
*Project.cpp/h
*TrackArtist.cpp/h
*TrackPanel.cpp/h

The following files were previously existing base classes that were subclassed to contain on-demand functionality.  They were slightly altered slightly by me.  

*BlockFile.cpp/h
*PCMAliasBlockFile.cpp/h
*SimpleBlockFile.cpp/h


In the following classes I have made some improvements that were not specific to on-demand.

Added an internal class called InvalidRegion which speeds up the redrawing caching procedure by a large factor.  I felt this was necessary since on-demand creates many redraws:
*WaveClip.cpp/h (also WaveTrack.cpp/h) - search for "Invalid" to find code

*TrackArtist.cpp/h - Added an alternate method of drawing to support OD

==User Documentation==
===Description===
The On-Demand feature allows users to begin playing, editing, and viewing audio from large uncompressed audio files immediately, as the waveform image is being computed in the background.  Previously, the user needed to wait up to several minutes for large files while the waveform image was computed.

The waveform image refreshes itself automatically as it is computed.  The "On-Demand" feature lets users dictate the point in the file from which the waveform should be computed from, thus allowing the user to view and edit any point in the file instantly.  

===Usage===
To use this feature, the "Read directly from file (faster)" option must be enabled under the "when reading uncompressed audio files" section of the "File Formats" in the preferences panel.
Once this has been verified, open an uncompressed file type, such as a .wav or .aiff, via any of the normal means of loading files (drag-and-drop, File->Open...)
You will see a sequence of blocks that are replaced by an audio waveform from left to right as time progresses.  If you want to change the point from which the waveform is being computed, click on the left area of the desired region.  The area should be redrawn with the waveform within a second or two.  The percentage of the waveform that has been computed is displayed on the status bar.  

In the future this feature will be extended to effects as well as compressed formats.

==What's Next And Sidetracked Observations==
*Adding a preference area for OD.  Possible items:
**Number of threads (maybe displayed to the users a "high, medium, low")
**Frequency of screen update (in seconds)
**display percentage of ODTasks per track?
**OD ondemand ordering - forward only, roll forward and back, etc.
**Add minimum length for OD - otherwise use traditional method.  (right now it is at 30 secs)

*Adding OD support for effects.

*Adding OD support for other files.  

*Making clicking/selection faster - This isn't really part of on-demand, but with the threads running against the GUI, I'm really starting to feel it on my 4 year old powerbook.  I noticed that we do a modifyState (which creates/deletes lots of objects in the undo manager) and TWO redraws of the trackpanel for every click.  Furthermore, when we redraw, we redraw EVERY pixel on the wavetrack, not just the dirty area.

*The loading of stereo wavs is done in a way that is not as efficient as it could be- we currently load the left and right channels independently.  Most wavs are interleaved, so  this means we are reading the left and right data from disk, tossing aside the right, computing the left, then reloading from disk the same to compute the right.  This was the way it was done before OD structures were put into place.  I notice loading 15 minutes of mono is much faster than loading 7 minutes of stereo.

==Discussion Question/Answer==

===How To Ask Questions===
*Q:[[User:Mchinen|Mchinen]]How do i put questions on the wiki?
*A:put your name in by using <nowiki>~~~</nowiki> and ask.  If desired create a header as in "How To Ask Questions"

*Q:[[User:Mchinen|Mchinen]]: How do I get around the IsAnotherRunning() thing when Audacity crashes (mac)?  I need to restart everytime I crash to get around that dialog.

===Comments===
* '''[[User:James|James]]:''' How's about using three tildas: <nowiki>~~~</nowiki> or four tildas: <nowiki>~~~~</nowiki> with comments, like is being done elsewhere now?
* [[User:Mchinen|Mchinen]]: Amazing.

===Threading===
* '''[[User:James|James]] 11:25, 14 June 2008 (PDT):''' I notice that you are using five threads to do the updates, and also that you delete threads and create new ones repeatedly.  This seems counterproductive to me.  It seems to me you get better results using just one thread.  Try nMaxThreads=1 for starters.  Then consider why bother delete the thread and recreate it?  Any comments?
**[[User:Mchinen|Mchinen]] 00:05, 15 June 2008 (PDT)That number is a maximum of concurrent tasks.  e.g. if I have a ComputeSummaryTask on the queue, then only one thread at a time is being executed (note that it is removed from the task array and added back on at the thread's termination).  If there are more active tasks then that number, then they play a priority queue musical chairs until they are less. The reason I want more than just one or two is so tasks will appear to be running simultaneously, if we set nMaxThreads=1, the tasks will appear to stop and go, which in my opinion is less friendly to the user, but that may just be a matter of taste.  What do you think users want here?  Right now there's only summary computation on the task queue, but hopefully this feature can encompass more tasks that block up the GUI.
** '''[[User:James|James]] 10:45, 15 June 2008 (PDT):''' I'm not sure I'm following.  If at the moment blocks for which we are doing ComputeSummary are handled strictly one at a time, then that's good.  There is no advantage to starting two blocks at once.  If we do, I'd argue we shouldn't.  I'm not yet convinced by the 'for the future' argument - though I see where it's going.  Suppose one task is ComputeSummary and one is ComputeVeryExpensiveReverb.  Having two threads means that ComputeVeryExpensiveReverb can progress at the same time as ComputeSummary...  Though both will progress more slowly.  It probably is a matter of taste whether we want that possibility.
** '''[[User:Galeandrews|Gale]] 15Jun08 18:40 UTC:''' I think it depends what the user experience is. If single threading means an effect progress dialogue is always switching between progress and "busy", that won't be a good experience.   


===Progress Bar===
: "A progress bar that sits on each track will be implemented to show the progess of load-on-demand, providing information about which sections have been loaded/etc. The intention is that this progressbar will be used in future render-on-demand additions."

See also [http://www.audacityteam.org/wiki/index.php?title=Quickload#Per-Track_Progress_Bar here]

The stripy wave display already is good at giving some feedback on progress.  

The motivation for the additional progress bar is:
* A place to get diagnostics from in debug builds.  E.g. click to show the block-data (name, type) of the block that we have here.  Another kind of click to immediately refresh/recompute the summary data for this block.
* Future plans beyond this GSoC to allow effects to follow the on-demand pattern.  We'll have more complex kinds of progress.  
* 'Green squares progressing from left to right' is an idiom for work progressing that users are already familiar with.  It should mean a more intuitive interface and hence less explaining needed for new users.



'''[[User:James|James]] 05:02, 25 June 2008 (PDT)''' Michael: For the GSoC mid term, as it is coming up quite quickly, I'd suggest that for now you implement this as an overlay (green squares/empty squares) on top of the stripy and non stripy sections.  Give them about the same size and spacing and colour as the squares in a normal progress dialog.  Draw each square as a series of vertical lines - less work than figuring out what to do across block boundaries.  I'd suggest a preference item off/auto/on (default auto) to show this overlay.  'auto' means show when there are any outstanding blocks, hide otherwise.  You should have enough time to get clicking behaviour sorted too.  I.e. if the progress squares are shown, clicking on them pops up a dialog that tells us what the block id is.  Does that seem the right way forward to you?
*[[User:Mchinen|Mchinen]] 10:15, 25 June 2008 (PDT) James, By stripy/non stripy sections do you mean the OD stripes on the waveform, or on a wx progressbar?  If it is the waveform, what happens when the user scrolls around?  I was leaning towards implementing it by putting the progress bar on the TrackInfo section, so that the user can always see it, whether or not the stripy sections are there or not.  This has the main disadvantage of taking up space.  Is a traditional progress bar too simple for displaying OD status?  It has the advantage of being more familiar to the new user.  If there is a wx progress bar element (non-dialog) that may be the simplest implementation.
*'''[[User:James|James]]''' I'm talking about tweaking the look (and functionality) of the OD Stripes.  The squares would be green where wave is loaded, white where we're in an OD stripe.  Same scale/positioning as the OD stripes - a roughly 14px high horizontal band painted over them.  That's sufficiently close to a standard progress bars that I think people will get the idea - especially with the current left-to-right fill order.  Scrolling is same behaviour as OD stripes.  It's just a slightly tweaked presentation and quite easy to draw as a series of lines.  On zooming the strip stays the same height and squares should stay the same size, not grow or shrink - so we would get more of them.  The strip of squares becomes a place we can click on for more info.  [The original concept was for a separate track rather than an overlay].
** A real progress bar comes with info about time-so-far and time-left.  If we want that it may be worth adding an optional 'diagnostics' modeless dialog that has all kinds of info about number of threads, and work in progress time-so-far and time-left.  That could use a wxGauge for progress.  I'd say only add it if later in the project you find it would help you in tracking what is going on when there is more than one background thing happening.  I don't want a Gauge taking space on the TrackInfo panel and I think a diagnostic dialog is very optional for what we have so far.
*[[User:Mchinen|Mchinen]] 13:39, 25 June 2008 (PDT) James, thanks, now I understand what you meant.  I like the easily recognizable green squares progressbar, and I think this makes sense for now, so I'm going to start implementing it.  I do have a few issues with it - the main one being that I dislike adding additional click functionality to the area where the waveform can be drawn.  Perhaps the ideal model would be to have the wavetrack expand it's border a little to make some dedicated space for such a progress bar.  I'm not going to implement that now, but I just wanted to bring it up for discussion's sake.
** '''[[User:Galeandrews|Gale]] 27Jun08, 04:14 UTC:''' I think in one way this progress bar does not educate at all; a progress bar usually suggests users must wait, so can't actually make a selection at the end of a 90 minute track/play it/apply an effect to it until loading is finished. It may be better than not having the bar, but how can we convey that you can start interacting with the audio anywhere immediately? <br>Once it's understood what the progress bar is getting across, IMO not everyone will want to see it if it's as strident as that, but we're only talking about switching it off in Prefs (not very discoverable). <br> To me it adds little if it merely mirrors the progress of the waveform, and I don't feel clicking to see a block number is helpful/understandable to most users, so I tend to share Michael's feeling that an ordinary progress bar might have been better. It's more understandable, could include an explanation of what's happening, and could (I assume) have a "don't show again" checkbox. Or can the unloaded audio be overlaid with five or six light grey side by side sections that say: <br><br><font color="gray">
:: LOADING...           
:: Click anywhere       
:: to play or edit        
</font>
<ul><ul><li>'''[[User:James|James]]''' Good points Gale.  I am thinking ahead slightly, to when we have multiple 'things' in progress, e.g. different 'layers' of effects - and might need multiple bars progressing at different rates.  Firefox has a true progress bar in the bottom right of the frame for when loading a file and an eight-blob circle in top right to show it's 'doing something'.   The text idea might be worth trying.  Shorter text could be 'Updating display'. </li> 
<li> '''[[User:Galeandrews|Gale]]''' Using the status bar as a file progress bar isn't a bad idea, but it can't easily say which of multiple files you are importing. I think quite a few audio editors (though they don't have on demand) simply use the on screen building of the waveform as a progress bar. I've never felt this concept needs an embedded stripe. Our having one could reinforce user interpretation that you must wait for loading before you can do anything. Not having the stripe removes the need for agonising over Preference options for it.  On Windows, the transparency problem with the stripe makes the block markers look like distinct peaks in the waveform which is confusing. On demand effects progress could be analogous by having the background pattern? To be blunt, unless the stripe enables some really useful functionality personally I would either not have it or confine it to debug builds. :=) <br>Stripe or not, there are two ideas to get across 1) Play or edit anywhere now 2) (when implemented) click to start waveform load from that point. "Play or edit anywhere" is more important as they will see the loading when they click. I assume drawing a selection region will start loading that rez`gion. I don't think "Updating display" or Michael's "Loading waveform image" are quite explicit enough as they could still imply waiting. One way might be to have "Loading on demand..." in the pattern background and an explanation of what that means in the status bar. </li>
<li> '''[[User:Galeandrews|Gale]] 14 July 08 06:59 UTC:''' Another thing just struck me about OD. There is absolutely no way to cancel it? For large files that benefit the most from OD, this is not trivial if you drag the wrong one in (just done it now). Do we have a plan to address this if we don't have a more normal progress dialogue?  </li></ul></ul>    
***[[User:Mchinen|Mchinen]] 15:46, 17 July 2008 (PDT) You can cancel it just by clicking the x button on the upper left hand of the track, or by closing the project.    


==Dev Log, Michael Status==
===6/07/08===
Created this Log.  Trying to finish coding for the alpha before I go on a trip (6/12 - 6/17).  If all goes well we should have background loading, but I never forget the lovely murphy's law.

===6/08/08=== 
Created the basic classes for background loading - ODTask, ComputePCMSummaryTask, ODManager, ODTaskThread.  Turns out we needed to make our own thread class because NonGuiThread doesn't support instance methods.  

===6/09/08=== 
Trying to integrate the classes with the display.  Having some trouble on finding out where to put the local queue for each WaveTrack/TrackPanel.  Finding out what to do when the user deletes a section of the WaveTrack containing ODPCMAliasBlockFiles - does it deref() them?  If so, we can do something smart about not computing them, but it also makes load order for the ODTask more complicated.

*I am leaning towards putting the Task Queue in the FIRST WaveTrack of the joined/team stereo track, and having all ODTasks go there.
** '''[[User:James|James]] 13:52, 9 June 2008 (PDT):''' Where does the task queue belong?  My view is that there is just one, not one per track, and it doesn't 'belong' in a track.  It's either a project level object or an application level object.  One can argue both cases.
*** '''[[User:mchinen|Michael]] 15:08, 9 June 2008 (PDT)::''' I should say that there is a global task queue in ODMananger, but there is also a local task queue in each visual track - because we want to be able to layer effects on-top of a track.  I see your point, and I'm thinking about the future when we use the on demand structure for effects.  If we want batch processing - we are going to be able to layer several effects upon the track - and (maybe) only the frontmost in the trackqueue will be active (sent to the global ODMananger queue).  I've reverted to putting one task queue PER track.  It is task-dependent whether or not the task is per track or per linked track pair.  For example, computing summary is done one task on every track for reasons below.  However I feel like effects should be done on a one-per track basis - this way they can be syncronized left and right.  

*should we disable splitting/joining until ODTask is complete?  I want to allow this because it is a popular task.  A workaround may be to pause the ODTask, remove it from the WaveTrack, duplicate the Task, and clean each up so that it only contains references to blockFiles in their own respective queue.  OR redesign ComputePCMSummaryTask to re-examine the WaveTrack each time it does a part of the task.  
** '''[[User:James|James]]:''' I think it's legit to pause the ODTask at the next block-complete stage, so there is no 'block in progress' to worry about.  It's legit to hold up the splitting/joining operation until then.  Each block completes quite fast, so the little delay will be hardly noticed.  Splitting/Joining will often add new summary tasks to the queue.  You could actually rather than having a queue, search the project file for summaries needed, starting each search from the place last finished.  That way you avoid complexity in reordering and editing the queue when there are changes.  That could be a good solution particularly if you also want to focus ODTasks to regions currently visible on screen - since you can start your search from a different place after any sideways scroll or zoom...



*So as a starting out point, I'm making ComputePCMSummary a per-wavetrack structure.  Then it doesn't matter if it is split.  Also, ComputePCMSummary analyzes the wavetrack for ODPCMAliasBlockFiles (we will need a flag for this).
** '''[[User:James|James]]:''' I think you are making avoidable work for yourself this way.  Also rather than a flag, a counter of how many outstanding ODPCMAliasBlockFiles there are would probably work better.  When it reaches zero, nothing more to do.
*** '''[[User:mchinen|Michael]]:''' The issue I am worried about is when these things get copied/pasted/split around the outstanding number of ODPCMAliasBlockFiles can change.  It might be more work to handle each of those cases by updating the counter when those events happen (although this may be more efficient... maybe.)  If we have one ComputePCMSummary per track that updates the list and order of ODPCMAliasBlockFiles to be computed by looking at the WaveTrack it is associated with it, then it will always be computing and writing summary data associated with the track.  Also, this will give a more meaningful estimate of ODTask::PercentComplete(), so when the progress bar is full, it means all Summary Data has been computed for this track.

===6/10/08===
*Noticed that we call AudacityProject::ModifyState and thus UndoManager::ModifyState when we click on the waveform to change cursor position.  This seems wrong - we are losing the last undo to a click (since the undo manager deletes the last undo state,) as well as wasting cpu time to copy tracks (although this is done efficiently).

*Tried using pthreads in place of wxthreads on macs.  Got some progress, but there are some other threading issues I am trying to sort out right now,  Been bugging the wx-users list about them.  Looks as though wxMac may have some bugs.

*Got pthreads working, but we have crashes I am investigating.  Several cases need to be taken care of:
**When the project window is closed an the ComputeSummaryTask is still running, it still contains references to the BlockFiles and Tracks that got deleted.  We need to handle Close() to be gentle here.  We could just throw a mutex and terminate in Project::Close().  
**There is a hard crash bad_access going on somewhere - it looks like we are deleting things when we shouldn't be.  Looking into it.
**Cut and delete also messes up the track because *I think* the BlockFiles are deleted or something weird in this case?  I think we will need to add Ref() and Deref() access.  Note that copy works fine.  

*The GUI doesn't auto-refresh yet, but if you zoom or scroll, you can see the background thread load it.  

*Managed to fix Cut/Delete by adding a new implementation of CalcSummary that doesn't use static arrays.  Whats left is to make closing the project while the ComputeSummaryTask is underway feasable.  

*The patch is up with todays progress at http://eamusic.dartmouth.edu/~michael/temp/ODPatch4AndNewFiles.zip

*Tomorrow morning I hope to fix the closing bug, and run it by the list before checking in.  My threading method is #defined to use pthreads for macs, and wxthreads for pc/linux, but I need to verify this, preferably before checking in.


===6/12/08===
*Fixed all the resource mutex bugs I could find.  Need to break some of the critical sections down (most are just spanning list iteration, but there are some higher level ones as in Sequence::Delete,) they may be too conservative.
*Verified things kind of work on other people's machines
*Wrote a drawing method for the PC-I hadn't realized only macs used buffered drawing until Leland pointed it out.  
*Checked in a big amount of code.  
** '''[[User:James|James]]''' I've done a quick sanity check on Windows and can confirm that basics are working there too.  A track with one clipped sample came out all in red in preview-mode, so the path through the drawing code that goes via clipping needs to be checked.
*Tried to get the auto-redraw update to work, but I couldn't find an elegant way to do it given the current framework.  I think it's going to take a few days to get that up.


===6/13-14/08===
*TODOS:
**Fix weird red sample bug on windows (check to see if it exists on mac) 
**Find out where those wxGUIMutexEnter/Leave things are coming from.  
***Seems like wxLogDebug certainly causes this assert to fail from a non main thread.  looking up other possible GUI functions that may be called.  My guess is that one of the wx functions we are calling indirectly calls wxLog of some sort.  While it looks messy and wasteful to do this, I'm changing all the wxLog calls that are in functions hit by our task thread code to printfs, which don't seem to break anything.  Alternatively it looks like it will work if we call wxGuiMutexEnter and Leave before/after the wxLog calls, but that also is ugly in a non-beautiful way.
**Look into thread priority.  We want low priority for background tasks.
**Gale mentioned how doing an effect like bassBoost really slows down while the summary is being computed - not only that, but the task stops once bassboost is complete - need to fix this.

*Dones:
**Removed the initialization/managing thread creation for the ODManager from AudacityApp::OnInit() - so no OD threads, not even the manager instance's will be running unless someone tries to load a file via ondemand.  This way if the user chooses to load wavs by copy, then it will be just as fast as if ondemand never existed.  However, once the ODManager is inited, its thread will remain in place until the App exits.
*Added the redrawing mechanism and event structure to the OD classes.  Now the thing should update automatically.  However, we are using the worst possible drawing method - it appears we cannot invalidate existing sections of the wavecache, and thus to get the waveform to redraw a blockfile we are deleting the cache and forcing a redraw.  This is wasteful, because it hits every summary file that exists (and is available) in the sequence, but to get around this we will need to change the structure of the wavecache.  That involves finding out which sample belongs to which blockfile.  If we can do this without looking thru the sample/blockfile cache, that would be great.  But as a starting point we probably will.
*changed the mac build back to using wxThreads- Found out why it was crashing the use of wxLog calls AND init of wxLogger classes breaks from an external thread.  
*Tried every possible combination to get thread priority to work in wxwidgets to no avail.  instead I just stuck a bunch of thread sleeps and yields in CalcSummary for ODPCMAliasBlockFile.  

===6/15/08=== 
*ack.  Threads don't work at all on mac release, although they do in debug.  This is almost definitely due to wxThreads.  Maybe we should switch back again.

*Priority scheduling works on linux.  sleeping works on mac and linux.  Priority scheduling is better.  What to do.

*Did a test to check wxMac priorities.  Turns out they do work, but threads at min priority only go about 10% slower than the default priority, and only 20% slower than the max priority - see [[wxThreads]].

*[[User:Mchinen|Mchinen]] 15:28, 15 June 2008 (PDT)Switched back to pthreads, works again.  Last commit before alpha in.

*[[User:Mchinen|Mchinen]] 23:02, 15 June 2008 (PDT) Need to add a ODTask Finished Event to update the GUI as well - the auto update only happens when the task is active, and thus the last bit often never gets redrawn.

===6/20/08===

*[[User:Mchinen|Mchinen]] I've been out for the past several of days with a souvenir I picked up from the water and food of Mexico City in the form of a stomach virus.  I expect to submit a few new things involving the wavecache and gui events mid-week at the latest, but my communication will be spotty at best at least until monday.  
** [[User:James|James]] No worries.  Get well soon.

===6/21/08===
*[[User:Mchinen|Mchinen]]Fixed the wave cache to take invalid regions, so it only hits the blockfiles/disk for pixels that happen to be in those regions.  This is much better than forcing a redraw by deleting the wave cache, (and especially so for large wave files where there are more blockfiles,) because in that case it looks up the summary data for each blockfile.

*[[User:Mchinen|Mchinen]]Added an event that is fired when the ODTask finishes.  This ensures that the waveform will be drawn completely.  Also in the future this message might trigger something useful.

*[[User:Mchinen|Mchinen]]Next: need to look at Opening/copying ODPCMAliasBlockFiles, or cases where they are locked and a new instance is born.  Probably the answer is simple, we just need to add a ODComputeSummaryTask to the wavetrack containing these instances.  those tasks are harmless - if it doesn't have any OD BlockFiles, the task will simply delete itself.
**[[User:Mchinen|Mchinen]]Saving/opening now has a minor bug.  We get the "X number of orphaned blockfiles" when opening a project that has an ODComputeSummaryTask running in the background, because after the save, more summary data is computed, generating extra blockfiles not refrenced in the project.  

===6/24/08===
*Added ODTask support for opening files.  Now if we open a project that was saved with partial summary data, it resumes computing when we re-open it.

*Looking into doing the same for copy/paste.

===6/25/08===
*Implemented simple progress bar overlay for MAC buffered drawing
**TODOS:
**Do for PC
**Make transparent so underlying waveform can be seen (easier with buffered RGB arithmetic than PC wxcalls)
**Currently ALWAYS on.  Turns out detecting whether or not a blockfile is used may be an O(N) problem.  Trying to find a smoother way around, as this smells like a classic CS problem.  We are using a wavecache which stores knowledge about each pixel's blockfile status.  Do we need to go through each pixel each time we invalidate the wavecache in its entirety or subsection?  Arg. It bugs me that I can't figure out how to look at the invalidated region to make this decision.  
***The best hack around all this schmancy computer science debate may be just to look in the ODManager queue to see if such a wavetrack exists in it, before we even call the drawing function.  duh.
***The correct answer to this problem is to keep a count of the number of pixels that are OD pixels.  Then every time a region is invalidated and recomputed, go over that region before hand and keep track of the number of OD pixels within the region.  After the recomputation, subtract the number of ODPixels that have vanished in this update from the total number of OD pixels.  When that number reaches zero, then we have the condition where there are no OD pixels and we don't need the progressbar.  duh.

'''[[User:James|James]]''' When you get a chance, please post a screenshot.  Translucency very optional.  It might even be clearer without!
*[[User:Mchinen|Mchinen]] 15:54, 25 June 2008 (PDT) Will do the next time I run, and then an after-shot too.  The RMS will be shadowed by the progressbar; this is super-annoying for my eyes, and if I can see the min/max/rms behind the progressbar with tranlucency, it may help calm me down.  On the other hand it may be really ugly to see tranlucency with bright green, so we'll see.

===6/26/08===
[[Image:ProgressBar1.png|thumb|opaque progress bar]]
[[Image:ODProgressBar2OpaqueClip.png|thumb|progress bar clips waveform]]
[[Image:ODProgressBar3Transparent.png|thumb|transparent bar allows waveform thru]]
[[Image:ODProgressBar4Finished.png|thumb|bar disappears on finish]]

*Posted a screen shot: The opaque progressbar can interfere with the display of audio with high min/max: 

*Fixed a bug that was causing the last pixel not to be drawn.  The wavecache arrays are for some reason accessible up to mArray[len], and not mArray[len-1] as you might expect.

*Got the auto-on-off to work, so when the wave is completely loaded, the progressbar vanishes.

*Added transparency - again, this might be harder to do for PCs.

*Added drawing support for PCs.  Fully functional, but the colors are way off.  I suspect this has something to do with theming and something about it that I am not understanding.

** Just use wxColour( R,G,B ) for now.  See [[Unusual_Header_Files]] for an explanation.  [[Theming HowTo|Theming]] can be made to work, but as it's due a rethink it's better to use 'old style for now' and we'll theme properly again when we revisit theming.


===7/7/08===
Remembered about this log.

*Fixed a bunch of warnings that were popping up.  These don't pop up on mac - perhaps we should change the project settings?

*Changed the update/redraw speed to about 2 times a second as requested - this really hurts performance on my power mac.  I think the frequency of update should be a user defined setting.  Audacity redraws every pixel on that waveform because it doesn't support dirty regions (I implemented invalid regions, which is a very different thing.)  

*TODO:preferences panel for OD - with use OD or no option, number of threads, frequency of update, thread priority (maybe.)  Will need to figure out how to add something to the panel.

'''[[User:James|James]]''' When you get nearer to doing this, please discuss on Audacity-devel.  Exposing a large number of 'tuning parameters' to the end user is (generally) a bad idea.  It may also make sense to use space on an existing panel.  If we don't reach consensus on how best to display 'progress'/'stripy'/'LOADING...' that could become a preference too.

===7/8/08===

*Fixed a bug that was preventing update events from getting drawn to the screen.

*removed - overrided some wxlog code that was probably causing the occasional wxGuiMutexEnter crashes.  So remember to yell if you see any of them.

===7/13/08===
* Added basic on-demand support (finally) - now, when you click on a part of the waveform, it begins loading from that point instantly.  I haven't added this to CVS because of the alpha freeze.

*There are some bugs I need to blog about so I don't forget:
**When you demand an area, it begins loading from that point on.  Then after it goes in reverse from the point you clicked on.  But that part that loads in reverse is getting drawn weird - it looks like it's missing the drawing of one of the pixels in each blockfile, only when it goes in reverse.  There must be an off by one bug in the wavecache code that is somehow covered when we do forward loading order.
**In a stereotrack, the second waveform isn't affected by ondemand - this is more of an annoying todo then a bug.  The code that moves the click cursor is in TrackPanel, so we don't really have access the to the pointer to the second wavetrack, since all those are in the project.  But it got me thinking again if we might want to have a special ODComputeStereoSummaryTask that keeps the pointers to both around and works on one thread instead of two.  
***'''[[User:Galeandrews|Gale]] 14 July 08 07:10 UTC:''' What happens if you actually demand a selection region by drawing one? After drawing that region, should it go forwards from the end of that region to the end of the track, before going backwards from the start of the region?     
****[[User:Mchinen|Mchinen]] 05:41, 14 July 2008 (PDT) Yes, that is what happens.  The code doesn't care about selection regions - it just looks at the beginning of the select area, and will process from that point onwards before wrapping back around.  Don't know if this is the best way or not, but it is the simplest way to implement it as a starting off point.

*Began looking at importflac.cpp to see if it can be OD'd.  I'll have to rename ComputeSummaryTask to be more specific, or I need to make it more general.

===7/16/08===
*Commited on-demand functionality
**Fixed aformentioned wavecache bug (was an off by one error)

*Made a major change to the OD system - now one ODTask can manage multiple WaveTracks.  So when we load a stereo wave, it happens on one thread, as opposed to the two that it used to.  There may be some bugs and I'm worried that this will introduce some non threadsafe issues, especially when we expose those wavetracks from the task.  For now though, it seems alright.  


===7/17/08===
*Changed some int/longs to sampleCount, hopefully making OD able to handle big files.
*Looked at ImportMP3 as well as ImportFlac - trying to decide which one to work on.  ImportFlac seems easily ODable because there is an obvious seek_absolute which lets you move to an arbitrary sample.  The MAD mp3 library has no obvious documentation on the first few pages of a google.  
*Decided on the structure for OD loading for non aliased and compressed files.
**Subclass SimpleBlockFile to ODSimpleBlockFile
***Don't write a file until the ODDecodeTask decodes the audio for the block.  
***Return dummy values for read until the file is written.
**ODDecodeTask is an abstract class with pure virtuals DecodeSamples(int start,int length, samplePtr, sampleFormat)
***The ImportXXX.cpp's Import function will have to guess how many samples the file has and then stick X number of ODSimpleBlockFiles into the wavetrack.  It will also have to figure out the number of channels etc and pass this to the ODDecodeTask so it knows how and where to decode/seek.
***In it's main loop (DoSomethingInternal) it pops one off the blockFile queue and calls DecodeSamples and then writes the file.  That simple.  The mutexes have to go in pretty much the same places as they did for wavs.  

===7/21/08===
*Created ODSimpleBlockFile, the class that will serve as an ondemand block file for compressed/encoded files.

===7/22/08===
*Changed it so that OnDemand's focal area is always the left side of the select region.

===7/23/08===
*Fixed Timeshift bug for single (non-split mono or stereo) tracks.  Now you can move a track with the timeshift tool and click on it and get the expected behavior.
*Noticed another minor wavecache bug that only occurs once in a while.  Sometimes a single pixel on the blockfile doesn't update - so this may be the off by one error on the left hand border of the invalid region - perhaps it is too small?  Looking into it.
[[Category:GSoC]]

===7/24/08===
*Added Remove Track/Undo/Redo import support.
*Going to deal with split track behavior.  My opinion is that when we split a track, we check to see if exists in the ODWaveTrackTaskeQueue list.  if it does, we take any ODWaveTrackTaskQueue that has the track in question being split, and then with careful mutexing, make a new ODWaveTrackTaskQueue with a duplicate ODTask for each split track.  When we join tracks together, it's a little trickier.  In that case, we may want to remove all associated tracks from the ODManager, then re-add them under one thread.  This will make sure all the original threads die out and start on a new one.

===7/25-26/08===
*Added support for the new split and merge stereo tracks.  It is smart and checks the ODTask queue for both tracks before merging them, detecting if they are mergeable or not.  Cases where the tracks have different ODTask queues mean that the two tasks can't merge.  

*After you split a track the ordering is a bit messed up.  Going to look at this tomorrow.
*There's still the lingering wavecache bug.  Can't figure out what it is for the life of me.

===7/29/08===
*Added slowly animated dummy waveform.  Note: I don't like the idea of dummy waveform so much now and want to change back to stripy blocks.  This is an aesthetic decision.  I may try to make the dummy waveform nicer - i.e. using a sinusoidal wavetable and varying the phase of several harmonics to draw the dummy so it has more pretty changing shape features.  But I'm still not sure.
**'''[[User:Galeandrews|Gale]] 30Jul08, 04:02 UTC:'''  FWIW I don't mind the stripy pattern at all, my problem was with the green progress stripe. I think the idea of those two colours in the pattern was good, maybe work something on those lines, perhaps with something that looks a lot more like a generic unzoomed waveform. Dare I suggest even the blue overlays the grey on and off every 2 seconds or something, making it look active (rather than the backwards progress which I find a bit confusing).       

*Added status bar updates as the ODTasks execute to show progress and give an explanation for what the ODTask is doing.

===8/4/08===
*Shocked to find that audacity doesn't import export waves bit identically.  Think some code I ran across in ImportPCM.cpp might be the cause, but can't be sure.  I want to follow this closely.

*The bug reported by gale/richard about 'freezing' on cancelling 'remove dependencies' during save was found to be something that existed before OD.  Not sure what the expected behavior should be, so this is under discussion.  My own personal opinion is that the cancel should cancel the entire save, and leave the project as being partially dependent on the aliased file (this is not 'risky' at all, as it may sound.)  

*Fixed the OD related freeze by adding a mutex to ReadData in odpcmaliasblockfile.

===8/5/08===
*Fixed some warnings.
*Short day.  TODO: make the display look better.  Also, for anyone reading this please comment on the save/removing dependencies dialog problem, or let me know what needs clarification.

===8/6/08===
*Fixed a bug that was preventing the per-track mouseover OD progress message from being displayed for unselected tracks.
*Fixed a bug that was preventing non-active projects (in the background) from having their waveforms being redrawn upon complete
**Currently the background projects don't get the auto-redraw updates every few seconds because I felt this was a waste of processing time due to the inefficient way audacity redraws.  Hope this is reasonable - at least to me it seems more reasonable now that the background projects are redrawn upon an associated ODTask completion.
*Replaced the dummy waveform with an animated (at one pixel per second) version of the old stripy block method.
*Removed various TODOs.
*Changed OD progress in the general (non-mouseover) text to Gale's suggestion:"Import(s) complete. Running n on-demand waveform
calculations: overall progress n% complete".

That looks like all the basic bugs that needed to be fixed, going to look at making flac on demand again.

===8/9/08===
*Added support for saving.  The behavior is a little complicated - I added a special flag to the UndoManager to notify of significant progress in an ODTask, or completion of an ODTask.  This will re-enable the save menu, ask to save on quit/close, etc.  The special flag was needed because ODTask progress is unlike any other change that enables the save because single block files should not be able to be undo'ed - it would take up the undo/redo stack and it isn't very useful.  
*Fixed a whole bunch of threading issues.  For some reason the save dialog really brought them out, since they are doing a whole lot of disk access and moving of files.  There may be one thread bug left in libsndfile's function, but it hasn't crashed on me since my fixes.    It's hard to tell if is thread-safe or not, but I saved a 2hr wave over 200 times to do all this debugging and I'm moving on for now.
*Next have to think about doing the profiling tool or do the Flac on demand.  My personal opinion is that the OD stuff seems fast enough, (and Martyn kindly did some tests for us already that shows this,) so I should focus on Flac and other bugs that will come up.  We only have about 5 days till the freeze, so I want to get focused.  I think adding flac support would be really useful for the OD structures as a design addition, since it would force me to make things less wave specific.

===8/11/08===
The following changes are pending on the ScoreAlign fix which is currently breaking mac builds:
*Added the old triangle dummy waveform back to the display, this time animated on top the stripes.
*Adjusted the frequency of the unsaved changes flag from a percentage of the total summary task to a constant.  This constant can be changed, it is called kNumBlocksPerDoSome in ODComputeSummaryTask.cpp.  Right now it is set at 36 blocks, which is about 3 minutes on a stereo 44.1kHz track.  I really don't care much about the frequency so feel free to go for it.  As Richard noted, this has some disadvantages in that the rate isn't constant over all machines, because it will take slower ones a longer time to compute the summaries for 36 blocks.
*Added two new ODTasks, ODDecodeTask, and ODDecodeFlacTask, and some decoding classes within the same file called ODFileDecoder and ODFlacDecoder, and renamed ODSimpleBlockFile as ODDecodeBlockFile.  
**I know that ODDecodeBlockFile should logically be named 'ODEncodedBlockFile', but I think a newcomer will be able to see the connection between ODDecodeBlockFile and ODDecodeTask if they are named that way.  
**These three classes should be all it takes to import flacs on demand.  
***ODDecodeFlacTask is a subclass of ODDecodeTask.  ODDecodeFlacTask should have almost nothing in it, except a type flag that identifies it as being able to handle flacs, and the ability to pump out ODDecodeFlacs as needed.  An ODDecodeTask, like ODComputeSummaryTask, is one-to-a mono or stereo track.   However, due to copy or paste, a given track might have any number of Flac files that need to be decoded.  For this reason it keeps track of filenames/lengths/etc in an array of ODFileDecoders.
***ODFileDecoder is a class that manages one file's header parsing, reading and writing. An ODFileDecoder has some general decoding implementation and some pure virtuals like Decode() which writes some samples into an external buffer and ParseHeader() which sets it up.  Of course, all ODFlacDecoder should have in it are those virtual methods.  
***ODDecodeBlockFile is pretty rough, and I'll update later on this.

===8/12/08===
*Added Profiler.cpp/h which allow the programmer to profile a certain tasks by inserting two macros at the beginning and end of the task.  It gets printed out to a text file with lovely filenames and linenumber and average time stats.
**Profiled the OD version of loading wavs against the old method.  Turns out for large files OD is faster, so it seems all around a better choice.  For small files, it doesn't really matter which method is used.  With OD the user can interact of course, so it is the clear winner.  A bit relieved to know that it isn't a big performance hit.
*Fixed some crashes/bug related to libsndfile not being threadsafe.  The crashes we got in ODPCMAliasBlockFile::ReadData were due to this - I had a suspicion that those sf_open calls weren't threadsafe, but I finally did the right googling and crashing to verify it.  Now they are safely mutexed.  This isn't ideal, since mutexes suck time out of the process.  It shouldn't matter when you are loading just one file at a time, but when you do two or more, that's when it hurts, since the mutexes start to lock eachother out.  In the future it would be nice if libsndfile could be made thread-safe or we use a different/custom lib for this.


===8/13-14/08===
*Fixed several memory leaks and subsequent crash bugs caused by the fixes.

===8/23/08===
*I fixed a bug that was causing crashes/freezes on my PPC that was being caused by memory leaks.