View source for Talk:Proposal: Earlier Disk Over-run Warnings

==Discussion transferred from long email thread==
'''Steve 14Apr16'''
I'd guess that what we need is 'one' piece of code that tests for disk space, 'one' piece of code that handles the error, then multiple places where the test is called.

The "test" needs to be available and sufficiently flexible to work in all cases. For example, when generating a sound from a built-in  effect we know in advance (almost) exactly how much space will be required, but when generating from a Nyquist effect we have no idea until Nyquist returns the data, and when recording we only know the rate at which data is written.

Getting the "test" and "error handling" to work correctly is the major task and imo should be the subject of bug 437.

Additional (smaller) bugs can then be raised for each additional situation where running out of disk space may occur, and fixing these should be relatively easy once the code for testing  and error handling is in place. For example, it may be appropriate to enable calling the test periodically from ProgressDialog.


'''Steve 14Apr16'''
Skimming Ed's old patch, it looks like he was trying to deal with the case: "write to disk has failed - stop now and deal with it". I think this is an important safety mechanism, but in most cases it would be best to catch "Not enough room for requested action" _before_the attempt to write is made.

For example, if the user attempts to generate 4 hours of stereo sound when they only have enough space to generate 3 hours 59 minutes, it would be better to catch that before we start generating.

Also, let's not try to use ALL of the available disk space. If the user is critically low on disk space (only a few tens of MB), then let's stop there and tell the user they need more space. Some other application or process could easily eat up the last bit of free space, causing the computer to crash and put the blame on Audacity. Best if we provide some protection against  that.


'''Peter 14Apr16:'''
I too (as an ex-developer in days of yore) am much in favour of a standardized centralized method - as discussed by Gale and Steve.

I'm kind of assuming that Mark's test for running out of space should be readily capable of being suroutine-ized (if it isn't so already) so it could deployed readily across the rest of Audacity.

A few things we have to remember it may be possible to standardize the error handling for most of the rest of the cases wher the end result is "no can do" an no action takes place - but I supspect that we need to retain the more subtle error handling that Mark is working on now for Record - as that doesn't stop an activity from starting instead it puts the brakes on an activity that is already running.

We also need to consider that there may be a need to provide different headroom allowances for different various commands in Audacity -  if so we will need a table and imo that should *not* be user settable.


I've started work on the background for a Proposal to track this grand-bug, at the moment I am drafting a set of tables to help us examine which commands are affected and which are likely to require the earliest attention.  Once I have the shell tables I will start some experimental work to ascertain the % growth absorbed by each command.  Thes tables can also be used to track the progress of each sub-bug within the grand-bug.



'''Peter 14Apr16:'''
It occurs to me that we have 3 sorts of commands in Audacity:

1) Open-ended ones like Record, Import, Open, Generate etc where when the command is first issue Audacity can have no idea of how much data will be generated

2) Commands that are likely to make a smallish increase in the data size a crossfade  or a fade out say

3) Commands that are extremly unlikely to increase the data size at all or even dectrease it.


It seems to me that what we should be doing is that whenever the user issues *any* command, then Audacity should first check to see if the disk in use still has more disk space than our pre-set headroom buffer - and if not then the command should be inhibited and an error message raised.

Additionally when Audacity is idling and the user is doing nothing Audacity should occasionally check the remaining available disk space - and if it is at or below or pre-set headroom buffer then an error message should be raised and all commands blocked.

For type 1) open-ended commands we will always need special code
*a)For recirding we need to chack as the recording progresss
*b)For Generators we should be able to make a suitable calculation based on the parameters the user sets
*c) For Import we should be able to calculate based on the size of the file and its type - bearing in mind that we "add water" to compressed files to bring them into Audacity working format.


BUT for type 2 and type 3 commands, if we set a chunky enough headroom buffer, we can with reasonable safety (certainly more safety than right now) - just blocking at command issuing time may be sufficient.

Or for type 2) commands Audacity may either need to make a pre-estimate of likely additional disk space usage and check that against available - or it may need to regularly check as it progresses.  But I think we could maybe dispense with this and just rely on the simple space check at the time the command is issued (if we set a suitably sized headroom buffer).

Doing this simple check could avoid us having to go into each and every command.

Does this sound like a feasible solution?



'''Federico Miyara 15Apr16:'''
I propose a hard limit (say, 50 Mb available space) after which no further recording can take place, let alone continue an already started recording. To ensure that, the available space in the dtive could be checked at the beginning of each, say, 10 Mb to be recorded. If the available space is > 60 Mb then the recording is allowed. Otherwise, it stops with a popup informing termination and its reason.

Besides, when the limit is smaller than, say, twice the hard limit, a warning should be issued indicating that the available space is too small so the recording may be terminated at any moment. In the event the user frees enough space, the recording would continue normally.

The hard limit should be chosen taking into account the likelihood that other programs or the operating system consume the available space so rapidly as to prevent the next 10 Mb to be saved.   

This would address the possibility of failure to write because of lack of space only. There may be other reasons that should be addressed in a different way. In a very friendly application, an attempt should be made to 1) Allow the user to take care of the situation and fix it without losing the whole project or forcing heroic rescue of the waveblocks, or better 2) Record rescue information to a temporary file on other available drive (this idea is taken from Cool Edit, which allowed two TEMP files on different drives).



'''Gale's response to Federico 15Apr16:'''
So in Audacity's case would that be a copy of the AUTOSAVE file?

Another possibility if we allowed a second temp folder could even be to switch writing of the block files to the second drive.

That is an enhancement rather than addressing the bug of course.



'''Gale 15Apr16:'''
An interesting case is running an effect on a two-hour track where the disk space check passed but some other operation external to Audacity then uses space while the effect is running. Presumably this is rather like the standard record case that checks available space ongoing.

In the case of an effect or an import or generate that runs out of space due to external circumstances, presumably we would cancel the operation. That would be better for the user than following through with the operation.



'''Peter 15Apr16:'''
My research/testing today indicates that most effects add an equivalent amount of data to the amount of selected audio that they are operating on.  See the propsal page

I.e. running an Amplify, say, on entire project will double the on-disk size of the project - I'm guessing that this is so that we can quickly handle the undo.

What I haven't experimented with are what is the cumulative effect of running multiple effects - but even without that we can see that running effects in a situation where disk space is limited can be more than somewhat a risky business right now.

This is something that we really should address if we possibly can.



'''Federico Miyara 15Apr16:'''
There arises the potential problem that the second drive might eventually get full as well. I'm not acquainted with how the autosave works. If it only uses a limited and known amount of storage (for instance a limited number of waveblocks), then this may work. If the temp files can occupy all available space, then there is a potential risk.

My proposal is:

Remaining disk space is checked at the end of each 10 Mb block

TEMP1 on the main drive may be written until a properly selected hard limit (for instance 50 Mb to disk full), minus 10 Mb, is reached. Then a first gentle warning is issued inviting the user to free space. During the next 10 Mb, as no check of the remaining space is done, the recording goes on. Once this block finishes, if no action has been taken by the user, the next check reveals the hard limit has been reached so no new 10 Mb block can be stored temporarily on TEMP1 and a warning is issued immediately explaining that the recording will stop within 1 minute.

TEMP2 on the secondary drive is a reserved 10 Mb file usually full of 0's. When TEMP1 reaches the hard limit, a last 10 Mb block goes to TEMP2 and the recording is terminated.

It may seem that the second TEMP on a different drive is an unnecessary complication, but the advantage is that the user has been warned with enough time (about 1 minute stereo at 44.1 kHz) to fix the situation or accept termination. If no TEMP2 were used, it might happen that several false positives would be issued while the user is working on freeing up disk space. In this case the warning is clear-cut: the next minute can still be recorded but there is no further extension.

The sizes of the grace period and the hard limit are just an example.

The hard limit could be estimated from the typical transfer rate that can be expected from an up-to-date hard disk.

Note: Once the recording ends, the last 10 Mb are in the TEMP2 file, so some procedure should be implemented to join both pieces together once some space has been freed up in the main drive.



'''Federico Miyara 15Apr16:'''
One more thing: For bold users, an "ignore warning" button could be provided with a strong warning, after which no check is done any longer with the risk of system crash an loss of information. This would provide an extra 4 or 5 minutes if the planets align and no process swallows any disk space.




'''Steve 15Apr15:'''
I think that the amount of space needs to be configurable. I'd guess that 50 MB is a reasonable limit for Windows machines, but Audacity will run under Linux on a Raspberry Pi, which may have only 32 GB disk space in total (and virtually nothing running in the background to consume disk space).

WxWidgets has a function wxGetDiskSpace. I don't know how reliable it is, but the reliability of the disk space test will be a crucial factor in determining the best solution (http://docs.wxwidgets.org/3.0/group__group__funcmacro__file.html#gab94c2e0044fd40d901ac250be547d0da)

An idea off the top of my head: perhaps we could write a 50 MB dummy file to the data directory on launch. If at any time a write fails due to lack of disk space, stop immediately, delete the dummy, have another go at writing the data, throw an error.


'''Steve 15Apr15:'''
My initial though was that a preference would be over-complicated, but on second thoughts a "safe minimum" for a typical Windows machine is probably massive overkill for a lean but tiny Linux box. If we do enforce disk space headroom, then I'm +1 for it being configurable (but happy for that setting to be buried deep in Preferences as most users should not need to touch it).
* '''Gale 15Apr16:''' I don't think the disk space threshold at which we stop or prevent an action should be user-configurable. I suggest that threshold should be less than the threshold for the Preferences warning "Low disk space at program start up", so we may want to increase the threshold for that preferences warning.<p> Pre-reserving small amounts of space seems a little pointless to me, but could itself cause a stop that otherwise would not occur, especially if we made the buffer larger. If we did it (I'm -1 at the moment) I think the size could be configurable in audacity.cfg without being exposed in Preferences. </p>

'''Gale 15Apr16:'''
Copy (to the Audacity or system clipboard) does not I believe take large amounts of space, because only pointers are copied.

'''Peter 17Apr16:''' Experiments seem to show that just the Paste too does not seem to increase the size on disk by any appreciable amount.  It's only when you actually do something with it like a Mix&Render that the size on disks actually increases.

'''Gale 17Apr16:''' I've just found confirmation that all copy does is write a pointer to memory, nothing to disk: https://sourceforge.net/p/audacity/mailman/message/14807839/.