Proposal Displaying Samples
|Proposal pages help us get from feature requests into actual plans. This page is a proposal to improve the display of samples in the zoomed-in track view, and how that relates to audio selection. It also proposes rules for aligning samples to track time.|
Proposal pages are used on an ongoing basis by the Audacity development team and are open to edits from visitors to the wiki. They are a good way to get community feedback on a proposal.
- Note: Proposals for Google Summer of Code projects are significantly different in structure, are submitted via Google's web app and may or may not have a corresponding proposal page.
In the Audacity 2.1.2 release, it appears ambiguous whether a specific sample at or near the end of a selection is within or outside of the selection. This makes sample accurate editing unnecessarily tricky and confusing for users.
The following image shows a short selection of three samples. The green dashed lines have been added to indicate clearly where the selection in the Timeline starts and ends. As indicated in the illustration, it would appear that the first sample that falls between the pair of green dashed lines is not selected. This looks strange because it is clearly within the selected region on the Timeline, but it is not highlighted in the track.
The final sample between the green dashed lines clearly looks to be selected because it falls within the selected region in the Timeline and it is highlighted in the track.
The screenshot was created by generating 1 second of silence in a mono track, cutting the silence and then pasting back at 0.007 seconds. Both the track sample rate and the project rate are 44100 Hz.
To see which samples actually are selected, we can apply an effect that causes a visible change in the sample values. In this example I will use a Nyquist command that adds an off-set of 0.5 to each sample (though any effect that causes a visible change in sample values could be used). The Nyquist command is:
(sum *track* 0.5)
Here we can see which samples were actually selected because they have been raised by 0.5 linear. (Note also the changed selection in the Timeline that occurs on processing). The first sample within the Timeline selection that appeared to be not selected, was in fact selected, and the final sample that appeared to be selected was in fact not selected. Considering that the total number of selected samples is 3, that is two very significant errors.
Why the problem occurs
Essentially the problem comes down to taking several different reference points for "sample position".
What we do now (Audacity 2.1.2)
- Timeline: The selection in the Timeline indicates continuously variable time positions relative to the start of the Timeline. It is not quantized to sample periods.
- Sample Dots: The dots on the zoomed in waveform view are a graphic device to assist visualizing discrete samples. In fact the entire waveform display is no more than a visual aid. Sound travelling through a fluid (such as air) is a longitudinal wave rather than a transverse wave, though displaying it as a transverse wave is helpful for visualization. (Transverse and Longitudinal waves - BBC tutorial). We place a dot at the start time for each sample period, and represent the numeric sample value by the vertical position of the dot in the audio track.
- Sample period: This is the length of time from one sample value to the next. Within an audio clip, this value is constant. It is the inverse of the sample rate. This is the shortest meaningful unit of time for PCM digital audio as a sample value either exists or does not exist - if we interpolate an "in between" value, then we are in effect resampling to a higher sample rate so that the sample period is shorter.
- Alignment of samples in a track: Within a single audio clip, all samples are exactly aligned to whole sample periods. Audio tracks in Audacity support one sample rate only. This sample rate is displayed in the track information panel. However, Audacity allows arbitrary alignment of audio clips in an audio track - samples within an audio clip need not be aligned to sample periods with respect to the start of the track. they need only to be aligned to sample periods relative to the start of the audio clip. I shall illustrate how this can cause unexpected problems and propose that we do not do this.
- Selection snapping within a track: Ignoring the selection "Snap" feature (we are looking at the behaviour when snapping is off), when dragging a selection in an audio track, the selected region snaps to sample periods relative to the start of the Timeline. Thus there is usually an offset between the bounds of the selection in the timeline and the bounds of the selection in the audio track (unless the selection in the audio track happens to exactly line up with a sample position in the audio track). The position of the start and end of the selection in the audio track may not line up with the actual samples in the audio track. This is because the selection is snapping with respect to sample periods relative to the start of the Timeline, but as described above, the samples in an audio clip may not be aligned to those positions. If there are tracks with different sample rates, the start / end of the selection may appeared staggered due to this "snapping".
Selection Across Tracks of Different Sample Rates
In this image we have three tracks. The first and last track have sample rates of 8000 Hz while the middle track has a sample rate of 44100 Hz. Note that between the start of the selection in tracks 1 and 3, and the start of the selection in track 2, there is a sample in track 2 that is clearly(?) not selected even though it is positioned well after the start of the selection in tracks 1 and 3. This begs the question "where does the selection actually start?" For many Audacity users, this must be confusing and I'm sure we can do better.
Joining split lines may cause audio to move
Steps to illustrate this issue:
- In a new project with a project rate of 44100 Hz, add a new track.
- From the track dropdown menu, set the track sample rate to 8000 Hz.
- In the Selection Toolbar, set the time units to hh:mm:ss + samples. Note that "samples" here means "sample periods at the project rate relative to the start of the Timeline.
- In the Selection Toolbar, set the start time to 0h 0m 0s + 3 samples.
- Generate a tone of exactly 1 second duration.
- In the Selection Toolbar, set the start time to exactly 0h 0m 10s.
- Generate another tone of exactly 1 second of duration.
- Set the Selection Toolbar to display the "End" time.
- Ctrl+A to select all. As expected, the End time shows as exactly 11 seconds.
- Ctrl+J to join the clips. The Selection Toolbar still shows an End time of exactly 11 seconds.
- Ctrl+A to re-select all. The Selection Toolbar now shows an End time of 10 seconds + 44097 samples.
Relative to the Timeline at the current project rate, we have lost 3 samples due to the second clip moving to the left so as to maintain constant sample periods throughout the new audio clip.