Back to files

Overview

This directory provides automatic testing of the audio output BSE synthesis networks produce. Since changing algorithms might result in subtle differences in the resulting audio files, 1:1 comparision of audio files (such as storing an md5 hash) is not a good choice for an automated testing framework.

Thus, the tests are based on comparing features of audio files that are not as fragile as a bytewise comparision, but still provide a good similarity measure, such as the spectrum or the average signal energy. To see in detail which features are supported, call bsefextract --help.

To run the currently available tests, there is a Makefile rule, so you can type:

make test

Adding a new test

To add a new audio test, lets call it foo, you need to perform the following steps:

create a foo.bse file with beast, which will generate appropriate output; ensure that the bse file terminates after some time (using the song sequencer is a good way to ensure this)
save foo.bse in this directory (add it to the CVS)
add a new target to Makefile.am (look at the other tests to see how its done)
add the target to the FEATURE_TESTS line in Makefile.am
run make foo-test once, the test will fail, but you will generate a foo.tmp file, which you can move to the reference file using

mv foo.tmp foo.ref

you need to add add the created foo.reference to CVS

6) run make test, the newly added test should be executed and succeed now

Note, that if you need multiple tests for the same .bse file (for instance because you want to set a different threshold for different features), you can do so by adding multiple targets foo1-test, foo2-test,... using difference reference file names for the tests, like foo1.ref, foo2.ref,...

Good features to use:

Generally, you can use bsefextract --help to find out about which features to use and what they do.

As a recommendation, there are three features that are generally useful, and should cover almost any case:

--avg-energy: since this evaluates the energy, it will ensure that the
test file will stay "as loud as it used to be"

this is generally a good thing to assume

--spectrum: this is a matrix feature which represents how the spectrum
changes over time - it represents approximately what you view in the fft scope and approximately what you hear (over time) when you listen to the file

                    using makes sense when your sound changes over time, and
                    you want to ensure that this stays the same

--avg-spectrum: is an average of the spectrum; if you have a sound which
doesn't change over time, you only need to compare this to ensure it sounds the same if you have a sound which does change over time, it is ok to add it as well

so you can safely always use this feature

If you use these features, you will get a good representation of "sounds the same", so it is the feature set that you can use for almost any test.

There are special features that can be useful for special cases, like for instance the --base-freq feature which determines the frequency of a signal (could be useful for testing an oscillator frequency), so its a good idea to at least read through the bsefextract help once, so you know whats there.

Tricks

volume trick: right now, the only time dependant feature is the spectrum;
however, spectrum comparisions normalize the data during compare, and this means that they will not capture differences in volume

                    so if you for instance have a test that consists of a sine
                    wave that becomes louder and louder, a spectrum comparision
                    with a signal from a sine wave that has constant volume
                    will succeed (!)

                    how to workaround this, if you want to do this test? simply
                    add another signal (for instance sine wave, with big
                    frequency distance to the signal you already have) with a
                    constant volume to your test signal

                    how does this help? now the spectrum will have a relative
                    difference, that is the strength of one frequency component
                    relative to the others will grow; this in turn is not
                    normalized away, and you have constructed the test you want

length trick: since bse can not guarantee that the signals produced will have
exactly the same length, the "spectrum" feature is not length sensitive - if you compare the spectrum of two different signals then only the start of the spectrum (that are present in both signals) is compared; so the shorter signal determines how much is actually compared

                    now usually your test will terminate at the same time, so
                    usually there is no problem; but what if beast decided to
                    play only the first second of your song and then exited?
                    how could you make the test detect it?

                    use the --end-time feature; since it indicates the position
                    of the last non-zero sample, together with --cut-zeros
                    (which will cut tail and head silence), it becomes the
                    length of the audible signal; so even if the --spectrum
                    feature will still successfully compare if one test run
                    just contains a fraction of the time of the other, the
                    --end-time feature will ensure that the complete
                    comparision will fail