FFmpeg bug testing

this document will try to explain the process used by bugtesters and developers to find and fix bugs in ffmpeg.

the process could be used in the future to automate bug testing. or find holes in our FATE coverage.

example 1 - bug 2414 - wmv2 decoding artifacts

• https://ffmpeg.org/trac/ffmpeg/ticket/2414
• sample: http://samples.ffmpeg.org/asf-wmv/wmv2decerror-short.asf

initial process

1. check if the official decoder/program handles the sample. it could be a broken file.
   1. mplayer -vc wmvdmo sample
2. test sample with latest ffplay/ffmpeg.
   1. sometimes its already fixed.
   2. make sure its not an ffplay-only bug by also testing with ffmpeg.
3. check with old version of ffmpeg to see if it is a regression bug.
4. check if its a demuxer problem by testing with the ffmpeg/mplayer demuxer and official decoder.
   1. to test ffmpeg demuxer - mplayer -demuxer lavf -vc wmvdmo
   2. to test mplayer demuxer - mplayer -demuxer asf -vc wmvdmo

now that we know its not a demuxer problem, we can move onto debugging the decoder.

advanced codec debugging

1. test if codec cpu optimizations are not binary identical to the c-version of the codec
   1. use ffmpeg -cpuflags 0 , ffmpeg has to be compiled with runtime cpu detection for this to work.
2. test if codec asm version is not binary identical to the c-version of the codec
   1. recompile ffmpeg with --disable-asm

since codec also reports "dc overflow" error, check what the decoder does when it encounters this error. it is ffmpeg policy to report these errors, but also to continue decoding, in an effort to play as much of a file as possible.

in this case, msmpeg4dec.c , when encountering a dc overflow error, would return -1.

1. try commenting out this in the source code, recompile, and test your changes to see if the picture changes for the better.

testing for regressions

1. run make fate to check for any regressions in the code
2. test variety of samples relating to the code that has changed. make sure they decode visually and/or framemd5 identical.

results and conclusion

there seems to be no way to switch asm optimizations on or off at runtime. its useful to have a seperate ffmpeg binary compiled with --disable-asm to test with.

ffmpeg should have some tests to make sure the various arch, asm and cpu optimizations of each codec are binary identical where possible.

required programs for bug testing

bug testers should have the following installed:

1. latest git ffmpeg binary with asm and cpu runtime detection enabled
2. latest git ffmpeg binary with --disable-asm and cpu runtime detection
3. older version ffmpeg binaries with and without asm
4. official 3rd party players
   1. realplayer
   2. quicktime
   3. windows media player
5. mplayer version with binary codec support
6. vlc