Difference between revisions of "Running FATE"

From MultimediaWiki
Jump to navigation Jump to search
(Regular Cycles: note about syncing FATE samples via cron)
(Clarify Libav/FFmpeg)
 
(13 intermediate revisions by 6 users not shown)
Line 1: Line 1:
 +
'''This page is written for Libav. For instructions on running FATE test for FFmpeg, see [http://ffmpeg.org/fate.html here].'''
 +
 
== Get The Samples ==
 
== Get The Samples ==
 
Sync the FATE samples:
 
Sync the FATE samples:
  
   rsync -aL rsync://rsync.mplayerhq.hu:/samples/fate-suite/ /path/to/fate/samples
+
   rsync -WaL rsync://fate-suite.libav.org/fate-suite/ /path/to/fate/samples
 +
 
 +
Please ignore any "Permission denied" errors, permissions of recently uploaded files are fixed regularly by a cron job.
  
 
== Configuration ==
 
== Configuration ==
Line 8: Line 12:
  
 
  slot=                                  # some unique identifier
 
  slot=                                  # some unique identifier
  repo=svn://svn.ffmpeg.org/ffmpeg/trunk        # the source repository
+
  repo=git://git.libav.org/libav.git      # the source repository
 
  samples=/path/to/fate/samples
 
  samples=/path/to/fate/samples
  fate_recv="ssh fate@fate.ffmpeg.org"   # command to submit report
+
workdir=                                # directory in which to do all the work
 +
  fate_recv="ssh -T fate@fate.libav.org" # command to submit report
 +
comment=                                # optional description
 
   
 
   
 
  # the following are optional and map to configure options
 
  # the following are optional and map to configure options
Line 26: Line 32:
 
  extra_conf=    # extra configure options not covered above
 
  extra_conf=    # extra configure options not covered above
 
   
 
   
  make=           # name of GNU make if not 'make'
+
  #make=         # name of GNU make if not 'make'
  workdir=       # directory in which to do all the work
+
  makeopts=       # extra options passed to 'make'
 +
#tar=          # command to create a tar archive from its arguments on stdout, defaults to 'tar c'
  
 
This file is parsed by the shell, so any shell features may be used. Any site-specific settings can be made here as well, such as PATH to compilers etc.  One such file is required per configuration to be tested.
 
This file is parsed by the shell, so any shell features may be used. Any site-specific settings can be made here as well, such as PATH to compilers etc.  One such file is required per configuration to be tested.
  
The $slot value can be anything not already in use.  I have chosen names of the form arch-os-compiler-version for mine, and I suggest others follow a similar scheme.  Allowed characters are alphanumeric, underscore, period, and hyphen.
+
The $slot value can be anything not already in use.  Names of the form arch-os-compiler-version are suggested.  Allowed characters are alphanumeric, underscore, period, and hyphen.
  
 
== Get the Script ==
 
== Get the Script ==
Fetch the file 'tests/fate.sh' from the ffmpeg repo.  This script checks out the source, builds it, runs the tests, and submits the report to the server. I suggest you watch it for changes and update keep an up to date version, although I hope this file will be rather stable.
+
Fetch the file 'tests/fate.sh' from the libav repo.  This script checks out the source, builds it, runs the tests, and submits the report to the server. It's a good idea to watch this file for changes and update your running copy as necessary.
  
 
== Test Run ==
 
== Test Run ==
Line 51: Line 58:
 
Delete the version file to force a retest of the same revision.
 
Delete the version file to force a retest of the same revision.
  
== Submitting Resuts ==
+
== Submitting Results ==
When everything is working properly, create and ssh key, send it to the FATE system adminstrator, and await confirmation. Make sure the ssh command used in the fate_recv setting uses this key for authentication. You will also need to accept the host key of the server. It's fingerprint is
+
When everything is working properly, create an ssh key, send it to the FATE system adminstrator, and await confirmation. Make sure the ssh command used in the fate_recv setting uses this key for authentication. You will also need to accept the host key of the server. Its fingerprint is:
  
 
   a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6
 
   a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6
Line 66: Line 73:
  
 
Adjunct to running the FATE script from cron, it's also useful to rsync the FATE samples directory via cron. This doesn't need to be performed as frequently. Once per day is adequate.
 
Adjunct to running the FATE script from cron, it's also useful to rsync the FATE samples directory via cron. This doesn't need to be performed as frequently. Once per day is adequate.
 +
 +
[[Category:FATE]]

Latest revision as of 21:41, 5 June 2014

This page is written for Libav. For instructions on running FATE test for FFmpeg, see here.

Get The Samples

Sync the FATE samples:

 rsync -WaL rsync://fate-suite.libav.org/fate-suite/ /path/to/fate/samples

Please ignore any "Permission denied" errors, permissions of recently uploaded files are fixed regularly by a cron job.

Configuration

Create a configuration file setting the following variables:

slot=                                   # some unique identifier
repo=git://git.libav.org/libav.git      # the source repository
samples=/path/to/fate/samples
workdir=                                # directory in which to do all the work
fate_recv="ssh -T fate@fate.libav.org"  # command to submit report
comment=                                # optional description

# the following are optional and map to configure options
arch=
cpu=
cross_prefix=
cc=
target_os=
sysroot=
target_exec=
target_path=
extra_cflags=
extra_ldflags=
extra_libs=
extra_conf=     # extra configure options not covered above

#make=          # name of GNU make if not 'make'
makeopts=       # extra options passed to 'make'
#tar=           # command to create a tar archive from its arguments on stdout, defaults to 'tar c'

This file is parsed by the shell, so any shell features may be used. Any site-specific settings can be made here as well, such as PATH to compilers etc. One such file is required per configuration to be tested.

The $slot value can be anything not already in use. Names of the form arch-os-compiler-version are suggested. Allowed characters are alphanumeric, underscore, period, and hyphen.

Get the Script

Fetch the file 'tests/fate.sh' from the libav repo. This script checks out the source, builds it, runs the tests, and submits the report to the server. It's a good idea to watch this file for changes and update your running copy as necessary.

Test Run

Perform a test run by leaving $fate_recv unset in the configuration and issue the command "fate.sh /path/to/config/file". This should perform all the steps except the final report submission. In the end, $workdir should contain the following files:

  • configure.log
  • compile.log
  • test.log
  • report
  • version

After a successful run, the 'report' file should contain one line for each test and a couple of header lines. Most of the content is base64-encoded, so it will look like gibberish.

If any of the files are missing, or if the report seems too sparse, check for errors that may have occurred.

Delete the version file to force a retest of the same revision.

Submitting Results

When everything is working properly, create an ssh key, send it to the FATE system adminstrator, and await confirmation. Make sure the ssh command used in the fate_recv setting uses this key for authentication. You will also need to accept the host key of the server. Its fingerprint is:

 a4:99:d7:d3:1c:92:0d:56:d6:d5:61:be:01:ae:7d:e6

Accepting the host key is most easily done by running the ssh command directly from a shell once. When it connects, simply kill it. You will not get a shell prompt.

Regular Cycles

You are now ready to commence automated testing. Set up a cron job, shell script, or use whatever means you prefer to run the command from step 3 at regular intervals. If using multiple configurations, a simple shell wrapper iterating over them may be convenient.

The fate.sh script will refuse to run if the workdir is locked by the presence of a fate.lock file there. If something dies leaving this file behind, you must delete it manually.

Adjunct to running the FATE script from cron, it's also useful to rsync the FATE samples directory via cron. This doesn't need to be performed as frequently. Once per day is adequate.