MultimediaWiki - User contributions [en]

FFmpeg codec HOWTO

2020-04-25T04:45:45Z

Rogerdpack: up to date!

This page is meant as an introduction to the internal codec API in [[FFmpeg]].
It will also show how the codecs are connected with the demuxers. This is by
no means a complete guide but enough to understand how to add a codec to FFmpeg.
[[RealAudio cook|Cook]] is used as an example throughout.

== registering the codec ==

=== libavcodec/avcodec.h ===
The first thing to look at is the AVCodec struct.

typedef struct AVCodec {
const char *name;
enum CodecType type;
enum CodecID id;
int priv_data_size;
int (*init)(AVCodecContext *);
int (*encode)(AVCodecContext *, uint8_t *buf, int buf_size, void *data);
int (*close)(AVCodecContext *);
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size,
uint8_t *buf, int buf_size);
int capabilities;
struct AVCodec *next;
void (*flush)(AVCodecContext *);
const AVRational *supported_framerates; ///array of supported framerates, or NULL if any, array is terminated by {0,0}
const enum PixelFormat *pix_fmts; ///array of supported pixel formats, or NULL if unknown, array is terminanted by -1
} AVCodec;

Here we can see that we have some elements to name the codec, what type it is (audio/video), the supported pixel formats and some function
pointers for init/encode/decode and close. Now lets see how it is used.

=== libavcodec/cook.c ===
If we look in this file at the bottom we can see this code:

AVCodec cook_decoder =
{
.name = "cook",
.type = CODEC_TYPE_AUDIO,
.id = CODEC_ID_COOK,
.priv_data_size = sizeof(COOKContext),
.init = cook_decode_init,
.close = cook_decode_close,
.decode = cook_decode_frame,
};

First we get an AVCodec struct named cook_decoder. And then we set the variables of cook_decoder. Note that we only set the variables that are needed. Currently there is no encoder so we don't set any. If we now look at the id variable we can see that CODEC_ID_COOK isn't defined in libavcodec/cook.c. It is declared in avcodec.h.

=== libavcodec/avcodec.h ===

Here we will find the CodecID enumeration.

enum CodecID {
...
CODEC_ID_GSM,
CODEC_ID_QDM2,
CODEC_ID_COOK,
CODEC_ID_TRUESPEECH,
CODEC_ID_TTA,
...
};

CODEC_ID_COOK is there in the list. This is the list of all supported codecs in FFmpeg, the list is fixed and used internally to id every codec. Changing the order would break binary compatibility.

This is all enough to declare a codec. Now we must register them for internal use also. This is done at runtime.

=== libavcodec/allcodecs.c ===
In this file we have the avcodec_register_all() function, it has entries like this for all codecs.

...
extern AVCodec cook_decoder; // must end with _decoder
...

This is expanded at configure time into a REGISTER_CODEC(...)

configure accomplishes this by running something like:

sed -n 's/^[^#]*DEC.*, *$.*$).*/\1_decoder/p' libavcodec/allcodecs.c

So adding an entry in allcodecs.c and reconfigure is enough to add the needed define. Now we have everything to hookup a codec.

=== libavcodec/Makefile ===
In this file we define the objects on which a codec depends. For example, cook uses fft and mdct code so it depends on the mdct.o and fft.o object files as well as the cook.o object file.

...
OBJS-$(CONFIG_COOK_DECODER) += cook.o mdct.o fft.o
...

== FFmpeg demuxer connection ==

: ''[[FFmpeg demuxer howto]]''

=== libavformat/rm.c ===

If we think of an imaginary rm file that ffmpeg is about to process, the first thing that happens is that it is identified
as a rm file. It is passed on to the rm demuxer ([http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/rmdec.c?view=markup rmdec.c]). The rm demuxer looks through the file and finds out that it is a
cook file.

...
} else if (!strcmp(buf, "cook")) {
st->codec->codec_id = CODEC_ID_COOK;
...

Now ffmpeg knows what codec to init and where to send the payload from the container. So back to cook.c and the initialization process.

== codec code ==

=== libavcodec/cook.c Init ===
After ffmpeg knows what codec to use, it calls the declared initialization function pointer declared in the codecs AVCodec struct. In
cook.c it is called cook_decode_init. Here we setup as much as we can before we start decoding. The following things should be handled in the init, vlc table initialization, table generation, memory allocation and extradata parsing.

=== libavcodec/cook.c Close ===
The cook_decode_close function is the codec clean-up call. All memory, vlc tables, etc. should be freed here.

=== libavcodec/cook.c Decode ===
In cook.c the name of the decode call is cook_decode_frame.

static int cook_decode_frame(AVCodecContext *avctx,
void *data, int *data_size,
uint8_t *buf, int buf_size) {
...

The function has 5 arguments:
* avctx is a pointer to an AVCodecContext
* data is the pointer to the output buffer
* data_size is a variable that should be set to the output buffer size in bytes (this is usually the number of samples decoded * the number of channels * the byte size of a sample)
* buf is the pointer to the input buffer
* buf_size is the byte size of the input buffer

The decode function shall return the number of bytes consumed from the input buffer or -1 in case of an error. If there is no error during decoding, the return value is usually buf_size as buf should only contain one 'frame' of data. Bitstream parsers to split the bitstream into 'frames' used to be part of the codec so a call to the decode function could have consumed less than buf_size bytes from buf. It is now encouraged that bitstream parsers be separate.

That's how it works without too much detail.

=== The Glue codec template ===
The imaginary Glue audio codec will serve as a base to exhibit bitstream reading, vlc decoding and other things.
The code is purely fictional and is sometimes written purely for the sake of example. No attempt is made to prevent invalid
data manipulation.

The Glue codec follows.

[http://wiki.multimedia.cx/index.php?title=FFmpeg_codec_howto&oldid=7347 non-colored version]

/* The following includes have the bitstream reader, various dsp functions and the various defaults */
#define ALT_BITSTREAM_READER
#include "avcodec.h"
#include "bitstream.h"
#include "dsputil.h"

/* This includes the tables needed for the Glue codec template */
#include "gluedata.h"

/* Here we declare the struct used for the codec private data */
typedef struct {
 GetBitContext gb;
 FFTContext fft_ctx;
 VLC vlc_table;
 MDCTContext mdct_ctx;
 float* sample_buffer;
} GLUEContext;

/* The init function */
static int glue_decode_init(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* This imaginary codec uses one fft, one mdct and one vlc table. */
 ff_mdct_init(&q->mdct_ctx, 10, 1); // 2^10 == size of mdct, 1 == inverse mdct
 ff_fft_init(&q->fft_ctx, 9, 1); // 2^9 == size of fft, 0 == inverse fft
 init_vlc (&q->vlc_table, 9, 24,
 vlctable_huffbits, 1, 1,
 vlctable_huffcodes, 2, 2, 0); // look in bitstream.h for the meaning of the arguments

 /* We also need to allocate a sample buffer */
 q->sample_buffer = av_mallocz(sizeof(float)*1024); // here we used av_mallocz instead of av_malloc
 // av_mallocz memsets the whole buffer to 0

 /* Check if the allocation was successful */
 if(q->sample_buffer == NULL)
 return -1;

 /* return 0 for a successful init, -1 for failure */
 return 0;
}

/* This is the main decode function */
static int glue_decode_frame(AVCodecContext *avctx,
 void *data, int *data_size,
 uint8_t *buf, int buf_size)
{
 GLUEContext *q = avctx->priv_data;
 int16_t *outbuffer = data;

 /* We know what the arguments for this function are from above
 now we just have to decode this imaginary codec, the made up
 bitstream format is as follows:
 12 bits representing the amount of samples
 1 bit fft or mdct coded coeffs, 0 for fft/1 for mdct
 read 13 bits representing the amount of vlc coded fft data coeffs
 read 10 bits representing the amount of vlc coded mdct data coeffs
 (...bits representing the coeffs...)
 5 bits of dummy data that should be ignored
 32 bits the hex value 0x12345678, used for integrity check
 */

 /* Declare the needed variables */
 int samples, coeffs, i, fft;
 float mdct_tmp[1024];

 /* Now we init the bitstream reader, we start at the beginning of the inbuffer */
 init_get_bits(&q->gb, buf, buf_size*8); //the buf_size is in bytes but we need bits

 /* Now we take 12 bits to get the amount of samples the current frame has */
 samples = get_bits(&q->gb, 12);
 
 /* Now we check if we have fft or mdct coeffs */
 fft = get_bits1(&q->gb);
 if (fft) {
 //fft coeffs, get how many
 coeffs = get_bits(&q->gb, 13);
 } else {
 //mdct coeffs, get how many
 coeffs = get_bits(&q->gb, 10);
 }

 /* Now decode the vlc coded coeffs to the sample_buffer */
 for (i=0 ; i<coeffs ; i++)
 q->sample_buffer[i] = get_vlc2(&q->gb, q->vlc_table.table, vlc_table.bits, 3); //read about the arguments in bitstream.h

 /* Now we need to transform the coeffs to samples */
 if (fft) {
 //The fft is done inplace
 ff_fft_permute(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 ff_fft_calc(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 } else {
 //And we pretend that the mdct is also inplace
 ff_imdct_calc(&q->mdct_ctx, q->sample_buffer, q->sample_buffer, mdct_tmp);
 }

 /* To make it easy the stream can only be 16 bits mono, so let's convert it to that */
 for (i=0 ; i<samples ; i++)
 outbuffer[i] = (int16_t)q->sample_buffer[i];

 /* Report how many samples we got */
 *data_size = samples;

 /* Skip the dummy data bits */
 skip_bits(&q->gb, 5);

 /* Check if the buffer was consumed ok */
 if (get_bits(&q->gb,32) != 0x12345678) {
 av_log(avctx,AV_LOG_ERROR,"Stream error, integrity check failed!\n");
 return -1;
 }

/* The decision between erroring out or not in case of unexpected data
should be made so that the output quality is maximized.
This means that if undamaged data is assumed then unused/resereved values
should lead to warnings but not failure. (assumption of slightly non compliant
file)
OTOH if possibly damaged data is assumed and it is assumed that the original
did contain specific values in reserved/unused fields then finding unexpected
values should trigger error concealment code and the decoder/demuxer should
attempt to resync.
The decision between these 2 should be made by using
AVCodecContext.error_recognition unless its a clear case where only one of
the 2 makes sense.
*/


 /* Return the amount of bytes consumed if everything was ok */
 return *data_size*sizeof(int16_t);
}

/* the uninit function, here we just do the inverse of the init */ 
static int glue_decode_close(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* Free allocated memory buffer */
 av_free(q->sample_buffer);

 /* Free the fft transform */
 ff_fft_end(&q->fft_ctx);

 /* Free the mdct transform */
 ff_mdct_end(&q->mdct_ctx);

 /* Free the vlc table */
 free_vlc(&q->vlc_table);

 /* Return 0 if everything is ok, -1 if not */
 return 0;
}

AVCodec glue_decoder =
{
 .name = "glue",
 .type = CODEC_TYPE_AUDIO,
 .id = CODEC_ID_GLUE,
 .priv_data_size = sizeof(GLUEContext),
 .init = glue_decode_init,
 .close = glue_decode_close,
 .decode = glue_decode_frame,
};

=== trouble shooting ===
Invalid pixel format '(null)'
Output pad "default" with type video of the filter instance "Parsed_null_0" of null not connected to any destination
Error opening filters!

meant encoder's description
.pix_fmts needs to end with AV_PIX_FMT_NONE

Assertion *(const AVClass **)avctx->priv_data == codec->priv_class failed at libavcodec/utils.c:1554

meant "your first member of your Context struct needs to be an AVClass *"

if it works but just says ""0 packets muxed"" you need to set the *got_packet return value to "1" or the muxer won't mux what you're giving it

== see also ==

* [[FFmpeg demuxer howto]]
* [[FFmpeg programming conventions]]
* [[FFmpeg_filter_HOWTO]]

[[Category:FFmpeg Tutorials]]
[[Category:Tutorials]]

FFmpeg codec HOWTO

2015-10-13T23:50:10Z

Rogerdpack: /* trouble shooting */

This page is meant as an introduction to the internal codec API in [[FFmpeg]].
It will also show how the codecs are connected with the demuxers. This is by
no means a complete guide but enough to understand how to add a codec to FFmpeg.
[[RealAudio cook|Cook]] is used as an example throughout.

== registering the codec ==

=== libavcodec/avcodec.h ===
The first thing to look at is the AVCodec struct.

typedef struct AVCodec {
const char *name;
enum CodecType type;
enum CodecID id;
int priv_data_size;
int (*init)(AVCodecContext *);
int (*encode)(AVCodecContext *, uint8_t *buf, int buf_size, void *data);
int (*close)(AVCodecContext *);
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size,
uint8_t *buf, int buf_size);
int capabilities;
struct AVCodec *next;
void (*flush)(AVCodecContext *);
const AVRational *supported_framerates; ///array of supported framerates, or NULL if any, array is terminated by {0,0}
const enum PixelFormat *pix_fmts; ///array of supported pixel formats, or NULL if unknown, array is terminanted by -1
} AVCodec;

Here we can see that we have some elements to name the codec, what type it is (audio/video), the supported pixel formats and some function
pointers for init/encode/decode and close. Now lets see how it is used.

=== libavcodec/cook.c ===
If we look in this file at the bottom we can see this code:

AVCodec cook_decoder =
{
.name = "cook",
.type = CODEC_TYPE_AUDIO,
.id = CODEC_ID_COOK,
.priv_data_size = sizeof(COOKContext),
.init = cook_decode_init,
.close = cook_decode_close,
.decode = cook_decode_frame,
};

First we get an AVCodec struct named cook_decoder. And then we set the variables of cook_decoder. Note that we only set the variables that are needed. Currently there is no encoder so we don't set any. If we now look at the id variable we can see that CODEC_ID_COOK isn't defined in libavcodec/cook.c. It is declared in avcodec.h.

=== libavcodec/avcodec.h ===

Here we will find the CodecID enumeration.

enum CodecID {
...
CODEC_ID_GSM,
CODEC_ID_QDM2,
CODEC_ID_COOK,
CODEC_ID_TRUESPEECH,
CODEC_ID_TTA,
...
};

CODEC_ID_COOK is there in the list. This is the list of all supported codecs in FFmpeg, the list is fixed and used internally to id every codec. Changing the order would break binary compatibility.

This is all enough to declare a codec. Now we must register them for internal use also. This is done at runtime.

=== libavcodec/allcodecs.c ===
In this file we have the avcodec_register_all() function, it has entries like this for all codecs.

...
REGISTER_DECODER(COOK, cook);
...

This macro expands to a register_avcodec() call which registers a codec for internal use.
Note that register_avcodec() will only be called when CONFIG_COOK_DECODER is defined.
This allows to not compile the decoder code for a specific codec.
But where is it defined? This is extracted by configure with this command line:

sed -n 's/^[^#]*DEC.*, *$.*$).*/\1_decoder/p' libavcodec/allcodecs.c

So adding a REGISTER_DECODER(NEW, new) entry in allcodecs.c and reconfigure is enough to add the needed define. Now we have everything to hookup a codec.

=== libavcodec/Makefile ===
In this file we define the objects on which a codec depends. For example, cook uses fft and mdct code so it depends on the mdct.o and fft.o object files as well as the cook.o object file.

...
OBJS-$(CONFIG_COOK_DECODER) += cook.o mdct.o fft.o
...

== FFmpeg demuxer connection ==

: ''[[FFmpeg demuxer howto]]''

=== libavformat/rm.c ===

If we think of an imaginary rm file that ffmpeg is about to process, the first thing that happens is that it is identified
as a rm file. It is passed on to the rm demuxer ([http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/rmdec.c?view=markup rmdec.c]). The rm demuxer looks through the file and finds out that it is a
cook file.

...
} else if (!strcmp(buf, "cook")) {
st->codec->codec_id = CODEC_ID_COOK;
...

Now ffmpeg knows what codec to init and where to send the payload from the container. So back to cook.c and the initialization process.

== codec code ==

=== libavcodec/cook.c Init ===
After ffmpeg knows what codec to use, it calls the declared initialization function pointer declared in the codecs AVCodec struct. In
cook.c it is called cook_decode_init. Here we setup as much as we can before we start decoding. The following things should be handled in the init, vlc table initialization, table generation, memory allocation and extradata parsing.

=== libavcodec/cook.c Close ===
The cook_decode_close function is the codec clean-up call. All memory, vlc tables, etc. should be freed here.

=== libavcodec/cook.c Decode ===
In cook.c the name of the decode call is cook_decode_frame.

static int cook_decode_frame(AVCodecContext *avctx,
void *data, int *data_size,
uint8_t *buf, int buf_size) {
...

The function has 5 arguments:
* avctx is a pointer to an AVCodecContext
* data is the pointer to the output buffer
* data_size is a variable that should be set to the output buffer size in bytes (this is usually the number of samples decoded * the number of channels * the byte size of a sample)
* buf is the pointer to the input buffer
* buf_size is the byte size of the input buffer

The decode function shall return the number of bytes consumed from the input buffer or -1 in case of an error. If there is no error during decoding, the return value is usually buf_size as buf should only contain one 'frame' of data. Bitstream parsers to split the bitstream into 'frames' used to be part of the codec so a call to the decode function could have consumed less than buf_size bytes from buf. It is now encouraged that bitstream parsers be separate.

That's how it works without too much detail.

=== The Glue codec template ===
The imaginary Glue audio codec will serve as a base to exhibit bitstream reading, vlc decoding and other things.
The code is purely fictional and is sometimes written purely for the sake of example. No attempt is made to prevent invalid
data manipulation.

The Glue codec follows.

[http://wiki.multimedia.cx/index.php?title=FFmpeg_codec_howto&oldid=7347 non-colored version]

/* The following includes have the bitstream reader, various dsp functions and the various defaults */
#define ALT_BITSTREAM_READER
#include "avcodec.h"
#include "bitstream.h"
#include "dsputil.h"

/* This includes the tables needed for the Glue codec template */
#include "gluedata.h"

/* Here we declare the struct used for the codec private data */
typedef struct {
 GetBitContext gb;
 FFTContext fft_ctx;
 VLC vlc_table;
 MDCTContext mdct_ctx;
 float* sample_buffer;
} GLUEContext;

/* The init function */
static int glue_decode_init(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* This imaginary codec uses one fft, one mdct and one vlc table. */
 ff_mdct_init(&q->mdct_ctx, 10, 1); // 2^10 == size of mdct, 1 == inverse mdct
 ff_fft_init(&q->fft_ctx, 9, 1); // 2^9 == size of fft, 0 == inverse fft
 init_vlc (&q->vlc_table, 9, 24,
 vlctable_huffbits, 1, 1,
 vlctable_huffcodes, 2, 2, 0); // look in bitstream.h for the meaning of the arguments

 /* We also need to allocate a sample buffer */
 q->sample_buffer = av_mallocz(sizeof(float)*1024); // here we used av_mallocz instead of av_malloc
 // av_mallocz memsets the whole buffer to 0

 /* Check if the allocation was successful */
 if(q->sample_buffer == NULL)
 return -1;

 /* return 0 for a successful init, -1 for failure */
 return 0;
}

/* This is the main decode function */
static int glue_decode_frame(AVCodecContext *avctx,
 void *data, int *data_size,
 uint8_t *buf, int buf_size)
{
 GLUEContext *q = avctx->priv_data;
 int16_t *outbuffer = data;

 /* We know what the arguments for this function are from above
 now we just have to decode this imaginary codec, the made up
 bitstream format is as follows:
 12 bits representing the amount of samples
 1 bit fft or mdct coded coeffs, 0 for fft/1 for mdct
 read 13 bits representing the amount of vlc coded fft data coeffs
 read 10 bits representing the amount of vlc coded mdct data coeffs
 (...bits representing the coeffs...)
 5 bits of dummy data that should be ignored
 32 bits the hex value 0x12345678, used for integrity check
 */

 /* Declare the needed variables */
 int samples, coeffs, i, fft;
 float mdct_tmp[1024];

 /* Now we init the bitstream reader, we start at the beginning of the inbuffer */
 init_get_bits(&q->gb, buf, buf_size*8); //the buf_size is in bytes but we need bits

 /* Now we take 12 bits to get the amount of samples the current frame has */
 samples = get_bits(&q->gb, 12);
 
 /* Now we check if we have fft or mdct coeffs */
 fft = get_bits1(&q->gb);
 if (fft) {
 //fft coeffs, get how many
 coeffs = get_bits(&q->gb, 13);
 } else {
 //mdct coeffs, get how many
 coeffs = get_bits(&q->gb, 10);
 }

 /* Now decode the vlc coded coeffs to the sample_buffer */
 for (i=0 ; i<coeffs ; i++)
 q->sample_buffer[i] = get_vlc2(&q->gb, q->vlc_table.table, vlc_table.bits, 3); //read about the arguments in bitstream.h

 /* Now we need to transform the coeffs to samples */
 if (fft) {
 //The fft is done inplace
 ff_fft_permute(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 ff_fft_calc(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 } else {
 //And we pretend that the mdct is also inplace
 ff_imdct_calc(&q->mdct_ctx, q->sample_buffer, q->sample_buffer, mdct_tmp);
 }

 /* To make it easy the stream can only be 16 bits mono, so let's convert it to that */
 for (i=0 ; i<samples ; i++)
 outbuffer[i] = (int16_t)q->sample_buffer[i];

 /* Report how many samples we got */
 *data_size = samples;

 /* Skip the dummy data bits */
 skip_bits(&q->gb, 5);

 /* Check if the buffer was consumed ok */
 if (get_bits(&q->gb,32) != 0x12345678) {
 av_log(avctx,AV_LOG_ERROR,"Stream error, integrity check failed!\n");
 return -1;
 }

/* The decision between erroring out or not in case of unexpected data
should be made so that the output quality is maximized.
This means that if undamaged data is assumed then unused/resereved values
should lead to warnings but not failure. (assumption of slightly non compliant
file)
OTOH if possibly damaged data is assumed and it is assumed that the original
did contain specific values in reserved/unused fields then finding unexpected
values should trigger error concealment code and the decoder/demuxer should
attempt to resync.
The decision between these 2 should be made by using
AVCodecContext.error_recognition unless its a clear case where only one of
the 2 makes sense.
*/


 /* Return the amount of bytes consumed if everything was ok */
 return *data_size*sizeof(int16_t);
}

/* the uninit function, here we just do the inverse of the init */ 
static int glue_decode_close(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* Free allocated memory buffer */
 av_free(q->sample_buffer);

 /* Free the fft transform */
 ff_fft_end(&q->fft_ctx);

 /* Free the mdct transform */
 ff_mdct_end(&q->mdct_ctx);

 /* Free the vlc table */
 free_vlc(&q->vlc_table);

 /* Return 0 if everything is ok, -1 if not */
 return 0;
}

AVCodec glue_decoder =
{
 .name = "glue",
 .type = CODEC_TYPE_AUDIO,
 .id = CODEC_ID_GLUE,
 .priv_data_size = sizeof(GLUEContext),
 .init = glue_decode_init,
 .close = glue_decode_close,
 .decode = glue_decode_frame,
};

=== trouble shooting ===
Invalid pixel format '(null)'
Output pad "default" with type video of the filter instance "Parsed_null_0" of null not connected to any destination
Error opening filters!

meant encoder's description
.pix_fmts needs to end with AV_PIX_FMT_NONE

Assertion *(const AVClass **)avctx->priv_data == codec->priv_class failed at libavcodec/utils.c:1554

meant "your first member of your Context struct needs to be an AVClass *"

if it works but just says ""0 packets muxed"" you need to set the *got_packet return value to "1" or the muxer won't mux what you're giving it

== see also ==

* [[FFmpeg demuxer howto]]
* [[FFmpeg programming conventions]]
* [[FFmpeg_filter_HOWTO]]

[[Category:FFmpeg Tutorials]]
[[Category:Tutorials]]

FFmpeg codec HOWTO

2015-10-13T21:15:53Z

Rogerdpack: add link

This page is meant as an introduction to the internal codec API in [[FFmpeg]].
It will also show how the codecs are connected with the demuxers. This is by
no means a complete guide but enough to understand how to add a codec to FFmpeg.
[[RealAudio cook|Cook]] is used as an example throughout.

== registering the codec ==

=== libavcodec/avcodec.h ===
The first thing to look at is the AVCodec struct.

typedef struct AVCodec {
const char *name;
enum CodecType type;
enum CodecID id;
int priv_data_size;
int (*init)(AVCodecContext *);
int (*encode)(AVCodecContext *, uint8_t *buf, int buf_size, void *data);
int (*close)(AVCodecContext *);
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size,
uint8_t *buf, int buf_size);
int capabilities;
struct AVCodec *next;
void (*flush)(AVCodecContext *);
const AVRational *supported_framerates; ///array of supported framerates, or NULL if any, array is terminated by {0,0}
const enum PixelFormat *pix_fmts; ///array of supported pixel formats, or NULL if unknown, array is terminanted by -1
} AVCodec;

Here we can see that we have some elements to name the codec, what type it is (audio/video), the supported pixel formats and some function
pointers for init/encode/decode and close. Now lets see how it is used.

=== libavcodec/cook.c ===
If we look in this file at the bottom we can see this code:

AVCodec cook_decoder =
{
.name = "cook",
.type = CODEC_TYPE_AUDIO,
.id = CODEC_ID_COOK,
.priv_data_size = sizeof(COOKContext),
.init = cook_decode_init,
.close = cook_decode_close,
.decode = cook_decode_frame,
};

First we get an AVCodec struct named cook_decoder. And then we set the variables of cook_decoder. Note that we only set the variables that are needed. Currently there is no encoder so we don't set any. If we now look at the id variable we can see that CODEC_ID_COOK isn't defined in libavcodec/cook.c. It is declared in avcodec.h.

=== libavcodec/avcodec.h ===

Here we will find the CodecID enumeration.

enum CodecID {
...
CODEC_ID_GSM,
CODEC_ID_QDM2,
CODEC_ID_COOK,
CODEC_ID_TRUESPEECH,
CODEC_ID_TTA,
...
};

CODEC_ID_COOK is there in the list. This is the list of all supported codecs in FFmpeg, the list is fixed and used internally to id every codec. Changing the order would break binary compatibility.

This is all enough to declare a codec. Now we must register them for internal use also. This is done at runtime.

=== libavcodec/allcodecs.c ===
In this file we have the avcodec_register_all() function, it has entries like this for all codecs.

...
REGISTER_DECODER(COOK, cook);
...

This macro expands to a register_avcodec() call which registers a codec for internal use.
Note that register_avcodec() will only be called when CONFIG_COOK_DECODER is defined.
This allows to not compile the decoder code for a specific codec.
But where is it defined? This is extracted by configure with this command line:

sed -n 's/^[^#]*DEC.*, *$.*$).*/\1_decoder/p' libavcodec/allcodecs.c

So adding a REGISTER_DECODER(NEW, new) entry in allcodecs.c and reconfigure is enough to add the needed define. Now we have everything to hookup a codec.

=== libavcodec/Makefile ===
In this file we define the objects on which a codec depends. For example, cook uses fft and mdct code so it depends on the mdct.o and fft.o object files as well as the cook.o object file.

...
OBJS-$(CONFIG_COOK_DECODER) += cook.o mdct.o fft.o
...

== FFmpeg demuxer connection ==

: ''[[FFmpeg demuxer howto]]''

=== libavformat/rm.c ===

If we think of an imaginary rm file that ffmpeg is about to process, the first thing that happens is that it is identified
as a rm file. It is passed on to the rm demuxer ([http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/rmdec.c?view=markup rmdec.c]). The rm demuxer looks through the file and finds out that it is a
cook file.

...
} else if (!strcmp(buf, "cook")) {
st->codec->codec_id = CODEC_ID_COOK;
...

Now ffmpeg knows what codec to init and where to send the payload from the container. So back to cook.c and the initialization process.

== codec code ==

=== libavcodec/cook.c Init ===
After ffmpeg knows what codec to use, it calls the declared initialization function pointer declared in the codecs AVCodec struct. In
cook.c it is called cook_decode_init. Here we setup as much as we can before we start decoding. The following things should be handled in the init, vlc table initialization, table generation, memory allocation and extradata parsing.

=== libavcodec/cook.c Close ===
The cook_decode_close function is the codec clean-up call. All memory, vlc tables, etc. should be freed here.

=== libavcodec/cook.c Decode ===
In cook.c the name of the decode call is cook_decode_frame.

static int cook_decode_frame(AVCodecContext *avctx,
void *data, int *data_size,
uint8_t *buf, int buf_size) {
...

The function has 5 arguments:
* avctx is a pointer to an AVCodecContext
* data is the pointer to the output buffer
* data_size is a variable that should be set to the output buffer size in bytes (this is usually the number of samples decoded * the number of channels * the byte size of a sample)
* buf is the pointer to the input buffer
* buf_size is the byte size of the input buffer

The decode function shall return the number of bytes consumed from the input buffer or -1 in case of an error. If there is no error during decoding, the return value is usually buf_size as buf should only contain one 'frame' of data. Bitstream parsers to split the bitstream into 'frames' used to be part of the codec so a call to the decode function could have consumed less than buf_size bytes from buf. It is now encouraged that bitstream parsers be separate.

That's how it works without too much detail.

=== The Glue codec template ===
The imaginary Glue audio codec will serve as a base to exhibit bitstream reading, vlc decoding and other things.
The code is purely fictional and is sometimes written purely for the sake of example. No attempt is made to prevent invalid
data manipulation.

The Glue codec follows.

[http://wiki.multimedia.cx/index.php?title=FFmpeg_codec_howto&oldid=7347 non-colored version]

/* The following includes have the bitstream reader, various dsp functions and the various defaults */
#define ALT_BITSTREAM_READER
#include "avcodec.h"
#include "bitstream.h"
#include "dsputil.h"

/* This includes the tables needed for the Glue codec template */
#include "gluedata.h"

/* Here we declare the struct used for the codec private data */
typedef struct {
 GetBitContext gb;
 FFTContext fft_ctx;
 VLC vlc_table;
 MDCTContext mdct_ctx;
 float* sample_buffer;
} GLUEContext;

/* The init function */
static int glue_decode_init(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* This imaginary codec uses one fft, one mdct and one vlc table. */
 ff_mdct_init(&q->mdct_ctx, 10, 1); // 2^10 == size of mdct, 1 == inverse mdct
 ff_fft_init(&q->fft_ctx, 9, 1); // 2^9 == size of fft, 0 == inverse fft
 init_vlc (&q->vlc_table, 9, 24,
 vlctable_huffbits, 1, 1,
 vlctable_huffcodes, 2, 2, 0); // look in bitstream.h for the meaning of the arguments

 /* We also need to allocate a sample buffer */
 q->sample_buffer = av_mallocz(sizeof(float)*1024); // here we used av_mallocz instead of av_malloc
 // av_mallocz memsets the whole buffer to 0

 /* Check if the allocation was successful */
 if(q->sample_buffer == NULL)
 return -1;

 /* return 0 for a successful init, -1 for failure */
 return 0;
}

/* This is the main decode function */
static int glue_decode_frame(AVCodecContext *avctx,
 void *data, int *data_size,
 uint8_t *buf, int buf_size)
{
 GLUEContext *q = avctx->priv_data;
 int16_t *outbuffer = data;

 /* We know what the arguments for this function are from above
 now we just have to decode this imaginary codec, the made up
 bitstream format is as follows:
 12 bits representing the amount of samples
 1 bit fft or mdct coded coeffs, 0 for fft/1 for mdct
 read 13 bits representing the amount of vlc coded fft data coeffs
 read 10 bits representing the amount of vlc coded mdct data coeffs
 (...bits representing the coeffs...)
 5 bits of dummy data that should be ignored
 32 bits the hex value 0x12345678, used for integrity check
 */

 /* Declare the needed variables */
 int samples, coeffs, i, fft;
 float mdct_tmp[1024];

 /* Now we init the bitstream reader, we start at the beginning of the inbuffer */
 init_get_bits(&q->gb, buf, buf_size*8); //the buf_size is in bytes but we need bits

 /* Now we take 12 bits to get the amount of samples the current frame has */
 samples = get_bits(&q->gb, 12);
 
 /* Now we check if we have fft or mdct coeffs */
 fft = get_bits1(&q->gb);
 if (fft) {
 //fft coeffs, get how many
 coeffs = get_bits(&q->gb, 13);
 } else {
 //mdct coeffs, get how many
 coeffs = get_bits(&q->gb, 10);
 }

 /* Now decode the vlc coded coeffs to the sample_buffer */
 for (i=0 ; i<coeffs ; i++)
 q->sample_buffer[i] = get_vlc2(&q->gb, q->vlc_table.table, vlc_table.bits, 3); //read about the arguments in bitstream.h

 /* Now we need to transform the coeffs to samples */
 if (fft) {
 //The fft is done inplace
 ff_fft_permute(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 ff_fft_calc(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 } else {
 //And we pretend that the mdct is also inplace
 ff_imdct_calc(&q->mdct_ctx, q->sample_buffer, q->sample_buffer, mdct_tmp);
 }

 /* To make it easy the stream can only be 16 bits mono, so let's convert it to that */
 for (i=0 ; i<samples ; i++)
 outbuffer[i] = (int16_t)q->sample_buffer[i];

 /* Report how many samples we got */
 *data_size = samples;

 /* Skip the dummy data bits */
 skip_bits(&q->gb, 5);

 /* Check if the buffer was consumed ok */
 if (get_bits(&q->gb,32) != 0x12345678) {
 av_log(avctx,AV_LOG_ERROR,"Stream error, integrity check failed!\n");
 return -1;
 }

/* The decision between erroring out or not in case of unexpected data
should be made so that the output quality is maximized.
This means that if undamaged data is assumed then unused/resereved values
should lead to warnings but not failure. (assumption of slightly non compliant
file)
OTOH if possibly damaged data is assumed and it is assumed that the original
did contain specific values in reserved/unused fields then finding unexpected
values should trigger error concealment code and the decoder/demuxer should
attempt to resync.
The decision between these 2 should be made by using
AVCodecContext.error_recognition unless its a clear case where only one of
the 2 makes sense.
*/


 /* Return the amount of bytes consumed if everything was ok */
 return *data_size*sizeof(int16_t);
}

/* the uninit function, here we just do the inverse of the init */ 
static int glue_decode_close(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* Free allocated memory buffer */
 av_free(q->sample_buffer);

 /* Free the fft transform */
 ff_fft_end(&q->fft_ctx);

 /* Free the mdct transform */
 ff_mdct_end(&q->mdct_ctx);

 /* Free the vlc table */
 free_vlc(&q->vlc_table);

 /* Return 0 if everything is ok, -1 if not */
 return 0;
}

AVCodec glue_decoder =
{
 .name = "glue",
 .type = CODEC_TYPE_AUDIO,
 .id = CODEC_ID_GLUE,
 .priv_data_size = sizeof(GLUEContext),
 .init = glue_decode_init,
 .close = glue_decode_close,
 .decode = glue_decode_frame,
};

=== trouble shooting ===
Invalid pixel format '(null)'
Output pad "default" with type video of the filter instance "Parsed_null_0" of null not connected to any destination
Error opening filters!

meant encoder's description
.pix_fmts needs to end with AV_PIX_FMT_NONE

Assertion *(const AVClass **)avctx->priv_data == codec->priv_class failed at libavcodec/utils.c:1554

meant "your first member of your Context struct needs to be an AVClass *"

== see also ==

* [[FFmpeg demuxer howto]]
* [[FFmpeg programming conventions]]
* [[FFmpeg_filter_HOWTO]]

[[Category:FFmpeg Tutorials]]
[[Category:Tutorials]]

FFmpeg codec HOWTO

2015-10-13T21:06:10Z

Rogerdpack: /* trouble shooting */

This page is meant as an introduction to the internal codec API in [[FFmpeg]].
It will also show how the codecs are connected with the demuxers. This is by
no means a complete guide but enough to understand how to add a codec to FFmpeg.
[[RealAudio cook|Cook]] is used as an example throughout.

== registering the codec ==

=== libavcodec/avcodec.h ===
The first thing to look at is the AVCodec struct.

typedef struct AVCodec {
const char *name;
enum CodecType type;
enum CodecID id;
int priv_data_size;
int (*init)(AVCodecContext *);
int (*encode)(AVCodecContext *, uint8_t *buf, int buf_size, void *data);
int (*close)(AVCodecContext *);
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size,
uint8_t *buf, int buf_size);
int capabilities;
struct AVCodec *next;
void (*flush)(AVCodecContext *);
const AVRational *supported_framerates; ///array of supported framerates, or NULL if any, array is terminated by {0,0}
const enum PixelFormat *pix_fmts; ///array of supported pixel formats, or NULL if unknown, array is terminanted by -1
} AVCodec;

Here we can see that we have some elements to name the codec, what type it is (audio/video), the supported pixel formats and some function
pointers for init/encode/decode and close. Now lets see how it is used.

=== libavcodec/cook.c ===
If we look in this file at the bottom we can see this code:

AVCodec cook_decoder =
{
.name = "cook",
.type = CODEC_TYPE_AUDIO,
.id = CODEC_ID_COOK,
.priv_data_size = sizeof(COOKContext),
.init = cook_decode_init,
.close = cook_decode_close,
.decode = cook_decode_frame,
};

First we get an AVCodec struct named cook_decoder. And then we set the variables of cook_decoder. Note that we only set the variables that are needed. Currently there is no encoder so we don't set any. If we now look at the id variable we can see that CODEC_ID_COOK isn't defined in libavcodec/cook.c. It is declared in avcodec.h.

=== libavcodec/avcodec.h ===

Here we will find the CodecID enumeration.

enum CodecID {
...
CODEC_ID_GSM,
CODEC_ID_QDM2,
CODEC_ID_COOK,
CODEC_ID_TRUESPEECH,
CODEC_ID_TTA,
...
};

CODEC_ID_COOK is there in the list. This is the list of all supported codecs in FFmpeg, the list is fixed and used internally to id every codec. Changing the order would break binary compatibility.

This is all enough to declare a codec. Now we must register them for internal use also. This is done at runtime.

=== libavcodec/allcodecs.c ===
In this file we have the avcodec_register_all() function, it has entries like this for all codecs.

...
REGISTER_DECODER(COOK, cook);
...

This macro expands to a register_avcodec() call which registers a codec for internal use.
Note that register_avcodec() will only be called when CONFIG_COOK_DECODER is defined.
This allows to not compile the decoder code for a specific codec.
But where is it defined? This is extracted by configure with this command line:

sed -n 's/^[^#]*DEC.*, *$.*$).*/\1_decoder/p' libavcodec/allcodecs.c

So adding a REGISTER_DECODER(NEW, new) entry in allcodecs.c and reconfigure is enough to add the needed define. Now we have everything to hookup a codec.

=== libavcodec/Makefile ===
In this file we define the objects on which a codec depends. For example, cook uses fft and mdct code so it depends on the mdct.o and fft.o object files as well as the cook.o object file.

...
OBJS-$(CONFIG_COOK_DECODER) += cook.o mdct.o fft.o
...

== FFmpeg demuxer connection ==

: ''[[FFmpeg demuxer howto]]''

=== libavformat/rm.c ===

If we think of an imaginary rm file that ffmpeg is about to process, the first thing that happens is that it is identified
as a rm file. It is passed on to the rm demuxer ([http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/rmdec.c?view=markup rmdec.c]). The rm demuxer looks through the file and finds out that it is a
cook file.

...
} else if (!strcmp(buf, "cook")) {
st->codec->codec_id = CODEC_ID_COOK;
...

Now ffmpeg knows what codec to init and where to send the payload from the container. So back to cook.c and the initialization process.

== codec code ==

=== libavcodec/cook.c Init ===
After ffmpeg knows what codec to use, it calls the declared initialization function pointer declared in the codecs AVCodec struct. In
cook.c it is called cook_decode_init. Here we setup as much as we can before we start decoding. The following things should be handled in the init, vlc table initialization, table generation, memory allocation and extradata parsing.

=== libavcodec/cook.c Close ===
The cook_decode_close function is the codec clean-up call. All memory, vlc tables, etc. should be freed here.

=== libavcodec/cook.c Decode ===
In cook.c the name of the decode call is cook_decode_frame.

static int cook_decode_frame(AVCodecContext *avctx,
void *data, int *data_size,
uint8_t *buf, int buf_size) {
...

The function has 5 arguments:
* avctx is a pointer to an AVCodecContext
* data is the pointer to the output buffer
* data_size is a variable that should be set to the output buffer size in bytes (this is usually the number of samples decoded * the number of channels * the byte size of a sample)
* buf is the pointer to the input buffer
* buf_size is the byte size of the input buffer

The decode function shall return the number of bytes consumed from the input buffer or -1 in case of an error. If there is no error during decoding, the return value is usually buf_size as buf should only contain one 'frame' of data. Bitstream parsers to split the bitstream into 'frames' used to be part of the codec so a call to the decode function could have consumed less than buf_size bytes from buf. It is now encouraged that bitstream parsers be separate.

That's how it works without too much detail.

=== The Glue codec template ===
The imaginary Glue audio codec will serve as a base to exhibit bitstream reading, vlc decoding and other things.
The code is purely fictional and is sometimes written purely for the sake of example. No attempt is made to prevent invalid
data manipulation.

The Glue codec follows.

[http://wiki.multimedia.cx/index.php?title=FFmpeg_codec_howto&oldid=7347 non-colored version]

/* The following includes have the bitstream reader, various dsp functions and the various defaults */
#define ALT_BITSTREAM_READER
#include "avcodec.h"
#include "bitstream.h"
#include "dsputil.h"

/* This includes the tables needed for the Glue codec template */
#include "gluedata.h"

/* Here we declare the struct used for the codec private data */
typedef struct {
 GetBitContext gb;
 FFTContext fft_ctx;
 VLC vlc_table;
 MDCTContext mdct_ctx;
 float* sample_buffer;
} GLUEContext;

/* The init function */
static int glue_decode_init(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* This imaginary codec uses one fft, one mdct and one vlc table. */
 ff_mdct_init(&q->mdct_ctx, 10, 1); // 2^10 == size of mdct, 1 == inverse mdct
 ff_fft_init(&q->fft_ctx, 9, 1); // 2^9 == size of fft, 0 == inverse fft
 init_vlc (&q->vlc_table, 9, 24,
 vlctable_huffbits, 1, 1,
 vlctable_huffcodes, 2, 2, 0); // look in bitstream.h for the meaning of the arguments

 /* We also need to allocate a sample buffer */
 q->sample_buffer = av_mallocz(sizeof(float)*1024); // here we used av_mallocz instead of av_malloc
 // av_mallocz memsets the whole buffer to 0

 /* Check if the allocation was successful */
 if(q->sample_buffer == NULL)
 return -1;

 /* return 0 for a successful init, -1 for failure */
 return 0;
}

/* This is the main decode function */
static int glue_decode_frame(AVCodecContext *avctx,
 void *data, int *data_size,
 uint8_t *buf, int buf_size)
{
 GLUEContext *q = avctx->priv_data;
 int16_t *outbuffer = data;

 /* We know what the arguments for this function are from above
 now we just have to decode this imaginary codec, the made up
 bitstream format is as follows:
 12 bits representing the amount of samples
 1 bit fft or mdct coded coeffs, 0 for fft/1 for mdct
 read 13 bits representing the amount of vlc coded fft data coeffs
 read 10 bits representing the amount of vlc coded mdct data coeffs
 (...bits representing the coeffs...)
 5 bits of dummy data that should be ignored
 32 bits the hex value 0x12345678, used for integrity check
 */

 /* Declare the needed variables */
 int samples, coeffs, i, fft;
 float mdct_tmp[1024];

 /* Now we init the bitstream reader, we start at the beginning of the inbuffer */
 init_get_bits(&q->gb, buf, buf_size*8); //the buf_size is in bytes but we need bits

 /* Now we take 12 bits to get the amount of samples the current frame has */
 samples = get_bits(&q->gb, 12);
 
 /* Now we check if we have fft or mdct coeffs */
 fft = get_bits1(&q->gb);
 if (fft) {
 //fft coeffs, get how many
 coeffs = get_bits(&q->gb, 13);
 } else {
 //mdct coeffs, get how many
 coeffs = get_bits(&q->gb, 10);
 }

 /* Now decode the vlc coded coeffs to the sample_buffer */
 for (i=0 ; i<coeffs ; i++)
 q->sample_buffer[i] = get_vlc2(&q->gb, q->vlc_table.table, vlc_table.bits, 3); //read about the arguments in bitstream.h

 /* Now we need to transform the coeffs to samples */
 if (fft) {
 //The fft is done inplace
 ff_fft_permute(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 ff_fft_calc(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 } else {
 //And we pretend that the mdct is also inplace
 ff_imdct_calc(&q->mdct_ctx, q->sample_buffer, q->sample_buffer, mdct_tmp);
 }

 /* To make it easy the stream can only be 16 bits mono, so let's convert it to that */
 for (i=0 ; i<samples ; i++)
 outbuffer[i] = (int16_t)q->sample_buffer[i];

 /* Report how many samples we got */
 *data_size = samples;

 /* Skip the dummy data bits */
 skip_bits(&q->gb, 5);

 /* Check if the buffer was consumed ok */
 if (get_bits(&q->gb,32) != 0x12345678) {
 av_log(avctx,AV_LOG_ERROR,"Stream error, integrity check failed!\n");
 return -1;
 }

/* The decision between erroring out or not in case of unexpected data
should be made so that the output quality is maximized.
This means that if undamaged data is assumed then unused/resereved values
should lead to warnings but not failure. (assumption of slightly non compliant
file)
OTOH if possibly damaged data is assumed and it is assumed that the original
did contain specific values in reserved/unused fields then finding unexpected
values should trigger error concealment code and the decoder/demuxer should
attempt to resync.
The decision between these 2 should be made by using
AVCodecContext.error_recognition unless its a clear case where only one of
the 2 makes sense.
*/


 /* Return the amount of bytes consumed if everything was ok */
 return *data_size*sizeof(int16_t);
}

/* the uninit function, here we just do the inverse of the init */ 
static int glue_decode_close(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* Free allocated memory buffer */
 av_free(q->sample_buffer);

 /* Free the fft transform */
 ff_fft_end(&q->fft_ctx);

 /* Free the mdct transform */
 ff_mdct_end(&q->mdct_ctx);

 /* Free the vlc table */
 free_vlc(&q->vlc_table);

 /* Return 0 if everything is ok, -1 if not */
 return 0;
}

AVCodec glue_decoder =
{
 .name = "glue",
 .type = CODEC_TYPE_AUDIO,
 .id = CODEC_ID_GLUE,
 .priv_data_size = sizeof(GLUEContext),
 .init = glue_decode_init,
 .close = glue_decode_close,
 .decode = glue_decode_frame,
};

=== trouble shooting ===
Invalid pixel format '(null)'
Output pad "default" with type video of the filter instance "Parsed_null_0" of null not connected to any destination
Error opening filters!

meant encoder's description
.pix_fmts needs to end with AV_PIX_FMT_NONE

Assertion *(const AVClass **)avctx->priv_data == codec->priv_class failed at libavcodec/utils.c:1554

meant "your first member of your Context struct needs to be an AVClass *"

== see also ==

* [[FFmpeg demuxer howto]]
* [[FFmpeg programming conventions]]

[[Category:FFmpeg Tutorials]]
[[Category:Tutorials]]

FFmpeg codec HOWTO

2015-10-13T19:12:29Z

Rogerdpack:

This page is meant as an introduction to the internal codec API in [[FFmpeg]].
It will also show how the codecs are connected with the demuxers. This is by
no means a complete guide but enough to understand how to add a codec to FFmpeg.
[[RealAudio cook|Cook]] is used as an example throughout.

== registering the codec ==

=== libavcodec/avcodec.h ===
The first thing to look at is the AVCodec struct.

typedef struct AVCodec {
const char *name;
enum CodecType type;
enum CodecID id;
int priv_data_size;
int (*init)(AVCodecContext *);
int (*encode)(AVCodecContext *, uint8_t *buf, int buf_size, void *data);
int (*close)(AVCodecContext *);
int (*decode)(AVCodecContext *, void *outdata, int *outdata_size,
uint8_t *buf, int buf_size);
int capabilities;
struct AVCodec *next;
void (*flush)(AVCodecContext *);
const AVRational *supported_framerates; ///array of supported framerates, or NULL if any, array is terminated by {0,0}
const enum PixelFormat *pix_fmts; ///array of supported pixel formats, or NULL if unknown, array is terminanted by -1
} AVCodec;

Here we can see that we have some elements to name the codec, what type it is (audio/video), the supported pixel formats and some function
pointers for init/encode/decode and close. Now lets see how it is used.

=== libavcodec/cook.c ===
If we look in this file at the bottom we can see this code:

AVCodec cook_decoder =
{
.name = "cook",
.type = CODEC_TYPE_AUDIO,
.id = CODEC_ID_COOK,
.priv_data_size = sizeof(COOKContext),
.init = cook_decode_init,
.close = cook_decode_close,
.decode = cook_decode_frame,
};

First we get an AVCodec struct named cook_decoder. And then we set the variables of cook_decoder. Note that we only set the variables that are needed. Currently there is no encoder so we don't set any. If we now look at the id variable we can see that CODEC_ID_COOK isn't defined in libavcodec/cook.c. It is declared in avcodec.h.

=== libavcodec/avcodec.h ===

Here we will find the CodecID enumeration.

enum CodecID {
...
CODEC_ID_GSM,
CODEC_ID_QDM2,
CODEC_ID_COOK,
CODEC_ID_TRUESPEECH,
CODEC_ID_TTA,
...
};

CODEC_ID_COOK is there in the list. This is the list of all supported codecs in FFmpeg, the list is fixed and used internally to id every codec. Changing the order would break binary compatibility.

This is all enough to declare a codec. Now we must register them for internal use also. This is done at runtime.

=== libavcodec/allcodecs.c ===
In this file we have the avcodec_register_all() function, it has entries like this for all codecs.

...
REGISTER_DECODER(COOK, cook);
...

This macro expands to a register_avcodec() call which registers a codec for internal use.
Note that register_avcodec() will only be called when CONFIG_COOK_DECODER is defined.
This allows to not compile the decoder code for a specific codec.
But where is it defined? This is extracted by configure with this command line:

sed -n 's/^[^#]*DEC.*, *$.*$).*/\1_decoder/p' libavcodec/allcodecs.c

So adding a REGISTER_DECODER(NEW, new) entry in allcodecs.c and reconfigure is enough to add the needed define. Now we have everything to hookup a codec.

=== libavcodec/Makefile ===
In this file we define the objects on which a codec depends. For example, cook uses fft and mdct code so it depends on the mdct.o and fft.o object files as well as the cook.o object file.

...
OBJS-$(CONFIG_COOK_DECODER) += cook.o mdct.o fft.o
...

== FFmpeg demuxer connection ==

: ''[[FFmpeg demuxer howto]]''

=== libavformat/rm.c ===

If we think of an imaginary rm file that ffmpeg is about to process, the first thing that happens is that it is identified
as a rm file. It is passed on to the rm demuxer ([http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/rmdec.c?view=markup rmdec.c]). The rm demuxer looks through the file and finds out that it is a
cook file.

...
} else if (!strcmp(buf, "cook")) {
st->codec->codec_id = CODEC_ID_COOK;
...

Now ffmpeg knows what codec to init and where to send the payload from the container. So back to cook.c and the initialization process.

== codec code ==

=== libavcodec/cook.c Init ===
After ffmpeg knows what codec to use, it calls the declared initialization function pointer declared in the codecs AVCodec struct. In
cook.c it is called cook_decode_init. Here we setup as much as we can before we start decoding. The following things should be handled in the init, vlc table initialization, table generation, memory allocation and extradata parsing.

=== libavcodec/cook.c Close ===
The cook_decode_close function is the codec clean-up call. All memory, vlc tables, etc. should be freed here.

=== libavcodec/cook.c Decode ===
In cook.c the name of the decode call is cook_decode_frame.

static int cook_decode_frame(AVCodecContext *avctx,
void *data, int *data_size,
uint8_t *buf, int buf_size) {
...

The function has 5 arguments:
* avctx is a pointer to an AVCodecContext
* data is the pointer to the output buffer
* data_size is a variable that should be set to the output buffer size in bytes (this is usually the number of samples decoded * the number of channels * the byte size of a sample)
* buf is the pointer to the input buffer
* buf_size is the byte size of the input buffer

The decode function shall return the number of bytes consumed from the input buffer or -1 in case of an error. If there is no error during decoding, the return value is usually buf_size as buf should only contain one 'frame' of data. Bitstream parsers to split the bitstream into 'frames' used to be part of the codec so a call to the decode function could have consumed less than buf_size bytes from buf. It is now encouraged that bitstream parsers be separate.

That's how it works without too much detail.

=== The Glue codec template ===
The imaginary Glue audio codec will serve as a base to exhibit bitstream reading, vlc decoding and other things.
The code is purely fictional and is sometimes written purely for the sake of example. No attempt is made to prevent invalid
data manipulation.

The Glue codec follows.

[http://wiki.multimedia.cx/index.php?title=FFmpeg_codec_howto&oldid=7347 non-colored version]

/* The following includes have the bitstream reader, various dsp functions and the various defaults */
#define ALT_BITSTREAM_READER
#include "avcodec.h"
#include "bitstream.h"
#include "dsputil.h"

/* This includes the tables needed for the Glue codec template */
#include "gluedata.h"

/* Here we declare the struct used for the codec private data */
typedef struct {
 GetBitContext gb;
 FFTContext fft_ctx;
 VLC vlc_table;
 MDCTContext mdct_ctx;
 float* sample_buffer;
} GLUEContext;

/* The init function */
static int glue_decode_init(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* This imaginary codec uses one fft, one mdct and one vlc table. */
 ff_mdct_init(&q->mdct_ctx, 10, 1); // 2^10 == size of mdct, 1 == inverse mdct
 ff_fft_init(&q->fft_ctx, 9, 1); // 2^9 == size of fft, 0 == inverse fft
 init_vlc (&q->vlc_table, 9, 24,
 vlctable_huffbits, 1, 1,
 vlctable_huffcodes, 2, 2, 0); // look in bitstream.h for the meaning of the arguments

 /* We also need to allocate a sample buffer */
 q->sample_buffer = av_mallocz(sizeof(float)*1024); // here we used av_mallocz instead of av_malloc
 // av_mallocz memsets the whole buffer to 0

 /* Check if the allocation was successful */
 if(q->sample_buffer == NULL)
 return -1;

 /* return 0 for a successful init, -1 for failure */
 return 0;
}

/* This is the main decode function */
static int glue_decode_frame(AVCodecContext *avctx,
 void *data, int *data_size,
 uint8_t *buf, int buf_size)
{
 GLUEContext *q = avctx->priv_data;
 int16_t *outbuffer = data;

 /* We know what the arguments for this function are from above
 now we just have to decode this imaginary codec, the made up
 bitstream format is as follows:
 12 bits representing the amount of samples
 1 bit fft or mdct coded coeffs, 0 for fft/1 for mdct
 read 13 bits representing the amount of vlc coded fft data coeffs
 read 10 bits representing the amount of vlc coded mdct data coeffs
 (...bits representing the coeffs...)
 5 bits of dummy data that should be ignored
 32 bits the hex value 0x12345678, used for integrity check
 */

 /* Declare the needed variables */
 int samples, coeffs, i, fft;
 float mdct_tmp[1024];

 /* Now we init the bitstream reader, we start at the beginning of the inbuffer */
 init_get_bits(&q->gb, buf, buf_size*8); //the buf_size is in bytes but we need bits

 /* Now we take 12 bits to get the amount of samples the current frame has */
 samples = get_bits(&q->gb, 12);
 
 /* Now we check if we have fft or mdct coeffs */
 fft = get_bits1(&q->gb);
 if (fft) {
 //fft coeffs, get how many
 coeffs = get_bits(&q->gb, 13);
 } else {
 //mdct coeffs, get how many
 coeffs = get_bits(&q->gb, 10);
 }

 /* Now decode the vlc coded coeffs to the sample_buffer */
 for (i=0 ; i<coeffs ; i++)
 q->sample_buffer[i] = get_vlc2(&q->gb, q->vlc_table.table, vlc_table.bits, 3); //read about the arguments in bitstream.h

 /* Now we need to transform the coeffs to samples */
 if (fft) {
 //The fft is done inplace
 ff_fft_permute(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 ff_fft_calc(&q->fft_ctx, (FFTComplex *) q->sample_buffer);
 } else {
 //And we pretend that the mdct is also inplace
 ff_imdct_calc(&q->mdct_ctx, q->sample_buffer, q->sample_buffer, mdct_tmp);
 }

 /* To make it easy the stream can only be 16 bits mono, so let's convert it to that */
 for (i=0 ; i<samples ; i++)
 outbuffer[i] = (int16_t)q->sample_buffer[i];

 /* Report how many samples we got */
 *data_size = samples;

 /* Skip the dummy data bits */
 skip_bits(&q->gb, 5);

 /* Check if the buffer was consumed ok */
 if (get_bits(&q->gb,32) != 0x12345678) {
 av_log(avctx,AV_LOG_ERROR,"Stream error, integrity check failed!\n");
 return -1;
 }

/* The decision between erroring out or not in case of unexpected data
should be made so that the output quality is maximized.
This means that if undamaged data is assumed then unused/resereved values
should lead to warnings but not failure. (assumption of slightly non compliant
file)
OTOH if possibly damaged data is assumed and it is assumed that the original
did contain specific values in reserved/unused fields then finding unexpected
values should trigger error concealment code and the decoder/demuxer should
attempt to resync.
The decision between these 2 should be made by using
AVCodecContext.error_recognition unless its a clear case where only one of
the 2 makes sense.
*/


 /* Return the amount of bytes consumed if everything was ok */
 return *data_size*sizeof(int16_t);
}

/* the uninit function, here we just do the inverse of the init */ 
static int glue_decode_close(AVCodecContext *avctx)
{
 GLUEContext *q = avctx->priv_data;

 /* Free allocated memory buffer */
 av_free(q->sample_buffer);

 /* Free the fft transform */
 ff_fft_end(&q->fft_ctx);

 /* Free the mdct transform */
 ff_mdct_end(&q->mdct_ctx);

 /* Free the vlc table */
 free_vlc(&q->vlc_table);

 /* Return 0 if everything is ok, -1 if not */
 return 0;
}

AVCodec glue_decoder =
{
 .name = "glue",
 .type = CODEC_TYPE_AUDIO,
 .id = CODEC_ID_GLUE,
 .priv_data_size = sizeof(GLUEContext),
 .init = glue_decode_init,
 .close = glue_decode_close,
 .decode = glue_decode_frame,
};

=== trouble shooting ===
Invalid pixel format '(null)'
Output pad "default" with type video of the filter instance "Parsed_null_0" of null not connected to any destination
Error opening filters!

meant encoder's description
.pix_fmts needs to end with AV_PIX_FMT_NONE

== see also ==

* [[FFmpeg demuxer howto]]
* [[FFmpeg programming conventions]]

[[Category:FFmpeg Tutorials]]
[[Category:Tutorials]]

HuffYUV

2015-01-29T19:14:01Z

Rogerdpack: add some "noted lacking" details LOL

''This page is based on the document 'Description of the HuffYUV (HFYU) Codec' by Roberto Togni found at [http://multimedia.cx/huffyuv.txt http://multimedia.cx/huffyuv.txt].''

* FourCC: HFYU
* Website: [http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html http://neuron2.net/www.math.berkeley.edu/benrg/huffyuv.html]
* Samples: [http://samples.mplayerhq.hu/V-codecs/HuffYUV/ http://samples.mplayerhq.hu/V-codecs/HuffYUV/]

HuffYUV is a lossless video compressor for [[Microsoft Audio/Video Interleaved|AVI]] files written by Ben Rudiak-Gould. Source code is available and distributed (mostly) under the GNU General Public License. This codec was created to temporarily store video data coming from a capture card for later editing, so its main features are:
* speed: uses simple predictors and original version is heavily optimized.
* quality: lossless, so image is not degraded if read/written many times.
* editable: every frame is intracoded (a.k.a., a keyframe) so there are no interframe dependencies and each frame can be seeked to independently and decoded on its own.

== Tech Notes From Author's Page (referring to v 2.1.1) ==

HuffYUV's algorithm is roughly the same as lossless JPEG: it predicts each sample and Huffman-encodes the error. The predictor functions are:
* "left" (predicts the previous sample from the same channel)
* "gradient" (predicts Left+Above-AboveLeft)
* "median" (predicts the median of Left, Above, and the gradient predictor). Channels are compressed separately, but in RGB mode the channels used are actually R-G, G, and B-G. This yields much better compression than R, G, B.

The error signal in each channel is encoded with its own Huffman table. On compression HuffYUV picks appropriate tables from its built-in collection. These tables are then stored in the output file and used when decompressing. This way future versions of HuffYUV can decompress old files without my having to explicitly support old tables. A Huffyuv-savvy application can also specify the Huffman tables to be used for compression instead of accepting the defaults.

== File Header ==

All information about the image format and predictor are stored into the [[BITMAPINFOHEADER]] structure of the AVI file header.

=== BITMAPINFOHEADER Layout ===

Only the interesting fields are shown here:

HUFFYUV_BITMAPINFOHEADER {
int biSize
[...]
short biBitCount
int biCompression
[...]
Extradata {
BYTE method
BYTE bpp_override
BYTE unused[2]
}
BYTE Compressed_Huffmantables[]
}

Extradata and Compressed_Huffmantables fields are not always there.

biCompression contains "HFYU" FOURCC.

biSize is the size of the whole structure and it is used to find out if extra fields are there, since size of standard BITMAPINFOHEADER is known.

To find out image format you need to get true bit count: it is stored in bpp_override if biSize > sizeof(BITMAPINFOHEADER), else (or if bpp_override is 0) it is stored in biBitCount. Then you can retrieve the image format with this table

+-----+-------+
| bit | image |
+-----+-------+
| 16 | YUY2 |
| 24 | RGB |
| 32 | RGBA |
+-----+-------+

In the rest of this description YUV will indicate image of type [[YUY2]], and RGB will be used for both RGB and RGBA.

To find out the prediction method you have to mask the 3 least significant bits of biBitCount (biBitCount & 0x07) and use them to index into this table:

+-----+-------------+
| bit | method |
+-----+-------------+
| 000 | see below |
| 001 | left |
| 010 | left decorr |
| 011 | gradient * |
| 100 | median |
+-----+-------------+

If the gradient method is selected (011) and the colorspace is RGB, the method is actually gradient with decorrelation.

If (biBitCount & 0x07) is 0, method is stored in extradata if biSize > sizeof(BITMAPINFOHEADER), else method is predict_old (the method used in version 1.x, it's the same as predict left). To find out method from extradata method byte use this table

+------+-----------------+
| byte | method |
+------+-----------------+
| -2 | old |
| 0 | left |
| 1 | gradient |
| 2 | median |
| 64 | left decorr |
| 65 | gradient decorr |
+------+-----------------+

Huffman tables may be stored in the file (v2.x) or not (v1.x). If biSize = sizeof(BITMAPINFOHEADER) tables are not stored into file, and to decode data the decoder must know classic tables, choosing
RGB or YUV as appropriate. Otherwise, tables are stored in file: if (biBitCount & 0x07) = 0 tables start at the end of BITMAPINFOHEADER, else they start after Extradata.

== Huffman Tables ==

Huffman tables are stored in file compressed with a run-length algorithm. This is what Ben Rudiak-Gould says about it in source code (tables.cpp)

// The Huffman tables are run-length encoded. (I could have Huffman-
// encoded them, but the madness has to end somewhere.) The
// decompression algorithm is as follows: Read a byte. The lower five
// bits are the value to repeat. If the upper three bits are zero, the
// repeat count is in the next byte; otherwise, the upper three bits are
// the repeat count. The tables are zero-terminated so that I can use
// string functions on them (a zero byte can never appear in the RLE
// data).

// Each array actually contains three Huffman tables, one for each sample
// type (e.g. Y,U,V, or R-G,G,B-G). Each table expands to 256 bytes, and
// each byte is a code length. The codes themselves are determined at
// run time and are not stored here--except for the "classic" tables,
// which don't fit the code-choosing algorithm.

When the 3 tables are available (either extracted and decompressed, or known because the file was compressed with v1.x of the codec) the decoder can extract data from the encoded byte stream.

Tables are stored in file in this order:
* YUV: Y_table, U_table, V_table
* RGB: B_table, G_table, R_table
* RGB with decorrelation: B-G_table, G_table, R-G_table

== Frame decompression ==

=== General info ===

One frame is stored per AVI chunk. All frames in file are marked as keyframes. You only need data from current chunk and file header to decompress a frame (no knowledge of previous frames or frame position in file is needed).

=== Operating Algorithm ===

The HuffYUV codec compresses an image by predicting the value of a pixel from its neighbors, and computes an error value (delta) by subtracting it from the effective pixel value. The result of this operation is then compressed with a Huffman algorithm, using a different table for each channel. To decompress a frame the decoder needs to reverse this process: extract the error value from the compressed Huffman stream, then reconstruct the pixel value by computing the predictor value and adding it to the error value. To make this process possible you need a start value (the first pixel in a frame) and a predictor that uses only values from past pixels (already decompressed). Obviously, all predictors used in HuffYUV satisfy this requirement.

All pixel component values must be treated as unsigned bytes. In all computations overflow or underflow causes a wrap-around (i.e., no saturation).

If decorrelation is used (RGB only), then instead of compressing R, G, B components the codec compresses R-G, G, B-G usually resulting in better compression. To reconstruct B and R values you have to add them the value of G taken from the same pixel.

=== Bitstream ===

The components of the image are compressed with different Huffman tables and then the bitstreams are interleaved.

As an example suppose there are 3 components (the values represent the difference between the pixel value and the predictor output). The byte sequence [ABC][DEF][GHI][LMN] is compressed in this way:
* Compress each byte of the first triplet [ABC] using the Huffan table associated with its component. The decoder will output 3 variable length codes. Suppose you get aaaa by compression of A, bbbbbbb from B and ccc from C. The output bitstream will be aaaabbbbbbccc.
* Repeat the process for every triplet, adding the new bits at the end of the stream. If compression of [DEF] will give dddd eee fffff, the output stream after second pixel will be aaaabbbbbbcccddddeeefffff.

To decompress this sequence, keep extracting bits from the stream until you get a valid code (the table contains all valid codes), and then you decode it using the table associated with that component. In the example above you will get the 4 bit aaaa and decode them to A. After decoding the current component you need to remove decoded bits from the stream (the bitstream now contains bbbbbbcccddddeeefffff), and start decoding the next component. When the first triplet [ABC] is decoded the stream will be ddddeeefffff, and the decoder is ready to start again with the first component of the next pixel.

=== Data order ===

This is how data is stored in then encoded stream. The meaning of the stream format line is:
* a capital letter is used to indicate an uncompressed value 1 byte long
* small letters represent Huffman-coded values
The length in bits of Huffman-coded values depend on the size of the code word (after decompression all delta values are 1byte long).

==== YUV Data ====
Image data is encoded and decoded left to right, top to bottom. Each pixel pair is represented by 4 bytes using YUY2 format. Frame width is always even.
Stream format: Y U Y V y u y v y u y v ....
where YUYV are the first two pixels, and y, u, v are the error values. y is decompressed with Y_table, u with U_table and v with V_table.

==== RGB Data ====
Image is stored left to right, bottom to top. Each pixel is represented by 3 bytes, using BGR24 format.
Stream format: X B G R b g r b g r ....
where X is unused, BGR is the first pixel, and b, g, r are the error values. b is decompressed with B_table, g with G_table and r with R_table.

==== RGB Data With Decorrelation ====
Image is stored left to right, bottom to top. Each pixel is represented by 3 bytes, using BGR24 format.
Stream format: X B G R g bg rg g bg rg ....
where X is unused, BGR is the first pixel, and g, bg, rg are the error values. g is decompressed with G_table, bg with B-G_table and rg with R-G_table.

==== RGBA Data ====
Image is stored left to right, bottom to top. Each pixel is represented by 4 bytes, using BGR24 format plus alpha channel.
Stream format: B G R A b g r a b g r a ....
where BGRA is the first pixel, and b, g, r, a are the error values. b is decompressed with B_table, g with G_table, r with R_table and a with R_table.

==== RGBA Data With Decorrelation ====
Image is stored left to right, bottom to top. Each pixel is represented by 4 bytes, using BGR24 format plus alpha channel.
Stream format: B G R A g bg rg a g bg rg a ....
where BGRA is the first pixel, and g, bg, rg, a are the error values. g is decompressed with G_table, bg with B-G_table, br with R-G_table and a with R-G_table.

== Predictors ==

Predictors are used to guess the value of the next pixel so that the codec needs to store only the error between the real pixel value and the prediction. If the prediction model is good the error will be small, and will compress better than full pixel values.

Valid predictors are:
* YUV: old, left, gradient, median.
* RGB: old, left, left with decorrelation, gradient with decorrelation.

Other combinations (e.g. RGB gradient without decorrelation), while technically feasible, are not used.

While describing predictors, the following symbols are used:
* [A B C]: components of a pixel
* Ayz: component A of pixel at row y and column z
* a, b, c: indices used for columns
* n, m: indices used for rows
* x: index used to indicate the last column of a frame

=== Predict Left, Predict Left With Decorrelation, And Predict Old ===

Predict old is the same as predict left (without decorrelation for RGB case).

The predictor for a pixel component is the same component from previous pixel (previous pair for U, V in YUV case).

Some examples:
* random pixel inside a row, RGB
...... [Ban Gan Ran] [Bam Gam Ram] ......
The predictor for Gam is Gan

* random pixel inside a row, YUV
.......[Y1an Uan Y2an Van] [Y1am Uam Y2am Vam] .......
The predictor for Uam is Uan, for Y1am is Y2an, for Y2am is Y1am

*first pixel in a row (not for first row), RGB
[Ba1 Ga1 Ra1] ........ [Bax Gax Rax]
[Bb1 Gb1 Rb1] ......
The predictor for Rb1 is Rax

*first pixel in a row (not for first row), YUV
[Y1a1 Ua1 Y2a1 Va1] ........ [Y1ax Uax Y2ax Vax]
[Y1b1 Ub1 Y2b1 Vb1] ......
The predictor for Ub1 is Uax, for Y2b1 is Y1b1, for Y1b1 is Y2ax

The first pixel of the picture is stored uncompressed, the rest of the frame is compressed with the predict left algorithm.

=== Predict gradient and predict gradient with decorrelation ===

The predictor for a pixel is given by the sum of the previous pixel (like predict left) and the above pixel (the pixel from the row above at the same column), minus the above left pixel (the pixel from the row above at the previous column).

Some examples:
* random pixel inside a row, RGB
...... [Ban Gan Ran] [Bam Gam Ram] ......
.......[Bbn Gbn Rbn] [Bbm Gbm Rbm]
The predictor for Gbm is (Gbn + Gam - Gan)

* random pixel inside a row, YUV
.......[Y1an Uan Y2an Van] [Y1am Uam Y2am Vam] .......
.......[Y1bn Ubn Y2bn Vbn] [Y1bm Ubm Y2bm Vbm] .......
** The predictor for Ubm is (Ubn + Uam - Uan)
** The predictor for Y1bm is (Y2bn + Y1am - Y2an)
** The predictor for Y2bm is (Y1bm + Y2am - Y1am)

* first pixel in a row (not for first row), RGB
[Ba1 Ga1 R11] ........ [Bax Gax Rax]
[Bb1 Gb1 Rb1] ........ [Bbx Gbx Rbx]
[Bc1 Gc1 Rc1] ......
** The predictor for Rc1 is (Rbx + Rb1 - Rax)

* first pixel in a row (not for first row), YUV
[Y1a1 Ua1 Y2a1 Va1] ........ [Y1ax Uax Y2ax Vax]
[Y1b1 Ub1 Y2b1 Vb1] ........ [Y1bx Ubx Y2bx Vbx]
[Y1c1 Uc1 Y2c1 Vc1] ......
** The predictor for Uc1 is (Ubx + Ub1 - Uax)
** The predictor for Y2c1 is (Y1c1 + Y2b1 - Y1b1)
** The predictor for Y1c1 is (Y2bx + Y1b1 - Y2ax)

The first pixel of the picture is stored uncompressed, and the remainder of the first row is compressed with the predict left algorithm. Other rows are compressed with predict gradient. The above left value for the first pixel of the second row is 0.

Maybe other pixels of the second row should ignore above left values, but I'm not sure about it.

=== Predict Median (YUV only) ===

The predictor for a pixel is given by the median value among the left pixel, the pixel above and the gradient predictor (sum of the previous pixel and the above pixel, minus the above left pixel). The median is obtained by sorting the three values and taking the one in the middle, and have nothing to do
with the mean value of the three numbers. For example, if the left pixel is 0x12, the above pixel is 0x4e and the gradient predictor is 0xaf, the median is 0x4e.

As Ben Rudiak-Gould says on his page, there is no technical reason for this method to be YUV only, he simply didn't implement it for RGB case.

Some examples:
* random pixel inside a row, YUV
.......[Y1an Uan Y2an Van] [Y1am Uam Y2am Vam] .......
.......[Y1bn Ubn Y2bn Vbn] [Y1bm Ubm Y2bm Vbm] .......
** The predictor for Ubm is the median value of Ubn, Uam and (Ubn + Uam - Uan)
** The predictor for Y1bm is the median value of Y2bn, Y1am and (Y2bn + Y1am - Y2an)
** The predictor for Y2bm is the median value of Y1bm, Y2am and (Y1bm + Y2am - Y1am)

* first pixel in a row (not for first row), YUV
[Y1a1 Ua1 Y2a1 Va1] ........ [Y1ax Uax Y2ax Vax]
[Y1b1 Ub1 Y2b1 Vb1] ........ [Y1bx Ubx Y2bx Vbx]
[Y1c1 Uc1 Y2c1 Vc1] ......
** The predictor for Uc1 is the median value of Ubx, Ub1 and (Ubx + Ub1 - Uax)
** The predictor for Y2c1 is the median value of Y1c1, Y2b1 and (Y1c1 + Y2b1 - Y1b1)
** The predictor for Y1c1 is the median value of Y2bx, Y1b1 and (Y2bx + Y1b1 - Y2ax)

The first pixel of the picture is stored uncompressed. The remaining of the first row, and the first 4 pixel (2 pairs, 8 bytes) of the second row are compressed with the predict left algorithm. The rest of the second row and other rows are compressed with predict median.

Technically the second pixel of the second row could be compressed using median predictor; it's compressed with left predictor because original code uses MMX instructions and can handle 8 bytes a time.

== Interlaced images ==

If image height is greater than 288 pixel, the image is treated as interlaced. The decoder must build an image with double width and half height, obtained by placing the two fields side by side, with the odd field on the left. The frame size reported in the file header is unchanged, the only way to find out interlacing is by comparing the image height with 288.

As an example, suppose your original frame is

1111111111111111
2222222222222222
3333333333333333
4444444444444444
5555555555555555
6666666666666666
7777777777777777
8888888888888888

then the frame to be compressed will be

11111111111111112222222222222222
33333333333333334444444444444444
55555555555555556666666666666666
77777777777777778888888888888888

In this way every line on a field gets the values for the predictors from the previous line of the same field, instead of using the previous line of the frame.

Please note that it makes a difference only for prediction methods that use values from the previous line (median and gradient algorithms): if a method refers only to the value of the previous pixel (left algorithms) interlacing can be ignored.

== Final notes ==

The frame width must be divisible by 4 (at least for the original codec).

It is likely that the header parsing logic can be simplified a lot since not every combination of options is valid. This is what I was able to create using dll v2.1.1 and 1.2.3/1.3.1 and VirtualDub under Windows:
* Old format (v1.x only) is only used with classical tables (not stored in file)
* Other methods (v2.1.1) always store tables in file, and uses extradata to store method.

The first two(?) pixels (or 4 bytes) of the second row in predict gradient could be stored in a different way, using only left and above pixel as predictor. None of my samples require this.

Also found on the web " the exact nature of the VLC bitstream: 32-bit words in little endian format, codewords placed starting from the MSB, Huffman codes allocated longest codeword first, and with a maximum codeword length of 31 bits." http://www.virtualdub.org/blog/pivot/entry.php?id=203

[[Category:Video Codecs]]
[[Category:Lossless Video Codecs]]

FFmpeg Summer of Code 2013

2013-03-31T00:42:18Z

Rogerdpack: fix typo, cleanup semicolons

[[Image:Ffmpeg-logo-gsoc.jpg]]

= Introduction =

FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter and stream audio and video. It includes libavcodec - the leading audio/video codec library.

[https://developers.google.com/open-source/soc/ Google Summer of Code (GSoC)] is a program that offers students stipends to write code for open source projects. Through the guidance of mentors, students gain valuable experience interacting with and coding for open source projects like FFmpeg. Additionally, the project and its users benefit from code created from students who often continue contributing as developers. FFmpeg participated to several past editions ([[FFmpeg Summer Of Code 2006|2006]], [[FFmpeg Summer Of Code 2007|2007]], [[FFmpeg Summer Of Code 2008|2008]], [[FFmpeg Summer Of Code 2009|2009]], [[FFmpeg Summer Of Code 2010|2010]], and [[FFmpeg / Libav Summer Of Code 2011|2011]]), and we are looking forward to being involved this year.

This is our ideas page for [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 Google Summer of Code 2013].

== Information for Students ==

=== Getting Started ===

0. '''Get to know FFmpeg.''' If you are a student and interested in contributing to an FFmpeg GSoC project it is recommended to start by subscribing to the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel] mailing-list, visiting our IRC channels (''#ffmpeg-devel'' and ''#ffmpeg''), and exploring the codebase and the development workflow. Feel free to [[#Contacting_FFmpeg|contact us]] if you have any questions.

1. '''Find a project.''' Listed on this page are mentored and unmentored projects. Mentored projects are well-defined and mentors have already volunteered. Unmentored projects are additional ideas that you may consider, but you will have to contact us to find a mentor. You may also propose your own project that may be a better match for your interest and skill level.

2. '''Contact us.''' If you find a project that you are interested in then get in touch with the community and let us know. In case you want to work on a qualification task, you should ask the appointed mentors so that the task can be claimed.

3. '''Apply.''' Student proposal period begins April 22, 2013 at 19:00 UTC and ends May 3rd at 19:00 UTC. See the [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#2._What_is_the_program_timeline GSoC timeline] for additional information.

=== Qualification Tasks ===

In order to get accepted you will be requested to complete a small task in the area you want to contribute. FFmpeg GSoC projects can be challenging, and a qualification task will show us that you are motivated and have the potential to finish a project.

The qualification task is usually shown in the project description. Contact the appointed mentors for assistance on getting a related qualification task or if you want to propose your own. See a list of [[Small FFmpeg Tasks]] or browse the [https://ffmpeg.org/trac/ffmpeg FFmpeg Bug Tracker] for qualification task ideas.

=== Contacting FFmpeg ===

If you have questions or comments feel free to contact us via our mailing list, IRC channel, or e-mail one of the FFmpeg GSoC admins:

* '''Mailing-list:''' [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel]
* '''IRC:''' ''#ffmpeg-devel'' on Freenode
* '''FFmpeg GSoC Admins:''' [[User:Stefanosa|Stefano Sabatini]] and [[User:Llogan|Lou Logan]]

You can also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.

= Mentored Projects =

This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor a mentored project rather than propose your own. Contact the appointed mentor(s) to get more information about the project and the requested qualification task.

== H.264 Multiview Video Coding (MVC) ==

<div class="floatright">[[Image:Mmspg-epfl-ch-double-camera.jpg]]</div>

'''Description:''' MVC samples exist and the codec is used on Blu-ray media, but FFmpeg is missing a decoder. Since this project also consists of some changes in the current architecture, it is especially important that this project is discussed on the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing list].

'''Expected results:''' Create MVC decoder and add a test for the FFmpeg Automated Testing Environment (FATE).

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' Perform work that demonstrates understanding of MVC and that is a subpart of the whole MVC implementation.

'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

'''Backup mentor:''' Kieran Kunhya (kierank on IRC)

== Animated Portable Network Graphics (APNG) ==

'''Description:''' FFmpeg currently does not support Animated PNGs.

'''Specification:''' https://wiki.mozilla.org/APNG_Specification

<div class="floatright">[[Image:Animated PNG example bouncing beach ball.png]]</div>

'''Expected results:'''
* APNG demuxer
** implement robust probing:
*** PNG images are not misdetected as APNG animations
*** APNG animations are not misdetected as PNG images
** splits stream into sensible packets (so they can be easily reused in APNG muxer)
** survives fuzzing (zzuf)
** add FATE coverage, coverage should be at least 70%
** test code under valgrind so no invalid reads/writes happen

* APNG decoder
** use existing PNG decoder code (write decoder in same file)
** implement parsing of all APNG chunks (acTL, fcTL, fdAT)
** error handling
** survives fuzzing (zzuf)
** add test for FATE, coverage should be at least 75%
** CRC checksum validation
** test code under valgrind so no invalid reads/writes happen

* APNG muxer && APNG encoder
** use existing PNG encoder code (write encoder in same file)
** write compliant files, make sure they play correctly in major web browsers that support APNG
** add test for FATE

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' Implement format autodetection for imagepipe & image demuxer

'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)

'''Backup mentor:''' [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)

== Misc Libavfilter extension ==

<div class="floatright">[[Image:Lavfi-gsoc-filter-vintage-illustration.jpg]]</div>

'''Description:''' Libavfilter is the FFmpeg filtering library. It currently supports audio and video filtering and generation support. This work may focus on porting, fixing, extending, or writing new audio and video filters from scratch.

Candidate filters for porting may be the remaining MPlayer filters currently supported through the mp wrapper, libaf MPlayer filters, and filters from other frameworks (e.g. mjpegtools, transcode, avisynth, virtualdub, etc.). In case of mp ports, the student should verify that the new filter produces the same output and is not slower.

Some ideas for more filters:
* a frequency filtering domain filter relying on the FFT utils in libavcodec
* a controller filter which allows to send commands to other filters (e.g. to adjust volume, contrast, etc.), e.g. like the sendcmd filter but through an interactive GUI
* a lua scripting filter, which allows to implement filtering custom logic in lua

For more ideas check:
[https://ffmpeg.org/trac/ffmpeg/query?status=new&status=open&status=reopened&component=avfilter&col=id&col=summary&col=status&col=type&col=priority&col=component&col=version&order=priority trac libavfilter tickets].

'''Expected results:''' Write or port audio and video filters and possibly fix/extend libavfilter API and design when required.

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

'''Qualification Task:''' write or port one or more filters

'''Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' in IRC)

'''Backup mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)

== Subtitles ==

'''Description:''' FFmpeg has been working on improving its subtitles support recently, notably by adding the support for various text subtitles and various hardsubbing (burning the subtitles onto the video) facilities. While the theme may sound relatively simple compared to audio/video signal processing, the project carries an historical burden not easy to deal with, and introduces various issues very specific to its sparse form.

<center>[[Image:Subtitles-sensei.jpg]]</center>

'''Expected results:'''
* Add support for new subtitles formats. Example: a demuxer for .SUP files, just like VobSub but for Blu-Ray, or a VobSub muxer.
* Improve text subtitles decoders. Typically, this can be supporting advanced markup features in SAMI or WebVTT.
* Update the API to get rid of the clumsy internal text representation of styles
* Proper integration of subtitles into libavfilter. This is the ultimate goal, as it will notably allow a complete subtitles rendering for applications such as ffplay.
* BONUS: if everything goes well, the student will be allowed to add basic support for teletext

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background in fansubbing area (notably ASS experience) would be a bonus but is not strictly required.

'''Qualification Task:''' write one subtitles demuxer and decoder (for example support for Spruce subtitles format). This is in order to make sure the subtitles chain is understood.

'''Mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)

'''Backup Mentor:''' Nicolas George (''Cigaes'' on IRC)

== Postproc optimizations ==

<div class="floatright">[[Image:PostProc.jpg]]</div>

'''Description:''' FFmpeg contains libpostproc, which is used to postprocess 8x8 DCT-MC based video and images (jpeg, mpeg-1/2/4, H.263 among others). The code though has been written a long time ago and its SIMD optimizations need to be updated to what modern CPUs support (AVX2 and SSE2+).

'''Expected results:'''
* Convert all gcc inline asm in libpostproc to YASM.
* Restructure the code so that it works with block sizes compatible with modern SIMD.
* Add Integer SSE2 and AVX2 optimizations for each existing MMX/MMX2/3dnow optimization in libpostproc.

'''Prerequisites:''' C coding skills, good x86 assembly coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' convert 1 or 2 MMX2 functions to SSE2 and AVX2.

'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

 

== Bayer RGB colorspaces ==

<div class="floatright">[[Image:350px-Bayer_pattern_on_sensor.svg.png ]]</div>

'''Description:''' Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmpeg's applicability to RAW still and video photography processing.

'''Expected Results:'''
* Rebase existing patches
* Implement high quality bayer transformations in libswscale (plain C)
* Add bayer formats to the libavutil pixfmt enumeration routines
* SIMD optimizations of the libswscale transformations
* Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)

Optional goodies:
* Extend TIFF decoder to support DNG-Bayer format
* Support a popular proprietary camera format (many to choose from; see dcraw project)

'''Prerequisites''': C coding skills, familiarity with git/source code control systems.

'''Qualification Task''': Implement a simple and working Bayer->RGB transform in libswscale

'''Mentor''': [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)

'''Backup Mentor''': [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

 

== MPEG-4 ALS encoder ==

<div class="floatright">http://upload.wikimedia.org/wikipedia/commons/e/e9/ATunes.png</div>
<div class="floatright">[http://commons.wikimedia.org/wiki/File%3AATunes.png]</div>

'''Description:'''
A MPEG-4 ALS decoder was implemented several years ago but an encoder is still missing in the official codebase. A rudimentary encoder has already been written and is available on [https://github.com/justinruggles/FFmpeg-alsenc.git github]. For this project, that encoder is first to be updated to fit into the current codebase of FFmpeg and to be tested for conformance using the [http://www.nue.tu-berlin.de/menue/forschung/projekte/beendete_projekte/mpeg-4_audio_lossless_coding_als/parameter/en/#230252 reference codec and specifications]. Second, the encoder is to be brought through the usual reviewing process to hit the codebase at the end of the project.

'''Expected results:'''

* Update the existing encoder to fit into the current codebase.
* Ensure conformance of the encoder by verifying using the reference codec and generate a test case for FATE.
* Ensure the FFmpeg decoder processes all generated files without warnings.
* Enhance the rudimentary feature set of the encoder.

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. A certain interest in audio coding and/or knowledge about the FFmpeg codebase could be beneficial.

'''Qualification Task:''' Add floating point support to MPEG-4 ALS decoder

'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)

 

== Hardware Acceleration (hwaccel) API v2 ==

<div class="floatright">[[Image:Hardware.jpg]]</div>

'''Description:''' FFmpeg supports hardware accelerated decoding through the internal hwacel API. Currently supported system hardware acceleration APIs are VA-API (Linux), DXVA2 (Windows) and VDA (MacOS X). However, the current approach requires client applications to allocate the underlying resources (e.g. hardware surfaces and context) themselves, and handing them over to FFmpeg. This incurs a few limitations: this is not scalable to new codecs, i.e. this requires new tokens for each newly supported codec; this incurs extra work in the client application, which tends to be duplicated over several client applications; and this prevents efficient fallback to software decoding mode if the hardware cannot handle a particular codec specification.

The goal of this project is to revamp the FFmpeg Hardware Acceleration API so that hardware resources are allocated and managed in the library, thus requiring the client application to only provide a single hardware context/device handle; provide a way to fallback early to software decoding mode if the underlying hardware won't be able to handle the bitstream; and make it possible to select a hardware accelerator by ID and not polluting the PixelFormats namespace.

'''Expected results:'''
* FFmpeg core library (libavcodec):
** Core API extensions and improvements
*** Add open/close hooks in a way that is backwards compatible with hwaccel v1 enabled applications
*** Add new tokens describing hardware accelerators
*** Add new flags exposing HW capabilities like download/upload
*** Investigate the benefits or impacts to provide a global map/unmap capability to FFmpeg video buffers
** Port hwaccels to v2 infrastructure
*** Port VA-API decoders to v2 infrastructure
*** Validate that VA-API decoders still work with existing applications supporting hwaccel v1
*** Provide download capability through ''vaGetImage()''
*** Validate that ffplay can support this feature with minor changes, and definitely no change to the existing SDL renderer
*** Port VDPAU decoders to hwaccel v2 (optional), and investigate ways to preserve compatibility with older applications

* FFmpeg applications:
** Integrate hardware acceleration into ffplay
*** Create a video-output (VO) infrastructure to ffplay
*** Port the SDL renderer to the new VO infrastructure
*** Add support for VA-API: VA renderer through ''vaPutSurface()'', add -hwaccel option to select "vaapi" renderer
*** Add support for VDPAU (optional): VDPAU renderer through ''VdpPresentationQueueDisplay()''
** Integrate hardware acceleration into ffmpeg
*** Add support for VA-API: use the VA/DRM API for headless (no-X display) decoding, use libudev to determine the device to use

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, hardware supporting VA-API.

'''Qualification Task:''' Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)

 

== Hardware Accelerated Video Encoding with VA-API ==

'''Description:''' FFmpeg already supports hardware accelerated decoding for multiple codecs but still lacks support for hardware accelerated encoding. The aim of the project is to add support for encoding with VA-API specifically, while keeping a generic enough approach in mind so that other hardware accelerators (TI-DSP, CUDA?) could be supported as well. This means that new ''hwaccel'' hooks are needed and two operational modes are possible: either ''(i)'' driver or hardware pack headers themselves, or ''(ii)'' lattitude is left to perform this task at the FFmpeg library level.

'''Expected results:''' Allow MPEG-2 and H.264 encoding with VA-API, while supporting variable bitrate (VBR) by default, and allowing alternate methods like constant bitrate (CBR) or constant QP (CQP) where appropriate or requested.
* MPEG-2 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** (Optionally) add support for interlaced contents
* H.264 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** Add support for constant QP (CQP, i.e. handle the ''-cqp'' option)
** Add support for more than one reference frame, while providing/using API to query the hardware capabilities
** Work on HRD conformance. May require to write an independent tool to assess that
** (Optionally) add configurability of the motion estimatation method to use. Define new types for HW accelerated encoding with at least two levels/hints for the accelerator.
* FFmpeg applications:
** Define common hwaccel interface for encoding
** Add initial support for hardware accelerated encoding to the ''ffmpeg'' application

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, hardware supporting VA-API for encoding.

'''Qualification Task:''' Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)

 

= Unmentored Projects =

This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also [[#Your_Own_Idea|propose their own project]].

== glplay ==

<div class="floatleft">[[Image:Opengl_logo.jpg]]</div>

'''Description:''' The SDL library that is used by FFplay has some deficiencies, adding OpenGL output to FFplay should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which may be relicensed under the LGPL.

'''Mentor:''' TBD Backup: Reimar Döffinger

 

== AAC ==

'''Description:''' FFmpeg contains an AAC encoder and decoder, both of them can be improved in various ways. This is enough work for more than one GSoC project, so one part of your submission would be to define on which task exactly you want to work.
* AAC LD decoder
* AAC BSAC decoder: This has already been started, but the existing decoder still fails on many samples
* AAC SSR decoder
* AAC 960/120 MDCT window
* AAC multi-channel encoding
See also the FFmpeg bug tracker for AAC issues, fixing one of them or rebasing the existing incomplete BSAC decoder for current git head fixing one or more existing bugs are possible qualification tasks.

== DTS / DCA ==

'''Description:''' FFmpeg contains a DTS decoder.
* DTS-HD decoder improvements: A possible qualification task is to implement ticket [https://ffmpeg.org/trac/ffmpeg/ticket/1920 #1920]
** Add support for X96 extension (96khz)
** Add support for XLL extension (lossless)
** Add support for pure DTS-HD streams that do not contain a DTS core
** Add support for multiple assets
** Add support for LBR extension

== TrueHD encoder ==

'''Description:''' FFmpeg currently does not support encoding to one of the lossless audio formats used on Bluray discs. This task consists of implementing a TrueHD encoder that allows to losslessly encode audio to play it on hardware devices capable of TrueHD decoding.

== Opus decoder ==

<div class="floatright">[[Image:Opus.png]]</div>

'''Description:''' Opus decoding is currently supported through the external libopus library
* Write a native decoder, continue working on the existing unfinished implementation
A possible qualification task is to port the existing incomplete decoder to current git head and improve it to show that you are capable of working on this task.

== VC-1 interlaced ==

'''Description:''' The FFmpeg VC-1 decoder has improved over the years, but many samples are still not decoded bit-exact and real-world interlaced streams typically show artefacts.
* Implement missing interlace features
* Make more reference samples bit-exact
As a qualification task, you should try to find a bug in the current decoder implementation and fix it.

== JPEG 2000 ==

<div class="floatleft">[[Image:Jpeg2000.jpg]]</div>

'''Description:''' FFmpeg contains an experimental native JPEG 2000 encoder and decoder. Both are missing many features, see also the FFmpeg bug tracker for some unsupported samples.
Work on an issue (for example from the bug tracker) as a qualification task to show that you are capable of improving the codec implementation.

 

== VP7 ==

'''Description:''' Not many [http://samples.mplayerhq.hu/V-codecs/VP7/ VP7 samples] are in the wild, but no open-source decoder exists although a [http://multimedia.cx/mirror/VP7_Data_Format_and_Decoder_Overview.pdf specification] exists. Write a decoder that reuses as much as possible of existing FFmpeg code: it is likely that functions of the existing decoders for On2-based formats will be useful.

== VP8L ==

'''Description:''' [[VP8L]] is a lossless format used in WebP. There is no support for this in FFmpeg.

== Your own idea ==

A student can propose a project. Ideas can also be found by browsing bugs and feature requests on our [https://ffmpeg.org/trac/ffmpeg/ bug tracker]. The work should last the majority of the GSoC duration, the task must be approved by the developers, and a mentor must be assigned.

Students can discuss an idea in the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing-list], the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins [[User:Stefanosa|Stefano Sabatini]] or [[User:Llogan|Lou Logan]] for more information.

[[Category:FFmpeg]]

Libavfilter

2012-07-16T00:13:14Z

Rogerdpack: /* See also */

==Reasons why a filter API is needed in FFmpeg==
FFmpeg's already well-known [[libavcodec]] module has become the de facto standard library for video decoding and encoding in free software projects. Unfortunately, no similar standard library has surfaced for audio/video filtering and otherwise working with audio/video stream once it has been decoded. Various multimedia projects (such as MPlayer, Xine, GStreamer, VirtualDub, etc.) have implemented their own filter systems to various degrees of success. What is needed is a high quality audio and video filter API - efficient, flexible enough to meet all the requirements which have led various projects to invent their own filter system, and yet easy to use or develop new filters with. This proposal is to implement a high quality video API and filter library for FFmpeg, where it can be easily used by other multimedia-related software projects.

== Possible known issues ==
* no frame rate estimation is supported (see post in -soc [http://lists.mplayerhq.hu/pipermail/ffmpeg-soc/2007-December/002283.html])
* make ffmpeg.c honor the aspect ratio outputted by the filter chain (thanks vdmsss for the bug report) -- '''Updated:''' probably fixed, but maybe not printing the right PAR in the terminal

== Feature wish list ==
'''Warning:''' This is not an official wish list. Before implementing any one of these items, ask first in the ffmpeg-devel mailing-list.

* See the [http://svn.mplayerhq.hu/soc/libavfilter/TODO?view=co TODO file] in SoC SVN tree
*Create a SDK (Software Development Kit) with templates for the A/V filter APIs
*Add support for audio filters
** See [[FFmpeg audio API]]
*Add support for subtitle filters
** Add a subtitle drawing support to simplify ffplay.c
* Create (or port) additional pre-process and post-process video filters to FFmpeg:
** General post-proc sources are [http://svn.mplayerhq.hu/mplayer/trunk/DOCS/tech/libmpcodecs.txt?revision=15625&view=markup MPlayer (libmpcodecs vf_*.c filters)], [http://xinehq.de/index.php/hackersguide#AEN864 Xine], [[ffdshow|FFdshow]], [[VLC media player|VLC]], [http://www.virtualdub.org/filtersdk VirtualDub], [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html GStreamer], [http://www.foobar2000.org foobar], and [http://www.xmms.org/ XMMS], vhook
**More [http://en.wikipedia.org/wiki/Image_scaling image scaling] methods:
***[http://en.wikipedia.org/wiki/Linear_interpolation linear interpolation]
***[http://en.wikipedia.org/wiki/Bilinear_interpolation bilinear interpolation]
***[http://en.wikipedia.org/wiki/Cubic_interpolation cubic interpolation]
***[http://en.wikipedia.org/wiki/Bicubic_interpolation bicubic interpolation]
***[http://en.wikipedia.org/wiki/Trilinear_interpolation trilinear interpolation]
** SSP (Statistical Post-Processing)
** DeBlocking
** DeRinging
** IVTC
** Sharpen / UnSharpen (Soften)
** ReQuantization
** Auto-Luminance
** Blurring / DeNoising / Spatial Blur / Temporal Blur
** Deinterlace (weave AND bob) filters
*** Possible sources: [http://sourceforge.net/projects/deinterlace DScaler] or [http://www.mir.com/DMG/Software/y4mscaler.html y4mscale]?
** 2:3 pull-down / ivtc (inverse telecine) for 24 progressive-frames on 30 FPS TV's
*** Possible sources: [http://sourceforge.net/projects/deinterlace DScaler] or [http://www.mir.com/DMG/Software/y4mscaler.html y4mscale]?
** NTSC => PAL, and PAL => NTSC frame-rate (FPS) adjust and reclock filter for NTSC <=> PAL conversion

== See also ==
* [[FFmpeg filter howto]]
* [[FFmpeg audio API]]
* [http://ffmpeg.org/trac/ffmpeg/wiki/FilteringGuide FFmpeg filtering Guide]

[[Category:FFmpeg]]

Small FFmpeg Tasks

2011-10-28T19:29:10Z

Rogerdpack: /* port Ut Video decoder/encoder */

This page contains ideas for small, relatively simple tasks for the [[FFmpeg]] project. People who might be interested in trying one of these tasks:
* Someone who wants to contribute to FFmpeg and needs to find a well-defined task to start with
* Someone who wishes to qualify for one of FFmpeg's coveted [[FFmpeg Summer Of Code|Summer of Code]] project slots
* An existing FFmpeg developer who has been away from the project for a while and needs a smaller task as motivation for re-learning the codebase

For other tasks of varying difficulty, see the [[Interesting Patches]] page.

'''If you would like to work on one of these tasks''', please take these steps:
* Subscribe to the [https://lists.ffmpeg.org/mailman/listinfo/ffmpeg-devel FFmpeg development mailing list] and indicate your interest
* Ask [[User:Multimedia Mike|Multimedia Mike]] for a Wiki account so you can claim your task on this Wiki

'''If you would like to add to this list''', please be prepared to explain some useful details about the task. Excessively vague tasks with no supporting details will be ruthlessly deleted.

=== Finish up a previous incomplete SoC project ===

Several SoC projects from previous years have not yet made it into FFmpeg. Taking any of them and finishing them up to the point that they can be included should make for a good qualification task. Check out the [[FFmpeg Summer Of Code]] overview page and look for the unfinished projects, like the TS muxer.

=== Generic Colorspace system ===
This task involves adding support more than 8 bits per component (Y on 10 bits, U on 10 bits, V on 10 bits for example)
and generic simple conversion to other colorspaces.

''Does this have to do with revising FFmpeg's infrastructure? If so, then it doesn't feel like a qualification task. If it's something simpler, then the vague description does not convey that simplicity. Please expound.'' --[[User:Multimedia Mike|Multimedia Mike]] 12:56, 25 February 2008 (EST)

''I don't think so, extending PixFmt to extended structure with finegrained description like depth, range values, colorspace, sample period, and write generic simple conversion from all formats to all others, like suggested by Michael on the mailing list. Conversion routine can be a good qualification task for video encoders/decoders. What do you think ?
--[[User:Bcoudurier|Baptiste Coudurier]] 00:30, 29 February 2008 (EST)

''* Adding the [[YCoCg]] colorspace (with different sized planes) for RGB sourced pictures would be nice too. [[User:Elte|Elte]] 07:15, 16 March 2009 (EDT)

=== Extend GIF Encoder and Decoder to support Animated GIFs ===

=== reverse engineer realplayer's ivr format and create a demuxer for it ===
samples on [[IVR]] page.

=== Implement a Vivo demuxer ===
Implement a demuxer for the [[Vivo]] file format. The best reference for understanding the format would be MPlayer's [http://svn.mplayerhq.hu/mplayer/trunk/libmpdemux/demux_viv.c?view=markup existing .viv demuxer].

This task corresponds to ticket 132: https://avcodec.org/trac/ffmpeg/ticket/132

''I am ready to help out with understanding MPlayer's demuxer, esp. MPlayer API stuff if necessary.
--[[User:Reimar|Reimar]] 15:46, 1 March 2008 (EST)

=== Port missing demuxers from MPlayer ===
MPlayer supports a few container formats in libmpdemux that are not yet present in libavformat. Porting them over and gettting them relicensed as LGPL or reimplementing them from scratch should make reasonable small tasks.

# TiVo --
# VIVO -- ''Daniel Verkamp has a patch for this''
# SL support for MPEG-TS (anyone got samples?)
# MNG

=== Optimal Huffman tables for (M)JPEG ===
This task is outlined at http://guru.multimedia.cx/small-tasks-for-ffmpeg/ and is tracked in the issue tracker: http://roundup.libav.org/issue267

=== M95 Playback System ===
This task is to implement a playback subsystem for [[M95]] files. This will entail writing a new file demuxer and video decoder (the audio is already uncompressed), both of which should be fairly easy by FFmpeg standards. [[M95|The M95 page]] contains the specs necessary to complete this task and points to downloadable samples.

=== BRP Playback System ===
This task is to implement a playback subsystem for [[BRP]] files. This will entail writing a new file demuxer as well as a video decoder that can handle at least 2 variations of format data. Further, write an audio decoder for the custom DPCM format in the file. All of these tasks are considered fairly easy by FFmpeg standards. [[BRP|The BRP page]] contains the specs necessary to complete this task and points to downloadable samples for both known variations.

=== 16-bit VQA Video Decoder ===
Westwood [[VQA]] files are already supported. However, there are three variations of its custom video codec, only the first two of which are supported. This task involves implementing support for the third variation. Visit the VQA samples repository: http://samples.multimedia.cx/game-formats/vqa/ -- The files in the directories Tiberian Sun VQAs/, bladerunner/, and dune2000/ use the 3rd variation of this codec. The [[VQA|VQA page]] should link to all the details you need to support this format.

Discussion/patch:
http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/89902/focus=90433

=== HNM4 Playback System ===
This task is to implement a playback subsystem for [[HNM4]] variant of the [[HNM]] format. This will entail writing a new file demuxer and video decoder, both of which are considered fairly easy by FFmpeg standards. [[HNM4|The HNM4 page]] contains the specs necessary to complete this task and links to downloadable samples.

=== Apple RPZA encoder ===
A patch was once sent to the ffmpeg-devel mailing list to include an encoder for the [[Apple RPZA]] video codec. That code can be found on the "[[Interesting Patches]]" page. This qualification task involves applying that patch so that it can compile with current HEAD and then cleaning it up per the standards of the project. Engage the mailing list to learn more about what to do.

=== QuickTime Edit List Support ===
Implement edit list support in the QuickTime demuxer (libavformat/mov.c). This involves parsing the 'elst' atom in a QuickTime file. For a demonstration of how this is a problem, download the file menace00.mov from http://samples.mplayerhq.hu/mov/editlist/ and play it with ffplay or transcode it with ffmpeg. Notice that the audio and video are ever so slightly out of sync. Proper edit list support will solve that. Other samples in that directory also presumably exhibit edit list-related bugs. The [http://xine.cvs.sourceforge.net/xine/xine-lib/src/demuxers/demux_qt.c?view=markup Xine demuxer] has support for this, it might be useful for hints.

(patch was submitted to ffmpeg-devel , around 14 March 2009)

=== Add wma fixed point decoder back into libavcodec ===
http://svn.rockbox.org/viewvc.cgi/trunk/apps/codecs/libwma/
Rockbox's fixed-point WMA decoder was adapted from the decoder in libavcodec.

=== VC1 timestamps in m2ts ===

Codec copy of VC1 from m2ts currently doesn't work. Either extend the VC1 parser to output/fix timestamps, or fix the timestamps from m2ts demuxing.

=== FLIC work ===

Revise the [[Flic Video]] decoder at libavcodec/flicvideo.c to support video transported in AVI or MOV files while making sure that data coming from the usual FLI files still works. 'AFLC' and 'flic' FourCC samples are linked from the [[Flic Video]] page.

=== flip flag for upside-down codecs ===

<pre>about the flip, a patch that decodes images fliped when
codec_tag == ff_get_fourcc("GEOX") is welcome.
its a metter of 2lines manipulating data/linesize of imgages after
get_buffer() or something similar
[...]
--
Michael GnuPG fingerprint: 9FF2128B147EF6730BADF133611EC787040B0FAB
</pre>
more info:
http://roundup.ffmpeg.org/roundup/ffmpeg/issue741

=== lavf-based concatenation tool ===

It would be nice to have some libavformat-based tool that would extract frames from multiple files (possible different containers as well) and put them into single one.

=== cljr and vcr1 encoders ===
According to this: http://lists.mplayerhq.hu/pipermail/ffmpeg-devel/2009-February/063555.html both of the encoders are disabled, and won't compile if enabled. Michael would prefer to keep them around, and have someone grow them into full encoders.

=== implement some colorspace fourcc/codecs ===
some colorspace formats were uploaded to http://samples.mplayerhq.hu/V-codecs/
including:
CYUV.AVI is 8 Bit Interleaved 4:2:2
a12v.avi is 4:2:2:4 10 Bit Interleaved
auv2.avi is 4:2:2:4 8 Bit Interleaved
and V-codecs/yuv8/MAILTEST.AVI .

it might decode with current pixfmts, a sample commit is 9853bbb21a19d540850de60d3e9cf7c6ef9da7dc

a sample commit for adding new input formats to swscale is 4884b9e50d416f84e64bfaf546a03e490cb83a2f
the hunks 3 and 5 you do not need, they are optional special converters
also the change to isSupportedOut() you do not need
above will add a new input format

another example for adding an input format is a43fb6b37efa5b01f2c9bdc414570691229bcfab

=== port libquicktime de/encoders ===

http://libquicktime.sourceforge.net/
some of these may be copies of already implemented codecs:
8 bit planar YUV 4:4:4 (v308)
8 bit Planar YUVA 4:4:4:4 (v408)
10 bit Packed YUV 4:4:4 (v410)
YUV 4:2:0 (yuv4)

See [http://ffmpeg.org/trac/ffmpeg/ticket/470 Ticket 470]

=== Implement Phantom Cine demuxer and Bayer format support for swscale ===
The format is described here:
http://wiki.multimedia.cx/index.php?title=Phantom_Cine
It will need support for Bayer -> RGB conversion in swscale to make the demuxer useful though.

=== support for [[YCoCg]]/RGB colorspace in FFV1 ===
Add support for [[YCoCg]] and [[RGB]] encoded sources for the [[FFV1]] codec

This would add a free lossless intra-frame RGB codec for all supported platforms (most important OS X + Windows) which is often asked for video editing in video forums (e.g. slashcam.de)

=== Metal Gear Solid Video format demuxer ===
Write a demuxer to play video files harvested from the game Metal Gear Solid: The Twin Snakes. The format is described on the wiki page [[Metal Gear Solid VP3]] (which also contains links to samples). This page is based on observations and conjecture, so remember to engage the mailing list or IRC with questions.

=== [[IFF#ANIM|IFF ANIM]] decoder ===
Modify libavformat/iff.c to handle this chunk and write a decoder for the format. The wiki page at [[IFF#ANIM|IFF ANIM]] has links to more information and source code. Samples can be found at http://www-user.tu-chemnitz.de/~womar/projects/iffanim/iffanim_samplepack.zip .

=== [[CDXL]] decoder ===
http://roundup.libav.org/issue1012

Write a decoder for this format using the information in the [[CDXL]] wiki page
Discussed for the 2009 SoC

=== port missing decoders/demuxers from other open source projects. ===

http://www.mega-nerd.com/libsndfile/#Features
Paris Audio File PAF
IRCAM SF
GNU Octave 2.0 MAT4
GNU Octave 2.1 MAT5
Portable Voice Format PVFSound
Designer II SD2
samples are here: http://www.mega-nerd.com/tmp/SoundFileCollection-20050711-0902.tgz

http://www.hawksoft.com/hawkvoice/
HVDI_VOICE_DATA- packet
[[GSM]]
LPC
CELP
LPC10

http://sourceforge.net/projects/vgmstream
150+ formats: http://vgmstream.svn.sourceforge.net/viewvc/vgmstream/readme.txt

http://www.imagemagick.org
http://www.graphicsmagick.org/formats.html
many image formats not supported yet.

http://gpac.sourceforge.net/
[[MPEG-4 BIFS]]
3GPP DIMS
[[LASeR]]
SAF
SVG
[[Synchronized Multimedia Integration Language|SMIL]]
VRML
X3D
XMT

http://adplug.sourceforge.net/
http://adplug.sourceforge.net/library/
many OPL2/OPL3 audio formats not supported yet.

http://mikmod.raphnet.net/
http://mikmod.raphnet.net/#features
many music pattern formats not supported yet.

http://www.fly.net/~ant/libs/audio.html#Game_Music_Emu
AY
GBS
GYM
HES
KSS
NSF, NSFE
SAP
[[SNES-SPC700 Sound Format]]
VGM, VGZ

=== port [[Ut Video]] decoder/encoder ===
gpl v2 decoder/encoder at wiki page some work has started http://www.ffmpeg.org/trac/ffmpeg/ticket/534

=== libswscale PAL8 output ===

See the thread: "[RFC] libswscale palette output implementation":
http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/101397

=== vloopback output support ===

vloopback is a linux kernel device which allows to create a virtual video device where
programs can write, and can be accessed as a normal video device:
http://www.lavrsen.dk/twiki/bin/view/Motion/VideoFourLinuxLoopbackDevice

This would allow to write the ffmpeg output to a vloopdevice and be displayed by some a
program reading from such device (e.g. skype, a voip client etc.).

An example of a program which uses vloopback:
http://www.ws4gl.org/

=== Port video filters from MPlayer/VLC/Mjpegtools/Effectv/etc etc to libavfilter ===

There are plenty programs providing their own filters, many of them may be easily ported to the
superior ;-) framework of libavfilter. Also may be possible to create wrappers around other libraries
(e.g. opencv, libgimp, libshowphoto, libaa).

=== rar/zip/gz/bz2 etc demuxer ===
there are still compressed files out there (zipped raw wav, multi-rar'ed videos etc). create a decompression demuxer for them.

=== Less common AAC decoder features ===

Add support to the AAC decoder for object type ER AAC LC or AAC LC 960.

=== emblaze demuxer/decoder from java code ===
samples and java decoder: http://samples.mplayerhq.hu/internets/emblaze/

[[Category:FFmpeg]]
[[Category:Libav]]

FFmpeg Wishlist

2011-09-20T16:59:09Z

Rogerdpack: done, I think

The various desired features for FFmpeg can be found in the following pages:
* [https://roundup.mplayerhq.hu/roundup/ffmpeg/issue?%40&%40columns=title&%40columns=topic&id=&%40columns=id&%40columns=activity&%40sort=activity&%40columns=priority&%40group=priority&type=2&status=2&%40columns=substatus&%40pagesize=50&%40startwith=0&%40action=search Open feature requests in the issue tracker]
* [http://svn.mplayerhq.hu/ffmpeg/trunk/doc/TODO?view=co TODO file in the SVN tree]
* [[FFmpeg Summer Of Code]] pages
* Finish and commit any code not yet commited at the [http://svn.mplayerhq.hu/soc/ SoC FFmpeg tree]

Also, other features requests can be found in:
* [https://roundup.mplayerhq.hu/roundup/ffmpeg/issue?%40&%40columns=title&%40columns=topic&id=&%40columns=id&%40columns=activity&%40sort=activity&%40columns=priority&%40group=priority&type=2&status=1&%40columns=substatus&%40pagesize=50&%40startwith=0&%40action=search Feature requests marked as "new"]
* Below in this page (mostly deprecated items)

See also the discussion about [[Ffmpeg audio api|Audio API]] TODOs.

= Old list =

The following is deprecated, please '''do not''' add new items to this list, use instead the issue tracker. Also, send a message to the mailing list before implementing one of those items. They could be work-in-progress or not wanted anymore.

Moving any of these items to a '''proper''' feature request in the issue tracker is welcome.

== Decoders ==
* integrate [[Speex]] (glue code or native)
* Fix "[rv20 @ 009C8BF0]unknown bit3 set" in [[RV20]] decoder
* [[xeb|XEB]] - the [[RatDVD]] video codec (stored in [[xvo|XVO]] container format)
* VNC decoder, files created by vncrec. Re-use code from [[VMware Video]] decoder http://www.sodan.org/~penny/vncrec/
* Additional game formats support:
** [[VQA]] v3 support, see http://www.gamers.org/pub/idgames2/planetquake/planetcnc/cncdz/
** [[Gremlin Digital Video]]
** [[M95]]

== Demuxers ==
* [[FluxDVD]] / [[RatDVD]] demuxer for [[xvo|XVO]] files (Note! [[RatDVD]] is the predecesor of [[FluxDVD]])

== Muxers ==
* DVB (MPEG-TS) muxer inside DVB containers
** MPEG-1/2 video-streams inside DVB containers
** MPEG-4 ASP video-streams inside DVB containers
** MPEG-4 AVC (H.264) video-streams inside DVB containers
** AC3 audio-streams inside DVB containers
*** Mutiple AC3 audio-streams inside DVB containers
** MP3 audio-streams inside DVB containers
*** Mutiple MP3 audio-streams inside DVB containers
* NSV muxer
* NSA muxer

== Features ==

* Improved existing documentation and add additional means to document
**Web
**WIKI
**manpage

== Misc ==
* Clean up the h263 rtp [[Interesting Patches#H.263 rtp patch|patch]].

== Streaming Media Network Protocols ==
Streaming Media Network Protocols (client and server-side) improvements/enhancements and related ideas for new features/functions.
* Create a common 'stream demuxer/parser library' for the client-side to receive input streams (and/or API for adding support for additional streaming formats?) - a LGPL'ed sub-library in FFmpeg with all stream demuxers/parsers gathered (similar to the libpostproc and libavutil). Call it "libstream" (or "stream" or whatever). Move FFmpeg's existing stream code there like HTTP and RTSP/RTP. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional streaming formats. All togther making it super easy for audio/video players using FFmpeg to add all-in-one streaming support to their player.
**Maybe use either [http://www.mplayerhq.hu MPlayer]'s "''stream''" library structure, [http://www.live555.com LIVE555], [http://curl.haxx.se cURL], or probebly the better [http://streaming.polito.it/client/library libnms] (from [http://streaming.polito.it/client NeMeSi]) as a base for such a common library?
*Add support for additional streaming protocols (on the client side) and improve/enhance support for existing protocols:
** HTTP (Hypertext Transfer Protocol) client
*** plus a SSL (Secure Sockets Layer) client support for HTTPS
** UDP (User Datagram Protocol) client
** RTSP - Real-Time Streaming Protocol (RFC2326) client
** RTP/RTCP - Real-Time Transport Protocol/RTP Control Protocol (RFC3550) client
** RTP Profile for Audio and Video Conferences with Minimal Control (RFC3551) client
** RealMedia RTSP/RDT (Real Time Streaming Protocol / Real Data Transport) client
** SDP (Service Discovery Protocol) / SSDP (Simple Service Discovery Protocol) client
** MMS (Microsoft Media Services) client
*** including the subprotocol mmsh (MMS over HTTP) and mmst (MMS over TCP)
*FFServer (streaming server) updating and improving:
**FFServer code hasn't been update for quite a while
**Support for RTSP interleaved RTP media
**RTSP over HTTP tunneling
**SLL (Secure Sockets Layer) support
**TLS (Transport Layer Security) support
**SCTP (Stream Control Transmission Protocol) support
***including tunnel SCTP over UDP
**Per-asset accounting options
**Profiling and performance improvements of the RTSP, HTTP and RTP server code
**Streaming to clients like WMP 9, 10 and 11 is broken
**MMS server streaming support in FFServer, (especially for Linux).
*** including the subprotocol mmsh (MMS over HTTP) and mmst (MMS over TCP)
*** Note that al3x has gotten something working with ffserver, you might want to ask him what needs to be done as well :) --[[User:Compn|Compn]] 14:22, 19 March 2007 (EDT)
***You should also take a look at the [http://streaming.polito.it/server FENG (RTSP Streaming Server)] code, [http://streaming.polito.it/embedded NetEmbryo (Embedded Open Media Streaming Library)], and also [http://curl.haxx.se cURL] --[[User:Gamester17|Gamester17]] 11:20, 29 March 2007 (GMT+1)

==See Also==
*[[FFmpeg Summer Of Code]] for more suggestions/requests (ideas for developers).
*[http://bugzilla.mplayerhq.hu/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=FFmpeg&content= FFmpeg bugs] for bugs in FFmpegs (codecs) that you can help fix or add addition information/samples to.
*[[:Category:Formats missing in FFmpeg]] for formats not implemented in ffmpeg yet

[[Category:FFmpeg]]

Talk:FFmpeg audio API

2011-09-08T17:36:55Z

Rogerdpack:

why does it need to be SIMD optimized? It might be that audio processing only accounts for a small amount of cpu used so won't be all too helpful...