FFmpeg filter HOWTO

From MultimediaWiki
Revision as of 12:35, 19 August 2007 by Koorogi (talk | contribs) (add information on filter links)
Jump to navigation Jump to search

This page is meant as an introduction of writing filters for libavfilter. This is a work in progress, but should at least point you in the right direction for writing simple filters.

Definition of a filter

AVFilter

All filters are described by an AVFilter structure. This structure gives information needed to initialize the filter, and information on the entry points into the filter code. This structure is declared in libavfilter/avfilter.h:

typedef struct
{
    char *name;         ///< filter name
    char *author;       ///< filter author

    int priv_size;      ///< size of private data to allocate for the filter

    int (*init)(AVFilterContext *ctx, const char *args, void *opaque);
    void (*uninit)(AVFilterContext *ctx);

    const AVFilterPad *inputs;  ///< NULL terminated list of inputs. NULL if none
    const AVFilterPad *outputs; ///< NULL terminated list of outputs. NULL if none
} AVFilter;

AVFilterPad

Let's take a quick look at the AVFilterPad structure, which is used to describe the inputs and outputs of the filter. This is also defined in libavfilter/avfilter.h:

typedef struct AVFilterPad
{
    char *name;
    int type;

    int min_perms;
    int rej_perms;

    int *(*query_formats)(AVFilterLink *link);

    void (*start_frame)(AVFilterLink *link, AVFilterPicRef *picref);
    AVFilterPicRef *(*get_video_buffer)(AVFilterLink *link, int perms);
    void (*end_frame)(AVFilterLink *link);
    void (*draw_slice)(AVFilterLink *link, int y, int height);

    int (*request_frame)(AVFilterLink *link);

    int (*config_props)(AVFilterLink *link);
} AVFilterPad;

The actual definition in the header file has doxygen comments describing each entry point, its purpose, and what type of pads it is relevant for. These fields are relevant for all pads:

Field Description
name Name of the pad. No two inputs should have the same name, and no two outputs should have the same name.
type Only AV_PAD_VIDEO currently.
query_formats Returns a list of colorspaces supported on the pad.
config_props Handles configuration of the link connected to the pad

Fields only relevant to input pads are:

Field Description
min_perms Minimum permissions required to a picture received as input.
rej_perms Permissions not accepted on pictures received as input.
start_frame Called when a frame is about to be given as input.
draw_slice Called when a slice of frame data has been given as input.
end_frame Called when the input frame has been completely sent.
get_video_buffer Called by the previous filter to request memory for a picture.

Fields only relevant to output pads are:

Field Description
request_frame Requests that the filter output a frame.

Picture buffers

Reference counting

All pictures in the filter system are reference counted. This means that there is a picture buffer with memory allocated for the image data, and various filters can own a reference to the buffer. When a reference is no longer needed, its owner frees the reference. When the last reference to a picture buffer is freed, the filter system automatically frees the picture buffer.

Permissions

The upshot of multiple filters having references to a single picture is that they will all want some level of access to the image data. It should be obvious that if one filter expects to be able to read the image data without it changing that no other filter should write to the image data. The permissions system handles this.

In most cases, when a filter prepares to output a frame, it will request a buffer from the filter to which it will be outputting. It specifies the minimum permissions it needs to the buffer, though it may be given a buffer with more permissions than the minimum it requested.

When it wants to pass this buffer to another filter as output, it creates a new reference to the picture, possibly with a reduced set of permissions. This new reference will be owned by the filter receiving it.

So, for example, for a filter which drops frames if they are similar to the last frame it output, it would want to keeps its own reference to a picture after outputting it, and make sure that no other filter modified the buffer either. It would do this by requesting the permissions AV_PERM_READ|AV_PERM_WRITE|AV_PERM_PRESERVE for itself, and removing the AV_PERM_WRITE permission from any references it gave to other filters.

The available permissions are:

Permission Description
AV_PERM_READ Can read the image data.
AV_PERM_WRITE Can write to the image data.
AV_PERM_PRESERVE Can assume that the image data will not be modified by other filters. This means that no other filters should have the AV_PERM_WRITE permission.
AV_PERM_REUSE The filter may output the same buffer multiple times, but the image data may not be changed for the different outputs.
AV_PERM_REUSE2 The filter may output the same buffer multiple times, and may modify the image data between outputs.

Filter Links

A filter's inputs and outputs are connected to those of another filter through the AVFilterLink structure:

typedef struct AVFilterLink
{
    AVFilterContext *src;       ///< source filter
    unsigned int srcpad;        ///< index of the output pad on the source filter

    AVFilterContext *dst;       ///< dest filter
    unsigned int dstpad;        ///< index of the input pad on the dest filter

    int w;                      ///< agreed upon image width
    int h;                      ///< agreed upon image height
    enum PixelFormat format;    ///< agreed upon image colorspace

    AVFilterPicRef *srcpic;

    AVFilterPicRef *cur_pic;
    AVFilterPicRef *outpic;
};

The src and dst members indicate the filters at the source and destination ends of the link, respectively. The srcpad indicates the index of the output pad on the source filter to which the link is connected. Likewise, the dstpad indicates the index of the input pad on the destination filter.

When two filters are connected, they need to agree upon the dimensions of the image data they'll be working with, and the format that data is in. Once this has been agreed upon, these parameters are stored in the link structure.

The srcpic member is used internally by the filter system, and should not be accessed directly.

The cur_pic member is for the use of the destination filter. When a frame is currently being sent over the link (ie. starting from the call to start_frame() and ending with the call to end_frame()), this contains the reference to the frame which is owned by the destination filter.

The outpic member is described in the following tutorial on writing a simple filter.

Writing a simple filter

Default filter entry points

Because the majority of filters that will probably be written will take exactly one input, and produce exactly one output, and output one frame for every frame received as input, the filter system provides a number default entry points to ease the development of such filters.

Entry point Actions taken by the default implementation
request_frame() Request a frame from the previous filter in the chain.
query_formats() on output pad Return a list of formats indicating that the format currently used on the input pad is the only supported output format.
start_frame() Request a buffer to store the output frame in. A reference to this buffer is stored in the outpic member of the link hooked to the filter's output. The next filter's start_frame() callback is called and given a reference to this buffer.
end_frame() Calls the next filter's end_frame() callback. Frees the reference to the outpic member of the output link, if it was set by (ie. if the default start_frame() is used). Frees the cur_pic reference in the input link.
get_video_buffer() Returns a buffer with the AV_PERM_READ permission in addition to all the requested permissions.
config_props() on output pad Sets the image dimensions for the output link to the same as on the filter's input.