MultimediaWiki - New pages [en]

FFmpeg bug testing

2013-04-09T16:22:10Z

Summary:

this document will try to explain the process used by bugtesters and developers to find and fix bugs in ffmpeg.

the process could be used in the future to automate bug testing. or find holes in our FATE coverage.

= example 1 - bug 2414 - wmv2 decoding artifacts =
*https://ffmpeg.org/trac/ffmpeg/ticket/2414
*sample: http://samples.ffmpeg.org/asf-wmv/wmv2decerror-short.asf

== initial process ==
#check if the official decoder/program handles the sample. it could be a broken file.
##mplayer -vc wmvdmo sample
#test sample with latest ffplay/ffmpeg.
##sometimes its already fixed.
##make sure its not an ffplay-only bug by also testing with ffmpeg.
#check with old version of ffmpeg to see if it is a regression bug.
#check if its a demuxer problem by testing with the ffmpeg/mplayer demuxer and official decoder.
##to test ffmpeg demuxer - mplayer -demuxer lavf -vc wmvdmo
##to test mplayer demuxer - mplayer -demuxer asf -vc wmvdmo

now that we know its not a demuxer problem, we can move onto debugging the decoder.

=== advanced codec debugging ===
#test if codec cpu optimizations are not binary identical to the c-version of the codec
##use ffmpeg -cpuflags 0 , ffmpeg has to be compiled with runtime cpu detection for this to work.
#test if codec asm version is not binary identical to the c-version of the codec
## recompile ffmpeg with --disable-asm

since codec also reports "dc overflow" error, check what the decoder does when it encounters this error. it is ffmpeg policy to report these errors, but also to continue decoding, in an effort to play as much of a file as possible.

in this case, msmpeg4dec.c , when encountering a dc overflow error, would return -1.
#try commenting out this in the source code, recompile, and test your changes to see if the picture changes for the better.

== testing for regressions ==
#run make fate to check for any regressions in the code
#test variety of samples relating to the code that has changed. make sure they decode visually and/or framemd5 identical.

== results and conclusion ==
there seems to be no way to switch asm optimizations on or off at runtime. its useful to have a seperate ffmpeg binary compiled with --disable-asm to test with.

ffmpeg should have some tests to make sure the various arch, asm and cpu optimizations of each codec are binary identical where possible.

= required programs for bug testing =
bug testers should have the following installed:
#latest git ffmpeg binary with asm and cpu runtime detection enabled
#latest git ffmpeg binary with --disable-asm and cpu runtime detection
#older version ffmpeg binaries with and without asm
#official 3rd party players
##realplayer
##quicktime
##windows media player
#mplayer version with binary codec support
#vlc

MO3

2013-02-25T20:12:05Z

Summary:

* created by Ian Luck (http://www.un4seen.com/)
* Amiga / PC modules with lossless, MP3 or OGG compression
* (un)official specifications and open source decoder : http://lclevy.free.fr/mo3/

FFmpeg Summer of Code 2013

2013-02-22T05:52:50Z

Summary: /* Introduction */ simplify note about project not accepted status

[[Image:Ffmpeg-logo-gsoc.jpg]]

= Introduction =

FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter and stream audio and video. It includes libavcodec - the leading audio/video codec library.

[https://developers.google.com/open-source/soc/ Google Summer of Code (GSoC)] is a program that offers students stipends to write code for open source projects. Through the guidance of mentors, students gain valuable experience interacting with and coding for open source projects like FFmpeg. Additionally, the project and its users benefit from code created from students who often continue contributing as developers. FFmpeg participated to several past editions ([[FFmpeg Summer Of Code 2006|2006]], [[FFmpeg Summer Of Code 2007|2007]], [[FFmpeg Summer Of Code 2008|2008]], [[FFmpeg Summer Of Code 2009|2009]], [[FFmpeg Summer Of Code 2010|2010]], and [[FFmpeg / Libav Summer Of Code 2011|2011]]), and we are looking forward to being involved this year.

This is our ideas page for [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 Google Summer of Code 2013].

'''FFmpeg has not been accepted this year''', so if you want to do any of the qualification tasks and projects, they are just for fun.

== Information for Students ==

=== Getting Started ===

0. '''Get to know FFmpeg.''' If you are a student and interested in contributing to an FFmpeg GSoC project it is recommended to start by subscribing to the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel] mailing-list, visiting our IRC channels (''#ffmpeg-devel'' and ''#ffmpeg''), and exploring the codebase and the development workflow. Feel free to [[#Contacting_FFmpeg|contact us]] if you have any questions.

1. '''Find a project.''' Listed on this page are mentored and unmentored projects. Mentored projects are well-defined and mentors have already volunteered. Unmentored projects are additional ideas that you may consider, but you will have to contact us to find a mentor. You may also propose your own project that may be a better match for your interest and skill level.

2. '''Contact us.''' If you find a project that you are interested in then get in touch with the community and let us know. In case you want to work on a qualification task, you should ask the appointed mentors so that the task can be claimed.

3. '''Apply.''' Student proposal period begins April 22, 2013 at 19:00 UTC and ends May 3rd at 19:00 UTC. See the [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#2._What_is_the_program_timeline GSoC timeline] for additional information.

=== Qualification Tasks ===

In order to get accepted you will be requested to complete a small task in the area you want to contribute. FFmpeg GSoC projects can be challenging, and a qualification task will show us that you are motivated and have the potential to finish a project.

The qualification task is usually shown in the project description. Contact the appointed mentors for assistance on getting a related qualification task or if you want to propose your own. See a list of [[Small FFmpeg Tasks]] or browse the [https://ffmpeg.org/trac/ffmpeg FFmpeg Bug Tracker] for qualification task ideas.

=== Contacting FFmpeg ===

If you have questions or comments feel free to contact us via our mailing list, IRC channel, or e-mail one of the FFmpeg GSoC admins:

* '''Mailing-list:''' [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel]
* '''IRC:''' ''#ffmpeg-devel'' on Freenode
* '''FFmpeg GSoC Admins:''' [[User:Stefanosa|Stefano Sabatini]] and [[User:Llogan|Lou Logan]]

You can also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.

= Mentored Projects =

This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor a mentored project rather than propose your own. Contact the appointed mentor(s) to get more information about the project and the requested qualification task.

== H.264 Multiview Video Coding (MVC) ==

<div class="floatright">[[Image:Mmspg-epfl-ch-double-camera.jpg]]</div>

'''Description:''' MVC samples exist and the codec is used on Blu-ray media, but FFmpeg is missing a decoder. Since this project also consists of some changes in the current architecture, it is especially important that this project is discussed on the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing list].

'''Expected results:''' Create MVC decoder and add a test for the FFmpeg Automated Testing Environment (FATE).

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' Perform work that demonstrates understanding of MVC and that is a subpart of the whole MVC implementation.

'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

'''Backup mentor:''' Kieran Kunhya (''kierank'' on IRC)

== Animated Portable Network Graphics (APNG) ==

'''Description:''' FFmpeg currently does not support Animated PNGs.

'''Specification:''' https://wiki.mozilla.org/APNG_Specification

<div class="floatright">[[Image:Animated PNG example bouncing beach ball.png]]</div>

'''Expected results:'''
* APNG demuxer
** implement robust probing:
*** PNG images are not misdetected as APNG animations
*** APNG animations are not misdetected as PNG images
** splits stream into sensible packets (so they can be easily reused in APNG muxer)
** survives fuzzing (zzuf)
** add FATE coverage, coverage should be at least 70%
** test code under valgrind so no invalid reads/writes happen

* APNG decoder
** use existing PNG decoder code (write decoder in same file)
** implement parsing of all APNG chunks (acTL, fcTL, fdAT)
** error handling
** survives fuzzing (zzuf)
** add test for FATE, coverage should be at least 75%
** CRC checksum validation
** test code under valgrind so no invalid reads/writes happen

* APNG muxer && APNG encoder
** use existing PNG encoder code (write encoder in same file)
** write compliant files, make sure they play correctly in major web browsers that support APNG
** add test for FATE

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' Implement format autodetection for imagepipe & image demuxer

'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)

'''Backup mentor:''' [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)

== Misc Libavfilter extension ==

<div class="floatright">[[Image:Lavfi-gsoc-filter-vintage-illustration.jpg]]</div>

'''Description:''' Libavfilter is the FFmpeg filtering library. It currently supports audio and video filtering and generation support. This work may focus on porting, fixing, extending, or writing new audio and video filters from scratch.

Candidate filters for porting may be the remaining MPlayer filters currently supported through the mp wrapper, libaf MPlayer filters, and filters from other frameworks (e.g. mjpegtools, transcode, avisynth, virtualdub, etc.). In case of mp ports, the student should verify that the new filter produces the same output and is not slower.

Some ideas for more filters:
* a frequency filtering domain filter relying on the FFT utils in libavcodec
* a controller filter which allows to send commands to other filters (e.g. to adjust volume, contrast, etc.), e.g. like the sendcmd filter but through an interactive GUI
* a lua scripting filter, which allows to implement filtering custom logic in lua

For more ideas check:
[https://ffmpeg.org/trac/ffmpeg/query?status=new&status=open&status=reopened&component=avfilter&col=id&col=summary&col=status&col=type&col=priority&col=component&col=version&order=priority trac libavfilter tickets].

'''Expected results:''' Write or port audio and video filters and possibly fix/extend libavfilter API and design when required.

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

'''Qualification Task:''' write or port one or more filters

'''Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)

'''Backup mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)

== Subtitles ==

'''Description:''' FFmpeg has been working on improving its subtitles support recently, notably by adding the support for various text subtitles and various hardsubbing (burning the subtitles onto the video) facilities. While the theme may sound relatively simple compared to audio/video signal processing, the project carries an historical burden not easy to deal with, and introduces various issues very specific to its sparse form.

<center>[[Image:Subtitles-sensei.jpg]]</center>

'''Expected results:'''
* Add support for new subtitles formats. Example: a demuxer for .SUP files, just like VobSub but for Blu-Ray, or a VobSub muxer.
* Improve text subtitles decoders. Typically, this can be supporting advanced markup features in SAMI or WebVTT.
* Update the API to get rid of the clumsy internal text representation of styles
* Proper integration of subtitles into libavfilter. This is the ultimate goal, as it will notably allow a complete subtitles rendering for applications such as ffplay.
* BONUS: if everything goes well, the student will be allowed to add basic support for teletext

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background in fansubbing area (notably ASS experience) would be a bonus but is not strictly required.

'''Qualification Task:''' write one subtitles demuxer and decoder (for example support for Spruce subtitles format). This is in order to make sure the subtitles chain is understood.

'''Mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)

'''Backup Mentor:''' Nicolas George (''Cigaes'' on IRC)

== Postproc optimizations ==

<div class="floatright">[[Image:PostProc.jpg]]</div>

'''Description:''' FFmpeg contains libpostproc, which is used to postprocess 8x8 DCT-MC based video and images (jpeg, mpeg-1/2/4, H.263 among others). The code though has been written a long time ago and its SIMD optimizations need to be updated to what modern CPUs support (AVX2 and SSE2+).

'''Expected results:'''
* Convert all gcc inline asm in libpostproc to YASM.
* Restructure the code so that it works with block sizes compatible with modern SIMD.
* Add Integer SSE2 and AVX2 optimizations for each existing MMX/MMX2/3dnow optimization in libpostproc.

'''Prerequisites:''' C coding skills, good x86 assembly coding skills, familiarity with git/source code control systems.

'''Qualification Task:''' convert 1 or 2 MMX2 functions to SSE2 and AVX2.

'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)

<br clear="all">

== Bayer RGB colorspaces ==

<div class="floatright">[[Image:350px-Bayer_pattern_on_sensor.svg.png ]]</div>

'''Description:''' Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmpeg's applicability to RAW still and video photography processing.

'''Expected Results:'''
* Rebase existing patches
* Implement high quality bayer transformations in libswscale (plain C)
* Add bayer formats to the libavutil pixfmt enumeration routines
* SIMD optimizations of the libswscale transformations
* Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)

Optional goodies:
* Extend TIFF decoder to support DNG-Bayer format
* Support a popular proprietary camera format (many to choose from; see dcraw project)

'''Prerequisites''': C coding skills, familiarity with git/source code control systems.

'''Qualification Task''': Implement a simple and working Bayer->RGB transform in libswscale

'''Mentor''': [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)

'''Backup Mentor''': [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)

<br clear="all">

== MPEG-4 ALS encoder ==

<div class="floatright">http://upload.wikimedia.org/wikipedia/commons/e/e9/ATunes.png</div>
<div class="floatright">[http://commons.wikimedia.org/wiki/File%3AATunes.png]</div>

'''Description:'''
A MPEG-4 ALS decoder was implemented several years ago but an encoder is still missing in the official codebase. A rudimentary encoder has already been written and is available on [https://github.com/justinruggles/FFmpeg-alsenc.git github]. For this project, that encoder is first to be updated to fit into the current codebase of FFmpeg and to be tested for conformance using the [http://www.nue.tu-berlin.de/menue/forschung/projekte/beendete_projekte/mpeg-4_audio_lossless_coding_als/parameter/en/#230252 reference codec and specifications]. Second, the encoder is to be brought through the usual reviewing process to hit the codebase at the end of the project.

'''Expected results:'''

* Update the existing encoder to fit into the current codebase.
* Ensure conformance of the encoder by verifying using the reference codec and generate a test case for FATE.
* Ensure the FFmpeg decoder processes all generated files without warnings.
* Enhance the rudimentary feature set of the encoder.

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. A certain interest in audio coding and/or knowledge about the FFmpeg codebase could be beneficial.

'''Qualification Task:''' Add floating point support to MPEG-4 ALS decoder

'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)

'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)

<br clear="all">

== Hardware Acceleration (hwaccel) API v2 ==

<div class="floatright">[[Image:Hardware.jpg]]</div>

'''Description:''' FFmpeg supports hardware accelerated decoding through the internal hwacel API. Currently supported system hardware acceleration APIs are VA-API (Linux), DXVA2 (Windows) and VDA (MacOS X). However, the current approach requires client applications to allocate the underlying resources (e.g. hardware surfaces and context) themselves, and handing them over to FFmpeg. This incurs a few limitations: this is not scalable to new codecs, i.e. this requires new tokens for each newly supported codec; this incurs extra work in the client application, which tends to be duplicated over several client applications; and this prevents efficient fallback to software decoding mode if the hardware cannot handle a particular codec specification.

The goal of this project is to revamp the FFmpeg Hardware Acceleration API so that hardware resources are allocated and managed in the library, thus requiring the client application to only provide a single hardware context/device handle; provide a way to fallback early to software decoding mode if the underlying hardware won't be able to handle the bitstream; and make it possible to select a hardware accelerator by ID and not polluting the PixelFormats namespace.

'''Expected results:'''
* FFmpeg core library (libavcodec):
** Core API extensions and improvements
*** Add open/close hooks in a way that is backwards compatible with hwaccel v1 enabled applications
*** Add new tokens describing hardware accelerators
*** Add new flags exposing HW capabilities like download/upload
*** Investigate the benefits or impacts to provide a global map/unmap capability to FFmpeg video buffers
** Port hwaccels to v2 infrastructure
*** Port VA-API decoders to v2 infrastructure
*** Validate that VA-API decoders still work with existing applications supporting hwaccel v1
*** Provide download capability through ''vaGetImage()''
*** Validate that ffplay can support this feature with minor changes, and definitely no change to the existing SDL renderer
*** Port VDPAU decoders to hwaccel v2 (optional), and investigate ways to preserve compatibility with older applications

* FFmpeg applications:
** Integrate hardware acceleration into ffplay
*** Create a video-output (VO) infrastructure to ffplay
*** Port the SDL renderer to the new VO infrastructure
*** Add support for VA-API: VA renderer through ''vaPutSurface()'', add -hwaccel option to select "vaapi" renderer
*** Add support for VDPAU (optional): VDPAU renderer through ''VdpPresentationQueueDisplay()''
** Integrate hardware acceleration into ffmpeg
*** Add support for VA-API: use the VA/DRM API for headless (no-X display) decoding, use libudev to determine the device to use

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, hardware supporting VA-API.

'''Qualification Task:''' Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)

'''Backup Mentor:''' Hendrik Leppkes (''nevcairiel'' on IRC)

<br clear="all">

== Hardware Accelerated Video Encoding with VA-API ==

'''Description:''' FFmpeg already supports hardware accelerated decoding for multiple codecs but still lacks support for hardware accelerated encoding. The aim of the project is to add support for encoding with VA-API specifically, while keeping a generic enough approach in mind so that other hardware accelerators (TI-DSP, CUDA?) could be supported as well. This means that new ''hwaccel'' hooks are needed and two operational modes are possible: either ''(i)'' driver or hardware pack headers themselves, or ''(ii)'' lattitude is left to perform this task at the FFmpeg library level.

'''Expected results:''' Allow MPEG-2 and H.264 encoding with VA-API, while supporting variable bitrate (VBR) by default, and allowing alternate methods like constant bitrate (CBR) or constant QP (CQP) where appropriate or requested.
* MPEG-2 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** (Optionally) add support for interlaced contents
* H.264 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** Add support for constant QP (CQP, i.e. handle the ''-cqp'' option)
** Add support for more than one reference frame, while providing/using API to query the hardware capabilities
** Work on HRD conformance. May require to write an independent tool to assess that
** (Optionally) add configurability of the motion estimatation method to use. Define new types for HW accelerated encoding with at least two levels/hints for the accelerator.
* FFmpeg applications:
** Define common hwaccel interface for encoding
** Add initial support for hardware accelerated encoding to the ''ffmpeg'' application

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, hardware supporting VA-API for encoding.

'''Qualification Task:''' Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)

'''Backup Mentor:''' Tushar Gohad

<br clear="all"/>

== AAC Improvements ==

'''Description:''' FFmpeg contains an AAC encoder and decoder, both of them can be improved in various ways. This is enough work for more than one GSoC project, so one part of your submission would be to define on which task exactly you want to work.
* AAC LD decoder
* AAC BSAC decoder: This has already been started, but the existing decoder still fails on many samples
* AAC SSR decoder
* AAC 960/120 MDCT window
* AAC multi-channel encoding

'''Qualification Task:''' See the FFmpeg bug tracker for AAC issues, fixing one of them or rebasing the existing incomplete BSAC decoder for current git head or fixing one or more existing bugs are possible qualification tasks.

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, knowledge about transform based audio coding would be useful.

'''Mentor:''' Baptiste Coudurier (''bcoudurier'' on IRC)

'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)

<br clear="all"/>

== DTS / DCA Improvements ==

'''Description:''' FFmpeg contains a DTS decoder.
* DTS-HD decoder improvements: A possible qualification task is to implement ticket [https://ffmpeg.org/trac/ffmpeg/ticket/1920 #1920]
** Add support for X96 extension (96khz)
** Add support for XLL extension (lossless)
** Add support for pure DTS-HD streams that do not contain a DTS core
** Add support for multiple assets
** Add support for LBR extension

'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.

'''Mentor:''' Benjamin Larsson (''merbanan'' on IRC)

'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)

<br clear="all"/>

= Unmentored Projects =

This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also [[#Your_Own_Idea|propose their own project]].

== glplay ==

<div class="floatleft">[[Image:Opengl_logo.jpg]]</div>

'''Description:''' The SDL library that is used by FFplay has some deficiencies, adding OpenGL output to FFplay should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which may be relicensed under the LGPL.

'''Mentor:''' TBD Backup: Reimar Döffinger

<br clear="all">

== TrueHD encoder ==

'''Description:''' FFmpeg currently does not support encoding to one of the lossless audio formats used on Bluray discs. This task consists of implementing a TrueHD encoder that allows to losslessly encode audio to play it on hardware devices capable of TrueHD decoding.

== Opus decoder ==

<div class="floatright">[[Image:Opus.png]]</div>

'''Description:''' Opus decoding is currently supported through the external libopus library
* Write a native decoder, continue working on the existing unfinished implementation
A possible qualification task is to port the existing incomplete decoder to current git head and improve it to show that you are capable of working on this task.

== VC-1 interlaced ==

'''Description:''' The FFmpeg VC-1 decoder has improved over the years, but many samples are still not decoded bit-exact and real-world interlaced streams typically show artefacts.
* Implement missing interlace features
* Make more reference samples bit-exact
As a qualification task, you should try to find a bug in the current decoder implementation and fix it.

== JPEG 2000 ==

<div class="floatleft">[[Image:Jpeg2000.jpg]]</div>

'''Description:''' FFmpeg contains an experimental native JPEG 2000 encoder and decoder. Both are missing many features, see also the FFmpeg bug tracker for some unsupported samples.
Work on an issue (for example from the bug tracker) as a qualification task to show that you are capable of improving the codec implementation.

<br clear="all">

== VP7 ==

'''Description:''' Not many [http://samples.mplayerhq.hu/V-codecs/VP7/ VP7 samples] are in the wild, but no open-source decoder exists although a [http://multimedia.cx/mirror/VP7_Data_Format_and_Decoder_Overview.pdf specification] exists. Write a decoder that reuses as much as possible of existing FFmpeg code: it is likely that functions of the existing decoders for On2-based formats will be useful.

== VP8L ==

'''Description:''' [[VP8L]] is a lossless format used in WebP. There is no support for this in FFmpeg.

== Your own idea ==

A student can propose a project. Ideas can also be found by browsing bugs and feature requests on our [https://ffmpeg.org/trac/ffmpeg/ bug tracker]. The work should last the majority of the GSoC duration, the task must be approved by the developers, and a mentor must be assigned.

Students can discuss an idea in the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing-list], the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins [[User:Stefanosa|Stefano Sabatini]] or [[User:Llogan|Lou Logan]] for more information.

[[Category:FFmpeg]]

Otrkey

2013-02-14T02:17:28Z

Summary:

* Website: http://www.onlinetvrecorder.com/
* Samples: http://otr.datenkeller.at/ http://otr.dwolp.de/
* (public domain?) Decoder: http://pyropeter.github.com/otrtool/

.otrkey files are encrypted video files from the onlinetvrecorder website.
the files are publically available, but require a key to decrypt them.

similar to wmv drm, it stores a key inside the file and requires a second key from the website.

A gpu brute force cracker may be able to decrypt such files.

Libav Summer Of Code 2013

2013-02-12T00:08:36Z

Summary: /* 1st Tier Project Proposals */

== How it works ==

Google's Summer of Code program is simple: you (the student) work on a project, full-time, during the whole summer, and you get assistance (advice, mentoring) from a seasoned Libav developer who knows the project and has considerable standing in the community. By doing so, you'll learn to operate in an open source project, you'll get relevant coding experience, and you'll have a chance at earning money while doing fun stuff during the summer. So, you need a project, a mentor, a qualification task (see below) so we can quickly assess how good a candidate we feel you'll be for the program, and then you can apply.

=== Selecting a project ===
Below, you'll find two lists of projects:
* Projects with a mentor
* Projects without a mentor
If you choose a project with a mentor, talk to that mentor (see below) and select a suitable qualification task. Once completed, you're eligible for participating in our Summer of Code program. If you choose a project without a mentor, your first job is to find a mentor (see below). Then, once you've found a mentor, continue as before. If you don't like any of the projects, you're free to define your own project and find a mentor as mentioned before (see below for caveats).

=== Contacting developers/mentors ===
Once you've found a project (with or without mentor), start talking to the developers of the Libav project. We can often be found on IRC, and you can talk to us on the mailing lists as well. Hop on irc.freenode.net, channel #libav-devel, or mail us at libav-devel@libav.org. There, you'll be able to ask around for persons to guide you on projects without a mentor if you need to. If you're trying to define your own project, explain (with reasonable amount of detail) what you intend to achieve and why you think your project should be in our Summer of Code program. Once you've found a mentor, you're good to start your qualification task.

Note that the self-selected mentor needs to have considerable standing in the community to be eligible for mentoring. Likewise, if you choose to define your own Summer of Code project, some community members of considerable standing need to vouch for your project.

=== Your qualification task ===
The goal of a qualification task is to see if the mentor and student feel that, together, they will be able to finish the project of their choice. More specifically, the mentor will want to test whether the student has the skills and work ethics to complete a large coding project in a limited amount of time. The student will want to make sure that the mentor provides (useful) assistance when necessary. Therefore, students should select a mentor and a Summer of Code project before starting their work on a qualification task. The qualification task is often (but not necessarily) related to the selected project. For example, if your project will be to reverse engineer a new video codec, the qualification task may be to write a partial bitstream parser for that codec. If your project is to write a muxer for a container format, the qualification task may be to write the code to write the stream header.

There will be a second qualification task for every student: Pick a file of moderate size and reformat it in proper K&R style. The goal of this task is twofold: First it familiarizes students with the style that they will have to write their code in, second it demonstrates that students are able to submit patches from Git and go through our review process.

=== Applying ===
While you're working on your qualification task, apply at http://www.google-melange.com/. The degree (and detail) with which you've finished your qualification task will determine how likely your project is to be selected. In the past few years, students that completely finished their qualification task always got selected as Summer of Code students, but that may vary depending on the number of slots we are allotted by Google, and the number of students that apply.

== 1st Tier Project Proposals ==
1st tier project proposals are project ideas that are reasonably well defined '''AND''' have a mentor.

=== Hardware Acceleration API ===

Libav has two different ways to provide hardware acceleration wrapping.

One, deprecated, is to make a full decoder and signal its capabilities somehow, the other, called '''hwaccel''', provides some API hooks to integrate the hardware acceleration in the stock decoder.

Currently '''hwaccel''' has some provision for transparent fallback, but lacks a clean way to setup and pass configuration options to it.

The project should improve hwaccel in this regard and add support for more accelerators.

* Draft the API (that will require knowledge of libavcodec).
* Extend the current implemented support to leverage the new features.
* Implement Freescale VPU support.
* Implement TI dce support.

'''Mentor: [[User:lu_zero|Luca Barbato]]'''

=== VP9 Native Decoder ===

The WebM team is refining the evolution of their VP8 codec using a number of additional tools and tunings. The project involves providing a baseline
decoder as first step and optimizing it to be at least as fast as the libvpx one on one of the most used architectures (ARM and x86_64).

[http://tools.ietf.org/html/draft-grange-vp9-bitstream-00 current draft]

'''Mentor: [[User:lu_zero|Luca Barbato]]'''

=== WebP Lossless Native codec and WebP Lossy decoder ===

WebP is a promising image format that could supercede [[JPEG]] and [[PNG]] as common format for web images, it leverages VP8 for lossy
encoding and uses a relatively simple original format for the lossless part.

The project is split in a number of easy tasks, such as writing a decoder for both the lossy and the lossless mode and a much harder task that is providing a good encoder for the lossless part, at least. (Exceptional praise for producing a good lossy encoder but it is not required.)

'''Mentor: [[User:lu_zero|Luca Barbato]]'''

=== Native DVD support ===

Most of the open source support for DVDs is available through libdvdread and libdvdnav. Currently there is some effort to unify them
in a single library (libdvd5). An additional step would be factoring the non-interactive part of into the libav codebase to leverage even more code and leave as stand alone library the parts that require some kind of interaction.

'''Mentor: [[User:flameeyes|Diego Pettenò]]'''

=== Restructuring the MPEG video family of codecs ===

Many encoders and decoders currently use the MpegEncContext structure and its associated API. This structure is a huge monolithic blob which combines the features of all those encoders and decoders, thus making it very fragile and hard to understand.

The goal of this project is splitting out parts of the MpegEncContext API into self-contained structures that would be easier to grasp, while still allowing code reuse where appropriate.

'''Mentor: [[User:elenril|Anton Khirnov]]'''

=== Spin off build system into a separate project ===

Our build system is neat enough to make into a more general solution to be reused by other projects.

The goal of this project is to achieve exactly that. Intermediate steps will be reading, understanding and documenting the current build system, refactoring parts that can be generalized further and finally making a prototype implementation for libpostproc.

You will require skills in POSIX shell, GNU Make and a firm command of English.

'''Mentor: [[User:DonDiego|Diego Biurrun]]'''

'''Co-Mentor: [[User:lu_zero|Luca Barbato]]'''

=== The Grand Refactoring (working title) ===

Many parts of the libav codebase are still unnecessarily monolithic. This makes custom builds that only support a subset of the multitude of formats we have larger than they need to be and slows down the compiling time on multicore systems.

The goal of this project will be to locate parts that can be separated and refactor the code so that each subpart can be compiled standalone so as to not increase the size of a custom configuration without the part.

A suitable qualification task for this project is picking a simple encoder/decoder pair or some part of dsputil (harder) and splitting it cleanly.

'''Mentor: [[User:DonDiego|Diego Biurrun]]'''

'''Co-Mentor: [[User:lu_zero|Luca Barbato]]'''

=== Rewrite the ASF muxer and demuxer ===

Our current ASF muxer and demuxer were written using reverse engineering when the specification was not yet available. Because of that, they are hard to understand, contain many bugs and do not support the format fully. The goal of this project would be to rewrite those old parts of the ASF muxer and demuxer so that the new code:
* conforms to the specification;
* is clear and readable;
* has less bugs;
* supports all useful features of the format.

A qualification task would be adding support for chapters (called 'markers' in ASF) to the current ASF muxer.

'''Mentor: [[User:elenril|Anton Khirnov]]'''

=== Support for concatenation in avconv ===

One of the most important features still missing from our multimedia transcoder avconv is proper support for concatenating (joining) media streams. The goals of this project would be to:
* add support for concatenating input streams when transcoding
* add support for concatenating outputs of the filtergraphs when transcoding
* (if time allows) add support for concatenating input streams when doing streamcopy.

A qualification task would be adding support for looping of arbitrary input files, to familiarize the student with avconv.

'''Mentor: [[User:elenril|Anton Khirnov]]'''

=== Rewrite the RealMedia demuxer ===

The current RM support is workable but it started from early attempts to support the yet to be documented format.
Some features (like multirate files) are not supported properly either.
The project aims to be a full blown rewrite to make a feature complete demuxer.

'''Mentor: [[User:Kostya|Kostya Shishkov]]'''

=== Extend the libavfilter filter collection ===
Libavfilter, our library for audio and video filtering, is slowly reaching the level of mature and usable code, but the number of filters in it is still very small. The goal of this project would be to find the most useful filters (under appropriate licences) in projects like Avisynth, SoX, ImageMagick, etc. and port them to libavfilter. Depending on the student's experience level, he or she can also write the filters from scratch.

A qualification task would be writing a very simple filter, to make the student familiar with libavfilter.

'''Mentor: [[User:elenril|Anton Khirnov]]'''

'''Co-Mentor: [[User:Jruggle|Justin Ruggles]]'''

=== Adobe DNG Decoder (Basic Support) ===

Adobe Digital Negative (DNG) is an attempt at a universal file format for raw camera images. Most camera manufacturers have their own proprietary raw image format. Adobe provides tools for converting these to DNG with minimal or no loss of information for more reliable long-term support in a format with an open specification.

The project goal would be to add features required for basic support of DNG files. Some of these include:
* test/improve TIFF and LJPEG 16-bit decoding support
* implement both variants of JPEG-in-TIFF in the TIFF decoder
* add basic handling for Bayer CFA pixel format(s), including demosaicing
* conversion from camera colorspace to RGB
* export of DNG/TIFF/Exif metadata

Resources:
* [http://en.wikipedia.org/wiki/Digital_Negative Wikipedia Article]
* [http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf Specification]
* DNG samples can be created from other raw formats using the free [http://www.adobe.com/products/photoshop/extend.displayTab2.html DNG Converter] program
* A good place to find raw camera samples is http://www.imaging-resource.com

'''Mentor: [[User:Jruggle|Justin Ruggles]]'''

=== Streaming protocol improvements ===

* Implement support for producing data for Adobe HTTP Dynamic Streaming and MPEG DASH
* Implement support for receiving the same

Alternatively you can also implement support for another new streaming protocol we currently don't support, or improve the ones we currently support.

'''Mentor: [[User:mstorsjo|Martin Storsjö]]'''

'''Co-Mentor: [[User:lu_zero|Luca Barbato]]'''

== 2nd Tier Project Proposals ==

None of the following project have a mentor confirmed yet.

=== Language bindings ===

Provide bindings for non-C language. The languages can be any among Perl, Python, Ruby, Go and such.

Tasks

* low-level abstraction with a 1:1 mapping to the C api.
* high-level abstraction matching the wrapping language idoms.
* example code leveraging it.

The bindings should be using only the public api.

'''Mentor: ???'''

=== Assembly Unit Testing Framework ===
* Libav has a lot of assembly and not enough tests for it. Your job is to write a unit testing framework for assembly.
* The framework should work across all supported architectures and operating systems.
* The framework should measure exactly how fast an individual function is (e.g. using START/STOP_TIMER).
* The framework should be able to test functions in isolation.
* x264's checkasm can be used as a reference.
* The qualification task will be to implement at least one unit test and have an idea of how to do the rest.

'''Mentor: Daniel Kang''' (Jumpyshoes on #libav-devel@chat.freenode.net; daniel.d.kang@gmail.com -- ping me on IRC and email me).

=== DTS-HD decoder ===

ETSI released the [http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf specification].
Your job is to complete the existing decoder with the following features.

(1) Add support for mixed Core + DTS-HD stream structure
(DtsCoreFrame+DtsHdFrame+DtsCoreFrame+DtsHdFrame+...), used by Blu-Ray main
and commentary tracks.
(2) Add support for XXCh extension (6.1 and 7.1 channels).
(3) Add support for X96 extension (96khz).
(4) Add support for XLL extension (lossless).
(5) Add support for a pure DTS-HD stream structure
(DtsHdFrame+DtsHdFrame+DtsHdFrame+...), used by Blu-Ray PiP tracks.
(6) Add support for XBR extension (extra bitrate).

'''Mentor: Benjamin Larsson'''

=== MPEG-4 ALS Roundup ===

This task is to update and enhance the existing ALS decoder as well as integrate
and enhance the rudimentary encoder found at:
https://github.com/justinruggles/FFmpeg-alsenc

Possible features are:

* implement rls-lms in the decoder
* do correct channel layout/sort handling in the decoder
* update to current master
* use codec private options
* implement encode2(), setting pts and duration
* document options and examples in encoders.texi
* come up with a good set of encoding tests for FATE
* implement mcc/channel sort in the encoder
* implement rls-lms in the encoder
* implement float support

'''Mentor: [[User:Jruggle|Justin Ruggles]]'''

=== Opus Decoder ===

Implement an independent Opus decoder using the publicly-available [http://tools.ietf.org/html/draft-ietf-codec-opus-11 specification]

* The reference source code should only be used as a normative document reference when required (i.e. this should not just be a port of libopus)
* Fully support Ogg/Opus mapping: https://wiki.xiph.org/OggOpus
* Handle CELT, SILK, and Hybrid modes (including transitions)
* Handle more than 2 channels
* (optional) Make sure opus-in-mkv and opus-in-nut work

The initial SILK code had been written as result of the previous attempt to support Opus

'''Mentor: [[User:Jruggle|Justin Ruggles]]'''

=== On2 VP7 decoder ===

VP7 is a DCT-based video codec. At the moment, it seems to lack a reference implementation, but we do have a spec and may take hint from libavcodec's VP8 and VP6 decoder and libvpx. MPlayer can decode VP7 by loading a binary. To begin working on the project, one has to setup a reference decoder against which to compare our output. This can be done by either writing a wrapper for the binary or (maybe) hacking libvpx itself.

You might want to discuss with us how and where to start. Drop by on IRC if you need help. It is not as difficult as it sounds.

* [http://multimedia.cx/mirror/VP7_Data_Format_and_Decoder_Overview.pdf Specification]
* [http://samples.libav.org/V-codecs/VP7/ Samples]

'''Mentor: [[User:Shahriman|Mashiat Sarker Shakkhar]]'''

=== Rewrite avserver ===

Libav needs a better system to serve streams, the current codebase had a number of design defect showing its age. The new avserver should be written from scratch, leveraging the knowledge piled up.

The implementation will be incrementally complex and possibly modular.

* Write a simple rtsp, http, rtmp redirector (listen for publish/announce and rebroadcast the received streams)
* Add the capability to serve on-demand content reading from a single path

Ideally the first implementation can be made using a poll/event loop and then moved to use threads.

avconv gained the ability to listen for incoming rtmp and rtsp connection as result of the previous year project.

'''Mentor: [[User:lu_zero|Luca Barbato]]'''

=== DTS-LBR decoder ===

This is the DTS equivalent to E-AC3 but not technically related to DTS/DTS-HD.
It is found in the following [http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf specification]. The task is to find a way of making the official decoder decode just the LBR stream (and not mix it into the main audio) and use that to verify decoder compliance of the decoder you wrote.
The spec may be incomplete or require parts to be reverse engineered from the binary.

'''Mentor: ???'''

[[Category:Libav]]

VP9

2013-01-29T07:42:14Z

Summary: initial information

* FOURCCs: VP90
* Description: ???
* Code : http://git.chromium.org/gitweb/?p=webm/libvpx.git;a=summary
* Samples: ???

VP9 is an open and royalty free video compression standard being developed by Google. VP9 had earlier development names of Next Gen Open Video (NGOV) and VP-Next. VP9 is a successor to VP8.

A test (not pretending to be scientific or reliable) from 17th January 2013 is present in the discussion thread
http://forum.doom9.org/showthread.php?p=1610941

Daala

2013-01-29T07:28:02Z

Summary: initial information

* FOURCCs: ?
* Description: https://wiki.xiph.org/Daala
* Code in progress : git clone https://git.xiph.org/daala.git
* Samples: none yet

Daala is the current working name of a next generation video codec. The effort is a collaboration between Mozilla Foundation, Xiph.Org Foundation and other contributors.

The goal of the project is to provide a free to implement, use and distribute digital media format and reference implementation with technical performance superior to h.265.

The development suffered at the end of 2012 due to the developers being involved in short term [[Opus]]-related activities. The availability of [[VP9]] since January 2013 also may influence the amount of resources being allotted to Daala.

Dalet Digital Media Solutions

2013-01-09T06:16:50Z

Summary:

Website: http://www.dalet.com/

[[Category:Multimedia-related Companies]]

AJA Kona 10-bit RGB Codec

2013-01-08T07:02:53Z

Summary:

* FourCCs: R10k, R10g
* Company: [[AJA Video Systems]]

This is a raw 10-bit RGB video codec. Each red, green, and blue color component is 10 bits wide. RGB pixels contain 30 bits of information and are stored in 32-bit blocks (so there are 2 unused bits). Each group of 32 bits is laid out as:

rrrrrrrr rrgggggg ggggbbbb bbbbbbxx

[[Category:Video Codecs]]
[[Category:Intermediate Video Codecs]]

Internet Engineering Task Force

2013-01-03T06:11:27Z

Summary: more standards; formatting reorg

* Website: http://www.ietf.org/

The Internet Engineering Task Force (IETF) is responsible for developing internet standards. The IETF publishes standards for a variety of multimedia technologies:

* RFC 3533: [[Ogg]] Container Format
* RFC 6716: [[Opus]] Audio Codec
* RFC 3550: [[RTP]]
* RFC 5215: [[RTP]] Payload Format for [[Vorbis]] Encoded Audio
* RFC 2326: [[RTSP]]
* RFC 6386: [[VP8]] Video Codec

[[Category:Multimedia-related Companies]]

Bink Video 2

2012-11-17T14:15:52Z

Summary: it's missing, obviously

* FourCC: KB2d (in [[Bink Container]])
* Company: [[RAD Game Tools]]
* Samples: ''ADD ME''

Bink Video 2 is a successor to [[Bink Video]].

(De)Compression details are presently unknown.

[[Category:Video Codecs]]
[[Category:Game Formats]]
[[Category:Formats missing in FFmpeg]]

VP8L

2012-10-29T01:49:49Z

Summary:

* Company: [[Google]]
* Specification: https://developers.google.com/speed/webp/docs/webp_lossless_bitstream_specification
* Sample implementation: https://code.google.com/p/webm/source/browse/src/dec/vp8l.c?repo=libwebp&r=2fc130157767e529fedd42fad0c70c13804058ab

From Google's [[WebP]] page: "Lossless WebP compression uses already seen image fragments in order to exactly reconstruct new pixels. It can also use a local palette if no interesting match is found. This palette is continuously updated to re-use recent colors. This compression mode is named "VP8L" and shares some common features with the so-called LZ77 compression algorithm."

[[Category:Video Codecs]]

MUSICompress

2012-10-29T01:16:27Z

Summary:

* Extensions: mcp
* Company: Soundspace Audio
* Homepage: [http://members.aol.com/sndspace Soundspace]
* [https://groups.google.com/forum/?fromgroups=#!topic/comp.music.research/uCu3xAi2k7o Introductory letter from author]
* Publication: A. Wegener, “MUSICompress: Lossless, lowMIPS audio compression in software and hardware”, in Proc. Int. Conf. Signal Processing Applications and Technology, San Diego, CA, USA, 1997 September 14-17.

Implementation also called WaveZip.

[[Category:Audio Codecs]]
[[Category:Undiscovered Audio Codecs]]
[[Category:Formats missing in FFmpeg]]

Sonarc

2012-10-29T01:06:11Z

Summary:

* Description: [http://www.speech.cs.cmu.edu/comp.speech/Section3/Software/sonarc.html Sonarc]

[[Category: Audio Codecs]]
[[Category: Lossless Audio Codecs]]
[[Category: Undiscovered Audio Codecs]]

WavArc

2012-10-29T01:04:38Z

Summary:

* Description: [http://www.firstpr.com.au/audiocomp/lossless/wavarc/ WavArc]

[[Category: Audio Codecs]]
[[Category: Lossless Audio Codecs]]
[[Category: Undiscovered Audio Codecs]]

Opus

2012-09-14T18:08:07Z

Summary:

* Company: [[Xiph]], IETF
* Extension: .opus
* Specification (codec): https://tools.ietf.org/html/rfc6716
* Specification (Ogg encapsulation): https://tools.ietf.org/html/draft-terriberry-oggopus-01
* Reference implementation: http://www.opus-codec.org/downloads/

Open standard for low-latency, scalable lossy audio coding based on the speech-oriented [[SILK]] codec and the low-latency [[CELT]] codec. Opus can transition between a linear prediction codec at low bitrates and a transform codec at high bitrates and supports a hybrid mode for a small bitrate range. Opus defaults to a latency that's 1/10th that of earlier music codecs (20ms vs. 200ms), making it possible for use in VoIP, and yet performs better in listening tests.

Note that Opus does not seem to perform better than [[CELT]] for music at bitrates over 75 kbit (http://www.hydrogenaudio.org/forums/index.php?showtopic=97913&st=0). This means that the unconditional merge of [[CELT]] into Opus doubled the complexity (both code size and CPU usage) even for applications which do not benefit from the addition.

== Links ==
* http://www.opus-codec.org/
* https://wiki.xiph.org/OpusFAQ

[[Category:Audio Codecs]]
[[Category:MDCT Audio Codecs]]
[[Category:Vocoders]]
[[Category:Multichannel Audio Codecs]]

Xact

2012-08-17T04:29:07Z

Summary: It's also a Multimedia API, I guess

* Extensions: xwb, xsb, xgs
* Company: [[Microsoft]]
* Website: http://msdn.microsoft.com/en-us/library/bb174772.aspx
* Wikipedia article: https://en.wikipedia.org/wiki/Cross-platform_Audio_Creation_Tool

Cross-platform Audio Creation Tool (XACT) is a audio framework released as part of the DirectX SDK.

Distributed XACT projects consist of three file formats:

== Wave Banks (.xwb) ==
A file format containing a collection of waves
* Waves - the raw wave data in [[Microsoft Wave|WAV]], [[Audio Interchange File Format|AIFF]], [[Windows Media Audio 9|WMA]], [[Microsoft xWMA|xWMA]] or [[XMA]] formats

Open source parsers:
* [http://aluigi.altervista.org/papers.htm#unxwb unxwb]
* [https://github.com/mono/MonoGame/blob/develop3d/MonoGame.Framework/Audio/WaveBank.cs MonoGame]

== Sound Banks (.xsb) ==
A collection of sounds and cues

* Sounds - a sound has one or more waves together with properties like volume and pitch. Sounds are made up of tracks.
** Tracks - tracks are made up of events E.g. the simplest track has a Play Wave event
** Events - various actions that take place within a track. Actions include: Play, Stop, Set Volume, Set Pitch etc.
* Cues - a cue is used in code to trigger sounds. Each cue is made up of one or more sounds

<pre>
sturct SoundBank {
struct SoundBankHeader header;

struct WaveBankNameTableEntry[header.numWaveBanks] wavebankNames;
struct SoundEntry[header.numSounds] sounds;
struct SimpleCueEntry[header.numSimpleCues] simpleCues;
struct ComplexCueEntry[header.numComplexCues] complexCues;

//...
}

struct SoundBankHeader {
uint32_t magic; // "SDBK"
uint16_t toolVersion; // 0x2E
uint16_t formatVersion; // 0x2B
uint16_t crc; // fcs16 checksum of all following data
uint32_t lastModifiedLow;
uint32_t lastModifiedHigh;
uint8_t platform; //0x12 ??
uint16_t numSimpleCues; //0x13
uint16_t numComplexCues; //0x15
uint16_t unkn3; //0x17
uint16_t numTotalCues; //0x19 ??
uint8_t numWaveBanks; //0x1b
uint16_t numSounds; //0x1c
uint32_t cueNameTableLen; //0x1e

uint32_t simpleCuesOffset; //0x22
uint32_t complexCuesOffset; //0x26
uint32_t cueNamesOffset; //0x2a
uint32_t unknOffset; //0x2e
uint32_t variationTablesOffset; //0x32
uint32_t unknOffset2; //0x36
uint32_t waveBankNameTableOffset; //0x3a
uint32_t cueNameHashTableOffset; //0x3e 16-bit hashes each
uint32_t cueNameHashValsOffset; //0x42
uint32_t soundsOffset; //0x46

char name[64];
};

struct WaveBankNameTableEntry {
char name[64];
}

sturct SoundEntry {
uint8_t flags; //0x0
uint16_t category; //0x1 ???
uint8_t unkn2; //0x3
uint16_t volume; //0x4 maybe pitch
uint8_t unkn3; //0x6
uint16_t entryLength; //0x7

if flags 1 (complex sound):
//"clips"
uint8_t numClips; //0x9
else:
uint16_t trackIndex; //0x9
uint8_t waveBankIndex; //0x11

if flags in 2/4/8:
uint16_t extraLen; //0x12
//then a rpc table for each flag
if flags in 16:
uint16_t extraLen; //0x12 7

//dsp preset table
uint8_t num;
uint32_t unkn[num];

if flags 1:
struct {
uint8_t unkn;
uint32_t clipOffset;
uint32_t unkn2;
}[numClips]
}

sturct SoundClip {
uint8_t numEvents;
struct ClipEvent[numEvents];
};

struct ClipEvent {
uint32_t eventInfo; //event id in lower 5 bits
}

//event id 1, size 16
struct EventPlayWave : ClipEvent {
char unkn[4]; //0x4
uint16_t trackIndex; //0x8
uint8_t waveBank; //0xa
uint8_t unkn; //0xb
uint16_t unkn2; //0xc
uint16_t unkn3; //0xe
};

//event id 3, size at least 13
struct EventPlayWaveVariation : ClipEvent {
char unkn[5];
int16_t unkn2;
int16_t unkn3;

struct VariationTable; ///wtf!?
}

//event id 4, size 40
struct EventPlayWavePitchVolumeFilterVariation : EventPlayWave {
struct PitchVolumeFilterVariation;
}

struct PitchVolumeFilterVariation {
int16_t unkn;
int16_t unkn2;
uint8_t unkn3;
uint8_t unkn4;
float unkn5;
float unkn6;
float unkn7;
float unkn8;
uint16_t flags;
}

struct SimpleCueEntry { //cue that maps straight to a sound
uint8_t flags;
uint32_t soundOffset;
};

//size 15
struct ComplexCueEntry {
uint8_t flags;
uint32_t variationTableOffset;
uint32_t transitionTableOffset;
struct InstanceLimit;

};
struct InstanceLimit {
uint32_t unkn;
uint8_t unkn2;
uint8_t unkn3; //dono if here or in ComplexCueEntry
};

enum VariationTableType {
Wave,
?,
?,
?,
CompactWave
}
struct VariationTable {
uint32_t flags;
//numEnties in lower 16 bits
//flags in upper 16 bits
// -table type in bits 3,4,5
uint8_t unkn;
uint16_t unkn2;
uint8_t unkn3;

table 0:
struct {
uint16_t trackIndex; //?
uint8_t waveBank;
uint8_t weightMin; //?
uint8_t weightMax; //?
}[numEntries];

table 1:
struct {
uint32_t soundOffset;
uint8_t weightMin; //?
uint8_t weightMax; //?
}[numEntries]

table 2:
struct {
uint32_t unkn;
uint8_t weightMin; //?
uint8_t weightMax; //?
}[numEntries]

table 3:
struct {
uint32_t soundOffset; //?
float weightMin;
float weightMax;
uint32_t unkn;
}[numEntries]

table 4:
struct {
uint16_t trackIndex; //?
uint8_t waveBank;
}[numEntries];
};

struct CueNameHashVal {
uint32_t nameOffset;
uint16_t unkn;
}
</pre>

== Global Settings (.xgs) ==
Defines rules and settings for sounds.

* Categories - sounds can be assigned to a category (only one each) that specifies certain rules like number of instances along with settings like volume. You could create a category for the sounds of one character in your game so they all have the same volume. There are three predefined categories: global, default and Music.
* Variables - these can be defined in the design stage and then referenced by the programmer in code to control Run-Time Parameter Controls
** Run-Time Parameter Controls - also known as 'sliders'. These allow the control of sound parameters as the sound plays. For example they could be used to control the pitch of a car engine sound so as the accelerator is pressed the pitch is changed
* DSP Effect Path Presets (DSPs) - allow effects like reverb to be applied to sounds
* Compression Presets - compression can be applied to waves or wave banks

Open source parsers:
* [https://github.com/espes/MonoGame/blob/develop3d/MonoGame.Framework/Audio/AudioEngine.cs MonoGame]

[[Category:Container Formats]]
[[Category:Game Formats]]
[[Category:Multimedia APIs]]

Micronas SC4

2012-07-25T19:22:43Z

Summary:

* Codec ID: 0x350
* Samples: http://samples.libav.org/A-codecs/Micronas%20SC4/

This is a simple speech codec supporting 32000, 16000, 11025 and 8000 Hz sampling rates.

It uses adaptive 6-point filtering and 1-5 or 30-byte blocks (interleaved in case of stereo).

[[Category:Audio Codecs]]
[[Category:Undiscovered Audio Codecs]]

FFmpeg Summer Of Code In Space 2012

2012-07-05T20:06:00Z

Summary: /* [[Libavfilter]] misc extensions */ Remove concatenat, it has been already integrated

ESA Summer of Code in Space 2012 (SOCIS 2012) is a program run by the European Space Agency. It aims at offering student developers stipends to write code for various space-related open source software projects. For more information read the page:

http://sophia.estec.esa.int/socis2012/

FFmpeg has been accepted in the SOCIS 2012 program:

http://sophia.estec.esa.int/socis2012/?q=node/13

and has been allocated one slot for one student/task.

If you want to apply as a student please read these documents:
* http://sophia.estec.esa.int/socis2012/?q=faq
* http://sophia.estec.esa.int/socis2012/?q=tos
* http://sophia.estec.esa.int/socis2012/?q=student_agreement
* http://sophia.estec.esa.int/socis2012/timeline

'''Candidate students application deadline is on 27/07 (11am UTC).'''

The FFmpeg administrator for SOCIS 2012 is Stefano Sabatini (<stefasab@gmail.com>, saste on IRC), and Michael Niedermayer (<michaelni@gmx.at>, michaelni on IRC) is the backup administrator. Please contact them if you have any related questions.

== Qualification tasks ==

For us to consider your application for SOCIS we require a completed qualification task. Many Summer-of-Code projects (in the list below) have specific qualification tasks. These tasks are meant to make you familiar with the code that you will be working with, are at approximately the same difficulty level as the actual Summer-of-Code project itself (just a lot smaller), and often already provide you with a jumpstart into your Summer-of-Code project. We suggest the following order of events:
* First, select a Summer-of-Code project (either from the list below, but in some cases you may also come up with your own)
* Second, discuss this project with the person that will mentor it. If a mentor is listed, talk to him on IRC, via email or so. If no mentor is listed, find one by emailing the FFmpeg-devel mailinglist.
* With your mentor, discuss the most appropriate qualification task for your Summer-of-Code project.
If no specific qualification task is listed for your project of interest, you can discuss with your mentor to choose a task from the [[Small FFmpeg Tasks|Small Tasks list]] or the [[Interesting Patches|Interesting Patches list]] instead. If your prospective mentor agrees, please send an email to the FFmpeg-devel mailing list to inform that you are working on it (to avoid duplicated work). The qualification task is considered completed when your patch is accepted to the main Git tree.

Before posting to the FFmpeg-devel mailing list, make sure you read and understand our netiquette guidelines, especially avoid top-posting and thread-hijacking. You should also be familiar with the diff, patch and git programs.

== 1st Tier Project Proposals ==
1st tier project proposals are project ideas that are reasonably well defined '''and''' have a mentor volunteered.

=== [[Libavfilter]] misc extensions ===

Libavfilter is the FFmpeg filtering library. It currently supports audio and video filtering and generation support.

The task would consist of writing or porting audio and video filters and eventually fix/extend libavfilter API and design.

Prerequisites: good C coding skills, familiarity with git/source code control systems, having some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

Qualification task: a port or an implementation from scratch of one or more filters.

The following list shows some ideas and objectives which could be pursued during the task. The effective task objectives will be discussed with the student, according to his/her interests and background.

* Port missing filters from MPlayer (remember to ask the authors of the original filter if it is ok to release them under the LGPL)
* Framework: implement dynamic-reconfiguration of the filterchain, for supporting dynamic size/format changes
* Port filters from other frameworks (mjpeg-tools, effectv, frei0r, virtualdub, vlc, etc...).
* Write wrappers for more image processing libraries and filtering frameworks (libgimp, libgraphicsmagic, weed), extend the libopencv wrapper for supporting more filters (this may need implemented float image support in libswscale)
* Write more filters (possibly starting from already posted filters which for a reason or another were never committed, e.g. fish, eval, posterize, elbg/posterize etc.)
* Write a frequency domain transform filter and inverse frequency domain transform using the DSP utils in libavcodec
* Write a Matlab/Octave/SAGE scripting wrapper (assuming it can be done)
* Add metadata injection in the filtergraph
* Fix [https://ffmpeg.org/trac/ffmpeg/query?status=new&status=open&status=reopened&component=avfilter&col=id&col=summary&col=status&col=type&col=priority&col=component&col=version&order=priority libavfilter trac tickets]

''Mentor: Stefano Sabatini - saste in IRC (possibly with a backup mentor)''

=== [[Error concealment]] improvements ===

When data is damaged beyond the capabilities of forward error correction codes or when it was damaged where there is no error correction like due to failing storage devices / RAM. Then when decoding/viewing the data error concealment can be applied to fill in lost areas. FFmpeg currently supports moderately advanced error concealment for most popular video codecs but lacks it for all image formats. This task is to add high quality error concealment to the image decoders where it's possible and improve resynchronization of the image decoders in light of data errors.

As qualification task, at least one image decoder's error concealment capabilities need to be significantly improved.

Prerequisites: good C coding skills, familiarity with git/source code control systems, having some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

''Mentor: Michael Niedermayer - michaelni in IRC (possibly with a backup mentor)''

== 2nd Tier Project Proposals ==

These proposals lack of a mentor.

=== Finish GSoC projects from previous years ===

Some projects are lingering in the dark unfinished. They should be picked up and made ready for inclusion. These projects are potentially less involved than starting from scratch, but also more useful for FFmpeg since the probability that the projects get finished should be higher. If some of them are deemed too easy, they could be combined.

For the current status of all GSoC projects up to date, see [[FFmpeg Summer Of Code]].

=== Forward Error Correction ===

Add FEC to libavutil.
* RTP FEC, standardised as SMPTE 2022M (Possibly the same as Pro-MPEG FEC) - (not 100% sure if this is for all types of RTP or just some)
* MPEG-TS 16 bytes extra - (not sure where this is standardised - in DVB perhaps?)

I (michaelni) am happy to help mentor this but i don't volunteer as primary mentor.

=== Automatic image classification ===

Research existing literature about image classification and implement the best algorithms in libavfilter with the purpose of allowing autonomous classification of images in terms of containing unexpected, interesting features or simply significant temporal changes. This would allow a probe to quicker and automatically detect interesting things or changes to image and transmit to earth.

I (michaelni) am happy to help mentor this but i don't volunteer as primary mentor.

=== libavfilter star scanner ===

Write a libavfilter that determines from a image of a random patch of the sky / a bunch of stars the exact orientation of the camera.

I (michaelni) am happy to help mentor this but i don't volunteer as primary mentor.

TechSmith Screen Capture Codec 2

2012-07-03T19:12:05Z

Summary: /* Frame format */

* FOURCC: TSC2
* Company: [[TechSmith]]

This codec was developed by TechSmith and used in their screen capturing products.

== Frame format ==

This codec operates on 16x8 macroblocks.

0 - frame type (1- coded, 0 - skip frame)
1 - quantiser1 (0-12)
2 - quantiser2 (0-12)
3 - ???
[macroblock properties]
[macroblock slices]

Macroblock properties:

4 bytes LE - chunk size
rest - RLE-coded mode in bytes in form (val << 6) | (count)

Macroblock slices begin with slice size. If the first byte is odd, then it's the full size. Otherwise read whole 32-bit word and divide it by two in order to obtain real macroblock data size.

Mode meaning: 0/3 - skip block, 1 - select quantiser1, 2 - select quantiser2

Macroblocks encode series of 4x4 DCT blocks (4x2), one component after another.

Block format:

DC - 8 bits for the first block, or VLC (for the difference from the previous DC)
number of AC coefficients - VLC
[AC coefficients] - VLC, bottom 4 bits - skip value, top 8 bits - AC value

Scan order is zigzag:

0 2 3 9
1 4 8 10
5 7 11 14
6 12 13 15

Quantisation:

Q0 Q1 Q0 Q1
Q1 Q2 Q1 Q2
Q0 Q1 Q0 Q1
Q1 Q2 Q1 Q2

where

Q0 = 4 * round(quant * 327.68)
Q1 = 4 * round(quant / 5 * 16384 * 0.1313064328597225) // about 4 * round(quant * 430.265)
Q2 = 4 * round(quant * 32768 * 0.1313064328597225^2 ) // about 4 * round(quant * 564.966)

Unquantising:

val = (quant * (qval << 6) + 0x10000) >> 17

IDCT matrix:

5 5 5 2
5 2 -5 -5
5 -2 -5 5
5 -5 5 -2

[[Category:Video Codecs]]
[[Category:Screen Capture Video Codecs]]

PCoIP

2012-06-25T14:56:44Z

Summary: more

*Company: [[Teradici]]
*Description: http://teradici.com/pcoip/pcoip-technology.php

PCoIP is a real-time Audio/Video/Keyboard/Mouse/USB streaming protocol intended for remote desktop access to physical and virtual machines. It exists primarily in the form of a Terradici hardware encoder/decoder ASIC that may be integrated into graphics cards and thin-clients. A software implementation exists within recent versions of the VMware View Agent and VMware View Client products. According to the website, ''PCoIP protocol continuously analyzes and decomposes image elements – graphics, text, icons, photographs, video, etc – and compresses them with the right codec for each and every pixel.''

= Management Channel (MGMT) =

The management channel establishes a PCoIP session between the client and server. The channel is created when the client connects to the server via SSL on port 4172. Existing PCoIP clients and servers perform validation of the SSL keys against the PCoIP Root Certificate Authority (see below). The channel is closed following negotiation of the data channel parameters. Parameters are listed below.

== Security ==
Existing server implementations check that the remote client's certificate is signed by the PCoIP Root Certificate Authority.

=== PCoIP Certificate Authority Key ===
//FIXME: Public key
//FIXME: Private key

=== PCoIP Client Key (libpcoip_client) ===
//FIXME: Public key
//FIXME: Private key

== Message Format ==
Each MGMT message consists of a 64-bit header followed by list of chunks. The first chunk is always the 'ssig' chunk, which identifies the purpose of the message.

=== Header Record ===
uint64_t size; Packet size excluding this header (bytes)
=== Chunk Record ===
uint32_t tag; FourCC code
uint32_t size; Chunk data size (bytes)
var data; Chunk data

== General FourCC Codes ==

{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| ssig || 4 || Purpose || 0=Invite, 1=Invite Ok, 2=Not Acceptable, 3=Ack, 4=Bye, 5=Bye Ok, 6=Ping, 7=Pong
|-
| byec || 4 || Bye reason ||
|-
| psec || 4 || security type || Integer; 0=NULL, 1=AES-128-GCM, 2=AES-256-GCM, 3=Salsa20-256-round12
|-
| 1key || 16 || AES 128 key ||
|-
| 1slt || 4 || AES 128 salt ||
|-
| 1spi || 4 || AES 128 spi ||
|-
| 2key || 32 || AES 256 key ||
|-
| 2slt || 4 || AES 256 salt ||
|-
| 2spi || 4 || AES 256 spi ||
|-
| s12k || 32 || Salsa20-256-round12 key ||
|-
| s12s || 4 || Salsa20-256-round12 salt ||
|-
| s12t || 4 || Salsa20-256-round12 spi ||
|-
| cipa || <= 255 || Connection IP address ||
|-
| cmac || 6 || Connection MAC address ||
|-
| cpri || 4 || Connection PRI address ||
|-
| ctag || <= 127 || Connection tag || VMware View Client 'token'
|-
| cprt || 2 || Connection port || Integer
|-
| pca1 || 32 || AES-128-GCM key ||
|-
| pca2 || 48 || AES-256-GCM key ||
|-
| pcs2 || 48 || Salsa20-256-round12 key ||
|-
| penc || <= 64 || PCoIP encoding || Array of bytes, where 0=pcoip_data_1, 1=pcoip_data_2
|-
| pcap || <= 64 || PCoIP encapsulation || Array of bytes, where 0=IP, 1=UDP, 2=TCP
|-
| pclr || 1 || Cleartext transport header supported || Boolean
|-
| psak || 1 || Selective ACK supported || Boolean
|-
| plnk || 4 || PCoIP link rate || Integer; BPS
|-
| pmtu || 4 || MTU size || Integer
|-
| pprf || <= 96 || PCoIP packet preference || Array of 3-byte records
|}

== Media FourCC Codes ==

When describing a specific media type (e.g. Audio), the following common chunks are present, followed by chunks specific to the media.

{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| mtyp || 4 || Media type || 0=USB, 1=Audio, 2=Video, 3=DDC, 4=KMP, 5=VChan
|-
| menc || var || Media encoding || (Media specific)
|-
| menb || 4 || Media || Boolean
|}

=== USB ===
Media Encoding:
{| border="1" cellpadding="2"
! Value !! Encoding
|-
| 0 || OHCI
|-
| 1 || EHCI
|-
| 3 || URB
|}

FourCCs:
{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| usbb || 4 || Client paramter version ||
|-
| usbp || 4 || Plugin version ||
|-
| usbc || 4 || Number of channels ||
|}

=== Audio ===
Media Encoding:
{| border="1" cellpadding="2"
! Value !! Codec || Frequency || Channels
|-
| 1 || ADPCM || 48000 || Stereo
|-
| 2 || ADPCM || 8000 || Mono
|-
| 3 || ADPCM || 16000 || Mono
|-
| 4 || PCM 16-bit LE || 48000 || Stereo
|-
| 5 || PCM 16-bit LE || 48000 || Mono
|}

FourCCs:
{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| audf || <= 100 || FEC mode || array of byes, where 0=FEC Mode 1
|-
| audi || 4 || Audio input || Boolean
|-
| audy || 4 || Standy mode ||
|}

=== Video ===
Media Encoding:
{| border="1" cellpadding="2"
! Value !! Encoding
|-
| 0 || Video 1
|-
| 1 || Video 2
|-
| 2 || Video 3
|}

FourCCs:
{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| vidt || 4 || Topology caching || Boolean
|-
| vidn || 4 || Max number of displays ||
|-
| vidv || 4 || Vertical extended motion || Boolean
|-
| vidh || 4 || Horizontal extended motion || Boolean
|-
| vidp || 4 || SACK || Boolean
|-
| vidl || || ? (deprecated) ||
|-
| vidb || 4 || Image cache size ||
|-
| vidu || 4 || User config || Boolean
|-
| vidy || 4 || Standby mode || Boolean
|-
| vidm || 4 || Monitor power saving || Boolean
|-
| vidC || 4 || ? ||
|-
| viCA || 4 || ? ||
|}

=== DDC ===
Always media encoding value 0

{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| ddce || 4 || Asynchronous EDID Update || Boolean
|}

=== KMP ===
Always media encoding value 0

{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| kmpa || <= 100 || Auto repeat modes || array of bytes, where 0=client auto repeat
|-
| kmpb || <= 100 || Pointer shape bitmap types || array of bytes, where 0=alpha, 1=color, 2=xor, 3=compressed alpha, 4=compressed color, 5=compressed xor
|-
| kmpc || 4 || Pointer shape caches ||
|-
| kmpx || 4 || Pointer shape cache size ||
|-
| kmps || 4 || Pointer shape max size || (x,y)
|-
| kmpf || 4 || Pointer fadeout || Boolean
|-
| kmpl || 4 || Multiple locale || Boolean
|-
| kmpu || 4 || Unicode key || Boolean
|-
| kmpk || 4 || Key block || Boolean
|}

=== VChan ===
Always media encoding value 0

{| border="1" cellpadding="2"
! FourCC !! Length (Bytes) !! Name || Data type
|-
| vchc || 4 || Number of channels ||
|-
| vchs || 4 || Buffer size ||
|}

== Key Exchange Obfuscation ==

PCoIP supports an optional method to obfuscate the the key exchange with the management channel. Obfuscation is achieved by xoring the client and server keys with values derived from a shared secret. The shared secret is communicated to the client and server via the [http://code.google.com/p/vmware-view-open-client/downloads/detail?name=VMware-view-client-protocol-spec-4.5.0-GA.pdf VMware View Client Protocol]. The shared secret is called the ''reservation_vcs_tag'' or ''PCoIP token''. The shared secret also contains a unique connection identifier called the ''reservation_ssig_tag'' or ''connection tag''. This is exchanged over the management channel via the 'ctag' chunk.

if (reservation_vcs_tag starts with "SCS1" && strlen(reservation_vcs_tag) >= 104) {
server key xor value = decodebase64(reservation_vcs_tag[4..47])
client key xor value = decodebase64(reservation_vcs_tag[48..91])
reservation_ssig_tag = reservation_vcs_tag[92..]
} else {
//server and client keys are exchanged verbatim
reservation_ssig_tag = reservation_vcs_tag
}

= Data Channel (DATA) =

The data channel exists to exchange encrypted data packets between the client and server. The channel is configured by the 'pcap' and 'penc' parameters negotiated in the MGMT channel. The following subsections relate to the (UDP, pcoip_data_2) configuration. In the default configuration, UDP messages are sent from client-to-server on UDP port 4172, and sent from the server-to-client on UDP port 50002. The network address and port are specified via the '''cipa''' and '''cprt''' chunks in the management channel.

== Encrypted Data Packet ==

PCoIP supports Salsa20/12, AES-128 and AES-256 encryption algorithms. Different algorithm and key may be used in each directions of the channel. Each encrypted data packet is sent as a UDP datagram and (when decrypted) contains a single data message. Note that each encryption algorithm uses a different packet format, but the resulting plaintext message is the same format.

=== Salsa20/12 Encapsulated Packet ===

A reference Salsa20/12 algorithm can be found here: http://cr.yp.to/snuffle/salsa20/ref/salsa20.c. Note the final four bytes of the plaintext data message is always equal to 0xDEADBEEF. These bytes are not considered part of the data message.

uint32_t spi; Unique indicator
uint32_t serial; Packet serial number, starts at zero
uint8_t iv[8]; Crypto IV
variable payload[] Ciphertext message

=== AES-128 Encapsulation Packet ===

//FIXME: Document

=== AES-256 Encapsulation Packet ===

//FIXME: Document

= Data Message Format =
Data message consist of a header followed by message-specific body.

== Header ==
uint8_t type; Message type (see table below)
uint8_t unknown[3];
uint16_t serial_no; Serial number; separate serial numbers maintained for each message type

== Packet Type ==
{| border="1" cellpadding="2"
! Packet Type !! Name || Description || Source(s)
|-
| 2 || IMG || Video Channel || Server
|-
| 6 || ? || Redirection services || Client,Server
|-
| 8 || DDC || Display Data Channel || Client,Server
|-
| 9 || ? || || Client,Server
|-
| 10 || ? || || Client,Server
|-
| 12 || HDA || Audio Channel || Client,Server
|}

[[Category:Container_Formats]]
[[Category:Undiscovered Codecs]]

UBB

2012-04-15T17:21:10Z

Summary:

* Extension: ubb , sometimes hnm
* Company: [[CRYO Interactive Entertainment]]
* Samples: '''ADD ME'''

UBB2 is another generation of FMV format developed by Cryo. It is an improved version of [[HNM4]] format. While general container format remains the same, video encoding is slightly improved.

== File Format ==
Remains the same as was used in [[HNM4]]. Following parts has changed:
* File signature is "UBB2" now.
* Video chunk has "IV" TwoCC.

== Video Compression ==
The video frame is encoded in a series of horizontal colums of variable width and height. However, columns height (and some of their coding properties) may only change at the beginning of new line.

<br>Encoded stream consist of column copy mode bytes:
bits:
7 - swap rows/columns
6 - vertical reverse
5 - horizontal reverse
4..0 - column width index (table lookup, see below)

Each of such code generally means ''"copy a width-by-height column from some other place to the current, using flags provided."''
<br>Flags behavior can be illustrated as follows (assuming 2x3 block and the source points to 'M' pixel):
Source Normal Horizontal Vertical Swap

ABCDE MN ML MN MR
FGHIJ RS RQ HI NS
KLMNO -> WX WV CD OT
PQRST
UVWXY

Multiple flags can be used together.

Encoded column width depends on column's height:
2: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,18,20,22,24,26,28,30,32,34,36,38,40,44,48,52,56
3: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,22,24,26,28,30,32,34,36,38,40,42,44
4: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,28,30,32,34,36,38

Source column offset may be an "absolute" offset - byte distance from the current column, or relative X:Y offset.
* For absolute offset read next 2-byte value (unsigned, little-endian) and add base oofset.
* For relative offset read next byte, extract X and Y deltas from it and add base deltas.

Certain combination used for special purposes. Those are:
* 0x20 - skip: read next byte (width) and copy width+1 column from previous frame
* 0x60 - solo: draw 1-pixel wide column using next bytes of input
* 0xA0 - draw: read next byte (width) and draw width+2 column using next bytes of input (top-down, left-right)
* 0xE0 - mode: examine next byte.
** if byte is zero, then read two bytes (width, color) and fill width+1 column with color.
** if byte is non-zero, change the decoding properties as follows:
+------+--------+--------+--------+----------+---------+--------+
| byte | height | source | offset | Y.X bits | X-base | Y-base |
+------+--------+--------+--------+----------+---------+--------+
| 1 | end of decoding |
+------+--------+--------+--------+----------+---------+--------+
| 2 | 2 | prev | -32768 | N/A | N/A | N/A |
| 3 | 3 | prev | -32768 | N/A | N/A | N/A |
| 4 | 4 | prev | -32768 | N/A | N/A | N/A |
| 5 | 2 | prev | N/A | 4.4 | -8 | -8 |
| 6 | 3 | prev | N/A | 4.4 | -8 | -8 |
| 7 | 4 | prev | N/A | 4.4 | -8 | -8 |
+------+--------+--------+--------+----------+---------+--------+
| 8 | 2 | prev | N/A | 4.4 | -2 | -8 |
| 9 | 2 | prev | N/A | 4.4 | -14 | -8 |
| 10 | 2 | prev | N/A | 4.4 | -8 | -2 |
| 11 | 2 | prev | N/A | 4.4 | -8 | -14 |
| 12 | 2 | prev | N/A | 4.4 | -2 | -2 |
| 13 | 2 | prev | N/A | 4.4 | -14 | -2 |
| 14 | 2 | prev | N/A | 4.4 | -2 | -14 |
| 15 | 2 | prev | N/A | 4.4 | -14 | -14 |
+------+--------+--------+--------+----------+---------+--------+
| 16 | 3 | prev | N/A | 4.4 | -2 | -8 |
| 17 | 3 | prev | N/A | 4.4 | -14 | -8 |
| 18 | 3 | prev | N/A | 4.4 | -8 | -2 |
| 19 | 3 | prev | N/A | 4.4 | -8 | -14 |
| 20 | 3 | prev | N/A | 4.4 | -2 | -2 |
| 21 | 3 | prev | N/A | 4.4 | -14 | -2 |
| 22 | 3 | prev | N/A | 4.4 | -2 | -14 |
| 23 | 3 | prev | N/A | 4.4 | -14 | -14 |
+------+--------+--------+--------+----------+---------+--------+
| 24 | 4 | prev | N/A | 4.4 | -2 | -8 |
| 25 | 4 | prev | N/A | 4.4 | -14 | -8 |
| 26 | 4 | prev | N/A | 4.4 | -8 | -2 |
| 27 | 4 | prev | N/A | 4.4 | -8 | -14 |
| 28 | 4 | prev | N/A | 4.4 | -2 | -2 |
| 29 | 4 | prev | N/A | 4.4 | -14 | -2 |
| 30 | 4 | prev | N/A | 4.4 | -2 | -14 |
| 31 | 4 | prev | N/A | 4.4 | -14 | -14 |
+------+--------+--------+--------+----------+---------+--------+
| 32 | 2 | curr | -65536 | N/A | N/A | N/A |
| 33 | 2 | curr | N/A | 3.5 | -16 | -8 |
| 34 | 2 | curr | N/A | 4.4 | -8 | -16 |
| 35 | 2 | curr | N/A | 4.4 | -24 | -16 |
| 36 | 2 | curr | N/A | 4.4 | +8 | -16 |
| 37 | 2 | curr | N/A | 5.3 | -4 | -32 |
| 38 | 2 | curr | N/A | 5.3 | -12 | -32 |
| 39 | 2 | curr | N/A | 5.3 | +4 | -32 |
+------+--------+--------+--------+----------+---------+--------+
| 40 | 3 | curr | -65536 | N/A | N/A | N/A |
| 41 | 3 | curr | N/A | 3.5 | -16 | -8 |
| 42 | 3 | curr | N/A | 4.4 | -8 | -16 |
| 43 | 3 | curr | N/A | 4.4 | -24 | -16 |
| 44 | 3 | curr | N/A | 4.4 | +8 | -16 |
| 45 | 3 | curr | N/A | 5.3 | -4 | -32 |
| 46 | 3 | curr | N/A | 5.3 | -12 | -32 |
| 47 | 3 | curr | N/A | 5.3 | +4 | -32 |
+------+--------+--------+--------+----------+---------+--------+
* prev means source is the previous frame, curr - current
* Y.X means "Y" value in upper Y bits of byte, "X" in lower X bits.

== Trivia ==
Apparently this format has another less-used name - HNM5.

[[Category:Game Formats]]
== Games Using UBB ==

* [http://www.mobygames.com/game/megarace-2 Megarace II]
* lots of others

[[Category:Game Formats]]
[[Category:Video Codecs]]
[[Category:Formats missing in FFmpeg]]

JangGu Bayer

2012-04-15T12:07:36Z

Summary:

*Source: http://os1a.cs.columbia.edu/lxr/source/drivers/media/video/se401.c
*FourCC: JBYR
*Sample:

Found in some kensington webcams.

Psygnosis SMV

2012-03-31T09:20:52Z

Summary:

* Extension: smv
* Company: [[Psygnosis]]
* Samples: http://cd.textfiles.com/cdaction/cdaction04/FILM/

SMV is a short-lived FMV format used in the PC game [http://www.mobygames.com/game/dos/wipeout Wipeout] by Psygnosis. It employs LZ-like compression to encode 8-bit palettized video and audio.

== File Format ==
An SMV file consist of a number of chunks with unique [[TwoCC]]. All numbers are little-endian.

u16 chunkid
u16 chunklen
u8 payload[chunklen]

=== ST chunk ===
File header. Encodes basic movie properties.

u16 width -- movie width
u16 height -- and height
u16 mbwidth -- macroblock width
u16 mbheight -- and height
u16 frames -- number of video frames
u16 colors -- number of used colors
u16 nibbles -- offset of frame's nibble data
-- if chunk size is >= 15
u8 codec -- video codec format

* Codec field may be absent, in such case assume codec 0.
* mbwidth is always 16, mbheight may be either 16 or 8.
* Movie playback rate is 12 FPS.

=== GP chunk ===
New palette. Contains a number of 6-bit [[RGB]] entries, as specified in ST chunk.

=== MU chunk ===
Sound chunk. Contains audio fragment in 15862Hz, 8-bit, unsigned, mono format. If the payload size is equal to the amount of audio data for a frame (i.e. the size is freq / fps, which is equal to 1321) then the audio data is stored in raw [[PCM]] format. Otherwise, it's additionally LZ-word-packed.

=== FR chunk ===
Frame chunk. Contains encoded video frame.

=== FE chunk ===
End of the movie. No payload.

== Video Compression ==

=== Video Codec 0 ===

LZUnpack(payload, temp, words)
DrawBlocks(temp)

=== Video Codec 1 ===

LZUnpack(payload, temp, bytes)
DrawBlocks(temp)

=== Video Codec 2 ===

LZUnpack(payload, temp, bytes)
UnpackPartitions(temp, temp2)
DrawBlocks(temp2)

UnpackPartitions decodes Macroblock's nibbles.

Source frame is broken on to several equal partitions, 2 across by 4 down. Encoded partitions format:

u8 pixels[16 * num_mblocks] -- passed as is to DrawBlocks()
for each partition
u8 codebook[256][2] -- nibbles codebook for each partition
for each partition
for each macroblock
u8 index[mb_width*mb_height/4] -- index of codebook entry

Each entry of codebook encodes 4 nibbles. First byte encodes two nibbles of upper macroblock's line, second - next line, and so on.

=== Video Codec 3 ===
This codec is never used, but the code is still in place.

if GetByte()
LZUnpack(payload, temp, bytes)
UnpackSpecial(temp, temp2)
DrawBlocks(temp2)
else
same as codec 1

UnpackSpecial performs advanced decompression. '''TODO''' ''this may be some common algorithm''

=== Macroblocks drawing ===
Macroblocks are stored in the following format:

u8 pixels[16 * num_mblocks]
u4 nibbles[num_mblocks]

Size of pixels[] is equal to ST chunk's ''nibbles''.
For each macroblock draw it by indexing its pixels by nibbles. First nibble stored in top 4 bits of byte.

== Common Algorithms ==
=== LZ decompression ===
This is a more-or-less common LZ77-like algorithm. It can operate in two modes: byte mode and words mode. Mode affects backchain offset encoding.
* "next bit" means next bit from the internal 8-bit bitreader queue. Queue refilled with the input stream bytes, bits are consumed starting from the highest.
* "next byte" means raw octet from the input stream

repeat

if next bit == 0
output next byte
else
if next bit == 0
offset = next byte
else
if byte mode
offset = next 3 bits || next byte + 1
if words mode
offset = next 2 bits || next byte + 1 *2

if next bit == 0 length = 2
else if next bit == 0 length = 3 + next bit
else if next bit == 0 length = 5 + next bit
else if next bit == 0 length = 7
else if next byte != 0 length = 7 + this byte
else finish unpacking

copy length bytes from output - offset to output
note that, bytes may overlap to repeat itself

[[Category:Game Formats]]
[[Category:Video Codecs]]
[[Category:Formats missing in FFmpeg]]

CSF

2012-03-14T16:44:36Z

Summary:

Currently discussed and sample here: https://ffmpeg.org/trac/ffmpeg/ticket/1060
Copy-and-pasted description:
It is a format based on PNG and partial re-encoding, as you can see when you look at it with e.g. http://sourceforge.net/projects/extractor-gtk/
Possibly contains some other encoding types, too.
All container structures seem to be little-endian.
Each "frame" contains a 4 byte size at offset 8.
For the PNG encoded ones, offsets 16 - 24 seem to contain 4 16 bit values that indicate the offset and size of the image the PNG updates. After that seems to follow yet another size field containing the remaining bytes.
Start of the following packet is always aligned to 16 bytes.
There seems to be some kind of index at 0x190.
This and the data start seems to be referenced in the header, at bytes 24 and 28.

CinemaDNG

2012-03-14T10:35:14Z

Summary: initial version

=Cinema Digital Negative=

* Company: [[Adobe]]

==Links==
* overview: http://labs.adobe.com/downloads/cinemadng.html#docs
* spezification: http://download.macromedia.com/pub/labs/cinemadng/cinemadng_spec_090211.pdf
* file format: http://download.macromedia.com/pub/labs/cinemadng/cinemadng_summary.pdf

[[Category:Video Codecs]]
[[Category:Container Formats]]

FFmpeg Summer of Code 2012

2012-02-18T20:52:40Z

Summary: remove another completed task

FFmpeg Summer of Code Proposals and Qualification Tasks.

==Timeline==
March 12th-16th: Project application evaluation.

NOTE: FFmpeg has not been accepted this year, so unless you want to do any of these, the small tasks or qualification tasks just for fun...

==Contacting Developers==
Find us on irc, server: irc.freenode.net channel: #ffmpeg-devel
or contact us by subscribing to the mailing list https://lists.ffmpeg.org/mailman/listinfo/ffmpeg-devel/

== Qualification Tasks ==
To be eligible for a Summer of Code project, we ask you to do a small programming task to prove you know the basics. FFmpeg is a large, complicated collection of code and it's not easy for beginners. There are some ideas for tasks on the [[Small FFmpeg Tasks]] page.

== 1st Tier Project Proposals ==
These are proposals with a mentor attached.

''Baptiste has also offered to mentor.''

=== hwaccel: global architecture ===

Revisit my older v2 proposal to completely get rid of HW pixel formats. The advantages were: this simplifies things, clean ups code, allows to move more code to the codec library (FFmpeg), allow for fallbacks to SW decoding (at the stream level if HW cannot meet the requirements, not while decoding a particular stream). The key point was to separate identification of the pixel format from the underlying HW accelerator. This will also allow for reading HW surfaces back, when requested if the selected pixel format.

Add encoding and post processing infrastructure.

* VA-API (hwaccel)

:Codecs related:
+ Add support {,M}JPEG decoding
+ Add support for VC-1 interlaced acceleration
+ Add support for H.264 interlaced

:Tools related:
:Add VA support to ffplay (see my older patches)
+ requires some VO infrastructure work
+ requires hwaccel hooks
+ requires enabling vaapi, dxva, bcm, tidsp, whatever

* Add VA support to ffmpeg
+ requires VA/DRI API, for no X dependency
+ dependencies: hwaccel v2 to construct pipelines (HW->HW, HW->SW, SW->HW, etc.), or at least allow for VA surface readback wherever necessary.

* VDPAU (hwaccel)

Update and push my older hwaccel-based VDPAU code. It should still work. :)
:Priority: hwaccel v2 infrastructure work first, ffplay enabling second.

''Mentor: Gwenole Beauchesne''

=== glplay ===
Add OpenGL output to ffplay, this should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which you may relicense under the LGPL.

''Mentor: Reimar Döffinger''

=== Improve the audio resampling/rematrixing/converting code ===

* Right now, we're using libswresample to resample/rematrix audio (samplerate / channels) and to resample the audio format (int, float, 16-bit, 32-bit).
* Both interleaved and planar audio sample formats are already supported
* We need SIMD optimization of popular conversions (float-int16, int16-float), (stereo-mono-5.1) and anything else that's frequently used.
* We need support for alternate conversion functions (e.g. sample format conversion with or without dithering)
* Fix bugs in current design (none known but there sure are some)

''Mentor: [[User:Michael|Michael Niedermayer]]''

=== Implement a H.265 / High efficiency video coding (HEVC) decoder ===

* Write a basic decoder supporting I, P, and, if time permits, B slices.
* It does not need to be ASM/SIMD optimized but its high level structure must permit such optimizations to be easily added later.
* As a qualification task you need to implement parsing headers and maybe a bit beyond that to demonstrate that you are qualified and understand the HEVC specification. This project requires a solid understanding of video coding and C, it's not something for the average SOC student.
* A draft spec is available at: http://phenix.it-sudparis.eu/jct/doc_end_user/documents/8_San%20Jose/wg11/JCTVC-H1003-v21.zip

''Mentor: [[User:Michael|Michael Niedermayer]]''

=== H.264 MVC ===

* Add MVC support to our H.264 decoder. MVC is used in 3D Blu-Rays.
* As qualification you have to do some work that demonstrates your understanding of MVC and that is a subpart of the whole MVC implementation.

''Mentor: [[User:Michael|Michael Niedermayer]]''

=== Libavfilter extension ===

Libavfilter is the FFmpeg filtering library that started as a 2007 SoC [[FFmpeg Summer Of Code#Video Filter API (AKA libavfilter)|project]]. It currently supports audio and video filtering and generation support.

The task would consist of writing or porting audio and video filters and eventually fix/extend libavfilter API and design.

In particular the work may focus on porting MPlayer filters which are currently integrated through the mp wrapper.
For each port the student should verify that the new filter produces the same output (by comparing the output generated by -vf mp=FILTER and -vf FILTER) and checking that the new integrated filter is not slower.

Prerequisites: good C coding skills, familiarity with git/source code control systems, having some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

For getting more ideas read also the [[FFmpeg_/_Libav_Summer_Of_Code_2011#Libavfilter_video_work|GSoC 2011 libavfilter video proposal]] and [https://ffmpeg.org/trac/ffmpeg/query?status=new&status=open&status=reopened&component=avfilter&col=id&col=summary&col=status&col=type&col=priority&col=component&col=version&order=priority trac libavfilter tickets].

Qualification task: a port or a new implementation of one or more filters.

''Mentor: Stefano Sabatini - saste on IRC (possibly with a backup mentor)''.

=== Bayer colorspace formats ===

Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmepg's applicability to RAW still and video photography processing. Tasks:
* Implement bayer transformations in libswscale (plain C)
* Add bayer formats to the libavutil pixfmt enumeration routines
* Extend TIFF decoder to support DNG-Bayer format
* Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)
* SIMD optimizations of the libswscale transformations
* Decoders/specs may be available in the [[Dcraw]] project

Qualification task: TBD

''Mentor: [[User:Suxen_drol|Peter Ross]]''

=== Extend image formats support ===

Improve FFmpeg support for image formats, adding missing formats (e.g. XPM) and extending support for the current ones (e.g. animated GIF, GIF compression, fix PNG todos, add support to animated PNG) etc.

Qualification task: TBD (possibly finally fixing and integrating Måns' zlib decoder that has been unmerged since ages? Or just starting with some small part of the task itself, or implementing format autodetection for imagepipe demuxer)

''Mentor: [[User:reimar|Reimar Döffinger]]''

=== DTS-HD decoder ===

* ETSI released specifications (http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf). Your job is to complete the existing decoder with the following features.

(1) Add support for mixed Core + DTS-HD stream structure
(DtsCoreFrame+DtsHdFrame+DtsCoreFrame+DtsHdFrame+...), used by Blu-Ray main
and commentary tracks.
(2) Add support for XXCh extension (6.1 and 7.1 channels).
(3) Add support for X96 extension (96khz).
(4) Add support for XLL extension (lossless).
(5) Add support for a pure DTS-HD stream structure
(DtsHdFrame+DtsHdFrame+DtsHdFrame+...), used by Blu-Ray PiP tracks.
(6) Add support for XBR extension (extra bitrate).

''Mentor: Benjamin Larsson''

=== Libavdevice API ===
Currently libavdevice use libavformat API. This is far from being useful for real-time video rendering. Your task is to write new API which resolves this and similar issues and port all already present input and output devices to the new API.

New API should have:
* Support for output video devices which do not provide own scaler.
* Minimal latency.

Qualification task: write one new video outdev, for example any of, but not limited to: aalib, caca, x11, xv, xover, vesa or gl2 using current libavdevice API.

''Mentor: [[User:Pbm|Paul B Mahol]]''

=== Redesigning the protocol reads ===
Eliminate active polling while allowing interrupts, and allowing selection on multiple handles.

''Mentor: Nicolas George''

== 2nd Tier (need mentor) Project Proposals ==
Some of the following proposals are also proposed by other organizations, we will try to coordinate this with them so as to avoid duplicate work.
We are also happy to hear your personal project ideas ...

=== AAC decoder improvements ===
Our AAC decoder does not support low-delay. Part of this task will be to also finish last year's BSAC task. A possible qualification task is to fix a crash in the current BSAC code with one of the samples from the BSAC testing suite.
:Sample: https://ffmpeg.org/trac/ffmpeg/ticket/113

=== AAC encoder improvements ===
Our AAC encoder does not produce competitive quality per bitrate.
Improve the encoder to be better than some other commonly used encoder like libfaac.
This requires solid understanding of things like psychoacoustics and rate distortion.
A qualification task for this could be to improve the encoder by at least 5% bitrate at the same quality
measured by some objective measure.

=== FF Fuzzer ===
Write a system like FATE that fuzzes and tests these fuzzed multimedia files under address sanitizer and valgrind with ffmpeg and ffplay.
When a crash, or other anomaly is found, it would use git bisect to identify which exact commit introduced the bug.
And either display this via some web frontend (similar to fate.ffmpeg.org) or just automatically send an email to some
dedicated mailing list. The system has to be robust (there will be infinite loops, OOM conditions and randomly occurring
crashes). Its also important that the system is easy to maintain and can filter out duplicates of the same issue.

=== VC1 interlaced ===
FFmpeg has code for interlaced VC1, but nearly all samples still do not decode correctly. The task is to finish last year's project. You should be able to find a possible qualification task by testing interlaced samples.

=== MPEG-4 ALS Roundup ===

This task is to update and enhance the existing ALS decoder as well as integrate
and enhance the rudimentary encoder found at:
https://github.com/justinruggles/FFmpeg-alsenc

Possible features are:

* implement rls-lms in the decoder
* do correct channel layout/sort handling in the decoder
* update to current master
* use codec private options
* implement encode2(), setting pts and duration
* document options and examples in encoders.texi
* come up with a good set of encoding tests for FATE
* implement mcc/channel sort in the encoder
* implement rls-lms in the encoder
* implement float support

=== Fix and improve FFserver ===
FFserver has been part of FFmpeg since a long time but due to lack of a motivated maintainer its a bit buggy.
For this project you would have to debug and fix many bugs. It requires good skills at reading and understanding other peoples code.
As a qualification task you will have to write functioning regression tests for FFserver which implicates some bugfixing to make ffserver produce the same output on all supported platforms.

=== Support for more subtitle formats ===
We have libass support now, either a parser (from mplayer) to convert subs into ass or something else.
* http://mailman.videolan.org/pipermail/vlc-devel/2011-September/081803.html
* http://ale5000.altervista.org/subtitles.htm

=== MKV ordered chapters / playlist support ===
Get playlist stuff into ffmpeg. Playlist is blocking a few things like quicktime edit list and .asx / .pls files.

=== Adobe fragmented http in/out ===
Adobe has a new streaming format.
* info/spec: http://www.adobe.com/products/httpdynamicstreaming/

=== libavfilter 9/10bit support ===
Make filters work with higher bitrate codecs/colorspaces

=== Extend paletted format support ===

Cleanup framework for handling better with paletted format, write a posterize filter, add support to libswscale palette output (possibly making use of libavcodec/elbg), add support for reading and saving a palette to a file and apply them to the input video (e.g. by creating ad-hoc filters).

=== Port formats/colorspace support from dcraw or make dcraw wrapper ===
[[Dcraw]] supports many raw camera formats that ffmpeg may not. Port or make a wrapper for this project.

[[Category:FFmpeg]]

Origin Flic Codec

2012-02-17T00:42:50Z

Summary: categorize

* FourCC: JYV1, JYA1, RRV1, RRV2
* Company: Origin Systems
* Samples: ask Mike

''Please rename this codec name to something more original''

Origin Flic Codec is a relatively simple palettized video codec used in PC version of Crusader: No Remorse game. Data is packaged inside standard [[AVI]] container and employs relatively simple [[RLE]] compression. Audio stored in uncompressed [[PCM]] form. It is rumored that codec's FourCCs are actually based on game developer's names (not confirmed.)

== Video Codec ==
A video movie uses fixed palette, stored in corresponding AVI Stream Format chunk.

Video frame is encoded in a series of slices, where each slice encodes a number of horizontal lines of picture. Number of slices depends on FourCC and is:
* JYV1, RRV1 - 5 slices
* RRV2 - 15 slices
Therefore, each slice contains VideoHeight/slices raster lines.

Frame chunk consist of the following parts:

u32 slicetable[num_slices]
-- for each slice --
u32 codes_size;
u8 codes[codes_size]
u8 raster[]
--------------------
where each entry of slicetable points to corresponding codes/raster block (offsets are relative to beginning of the chunk.)

Video decompression is performed as follows (for each slice):

flipflop = flip
while slice is not complete
idx = GetBits(4);
blocksize = BaseLength[idx];
if idx != 0 and idx != 8
blocksize += GetBits(FineLengthBits[idx]);
if flipflop == flip
leave blocksize pixels unchanged
else
draw blocksize pixels from raster[]
flip flop

BaseLength[] = {0, 1<<7, 1<<3, 0, 1<<1, 0, 1<<5, 0,
1, 1<<8, 1<<4, 0, 1<<2, 0, 1<<6, 0};
FineLengthBits[] = {0, 7, 3, 0, 1, 16, 5, 0,
1, 8, 4, 0, 2, 24, 6, 0};

GetBits() reads variable number of bits from codes[], highest bits of lowest byte first.
For RRV* FourCCs 'leave' and 'draw' command skips/doubles each pixel when decoding a keyframe (i.e. a keyframe should be upscaled horizontally.)

[[Category:Game Formats]]
[[Category:Video Codecs]]
[[Category:Formats missing in FFmpeg]]

HNM6

2012-02-05T17:30:53Z

Summary: /* Normal decoding */

* Extension: hnm
* Company: CRYO Interactive Entertainment
* Samples:

HNM6 is the latest variant of [[HNM]] video format by Cryo. Unlike its previous versions, it has Hi-color video support.

== File Format ==
File consist of a main header, followed by frame chunks. Each frame chunk consist of individual audio and video chunks. All numbers are little-endian.

u8 sig[4] -- file signature, "HNM6"
u8 reserved[2] -- usually 0
u8 audioflags -- nonzero value indicates file has APC sound (not authoritative), bits meaning:
7 : stereo
6..5 : frequency in units of 11025Hz (i.e. frequency = value * 11025)
u8 bpp
u16 width
u16 height
u32 filesize
u16 frames -- number of frames
u16 frames2 -- upper part of frame counter, ignored by some implementations
u32 reserved3 -- usually 0, but sometimes "V107", "V108" - version?
u16 speed -- playback speed in fps, may be zero (?assume 15 fps then?)
u16 maxbuffer -- number of frame buffers used
u32 maxchunk -- max frame chunk size
u8 note[16]
u8 copyright[16] -- "-Copyright CRYO-"

Each frame chunk begins with u32 chunk size (including this size field), then followed by frame's individual chunks:

u32 chunksize -- chunk size including this field, excluding padding
u16 chunkid -- TWOCC chunk id
u16 reserved
u8 data[]
u8 padding[] -- pads chunk to 4-byte boundary

=== AA Chunk ===
APC Audio. See [[CRYO APC]]

=== BB Chunk ===
Audio continuation.

=== IW Chunk ===
Video frame, WARP format.

=== IX Chunk ===
Video frame, normal format.

== Format modifications ==

At least one known game (Riverworld) uses slightly different HNM6 container format, most likely due to different sound encoding. This format is just a small enhancement of the original, but it's not backward compatible.

Standard HNM6 header continues with additional audio description header:

u32 freq
u32 bits
u32 channels
u32 ?
u32 ?
u8 copyright[28] -- "HNMS 1.1 by SARRET Hubert\0\0\0"
u32 ? -- some initial audio samples?
u32 ?
u32 ?
u32 ?

Each frame chunk followed by an audio chunk, however, this audio chunk size is not counted by frame's chunk size field.
Audio chunk format:

u32 sig[4] -- "SOUN"
u32 chunksize -- including this preamble
u8 data[]

== Video Format ==
The video compression relies on a concept of Key-blocks, encoded with [[JPEG]]-like algorithm and motion blocks, derived from previously drawn blocks. The frame is encoded in 8x8 or smaller blocks. The Key-blocks are always 8x8 and directly encoded, while motion blocks heavily rely on [[D&C]] approach and various block transformation techniques. Generally, this codec is quite similar to [[4XM]] codec and [[Mobiclip]] codecs, because it's written by the same people.

There are two variants of HNM6 video codec.
* WARP codec. Prototype codec, simplified version. Used in a very little number of games circa 1997.
* Normal (yes, that's how it's called internally) codec. Fully-featured version, widely used.

Both version has identical bitstream layout. Frame data begins this the following header:

s32 quality -- JPEG quality index, negative value indicates that the frame is a keyframe
u32 bitbuffer -- offset of bitbuffer
u32 motionbuffer -- offset of motion vector buffer
u32 shortmotionbuffer -- offset of short motion vector buffer
u32 jpegbuffer -- offset of jpeg data buffer
u32 jpegend -- offset of jpeg data buffer end

All offsets are little-endian and relative to this header.

* quality is a standard [[JPEG]] quality value (0..100%). Negative value is used to specify a keyframe (for Normal codec only.)
* bitbuffer points to frame's VLC-encoded macroblocks. This kind of data is accessed by a bit-reader with 32-bit internal queue, MSB bit comes out first.
* motion buffer points to array of u16 motion vectors, used to copy blocks from various areas of the frame
* short motion buffer is used for accessing blocks near the current macroblock. Each entry of buffer is 12 bits long, accessed in LSB order.
* jpeg buffer points to array of encoded [[JPEG]] macroblocks

=== Key-blocks decoding ===
Key-blocks are JPEG-encoded blocks in YUV 4:4:4 format. The requantization, [[IDCT]] and colorspace conversion steps are identical to standard [[JPEG]]. Standard zigzag (0,1,8,16,9,...) and quantization tables (16,11,10,16, 24,...; 17,18,24,47,99,...) used as well.

The only custom feature used is the coefficients encoding. Instead of Huffman coding they are stored using RLE scheme.

Macroblock's jpeg data is accessed by 4-bit nibbles, lower nibble of each byte first. Also, the following primitives are used:
* half - half of a nibble, upper part first, then lower (i.e. read a nibble, process upper part of it upon first use, lower part upon second)
* s2 - signed two-bit 2's complement value of a half, no zero point (i.e. 0b00 -> 1, 0b01 -> 2, 0b10 -> -2, 0b11 -> -1)
* s4 - signed four-bit 2's complement value of a nibble, no zero point
* s44 - signed eight-bit value of two nibbles, with zero point (this is basically (signextend(nibble1) << 4) | nibble2)

For each plane of 8x8 coefficients, the first coefficient is s44, then read nibbles and fill the rest of coefficients as follows until the whole plane is complete. Repeat for each Y, U and V plane.

0 - fill remainder of a plane with zeros
1 - 0, 0, 0, 0
2 - 0, 1
3 - 0, -1
4 - 0, 0, 1
5 - 0, 0, -1
6 - 0, 0, 0, 1
7 - 0, 0, 0, -1
8 - zeros = half, tail = half
fill (zeros+1) coefficients with 0
fill (tail+2) coefficients with s2
9 - zeros = half, tail = half
fill (zeros+1) coefficients with 0
fill (tail+1) coefficients with s4
10 - s2, s2
11 - s4
12 - s4, s4
13 - s4, s4, s4
14 - 0
15 - s44

=== WARP decoding ===
Macroblocks are encoded using the following VLC codes:

{| border="1" cellpadding="5" style="border-collapse: collapse; border-style: dashed; border-color: #2f6fab;"
|- bgcolor="#f0f0f0" |
! Block type !! 8x8 !! 4x4
|-
| Keyblock || 11 || n/a
|-
| Motion || 10 || 0
|-
| CrossCut || 0 || 1
|}

* Keyblock is a JPEG-encoded macroblock.
* CrossCut means that the current block should be cut on to 4 smaller subblocks and each one processed independently, using the same scheme recursively.
* Motion means that the block is motion-coded and should be copied from some other part of the frame. To do so, read the next entry of the motion buffer and process it as follows:
** transformation mode = next 2 bits from bitbuffer (upper part) combined with bit 15 of motion (lower bit)
** xmotion = 128 - (bits 7..14 of motion)
** ymotion = (if block is not 8x8, then 4) - (bits 0..6 of motion)
* 2x2 blocks are always short motion coded, using no extra VLC:
** transformation mode = bits 1..3 of short motion
** motion index = bit 0 || bits 4..7 || bits 8..11 (total 9 bits)
** decode the index as follows:
if index < 12 * 8
xmotion = -2 - index % 12
ymotion = 6 - index / 12
else
index -= 12 * 8
xmotion = 19 - index % 32
ymotion = -2 - index / 32
if ymotion <= -8
ymotion = ymotion - 1

Block's source coordinates are the sum of current 8x8 macroblock's coordinates (not its subblock's!) and the xmotion and ymotion respectively. If x-coordinate happens to go out of frame, it should be wrapped around the line. Copy block at the following coordinates to the current, applying the transformation as specified.

=== Normal decoding ===
Normal compression introduces interframes, more split modes and different short motion encoding.

Macroblocks are encoded using the following VLC codes:

{| border="1" cellpadding="5" style="border-collapse: collapse; border-style: dashed; border-color: #2f6fab;"
|- bgcolor="#f0f0f0" |
! Block type !! 8x8 !! 4x8 !! 8x4 !! 4x4 !! 2x4 !! 4x2 !! 2x2
|-
| colspan="8" style="text-align: center;" bgcolor="#f0f0f0" | Keyframe
|-
| Keyblock || 110 || n/a || n/a || n/a || n/a || n/a || n/a
|-
| HorizontalCut || 00 || 0 || n/a || 01 || 1 || n/a || n/a
|-
| VerticalCut || 01 || n/a || 0 || 10 || n/a || 1 || n/a
|-
| CrossCut || 10 || n/a || n/a || 11 || n/a || n/a || n/a
|-
| Motion || 111 || 1 || 1 || 00 || 0 || 0 || n/a
|-
| colspan="8" style="text-align: center;" bgcolor="#f0f0f0" | Interframe
|-
| Keyblock || 011 || n/a || n/a || n/a || n/a || n/a || n/a
|-
| HorizontalCut || 100 || 10 || n/a || 101 || 111 || n/a || n/a
|-
| VerticalCut || 101 || n/a || 10 || 110 || n/a || 111 || n/a
|-
| CrossCut || 1110 || n/a || n/a || 111 || n/a || n/a || n/a
|-
| Skip || 110 || 11 || 11 || 100 || 110 || 110 || 11
|-
| ShortMotion || 00 || 00 || 00 || 00 || 0 || 0 || 0
|-
| Motion || 010 || 01 || 01 || 01 || 10 || 10 || 10
|}

* HorizontalCut splits the block horizontally, producing two subblocks of smaller height. Each one then processed recursively.
* VerticalCut is identical to HorizontalCut, but splits the block vertically.
* Skip simply copies the block from the previous frame.
* ShortMotion copies block from previous frame, using the spiral-encoded motion. Unlike regular motion, short motion is relative to subblock's coordinates, not the whole macroblock. Read ShortMotion index, then obtain block's relative coordinates as per illustration:
[[Image:HNM6Spiral.png]]
* Motion blocks are similar to WARP motion block, with minor enhancements:
** For keyframe motion is the same as for WARP, with the exception for blocks smaller than 4x4. For such blocks:
*** transformation mode = bits 12.14 of motion
*** xmotion = 63 - (bits 0..6 of motion)
*** ymotion = 6 - (bits 7..11 of motion)
** Keyframe's 2x2 blocks are always motion-coded as above.

** For interframe motion is different. For blocks 4x4 and larger:
*** transformation mode = next 3 bits from bitbuffer
*** if bit 15 of motion is zero then source is the previous frame, current frame otherwise
*** xmotion = 128 - (bits 7..14 of motion)
*** ymotion = (if from previous frame then 64, otherwise if block is not 8x8, then 4) - (bits 0..6 of motion)
** For blocks smaller than 4x4:
*** if bit 15 of motion is zero then source is the previous frame, current frame otherwise
*** if from current frame - same as for keyframe, see above. for previous frame:
*** transformation mode = next 3 bits from bitbuffer
*** xmotion = 31 - (bits 0..5 of motion)
*** ymotion = 31 - (bits 6..11 of motion)
** Don't forget to wrap x motion around the line if it goes out of frame.

=== Block transformation ===
During block copying, it can be additionally transformed. Transformation modes:
==== 0 - Normal copy ====
Copy block as is.
==== 1 - Horizontal flip ====
Swap left and right:
abc -> cba
==== 2 - Vertical flip ====
Swap up and down:
a c
b -> b
c a
==== 3 - Cross flip ====
Swap both up-down and left-right:
ab fe
cd -> dc
ef ba
==== 4 - Forward flip ====
Swap block around / :
ab -> db
cd ca
==== 5 - Forward rotate ====
Rotate block 90 degrees clockwise:
ab eca fe
cd -> fdb -> dc
ef ba
==== 6 - Backward rotate ====
Rotate block 90 degrees counterclockwise:
ab bdf fe
cd -> ace -> dc
ef ba
==== 7 - Backward flip ====
Swap block around \ :
ab -> ac
cd bd

=== Bitexact decoding ===
While you can use stock [[JPEG]] routines to decode Key-blocks, if you want to produce bitexact output you need to use very specific code to do so.

[[Category:Video Codecs]]
[[Category:Game Formats]]
[[Category:Formats missing in FFmpeg]]

FFmpeg / Libav Summer Of Code 2012

2012-01-27T09:52:16Z

Summary: fix category

== How it works ==

Google's Summer of Code program is simple: you (the student) work on a project, full-time, during the whole summer, and you get assistance (advice, mentoring) from a seasoned Libav developer who knows his way around the project and has considerable standing in the community. By doing so, you'll learn to operate in an opensource project, you'll get relevant coding experience, and you'll have a chance at earning money while doing fun stuff during the summer. So, you need a project, a mentor, do a qualification task (see below) so we can quickly assess how good a candidate we feel you'll be for the program, and then you can apply.

=== Selecting a project ===
Below, you'll find two lists of projects:
* Projects with a mentor
* Projects without a mentor
If you choose a project with a mentor, talk to that mentor (see below) and select a suitable qualification task. Once completed, you're eligible for participating in our Summer of Code program. If you choose a project without a mentor, your first job is to find a mentor (see below). Then, once you've found a mentor, continue as before. If you don't like any of the projects, you're free to define your own project and find a mentor as mentioned before (see below for caveats).

=== Contacting developers/mentors ===
Once you've found a project (with or without mentor), start talking to the developers of the Libav project. We can often be found on IRC, and you can talk to us on mailinglists also. Hop on irc.freenode.net channel #libav-devel, or talk to us on libav-devel@libav.org. Here, you'll be able to ask around for mentors for projects without a mentor if you need to. If you're trying to define your own project, explain (with reasonable amount of detail) what you intend to achieve and why you think your project should be in our Summer of Code program. Once you've found a mentor, you're good to start your qualification task.

Note that the self-selected mentor needs to have considerable standing in the community to be eligible for mentoring. Likewise, if you choose to define your own Summer of Code project, some community members of considerable standing need to vouch for your project.

=== Your qualification task ===
The goal of a qualification task is to see if the mentor and student feel that, together, they will be able to finish the project of their choice. More specifically, the mentor will want to test whether the student has the skills and work ethics to complete a large coding project in a limited amount of time. The student will want to make sure that the mentor provides (useful) assistance when necessary. Therefore, students should select a mentor and a Summer of Code project before starting their work on a qualification task. The qualification task is often (but not necessarily) related to the selected project. For example, if your project will be to reverse engineer a new video codec, the qualification task may be to write a partial bitstream parser for that codec. If your project is to write a muxer for a container format, the qualification task may be to write the code to write the stream header.

There will be a second qualification task for every student: Pick a file of moderate size and reformat it in proper K&R style. The goal of this task is twofold: First it familiarizes students with the style that they will have to write their code in, second it demonstrates that students are able to submit patches from git and go through our review process.

=== Applying ===
While you're working on your qualification task, apply at http://www.google-melange.com/. The degree (and detail) with which you've finished your qualification task will determine how likely your project is to be selected. In the past few years, students that completely finished their qualification task always got selected as Summer of Code students, but that may vary depending on the number of spots we get assigned by Google, and the number of students that apply.

== 1st Tier Project Proposals ==
1st tier project proposals are project ideas that are reasonably well defined '''AND''' have a mentor volunteered.

=== Assembly Unit Testing Framework ===
* Libav has a lot of assembly and not enough tests for it. Your job is to write a unit testing framework for assembly.
* The framework should work across all supported architectures and operating systems.
* The framework should measure exactly how fast an individual function is (e.g. using START/STOP_TIMER).
* The framework should be able to test functions in isolation.
* x264's checkasm can be used as a reference.
* The qualification task will be to implement at least one unit test and have an idea of how to do the rest.

''Mentor: Daniel Kang'' (Jumpyshoes on #libav-devel@chat.freenode.net; daniel.d.kang@gmail.com -- ping me on IRC and email me).

=== HEVC/H265 decoder ===

* HEVC is the next-generation video standard from the MPEG standards committee. Your job is to write a decoder that can playback HEVC files.
* It does not need to be SIMD-optimized, but preferably has relevant functions separated in DSP contexts so they're easy to optimize later on.
* This task will depend on the current standardisation status of HEVC. Final publication is in 2013 but there may be parts which will not change much anyway.
* Draft spec: http://www.itscj.ipsj.or.jp/sc29/open/29view/29n12612c.htm
* HM (draft reference decoder) git mirror: http://hevc.kw.bbc.co.uk/git/w/jctvc-hm.git

Note that this is a difficult project. Do not try to apply unless you already have experience with H.264 as well as reading and writing complex C code.

''Mentor: [[User:Ronald_S._Bultje|Ronald S. Bultje]]''

=== DTS-HD decoder ===

* ETSI released specifcations (http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf). Your job is to complete the existing decoder with the following features.

(1) Add support for mixed Core + DTS-HD stream structure
(DtsCoreFrame+DtsHdFrame+DtsCoreFrame+DtsHdFrame+...), used by Blu-Ray main
and commentary tracks.
(2) Add support for XXCh extension (6.1 and 7.1 channels).
(3) Add support for X96 extension (96khz).
(4) Add support for XLL extension (lossless).
(5) Add support for a pure DTS-HD stream structure
(DtsHdFrame+DtsHdFrame+DtsHdFrame+...), used by Blu-Ray PiP tracks.
(6) Add support for XBR extension (extra bitrate).

''Mentor: Benjamin Larsson''

=== MPEG-4 ALS Roundup ===

This task is to update and enhance the existing ALS decoder as well as integrate
and enhance the rudimentary encoder found at:
https://github.com/justinruggles/FFmpeg-alsenc

Possible features are:

* implement rls-lms in the decoder
* do correct channel layout/sort handling in the decoder
* update to current master
* use codec private options
* implement encode2(), setting pts and duration
* document options and examples in encoders.texi
* come up with a good set of encoding tests for FATE
* implement mcc/channel sort in the encoder
* implement rls-lms in the encoder
* implement float support

''Mentor: [[User:Jruggle|Justin Ruggles]]''

=== Opus Decoder ===

Implement an independent Opus decoder using the publicly-available specification at:
http://tools.ietf.org/html/draft-ietf-codec-opus-11

* The reference source code should only be used as a normative document reference when required (i.e. this should not just be a port of libopus)
* Fully support Ogg/Opus mapping: https://wiki.xiph.org/OggOpus
* Handle CELT, SILK, and Hybrid modes (including transitions)
* (optional) Handle more than 2 channels

''Mentor: [[User:Jruggle|Justin Ruggles]]''

=== Adobe DNG Decoder (Basic Support) ===

Adobe Digital Negative (DNG) is an attempt at a universal file format for raw camera images. Most camera manufacturers have their own proprietary raw image format. Adobe provides tools for converting these to DNG with minimal or no loss of information for more reliable long-term support in a format with an open specification.

The project goal would be to add features required for basic support of DNG files. Some of these include:
* test/improve TIFF and LJPEG 16-bit decoding support
* implement both variants of JPEG-in-TIFF in the TIFF decoder
* add basic handling for Bayer CFA pixel format(s), including demosaicing
* conversion from camera colorspace to RGB
* export of DNG/TIFF/Exif metadata

Resources:
* [http://en.wikipedia.org/wiki/Digital_Negative Wikipedia Article]
* [http://wwwimages.adobe.com/www.adobe.com/content/dam/Adobe/en/products/photoshop/pdfs/dng_spec.pdf Specification]
* DNG samples can be created from other raw formats using the free [http://www.adobe.com/products/photoshop/extend.displayTab2.html DNG Converter] program
* A good place to find raw camera samples is http://www.imaging-resource.com

''Mentor: [[User:Jruggle|Justin Ruggles]]''

=== On2 VP7 decoder ===

VP7 is a DCT-based video codec. At the moment, it seems to lack a reference implementation, but we do have a spec and may take hint from libavcodec's VP8 and VP6 decoder and libvpx. MPlayer can decode VP7 by loading a binary. To begin working on the project, one has to setup a reference decoder against which to compare our output. This can be done by either writing a wrapper for the binary or (maybe) hacking libvpx itself.

You might want to discuss with us how and where to start. Drop by on IRC if you need help. It is not as difficult as it sounds.

* [http://multimedia.cx/mirror/VP7_Data_Format_and_Decoder_Overview.pdf Specification]
* [http://samples.libav.org/V-codecs/VP7/ Samples]

''Mentor: [[User:Shahriman|Mashiat Sarker Shakkhar]]''

=== RTMP[E|S|T|TE] protocol implementation ===

Currently librtmp is required for RTMPE, RTMPS, RTMPT, RTMPTE. The goal of this project is to implement these protocol variants natively in libavformat.

''Mentor: [[User:mstorsjo|Martin Storsjö]], [[User:lu_zero|Luca Barbato]]''

=== spin off build system into a separate project ===

Our build system is neat enough to make into a more general solution to be reused by other projects.

The goal of this project is to achieve exactly that. Intermediate steps will be reading, understanding and documenting the current build system, refactoring parts that can be generalized further and finally making a prototype implementation for libpostproc.

You will require skills in POSIX shell, GNU Make and a firm command of English.

''Mentor: [[User:DonDiego|Diego Biurrun]]''

=== Hardware Acceleration API ===

Libav has two different and incomplete api to provide hardware acceleration wrapping, the project should provide a better abstraction and migrate the currently implemented ones and additional provide more.

* Draft the API (that will require knowledge of libavcodec)
* Port vaapi/hwaccel to the new API.
* Port vdpau to the new API
* Implement Freescale VPU support
* Implement TI dce support

''Mentor: [[User:lu_zero|Luca Barbato]]''

=== Rewrite avserver ===

Libav needs a better system to serve streams, the current codebase had a number of design defect showing its age. The new avserver should be written from scratch, leveraging the knowledge piled up.

The implementation will be incrementally complex and possibly modular.

* Implement support to rtsp record/announce
* Expose RTMP and RTSP specific API (so the server won't have to use private calls)
* Write a simple rtsp, http, rtmp redirector (listen for publish/announce and rebroadcast the received streams)
* Add the capability to serve on-demand content reading from a single path

Ideally the first implementation can be made using a poll/event loop and then moved to use threads.

''Mentor: [[User:lu_zero|Luca Barbato]]''

=== Swscale Cleanup and Additions ===

Libswscale is an open source library for colourspace conversion. However, it requires significant cleanup to allow
it to support modern features. Knowledge of x86 assembly is preferred, though it can be learnt as part of GSOC.

* Support for more interleaved colourspaces. This is useful because interleaved chroma provides improved cache coherency and would form the basis for future optimisations in libav's decoders.
* Support for the vast number of packed pixel formats (e.g r210, v210). Many are currently implemented as decoders but it is agreed that all uses of such formats are effectively colourspace conversions, they should be included in swscale. This task will involve collaborating with the student of "Adobe DNG Decoder" (if applicable) to rewrite the pixel format headers to support more complex pixel formats.
* Inline asm to yasm conversion
* RGB work (todo: expand this)
* Bayer layouts
* Other general cleanup

''Mentor: Kieran Kunhya (kierank)''

== 2nd Tier Project Proposals ==

=== DTS-LBR decoder ===

This is the DTS equivalent to E-AC3 but not technically related to DTS/DTS-HD. It is found in the following specification: http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf). The task is to find a way of making the official
decoder decode just the LBR stream (and not mix it into the main audio) and use that to verify decoder compliance of the decoder you wrote.
The spec may be incomplete or require parts to be reverse engineered from the binary.

''Mentor: ???''

[[Category:Libav]]

Maxis UTalk

2012-01-15T03:47:21Z

Summary:

* Extension: UTK
* Company: Maxis
* Decompression code: http://wiki.niotso.org/UTK
* Samples: http://niotso.org/utalk/

Maxis UTalk is a speech-oriented audio codec used in The Sims Online, which is based on CELP and operates at 32kbit/s.

== UTK File header ==
<pre>struct UTKHeader
{
char sID[4];
DWORD dwOutSize;
DWORD dwWfxSize;
/* WAVEFORMATEX */
WORD wFormatTag;
WORD nChannels;
DWORD nSamplesPerSec;
DWORD nAvgBytesPerSec;
WORD nBlockAlign;
WORD wBitsPerSample;
DWORD cbSize;
};
</pre>

* '''sID''' - A 4-byte string identifier equal to "UTM0"
* '''dwOutSize''' - The decompressed size of the audio stream
* '''dwWfxSize''' - The size in bytes of the WAVEFORMATEX structure to follow; must be 20
* '''wFormatTag''' - The decoded audio format; set to WAVE_FORMAT_PCM (0x0001)
* '''nChannels''' - Number of channels in the decoded audio data
* '''nSamplesPerSec''' - Sampling rate used in the decoded audio data
* '''nAvgBytesPerSec''' - Bytes per second consumed by the decoded audio data; equal to nChannels*nSamplesPerSec*wBitsPerSample/8 or nSamplesPerSec*nBlockAlign
* '''nBlockAlign''' - The number of bytes consumed by an audio frame (one sample for each channel) in the decoded audio data; equal to nChannels*wBitsPerSample/8
* '''wBitsPerSample''' - The bits per sample for one audio channel in the decoded audio data; 8-, 16-, or 24-bit, etc.
* '''cbSize''' - The size in bytes of extra format information appended to the end of the WAVEFORMATEX structure; must be 0. Note that in the original WAVEFORMATEX, this parameter is a WORD, not a DWORD.

Note that the part of the header from wFormatTag below is a Microsoft WAVEFORMATEX structure.

== Naming notes ==
The UTK files in The Sims Online were stored in DBPF archives without filenames. However, we can deduce the codec's name and file extension from this text contained in sys/tsoaudio.ini:

<pre>;kGZCLSID_cWaveXA = 0x1d07eb4b
;kGZCLSID_cGZSndSegmentDataSourceUTalk = 0x1b6b9806
;kGZCLSID_cGZWave = 0xbb7051f5
;kGZCLSID_cHitMp3DecoderDataSource = 0x3cec2b47
1=0x1d07eb4b,XA
2=0x1b6b9806,UTK
3=0xbb7051f5,WAV
4=0x3cec2b47,MP3</pre>

WAV and MP3 are common audio formats, and XA refers to the [[Maxis XA]] ADPCM codec (with extension ".xa" in The Sims 1); and most importantly, the UTK files in the DBPF archives have a Type ID that match 0x1b6b9806.

[[Category:Vocoders]]
[[Category:Audio Codecs]]
[[Category:Formats missing in FFmpeg]]
[[Category:Game Formats]]

Libav/ReleaseHowto

2011-12-31T14:25:15Z

Summary: Replace backtics by superior $() syntax.

== LibavReleaseHowto ==

These are more or less personal notes how Libav releases and point release tarballs are rolled.

=== Checklist for release preparation ===

* If necessary, create a new release tracking bug
* (constantly) Backport interesting patches from trunk
* check the bugzilla release tracking bug
* update Changelog, use 'git log --oneline' for inspiration
* Write/Review News entry for http://libav.org

=== Roll, sign and publish tarballs ===

* For 0.6 and later, update the VERSION file:
<code>
echo '$VERSION' > VERSION<br/>
git add VERSION<br/>
git commit -m"Update Version identification for $(cat VERSION)"
</code>

* push the changelog to repository
** git push ...

* create an annotated tag:
** git tag -a v$VERSION -m"$VERSION Release"

* For 0.7 and later, create a VERSION file from the RELEASE file (don't commit/push this)
** <code>cp -v RELEASE VERSION</code>

* Create the release tarball
** <code>tar czv --xform="s,^./,libav-$VERSION/," --exclude=.git -f /tmp/libav-$VERSION.tar.gz .</code>

* Testcompile this tarball

* Create the xz variant:
** <code>zcat /tmp/libav-$VERSION.tar.gz | xz > /tmp/libav-$VERSION.tar.xz</code>

* GPG signatures:
<code>
gpg -ab /tmp/libav-$VERSION.tar.gz<br/>
gpg -ab /tmp/libav-$VERSION.tar.xz
</code>

* Copy release notes and changelog:
<code>
cp -v Changelog /tmp/libav-$VERSION.changelog<br/>
cp -v doc/RELEASE_NOTES /tmp/libav-$VERSION.release<br/>
</code>

* Install tarballs to libav.org:
** <code>scp /tmp/libav-$(cat VERSION).* libav.org:releases</code>

=== Publish the release ===

* push the created, *annotated* tags
* Publish the prepared News entry for http://libav.org

* Enjoy!

Codecs using unitialized memory

2011-12-25T15:18:04Z

Summary:

Removing memset(128) from video_get_buffer() or avconv's own get_buffer() breaks some FATE tests. This means that those decoders are using unitialized memory; this should be fixed. The following tests fail:
* bethsoft-vid
* cdgraphics
* ansi
* aasc
* fraps-v1
* qtrle-1bit

Dxtory

2011-12-09T10:24:40Z

Summary:

* FourCC: xtor
* Company: [http://dxtory.com/ Dxtory]

The Dxtory video codec is used to store in real-time data that is captured from high frame-rate computer games played on Windows PCs using DirectX/OpenGL.

Frame data consists of 16-byte header and YV12 blocks (2x2 block of luma and two bytes for chroma).

[[Category:Video Codecs]]
[[Category:Screen Capture Video Codecs]]

Libavfilter Shortcomings

2011-12-04T19:53:48Z

Summary: remove addresed stuff

Here a list of Libavfilter shortcomings that should be addressed:

* No support for reconfiguration
* Avisynth-like features (script files, random access to source frames)
* higher-bit-depth support in most scenarios

High Efficiency Video Coding

2011-11-30T20:03:04Z

Summary: minor rewording

High Efficiency Video Coding (HEVC) is an emerging video coding standard that is scheduled to succeed [[H.264]]. As the successor to H.264, it is sometimes called H.265, though that is not an official name.

[[Category:Video Codecs]]

Libav libavcodec HOWTO

2011-11-16T02:43:40Z

Summary:

: ''This page will hold information on how libavcodec is structured an how to add demuxer and protocols to it''
: ''please make this page more complete if you can, thanks''

== Codec structure ==

== Introducing a new codec ==

=== libavcodec/Makefile ===

=== libavcodec/allcodecs.c ===

== Example code ==

[[Category:Libav Tutorials]]
[[Category:Tutorials]]
[[Category:Work In Progress]]

Libav demuxer HOWTO

2011-11-15T03:16:58Z

Summary:

: ''This page will hold information on how libavformat is structured an how to add demuxer and protocols to it''
: ''please make this page more complete if you can, thanks''

Libavformat provides means to retrieve codec data and stream metadata from container formats and network streams.

Demuxers let the application access or store the codec data and metadata within a container format providing at least timing information.

Protocols let you access container formats through any kind of storage or delivery system, being that a filesystem access (e.g. pipe, file), a low level network protocol (tcp, udp) or a application level protocol (rtmp, http, tls)

In general (TODO: make a diagram out of it)

Protocol -> Demuxer -> Encoded data with timing information (video frames, audio samples, timed metadata) -> Decoders -> Raw data with timing information -> Output

Input -> Raw data with timing information -> Encoders -> Coded data with timing information -> Muxer -> Protocol

== Demuxer structure ==

== Protocol structure ==

== Introducing a new demuxer or a new protocol ==

In order to make available the new demuxer or the new protocol you have to edit '''libavformat/allformats.c''' and '''libavformat/Makefile'''.
The former holds a list of available demuxers and protocol and is parsed by configure to generate build system
variables for it, the latter contains the actual build directives.

=== libavformat/Makefile ===

==== Demuxer specific ====

The convention regarding the file structure is to use, for a format named '''name''', '''name'''enc.c for muxer code and '''name'''dec.c for demuxer code, common code should go in '''name'''.c

Add
OBJS-$(CONFIG_NAME_MUXER) += nameenc.o name.o
OBJS-$(CONFIG_NAME_DEMUXER) += namedec.o name.o

to the Makefile in order to build.

==== Protocol specific ====

The convention regarding the file structure is to use use directly the protocol name, so for a protocol named '''name''', '''name'''.c.
If the protocol is tightly bound to a specific demuxer or could be mismatched is advised to use '''name'''proto.c (e.g. rtmpproto.c)

Add
OBJS-$(CONFIG_NAME_PROTOCOL) += nameproto.o

to the Makefile in order to build.

=== libavformat/allformats.c ===

Modifying allformats.c will trigger a build system warning:

'''Run configure in the top level directory and make clean, as config.h will be automatically modified'''.

It is advised to run:

make clean
make configure
make

In order to build the new code. Make sure you properly edited Makefile and allformats.c .

==== Demuxer specific ====

In order to add a demuxer:
REGISTER_DEMUXER(NAME, '''name''')

In order to add a muxer:
REGISTER_MUXER(NAME, '''name''')

In order to add muxer and demuxer in a single line:
REGISTER_MUXDEMU(NAME, '''name''')

The macro will add ff_'''name'''_demuxer or/and ff_'''name'''_muxer to the available formats.

==== Protocol specific ====

In order to add a protoco:
REGISTER_PROTOCOL(NAME, '''name''')

The macro will add ff_'''name'''_protocol to the available protocols.

== Example code ==

This section is merely an overview; please look at an actual demuxer for a more in depth and up to date example.
libavformat/avformat.h shows the different structure of a muxer (AVOutputFormat) and a demuxer (AVInputFormat)

WIP

[[Category:Libav Tutorials]]
[[Category:Tutorials]]
[[Category:Work In Progress]]

Libav technical

2011-11-14T02:16:06Z

Summary: Update tutorial

== Demuxer stuff ==
=== How to use demuxer with raw data ===
In order to demux raw data you need to parse the raw bistream and extract two kind of information: the frame data and the codec extradata.
We name those two activities parse and split. '''parse''' is used to reconstruct precisely one frame from raw packets, '''split''' is used for extracting extradata from the same stream.

An AVParser declarations for the bitstream XXX will look like this:

typedef struct XXXParseContext{
ParseContext pc; /* always include this first */
another data
};
AVParser some_parser = {
{ CODEC_ID_1 [CODEC_ID_2, ...] },
sizeof (XXXParseContext),
NULL, /* usually there is no need in parser_open */
xxx_parse,
ff_parse_close, /* again, use standard close function */
xxx_split,
};

And here is the code for some parser which splits frame by some markers:

static int xxx_find_frame_end(XXXParseContext *pc1, const uint8_t *buf,
int buf_size) {
int start_found, i;
uint32_t state;
ParseContext *pc = &pc1->pc;
start_found= pc->frame_start_found;
state= pc->state;
i=0;
if(!start_found){
for(i=0; i<buf_size; i++){
state= (state<<8) | buf[i];
if(state == MARKER){
start_found=1;
break;
}
}
}
if(start_found){
for(; i<buf_size; i++){
state= (state<<8) | buf[i];
if(state == MARKER){
pc->frame_start_found= 0;
pc->state= -1;
return i-3;
}
}
}
pc->frame_start_found= start_found;
pc->state= state;
return END_NOT_FOUND;
}

static int xxx_parse(AVCodecParserContext *s,
AVCodecContext *avctx,
uint8_t **poutbuf, int *poutbuf_size,
const uint8_t *buf, int buf_size)
{
XXXParseContext *pc1 = s->priv_data;
ParseContext *pc = &pc1->pc;
int next;
if(s->flags & PARSER_FLAG_COMPLETE_FRAMES){
next= buf_size;
}else{
next= xxx_find_frame_end(pc1, buf, buf_size);
if (ff_combine_frame(pc, next, (uint8_t **)&buf, &buf_size) < 0) {
*poutbuf = NULL;
*poutbuf_size = 0;
return buf_size;
}
}
*poutbuf = (uint8_t *)buf;
*poutbuf_size = buf_size;
return next;
}

== Decoder stuff ==

=== How to use delta frames ===
There are two ways: either use your own buffer for frame data or reget old frame and modify it.
This can be done by calling reget_buffer() instead of get_buffer().
Usual code:

if(avctx->get_buffer(avctx, &c->pic) < 0){
av_log(avctx, AV_LOG_ERROR, "get_buffer() failed\n");
return -1;
}

New code:

c->pic.reference = 1;
c->pic.buffer_hints = FF_BUFFER_HINTS_VALID | FF_BUFFER_HINTS_PRESERVE | FF_BUFFER_HINTS_REUSABLE;
if(avctx->reget_buffer(avctx, &c->pic) < 0){
av_log(avctx, AV_LOG_ERROR, "reget_buffer() failed\n");
return -1;
}

=== How to use bitstream ===
There are two bitstream reading methods - MSB and 32-bit little-endian word. In order to use the second you need to #define ALT_BITSTREAM_READER_LE before including bitstream.h.

Simple bitstream writing:

PutBitContext pb;
init_put_bits(&pb, buffer, max_bytes); // max_bytes is buffer size to avoid out-of-bounds write
put_bits(&pb, 13, value); // Note that the second argument is value size in bits, not actual value
size = put_bits_count(&pb); // get the number of bits written so far
flush_put_bits(&pb); // call this to finish bit writing so bits won't be lost in buffer

Simple bitstream reading:

GetBitContext gb;
init_get_bits(&gb, buffer, buffer_size * 8); // init_get_bits() expects buffer size in _bits_
a = get_bits(&gb, 16); // read some bits from stream
b = get_sbits(&gb, 15); // read some bits from stream as signed integer
c = get_bits_long(&gb, 18); // get_bits() is guaranteed to work only for bits <= 17, for greater values use get_bits_long()
d = show_bits(&gb, 5); // peek bits but don't change position in bitstream, also has show_bits_long() counterpart

Variable-length codes reading is also quite simple.

VLC vlc;
init_vlc(&vlc, MAX_BITS, codes_size,
bits, bits_skip, sizeof(bits[0]),
codes, codes_skip, sizeof(codes[0]), flags);
value = get_vlc2(&gb, vlc.table, MAX_BITS, wrap);
free_vlc(&vlc);

Notes on VLC reading:
* MAX_BITS is the maximum code length but not greater than 9 (i.e. if max codeword length is 5 then MAX_BITS=5 but if max codeword length is 15 then MAX_BITS=6)
* *_skip should be either equal to sizeof() if bits and codes are stored in separate tables or = ((uint8_t*)bits[1])-((uint8_t*)bits[0]). For example if you store all in struct { char bits; short value;} codes[100] you should call

init_vlc(&vlc, MAX_BITS, 100, &codes[0].bits, sizeof(codes[0]), 1, &codes[0].value, sizeof(codes[0]), 2, 0);

* flags may be:
** INIT_VLC_USE_STATIC - init codes statically ( free_vlc() is not needed)
** INIT_VLC_LE - use alternative VLC reading mode

* wrap = ceil(REAL_MAX_BITS/MAX_BITS) and should not be greater than 3. For example is maximum codeword size is 15 then wrap = ceil(15/9) = 2

=== How to use frame reordering ===
The simpliest way is to adapt decode_frame() from h263dec.c to your needs (that will also require to include MpegEncContext into your codec context).

=== Some example decoder ===
The simpliest decoder that does not even use bitstream reading is ATI VCR1 decoder, file libavcodec/vcr1.c.

For simple codec with VLC reading look at WNV1 (file libavcodec/wnv1.c).

For simple delta-frame codec look at RPZA (file libavcodec/rpza.c).

== MDCT ==
'''I would like to know how to do that properly''', for now this is the stuff borrowed from wmadec.c and wmaenc.c.

In order to perform N-point transform you need those variables:

MDCTContext mdct;
DSPUtil dsp;
float window[N];
float output[2*N];
float saved[2*N];
float tmp[N];
float coefs[N]; // result of MDCT will be stored here

Forward windowed MDCT:

memcpy(output, saved, sizeof(float)*N);
for (i = 0; i < N; i++){
output[i+N] = audio[i] / (N/2) * window[N - i - 1];
saved [i] = audio[i] / (N/2) * window[i];
}
ff_mdct_calc(&mdct, coefs, output, tmp);

Backward windowed MDCT:

ff_imdct_calc(&mdct, output, coefs, tmp);
dsp.vector_fmul_add_add(out, output, window, saved, 0.0f, N, 1);
dsp.vector_fmul_reverse(saved, output+1024, window, N);

[[Category:Libav Tutorials]]

MLC

2011-11-11T23:51:49Z

Summary:

* FourCC: MLCY
* Website: http://www.linek.sk/mlc/

MLC is a lossless video codec which supports YUY2, YV12, RGB24 and RGB32 colorspaces. The video codec is open source and released under the GNU GPL v3.

[[Category:Video Codecs]]
[[Category:Formats missing in FFmpeg]]

VBLE

2011-11-09T19:45:19Z

Summary:

* FourCC: VBLE

VBLE is a lossless codec for YUV colourspace written by a person known as Mark FD.
It employs standard median prediction and coding components with reduced number of bits.

== Frame format ==

Bitstream is stored in little-endian words LSB first.

Image data is coded in that order: the line of luma values, two lines of chroma values, all data is coded as pixel quads.

In the beginning of frame data there are four unary codes for each quad element size in bits.
The rest of data are quad values written with that number of bits.
After decoding pixel values first are renormalised by formula <code>pix & 1 ? 255 - (pix >> 1) : (pix >> 1)</code> and for each plane median restoration is applied.

[[Category:Video Codecs]]
[[Category:Video FourCCs]]

XMPG

2011-10-18T05:02:54Z

Summary:

* FourCC: xmpg
* Samples: http://samples.mplayerhq.hu/V-codecs/xmpg/

XMPG = Xing MPEG (I-Frame only)

[[Category:Video Codecs]]
[[Category:Video FourCCs]]

Space Adventure MOV

2011-10-01T04:59:51Z

Summary:

* Extension: mov
* Company: Knowledge Adventure
* Samples: http://samples.mplayerhq.hu/game-formats/space-adventure-mov/

Space Adventure is a 1993 CD-ROM title by Knowledge Adventure. It uses a multimedia format with the extension .mov which is not the [[Apple QuickTime]] format. It is likely a video-only format as a different directory on the CD-ROM contains [[Creative Voice]] files with similar filenames.

[[Category:Game Formats]]
[[Category:Video Codecs]]
[[Category:Undiscovered Game Formats]]
[[Category:Undiscovered Video Codecs]]

FATE failures

2011-09-17T16:57:39Z

Summary: /* Other samples from the H264 conformance suite that don't play back correctly */ cropping is supported now

Although FATE can appear green, in fact many of the samples in FATE are not bit-exact.

The following samples are not bit-exact.

=== H.264 ===

tandberg 4/5 have large codec delays because of misordered frames, and the delay jumps randomly within the file. We don't preroll well enough to know the correct delay in advance and so at each point where we return a frame too far ahead in the future, we drop the preceeding frames once we get around to decoding them because we've already passed that point in units returned to the application. The correct solution for these would be to increase delay dynamically whenever we encounter a gap until either we're at max-delay, or until we've resolved the gap. almost certainly, max-delay should be settable in user applications so we can disable it for live/low-latency purposes. Q: should we return old frames also? The user may want to reorder himself (although this doesn't make much sense). Fate currently uses -strict 1 as a workaround for these samples, which simply forces codec-delay to 16. This is not the correct solution.
MAIN/MR4_TANDBERG_C.264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal
MAIN/MR5_TANDBERG_C.264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal

=== Other samples from the H264 conformance suite that don't play back correctly ===

FMO:
MAIN/FM1_FT_E.264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal
MAIN/FM2_SVA_C.264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal
MAIN/FM1_BT_B.h264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal

For this one, we output only 12 of a few hundred-or-so frames, unanalyzed beyond that:
MAIN/sp1_bt_a.h264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal

Looks like garbage, unanalyzed beyond that:
MAIN/H26L/BitstreamExchange/sp2_bt_b.h264 ... JF ( yuv420p @ 176x144 / 25 fps) not equal

Looks like CAVLC-444 isn't completely implemented:
PP/PPCV444I-7.264 ... JF ( yuv420p @ 48x80 / 25 fps) not equal
PP/PPCV444I4_Mitsubishi_A.264 ... JF ( yuv420p @ 48x48 / 25 fps) not equal
PP/PPCV444I5_Mitsubishi_A.264 ... JF ( 50 fps @ 25 tbr / 25 fps) not equal
PP/PPCV444I6_Mitsubishi_A.264 ... JF ( yuv420p @ 48x48 / 25 fps) not equal
PP/Professional_profiles/PPCV444I1_Thomson_A.bits ... JF ( 50 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPCV444I2_Thomson_A.bits ... JF ( 50 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPCV444I3_Thomson_A.bits ... JF ( 50 fps @ 25 tbr / 25 fps) not equal

Missing 14-bits support:
PP/PPH444I4_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444I5_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444I6_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444P6_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444P7_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444P8_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444P9_Mitsubishi_A.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444I1_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444I2_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444I3_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444P1_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444P2_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444P3_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444P4_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/Professional_profiles/PPH444P5_Thomson_A.bits ... JF ( 25 fps @ 25 tbr / 25 fps) not equal

FMO + 14-bits:
PP/PPH444I-7.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal
PP/PPH444P-10.264 ... JF ( 25 fps @ 25 tbr / 25 fps) not equal

[[Category:FATE]]

Libav starting tasks

2011-09-04T10:54:03Z

Summary: /* Documentation */

Want to get started doing something useful in libav, but not sure where to start? Here's some basic tasks you can do that don't require much detailed knowledge of libav's internals. Need help? Ask on IRC and someone can answer questions and help walk you through your task of choice.

==Task categories==

===QA===

(TODO: make this a bit more detailed so it doesn't require as much searching/knowledge to do.)

Finding bugs in libav is always welcome, and requires little to no knowledge of libav itself. Here's one possible process:

* Make a build of libav with debug symbols.
* Download a sample file from the samples archive.
* Download zzuf and use it to fuzz libav on the file.
* When you find a crash, do a gdb backtrace and report the bug to the libav bug tracker following the bug tracker guidelines.

If a sample file doesn't seem to be giving any crashes, try a different one. Some decoders are more buggy than others.

===Documentation===

==== Low level API ====

==== Tutorials ====

===== Basic stuff =====

Would be nice have more increasingly complex example of our low level api usage.

- demux a container format and get the basic format packet information

Example https://github.com/simock85/libav/wiki/file-opening-and-packet-inspection

- extract a single track out a multi track file

TBD

- generate some synthetic audio and encode it

- generate some synthetic video and encode it

- generate an audio and video and encode it to a file

- add some subtitles in the mix

===Coding===

Noctropolis VID

2011-08-06T13:08:36Z

Summary: add another game

: ''See also [[VID]]''

* Extensions: vid
* Company: Flashpoint Productions, Inc.
* Year: 1994
* Samples: '''TODO'''

VID is an FMV format first appeared in the computer game [http://www.mobygames.com/game/dos/noctropolis Noctropolis]. Uses basic [[RLE]] compression.

A file consists of a header and a number of frame chunks. All numbers are little endian.

== File header ==

+ 0 s8 Signature[4] -- "VID"
+ 4 s8 Version -- usually 2
+ 5 s16 Frames -- number of video frames (not chunks!)
+ 7 s16 Width -- frame width and height
+ 9 s16 Height
+ B s16 Unknown -- related to playback speed?
+ D s16 Rate -- playback rate (?)

== Frame chunks ==
Each chunk begins with a byte chunk type then followed by payload of variable length.

=== 0x00 - Intraframe ===

u16 sync value, followed by Width * Height frame raster

=== 0x01 - Interframe ===

u16 sync value, followed by RLE-compressed raster. Version 2 movies has extra zero byte after it (if not ended permaturely by a stop-code.) Decompression algorithm:

while frame is not complete
len = next byte of input
if len is 0 finish
if high bit of len is set
leave (len & 0x7F) pixels unchanged
else
draw next (len) pixels

=== 0x02 - Palette ===

256 6-bit [[RGB]] [[DAC]] triplets.

=== 0x03 - RLE Intraframe ===

Same as 0x01, but no stop-codes and instead of skipping pixels, fill them with the next byte of input.

=== 0x04 - Partial Interframe ===

Same as 0x01, but has an extra u16 value after the sync field denotes how many horizontal lines to skip before start of decoding.

=== 0x7C - Audio Start ===

u16 sync, followed by u8 sampling frequency in SoundBlaster [[VOC]] format, then followed by u16 size of data and corresponding amount of actual [[PCM]] data in Mono, 8-bit Unsigned format.

=== 0x7D - Audio Continue ===

Same as 0x7C, but no sync/frequency fields.

== Games Using VID ==

* [http://www.mobygames.com/game/dos/noctropolis Noctropolis]
* [http://www.mobygames.com/game/synnergist Synnergist]

[[Category:Game Formats]]
[[Category:Video Codecs]]
[[Category:Formats missing in FFmpeg]]

Cyclemania Video

2011-07-12T05:15:38Z

Summary: typo

* Samples: http://samples.mplayerhq.hu/game-formats/cyclemania/

[http://www.mobygames.com/game/cyclemania Cyclemania] is a 1994 DOS game published by Accolade. It's a motorcycle racing game with a full motion video backdrop. Video in the game is handled with 4 files per animation:

* .COL file: color palette, always 776 bytes (2 32-bit numbers followed by 256 RGB triplets)
* .FRM file: unknown
* .HED file: header data including width and height and some other information
* .VID file: the size and extension of this file indicate that it hold the raw video data

[[Category:Game Formats]]
[[Category:Undiscovered Video Codecs]]

Interplay ACMP

2011-07-12T05:02:33Z

Summary: correct categories

* Company: [[Interplay Entertainment]]
* Samples: http://samples.mplayerhq.hu/game-formats/interplay-acmp/

Interplay ACMP is an audio compression file format used in some games from Interplay. The specific audio coding format is presently unknown but is suspected to be [[ACOMP]].

== Interplay ACMP Format ==
Multi-byte numbers are stored in big endian format.

bytes 0-19 Signature string: 'Interplay ACMP Data\x1A'
bytes 20-21 sample rate
bytes 22-23 unknown, always seems to be 0 and might be part of sample rate
byte 24 unknown, always seems to be 0x28
byte 25 unknown, always seems to be 0
bytes 26-29 possibly the size of decompressed audio
bytes 30.. suspected to be the encoded audio data

== Games Using Interplay ACMP Files ==
* [http://www.mobygames.com/game/star-trek-judgment-rites Star Trek: Judgment Rites]

[[Category:Audio Codecs]]
[[Category:Game Formats]]
[[Category:Undiscovered Audio Codecs]]

FFmpeg / Libav Summer Of Code In Space 2011

2011-07-10T19:36:58Z

Summary: Add IRC handle to page

== Qualification tasks ==

For us to consider your application for SOCIS we require a completed qualification task. Many Summer-of-Code projects (in the list below) have specific qualification tasks. These tasks are meant to make you familiar with the code that you will be working with, are at approximately the same difficulty level as the actual Summer-of-Code project itself (just a lot smaller), and often already provide you with a jumpstart into your Summer-of-Code project. We suggest the following order of events:
* First, select a Summer-of-Code project (either from the list below, but in some cases you may also come up with your own)
* Second, discuss this project with the person that will mentor it. If a mentor is listed, talk to him on IRC, via email or so. If no mentor is listed, find one by emailing the FFmpeg/Libav mailinglists.
* With your mentor, discuss the most appropriate qualification task for your Summer-of-Code project.
If no specific qualification task is listed for your project of interest, you can discuss with your mentor to choose a task from the [[Small FFmpeg Tasks|Small Tasks list]] or the [[Interesting Patches|Interesting Patches list]] instead. If your prospective mentor agrees, please send an email to the FFmpeg and Libav mailing list to inform that you are working on it (to avoid duplicated work). The qualification task is considered completed when your patch is accepted to the main Git tree.

Before posting to the FFmpeg/Libav mailing lists, make sure you read and understand our netiquette guidelines, especially avoid top-posting and thread-hijacking (note that if you don't understand one of those terms, make sure to have understood them before writing your first post). Before you send us your patch, read our [http://www.ffmpeg.org/general.html#SEC23 development guidelines] and make sure your patch fulfills all the requirements stated there. You should also familiar with the programs diff, patch and Git. You have to learn these basics on your own before you start, we will not teach them to you during the application process.

== 1st Tier Project Proposals ==
1st tier project proposals are project ideas that are reasonably well defined '''AND''' have a mentor volunteered.

=== [[Libavfilter]] video work ===

Libavfilter is the FFmpeg filtering library that started as a 2007 SoC [[FFmpeg Summer Of Code#Video Filter API (AKA libavfilter)|project]]. It replaced the now removed vhook subsystem. Most of it is already part of the FFmpeg main source tree, but there a few bits remaining. This project would consist in a combination of the following tasks:

* Get the remaining bits of the SoC tree committed (this includes: the rotate, and fps filters)
* Port all the missing filters from MPlayer (do not forget asking the authors if it is ok to release them under the LGPL)
* Framework: implement dynamic-reconfiguration of the filterchain, for supporting dynamic size/format changes
* Port filters from other frameworks (mjpeg-tools, effectv, frei0r, virtualdub, vlc, etc...)
* Write wrappers for more image processing libraries and filtering frameworks (libgimp, libgraphicsmagic, weed), extend the libopencv wrapper for supporting more filters (this may need implemented float image support in libswscale)
* Write more filters (possibly starting from already posted filters which for a reason or another were never committed, e.g. concatenate, fish, 2xsai, eval, posterize, elbg/posterize etc.)
* Implement a compose filter (suggestion for a better name?), for creating a mosaic of the input video streams
* Write a movie sink (e.g. it could be useful for implementing a display functionality, e.g. when the video is dumped to an output device)
* Write a frequency domain filter using the FFT implementation in libavcodec
* Write a Matlab/Octave/SAGE scripting wrapper (assuming it can be done)
* More ideas?

''Mentor: Stefano Sabatini (possibly with a backup mentor)''

=== Assembly Testing Framework ===

FFmpeg/Libav use an extensive amount of assembly to optimize video and audio decoding. For scientific applications and high-assurance systems (e.g. a spacecraft), accuracy and speed are vital. Assembly is difficult to code, and lends itself to mistakes. As FFmpeg/Libav has no framework for testing assembly functions, development of assembly is often hindered by each developer having to create his or her own tests. In addition, functions are not tested for speed. While FATE covers some functions, there are many that simply aren't tested. The samples in FATE do not cover all features in every decoder that has assembly functions (e.g. several 10-bit H.264 predict functions are not used in any FATE sample). FATE also does not test functions in isolation, which impedes assembly development. In addition, FATE might not cover all assembly versions of a function if multiple different levels of optimizations exist (e.g. MMX, MMX2, SSE, SSE2, SSSE3, SSE4, AVX, etc).

* Qualification task: describe how you will implement the system (probably based on x264's checkasm). It should test accuracy, speed, and be extensible.
* Actual task: implement the system. At least all H.264 assembly functions should be covered and the framework should be easily extensible.
* The framework should work across all supported architectures and operating systems.
* The framework should measure exactly how fast an individual function is (e.g. using START/STOP_TIMER).
* The framework should be able to test functions in isolation.
* x264's checkasm can be used as a reference.

''Mentor: Daniel Kang'' (Jumpyshoes on #libav-devel@chat.freenode.net; IRC is generally the best way to contact me).

=== Add Support for GeoTIFF metadata ===

FFmpeg/Libav supports decoding of TIFF images, but currently skips GeoTIFF tags which contain geographic location data, which is commonly used for satellite and aerial imagery. This would be a rather small project to export this metadata in textual form and parse the metadata for encoding.

* Qualification task: Add support in the TIFF decoder for the DOUBLE data format, which is used by many GeoTIFF tags.
* Implement simple parsing of GeoTIFF tags, just printing basic info to the console.
* Print the output in a more human-readable format.
* Add support for decoder export of metadata.
* Export each GeoTIFF tag as a metadata entry.
* Document the metadata format for each tag.
* Add support for encoder import of metadata.
* Import/validate GeoTIFF tags from metadata and write to TIFF output.
* reference: http://trac.osgeo.org/geotiff/

''Mentor: Justin Ruggles''

=== Libav Usage Example, Tutorial and test suite ===

* Prepare a serie of example program of increasing complexity
** Simple remuxer
** Simple decoding loop
** Simple encoding loop
** Simple transcoding loop
** Simple producer + consumer scenario with the network protocols
** ...
** Optionally it would lead to prepare a leaner equivalent to ffmpeg and ffplay

* Examine when there are inconsistencies with the current API (e.g. unexpected timestamp mangling, packet splitting) and fix the issues.

* Integrate the example programs as components of FATE.

* Document all the previous steps to reduce the learning curve of future contributors.

''Mentor: [[User:Lu_zero|Luca Barbato]]''

== 2nd Tier Project Proposals ==

These proposals lack of a mentor.

=== Finish GSoC projects from previous years ===

Some projects are lingering in the dark unfinished. They should be picked up and made ready for inclusion. These projects are potentially less involved than starting from scratch, but also more useful for FFmpeg/Libav since the probability that the projects get finished should be higher. If some of them are deemed too easy, they could be combined.

For the current status of all GSoC projects up to date, see [[FFmpeg Summer Of Code]].

=== Forward Error Correction ===

Add FEC to libavutil.
* RTP FEC, standardised as SMPTE 2022M (Possibly the same as Pro-MPEG FEC) - (not 100% sure if this is for all types of RTP or just some)
* MPEG-TS 16 bytes extra - (not sure where this is standardised - in DVB perhaps?)
''Mentor: ???''

FFmpeg/Libav Summer Of Code In Space

2011-07-10T19:16:59Z

Summary: [[FFmpeg/Libav Summer Of Code In Space]] moved to [[FFmpeg Summer Of Code In Space]]: Since for this year is pretty unlikely that FFmpeg will run a joint application with Libav, rename the page to be FFmpeg oriented, and less confusing for the uninformed

[http://sophia.estec.esa.int/socis2012/?q=about ESA Summer of Code in Space (SOCIS)] is a program run by the Advanced Concepts Team of the European Space Agency that offers student developers stipends to write code for various space-related open source software projects.

FFmpeg run a joint application with Libav for the pilot program of 2011.

* [[FFmpeg / Libav Summer Of Code In Space 2011|2011 project page]]
* [[FFmpeg Summer Of Code In Space 2012|2012 project page]]

Each accepted project is developed in its own sandbox, separate from the main FFmpeg codebase. Naturally, the end goal of each of the accepted projects ought to be to have that code in shape for acceptance into the production codebase. This page tracks the status of each project and how well each student did.