Libav Summer Of Code 2012
- 1 How it works
- 2 1st Tier Project Proposals
- 3 2nd Tier (need mentor) Project Proposals
- 4 ideas from previous years
- 4.1 BSAC AAC Decoder
- 4.2 Libav Usage Example, Tutorial and test suite
- 4.3 VC-1 decoder missing features
- 4.4 DTS Encoder
- 4.5 Hardware accelerated decoder for Android devices
- 4.6 Multithreading support
- 4.7 Libavfilter video work
- 4.8 Libavfilter audio work
- 4.9 Flash screen video (1)/2 encoder
- 4.10 MXF support
- 4.11 Standards Compliant TS Muxer
- 4.12 NEW Seeking API
- 4.13 h264 decoder optimizations
- 4.14 Add support for Bayer RGB colour format
- 4.15 Finish SoC projects from previous years
- 4.16 Implement a Windows Video Presentation 2 decoder
- 4.17 RALF Realaudio Lossless
- 4.18 libvo
- 4.19 Speex Decoder
- 4.20 CELT/Opus Decoder
- 4.21 AMR-NB Encoder
- 4.22 VP6 Encoder
- 4.23 WMV3 Encoder
- 4.24 Improve subtitle support
- 4.25 Improve Ratecontrol
- 4.26 SILK decoder and/or encoder
- 4.27 ATRAC3plus
- 4.28 MPEG-4 SLS
- 4.29 DirectShow capture
- 4.30 AAC-LD
- 4.31 SSE2 swscale
- 4.32 Forward Error Correction
How it works
Google's Summer of Code program is simple: you (the student) work on a project, full-time, during the whole summer, and you get assistance (advice, mentoring) from a seasoned FFmpeg/Libav developer who knows his way around the project and has considerable standing in the community. By doing so, you'll learn to operate in an opensource project, you'll get relevant coding experience, and you'll have a chance at earning money while doing fun stuff during the summer. So, you need a project, a mentor, do a qualification task (see below) so we can quickly assess how good a candidate we feel you'll be for the program, and then you can apply.
- Please detail the deadlines for applying and being a student/mentor.
Selecting a project
Below, you'll find two lists of projects:
- Projects with a mentor
- Projects without a mentor
If you choose a project with a mentor, talk to that mentor (see below) and select a suitable qualification task. Once completed, you're eligible for participating in our Summer of Code program. If you choose a project without a mentor, your first job is to find a mentor (see below). Then, once you've found a mentor, continue as before. If you don't like any of the projects, you're free to define your own project and find a mentor as mentioned before (see below for caveats).
Once you've found a project (with or without mentor), start talking to the developers of the FFmpeg and Libav projects. We can often be found on IRC, and you can talk to us on mailinglists also. Hop on irc.freenode.net channels #ffmpeg-devel or #libav-devel, or talk to us on firstname.lastname@example.org or email@example.com. (At this point, you're probably wondering why there are 2 IRC channels and 2 mailinglists. Ask your mentor, it'll make for a good first conversation topic.) Here, you'll be able to ask around for mentors for projects without a mentor if you need to. If you're trying to define your own project, explain (with reasonable amount of detail) what you intend to achieve and why you think your project should be in our Summer of Code program. Once you've found a mentor, you're good to start your qualification task.
Note that the self-selected mentor needs to have considerable standing in the community to be eligible for mentoring. Likewise, if you choose to define your own Summer of Code project, some community members of considerable standing need to vouch for your project.
Your qualification task
The goal of a qualification task is to see if the mentor and student feel that, together, they will be able to finish the project of their choice. More specifically, the mentor will want to test whether the student has the skills and work ethics to complete a large coding project in a limited amount of time. The student will want to make sure that the mentor provides (useful) assistance when necessary. Therefore, students should select a mentor and a Summer of Code project before starting their work on a qualification task. The qualification task is often (but not necessarily) related to the selected project. For example, if your project will be to reverse engineer a new video codec, the qualification task may be to write a partial bitstream parser for that codec. If your project is to write a muxer for a container format, the qualification task may be to write the code to write the stream header.
There will be a second qualification task for every student: Pick a file of moderate size and reformat it in proper K&R style. The goal of this task is twofold: First it familiarizes students with the style that they will have to write their code in, second it demonstrates that students are able to submit patches from git and go through our review process.
While you're working on your qualification task, apply at http://www.google-melange.com/. The degree (and detail) with which you've finished your qualification task will determine how likely your project is to be selected. In the past few years, students that completely finished their qualification task always got selected as Summer of Code students, but that may vary depending on the number of spots we get assigned by Google, and the number of students that apply.
1st Tier Project Proposals
1st tier project proposals are project ideas that are reasonably well defined AND have a mentor volunteered.
- HEVC is the next-generation video standard from the MPEG standards committee. Your job is to write a decoder that can playback HEVC files.
- It does not need to be SIMD-optimized, but preferably has relevant functions separated in DSP contexts so they're easy to optimize later on.
- This task will depend on the current standardisation status of HEVC. Final publication is in 2013 but there may be parts which will not change much anyway.
Mentor: Ronald S. Bultje
- ETSI released specifcations (http://www.etsi.org/deliver/etsi_ts/102100_102199/102114/01.03.01_60/ts_102114v010301p.pdf). Your job is to complete the existing decoder with the following features.
(1) Add support for mixed Core + DTS-HD stream structure (DtsCoreFrame+DtsHdFrame+DtsCoreFrame+DtsHdFrame+...), used by Blu-Ray main and commentary tracks. (2) Add support for XXCh extension (6.1 and 7.1 channels). (3) Add support for X96 extension (96khz). (4) Add support for XLL extension (lossless). (5) Add support for a pure DTS-HD stream structure (DtsHdFrame+DtsHdFrame+DtsHdFrame+...), used by Blu-Ray PiP tracks. (6) Add support for XBR extension (extra bitrate).
Mentor: Benjamin Larsson
MPEG-4 ALS Roundup
This task is to update and enhance the existing ALS decoder as well as integrate and enhance the rudimentary encoder found at: https://github.com/justinruggles/FFmpeg-alsenc
Possible features are:
- implement rls-lms in the decoder
- do correct channel layout/sort handling in the decoder
- update to current master
- use codec private options
- implement encode2(), setting pts and duration
- document options and examples in encoders.texi
- come up with a good set of encoding tests for FATE
- implement mcc/channel sort in the encoder
- implement rls-lms in the encoder
- implement float support
Mentor: Justin Ruggles
On2 VP7 decoder
VP7 is a DCT-based video codec. At the moment, it seems to lack a reference implementation, but we do have a spec and may take hint from libavcodec's VP8 and VP6 decoder and libvpx. MPlayer can decode VP7 by loading a binary. To begin working on the project, one has to setup a reference decoder against which to compare our output. This can be done by either writing a wrapper for the binary or (maybe) hacking libvpx itself.
You might want to discuss with us how and where to start. Drop by on IRC if you need help. It is not as difficult as it sounds.
Mentor: Mashiat Sarker Shakkhar
RTMP[E|S|T|TE] protocol implementation
Currently librtmp is required for RTMPE, RTMPS, RTMPT, RTMPTE. The goal of this project is to implement these protocol variants natively in libavformat.
Mentor: Martin Storsjö
spin off build system into a separate project
Our build system is neat enough to make into a more general solution to be reused by other projects.
The goal of this project is to achieve exactly that. Intermediate steps will be reading, understanding and documenting the current build system, refactoring parts that can be generalized further and finally making a prototype implementation for libpostproc.
You will require skills in POSIX shell, GNU Make and a firm command of English.
Mentor: Diego Biurrun
Improve the audio resampling/rematrixing/converting code
- right now, we're using libswresample to resample/rematrix audio (samplerate / channels) and to resample the audio format (int, float, 16-bit, 32-bit).
- both interleaved and planar audio sample formats are already supported
- We need SIMD optimization of popular conversions (float-int16, int16-float), (stereo-mono-5.1) and anything else thats frequently used.
- We need support for alternate conversion functions (e.g. sample format conversion with or without dithering)
- fix bugs in current design (none known but there sure are some)
Mentor: Michael Niedermayer
2nd Tier (need mentor) Project Proposals
- this task is to get libbluray integrated and working with as many discs as possible.
- Add the ability to encode and decode using Advanced Access Content System.
- Specifications: http://www.aacsla.com/specifications/
- existing implementation e.g. DumpHD: http://forum.doom9.org/showthread.php?t=123111
- Most parts (BD-J, MKB, title key generation) probably do not belong in libavcodec/libavformat, this should be discussed with us before submitting an application
- possible solution: only implement "lowest" level (decode given the correct title key) but implement CSS en- and decryption as secondary goal
ideas from previous years
BSAC AAC Decoder
- Separate AAC Huffman decoding and dequantization from the rest of AAC decoder.
- Write a BSAC AAC decoder.
- Reuse as much of the existing AAC decoder and libavcodec as possible.
Libav Usage Example, Tutorial and test suite
- Prepare a series of example program of increasing complexity
- Simple remuxer
- Simple decoding loop
- Simple encoding loop
- Simple transcoding loop
- Simple producer + consumer scenario with the network protocols
- Optionally it would lead to prepare a leaner equivalent to avconf and avplay
- Examine when there are inconsistencies with the current API (e.g. unexpected timestamp mangling, packet splitting) and fix the issues.
- Integrate the example programs as components of FATE.
VC-1 decoder missing features
- Add support for interlaced streams as used in Bluray recordings to the VC-1 decoder.
- Add support for chroma/luma scaling, used in RTP/VC1.
- Add support for slice decoding, used in RTP/VC1.
- Add optimizations for x86-32/64 (or ARM), so that HD streams play back faster.
- This includes fixing some reference streams
- Specification: http://wiki.multimedia.cx/index.php?title=Mirror
- Sample files: http://samples.libav.org/A-codecs/DTS/
- Qualification: Extend the encoder in the SoC tree (port the float transform from the decoder, fix the LFE channel generation).
- Primary goal: Encoder that can produce multi sample rate, multi channel files and multi bit-rate. Wav and raw muxing support.
- Secondary goal: Techniques from specification implemented, optimal codebook usage, vector quantization, simple psycho acoustics.
- Third goal: Advanced psycho acoustics.
Hardware accelerated decoder for Android devices
Most Android phones supports decoding H264 via an OpenMAX decoder interface. This interface is exposed through the Stagefright multimedia framework. The flashplugin uses this to be able to have hardware accelerated decode of H264 bitstreams from Android user space. As many Android devices will not be upgraded to Android 2.3 or users don't want to root/reflash their firmware this will enable a lot of devices to install a libavcodec based media player while still having hardware accelerated H264 decode.
The goal of this task will be to improve the multithreading support. The task is relatively unspecific, but your basic goal will be to add multithreading support to a variety of decoders, by using existing infrastructure. Example goals could include:
- Fix problems with the libavcodec slice-based H.264 decoder threading (ask Jason for details here)
- Implement functions in the avconv/avplay binaries to autodetect the number of CPUs/cores and autodetect the number of threads to use (if unspecified)
- define ways to use both slice-based and frame-based multithreading at the same time if decoders (and files) support both
- Multithreading in other decoders that currently don't support it
- implement slice-based and/or frame-based threading in the libavcodec VP8 decoder
- implement frame-based threading in MJPEG decoder
- implement frame-based (or if you want to hurt yourself, try slice-based) multithreading in VC-1 decoder
Libavfilter video work
Libavfilter is the filtering library that started as a 2007 SoC project. It replaced the now removed vhook subsystem. Most of it is already part of the main source tree, but there a few bits remaining. This project would consist in a combination of the following tasks:
- Get the remaining bits of the SoC tree committed (this includes: the rotate and fps filters)
- Port all the missing filters from MPlayer (do not forget asking the authors if it is ok to release them under the LGPL)
- Framework: implement dynamic-reconfiguration of the filterchain, for supporting dynamic size/format changes
- Port filters from other frameworks (mjpeg-tools, effectv, frei0r, virtualdub, vlc, etc...)
- Write wrappers for more image processing libraries and filtering frameworks (libgimp, libgraphicsmagic, weed), extend the libopencv wrapper for supporting more filters (this may need implemented float image support in libswscale)
- Write more filters (possibly starting from already posted filters which for a reason or another were never committed, e.g. concatenate, fish, 2xsai, select, lut, eval, posterize, elbg/posterize etc.)
- Implement a compose filter (suggestion for a better name?), for creating a mosaic of the input video streams
- Write a movie sink (e.g. it could be useful for implementing a display functionality, e.g. when the video is dumped to an output device)
- Write a frequency domain filter using the FFT implementation in libavcodec
- Write a Matlab/Octave/SAGE scripting wrapper (assuming it can be done)
- More ideas?
Libavfilter audio work
At the moment, libavfilter has incomplete support for handling audio. This task would consist of a combination of the following tasks:
- Fix the remaining outstanding issues and complete the audio framework integration (check saste's audio-filters branch on gitorious)
- Make the resampling filter works for several combinations of sample formats and channels
- Write a visualization filter as proof-of-concept of a filter that works with both video and audio
- Extend/improve libav* audio resampling/rematrixing/requantization capabilities (this may need to re-design the audio API, maybe it deserves a task on its own)
- Write/port basic audio filters (check for example MPlayer/libaf)
- Write wrapper for audio frameworks (sox, ladspa, ...)
Flash screen video (1)/2 encoder
- Enhance the encoder patch posted to ffmpeg-devel so it can be committed
- Write optimal 2-pass encoding support for the flash screen video 1 and possibly 2 encoder (dijkstra graph)
- Support Clip Wrapping, Op1B, Avid proprietary files
- Support muxing DV, H.264
- Seeking support using index
Standards Compliant TS Muxer
- This is harder than it might seem at first because of the myriad of codecs and muxing rules.
- In order to complete this task you must find (or ask for) the spec, read it and understand the buffering models and what you will need to do for each codec. It's much easier to start with MPEG codecs to begin with. In most cases this will involve extracting or guessing information from the input stream.
- Sometimes spec-incompliant modes might be ok such as H.264 in TS files without HRD for iPod streaming.
- Later work can focus on remuxing
NEW Seeking API
- primary goal: implement a new seeking API in libavformat
- implement av_seek_file in libavformat
- implement compatible new seek_file for all AVInputFormat porting existing seek function if possible.
- implement av_build_index function which will build an AVIndex for the file
- implement av_export_index function which will save AVIndex in a file which can be loaded later.
h264 decoder optimizations
- Quite simply, make our decoder faster
- Qualification task: make our decoder 1% faster
Add support for Bayer RGB colour format
Since it is not even clear if this should be implemented in libswscale or libavcodec, this should be discussed before submitting. There was a related discussion on mplayer-devel once and at least two related FFmpeg issues:
- Contact Peter Ross for his latest working patch (Jan-2011)
Finish SoC projects from previous years
Some projects are lingering in the dark unfinished. They should be picked up and made ready for inclusion. These projects are potentially less involved than starting from scratch, but also more useful since the probability that the projects get finished should be higher. If some of them are deemed too easy, they could be combined.
Unfinished projects from previous years are:
For the current status of all SoC projects up to date, see FFmpeg / Libav Summer Of Code.
Implement a Windows Video Presentation 2 decoder
- Fourcc: WVP2
- Samples: http://samples.libav.org/V-codecs/WVP2/
- Suggested qualification task: Reverse engineer the frame header parsing or the actual frame data parsing
RALF Realaudio Lossless
- RE and implement a decoder for this format
- Reuse as much libavcodec code as possible
- Port MPlayer's libvo to ffplay
- Note that this does not just mean to produce a working hack so that ffplay can use xv, but a clean and acceptable wrapper for (most of) libvo.
- An alternative might be to only port the OpenGL part of libvo into a new glplay. This should be discussed before submission.
Also see Speex.
- Specification: http://celt-codec.org/docs
Also see CELT.
- Specification: http://www.3gpp.org/ftp/Specs/html-info/26-series.htm
- Samples: http://samples.libav.org/A-codecs/amr/
Also see AMR.
- Clearly defined task
- Primary goal: Encode video sequences such that they can be decoded by a Windows Media player.
This could either be done by improving this patch or by writing the encoder from scratch.
Improve subtitle support
- Add text-to-bitmap conversion functions
- One with hard-coded bitmaps for characters
- One that utilizes freetype
- Function used will be chosen upon compilation
Adjust existing subtitle support to new ABI
- Primary goal 1: Fast heuristic VBV compliant per macroblock ratecontrol which has a better PSNR/bitrate and better subjective quality/bitrate than the current code.
- Primary goal 2: VBV compliant, rate distortion optimal per macroblock ratecontrol using the viterbi algorithm.
- Secondary goal 1: Fast heuristic scene change detection which detects scene changes more accurately, has better PSNR/bitrate and subjective quality/bitrate than the current heuristic.
- Secondary goal 2: Rate distortion optimal (for the current picture) scene change detection.
- Secondary goal 3: B frames decision which is faster and or has a higher PSNR/bitrate and subjective quality/bitrate than the current code.
SILK decoder and/or encoder
- A specification was published: http://tools.ietf.org/html/draft-vos-silk-01
- Implement a decoder for ATRAC3plus.
- libavcodec has currently no support for MPEG-4 SLS
- Implement an indev for DirectShow capture on Windows (and support A/V synchronisation).
- Qualification: Get GDI and waveform capture integrated into master with A/V synchronisation. Both patches are almost finished.
- Implement an AAC-LD decoder.
- swscale largely implements MMX. Add SSE2 support using the yasm macro library.
- Difficult variants: add OpenCL, high bit-depth support, gamma-accurate conversion, better dithering, or edge-directed interpolation.
Forward Error Correction
Add FEC to libavutil - this is relevant to ESA summer of code.
- RTP FEC, standardised as SMPTE 2022M (Possibly the same as Pro-MPEG FEC) - (not 100% sure if this is for all types of RTP or just some)
- MPEG-TS 16 bytes extra - (not sure where this is standardised - in DVB perhaps?)