FFmpeg Summer of Code 2013: Difference between revisions

From MultimediaWiki
Jump to navigation Jump to search
(→‎Information for Students: subheadings, contact info)
(move bayer and apng to finished, theres a patch for apng and bayer was committed.)
 
(41 intermediate revisions by 9 users not shown)
Line 5: Line 5:
FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter and stream audio and video. It includes libavcodec - the leading audio/video codec library.
FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter and stream audio and video. It includes libavcodec - the leading audio/video codec library.


[https://developers.google.com/open-source/soc/ Google Summer of Code (GSoC)] is a program that offers students stipends to write code for open source projects. Through the guidance of mentors, students gain valuable experience interacting with and coding for open source projects like FFmpeg. Additionally, the project and it's users benefit from code created from students who often continue contributing as developers. FFmpeg participated to several past editions ([[FFmpeg Summer Of Code 2006|2006]], [[FFmpeg Summer Of Code 2007|2007]], [[FFmpeg Summer Of Code 2008|2008]], [[FFmpeg Summer Of Code 2009|2009]], [[FFmpeg Summer Of Code 2010|2010]], and [[FFmpeg / Libav Summer Of Code 2011|2011]]), and we are looking forward to being involved this year. This is our ideas page for [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 Google Summer of Code 2013].
[https://developers.google.com/open-source/soc/ Google Summer of Code (GSoC)] is a program that offers students stipends to write code for open source projects. Through the guidance of mentors, students gain valuable experience interacting with and coding for open source projects like FFmpeg. Additionally, the project and its users benefit from code created from students who often continue contributing as developers. FFmpeg participated to several past editions ([[FFmpeg Summer Of Code 2006|2006]], [[FFmpeg Summer Of Code 2007|2007]], [[FFmpeg Summer Of Code 2008|2008]], [[FFmpeg Summer Of Code 2009|2009]], [[FFmpeg Summer Of Code 2010|2010]], and [[FFmpeg / Libav Summer Of Code 2011|2011]]), and we are looking forward to being involved this year.  
 
This is our ideas page for [http://www.google-melange.com/gsoc/homepage/google/gsoc2013 Google Summer of Code 2013].
 
'''FFmpeg has not been accepted this year''', so if you want to do any of the qualification tasks and projects, they are just for fun.


== Information for Students ==
== Information for Students ==


=== Getting Started ===
=== Getting Started ===
0. '''Get to know FFmpeg.''' If you are a student and interested in contributing to an FFmpeg GSoC project it is recommended to start by subscribing to the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel] mailing-list, visiting our IRC channels (''#ffmpeg-devel'' and ''#ffmpeg''), and exploring the codebase and the development workflow. Feel free to [[#Contacting_FFmpeg|contact us]] if you have any questions.
1. '''Find a project.''' Listed on this page are mentored and unmentored projects. Mentored projects are well-defined and mentors have already volunteered. Unmentored projects are additional ideas that you may consider, but you will have to contact us to find a mentor. You may also propose your own project that may be a better match for your interest and skill level.
2. '''Contact us.''' If you find a project that you are interested in then get in touch with the community and let us know. In case you want to work on a qualification task, you should ask the appointed mentors so that the task can be claimed.
3. '''Apply.''' Student proposal period begins April 22, 2013 at 19:00 UTC and ends May 3rd at 19:00 UTC. See the [http://www.google-melange.com/gsoc/document/show/gsoc_program/google/gsoc2013/help_page#2._What_is_the_program_timeline GSoC timeline] for additional information.


=== Qualification Tasks ===
=== Qualification Tasks ===


FFmpeg GSoC tasks can be challenging, and a qualification task will show us that the student has the potential and motivation to finish his or her task.
In order to get accepted you will be requested to complete a small task in the area you want to contribute. FFmpeg GSoC projects can be challenging, and a qualification task will show us that you are motivated and have the potential to finish a project.
 
The qualification task is usually shown in the project description. Contact the appointed mentors for assistance on getting a related qualification task or if you want to propose your own. See a list of [[Small FFmpeg Tasks]] or browse the [https://ffmpeg.org/trac/ffmpeg FFmpeg Bug Tracker] for qualification task ideas.


=== Contacting FFmpeg ===
=== Contacting FFmpeg ===
Line 23: Line 37:
* '''FFmpeg GSoC Admins:''' [[User:Stefanosa|Stefano Sabatini]] and [[User:Llogan|Lou Logan]]
* '''FFmpeg GSoC Admins:''' [[User:Stefanosa|Stefano Sabatini]] and [[User:Llogan|Lou Logan]]


= Proposing Ideas =
You can also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.
 
If you have a project idea please contact a developer first at the #ffmpeg-devel IRC channel on Freenode or via the [http://ffmpeg.org/contact.html ffmpeg-devel mailing list]. A good source of ideas is the [https://ffmpeg.org/trac/ffmpeg/ FFmpeg bug tracker] and [[FFmpeg_Summer_of_Code_2012|FFmpeg Summer of Code 2012 Ideas List]].
 
When adding an idea follow this template for consistency:
 
<pre>
== Task Title ==
 
'''Description:''' A few sentences or a short paragraph describing the task.
 
'''Expected results:''' Bulleted list or paragraph describing what the student is expected to achieve.
 
'''Prerequisites:''' Skills or knowledge required by student.
 
'''Qualification Task:''' Small task student should perform to demonstrate understanding.
 
'''Mentor:''' List mentor and backup mentor if there is one and contact info such as IRC name or email address.
</pre>


'''Note:''' This section will be removed upon our organization application.
= Mentored Projects =


= Mentored tasks =
This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor a mentored project rather than propose your own. Contact the appointed mentor(s) to get more information about the project and the requested qualification task.


== H.264 Multiview Video Coding (MVC) ==
== H.264 Multiview Video Coding (MVC) ==
Line 51: Line 47:
<div class="floatright">[[Image:Mmspg-epfl-ch-double-camera.jpg]]</div>
<div class="floatright">[[Image:Mmspg-epfl-ch-double-camera.jpg]]</div>


'''Description:''' MVC samples exist and the codec is used on Bluray media, but FFmpeg is missing a decoder. Since this task also consists of some changes in the current architecture, it is especially important that this task is discussed on the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing list].
'''Description:''' MVC samples exist and the codec is used on Blu-ray media, but FFmpeg is missing a decoder. Since this project also consists of some changes in the current architecture, it is especially important that this project is discussed on the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing list].


'''Expected results:''' Create MVC decoder and add a test for the FFmpeg Automated Testing Environment (FATE).
'''Expected results:''' Create MVC decoder and add a test for the FFmpeg Automated Testing Environment (FATE).
Line 61: Line 57:
'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)
'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)


== Animated Portable Network Graphics (APNG) ==
'''Backup mentor:''' Kieran Kunhya (''kierank'' on IRC)
 
'''Description:''' FFmpeg currently does not support Animated PNGs.
 
'''Specification:''' https://wiki.mozilla.org/APNG_Specification
 
<div class="floatright">[[Image:Animated PNG example bouncing beach ball.png]]</div>
 
'''Expected results:'''
* APNG demuxer
** implement robust probing:
*** PNG images are not misdetected as APNG animations
*** APNG animations are not misdetected as PNG images
** splits stream into sensible packets (so they can be easily reused in APNG muxer)
** survives fuzzing (zzuf)
** add FATE coverage, coverage should be at least 70%
** test code under valgrind so no invalid reads/writes happen


* APNG decoder
** use existing PNG decoder code (write decoder in same file)
** implement parsing of all APNG chunks (acTL, fcTL, fdAT)
** error handling
** survives fuzzing (zzuf)
** add test for FATE, coverage should be at least 75%
** CRC checksum validation
** test code under valgrind so no invalid reads/writes happen
* APNG muxer && APNG encoder
** use existing PNG encoder code (write encoder in same file)
** write compliant files, make sure they play correctly in major web browsers that support APNG
** add test for FATE
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.
'''Qualification Task:''' Implement format autodetection for imagepipe & image demuxer
'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)


== Misc Libavfilter extension ==
== Misc Libavfilter extension ==
Line 119: Line 80:
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.


'''Qualification task:''' write or port one or more filters
'''Qualification Task:''' write or port one or more filters


'''Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' in IRC)
'''Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)


'''Backup mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)
'''Backup mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)
Line 127: Line 88:
== Subtitles ==
== Subtitles ==


'''Description:''' FFmpeg has been working on improving its subtitles support recently, notably by adding the support for various text subtitles and various hardsubbing (burning the subtitles onto the video) facilities. While the theme may sound relatively simple compared to audio/video signal processing, the project carry an historical burden not easy to deal with, and introduces various issues very specific to its sparse form.
'''Description:''' FFmpeg has been working on improving its subtitles support recently, notably by adding the support for various text subtitles and various hardsubbing (burning the subtitles onto the video) facilities. While the theme may sound relatively simple compared to audio/video signal processing, the project carries an historical burden not easy to deal with, and introduces various issues very specific to its sparse form.


<center>[[Image:Subtitles-sensei.jpg]]</center>
<center>[[Image:Subtitles-sensei.jpg]]</center>
Line 140: Line 101:
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background in fansubbing area (notably ASS experience) would be a bonus but is not strictly required.
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. Some background in fansubbing area (notably ASS experience) would be a bonus but is not strictly required.


'''Qualification task:''' write one subtitles demuxer and decoder (for example support for Spruce subtitles format). This is in order to make sure the subtitles chain is understood.
'''Qualification Task:''' write one subtitles demuxer and decoder (for example support for Spruce subtitles format). This is in order to make sure the subtitles chain is understood.


'''Mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)
'''Mentor:''' [[User:Ubitux|Clément Bœsch]] (''ubitux'' on IRC)
Line 159: Line 120:
'''Prerequisites:''' C coding skills, good x86 assembly coding skills, familiarity with git/source code control systems.
'''Prerequisites:''' C coding skills, good x86 assembly coding skills, familiarity with git/source code control systems.


'''Qualification task:''' convert 1 or 2 MMX2 functions to SSE2 and AVX2.
'''Qualification Task:''' convert 1 or 2 MMX2 functions to SSE2 and AVX2.


'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)
'''Mentor:''' [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)
'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)


<br clear="all">
<br clear="all">
== Bayer RGB colorspaces ==
<div class="floatright">[[Image:350px-Bayer_pattern_on_sensor.svg.png ]]</div>
'''Description:''' Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmpeg's applicability to RAW still and video photography processing. Tasks:
'''Expected Results:'''
* Rebase existing patches
* Implement bayer transformations in libswscale (plain C)
* Add bayer formats to the libavutil pixfmt enumeration routines
* SIMD optimizations of the libswscale transformations
* Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)
Optional goodies:
* Extend TIFF decoder to support DNG-Bayer format
* Support a popular proprietary camera format (many to choose from; see dcraw project)
'''Qualification task''': TBD
'''Mentor''': [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)
'''Backup Mentor''': [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)


<br clear="all">
<br clear="all">
Line 196: Line 136:


'''Description:'''  
'''Description:'''  
A MPEG-4 ALS decoder was implemented several years ago but an encoder is still missing in the official codebase. A rudimentary encoder has already been written and is available on [https://github.com/justinruggles/FFmpeg-alsenc.git github]. For this task, that encoder is first to be updated to fit into the current codebase of FFmpeg and to be tested for conformance using the [http://www.nue.tu-berlin.de/menue/forschung/projekte/beendete_projekte/mpeg-4_audio_lossless_coding_als/parameter/en/#230252 reference codec and specifications]. Second, the encoder is to be brought through the usual reviewing process to hit the codebase at the end of the project.
A MPEG-4 ALS decoder was implemented several years ago but an encoder is still missing in the official codebase. A rudimentary encoder has already been written and is available on [https://github.com/justinruggles/FFmpeg-alsenc.git github]. For this project, that encoder is first to be updated to fit into the current codebase of FFmpeg and to be tested for conformance using the [http://www.nue.tu-berlin.de/menue/forschung/projekte/beendete_projekte/mpeg-4_audio_lossless_coding_als/parameter/en/#230252 reference codec and specifications]. Second, the encoder is to be brought through the usual reviewing process to hit the codebase at the end of the project.


'''Expected results:'''  
'''Expected results:'''  
Line 207: Line 147:
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. A certain interest in audio coding and/or knowledge about the FFmpeg codebase could be beneficial.
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems. A certain interest in audio coding and/or knowledge about the FFmpeg codebase could be beneficial.


'''Qualification task:''' Add floating point support to MPEG-4 ALS decoder
'''Qualification Task:''' Add floating point support to MPEG-4 ALS decoder


'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)
'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)
'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)


<br clear="all">
<br clear="all">
Line 215: Line 157:
== Hardware Acceleration (hwaccel) API v2 ==
== Hardware Acceleration (hwaccel) API v2 ==


'''Description:''' FFmpeg supports hardware accelerated decoding through the internal hwacel API. Currently supported system hardware acceleration APIs are VA-API (Linux), DXVA2 (Windows) and VDA (MacOS X). However, the current approach requires client applications to allocate the underlying resources (e.g. hardware surfaces and context) themselves, and hanging them over to FFmpeg. This incurs a few limitations: this is not scalable to new codecs, i.e. this requires new tokens for each newly supported codec ; this incurs extra work in the client application, which tends to be duplicated over several client applications ; and this prevents efficient fallback to software decoding mode if the hardware cannot handle a particular codec specification.
<div class="floatright">[[Image:Hardware.jpg]]</div>


The goal of this project is to revamp the FFmpeg Hardware Acceleration API so that hardware resources are allocated and managed in the library, thus requiring the client application to only provide a single hardware context/device handle ; provide a way to fallback early to software decoding mode if the underlying hardware won't be able to handle the bitstream ; and make it possible to select a hardware accelerator by ID and not polluting the PixelFormats namespace.
'''Description:''' FFmpeg supports hardware accelerated decoding through the internal hwacel API. Currently supported system hardware acceleration APIs are VA-API (Linux), DXVA2 (Windows) and VDA (MacOS X). However, the current approach requires client applications to allocate the underlying resources (e.g. hardware surfaces and context) themselves, and handing them over to FFmpeg. This incurs a few limitations: this is not scalable to new codecs, i.e. this requires new tokens for each newly supported codec; this incurs extra work in the client application, which tends to be duplicated over several client applications; and this prevents efficient fallback to software decoding mode if the hardware cannot handle a particular codec specification.
 
The goal of this project is to revamp the FFmpeg Hardware Acceleration API so that hardware resources are allocated and managed in the library, thus requiring the client application to only provide a single hardware context/device handle; provide a way to fallback early to software decoding mode if the underlying hardware won't be able to handle the bitstream; and make it possible to select a hardware accelerator by ID and not polluting the PixelFormats namespace.


'''Expected results:'''
'''Expected results:'''
Line 248: Line 192:
'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)
'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)


<br clear="all"/>
'''Backup Mentor:''' Hendrik Leppkes (''nevcairiel'' on IRC)
 
<br clear="all">
 
== Hardware Accelerated Video Encoding with VA-API ==


= Unmentored tasks =
'''Description:''' FFmpeg already supports hardware accelerated decoding for multiple codecs but still lacks support for hardware accelerated encoding. The aim of the project is to add support for encoding with VA-API specifically, while keeping a generic enough approach in mind so that other hardware accelerators (TI-DSP, CUDA?) could be supported as well. This means that new ''hwaccel'' hooks are needed and two operational modes are possible: either ''(i)'' driver or hardware pack headers themselves, or ''(ii)'' lattitude is left to perform this task at the FFmpeg library level.


This is a list of ideas that students are encouraged to consider if a mentored task is unavailable or not within the student's skill or interests. A mentor will then be assigned for the task. A student can also [[#Your_Own_Idea|propose their own task]].
'''Expected results:''' Allow MPEG-2 and H.264 encoding with VA-API, while supporting variable bitrate (VBR) by default, and allowing alternate methods like constant bitrate (CBR) or constant QP (CQP) where appropriate or requested.
* MPEG-2 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** (Optionally) add support for interlaced contents
* H.264 encoding:
** Add basic encoding with I/P frames (handle the ''-g'' option)
** Add support for B frames (handle the ''-bf'' option)
** Add support for constant bitrate (CBR, i.e. ''maxrate == bitrate'' and ''bufsize'' set)
** Add support for constant QP (CQP, i.e. handle the ''-cqp'' option)
** Add support for more than one reference frame, while providing/using API to query the hardware capabilities
** Work on HRD conformance. May require to write an independent tool to assess that
** (Optionally) add configurability of the motion estimatation method to use. Define new types for HW accelerated encoding with at least two levels/hints for the accelerator.
* FFmpeg applications:
** Define common hwaccel interface for encoding
** Add initial support for hardware accelerated encoding to the ''ffmpeg'' application


== glplay ==
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, hardware supporting VA-API for encoding.


<div class="floatleft">[[Image:Opengl_logo.jpg]]</div>
'''Qualification Task:''' Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.


'''Description:''' The SDL library that is used by FFplay has some deficiencies, adding OpenGL output to FFplay should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which may be relicensed under the LGPL.
'''Mentor:''' [[User:Gwenole_Beauchesne|Gwenole Beauchesne]] (''__gb__'' on IRC)


'''Mentor:''' TBD Backup: Reimar Döffinger
'''Backup Mentor:''' Tushar Gohad


<br clear="all">
<br clear="all"/>


== AAC ==
== AAC Improvements ==


'''Description:''' FFmpeg contains an AAC encoder and decoder, both of them can be improved in various ways. This is enough work for more than one GSoC project, so one part of your submission would be to define on which task exactly you want to work.
'''Description:''' FFmpeg contains an AAC encoder and decoder, both of them can be improved in various ways. This is enough work for more than one GSoC project, so one part of your submission would be to define on which task exactly you want to work.
Line 272: Line 236:
* AAC 960/120 MDCT window
* AAC 960/120 MDCT window
* AAC multi-channel encoding
* AAC multi-channel encoding
See also the FFmpeg bug tracker for AAC issues, fixing one of them or rebasing the existing incomplete BSAC decoder for current git head fixing one or more existing bugs are possible qualification tasks.


== DTS / DCA ==
'''Qualification Task:''' See the FFmpeg bug tracker for AAC issues, fixing one of them or rebasing the existing incomplete BSAC decoder for current git head or fixing one or more existing bugs are possible qualification tasks.
 
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems, knowledge about transform based audio coding would be useful.
 
'''Mentor:''' Baptiste Coudurier (''bcoudurier'' on IRC)
 
'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)
 
<br clear="all"/>
 
== DTS / DCA Improvements ==


'''Description:''' FFmpeg contains a DTS decoder.
'''Description:''' FFmpeg contains a DTS decoder.
Line 283: Line 256:
** Add support for multiple assets
** Add support for multiple assets
** Add support for LBR extension
** Add support for LBR extension
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.
'''Mentor:''' Benjamin Larsson (''merbanan'' on IRC)
'''Backup Mentor:''' [[User:Stefanosa|Stefano Sabatini]] (''saste'' on IRC)
<br clear="all"/>
= Unmentored Projects =
This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also [[#Your_Own_Idea|propose their own project]].
== glplay ==
<div class="floatleft">[[Image:Opengl_logo.jpg]]</div>
'''Description:''' The SDL library that is used by FFplay has some deficiencies, adding OpenGL output to FFplay should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which may be relicensed under the LGPL.
'''Mentor:''' TBD Backup: Reimar Döffinger
<br clear="all">


== TrueHD encoder ==
== TrueHD encoder ==
Line 320: Line 315:
'''Description:''' [[VP8L]] is a lossless format used in WebP. There is no support for this in FFmpeg.
'''Description:''' [[VP8L]] is a lossless format used in WebP. There is no support for this in FFmpeg.


== Your Own Idea ==
== Your own idea ==


A student can propose a task. Ideas can also be found by browsing bugs and feature requests on our [https://ffmpeg.org/trac/ffmpeg/ bug tracker]. The work should last the majority of the GSoC duration, the task must be approved by the developers, and a mentor must be assigned.
A student can propose a project. Ideas can also be found by browsing bugs and feature requests on our [https://ffmpeg.org/trac/ffmpeg/ bug tracker]. The work should last the majority of the GSoC duration, the task must be approved by the developers, and a mentor must be assigned.


Students can discuss an idea in the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing-list], the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins [[User:Stefanosa|Stefano Sabatini]] or [[User:Llogan|Lou Logan]] for more information.
Students can discuss an idea in the [http://ffmpeg.org/mailman/listinfo/ffmpeg-devel ffmpeg-devel mailing-list], the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins [[User:Stefanosa|Stefano Sabatini]] or [[User:Llogan|Lou Logan]] for more information.
= Finished Projects =
== Animated Portable Network Graphics (APNG) ==
'''Description:''' FFmpeg currently does not support Animated PNGs.
'''Specification:''' https://wiki.mozilla.org/APNG_Specification
<div class="floatright">[[Image:Animated PNG example bouncing beach ball.png]]</div>
'''Expected results:'''
* APNG demuxer
** implement robust probing:
*** PNG images are not misdetected as APNG animations
*** APNG animations are not misdetected as PNG images
** splits stream into sensible packets (so they can be easily reused in APNG muxer)
** survives fuzzing (zzuf)
** add FATE coverage, coverage should be at least 70%
** test code under valgrind so no invalid reads/writes happen
* APNG decoder
** use existing PNG decoder code (write decoder in same file)
** implement parsing of all APNG chunks (acTL, fcTL, fdAT)
** error handling
** survives fuzzing (zzuf)
** add test for FATE, coverage should be at least 75%
** CRC checksum validation
** test code under valgrind so no invalid reads/writes happen
* APNG muxer && APNG encoder
** use existing PNG encoder code (write encoder in same file)
** write compliant files, make sure they play correctly in major web browsers that support APNG
** add test for FATE
'''Prerequisites:''' C coding skills, familiarity with git/source code control systems.
'''Qualification Task:''' Implement format autodetection for imagepipe & image demuxer
'''Mentor:''' [[User:Pbm|Paul B Mahol]] (''durandal_1707'' on IRC)
'''Backup mentor:''' [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)
== Bayer RGB colorspaces ==
<div class="floatright">[[Image:350px-Bayer_pattern_on_sensor.svg.png ]]</div>
'''Description:''' Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmpeg's applicability to RAW still and video photography processing.
'''Expected Results:'''
* Rebase existing patches
* Implement high quality bayer transformations in libswscale (plain C)
* Add bayer formats to the libavutil pixfmt enumeration routines
* SIMD optimizations of the libswscale transformations
* Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)
Optional goodies:
* Extend TIFF decoder to support DNG-Bayer format
* Support a popular proprietary camera format (many to choose from; see dcraw project)
'''Prerequisites''': C coding skills, familiarity with git/source code control systems.
'''Qualification Task''': Implement a simple and working Bayer->RGB transform in libswscale
'''Mentor''': [[User:Suxen_drol|Peter Ross]] (''pross-au'' on IRC)
'''Backup Mentor''': [[User:Michael|Michael Niedermayer]] (''michaelni'' on IRC)


[[Category:FFmpeg]]
[[Category:FFmpeg]]

Latest revision as of 11:19, 20 November 2014

Ffmpeg-logo-gsoc.jpg

Introduction

FFmpeg is the universal multimedia toolkit: a complete, cross-platform solution to record, convert, filter and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Google Summer of Code (GSoC) is a program that offers students stipends to write code for open source projects. Through the guidance of mentors, students gain valuable experience interacting with and coding for open source projects like FFmpeg. Additionally, the project and its users benefit from code created from students who often continue contributing as developers. FFmpeg participated to several past editions (2006, 2007, 2008, 2009, 2010, and 2011), and we are looking forward to being involved this year.

This is our ideas page for Google Summer of Code 2013.

FFmpeg has not been accepted this year, so if you want to do any of the qualification tasks and projects, they are just for fun.

Information for Students

Getting Started

0. Get to know FFmpeg. If you are a student and interested in contributing to an FFmpeg GSoC project it is recommended to start by subscribing to the ffmpeg-devel mailing-list, visiting our IRC channels (#ffmpeg-devel and #ffmpeg), and exploring the codebase and the development workflow. Feel free to contact us if you have any questions.

1. Find a project. Listed on this page are mentored and unmentored projects. Mentored projects are well-defined and mentors have already volunteered. Unmentored projects are additional ideas that you may consider, but you will have to contact us to find a mentor. You may also propose your own project that may be a better match for your interest and skill level.

2. Contact us. If you find a project that you are interested in then get in touch with the community and let us know. In case you want to work on a qualification task, you should ask the appointed mentors so that the task can be claimed.

3. Apply. Student proposal period begins April 22, 2013 at 19:00 UTC and ends May 3rd at 19:00 UTC. See the GSoC timeline for additional information.

Qualification Tasks

In order to get accepted you will be requested to complete a small task in the area you want to contribute. FFmpeg GSoC projects can be challenging, and a qualification task will show us that you are motivated and have the potential to finish a project.

The qualification task is usually shown in the project description. Contact the appointed mentors for assistance on getting a related qualification task or if you want to propose your own. See a list of Small FFmpeg Tasks or browse the FFmpeg Bug Tracker for qualification task ideas.

Contacting FFmpeg

If you have questions or comments feel free to contact us via our mailing list, IRC channel, or e-mail one of the FFmpeg GSoC admins:

You can also contact a mentor directly if you have questions specifically related to one of the projects listed on this page.

Mentored Projects

This section lists well-defined projects that have one or more available mentors. If you are new to FFmpeg, and have relatively little experience with multimedia, you should favor a mentored project rather than propose your own. Contact the appointed mentor(s) to get more information about the project and the requested qualification task.

H.264 Multiview Video Coding (MVC)

Mmspg-epfl-ch-double-camera.jpg

Description: MVC samples exist and the codec is used on Blu-ray media, but FFmpeg is missing a decoder. Since this project also consists of some changes in the current architecture, it is especially important that this project is discussed on the ffmpeg-devel mailing list.

Expected results: Create MVC decoder and add a test for the FFmpeg Automated Testing Environment (FATE).

Prerequisites: C coding skills, familiarity with git/source code control systems.

Qualification Task: Perform work that demonstrates understanding of MVC and that is a subpart of the whole MVC implementation.

Mentor: Michael Niedermayer (michaelni on IRC)

Backup mentor: Kieran Kunhya (kierank on IRC)


Misc Libavfilter extension

Lavfi-gsoc-filter-vintage-illustration.jpg

Description: Libavfilter is the FFmpeg filtering library. It currently supports audio and video filtering and generation support. This work may focus on porting, fixing, extending, or writing new audio and video filters from scratch.

Candidate filters for porting may be the remaining MPlayer filters currently supported through the mp wrapper, libaf MPlayer filters, and filters from other frameworks (e.g. mjpegtools, transcode, avisynth, virtualdub, etc.). In case of mp ports, the student should verify that the new filter produces the same output and is not slower.

Some ideas for more filters:

  • a frequency filtering domain filter relying on the FFT utils in libavcodec
  • a controller filter which allows to send commands to other filters (e.g. to adjust volume, contrast, etc.), e.g. like the sendcmd filter but through an interactive GUI
  • a lua scripting filter, which allows to implement filtering custom logic in lua

For more ideas check: trac libavfilter tickets.

Expected results: Write or port audio and video filters and possibly fix/extend libavfilter API and design when required.

Prerequisites: C coding skills, familiarity with git/source code control systems. Some background on DSP and image/sound processing techniques would be a bonus but is not strictly required.

Qualification Task: write or port one or more filters

Mentor: Stefano Sabatini (saste on IRC)

Backup mentor: Clément Bœsch (ubitux on IRC)

Subtitles

Description: FFmpeg has been working on improving its subtitles support recently, notably by adding the support for various text subtitles and various hardsubbing (burning the subtitles onto the video) facilities. While the theme may sound relatively simple compared to audio/video signal processing, the project carries an historical burden not easy to deal with, and introduces various issues very specific to its sparse form.

Subtitles-sensei.jpg

Expected results:

  • Add support for new subtitles formats. Example: a demuxer for .SUP files, just like VobSub but for Blu-Ray, or a VobSub muxer.
  • Improve text subtitles decoders. Typically, this can be supporting advanced markup features in SAMI or WebVTT.
  • Update the API to get rid of the clumsy internal text representation of styles
  • Proper integration of subtitles into libavfilter. This is the ultimate goal, as it will notably allow a complete subtitles rendering for applications such as ffplay.
  • BONUS: if everything goes well, the student will be allowed to add basic support for teletext

Prerequisites: C coding skills, familiarity with git/source code control systems. Some background in fansubbing area (notably ASS experience) would be a bonus but is not strictly required.

Qualification Task: write one subtitles demuxer and decoder (for example support for Spruce subtitles format). This is in order to make sure the subtitles chain is understood.

Mentor: Clément Bœsch (ubitux on IRC)

Backup Mentor: Nicolas George (Cigaes on IRC)

Postproc optimizations

PostProc.jpg

Description: FFmpeg contains libpostproc, which is used to postprocess 8x8 DCT-MC based video and images (jpeg, mpeg-1/2/4, H.263 among others). The code though has been written a long time ago and its SIMD optimizations need to be updated to what modern CPUs support (AVX2 and SSE2+).

Expected results:

  • Convert all gcc inline asm in libpostproc to YASM.
  • Restructure the code so that it works with block sizes compatible with modern SIMD.
  • Add Integer SSE2 and AVX2 optimizations for each existing MMX/MMX2/3dnow optimization in libpostproc.

Prerequisites: C coding skills, good x86 assembly coding skills, familiarity with git/source code control systems.

Qualification Task: convert 1 or 2 MMX2 functions to SSE2 and AVX2.

Mentor: Michael Niedermayer (michaelni on IRC)

Backup Mentor: Stefano Sabatini (saste on IRC)



MPEG-4 ALS encoder

Description: A MPEG-4 ALS decoder was implemented several years ago but an encoder is still missing in the official codebase. A rudimentary encoder has already been written and is available on github. For this project, that encoder is first to be updated to fit into the current codebase of FFmpeg and to be tested for conformance using the reference codec and specifications. Second, the encoder is to be brought through the usual reviewing process to hit the codebase at the end of the project.

Expected results:

  • Update the existing encoder to fit into the current codebase.
  • Ensure conformance of the encoder by verifying using the reference codec and generate a test case for FATE.
  • Ensure the FFmpeg decoder processes all generated files without warnings.
  • Enhance the rudimentary feature set of the encoder.

Prerequisites: C coding skills, familiarity with git/source code control systems. A certain interest in audio coding and/or knowledge about the FFmpeg codebase could be beneficial.

Qualification Task: Add floating point support to MPEG-4 ALS decoder

Mentor: Paul B Mahol (durandal_1707 on IRC)

Backup Mentor: Stefano Sabatini (saste on IRC)


Hardware Acceleration (hwaccel) API v2

Hardware.jpg

Description: FFmpeg supports hardware accelerated decoding through the internal hwacel API. Currently supported system hardware acceleration APIs are VA-API (Linux), DXVA2 (Windows) and VDA (MacOS X). However, the current approach requires client applications to allocate the underlying resources (e.g. hardware surfaces and context) themselves, and handing them over to FFmpeg. This incurs a few limitations: this is not scalable to new codecs, i.e. this requires new tokens for each newly supported codec; this incurs extra work in the client application, which tends to be duplicated over several client applications; and this prevents efficient fallback to software decoding mode if the hardware cannot handle a particular codec specification.

The goal of this project is to revamp the FFmpeg Hardware Acceleration API so that hardware resources are allocated and managed in the library, thus requiring the client application to only provide a single hardware context/device handle; provide a way to fallback early to software decoding mode if the underlying hardware won't be able to handle the bitstream; and make it possible to select a hardware accelerator by ID and not polluting the PixelFormats namespace.

Expected results:

  • FFmpeg core library (libavcodec):
    • Core API extensions and improvements
      • Add open/close hooks in a way that is backwards compatible with hwaccel v1 enabled applications
      • Add new tokens describing hardware accelerators
      • Add new flags exposing HW capabilities like download/upload
      • Investigate the benefits or impacts to provide a global map/unmap capability to FFmpeg video buffers
    • Port hwaccels to v2 infrastructure
      • Port VA-API decoders to v2 infrastructure
      • Validate that VA-API decoders still work with existing applications supporting hwaccel v1
      • Provide download capability through vaGetImage()
      • Validate that ffplay can support this feature with minor changes, and definitely no change to the existing SDL renderer
      • Port VDPAU decoders to hwaccel v2 (optional), and investigate ways to preserve compatibility with older applications
  • FFmpeg applications:
    • Integrate hardware acceleration into ffplay
      • Create a video-output (VO) infrastructure to ffplay
      • Port the SDL renderer to the new VO infrastructure
      • Add support for VA-API: VA renderer through vaPutSurface(), add -hwaccel option to select "vaapi" renderer
      • Add support for VDPAU (optional): VDPAU renderer through VdpPresentationQueueDisplay()
    • Integrate hardware acceleration into ffmpeg
      • Add support for VA-API: use the VA/DRM API for headless (no-X display) decoding, use libudev to determine the device to use

Prerequisites: C coding skills, familiarity with git/source code control systems, hardware supporting VA-API.

Qualification Task: Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

Mentor: Gwenole Beauchesne (__gb__ on IRC)

Backup Mentor: Hendrik Leppkes (nevcairiel on IRC)


Hardware Accelerated Video Encoding with VA-API

Description: FFmpeg already supports hardware accelerated decoding for multiple codecs but still lacks support for hardware accelerated encoding. The aim of the project is to add support for encoding with VA-API specifically, while keeping a generic enough approach in mind so that other hardware accelerators (TI-DSP, CUDA?) could be supported as well. This means that new hwaccel hooks are needed and two operational modes are possible: either (i) driver or hardware pack headers themselves, or (ii) lattitude is left to perform this task at the FFmpeg library level.

Expected results: Allow MPEG-2 and H.264 encoding with VA-API, while supporting variable bitrate (VBR) by default, and allowing alternate methods like constant bitrate (CBR) or constant QP (CQP) where appropriate or requested.

  • MPEG-2 encoding:
    • Add basic encoding with I/P frames (handle the -g option)
    • Add support for B frames (handle the -bf option)
    • Add support for constant bitrate (CBR, i.e. maxrate == bitrate and bufsize set)
    • (Optionally) add support for interlaced contents
  • H.264 encoding:
    • Add basic encoding with I/P frames (handle the -g option)
    • Add support for B frames (handle the -bf option)
    • Add support for constant bitrate (CBR, i.e. maxrate == bitrate and bufsize set)
    • Add support for constant QP (CQP, i.e. handle the -cqp option)
    • Add support for more than one reference frame, while providing/using API to query the hardware capabilities
    • Work on HRD conformance. May require to write an independent tool to assess that
    • (Optionally) add configurability of the motion estimatation method to use. Define new types for HW accelerated encoding with at least two levels/hints for the accelerator.
  • FFmpeg applications:
    • Define common hwaccel interface for encoding
    • Add initial support for hardware accelerated encoding to the ffmpeg application

Prerequisites: C coding skills, familiarity with git/source code control systems, hardware supporting VA-API for encoding.

Qualification Task: Anything related to the Hardware Acceleration (hwaccel) API, or to its related users. e.g. port VDPAU acceleration to use hwaccel, add JPEG decoding support with VA-API, etc.

Mentor: Gwenole Beauchesne (__gb__ on IRC)

Backup Mentor: Tushar Gohad


AAC Improvements

Description: FFmpeg contains an AAC encoder and decoder, both of them can be improved in various ways. This is enough work for more than one GSoC project, so one part of your submission would be to define on which task exactly you want to work.

  • AAC LD decoder
  • AAC BSAC decoder: This has already been started, but the existing decoder still fails on many samples
  • AAC SSR decoder
  • AAC 960/120 MDCT window
  • AAC multi-channel encoding

Qualification Task: See the FFmpeg bug tracker for AAC issues, fixing one of them or rebasing the existing incomplete BSAC decoder for current git head or fixing one or more existing bugs are possible qualification tasks.

Prerequisites: C coding skills, familiarity with git/source code control systems, knowledge about transform based audio coding would be useful.

Mentor: Baptiste Coudurier (bcoudurier on IRC)

Backup Mentor: Stefano Sabatini (saste on IRC)


DTS / DCA Improvements

Description: FFmpeg contains a DTS decoder.

  • DTS-HD decoder improvements: A possible qualification task is to implement ticket #1920
    • Add support for X96 extension (96khz)
    • Add support for XLL extension (lossless)
    • Add support for pure DTS-HD streams that do not contain a DTS core
    • Add support for multiple assets
    • Add support for LBR extension

Prerequisites: C coding skills, familiarity with git/source code control systems.

Mentor: Benjamin Larsson (merbanan on IRC)

Backup Mentor: Stefano Sabatini (saste on IRC)


Unmentored Projects

This is a list of projects that students are encouraged to consider if a mentored project is unavailable or not within the students skill or interests. The student will have to find a mentor for the project. A student can also propose their own project.

glplay

Opengl logo.jpg

Description: The SDL library that is used by FFplay has some deficiencies, adding OpenGL output to FFplay should allow for better performance (and less bugs at least for some hardware / driver combinations). This could be a new application (glplay), but it is probably simpler to extend ffplay to use OpenGL. You can use code from MPlayer's OpenGL vo module which may be relicensed under the LGPL.

Mentor: TBD Backup: Reimar Döffinger


TrueHD encoder

Description: FFmpeg currently does not support encoding to one of the lossless audio formats used on Bluray discs. This task consists of implementing a TrueHD encoder that allows to losslessly encode audio to play it on hardware devices capable of TrueHD decoding.

Opus decoder

Opus.png

Description: Opus decoding is currently supported through the external libopus library

  • Write a native decoder, continue working on the existing unfinished implementation

A possible qualification task is to port the existing incomplete decoder to current git head and improve it to show that you are capable of working on this task.

VC-1 interlaced

Description: The FFmpeg VC-1 decoder has improved over the years, but many samples are still not decoded bit-exact and real-world interlaced streams typically show artefacts.

  • Implement missing interlace features
  • Make more reference samples bit-exact

As a qualification task, you should try to find a bug in the current decoder implementation and fix it.

JPEG 2000

Jpeg2000.jpg

Description: FFmpeg contains an experimental native JPEG 2000 encoder and decoder. Both are missing many features, see also the FFmpeg bug tracker for some unsupported samples. Work on an issue (for example from the bug tracker) as a qualification task to show that you are capable of improving the codec implementation.


VP7

Description: Not many VP7 samples are in the wild, but no open-source decoder exists although a specification exists. Write a decoder that reuses as much as possible of existing FFmpeg code: it is likely that functions of the existing decoders for On2-based formats will be useful.

VP8L

Description: VP8L is a lossless format used in WebP. There is no support for this in FFmpeg.

Your own idea

A student can propose a project. Ideas can also be found by browsing bugs and feature requests on our bug tracker. The work should last the majority of the GSoC duration, the task must be approved by the developers, and a mentor must be assigned.

Students can discuss an idea in the ffmpeg-devel mailing-list, the #ffmpeg-devel IRC channel, or contact the FFmpeg GSoC admins Stefano Sabatini or Lou Logan for more information.

Finished Projects

Animated Portable Network Graphics (APNG)

Description: FFmpeg currently does not support Animated PNGs.

Specification: https://wiki.mozilla.org/APNG_Specification

Animated PNG example bouncing beach ball.png

Expected results:

  • APNG demuxer
    • implement robust probing:
      • PNG images are not misdetected as APNG animations
      • APNG animations are not misdetected as PNG images
    • splits stream into sensible packets (so they can be easily reused in APNG muxer)
    • survives fuzzing (zzuf)
    • add FATE coverage, coverage should be at least 70%
    • test code under valgrind so no invalid reads/writes happen
  • APNG decoder
    • use existing PNG decoder code (write decoder in same file)
    • implement parsing of all APNG chunks (acTL, fcTL, fdAT)
    • error handling
    • survives fuzzing (zzuf)
    • add test for FATE, coverage should be at least 75%
    • CRC checksum validation
    • test code under valgrind so no invalid reads/writes happen
  • APNG muxer && APNG encoder
    • use existing PNG encoder code (write encoder in same file)
    • write compliant files, make sure they play correctly in major web browsers that support APNG
    • add test for FATE

Prerequisites: C coding skills, familiarity with git/source code control systems.

Qualification Task: Implement format autodetection for imagepipe & image demuxer

Mentor: Paul B Mahol (durandal_1707 on IRC)

Backup mentor: Peter Ross (pross-au on IRC)

Bayer RGB colorspaces

350px-Bayer pattern on sensor.svg.png

Description: Several image and video format store pixels using Bayer-pattern colorspaces. Supporting these format would broaden FFmpeg's applicability to RAW still and video photography processing.

Expected Results:

  • Rebase existing patches
  • Implement high quality bayer transformations in libswscale (plain C)
  • Add bayer formats to the libavutil pixfmt enumeration routines
  • SIMD optimizations of the libswscale transformations
  • Complete PhotoCINE demuxer to support Bayer format; (or another format of your choosing)

Optional goodies:

  • Extend TIFF decoder to support DNG-Bayer format
  • Support a popular proprietary camera format (many to choose from; see dcraw project)

Prerequisites: C coding skills, familiarity with git/source code control systems.

Qualification Task: Implement a simple and working Bayer->RGB transform in libswscale

Mentor: Peter Ross (pross-au on IRC)

Backup Mentor: Michael Niedermayer (michaelni on IRC)