Difference between revisions of "FFmpeg Summer Of Code 2007"

From MultimediaWiki
Jump to navigation Jump to search
(Added Justin as mentor for E-AC3)
(Monkey's Audio Decoder (APE): this was completed)
 
(78 intermediate revisions by 18 users not shown)
Line 1: Line 1:
 
Google is sponsoring their third annual [http://code.google.com/soc/ Summer of Code] for the summer of 2007. This entails sponsoring students to work on assorted open source projects as well as sponsoring mentors in those same projects. Everyone wins.
 
Google is sponsoring their third annual [http://code.google.com/soc/ Summer of Code] for the summer of 2007. This entails sponsoring students to work on assorted open source projects as well as sponsoring mentors in those same projects. Everyone wins.
  
[[FFmpeg]] was a Summer of Code participant in the [http://code.google.com/soc/2006/ffmpeg/about.html summer of 2006] (here is the [[FFmpeg Summer Of Code|corresponding Wiki page]]).
+
[[FFmpeg]] was a Summer of Code participant in the [http://code.google.com/soc/2006/ffmpeg/about.html summer of 2006] (here is the [[FFmpeg Summer Of Code 2006|corresponding Wiki page]]).
  
 
[[User:Multimedia Mike|Mike Melanson]] (mike -at- multimedia.cx) is the administrator and main point of contact for matters relating to the FFmpeg Summer of Code.
 
[[User:Multimedia Mike|Mike Melanson]] (mike -at- multimedia.cx) is the administrator and main point of contact for matters relating to the FFmpeg Summer of Code.
Line 10: Line 10:
  
 
* You have to know how to program in C fairly well.
 
* You have to know how to program in C fairly well.
* We would like you to submit a patch that fixes a bug or adds a feature to FFmpeg. By doing that we will know that you are qualified for the task or not. On this page there is a list of [[Summer Of Code 2007#Qualification_tasks|Qualification Tasks]] that can be done. But you are free to submit anything you feel might be of value to FFmpeg. The qualification task can be done after you have filed you application.
+
* We would like you to submit a patch that fixes a bug or adds a feature to FFmpeg. By doing that we will know that you are qualified for the task or not. On this page there is a list of [[FFmpeg Summer Of Code 2007#Qualification_tasks|Qualification Tasks]] that can be done. But you are free to submit anything you feel might be of value to FFmpeg. The qualification task can be done after you have filed you application (up until around April 7 as the list of accepted students is scheduled to be posted by Google on April 9).
 
* Submit a good application through the formal Google Summer of Code process during the application timeframe (March 14-24, 2007).
 
* Submit a good application through the formal Google Summer of Code process during the application timeframe (March 14-24, 2007).
 
* You have to have >35 hours per week to put into the project.
 
* You have to have >35 hours per week to put into the project.
Line 19: Line 19:
 
* March 6, 2007: Mike Melanson submitted FFmpeg mentoring application.
 
* March 6, 2007: Mike Melanson submitted FFmpeg mentoring application.
 
* March 14, 2007: Google are now accepting student Applications.
 
* March 14, 2007: Google are now accepting student Applications.
* March 15, 2007: FFmpeg got accepted as mentoring organization; now accepting student applications until March 24, 2007.
+
* March 15, 2007: FFmpeg got accepted as mentoring organization; now accepting student applications until March 24, 2007. Note that this is '''NOT''' the deadline to complete a qualification task; final student selections are to be made by April 9.
 +
* April 12, 2007: Google has allocated 8 project slots to FFmpeg.
  
 
== Project Proposals ==
 
== Project Proposals ==
Line 90: Line 91:
  
 
''Mentor: Justin Ruggles''
 
''Mentor: Justin Ruggles''
 +
 +
  
 
=== Ideas for more projects to be determined ===
 
=== Ideas for more projects to be determined ===
Line 98: Line 101:
 
* Samples: http://samples.mplayerhq.hu/jpeg2000/
 
* Samples: http://samples.mplayerhq.hu/jpeg2000/
  
==== Monkey's Audio Decoder ====
+
==== Monkey's Audio Decoder (APE) ====
 +
'''Completed'''
 
* Sources: [http://www.monkeysaudio.com/files/MAC_SDK_399.zip original sources], [http://sourceforge.net/projects/mac-port/ original sources port for non-win32 platforms], [http://jmac.sourceforge.net/ LGPLed Java implementation]
 
* Sources: [http://www.monkeysaudio.com/files/MAC_SDK_399.zip original sources], [http://sourceforge.net/projects/mac-port/ original sources port for non-win32 platforms], [http://jmac.sourceforge.net/ LGPLed Java implementation]
  
==== Finish LC AAC decoder and implement HE-AAC(+) ====
+
==== Finish LC-AAC decoder and implement HE-AAC decoder (LGPL) ====
* Code: http://svn.mplayerhq.hu/aac/
+
* LGPL code: http://svn.mplayerhq.hu/aac/
 
+
* GPL decoder (libfaad2): http://www.audiocoding.com
 +
* Possible aacPlus (a.k.a. AAC+) sub-project:
 +
** [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] decoder (aacPlus v1 support)
 +
** [http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] decoder (for aacPlus v2 support)
  
 
==== GSM Decoder ====
 
==== GSM Decoder ====
Line 119: Line 126:
  
 
==== NUT Muxer ====
 
==== NUT Muxer ====
 +
* General improvements and enhancements
  
 
==== DPX/Cineon Encoder AND Decoder ====
 
==== DPX/Cineon Encoder AND Decoder ====
Line 135: Line 143:
 
* Sources and specification: http://www.microsoft.com/whdc/xps/hdphotodpk.mspx
 
* Sources and specification: http://www.microsoft.com/whdc/xps/hdphotodpk.mspx
 
* http://en.wikipedia.org/wiki/HD_Photo
 
* http://en.wikipedia.org/wiki/HD_Photo
 +
 +
==== mp3PRO & aacPlus & MPEG Surround decoders ====
 +
* [http://www.codingtechnologies.com/products/mp3pro.htm mp3PRO] decoder. Note: mp3PRO decoding means MP3 + [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] demuxing/decoding. (Standard MP3 decoders can decode mp3PRO encoded files/streams but without [http://www.codingtechnologies.com/products/sbr.htm SBR] you do not get the full quality. By adding a [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder to FFmpeg and coupling it with the existing MP3 decoder you could playback mp3PRO at full quality? A [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder could be shared with a aacPlus decoder as aacPlus also uses [http://www.codingtechnologies.com/products/sbr.htm SBR]).
 +
* [http://www.codingtechnologies.com/products/aacPlus.htm aacPlus] (a.k.a. AAC+) decoder. Note: aacPlus v1 decoding means HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] demuxing/decoding, and aacPlus v2 decoding means HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] + [http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] demuxing/decoding. (Standard HE-AAC decoders can decode aacPlus encoded files/streams but without [http://www.codingtechnologies.com/products/sbr.htm SBR] and [http://www.codingtechnologies.com/products/paraSter.htm PS] you do not get the full quality. By adding a [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder and a [http://www.codingtechnologies.com/products/paraSter.htm PS] decoder to FFmpeg and coupling it with an existing HE-AAC decoder you could playback aacPlus at full quality? A [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder could be shared with a mp3PRO decoder as mp3PRO also uses [http://www.codingtechnologies.com/products/sbr.htm SBR]).
 +
* [http://www.mpegsurround.com MPEG Surround] decoder/parser (for all audio but especially MP3/mp3PRO and AAC/aacPlus as those are in use today). [http://www.codingtechnologies.com/products/mpgsrnd.htm MPEG Surround technology] share similar characteristics with [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] and [http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] demuxing/decoding, which [http://www.codingtechnologies.com/products/mp3pro.htm mp3PRO] and [http://www.codingtechnologies.com/products/aacPlus.htm aacPlus] decoders also use, so if SBR and PS decoders was added to FFmpeg then those could probebly share common code with a [http://www.mpegsurround.com MPEG Surround] decoder/parser. ([http://www.divx.com DivX Inc.] is one company that uses MPEG Surround technology to achieve 5.1 channel surround sound in smaller files).
 +
 +
==== Native DirectShow support ====
 +
Option to build FFmpeg decoder/encoder/demuxer/muxer and post-processing filters for the DirectShow API for Windows by Microsoft, (the native DirectX 8/9 Direct3D overlay for video playback), so that FFmpeg has native support to be compiled for DirectShow and thus be used directly by players that use DirectShow.
 +
* http://en.wikipedia.org/wiki/DirectShow
 +
http://en.wikipedia.org/wiki/Ffdshow
 +
* Specifications: http://msdn2.microsoft.com/en-us/library/ms783323.aspx
 +
**http://msdn2.microsoft.com/en-us/library/ms788119.aspx
 +
* Sources: http://sourceforge.net/projects/ffdshow-tryout/
 +
http://sourceforge.net/projects/drdivx/ (drffmpeg)
 +
 +
==== DirectX Video Acceleration (DXVA) 1.0 AND 2.0 for video decoding ====
 +
Support Microsoft DirectX VA (DXVA) API nativly for GPU assisted decoding under Windows.
 +
Note! For this, native support for the above mentioned DirectShow API is needed first.
 +
* http://en.wikipedia.org/wiki/DXVA
 +
* DXVA 1.0 (DirectX SDK) specifications: http://msdn2.microsoft.com/en-us/library/ms798379.aspx
 +
* DXVA 2.0 (Windows SDK) specifications: http://msdn2.microsoft.com/en-us/library/ms788119.aspx
 +
http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/MED134_WH06.ppt
 +
 +
==== Additional subtitle support ====
 +
*  Create a common 'subtitles parser library' (and/or an API system for adding support for additional subtitle formats?) - a common sub-library to FFmpeg with all subtile decoders/demuxers/parsers gathered (similar to the libpostproc and libavutils). Call it "libsubs" (or "libsub", "libsubtitles" or whatever). Move FFmpeg's existing VobSub and DVBsub code there, so no matter if they are bitmap or text-based subs all existing and future subtile code is collected there. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional subtitles.
 +
**Maybe use MPlayer's recently added "'''libass'''" (SSA/ASS subtile reader) as a base for such a common library?
 +
* Support for advanced SSA/ASS rendering
 +
**Possible source are libass or the asa library
 +
* Support bold, italic, underline, RGB colors, size changes and font changes for a whole line or part of one line
 +
* [http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=284541 Line 23 signal (a.k.a. "Wide-screen signal")] detecting and use for DVD-Video (VobSub)
 +
* Support for the subtitles HTML tags
 +
* Capability of displaying subtitles with no video enabled (for example for audio-books)
 +
* Support for Karaoke subtitles (for kar and cdg, etc.)
 +
* Dual-subtitle-display (display two subtitles/languages at the same time, one at the bottom as normal plus one at the top of the screen)
 +
* Capability of moving the subtitles in the picture (freetype renderer)
 +
*Support more subtitle formats (text and bitmap-based):
 +
**  [http://en.wikipedia.org/wiki/Closed_captioning Closed captioning (CC)] subtile support - (Closed captions for the deaf and hard of hearing, also known as "Line 21 captioning", uses VobSub bitmaps)
 +
***[http://www.xinehq.de xine] have a SPU decoder for subpictures and Closed Captions software decoding
 +
**  DirectVobSub (VSFilter) - standard VobSubs (DVD-Video subtitles) embedded in AVI containers
 +
**  [http://en.wikipedia.org/wiki/DivX#DivX_Subtitles_.28XSUB.29 DivX Subtitles (XSUB)] display/reader/decoder (Note: bitmap based subtitle, similar to VobSub)
 +
**  [http://en.wikipedia.org/wiki/SubRip SubRip (.srt)] subtile support (Note: simple text-based based subtitle with timestamp)
 +
**  Subviewer (.sub) subtile support (Note: simple text-based based subtitle with timestamp)
 +
**  MicroDVD (.sub) subtile support (Note: simple text-based based subtitle with timestamp
 +
**  [http://en.wikipedia.org/wiki/Sami_%28subtitle_tool%29 Sami (.smi)] subtile support (Note: simple text-based based subtitle with timestamp)
 +
**  [http://en.wikipedia.org/wiki/SubStation_Alpha SubStation Alpha (.ssa+.ass)] subtile support (Note: advanced text-based based subtitle with timestamps and XY location on screen)
 +
**  [http://en.wikipedia.org/wiki/Synchronized_Multimedia_Integration_Language RealText (.rt)] subtile support
 +
**  PowerDivx (.psb) subtile support
 +
**  [http://en.wikipedia.org/wiki/Universal_Subtitle_Format Universal Subtitle Format (.usf)] subtile support
 +
**  Structured Subtitle Format (.ssf) subtile support
 +
 +
==== libstream (a common 'stream client' library) ====
 +
*  Create a common 'stream demuxer/parser library' for the client-side to receive input streams (and/or API for adding support for additional streaming formats?) - a LGPL'ed sub-library in FFmpeg with all stream demuxers/parsers gathered (similar to the libpostproc and libavutil). Call it "libstream" (or "stream" or whatever). Move FFmpeg's existing stream code there like HTTP and RTSP/RTP. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional streaming formats. All togther making it super easy for audio/video players using FFmpeg to add all-in-one streaming support to their player.
 +
**Maybe use either [http://www.mplayerhq.hu MPlayer]'s "''stream''" library structure, [http://www.live555.com LIVE555], [http://curl.haxx.se cURL], or probably the better [http://live.polito.it/documentation/libnemesi libnemesi] as a base for such a common library?
 +
*Add support for additional streaming protocols (on the client side) and improve/enhance support for existing protocols:
 +
** HTTP (Hypertext Transfer Protocol) client
 +
***plus a SSL (Secure Sockets Layer) client support for HTTPS
 +
** UDP (User Datagram Protocol) client
 +
** RTSP - Real-Time Streaming Protocol (RFC2326) client
 +
** RTP/RTCP - Real-Time Transport Protocol/RTP Control Protocol (RFC3550) client
 +
** RTP Profile for Audio and Video Conferences with Minimal Control (RFC3551) client
 +
** RealMedia RTSP/RDT (Real Time Streaming Protocol /  Real Data Transport)  client
 +
** SDP (Service Discovery Protocol) / SSDP (Simple Service Discovery Protocol)  client
 +
** MMS (Microsoft Media Services) client
 +
*** including the subprotocol mmsh (MMS over HTTP) and mmst (MMS over TCP)
 +
 +
==== A/V filter API (audio and video pre-process/post-process filters API system) ====
 +
FFmpeg's already well-known [[libavcodec]] module has become the de facto standard library for video decoding and encoding in free software projects. Unfortunately, no similar standard library has surfaced for audio/video filtering and otherwise working with audio/video stream once it has been decoded. Various multimedia projects (such as MPlayer, Xine, GStreamer, VirtualDub, etc.) have implemented their own filter systems to various degrees of success. What is needed is a high quality audio and video filter API - efficient, flexible enough to meet all the requirements which have led various projects to invent their own filter system, and yet easy to use or develop new filters with. This proposal is to implement a high quality audio/video filter library for FFmpeg, where it can be easily used by other multimedia-related software projects.
 +
 +
''Mentor: A'rpi (has expressed interest of possibly helping with implementing a filter API in FFmpeg, he also volunteering to help porting the MPlayer filters too if a such API becomes available http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2007-April/051164.html)''
 +
 +
* Adopt MPlayer's A/V filter system or create a new API 'from scratch' for pre-process and post-process audio/video filters:
 +
** See http://article.gmane.org/gmane.comp.video.ffmpeg.devel/39130 for michaelni's idea of what to do.
 +
***Also read this discussion thread on MPlayer's mailing-list:
 +
****http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2007-April/051142.html
 +
*****http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2007-April/thread.html#51142
 +
**Take a look at other eixsting players API for filter plugins, like for example;
 +
*** [http://svn.mplayerhq.hu/mplayer/trunk/DOCS/tech/libmpcodecs.txt?revision=15625&view=markup MPlayer (libmpcodecs vf_*.c filters)], [http://xinehq.de/index.php/hackersguide#AEN864 Xine], [[ffdshow|FFdshow]], [[VLC media player|VLC]], [http://www.virtualdub.org/filtersdk VirtualDub], [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html GStreamer], [http://www.foobar2000.org foobar], and [http://www.xmms.org/ XMMS]
 +
** Decide on name of a such A/V filter API.
 +
*** libavfilter (conflicts with LAVF)? libavmunge?
 +
* Create (or port) additional pre-process and post-process video filters to FFmpeg:
 +
** General post-proc sources are [http://svn.mplayerhq.hu/mplayer/trunk/DOCS/tech/libmpcodecs.txt?revision=15625&view=markup MPlayer (libmpcodecs vf_*.c filters)], [http://xinehq.de/index.php/hackersguide#AEN864 Xine], [[ffdshow|FFdshow]], [[VLC media player|VLC]], [http://www.virtualdub.org/filtersdk VirtualDub], [http://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html GStreamer], [http://www.foobar2000.org foobar], and [http://www.xmms.org/ XMMS]
 +
**More [http://en.wikipedia.org/wiki/Image_scaling image scaling] methods:
 +
***[http://en.wikipedia.org/wiki/Linear_interpolation linear interpolation]
 +
***[http://en.wikipedia.org/wiki/Bilinear_interpolation bilinear interpolation]
 +
***[http://en.wikipedia.org/wiki/Cubic_interpolation cubic interpolation]
 +
***[http://en.wikipedia.org/wiki/Bicubic_interpolation bicubic interpolation]
 +
***[http://en.wikipedia.org/wiki/Trilinear_interpolation trilinear interpolation]
 +
** Croping
 +
** SSP (Statistical Post-Processing)
 +
** DeBlocking
 +
** DeRinging
 +
** IVTC
 +
** Sharpen / UnSharpen (Soften)
 +
** ReQuantization
 +
** Auto-Luminance
 +
** Blurring / DeNoising / Spatial Blur / Temporal Blur
 +
**  Deinterlace (weave AND bob) filters
 +
*** Possible sources: [http://sourceforge.net/projects/deinterlace DScaler] or [http://www.mir.com/DMG/Software/y4mscaler.html y4mscale]?
 +
** 2:3 pull-down / ivtc (inverse telecine) for 24 progressive-frames on 30 FPS TV's
 +
*** Possible sources: [http://sourceforge.net/projects/deinterlace DScaler] or [http://www.mir.com/DMG/Software/y4mscaler.html y4mscale]?
 +
** NTSC => PAL, and PAL => NTSC frame-rate (FPS) adjust and reclock filter for NTSC <=> PAL conversion
 +
*** NTSC <=> PAL frame-rate adjust FPS ratios?: 23.97 <=> 25, 24 <=> 25, 30 <=> 25, 25 <=> 30
 +
* Create (or port) additional pre-process and post-process audio filters:
 +
** Psychoacoustic audio processing
 +
** Artificial reverberation
 +
** Dolby Prologic 2 decoding
 +
** Audio re-sampler (sample rate converter) filter
 +
*** Possible source is [http://www.mega-nerd.com/SRC/ SRC (Secret Rabbit Code)]
 +
*Create a SDK (Software Development Kit) with templates for the a/v filter API
  
 
== Qualification tasks ==
 
== Qualification tasks ==
  
 
Add a note if you choose to work on a Qualification task to avoid duplicate work.
 
Add a note if you choose to work on a Qualification task to avoid duplicate work.
 +
 +
=== Quicktime IMA ADPCM encoder ===
 +
<B>COMPLETED</B>
 +
* Specification: ffmpeg decoder source, libavcodec/adpcm.c
 +
* Samples: http://samples.mplayerhq.hu/A-codecs/ima-adpcm/
  
 
=== TIFF encoder ===
 
=== TIFF encoder ===
 +
 +
'''Completed'''
 +
 
* Specification: http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf  
 
* Specification: http://partners.adobe.com/public/developer/en/tiff/TIFF6.pdf  
* Samples: http://samples.mplayerhq.hu/mov/tiff/
+
* Samples: http://samples.mplayerhq.hu/mov/tiff/
+
''Kamil Nowosad is working on this''
 +
 
 +
''Bartlomiej Wolowiec is working on this''
 +
 
 
=== Vivo demuxer ===
 
=== Vivo demuxer ===
 
* Specification: look at the [[MPlayer]] [[vivo]] demuxer [http://svn.mplayerhq.hu/mplayer/trunk/libmpdemux/demux_viv.c?revision=22301&view=markup]
 
* Specification: look at the [[MPlayer]] [[vivo]] demuxer [http://svn.mplayerhq.hu/mplayer/trunk/libmpdemux/demux_viv.c?revision=22301&view=markup]
* Samples: http://samples.mplayerhq.hu/vivo/
+
* Samples: http://samples.mplayerhq.hu/vivo/
+
''Alex Kalouguine is working on this task.''
 +
 
 
=== IFF/8SVX 8-bit audio demuxer ===
 
=== IFF/8SVX 8-bit audio demuxer ===
 
* Specification: http://netghost.narod.ru/gff/vendspec/iff/iff.txt, http://sox.sourceforge.net/AudioFormats-11.html, and [http://xine.cvs.sourceforge.net/xine/xine-lib/src/demuxers/demux_iff.c?view=markup xine demuxer]  
 
* Specification: http://netghost.narod.ru/gff/vendspec/iff/iff.txt, http://sox.sourceforge.net/AudioFormats-11.html, and [http://xine.cvs.sourceforge.net/xine/xine-lib/src/demuxers/demux_iff.c?view=markup xine demuxer]  
 
* Samples: http://aminet.net/mods/smpl/
 
* Samples: http://aminet.net/mods/smpl/
 +
 +
''Tyler Williams is working on this task.''
  
 
=== Port SGI image support to new API ===
 
=== Port SGI image support to new API ===
 +
 +
'''Completed'''
 +
 
* Code: http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/sgi.c?revision=7867&view=markup&pathrev=7972
 
* Code: http://svn.mplayerhq.hu/ffmpeg/trunk/libavformat/sgi.c?revision=7867&view=markup&pathrev=7972
 
FFmpeg changed image format APIs, but the SGI file format was never ported to the new API.
 
FFmpeg changed image format APIs, but the SGI file format was never ported to the new API.
 +
**patch pending on ffmpeg-devel
  
 
=== Optimize some code ===
 
=== Optimize some code ===
Do you think some code in ffmpeg could run faster? We would love to get some faster decoders or encoders.
+
Do you think some code in FFmpeg could be made to run faster? We always love to get faster decoders or encoders. Note that this will require some ASM (assembly) skills and using timer code to benchmark and compare. Please make sure that any new code do not break compiling for other platforms.
 
 
This requires some ASM skills and using timer code to benchmark.
 
 
 
Speedups needed in:
 
* h264 decoder
 
* vc-1 decoder
 
  
 +
Speedups via optimizations (like SIMD for [http://en.wikipedia.org/wiki/3DNow%21 3DNow], [http://en.wikipedia.org/wiki/MMX MMX/MMX2], [http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions SSE/SSE2/SSE3] and [http://en.wikipedia.org/wiki/AltiVec AltiVec]) are needed in FFmpeg's:
 +
* [[H.264]] video decoder optimizations.
 +
* [[VC-1]] video decoder optimizations.
 +
* [[AAC|AAC (LE-AAC and HE-AAC)]] audio decoder optimizations.
 +
* cat libavcodec/*.c | grep -i optimize for more files that need optimization.
 +
**''Andrew Savchenko is working on this''
  
 
=== BFI Playback System ===
 
=== BFI Playback System ===
Line 172: Line 307:
  
 
=== THP Playback System ===
 
=== THP Playback System ===
 +
 +
'''Completed'''
 +
 
Add FFmpeg playback capability for the [[THP]] format. This entails writing a new file demuxer and leveraging existing JPEG and ADPCM decoders to handle the video and audio data inside.
 
Add FFmpeg playback capability for the [[THP]] format. This entails writing a new file demuxer and leveraging existing JPEG and ADPCM decoders to handle the video and audio data inside.
  
=== Game Formats ===
+
''Marco Gerards is working on this (one patch applied, one patch pending)''
Several game formats are documented in this Wiki, but not yet implemented in FFmpeg. Some of them are [[Bethsoft VID]], ...
+
 
 +
=== Bethsoft VID ===
 +
 
 +
'''Completed'''
 +
 
 +
Add FFmpeg playback capability for the [[Bethsoft VID]] format (new demuxer, new video decoder).
 +
: patch applied. --Nicholas
 +
 
 +
=== Other Game Formats ===
 +
Several game formats are documented in this Wiki, but not yet implemented in FFmpeg. Investigate via the [[:Category:Game Formats]] page.
 +
 
 +
=== Theora in Matroska ===
 +
 
 +
'''Completed'''
 +
 
 +
The current mkv demuxer supports Vorbis but not Theora.  Add support for Theora.  This requires parsing the matroska extradata to extract the three header packets, and correctly passing these to the Theora decoder.  You might want to read [http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2006-September/046115.html this thread] on the MPlayer list:
 +
 
 +
''David Conrad is working on this''
  
 
==See Also==
 
==See Also==
[[FFmpeg Wishlist]] for more tasks or ideas.
+
*[[FFmpeg Wishlist]] for more tasks or ideas.
 +
* The main [[FFmpeg Summer Of Code]] MultimediaWiki page
 +
*[http://bugzilla.mplayerhq.hu/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=FFmpeg&content= FFmpeg bugs] for ffmpeg bugs.
 +
 
  
[http://bugzilla.mplayerhq.hu/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=FFmpeg&content= FFmpeg bugs] for ffmpeg bugs.
+
[[Category:FFmpeg]]

Latest revision as of 09:43, 18 March 2008

Google is sponsoring their third annual Summer of Code for the summer of 2007. This entails sponsoring students to work on assorted open source projects as well as sponsoring mentors in those same projects. Everyone wins.

FFmpeg was a Summer of Code participant in the summer of 2006 (here is the corresponding Wiki page).

Mike Melanson (mike -at- multimedia.cx) is the administrator and main point of contact for matters relating to the FFmpeg Summer of Code.

How to apply

Before you can apply make sure you are qualified enough to apply. Last year 50% of the applicants weren't qualified for the task they applied for.

  • You have to know how to program in C fairly well.
  • We would like you to submit a patch that fixes a bug or adds a feature to FFmpeg. By doing that we will know that you are qualified for the task or not. On this page there is a list of Qualification Tasks that can be done. But you are free to submit anything you feel might be of value to FFmpeg. The qualification task can be done after you have filed you application (up until around April 7 as the list of accepted students is scheduled to be posted by Google on April 9).
  • Submit a good application through the formal Google Summer of Code process during the application timeframe (March 14-24, 2007).
  • You have to have >35 hours per week to put into the project.
  • You can't have another job at the same time as the SoC project.

Current Status

  • March 5-12, 2007: Application period for mentoring organizations.
  • March 6, 2007: Mike Melanson submitted FFmpeg mentoring application.
  • March 14, 2007: Google are now accepting student Applications.
  • March 15, 2007: FFmpeg got accepted as mentoring organization; now accepting student applications until March 24, 2007. Note that this is NOT the deadline to complete a qualification task; final student selections are to be made by April 9.
  • April 12, 2007: Google has allocated 8 project slots to FFmpeg.

Project Proposals

Overview

Qualifications for a good Summer of Code proposal:

  • discrete, well-defined, modular
  • comprised of a series of measurable sub-goals
  • based on open specs that are available free of charge
  • based on complete specs

An example of a good proposal is the implementation of a decoder or demuxer for an as yet unsupported multimedia format, or an encoder or muxer for a format that can already be demuxed/decoded by FFmpeg.

An example of a less desirable proposal is one that's not as measurable, such as refactoring APIs. Bad proposals tend to be ones that would require touching a lot of core code.

To re-iterate:

  • Localized/isolated code projects = good
  • Global code refactoring = bad

Note

THIS LIST IS NOT THE PROPOSALS WE ARE SUBMITTING!

Most of this list is just some ideas we are kicking around.

Projects with Mentors (official projects)

QCELP Decoder

Mentor: Benjamin Larsson

Matroska Muxer

Mentor: Aurelien Jacobs; Backup mentors: Steve Lhomme, Ronald S. Bultje

MPEG TS/DVB Muxer

  • Specification: ISO 13818-1

Mentor: Baptiste Coudurier

MXF Muxer

  • Specification: SMPTE 377M

Mentor: Baptiste Coudurier

RV40 Decoder

  • RealVideo 4 is steadily being reverse engineered and should be a reasonable candidate for re-implementation by the summer.

Mentor: Mike Melanson

PAFF decoding for H.264/AVC

Mentor: Loren Merritt

Dirac Encoder AND Decoder

Mentor: Luca Barbato

E-AC3 Decoder

Mentor: Justin Ruggles


Ideas for more projects to be determined

JPEG2000

Monkey's Audio Decoder (APE)

Completed

Finish LC-AAC decoder and implement HE-AAC decoder (LGPL)

GSM Decoder

i263 Decoder

VP6 Encoder

NUT Muxer

  • General improvements and enhancements

DPX/Cineon Encoder AND Decoder

OpenEXR Encoder AND Decoder

HD Photo Encoder AND Decoder

mp3PRO & aacPlus & MPEG Surround decoders

  • mp3PRO decoder. Note: mp3PRO decoding means MP3 + SBR (Spectral Band Replication) demuxing/decoding. (Standard MP3 decoders can decode mp3PRO encoded files/streams but without SBR you do not get the full quality. By adding a SBR decoder to FFmpeg and coupling it with the existing MP3 decoder you could playback mp3PRO at full quality? A SBR decoder could be shared with a aacPlus decoder as aacPlus also uses SBR).
  • aacPlus (a.k.a. AAC+) decoder. Note: aacPlus v1 decoding means HE-AAC + SBR (Spectral Band Replication) demuxing/decoding, and aacPlus v2 decoding means HE-AAC + SBR (Spectral Band Replication) + PS (Parametric Stereo) demuxing/decoding. (Standard HE-AAC decoders can decode aacPlus encoded files/streams but without SBR and PS you do not get the full quality. By adding a SBR decoder and a PS decoder to FFmpeg and coupling it with an existing HE-AAC decoder you could playback aacPlus at full quality? A SBR decoder could be shared with a mp3PRO decoder as mp3PRO also uses SBR).
  • MPEG Surround decoder/parser (for all audio but especially MP3/mp3PRO and AAC/aacPlus as those are in use today). MPEG Surround technology share similar characteristics with SBR (Spectral Band Replication) and PS (Parametric Stereo) demuxing/decoding, which mp3PRO and aacPlus decoders also use, so if SBR and PS decoders was added to FFmpeg then those could probebly share common code with a MPEG Surround decoder/parser. (DivX Inc. is one company that uses MPEG Surround technology to achieve 5.1 channel surround sound in smaller files).

Native DirectShow support

Option to build FFmpeg decoder/encoder/demuxer/muxer and post-processing filters for the DirectShow API for Windows by Microsoft, (the native DirectX 8/9 Direct3D overlay for video playback), so that FFmpeg has native support to be compiled for DirectShow and thus be used directly by players that use DirectShow.

http://en.wikipedia.org/wiki/Ffdshow

http://sourceforge.net/projects/drdivx/ (drffmpeg)

DirectX Video Acceleration (DXVA) 1.0 AND 2.0 for video decoding

Support Microsoft DirectX VA (DXVA) API nativly for GPU assisted decoding under Windows. Note! For this, native support for the above mentioned DirectShow API is needed first.

http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/MED134_WH06.ppt

Additional subtitle support

  • Create a common 'subtitles parser library' (and/or an API system for adding support for additional subtitle formats?) - a common sub-library to FFmpeg with all subtile decoders/demuxers/parsers gathered (similar to the libpostproc and libavutils). Call it "libsubs" (or "libsub", "libsubtitles" or whatever). Move FFmpeg's existing VobSub and DVBsub code there, so no matter if they are bitmap or text-based subs all existing and future subtile code is collected there. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional subtitles.
    • Maybe use MPlayer's recently added "libass" (SSA/ASS subtile reader) as a base for such a common library?
  • Support for advanced SSA/ASS rendering
    • Possible source are libass or the asa library
  • Support bold, italic, underline, RGB colors, size changes and font changes for a whole line or part of one line
  • Line 23 signal (a.k.a. "Wide-screen signal") detecting and use for DVD-Video (VobSub)
  • Support for the subtitles HTML tags
  • Capability of displaying subtitles with no video enabled (for example for audio-books)
  • Support for Karaoke subtitles (for kar and cdg, etc.)
  • Dual-subtitle-display (display two subtitles/languages at the same time, one at the bottom as normal plus one at the top of the screen)
  • Capability of moving the subtitles in the picture (freetype renderer)
  • Support more subtitle formats (text and bitmap-based):
    • Closed captioning (CC) subtile support - (Closed captions for the deaf and hard of hearing, also known as "Line 21 captioning", uses VobSub bitmaps)
      • xine have a SPU decoder for subpictures and Closed Captions software decoding
    • DirectVobSub (VSFilter) - standard VobSubs (DVD-Video subtitles) embedded in AVI containers
    • DivX Subtitles (XSUB) display/reader/decoder (Note: bitmap based subtitle, similar to VobSub)
    • SubRip (.srt) subtile support (Note: simple text-based based subtitle with timestamp)
    • Subviewer (.sub) subtile support (Note: simple text-based based subtitle with timestamp)
    • MicroDVD (.sub) subtile support (Note: simple text-based based subtitle with timestamp
    • Sami (.smi) subtile support (Note: simple text-based based subtitle with timestamp)
    • SubStation Alpha (.ssa+.ass) subtile support (Note: advanced text-based based subtitle with timestamps and XY location on screen)
    • RealText (.rt) subtile support
    • PowerDivx (.psb) subtile support
    • Universal Subtitle Format (.usf) subtile support
    • Structured Subtitle Format (.ssf) subtile support

libstream (a common 'stream client' library)

  • Create a common 'stream demuxer/parser library' for the client-side to receive input streams (and/or API for adding support for additional streaming formats?) - a LGPL'ed sub-library in FFmpeg with all stream demuxers/parsers gathered (similar to the libpostproc and libavutil). Call it "libstream" (or "stream" or whatever). Move FFmpeg's existing stream code there like HTTP and RTSP/RTP. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional streaming formats. All togther making it super easy for audio/video players using FFmpeg to add all-in-one streaming support to their player.
    • Maybe use either MPlayer's "stream" library structure, LIVE555, cURL, or probably the better libnemesi as a base for such a common library?
  • Add support for additional streaming protocols (on the client side) and improve/enhance support for existing protocols:
    • HTTP (Hypertext Transfer Protocol) client
      • plus a SSL (Secure Sockets Layer) client support for HTTPS
    • UDP (User Datagram Protocol) client
    • RTSP - Real-Time Streaming Protocol (RFC2326) client
    • RTP/RTCP - Real-Time Transport Protocol/RTP Control Protocol (RFC3550) client
    • RTP Profile for Audio and Video Conferences with Minimal Control (RFC3551) client
    • RealMedia RTSP/RDT (Real Time Streaming Protocol / Real Data Transport) client
    • SDP (Service Discovery Protocol) / SSDP (Simple Service Discovery Protocol) client
    • MMS (Microsoft Media Services) client
      • including the subprotocol mmsh (MMS over HTTP) and mmst (MMS over TCP)

A/V filter API (audio and video pre-process/post-process filters API system)

FFmpeg's already well-known libavcodec module has become the de facto standard library for video decoding and encoding in free software projects. Unfortunately, no similar standard library has surfaced for audio/video filtering and otherwise working with audio/video stream once it has been decoded. Various multimedia projects (such as MPlayer, Xine, GStreamer, VirtualDub, etc.) have implemented their own filter systems to various degrees of success. What is needed is a high quality audio and video filter API - efficient, flexible enough to meet all the requirements which have led various projects to invent their own filter system, and yet easy to use or develop new filters with. This proposal is to implement a high quality audio/video filter library for FFmpeg, where it can be easily used by other multimedia-related software projects.

Mentor: A'rpi (has expressed interest of possibly helping with implementing a filter API in FFmpeg, he also volunteering to help porting the MPlayer filters too if a such API becomes available http://lists.mplayerhq.hu/pipermail/mplayer-dev-eng/2007-April/051164.html)

Qualification tasks

Add a note if you choose to work on a Qualification task to avoid duplicate work.

Quicktime IMA ADPCM encoder

COMPLETED

TIFF encoder

Completed

Kamil Nowosad is working on this

Bartlomiej Wolowiec is working on this

Vivo demuxer

Alex Kalouguine is working on this task.

IFF/8SVX 8-bit audio demuxer

Tyler Williams is working on this task.

Port SGI image support to new API

Completed

FFmpeg changed image format APIs, but the SGI file format was never ported to the new API.

    • patch pending on ffmpeg-devel

Optimize some code

Do you think some code in FFmpeg could be made to run faster? We always love to get faster decoders or encoders. Note that this will require some ASM (assembly) skills and using timer code to benchmark and compare. Please make sure that any new code do not break compiling for other platforms.

Speedups via optimizations (like SIMD for 3DNow, MMX/MMX2, SSE/SSE2/SSE3 and AltiVec) are needed in FFmpeg's:

  • H.264 video decoder optimizations.
  • VC-1 video decoder optimizations.
  • AAC (LE-AAC and HE-AAC) audio decoder optimizations.
  • cat libavcodec/*.c | grep -i optimize for more files that need optimization.
    • Andrew Savchenko is working on this

BFI Playback System

Add FFmpeg playback capability for the BFI format. This entails writing a new file demuxer and simple video decoder.

DrV said he was working on this.

THP Playback System

Completed

Add FFmpeg playback capability for the THP format. This entails writing a new file demuxer and leveraging existing JPEG and ADPCM decoders to handle the video and audio data inside.

Marco Gerards is working on this (one patch applied, one patch pending)

Bethsoft VID

Completed

Add FFmpeg playback capability for the Bethsoft VID format (new demuxer, new video decoder).

patch applied. --Nicholas

Other Game Formats

Several game formats are documented in this Wiki, but not yet implemented in FFmpeg. Investigate via the Category:Game Formats page.

Theora in Matroska

Completed

The current mkv demuxer supports Vorbis but not Theora. Add support for Theora. This requires parsing the matroska extradata to extract the three header packets, and correctly passing these to the Theora decoder. You might want to read this thread on the MPlayer list:

David Conrad is working on this

See Also