Difference between revisions of "FFmpeg Wishlist"

From MultimediaWiki
Jump to navigation Jump to search
(See Also)
(Add back GSM decoder.)
Line 7: Line 7:
 
* [[VC-1]] (a.k.a. Microsoft Windows Media Video 3 or 9) decoder improvements/enhancements:
 
* [[VC-1]] (a.k.a. Microsoft Windows Media Video 3 or 9) decoder improvements/enhancements:
 
**Assembly optimizations (like SIMD for [http://en.wikipedia.org/wiki/3DNow%21 3DNow], [http://en.wikipedia.org/wiki/MMX MMX/MMX2], [http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions SSE/SSE2/SSE3] and [http://en.wikipedia.org/wiki/AltiVec AltiVec])
 
**Assembly optimizations (like SIMD for [http://en.wikipedia.org/wiki/3DNow%21 3DNow], [http://en.wikipedia.org/wiki/MMX MMX/MMX2], [http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions SSE/SSE2/SSE3] and [http://en.wikipedia.org/wiki/AltiVec AltiVec])
* LGPL'ed LC-AAC and HE-AAC ([[AAC|Advanced Audio Coding]]) decoder
+
* LGPL'ed LE-AAC and HE-AAC ([[AAC|Advanced Audio Coding]]) decoder
 
**Also add a aac parser so -acodec copy to mp4/mov will work
 
**Also add a aac parser so -acodec copy to mp4/mov will work
 
**Assembly optimizations (like SIMD for [http://en.wikipedia.org/wiki/3DNow%21 3DNow], [http://en.wikipedia.org/wiki/MMX MMX/MMX2], [http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions SSE/SSE2/SSE3] and [http://en.wikipedia.org/wiki/AltiVec AltiVec])
 
**Assembly optimizations (like SIMD for [http://en.wikipedia.org/wiki/3DNow%21 3DNow], [http://en.wikipedia.org/wiki/MMX MMX/MMX2], [http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions SSE/SSE2/SSE3] and [http://en.wikipedia.org/wiki/AltiVec AltiVec])
* [http://www.codingtechnologies.com/products/aacPlus.htm aacPlus] (a.k.a. AAC+) decoder Note: aacPlus v1 is HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR], aacPlus v2 is HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR] + [http://www.codingtechnologies.com/products/paraSter.htm PS].
+
* [http://www.codingtechnologies.com/products/aacPlus.htm aacPlus] (a.k.a. AAC+) decoder (Note: aacPlus v1 is HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR], aacPlus v2 is HE-AAC + [http://www.codingtechnologies.com/products/sbr.htm SBR] + [http://www.codingtechnologies.com/products/paraSter.htm PS]. Standard HE-AAC decoders can decode aacPlus encoded files/streams but without [http://www.codingtechnologies.com/products/sbr.htm SBR] and [http://www.codingtechnologies.com/products/paraSter.htm PS] you do not get the full quality. By adding a [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder and a [http://www.codingtechnologies.com/products/paraSter.htm PS] decoder to FFmpeg and coupling it with an existing HE-AAC decoder you could playback aacPlus at full quality? A [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder could be shared with a mp3PRO decoder as mp3PRO also uses [http://www.codingtechnologies.com/products/sbr.htm SBR]).
* [http://www.codingtechnologies.com/products/mp3pro.htm mp3PRO] decoder (Note: mp3PRO is MP3 + [http://www.codingtechnologies.com/products/sbr.htm SBR]. Standard MP3 decoders can decode mp3PRO encoded files/streams but without [http://www.codingtechnologies.com/products/sbr.htm SBR] you do not get the full quality.  
+
**[http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] decoder
* [http://www.mpegsurround.com MPEG Surround] decoder/parser (for all audio but especially MP3/mp3PRO and AAC/aacPlus as those are in use today).
+
**[http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] decoder
 +
* [http://www.codingtechnologies.com/products/mp3pro.htm mp3PRO] decoder (Note: mp3PRO is MP3 + [http://www.codingtechnologies.com/products/sbr.htm SBR]. Standard MP3 decoders can decode mp3PRO encoded files/streams but without [http://www.codingtechnologies.com/products/sbr.htm SBR] you do not get the full quality. By adding a [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder to FFmpeg and coupling it with the existing MP3 decoder you could playback mp3PRO at full quality? A [http://www.codingtechnologies.com/products/sbr.htm SBR] decoder could be shared with a aacPlus decoder as aacPlus also uses [http://www.codingtechnologies.com/products/sbr.htm SBR]).
 +
**[http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] decoder
 +
**[http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] decoder
 +
* [http://www.mpegsurround.com MPEG Surround] decoder/parser (for all audio but especially MP3/mp3PRO and AAC/aacPlus as those are in use today). [http://www.codingtechnologies.com/products/mpgsrnd.htm MPEG Surround technology] share similar characteristics with [http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] and [http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)], which [http://www.codingtechnologies.com/products/mp3pro.htm mp3PRO] and [http://www.codingtechnologies.com/products/aacPlus.htm aacPlus] also use, so if SBR and PS decoders was added to FFmpeg then those could probebly share common code with a [http://www.mpegsurround.com MPEG Surround] decoder/parser.
 +
**[http://www.codingtechnologies.com/products/sbr.htm SBR (Spectral Band Replication)] decoder
 +
**[http://www.codingtechnologies.com/products/paraSter.htm PS (Parametric Stereo)] decoder
 
* ALAC decoder improvements/enhancements:
 
* ALAC decoder improvements/enhancements:
 
**Clean up the existing alac decoder code
 
**Clean up the existing alac decoder code
Line 26: Line 32:
 
**[http://jmac.sourceforge.net/]LGPLed Java implementation
 
**[http://jmac.sourceforge.net/]LGPLed Java implementation
 
* JPEG2000 decoder
 
* JPEG2000 decoder
 +
* [[GSM]] decoder
 
* QCELP decoder [http://www.3gpp2.org/Public_html/specs/alltsgscfm.cfm spec] is c.s0020 and source is c.r0020
 
* QCELP decoder [http://www.3gpp2.org/Public_html/specs/alltsgscfm.cfm spec] is c.s0020 and source is c.r0020
 
* AMV decoder, http://scrub50187.com/ has the creator. wikipedia has [http://en.wikipedia.org/wiki/AMV_video_format articles] about the format also.
 
* AMV decoder, http://scrub50187.com/ has the creator. wikipedia has [http://en.wikipedia.org/wiki/AMV_video_format articles] about the format also.
Line 66: Line 73:
 
http://sourceforge.net/tracker/index.php?func=detail&aid=1097094&group_id=53761&atid=471491 also look at http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/25207/focus=25224 and http://www.maxconsole.net/?mode=news&newsid=411 for hints/tips
 
http://sourceforge.net/tracker/index.php?func=detail&aid=1097094&group_id=53761&atid=471491 also look at http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/25207/focus=25224 and http://www.maxconsole.net/?mode=news&newsid=411 for hints/tips
 
*  AMV demuxer, http://scrub50187.com/ has the creator. wikipedia has [http://en.wikipedia.org/wiki/AMV_video_format articles] about the format also.
 
*  AMV demuxer, http://scrub50187.com/ has the creator. wikipedia has [http://en.wikipedia.org/wiki/AMV_video_format articles] about the format also.
* [[RatDVD]] demuxer (for [[Xvo]] files (mpeg variant?)
+
* ratdvd demuxer (for XVO files (mpeg variant?)
  
 
== Muxers ==
 
== Muxers ==
Line 208: Line 215:
 
*[[FFmpeg Summer Of Code|FFmpeg's Google SoC (Summer of Code) 2006 list of tasks]] for more suggestions/requests (ideas for developers).
 
*[[FFmpeg Summer Of Code|FFmpeg's Google SoC (Summer of Code) 2006 list of tasks]] for more suggestions/requests (ideas for developers).
 
*[http://bugzilla.mplayerhq.hu/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=FFmpeg&content= FFmpeg bugs] for bugs in FFmpegs (codecs) that you can help fix or add addition information/samples to.
 
*[http://bugzilla.mplayerhq.hu/buglist.cgi?query_format=specific&order=relevance+desc&bug_status=__open__&product=FFmpeg&content= FFmpeg bugs] for bugs in FFmpegs (codecs) that you can help fix or add addition information/samples to.
*[[:Category:Formats_missing_in_FFmpeg]] for formats not implemented in ffmpeg yet
 

Revision as of 04:43, 21 April 2007

Temporary FFmpeg wish/todo list:

Decoders

Encoders

  • Snow
    • multiple reference frames improvements
      • decide which frames to keep (e.g. long-term refs)
      • some changes to the mv prediction code
    • non-translational motion-compensation
      • estimate non translational parameters per block by using surrounding motion vectors
      • add a ac coded bit per block to switch between translational and non-translational MC
      • borrow the non translational MC code from libmpcodecs/vf_perspective.c
      • some changes to the encoder to decide between translational and non t.
    • Trellis quantization (select quantized coefficient so as to minimize the rate distrortion
    • 4x4 sized block support (we have 16x16 and 8x8 currently)
    • 1/8 pel motion compensation / estimation support (pretty much just encoder changes needed which in case of the iterative me should be trivial)
    • improve the intra color decision

Demuxers

  • iff demuxer (with anim and sound decoding)
    • xine has a demuxer/decoder for iff
  • g723.1 / rtp demuxer
  • g729 / rtp demuxer
  • VIVO demuxer, look at the mplayer vivo demuxer for reference
  • XMV / FMV (Xbox Media Video) demuxer (from Microsoft and based on WMV8)
    • Open source and legal demuxer/decoder but copyrighted source/specification:

http://sourceforge.net/tracker/index.php?func=detail&aid=1097094&group_id=53761&atid=471491 also look at http://thread.gmane.org/gmane.comp.video.ffmpeg.devel/25207/focus=25224 and http://www.maxconsole.net/?mode=news&newsid=411 for hints/tips

Muxers

  • DVB (MPEG-TS) muxer inside DVB containers
    • MPEG-1/2 video-streams inside DVB containers
    • MPEG-4 ASP video-streams inside DVB containers
    • MPEG-4 AVC (H.264) video-streams inside DVB containers
    • AC3 audio-streams inside DVB containers
      • Mutiple AC3 audio-streams inside DVB containers
    • MP3 audio-streams inside DVB containers
      • Mutiple MP3 audio-streams inside DVB containers
  • NSV muxer
  • NSA muxer

Microsoft DirectShow and DirectX and MediaFoundation

Native DirectShow support

Option to build FFmpeg decoder/encoder/demuxer/muxer and post-processing filters for the DirectShow API for Windows by Microsoft, (the native DirectX 8/9 Direct3D overlay for video playback), so that FFmpeg has native support to be compiled for DirectShow and thus be used directly by players that use DirectShow.

Native MediaFoundation support

  • Microsoft Media Foundation API usage for optimized digital media playback on Microsoft Windows Vista™
  • Multimedia Class Scheduler Service (MMCSS) class
  • Enhanced Video Renderer (EVR) class
  • Streaming Audio Renderer (SAR) class

DirectX Video Acceleration (DXVA)

DirectX Video Acceleration (DXVA) 1.0 AND 2.0 support, (for GPU accelerated video decoding under Windows).
Note! Native DirectShow support in FFmpeg is before DirectX VA (DXVA) video decoding support can be added!

http://download.microsoft.com/download/5/b/9/5b97017b-e28a-4bae-ba48-174cf47d23cd/MED134_WH06.ppt

Features


Subtitles

  • Create a common 'subtitles parser library' (and/or an API system for adding support for additional subtitle formats?) - a common sub-library to FFmpeg with all subtile decoders/demuxers/parsers gathered (similar to the libpostproc and libavutils). Call it "libsubs" (or "libsub", "libsubtitles" or whatever). Move FFmpeg's existing VobSub and DVBsub code there, so no matter if they are bitmap or text-based subs all existing and future subtile code is collected there. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional subtitles.
    • Maybe use MPlayer's recently added "libass" (SSA/ASS subtile reader) as a base for such a common library?
  • Support for advanced SSA/ASS rendering
    • Possible source are libass or the asa library
  • Support bold, italic, underline, RGB colors, size changes and font changes for a whole line or part of one line
  • Line 23 signal (a.k.a. "Wide-screen signal") detecting and use for DVD-Video (VobSub)
  • Support for the subtitles HTML tags
  • Capability of displaying subtitles with no video enabled (for example for audio-books)
  • Support for Karaoke subtitles (for kar and cdg, etc.)
  • Dual-subtitle-display (display two subtitles/languages at the same time, one at the bottom as normal plus one at the top of the screen)
  • Capability of moving the subtitles in the picture (freetype renderer)
  • Support more subtitle formats (text and bitmap-based):
    • Closed captioning (CC) subtile support - (Closed captions for the deaf and hard of hearing, also known as "Line 21 captioning", uses VobSub bitmaps)
      • xine have a SPU decoder for subpictures and Closed Captions software decoding
    • DirectVobSub (VSFilter) - standard VobSubs (DVD-Video subtitles) embedded in AVI containers
    • DivX Subtitles (XSUB) display/reader/decoder (Note: bitmap based subtitle, similar to VobSub)
    • SubRip (.srt) subtile support (Note: simple text-based based subtitle with timestamp)
    • Subviewer (.sub) subtile support (Note: simple text-based based subtitle with timestamp)
    • MicroDVD (.sub) subtile support (Note: simple text-based based subtitle with timestamp
    • Sami (.smi) subtile support (Note: simple text-based based subtitle with timestamp)
    • SubStation Alpha (.ssa+.ass) subtile support (Note: advanced text-based based subtitle with timestamps and XY location on screen)
    • RealText (.rt) subtile support
    • PowerDivx (.psb) subtile support
    • Universal Subtitle Format (.usf) subtile support
    • Structured Subtitle Format (.ssf) subtile support


Misc

Streaming Media Network Protocols

Streaming Media Network Protocols (client and server-side) improvements/enhancements and related ideas for new features/functions.

  • Create a common 'stream demuxer/parser library' for the client-side (and/or API for adding support for additional streaming formats?) - a LGPL'ed sub-library in FFmpeg with all stream demuxers/parsers gathered (similar to the libpostproc and libavutil). Call it "libstream" (or "stream" or whatever). Move FFmpeg's existing stream code there like HTTP and RTSP/RTP. This will help reduce future code replication by sharing common code, thus making it easier to add support for additional streaming formats. All togther making it super easy for audio/video players using FFmpeg to add all-in-one streaming support to their player.
    • Maybe use either MPlayer's "stream" library structure, LIVE555, or probebly the better libnms (from NeMeSi) as a base for such a common library?
  • Add support for additional streaming protocols (on the client side) and improve/enhance support for existing protocols:
    • HTTP (Hypertext Transfer Protocol) client
    • UDP (User Datagram Protocol) client
    • RTSP - Real-Time Streaming Protocol (RFC2326) client
    • RTP/RTCP - Real-Time Transport Protocol/RTP Control Protocol (RFC3550) client
    • RTP Profile for Audio and Video Conferences with Minimal Control (RFC3551) client
    • RealMedia RTSP/RDT (Real Time Streaming Protocol / Real Data Transport) client
    • SDP (Service Discovery Protocol) / SSDP (Simple Service Discovery Protocol) client
    • MMS (Microsoft Media Services) client
  • FFServer updating (and improving)
    • FFServer code hasn't been update for quite a while.
    • Streaming to clients like WMP 9, 10 and 11 is broken.
    • MMS server streaming support in FFServer, (especially for Linux).
      • Note that al3x has gotten something working with ffserver, you might want to ask him what needs to be done as well :) --Compn 14:22, 19 March 2007 (EDT)
      • You should also take a look at the FENG (RTSP Streaming Server) code, and [NetEmbryo (Embedded Open Media Streaming Library)] --Gamester17 11:20, 29 March 2007 (GMT+1)

Audio and video (pre-process/post-process) filters

  • Adopt MPlayer's A/V filter system or create new one.
  • Create (or port) additional pre-process and post-process video filters:
  • Create (or port) additional pre-process and post-process audio filters:

See Also