RE process

From MultimediaWiki
Revision as of 01:36, 10 July 2009 by Merbanan (talk | contribs) (Initial how to RE codecs article)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

The not so simple introduction to RE'ing of multimedia codecs.

Information gathering

Try to collect as much public knowledge as possible.

  • The first thing to do is to collect/create sample files. Without samples files there is nothing to test/verify a reimplementation.
  • The second thing is to collect different decoder for the codec. Sometimes debug symbols are available in one binary but not the other.
  • Read the product/codec whitepapers, they are mostly useless but can give a hint of the techniques used in the codec.

Get it running

If you get a binary running it is much more easy to figure out how it works exactly. Getting it running in a controlled environment is even better.

  • Mplayer has a dll loader which makes it very easy to load acm and dmo codecs.
  • Use the technique described here to load a binary under Linux:
  • Run it on the original platform in some form.

Picking it apart

Load up the codec in a dissassembler.

Hijack code flow

If the codec is running in a deterministic way it is possible to runtime patch the code to replace call addresses. The following macros can be used:

#define insert_native_addr(address, replacement) \
{ \
    unsigned int *padd = (unsigned int *)address;       \
    *padd = (unsigned int*)replacement; \

#define insert_native_call(address, replacement) \
{ \
      *((uint8_t*)address) = 0xe8; \
      *((int32_t*)(address+1)) = (int) replacement - address - 5; \

When you redirect the functions make sure you declare your replacement function with the right call convention.


Most codecs work in a init, decode, close fashion. The init step allocated a codec private context that is then passed to the decode function. The private context is used by the codec to store it's internal state needed for decoding. The close function just cleans up the context. First start with reverse engineering the init and use a malloc wrapper to figure out the structure of the private context. Then go for the main decode function. It usually takes the private context as arguments and the indata buffer and size. From here the progress is open ended, try to tackle functions at the end of the decode call tree and at the start. That way knowledge about parameters can propagate through the code.


When all the code is understood replace the code with lavc equivalents and re implement the stuff that is missing.