MultimediaWiki - User contributions [en]

Xmf

2010-07-28T16:34:38Z

Xperroni:

#Redirect [[Extensible Music Format (XMF)]]

Extensible Music Format (XMF)

2010-07-28T16:04:07Z

Xperroni: /* Simple Example Parser */

The '''eXtensible Music Format (XMF)''' is a tree-structured container format developed by the [http://www.midi.org/ MIDI Manufacturer's Association] from 2001 to 2004. Historically it was promoted as a way to package together MIDI tunes and DLS instrument sounds for distribution, ensuring a consistent playback experience across environments<ref name="WhatXMFIsFor">[http://www.beatnik.com/pdf_files/xmf_datasheet.pdf eXtensible Music Format]</ref>. It saw ample adoption on early "polyphonic" mobile phones, but as handsets achieved support for MP3 and other digital audio formats, the consumer market for instrument-based formats all but disappeared. Despite that it remains widely supported: even recent mobile architectures such as Google's Android include XMF playing technology<ref name="XMFInAndroid">[http://www.sonivoxrocks.com/google.html SONiVOX Contributes audioINSIDE Technology to Google Open Handset Alliance and Android Platform]</ref>.

== File Structure ==

Internally, XMF files are composed of a ''header chunk'' and a collection of ''node chunks'' ordered in [http://en.wikipedia.org/wiki/Tree_traversal#Depth-first_Traversal preorder]. Nodes can be either of type ''folder'' (in which case they will have at least one child node) or ''contents'' (in which case they encapsulate a data stream, such as MIDI or DLS contents). Most data in XMF nodes is encoded in [http://en.wikipedia.org/wiki/Variable-length_quantity variable-length quantity (VLQ)], a universal code that uses an arbitrary number of binary octets (eight-bit bytes) to represent an arbitrarily large integer. The Python code snippet below illustrates how a VLQ-encoded integer can be converted back to its native representation:

def vlq2int(data):
r'''Read one VLQ-encoded integer value from an input data stream.
'''
value = 0
while True:
# Read one numeric octet from the input stream
octet = ord(data.read(1))

# Appends the octet to the output value, discarding the continuation bit
value = (value << 7) + (octet & 0x7F)

# If the continuation bit is zero, this was the last octet
if (octet & 0x80) == 0:
break

return value

The diagram below illustrates the structure of the XMF format, displaying an example file's internals both in contiguous and tree forms.

<code>
{| class="wikitable" border="0" width="50%"
|-
|align="center" colspan="2"|ut_60-dls.xmf
|-
|
{| class="wikitable" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-
|Header Chunk
|-
|Root Node
|-
|Folder Node 1
|-
|Contents Node 1
|-
|ut.mid
|-
|Folder Node 2
|-
|Contents Node 2
|-
|mobile60.dls
|}
|[[Image:Ut_60-dls.xmf.png]]
|}
</code>

The structure above is typical of several XMF authoring tools, including Beatnik's Mobile Sound Builder and Nokia's [http://www.forum.nokia.com/info/sw.nokia.com/id/9395846c-262c-47e0-9420-3cd5dec627af/Nokia_Audio_Suite.html Audio Suite] plugin for [http://www.steinberg.net/en/support/downloads/cubase_sx_3.html Steinberg Cubase SX]. It's far from optimal, however, as it unnecessarily nests the media contents inside exclusive folder nodes, when they could very well be put directly under the root node (and in fact some mobile players are known to choke on those extra nodes).

=== Header Chunk ===

The table below relates the fields of a XMF header chunk, in order of occurrence.

{| class="wikitable" width="70%" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-align="left"
!width="20%"| Field
!width="7%"| Type
! Value
|-
|Format ID
|<code>byte[4]</code>
|<code>"XMF_" (0x584d465f)</code>
|-
|Format version
|<code>byte[4]</code>
|One of <code>"1.00" (0x312E3030)</code>, <code>"1.01" (0x312E3031)</code> or <code>"2.00" (0x322E3030)</code>
|-
|File Type ID
|<code>int32</code>
|Magic number indicating the XMF file type1
|-
|File Type Revision ID
|<code>int32</code>
|Magic number indicating the XMF file type's revision1
|-
|File size
|<code>VLQ</code>
|The file size in bytes
|-
|MTT length
|<code>VLQ</code>
|Length of the Metadata Types Table (MTT), in bytes
|-
|MTT Contents
|N/A
|Binary-encoded contents of the MTT
|-
|Tree Start Offset (TFO)
|<code>VLQ</code>
|The root node's absolute offset within the XMF file
|}

1. Included as of version 2.0 of the spec. Not present in files encoded to earlier versions.

=== Node Chunk ===

The table below relates the fields of a XMF node chunk, in order of occurrence.

{| class="wikitable" width="70%" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-align="left"
!width="20%"| Field
!width="7%"| Type
! Value
|-
|Node length
|<code>VLQ</code>
|The node's length in bytes, including the length of all child nodes (for folder nodes) or encapsulated contents (for content nodes)
|-
|Item count
|<code>VLQ</code>
|Children node count, must be zero for content nodes
|-
|Node header length
|<code>VLQ</code>
|Length of this node's header, in bytes
|-
|Node metadata length
|<code>VLQ</code>
|Length of the node's metadata, in bytes
|-
|Node metadata contents
|N/A
|Binary-encoded node metadata contents
|-
|Reference type
|<code>VLQ</code>
|Indicates whether the following contents constitute an actual data buffer (<code>0x1</code>) or an offset to somewhere else within the file (<code>0x2</code>)
|-
|Contents
|N/A
|Either a data buffer or an offset value within the file, depending on the reference type
|}

=== Simple Example Parser ===

As way of illustration, the Python script below decodes an input XMF file, printing various statistics about its contents:

from os import SEEK_CUR, SEEK_SET

class xmfnodemetadata(object):
def __str__(self):
return '\nMetadata Size : ' + str(self.size) + ' bytes'

class xmfnodeheader(object):
def __init__(self):
self.metadata = xmfnodemetadata()

def __str__(self):
return \
'\nHeader Size : ' + str(self.size) + ' bytes' + \
str(self.metadata)

class xmfnode(object):
def __init__(self):
self.header = xmfnodeheader()
self.children = []

def __str__(self):
return \
'\nNode Size : ' + str(self.size) + ' bytes' + \
str(self.header) + \
'\nReference Type : ' + hex(self.reftype) + \
'\nContents : ' + str(self.contents) + \
'\nChildren Count : ' + str(len(self.children)) + \
'\n'.join([str(child).replace('\n', '\n ') for child in self.children])

class mtt(object):
def __str__(self):
return '\nMTT Length : ' + str(self.size) + ' bytes'

class xmf10(object):
def __init__(self):
self.mtt = mtt()

def __str__(self):
return \
'\nFormat ID : ' + self.formatid + \
'\nFormat Version : ' + self.version + \
'\nFile Size : ' + str(self.size) + ' bytes' + \
str(self.mtt) + \
'\n' + \
'\nContent Nodes:' + \
'\n' + \
str(self.root)

def vlq2int(data):
r'''Read one VLQ-encoded integer value from an input data stream.
'''
value = 0
while True:
# Read one numeric octet from the input stream
octet = ord(data.read(1))

# Appends the octet to the output value, discarding the continuation bit
value = (value << 7) + (octet & 0x7F)

# If the continuation bit is zero, this was the last octet
if (octet & 0x80) == 0:
break

return value

def parsenode(data):
offset = data.tell()

node = xmfnode()
node.size = vlq2int(data)
childcount = vlq2int(data)
node.header.size = vlq2int(data)
node.header.metadata.size = vlq2int(data)

data.seek(offset + node.header.size, SEEK_SET)
node.reftype = vlq2int(data)
node.contents = '(folder)'
node.children = []

if childcount > 0:
node.children = [parsenode(data) for i in range(0, childcount)]
elif node.reftype == 1:
node.contents = data.read(4)
else:
raise Exception('Invalid type ' + hex(node.reftype))

data.seek(offset + node.size, SEEK_SET)

return node

def parse(path):
with open(path, 'rb') as data:
xmf = xmf10()

xmf.formatid = data.read(4)
xmf.version = data.read(4)
if xmf.version != '1.00':
raise Exception('Invalid version \"' + xmf.version + '\"')

xmf.size = vlq2int(data)
xmf.mtt.size = vlq2int(data)
data.seek(xmf.mtt.size, SEEK_CUR)

tfo = vlq2int(data)
data.seek(tfo, SEEK_SET)
xmf.root = parsenode(data)

return path + '\n' + str(xmf)

if __name__ == '__main__':
from sys import argv

parsed = parse(argv[1])
print(parsed)

== Tools ==

The MMA provides a [http://www.midi.org/techspecs/xmf/xmf_products.php list of authoring tools for XMF]; however, virtually every one of them is currently unsupported, their makers having either moved past the format or gone with it. Some can still be found around the Internet, however:

# The [http://www.beatnik.com/pdf_files/msb_datasheet.pdf Mobile Sound Builder] was [http://www.beatnik.com/ Beatnik]'s solution for authoring and playing XMF files. It's hard to tell from their half-functioning company website whether it's still supported, but binaries for version 1.2 often turn up in Google searches;
# [http://www.forum.nokia.com/info/sw.nokia.com/id/9395846c-262c-47e0-9420-3cd5dec627af/Nokia_Audio_Suite.html Nokia Audio Suite] is a plug-in for [http://www.steinberg.net/en/support/downloads/cubase_sx_3.html Steinberg Cubase SX] which adds the ability to author XMF files. It's a free download, however license keys for version 2.0 are issued only to users outside of Japan and US, while keys for version 1.1 aren't even available anymore – so effectively it can't be used by neither Japanese nor American developers;
# [http://www.faith-inc.com/ Faith] reportedly<ref name="FaithmXMFTool">[http://www.faith-inc.com/ir/pdf/070612rm_faithinsound4.pdf Mobile Game Audio Gets Boost from Faith West’s mXMFTool and New Royalty-Free Game Sounds Collection]</ref> once provided the mXMFTool as a free download, however it was removed when their San Francisco office (Faith West) was closed. Currently it's nowhere to be found, though it seems you can talk them into mailing you a copy<ref name="FaithmXMFToolGet">[http://www.j2meforums.com/forum/index.php?PHPSESSID=a41099ffd022ae3b4afb8e2838c0a398&topic=22589.msg104916#msg104916 Mobile eXtensible Music File]</ref>;
# The [http://www.pac.fi/ Professional Audio Company] apparently provided some sort of XMF-related tool in the past, but no further information seems to be available. The company relinquished their MMA Manufacturer ID<ref name="PACMMAID">[http://www.midi.org/techspecs/manid.php MANUFACTURER'S ID NUMBERS]</ref>, so it's very unlikely they still work with it;
# [http://www.crimsontech.jp/eng/home.html Crimson Technology] still provides a demo version for their [http://www.crimsontech.jp/eng/dls/ DLS tools], however it's unclear what XMF features (if any) they enable.

== References ==

<references />

== External links ==

# [http://www.midi.org/techspecs/xmf/ XMF - eXtensible Music Format]

Extensible Music Format (XMF)

2010-07-28T16:02:05Z

Xperroni: /* File Structure */

2010-07-26T14:41:00Z

Xperroni:

The '''eXtensible Music Format (XMF)''' is a tree-structured container format developed by the [http://www.midi.org/ MIDI Manufacturer's Association] from 2001 to 2004. Historically it was promoted as a way to package together MIDI tunes and DLS instrument sounds for distribution, ensuring a consistent playback experience across environments<ref name="WhatXMFIsFor">{{cite web |url=http://www.beatnik.com/pdf_files/xmf_datasheet.pdf |title=eXtensible Music Format |date=2003-01-30 |accessdate=2010-07-23}}</ref>. It saw ample adoption on early "polyphonic" mobile phones, but as handsets achieved support for MP3 and other digital audio formats, the consumer market for instrument-based formats all but disapeared. Despite that it remains widely supported: even recent mobile architectures such as Google's Android include XMF playing technology<ref name="XMFInAndroid">{{cite web |url=http://www.sonivoxrocks.com/google.html |title=SONiVOX Contributes audioINSIDE Technology to Google Open Handset Alliance and Android Platform |date=2007-11-01 |accessdate=2010-07-23}}</ref>.

== File Structure ==

Internally, XMF files are composed of a ''header chunk'' and a collection of ''node chunks'' ordered in [http://en.wikipedia.org/wiki/Tree_traversal#Depth-first_Traversal preorder]. Nodes can be either of type ''folder'' (in which case they will have at least one child node) or ''contents'' (in which case they encapsulate a data stream, such as MIDI or DLS contents). Most data in XMF nodes is encoded in [http://en.wikipedia.org/wiki/Variable-length_quantity variable-length quantity (VLQ)], a universal code that uses an arbitrary number of binary octets (eight-bit bytes) to represent an arbitrarily large integer. The Python code snippet below illustrates how a VLQ-encoded integer can be converted back to its native representation:

def vlq2int(data):
r'''Read one VLQ-encoded integer value from an input data stream.
'''
value = 0
while True:
# Read one numeric octet from the input stream
octet = ord(data.read(1))

# Appends the octet to the output value, discarding the continuation bit
value = (value << 7) + (octet & 0x7F)

# If the continuation bit is zero, this was the last octet
if (octet & 0x80) == 0:
break

return value

The diagram below illustrates the structure of the XMF format, displaying an example file's internals both in contiguous and tree forms.

<code>
{| class="wikitable" border="0" width="50%"
|-
|align="center" colspan="2"|ut_60-dls.xmf
|-
|
{| class="wikitable" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-
|Header Chunk
|-
|Root Node
|-
|Folder Node 1
|-
|Contents Node 1
|-
|ut.mid
|-
|Folder Node 2
|-
|Contents Node 2
|-
|mobile60.dls
|}
|[[Image:Ut_60-dls.xmf.png]]
|}
</code>

The structure above is typical of several XMF authoring tools, including Beatnik's Mobile Sound Builder and Nokia's [http://www.forum.nokia.com/info/sw.nokia.com/id/9395846c-262c-47e0-9420-3cd5dec627af/Nokia_Audio_Suite.html Audio Suite] plugin for [http://www.steinberg.net/en/support/downloads/cubase_sx_3.html Steinberg Cubase SX]. It's far from optimal, however, as it unnecessarily nests the media contents inside exclusive folder nodes, when they could very well be put directly under the root node (and in fact some mobile players are known to choke on those extra nodes).

=== Header Chunk ===

The table below relates the fields of a XMF header chunk, in order of occurrence.

{| class="wikitable" width="70%" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-align="left"
!width="20%"| Field
!width="7%"| Type
! Value
|-
|Format ID
|<code>byte[4]</code>
|<code>"XMF_" (0x584d465f)</code>
|-
|Format version
|<code>byte[4]</code>
|One of <code>"1.00" (0x312E3030)</code>, <code>"1.01" (0x312E3031)</code> or <code>"2.00" (0x322E3030)</code>
|-
|File Type ID
|<code>int32</code>
|Magic number indicating the XMF file type1
|-
|File Type Revision ID
|<code>int32</code>
|Magic number indicating the XMF file type's revision1
|-
|File size
|<code>VLQ</code>
|The file size in bytes
|-
|MTT length
|<code>VLQ</code>
|Length of the Metadata Types Table (MTT), in bytes
|-
|MTT Contents
|N/A
|Binary-encoded contents of the MTT
|-
|Tree Start Offset (TFO)
|<code>VLQ</code>
|The root node's absolute offset within the XMF file
|}

1. Included as of version 2.0 of the spec. Not present in files encoded to earlier versions.

=== Node Chunk ===

The table below relates the fields of a XMF node chunk, in order of occurrence.

{| class="wikitable" width="70%" border="1" style="margin: 1em 1em 1em 0; background: #f9f9f9; border: 1px #aaa solid; border-collapse: collapse"
|-align="left"
!width="20%"| Field
!width="7%"| Type
! Value
|-
|Node length
|<code>VLQ</code>
|The node's length in bytes, including the length of all child nodes (for folder nodes) or encapsulated contents (for content nodes)
|-
|Item count
|<code>VLQ</code>
|Children node count, must be zero for content nodes
|-
|Node header length
|<code>VLQ</code>
|Length of this node's header, in bytes
|-
|Node metadata length
|<code>VLQ</code>
|Length of the node's metadata, in bytes
|-
|Node metadata contents
|N/A
|Binary-encoded node metadata contents
|-
|Reference type
|<code>VLQ</code>
|Indicates whether the following contents constitute an actual data buffer (<code>0x1</code>) or an offset to somewhere else within the file (<code>0x2</code>)
|-
|Contents
|<code>N/A</code>
|Either a data buffer or an offset value within the file, depending on the reference type
|}

=== Simple Example Parser ===

As way of illustration, the Python script below decodes an input XMF file, printing various statistics about its contents:

from os import SEEK_CUR, SEEK_SET

class xmfnodemetadata(object):
def __str__(self):
return '\nMetadata Size : ' + str(self.size) + ' bytes'

class xmfnodeheader(object):
def __init__(self):
self.metadata = xmfnodemetadata()

def __str__(self):
return \
'\nHeader Size : ' + str(self.size) + ' bytes' + \
str(self.metadata)

class xmfnode(object):
def __init__(self):
self.header = xmfnodeheader()
self.children = []

def __str__(self):
return \
'\nNode Size : ' + str(self.size) + ' bytes' + \
str(self.header) + \
'\nReference Type : ' + hex(self.reftype) + \
'\nContents : ' + str(self.contents) + \
'\nChildren Count : ' + str(len(self.children)) + \
'\n'.join([str(child).replace('\n', '\n ') for child in self.children])

class mtt(object):
def __str__(self):
return '\nMTT Length : ' + str(self.size) + ' bytes'

class xmf10(object):
def __init__(self):
self.mtt = mtt()

def __str__(self):
return \
'\nFormat ID : ' + self.formatid + \
'\nFormat Version : ' + self.version + \
'\nFile Size : ' + str(self.size) + ' bytes' + \
str(self.mtt) + \
'\n' + \
'\nContent Nodes:' + \
'\n' + \
str(self.root)

def vlq2int(data):
r'''Read one VLQ-encoded integer value from an input data stream.
'''
value = 0
while True:
# Read one numeric octet from the input stream
octet = ord(data.read(1))

# Appends the octet to the output value, discarding the continuation bit
value = (value << 7) + (octet & 0x7F)

# If the continuation bit is zero, this was the last octet
if (octet & 0x80) == 0:
break

return value

def parsenode(data):
offset = data.tell()

node = xmfnode()
node.size = vlq2int(data)
childcount = vlq2int(data)
node.header.size = vlq2int(data)
node.header.metadata.size = vlq2int(data)

data.seek(offset + node.header.size, SEEK_SET)
node.reftype = vlq2int(data)
node.contents = '(folder)'
node.children = []

if childcount > 0:
node.children = [parsenode(data) for i in range(0, childcount)]
elif node.reftype == 1:
node.contents = data.read(4)
else:
raise Exception('Invalid type ' + hex(node.reftype))

data.seek(offset + node.size, SEEK_SET)

return node

def parse(path):
with open(path, 'rb') as data:
xmf = xmf10()

xmf.formatid = data.read(4)
xmf.version = data.read(4)
if xmf.version != '1.00':
raise Exception('Invalid version \"' + xmf.version + '\"')

xmf.size = vlq2int(data)
xmf.mtt.size = vlq2int(data)
data.seek(xmf.mtt.size, SEEK_CUR)

tfo = vlq2int(data)
data.seek(tfo, SEEK_SET)
xmf.root = parsenode(data)

return path + '\n' + str(xmf)

if __name__ == '__main__':
from sys import argv

parsed = parse(argv[1])
print(parsed)

== Tools ==

The MMA provides a [http://www.midi.org/techspecs/xmf/xmf_products.php list of authoring tools for XMF]; however, virtually every one of them is currently unsupported, their makers having either moved past the format or gone with it. Some can still be found around the Internet, however:

# The [http://www.beatnik.com/pdf_files/msb_datasheet.pdf Mobile Sound Builder] was [http://www.beatnik.com/ Beatnik]'s solution for authoring and playing XMF files. It's hard to tell from their half-functioning company website whether it's still supported, but binaries for version 1.2 often turn up in Google searches;
# [http://www.forum.nokia.com/info/sw.nokia.com/id/9395846c-262c-47e0-9420-3cd5dec627af/Nokia_Audio_Suite.html Nokia Audio Suite] is a plug-in for [http://www.steinberg.net/en/support/downloads/cubase_sx_3.html Steinberg Cubase SX] which adds the ability to author XMF files. It's a free download, however license keys for version 2.0 are issued only to users outside of Japan and US, while keys for version 1.1 aren't even available anymore – so effectively it can't be used by neither Japanese nor American developers;
# [http://www.faith-inc.com/ Faith] reportedly<ref name="FaithmXMFTool">{{cite web |url=http://www.faith-inc.com/ir/pdf/070612rm_faithinsound4.pdf |title=Mobile Game Audio Gets Boost from Faith West’s mXMFTool and New Royalty-Free Game Sounds Collection |date=2007-06-12 |accessdate=2010-07-26}}</ref> once provided the mXMFTool as a free download, however it was removed when their San Francisco office (Faith West) was closed. Currently it's nowhere to be found, though it seems you can talk them into mailing you a copy<ref name="FaithmXMFToolGet">{{cite web |url=http://www.j2meforums.com/forum/index.php?PHPSESSID=a41099ffd022ae3b4afb8e2838c0a398&topic=22589.msg104916#msg104916 |title=Mobile eXtensible Music File |date=2010-01-09 |accessdate=2010-07-26}}</ref>;
# The [http://www.pac.fi/ Professional Audio Company] apparently provided some sort of XMF-related tool in the past, but no further information seems to be available. The company relinquished their MMA Manufacturer ID<ref name="PACMMAID">{{cite web |url=http://www.midi.org/techspecs/manid.php |title=MMANUFACTURER'S ID NUMBERS |date=2010-06-01 |accessdate=2010-07-26}}</ref>, so it's very unlikely they still work with it;
# [http://www.crimsontech.jp/eng/home.html Crimson Technology] still provides a demo version for their [http://www.crimsontech.jp/eng/dls/ DLS tools], however it's unclear what XMF features (if any) they enable.

== External links ==

# [http://www.midi.org/techspecs/xmf/ XMF - eXtensible Music Format]

File:Ut 60-dls.xmf.png

2010-07-26T14:37:41Z

Xperroni: Internal structure of XMF file ut_60-dls.xmf in tree form.

Internal structure of XMF file ut_60-dls.xmf in tree form.