9. RAW Metadata Files

A RAW file holds nothing but the sample values, so Dragonfly pairs it with a sibling metadata file that describes the geometry, shape and calibration of the data. Dragonfly both writes and reads this pair.

While there are several other metadata formats out there, too many to be listed here, and while some of them are supported by Dragonfly, this page describes the two “homemade” ones.

Important

Both of them live in a file with the .dat extension. They are two encodings of the same metadata, not two kinds of file:

• an XML encoding, which is what Dragonfly writes today;

• a legacy plain-text encoding, which is still read but is no longer written.

On import, Dragonfly first tries to parse the file as XML with a RAWFileData root element. Only if that fails does it fall back to the plain-text parser. The extension must be .dat either way; naming the file .xml will not work, as the import wizard only offers a DAT (*.dat) filter and recognises the format by extension alone.

9.1. Exporting to RAW

From the user interface, use the Export Images command on a dataset and pick the RAW format, or Export Images with Contrast/Brightness to bake the current window level and gamma into the exported samples. From Python, the same export is available as:

from OrsPlugins.orsimagesaver import OrsImageSaver

OrsImageSaver.exportDatasetToRAWFile(channel.getGUID(), r'C:\data\brain.raw',
                                     exportMultiFrameFile=True,
                                     wlw=0, wlc=0, gamma=0, showProgress=False)

The convenience method OrsImageSaver.exportChannel(channel, fileName, extension='dat') calls the same code.

Dragonfly writes the sample values to <name>.raw, adding the extension itself if it is missing, and then writes the metadata next to it as <name>.dat.

If wlw, wlc and gamma are not all zero, the window level and gamma transformation is baked into the exported samples. The exported file is then no longer the original data.

9.1.1. Layout of the RAW data

The RAW file is a bare dump of the samples, with no header, no padding and no separator. Samples are written in X, then Y, then Z, then T order, so a file holds exactly:

X * Y * Z * T * bytes_per_sample

bytes, where bytes_per_sample follows Format: 1 for UCHAR, 2 for USHORT, 4 for UINT and FLOAT. Values are written in the machine’s native byte order, which is little-endian on every platform Dragonfly supports.

9.1.2. Multiple files

When the dataset has several timesteps and exportMultiFrameFile is False, one RAW file is written per timestep. The first one keeps the name you chose and the following ones get the timestep index appended to the base name:

brain.raw     brain1.raw     brain2.raw     ...     brain.dat

A single .dat describes the whole set, and lists the files in order under an ObjectFileNames element.

With exportMultiFrameFile set to True, all timesteps go into one RAW file, one after the other, and the .dat names that single file.

9.2. XML format

This is the format Dragonfly writes. Here is a complete file, produced by exporting a two-timestep dataset from Dragonfly 2027.1:

<?xml version="1.0"?>
<RAWFileData>
    <Version>1</Version>
    <ObjectFileName>time step.raw</ObjectFileName>
    <Format>UCHAR</Format>
    <DataSlope>1</DataSlope>
    <DataOffset>0</DataOffset>
    <Unit>Density</Unit>
    <Resolution X="982" Y="649" Z="1264" T="2" />
    <Spacing X="2.7571400000000001684558395553814591494301566854119301e-05" Y="2.7571400000000001684558395553814591494301566854119301e-05" Z="2.7571400000000001684558395553814591494301566854119301e-05" />
    <Orientation X0="1" X1="0" X2="0" Y0="0" Y1="1" Y2="0" Z0="0" Z1="0" Z2="1" />
    <Position P1="0" P2="0" P3="0" />
</RAWFileData>

This one was exported as a multi-frame file, so both timesteps sit in time step.raw, whose size is exactly 982 * 649 * 1264 * 2 * 1 bytes.

Note

Current versions write UTF-8 without a BOM, and the XML declaration does not name an encoding. Older files may instead be UTF-16 with an explicit encoding="UTF-16" in the declaration, and may be written on a single line. Do not assume either when reading a file you did not produce.

Note

Doubles are written with full round-trip precision, which is why the spacings above run to fifty digits. Any XML parser will read them back correctly, but a hand-rolled reader should expect long decimal strings with an exponent.

9.2.1. Elements

All elements are children of the RAWFileData root, in the order below. Everything except ObjectFileName (or ObjectFileNames), Format, Resolution and Spacing is optional.

Name	Meaning	Type
Version	Format version of the metadata file. Dragonfly writes 1. See the warning below: an absent or zero version means the file predates version 1.0 and its spacings are read as millimetres.	double
ObjectFileName	File name of the RAW data file, with no path. The file is assumed to sit in the same folder as the metadata file. Use this form when there is a single RAW file.	string
ObjectFileNames	Parent element holding one ObjectFileName child per RAW file, in order. Written instead of a lone ObjectFileName when the export produced several files (see Multiple files below).	element
Format	Sample data type, one of UCHAR, USHORT, UINT or FLOAT. Compared case-insensitively on import.	string
DataSlope	Slope of the linear calibration applied to the samples. A value of 0 is ignored on import.	double
DataOffset	Offset of the linear calibration applied to the samples.	double
Unit	Unit of the calibrated data values, for example HU or Density. This is the data unit, not the spatial unit.	string
Resolution	Dimensions of the dataset, as the attributes X, Y, Z and T.	4 uint
Spacing	Size of one voxel in metres, as the attributes X, Y and Z.	3 double
Orientation	Orientation of the whole dataset: three unit vectors, given as the attributes X0/X1/X2, Y0/Y1/Y2 and Z0/Z1/Z2, respectively the direction, up and normal axes.	9 double
Position	Position of the dataset in world coordinates in metres, as the attributes P1, P2 and P3. Dragonfly writes the centre of voxel (0, 0, 0), that is the box origin plus half a voxel.	3 double
SliceData	Optional parent element holding a per-slice orientation and position. See Per-slice geometry below.	element

Warning

Spacings and positions are in metres. Dragonfly works in SI units internally and writes them as such.

Version is what decides this. If it is absent or 0, the file is treated as predating version 1.0 and every value in Spacing is multiplied by 0.001, that is, read as millimetres. Always write <Version>1</Version> in files you generate yourself, or your voxels will come out a thousand times too small.

9.2.2. Per-slice geometry

When each slice has its own orientation and position, they are listed under a SliceData element. It holds one child element per slice, named Slice0, Slice1 and so on, in increasing Z and then increasing T order:

<SliceData>
  <Slice0 X0="1" X1="0" X2="0" Y0="0" Y1="1" Y2="0" Z0="0" Z1="0" Z2="1"
          P1="0" P2="0" P3="0" SPX="0.00072" SPY="0.00072"/>
  <Slice1 .../>
</SliceData>

Each slice element carries these attributes:

Name	Meaning	Type
X0, X1, X2	Direction axis of the slice.	3 double
Y0, Y1, Y2	Up axis of the slice.	3 double
Z0, Z1, Z2	Normal axis of the slice.	3 double
P1, P2, P3	Position of the slice in world coordinates, in metres.	3 double
SPX	In-plane spacing along the slice direction axis, in metres. Ignored on import when it is 0.	double
SPY	In-plane spacing along the slice up axis, in metres.	double

Note

The element names are read positionally, not by their number. The first child is slice 0 of timestep 0, and the reader moves to the next timestep every Resolution/@Z children. For a dataset with more than one timestep the names therefore repeat, Slice0 through SliceN once per timestep.

9.2.3. Schema

<xsd:schema xmlns:xsd="http://www.w3.org/2001/XMLSchema">

  <xsd:element name="RAWFileData" type="RAWFormat" />

  <xsd:complexType name="RAWFormat">
    <xsd:sequence>
      <!-- Version 1 means metres. Omitting it means millimetres, see the warning above -->
      <xsd:element name="Version" type="xsd:double" minOccurs="0" maxOccurs="1"/>
      <!-- Exactly one of ObjectFileName or ObjectFileNames -->
      <xsd:choice>
        <xsd:element name="ObjectFileName" type="xsd:string"/>
        <xsd:element name="ObjectFileNames" type="fileNameList"/>
      </xsd:choice>
      <!-- Format holds the sample data type -->
      <xsd:element name="Format" type="pixelType"/>
      <xsd:element name="DataSlope" type="xsd:double" minOccurs="0" maxOccurs="1"/>
      <xsd:element name="DataOffset" type="xsd:double" minOccurs="0" maxOccurs="1"/>
      <!-- Unit of the data values, not of the geometry -->
      <xsd:element name="Unit" type="xsd:string" minOccurs="0" maxOccurs="1"/>
      <xsd:element name="Resolution" type="imageResolution"/>
      <xsd:element name="Spacing" type="imageSpacing"/>
      <xsd:element name="Orientation" type="imageOrientation" minOccurs="0" maxOccurs="1"/>
      <xsd:element name="Position" type="imagePosition" minOccurs="0" maxOccurs="1"/>
      <!-- See note below -->
      <xsd:element name="SliceData" minOccurs="0"/>
    </xsd:sequence>
  </xsd:complexType>

  <xsd:complexType name="fileNameList">
    <xsd:sequence>
      <xsd:element name="ObjectFileName" type="xsd:string" maxOccurs="unbounded"/>
    </xsd:sequence>
  </xsd:complexType>

  <xsd:simpleType name="pixelType">
    <xsd:restriction base="xsd:string">
      <xsd:enumeration value="UCHAR" />
      <xsd:enumeration value="USHORT" />
      <xsd:enumeration value="UINT" />
      <xsd:enumeration value="FLOAT" />
    </xsd:restriction>
  </xsd:simpleType>

  <xsd:complexType name="imageResolution">
    <xsd:attribute use="required" name="X" type="xsd:integer" />
    <xsd:attribute use="required" name="Y" type="xsd:integer" />
    <xsd:attribute use="required" name="Z" type="xsd:integer" />
    <xsd:attribute use="required" name="T" type="xsd:integer" />
  </xsd:complexType>

  <!-- What we call spacing is the voxel size, in metres -->
  <xsd:complexType name="imageSpacing">
    <xsd:attribute use="required" name="X" type="xsd:double" />
    <xsd:attribute use="required" name="Y" type="xsd:double" />
    <xsd:attribute use="required" name="Z" type="xsd:double" />
  </xsd:complexType>

  <!-- Orientation is 3 vectors, respectively direction, up and normal axis -->
  <xsd:complexType name="imageOrientation">
    <xsd:attribute use="required" name="X0" type="xsd:double" />
    <xsd:attribute use="required" name="X1" type="xsd:double" />
    <xsd:attribute use="required" name="X2" type="xsd:double" />
    <xsd:attribute use="required" name="Y0" type="xsd:double" />
    <xsd:attribute use="required" name="Y1" type="xsd:double" />
    <xsd:attribute use="required" name="Y2" type="xsd:double" />
    <xsd:attribute use="required" name="Z0" type="xsd:double" />
    <xsd:attribute use="required" name="Z1" type="xsd:double" />
    <xsd:attribute use="required" name="Z2" type="xsd:double" />
  </xsd:complexType>

  <!-- Position is in world coordinates and in metres, P1 -> X, P2 -> Y, P3 -> Z -->
  <xsd:complexType name="imagePosition">
    <xsd:attribute use="required" name="P1" type="xsd:double" />
    <xsd:attribute use="required" name="P2" type="xsd:double" />
    <xsd:attribute use="required" name="P3" type="xsd:double" />
  </xsd:complexType>

  <!-- When you have slice orientation and position, this is where you define them.
       Because our XML is poorly structured for the slice data, we cannot describe it
       in the XSD.

       You start with an empty node named "SliceData".
       For each slice, add a node thus:

       SliceN X0= X1= X2= Y0= Y1= Y2= Z0= Z1= Z2= P1= P2= P3= SPX= SPY=

       where N is the slice number (0 based)
  -->
</xsd:schema>

9.3. Legacy plain-text format

Warning

Dragonfly no longer writes this format. It is read for backward compatibility only and should not be your choice for new files. Use the XML format above.

The legacy metadata file is a straight text file where each line has a field name and a value, separated by a colon:

field: value

Lines starting with # are ignored. Field names are matched case-insensitively, and tabs are treated as spaces. Here is a small example:

# A legacy RAW metadata file
ObjectFileName: brain.raw
Resolution: 512 512 108
TimeSteps: 1
Format: UCHAR
MeasurementUnit: MILLIMITER
SliceThickness: 0.72 0.72 0.72
Orientation: 1 0 0 0 1 0 0 0 1
Position: 0 0 0
DataSlope: 1
DataOffset: 0
DataUnit: HU

These are the valid fields, their meaning and the expected type of value.

Name	Meaning	Type
ObjectFileName	File name of the RAW data file, with no path. The file is assumed to sit in the same folder as the metadata file.	string
Resolution	X, Y and Z dimensions.	3 int
TimeSteps	T dimension.	int
Format	Sample data type, one of UCHAR, USHORT, UINT or FLOAT.	string
SliceThickness	Voxel size along X, Y and Z, expressed in the unit given by MeasurementUnit.	3 double
MeasurementUnit	Spatial unit of SliceThickness, either METER or MILLIMITER. Note the spelling: the parser really does look for MILLIMITER, not MILLIMETER. Any other value, or no field at all, is treated as millimetres.	string
Orientation	Three vectors of size 3, respectively the direction, up and normal axes of the dataset. If the first vector is all zeros the whole orientation is reset to the identity.	9 double
Position	Position of the dataset in world coordinates.	3 double
DataSlope	Slope of the linear calibration applied to the samples.	double
DataOffset	Offset of the linear calibration applied to the samples.	double
DataUnit	Unit of the calibrated data values, for example HU.	string
Slice Orientation/Position	Geometry of one slice. Note that this key is two words, not one. See the syntax below.	see below

The per-slice geometry lines have this shape, one line per slice:

Slice Orientation/Position: <index> X0 X1 X2 Y0 Y1 Y2 Z0 Z1 Z2 P1 P2 P3 SPX SPY

That is a 1-based slice index followed by fourteen doubles: nine for the orientation, three for the position and two for the in-plane spacing, in the same meaning as the XML attributes of the same name. The index runs continuously across timesteps, so slice z of timestep t is written with index t * Z + z + 1.

Note

Because this format has no version field, its SliceThickness is read as millimetres unless MeasurementUnit: METER says otherwise.

Note

Write exactly one space after the colon. ObjectFileName is read as everything past that one space, so a second space ends up in the file name, and Orientation, Position and SliceThickness look for the literal sequence ": " to find where their values start.

Note

To avoid spending a long time on a file that is not really metadata, the parser gives up once it has read 2% of a file larger than 50 MB.

9.4. Importing

Select the .dat file in the import wizard, not the .raw file. Dragonfly resolves the RAW file names relative to the folder holding the .dat, which is why they must be stored without a path.

Selecting several .dat files at once is supported. Dragonfly checks that they all describe the same geometry, dimensions, voxel size, data type and timestep count, and refuses the set otherwise.