OMK Logo
OMK Title

Chimaera

Usage Scenarios

If you have read the firmware documentation, you now know, that the device converts the raw sensor data to a stream of on, off and set event signals. Before the event signals reach your host computer running your audio framework, there is one more step involved: the serialization of the event data into a given transport protocol.

We use the extended Open Sound Control specification 1.0 over UDP and TCP as low-level protocol. Open Sound Control is an open protocol, and there exist infinite ways to serialize our event data. Depending on the robustness of the network connection and on the target programs on the host listening for incoming messages, we provide different higher-level protocols.

The higher-level protocols are implemented as output engines of the blob-tracking procedures of the Chimaera. An arbitrary amount of engines can be active concurrently. They are aggregated into one single Open Sound Control bundle at a given iteration step. As the firmware is free/libre and open source, the user can implement its own protocol, if the existing ones should not fit his or her needs. The higher-level protocols can be divided into two groups: the ones insusceptible to lost messages, the others not.

UDP is a connection-less protocol, which makes it the ideal choice for real-time networking. However, UDP packets are not guaranteed to arrive at the destination or they may arrive out-of-order. This is unlikely to happen in a simple LAN using direct connections or switches, it may become important in a more complex and/or wireless set-up using routers and NATs, though. In this case, it is recommended to switch to TCP instead. If you tend to loose UDP packets on your network, either use one of the higher-level protocols that are insusceptible to lost packets, e.g. TUIO 1.0/2.0 (or switch to TCP). Those engines serialize data in a way that enables the receiver to recover from lost information. This comes at a loss, however, as the host has to do some book-keeping, the algorithms involved are quite simple, though.

Out-of-order packets pose no problem, as the Open Sound Control bundles sent from the device are all timestamped. The timestamps are guaranteed to be monotonic and can be used to sort out-of-order packets. This works for both types of output engines. For this to work, clocks of the device and the host have to run in sync, which is transparently handled by the Precision Time Protocol.

If you have a stable connection with none or only a few lost packets, you can also use one of the other output engines. They tend to be more efficient as the host has to do no or less book-keeping.

As more and more audio software written today implements the Open Sound Control protocol (not only open-source, but also proprietary programs), the Chimaera is able to talk in their native language. The scsynth engine is such an example which enables the Chimaera to directly create and manipulate synths on a SuperCollider server which in turn leverages the host and can lower latency considerably. The OSC-MIDI output engine transmits raw MIDI data via Open Sound Control which can be easily dispatched 1to1 to any MIDI device at the host.

The output engines have all in common that the receiving end can reconstruct which blobs have appeared, disappeared or need updating. What happens with this information is pretty much up to you. The event signals can be combined in any way possible.

The most trivial example is to start a sine wave generator at an ON event, update its pitch and volume with the position and distance signals at a SET event and stop the wave at an OFF event.

You can do much more crazy things, of course, e.g. controlling a single synth with two events: pitch and volume with south-polarized magnets and filter-cutoff and resonance of a low-pass filter with north-polarized magnets. Divide the sensor array into segments associated to different instruments. Steer the synthesizer with one part of the device and down-stream effect plugins with another part. You may want to use the raw output of single sensors only. You could steer some mixer channels with the Chimaera or ...

Below you can find an overview of the standard output engines, their formatting and usage scenarios. If none should fit your application, there still is the possibility to write your own, of course.

Dump

The dump output engine writes out the raw sensor data as a binary blob. The engine is insusceptible to lost and out-of-order packets. There is a single bundled OSC message of the form:

/dump

TypeDescriptionMinMax
Int32 Frame ID (encoded as unsigned Int32) 0UINT32_MAX
Blob Binary blob of sensory dump --

The frame ID is a counter starting at 0 at reset. it is increased at each update step and can be used as a means of detecting missing and/or out-of-order bundles.

Each sensor value is given as a signed 16-bit integer in network byte-order, with its quiescent value already subtracted. There is no sensitivity scaling involved in this output, it's just the raw 12-bit (11-bit + sign) sensor value. For a device with 96 sensors, e.g, the binary blob would have a size of 96*2=192 bytes.

This output engine is used during both hardware and software sensor calibration, it is also useful for users that want to do their own processing of the raw sensor data. This is the fastest output engine, as there is no costly blob detection ongoing. It can be activated through the configuration system: /engines/dump/enabled

SuperCollider Sensor Dump

Chimaera graphical sensor dump implemented in SuperCollider showing 4 present magnetic sources. Get the code at our SuperCollider repository

TUIO 1.0

Learn more about the TUIO 1.0 protocol specification. We use the 2Dobj profile of the TUIO 1.0 spec as default. It is the profile which best fits the information coming from the Chimaera. The angular velocity and acceleration are meaningless for the device. But as those values are part of the 2DObj profile and need to be sent, the Chimaera just sets them to zero.

If your TUIO 1.0 client supports custom profiles, we suggest to use those instead, to not send redundant data and save some bandwidth. The profile we provide is: _sixya. The active profile can be chosen via: /engines/tuio1/custom_profile.

There is a single OSC bundle with at least an "alive" and "fseq" message. The "alive" message lists all currently active blobs according to their ID. Blob IDs are unique and represented by an increasing counter (starting at 0). Subsequently there may be an arbitrary number (also 0) of "set" messages, specifying blob position (x, y), polarity (angle) and attributes (blob ID, group ID) for each active blob. The bundle is finalized with the "fseq" message, carrying an increasing frame counter (starting at 1). The latter can be used to identify lost and/or out-of-order packets.

This output engine does not directly serialize on and off events. The host instead has to do some bookkeeping and decipher newly appeared and disappeared blobs based on the current and previous "alive" messages.

This output engine is insusceptible to lost and out-of-order packets and thus save to use over UDP.

/tuio/2Dobj or /tuio/_sixya

TypeDescriptionMinMax
String "alive" 0-
Int32 Blob ID 0 (encoded as unsigned Int32) 0UINT32_MAX
.. Blob ID 1 0UINT32_MAX
Int32 Blob ID N 0UINT32_MAX

/tuio/2Dobj or /tuio/_sixya

TypeDescriptionMinMax
String "set" 0-
Int32 Blob ID (encoded as unsigned Int32) 0UINT32_MAX
Int32 Group ID 0INT32_MAX
Float x coordinate 0.01.0
Float y coordinate 0.01.0
Float Angle (north:0 or south:Pi) 0.03.1415926
Float x velocity -inf+inf
Float y velocity -inf+inf
Float angular velocity (not used) 0.00.0
Float motion acceleration -inf+inf
Float angular acceleration (not used) 0.00.0

/tuio/2Dobj or /tuio/_sixya

TypeDescriptionMinMax
String "fseq" 0-
Int32 Frame ID (encoded as unsigned Int32) 0UINT32_MAX

There are numerous TUIO 1.0 client implementations for any imaginable language or software package out there, get an overview at tuio.org.

Note: Not all implementations may have been designed (and implemented accordingly) to handle such low latencies and high update rates as the Chimaera is capable of.

The engine is activated via: /engines/tuio1/enabled

TUIO 2.0

Learn more about the TUIO 2.0 protocol specification. This is the successor to TUIO 1.0. It is pretty similar to the 1.0 spec, but more modularized with a much cleaner name space. The spec is new and not yet widely used. We prefer it compared to the 1.0 spec, as it is more efficient for simple applications like ours (lower bandwidth and less to parse). We use tok component messages to transport blob events.

There is a single OSC bundle with at least a frm and alv message. The frm message carries an increasing frame counter (starting at 1) and the time stamp, the message was generated at. This information can be used to identify lost and/or out-of-order packets and to estimate velocity and acceleration signals. Subsequently there may be an arbitrary number (also 0) of tok component messages, specifying blob position (x, y), polarity (angle) and attributes (blob ID, group ID, type ID) for each active blob. The alv message lists all currently active blobs according to their ID. Blob IDs are unique and represented by an increasing counter (starting at 0).

This output engine does not directly serialize on and off events. The host instead has to do some bookkeeping and decipher newly appeared and disappeared blobs based on the current and previous alv messages.

This output engine is insusceptible to lost and out-of-order packets and thus save to use over UDP.

/tuio2/frm

TypeDescriptionMinMax
Int32 Frame ID (encoded as unsigned Int32) 0UINT32_MAX
Timetag OSC Timestamp at sensor readout 00xFFFFFFFF.FFFFFFFF
Int32 Sensor dimensions (encoded as 2x unsigned Int16) 0UINT32_MAX
String NAME:ID@MAC 0-

/tuio2/tok

TypeDescriptionMinMax
Int32 Blob ID (encoded as unsigned Int32) 0UINT32_MAX
Int32 Group ID 0INT32_MAX
Int32 Type ID (upper unsigned Int16, north:0x100, south:0x80), User ID (lower unsigned Int16) 0UINT32_MAX
Float x coordinate 0.01.0
Float y coordinate 0.01.0
Float Angle (north:0, south:Pi) 0.03.1415926
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf
Float Angular velocity 00
Float Acceleration -inf+inf
Float Angular acceleration 00

/tuio2/alv

TypeDescriptionMinMax
Int32 Blob ID 0 (encoded as unsigned Int32) 0UINT32_MAX
.. Blob ID 1 0UINT32_MAX
Int32 Blob ID N 0UINT32_MAX

Ready to use SuperCollider classes and sketches using the TUIO 2.0 engine can be found at our SuperCollider repository

The engine is activated via: /engines/tuio2/enabled. Output of derivative signals can be toggled via: /engines/tuio2/derivatives.

SuperCollider

We use SuperCollider as one of our main software synthesizers, we therefore implemented this output engine to be able to directly send event data to the SuperCollider synthesis servers scsynth and supernova. To fully understand this engine, you may need to get familiar with the SuperCollider server messaging system first. There is an overview of the latter at http://doc.sccode.org/.

The usage scheme for this engine is to define all the synthdefs (aka instruments) with sclang (or another client) and then start, control and stop them from the device directly.

For optimal control, groups on the device can be configured in different profiles. A group may be used to trigger the creation and modulation of a single synth instance on the scsynth server. Or a group may be used to only modulate already running synths on the server. For this, individual groups can be configured to allocate or not-to-allocate a synth instance on the server and to trigger or not-to-trigger a gate signal on the server.

As the creation of a synth on the server needs a synth name, synth ID, group add action, group ID, out channel and whether the sensor array range should control a synth instance or a synth group, all these have to be provided to the device before-hand via /engines/scsynth/attributes/*.

/s_new

is triggered at a ON event and only sent with synth allocation enabled

TypeDescriptionMinMax
String Synth name --
Int32 Synth ID 0INT32_MAX
Int32 Group add action 04
Int32 Group ID 0INT32_MAX
Int32 parameter #(offset + 4) 0INT32_MAX
Int32 Polarity 128256
String "gate" --
Int32 Boolean 11
String "out" --
Int32 Out channel 0INT32_MAX

/n_setn

is triggered at an ON event and only sent with synth gating enabled

TypeDescriptionMinMax
Int32 Synth ID 0INT32_MAX
Int32 parameter #(offset + 0) 0INT32_MAX
Int32 parameter number 24
Float x coordinate 0.01.0
Float y coordinate 0.01.0
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf

/n_setn

is triggered at a SET event and always sent

TypeDescriptionMinMax
Int32 Synth ID 0INT32_MAX
Int32 parameter #(offset + 0) 0INT32_MAX
Int32 parameter number 24
Float x coordinate 0.01.0
Float y coordinate 0.01.0
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf

/n_set

is triggered at an OFF event and only sent with synth gating enabled

TypeDescriptionMinMax
Int32 Synth ID 0INT32_MAX
String "gate" --
Int32 Boolean 00

This output engine is susceptible to lost packets and should only be used on small and stable networks with UDP, works fine with TCP, though.

Ready to use SuperCollider classes and sketches using the scsynth engine can be found at our SuperCollider repository

The engine is activated via: /engines/scsynth/enabled. Output of derivative signals can be toggled via: /engines/scsynth/derivatives.

Dummy

The Dummy output engine is a direct 1to1 translation of the blob event stream to Open Sound Control messages. For an On event, there is an /on message, for an Off event an /off message and for a Set event a /set message. Apart from the latter ones, the engine also outputs an /idle message at a fixed interval when no events are present which may help the host to recover from lost Off events.

Note: This is a good starting point for writing your own output engine because it's compressed to a bare minimum of event handling.

This output engine is susceptible to lost packets and should only be used on small and stable networks with UDP, works fine with TCP, though.

/idle

TypeDescriptionMinMax

/on

TypeDescriptionMinMax
Int32 Blob ID 0INT32_MAX
Int32 Group ID 0INT32_MAX
Int32 Type ID (north: 256 or south:128) 128256
Float x coordinate 0.01.0
Float y coordinate 0.01.0
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf

/off

TypeDescriptionMinMax
Int32 Blob ID 0INT32_MAX

/set

TypeDescriptionMinMax
Int32 Blob ID 0INT32_MAX
Float x coordinate 0.01.0
Float y coordinate 0.01.0
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf

The engine can be run in a redundant data mode where both group ID and type ID are sent for /set and /off messages via /engines/dummy/redundancy. This has the advantage that the receiver does not have to do any bookkeeping at all, e.g. does not have to remember any blob-group-type ID associations as they are inherently included in each single message. Enabling redundancy however slightly increases used bandwidth.

/off

TypeDescriptionMinMax
Int32 Blob ID 0INT32_MAX
Int32 Group ID 0INT32_MAX
Int32 Type ID (north: 256 or south:128) 128256

/set

TypeDescriptionMinMax
Int32 Blob ID 0INT32_MAX
Int32 Group ID 0INT32_MAX
Int32 Type ID (north: 256 or south:128) 128256
Float x coordinate 0.01.0
Float y coordinate 0.01.0
With derivatives
Float x velocity -inf+inf
Float y velocity -inf+inf

Ready to use SuperCollider classes and sketches using the Dummy engine can be found at our SuperCollider repository

Ready to use Pure Data sketches using the Dummy engine can be found at our Pure Data repository

The engine is activated via: /engines/dummy/enabled. Output of derivative signals can be toggled via: /engines/dummy/derivatives.

OSC-MIDI

There are numerous ways to translate the event data into MIDI commands, here we just provide the most common ones. If you need an alternate conversion theme, it makes sense to use one of the other output engines and only convert to MIDI on the host. Or you may adapt this output engine to your needs.

This output engine exists for convenience, if you want to use the device mostly to steer MIDI devices. Keep in mind that this output engine looses some of the event information, like floating point resolution of the x and y dimensions and magnetic field polarization.

This engine translates event data into NOTEON, NOTEOFF, PITCHBEND, CONTROLLER, NOTE PRESSURE or CHANNEL PRESSURE MIDI messages. It maps x-dimension to frequency by default (using PITCHBEND), and y-dimension to note/channel pressure or an arbitrary controller. The engine can be configured by the following parameters:

If the receiving end supports multiple arguments, multiple MIDI messages can be packed into a single OSC message to lower bandwidth. If the receiving end only supports single arguments, disable multiple arguments via /engines/oscmidi/multi.

You can choose between a serialization to the special OSC MIDI type ('m'), standard OSC BLOB type ('b') or OSC INT32 ('i') type, via /engines/oscmidi/format. The extended Open Sound Control specification features the MIDI argument type 'm'. It is a four byte array. First byte stores MIDI port, second byte stored MIDI system command with channel, third and optional fourth byte store the data for the given command. The OSC MIDI type is not that widely supported, we thus recommend to use the standard OSC BLOB type in such cases. Alternatively, you may want to use the OSC INT32 type if your host should also not support the OSC BLOB type. There, the MIDI bytes are encoded into a single 32-bit integer in least signifiant byte first order, e.g. a MIDI NOTEON message (0x90-0x30-0x7f) thus would be encoded into the INT32 0x7f3090.

The remaining attributes can be set on a per group basis via /engines/oscmidi/attributes/0. E.g. the MIDI note offset corresponding to x=0 can be a floating point number from 0-127. To set this to a low C minus a quarter tone, you would set it to 24-0.5=23.5. Pitch-bend note range for the a given sensor-array range corresponds to e.g x=0 to x=1. This can be a floating point number from 0-127. If you want to configure a 144-sensor array to span 4 octaves, you would set this to 48. What do you want the y-dimension to map to? Choose one of the available mappings: note pressure (polyphonic aftertouch), channel pressure (channel aftertouch) or an arbitrary controller. For the controllers that support double-precision, the output engine maps the [0-1] range of the y-dimension to a 14-bit value and emits both the least significant and the most significant 7 bits. For the controllers that only support single-precision, the [0-1] range is mapped to a single 7-bit value only. Note and channel pressure only support 7 bit resolution, so keep in mind that these capture only a fraction of the available resolution.

The MIDI channel is set to the blob group by default. You can change that behaviour by enabling the MPE (Multidimensional Polyphonic Expression) mode via /engines/oscmidi/mpe.

The path of the message defaults to /midi and can be changed via /engines/oscmidi/path.

This output engine is susceptible to lost packets (that is the nature of MIDI) and should only be used on small and stable networks with UDP, works fine with TCP, though.

/midi

TypeDescription
MIDI OSC MIDI argument 0
.. OSC MIDI argument 1
MIDI OSC MIDI argument N

Ready to use SuperCollider classes and sketches using the OSC-MIDI engine can be found at our SuperCollider repository

Ready to use Pure Data sketches using the OSC-MIDI engine can be found at our Pure Data repository

Ready to use Renoise sketches using the OSC-MIDI engine can be found at our Renoise repository

The engine is activated via: /engines/oscmidi/enabled

Custom

The Custom output engine was designed as a compromise when you need a special output format but do not want to (or cannot) implement it into the firmware directly.

For each of the six destination hooks frame, idle, on, off, set and end you can define the path and format of the output dynamically. The format is specified in OSC postfix notation, compiled to byte code so it can efficiently be run in a virtual machine on-the-fly. Note: Make sure to understand first how OSC postfix notation works before proceeding.

The Chimaera defines the following variables to be used in the OSC postfix notation:

VariableDescription
$n Sensor number
$f Frame Id
$b Blob Id
$g Group Id
$p Polarity (128 or 256)
$x Position x [0,1]
$z Vicinity z [0,1]
$X Velocity x
$Z Velocity z

The formats can be reset and set with the following OSC methods: /engines/custom/reset and /engines/custom/append. The format strings themselves are not stored in RAM and can thus not be queried. The generated byte code though is stored and preserved across boots when saved previously.

This output engine may be susceptible to lost packets depending on what output formats are used and thus may only be used on small and stable networks with UDP, works fine with TCP, though.

Ready to use SuperCollider classes and sketches using the Custom engine can be found at our SuperCollider repository

Ready to use Renoise sketches using the OSC-MIDI engine can be found at our Renoise repository

The engine is activated via: /engines/custom/enabled

Example

The following configuration e.g. can be used to send /gate messages for on and off callbacks and /cv messages for set callbacks. The blob id $b is mapped to the range [0,7]. Position $x is inverted from [0,1] to [1,0]. If vicinity $z is less than 0.5, the second expression returns -1 or 1 otherwise.

$ oscsend ... /engines/custom/reset i $[RANDOM]
$ oscsend ... /engines/custom/append isss $[RANDOM] 'on' '/gate' 'i($b 8%) i(1)'
$ oscsend ... /engines/custom/append isss $[RANDOM] 'off' '/gate' 'i($b 8%) i(0)'
$ oscsend ... /engines/custom/append isss $[RANDOM] 'set' '/cv' 'i($b 8%) f(1 $x-) i(1~ 1 $z 0.5<?)'

Groups

To ease handling of multi-touch events, we have introduced the concepts of blob groups. A blob is only detected if it is part of a given group. A group association is only triggered when a blobs position falls within the range the group was defined for and when the magnetic polarity of the blob matches the groups one, too. A given group can respond to a single or both magnetic polarities.

Groups have increasing group IDs (starting at 0). By default, just one group is defined: group 0 responds to both magnetic fields over the whole sensor array (0..1). A given blob is associated to the first group that has matching criterias.

There is the option to instruct the device to rescale positions of groups from a subrange to a complete range. If we define a group to respond to the x-dimension range (0.2 .. 0.8), enabled scaling would output a positional value of 0 for a raw value of 0.2, and 1 for a raw value of 0.8. With disabled scaling, the raw values are output unmodified. e.g. 0.2 and 0.8.

SuperCollider Group Configurator

Chimaera graphical group configurator implemented in SuperCollider showing 8 distinct groups, each group only responding over a fourth of the whole sensor array, odd groups responding to north polarized magnetic fields, even groups responding to south polarized magnetic fields, odd groups being scaled, even groups not. Get the code at our SuperCollider repository

Group settings can be changed through /sensors/group/attributes/*, loaded through /sensors/group/load and saved across reboots through /sensors/group/save.

Software calibration

Start calibration routine by getting the device into calibration mode.

/calibration/start i $[RANDOM]

Before submitting the following command, wait some seconds and keep any magnetic sources clear of the sensor array. This step calibrates for the quiescent sensor values, e.g. the values the sensors spit out when no magnetic source is present.

/calibration/zero i $[RANDOM]

The next step calibrates for the threshold value of each sensor, e.g. the minimal value, a sensor will respond to. Before submitting the following command, glide along the whole sensor array with both poles of your magnetic source at a fixed vicinity. This method will return a sensor number, the next four steps will be based on. The sensors are numbered beginning at the end of the device without connectors.

/calibration/min i $[RANDOM]

The next four steps are needed to calibrate for the distance-magnetic-flux relationship. As we do a five-point curve fitting, apart from threshold and maximal points, we need three additional points. Those are recorded in the following steps. Glide along the sensor array in the region above the sensor of interest (known from the previous step) with both poles of your magnetic source at a fixed, increasing, but varying vicinity, e.g. at 0.25, 0.5 and 0.75.

/calibration/mid if $[RANDOM] 0.25
/calibration/mid if $[RANDOM] 0.5
/calibration/mid if $[RANDOM] 0.75

The curve-fitting is finalized by providing the maximal values for the sensor of interest. Before submitting the following command, glide directly along the surface (if this is your desired maximal vicinity) in the region above the sensor of interest with both poles of your magnetic source.

/calibration/max $[RANDOM]

The sensitivity of each sensor is calibrated for in the last step. Before submitting the following command, glide along the whole sensor array with both poles of your magnetic source at a fixed vicinity, e.g. 0.75.

/calibration/end if $[RANDOM] 0.75

Calibration now is done. You may want to test your new calibration and then save it to one of the three slots (0, 1 or 2) on the EEPROM.

/calibration/save ii $[RANDOM] 0

Reset

The Chimaera can boot up in three modes: in soft, in hard or in flash mode. In soft mode, the Chimaera loads its previously stored configuration from EEPROM, whereas in hard mode, the device boots up with factory presets. A soft reset may be necessary for some configuration settings to take effect. A hard reset may be useful in cases where you screwed up the configuration and want to start from scratch. In flash mode, the device can be flashed with an up-to-date or custom firmware.

If the Chimaera is (re)powered, it boots up and loads its saved configuration. A normal startup therefore is equivalent to soft mode.

WANTED:
Beta Testers

After a high interest at Maker Faire Rome, we are now running a beta-testing campaign to collect more comprehensive feedback of first-hand experiences of our final Chimaera prototype design from interested individuals. Get in contact with us. Now.

Please support free/libre software and hardware designs.

or

donate bitcoins

or

request bank credentials via encrypted mail for SEPA transfers.

Last update - 16 Sep 2017

GitHub Vimeo

Copyright © 2014-2017, Hanspeter Portner, Open Music Kontrollers, cc-by-sa 4.0. Uses libre javascript.