real time sonification of motion streams
Direct audio feedback during general movements or sport exercises opens a fascinating field of sonification in sport training and rehabilitation. An important problem in this context is to improve the aesthetic quality of realtime sonification. You can download the audio examples as a gzipped tar-archive and a realization of a library for this purpose on the basis of PD at the bottom of this page.
In addition to the IV'05 paper (available for download here) three videos were rendered to demonstrate the capabilities of the MotionLab Sonify system. Because of their complexity, all videos are available with and without an additional comment, so you may want to start with the commented version before looking, i.e. listening, to pure sonification. (All videos use the 3ivx Codec. So if you do not see the video, you need to install the proper codec. It can be found here in the Windows version)
2 segments video
Section three of the paper gives a thorough introduction to the system, but since the description is rather abstract watching/listening to one of the following videos is a good addition. The videos are ment as a simple introduction to sonification in general and to the system in special and are highly recommended, especially to the novice in sonification to get used to the sounds produced.
- video_2seg_c.zip (audio comment)
- video_2seg_ct.zip (text comment)
- video_2seg.zip (uncommented)
Sonification of a steady walk
This video is based on a motion capture set from the Carnegie Mellon Library and it shows a walking human figure.
Since an object's velocity can be observed quite well, first of all the left foot's velocity is sonified. After a short period of time, the right foot's velocity is sonified as well. After this introduction to basic locomotion sonification, the right foot's sonification is switched off again and the left foot's sonification is changed from velocity to force. You will notice that the resulting forces from the push off as well as the foot's landing on the ground can be heard well.
After some cycles of force sonification, the sound is switched to piano to demonstrate the effect of insufficient number of different available pitches. The sonification is much more prominent than before and therefore more distracting. Additional, the force of the other foot is sonified as well. After several more cycles, both patches are switched back to synthesizer sounds.
(Due to the short duration of the motion capture set, one cycle has been looped in MotionLab, therefore you will experience minor visual glitches, that do not affect sonification)
- video_walk_c.zip (audio comment)
- video_walk_ct.zip (text comment)
Sonification of a cart wheel
This video is based on a motion capture set from the Carnegie Mellon Library and it shows a human figure performing two cart wheels.
First, the motion is presented without any sonification; see if you notice any differences between the first and the second cartwheel. After a few cycles, sonification of the left foot's velocity starts. If it was not obvious prior to the sonified motion, it definitely is now: Sonificaton of the velocity reveals, that the second cartwheel is performed slightly faster than the first one - thus resulting in a higher pitch. This is a good example of the finer resolution of the hearing sense in comparision to visual sense and may be used to show that sonification not only adds additional information to a motion, but that this information contains more value to our brain.
After this example, sound is switched off again. Then, the right foot's force is sonified. Especially the second cartwheel results in a good sonification of the foot's pushing-off- and landing-forces. After a few cycles, sonification of the other foot's force is switched on as well. Here, the most audible feature is the last step after the second cartwheel.
This shows, that although inverse dynamics works well with all motions, that especially the carnegie mellon files contain much noise and motion capture data should be of high quality if complex sonification is wished or required.