real time sonification of motion streams

Direct audio feedback during general movements or sport exercises opens a fascinating field of sonification in sport training and rehabilitation. An important problem in this context is to improve the aesthetic quality of realtime sonification. You can download the audio examples as a gzipped tar-archive and a realization of a library for this purpose on the basis of PD at the bottom of this page.

MotionLab Sonify

In addition to the IV'05 paper (available for download here) three videos were rendered to demonstrate the capabilities of the MotionLab Sonify system. Because of their complexity, all videos are available with and without an additional comment, so you may want to start with the commented version before looking, i.e. listening, to pure sonification. (All videos use the 3ivx Codec. So if you do not see the video, you need to install the proper codec. It can be found here in the Windows version)

2 segments video

Section three of the paper gives a thorough introduction to the system, but since the description is rather abstract watching/listening to one of the following videos is a good addition. The videos are ment as a simple introduction to sonification in general and to the system in special and are highly recommended, especially to the novice in sonification to get used to the sounds produced.

  • (audio comment)
  • (text comment)
  • (uncommented)

Sonification of a steady walk

This video is based on a motion capture set from the Carnegie Mellon Library and it shows a walking human figure.

Since an object's velocity can be observed quite well, first of all the left foot's velocity is sonified. After a short period of time, the right foot's velocity is sonified as well. After this introduction to basic locomotion sonification, the right foot's sonification is switched off again and the left foot's sonification is changed from velocity to force. You will notice that the resulting forces from the push off as well as the foot's landing on the ground can be heard well.

After some cycles of force sonification, the sound is switched to piano to demonstrate the effect of insufficient number of different available pitches. The sonification is much more prominent than before and therefore more distracting. Additional, the force of the other foot is sonified as well. After several more cycles, both patches are switched back to synthesizer sounds.

(Due to the short duration of the motion capture set, one cycle has been looped in MotionLab, therefore you will experience minor visual glitches, that do not affect sonification)

  • (audio comment)
  • (text comment)

Sonification of a cart wheel

This video is based on a motion capture set from the Carnegie Mellon Library and it shows a human figure performing two cart wheels.

First, the motion is presented without any sonification; see if you notice any differences between the first and the second cartwheel. After a few cycles, sonification of the left foot's velocity starts. If it was not obvious prior to the sonified motion, it definitely is now: Sonificaton of the velocity reveals, that the second cartwheel is performed slightly faster than the first one - thus resulting in a higher pitch. This is a good example of the finer resolution of the hearing sense in comparision to visual sense and may be used to show that sonification not only adds additional information to a motion, but that this information contains more value to our brain.

After this example, sound is switched off again. Then, the right foot's force is sonified. Especially the second cartwheel results in a good sonification of the foot's pushing-off- and landing-forces. After a few cycles, sonification of the other foot's force is switched on as well. Here, the most audible feature is the last step after the second cartwheel.

This shows, that although inverse dynamics works well with all motions, that especially the carnegie mellon files contain much noise and motion capture data should be of high quality if complex sonification is wished or required.


Erik Peper, Björn Krüger, Esther Gokhale, and Richard Harvey
In: Biofeedback (2019)
Icxa Khandelwal, Katharina Stollenwerk, Björn Krüger, and Andreas Weber
In proceedings of Computational Science and Its Applications -- ICCSA 2019, 2019
Katharina Stollenwerk, Jonas Müller, André Hinkenjann, and Björn Krüger
In: Sensors (Aug. 2019)
Erik Peper, Björn Krüger, and Esther Gokhale
Association for Applied Psychophysiology and Biofeedback, Mar. 2019
In proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces, Tokyo, Japan, pages 305-310, ACM, 2018
Susanna Mezzarobba, Michele Grassi, Lorella Pellegrini, Mauro Catalan, Björn Krüger, Giovanni Furlanis, Paolo Manganotti, and Paolo Bernardis
In: Frontiers in Neurology (Jan. 2018), 8(723)
J. Wiemeyer and A Seyfarth (Editors)
U. Fehse, A. Effenberg, Gerd Schmitz, and Björn Krüger
In proceedings of Human Movement and Technology. 11th joint dvs- Conference on Motor Control & Learning, Biomechanics & Training, Darmstadt (Germany), pages 157-159, Sept. 2016
Susanna Mezzarobba, Michele Grassi, Mauro Catalan, Lorella Pellegrini, Roberto Valentini, Björn Krüger, and Paolo Bernardis
In proceedings of 20th International congress of Parkinson's Disease and Movement Disorders, Berlin, Germany, June 2016
Alfred O. Effenberg, U. Fehse, Gerd Schmitz, Björn Krüger, and H. Mechling
In: Frontiers in Neuroscience (May 2016), 10:219
In proceedings of 2014 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Copenhagen, Denmark, July 2014
Dissertation, Universität Bonn, Apr. 2012
Heike Brock, Gerd Schmitz, Jan Baumann, and Alfred O. Effenberg
In proceedings of 9th Conference of the International Sports Engineering Association (ISEA), Elsevier, Jan. 2012
Björn Krüger, Jan Baumann, Mohammad Abdallah, and Andreas Weber
In proceedings of Workshop on Virtual Reality Interaction and Physical Simulation (VRIPHYS), Lyon, France, Eurographics Association, Dec. 2011
In proceedings of Workshop on Virtual Reality Interaction and Physical Simulation (VRIPHYS), Lyon, France, Eurographics Association, Dec. 2011
In proceedings of SCA'11: Poster Proc. ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Vancouver, Canada, Aug. 2011
Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helten, Meinard Müller, Hans-Peter Seidel, and Bernd Eberhardt
In: ACM Trans. Graph. (May 2011), 30:3(18:1-18:12)
In proceedings of 2010 ACM SIGGRAPH/Eurographics Symposium on Computer Animation, Madrid, Spain, pages 1-10, Eurographics Association, July 2010
Alfred O. Effenberg, Andreas Weber, K. Mattes, U. Fehse, and H. Mechling
In: Supplement to Journal of Sport & Exercise Psychology (2007), 29:66
Christoph Henkelmann, Arno Zinke, and Andreas Weber
In: Workshop VR/AR 2007 (2007)
Christoph Henkelmann
Universität Bonn, Technical Report number CG-2007-4, Oct. 2007
A. Effenberg, J. Melzer, Andreas Weber, and Arno Zinke
In proceedings of The 9th International Conference on Information Visualisation (IV'05), pages 17-23, IEEE Press, July 2005
Christoph Brzozowski, S. Klemme, J. Melzer, and H. Nguyen
Universität Bonn, Technical Report number CG-2004-1, June 2004