Multimodal Perception of Material Properties

In proceedings of ACM SIGGRAPH Symposium on Applied Perception, Tübingen, Germany, pages 33-40, ACM, Sept. 2015
 

Abstract

The human ability to perceive materials and their properties is a very intricate multisensory skill and as such not only an intriguing research subject, but also an immense challenge when creating realistic virtual presentations of materials. In this paper, our goal is to learn about how the visual and auditory channels contribute to our perception of characteristic material parameters. At the center of our work are two psychophysical experiments performed on tablet computers, where the subjects rated a set of perceptual material qualities under different stimuli. The first experiment covers a full collection of materials in different presentations (visual, auditory and audio-visual). As a point of reference, subjects also performed all ratings on physical material samples. A key result of this experiment is that auditory cues strongly benefit the perception of certain qualities that are of a tactile nature (like “hard–soft”, “rough–smooth”). The follow-up experiment demonstrates that, to a certain extent, audio cues can also be transferred to other materials, exaggerating or attenuating some of their perceived qualities. From these results, we conclude that a multimodal approach, and in particular the inclusion of sound, can greatly enhance the digital communication of material properties.

Keywords: multisensory perception and integration, visual psychphysics

Images

Download Paper

Download Paper

Bibtex

@INPROCEEDINGS{martin2015,
     author = {Mart{\'i}n, Rodrigo and Iseringhausen, Julian and Weinmann, Michael and Hullin, Matthias B.},
      pages = {33--40},
      title = {Multimodal Perception of Material Properties},
  booktitle = {ACM SIGGRAPH Symposium on Applied Perception},
     series = {SAP '15},
       year = {2015},
      month = sep,
  publisher = {ACM},
   location = {T{\"u}bingen, Germany},
    address = {New York, NY, USA},
   keywords = {multisensory perception and integration, visual psychphysics},
   abstract = {The human ability to perceive materials and their properties is a very intricate multisensory skill
               and as such not only an intriguing research subject, but also an immense challenge when creating
               realistic virtual  presentations of materials.
               In this paper, our goal is to learn about how the visual and auditory channels contribute to our
               perception of characteristic material parameters. At the center of our work are two psychophysical
               experiments performed on tablet computers, where the subjects rated a set of perceptual material
               qualities under different stimuli. The first experiment covers a full collection of materials in
               different presentations (visual, auditory and audio-visual). As a point of reference, subjects also
               performed all ratings on physical material samples.
               A key result of this experiment is that auditory cues strongly benefit the perception of certain
               qualities that are of a tactile nature (like ``hard--soft'', ``rough--smooth''). The follow-up
               experiment demonstrates that, to a certain extent, audio cues can also be transferred to other
               materials, exaggerating or attenuating some of their perceived qualities. From these results, we
               conclude that a multimodal approach, and in particular the inclusion of sound, can greatly enhance
               the digital communication of material properties.},
       isbn = {978-1-4503-3812-7},
        url = {http://doi.acm.org/10.1145/2804408.2804420},
        doi = {10.1145/2804408.2804420}
}