Skip to the main content

Click for FAST Consulting's home page
FAST Consulting
FAST Consulting Home Web UIs Desktop UIs Services Websites About Our Team Amusements
FAST Consulting

Preface to the Web Application Design Handbook

Too Many Graphic Elements to Show Them All

Sounds & Graphics:
Using Sounds & Graphics in Applications
Yeah, I Hear You: Workshop Results
FAST Consulting

You are here: Home ~ Web UIs ~ Yeah, I Hear You

Yeah, I hear you: Why aren’t there more sounds and graphics in our applications?

Workshop/Advanced Topic Summary
UPA Conference, June 7-11, 2004

By Alice Preston and Susan Fowler

The Tip of the Iceberg

This page presents a summary of the work done by the attendees (the authors and Julie Bzostek, Millicent Cooley, and John Flowers) of a 2004 UPA workshop, “Yeah, I Hear You. Why Aren’t There More Sounds and Graphics in Our Interfaces?”

The workshop collected information on the use of graphics and sound in complex, data-intensive, and mission-critical designs. The participants shared information about the neuropsychology of visualization and auralization; when multiple media are and are not useful; and the challenges of adding multimedia to applications.

When Alice Preston and I proposed the workshop, we expected to collect information about using sound in interfaces and then create a set of guidelines. Instead, the workshop members convinced us that it was far too early for guidelines, that trying to come up with rules at this point would cut off experimentation and constrain rather than expand our knowledge.

John Flowers added that, in addition to being far too early for guidelines, "an equally important impediment to making a concise set of guidelines is that guidelines are task- and application-dependent. For example, principles for good design of displays for online patient monitoring may be quite different from those for designing auditory displays for exploratory data analysis, and principles for designing effective auditory alarms may be very different from the previous two. While there are some general principles of auditory perception and attention that may be important across most all tasks and applications, there will be design considerations unique to specific tasks."

So, instead, we described, half jokingly and half seriously, the knowledge dissemination “iceberg”:

  1. Initially, academics do research and publish their results in peer-reviewed journals.
  2. Other researchers start writing literature reviews, summarizing and comparing research areas.
  3. Professors like Ben Shneiderman begin to publish instructional digests for graduate students, who then do more research.
  4. Articles start to appear in professional journals and trade magazines.
  5. Authors write trade books on the topic: "How To Create Sound User Interfaces," "SUI Design Handbook," "SUI Bloopers," etc.
  6. Finally, "Sound Interfaces for Dummies" appears.

Sound interface design is still only at the top of the iceberg, no lower than number 2. A possible exception is Susan Weinschenk’s Designing Effective Speech Interfaces (Feb. 2000), which covers verbal interfaces. However, our concentration during the workshop was on non-speech sound.

Go to top of page

Sound Categories

Once we discovered how far away we were from a set of guidelines, we decided to at least collect and categorize examples of sounds used in software and other artifacts. We came up with these categories:

Data Presentation

Assistive Technology

Tangible Products


Synthesized Sonic Feedback


Eyes-Busy, Hands-Busy Applications


Note that these categories overlap, so any particular application may actually fall into more than one. For example, the BMW car door is both a tangible product and an example of branding; speech applications may fall into both assistive technologies and eyes-busy/hands-busy applications; and so on.

Following are representative (and by no means exhaustive) examples of papers and samples in the various categories. We have also added sections on tools for communicating about sound; auditory software tools; and references.

Go to top of page

Data Presentation

Also called “sonification of complex data," displaying charts and graphs with sound.

Go to top of page

Assistive Technology

Assistive technology overlaps with almost all of the other categories. Also, this list isn't restricted to aural technologies.

Go to top of page

Tangible, Ambient Intelligent Products


Go to top of page

Synthesized Sonic Feedback

  • Formerly real feedback: Bubble noise of coffeemakers, cell-phone rings, the "film advancing" sound that was added to many digital cameras. (Note: On the plane home, someone had his cell phone ringer set to sound like a "real" phone—everyone turned around to look. SLF.)
  • Totally synthetic feedback: earcons and auditory icons. See Chapter 6, "Auditory Icons," in William Buxton, Sara Bly, William Gaver, 1994, Auditory Interfaces: The Use of Non-Speech Audio at the Interface.


Eyes-Busy and Hands-Busy Applications


Some companies design the sounds of their products or services. Here are some examples.

Go to top of page

Tools for Communicating about Sound

During the workshop, Millicent Cooley pointed out that people rarely have a vocabulary with which they can talk about sounds. Customers may decide against including sounds in a website or product if they cannot communicate effectively about them.

So that one of her project teams could listen to and make decisions about sounds, she developed a sound map that she presented to the workshop. She also demonstrated how sound can change one's perception of visuals. See

Tools for Making Sounds

Csound from MIT, freeware: see

Musoft Builder: A Musical Generator that creates music and MIDI files from fractals, pictures, text and numbers.

Sonification Sandbox, Bruce Walker, Georgia Institute of Technology.

Sphinx-4 Java open-source speech recognition system:

vOICe Mapping, a Java applet for presenting visual items with sounds:

Go to top of page

Additional References

Arrabito, G.R., Mondor, T.A., Kent, K.J. (2004). Judging the urgency of non-verbal auditory alarms: a case study. Ergonomics, 47(8), 821-840.

Buxton, William, Sara Bly, William Gaver. 1994. Auditory Interfaces: The Use of Non-Speech Audio at the Interface (online book).

Chion, Michel, Audio-Vision: Sound on Screen, Columbia University Press, 1994.

Edworthy, J., & Stanton, N. (1995). A user-centred approach to the design and evaluation of auditory warning signals; methodology. Ergonomics, 38, 2262-2280.

Edworthy, J. (1994). The design and implementation of non-verbal auditory warnings. Applied Ergonomics, 25, 202-210.

Flowers, J. H., Whitwer, L. E., Grafel, D. C., & Kotan, C. A. 2001. Sonification of daily weather records: Issues of perception, attention and memory in design choices. Proceedings of the 2001 International Conference on Auditory Display, 222-226.

Flowers, J. H. & Grafel, D. C. 2002. Perception of sonified daily weather records. Proceedings of the Human Factors and Ergonomic Society 46th Annual Meeting, 2002, 1579-1583.

Gaver, W.W. (1993). What in the world do we hear? An ecological approach to auditory source perception. Ecological Psychology, 5(1), 1-29.

Gaver, W.W. (1997). Auditory interfaces. In: Helander, M.G., Landauer, T.K., and Prabhu, P. (eds.). Handbook of Human-Computer Interaction, 2nd edition. Amsterdam: Elsevier Science.

Kramer, Geoffrey (ed.). Auditory Display: Sonification, Audification, and Auditory Interfaces. Santa Fe Institute Studies in the Sciences of Complexity, Proc. Vol. XVIII. Reading, MA: Addison-Wesley, 1994.

Kramer, Geoffrey (ed.), et al. 1997. Sonification Report: Status of the Field and Research Agenda. National Science Foundation.

Vickers, Paul. Undated. Lexicon of Auditory Display.

And a book by one of the workshop attendees (Jettie. Hoonhout): Zwaga, H. J. G., Theo Boersema, Henriette C. M. Hoonhout, Visual Information for Everyday Use: Design and Research Perspectives, CRC Press, 1998.

Go to top of page

Your Comments and Questions

Please write to us and we'll add your comments to this page.

Go to top of page


FAST Consulting
FAST Consulting Home | Site Map | Accessibility | Web UIs | Desktop UIs | Services | Websites | About Our Team | Amusements
FAST Consulting
© 2008 FAST Consulting All FAST ConsultingRights Reserved Privacy Statement