Designing With Audio: What Is Sound Good For?

Advertisement

Our world is getting louder. Consider all the beeps and bops from your smartphone that alert you that something is happening, and all the feedback from your appliances when your toast is ready or your oven is heated, and when Siri responds to a question you’ve posed.

Today our technology is expressing itself with sound, and, as interaction designers, we need to consider how to deliberately design with audio to create harmony rather than cacophony. The cacophony is beautifully captured in Chris Crutchfield’s video1, in which he interprets the experience of receiving email, SMS texts, phone calls, Facebook messages and tweets all at the same time:

In this article, we’ll explore some of the uses of audio, where we might find it and when it is useful. This is meant not as a tutorial but rather as a discussion of some basics on using audio feedback.

Audio is a form of feedback that can be used either in combination with other forms, such as haptics, visual displays and LEDs, or on its own. We have to weigh several factors when designing feedback mechanisms: the scenario, the device and the interaction, where and how the device will be used, whether the user has a screen or display, whether the device has physical buttons or a touchscreen, where the user is relative to the device, and so on.

For every action of the user, a good experience will include feedback that the action has been registered; for example, pressing a number key on a mobile phone would play a sound and show the number being pressed. Audio is particularly useful when there is no screen or when looking at the screen is not possible or not desirable (such as when users want to multitask). It’s interactive, creating a dialog with the user. It is also particularly good at providing feedback as “shared audio,” a form of feedback that reaches multiple people at once, such as a PA system or a citywide emergency warning system.

Audio is not always warranted. Something that makes noise repeatedly when other feedback would suffice is annoying. Audio that is private and intended for you but is heard by others is embarrassing, such as when your phone rings and announces a “Call from Sexy Neighbor.” Audio design has many ins and outs, but let’s start with some common uses of audio feedback.

Where We Find Audio

Many of us who work on interaction, mobile, device or game design have already discovered the importance of designing audio — audio is everywhere.

Ear buds2
(Image credit: Fey Ilyas3)

Mobile

Much of the Web is moving to mobile, which of course entails smaller screens and people on the go. But besides creating mobile-specific websites, there are ways to augment the mobile experience with audio when people aren’t looking at or can’t interact with the screen. A great example is GPS and turn-by-turn navigation systems that speak directions (either as part of a dedicated device or from a smartphone app). While audio isn’t yet native to mobile websites and apps, it is native to smartphones to indicate new email, incoming text messages and calendar events.

Gaming

For those who play video games, audio is integral to setting the mood, environment and situation, and it engages the user tremendously. First-person shooter games such as Halo and Call of Duty rely on audio feedback to show cause and effect — for example, the sound of a gun shooting and the moment of impact on the enemy. Or consider Wii Sports: the smash of the ball in tennis, the crack of the bat in baseball, and the cheer of fans all help to blur the line between the very physical game and the digital world.

Consumer Devices

As more appliances become smarter and connected, they might have more to say. Today a set of beeps tells you that the refrigerator door is open, but in the future you can expect notifications that the milk has gone bad or that you need to pick up eggs if you want to make that cake for your spouse’s birthday on Tuesday.

More and more of our everyday devices use audio feedback: a Bluetooth headset tells you who is calling, Nike+ tells you your current distance travelled and pace, and cars beep to help you park.

Speech Recognition and Robots

Voice interaction such as Siri’s is revolutionizing the way people interact with their iPhone and will help to change future interactions with all devices and information sources. People are beginning to talk to their devices and expect some audio feedback in return. Siri is just the start; we’re starting to see speech recognition in Xbox Kinect, Samsung TVs and more. Audio feedback is a natural way to let the user know that the system or device has heard them, is processing their request and so on.

Think of your favorite robots — HAL, Wall-E or any of the personal robotic devices that are emerging. These robots are developing human characteristics, with sounds being one of the strongest ways to deliver emotion. Leila Takayama4 of Willow Garage has talked about the “design challenge in communicating internal robot states and requests to effectively reach the robot’s assigned goals.” Willow Garage has created a set of sound libraries5 for communication between people and robots that might help make robots “more appealing.” Then there are other robots that speak English and other languages, such as the new Autom weight-loss coach. Studies have shown6 that people who use Autom stick with their diet and exercise routines for twice as long as people who use traditional weight-loss methods, perhaps partly because of its human-like interactions.

Why Use Audio?

There are numerous principles to determine why and when to use audio in designing interactions for devices. Being conscious about adding sound to a device is the first step in designing it right. The point is to do it deliberately, not as an afterthought, so that the audio means something and is not annoying. Here are some of the many scenarios in which you should consider using audio.

Instructions and Information

Audio is used to give instructions, especially where there is no screen or where looking at a screen would be difficult, unsafe or impossible. Again, think turn-by-turn directions. Or it can be used to augment visuals. The parking machine below obviously has visual instructions for entering a credit card, but they weren’t sufficient to get people to enter it correctly.

Audio can be used to offer information, either when no screen is available or when certain details would be better captured as audio. The Jambox by Jawbone7 tells the user when they need to recharge the battery. The Leapfrog LeapPad takes this one step further by specifying the type of batteries it needs!

Feedback and Interaction

As mentioned, audio is used as a feedback mechanism when the user takes action. This could be feedback for when the user pushes a button, such as when turning on a Jambox speaker, or to tell a driver that they are getting close to a parked car.

It’s also used to allow for interaction and conversation with our devices. We’re used to interacting with speech-recognition systems when we call an airline or a bank, and now sending a text message with your voice from a Windows phone is just as easy. The audio from these services and devices create a dialog that enables users to get things done.

Personalization and Customization

Audio allows for personalization of a device, helping to engage users and create an emotional attachment. Siri learns its user’s name and uses it in its replies, adding a personal connection to the interaction. Garmin and TomTom let users download all kinds of voices to their GPS devices, from Bert and Ernie to Star Wars characters to Kitt from Knight Rider8, with the goal of creating more engaging experiences. Jawbone device owners can download different languages and characters to their Bluetooth headset and speaker, with the device volunteering such responses as, “A bombshell is whispering in my ear… And yes, I’m blushing.”

Audio also helps to establish personality and to humanize a device. Ford and other electric vehicle manufacturers are dealing with a proposed bill9 that would require electric vehicles to make some sound to ensure pedestrian safety. Ford has asked the public to vote on four different sounds that would essentially shape the personality of its cars. Here’s one of them:

In another example, to show the power of talking devices, Radio Lab reported10 on an experiment that pitted a hamster against a Barbie doll and a Furby (the popular furry electronic talking robot) to see how long kids could hold each of them upside down. While all of the five kids in the experiment could hold Barbie upside down “almost forever,” they treated Furby much more like the living hamster than the Barbie. Why? Well, when you hold Furby upside down, he says, “Me scared,” giving human-like characteristics to the toy. The kids said afterwards that they “didn’t want him to be scared.”

Conclusion

Audio is everywhere, and there are good reasons to use it: to instruct, enhance and engage and to personalize experiences. But if poorly designed or used inappropriately, it can detract from the experience and be annoying. We’ve covered the why and the where of audio. Next time, we’ll review some guidelines and principles on the ins and out of designing with audio.

(al)

↑ Back to topShare on Twitter

Karen Kaushansky is a Principal Device Interaction Designer at Jawbone where she creates rich interactive experiences for consumer devices. Karen is formerly of Microsoft/Tellme and is a 14 year veteran of the speech recognition industry. Over the years, her work has spanned from traditional phone-based speech recognition applications, to voice biometrics, to multimodal experiences. You can connect with her on twitter.

  1. 1

    Great article and audio definitely has been underused in the past but now as we got the multi tasking down and see the proliferation of portable devices – audio will become more and more important.

    An awesome keynote that I saw at SXSW by Amber Case talked about just that, using audio to inform and to liberate us and explore our environment a little bit more. So for example actually looking at the landmarks and where you’re going instead of staring at the small mobile screen all the time.

    Alternatively as mentioned in the GPS example – using audio to control the device, such as siri commands to your iPhone is much better than fiddling around with small phone controls in the car. A concern with the latter though is having speech recognition. Although I have to say it has been getting pretty darn good as of late.

    1
  2. 2

    Don’t forget where possible audio should not be used as a sole identifier of instruction or action. Developers will be familiar if building to WCAG2 guidelines.

    http://www.w3.org/TR/WCAG/#content-structure-separation-understanding

    Remember to consider assistive technology when building; don’t want unwanted sounds playing over screen reader voices.

    0
  3. 3

    One of the key points in this article is that, if poorly used, it is VERY distracting. Audio is one of those design elements that should ONLY be used if it has a positive effect. When used correctly, however, it can be a very useful way to enhance a design. Good article, I hope to see designers come up with creative and helpful ways to introduce audio into their creations.

    0
  4. 4

    Martin Gonzales

    April 18, 2012 1:10 pm

    I still remember that back in the days, having songs or music on a website was so cool. Today, not anymore – and I tend to close these websites with music because it’s irritating as it interferes with my music player.

    0
  5. 5

    Great way of using sounds in design ;)

    http://www.youtube.com/watch?v=sXqXpwyBI1k

    0
    • 6

      I completely agree. It may not be entirely necessary, but the quality and selection of the sounds in this particular example are pleasing and offer even more “feedback” that a touch screen by itself lacks. Touch animations, etc. are nice, but when you think about it, one of the most satisfying sounds we hear in day-to-day use is the sound of physical keyboards.

      On an iOS device, the keyboard ‘clicks’ when you type, and to me, it’s super satisfying to hear that connection with the key that I pressed.

      -1
  6. 7

    There’s a website that I often (have to) visit, which uses sound when you hover over the menu. Very annoying, especially if you are listening to music while browsing.
    I guess audio can be a good addition to websites, but it has to be thought out well. Not all websites will benefit from it.

    0
  7. 8

    Great article and it is awesome to think that maybe my distinction for creative music production may come in handy for web design now! yey to me!

    0
  8. 9

    Adonis Raul Raduca

    April 23, 2012 4:19 am

    Very nice article !

    I think this kind or sound based/augmented interaction is not for the web in general, we can find the explanation in the people browsing behavior. Almost all the time people are using multiple tabs, also beside the browser they have some other applications opened, they can also browse at work when they must be productive, and so on …
    But for different home appliances or small gadgets, a well designed audio experience add a new dimension of expression in terms of user interaction. Thats open the gates for a better user experience.

    By the way
    I’m looking for a good audio tool to produce such kind UI sounds.
    Since I’m not an audio expert, I’m looking a tool with a simple interface focused on this particular task.
    Or maybe a book for beginners regarding UI sounds creation, since there are a lot about visual stuff but almost no one for audio.
    Any idea ?

    0
    • 10

      Karen Kaushansky

      April 25, 2012 8:00 am

      Exactly right – using sound to enhance the experience is not focused on the web where there are rich visuals, though there are situations where audio can enhance visuals (especially mobile web).

      I haven’t found a really good source for UI sounds creation, but am sifting through some possibilities as I write the next article. Will keep you posted!

      0
  9. 11

    Great article although I think you missed one of the greatest advantages with sound, namely providing users with an alternative way to access the content of a website and/or app. I think a great example of this is how Kidshealth.org has added speech to their website providing kids with for example reading or learning disabilities with another option besides being able to read the text. Another great example would be if a newspaper made it possible for me to listen to article while cooking food or driving.

    0
    • 12

      Karen Kaushansky

      April 25, 2012 8:04 am

      Great point! There are definitely services out there for mobile phones that do read news headlines, weather, etc… for people who would rather hear this kind of information instead of reading it. I’ll be including some examples in the next article.

      0
  10. 13

    Catherine Khetagurova

    May 1, 2012 11:13 am

    I agree that the audio is firmly part of our lives, and often does help to better understand the information. This is especially important if the person has poor vision or better perceive information at the hearing. However, in my experience, the residents of cities are increasingly getting rid of extra sounds – including vibration signal on the phone, turn off the sound signals to household appliances, etc. The abundance of sound becomes annoying and becomes a common cause of stress.
    Nevertheless, the presence of, for example, the audio version of the website or audio e-book several times increases site traffic and sales. The main thing – do not forget that the user or reader should be able to turn off the audio or customize it to fit your requirements.

    0
  11. 14

    how can i add audio to mobile devices?

    1

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top