Converting Our Stories Into Multi-Screen Experiences

Advertisement

Storytelling takes many forms. In the past, stories were told orally, with people telling and retelling myths, fables and even histories. As writing technology became more prevalent, we began to record our stories, and we told them in the pages of books. Now, our society is awash in different devices and technologies, and those traditions of spoken stories and printed stories are blurring.

Multi-screen narratives are being told across all kinds of platforms, pages and devices, making for truly immersive experiences. We are watching them, tapping them and learning from them. They immerse us in the storyteller’s world. This article outlines what I believe are the five essentials of telling multi-screen stories.

How I Fell In Love With Interactive Storytelling

First, a little background. My childhood was spent in Nigeria, West Africa. I am a member of the Tiv tribe, a group of about 6 million people clustered in Nigeria’s Benue River Valley. As a child, I heard a lot of Nigerian folktales, about animals, humans and even magic. In Nigerian narrative tradition, stories are often told orally, in front of a gathered audience. During festivals and cultural events, men even dress up in elaborate costumes and perform stories for the crowds. I have vivid memories of these stories and have always been curious about how they could be translated into something digital and interactive.

kwagh-hir-nigerian-masquerade
The Kwagh Hir, or Thing of Magic, my tribe’s largest cultural festival (Image: Naptu2)

Those fables are a piece of my cultural inheritance. They always seemed to contain essential truths about humans. Take the story of the Ear and the Mosquito. One day, the Ear steals food from the Mosquito and refuses to pay it back. In anger, the Mosquito visits the Ear every evening, demanding the food to be returned, annoying Ear all night with his buzzing. It’s an old tale, with many versions, but the moral is consistent: don’t steal from your friends.

Creating modern, interactive versions of these stories is possible, but how exactly do we do that? Let’s begin by talking about what I mean by the word “multi-screen.”

A Bit About Context And Screens

When speaking about multi-screen storytelling, remember that screens have different contexts, not only different capabilities. The same screen on which you carelessly watch videos at home becomes a closely guarded viewport when you’re watching a movie on a crowded train. The context in which people view stories is more important than the device’s specifications. When we tell interactive stories, we need to be aware of this, and embrace it.

I like to focus on the following screens:

  • Sensors (Twine, GPS, Arduino, motion detectors, etc.)
  • Mobiles and tablets (phones, tablets, laptops and everything in between)
  • Flat-screens (desktops, TVs, etc.)
  • Public and immersive displays (store kiosks, large stadium screens, projectors, Kinects, etc.)

Not all of these need to be used at the same time, because they won’t all be appropriate to the story you are telling. Context is extremely important.

Now, as promised, here are the five essentials of multi-screen storytelling.

1. Divide Your Story Into Separate Content Blocks

When we create multi-screen narratives, we need to find natural breakpoints in the story, places where the visual or narrative content can easily be separated. This enables us to deliver different segments to different devices, in different contexts.

Kolobok is a Slavic children’s story about the adventures of a round yellow cake. For the Moscow International Festival, a large team of designers and animators from SilaSveta Studio incorporated it into a truly fascinating demonstration of storytelling. Before the show, the team set up a large touchscreen at the children’s height. With their hands, the kids could manipulate parts of the animation by adding movement and color.

kolobok-story-on-touchscreen
A public display for children to play with

For the show itself, the full story was projected onto the facade of a large building, allowing the crowd of adults and their children to watch the narrative unfold. Along with sound, it made for another discrete content block, one that closely resembled a 3-D movie.

kolobok-story-on-building
The full animated story in front of a large festival crowd

While the touchscreen and the movie were different views of the same content, they could exist as independent pieces and did not have to appear next to each other. The SilaSveta team found the natural breakpoints in its story and created two separate visual experiences to match them.

Questions to Ask

  • Where are the breakpoints in the story? Divide your content so that it makes sense in context. The practice of responsive design gives us numerous guidelines on how to do this.
  • Can those content blocks exist independently? Sometimes, the answer is no, but it depends on the story. In the Kolobok example above, they can. In other interactive stories, such as Snow Fall from the New York Times, the blocks are chapters in a single story and should be kept together.

2. Offer People Multiple Perspectives

Bear 71 is an award-winning multi-screen experience created by Jeremy Mendes and Leanne Allison. The creators tell the story of a bear living in Banff National Park in Canada. It feels like a cross between a role-playing game and a TV documentary, and as a linear narrative with a non-linear interface, it works beautifully.

bear71-story-site
The Bear 71 story is told in a highly abstracted interface.

Multiple viewpoints are accessible. Online, you roam in a stylized landscape, watch crittercam footage from the forest, and otherwise live as bears do — freely. Even though it may look like a game interface, you are not so much “playing” as you are participating in a story. Watching real crittercam footage, you see what the forest silently sees. You also have the option to turn on your webcam (“stealth mode”) to see other users around the world, all watching the same story online.

bear71-AR-installation

During shows and installations, the team responsible for Bear 71 set up augmented-reality applications and motion-detection cameras, so that visitors could experience what it’s like to have their pictures taken with one. By playing with the augmented-reality apps and the motion-detection cameras at the installations, users got a bit of the same physical experience that the bears had.

Questions to Ask

  • Does the narrative change when viewed from a different perspective? A variety of perspectives can make a narrative much more fascinating. Bear 71 forces us to see the world first from the bear’s perspective and to sympathize with its loss of habitat, but other viewpoints take a slightly different angle. The voyeurism common in our digital sharing culture takes on a different meaning when used for animal surveillance.
  • What data sets can be used in the narrative? Bear 71 cleverly combines crittercam video, GPS data, cell tower data, augmented reality, and topographical data. The photographs of visitors to the installation provide an additional emotional layer of data. The data we bring into our stories helps to define additional viewpoints and characters.

3. Redefine A Tradition

As Western culture has moved more deeply into Nigeria, Nigeria’s traditions are weakening. I wanted to take a piece of my culture and put it in the cloud, instead of leaving it locked in the heads of our oral storytellers. That meant redefining how the stories are relayed, how they are saved and, most importantly, what messages they convey to the audience.

In 2011, I started a project named Pixel Fable in which I take those traditional stories and reinterpret them online. In essence, I’m creating an interactive archive of Nigerian stories. As mentioned earlier, the oral histories of Nigerians are rich, but capturing them and translating them into digital stories means they will reach a wider audience. About 25% of my website’s visitors come from the US, while another 25% come from Japan. Canada, France and Germany also send a fair amount of traffic.

Pixel Fable uses responsive websites, iOS apps and augmented-reality animations to reinterpret Nigeria’s oral history.

pixel-fable-logo
An introductory screen from “Cricket and Mud Brick,” a new Pixel Fable story built with the Tapestry app

I’ve relied on two primary contexts to reinterpret the old Nigerian storytelling tradition. The first is people on their tablets and phones, clicking on and reading the stories. The spread of mobile devices makes this inevitable — why not tell African fables in a more accessible context? The second is my attempt to update the moral lessons for our modern age. While the original story of the Ear and the Mosquito may be a funny tale about annoying insects, the lesson can be updated to speak about how mosquitoes spread malaria in Nigeria. There’s room to redefine our old myths for the 21st century.

Questions to Ask

  • Will people love or hate the reimagined version? Not every fable or myth can (or should) be recreated digitally. However, if people have an emotional reaction to a story that you have designed and pushed out to multiple screens, that is usually a good sign.
  • If people talk about your narrative, will it bring about change in society? Each Pixel Fable story has a message. Most fables do. Some revolve around love triangles, others around the wisdom of elders, and there’s even one about why you shouldn’t get angry at your friends. They are small messages, but put together, they force us to reconsider how we treat the narrative history of people in Nigeria and West Africa.

4. Immerse People In The Narrative

The Walking Dead, the famous comic and now TV show, used a polling Web app (AMC’s Story Sync, if you like marketing-speak) to ask viewers questions and show related content as an episode was being broadcast on TV. While the app was simply timed to each scene, it was an experiment in multi-screen storytelling that invited audience participation, not just audience attention. Polling has a gimmicky feel to it, but that probably came about as a result of Hollywood pressure and doesn’t reflect the value of the concept in general.

walking-dead-story-sync
Polling and syncing apps extend narratives from the TV to the couch.

The creators also added mobile gaming to the mix, bringing viewers “into” the story in a completely different way, in different contexts.

All of these facets of each story’s arc enabled people to immerse themselves in this apocalyptic narrative. Jason Spero of Google notes the need for a seamless experience as users move between devices. Other people, however, say that a second-screen experience can be extremely distracting, forcing viewers to miss key parts of the TV show. It is the opposite of an immersive experience, they say, and is confusing to use. In my opinion, each content block should work independently to avoid putting users in this position.

Questions to Ask

  • Will people forget where they are? I’ll be the first to admit that this is not always a good thing. I can’t count the number of times that I’ve almost missed my train stop because my head was buried in my phone. Context, not only device capability, is key. Do you want users to get lost in the story or just engage in a manageable chunk?
  • Do the screens you have chosen feel natural? By this, I’m not referring to pixel density. That is simply impossible to control, and if the story if good, it won’t matter anyway. The screens that people choose will depend a lot on the tasks they want to complete, so make your story feel natural for whatever content block they are interacting with.

5. Design Contextual Interactions

Recently, a number of storytelling apps have relied on location to serve additional content, much in the same way that Foursquare or Google Maps do. The Silent History is a dystopian science-fiction story about children who do not speak. The iPad app contains the whole story, but by visiting certain geo-tagged locations, users can access additional content.

silent-history-story

For a novel about children all over the US, inviting readers to physically go to where the story’s kids are makes perfect sense. The additional contextual interaction makes the story more layered and thought-provoking, in a way that a simple app would not be.

We use map data every day, to look for restaurants, check the weather, see road conditions and even check for public transit delays. Other contextual interactions make sense when creating digital stories: taking photographs, texting, sharing and saving information, even body motion. Use these, along with your UI and UX skill sets, to devise new storytelling methods.

Questions to Ask

  • Does the device matter? With the rise of responsive design workflows, our content should not be device-dependent. Some things, however, such as camera or GPS functionality, may be integral to a part of your story, and so the device would need to be factored in.
  • Should the interface be designed as a seamless part of the narrative? As people who work on the Web, we really have a strength here. If we choose to make the interface part of the story, then we can rely on our experience in building websites and content management systems.
  • Will your story “remember” anything? As a child, I folded over the top corner of the page when I had to put a book down. It was a simple way to keep my place. With a narrative split across multiple devices, it might be necessary to design an interaction that flags where you’ve gotten to and then returns you there when you visit again. That all depends on the content, but the question does need to be asked. Everyone hates losing their place on the Internet and having to navigate back, so perhaps we should enable a memory in our stories as well.

Conclusion

We have conceptualized different uses of multiple screens to tell stories. All of us, from every corner of the globe, have intensely rich cultures filled with stories and fables. Using them to create interactive narratives is another way to explore the power of the Web, to wow people and to record our cultural history.

I would love to see what you come up with, or hear about other examples of clever digital storytelling.

(al) (ea)

↑ Back to top

Senongo is a Nigerian-American designer and illustrator in New York City. He designs and builds digital experiences, and is the founder of Pixel Fable, an interactive archive of African stories.

  1. 1

    Daniel Schwarz

    May 13, 2013 9:34 am

    Senongo,

    I was really inspired by your beautiful story, and the way you told it. I think the way that you’ve explored the web has only been seen in a vision by most creatives, and when we decide to adopt these ideas, it truly will be a new era of digital design.

    I had this idea of “painting” the imagery of poems in an interactive way, but this was years ago and the technology just wasn’t as advanced then. I’m glad to see that it is now, and that somebody is doing it.

    -Daniel (Founder of Airwalk Design).

    2
    • 2

      David,
      Thank you for the kind words. There are a lot of artists who talk about synesthesia- perhaps your idea of painting the imagery behind words alludes to that? The technology will always be a little behind what designers and artists want to do- I would encourage you to take a shot at developing the “imagery of poems”, as you put it. It would be very cool to see.

      2
  2. 4

    Very happy to see this kind of post on Smashing Mag. I work in webdocumentary and love to explore its relationship with webdesign, always though it would make good posts material for SM. Thank you Senongo!

    1
    • 5

      Jessic,
      The example of Bear71 was a particularly interesting one for the documentary format. I Love Your Work by Jonathan Harris is another one, although the subject matter is decidedly more adult. Thanks for reading and commenting!

      0
  3. 6

    Wonderful work. Really appreciate your approach to outlining an actual methodology and framework. Hope to see more. Again, kudos.

    2
    • 7

      James,
      Though this touched mainly on the multi-screen aspects of storytelling, I think there is still room to talk about the construction of narratives themselves, with foreshadowing, character development, and story climaxes/anti-climaxes. Although the music is not for everyone, you might look at storytelling in hip hop for great examples of constructed narratives- Slick Rick, Masta Ace, or Redman are three artists that immediately come to mind.

      0
  4. 8

    Hi Senongo,

    Wonderful article.
    As a interaction designer and comic artist I like it when articles like these appear on this site.

    Right now I am toying with the idea of using interactive geographical maps to guide a reader through a story line. How it should work I can figure out on my own, but what do you use in terms of technical aids (code libraries, computer programmes, data resources etc.)?

    Also good luck with your awesome company!

    1
    • 9

      Hi Sanne,
      Your idea sounds really great. I dont have any specific resources to provide, other than to say you should look at openstreetmap, and Geoloqi from Amber Case and Aaron Parecki. There might be some cool stuff around those that developers have built.
      Good luck with the project- please keep us informed of how you get along. I’m sure the ideas you generate will be a great help to the community.

      0
  5. 10

    Great article,
    as a game designer, i think that sport practice could be really refreshed by this kind of interaction.
    What do you think about modern sport practice and gamification of life?
    Maybe those things could merge in a new experience called sportification of life, and this is obviously a multi-screen experience. In this case, real activity defines breakpoints and you can narrate in those breaks.
    Sports like running or triathlon are activities ready to be narrated from a new point of view, with g-glass or not, people want to try those experiences without leaving the sofa.
    Thanks Senongo

    1
    • 11

      Panwooders,
      Have you heard of the Rocket Racing League? It’s not physical activity, but perhaps provides/provided a model for crowd participation in competition?
      As for games in every day life, there is definitely a place for it. Humans play little word games, mental games with themselves and others all the time, like subtly competing with a stranger to see who will get to an elevator door first, that kind of thing.

      My only concern would be that the act of competition is quite mentally taxing for a lot of people, and the emphasis on gamification obscures the real seriousness of life! Maybe thats why Nike Fuelbands and the like are successful, because you play against yourself more than anything else?

      0
  6. 12

    Hi Senongo,

    Love the article, and find your efforts, to preserve our African oral technique in the digital ecosystem, to be quite commendable. I loved kwagh-hir and ki-kyagh (riddles), as a kid growing up in Africa, I still do… these cultural activities served other functions beyond entertaining as most fables do, they were seen as a moral compass for individuals in the society, they embodied some deep philosophies and could be considered highly educative… re-telling them in the digital medium, and across multiple screens is to be viewed as a neat innovation… kudos!

    0
    • 13

      Terfa,
      Msugh we. M fah zwa Tiv dedo ga! (for all you non-Tiv speakers: Hello. I dont speak Tiv that well!)

      The moral lessons have been somewhat difficult for me to pick up, but I think we can take these stories and update those lessons, rather than relying only on the old interpretations.

      Thanks for reading the article, and I hope you check back for more Pixel Fable stories this summer.

      0
      • 14

        Thank you Senongo, its alright if you don’t (I mean speak Tiv very well). This leads me to the part where you initially asked for suggestions on clever storytelling. Apart from screens, the language of the story also has a large impact on the way the story is conveyed, context can be misunderstood, so translation of such stories in multiple languages with preservation of context should also be noted.

        If a story was initially conceived in Tiv for instance and needs to be told to a French or English speaker (who doesn’t speak Tiv), these devices should carry embedded features or algorithms for language translation as well. I already know that work is being done on efficient translators that could take a story told in Chinese and tell it in German fluently. I promise to check back for more. Thank you once again.

        1
  7. 15

    Great article. Interaction design and good user experience are in need of learning from classic storytelling methods.

    0

Leave a Comment

Yay! You've decided to leave a comment. That's fantastic! Please keep in mind that comments are moderated and rel="nofollow" is in use. So, please do not use a spammy keyword or a domain as your name, or else it will be deleted. Let's have a personal and meaningful conversation instead. Thanks for dropping by!

↑ Back to top