Augmented Reality: The What-If Game

Most of us have had experiences playing with traditional playsets. In fact, many of us have fond memories and attachments to particular figurines or characters. Reminiscing about these experiences may bring up memories of your childhood imagination, your love to create stories and explore imaginary worlds, or even a desire to share these experiences with another person. If you’re like the two of us, you might have a couple cherished pieces sitting in a box somewhere hoping that one day someone will find them as meaningful as you do.

Let’s play the “what if” game for a minute. What if your playset knew you were playing with it? What if there was a “magic window” you could look through to see your playthings “come to life”? What if that character you just moved around could dynamically react to the scenarios you set up, talk to you, ask you questions, and encourage you to engage in other creative ways?

Sesame Workshop is asking the same questions by merging augmented reality with traditional playsets.  Last fall, Sesame Workshop’s new Content Innovation Lab partnered with Qualcomm to explore how augmented reality technology can support and enhance pretend play.  The Sesame Street Prototype Playset, (see a video of the game here) unveiled at the 2012 Consumer Electronics Show, links mobile devices to a traditional playset by digitally bringing physical playsets to life. Using Qualcomm’s Vuforia augmented reality platform, children can hold up a tablet or smartphone to a Sesame Street playset and engage with their favorite Sesame Street characters through digitally supported pretend play.  For example, when a tablet detects a child’s Ernie figurine, Ernie will look at the child and say, “Hiya!  It’s great to see you!  Know what?  I’ve got a great idea!  Take all this stuff here, the floor mats, the furniture and your Sesame Street friends, and use them while you tell your story!”

The experience currently supports two modes of play: free play (where children can make up their own story) and guided, goal-directed play (where Ernie or other characters prompt children to take particular actions in an existing story or game). We know that both kinds of play—child-directed and guided—serve an important function in child development. Play enables children to learn about themselves, their environment, and those around them. It helps children learn to solve problems and get along with others.

Sesame Workshop’s Education and Research department and Content Innovation Lab are exploring the intersection of augmented reality and play. True to the Sesame Workshop model, we utilized formative research throughout the prototype development. Our initial research explored how children create stories through imaginative play. We sat in a room, on the floor to be exact, for hours playing with children and a basic playset consisting of classic 1970’s Sesame Muppet figurines, a wide selection of playset furniture, and a range of 8” x 10” mats that served as the footprints of rooms. After a day of “testing,” we realized through reviewing the data and video footage that there were trends developing in how we, as adults, scaffolded and encouraged children to describe their imaginative play. Taking the notes from formative research, we adapted the prompts used in a simple play setting to scaffold and structure our script, game logic, and timeouts for the prototype.

Weeks later we found ourselves in the same room, sitting on the same floor, watching children engage with a new iteration of the original playset featuring an early build of the augmented reality application. As with most novel experiences, the reactions elicited were priceless. A typical scene was a child peering over the top of the tablet to see if the playset was really coming to life. We confirmed that our script was on target, but as with any formative testing, we discovered areas that needed extra support. We also realized that various technical components (such as the weight of the tablet) drastically change the play experience for the child. We made adjustments by testing on a smaller device and editing our script and prompts, and learned new ways to support children in their explorative play. One thing we did confirm was that children enjoyed and were captivated by the innovative experience.

Going forward, we are excited to see where our “what if” question takes us.