As a speech-language pathologist, I am fascinated with how humans and voice interfaces (such as Amazon’s Echo, “Alexa”) communicate with each other. I was fortunate to be a member of a research team based at the University of Washington, in which we recruited 10 diverse families to incorporate an Amazon Echo Dot into their homes for the first time. As researchers, we wanted to learn and understand how families incorporate the fast-growing technology of dedicated, home-based, voice interfaces in their homes. None of the families we recruited had ever had a dedicated voice interface like the Echo or Echo Dot in their home before, and all of the families had children living in their homes.

During our study, we found that all 10 families experienced communication breakdowns with the Echo Dot (“Alexa”). A communication breakdown in these instances is when the human communication partner says something to Alexa, and Alexa misunderstands what was said, resulting in an inappropriate response or no response at all from Alexa. I used my background as a speech-language pathologist to analyze how communication between families and Alexa would breakdown, and why those breakdowns occurred.

A major concept in analyzing communication breakdowns is pragmatics. In the context of human communication interactions, pragmatic skills involve the social use of communication, such as choosing the right vocabulary for your communication partner. The ability to adjust your communication to be appropriate for the person you are communicating with is a form of code switching. For example, when a teenager talks to her friends, she will use vocabulary, sentence structures, and a tone of voice that is appropriate for her peers. In contrast, a teenager is likely to use a different vocabulary, sentence structure, and tone of voice when speaking to a teacher or a parent.

Our research shows that the voice interface our families used (whom I’ll refer to as “Alexa”) currently has problems with pragmatics, including code switching between children and adults. This may not be surprising since Alexa is not actually a person with a brain, but nonetheless, communication breakdowns with Alexa could be addressed, at least in part, if designers and developers could build Alexa’s communication skills in the area of pragmatics. In addition, Alexa has difficulty with assisting her human communication partners in repairing communication breakdowns. Alexa often provides a neutral response to communication breakdowns, such as “hmmm…I’m not sure” rather than acting on a misunderstanding or providing a specific response. Neutral responses do not provide any contextual cues about where the breakdown occurred, whereas when Alexa acts on misunderstood information, her communication partners can at least get some clues as to where the communication breakdown occurred. In our research paper, “Communication Breakdowns with Alexa” I describe how parents and children collaborate to repair communication breakdowns, and how Alexa’s responses impact families’ abilities to repair communication breakdowns successfully.

In this post, I’m going to delve into the concept of figurative language, words with multiple meanings, and code switching, and we’ll explore how important these concepts are for voice interfaces that interact with children.

To start, let’s take a look at an excerpt that we recorded from one of our families, in which we see how Alexa has difficulty with words that have multiple meanings. In this instance, the mother and child have tried out different requests with Alexa, and the mother urged the child to see if Alexa can speak in other languages.

Mother: Go on ask her. Ask her if she speaks Spanish.
Child: Alexa, count in Spanish.
Alexa: Count in Spanish is …contar. (word spoken in Spanish)

In this exchange between parent, child, and Alexa, we see how Alexa takes the child’s request literally, by translating the word “count” instead of performing the action “to count.” This demonstrates a concept we observed during our research study: Alexa has difficulty following directions which involve words with multiple meanings. Without additional context, Alexa has difficulty knowing which meaning to interpret in the command. However, this doesn’t stop Alexa from using words with multiple meanings herself, which also result in communication breakdowns.

Let’s consider the concept of code switching. When thinking of the different types of human communication partners Alexa was talking with in our research study, we need to think about the communication differences between children and adults. Children are still developing their language skills, and therefore figurative language and understanding how words may have multiple meanings can be challenging concepts for children, especially young children. Yet, in our study, we found that Alexa used figurative language, particularly when it came to jokes. Often, children in our study asked Alexa to tell jokes, and one child in particular asked Alexa to tell jokes throughout the entire four-week research period. Here’s an example of one of the jokes that Alexa told this child:

A man walks into a bar. Crank. It was a heavy metal bar.

Did you get the joke? How about this next one?

Why shouldn’t you tell a secret on a farm?
Because the potatoes have eyes and the corn has ears! And the beans stalk.

Both of these jokes were told to the same child during the course of our study. Do you think the child got these jokes? How about if I told you the child was four years old? When we interviewed this child at the end of our study, we asked if they understood the jokes that Alexa told, and why the jokes were funny. The child’s response was “No, I don’t know why.” This is a classic illustration of how lack of code switching can significantly impact communication interactions. In this case, Alexa’s jokes were not age-appropriate for young children, relying often on figurative language as a key element for humor—a form of humor that is not yet developed in young children (see the American Speech-Language-Hearing-Association’s Communication Reference Sheets for more details). Alexa failed to properly identify her communication partner (a child) and as a result, failed to adjust her communication so that she could be understood.

Just for fun, let’s think of how Alexa, as an entity, could improve her communication skills. I’ve created a communication plan for Alexa to help:

Goal: Alexa will correctly identify her communication partner at least 80% of the time.

• Objective 1: Alexa will identify if her communication partner is an adult or a child (approximately age 5 or under) based on fundamental frequency¹.
• Objective 2: Alexa will identify if her communication partner is an adult or a child based on sentence length.
• Objective 3: Alexa will identify if her communication partner is an adult or a child based on key words used.

Goal: Alexa will provide appropriate contextual cues to her communication partner when a communication breakdown occurs at least 90% of the time.

• Objective 1: Alexa will state when the request was too long for her to understand.
• Objective 2: Alexa will request clarification if the topic of the request is not known.
• Objective 3: When Alexa is less than 60% certain of the human’s request, Alexa will repeat the information she understood, and ask her communication partner if this is what they said.
• Objective 4: Alexa will provide a specific response, or act on a misunderstanding, whenever possible to assist her human communication partner in identifying the communication breakdown.

This communication plan is really an exercise in helping technology designers think of different ways in which they might create avenues for voice interfaces, such as Alexa, to improve communication interactions with humans.  However, it highlights just how complex human communication is, and how many of us take verbal communication and the many processes involved in verbal communication, for granted. Human communication is complex and challenging, even for other humans… No wonder technology is having a hard time with it as well.

For more details on our research study, and on the communication breakdowns that occurred between Alexa and families, you can find our research paper, “Communication Breakdowns Between Families and Alexa,” here.

 

¹ In this case, fundamental frequency refers to a measure of vocal fold vibration during speech production. Although not exactly the same, you can think of it as overall pitch of a voice. I suggest this as one empirical measure that an entity such as Alexa can refer to in order to help identify the sound characteristics of her communication partner: differentiating between a young child vs an adult, who generally have different fundamental frequencies.

 

Erin Beneteau is a speech-language pathologist (SLP) and a PhD student at the Information School at the University of Washington. Her primary areas of clinical practice include working with preschool age children, as well as working with people of all ages who use assistive technologies for communication. Erin has practiced in the United States, New Zealand, and Ireland as an SLP. and she has also worked in the technology industry as an instructional designer. Her recent research on family interactions with the Amazon Echo Dot has been an exciting way to blend her interests in technology and human communication.

Let me set the scene. You’ve been invited to a roundtable conversation with 17 international thought leaders working at the intersection of child development, early learning and children’s media. As you look around the table, you see influential early childhood educators, researchers, academics, pediatricians, children’s media producers, advocates and policy experts. It is immediately clear that these leaders and innovators share a commitment to young children and child development first, technology second.

Milton Chen, senior fellow at the George Lucas Education Foundation, sets the table by offering this provocation in his foreword to the new book, Exploring Key Issues in Early Childhood and Technology: Evolving Perspectives and Innovative Approaches .

“Imagine yourself sitting at this distinguished table and how you might contribute to the conversation, as a researcher, teacher, media professional, activist, student, parent or caregiver… Interesting projects might merge for how families and communities can create the better digital environment we deserve. What could be more important to our democracy that the environment of ideas our children are raised in?”

This imagined dinner party conversation is a metaphor that captures the spirit of my third edited book with contributions from thought leaders in the United States, Australia, Great Britain, Israel, Scotland and Norway. Each author contributed a 2,000-word essay to this collection of powerful ideas about lessons learned, promising practices, innovative approaches and reflections on research and practice. Taken together, the essays offer a look at the groundbreaking work these innovators have done, and continue to do, to help define beneficial screen time and what positive experiences with interactive media should look like and include.

“When one enters a new domain of knowledge, one initially encounters a crowd of new ideas. Good learners are able to pick out those which are powerful.”
Seymour Papert*

Here are a few of the new ideas from the book that are powerful: 

• In his essay, “Five Things that Haven’t Changed (Much),” David Kleeman, senior vice president for Global Trends for Dubit, suggests that “The concept of “screen time” has become meaningless in a world where screens bring entertainment, learning, discovery, communication, play, creation and more.”
• Lydia Plowman, from the University of Edinburgh in Scotland, writes “Designing for young children in a world of ambient computing provides a wonderful opportunity to explore the ways in which technology can provide fun, pleasure and play,” in her essay “When the Technology Disappears.” 
• Kevin A. Clark, director of the Center for Digital Media Innovation and Diversity at George Mason University, contributed “Seeing is Believing: Racial Diversity in Children’s Media,” in which he offers this provocation: “Imagine what would happen if all children read books, saw television programs, watched movies, and played games that presented positive portrayals of people who look like them.”
• Michael H. Levine, former executive director of the Joan Ganz Cooney Center at Sesame Workshop, wrote an essay on “Digesting the iScreen Decade: What Should Media Makers, Policymakers and Philanthropy Do Next?” He writes: “Kids’ enthusiasm for digital activities presents a great “hook” for teachers and busy parents to manage their responsibilities, but if educators and parents themselves do not become technically proficient, the full range of digital possibilities—and the access to new technological tools, will effectively be reserved for the more privileged.”
• Lisa Guernsey, director of the Learning Technologies Project and senior advisor to the Early and Elementary Education Policy program at New America wrote an essay titled, “A Mission for Media Mentors: Creating Critical Thinkers.” “With ideas and guidance from media mentors, educators and parents can learn to take advantage of children’s natural curiosity to build their critical thinking skills about all the message they see and hear.”

I invite you to read these essays, encounter new ideas, and “pick out those which are powerful” for your work with young children and technology.

I started this blog post with words from Milton Chen’s foreword, and it seems fitting to leave you with his powerful and timely words.

“Perhaps it’s time to acknowledge that the digital environment is now as immersive, tangible, and vital to life as the physical environment. Creating a sustainable future is becoming as urgent for our children’s digital lives as the crises threatening our ecosystems, where extreme weather, air and water pollution, and climate change are on the rise. As Internet hacking, fake news, and divisive social media have shown, the climate is changing for our digital world, as well.”

 

Thanks to the thought leaders who contributed their powerful ideas and innovative approaches to this book: Marina Bers, Lewis Bernstein, Warren Buckleitner, Milton Chen, Kevin Clark, Susan Edwards, Shuli Gilutz, Lisa Guernsey, David Kleeman, Natalia Kucirkova, Michael Levine, Sonia Livingstone, Elena Lopez, Jackie Marsh, Lydia Plowman, Jenny Radesky, and Ellen Wartella

 

 

 

*Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas, p. 137. New York, NY: Basic Books.

 

Chip Donohue, PhD, is no longer at TEC Center at Erikson Institute but remains as Founding Director. He is a Senior Fellow and Member of the Advisory Board of the Fred Rogers Center for Early Learning and Children’s Media at Saint Vincent College, where he co-chaired the working group that revised the 2012 NAEYC & Fred Rogers Center Joint Position Statement on Technology and Interactive Media as Tools in Early Childhood Programs Serving Children from Birth through Age 8. Chip is the editor of Exploring Key Issues in Early Childhood and Technology: Evolving Perspectives and Innovative Approaches (2019),Family Engagement in the Digital Age: Early Childhood Educators as Media Mentors (2017), and Technology and Digital Media in the Early Years: Tools for Teaching and Learning (2015) co-published by Routledge/NAEYC. In 2012 he received the Bammy Award and Educators Voice Award as Innovator of the Year from the Academy of Education Arts & Sciences. In 2015, he was honored as a children’s media Emerging Pioneer at the KAPi (Kids At Play International) Awards. Follow Chip’s digital footprints as he travels the globe at @chipdono.

Asha and her 6-year-old daughter, Zara, take deep breaths and blow onto the sails of their sail cars. The cars roll away…quickly at first before slowing to a stop. Zara cheers and exclaims, “I won! Mine went farthest!”
Asha agrees and asks, “What was your prediction?”
Zara reflects and grabs a yardstick, “Well, I thought it would go 16 inches. But it went a lot more.” She measures the distance and proudly announces, “25 inches!”
Asha is impressed at the result and asks her daughter, “What made your car go that far?”
Zara explains, “My breath is a force. It’s like the wind. It pushed on the sail and made it go.”[1]

 

Project Background

Asha and Zara are participating in Ruff Family Science, a project that aims to foster joint media engagement and hands-on science exploration among diverse, low-income parents and their 4- to 8- year old children. Funded by the National Science Foundation and based on the popular PBS KIDS animated character Ruff Ruffman, the project is a collaboration among public media producers at WGBH, researchers at Education Development Center (EDC), and three implementation partners: National Center for Families Learning, Kentucky Educational Television, and Alabama Public Television. Together, these partners used an iterative research-design process to create an implementation model and prototype resources that meet the needs of families with a parent enrolled in an adult education program, addressing parents as both adult learners and as caregivers who can nurture their children’s development in science. In doing so, the project’s ultimate aim is to build new knowledge about the potential for digital media to inspire and support intergenerational science learning among families.

Ruff Family Science teaches fundamental science concepts to adult learners in different types of adult education classes: English as a second language, parenting, or adult basic education. Using the Ruff model, parents are first introduced to the resources in an adult education class, and then have the opportunity to explore the same resources with their children in a facilitated family workshop and at home. This model enables parents to build their confidence while exploring science topics among peers before introducing topics to their children. The resources include:

• Videos that feature Ruff Ruffman and introduce the science topic for investigation. They include animated and live-action segments that show real families engaged in science exploration.
• Hands-on activities that use household materials and include simple instructions so families can investigate science concepts together.
• At-home tools, including a handout containing tips and key science vocabulary translated into several languages; and an app containing activities, additional videos, and a 2-player game intended to reinforce the featured science content.
• Educator supports to facilitate integration of the above materials into various adult and family education programs. They include videos, two different versions of educator guides (for adult educators and family educators), and alignment to Career and College Readiness Standards for Adult Education (CCRSAE) and English Language Proficiency (ELP) standards.

Lessons Learned

EDC conducted an initial needs assessment and led several rounds of testing at adult and family education sites across the country. EDC collected information via site visits, focus groups, educator and parent interviews, and surveys. Testing was conducted in a range of settings, including community schools, adult education programs, poverty reduction programs, and immigrant and refugee assistance programs. In addition, Concord Evaluation Group (CEG) conducted a series of adult educator focus groups, and WonderWhy Consulting conducted user experience testing of at-home materials. Below is a summary of the findings and recommendations for developers.

Educators and adult education programs

• Adult and family education programs focus their instruction on literacy, parenting, and life skills. Intergenerational learning programs were most receptive to introducing science when it can be integrated into the programs’ existing focus. Educators appreciate materials that include vocabulary support and prompts for reading, speaking, listening, and writing about science. Visual instructions for low literacy learners and vocabulary translations for English language learners were also helpful.
• Adult and family educators need resources and support to effectively integrate science into their programs. Educators often have outdated curricula, small budgets, limited background knowledge, and a lack of information about science topics. Educators need stand-alone resources that are easily accessible, free of charge or inexpensive, and use materials that are easy to find.
• Educators are enthusiastic about teaching science but need training. Many adult and family educators have limited to no experience teaching science. However, they are interested in attending STEM-related trainings, providing STEM activities and materials to their students to use in class and at home with their children, and inspiring a love of science in their adult students. Educators also expressed interest in receiving support for teaching with media. Therefore, instructional materials should include a variety of educator training and supports to help them teach science with media: videos, webinars, and versions of educator guides that are specific to particular audiences.

Adult learners

• Parents build confidence and are more prepared to teach science to their children when previously introduced to science concepts in adult education settings. Many adult learners have low literacy levels, special needs, or limited education. They are often intimidated by science and worry that they will make mistakes, get the wrong answer, will not learn as quickly as their children, or not grasp the concepts. However, they believe science education is important for themselves and their children and want to help their children learn science at home. Programs should give parents an opportunity to complete the hands-on activities in an adult education setting prior to engaging with their children at home. This approach enables them to learn the content, vocabulary, and practices needed to support their children and build their own confidence.
• Resources should expand adult learners’ perception and interpretation of science. Parents often don’t realize that science is about is about hypothesizing, wondering about their surroundings, and testing their ideas. Science materials should enable parents and their children to make predictions, observe, record data, discuss concepts, and use scientific vocabulary, in an informal and fun environment. This approach encourages families to “think scientifically” about the things they do every day, such as cooking or riding a bike.
• Materials should emphasize hands-on, collaborative learning. Adult learners enjoy engaging in hands-on learning experiences and gameplay. They prefer to work collaboratively with their peers and their children rather than work independently or compete against others.
• Parents recognize the value of playing with their children as a way to help them learn. However, time is a big constraint to engaging with their children. Parents must juggle jobs with irregular schedules or unconventional hours, household chores, and homework support. Multigenerational science activities must be highly valued by families. They should be fun, hands-on, engaging, playful, and effective if parents are to prioritize and include them in their lives.

Family Learning with Media

• Playful and relatable videos can set the stage for learning. Families and adult learners respond enthusiastically to videos that mix humor and science learning. When creating videos, depict real families from a variety of cultural and ethnic backgrounds and show them exploring science in ways that feel natural and authentic.
• Translation should be approached strategically. Although intergenerational science programs serve a large proportion of non-native English speakers, they use English-language materials almost exclusively to support English language and literacy development. Translating science keywords into multiple languages may be more helpful than translating complete videos or activities into other languages.
• Digital games can support families in exploring science at home. Access to and familiarity with technology varies greatly among families in intergenerational learning programs. However, educators and families enjoy using online games and apps to extend family science exploration at home. Games can scaffold play for beginning science learners by offering hints or levels according to players’ ability.

In Summary
The Ruff Family Science model shows that using an intergenerational approach to teach fundamental science concepts can be effective on many levels. Science is rarely taught to adult learners enrolled in ESL, parenting, or basic skills classes; however, when integrated into existing programs, understandings about science and enthusiasm toward science learning can grow. When science concepts are taught using a variety of media­–animated and live action video, engaging hands-on activities, online games, and printed resources­–adults gain confidence in their own abilities and are excited to bring science home and try it with their children. By engaging in science together, parents not only strengthen the bond with their children, they also teach them science practices and life skills…to hypothesize, question, experiment, collect data, and wonder about the world.[2]

 

 

[1] This is a fictional example, based on conversations, interactions, and observations during testing.

[2] This material is based upon work supported by the National Science Foundation under Grant No. 1713494. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

 

Lisa Hamanaka Ellsworth has over 25 years of experience in education, both in the classroom and in educational publishing, and holds a master’s degree from Harvard University. She currently works as a digital producer at WGBH Educational Foundation where she contributes to public media projects in science and engineering, computational thinking, and social studies. Lisa also freelances as an instructional designer and content developer, working with industry leading clients such as FableVision Studios and iCivics.