Usability testing_ Duolingo

Team Project : Desarae Veit, Young Ju Cho, Nick Sturtz

CI 504 spring 2016 Iowa State University


  • Product Description
  • Target Audience Assumption
  • Intended Outcomes
  • Preliminary Evaluation
  • User Try-Out Plan
  • Matrix of Objectives & Goal Criteria
  • Pilot Test Results
  • User Try-Out Implementation
  • Results and Finding

About Duolingo

Duolingo was created by a team of Carnegie Mellon students lead by PhD students, Luis von Ahn and Severin Hacker (Siegler, 2011). The Python based web application won App of the Year awards for best of iPhone and Android in 2013 and again in 2014 for Android 2014 (Gigaom, 2013). This award winning education application has been sponsored by several organizations including Union Square Ventures, New Enterprise Associates, and Google Capital (Siegler, 2011; Todd, 2012; Venture Capital Dispatch, 2012)

Duolingo is a Python application, which allows a user to learn to speak, read, write, and understand new language(s). Duolingo website states “With over 100 million users, Duolingo has organically become the most popular way to learn languages online in only 2 years” (von Ahn, Luis, 2011).  

Duolingo may be run on in a native mobile application, or accessed on desktop or mobile browser. The desktop website (Appendix B) and mobile browser versions of the application (Appendix C) are very similar to the three native mobile versions Android, iOS, and Windows (Appendix D).  This evaluation will primarily focus on the Apple iOS version of the Duolingo application, with some cross references to the browser desktop version to note additional resources and features. Duolingo offers courses in 22 different languages and has created special targeted versions of the Duolingo application geared toward self-learning or schools and one towards business with a test in English fluency.

Target Audience

1.     People wanting to learn a new language for their own enjoyment/benefit

2.     People learning a new language for their job

3.     Children learning a new language in their school

4.   Adults learning a second language at a community college or community course with         their progress tracked in-app by the instructor

Intended Outcomes

Duolingo is a free language learning tool and a tech based platform aiming to facilitate and assist the learning progress of foreign languages. It supports primarily the curriculum and language services for the early learners. It is currently offering 11 language courses to English speakers (Latin American Spanish, French, German, Brazilian Portuguese and Italian), as well as a variety of other courses (mostly American English, but also Spanish and French) to native speakers of other languages, such as Spanish, French, German, Portuguese, Italian, and many more.

The learning process is accompanied with visual elements and provides a number of different ways to study for new language skills such as listening, reading, speaking and writing which helps users not get bored. Also users can cover all the aspect of the language that make a balance of the learning process. By progressing sets of exercise like answering questions and completing lesson, a user can improve vocabulary and grammar as drills and exercises get progressively more complex and harder. 

Each lesson contains 4 types of exercises for translation, listening, matching, and speaking. The translation exercise is to translate from your native language to your target language or vice-versa. The activity for a listening exercise is to listen to a short phrase being spoken and write what is heard. In the case of a matching exercise, users are introduced to new vocabulary and need to match it with the right photo or vice-versa. The speaking exercise is an activity in which a user reads out sentences in the learning language.    

Cognitive Objectives – Knowledge

      Increase the skills of listening, reading, writing  and number of vocabulary

      Be able to understand what they listen

      Be able to type what they hear

      Be able to translate a target language

      Be able to speak with correct accent and pronunciation

      Be able to write a sentence with right grammar

Affective Objective - Attitude

      Gain confidence with gamification factor of the program.

      Increase concentration with simple User Interface.

      Be self-motivated - motivation is a key factor in language acquisition.

      Experience a self-directed, tech-based learning work

      Grow ability of independent study

Psychomotor Objectives – Skills

      Be able to have relatively simple conversations with new language.

      Be able to create simple sentences straight away.

      Identify all correct translations of a chunk of spoken or written text

      Identify the English meaning of a vocabulary word

Preliminary Evaluation

Appropriateness of the materials to the task and audience

Learning a new language can be very difficult.  The traditional, classroom-based approach can be very intimidating to a new learner.  That approach usually starts with learning simple noun/verb combinations such as conjugating the verb to be (I am, he is, we are, etc.) and builds from there to alphabets, numbers, and simple phrases.  Duolingo changes the approach to begin with simple matching exercises and repetition of simple words such as man, woman, boy, etc., going between the user’s native language and the language being learned.

Duolingo puts new language programs through a stringent user testing process. In phase one, volunteer contributors decide on preferred words and topics for the program. Then the program is released into beta for testing. In the beta testing phase, actual users are given the opportunity to try out the program. It has been noted that sometimes users try to prematurely reverse engineer the program to attempt to learn the other language. This happened in the Korean for English speaker’s beta testing. English speakers who wanted to learn Korean got frustrated waiting for the app to be released and attempted to use the English for Korean Speakers program to learn Korean. The results set the beta testing back by months. 

Duolingo requires a certain amount of successful course completions and a minimum count of completed courses without sections being flagged for inaccuracy. This was noted frequently in the commentary of Duolingo’s discussion boards, as moderators begged users to only accurately flag items and for English speakers to please wait for their own application because the English for Korean Speakers needed to be released first and Korean for English Speakers was meant to follow up after Phase 3 of English for Korean Speakers. In phase 3 the course within the application graduates and is released for all users. 

Visual Design

The visual design is very child-like but the adorable owl figure is a brand icon within the Duolingo application.

The application uses gamification aspects that are becoming more and more common within websites, alternate reality gaming, pervasive games, daily life (Szulborski, 2005), and mobile applications. In Duolingo, Duo the owl, can be dressed up by visiting the application’s shop section. Here, users can also “purchase” more courses or skills to learn. These items may be purchased using a version of currency specific to Duolingo called Lingots. Lingots may be earned by completing courses within Duolingo or practicing and honing skills. 


Evaluative Questions and Matrix Creation

Evaluative Instrument Selection Reasons

The team chose our evaluation instruments so we could collect data on the design of Duolingo in different ways and could triangulate on whether or not the design is usable.  Triangulation allows us to look at data from different ways to hopefully show patterns in the data to support the hypothesis.

Here are the reasons we chose our evaluative instruments:

      We chose to use a survey to get quantitative data on what the users thought of the design of Duolingo and how it affects their perceived usability (see survey in Appendix F).  We could easily compare the answers provided on the survey to see what users thought of the design of the app and see if the design was actually good.  We also chose surveys because they are a good way to reach a larger audience and hopefully get a larger dataset to study.

      We chose observation so we could see actual users interacting with the app and could watch for places where the design of the app caused issues for the users.  Other instruments require the user to remember and report later, so they could forget times when the design of the app was bad or could not provide accurate answers for various reasons.  Observation will allow us to see them working in the app and collect data without requiring the user to self-report.

      We chose interviews as a way to get more qualitative data from the user, allow them to ask questions of the evaluation team, and provide a dialogue between the user and the team so we can gather data on how the users feel about the design of Duolingo and why they might feel that way.  See interview questions in Appendix G.

Description of the Data Collection Process

Date: April 7 - 10, 2016
Time allotted: 30 minutes per each participant
Location: Room 303 at College of Design, Urbandale Public Library Room D, Residence in Ames
Evaluators: Young Ju Cho, Nick Sturtz
Participants: Six participants with different level of familiarity of the application, Duolingo.  The users will be recruited by the evaluators and will be friends, family members, or classmates of the evaluators.
Materials: iPhone, Video camera
Evaluation Tools: Survey Questionnaire, observation checklist, List of interview questions
Procedure: Welcome Script and instructions to participant (5 minutes), Try-Out (20 minutes), Survey Questionnaire (5-10 minutes), Wrap Up (5 minutes)

Conclusion and Debriefing of the user tryout session

After the test of Duolingo has been completed, participants will be asked to fill out post survey questionnaire about their overall experience and impression of the app. After the survey, the evaluator will have a short interview with the participants, ask if they have any additional comments, and appreciate their time.

Results and Findings

Observation Results

 We observed our users completing several tasks in the app and below are the results from our observations:

      The users were able to easily navigate through the app and find and complete exercises

      The users were able to easily switch to learning a new language and complete an exercise in that new language

      The app provided good feedback on the meaning of new words being introduced in an exercise, giving the user the ability to touch the new word and see what it meant

      The app allowed the user to repeat any words/phrases that were played via audio so they could hear the word repeated

      If the user needed a new keyboard installed on their iPhone for the language they were studying, the app did give feedback on how to install it, but the notice was missed by at least one user

      The users found it quite difficult to change the language of the app, and a majority were unable to do that

During observation, we watched for the number of errors that the users received

Survey Results

After observing, we administered a survey to our users.  Below are the results from our survey on Duolingo.  The survey gathered demographic data and also asked what the users thought about various aspect of the design of Duolingo.

Interview Question Results

            After observing the users, and having them complete our survey, we administered an interview to get more information on what the users thought about the usability of Duolingo.  Below are the results of the survey:

      Most users said they thought Duolingo was a very usable app

      They had mostly positive things to say about the app and struggled when asked what they disliked about the app

      They liked the way it teaches the language through images, sounds, and that it introduces words and phrases rather than more traditional language learning techniques


Based on our evaluation, we would make the following recommendations for Duolingo:

             ●      Make it easier to change the language of the app

             ●    Give more time before progress is lost

             ●      Give a syllabus overview of lessons for teachers

             ●      Explain the purpose of XP points

             ●      Bring back “Victory” sounds

             ●      Make it easier to add friends

             ●      Keep up the good work!


Stop Bullying

The video aims to prevent bullying in school. By utilizing the method of the billiad pool game, the message would be delivered metaphorically. The white ball represents the BULLY, which is used as a tool to hit the other balls in the actual billiard game and other colored balls play roles as victims. Each scene shows that the white ball hits the other victim ball with script and sound effects. The white BULLY ball and the eight ball are zoomed in and the white ball tries to hit the eight ball, but it rolls down into a hole itself. Finally, the large size of the eight ball is rolling out slowly and the message follows after the ball. 

 By Young Ju Cho

Experiment / Low-tech intervention

Interactive Public Sound Art: David Birchfield, Kelly Phillips

Visual and Acoustic Space: Marshall McLuhan


As the life span of people is getting longer, health is one of the issues people pay more attention to. Even though people know the importance of the health, it is hard for people to get involved in the physical activities like fitness due to the limitations of time and cost. Over the course of several days I observed that many people walked into the Design building and automatically turned left to wait for the elevator. Even if, they need to wait for an elevator for a few minutes, they do not consider going to the stairs. Thus, the goal of the experiment I carried out was to encourage people going in and out in the Design building to use stairs than the elevator.

I used HOPSCOTCH to provide people who usually take elevator with the experience of using stairs in an interesting ways. I installed the hopscotch lines on the main floor of Design building by using duck tape and put the numbers on each box without any sound effects. The hopscotch boxes lead to the stairs, which is my intention of leading people go to the stairs through playing hopscotch.

I observed really interesting findings during the experiment. While most students I watched passed the lines and were not intrigued with the hopscotch I drew with tape, people who are older than undergraduate students played this “analogue style game” because they have a memory of playing hopscotch when they were young. Also, I had some feedback from people who played the hopscotch. They suggested that there should be something to throw to the square and jump to catch, just like how they played when they were young.

Based on the interesting observation from this experiment, I searched other examples in which the hopscotch was used as a method related to memory, interaction, and childhood. I found interesting projects with those themes.

Interactive Hopscotch for ART && CODE 3D from Reha Discioglu on Vimeo.

“For us, childhood means memories. Few are clear, some blurry, and some flashy. Childhood also means playing and games. Skipping rope, hide-and-seek and hopscotch… Interactive Hopscotch aims to trigger childhood memories in a playful manner by using a child game.”

Hopscotch game was installed to surround the visitor with images and sound. When the visitors jump, they are surprised with the responses of squares of the Hopscotch. This activity provides interactions between the visitors and their memories.

Through the experiment, I learned that connecting people with their memory in the childhood can be an effective way to be used for behavior changes. For the next step, other sensory effects will be applied to make this project more attractive to people and to provide the connection to their memory. Visual and auditory effects like using different patterns and colors of hopscotch, and sounds from old video games may be included.

Awakening the Senses through Technology

Interactive Public Sound Art: David Birchfield, Kelly Phillips, Assegid Kidané
Visual and Acoustic Space: Marshall McLuhan

                                                                                                                                                                 By Young Ju Cho

The rapid advance of technologies gives limitless convenience to human life and brings different ways of human perception. It also provides us with a variety of media for communications and interactions. According to McLuhan, he predicted that the development of technology would change whole our society by allowing us to use other senses such as touch, sound, and smell, rather than only visual sense which we had heavily relied on for a long time due to an advance of the print media. I think the time McLuhan mentioned has just arrived now. 

Another revolutionary emergence includes the internet and smart phones which give us countless benefits and convenience, but there are also many negative consequences, such as breach of privacy and theft of person information, and insecurity. However,through the McLuhan’s theory, I can expect the future in which imaginary ideas come true through a variety of media rather than concerning about the negative effects of the new media. There are many studies to find the way to use all the senses for restoring the value of human emotion in many areas. I think it is all the same reason why we are learning about sounds and trying to find the connection to the design. 

Interactive public art in Tampa looks a very interesting project that matches well with McLuhan’s theory, which emphasized the role of the media. Audiences interact with the musical structures, which is installed in bus stops, by moving their body as a tool. Through this playful interaction and engaging one another, people can be more active instead of stay a bench for waiting a bus. Also, they have a chance to communicate during the time of engaging and interacting. The use of the musical structures was limited in bad weather, but it was overcome and continued to function by application of technology like using solar energy and battery. The musical structure was inspired by the metaphor of water using blue background, and ripples from a disturbance in the water, which may lead more interest to people living in the desert areas. Like this musical structure, public art leads the active participation of the audience to become one of the elements that directly completed the artwork, not just passively accept information in the project. In addition, it helps the audience wake up all the senses and makes sensory experience unforgettable for a long time.

Development of the media during the past few decades is beyond imagination with new forms which resulted in even change of society. One of examples is the electronic media such as virtual reality and artificial space which become common in our daily life. Before long, I think we will see another change in society. Probably, we can control the information in the future just by simple actions, like looking at an object or moving head. Also, the information will be delivered through combined and diverse senses rather than simple visual or auditory senses. Our capability of controlling complicated media may lead to the expansion of countless human experiences through these physical interactions. 

At this point, it is needed to think if the new things are always good while the old ones are bad, or vice versa. It is irony that we living with high technologies in the 21century consider that the primitive age was the ideal time in terms of using senses, but it is important to keep it in mind that the basic human values are not subject to the machine.

Then, what is the role of a designer in the modern society full of technologies and tools? I think that awakening the senses in a variety of ways and helping people to communicate by using the technologies beneficial to humanity are the most important role as a designer.


The object I used to create sounds was a plastic bottle containing vitamin pills. Based on the sounds of turning the cap, scratching the surface, and shaking the bottle, I designed diverse sounds given the design principles. At first, I scratched the embossed letters on the top of the cap with a brush stick to produce the rough and rhythmical sound for texture. I made a pattern using the sound created on turning the cap with intervals, and scratched fast the smooth surface on the side of the bottle with a disposable plastic spoon for the expression of direction by increasing the intensity of sound and making sliding sounds. For gravity, I held the bottle horizontally and slowly tilted to one side and the other side, and used the sound of pills rolling down when moving the bottle in a seesaw manner. In the case of asymmetry, I recorded the sound generated when I tabbed the bottle irregularly. For symmetry principle, I made the scratching sound, which was used for texture, in regular time intervals and with balanced sound. It would be interesting when each sound created for texture, pattern, direction, gravity, asymmetry, and symmetry are combined to express harmony, density, anomaly, and scale on the basis of design principles.

Listening Map / Project 1

This is a listening map in which I visualized the sounds heard from children’s section and young adult’s section at the Ames public library.  The reason I choose this place is that generally people refrain from making sounds and make their voice quieter than usual in the library. Thus, I think that the library is an interesting place to make a listening map, and I am also curious about what the listening map looks like.

I visited the library to sketch a listening map at 6pm on Tuesday, January 27, 2015. Unlike my expectation, there were still a lot of little kids playing around with their parents or guardians at that time. I was sitting between kids section and teens room, and then tried to listen to the sounds. The most dominant sound was human voices from kids, their parents, librarians, and teenagers. Since the library was renovated a few months ago and the newly installed carpet seems to absorb the sound, I feel that all the sounds were really warm and cozy, and readily disappeared. The sounds did not bother me even though I can still hear lot of high pitch voices of kids and the voices resonated.

My next visit to the library occurred at 5pm on Saturday, January 31. It was just an hour before the library closed. There were less people in the library compared to Tuesday maybe because it was close to the closing time and a snow storm warning was issued. I had a similar feeling of warmness and coziness on the sounds that I had had on Tuesday but with less voices and noises. Because of this, I felt the background sound, heater sound, louder than Tuesday.

For making the listening map, I used colors and round circles to describe all the voices that I heard. In order to show the warm and cozy feeling sounds, I used bright colors, and the opacity was used to show the pitches. The large circles with light gray and skin colors represent the low and small sound, respectively. Small bright dots show the high pitch sounds of children’s voices. Yellow squares are the sounds of the librarian shelving books. There are large yellow circles and lines depicting the heating sounds on the right side of the map. I think that the adult’s voices and heater sounds were more to a background sounds which I could hear constantly with low sounds.

The green circles on right side of the map showed the mid tone of teen’s voices. Although there was a wall to the teenager’s room, I could hear the voices very well. There are the red starbursts right at the top of the upper map show the little kid’s crying sounds on the Tuesday map. On Saturday, before the library was closed, the library was filled with the large announcement that they are closing which were showed with red radiation lines the center of the map.

We are communicating with the world through five senses. Among them, we perceive objects mainly by vision, but auditory sense can provide us a great amount of information and details. When hearing sounds, we imagine the associated activities and objects. For example, in the dark or behind the back where the visional sense is futile, we totally depend on hearing.  If you have been in the beach, you can envision the scenery around the beach from wave sounds. Similarly, you can guess a species of the bird from the bird sound, the surprised face of a driver from brake sounds, the face of a child from crying sounds, and an image of a librarian re-shelving from the sound of placing books on the shelves.

Information given by sounds which are abstract is visualized in a listening map and expressed in concrete form. On the contrary, any sound maps make us to feel as if we hear the sounds by looking at it.

Listening Map / Exercise

This is a map of sound I heard on the fourth floor in the design building around 8pm on January 26. I tried to visualize all the sounds by using geometric shapes.

Since the time was a little late, only a few students were there and it was quiet. The place on the fourth floor lobby was divided by a wall divider, and I was sitting inside of the wall divider, so, most of the sounds, I could clearly identify what sound it was, and where the sound was coming from. Elevators, voices and stepping sounds were the most dominant at that time. Probably a lot more background sounds were there, but I could not notice them. The right side from where I sat, the elevators were going up and down consistently with a really annoying sound of the door opening, which was loud, high pitched and squeaky, but the bell ringing sound from the elevator was at an acceptable level of sound. The sound of the elevator door was represented by porky triangle shapes, which was fully colored and the sound of the bell ringing was visualized by circle shapes with empty inside, because I wanted show resonating bell sound. I could hear the students stepping clearly from behind the wall divider. Only several students walked through the hall and I realized that most of stepping sounds were not what I expected, like “Tap, Tap, Tap”. “S~s~S~s~”is the sounds of dragging shoes that I heard, so all the sounds were connected like a long line until the sounds disappeared. I drew these with faint outlines of scribbled circles because the stepping sounds were not clearly separated. Can you see the square on the left, bottom of the map? Somebody dropped his backpack and books on the floor at that point.

We often listen to things that we need to hear, so we miss a lot of sounds around us. Close your senses for a moment, and open your ears wider to the world around you. You can find another world of sound.