Usability testing_ Duolingo

Team Project : Desarae Veit, Young Ju Cho, Nick Sturtz

CI 504 spring 2016 Iowa State University

Overview

  • Product Description
  • Target Audience Assumption
  • Intended Outcomes
  • Preliminary Evaluation
  • User Try-Out Plan
  • Matrix of Objectives & Goal Criteria
  • Pilot Test Results
  • User Try-Out Implementation
  • Results and Finding

About Duolingo

Duolingo was created by a team of Carnegie Mellon students lead by PhD students, Luis von Ahn and Severin Hacker (Siegler, 2011). The Python based web application won App of the Year awards for best of iPhone and Android in 2013 and again in 2014 for Android 2014 (Gigaom, 2013). This award winning education application has been sponsored by several organizations including Union Square Ventures, New Enterprise Associates, and Google Capital (Siegler, 2011; Todd, 2012; Venture Capital Dispatch, 2012)

Duolingo is a Python application, which allows a user to learn to speak, read, write, and understand new language(s). Duolingo website states “With over 100 million users, Duolingo has organically become the most popular way to learn languages online in only 2 years” (von Ahn, Luis, 2011).  

Duolingo may be run on in a native mobile application, or accessed on desktop or mobile browser. The desktop website (Appendix B) and mobile browser versions of the application (Appendix C) are very similar to the three native mobile versions Android, iOS, and Windows (Appendix D).  This evaluation will primarily focus on the Apple iOS version of the Duolingo application, with some cross references to the browser desktop version to note additional resources and features. Duolingo offers courses in 22 different languages and has created special targeted versions of the Duolingo application geared toward self-learning or schools and one towards business with a test in English fluency.

Target Audience

1.     People wanting to learn a new language for their own enjoyment/benefit

2.     People learning a new language for their job

3.     Children learning a new language in their school

4.   Adults learning a second language at a community college or community course with         their progress tracked in-app by the instructor

Intended Outcomes

Duolingo is a free language learning tool and a tech based platform aiming to facilitate and assist the learning progress of foreign languages. It supports primarily the curriculum and language services for the early learners. It is currently offering 11 language courses to English speakers (Latin American Spanish, French, German, Brazilian Portuguese and Italian), as well as a variety of other courses (mostly American English, but also Spanish and French) to native speakers of other languages, such as Spanish, French, German, Portuguese, Italian, and many more.

The learning process is accompanied with visual elements and provides a number of different ways to study for new language skills such as listening, reading, speaking and writing which helps users not get bored. Also users can cover all the aspect of the language that make a balance of the learning process. By progressing sets of exercise like answering questions and completing lesson, a user can improve vocabulary and grammar as drills and exercises get progressively more complex and harder. 

Each lesson contains 4 types of exercises for translation, listening, matching, and speaking. The translation exercise is to translate from your native language to your target language or vice-versa. The activity for a listening exercise is to listen to a short phrase being spoken and write what is heard. In the case of a matching exercise, users are introduced to new vocabulary and need to match it with the right photo or vice-versa. The speaking exercise is an activity in which a user reads out sentences in the learning language.    

Cognitive Objectives – Knowledge

      Increase the skills of listening, reading, writing  and number of vocabulary

      Be able to understand what they listen

      Be able to type what they hear

      Be able to translate a target language

      Be able to speak with correct accent and pronunciation

      Be able to write a sentence with right grammar

Affective Objective - Attitude

      Gain confidence with gamification factor of the program.

      Increase concentration with simple User Interface.

      Be self-motivated - motivation is a key factor in language acquisition.

      Experience a self-directed, tech-based learning work

      Grow ability of independent study

Psychomotor Objectives – Skills

      Be able to have relatively simple conversations with new language.

      Be able to create simple sentences straight away.

      Identify all correct translations of a chunk of spoken or written text

      Identify the English meaning of a vocabulary word

Preliminary Evaluation

Appropriateness of the materials to the task and audience

Learning a new language can be very difficult.  The traditional, classroom-based approach can be very intimidating to a new learner.  That approach usually starts with learning simple noun/verb combinations such as conjugating the verb to be (I am, he is, we are, etc.) and builds from there to alphabets, numbers, and simple phrases.  Duolingo changes the approach to begin with simple matching exercises and repetition of simple words such as man, woman, boy, etc., going between the user’s native language and the language being learned.

Duolingo puts new language programs through a stringent user testing process. In phase one, volunteer contributors decide on preferred words and topics for the program. Then the program is released into beta for testing. In the beta testing phase, actual users are given the opportunity to try out the program. It has been noted that sometimes users try to prematurely reverse engineer the program to attempt to learn the other language. This happened in the Korean for English speaker’s beta testing. English speakers who wanted to learn Korean got frustrated waiting for the app to be released and attempted to use the English for Korean Speakers program to learn Korean. The results set the beta testing back by months. 

Duolingo requires a certain amount of successful course completions and a minimum count of completed courses without sections being flagged for inaccuracy. This was noted frequently in the commentary of Duolingo’s discussion boards, as moderators begged users to only accurately flag items and for English speakers to please wait for their own application because the English for Korean Speakers needed to be released first and Korean for English Speakers was meant to follow up after Phase 3 of English for Korean Speakers. In phase 3 the course within the application graduates and is released for all users. 

Visual Design

The visual design is very child-like but the adorable owl figure is a brand icon within the Duolingo application.

The application uses gamification aspects that are becoming more and more common within websites, alternate reality gaming, pervasive games, daily life (Szulborski, 2005), and mobile applications. In Duolingo, Duo the owl, can be dressed up by visiting the application’s shop section. Here, users can also “purchase” more courses or skills to learn. These items may be purchased using a version of currency specific to Duolingo called Lingots. Lingots may be earned by completing courses within Duolingo or practicing and honing skills. 

 

Evaluative Questions and Matrix Creation

Evaluative Instrument Selection Reasons

The team chose our evaluation instruments so we could collect data on the design of Duolingo in different ways and could triangulate on whether or not the design is usable.  Triangulation allows us to look at data from different ways to hopefully show patterns in the data to support the hypothesis.

Here are the reasons we chose our evaluative instruments:

      We chose to use a survey to get quantitative data on what the users thought of the design of Duolingo and how it affects their perceived usability (see survey in Appendix F).  We could easily compare the answers provided on the survey to see what users thought of the design of the app and see if the design was actually good.  We also chose surveys because they are a good way to reach a larger audience and hopefully get a larger dataset to study.

      We chose observation so we could see actual users interacting with the app and could watch for places where the design of the app caused issues for the users.  Other instruments require the user to remember and report later, so they could forget times when the design of the app was bad or could not provide accurate answers for various reasons.  Observation will allow us to see them working in the app and collect data without requiring the user to self-report.

      We chose interviews as a way to get more qualitative data from the user, allow them to ask questions of the evaluation team, and provide a dialogue between the user and the team so we can gather data on how the users feel about the design of Duolingo and why they might feel that way.  See interview questions in Appendix G.

Description of the Data Collection Process

Date: April 7 - 10, 2016
Time allotted: 30 minutes per each participant
Location: Room 303 at College of Design, Urbandale Public Library Room D, Residence in Ames
Evaluators: Young Ju Cho, Nick Sturtz
Participants: Six participants with different level of familiarity of the application, Duolingo.  The users will be recruited by the evaluators and will be friends, family members, or classmates of the evaluators.
Materials: iPhone, Video camera
Evaluation Tools: Survey Questionnaire, observation checklist, List of interview questions
Procedure: Welcome Script and instructions to participant (5 minutes), Try-Out (20 minutes), Survey Questionnaire (5-10 minutes), Wrap Up (5 minutes)

Conclusion and Debriefing of the user tryout session

After the test of Duolingo has been completed, participants will be asked to fill out post survey questionnaire about their overall experience and impression of the app. After the survey, the evaluator will have a short interview with the participants, ask if they have any additional comments, and appreciate their time.

Results and Findings

Observation Results

 We observed our users completing several tasks in the app and below are the results from our observations:

      The users were able to easily navigate through the app and find and complete exercises

      The users were able to easily switch to learning a new language and complete an exercise in that new language

      The app provided good feedback on the meaning of new words being introduced in an exercise, giving the user the ability to touch the new word and see what it meant

      The app allowed the user to repeat any words/phrases that were played via audio so they could hear the word repeated

      If the user needed a new keyboard installed on their iPhone for the language they were studying, the app did give feedback on how to install it, but the notice was missed by at least one user

      The users found it quite difficult to change the language of the app, and a majority were unable to do that

During observation, we watched for the number of errors that the users received

Survey Results

After observing, we administered a survey to our users.  Below are the results from our survey on Duolingo.  The survey gathered demographic data and also asked what the users thought about various aspect of the design of Duolingo.

Interview Question Results

            After observing the users, and having them complete our survey, we administered an interview to get more information on what the users thought about the usability of Duolingo.  Below are the results of the survey:

      Most users said they thought Duolingo was a very usable app

      They had mostly positive things to say about the app and struggled when asked what they disliked about the app

      They liked the way it teaches the language through images, sounds, and that it introduces words and phrases rather than more traditional language learning techniques

Recommendations

Based on our evaluation, we would make the following recommendations for Duolingo:

             ●      Make it easier to change the language of the app

             ●    Give more time before progress is lost

             ●      Give a syllabus overview of lessons for teachers

             ●      Explain the purpose of XP points

             ●      Bring back “Victory” sounds

             ●      Make it easier to add friends

             ●      Keep up the good work!