Results

I conducted the user tests with two of my peers and committee members who agreed to be recorded and identified: Nissa and Nicole. Both tests occurred as predicted on the script (see: User Tests) and I here present my reflections on the results.

User Observation

Performance

The two users behaved and performed very similarly during the test, even though they have different native languages (English and Spanish) and language learning experiences.

Indeed, their performance was above the average, compared to any other Portuguese student I have had. First, their pronunciation was very accurate, especially for learners who were taking a first lesson in Portuguese. Second, during Practicing and Recording sessions (see: User Tests), their retention of meanings/translations was approximately at a 100% rate.

Focus

Both users seemed relaxed, joyful and often laughed during most of the test. In fact, on the post-test questionnaire, Nicole has mentioned:

Nicole: “I remember having a lot of fun (although I don’t look it in that video o.0)”

Image 24 - Indicators of focus/concentration

But when asked to perform tasks by themselves (during the Practicing and Recording sections of the test), they also showed clear signs being focused or challenged:

• Nissa and Nicole would stop smiling,
• Nissa would often touch his face or pull his hair, and
• Nicole would slightly hit the table with her hands).

Recording

Beginning to record Nissa’s and Nicole’s speeches did not seem to affect the excellent quality of their performances.

Notes

When practicing the self-introduction sentences, Nissa decided to write down the information he would need to complete the blank lines.

For example, in the sentence “Meus olhos são __”, which means “My eyes are (color)”. Nissa had learned that “castanho” means brown and applies to the color of his eyes.

However, that word was not written on the slides. Nissa had asked me how to say “brown” in Portuguese and so I told him “castanho”. Therefore, he was only introduced to the sound of that new word, not to its spelling.

On Nissa’s sketch page, he wrote the word “castanho” intuitively as “casteño”, which resembles Spanish and even includes the “ñ” character, non-existent in the Portuguese alphabet.

Image 25 - Nissa’s notes

Nicole did not ask or attempted to write down her answers at any point. One of the reasons was that, besides the new vocabulary in Portuguese, Nissa was also introduced to his weight and height in the metric system, while Nicole already knew these values.

Brief Interview

In brief and informal conversations after the user tests, Nissa and Nicole have stated that they enjoyed their experiences. They both shared that they were happy with how much progress was accomplished in such a short time-frame.

In addition to that, Nissa shared that it would have been nice if he had reviewed all the content after the Practicing and Recording sessions. According to him, when he was speaking and translating by himself, he had many questions (such as “Am I pronouncing this correctly?” and “Is this the appropriate translation?”). However, since after one content practice to another we have moved to another content practice, he did not have a chance to have those questions answered.

Upload of videos

On the day of the tests, I uploaded the recorded videos on Youtube and shared the links with each user. The playlists with such videos can be accessed through the following links:

• Nissa: http://tinyurl.com/Nissa-LBug
• Nicole: http://tinyurl.com/Nicole-LBug

To protect the privacy of the users (who agreed to be recorded), users are only identified by their first name and the videos were shared as “unlisted” (a direct URL link is necessary to access them).

Questionnaire

The questionnaire was sent two days after the user tests.

Interest

It called my attention that Nissa and Nicole have both declared that their interest in learning Portuguese had increased after the tests. This may indicate that the approach used by LanguageBug can be very effective for motivating learners.

Image 26 - Both users feel they have learned during tests

Self-consciousness

When prompted to describe how they felt when watching themselves speaking Portuguese, Nissa and Nicole both mentioned “pride” and a certain “self-consciousness”.

Nicole: proud yeah, funny also, I sound so formal haha

Nissa: I felt sort of self-conscious, kind of embarrassed to watch myself, but also kind of proud of the actual language work

This embarrassment was expected, since it was a completely new language for both users.

Not great… yet

It is interesting to observe that although Nissa’s and Nicole’s performance were among the best among any student I have had, neither of them has qualified their performance great.

Image 27 - Both users feel they could have performed even better

One logical explanation for that could be that Nissa and Nicole had noticed that their speech was not perfect at all times. Indeed, the two users spoke Portuguese with very clear traces of a non-native, beginner speaker (hard-to-understand accents, slow pace, unusual rhythm, etc.).

This might have made Nissa and Nicole compare their speech with the perfect and desired speech, that is, a native speaker’s speech (e.g. my speech). By doing so, they became aware of their shortcomings, rather than extremely proud of their outstanding progress.

Image 28 - Confidence in their own language learning skills

On the other hand, they believe that their performance could be improved by using LanguageBug a few more times, which indicates that they trust their language learning abilities and feel capable of reaching even better results (that they would probably qualify as great).

Sharing videos

• Have you shared the videos of your speech with anyone else?
• Would you be comfortable sharing your language learning progress with a network of people?

Nissa and Nicole have both answered “Yes” to these questions. Nissa added that:

Nissa: I would definitely be comfortable sharing learning progress with a network, as long as I could choose if it was totally public to the whole network or if it was only within my friends (who I had selected).

In addition to that, Nissa has shared with me on a private messaging channel that one member of his family felt very happy and proud to see him speaking Portuguese.

Conclusions

One immediate conclusion is that user tests such as this one are extremely helpful and should be done more often. Besides this, a few design suggestions and considerations have emerged, such as:

• All new content should be displayed on a written form, including information that varies from user to user, and
• Users should be given the chance to review the content after having practiced and recorded their performance.

Most importantly, the self-assessment video feature seemed to have produce feelings of shyness, but also pride. Both users were happy with the opportunity to review and assess their own performance, which indicates that this feature could actually be incorporated on LanguageBug.

Both users have shared their performances with other people, which evidences that there is a social aspect in the self-assessment video feature. This aspect could be further explored in other user tests.