Supporting Listening Comprehension and Vocabulary Acquisition with Multimedia Annotations: The Students' Voice

LINDA C. JONES
University of Arkansas

Abstract:
This study extends Mayer's (1997, 2001) generative theory of multimedia learning and investigates under what conditions multimedia annotations can support listening comprehension in a second language. This paper highlights students' views on the effectiveness of multimedia annotations (visual and verbal) in assisting them in their comprehension and acquisition of vocabulary from aural texts. English-speaking college students listened to a 2 min 20 sec historical account in French presented by a computer program. Participants were randomly assigned to one of four listening treatments: the aural text (a) with no annotations, (b) with only verbal annotations, (c) with only visual annotations, and (d) with both visual and verbal annotations. For purposes of this paper, 20 students were purposively selected to participate in interviews. Overall, students remembered word translations and recalled the passage best when they had selected both verbal and visual annotations while listening. Students' voices reflected these results and revealed that they should have options for viewing material in both a visual mode and a verbal mode in a multimedia listening comprehension environment. This study provides qualitative evidence for a generative theory of multimedia learning that suggests that the availability and the choice of visual and verbal annotations in listening comprehension activities enhances students' abilities to comprehend the material presented and to acquire vocabulary.

INTRODUCTION

Listening comprehension activities provide students with the aural component of the target language to help them better hear the intricate sounds, enunciations, and content and develop their abilities to communicate with others in a target language. Educators try to help students enhance their listening skills by assigning them videotape, audiotape or computer-based activities to complete either at home or in the language lab setting. With these materials, students can practice hearing vocabulary words, sentence structures, and dialogues in the target language. Technology and language teaching have changed in recent years. Now, second language (L2) multimedia packages developed by researchers (e.g., Larson & Bush, 1992; Otto & Pusack, 1992) and by textbook publishing companies provide students with various listening comprehension activities and the learning aids needed to process them. Researchers have also called for an increase in research on L2 listening comprehension (e.g., Cauldwell, 1996; Field, 1997; Joiner, 1997; Lynch, 1998; Mendelsohn, 1998) and an increase in research on technology to better understand how we can utilize the attributes of multimedia to enhance various aspects of language learning, including listening comprehension (e.g., Brett, 1995, 1997; Hoven, 1999; Joiner, 1997; Jones & Plass, 2002; Lynch, 1998; Meskill, 1996; Purdy, 1996; Pusack & Otto, 1997; Salaberry, 2001).

To answer these calls, this paper specifically investigates students' thoughts and opinions concerning how verbal and visual annotations in a multimedia environment can assist them in their acquisition of new vocabulary from and comprehension of an aural L2 passage. Many related studies in the past pursued a purely quantitative approach (Chun & Plass, 1996a, 1996b, 1997a; Mayer & Sims, 1994; Plass, Chun, Mayer, & Leutner, 1998). While highly pertinent, these studies left many questions unanswered concerning students' views and experiences in a multimedia environment. This paper, therefore, more closely examines the students' voices, qualitative data that can provide immeasurable and even unanticipated information concerning the design and effectiveness of multimedia tools for listening comprehension. The results establish a clearer picture of the current needs and attitudes of L2 students in a multimedia environment and clarify how developers can better design listening comprehension materials and activities to enhance students' aural comprehension and make them feel as though they are indeed learning.

To bring the qualitative perspective to light, the research strategies used for both the quantitative and qualitative components will be fully disclosed while the statistical information, more completely discussed in Jones & Plass (2002), will be summarized. Students' voices will be represented by their exact quotes, and their remarks will be further highlighted by an appropriate literature review. It is through research such as this that we can promote change in our pedagogical and technological strategies and can find ways to facilitate students' acquisition of L2 aural skills in a multimedia environment.

METHODOLOGY

Quantitative Component

Participants and Design

The participants in this study were 171 English-speaking students enrolled in second semester beginning French at the University of Arkansas. At the time of testing, their mean estimated French GPA was 2.92 (4.0 = A, 3.0 = B, 2.0 = C, 1.0 = D, 0.0 = F).

A pretest/posttest control group (between-subject) design was employed to observe the effects of two factors—the absence or presence of verbal annotations and the absence or presence of visual annotations—on students' comprehension of the aural passage and their acquisition of vocabulary. Participants were randomly assigned to one of four treatments: the aural text (a) without access to annotations, (b) with only verbal annotations available, (c) with only visual annotations available, and (d) with both visual and verbal annotations available.

Dependent Measures and Scoring

The effects of the four aural treatments on students' comprehension and vocabulary acquisition were measured using an immediate multiple-choice vocabulary posttest and an immediate recall protocol posttest (dependent measures) then again, as delayed tests, 3 weeks later. The multiple choice vocabulary posttest was made up of 25 of the 27 keywords visible on the five computer screens of each treatment. The keywords were selected from the aural passage because of their importance in the text and because they could be represented in both a visual and a text-based format. Using the split-half reliability method, the internal consistency of the vocabulary test was .82.

The recall protocol comprehension test instructed students to summarize, in English, the aural passage they listened to. Two French professors chose the 63 propositions that represented the idea units of the passage. The test was scored based on the number of correct propositions given by each student, up to a maximum of 63 points. The interrater reliability of this measure was .97.

Materials

The listening comprehension software was developed using Adobe Premiere 4.2 and Authorware 4.0. The apparatus for presenting the materials consisted of a 22-station Macintosh computer language lab, arranged such that students could view only their own computer screen. Each treatment began with an introductory screen that placed the historical event in context (advance organizer) and instructed students on how to use the program. The five separate screens that followed contained a total of 27 keywords, positioned on the left side of the screen, accompanied by ellipses to indicate missing words and thus to emulate the flow of the dialogue. Audio play buttons were positioned to the left of each text segment such that students could have equally available access to the predivided syntactic chunks of the passage (Meskill, 1996; O'Malley, Chamot, & Küpper, 1989). An icon of a speaker was present on each screen so that students could drag a keyword to it to hear the word pronounced.

The four treatments differed as follows:

1. In the Control group (Trt. 1), no annotations for the key vocabulary words were available except for the option of hearing them pronounced.

2. In the Visual group (visual annotations only, Trt. 2), a camera icon was present to the left of the pronunciation icon. Students could drag a keyword to the camera icon to view its image.

3. In the Verbal group (verbal annotations only, Trt. 3), a text icon was present to the right of the pronunciation icon. Students could drag a keyword to the text icon to view its text-based translation.

4. The Visual and Verbal group (visual and verbal annotations, Trt. 4) contained both the camera and the text icons, in addition to the pronunciation icon. Students could drag a keyword to the camera icon to view its visual annotation or to the text icon to view its verbal annotation of the keyword (see Figure 1).

The visual annotations consisted of 14 color drawings and 13 color photos. The text annotations were written in bold, 14 point Helvetica font. Students could select any annotation available in their treatment as often as desired, before, during, or after presentation of each aural segment.

Visual and Group Treatment

Procedures

Students first participated in this segment of the study during two consecutive 50-minute class periods (25 minutes in the first class period, 50 minutes in the second). On the first day, students filled out a paper-and-pencil demographic questionnaire and were then given 8 minutes to complete a vocabulary pretest. On the second day, students were randomly assigned to one of the four treatment groups. They were given 14 minutes to listen to the passage and were instructed to look up all annotations available in their treatment. They were then given 8 minutes to summarize all that they could from the passage and then 8 minutes to complete a vocabulary posttest, identical to the pretreatment vocabulary test. Three weeks later, students completed unannounced delayed vocabulary and comprehension tests that were administered in the same way as the immediate measures.

Qualitative Methods

Participants

Twenty students were purposively selected to participate in interviews so as to gain a clearer understanding of the effects of the different media types on students' comprehension and vocabulary acquisition

 Interviews

Throughout each interview, the researcher remained unbiased and accepted each student's remarks openly. Member checks ensured that what the students said indeed reflected their true beliefs. Unfortunately, all interviews took place 4 to 5 weeks after students' experiences with the treatments, an unavoidable situation of "give and take" so as to eliminate any influence of the interviews on the delayed comprehension and acquisition tests. Therefore, the "lateness" of the interviews potentially lessened students' expression of their experiences with their respective treatments.

Data Analysis

All interviews were closely analyzed to identify any emergent data patterns. Since the data were processed electronically, the researcher looked for consistent global themes; in particular, information that underscored students' experiences with the material and their attitudes toward the different annotations available. Once prepared, the transcripts were more closely reviewed numerous times to identify further unanticipated patterns and individual remarks relating to the quantitative results. The identified information was numerically coded and organized based on themes which were more deeply analyzed to reveal any subtleties that could further support the statistical outcomes. Primary themes were retained for discussion based on their relevancy, either in a supportive or a contradictory manner. Qualitative analysis thus provided a more triangulated approach, and exact quotes are presented in the results and discussion section below.

RESULTS AND DISCUSSION

The underlying theme of this study focused on aural information processing in French as influenced by access to verbal (word translations) and/or visual (images) presentation modes. It was believed that students would comprehend an aural passage and acquire vocabulary best when they had access to and actively selected visual and verbal annotations accompanying the aural material, while students who had access to only one mode (visual or verbal) would perform at a moderate level, and those without access to any annotated information would perform the poorest. Similar research on the comprehension of written texts in a multimedia environment suggests that when students access multiple annotation types (visual and verbal) that accompany a reading comprehension passage, learning is more likely (Chun & Plass, 1996a, 1996b, 1997a; Plass et al., 1998). However, these studies were conducted without inclusion of students' opinions toward the material with which they worked. The focal point of the study presented here is: What can students tell us to better clarify how visual and verbal annotations might help them to comprehend information aurally presented in a multimedia environment? To understand students' views more fully, each hypothesis will be presented followed by a summary of the quantitative results obtained. The remainder of the discussion will then focus on students' views of the helpfulness of visual and verbal annotations for supporting listening comprehension in a multimedia environment.

Hypothesis 1

The first hypothesis argues that students who complete a listening comprehension activity that contains a choice of visual and verbal annotations will recall more propositions of a listening text than those who complete listening tasks with single annotations (visual or verbal) or no annotations. The quantitative results of the study support this hypothesis.

Students' construction of knowledge from an aural text requires "learning on a higher level, including understanding words in context as well as propositions" (Plass et al., 1998, p. 27). Thus, the deeper processing required of and supported by images leads to greater knowledge construction. Initially, this process seems different from vocabulary acquisition where students often rely on rote memorization of word translations to learn the meanings of words. Despite the shallower reputation of vocabulary acquisition, the availability of different annotation types (visual and verbal) and the amount of invested mental effort (Salomon, 1983) that students give to a particular annotation mode suggest that deeper processing may also be key to acquiring vocabulary. Therefore, it is a discussion of the second hypothesis and vocabulary acquisition in an annotated listening comprehension environment to which we now turn.

Hypothesis 2

The second hypothesis examines how student’s best acquire vocabulary when the availability of annotation types varies from treatment to treatment. It was proposed that students who completed a listening comprehension activity that contained visual and verbal annotations would acquire more vocabulary words than those who completed the listening task with single annotations (visual or verbal) or no annotations.

SUMMARY

Throughout this study, students performed best when they had access to visual and verbal annotations, moderately well when they had access to visual or verbal annotations alone, and poorest when no annotations were available to them. In most instances, access to multiple annotations led to significantly higher results than the other annotations. Students' comments supported these results in that those who worked without annotations had the lowest opinion of the activity as well as the lowest scores on all dependent measures; the absence of annotations prevented them from understanding the passage and learning the vocabulary. The Verbal group (Trt. 3) was more positive and believed that the word translations helped them understand and recall the vocabulary. The Visual group (Trt. 2) and the Visual and Verbal group (Trt. 4) believed that visual annotations allowed for deeper processing of the aural passage and thus longer retention of the material and greater vocabulary knowledge. Images, in fact, did lead to deeper processing of the aural text; the mapping of visual input with the aural input into a mental model resulted in stronger recall than the mapping of words. As a result, these two groups rarely differed significantly from each other though access to both annotations consistently led to the highest results on all dependent measures.

Students in the Visual and Verbal group also confirmed that choosing from multiple annotation types created a more individualized, balanced, and interactive approach to the activity. The choice of annotations led to more than one retrieval route to the information, a strategy that was particularly helpful when either the image or the translation was not clear or when students' cognitive abilities were low for a particular annotation type. The presence of visual and text annotations allowed students to select from the material and to build referential connections between two, if not three, systems, thereby resulting in more retrieval routes to the vocabulary items and added effects on recall.

The qualitative data also highlighted students' views of the amount of mental effort needed to process visual or verbal annotations. Overall, those who worked with any annotation type were positive about their treatment. However, some believed that English translations were more effective and efficient for learning vocabulary than images because images required more effort to process word meanings. Students' remarks appear to corroborate Salomon's (1983) theory that the less mental effort placed on a given task, the less successful the outcome. Students who believed that verbal annotations were easier to process may have expended less effort with such annotation types and, therefore, retained less vocabulary on the delayed vocabulary test and only slightly understood the passage. Images, on the other hand, were viewed as the more difficult form of annotation with which to work. Students may have used more mental effort to process these annotations, and therefore performed significantly better on all recall protocol measures than those with verbal annotations alone. This same attitude may have helped them to perform better on the delayed posttreatment vocabulary test than those with verbal annotations alone. Students in the Visual and Verbal group (Trt. 4) also reconfirmed that choosing from multiple annotation types created a more individualized and balanced approach to the activity, a strategy that was particularly helpful when either the image or the translation was not clear. This outcome, once again, reconfirms Mayer's (1997, 2001) generative theory of multimedia learning and demonstrates that the treatment design itself benefited students' comprehension and vocabulary acquisition more so than did the amount of invested mental effort. The presence of visual and text annotations allowed students to select from the material and build referential connections between multiple systems, thereby resulting in more retrieval routes to the vocabulary items and greater comprehension of the aural passage.

CONCLUSION

In a pilot study conducted during the Fall of 1998, students remarked that they struggled with listening comprehension activities when there was a lack of choice for information. In particular, students struggled with aural exercises when they had no preference for the pedagogical design nor could receive any relevant contextual assistance to help them better comprehend the passage. The absence of choice compromised their ability to understand the spoken message. A theoretical implication for combining listening comprehension activities with multimedia is that organizing aural material into working memory seems to be aided by students making connections between the visual and verbal systems. As stated by the students themselves, the presence of visual and verbal annotations helped them to link information with the aural message and thus better retain information in long-term memory for later comprehension and vocabulary recall. Students' voices, in addition to the quantitative results summarized above, extend Mayer's (1997, 2001) generative theory of multimedia learning to listening comprehension since selection from the material available and construction of referential connections between visual, verbal, and aural representations gave students more retrieval routes to the information to recall vocabulary words and to comprehend the passage.