Using Formative Evaluation in the Development of Learner-centered Materials

Carl S. Blyth
University of Texas
James N. Davis
University of Arkansas

Abstract:
In this article we report on an 8-year process that included three successive iterations of the following cycle: (a) development of instructional technology, (b) formative evaluation, and (c) modification of the technology. From the first formative evaluation to the last, our students told us that they found heavily contextualized language difficult to learn and frequently requested more decontextualized language for textbook presentations and for practice. With the aid of formative evaluation data (e.g., performance data based on think aloud protocols, attitudinal data, retrospective interviews, and course surveys), we tried to strike a balance between what students said they wanted (i.e., more decontextualized language input and practice) and what we as language teachers and curriculum developers believed that they needed (i.e., more contextualized language input and practice). Three theoretical constructs proved particularly relevant in helping us interpret the data: activity theory (Lantolf & Appel, 1994; Lantolf, 2000); the naïve lexical hypothesis (Bland, Noblitt, Armington, & Gay, 1990); and the lexical approach (Lewis, 1993), also known as the lexical syllabus (Willis, 1990). We argue that when formative evaluation becomes a central part of the development of instructional technology, the results are a more learner-centered curriculum with more user-friendly technology.

INTRODUCTION

In this article we report on an 8-year process that resulted in a technology-enhanced beginning French program at the University of Texas at Austin. In order to understand better what worked and what did not, the development process included successive iterations of formative evaluation followed by modifications to technological tools and classroom practices. We describe this longitudinal process by focusing on three iterations of development and evaluation. In the first iteration, the developers attempted to make the content of a textbook more accessible and understandable.¹ In the second, they tried to develop a useable pedagogical reference grammar.² In the third, they built an online program for beginning French complete with audio and video clips, self-correcting grammar exercises, downloadable textbook, and web-based writing activities. We argue that when formative evaluation becomes a central part of the development of instructional technology, the results are a more learner-centered curriculum with more user-friendly technology. Nevertheless, formative evaluation also presented thorny challenges. For example, do students really know how they learn best? How can developers use formative evaluation as a tool to help discern when student desires indicate legitimate student needs?

LITERATURE SURVEY

The present study is also motivated by a paucity of literature on evaluation in computer-assisted language learning (CALL). Ehrmann's comment about limited information on the evaluation of all types of instructional technology argues for more studies that might provide suggestions for developers. With regard to CALL, this assertion was supported in recently published research by Zhao (2003, p. 7) who noted--in a meta-analytic investigation--that "the number of systematic, well designed empirical evaluative studies of technology in language learning is very small," as well as by Chenoweth and Murday (2003, p. 287), who affirmed that "evaluation of student learning in CALL environments remains relatively rare."

Evaluation: Summative and Formative

The role of evaluation, as opposed to applied primary research, is the focus of this article. As Patton (2001) suggests, evaluation differs from research in several fundamental ways: (a) evaluation is "decision driven," while research is "hypothesis driven;" (b) evaluation is aimed at influencing the making of decisions in a specific context, while research tends to have broader implications; and (c) evaluation typically uses a wider variety of information elicitation methods than does primary research.

Evaluation is typically spoken of as being either formative or summative. Evaluation is deemed formative when "the intention is to identify scope and potential for improvement" (George & Cowan, 1999, p. 1). In the case of educational technology, formative evaluation focuses on the collection of information that will enable designers to improve the materials. In formative evaluation, then, the emphasis is on the development process. In contrast, summative evaluation focuses on an end product, with no implications for change.

Examples of CALL Evaluative Studies

The investigations noted above illustrate Patton's statement that evaluators often use a wide variety of procedures to collect information. We also employed several different methods to collect information on user satisfaction and performance (e.g., think aloud protocols, retrospective interviews, attitudinal surveys, and instructors' logs). The way our account differs from those cited above is in its emphasis on the way formative evaluation was directly linked to modifications of technological tools and classroom practices. Furthermore, most of the evaluative studies mentioned above report on single, short-term projects while ours reports on a longitudinal process involving successive iterations of development and evaluation over an 8-year period.

PROJECT DEVELOPMENT HISTORY

In this section, we recount three successive iterations of project development followed by formative evaluation and then modification. The first iteration (1994-1997) involved the creation of online grammar drills, a CD-ROM, and web-based writing activities to accompany a commercially produced first-year French textbook. The second iteration (1998-2000) resulted in an online French reference grammar. Finally, the third iteration (2001-2005) focused on a complete online first-year French course with multiple media components (e.g., audio, video, downloadable textbook, and self-correcting exercises).

First Iteration (1994-97): Parallèles Interactive

In 1994, the lower division French program at the University of Texas at Austin adopted a new textbook called Parallèles: Communication et culture (Allen & Fouletier-Smith, 1995) due in large part to its cultural emphasis. While the instructors approved of the textbook's content, they complained that the book was less than user friendly. The most problematic aspect of the book was reportedly the opening section of each chapter. This section presented vocabulary and grammar in heavily contextualized formats, which were not only quite lengthy but, most crucially, lacked glossing. Moreover, while these texts were recorded on an accompanying audiocassette, many students (as well as their instructors) reported that this component was rarely used. In addition to the aforementioned issues with the opening sections of the chapters, students often expressed a desire for more form-focused grammar exercises than were available in the textbook and accompanying workbook. Furthermore, many students found the grammatical explanations unclear because they were not familiar with the textbook authors' metalanguage (i.e., the names and functions of parts of speech). And, finally, both students and instructors found the end-of-chapter synthetic activities well beyond the proficiency level of beginners.

Development

The developers decided to address the problems directly by creating materials that would make the textbook's presentations and activities more accessible. To that end, they developed three different computer tools: online grammar drills, a multimedia CD-ROM, and web-based activities.⁴ The grammar drills followed the textbook's grammatical syllabus. The drills required students to fill in a blank and submit the set of answers for automatic correction and feedback. In contrast to the discrete point grammar drills, the CD-ROM's aim was to contextualize the vocabulary and grammar with the aid of texts, realia and photos⁵ (for further information, see http://www.laits.utexas.edu/fr/pi).

A typical screen in the Parallèles Interactive CD-ROM included a color picture or graphic (e.g., a journalist from Senegal) illustrating a short dialogue or text (see Figure 1).

Students were encouraged to listen to the dialogue, to check their comprehension by answering questions, and to record their pronunciation of words and phrases to be compared with the audio. By clicking on any underlined word or group of words within the text, students could gain access to lexical, grammatical, pragmatic, and cultural notes which appeared in the two boxes on the lower right of the screen. Buttons at the top of every screen in the program gave access to various functions. From left to right these functions included

1. an image or diagram specific to the chapter,

2. a database of Internet sites thematically correlated with each chapter,

3. the recording function,

4. the dictionary function,

5. comprehension questions (Q1-Q4), and

6. navigation buttons.

In addition to the grammar drills and the CD-ROM, the developers built web-based activities to help students to synthesize information gleaned from practice with the first two components. Students were typically required to produce a paragraph or more of written French in response to specific questions related to the websites provided. These end-of-chapter activities obliged the students to take part in a communicative task.

Formative evaluation 1

Surveys. To examine attitudes and practices concerning the computer tools, an end-of-the-semester survey was administered to 216 students enrolled in the first-semester course. The survey items were aimed at uncovering different kinds of information: general level of computer knowledge (e.g., "I have a good aptitude for computers" and "I often use email"), students actual use of computers (e.g., "I recorded myself frequently …"), and student assessment of the effectiveness of the computer technology (e.g., "The computer lab has improved my writing skills"). The survey also included a few open-ended questions to elicit general comments about how to improve the use of computers in the course. The results demonstrated a largely positive reaction to the use of the new technology. Furthermore, the data showed that the students (a) preferred using the online grammar drills, which were discrete point and provided immediate feedback, (b) liked--although to a lesser extent--the CD-ROM, which was intended to facilitate access to the contextualized grammar and vocabulary, and (c) disliked the web-based activities, which were open ended and attempted to synthesize the entire chapter.

Think aloud protocols. Following the surveys conducted in the spring of 1996, a dissertation study was conducted during the summer on 10 students enrolled in a 6-week summer session of the same course (see McCormick, 1997). This investigation utilized two well known introspective techniques to assess as directly as possible students' attitudes and behaviors: think aloud protocols and student interviews, commonly referred to as concurrent and retrospective verbalization in the psychological literature (Ericsson & Simon, 1994). The treatment consisted of individual sessions lasting approximately 20 minutes each during which the student was asked to work with the CD-ROM to preview the chapter.

Retrospective interviews. In addition to the think aloud protocols, retrospective interviews were conducted to provide richer profiles of student-computer interaction. The interview data confirmed the positive user reactions found in the surveys. The CD-ROM materials also received favorable reviews but were reported to be less useful than the grammar drills. However, most students expressed clear dissatisfaction with the web-based activities since they were judged to be too time consuming for their worth.

Activity theory. Although McCormick's analysis was not framed in terms of activity theory, we found that this approach to understanding student behavior provided a better explanation of the data. A construct of Vygotskyan sociocultural theory, activity theory posits three distinct levels of analysis in order to account for human behavior: activity, action, and operation. The level of activity is closely linked to motives and to the larger sociocultural setting (e.g., work, play, education, leisure, etc.). In the context of our research, the activity was the university language classroom setting, the action was the student's concrete goal (which differed from the developers'), and the operation was the actual student behavior while using the CD-ROM.

Summary. The various data collection procedures used in the formative evaluation of the first iteration of the materials yielded important information about the product. Both students and instructors expressed high levels of satisfaction with the computer tools. At the same time, convergent data from the student surveys, the think aloud protocols, and individual interviews corroborated the need for significant changes to the CD-ROM, the chapter-ending web-based activities, as well as the ways in which instructors used this technology in their classes. An analysis of student data informed by activity theory revealed that students understood the CD-ROM as a tool for vocabulary learning. The pedagogical goal that developers had originally envisioned for the CD-ROM, a tool to help students explore and deepen their understanding of the relations between vocabulary, grammar and culture, proved to be beyond the developmental level of the students whose mental conception of the L2 was consistent with the naïve lexical hypothesis.

Modifications

The formative evaluation data indicated that no modifications of the online grammar drills were called for. However, significant modifications of the CD-ROM and Internet activities were deemed necessary.

1. The CD-ROM's glosses were greatly increased while the glosses themselves were limited to straightforward translations in order to conform to the students' overwhelming preference for lexical information as opposed to grammatical or pragmatic information while reading target language texts.

2. The pedagogical goals of the CD-ROM were made more explicit, and different tasks were developed to motivate students to use more of the available functions of the computer program (e.g., students were told to read the multimedia texts in preparation for an oral interview with their instructor to motivate greater use of the listening and recording functions).

3. The web-based activities were significantly edited by reducing the number of websites required for each activity, eliminating problematic websites, and embedding relevant vocabulary/grammar reminders in the directions of each activity.

Second Iteration (1998-2000): Tex's French Grammar

As noted above, during the first round of formative evaluation, students had reported high levels of satisfaction with the online grammar drills, praising their relevance and ease of use. In contrast, they had reported low levels of satisfaction with the textbook's grammar explanations. The most common complaint was that the grammar explanations found in the textbook assumed knowledge (e.g., names of parts of speech) that many students did not possess. Furthermore, faculty teaching upper level courses asserted that students coming from the lower division courses required increasing grammatical remediation.

In light of the widespread perception that this component of the beginning language program was highly problematic, it was decided that the textbook should be accompanied by a grammar supplement produced in house. It is also important to note that the developers were strongly encouraged to seek a technological solution to these issues because of campus-wide interest in and readily available monies for courseware development.

Development

n order to provide more coherence to the pedagogy and grammatical content of the first two years of the French language program, it was decided that a website should be built. Given that the second year of the French language program comprised many different courses, each using a different textbook, the website was conceived of as a stand-alone product. The resulting website, called Tex's French Grammar, is an online pedagogical reference grammar that combines grammatical explanations with cartoon images and humorous dialogues woven together to tell a hyperlinked story.Arranged like many traditional reference grammars with the parts of speech (e.g., nouns and verbs) used to categorize specific grammar items (e.g., gender of nouns and irregular verbs), the website features explanations in English, recorded French dialogues with translations, and self-correcting fill-in-the-blank exercises. To facilitate reference and learning, all grammar items are thoroughly cross linked. In direct response to upper-level course professors' complaints about class time being used for grammatical remediation, the developers decided to define all grammatical terminology more completely than most commercially produced materials. For example, the definitions of infinitive, conjugation, and paradigm can all be found in the Introduction to Verbs page.

Formative evaluation 2

The first round of formative evaluation had employed many different methods for data collection (e.g., think aloud protocols, interviews, observations, and surveys). In contrast, the second round of evaluation relied solely on course evaluations that included questions pertaining to Tex's French Grammar. Different evaluation methods were chosen in keeping with the higher levels of student familiarity with computer-mediated materials. Since foreign language educational CD-ROMs and websites were still relatively new in the mid 1990s, the developers were unsure how students at that time were using Parallèles Interactive and what they thought about it. Several years later, students were much more knowledgeable about the Internet as well as about its pedagogical uses. More important, the developers were confident that the students were already familiar with the methods and practices inherent in Tex's French Grammar (i.e., explicit grammar presentation followed by discrete-point form-focused drills).

Modifications

While students praised the content of the site (e.g., clear explanations, useful exercises, etc.), they complained that they had difficulties using the site successfully (given issues of confusing navigation and opaque terminology). How could the developers help grammatically naive students to use a reference grammar that was organized according to traditional grammatical categories? As previously noted, unlike most pedagogical grammars that simply follow a textbook's scope and sequence and do not require any previous background knowledge of grammar, Tex's French Grammar was conceived of as a stand-alone reference grammar. After much discussion the developers made several modifications to the interface along the lines suggested by Bland et al. (1990).

1. Pop-up navigation windows were added that automatically linked related grammar items to show more explicitly how grammatical categories interrelated. For example, when students access the possessive determiners page, links for all articles and determiners are displayed in a pop-up window at the top right hand corner of the screen.

2. A key word search function was added to allow students to access grammatical items as vocabulary words in keeping with the naïve lexical hypothesis.

3. Icons were added to navigate major sections of the site (e.g., homepage, index, help, search, etc.).

4. The website was more carefully integrated into each course's syllabus and printed materials (i.e., website grammar points were numbered and indexed to sections in the textbook).

5. A workshop on how to use the website was added to each course at the beginning of every semester.

Third Iteration (2001-2005): Français Interactif

Student feedback during the successive iterations of formative evaluation had often included ideas that were not realized in any commercially produced materials. The recurring theme in this feedback was that users wanted a clearer and more deliberate progression in each chapter from decontextualized vocabulary words to contextualized discourse. In particular, students kept asking for more practice aimed at the word level so that they would feel more comfortable tackling contextualized language samples.

Lexical approach. As the developers searched for a new role for lexis (i.e., words, collocations, and unanalyzed phrases) in the instructional program, they were guided the so-called lexical approach. The lexical approach as described in Lewis (1993) and Willis (1990) is an alternative to grammar-based approaches to language learning. Moudraia (2001, p. 1) claims that the essential feature of the Lexical Approach is "the idea that an important part of language learning is the ability to comprehend and produce lexical phrases as unanalyzed wholes, or 'chunks,' and that these chunks become the raw data by which learners perceive patterns of language traditionally thought of as grammar." Lewis (1993, p. vi) enumerates several key principles of this approach.

Development

The developmental team set out to design a program based on suggestions from students and guided by their understanding of the lexical approach. Lewis (1993) notes that language teaching has traditionally been organized around the supposedly central grammatical system. In the lexical approach input and practice are structured in order to guide students to collocate words and to grammaticalize meaning from word to sentence to discourse. The developers began by focusing on the aforementioned concerns expressed by students from the time of the very first iteration: (a) the lack of pedagogical sequences that slowly increased the degree of language contextualization and (b) the lack of an obvious relationship between the parts (vocabulary and grammar items) and the whole (communicative tasks). Next, they identified various communicative tasks that they wanted their students to be able to perform (e.g., identifying the members of one's family, describing a friend's personality, narrating the events of the day, etc.). Then, they videotaped native speakers performing these tasks and inventoried the vocabulary and grammar spontaneously used to perform the task. These inventories became the basis for the lexical and grammatical syllabus for each chapter.

Formative evaluation 3

Judging from the positive, yet informal, comments from students and instructors, our online program entitled Français Interactif was well received. Other than such indirect, anecdotal evidence, however, there was no explicit feedback regarding user satisfaction. It was deemed necessary, then, to conduct another evaluation study. As always, the aim was not to overhaul the materials completely, but rather to pinpoint elements still needing modification. Given this goal, another survey of student attitudes seemed the most efficient means of collecting the desired feedback. Examination of the student comments confirmed that the web-based writing activities remained the least popular component of the online materials. A few of the written remarks indicated that some students had at least understood our rationale for including these challenging activities (e.g. "I like the internet writing activities the best because you get to sort of incorporate what you're learning with something you like (i.e., fav. Artist, place to visit, etc.)"). Criticism of the Internet writing activities tended to focus upon the difficulty of finding and using "the actual things on the website (i.e., the elements students were to draw upon to complete the writing activities)." Others expressed concern over problems comprehending and utilizing authentic materials such as those found at websites (e.g., "I did not like some of the internet activities because I did not know a lot of the needed vocabulary and felt I was only copying what the website said rather than using my own thoughts.").

Modifications

Most of the changes made to the original version of Français interactif reflected, yet again, the desire to take into account student desires (in particular, more emphasis on vocabulary learning and a slower progression from decontextualized to contextualized language) as well as student needs (most especially, the ability to comprehend and use the target language in context).

CONCLUSION: LESSONS LEARNED

This 8-year iterative process of development and evaluation demonstrated that students wanted two things: a greater emphasis on vocabulary as opposed to grammar and more decontextualized language used in presentation and practice. These student preferences ran counter to the developers' presuppositions about how language should be taught, that is, with a greater emphasis on contextualized grammar from the beginning. As a direct consequence of formative evaluation, the program's materials evolved from a computer-based ancillary for a printed textbook to an online multimedia instructional program. The developers believe that their materials have also improved and that the improvements are directly attributable to a clearer understanding of their students' preferences. Clearly, research on the role of formative evaluation in CALL is still in its infancy as demonstrated by the paucity of related studies. As Chenoweth and Murday (2003) and Zhao (2003) have contended, many more studies are needed for guidance in projects such as the one described here.

The data collected in three cycles of formative evaluation provided clear guidelines for meaningful improvements to our materials. Convergent data (in the first formative evaluation) and recurrent themes (throughout the entire development process) were especially helpful in this endeavor. Different types of data were needed at different moments in the development process. For instance, protocol analyses were not conducted in the second and third formative evaluations since the developers believed that they already had an adequate understanding of student-computer behaviors in the context of their materials.

In order to make sense of the student data collected over the years and to generalize these findings beyond the local context, three theoretical constructs proved particularly useful: activity theory, the naïve lexical hypothesis, and the lexical approach. Activity theory helped the developers to understand students' behavior by relating it to the students' specific goals for a pedagogical task and to their general motives for language learning at the college level. The naïve lexical hypothesis accounted for beginning students' preference for lexical information (as opposed to grammatical or pragmatic information) when attempting to read or compose in a foreign language. The developers came to accept that this preference was simply an unavoidable consequence of the beginning stage of language learning. The lexical approach was key in helping the developers to resolve their professional misgivings about the inclusion of decontextualized lexis in pedagogical materials. While the current program does not follow the lexical approach in all respects, it has clearly been informed by many of its principles, namely, that lexis is the logical starting point for beginning language learning.

In conclusion, we argue that one of the principal engines of innovation in the production of foreign language materials resides in locally produced projects such as those described here. Anyone attending a professional meeting hears criticisms of currently available textbooks as being conservative and generic and, thus, ill fitted to a given institution's particular (and actual) needs. This state of affairs is largely an artifact of the complicity that exists among the various stakeholders: commercial publishers, textbook writers, reviewers, and language program directors.

The 8-year project reported on here has suggested alternative solutions to the conundrum described above in at least two important ways: (a) the developers produced their own language teaching materials because they realized that no commercially produced materials met their program's needs, and (b) a large part of this development process entailed listening to students' voices with discernment.

Throughout this process the developers discovered that they often thought they knew how their students learned best, but in actual fact, they often did not. The lesson here is that teachers and curriculum developers will never know with any certainty without asking.