A Visual Display for the Teaching of Intonation

Gerard W. G. Spaai
and Dik J. Hermes
Institute for Perception Research/IPO, Eindhoven, The Netherlands

Abstract:
Those learning foreign languages often experience problems in generating correct intonation. Therefore, several attempts have been made to support the teaching of intonation with the help of devices that display the pitch of speech. This paper deals with the development of a visual intonation-display system. In this system, the visual feedback of intonation is given as a continuous representation of the pitch contour containing only the perceptually relevant aspects of the pitch contour. The course of the pitch contour is approximated by a small number of straight lines. This pitch-contour representation is supposed to facilitate the interpretation of the visual feedback of the pitch contour. In many languages, correct positioning of the pitch movements with respect to the vowel onsets is crucial to proper intonation. Vowel onsets are therefore measured and displayed, too. A description of this visual-intonation display system is given, and its possible uses for teaching intonation are discussed.

INTRODUCTION

In teaching a student to speak a foreign language most attention is devoted to the correct pronunciation of sounds and isolated words. Generally speaking, much less attention is paid to a correct production of intonation. However, it is well known that linguistic, syntactic and semantic information is more easily conveyed when a speaker produces the correct variations in pitch in a speech utterance (Crystal, 1981). For instance, correct production of pitch variations helps the listener to signal the location of clause and phrase boundaries, i.e. syntactic boundaries (Cooper and Sorensen, 1977), to identify prominent syllables (Bolinger, 1958; Van Katwijk, 1974), to differentiate questions from statements (Eady and Cooper, 1985), and to make new distinctions (Crystal, 1981). Listeners also use pitch information in non-linguistic tasks such as determining a speaker's emotional state (Lieberman and Michaels, 1962), personality traits and the professional status (Crystal, 1981). It is also argued that intonation can facilitate interactions in conversations by signalling a person to respond or by signalling the end of a conversation (Sacks, Schegloff and Jefferson, 1974; Geluykens and Swerts, 1992). Thus, intonation serves three important communicative functions which are relevant to the task of deciphering connected speech, vis.: a pragmatic function, a grammatical function, and a comprehension function.

Typically, an intonation contour is simply presented for imitation without any reference to meaning, or any contrast with other possibilities. O'Connor and Arnold (1973) claim that after a period of drill, students will be able to automatically choose an appropriate melodic contour. However, training a particular intonation contour on a set of unrelated sentences inhibits any insight into the contexts in which the pattern in question is appropriate or the underlying general meaning. Exposure to phonetic manipulation of a particular intonation without any kind of contextual justification is in fact likely to lead to faulty generalizations, because of the absence of feedback as to when it is not applicable.

There is a dear need for intonation to be taught with sufficient verbal and non-verbal context for meaning to be apparent, and for contrasts in form and meaning to be made continually throughout the teaching. Thus, an important drawback of the drill method may be that the learners are not capable of generating new instances not yet encountered. Furthermore, it is assumed that the learner will not acquire any insight into the underlying system of rules, nor will the learner be able to make comparisons with the intonational system of the mother language and the intonational system of the target language. Instead of a drill method it could be more profitable to use a more cognitive approach in the teaching of intonation. Such a method aims at making the students conscious of intonational structures of the target language by providing them with an explicit description of this particular aspect of the language in the form of a rule-based intonation course or grammar and by training them in listening analytically to pitch phenomena. Unfortunately, however, the formulation of explicit rules for the generation of acceptable intonation courses has not been achieved yet for many languages. Thus, the use of a cognitive approach for teaching intonation is often not possible.

VISUAL FEEDBACK OF INTONATION

Acquiring the intonation of a foreign language is difficult. Or, in the words of Leon and Martin (1972) "Of all the elements of a target language, the intonation appears to be the most difficult to acquire...." There are two important factors that contribute to the problem of acquiring the intonation of a foreign language. First, because the intonation in infants is learned at a very early stage in the language-acquisition process Liebermann, 1967), it is most resistant to change. Crystal (1969) pointed out that at a very early age, children respond to the intonation of the voice at the expense of speech content and that non-segmental patterns are understood and produced prior to anything conventionally grammatical. Second, as a result of the fact that suprasegmental patterns are particularly deep-rooted, foreign language learners often superimpose the prosodic features of their mother language on the sounds of the foreign language. For this reason foreign-language learners are often not aware of any differences in intonation between the mother language and the foreign language. Visualization of intonation may help the foreign-language learner to become more aware of these differences and may also support the teaching of intonation to foreign-language learners. The idea of visualizing intonation is not a new one. Anderson (1969) and Abberton and Fourcin (1975), for instance, used it for the teaching of intonation to deaf persons and Vardanian (1964) used it in an attempt to teach English as a second language. Several studies have reported positive effects of the use of visual intonation-display systems on the learning of intonation to foreign-language learners (Lane and Buiten, 1966; Hengstenberg, 1980) whereas no such effects were found in other studies (Vardanian, 1964; Wichern and Boves, 1980; De Bot, 1982). None of these display systems came into widespread use. Although some of them might be useful in the teaching of intonation, they are not widely used for the following reasons: (1) the unreliability of the measurement of pitch, (2) the high cost of the systems, (3) the lack of a systematic curriculum on teaching intonation to foreign language learners, and (4) the lack of feedback processing, as in most cases, pitch was measured and directly fed back to the learner. In teaching applications, such perceptually irrelevant pitch variations, should be removed from the visual presentation. In conclusion, for the purpose of teaching intonation the visual feedback of the pitch contour should be provided as a continuous representation from which the perceptually irrelevant pitch variations are left out in order to facilitate the interpretation of the visual feedback of the pitch contour.

The second difficulty that arises when the second-language learner is provided with the unprocessed visual feedback of the pitch contour, is that the perceptual identity of a pitch movement in, for instance Dutch, depends on its position within a syllable, in particular with respect to the vowel onset. It appears that the vowel onset is one of the most relevant anchor points in this respect ('t Hart and Collier, 1975;'t Hart, Collier and Cohen, 1990).

For example, one of the rising pitch movements used for accentuation (i.e., a prominence-lending pitch movement) in Dutch starts about 70 ms before the vowel onset. Another prominence-lending pitch movement starts about 80 ms after the vowel onset. A continuation rise, as occurs at a major phrase boundary, starts even later in the syllable and does not lend prominence to a syllable. Hence, a correct interpretation of the pitch contour of a sentence requires that the pitch movements are presented in relation to the vowel onsets of the syllables in which they occur. Visual feedback of the pitch contour should, therefore, show the moments of occurrence of the vowel onsets.

THE INTONATION METER¹

In order to facilitate the interpretation of the visual feedback of the intonation contour, a visual intonation-display system was developed, the so-called Intonation Meter (Spaai, Storm and Hermes, 1991), that presents visual feedback of the intonation as a continuous representation of the pitch contour and contains only the perceptually relevant aspects of the intonation pattern. The course of the pitch contour is approximated by a small number of straight lines resulting in a so-called stylized pitch contour('tHart,CollierandCohen,1990). This means that unvoiced parts are interpolated and the original pitch contour is simplified into a sequence of straight fines without affecting the perceptually relevant properties. It is assumed that stylized pitch contours are more suited for the purpose of teaching intonation than unprocessed pitch contours since they will be easier to interpret. In addition the vowel onsets are measured and indicated. The measurement of vowel onsets is based on the assumption that vowel onsets are characterized by the appearance of strong resonance (format) peaks in the amplitude spectrum of the speech signal. This leads to the concept of "vowel strength," i.e. the combined strength of the spectral peaks below 2500 Hz. The instants at which the vowel strength rises rapidly are assumed to be the vowel onsets (Hermes, I 990).

During operation the Intonation Meter displays a pitch contour example produced by the foreign-language teacher on the upper part of a computer screen. This pitch-contour example must be imitated by the student whose performance is shown on the lower part of the screen. While imitating, direct unprocessed feedback is given by means of real-time pitch measurements. After finishing the utterance, the stylized contour including indications of moments of occurrence of the vowel onsets, is displayed too. Furthermore, the second-language learner can listen to the model utterance and their own imitation as many times as she or he wishes. The pitch movements in speech intonation are expressed on a psycho-acoustic scale which takes the frequency selectivity of the auditory system into account. The psycho-acoustic frequency scale is an intermediate between a linear and a logarithmic frequency scale. This frequency scale is used because it is the only one whereby pitch movements with similar excursions but presented in different pitch registers lead to the same degree of prominence (Hermes and Van Gestel, 1991). Thus, the pitch contours produced by a teacher of foreign languages and those produced by a student, or the pitch contours produced by a man and those produced by a woman, can be compared directly.

A detailed description of the Intonation Meter can be found in Spaai, Storm and Hermes (1991).

FUTURE DEVELOPMENTS

The visual intonation-display system will be used in a program for training the intonation of a foreign language. In this training program the foreign-language learner will be made conscious of differences in patterns of intonation between the mother language and the foreign language and will learn to produce the intonation patterns of the foreign language. This training program contains, among other things, a specification of the sequence in which learning materials are to be presented and the different intonation contours are to be taught. Furthermore, it will contain three different sets of exercises (Brazil, Coultheard and Johns, 1980) that can, in part, be implemented in the visual intonation-display system.

In the first set of exercises, auditory discrimination, the foreign-language learner will be made aware of the intonation patterns of the foreign language and the mother language. It is also necessary that the foreign language learners become familiar with the different functions of these intonation contours. This may help the student to acquire the intonation of a foreign language. MacCarthy (1976) makes several proposals for an auditory training program aiming at teaching foreign language learning. It is suggested that such a program should start presenting speech sounds to stimulate pure sound perception and then gradually shift to more language-like sound material. A similar approach is suggested by Collier and 't Hart (1977). In order to introduce the students to common patterns of Dutch intonation they present several pitch patterns in isolation and in combination. They start with single artificial vowels and end up with words and sentences.

In the second set of exercises, imitation, the learner will be instructed to imitate on the visual display the pitch contour of an utterance produced by the teacher or the computer tape recorder. Visual feedback in a stylized form of the pitch contour of the student's utterance is presented almost without a delay. The moments of occurrence of the vowel onsets can be displayed in order to train the learner to time the pitch movement correctly. Furthermore, the student can listen to his or her imitation as many times as she or he wishes.

In the third set of exercises, production on demand, the learner will be taught to produce various intonation contours "spontaneously." These productions can be elicited by means of visual cues (e.g., sentences, parts of stories or dialogues). Furthermore, she or he will be prompted to make a judgment regarding the correctness of his or her speech production which is initially based on the auditory and the visual feedback of the pitch contour. Later on, the learner is requested to judge the correctness of the speech production merely by the use of the internal feedback modalities.

The exercises described above will be integrated into an educational software package in which instructions on speech materials and pitch contours are presented in increasing order of difficulty. Either the language teacher or the computer program will determine what instruction material (e.g. exercises, speech materials) will be used for the instruction of each individual student. On the one hand the possibility of selection of instructional materials by the computer requires it to keep records of performance of each individual student. However, for this purpose it is necessary to keep records of the performance of each individual student. On the other hand, the possibility for a teacher to determine the selection of the instructional materials makes the development of a friendly user-interface necessary.