Rating learners of English as a foreign language: Rating Scales vs. Rapid Profile

Contents

1 Introduction

2 Language Testing: Requirements, Types and Properties
2.1. Test requirements
2.2. Types of tests
2.3. Test properties

3 Rating Scales

4 Rapid Profile
4.1. What is Rapid Profile?
4.2. The theory behind it

5 Rating Scales vs. Rapid Profile

6 Conclusion

7 Bibliography

1 Introduction

Rating the learner is an important part of language teaching. Tests can be used to place students in courses according to their knowledge, to diagnose their strengths and weaknesses, to identify their proficiency etc. There are many theories about what tests should analyse and how they should be taken. Depending on the relevance of the results, the ratings need to be more or less accurate. Furthermore, they should have a beneficial backwash on teaching. It is thus at least as important to examine rating itself as it is to look at teaching methods.

As there are many different ways of rating language students, this work will concentrate on two specific approaches: rating scales and Rapid Profile. Whereas the first have been in use for a while, Rapid Profile is a newly developed computer programme which has not been discussed very much so far. It is thus particularly interesting to compare the two rating methods in order to investigate their advantages and disadvantages.

First, I will give an overview of the types of tests and present the basic properties any test should have. I will then introduce the concept of rating scales and Rapid Profile in order to be able to compare them.

2 Language Testing: Requirements, Types and Properties

2.1. Test requirements

In order for a test to be accurate, there are two principles that need to be taken into consideration: reliability and validity. Reliability refers to the consistency of the test. Even if the results will never be exactly the same because of human inconsistency they should be as similar as possible. Validity describes in how far the test fulfils its purpose. This principle is divided into several subcategories: concurrent validity, or the extent to which the results of a test correlate with those of another one; predictive validity, the extent to which the outcome of a test predicts the performance in a future situation; face validity, if a test mirrors what it is testing; content validity, if a test examines a representative sample of what was taught; construct validity, the extent to which the test bases on a theory of language acquisition, and finally consequential validity, or the backwash expected from a test.

2.2. Types of tests

Language tests can be divided into several types according to their purpose. Proficiency tests judge the ability of communication in a foreign language. They can be designed according to the purpose for which the knowledge is needed, or they can be general. These tests are not based on any specific courses.

Achievement tests relate to courses taken before the testing. They can be relatively informal progress achievement tests, or final achievement tests taken at the end of a course. The latter can be based either on the syllabus content, covering the material the students have dealt with in class, or on the objectives of the course.

Diagnostic tests are designed to analyse students’ strengths and weaknesses. However, whereas they are sufficient for testing basic skills they are not reliable on complex structures.

Placement tests are used to allocate students to different levels according to their competence. To ensure their accuracy they need to be designed especially for particular situations.

2.3. Test properties

There are different ways of testing depending on the requirements. Testing can be direct or indirect, discrete point or integrative, norm-referenced or criterion-referenced, objective or subjective, or computer adaptive.

Direct tests examine the performance of a specific skill (i.e. reading, writing etc.) measuring only one ability. Indirect tests measure a number of abilities that underlie the skills asked for.

Discrete point tests analyse individual grammatical items or structures, normally indirectly. Integrative tests are usually direct; this kind of test presupposes the knowledge of an infinite number of language elements.

Criterion-referenced testing provides information about a student’s ability to fulfil a task; the result is a pass or a fail. Norm-referenced tests do not measure competence, but it reflects whether one student is better or worse than others by comparing their performances.

Tests are called objective if their results are unambiguous, e.g. in multiple choice tests. If the results vary depending on the testers’ judgement the testing is said to be subjective.

Unlike the majority of paper and pencil tests, computer adaptive tests adjust to the learners’ abilities. Depending on whether the test subjects answer the first question (of average difficulty) correctly or not, the next questions will be easier or more difficult.

3 Rating Scales

Rating scales consist of an ordered series of level descriptors which describe the requirements for each stage of proficiency. They determine the criteria which are going to be focussed on. Their number usually ranges from 3 to 9 between zero knowledge and mastery^[1]. Ideally, these levels are independent definitions which are not interrelated, however, very often they are comparative. The definition of mastery is normally the performance of an idealised native speaker^[2]. As rating scales can be used for many purposes, the definitions given by the level descriptors differ accordingly. They not only describe levels of proficiency, but they are also guides for test constructors and raters^[3]. Rating scales can identify the testees’ competence in both active and passive skills, i.e. reading, writing, listening and speaking. This proficiency can be measured as an achievement as a continuum or at certain levels of competence seen as hurdles or cut-points. There are several possibilities to carry out the rating: through a formal exam or through observations made during the instruction of the candidates. The results can be either norm-referenced or criterion-referenced. Depending on the information needed, ratings can be either holistic or analytic. The first reflect on a performance in general, whereas the latter give evidence about the competence in specific aspects of language. Usually, the scores achieved in the analytic rating are merged into a single overall score which can be compensatory or non-compensatory. In the first case, the candidates’ strengths make up for weaknesses giving them the chance to pass even if not all aspects were satisfactory. In the second case, what is relevant is the lowest score, strengths are not counted as bonuses. Rating scales measuring proficiency focus either on decontextualised abilities or on real-life performances (Hudson, 2005).

[...]

^[1] Bachman, L. & A. Palmer. Language Testing in Practice. Oxford: OUP, 1996.

^[2] This is not necessarily accurate as performances of native speakers vary considerably according to their education etc.

^[3] Hudson, Thom. “Trends in Assessment Scales and Criterion-Referenced Language Assessment”. Annual review of applied linguistics 25 (2005): 205-227.