Principled Washback – IELTS and Academic Reading

In a previous post, I looked at research that analyzed the IELTS writing tasks in juxtaposition with university writing tasks. In this post, I will summarize similar research on academic reading tasks, apply this research to briefly analyzing TOEFL reading tasks, and then discuss implications for principled washback.

Introduction

Moore, Morton, and Price (2012) created an IELTS reading task corpus, created an academic reading task corpus, and interviewed a number of faculty members in order to get a holistic and in-depth view of the demands of academic reading. Their work is exhaustive and long (89 pages), but I hope to summarize it here as concisely as possible.

The purpose of their research was to analyze the validity of the IELTS reading section and make suggestions for improvements. One impetus for this was IELTS washback on ESL and EAP courses. According to the authors, “Washback is considered harmful then when there is a serious disjunct between a test’s construct of reading and the broader demands of real world or target language tasks” (p. 6). Therefore, this work is very important in deciding how much of a disjunct, if any, exists.

Framework of Analysis

A framework was developed to analyze the writing tasks. This framework was based on the level of engagement reading requires (i.e. sentence, paragraph, or whole text) and type of engagement (i.e. “what was needed to be done with texts” [p. 10] – which can be seen as the type of critical or non-critical skills that need to be applied). Their framework looks like this:

levelandtypeofengagement

  • Level of Engagement moves along the x-axis as how much needs to be understood moves from local to global: word → sentence → paragraph → whole text → multiple texts
  • Type of Engagement moves along the y-axis as what needs to be done with the text moves from literal to interpretative: basic comprehension → critical or personal reflection

IELTS Reading Tasks

The article contains examples and graphs for each task type and analysis. For the purpose of this summary, a simple table has been made below:

Task Type Level of Engagement Type of Engagement Frequency Notes
True/ False/ Not Given TF: local
Not Given: more global
 literal 26%
Section-summary match more global Academic: more literal

Journalistic: more interpretive

16% It was noted that there were both academic- and journalistic-style prompts
Gapped summary local literal 16% longer sections increase the level of engagement
Information- category match local literal 12% level and type increase slightly if matching scholars rather than information
Multiple choice varies varies 10% these questions represented various dimensions
Short answer literal local 7% most literal and local
Other not analyzed not analyzed 17% the different tasks has a “minimal presence”

In summary, the most task types required mostly local-literal engagements with the text, though there was some variation. The authors noted that multiple choice items were “the only items in our corpus that clearly traversed the ‘local-literal’ domain” (p. 40).

Academic Reading Tasks

Academic tasks, by nature of the different disciplines they are situated in, vary significantly. Nevertheless, reading is important for all disciplines and it is therefore necessary to understand what this reading requires of students. The authors investigated the reading tasks from 12 different disciplines representing hard (physics, biology, economics, etc.) and soft (history, management, lingusitcs) domains. For each discipline, a faculty member was also interviewed to get a deeper understand of the task. The general findings are reported below.

Task Type Level of Engagement Type of Engagement Notes
weekly reading and question task local literal checking understanding of key concepts
exam question local literal testing concepts and knowledge
summary tasks – single text global literal rare

focus on ideas as part of larger concept

summary tasks – multiple texts global literal more common

focus on ideas as part of larger concept

summary of research findings global literal
application tasks local interpretative these tasks ask students to apply concepts to new situations
evaluation tasks  local interpretative  assessment of the “value, worth, benefit, etc”
 assignment tasks  global interpretative essays, reports, text analysis.

Commentary

Besides the analysis of the tasks themselves, the authors also interviewed 12 faculty members from 12 different disciplines. The interview data was used to help explain different academic reading tasks, including the rationale and purposes behind them. While they served to provide insight into tasks, the interview data also revealed insight into other aspects of academic reading. Below are a few that I found interesting:

On Reading

  • Reading is considered important by all disciplines. And, all instructors in these disciplines feel frustrated by the lack of effort students (not just international students) put into their readings:
    • “It is a constant struggle to get students to do the reading these days.” (Linguistics, p. 46)
    • “I think quite a lot of students [aren’t] actually doing the reading.” (Computer science, p. 46)
    • “So reading in fact has become a major problem. Students are just doing less reading than they’ve ever done before…” (Media studies, p. 46)
    • “They just don’t read this stuff in any serious and sustained way.” (Media studies, p.47)
    • “Students only have to read the textbook and the powerpoint slides to be
      successful in this subject nowadays.” (Engineering, , p. 47)
  • A lot of the changes in reading habits is due to the influence of the internet:
    • “There is a lot of material now that students access that they just typically
      browse. It’s a kind of trawling for information.” (Media studies, , p. 47)
    • “Research is showing that the evaluation and management of material that’s coming out over the internet is the biggest issue. And some students do not have particularly well-developed evaluation skills.” (Communications, p. 47)
    • “We actually have discussions in tutorials. How can we tell whether this is a reliable site or not? So its evaluation of who is producing this, in what context, and for what purpose.” (History, p.47)
    • “What I try to teach [students] is to get them to be selective in their reading of the media.” (Media studies, p.47)
  • Some see the changing nature of reading as an impetus to change course structure:
    • “I’ve had to tone down the volume of reading and that’s in response to the changing student mix and changing student behaviour. I have probably shifted it more to use of business press material, less academic material.” (Management, p.47)
    • “So I think we in the university have to learn more about student’s reading habits and practices and to rethink our assumptions. And we are probably going to have to make big adjustments about what it is that students do these days when they study.” (Linguistics, p.47)

On IELTS

  • Some hard disciplines were found to have quite similar tasks as the IELTS. Instructors in these fields mostly felt the IELTS was good preparation. Even some in the soft disciplines felt the same way.
    • “I think the skills required here [on the IELTS test] would be very closely aligned to what I would expect a student in first-year biology to come to terms with.” (Biology, p. 61)
    • “Our exam questions are not dissimilar to some of the questions [on IELTS].” (Computer science, p. 61)
    • If anything, we’re expecting less of students in terms of reading. The test is definitely relevant and having it at a higher level than what we’re asking for in the course is a good thing.” (Computer science, p. 62)
    • “I think [the IELTS Academic Reading test] would be good preparation actually. I found the science-based articles and items quite complicated actually. If I had to answer questions about the science, I’d have to go back and read it twice.” (Communications, p. 62)
    • “These sorts of skills [tested in IELTS] would definitely be useful in a generic sense … and I can see that it would be good preparation for what we require on our course.” (Engineering, p. 62)
    • “I‘d see this as all useful. The test is very focused on reading comprehension …
      that is a basic pre-requisite for our courses. It doesn’t cover the quite discipline-specific methods of reading we’re concerned with …” (History, p. 62)
  • However, the IELTS was not without its flaws in these disciplines:
    • “The IELTS tasks seem more arbitrary in what they pick out from the text … and
      seem to be mainly about pattern recognition.” (Physics, p. 63)
    • “[To use sources appropriately] students need to see concrete examples to know what is acceptable and what’s not … I can’t see much evidence in the test of this aspect of academic study, and this would certainly be helpful.” (Management, p. 63)
  • Still others were very critical of the test:
    • “In the tasks on the test, it seems to me that students are really just dealing with information….And then it’s mainly about regurgitating the information. …I see it as pretty low level.” (Media studies, p. 63)
    • There is a broader context for interpreting the reading which university
      students have because they have a purpose for assignments…[On the IELTS] the tasks are completely out of context. What is missing is the purpose for knowing this information.” (Linguistics, p. 64)

Discussion

Overall, the research found that the IELTS focuses mostly on local-literal tasks, with very minor variability. In general, the authors consider the IELTS a local-literal test. Academic reading tasks, by contrast, vary widely depending on purpose and discipline. The disciplines that fall on the “hard” side of the spectrum tend to require more local-literal reading. The “soft” disciplines require more global-interpretative.

The skills the IELTS tests are very basic, but important. They are the groundwork for both the hard and soft disciplines, with the soft disciplines forcing students to move beyond the local-literal. However, there is a difference of purpose. The authors point out that IELTS reading is reading for information whereas academic reading is characterized by reading for knowledge (concepts, ideas, etc.). This lends itself to what the authors term the “reading-writing nexus” in which most academic reading tasks involved academic writing, that is, a place to apply and demonstrate the knowledge read.

Reading only for information purposes points to an “apparent lack of an underlying intellectual purpose for the particular questions posed in tasks” (p. 67). While this may seem fine for a test which needs to be broad and is about testing reading not knowledge, there is still the issue of validity. The authors point out that the results do not necessarily reflect a deficit within the IELTS, but show room for improvement, such as having tasks that represent greater variety in level and type of engagement.

Principled Washback

Washback is not inevitable. Nor is it necessarily warranted. However, denying a focus on testing does a disservice to students who need to take the test. (Despite research that shows non-test prep EAP programs are just as effective as test-prep programs, students may not feel at ease.) Still, offering test-prep within an EAP setting can also be doing students a disservice, giving them conflicting views of what academic skills really are or sacrifing time that could be spent on said skills.

As the authors pointed out: “Washback is considered harmful then when there is a serious disjunct between a test’s construct of reading and the broader demands of real world or target language tasks” (p. 6).

The point of principled washback is to address students wants for test preparation while attending to their needs for academic skills. It is important, therefore, to investigate overlap between testing and academia in order to find a place from which to begin. The IELTS study summarized above points to such an overlap (and provides a framework for future investigations). Clearly, basic comprehension skills are essential. Without these, there is no IELTS and there is no academic reading. However, these skills only go so far. Students also need to learn to read for knowledge, not information. They need to be able to summarize, synthesize, evaluate, apply, and utilize.

So, how does one apply principled washback to the case of academic reading? It’s an open question and I have only the beginnings of an idea, not a formulation of an answer. My primary suggestion would be to exploit texts across the engagement spectrum. Begin at the literal end with both local (sentence, paragraph) and global (whole text) focuses on comprehension, which can be modeled in the IELTS style (or whatever test style they are going to take) among others. Then, move beyond the literal and begin more interpretative (i.e. critical thinking) approaches to understanding the texts. In this way, there can also be a move from information mining to knowledge construction – students learning how to transform information into learning.

In these ways, one can conceivably address important test skills as a way of bridging into more academic skills. Students get the test prep they want and the academic skills they need. This is principled washback.