washback (n.) the impact of a test on teaching
“Washback is considered harmful…when there is a serious disjunct between a test’s construct…and the broader demands of real world or target language tasks” (Moore, Morton, & Price, p. 6)
principled (adj.) 1. based on the principles of pedagogy 2. based on research
“Principled pragmatism is based on the pragmatics of pedagogy…Principled pragmatism thus focuses on how classroom learning can be shaped and managed by teachers as a result of informed teaching and critical appraisal” (Kumaravadivelu, 1994)
principled washback (n.) focuses on how test preparation can be shaped and managed by teachers as a result of informed teaching (through research and pedagogy) and critical appraisal (of both tests and academic skills)
Many educators in EAP have the dual role of preparing students for success in the university classroom as well as preparing them for high-stakes gatekeeping tests like IELTS and TOEFL. Whether we like these tests or not, that students’ entrance into the academic world depends on these tests makes our job makes our job both more important and more difficult. If we focus too much on the test, we are sacrificing important long-term skills students will need to survive in academia. If we focus solely on academic skills, students might be OK, but they may not feel prepared for the test or satisfied with their classes, which are perceived as not meeting their (short-term) needs. Principled washback is meant to find a happy middle ground that addresses test prep skills en route to addressing academic skills.
Principled washback considers the academic demands of the classroom, the academic demands of the test, and then looks for overlaps in order to focus and frame instruction. These overlaps serve as starting points of instruction that reference test skills and perhaps emulate test questions but actually move students along to important work academic areas not addressed by these tests.
The IELTS and TOEFL are broken up into four parts: speaking, reading, listening, and writing. The IELTS treats these as separate (which speaks to its validity a bit) while the TOEFL iBT separates them AND integrates them. For my presentation and this blog post, I will separate them and then offer some ideas for integration.
Ferris and Tagg found that the speaking demands in a class often depend on the class size, discipline, and the instructor’s style, but, in general, important university tasks that require speaking, as rated by faculty and students, include:
- Oral presentations
- Ferris and Tagg (1994a) found that these were not as common as one would think and usually done in groups or in pairs. That was in 1994. If my own college experience counts as anything besides anecdotal, that is still true today. Presentations are common, but not as common as one would think. However, many students rated having the most difficulty with this type of oral task. This may be because students don’t get a lot of practice doing this sort of thing.
- Class participation
- This includes working in groups, whole-class discussion, and answering questions from the instructor
- Other skills that were deemed important because they happen amongst the different types of participation:
- question asking/answering
- summarizing information
- giving and supporting opinions
- explaining or informing
- Other tasks that were reported as important were meeting professors during office hours.
- Obviously, making everyday conversations is also important
|3 Tasks – Live Interviewer – 11-14 Minutes||Independent Tasks|
Where’s the overlap?
Each test task addresses some of the academic skills that are needed. For example, IELTS Task 1 and TOEFL Independent Task 1 require general English ability (e.g. the ability to hold a conversation) which is also required of university students for everyday life. Pretty simple, right? IELTS Task 2 and the integrated tasks of the TOEFL assess important skills such as giving and opinion, explaining, and summarizing, which are useful for class participation and even presentation skills (though research has shown that these tests, although they require monologues, are not valid as assessors of presentation skills CITATION.) As you can see, you can easily find overlaps. Of course, the test does not cover all academic speaking skills, and not necessarily with any depth, but it is a good starting place, and this is what principled washback is about.
Principled Washback and Speaking
From this type of analysis, and from the research on these demands, we can make some of the following recommendations. If they seem common sense, it is because they are. They are based on good pedagogy.
- Academic content vs language
- When possible, you should focus more on working with content, not language. Being able to demonstrate learning or knowledge is far more important than having 100% correct grammar. Of course, when grammar does get in the way of relaying this knowledge, then it is obviously something to be addressed.
- Demonstration of knowledge
- Classroom tasks should get students to demonstrate knowledge, not only linguistic proficiency
- Summarization / explaining / evaluating / justifying
- Classroom tasks should involve these skills
- Orally summarize a reading or listening
- Explain a concept to a classmate
- Evaluate a reading or listening – agree or disagree with it and give justification (support)
- Classroom tasks should involve these skills
- Confidence building (Ferris and Tagg, 1996a)
- Both teachers and students realized they needed to be more confident. This also applied to the test, which can be nerve-racking. Work to build social, emotional, and linguistic confidence in your students. One way to do this is to set tasks that students can and will succeed at, even if they are challenging. Positive feedback is also helpful.
- Communicative / discourse strategies (Ferris and Tagg, 1996a)
- Teaching important phrases and strategies for sharing ideas, holding the floor, asking questions, etc. is important.
- Whole-class discussion (Kim, 2005)
- Kim recommends trying to emulate whole-class discussion (on serious academic content) whenever possible to simulate the environment of the university classroom. Whether or not whole-class is good pedagogy is a discussion outside of the presentation, especially because not all professors are good teachers!
We will come back to more ideas at the end of this post.
It’s difficult to say which skill is most important for the university classroom, but reading must rank highly because it is one of the main sources of input – the textbook and course readings.
Important reading skills, as rated by faculty and students include (Rosenfield, Leung, Oltman, 2001):
- Determining the theme/main idea of a passage
- Understanding and remembering main ideas, especially without the text present
- Understanding assignments and directions
- Locating and understanding information in the text
- Distinguishing fact from opinion
- Comparing and contrasting texts
These are important reading skills, but what about the actual tasks? Moore, Morton and Price (2012) surveyed faculty in a number of disciplines at a major university and found these tasks to be common:
- Weekly reading questions
- This would be reading the assigned readings and answering basic questions about them
- Exam questions
- Short answer and possible essay questions on exams
- Multi-text summaries
- Typically, students need to summarize several texts along a common theme, which may also include comparing and contrasting them
- Application and evaluation
- Application tasks means tasks in which students must apply concepts they read about to new situations
- Evaluation tasks ask students to evaluate the information they just read
- Essay/report writing
To analyze these tasks so that they could be compared to test tasks, the authors devised a scale that looks at the level of engagement (i.e. is it one small piece of the text, the whole texts, or beyond the text?) and the type of engagement (e.g. are they engaging in a literal understanding of the text or are they doing more interpretative thinking, moving beyond the literal meanings?).
The scale is represented here below. I have written a bit more in-depth on the topic here, in which I compare university and IELTS tasks, and include comments about reading from university faculty. Below, I will briefly reproduce related information.
Academic Reading Tasks
|Task Type||Level of Engagement||Type of Engagement||Notes|
|weekly reading and question task||local||literal||checking understanding of key concepts|
|exam question||local||literal||testing concepts and knowledge|
|summary tasks – single text||global||literal||rarefocus on ideas as part of larger concept|
|summary tasks – multiple texts||global||literal||more commonfocus on ideas as part of larger concept|
|summary of research findings||global||literal|
|application tasks||local||interpretative||these tasks ask students to apply concepts to new situations|
|evaluation tasks||local||interpretative||assessment of the “value, worth, benefit, etc”|
|assignment tasks||global||interpretative||essays, reports, text analysis.|
IELTS Reading Questions
|Task Type||Level of Engagement||Type of Engagement||Frequency||Notes|
|True/ False/ Not Given||TF: local
Not Given: more global
|Section-summary match||more global||Academic: more literalJournalistic: more interpretive||16%||It was noted that there were both academic- and journalistic-style prompts|
|Gapped summary||local||literal||16%||longer sections increase the level of engagement|
|Information- category match||local||literal||12%||level and type increase slightly if matching scholars rather than information|
|Multiple choice||varies||varies||10%||these questions represented various dimensions|
|Short answer||literal||local||7%||most literal and local|
|Other||not analyzed||not analyzed||17%||the different tasks has a “minimal presence”|
TOEFL Reading Questions and Tasks
|Independent Reading Tasks||Integrated Reading Tasks|
(see SPEAKING above)
Where’s the overlap?
The overlap is really based on discipline. As Moore, Morton, and Price point out, “hard” sciences demand academic reading skills more akin to the IELTS skills – the local/literal variety, while “soft” sciences often require more interpretative tasks. Universal to any discipline is at least basic comprehension (of the local-global/literal variety), but the main difference is IELTS/TOEFL requires the reading for information whereas university reading tasks require reading for knowledge construction. IELTS/TOEFL reading is one-off but academic reading can be seen as accumulative and will eventually require the demonstration of learning. Despite these differences, there are a number of areas of overlap that can make addressed test-prep more principled.
Principled Washback and Reading
- Use IELTS/TOEFL questions for checking reading comprehension
- Any IELTS or TOEFL website will give you the proper format for these questions. Exposure to these questions and how to find the answer is important for not only test-prep but for understanding a text in general.
- After comprehension, move to critical thinking questions
- These types of questions are more interpretative and require evaluation, questioning, analysis, and inference
- Address all levels and types of engagement
- In working with a reading, try to give students exposure to a bit of eveyrthing:
- Local/literal – basic comprehension questions
- Local/global – summary and comparison
- Local/interpretative – evaluating or inferring specific ideas
- Global/interpretative – critical writing from reading
- In working with a reading, try to give students exposure to a bit of eveyrthing:
- Use readings for writing, as this is the most common type of task
- This could include summaries, summary and responses, short answer essays, comparisons, and full research papers
- Use readings for discussion
- Get students used to orally summarizing, explaining ideas and concepts to each other, and evaluating the information
More ideas will follow at the end of this post.
Aside from reading, listening is one of the most crucial skills in the university classroom. Sawaki and Nissan (2009), in their extensive literature review and research among undergraduate and graduate students, found the following skills to be important in academic listening:
- Understanding language and content simultaneously
- Processing visual information such as presentations, charts, graphs, etc.
- Handling a variety of lecture styles, including long, uninterrupted lectures
- Discourse markers and other cues which signal main ideas, details, examples, rhetorical questions, etc.
- Understanding explanations of assignments
- Applying concepts to complete tasks
- Tasks might include small group or whole class discussion, as well as answering and asking questions
- Multiple choice tests – found to be the most common test format
- Short answer – also very common
- Ability to synthesize lecture content with reading
Both the IELTS and TOEFL use sources that are based on either campus conversations (course announcements, campus information, campus organizations, etc.) or short academic lectures. In addition, listening questions on these exams are very similar to reading questions and can be analyzed similarly in terms of level and type of engagement.
|IELTS Listening Questions||TOEFL Independent Listening Questions|
|TOEFL Listening (see above sections)|
Where’s the overlap?
Listening is a demanding skills that requires comprehension and application. The IELTS and TOEFL independent listening test tests comprehension skills while the TOEFL integrated section moves more toward application. Both the tests and academic study require good note-taking skills, which include an idea about organizing notes and lecture organization. In addition, the listening sources for these tests represent similar sources for university students: campus life and academic lectures.
Principled Washback and Listening
Note-taking is the fundamental skill that needs to be addressed first. Note-taking is difficult because it requires a division of focus. Listening, comprehending, and writing at the same time needs to be practiced (even for native speakers) and the classroom should give ample time for this. Note-taking practice could include who to abbreviate, how to outline appropriately and perhaps even how to use Cornell notes. I have a good post about research on note-taking here.
Other ideas for principled washback include:
- Working with a range of listening contexts: academic lectures (e.g. from MIT Open Courses, Five Minute Lectures, or TED), campus-based conversations, and even general English contexts
- Exposure to various accents. The tests use various native-English speaking accents (General American, Received Pronunciation, Australian, etc.) but students will also need exposure to Arab, Spanish, African, Indian, East Asian, etc.
- Students should have tasks that ask them to apply what they have learned from their listening tasks
- Comprehension check questions can come in the form of IELTS or TOEFL questions and then move on to more critical thinking questions, which can include summarization and evaluation – in both reading and writing
More ideas will follow at the end of this post.
Writing is probably the most robust, demanding and varied skill that becomes developed during academic study, especially as one moves deeper into their discipline and its writing specifics. Broadly speaking, undergraduate university students need to write, a lot, and in different ways.
Moore and Morton (2005) investigated 155 writing tasks from 79 academics and analyzed them following a framework that looked at genre (e.g essay, report, bibliography), information source (e.g. primary, secondary, prior knowledge), and rhetorical function (e.g. classification, comparison, evaluation), and object of enquiry (e.g. concrete vs abstract topics). They found the following for the first three aspects of the framework (I will skip object of enquiry for now):
- 60% of assignments fall within the essay genre, followed by case studied and short exercises. However, there is great variation, especially dependent on discipline.
- Information source
- Primary or secondary sources made up 97% of all source writing is based on
- Rhetorical function
- Most assignments consist of several rhetorical functions
- The most common rhetorical functions are evaluation (67%), description, summarization, comparison, explanation, recommendation (23%), hortation (“Should…”) (15%)
The authors applied this same framework to a corpus of IELTS writing tasks. Here are their findings:
|IELTS Wrting||Task 1||Task 2|
|Information source||Primary (chart or graph)||Prior knowledge|
|Examples||The line graph below shows changes in the amount and type of fast food consumed by Australian teenagers from 1975 to 2000.
Summarize the information by selecting and reporting the main features and make comparisons where relevant.
|Many people believe that media coverage of celebrities is having a negative effect on children.
To what extent do you agree?
We can apply the same framework to the TOEFL
|TOEFL Writing||Independent Task||Integrated Tasks|
|Information Source||Prior knowledge||Secondary sources (lecture, reading)|
|Examples||Do you agree or disagree with the following statement?
Always telling the truth is the most important consideration in any relationship
Use specific reasons and examples to support your answer.
Principled Washback and Writing
It seems as though only Task 1 of the IELTS and the integrated TOEFL writing tasks exhibit some overlap with real university writing demands. As Moore and Morton say, most IELTS writing tasks are more similar to “letters to the editor” you would find in a newspaper than academic writing. While having the ability to writing based on one’s sole opinion is not doubt useful for more informal writing, writing really needs to be focused on incorporating outside sources through multiple rhetorical functions. This includes summarization, description, evaluation, and synthesis of both written and verbal textual sources.
Nevertheless, there are also some skills that can be applied to both academic writing and the very non-academic writing that these tests demand. Students, no matter what they are writing, should have a strong ability to organize writing logically; understand and apply the different rhetorical functions, even in isolation (this includes identifying them in assignment tasks); and the ability to use formulaic language in order to enhance organization, cohesion, unity, etc. By formulaic language, I mean discourse phrases that can show logical order and transition(first, in addition, furthermore), comparing and contrasting (likewise, on the other hand), concession (although), and so forth. These skills are applicable in any context. And, as I have argued elsewhere, by practicing and internalizing these skills and phrases, it can improve overall organization (by using such phrases, they can help set the organizational flow of a writing piece) and increase writing speed (something that could be helpful on in-class essay exams.
BROADLY APPLYING PRINCIPLED WASHBACK
Recall that the idea behind principled washback is to address relevant testing skills en route to addressing academic skills. And, principled washback focuses mostly on the skills that overlap. To apply principled washback simply means to be more aware of the different contexts in which certain activities are useful. If a task is useful for both a portion of the test and some academic skill, it is a good idea to state that explicitly. This can heighten students awareness of why they are doing a certain task – perhaps even seeing more meaning in it since it not only prepares them for their long-term needs (which they may not even be considering) but also their short-term needs (which are likely causing them much stress.)
Principled washback also means to sometimes use testing skills as a base to build from before addressing more integrated, higher-order, or critical skills. This can be achieved by using test question format when phrasing questions, being mindful of the question’s purpose and validity.
Throughout any application of principled washback, its important to try to emulate academic tasks and integrate skills whenever possible. Two great ways you can do this is through Audio Diaries and Academic Reading/Discussion Circles. The posts I have linked to contain in-depth information on how I have used these projects. I will also give a short breakdown here. Afterwords, I will briefly describe several other techniques that might be useful:
The idea behind audio diaries is to give delayed corrective feedback and opportunities for uptake of correction. This is achieved through having students record a monologue for 1-3 minutes, providing feedback through some means (Soundcloud, Google Classroom, Blackboard, email, etc.) and then having them re-record their monologue, attending to their errors. This has worked very well for helping students improve their spoken grammatical accuracy and pronunciation. However, it can also be used to address content.
In my usage of Audio Diaries, topics have been relatively loose and unorganized. However, with some mindfulness and an eye toward principled washback, the Audio Diary tasks can emulate the speaking tasks that are similar to both TOEFL and academic speaking demands. If they are assigned in conjunction with a reading or lecture, you can ask students to summarize, evaluate, discuss, or explain the ideas they have read/heard. Feedback can focus not so much on accuracy (which is important) but whether or not they have applied the required skills effectively. This could also be an additional opportunity for them to practice discourse strategies and academic phrases.
Academic Reading / Discussion Circles
This is an intensive and collaborative project in which students discuss a text (written or spoken) following different roles. These roles require students to explicitly focus on a particular set of skills, all of which are useful for both the test and even more so for the academic classroom. After investigating their text following their roles, students come together in small groups to discuss the text. This builds up their academic speaking skills, which is clearly useful for all contexts.
The roles and skills are as follows:
- Leader – writings comprehension/critical thinking questions, practices the ability to ask questions and provide oral and written summaries, gets other students to practicing answering, evaluating, agreeing, disagreeing, justifying, etc.
- Highlighter – focuses on language usage by identifying general and topical vocabulary, enhances vocabulary awareness, practices explaining and defining terms
- Contextualizer – researches contextual references about people, events, or ideas, practices summarizing, describing, and explaining, as well as important research skills
- Visualizer – finds visual ways in which to represent the text, including the description and interpretation of visuals such as charts and graphs
- Connector – finds meaningful connections between the text and other topics, other texts, and current events, practices comparison, description, explanation
For this discussion project, students are intensely involved in discussion for quite some time (~60 minutes) and are likely practicing many more skills than I have written here. The point is that it is a great project which helps students have a more academic focus on outside sources while engaging in deliberate and academic speaking practice. At the same time it builds up a number of different skills which students will need both for the test and for academic success.
Principled Washback Task Sequencing
It is safe to say that tests like the IELTS and TOEFL iBT test very basic comprehension skills (and very, very basic critical thinking skills in the case of the TOEFL). Therefore, it may be wise to begin tasks at this level and then use these basic comprehension tasks to move towards tasks that require higher order thinking. Here is very basic task sequence that one could use:
- Begin with a reading or listening text
- Draft questions following TOEFL or IELTS format. Be aware of what types and levels of engagement these focus on, and what skills they address. Try to write enough questions to test a wide range of skills.
- Work on written or oral summaries, summary and responses, and explanations (e.g. pair teaching)
- Move to asking or having the students ask questions that require more critical thinking. Build this into oral discussion work:
- evaluating and justifying
- Finally, move to whole class discussion or writing based on the text
- An additional step could be other summarization activities that take into account the small group or whole class discussions
Another activity that can helps students in formulating an intelligent academic response applicable to test speaking tasks or in-class speaking tasks is following an academic response formula. A colleague in my department introduced me to “idea + opinion + support” formula in which an idea is introduced, an academic opinion is given, and then support is used to justify the opinion. A participant during this presentation pointed out a similar structure called CQC – claim, quotation, commentary (more info here and here). The benefits of teaching such formulaic responses are several:
- It gives students a framework from which they can build verbal academic responses during spoken or written tasks
- vocabulary, grammar, and discourse strategies can be taught for each aspect of the framework
- internalizing the framework and automatizing the relavent phrases can likely lead to quicker more fluent responses that require less cognitive demand
- these types of responses are applicable to IELTS Task 3 and most of the TOEFL iBT integrated tasks
- these types of responses can be used for when a professor calls on a student or if the student wants to offer an opinion during whole-class or small-group discussion
- these types of responses can be built upon in various ways
Principled washback is not a new or radical idea. It can’t be called a technique or methodology either. It is simply having a more informed understanding of the differing contexts for which our students must be prepared and then reacting to those contexts in a way that facilitates both of them together. Returning to the definition of washback at the beginning of this post: “Washback is considered harmful…when there is a serious disjunct between a test’s construct…and the broader demands of real world or target language tasks” (Moore, Morton, & Price, p. 6). However, if washback bridges a tests’ construct and links it to target language tasks, then this washback can hopefully be considered beneficial, and even principled.
REFERENCES and FURTHER READING
Ferris, D. (1998). Students’ views of academic aural/oral skills: A comparative needs analysis. TESOL Quarterly, 32(2), 289-318.
Ferris, D., & Tagg, T. (1996a). Academic oral communication needs of EAP learners: What subject‐matter instructors actually require. TESOL Quarterly, 30(1), 31-58.
Ferris, D., & Tagg, T. (1996b). Academic listening/speaking tasks for ESL students: Problems, suggestions, and implications. TESOL Quarterly, 297-320.
Gomez, P. G., Noah, A., Schedl, M., Wright, C., & Yolkut, A. (2007). Proficiency descriptors based on a scale-anchoring study of the new TOEFL iBT reading test. Language Testing, 24(3), 417-444.
Kim, S. (2006). Academic oral communication needs of East Asian international graduate students in non-science and non-engineering fields. English for Specific Purposes, 25(4), 479-489.
Kumaravadivelu, B. (1994). The postmethod condition: (E)merging strategies for second/foreign language teaching. TESOL quarterly, 28(1), 27-48.
Moore, T., Morton, J., & Price, S. (2012). 4 Construct validity in the IELTS Academic Reading test: A comparison of reading requirements in IELTS test items and in university study. IELTS collected papers, 2, 120-211.
Rosenfeld, M., Leung, S., & Oltman, P. K. (2001). The reading, writing, speaking, and listening tasks important for academic success at the undergraduate and graduate levels. Educational Testing Service.
Sawaki, Y., & Nissan, S. (2009). Criterion‐related validity of the TOEFL iBT listening section. ETS Research Report Series.