Deconstructing the Duolingo English Test (DET)

Duolingo is a great language learning tool. It can introduce you to the basics of a number of different languages through a fun, game-like app in which grammar and vocabulary are built up and reinforced through translation practice. I can thank Duolingo for my basic ability in Spanish and my deeper understanding of Polish grammar. The Duolingo English Test (DET), on the other hand, is absolutely terrible. Last year, it made some buzz on the internet as a kind of TOEFL/IELTS killer, a serious competitor to the big tests – one which was affordable (at $30) and accurate. I had a chance to take the test last week to see if it could serve our institute and students. I went into it very excited and came away with a very bad taste in my mouth. Continue reading

Principled Washback: Integrating Test Prep to Foster Academic Skills

(This post is a companion to the presentation I gave at the 2016 Toronto TESOL Conference. To download my presentation click here. To download my handout, click here.)

DEFINITIONS

washback (n.) the impact of a test on teaching

“Washback is considered harmful…when there is a serious disjunct between a test’s construct…and the broader demands of real world or target language tasks” (Moore, Morton, & Price, p. 6)

principled (adj.) 1. based on the principles of pedagogy 2. based on research

“Principled pragmatism is based on the pragmatics of pedagogy…Principled pragmatism thus focuses on how classroom learning can be shaped and managed by teachers as a result of informed teaching and critical appraisal” (Kumaravadivelu, 1994)

principled washback (n.) focuses on how test preparation can be shaped and managed by teachers as a result of informed teaching (through research and pedagogy) and critical appraisal (of both tests and academic skills)

INTRODUCTION

Many educators in EAP have the dual role of preparing students for success in the university classroom as well as preparing them for high-stakes gatekeeping tests like IELTS and TOEFL. Whether we like these tests or not, that students’ entrance into the academic world depends on these tests makes our job makes our job both more important and more difficult. If we focus too much on the test, we are sacrificing important long-term skills students will need to survive in academia. If we focus solely on academic skills, students might be OK, but they may not feel prepared for the test or satisfied with their classes, which are perceived as not meeting their (short-term) needs. Principled washback is meant to find a happy middle ground that addresses test prep skills en route to addressing academic skills.

Principled washback considers the academic demands of the classroom, the academic demands of the test, and then looks for overlaps in order to focus and frame instruction. These overlaps serve as starting points of instruction that reference test skills and perhaps emulate test questions but actually move students along to important work academic areas not addressed by these tests.

The IELTS and TOEFL are broken up into four parts: speaking, reading, listening, and writing. The IELTS treats these as separate (which speaks to its validity a bit) while the TOEFL iBT separates them AND integrates them. For my presentation and this blog post, I will separate them and then offer some ideas for integration. Continue reading

Principled Washback – IELTS and Academic Reading

In a previous post, I looked at research that analyzed the IELTS writing tasks in juxtaposition with university writing tasks. In this post, I will summarize similar research on academic reading tasks, apply this research to briefly analyzing TOEFL reading tasks, and then discuss implications for principled washback.

Introduction

Moore, Morton, and Price (2012) created an IELTS reading task corpus, created an academic reading task corpus, and interviewed a number of faculty members in order to get a holistic and in-depth view of the demands of academic reading. Their work is exhaustive and long (89 pages), but I hope to summarize it here as concisely as possible.

The purpose of their research was to analyze the validity of the IELTS reading section and make suggestions for improvements. One impetus for this was IELTS washback on ESL and EAP courses. According to the authors, “Washback is considered harmful then when there is a serious disjunct between a test’s construct of reading and the broader demands of real world or target language tasks” (p. 6). Therefore, this work is very important in deciding how much of a disjunct, if any, exists. Continue reading

Principled Washback – TOEFL, IELTS, and Academic Writing

I have already written about teaching students formulaic language to help with their writing, especially in terms of organization, unity, and coherence. However, I may have made a fundamental mistake in assuming that preparing students for test-based and academic writing at the same time was a good idea. In other words, I fell victim to the assumption that test-based writing had anything in common with academic writing, and hence there was room for transferability of skills (in either direction).

As I pointed out in my last post on IELTS and academic writing, it turns out I was wrong. but perhaps not entirely wrong. It really depends on which test students will be attempting. The research I summarized in my last post provides a good model by which to judge how academic a task is. According to the research, an academic task should have the following elements:

  • Genre: it should be an essay, case study, review, literature review, research report or some other academic genre
  • Information source: It must be written based on information outside the student. It must require understanding primary or secondary outside sources and not rely on prior knowledge (opinions, values, feelings).
    • Correctness: Related to the information source, content must have right or wrong answers. (note: this is based on my own interpretation of the research above)
  • Rhetorical function: The essay should employ evaluation, description, summarization, comparison or explanation. It should avoid hortation (should questions).

For the IELTS, as will be explained below, Writing Task 2 (which carries the most assessment weight) bears little resemblance to academic writing expected in university coursework. TOEFL iBT writing, on the other hand, may be more aligned with university work, as my anecdotal evidence shows. Both of these findings have clear implications for EAP and principled washback. Nevertheless, I still believe that, by covering essential academic skills such as argumentation, logic, and formulaic language, students will become better prepared for test-based writing en route to academic writing (and this is the point of principled washback). This belief is supported by evidence that EAP programs without a test focus are just as effective as IELTS prep courses.

Below, I briefly outline which tests I think are and are not related, compatible, comparable, or conducive to academic writing. Finally, I list a few essential skills that are common to both and should be taught in any EAP writing course.

IELTS Task 1 – Yes

For IELTS Task 1, students look at visual data and must summarize it, focusing on major trends and explaining differences. This clearly employs academic skills and represents an academic task common to many disciplines. One important aspect that makes this task align more with academic tasks is the fact that students must be able to interpret, evaluate, and compare information external to their own knowledge. Most academic tasks require working with knowledge external to the student while most test-based tasks only work with student knowledge. Indeed, this is one of the major criticisms of test writing discussed in my previous article. Unfortunately, while IELTS Task 1 represents an academic task, the value and importance to the overall IELTS score is overshadowed by Task 2.

IELTS Task 2 – No

Task 2 has been discussed at length here.

TOEFL iBT Reading-Listening-Writing Integrated Task – Yes

For the first TOEFL iBT writing task, students are given a passage to read. Then listen to a brief lecture. The reading and lecture are on the same topic, but offer differing or contradictory perspectives . The reading passage reappears alongside the writing task. The writing task is almost always: “Summarize the points made in the lecture you just heard, explaining how they cast doubt on points made in the reading.” Sometimes it will ask you to explain how they “challenge”, “support”, “answer”, or “strengthen” some proposition. Students are given 20 minutes to write a 150-225 word essay.

For this task, I think ETS were trying to do their best simulating academic writing while controlling for content-specific knowledge. In other words, they seem to have tried to focus on certain academic writing skills such as summarization, evaluation, explanation, and argumentation while limiting the need to understand subject-specific content (which would negatively affect the validity of the test). In addition, this task relies on understanding both textual and aural outside sources and is therefore a good simulation of academic work.

TOEFL iBT Independent Writing Task – No
TOEFL PBT Test of Written English – No

Both of these writing tasks are similar in nature to IELTS Task 2 above and therefore do not represent academic writing.

Essential Skills

So, how can these sorts of tests be addressed in the EAP classroom? As I have argued before, although EAP is just as effective in preparing students for these tests as IELTS prep courses, and therefore we could choose to avoid test prep all together, we still must contend with the fact that students demand of us preparation for both academia and tests. We need to deal with these demands in a principled way, teaching essential skills that prepare students for both, or teaching academic skills disguised as test skills. So, what kind of skills should we be teaching?

  • Organization – Getting students to understand logical organization, that is how to organize elements to present ideas in a clear manner no matter the genre or rhetorical function, is a major part of ESL and EAP writing. Whether students are writing a compare/contrast essay, an argumentative essay, or a research paper, organization still follows a basic logic – topics need to be introduced, background needs to be given, a thesis (central idea) needs to be formed and then supported, ideas need to connect back to this thesis, and the paper needs to be summarized and concluded. So, if students understand the basic requirements of English writing, they can be applied and expanded upon further whether it is for a test or for a multi-page research project.
  • Formulaic language – To help with organization, cohesion, unity, and flow, students should learn to use academic phrases and syntax that can aid them in making their writing clear and logical. These are applicable in almost any domain. It is worth taking the time to teach students how to utilize the Academic Phrasebank, which can help them greatly in their academic writing careers.
  • Visual Literacy – Students need to learn to understand and interpret data from charts, graphs, and other visual sources of information. This is key not only for IELTS Task 1, but across the disciplines. Some ideas include working with infographics or having students make their own.
  • Argumentation – Students need to learn how to make a sound argument in their writing. Argumentative writing is a primary writing skill on most test tasks and is also very common in academic writing. Having students work on making logical arguments supported by their own ideas is a great practice task. Having them further support their ideas through research moves the task into the academic realm. Being able to make a clear and concise argument based on outside information is a key that students must have. If students are being prepared for making logical arguments using evidence, then students should have no trouble at all “dumbing down” their skills on tests and writing solely from prior experience. The point here is not so much the information source, but making logical arguments and learning to fully support a thesis.
  • Writing from sources – Clearly, writing from sources is the sine qua non of academic writing. Students should be able to comprehend sources, especially secondary sources, and then learn how to summarize them, evaluate them, compare them with other sources, and synthesize them in order to use them in their writing. These essential skills are imperative for academic success, and along the way students will become prepared for writing on the TOEFL iBT as well.

The idea here is to teach EAP mostly as usual, but be willing to explicitly explain how EAP skills will be useful or can be modified for tests. Some activities, such as listening or reading comprehension, can be modified to incorporate elements of testing, especially if those elements are conducive to learning academic skills (e.g. reading comprehension questions or understanding implied meaning through paraphrase). Students will feel they are getting more bang for their buck and you can rest assured you are not falling victim to true washback and are therefore cheating your students’ education.

Research Bites: IELTS and Academic Writing Compared

EAP teachers have to contend with two goals that are often at odds with each other: preparing students for academic writing and preparing students for high-stakes academic tests such as the IELTS and TOEFL. While not mutually exclusive, it is important to consider the nature of these tests and what should and should not be incorporated into EAP pedagogy. I have been arguing elsewhere that both elements can be combined in the classroom in a way that fosters academic skills primarily and test-taking skills secondarily. Again, it depends on what is being tested.

The IELTS test has grown in popularity in recent years and, as its popularity has become more global, it has become a greater concern for more students, and therefore, more EAP teachers. An important question to consider is: should we spend valuable class time preparing students for IELTS writing tasks? For Task 1, I would argue yes, as it clearly represents an academic writing skill: interpreting and summarizing visual data. For Task 2, which holds more weight in terms of test scores, maybe not. The research below explains why. Continue reading

Principled Washback – Improving Writing with Formulaic Language

(This post is the fourth in a series of posts on “principled washback,” which I introduced here.)

A: How are you?
B: I’m fine, thank you. And you?

Just to get it out of the way, this is not the type of formulaic language I’m talking about.

This term has been particularly interesting because I am working with students who are under extreme pressure to achieve a certain TOEFL/IELTS score or risk losing their scholarships and hence their ability to attend a university in the US. What’s more, they have been here for a year and many have hit a plateau, or more like they are sliding back down the mountain as pressure and stress have actually made their scores decrease.

I am the writing teacher for some of these students and I noticed a major problem in their writing. Despite several terms of writing instruction, they still struggle with clear and concise writing. For example, one of my students has a very large vocabulary and very advanced grammar skills. This seems to actually be a detriment to his writing, as he tries to utilize these skills at the expense of clarity. (My mantra with him this term has been “Be direct, short, and simple.”) Eventually, this lack of clarity moves from just affecting the meaning of a sentence to negatively affecting the entire paragraph or essay.

How can a student like this be expected to succeed academically, let alone score high on the iBT or IELTS? The problem I identified, which is occurring with other students as well, is too much language play. Play is a great thing – especially with language, but sometimes structure is needed. And in writing, the structure needed here was formulaic language.

By formulaic language, I mean the phrases, chunks, and patterns that our language is made up of – those that we naturally use when we are speaking or writing. For example, formulaic language includes phrases like “in terms of,” “the extent to which,” “as a result of,” and so on. There has been much research about formulaic language running the gamut of ELT – from corpus linguistics, to SLA, to pedagogy – with as many perspectives on it. To my surprise, there is actually a lot of research against the use of formulaic language in writing. One reason is because, while a great deal of our native language use is formulaic (meaning phrase- or pattern-based), the there is still a great deal that is based on “language play” (Bell, 2012). For instance, Schenck and Choi (2015) found that “learners’ use of formulaic language reveals academic writing that is only mechanically proficient.” Native speakers, it turns out, are less formulaic than it seems.

This brings to mind several questions:

  • Does mechanical proficiency precede a more pragmatic, discourse-based proficiency?
  • Can you teach pragmatic academic writing without teaching formulaic expressions?
  • If students haven’t even reached mechanical proficiency, does it become a valid goal?
  • As Swan (2006) points out, should we expect learners to meet a “native-like” target, or should we be happy with clear and understandable communication?

There are a lot of things that can be discussed here. My focus will be on the last two points. Based on my experience, I believe that certain types of formulaic language in writing can have benefits for students and therefore should be incorporated into EAP writing instruction. These benefits include:

  • increasing clarity in writing for academic and testing purposes
  • improving organization in writing for academic and testing purposes
  • time management in writing for testing purposes

Which Formulas?

(Those with access to journal databases can find an Academic Formulas List (AFL) here or here. The formulas I am considering, however, are more systematic than the AFL. The Academic Phrasebank has some good formula resources.)

I am talking about two kinds of formulas. One concerns short academic phrases that serve certain discourse functions. These are usually subordinating conjunctions, prepositions phrases, or adverbial connectors such as:

Contrast However,
On the other hand,
In contrast to
Result As a consequence,
Therefore,
Explanation That is,
In other words,
, which means

See more examples here.

The other types of formulas I am talking about pertain both to language and organization. These include ways to write thesis statements, topic sentences, and even conclusions. Here are a few examples.

Argumentative thesis statement  

There are three reasons why ________ should/should not ….
There are several reasons why _______ should/should not …
________ should/should not ________ because ….

Topic Sentence One reason why _______ is because …
One reason _______ is that…
The main reason that ____ is …
Topic Sentences with Transitions In addition to ______, another reason for ______ is ….
Besides _____, a further reason why ______ is …

Clarity

Formulaic expressions can help students clearly express their ideas. Each phrase denotes a special function. By considering the function of the phrase, they can organize their sentence in a way that expresses their meaning clearly, especially in relation to other ideas. Embedded within these formulas are grammatical clues that can help students maintain grammatical accuracy to aid in their clarity.

Organization

It is not rare that I find a paragraph that has absolutely no point to it, either as a paragraph on its own or in terms of unity with a larger piece of writing. One reason this usually happens is that the writer forgets to include a topic sentence. By learning the more organization formulaic sentences above, students are much more likely to employ them. Hence, they are more likely to include essential elements that are required to organize their writing and convey meaning. These formulas give students an easy plug-and-play rule for writing thesis statements, topic sentences, and transitions. With practice using various structures, students will eventually move from sounding “mechanical” (but clear!) to varied and more natural.

Time Management

Writing a timed essay as a native speaker is a challenging task. Imagine how our students feel. They have to comprehend and then analyze the task, organize their ideas, figure out how to express these ideas in another language, and write these these ideas to include the writing conventions expected in this other language. This isn’t even including stress, fatigue, slow typing skills, lack of comprehension of the prompt, or lack of ideas. The writing practice in the classroom should be geared to dealing with many of these, but we rarely put our students under such time constraints. So, this is where formulaic language, especially organizational formulaic language, can come in handy. By having these chunks and plug-and-play sentences ready, students have to spend less time thinking about organization and conveying their main ideas. They have these ready-made sentences that they know how and where to employ. This will not only help them organize and write their essays faster, but it can help ensure they have some of the vital elements necessary for a high score. It’s a clear advantage to teach formulaic language in order to better prepare students for writing exams. This skill can also be extended outside of TOEFL/IELTS into university classes that employ in-class essay tests. Professors are going to be more focused on the content than the ideas. By using formulaic language, it could allow less cognitive resources to be spent on organization and the basic mechanics of an essay and instead allow more time to focus on explaining content.

How Principled is this Washback?

Formulaic language is clearly useful for testing, but, what about for the academic classroom? Some research finds it useful, and some does not, so the answer to this question is currently contentious. If students understand where, how, and why to use these phrases, as well as their accompanying grammar, I don’t see a major issue. If students are entering the sciences, mechanical writing might actually be more preferred. If not, I feel that once students grasp the use of this kind of formulaic language, they can start to compare their work to the works of others (through readings) and begin develop new styles of writing. Could they do this without going the formulaic route? Possibly, but this assumes they have a good grasp of structure, organization, and pragmatics. If they don’t understand that, they won’t understand the more preferred styles of writing.

So the main idea here is to teach direct, clear, and concise writing skills using formulaic expressions. This could be helpful for students for both academic and testing success.

References

Bell, N. (2012). Formulaic language, creativity, and language play in a second language. Annual Review of Applied Linguistics, 32, 189-205. Retrieved from here.

Schneck, A.D. and Choi, W. (2015). Improving EFL writing through study of semantic concepts in formulaic language. English Language Teaching, (8)1, 142-154. Retrieved from here.

Swan, M. (2006). Chunks in the classroom: Let’s not go overboard. The Teacher Trainer (20)3. Retrieved from http://www.mikeswan.co.uk/elt-applied-linguistics/chunks-in-the-classroom.htm.

Principled Washback – A TOEFL Review Game…with music!

(This post is the third in a series of several posts on “principled washback” which I introduced here.)

This post will deviate a little from my principled washback theme as it is not really focusing on an integrated activity that can be used for both academic study and test prep. However, it is introducing a unique way to do some test review in an extremely fun manner: a TOEFL game…with music!

This is a PowerPoint game I have been using for several years, modified every eight weeks or so. It is easily modified and adapted to suit learner level, class time, skill focus, music preferences, etc. Each game focuses on a specific skill, though it could easily be a mixed skills game. The listening game offers one listening track (or one lecture and one question) and a group of answer choices per slide. The reading game offers one question per slide along with a sentence, paragraph, or grammar problem.

Downloads

Materials Needed

  • Computer with projector (or smartboard)
  • PowerPoint
  • Optional: mini whiteboards

Procedure

  1. Set-up
    1. Form groups of three or four and have students decide on team names.
    2. Write these team names on the board in a way in which you can keep score (note: one of the PPT versions has a score keeper on it, but I recommend writing down points – just in case).
  2. Game Play
    1. Students will see numbers.
    2. Each number is a question.
    3. One group will choose a number.
    4. All groups will have 20 seconds to 1 minute (it varies by question) to discuss the question and decide on an answer.
      1. Press the space bar to start the timer.
    5. The answers are multiple choice: 1, 2, 3, 4 or A, B, C, D
    6. When the time is up, all students should either show their answer by holding up fingers (1/A, 2/B, 3/C, 4/D) or by writing it on a mini whiteboard.
    7. Play goes counterclockwise, with one group at a time choosing, but all groups may answer.
  3. Music
    1. If students get a question with a music image, all groups will hear a 30-second song clip.
      1. The first group to guess the name of the song and the artist(s) will get 2 points.
      2. Press the space bar to remove the music icon after the clip has been played.
      3. I try to keep the music clips updated based on what is popular at the moment. I typically use only American music, though I have used Korean music when I used this game in Korea.
        1. If you want to make your own clips, I use the site Clip Converter.
      4. If I realize none of the students know the music, I will let them “cheat” by using music recognition apps. Whoever find the answer first is the winner!
  4. Point Deductions
    1. If students see a specific icon (in the versions below, they may see a zombie or broken heart) they lose a point.
  5. Game Time
    1. Each game lasts about 40 minutes.
    2. If we finish early, I make sure to review all questions.
    3. Sometimes I have students write down the numbers of the difficult questions while playing and review the specific questions afterwards.

The game is pretty simple to play and very fun for students. It gives them direct practice with the TOEFL material, but it does so in a way that involves communication and negotiation while discussing the answer, and it does so in a way that is fun. In my experience, this game definitely lowers affective filters and students get very into it. The game really loses whatever “TOEFLness” it may have and it becomes a meaningful language game instead.