Underrepresentation of listening comprehension
Why is language testing for the ICAO LPRs in such a poor state? One reason is a lack of assessment literacy among test service providers and regulators which has led to the inadequate treatment of listening comprehension in aviation English tests.
Pilots and controllers do A LOT of listening on the radio. Listening comprehension is at least 50% of the communicative load. Listening is a complex process that results in a partial, unique, fleeting and invisible picture in the listener’s mind. Because the listening process is not directly observable, we need carefully designed instruments which present:
Plenty of pre-recorded text which is representative of radio communications; and
A variety of tasks designed to tap listening specifically.
Good tests of listening comprehension take time to administer. The higher the level at which we aim to assess, the more comprehensive and therefore the longer our listening tests need to be.
On scanning the websites of test service providers, two issues are evident:
1. Many assess the ability of pilots and controllers to understand radiotelephony communications:
With very limited comprehensible input; and/or
Without reference to recordings of radiotelephony communications at all.
2. Many assess listening through performance in speaking tasks. This results in poor measurement because:
Speaking tasks cannot present the quantity or quality of comprehensible input required to enable robust measures of listening; and
What people say is influenced by many things, only one of which is what they understand.
Human rating of listening through a test taker's performance in tasks designed primarily to elicit speaking leads to indirect, muddied and error-prone measurement.
Until test service providers better understand the listening construct and offer tests which adequately address listening comprehension in the specified domain, pilots and controllers will continue to take poorly-constructed language tests that fail to address aeronautical radiotelephony communication.
Download this collection of blog posts here:
