User perceptions and assessment literacy
There is a lack of assessment literacy coupled with competing priorities on the part of the organisations and individuals who use aviation language tests.
We would like airlines, ANSPs, ATOs and individual test takers to choose good testing practice over bad, but it is perhaps unreasonable to expect them to. To identify good practice, you need genuine interest and field-specific expertise. Test users are in the business of flying aeroplanes, controlling traffic, keeping costs to a minimum and getting on with their lives. Unless they are involved with test development and administration themselves, test users cannot be expected to be experts in language testing any more than I can be expected to be an expert in medicine when I visit the doctor.
Some argue that test users don’t care. This is not true - many care very much. But we have to accept that the majority of test users are busy people with different priorities and motivations. I would go further and argue that any ignorance that we see manifesting in expediency and apathy has been strongly exacerbated by naïve and lazy regulation frequently exploited by amateur testing practice. Weak regulatory standards and sloppy testing have contributed to any perceptions held in the industry that language proficiency is not a central part of the safety picture, but a box to be ticked, a necessary evil.
Where does the onus to improve standards lie? Is it with ICAO who is responsible for the standard itself? Is it with the regulators who oversee it? Or is it the TSPs who, after all, are supposed to be the experts? Whoever is responsible, test users can only choose from what’s available, if indeed they can choose at all. And where choice exists, poor practice lurks all too often behind edifices of reputability, and glossy websites with spurious claims for validity and promises of low prices and instant results which are so seductive that test users cannot see the wood for the trees.
Until regulators and TSPs better understand their professional responsibilities and make a more concerted effort to ensure the provision of relevant and meaningful language assessment that meets established standards for quality, pilots and controllers will continue to take poorly-constructed language tests that fail to address aeronautical radiotelephony communication. For the thousands of professional pilots and controllers who keep our skies safe, that is unjust.
Conclusion
This collection of short essays summarises my personal incredulity at the state of language testing in aviation. While revision of the standard and associated guidance would help, the real problem is a deep lack of assessment literacy amongst those responsible for the provision and oversight of testing services. The result is that many regulators fail to regulate which means that:
Untrained individuals conduct casual layperson assessments on the fly;
Non-commercial TSPs operate without adequate training and support; and
Commercial TSPs (sometimes shamelessly) prioritise profit over professional responsibility.
The widespread amateurism is lamentable. Due to ignorance, wilful or otherwise, most aviation language testing does a disservice to not only the industry and individuals it aims to serve, but to aviation safety.
‘Language testing has evolved into an independent field that is characterised by well-articulated theories of validity and sophisticated validation methodologies’.1 This translates into good practice in other domains (e.g. medical English, academic English), and it should in aviation, especially given the stakes involved. The fact is that it hasn’t: over the last 19 years, in spite of robust industry tools (e.g. ICAO recognition) and guidance (e.g. the ICAEA Test Design Guidelines) things are getting worse. Today, professional TSPs do exist, but they are vastly outnumbered by poor ones. The profoundly negative impact that this has on training continues to undermine the whole purpose of the ICAO LPRs.
Critiquing is easy. Putting things right is not. So, what to do? Perhaps the first step is raising awareness across all industry stakeholders - regulators, TSPs, industry organisations and test users - that something’s wrong. The more noise we make, the more likely it is that things will improve, and the more likely it is that any earnest efforts to promote better standards will gain traction. If you’ve taken the time to read any of the articles in this series, and if they have resonated with you in any way, then please don’t let your silence be interpreted as acceptance. The status quo is woeful, and things will only change if we call it out. Let’s hope that change comes before we have another Tenerife, Cali or Charkhi Dadri on our hands.
1Xi, X. (2014) Methods of test validation in Sohamy, E. & Hornberger, N. Encyclopedia of Language and Education (2nd Edition) Volume 7: Language Testing and Assessment (pp. 177-196) Springer: New York
Download this collection of blog posts here:
