Hans van Loenhoud

Translating Exam Questions

No Double Dutch! [An article of the Inside IREB series]

Many Dutch students struggle with the IREB CPRE FL exam, which, in the Netherlands and Begium, was available in English only. They commented that the exam does not only test their knowledge of the IREB syllabus, but also tests their language skills. They certainly have a point: some alternatives in multiple choice questions may seem very similar to each other for those who are not native English speakers. IREB responded to these comments with an initiative to have all prevailing Foundation level exam questions translated into Dutch.

An exam is a means to evaluate the knowledge and skill level of a student. It is a way to ascertain whether the candidate has mastered the intended learning objectives as defined in the course syllabus. The best way to do so might be to discuss the separate subjects at the end of the course individually between an examiner and the student, finding out what the student really understands about the material. But, of course, in a worldwide operation like IREB, this is not feasible: it would be impossible to recruit enough skilled examiners, and to guarantee objectivity and independence at reasonable cost.

Thus, IREB CPRE Foundation level exams are provided in the form of multiple choice exam questions. IREB has chosen to standardize on a limited set of questions to be presented in a similar way to candidates all over the world. This ensures that everybody who passed the exam has a sufficient minimum level of understanding about the principles of requirements engineering.

In the Netherlands and Belgium candidates faced an additional challenge because the exam was not available in Dutch. As most of us will agree, IREB exam questions are not that easy. Which is how it should be, because requirements engineering itself is not easy. Requirements engineering is a subtle profession and skill, in which good communication is key. We teach our students that when natural language is used for documenting requirements, a requirements engineer must take care to communicate the concepts in an unambiguous way, avoiding transformational effects as much as possible. Our exam questions are written in natural language. And they are not unambiguous, at least not at first glance. The alternatives in a multiple choice question may seem quite similar to the correct answer. Only the candidate who really understands the subject may be able to distinguish between the correct answer and the so called distractors. But if the question is not written in your native tongue you can easily miss the subtleties that make the difference between a good and a bad answer.

In the Netherlands, most students who attend an IREB CPRE course do pretty well on English. They read the syllabus and the textbook in English, and are quite confident that they understand the subject matter. At least, that’s what they think, until they start working on questions in the practice exam. Then they find out that they really need to understand the exact meaning of all alternatives of an exam question in order to pick the right answer. And then, on (too) many occasions, their knowledge of the English language proves to be deficient. Thus, candidates may fail their exam, not because of their lack of the required knowledge on requirements engineering, but because they do not fully understand the exact wording in certain exam questions. This is an undesirable outcome, as the exam should test requirements engineering, not their proficiency in English grammar.

IREB CPRE Foundation level courses are quite popular in the Netherlands, and a growing number of students faced the additional challenge of examination in a foreign language. Several trainers observed a demand for the exams in Dutch and discussed this with IREB. IREB was open to discussing the problem and welcomed a solution, especially if the Dutch requirements engineering community would contribute. To this end, IREB started a working group for the translation in which several training providers from the Netherlands participated.

As a pilot project we started with the 34 questions of the practice examination. The questions were equally divided amongst the participants, and translator / reviewer teams from different providers were formed. As soon as the individual teams finalized their translations, the questions were merged into a complete set and harmonized in style, spelling, etc. Subsequently, two reviewers did a final review to guarantee consistency over the complete set.

The pilot translation revealed several attention points: it appeared that for a good translation, working with the English version is not sufficient; a comparison with the original German text was required. Also, frequent consulting of the IREB glossary is a must in order to maintain consistency in terminology. Fortunately, the present glossary already contains an English – Dutch translation for all relevant terms.
The translation of the practice exam went smoothly and took about one and a half month: quite satisfying considering the fact that it all was done on top of normal working activities. Careful planning and monitoring of progress proved to be successful in order to avoid time being wasted in waiting for each other.

After the successful translation of the practice examination, we started with the ‘real’ exam questions. This introduced an extra complication with regards to confidentiality. It is vital to keep the exam questions secret to the outside world in order to guarantee a level playing field for all candidates. Also, training providers having access to the exam questions in theory would have an advantage over others. Therefore, it was decided that no single provider was granted access to the complete set of questions; both translators and reviewers were allowed to see only a limited number of questions.

Thus, all exam questions were provided in DropBox in chunks; every translator got access to only a part of the question set. The files containing the questions and the resulting translations were encrypted; the passwords were created by IREB and distributed on a ‘need-to-know’ basis.

With the experience of the practice exam in mind, the progress was even better; we managed to translate the complete set of questions within a month. Overall, it took 15 to 30 minutes to translate a single exam question, reviewing on average was done in half that time. In order to maintain the confidentiality of the question set, the task of integrating the translations in one complete set was delegated to an exam provider for IREB in the Netherlands and Belgium.

Coincidentally, a Dutch textbook ^[CAN14] based on the IREB syllabus has recently been published. As a spin-off, the authors of this book provided a translation of the Foundation level syllabus as well. From now on Dutch students are able to follow the complete course in Dutch: read it in Dutch, practice it in Dutch and then take the exam in Dutch. IREB and the exam provider will carefully analyze the exam results to see whether this translation proves to be beneficial. Of course, our expectation is that the success rate will improve. That would mean that more certified, better educated requirements engineers will become available in the Netherlands and Belgium.

In the end, better software will be the result!

References

• [CAN14] Cannegieter, J. J., N. de Swart & J. Zandhuis. Grip op requirements (Eburon, 2014).