What's wrong with multichoice exams for ITIL V3?

APMG and the ITIL Qualifications Board are insisting on using multi-choice exams to certify ITIL V3 competence for Intermediate and Expert level qualifications. We have discussed this already across the blog but allow me to summarise the arguments against it here.

As argued elsewhere on this blog, by me and others, multichoice:

  • might be wrong, or open to debate (the examiners are not perfect), but there is no forum/medium for arguing a better answer. It is extremely patronising to assume that the examiners always know better and are always completely right in a complex domain like ITSM. There are anecdotal reports of people arguing with the orthodox position in their V2 Manager's exam essays and getting a pass because of it
  • should be crisply back or white. the V3 exams reportedly have a very right answer and a less-right answer. This sort of subjectivity indicates that the answers are opinion-based not fact-based
  • test the ability to recall sacred texts or guess eight correct letters. This is not a test of somebody's ability to communicate complex ideas, to write articulately, or to argue subtle multi-faceted positions, all of which I would regard as essential in Expert and probably also Intermediate level roles

Multichoice is good for testing basic facts with one clear right answer. How many questions can you come up with in the ITSM domain that meet that criterion, especially questions suitable to test above Foundation level?

Here is a Real ITSM exam question:
7) For which audience are multi-choice questions best suited?
a) kindergarten
b) primary school
c) trade apprentices
d) gossip magazine sex questionnaires

Comments

Why would they do that?

Furthe rto Ken's comment at the top about multi-choice only being as good as the content, I saw this ITIL V3 Foundation exam tip on MJ's blog (another good one to be following)

Memorize your definitions - The questions on the exam will provide 2 out of 4 answers that make sense and they will look very similar/plausible. If you don’t have the term memorized, you’ll be stuck. Trust me; app. 15 questions on the exam will be just like this. But MJ, that doesn’t test my knowledge of ITIL at all?! Why would they do that? I don’t know.

You are right, MJ. They still have a LOT of explaining to do...

Disappointment is always driven by expectations...

As a v3 Expert and v2 Service Manager, I do have to say that I'm a bit sorry to see the old hand-written Service Manager exam format go by the wayside.

But, here are some factual (eek!) clarifications that may be helpful. (I'm starting to get the gist of how the game is played here: The Skeptic puffs up with florid indignation around a flabby sort of issue...and then the rest of us get to satellite around him. All good fun!)

Multi-choice exams are nothing new in the ITIL world. In fact, since as yet there is no exam at the Expert level except the bridging exam, there has been almost no net change in the format of exam delivery. All v2 practitioner exams were also multi-choice, just as the v3 intermediate ones are. So, maybe more smoke than fire here, eh?

Second, the exams are just a tool for focusing attention. They don't make you any more of a service management professional than passing the driver's exam makes you a race car driver. That's one reason I like tough exams and why I really appreciated the old format of the ITIL v2 Service Manager courses: they provide at least a fleeting (unfortunately anxiety-induced) opportunity to get people to really engage the material. I teach those (and the practitioner classes) with a real emphasis on getting people to engage aggressively with case study problems, doing the role plays, etc. I'm typically pretty bullish on people (all in keeping with good humor) as a means of really encouraging them to skull through not only the material, but the environment in which they're being asked to apply it. Good habits of application can begin in the classroom...if you make it so!

As a person who is fairly familiar with the internal working of the exam mill, I'll say that in general the APMG machinery for generating test questions is pretty primitive at best. Initially, there were serious conflict of interest issues with the mechanism by which exam questions were vetted for v3. This has now changed. Still, the overall pool of exam questions is pretty small and the tests are not adaptive. Expanding the pool of questions would be great and using an adaptive engine would be a big improvement.

In my view, there's no clearly superior format for testing. Sure, multi-choice has big drawbacks, but what's better? It pays to consider the attendant problems of other formats. (To The Skeptic's anecdotal claim regarding the possibility of challenging the old v2 SM exam, I'll simply say that I personally know of no cases in which that occurred. Much more commonly, a challenge to the v2 SM exam meant a guaranteed fail.)

So, my advice is to get real about what we expect from exams in the first place. They're simply (minor) milestones on the way to developing better skills. That development happens as we really work the material, discuss it (Go Skeptic!), take it apart, and put it back together in ways that help us get things done.

Written v Multi-choice

"All v2 practitioner exams were also multi-choice, just as the v3 intermediate ones are."
- ISEB Practitioner exams consist of part multiple choice (often requiring analysis of provided information, not just a memory test) AND a 1 hour written paper - a good balance, IMHO.
"To The Skeptic's anecdotal claim regarding the possibility of challenging the old v2 SM exam, I'll simply say that I personally know of no cases in which that occurred. Much more commonly, a challenge to the v2 SM exam meant a guaranteed fail."
As a member of the marking panel, I can say that we DO look for reasonable, well-argued answers which depart from the accepted wisdom. Sure, if someone wrote that ITIL was all rubbish, they would not pass, but the arguments, for example, about the practicality of achieving a CMDB as described in the books, such as discussed on this website, would certainly be marked favourably, if the question was about implementing such a database.
Liz Gallacher
Freelance Trainer and Consultant

V2 versus V3 exams

There is in my opinion a number of problems with the V3 exams:
- the quality of the current multi point choice exams V3, is far less than the quality of the EXIN exams in V2. I think it has been a wrong desicion of OGC to take AMPG (who had NO experience in ITIL, nor in higher level exams like the servive manager) in stead of EXN who had a far larger base of exams questions and a much more developped ability to translate the exams. In French there is actually only ONE version of the V3 foundation exam!!!!!!!!!!
- the choice of no longer demanding the In Course Assessement during a Service Manager is a strong degradation of the level of the SM exam
- Examining the Expert level with Multi Point Choice is an absolute mistake. Not only becasue of the poor quality of the actual questions but also because at the management(expert) level one also has to show his ability to give arguments for an answer.

I guess APMG has just been chosen for the fact that they are english and not based on proven track record in ITIL

Just to ad to my previous

Just to ad to my previous comment (I forgot to sign in, Pjotrg is my nickname, my real name is Peter Gerritsen)

Trainings and certifications have to respond what customers want. This also is one of the points I have with the current V3 qualification scheme. Does it really respond to what customers want?

When looking at what the customer wants there are perhaps different customers:

- companies who want to train a lot of staff as they are implementing ITIL: they are looking for a training that makes their staff understand "how, why and what" about ITIL. For this group ITIL V3 foundation is way overdone even counter productive to my opinion! I'd say that even a V2 foundation is too much

- people who already have an understanding of the ITIL principles and want to concentrate on the more detailed implementation of one ore more processes. For this group the V2 (clustered) practitioners might be usefull. I've understood that the capablity modules of V3 point the same direction, but I have had no change to study the syllabi yet (was a bit to busy giving training ;-)

- people who want to become consultants and or trainers. They might want to have more in depth knowledge about ITIL. The V2 Service Manager training is a good means to it.........provided the person can integrate that theoretical knowledge with his own experience, skills, education and background (preferable on some sort of MBA level). The life cycle modules cannot IMO Perhaps the "Managing through the life cycle syllabus will open our eyes, who knows???

Oh and please have pity with us, poor trainers. To be worth something in the new "ITIL world" we have to pass all 10 intermediate and expert level exams!!! You have to have the certificate in the exam you're training and not just the level above like you had to have in ITIL V2. What is the use of passing the capability exams if you are an experienced expert in ITL and have for instance already passed the 5 lifecycle exams plus managing through the life cycle?

Seen in this light the certification scheme of AMPG is not completely (or completely not) responding to the customers unless.......APMG's customer are the ATO's (who want to sell training), exams institutes (who want to sell certifications) and last but not least OGC!!,

not about the trainees

Peter I hope you are not being facetious. Of course "APMG's customer are the ATO's (who want to sell training), exams institutes (who want to sell certifications) and last but not least OGC". I've said it before. the industry is not about the trainees.

I'm bloody serious !

Not at all Skep. I'm bloody serious ! Our customers have their employees follow an ITIL training because they want them to understand what ITIL is about and to prepare them to work in an "ITILized" environnement or project. For them the certificate should give a proof of having understood all this. And all Iam worrying about is that we are not delivering this proof through the current APMG certificates!

The old V2 manager approach including In Course Assessment (of course conducted properly by a serious trainer, like me ;-) LoL ) gave at least somewhat more of the drivers licence aproach

Despairing

I don't know how many readers of this blog attended the itSMF UK conference last week but I came away from the open forum on the exam schemes in a state of deep despair, which I know was shared by others I spoke to. Even one [edited here:] person actively involved[] told me off the record that they found it hard to stand up and defend the multiple choice approach. Particularly worrying was a throw away comment that a qualification might be withdrawn if it did not get sufficent support - so early adopters run the risk of spending time and money obtaining a qualification that could rapidly become obsolete.

Peter missed out one type of customer - the one who in a time of recession wants a training course that will equip their staff to deliver positive outcomes, rather than to become experts in an ivory tower model.

As a direct response to the conference I found myself drafting the outline for such a course.

James

Intermediate exams

I too have to sit all the exams, in order to teach each course. I have V2 Manager (distinction), V2/V3 Manager's bridge(and am therefore an official ITIL Expert), and have put this all into practice for many years, as an employee or consultant. I am on the V2 managers examiners panel, have taught V2 at all levels, and V3 Foundation and bridge courses. I have also produced course materials which have passed accreditation. All this means nothing, however, when faced with a so-called complex multiple choice exam. I have sat a number of these, and my scores have varied from 90% to fail. You have no feeling when sitting the exam which of these you will achieve! For those I need to resit, it is not a question of studying harder - it is somehow identifying which is the examiner's favourite answer. One paper in particular gave a number of virtually identical answers, each about a page long, which you had to go through, crossing out all the duplicated sentences,until you were left with the odd phrase that differed, then try and decide which of these was better.
I do not have faith in those compiling the questions that they are any more expert than many sitting the exam, or that they have had any experience/training in the subtle art of writing good questions.
As a trainer, I am also concerned that one could write a course which covered the syllabus, have really good, practical exercises, which students enjoyed, but which would not/could not prepare them to pass the exam at the end. (And that is how the course will be remembered - as leading to a fail)
Liz Gallacher
Freelance Trainer and Consultant

Whats wrong with exams

Another recurring theme from me..

Whats wrong is the concept of certification. What is certification ? What does it deliver ?

Understanding ? NO
Capability ? NO
Knowledge ? NO
Attendance ? Maybe

If certification is the only system we have to trying to validate our capabilities as professionals, perhaps its best we don't have anything at all..

$0.02 (which is worth a lot less in the last month)

Brad Vaughan

Has anyone mentioned

the cost? Multiple choice exams are cheap. Cheap to run, cheap to mark and cheap to quality assure.

Exams only ever test somebody's ability to do the exam. They are always a poor proxy for the actual things you want to assess. How poor a proxy depends upon what you are trying to assess and the method employed.

For a middle-class englishman like myself a written exam with essay type answers is the comfort zone. I spent 7 years getting better and better at them from age 14 to my university finals. So it isn't surprising that I passed the V2 Managers Certificate with a distinction first time around.

So with so many poor choices it seems fairly obvious to me why multiple choice was the winner. It's cheaper and therefore generates greater profit. Nothing to do with quality. Nothing to do with testing the skills that should be under examination.

Alex Jones

if all you know is how to tick one of four

I hear you Alex. i've mentioned a number of times that the choice of multichoice is profit-driven.

I also understand that passing any kind of exam is a developed skill (I've posted about how to optimise chances in multichoice). if you are skilled at writing essay answers I think you are better equipped to write proposals, process descriptions, promotional newsletters etc than if all you know is how to tick one of four.

Quality of Multi choice

I agree with your arguments having (as a trainer) just sat the OSA and SOA Capability exams and found the quality varies enormously. What doesn't help is that the rationale used to justify the answers in the Mock exams is also frequently wrong and open to debate to the extent that we all agreed it was pointless to share the rationale with the students. Doing so would lead to further discussion and the potential undermining of the trainer's credibility as we felt that the rationale was wrong in some cases and answers with a lower mark were, in some instances, stronger than the top answer. If the mocks are wrong then what value can we place on the QA for the exams?

As it is, I found the SOA exam quite easy to do and achieved 36/40. It felt straighforward and the answers seemed quite obvious.

However, I found the OSA exam to be difficult and will be surprised if I have passed. Although the scenarios were straightforward, the 4 answers to each question tended to be very similar and, in some cases, were quite convoluted (switching phrases around), wordy and certainly demanded full attention.

It also appeared, in some instances, that the distinction between the most correct and next correct answer seemed to hinge around the interpretation of a particular word or phrase.

Whilst I can see the value of this type of testing, it only works when, as you say, there is a clear distinction between each answer. Producing questions with this level of rigour is likely to result in a lot of frustration as most students will probably give up and just guess!

If I'm struggling as a trainer, what chance do the students have?

btw - the captcha is very difficult to read (10 bl***y times I've got it wrong - why can't you use multiple choice!)

Depends Upon Qualty...

Skep,

1. Just because you have not seen or taken a multi-choice exam with your desired characteristics or way of evaluating advanced content concepts does not mean that they don't exist -- they do! I've taken a number of them. The rigor in exams doesn't come from the form of the question, it comes from the content and what it's intended to address.
2. You said:

This is not a test of somebody's ability to communicate complex ideas, to write articulately, or to argue subtle multi-faceted positions, all of which I would regard as essential in Expert and probably also Intermediate level roles

  • Is this really a requirement for someone to be able to understand and leverage the concepts in their work or for their organization? [No, I am not arguing the utility or value of these capabilities.]
  • How does arguing a position allow you to be better able to serve your customer (internal or external)?

Again, if you're not clear about what you're trying to test (or why for that matter), any form of question can be a "bad" one.

kengon

Foundations Exam

I recently passed the v3 Foundations exam, which is probably the first multiple choice test I have taken in 20 years.

First, while perhaps multiple choice test is ok for the Foundations exam I would not want it to be the sole test for the expert level exams.

Second, I was shocked to see that one only needed 65% to pass. When I was in school that was at least a D! (I passed with 90%)

bill

let loose on the roads

Hi Ken, I see your point about being able to "understand and leverage the concepts in their work " - if that is all the certification is for then just grasping the concepts is enough. To me a OSA Intermediate qualification should indicate someone is fit to operate a Service Desk or an internal expert/champion or the owner of related processes. I'd say a process owner needs to be able to document, teach and promote the process.

At Expert level I think the expectation is that it qualifies someone to be a consultant or an internal expert/champion of all of ITIL and there is most certainly an expectation that they can communicate concepts to others articulately.

In both levels, the real world will present complex situations which will not be as clearcut as the multichoice, where subtle decisions have to be made. the ability to argue and reason and weigh factors is just as important as a grasp of concepts. Multichoice does not test this. Essays do.

If the certifications only indicate that the person has a basic grasp of core concepts then multichoice is probably enough. I'm not sure that is how they are perceived, especially the Expert one. We need some sort of professional certification and obviously APMG and co are not out to provide it. But in the absense of anything else I think the ITIL certifications are being used for more than certiftying basic facts.

In most countries a driver's licence consists of a multichoice, an eyesight test, medical certification and a practical driving test. The licence indicates you are able to operate a car. And yet for ITIL they only have to pass the multichoice, no practical exams, no test of basic capability, before they are let loose on the "roads".

The Fundamental Question...

Ultimately, this brings us to a fundamental (multi-choice) question:

1. What is the ITIL qualification scheme for?
a) A scheme to generate revenue
b) Demonstrate knowledge of basic ITIL facts
c) Evaluate candidate capabilities to operate as an internal/external ITIL champion
d) Build/reinforce brand loyalty
e) a and b
f) a and c
g) none of the above

The way things are, each person needs to find the answer to this question themselves, regardless of what any official answer there may be. The more ambiitious the goal of the qualification scheme, the more diligence is required to build an appropriate evaluation mechanism.

In most countries a driver's licence consists of a multichoice, an eyesight test, medical certification and a practical driving test. The licence indicates you are able to operate a car. And yet for ITIL they only have to pass the multichoice, no practical exams, no test of basic capability, before they are let loose on the "roads".

Your analogy is an important one to investigate. Yes, this does happen today, to the detriment of the entire community. Is this a failure of the qualfication scheme or is it a misuse of it?

Perhaps the real failure lies in a set of improperly set expectations with all of the stakeholders. We expect more than this actually provides, regardless of what (currently) works/doesn't work about it.

If you're looking for something that this isn't, perhaps it's time to look for something more appropriate, instead of trying to "put lipstick on a pig" (gratuitous U.S. election humor)

kengon

Basic content OK

If the idea is to provide a mechanism for non-professionals in a field to obtain some sort of knowledge on a topic, then I think the current training and examination mechanism can work. I would even say the "Short answer" question model of the higher level certifications is way overkill. I have seen project managers, software developers etc.. get some good value from foundation level courses and exams, both from application in there jobs and also empathy for the roles of other people in the deployment lifecycle.

The failure of the process in my mind is when you expect real capability from the person. The training, exam and certification process fails in this respect.

Brad Vaughan

Syndicate content