Should we spend more time designing multiple choice questions? a) Yes b) No

I am the first to admit that when I plan my lesson resources, I spend far too long making them aesthetically pleasing. Of course, I try to ensure that my instructional design is efficient and the content is challenging, but I enjoy making the resources look great too. There are probably many others, just like me.

 

In this post I’d like to look at formative assessment design, specifically multiple choice questions (MCQs), and I will argue why I need to spend more time focussing on designing these and probably less time on how ‘funky’ my resources look.

knowing_online_assessment

Let me explain the benefits* of multiple choice questions before I go on to how you might approach the design of them:

  1. The ‘testing effect’ – Frequent quizzing has shown to enhance later exam performance (McDermott et al, 2014) as learners are provided with the opportunity of retrieval practice. There is, however, research that indicates that MCQs might not be as effective for retrieval compared to short answer response questions due to the answer being available, thus learners are not required to think as hard (Kang, McDermott and Roediger, 2007).
  2. Identifying gaps in knowledge – We can quite reliably use multiple choice tests to identify the gaps learners have in their knowledge. We can use this information to close the gaps in knowledge for groups or individuals. Whilst there is the argument for guessing answers, if we increase the number of plausible incorrect answers and the number of questions to respond to, we do increase reliability. This is illustrated in the work of Burton et al (1991):
  3. 00001Furthermore, when we ask questions to learners in class, each learner is usually asked something completely different and therefore, this results in completely different answers. Whilst I think questioning is useful, it isn’t a reliable measure of learner understanding, plus we only know the response of that one learner we ask, not the others.
  4. They can be used as a diagnostic tool – If the questions are written as such that we can determine why the learner is selecting a particular answer, then we can start to diagnose problems with their cognitive processing (Wylie and Wiliam, 2006). Wiliam advocates using a small number of these questions at a hinge point in the session, whereas I’d argue that they’re useful at any point. The problem is tha they are quite challenging and time-consuming to create, as Harry Fletcher-Wood and others have found. I’m still developing my thinking on these, but here is a video by Wiliam which explains it in a little more detail – “Kids cannot get it right for the wrong reason”.
  5. Quick, visible responses  – When the learners answer these questions, we need to be able to see the responses of all individuals. This can be done quickly and with ease through the use of mini whiteboards or holding fingers up. The teacher can view the whole class in a few seconds and determine whether they can move on, or revisit information accordingly. The wealth of formative assessment online tools also allow for MCQs to be administered to all, and automated marking can generate clear analytics quickly. I personally like Google Forms, but there are many other online tools offering similar features.

 

Designing the Stem

In designing effective MCQ’s I have found several research articles and documents (1,2,3,4). Each of which offer similar advice when writing questions and answers. Here are some of the key things to consider when writing the stem:

  1. The stem should be meaningful by itself and should include the main idea.

Basically, this means that the main thing you’re trying to find out about should be in the stem. E.g. What chamber does deoxygenated blood enter in the heart? (I am trying to find out if they know about chambers of the heart).

 

  1. The stem should not contain irrelevant material.

This will just serve to confuse learners and this can cause more harm than good. Learners may create ‘false knowledge’ if the information is not relevant.

 

  1. Avoid a negatively written stem.

Where negatives are used in the stem, this can make the question easier according to Harasym, Price, Brant, Violato, and Lorscheider (1992). Furthermore, negatives can cause ambiguity in what is being asked and just because a learner knows an incorrect answer, this doesn’t mean that they know the correct one. Here’s an example by Burton et al (1991).

Which of the following is not true of George Washington?

  1. He served only two terms as president.
  2. He was an experienced military officer before the Revolutionary War.
  3. He was born in 1732.
  4. He was one of the signers of the Declaration of Independence?

 

Designing the Answers

In a meta analysis of MCQ research, Rodriguez (2005) informs us that it is not the number of distractors but the quality of distractors that are important when designing answers for MCQs. It was found that writing more than 2 distractors can become challenging and is not significantly more effective that having more, so having A-C is fine if you are struggling to produce A-D. The key elements that should be addressed when writing stem answers are:

  1. All alternatives should be plausible.

Essentially, each incorrect answer should be plausible. In the example below, there is clearly one implausible response:

In what year was Winston Churchill first chosen as Prime Minister?

  1. 1700
  2. 1940
  3. 1941
  4. 1942

 

  1. Alternatives should be stated clearly and concisely.

Try to avoid unnecessary ‘waffle’, so that in interpreting the question, the cognitive burden is reduced.

 

  1. Alternatives should be mutually exclusive.

There should not be more than one answer that can be defended as a correct response by using correct reasoning. An example by Burton et al (1991) shows two possible correct answers:

How long does an annual plant generally live?

*a. It dies after the first year.

  1. It lives for many years.
  2. It lives for more than one year.

*d. It needs to be replanted each year.

 

  1. Alternatives should be free from clues about which response is correct.

Avoid including a word from the stem in the answers. This can provide a clue to the answer, and for some, they may think of it as a trick question, thus go with an alternative answer. For example:

What muscle is the agonist on a bicep curl?

  1. Bicep
  2. Deltoid
  3. Hamstring
  4. Tricep

 

  1. The alternatives “all of the above” and “none of the above” should not be used.

Speaks for itself as more often than not, this option is the correct answer in MCQs.

 

  1. The alternatives should be presented in a logical order.

The best approach suggested is numerical or alphabetical to avoid any clues as to which is the correct response. When working with City and Guilds on a project a few years ago, I was also advised to avoid having the starting letters of each answer show an obviously different response. For example, in the first set of answers, Hungary clearly stands out and this might lead learners to respond with that, whereas in the second set of responses, each starting letter is different and leaves no clues:

  1. Germany                 a. Germany
  2. Ghana                      b. Hungary
  3. Greenland               c. Poland
  4. Hungary                   d. Russia

Summary

As can be seen, writing MCQs isn’t something you can throw together 5 minutes before a lesson. To make them effective, it requires time and a number of elements need to be addressed. I’d suggest working with your colleagues to build a bank of questions.

 

*I have stumbled across a couple of problems with MCQs which are worth examining. @surrealyno has written a short piece on the disadvantages of MCQ’s and Roediger and Marsh (2005) also found that using MCQs could lead to ‘false knowledge’ in some students, where they believe an incorrect answer to be true. With due consideration of the abovementioned points, it certainly will mitigate against some of these concerns and I’d argue that the advantages of using MCQs outweighs the disadvantages, particularly compared to alternative methods of assessment.

 

Advertisements

7 thoughts on “Should we spend more time designing multiple choice questions? a) Yes b) No

  1. Really interesting, thank you! And I’m reading this whilst writing up a study I did where one test section was about avoidance of false friends, where there was a kind of MCQ (all correct answers and one distractor for each in a box at the top to choose from). It turned out to not be a reliable test, but it’s hard to think of other ways of testing false friends – so these tips will be very helpful in designing the next tests!
    Clare

  2. Pingback: Own your room

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s