Short-Answer Questions

Short-answer questions (SAQ) are routinely used to assess reading and listening comprehension. The questions can target different types of constructs: comprehension of gist/main idea, supporting ideas, specific detail, inferencing, deducing meaning of unknown vocabulary from context.

One advantage of these items is that they eliminate the possibility of guessing – the problem with all selected-response task types. SAQs are also thought to probe deeper into test-taker’s ability. SAQs are also easier to produce compared to selected-response task types, especially multiple-choice items.

On the minus side, marking SAQs might be difficult because there are normally many ways to express the same idea. That is why the answer key for SAQ tasks should, ideally, contain all possible response permutations. However, even most careful of item writers will not be able to anticipate all acceptable alternative answers. Therefore, SAQs should always be carefully trialled to reveal acceptable but unanticipated responses. Marking SAQs should also involve careful consideration of all responses, even if they are not included in the answer key. That is why large-scale tests often avoid the SAQ format. In such tests, responses are normally marked by clerical markers (people who mechanically mark against the key) or machines. 

Another problem with SAQs – similar to all constructed-response tasks – is that they might introduce construct-irrelevant variance. This is because SAQs require written responses – answering correctly, at least in part, depends on the test-takers’ writing ability. When SAQs are used to assess, for example, listening or reading comprehension, writing ability is irrelevant. 

Recommendations for writing SAQ items

• Make it clear in the instructions what is expected of the test-takers:
  •  Must they respond with a number, a word, a phrase, or a complete sentence? E.g., “Answer with a word or a short phrase”.
  • What is the maximum number of words the response can contain?  E.g., “Use a maximum of three words”.
  • Should the words come from the text or can test-takers paraphrase/use synonyms? E.g., “Use the words from the text”.

• Always provide an example. The example should be a good illustration of what responses are expected. For example, if the instructions require test-takers to provide 1 to 3-word responses, the example cannot be a 4-word response – this will confuse test-takers.

• Limit the required length of responses. One or two words for lower-proficiency levels and up to 5 words for higher-proficiency levels are recommended. This will help with producing an exhaustive answer key.

• Answering correctly should not depend on accurate grammar or spelling when grammar/spelling are not part of the construct targeted in items. All recognisable spelling and grammar that do not change the meaning of the response should be accepted.

• Formulate the questions in such a way as to limit the number of acceptable responses. The questions should be worded as precisely and concisely as possible. E.g.: “When did John arrive in Prague?” can be answered with “on 12^th December”, “on Monday”, “at 7pm”, “in the evening”. Better question: “At what time did John arrive in Prague?” Note that both “at 7pm” and “7pm” should be accepted as correct responses.

• Do not use close-ended (yes/no) questions. Using yes/no questions will turn a SAQ into a True/False item.

• Pre-test all SAQs with learners who are similar in their ability to prospective test-takers. Often times, what is perfectly clear to the person who wrote the items is not at all clear to people who take the test.

---

Suggestions for the item revision:

This item should be split into two individual SAQs:

Further reading

Miller, M. D., Linn, R. L., & Gronlund, N. E. (2009). Measurement and assessment in teaching (10^th ed.). Pearson Education Ltd. – pp.172-179 of this book contain some useful recommendations for constructing short-answer items, as well as a checklist for task review.