Short-Answer Questions

Short-answer questions (SAQ) are routinely used to assess reading and listening comprehension. The questions can target different types of constructs: comprehension of gist/main idea, supporting ideas, specific detail, inferencing, deducing meaning of unknown vocabulary from context.

The example is cited from Hughes & Hughes, 2020, p.83

For example, in these SAQs, question 1 targets understanding referencing (textual cohesion); question 2 – specific detail; question 3 – textual inference.

One advantage of these items is that they eliminate the possibility of guessing – the problem with all selected-response task types. SAQs are also thought to probe deeper into test-taker’s ability. SAQs are also easier to produce compared to selected-response task types, especially multiple-choice items.

On the minus side, marking SAQs might be difficult because there are normally many ways to express the same idea. That is why the answer key for SAQ tasks should, ideally, contain all possible response permutations. However, even most careful of item writers will not be able to anticipate all acceptable alternative answers. Therefore, SAQs should always be carefully trialled to reveal acceptable but unanticipated responses. Marking SAQs should also involve careful consideration of all responses, even if they are not included in the answer key. That is why large-scale tests often avoid the SAQ format. In such tests, responses are normally marked by clerical markers (people who mechanically mark against the key) or machines. 

Another problem with SAQs – similar to all constructed-response tasks – is that they might introduce construct-irrelevant variance. This is because SAQs require written responses – answering correctly, at least in part, depends on the test-takers’ writing ability. When SAQs are used to assess, for example, listening or reading comprehension, writing ability is irrelevant. 

Recommendations for writing SAQ items

  • Make it clear in the instructions what is expected of the test-takers:
    •  Must they respond with a number, a word, a phrase, or a complete sentence? E.g., “Answer with a word or a short phrase”.
    • What is the maximum number of words the response can contain?  E.g., “Use a maximum of three words”.
    • Should the words come from the text or can test-takers paraphrase/use synonyms? E.g., “Use the words from the text”.
  • Always provide an example. The example should be a good illustration of what responses are expected. For example, if the instructions require test-takers to provide 1 to 3-word responses, the example cannot be a 4-word response – this will confuse test-takers.
  • Limit the required length of responses. One or two words for lower-proficiency levels and up to 5 words for higher-proficiency levels are recommended. This will help with producing an exhaustive answer key.
  • Answering correctly should not depend on accurate grammar or spelling when grammar/spelling are not part of the construct targeted in items. All recognisable spelling and grammar that do not change the meaning of the response should be accepted.
  • Formulate the questions in such a way as to limit the number of acceptable responses. The questions should be worded as precisely and concisely as possible. E.g.: “When did John arrive in Prague?” can be answered with “on 12th December”, “on Monday”, “at 7pm”, “in the evening”. Better question: “At what time did John arrive in Prague?” Note that both “at 7pm” and “7pm” should be accepted as correct responses.
  • Do not use close-ended (yes/no) questions. Using yes/no questions will turn a SAQ into a True/False item.
  • Pre-test all SAQs with learners who are similar in their ability to prospective test-takers. Often times, what is perfectly clear to the person who wrote the items is not at all clear to people who take the test.

-The instructions do not provide any guidance in answering the questions;

-There is no example item;

-Some questions can be answered using prior knowledge, while what is being tested is reading comprehension, so background knowledge is construct-irrelevant and also gives unfair advantage to those students who have it;

-It is not clear if the answers should come verbatim from the text or any acceptable variations in wording can be used.

There are two questions in one item. Both questions are very broad and will elicit a wide range of correct responses. It will be very difficult to create an answer key for this item

Suggestions for the item revision:

This item should be split into two individual SAQs:

Question 1 is much more concrete compared to the original one. It can be answered with one word.

Question 2 uses a number of item-writing techniques to limit the range of acceptable responses. Firstly, the question is concrete – it asks for “three things that beavers do”. Secondly, it provides an example of what response format is expected (a verbal phrase). Thirdly, it indicates the response length and states that the words should come from the text. It will be much easier to create the answer key for this item, compared to the original one.

Further reading

Miller, M. D., Linn, R. L., & Gronlund, N. E. (2009). Measurement and assessment in teaching (10th ed.). Pearson Education Ltd. – pp.172-179 of this book contain some useful recommendations for constructing short-answer items, as well as a checklist for task review.