Multiple selection lists: “Check all that apply” Vs “forced choice”

The survey Academy Series: 6th Chapter

In any type of questionnaire, it is usually necessary to ask the respondent to choose several options in a set of available alternatives. A typical format for this kind of questions is “check-all-that-apply”; it is used when we have a list – generally a long one – of items from which we want to know all the answers that apply for each respondent. For example, from a list of n brands we would like to know which ones the respondent knows and which ones have been bought during the last 12 months. Similarly, from a list of n activities, we could ask which options the respondent practise at least once a week, the activities he/she has done at least once in his/her life, etc.

The “check-all-that-apply” questions are usually presented like this:

Q1 - Please, tick all the brands you have bought during the last 12 months:

  Brand 1

  Brand 2

 ....

 Brand n

In online questionnaires, the element that allows respondents to tick each item (known as “input type”) is normally a “checkbox”, which is a squared box that can be selected and unselected easily by clicking on it. Using this element to choose more than one option from a list is a standard of web communication.

An alternative way to ask the same question is to list each item and request the respondent to answer “yes” or “no” for each of them. This kind of question is known as “forced-choice format” because the respondent must indicate whether this applies or not. It is usually presented as following:

Q1 alternative - Please, for each of the following brands, indicate whether you have bought their products in the last 12 months:

In order to have a question with this format in an online survey it is normal to use pairs of “radio-buttons”; one for the option “yes” and the other for the option “no”. The “radio-buttons” have a circular shape and respondents can only select one element of the group. So, when one “radio-button” is picked, the rest become automatically unavailable (in this case, if we pick “yes” we won’t be able to pick “no”). This does not mean that the respondent will not be able to change the chosen answer.

In face-to-face surveys (CAPI) or phone surveys (CATI), the second alternative “forced-choice” is more used as it would be very difficult for the responder to hear a long list of items and remember all the options that apply in order to number them at the end. It is much easier to answer “yes” or “no” as the respondent listens to each item, before moving on to assess the next one. However, on paper or online surveys where the respondent can see all the items written out, this memory problem does not exist. That is why a lot of surveys use the “check-all-that-apply” format.

Often, both formats are used as if they were equivalent or interchangeable. Many researchers, when they migrate surveys from paper to CAPI/CATI or from CAPI/CATI to online, they only transform questions from one format to the other, without worrying about the consequences that this change can have in the results.

An issue with the “check-all-that-apply” format is that if the respondents do not make enough effort to answer properly the question (reading and evaluating conveniently each of the given options), they could end up choosing less items than the ones that really apply (known by “weak satisficing”). For example, a respondent selects 3 options from a list of 10 and stops answering the question, thinking that he/she has done his/her duty and that it is not necessary to keep making efforts in reading and evaluating all of the other options.

On the other hand, it has been proven that people have a tendency to answer yes (“acquiescence” or “yes-saying”), so when we ask a respondent to indicate “yes” or “no” in a “forced-choice” format, we can cause respondents to choose “yes” more often than what in fact corresponds to reality.

To check whether those phenomena occur in practice, Smyth et al (2006) compared 16 experiments in two online and one on paper survey (2002-2003) in the US. After analyzing the results, they found that the 2 formats do not act in an equivalent way: the respondents choose more items with the “yes/no” format, rather than when using a “check-all-that-apply” format. Smyth et al (2006) shows that the problem comes from the “satisficing” experienced in the “check-all-that-apply” format, and not from the tendency of answer “yes” in the format “yes/no”. Therefore, they conclude that the format “forced-choice” is preferable, or in other words, it gets results that are closer to reality.

Since both the tendency of saying “yes” and the “weak satisficing” can vary between countries, we in Netquest – within the scope of the of our Research Lab – have carried out experiments in different countries of the Ibero-America region: Spain, Mexico and Colombia. In table 1, we can observe a list of 7 items asking about political activities (e.g.: whether the respondent has ever signed a petition to a government, whether the respondent has ever contact a politician...). Here we can see an average of how many times the respondents said yes with the format “yes/no” and with the format “check-all-that-apply”, in each of the 3 countries.

Note: CATA = check-all-that-apply

The results are similar to those obtained by Smyth et al (2006): there are more political actions reported when using format “yes/no” than when using the “check-all-that-apply” format.

The check-all-that-apply format leads to less items selected than the yes/no format: but which result is closer to reality? To know which of these 2 formats is preferable, we have performed a test of external validity, considering the correlations between the total number of items that a respondent affirms to have done and a variable with which we theoretically should find a high correlation; in this case, we use a question about the interest that the respondent has towards politics (B1- To what extend would you say you are interested in politics? A lot, quite a lot, not much, not at all). Supposedly, the greater interest in politics, the higher the number of activities related to politics should be. Thus, the format that gets a higher correlation is the one we should consider as preferable.

The correlations are similar in Colombia, but in Mexico and Spain are higher in the “yes/no” format”, which means that this format is more preferable than the “check-all-that-apply” one. Therefore, the experiments carried out in the Ibero-America region agree with the ones found by Smyth et al (2006) in USA.

In conclusion, in different countries seem that respondents do not make the maximum effort to answer properly when the “check-all-that-apply” format is used, as they choose less items than the ones that really apply. Clearly, both formats are not equivalent and so they are not interchangeable.

If you would like to read more about collecting data online, you can read our free ebook by clicking the banner below.

 Bibliographic references

Jolene D. Smyth, Don A. Dillman, Leah Melani Christian, and Michael J. Stern (2006). Comparing Check-All and Forced-Choice Question Formats in Web Surveys. Public Opinion Quarterly (Spring 2006) 70(1): 66-77 doi:10.1093/poq/nfj007