Measurement of health and social behaviors in schoolchildren: Randomized study comparing paper versus electronic mode

Information and communication technology has become an ever more demanded working tool to enhance the management, efficiency, and quality of surveys on health and social phenomena. There are several kinds of electronic questionnaires – online access, mobile device administered by the researcher, or computer/device handled by respondent. The responses can be collected by participant, researcher or a proxy (if a participant is minor). Overall digitization of social life and communication suggests ever-increasing pressure to conduct digital surveys and, therefore, it is essential to assess how reliable and valid the digital methods are and, if they replace paper-and-pencil method, are the findings comparable?

The online mode reduces the study costs by saving on the costs of paper and printing as well as from transportation (1). Besides, it ensures quick data with virtually no errors and suggests fewer no-response answers (2). Another important point is that these devices permit automatic checking of responses and complex skip patterns. However, in the digital survey mode, it is essential to ensure who is filling in the questionnaire, which is not always feasible.

The literature on the effects of digital-based and computer-adaptive testing suggests that digitization of standardized tests is a precise and appropriate research mode both from a scientific and logistic point of view (3, 4). Nonetheless, some researchers propose that the reliability of data obtained by the web-based approach should be determined (5). There is also a potential for selection bias, where a particular type of participant may be more prone to a particular survey mode (e.g. preference for digital mode among younger, more affluent or educated people). Moreover, in online mode, the participants can be unknown, not meet eligibility criteria or make double entries. Therefore, due to the potential for selection bias a randomized controlled design could be regarded as the main choice in studies on potential mode effects.

Even though many studies analysing the issue of mode effect on study results use randomization, quite a lot of them address the issue of response rate foremost, while content-specific comparison receives less attention. Also, such studies rarely investigate younger groups and the majority of them do not use randomization. For example, in the international Health Behaviour in School-aged Children (HBSC) study some countries use mixed mode design for more than a decade, e.g. Belgium (6), but they usually do not randomize the schools or children, leaving the choice of mode up to the school’s or child’s preference – which may be a subject to bias.

Thus, even though research on the validity and reliability of digital versus paper mode is quite extensive, such assessment in adolescents is rarely addressed. Moreover, the randomized approach in the research of mode effect is not always applicable, leaving the findings with a potential for self-selection or school-specific bias. In addition, the health perceptions and behaviours have also been under-investigated from this perspective. Therefore, the objective of our study is to compare the findings from paper and electronic mode using a randomized controlled design among schoolchildren.

The study sample comprised 531 schoolchildren – 261 filled the electronic questionnaire and 270 the paper version. The overall response rate was 83.0% with higher rates among girls and elder schoolchildren. A detailed comparison of study groups by gender, grade, and school are presented in Table 1. Regardless of randomization, there were some differences observed between study groups and since they were definitely random (by design of the study) their statistical significance was not calculated.

In this study, the internal consistency of scales and subscales was acceptable and the difference between the modes was not more than .07 points – with no consistent superiority of either mode (Table 2).

Electronic research mode is very convenient for large-scale studies. In order to address the possible effect of survey mode on its results, we conducted a randomized study to eliminate the potential for selection bias within the study sample. This is the main strength of our study since the previous research has quite frequently neglected the issue of self-selection bias that arises in non-randomized studies. This is especially relevant across different social conditions such as schools, where some of them may have better resources to prefer online mode, either through better financing or through higher quality of educational services. So, by the study design, our study avoided the possible self-selection bias or school-specific differences by randomizing the schoolchildren within classes. We also adjusted the calculations by main sociodemographic indicators that could affect the differences. Besides, the inclusion of different size schools from bigger and smaller urban areas also increased the diversity of schoolchildren.

However, when discussing our study weaknesses, we had a limited sample size, which potentially led to an underestimating of the statistical significance of differences, especially when controlling for multiplicity. Nevertheless, we found that in the majority of cases the differences between the modes were small and did not exceed a 5% point. For such five percent differences to detect as statistically significant at P<0.05 level, we would have needed the sample from 431 to 1,559 per arm – and this without the multiplicity correction that was applied in our study (for P<0.01 640 and 2,315 participants would be needed (15), respectively, and for P<0.001 even more). Compared to previous studies, our sample size was rather medium, and we had no intention to find minor differences as statistically significant. After all, the fact that absolute differences between the modes were inconsistent (i.e. not showing better health behaviours in either mode) suggests the likely absence of substantial differences.

Another limitation of our study was the lack of replicability since our study participants had an opportunity to fill in the questionnaire only in one of the modes. Therefore, the assessment of the consistency of results within individuals was not possible. This occurred because we raised no question regarding the particular subject’s replicability of responses – rather, we had an interest in comparing the population (i.e. study sample) estimates.

Overall, our study findings revealed that differences by study mode are virtually absent and in the majority of cases do not exceed five percent difference between the modes. These findings do not have many studies to compare with since the schoolchildren’s health behaviours have rarely been addressed in previous research on survey modes. The study on the HBSC sample was previously reported by Vereecken and Maes (6) in Belgium. Their findings showed some differences by mode, but our results did not support them. We did not detect those differences not only due to a smaller sample size (i.e. lower power of the study) but rather due to the absence of absolute difference.

That same study (6) noted that for several questions about feelings and affective states more socially desirable responses were found in paper format. However, in our case, this was not observed. Even though we saw some larger differences when assessing social support measures, this was also inconsistent. The fact that the adolescents provide equivalent responses in paper and computer formats was also found elsewhere (16).

It was found that adolescents were more likely to report substance use and less desirable aspects of psychological well-being using a digital format (17). However, we found that subjective health was reported as slightly better in paper mode (like Smith et al. (18) in the military sample), while higher life satisfaction was reported as better in electronic mode, which does not suggest the consistency of mode effect.

The issue of different responses by survey mode has been addressed with other samples as well. For example, patients after knee surgery reported similar levels of daily functioning, quality of life, pain intensity as well as symptoms (19). Similarly, for college students in education and psychology, the survey mode did not have strong differential effects on data quality regarding the learning environment and perceptions (20). One study on military participants found some differences in health behaviours by mode, though like our study they did not exceed five percent (18).

We also compared the internal consistency of scales. Previous studies demonstrated that electronic mode is likely to show higher internal consistency compared to the paper, with differences by up to .30 (16), however, in our study there was no superiority of either mode (differences did not exceed .07). Some other studies also showed no relevant differences in psychometric properties by mode (21).

Previous research suggests that young people are keener to choose the electronic than the paper version (22), while the studies of other samples are rather ambivalent: for instance, the study on people who take supplements and vitamins found the electronic version as more acceptable (1, 23), while HIV patients preferred the paper version (24). Interestingly, our procedure-specific findings also indicate ambiguity, since some children preferred to move to electronic, while others to paper mode. This was rather unexpected due to the hypothesized preference of digital natives toward electronic mode. It could be explained by the fact that maintaining privacy was an issue during this study, especially when filling in electronic questionnaires on desktop computers: schoolchildren were able to see the answers of adjacent classmates, which could have made them feel insecure. Additionally, in some classes, the teachers refused to leave the room, which may interfere with the confidentiality perceptions of children and the sincerity of their responses. The fact that some schoolchildren complained about having different survey modes across the class implies that, in the future, a class as an entity should preferably be investigated using the same mode.

Regarding the cost-effectiveness of the shift from paper mode to an electronic mode in our study, the main difference was related to expenses for paper and printing the questionnaire as well as typing in the responses from paper to database. In addition, the probability of data typing errors in case of the electronic mode is virtually zero. Nevertheless, the shift toward electronic mode should be approached carefully: if the survey is going to be uploaded online with a non-restricted access, the study participants cannot be controlled. This may further result in a situation where some subjects submit several questionnaires, or they do not meet the eligibility criteria for age or other relevant characteristics. It should be emphasized that in online surveys the basic concern is related to the problem of who really fills in the questionnaire and if they meet the eligibility criteria of the study. This should be controlled whenever possible.

Summarizing our study, it can be stated that the comparison of electronic and paper mode in the research of health and social behaviours among schoolchildren revealed no consistent differences between the modes. There were some items or questions that had larger differences between the survey modes, however, they did not have a trend to be healthier or more socially desirable in one particular mode. This suggests that, in the future, the use of electronic questionnaires in surveys of schoolchildren may provide findings that are comparable with concurrent or previously conducted paper surveys. However, this does not relieve the concerns related to electronic surveys where the study participants are not controlled in terms of eligibility criteria. Thus, when the electronic survey responders are unknown, this still threatens the validity of study findings.