Electronic survey mode has become a more common tool of research than it used to be previously. This is strongly associated with the overall digitization of modern society. However, the evidence on the possible mode effect on study results has been scarce. Therefore, the aim of this study is to investigate the comparability of findings on health and behaviours using a paper-versus-electronic mode of survey with randomization design among schoolchildren.
A randomized study was conducted using a mandatory questionnaire on international Health Behaviour in School-aged Children (HBSC) study in Lithuania, enrolling 531 schoolchildren aged 11–15 years. The questionnaire included health and social topics about physical activity, risk behaviours, self-reported health and symptoms, life satisfaction, bullying, fighting, family and school environment, peer relationships, electronic media communication, sociodemographic indicators, etc. The schoolchildren within classes were randomly selected for electronic or paper mode.
It was found that by study mode differences are inconsistent and in the majority of cases do not exceed 5%-point difference between the modes. The only significant difference was that in the paper survey the participants reported more exercise than in the electronic survey (OR=8.08, P<.001). Other trends were nonsignificant and did not show a consistent pattern – in certain behaviours the paper mode was related to healthier choices, while in others - the electronic.
The use of electronic questionnaires in surveys of schoolchildren may provide findings that are comparable with concurrent or previously conducted paper surveys.
Information and communication technology has become an ever more demanded working tool to enhance the management, efficiency, and quality of surveys on health and social phenomena. There are several kinds of electronic questionnaires – online access, mobile device administered by the researcher, or computer/device handled by respondent. The responses can be collected by participant, researcher or a proxy (if a participant is minor). Overall digitization of social life and communication suggests ever-increasing pressure to conduct digital surveys and, therefore, it is essential to assess how reliable and valid the digital methods are and, if they replace paper-and-pencil method, are the findings comparable?
The online mode reduces the study costs by saving on the costs of paper and printing as well as from transportation (1). Besides, it ensures quick data with virtually no errors and suggests fewer no-response answers (2). Another important point is that these devices permit automatic checking of responses and complex skip patterns. However, in the digital survey mode, it is essential to ensure who is filling in the questionnaire, which is not always feasible.
The literature on the effects of digital-based and computer-adaptive testing suggests that digitization of standardized tests is a precise and appropriate research mode both from a scientific and logistic point of view (3, 4). Nonetheless, some researchers propose that the reliability of data obtained by the web-based approach should be determined (5). There is also a potential for selection bias, where a particular type of participant may be more prone to a particular survey mode (e.g. preference for digital mode among younger, more affluent or educated people). Moreover, in online mode, the participants can be unknown, not meet eligibility criteria or make double entries. Therefore, due to the potential for selection bias a randomized controlled design could be regarded as the main choice in studies on potential mode effects.
Even though many studies analysing the issue of mode effect on study results use randomization, quite a lot of them address the issue of response rate foremost, while content-specific comparison receives less attention. Also, such studies rarely investigate younger groups and the majority of them do not use randomization. For example, in the international Health Behaviour in School-aged Children (HBSC) study some countries use mixed mode design for more than a decade, e.g. Belgium (6), but they usually do not randomize the schools or children, leaving the choice of mode up to the school’s or child’s preference – which may be a subject to bias.
Thus, even though research on the validity and reliability of digital versus paper mode is quite extensive, such assessment in adolescents is rarely addressed. Moreover, the randomized approach in the research of mode effect is not always applicable, leaving the findings with a potential for self-selection or school-specific bias. In addition, the health perceptions and behaviours have also been under-investigated from this perspective. Therefore, the objective of our study is to compare the findings from paper and electronic mode using a randomized controlled design among schoolchildren.
2.1 Study Process and Sample
The randomized controlled study was conducted in May 2017 at five secondary schools in Lithuania. All study subjects were informed about the details of the study and that the return of the filled questionnaire will be treated as the informed consent. The anonymity of study participants and confidentiality of the data was ensured. The study was conducted as a pilot project for an oncoming 2018 Health Behaviour in School-aged Children study in Lithuania. The schools were randomly selected from the national schools’ list, by choosing the first five schools who agreed to participate in the study. The schools were from the second-largest city, other cities, and one town. In every school, the questionnaire was administered to 5th, 7th, and 9th grades (predominant age of children 11, 13 and 15 years, respectively). Then, the randomization was applied for every class in the school, with one-half of students filling the questionnaire in paper mode and the other half in electronic mode. Every class was randomized to define which half of the students’ list filled the online and which the paper version of the questionnaire.
Questionnaires (both electronic and paper mode) were administered in school classrooms by trained researchers who complied with written instructions. The electronic version of the questionnaire was uploaded to Google Forms, which was available only to the researchers. During the survey, the researchers shared the web link to study participants. The online questionnaire was filled in on desktop or tablet computers. The places for survey were usually classrooms, computer rooms or libraries. In some cases, the survey of paper and online mode was conducted simultaneously in the same room. Every researcher wrote the notes about the procedure of survey.
The tool for the study was based on the then-current version of the standardized international HBSC research protocol (7). The HBSC questionnaire covers a wide range of health and social topics about schoolchildren’s physical activity, risk behaviours, self-reported health and symptoms, life satisfaction, bullying, fighting, family, school environment, peer relationships, electronic media communication, sociodemographic indicators, etc. Only the mandatory items were included. The sequence, formulation, and overall visualization of items did not differ by mode.
Some items of the questionnaire were used from particular scales or subscales:
HBSC symptom checklist, 8 items (7),
Family Affluence Scale, 6 items (8),
Multidimensional Scale of Perceived Social Support: Family, 4 items (9),
Multidimensional Scale of Perceived Social Support: Friends, 4 items (adapted from (9)),
Teacher and Classmate Support Scale: Classmates, 3 items (adapted from (10)),
Teacher and Classmate Support Scale: Teachers, 3 items (adapted from (10)),
Online contact with friends and others, 4 items (11),
Preference for online communication, 3 items (12),
Social media addiction, 9 items (13).
2.3 Data Analysis
Data was processed using MS Excel 2010 and analysed using IBM SPSS Statistics, version 20. The descriptive analysis included the calculation of the prevalence of different health behaviours (%). The items were dichotomized based on the cut-offs used in the 2014 Health Behaviour in School-aged Children study report (14). The main purpose of the analysis was to estimate whether various health-related items are similarly distributed among study groups in schoolchildren that filled in the questionnaire in paper-versus-electronic mode. For this, the percentage point differences were calculated, and logistic regression was used with the calculation of certain behaviours’ risk when comparing the modes. The differences between the modes were estimated using percentage point difference and odds ratios with the reference group being electronic mode (OR=1.00). Given that despite randomization there were some imbalances between the study groups by gender, grade, and school, these indicators were adjusted for in the multivariate logistic regression model.
Due to multiple comparisons of different indicators, the Bonferroni correction was used: in total, 78 variables were compared, so the conventional significance level of P<0.05 was decreased to P<0.001 (0.05/78=0.00064). The P-values between 0.001 and 0.05 were reported as trends.
The study sample comprised 531 schoolchildren – 261 filled the electronic questionnaire and 270 the paper version. The overall response rate was 83.0% with higher rates among girls and elder schoolchildren. A detailed comparison of study groups by gender, grade, and school are presented in Table 1. Regardless of randomization, there were some differences observed between study groups and since they were definitely random (by design of the study) their statistical significance was not calculated.
The main characteristics of study sample.
|Characteristic||Electronic mode||Paper mode||n||Response rate|
|#1 (large city)||47.3||52.7||74||89.2|
|#2 (large city)||48.7||51.3||224||94.5|
In this study, the internal consistency of scales and subscales was acceptable and the difference between the modes was not more than .07 points – with no consistent superiority of either mode (Table 2).
Internal consistency of study scales and subscales by survey mode.
|Scale||Number of items||Internal consistency (α)|
|Electronic mode||Paper mode|
|HBSC symptom checklist||8||.78||.79|
|Family Affluence Scale||6||.52||.58|
|Multidimensional Scale of Perceived Social Support: Family||4||.76||.69|
|Multidimensional Scale of Perceived Social Support: Friends||4||.90||.85|
|Teacher and Classmate Support Scale: Classmates||3||.77||.70|
|Teacher and Classmate Support Scale: Teachers||3||.75||.74|
|Online contact with friends and others||4||.54||.54|
|Preference for online communication||3||.84||.81|
|Social media addiction||9||.75||.76|
3.1 Health Behaviours
In the field of health behaviours (Table 3), the largest difference depending on survey mode was observed in extensive physical activity – in paper mode, the schoolchildren more frequently reported daily exercise until getting out of breath or sweating (OR=8.08, P<.001). Other indicators had no differences except the trends that students in paper mode more frequently reported, such as having a regular breakfast on weekends (OR=1.93, P=.009). Almost all aspects of health behaviours differed between the survey modes by no more than 5% points.
Health behaviours of schoolchildren by survey mode.
|Characteristic||Prevalence, %||% difference||OR||P|
|Having breakfast during the weekdays||Every day||58.8||62.7||3.9||1.18||.366|
|Having breakfast during the weekends||Every day||79.6||87.7||8.1||1.93||.009|
|Having breakfast with parents||Every day||41.0||40.7||-.3||1.00||.982|
|Having dinner with parents||Every day||47.1||45.6||-1.5||.96||.816|
|Eating fruits||Every day||41.8||38.5||-3.3||.87||.446|
|Eating vegetables||Every day||32.6||34.2||1.6||1.07||.707|
|Eating sweets||Every day||16.1||13.8||-2.3||.83||.453|
|Drinking soft drinks||Every day||5.4||6.3||.9||1.23||.593|
|Drinking energy drinks||Every day||2.3||.4||-1.9||.16||.097|
|Health and well-being|
|Subjective health assessment||Good||88.5||91.8||3.3||1.59||.132|
|Life satisfaction||6–10 (10 pts scale)||87.7||85.8||-1.9||.84||.510|
|Irritability or bad temper||Rarely||72.0||76.5||4.5||1.39||.115|
|Difficulties in getting to sleep||Rarely||79.7||83.2||3.5||1.33||.214|
|Brushing the teeth||More than once a day||61.3||62.8||1.5||1.05||.809|
|Body image||A bit too thin||11.9||15.0||3.1||.71||.200|
|A bit too fat||29.1||30.7||1.6||.91||.654|
|About the right size||59.0||54.3||-4.7||1.00||-|
|Physical activity at least 60 minutes per day (last week)||7 days||18.9||20.6||1.7||1.16||.518|
|Exercise in free time until getting out of breath or sweating||Every day||3.1||19.7||16.6||8.08||<.001|
|Cigarette smoking (lifetime)||Never||73.2||77.8||4.6||1.37||.187|
|Cigarette smoking (last month)||Never||88.1||92.3||4.2||1.75||.097|
|Alcohol drinking (lifetime)||Never||62.8||68.4||5.6||1.37||.129|
|Alcohol drinking (last month)||Never||88.9||88.0||-.9||.89||.706|
|Cannabis taking (lifetime)||Never||94.3||97.0||2.7||2.22||.101|
|Cannabis taking (last month)||Never||98.1||99.2||1.1||2.13||.376|
3.2 Social Behaviours and School
The selected indicators of social behaviours under study showed slightly bigger differences than health behaviours, though they were inconsistent and nonsignificant (Table 4). Here the trend in paper mode was that the children were more likely to report having friends to share joys and sorrows, but also more cyber-bullying and more treatment-needed injuries (.001<P<.05).
School behaviours of schoolchildren by survey mode.
|Characteristic||Prevalence, %||% difference||OR||P|
|Counting on friends||Agree||75.1||78.3||3.2||1.18||.440|
|Having friends to share joys and sorrows||Agree||78.2||87.7||9.5||1.98||.005|
|Being able to talk about problems with friends||Agree||70.5||76.4||5.9||1.35||.138|
|Enjoy being together with students||Agree||59.8||54.1||-5.7||.81||.243|
|Students in class are kind and helpful||Agree||53.3||52.8||-.5||1.00||.982|
|Students accepting one as he/she is||Agree||67.4||61.8||-5.6||.80||.231|
|Teachers accepting one as he/she is||Agree||75.1||78.1||3.0||1.23||.348|
|Feeling a lot of trust in teachers||Agree||65.1||67.5||2.4||1.16||.462|
|Feeling about school||I like it a lot||81.1||82.9||1.8||1.14||.586|
|Pressure by schoolwork||Some or a lot||72.2||72.7||.5||.99||.947|
|Taking part in bullying another student||Yes||44.2||42.9||-1.3||.97||.889|
|at school, last two months|
|Being bullied at school, last two months||Yes||50.2||46.5||-3.7||.86||.402|
|Taking part in cyber-bullying, last two months||Yes||20.3||18.7||-1.6||.92||.708|
|Being cyber-bullied, last two months||Yes||14.1||22.3||8.2||1.82||.011|
|Having a physical fight, last year||Yes||31.7||29.4||-2.3||.92||.688|
|Being injured with treatment needed, last year||Yes||48.8||57.6||8.8||1.45||.036|
3.3 Family Environment
The evaluation of schoolchildren’s family environment revealed that there were almost no differences depending on survey mode (Table 5). The children in paper mode reported slightly better family communication and support, but this was nonsignificant (P=0.068). All other indicators did not reach a 5%-point difference and, regarding items on family affluence, the differences by paper mode were also minor.
Family-related perceptions of schoolchildren by survey mode.
|Characteristic||Prevalence, %||% difference||OR||P|
|Ease to talk about things that really bother: to father||Easy||63.2||69.7||6.5||1.43||.068|
|Ease to talk about things that really bother: to stepfather||Easy||11.5||15.5||4.0||1.48||.221|
|Ease to talk about things that really bother: to mother||Easy||79.7||82.0||2.3||1.15||.526|
|Ease to talk about things that really bother: to stepmother||Easy||9.6||10.1||.5||1.08||.848|
|Family really tries helping||Agree||92.0||93.7||1.7||1.29||.461|
|Getting emotional help and support from family||Agree||83.9||88.0||4.1||1.43||.164|
|Being able to talk about problems with family||Agree||69.3||71.2||1.9||1.13||.537|
|Family is willing to help in making decisions||Agree||86.6||90.3||3.7||1.45||.183|
|Dishwasher at home||Yes||62.5||59.7||-2.8||.87||.496|
|Bathrooms at home||One or more||97.7||97.8||.1||1.03||.958|
|Family car||One or more||95.0||93.7||-1.3||.78||.525|
|Computers at home||One or more||97.3||97.7||.4||1.15||.806|
|Family travel abroad for vacation, last year||Once or more||86.5||81.6||-4.9||.68||.129|
3.4 Electronic Media Communication
The survey included three main aspects of electronic communication – online contact with friends, preference for online communication, and social media addiction (Table 6). Here there were two trends: in electronic mode, children reported using social media more as a way to escape from negative feelings and having conflicts with family members because of social media use (.001<P<.05). All other items were indifferent by mode and rarely exceeded a 5%-point difference.
Electronic media communication of schoolchildren by survey mode.
|Characteristic||Prevalence, %||% difference||OR||P|
|Online contact with friends and others|
|Close friend(s)||Every day||66.7||63.3||-3.4||.81||.277|
|Friends from a larger friend group||Every day||37.9||40.6||2.7||.12||.535|
|Friends that you got to know through the internet but didn’t know before||Every day||11.5||14.7||3.2||1.32||.302|
|Other people than friends||Every day||44.8||47.7||2.9||1.11||.563|
|Preference for online communication|
|On the internet, I talk more easily about secrets than in a face-to-face encounter||Agree||27.3||25.7||-1.6||.93||.725|
|On the internet, I talk more easily about my inner feelings than in a face-to-face encounter||Agree||26.8||22.4||-4.4||.78||.238|
|On the internet, I talk more easily about my concerns than in a face-to-face encounter||Agree||27.6||20.1||-7.5||.67||.054|
|Social media addiction|
|Regularly felt dissatisfied because you wanted to spend more time on social media||Yes||16.9||18.0||1.1||1.03||.912|
|Often felt bad when you could not use social media||Yes||27.2||25.7||-1.5||.88||.528|
|Tried to spend less time on social media, but failed||Yes||28.7||25.3||-3.4||.81||.284|
|Regularly neglected other activities (e.g. hobbies, sport) because you wanted to use social media||Yes||13.8||12.4||-1.4||.90||.700|
|Regularly had arguments with others because of your social media use||Yes||14.9||12.0||-2.9||.76||.290|
|Regularly lied to your parents or friends about the amount of time you spend on social media||Yes||17.2||16.1||-1.1||.89||.628|
|Often used social media to escape from negative feelings||Yes||30.3||22.5||-7.8||.64||.030|
|Had serious conflict with your parents, brother(s) or sister(s) because of your social media use||Yes||21.8||13.5||-8.3||.49||.004|
|Regularly found that you can’t think of anything else but the moment that you will be able to use social media again||Yes||40.6||39.9||-.7||.93||.704|
3.5 Procedure-Specific Findings
In this study, the environment and circumstances of the survey were also documented in order to depict the procedure-specific findings. So, during the survey and especially in online mode some participants were able to see the adjacent participants’ responses, thus infringing the privacy of other responders. In addition, the teachers sometimes refused to leave the classroom even when asked.
It was also observed that some schoolchildren were not content with the assigned mode of the survey. However, this was not mode-specific: some adolescents expressed the wish to move from paper mode to electronic, while others vice versa. The former ones were keener to choose the electronic device (computer or tablet) instead of paper, while the latter preferred more privacy. Of note, some students were concerned about the split of the class into different modes as if treated unequally.
Electronic research mode is very convenient for large-scale studies. In order to address the possible effect of survey mode on its results, we conducted a randomized study to eliminate the potential for selection bias within the study sample. This is the main strength of our study since the previous research has quite frequently neglected the issue of self-selection bias that arises in non-randomized studies. This is especially relevant across different social conditions such as schools, where some of them may have better resources to prefer online mode, either through better financing or through higher quality of educational services. So, by the study design, our study avoided the possible self-selection bias or school-specific differences by randomizing the schoolchildren within classes. We also adjusted the calculations by main sociodemographic indicators that could affect the differences. Besides, the inclusion of different size schools from bigger and smaller urban areas also increased the diversity of schoolchildren.
However, when discussing our study weaknesses, we had a limited sample size, which potentially led to an underestimating of the statistical significance of differences, especially when controlling for multiplicity. Nevertheless, we found that in the majority of cases the differences between the modes were small and did not exceed a 5% point. For such five percent differences to detect as statistically significant at P<0.05 level, we would have needed the sample from 431 to 1,559 per arm – and this without the multiplicity correction that was applied in our study (for P<0.01 640 and 2,315 participants would be needed (15), respectively, and for P<0.001 even more). Compared to previous studies, our sample size was rather medium, and we had no intention to find minor differences as statistically significant. After all, the fact that absolute differences between the modes were inconsistent (i.e. not showing better health behaviours in either mode) suggests the likely absence of substantial differences.
Another limitation of our study was the lack of replicability since our study participants had an opportunity to fill in the questionnaire only in one of the modes. Therefore, the assessment of the consistency of results within individuals was not possible. This occurred because we raised no question regarding the particular subject’s replicability of responses – rather, we had an interest in comparing the population (i.e. study sample) estimates.
Overall, our study findings revealed that differences by study mode are virtually absent and in the majority of cases do not exceed five percent difference between the modes. These findings do not have many studies to compare with since the schoolchildren’s health behaviours have rarely been addressed in previous research on survey modes. The study on the HBSC sample was previously reported by Vereecken and Maes (6) in Belgium. Their findings showed some differences by mode, but our results did not support them. We did not detect those differences not only due to a smaller sample size (i.e. lower power of the study) but rather due to the absence of absolute difference.
That same study (6) noted that for several questions about feelings and affective states more socially desirable responses were found in paper format. However, in our case, this was not observed. Even though we saw some larger differences when assessing social support measures, this was also inconsistent. The fact that the adolescents provide equivalent responses in paper and computer formats was also found elsewhere (16).
It was found that adolescents were more likely to report substance use and less desirable aspects of psychological well-being using a digital format (17). However, we found that subjective health was reported as slightly better in paper mode (like Smith et al. (18) in the military sample), while higher life satisfaction was reported as better in electronic mode, which does not suggest the consistency of mode effect.
The issue of different responses by survey mode has been addressed with other samples as well. For example, patients after knee surgery reported similar levels of daily functioning, quality of life, pain intensity as well as symptoms (19). Similarly, for college students in education and psychology, the survey mode did not have strong differential effects on data quality regarding the learning environment and perceptions (20). One study on military participants found some differences in health behaviours by mode, though like our study they did not exceed five percent (18).
We also compared the internal consistency of scales. Previous studies demonstrated that electronic mode is likely to show higher internal consistency compared to the paper, with differences by up to .30 (16), however, in our study there was no superiority of either mode (differences did not exceed .07). Some other studies also showed no relevant differences in psychometric properties by mode (21).
Previous research suggests that young people are keener to choose the electronic than the paper version (22), while the studies of other samples are rather ambivalent: for instance, the study on people who take supplements and vitamins found the electronic version as more acceptable (1, 23), while HIV patients preferred the paper version (24). Interestingly, our procedure-specific findings also indicate ambiguity, since some children preferred to move to electronic, while others to paper mode. This was rather unexpected due to the hypothesized preference of digital natives toward electronic mode. It could be explained by the fact that maintaining privacy was an issue during this study, especially when filling in electronic questionnaires on desktop computers: schoolchildren were able to see the answers of adjacent classmates, which could have made them feel insecure. Additionally, in some classes, the teachers refused to leave the room, which may interfere with the confidentiality perceptions of children and the sincerity of their responses. The fact that some schoolchildren complained about having different survey modes across the class implies that, in the future, a class as an entity should preferably be investigated using the same mode.
Regarding the cost-effectiveness of the shift from paper mode to an electronic mode in our study, the main difference was related to expenses for paper and printing the questionnaire as well as typing in the responses from paper to database. In addition, the probability of data typing errors in case of the electronic mode is virtually zero. Nevertheless, the shift toward electronic mode should be approached carefully: if the survey is going to be uploaded online with a non-restricted access, the study participants cannot be controlled. This may further result in a situation where some subjects submit several questionnaires, or they do not meet the eligibility criteria for age or other relevant characteristics. It should be emphasized that in online surveys the basic concern is related to the problem of who really fills in the questionnaire and if they meet the eligibility criteria of the study. This should be controlled whenever possible.
Summarizing our study, it can be stated that the comparison of electronic and paper mode in the research of health and social behaviours among schoolchildren revealed no consistent differences between the modes. There were some items or questions that had larger differences between the survey modes, however, they did not have a trend to be healthier or more socially desirable in one particular mode. This suggests that, in the future, the use of electronic questionnaires in surveys of schoolchildren may provide findings that are comparable with concurrent or previously conducted paper surveys. However, this does not relieve the concerns related to electronic surveys where the study participants are not controlled in terms of eligibility criteria. Thus, when the electronic survey responders are unknown, this still threatens the validity of study findings.
We thank all study participants and schools who agreed to enter this study. We also would like to express gratitude to reviewers whose comments and suggestions improved the manuscript.
Conflicts of Interest
The authors declare that no conflicts of interest exist.
The study was financed by internal institutional sources.
The study protocol was approved by the Ethics Committee at the Lithuanian University of Health Sciences, reference number BEC-SP(B)-129, and it conforms to the provisions of the Declaration of Helsinki in 1995 (as revised in Edinburgh in 2000).
Inchley J, Currie D, Cosma A, editors. Health Behaviour in School-aged Children (HBSC) study protocol: background, methodology and mandatory items for the 2017/18 survey. St Andrews: CAHRU, 2018.
Mascheroni G, Ólafsson K. Net children go mobile: cross-national comparisons: report D3.3. Milano: Educatt, 2014.
Inchley J, Currie D, Young T, Samdal O, Torsheim T, Augustson L, et al. Growing up unequal: gender and socioeconomic differences in young people’s health and well-being. World Health Organisation, Health Policy for Children and Adolescents, 2016;7. Accessed March 19th, 2018 at: http://www.euro.who.int/__data/assets/pdf_file/0003/303438/HSBC-No.7-Growing-up-unequal-Full-Report.pdf
Chow S, Shao J, Wang H. Sample size calculations in clinical research. 2nd ed. Chapman & Hall/CRC Biostatistics Series, 2008:89.