The outcome-based iCAN! / theyCAN! feedback paradigm differentiates strong and weak learning outcomes, learner diversity, and the learning outcomes of each learner: A shift to metacognitive assessment

An outcome-based education is a sine qua non in today’s teaching and learning [1]. The medical education has gradually changed from a content-based education to an outcome-based education. The “European Core Curriculum: the Students’ Perspective” [2], and mainly “The Tuning Project (medicine): Learning Outcomes/ Competences for Undergraduate Medical Education in Europe” [3] could be considered as landmarks of this trend. The Tuning Project, developed by the thematic network of ninety-three European medical schools on Medical Education in Europe (MEDINE) and presented to the European Commission, is the teachers’ perspective. Based on these documents, the iCAN!, a self-administered questionnaire measuring the outcomes/ competences and the professionalism of the medical graduates, was developed [4]. By using the iCAN! on the first day in a medical school, the new students get informed of what is expected of them upon their graduation. By using it at any time during their studies, the students are informed on their progress and on the material that remains to be covered. And, by using it at graduation, the graduates know what has been finally gained and what remains uncovered. In order to measure the learning outcomes of each specific course of medical studies, a series of tools, iCAN!-Paediatrics [5], iCAN!-Pain [6], iCAN!-Normal-child [7], was developed.

Oncology is not a distinct core curriculum subject in any of the seven medical schools in Greece. Extracurricular seminars and a dense program of focused courses and national conferences complement undergraduate and postgraduate oncology training [8-9]. The “Hellenic Medical Student Oncology Meeting” is a focused program for which learning outcomes had been identified and transformed into the iCAN!-Oncology questionnaire for student self-assessment before and after training [10]. In addition, the teachers of the seminar complete the purposely designed theyCAN!-Oncology questionnaire in order to evaluate students’ progress.

The comparison between the students’ and the teachers’ perceptions before and after training on the same set of learning outcomes is the aim of this study. Can learning outcomes be transformed into useful tools revealing the strong and weak learning outcomes, the strong and weak learners and teachers, and the strong and weak learning outcomes of each learner or teacher? Can they report student self-assessment overestimation? Can they inform formative feedback and summative examinations? Can this paradigm support evidence-based teaching/learning SWOT policy?

Under the auspice of the Faculty of Medicine of the University of Crete, Greece, the Greek Oncology Research Group has been organizing for the last 15 years the extra-curricular seminar “Hellenic Medical Student Oncology Meeting” (hereinafter, meeting), aiming to a better undergraduate education in the principles of cancer biology, epidemiology, early detection, and care of patients with neoplasia. The organizing committee accepts the applications of medical students of the 4^th–6^th year of studies from all seven Greek medical schools, based on their curriculum vitae and a motivation letter.

Based on the European Society of Medical Oncology (ESMO) / American Society of Clinical Oncology (ASCO) global curriculum for training in oncology logbook [11], translated in Greek [12], twelve level-one learning outcomes (L1LOs), which were then expanded to 66 refined level-two learning outcomes (L2LOs), were adjusted to the 2013 meeting program. These L2LOs were transformed into the iCAN!-Oncology questionnaire for students (e.g., “I can correctly describe the process of carcinogenesis”) and the theyCAN!-Oncology questionnaire for teachers (e.g., “These students, in average, can correctly describe the process of carcinogenesis”). Each questionnaire was available online through Google Drive Form, and was completed anonymously by the students and the teachers respectively, before and after the five-day full-time meeting.

The responses were coded in the 0–100 scale: absolutely disagree 0; disagree 20; rather disagree 40; rather agree 60; agree 80; absolutely agree 100. Then, the question mean score (corresponding to one L2LO), the subscale mean score (corresponding to one L1LO), and the whole questionnaire mean score were calculated. All scores are given in the standard 0–100 scale, the easiest to grasp, interpret and remember without any need for further training (% is a universal scale). While a single score describes a point within the 0–100 scale, the term “percentage points” (pp) describes the difference between two single scores; and the term “improvement” describes the division of this difference by the baseline score multiplied by 100 [13].

If a score was lower than the pass/fail cut-point (50), it was interpreted as a student failure; if it was at least 50 but lower than 60 (50-59), it was borderline; 60-69 moderate; 70-79 good; 80-89 very good; 90-100 excellent. And, following the quality of life practice [14-16], a difference greater than a minimal important difference of 5 pp was considered educationally significant (educationally minimal important difference, EMID).

Seventy-two students answered to the iCAN!-Oncology questionnaire before the meeting (95% of the 76 who had been chosen to attend the meeting) and 33 after (46% of the 72 who finally attended). Eighteen teachers answered to the theyCAN!-Oncology questionnaire before and 7 after the meeting (38% and 15% respectively of the 47 teachers). The participants’ distribution by school (from all seven country’s medical schools), gender (from both genders), students’ year of study (5^th and 6^th), and teachers’ age (30-69) are given in Additional material 1.

The mean scores and the added values are presented in Table 1 (see legend for details). As it can be seen in the first raw (TOTAL), the students assessed themselves with a total mean score of 55 before and of 70 after the meeting (15 pp increase, 27% improvement), while their teachers assessed the students’ total ability with 43 before and 69 after the meeting (25 pp increase, 60% improvement). Therefore, the overall meeting’s added value as it was estimated by teachers was about double the students’ estimation, measured either by pp (25 ≈ 2x15) or by improvement (60% ≈ 2x27%). Any L1LO (i, ii, iii etc.) or L2LO (i1, i2, i3 etc.) mean score can be read this way.

The distribution of all 66 L2LOs in six interpretation zones is seen in Table 2. In the first raw, before the meeting, the ability of the students in 46 out of the 66 L2LOs (70%) was marked by their teachers as failed; 13 (20%) borderline; 6 (9%) moderate; 1 (2%) good; none very good; none excellent. According to the students’ self-assessment before the meeting, they failed in only 20 L2LOs (30%), they were borderline in 24 (36%), etc. After the meeting, no outcome was marked as failed or as excellent neither by the students nor by the teachers,

while about 90% of the outcomes were marked as moderate and as good.

A further analysis of the L1LOs on what students gained, according to their own and their teachers’ perceptions (perceived added value), is given in the Additional material 2. A similar analysis can be easily expanded to any L2LO.

Figure 1 illustrates the learning outcome scores from the highest to the lowest. Before meeting, the students self-assessed the L1LOs from “vi Staging procedures” (best) to “iv Clinical research” (worst) and the L2LOs from “vi1 TNM system” to “iv5 Cancer research protocol”, while the teachers assessed the L1LOs from “i Cancer biology” to “iv Clinical research” and the L2LOs from “i1 Biology of normal cells” to “iv5 Cancer research” protocol. After the meeting, the students self-assessed the L1LOs from “ix Oncology emergencies” to “viii Paraneoplastic syndromes” and the L2LOs from “vi1 TNM system” to “viic6 Anticancer-agent pharmaco-kinetics/dynamics/genetics”, while the teachers assessed the L1LOs from “viia Surgical therapy” to “viii Paraneoplastic syndromes” and the L2LOs from “iv3 Moral-ethical issues” to “viic6 Anticancer-agent pharmaco-kinetics/dynamics/genetics”, respectively. That is, the score ranks are not the same between the students and the teachers, and they do not remain the same after the meeting.

Figure 1

Comparing the two before the meeting graphs, the teachers’ colored bars are narrower than the students’; i.e., the teachers were more tightfisted at rating the students’ abilities than the students themselves (in total 43 versus 55, 12 pp difference, greater than 2 EMIDs). This can also be seen in the amount of red bars (failure): there is much more red color in the teachers’ graph. After the meeting, the width of the colored bars is about the same for the teachers and the students (in total 69 versus 70, 1 pp difference, much less than 1 EMID, i.e., negligible).

Neither the teachers nor the students had scored in the excellent area 90-100 (green), even after the meeting; i.e., the students did not overestimate their abilities.

The plot area of any graph of the figure – and this apply to all graphs of all figures – is divided in two main parts, the colored part on the left, increasingly narrowing top-down, and the complementary black part on the right, increasingly widening top-down. The colored part represents the percentage of the learning outcomes that has been met, while the black part represents the percentage of the learning outcomes that remains unmet. Thus, one can have at a glance the relation between met and unmet outcomes, at any level, from a specific L2LO to a L1LO and in total. Comparing the graphs after the meeting to those before the meeting, one can have at a glance the added value of the meeting.

Figure 2 illustrates the learner and the teacher scores. Emphasis is given on each specific learner or teacher. The graphs reveal the diversity within the students and the diversity within the teachers.

Figure 2

Before the meeting, the student who completed the iCAN! questionnaire 68^th self-assessed themselves as 88% (the highest), while the student who completed it 12^th self-assessed themselves as 31% (the lowest). After the meeting, the self-assessment ranged from 88% (highest) to 54% (lowest) for the students who completed the same questionnaire 25^th and 6^th, respectively. No student self-assessed themselves in the excellent zone (green), neither before nor after the course.

Before the meeting, the teacher who completed the theyCAN! questionnaire 3^rd assessed the students with 66% in average (highest), while the teacher who completed it 8^th assessed the students with 29% in average (lowest). After the meeting, the teachers’ student-assessment ranged from 94% (highest) to 55% (lowest) for teachers who completed the same questionnaire 4^th and 6^th, respectively. The only excellent score (green) was given by a teacher (4^th).

Figure 3 illustrates, through four examples (two students, two teachers; two tightfisted, two generous), the individual learner and individual teacher learning outcome scores. The emphasis is turned to each learning outcome of each specific learner or teacher. We can have the exact individual profile of each education partenaire.

Figure 3

The student, who completed the iCAN! 64^th before the meeting, self-assessed themselves with 54 in total (a rather self-tightfisted student), from 80 to 23 in the L1LOs (ix to i), from 80 to 20 in the L2LOs (viid5 to iv4). The teacher, who completed the theyCAN! 7^th after the meeting, assessed the students with 70 in total (a rather generous teacher), from 80 to 56 in the L1LOs (ii to ix), from 100 to 20 in the L2LOs (vi1 to viid11).

The graph can be read in the same way for any student (e.g., the one who completed the iCAN! 12^th after the meeting, a rather self-generous student) and any teacher (e.g., the one who completed the theyCAN! 5^th before the meeting, a rather tightfisted teacher). That is, the iCAN! and theyCAN! tools can give the exact individual profile for any student and any teacher.

Through the well established ESMO/ASCO oncology training global curriculum, the L1LOs and L2LOs were determined for a well-structured five-day full-time undergraduate oncology meeting. The L2LOs were transformed into the iCAN!-Oncology and the theyCAN!-Oncology questionnaires in order to assess the students’ and the teachers’ perceptions on students’ knowledge and abilities. The students completed the iCAN!-Oncology and the teachers the theyCAN!-Oncology questionnaires, before and after the meeting. The students had in their hands from the beginning of the meeting the learning outcomes they should be able to achieve at the end of it, and a tool, the iCAN!-Oncology, to self-assess themselves, enjoying an immediate (online) feedback at the same time. They also enjoyed very much the teachers’ before and after the meeting formative feedback on their (students’) abilities, through the theyCAN!-Oncology questionnaire. The students assessed themselves with a total score of 55% before and with 70% after the meeting (15 pp increase, 27% improvement), while their teachers assessed the students with 43% before and 69% after the meeting (25 pp increase, 60% improvement). Though the students themselves and the teachers assessed the students’ prior abilities with a difference of 12 pp (more than 2 EMIDs), they both assessed the students’ after the meeting abilities similarly (only 1 pp difference, much 1 less than 1 EMID, i.e. negligible). The tools discerned well the strong and the weak learning outcomes, the strong and the weak learners (the tightfisted and the generous teachers too), and the strong and the weak learning outcomes of each learner (and teacher). Thus, these tools can support a SWOT (strengths, weaknesses, opportunities, threats) analysis for an informed evidence-based educational policy [17].

Both teachers and students agreed that the biggest improvement was in the outcome “Clinical Research”, while the smallest one was in “Cancer Biology”. This is perhaps because the students already had prior knowledge on the cancer biology, while it was the first time they were exposed to clinical research. This could also be due to the different teaching methods; the cancer biology teacher delivered a lecture, while the clinical research teacher had prepared a small group teaching with student cooperation opportunities, more time for discussion and more interactive methods of teaching: Despite the fact that there is no agreement on the percentage of learning retention [18], lecturing is considered less effective compared to an active student engagement. In any case, the two-questionnaire system revealed the difference.

The worst, recently pinpointed by 803 students from all seven Greek medical schools, option of the educational environment was “The teachers are good at providing feedback to students” (score 15%) and the seventh worst was “I am clear about the learning objectives of the course” (41%) [19]. Both aspects were perfectly resolved using the iCAN!-Oncology and the theyCAN!-Oncology tools; and they can be resolved with the combination of the iCAN! / theyCAN! for any subject, core or selective, seminar or meeting.

As both tools are based on perceptions, they are subjective. A subjective and an objective test, before and after training, would be better. The iCAN! / theyCAN! can attend to the subjective part. An OSCE (objective structured clinical examination; [20]) could minister to the second part. However, our behavior is based upon our perceptions of the world rather than the real world itself; i.e., the subjective part might be more important to learners than the objective one. Secondly, students’ curiosity to compare their own perceptions to those of their teachers is a good motivation for learning, and the teachers’ curiosity to compare their own perceptions to those of their students is a good motivation for better teaching. Thirdly, while OSCEs promote hetero-assessment (unfortunately, at the end of the day, hetero-assessment teaches students how to pass the exams), iCAN! promotes self-assessment (fortunately, at the end of the day, self-assessment teaches students to see themselves critically). Fourthly, external assessment puts students against others, while self-assessment puts students against themselves (the famous Socrates’ “know yourself” [21] and “I know that I know nothing” [22]). Moving from hetero-assessment to self-assessment is a dynamic from the cognitive level (“knowing what”) to the metacognitive level (“knowing whether knowing”) [23], where the role of the teacher is not to fill an empty vessel [24], but to help students become better self-assessors. iCAN! / theyCAN! are invaluable tools on the metacognitive level (see Additional Material 3 for a definition of metacognition).

Furthermore, iCAN! / theyCAN! could minister not only to formative feedback, but also summative examinations. A student could present to their assessor their eponymously completed iCAN! and receive his/her theyCAN! completed by his/her teacher(s), as in Figure 3. The student and the assessor could then compare the self- and the hetero-assessment, discuss the weak (and strong) areas, the reasons why those areas remained unmet, what should be done to fill the gap, etc. Thus, the exams could be transformed from a punishing experience on the cognitive level (do students know what they should know?) into a constructive experience on a metacognitive level (do students self-assess themselves accurately?).

Although the usefulness of the iCAN! / theyCAN! paradigm does not depend on the low number of teacher after the meeting response, this might be a weakness of this study. However, even in the final summative exams, students are usually examined by one examiner (rarely two). Nevertheless, better logistics (protected time for the completion and immediate feedback, advertisement, involvement) could solve this problem.

Immediate feedback is highly important. It would be best if the first and the last hours of the course were devoted to questionnaires. During the first hour, the students and the teachers would realize what students should be able to do at the end of the course (outcomes) and the target would be clear to all; what they (the students and the teachers) think about the students’ abilities at the beginning; in which areas the students feel weak and strong; in which areas the teachers feel the students are weak and strong; why teachers’ perceptions are different to those of the students; etc. The last hour of the course should also be dedicated to the online questionnaire completion by both the students (iCAN!) and the teachers (theyCAN!). The results should be comparatively presented and discussed: Were the preset targets achieved? Why, if not? What should have been done differently? etc. Since no protected time was predicted for this study, solving the problem through reminders was not that effective. The completion and the immediate discussion and feedback were not considered as major teaching and learning procedures in order to build the teacher-learner coalition before the course and to share business gains and losses after it had ended. The theyCAN!-Oncology is a very important counterpart of the iCAN!-Oncology for students; they both consist a two-source feedback pair, a step towards multi-source and the ideal 360^o feedback.

Finally, a good tool should be valid, reliable, sensitive, and responsive [25]; in Additional material 4 we discuss why we believe both questionnaires meet these psychometric qualities and the extent to which they are met. It was not included in the main text because this paper’s aim was rather to discuss the before and after iCAN! / theyCAN! educational philosophy than to check the specific iCAN!-Oncology / theyCAN!-Oncology questionnaire properties.

Meanwhile, it has been revealed that students do not know that they don’t know (DNKDNK; the Socrates bias) and thus they overestimate their knowledge and abilities before they having been taught [26]. Indeed, a 15 pp increase after a 5-day full-time meeting seems pretty low. Incorporating an a-posteriori-before iCAN! [26] could give a more accurate estimation of the added value of the whole procedure.

The twins iCAN! / theyCAN! are exceptional teaching and learning tools. Given that clear learning outcomes have been set, their transformation into iCAN! / theyCAN! brings the outcome-based education theory to daily practice and offers students the possibility to self-assess themselves (iCAN!) at any point of their learning procedure and to compare self-assessment with the teachers’ student-assessment (theyCAN!). Completing them in advance, the starting point (what is already known) and the course’s learning outcomes (what should be learned) become clear to both learners and facilitators, and they will be happy to strive for them. Completing them immediately after the end point (what is now known), the course’s added value (what has been gained) and what has remained unmet become clear to both learners and facilitators, and they will be happy for the effort made; they will also have the opportunity to think about why the unmet outcomes remained so. Not only the strong and weak learning outcomes are well differentiated by the tools, but also the strong and weak learners, the generous and tightfisted teachers, and the strong and weak learning outcomes of each learner or teacher (their detailed individual profile). These properties make the tools very useful to both teachers and students, for immediate two-source formative feedback but also for the final summative exams at a higher (metacognitive) level. Thus, with the nowadays information technology available, the easily applicable iCAN! / theyCAN! paradigm supports the evidence-based SWOT analysis and the educational policy, generalizable to any course (core or selective, curricular or extra-curricular, undergraduate or postgraduate or any continuing professional development, oncology or non-oncology, medical or not).