Teacher evaluations of open educational resources designed to support dialogic cultural literacy learning in schools
Article Category: Research Article
Published Online: Aug 05, 2023
Page range: 136 - 147
DOI: https://doi.org/10.2478/eurodl-2023-0011
Keywords
© 2023 Maria Zimmermann et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Open educational resources (OER) are learning, teaching and research materials that are freely available online and can be flexibly used by anyone (Atkins et al., 2007; Hylén, 2006; Wiley & Hilton, 2018). As more and more people have started using OER (Downes, 2007; Otto, 2019), they are playing a key role in facilitating widespread access to education and enabling innovative teaching and learning (Otto, 2019). Nevertheless, researchers claim that OER use is still in its infancy (Otto, 2019). Furthermore, the literature suggests that a main challenge with using OER is the quality of the materials (Admiraal, 2022; Belikov & Bodily, 2016; Clements & Pawlowski, 2011; Luo et al., 2020).
Within the context of the European Horizon 2020 project ‘Dialogue and Argumentation for Cultural Literacy Learning in Schools’ (DIALLS), we developed a set of OER to foster students’ dialogic practices around learning cultural literacy in schools. As having OER of high quality is crucial for the long-term and meaningful use of the materials (Belikov & Bodily, 2016; Otto, 2019), in this study, we evaluated the quality of these materials as ‘stand-alone’ resources to be provided as OER on the DIALLS project’s website. In accordance with research on quality aspects of OER, we had teachers test the DIALLS materials in their lessons and give systematic feedback on the materials’
DIALLS was a 3-year project aiming at supporting the development of children’s cultural literacy through teaching them the skills of dialogue and argumentation. Reconceptualising cultural literacy as a dialogic social practice underpinned by tolerance, empathy and inclusion (Maine et al., 2019), the project worked with teachers in seven countries in and around Europe to support students to become more culturally literate. A key phase of the project was to develop materials to be freely accessible and available for any teacher to use, and a unique innovation included the use of wordless films that had been created by European directors and licensed for inclusion. Drawing on research in dialogue and argumentation (see e.g., Alexander, 2008; Mercer et al., 1999; Reznitskaya & Wilkinson, 2017), a programme of lessons called the Cultural Literacy Learning Programme (CLLP) was created to include a progression of learning for three age groups (4–7, 8–11, and 12–15). Additional OER that supported the CLLP included Professional Development (PD) materials and the Scale of Progression for Cultural Literacy Learning (SPCLL), a tool to support assessment and planning.
The purpose of using OER can vary: Teachers report that they use OER to get new ideas and inspiration, to prepare for and develop their teaching or to supplement their lessons (De los Arcos et al., 2016). What all these purposes have in common is that teachers use them (and are motivated to use them) autonomously (Werth & Williams, 2021).
A variety of opportunities come with OER (cf. Belikov & Bodily, 2016), beyond allowing education to be widely accessed and innovated (Otto, 2019). Other benefits of using OER are that they do not require institutional support or many resources (e.g., they are low cost), and they offer pedagogical benefits (e.g., flexibility to adjust OER to respective needs) (Belikov & Bodily, 2016; Bliss et al., 2013). Educators stress that OER can enhance their teaching by providing information about recent incidents, a variety of topics and a broad array of perspectives (Bliss et al., 2013; Richter & Ehlers, 2011). Thus, with the help of ready-to-use OER, educators think they can provide better education for their students, e.g., customised to their learning needs (Bliss et al., 2013).
Another benefit of OER is that they can lead to exchanges among educators (Bliss et al., 2013). Using OER may increase collaborations among educators, which might save them time and allow them to continuously adapt their pedagogical practices as well as their educational knowledge and resources to an ever-changing society (Hylén, 2006; Otto, 2019). Finally, not only does such exchange invite educators to share their experiences (Bliss et al., 2013) but OER may also encourage teachers to reflect on the way they teach (De los Arcos et al., 2016).
As mentioned earlier, teachers are autonomously motivated to use OER (Werth & Williams, 2021). As such, using OER requires teachers to work with the shared resources independently and flexibly at their own pace, addressing their students’ specific needs as well as their teaching requirements (Otto, 2019). Selfregulated learning is of particular importance in informal learning settings at the workplace (Sitzmann & Ely, 2011). It is crucial for facilitating change, e.g., acquiring new knowledge about teaching practices from OER (Littlejohn & Hood, 2017). This can be challenging but motivating at the same time (Belikov & Bodily, 2016; Bliss et al., 2013; Kimmons, 2015); for example, Kimmons (2015) found that compared to traditional textbooks, OER were able to meet teachers’ expectations when the materials could be remixed, adapted or modified. All this suggests that it is important for OER to facilitate selfregulated use. When teachers are willing to engage with them, the resources themselves must allow teachers to act autonomously by providing sufficient information and possibilities for teachers to independently engage.
The importance of self-regulated use in context with OER is supported by self-determination theory (SDT). This theory suggests that a teacher’s motivation may not only depend on their basic psychological needs for competence, relatedness and autonomy but also on the environment (Deci & Ryan, 2008). As such, teachers may act in a self-regulated and internally motivated way because they identify with the activity’s value (Deci & Ryan, 2008) or when OER are created in a way that allows teachers to use them autonomously and flexibly (Salikhova et al., 2020).
The benefits teachers can realise from using OER exist alongside challenges that may hinder their uptake (Otto, 2019). As the popularity of OER increases, the need for quality control is also growing (Admiraal, 2022; Clements & Pawlowski, 2011). In this vein, the steadily rising number of OER (Creative Commons, 2017; Downes, 2007) makes it increasingly difficult for users to systematically assess OER’s quality; users, for example, might not have the time to effortfully engage in quality checks (Belikov & Bodily, 2016). Furthermore, OER are not necessarily reviewed before or even after publication, which thus hinders their use, and some users have expressed concern regarding the quality of the materials (Admiraal, 2022; Belikov & Bodily, 2016; Luo et al., 2020).
In this context, it is important to define what ‘quality’ OER means. Quality should not necessarily and solely be seen as an objective measure (i.e., concerned with the correctness of information and the validity of the materials) (Kursun et al., 2014; Richter & Ehlers, 2011), but it should also encompass the materials’ value as perceived by teachers (Baas et al., 2022; Clements & Pawlowski, 2011; Cox & Trotter, 2017). In this sense, to assess quality, researchers need empirical studies on teachers’ perceptions of OER (Belikov & McLure, 2020; Bliss et al., 2013; Leighton & Griffioen, 2021).
While many approaches exist on how to develop and evaluate OER (Jung et al., 2016; Nikoi et al., 2011), there is no common recommendation for which criteria should be considered in an evaluation of OER quality or, further, how these criteria should be conceptualised and assessed; studies have included widely varying numbers of quality dimensions and related criteria (Zawacki-Richter & Mayrberger, 2017). Among the many quality dimensions/aspects mentioned in the literature (e.g., licensing, openness, provenance, fit of purpose, navigation, content, usability, design and sustainability: see Zawacki-Richter and Mayrberger (2017), for an overview of OER evaluation frameworks), it seems crucial to consider not only the aspects mentioned regularly but also those particular to a specific learning context.
In this case, the learning context is teachers’ PD through self-regulated learning to promote students’ cultural literacy learning with dialogue and argumentation. OER not only are materials to use but also include information for teachers to acquire knowledge about a new teaching practice, i.e., dialogue and argumentation.
Because the DIALLS learning materials were developed as open-access materials, in this study, we did not focus on assessing whether the materials can be accessed openly (cf. Haughey & Muirhead, 2005; Jung et al., 2016). Instead, we aimed to learn whether teachers could use the materials sensibly to promote their students’ cultural literacy through dialogue and argumentation in classes, i.e., our specific learning context. Hence, of the dimensions for evaluating OER quality that have received attention in the literature, we focussed on five:
Since OER are considered open materials that can be flexibly used by anyone, as described earlier (Atkins et al., 2007; Hylén, 2006; Wiley & Hilton, 2018), their ability to be applied effectively, efficiently and independently is particularly important. OER scoring high on application would allow educators to appropriately use the materials, such as by including instructions on how to use and adapt the materials to specific learning situations. In this sense, an evaluation of educational materials may address how well the materials themselves or any additional information provide support that enables users to use the materials flexibly and appropriately (Cox & Trotter, 2017; Haughey & Muirhead, 2005). Thus, on the application level, we were interested in whether the teachers experience an easy and straightforward application of the (knowledge gained from the) materials in practice, i.e., whether they were able to transfer their new knowledge about teaching dialogue and argumentation to their classrooms.
Among the diverse quality criteria related to the content of OER (e.g., correctness, completeness or currency; Zawacki-Richter & Mayrberger, 2017), we were particularly interested in whether the content of the DIALLS materials covers the concepts of the intended learning, i.e., cultural literacy and dialogic practice, as defined in the project, and whether the content helps educators gain a deeper understanding of corresponding pedagogical practices (Jung et al., 2016). As the materials were developed during the DIALLS project and are based on research on argumentation, dialogue and cultural literacy, the content legitimacy of the materials is assured by the research done during the 3-year project. In addition to the theoretical and empirical foundation, it is also important that the material reflect this content appropriately.
Design and appearance are often seen as the first hurdle that can discourage users from using these materials, e.g., when the materials are of low visual appeal (Baas et al., 2022; Bugler et al., 2017). According to Baas et al. (2022) and Kurilovas et al. (2011), the design of learning materials, however, refers not only to the ‘look’ of the materials but also to the structure of information (e.g., whether reading the information is confusing), which is an important prerequisite for selfregulated learning. Hence, by referring to the design of the DIALLS OER, we were interested in whether the materials encourage educators to engage with them.
The evaluation aspect
With respect to the sustainability of OER, the literature often describes aspects related to the openness of materials (Downes, 2007; Tlili et al., 2023; Wiley, 2007). However, teachers assess OER’s quality not only before but also after using them (Clements & Pawlowski, 2011). Therefore, for the sustainable use of OER, we think that it is important to consider an integrated view of the aforementioned aspects: While the quality of OER in terms of
By investigating teachers’ perceptions of OER’s quality, this study focussed on teachers as experts in teaching (Baas et al., 2022; Clements & Pawlowski, 2011). Methodologically, this means that we applied the principle of peer review to our OER (Zawacki-Richter & Mayrberger, 2017). Firstly, teachers gave feedback on the quality of the DIALLS OER by answering surveys that systematically asked about quality criteria of OER derived from the literature, namely,
Overall,
During the DIALLS project, between September 2020 and February 2021, participants were asked to work with and subsequently evaluate the OER at their own pace. Specifically, teachers were asked to
1. engage with the PD material,
2. conduct up to 10 lessons from the CLLP with their group(s) of students by using the lesson prompt materials,
3. engage with the SPCLL and
4. give systematic feedback on the respective resources through designated online surveys.
Access to all online resources (i.e., the PD materials, lesson prompts, the SPCLL and the evaluation surveys) was centralised in a password-protected members’ area on the DIALLS website. All surveys were administered with Questback’s software Unipark (
In the following, we introduce the materials developed in the project and show how their characteristics relate to the evaluation criteria.
The CLLP included a bibliography with 30 wordless films representing a breadth of topics from the Cultural Analysis Framework (
The PD material included three separate but related materials about (1) exploring cultural literacy, (2) promoting and building dialogue and argumentation and (3) mediating wordless films. These materials aimed to help teachers flexibly plan their lessons (
The SPCLL comprises two tools, namely, the Dialogue Progression Tool and the Cultural Learning Progression Tool. They include indicators to highlight how to improve students’ learning (
We collected quantitative and qualitative data to answer our research questions. The goals of the evaluation were to assess teachers’ perception of the OER in terms of
We inferred five predominantly used levels of OER evaluation from the literature to answer RQ1 on how teachers perceive the application, content, design and value as well as sustainability of the DIALLS materials and adapted these according to our learning context. On the
In addition, and as outlined earlier, we assessed the
Reliability scores for evaluation scales for DIALLS OER
| Material | Cronbach’s alpha |
|---|---|
| PD (for three materials packages) | 0.87 – 0.88 |
| Lesson Prompts | 0.86 |
| SPCLL | 0.94 |
As mentioned earlier, teachers had the opportunity to provide open feedback at the end of each survey to add any aspects that had not been evaluated. These served as the foundation for gaining more insights into what teachers value when evaluating and working with OER (RQ2). We developed a coding scheme to help answer this question. The coding scheme incorporated four categories derived from the literature as outlined and defined before: (1)
All data are openly available on Zenodo (Mayweg-Paus et al., 2021).
The descriptive statistics for each material are summarised in Table 2. Teachers’ impressions of the four resources were positive overall. They rated the lesson prompts highest (M = 4.31, SD = 0.53), followed by the PD material (M = 4.07, SD = 0.50) and the SPCLL (M = 3.99, SD = 0.60). This means that teachers strongly agreed that they were able to use the lesson prompts, the SPCLL, and the PD materials; that they liked the materials’ content and design; and that they appreciated the materials’ value.
Descriptive statistics for DIALLS OER
| Material and subscale | n | Min | Max | M | SD |
|---|---|---|---|---|---|
| PD Total—Application | 2.11 | 5.00 | 3.96 | 0.06 | |
| PD Total—Content | 2.11 | 5.00 | 4.02 | 0.04 | |
| PD Total—Design | 2.00 | 5.00 | 4.20 | 0.06 | |
| PD Total—Value | 1.22 | 5.00 | 4.11 | 0.04 | |
| PD—Total | 121 | 2.47 | 5.00 | 4.07 | 0.50 |
| Lesson Prompts | |||||
| Application | 1.33 | 5.00 | 4.25 | 0.71 | |
| Content | 2.33 | 5.00 | 4.40 | 0.56 | |
| Design | 2.33 | 5.00 | 4.28 | 0.60 | |
| Value | 1.67 | 5.00 | 4.30 | 0.74 | |
| Lesson Prompts—Total | 77 | 2.83 | 5.00 | 4.31 | 0.53 |
| SPCLL | |||||
| Application | 1.67 | 5.00 | 3.94 | 0.70 | |
| Content—Cultural Learning Tool | 1.00 | 5.00 | 4.09 | 0.70 | |
| Content—Dialogue Progression Tool | 1.00 | 5.00 | 4.04 | 0.74 | |
| Design | 2.00 | 5.00 | 3.86 | 0.66 | |
| Value | 2.33 | 5.00 | 4.01 | 0.65 | |
| SPCLL—total | 65 | 1.80 | 5.00 | 3.99 | 0.60 |
Regarding the
As part of the survey, we asked teachers what they would like to add about the materials and the CLLP in general. With this, we aimed at investigating which aspects are not present in the literature that teachers value when evaluating the quality of OER (RQ2). To identify any missing elements in our evaluation of the OER, we systematically analysed the open feedback from the surveys. The teachers’ feedback was divided into units of meaning, in which consistent themes or ideas were identified (Clarà & Mauri, 2010). Based on the coding scheme (see above), one researcher assessed all
The descriptive frequencies of the coding categories are displayed in Table 3. These show that
Absolute and relative frequencies of coding categories
| Coding category | Absolute frequencies | Cumulative absolute frequencies | Relative frequencies | Cumulative relative frequencies |
|---|---|---|---|---|
| Application | 54 | 54 | 0.22 | 0.21 |
| Content | 26 | 80 | 0.10 | 0.31 |
| Design | 48 | 128 | 0.18 | 0.49 |
| Value | 41 | 169 | 0.16 | 0.65 |
| Adaptability | 26 | 195 | 0.10 | 0.75 |
| Reflection | 22 | 217 | 0.08 | 0.83 |
| Other | 19 | 236 | 0.07 | 0.90 |
| Off-Topic | 25 | 261 | 0.10 | 1.00 |
With respect to the first research question on how teachers perceive the DIALLS materials in terms of their application, content, design, value and sustainability (RQ1), the quantitative evaluation of the materials indicates that the teachers, as experts of teaching and learning, were highly satisfied with all the DIALLS OER. In particular, the quantitative data on how teachers perceived the design and application of the material indicate that these materials were successfully developed as stand-alone resources that allow teachers to use them in a self-regulated way (RQ1). Such positive perceptions towards the materials were also reflected in teachers’ qualitative feedback about the materials, in which they, for instance, stated that they were highly motivated to use the DIALLS learning materials. Interestingly, teachers especially appreciated the lesson prompts as well as the PD material and favoured the respective concepts about cultural literacy, argumentation and dialogue, and the mediating films. Similarly, their open feedback on the lesson prompts showed that the participants perceived the prompts to be helpful to flexibly prepare a lesson around cultural themes and discussions; for instance, one teacher stated that ‘The lesson prompts are engaging, flexible, and support the sessions’. Finally, the teachers also reported that the materials helped in cultivating their students’ cultural literacy and argumentation skills (e.g., ‘The students have made progress and learned in every lesson’. Or ‘The children whom I teach enjoy the lessons and the videos very much and relate to them and [to my great surprise] manage to understand many of the values and messages presented […]’). The quantitative results in terms of the SPCLL showed that teachers thought the tools were very helpful not only to plan a DIALLS lesson around argumentation but also to identify progress in their students’ skill progression. Similarly, some teachers mentioned that the right prompts led to fruitful discussions among the students. They were able to improve their students’ skills over the scope of DIALLS. Overall, most of the teachers expressed their future willingness to use the DIALLS materials to educate their students’ cultural literacy and argumentation skills, which altogether emphasises their satisfaction with the materials as well as indicates that the materials are useful for educating their students’ cultural literacy through argumentation and dialogue.
With respect to the second research question on what else teachers think is important when it comes to evaluating the quality of OER (RQ2), in their open responses at the end of the surveys, the participating teachers most often mentioned quality aspects that were already covered in the evaluation questions during the quantitative assessment (i.e., aspects related to the
Together, the findings of the quantitative and qualitative analyses are in line with the literature on the importance of teachers’ self-regulated use of OER (Kimmons, 2015) and support the assumption that OER should fulfil the requirements for self-regulated learning by allowing teachers to use them flexibly for their own teaching and to adapt them to their students’ needs (Kimmons, 2015).
Regarding the content analysis of the open responses and, thus, the deductively as well as inductively derived coding categories, it should be noted that the interrater reliability was only moderately high. In detail, this means that of all the categories,
Furthermore, it is also important to mention that the number of teachers who evaluated the specific materials decreased from survey to survey, such that the study possibly describes a distorted drop-out sample. Since the evaluation of each piece of material was linked to the teachers’ use of the materials in real classroom settings with their students and from September 2020 to February 2021, this study was highly affected by the COVID-19 pandemic and school lockdowns. Of the 140 teachers in total, between 64 and 121 teachers responded to each of the surveys (i.e., to each of the specific materials). This means, on the one hand, that we do not know whether teachers who might not have positive attitudes towards the project and the materials were, perhaps, less motivated to continue giving their feedback on the materials and, thus, more likely to end their participation. On the other hand, this also means that teachers who continued their participation tested the materials with diverse students and even under the challenges of the COVID-19 pandemic, indicating that the DIALLS materials are appropriate for being flexibly adapted towards specific teaching and learning needs. This promises a successful, long-term and adaptable use of the materials in the future.
Another limitation concerns the statistical reliability of the results from analysing a heterogeneous sample, i.e., 140 teachers from 87 schools in different countries. Simultaneously, though, such samples are common in large, practical research projects and benefit from high external validity.
Firstly, this study has implications for teachers’ use of the DIALLS materials. By considering OER users’ challenges in evaluating the quality of OER (Belikov & Bodily, 2016; Clements & Pawlowski, 2011), the systematic evaluation of the DIALLS materials was a crucial part of the DIALLS project to provide open and high-quality teaching materials around dialogic practice and cultural literacy learning in schools. The findings of the evaluation suggest that the self-reliant use of the DIALLS materials by European teachers is sustainable, which is important for the use of these materials even beyond the duration of the DIALLS research project (Hylén, 2006). The evaluation was carried out to determine whether the materials were also considered useful by teachers and whether they could be used independently and flexibly, thereby addressing the need for high-quality assurance of OER at the material development stage (Belikov & Bodily, 2016). Our systematic evaluation considered quantitative and qualitative data from teachers and shows that overall, the feedback in terms of the materials’ application, content, design and value, as well as its adaptability and its potential to spark reflection about one’s own teaching, indicate that the teachers, as experts of teaching and learning, were highly satisfied with all the materials. Furthermore, teachers were willing to use and recommend the DIALLS materials in the future, indicating a high sustainability of the developed DIALLS OER. Since teachers tested the materials with diverse students (e.g., diverse age groups or in rural or urban regions) and even under the challenges of the COVID-19 pandemic (e.g., distance learning), the results indicate that the DIALLS materials are appropriate for children between 4 years and 15 years and for the flexible adaptation towards specific teaching and learning needs, which again emphasises the DIALLS materials’ potential for being used by teachers in a self-regulated way.
Secondly, the study highlights that when teachers use OER, the most important aspects they consider include those derived from the literature (i.e.,
After evaluating the DIALLS OER, we can conclude that the materials enable a self-regulated use by teachers to teach cultural literacy with dialogue and argumentation. From the evaluation, we take away that keeping the learning context of the OER in mind as well as considering aspects not only related to theory and practice but to the acquisition of new knowledge, e.g., reflection and adaptation.