Specialty training in family medicine in Montenegro – An evaluation of the programme by the first generation of trainees
Article Category: Original scientific article
Published Online: Apr 12, 2018
Page range: 96 - 105
Received: Oct 16, 2017
Accepted: Mar 12, 2018
DOI: https://doi.org/10.2478/sjph-2018-0013
Keywords
© 2018 National Institute of Public Health, Slovenia
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 License.
For decades, the World Health Organisation has underlined the importance of strong primary health care for better health of the population, for reducing inequality, for improving access to health care, and for lower costs (1, 2). Systems which are oriented towards secondary and tertiary health care are both more expensive and less effective (3).
In Montenegro, one of the former Yugoslavian countries, primary health care is community-oriented (4) and declared a priority by law (5, 6). The Ministry of Health of Montenegro prepared health care reform in 2003 and introduced new roles and responsibilities for primary health care teams. One of the main changes was that each patient must select a (personal) doctor who has a gate-keeping role and responsibility for the quality of service. One of the goals of health care reform was also to implement a curriculum of family medicine (FM) into undergraduate education, and to develop and implement the specialisation of family medicine. This process has been described in detail elsewhere (7).
The first step in implementing a specialisation in family medicine was an intensive 4-month course for almost all the primary care teams in the country, which started in 2005 with the help of Slovenian educators. From 2005 to 2011, eleven courses were organised, and 222 primary health care teams finished the training. The first results showed that the quality of work increased; primary care was more accessible, the organisation of the service improved, and patient satisfaction was higher (7).
The next step was the organisation of an additional one-year training programme for the most experienced General Practitioners (GPs) who were motivated towards teaching; these became the first specialists in family medicine. In 2012, twenty-four GPs completed this programme. It was intended as a short-term solution to create mentors for future trainees in family medicine. The programme was led by the Department of Family Medicine at the Faculty of Medicine, of the University of Ljubljana, as a part of a project by the World Bank (8).
The first generation of trainees to start the full four-year specialty training began their course in January 2013. The specialisation course in family medicine was led by a coordinator from the Faculty of Medicine of the University of Montenegro, in Podgorica, who is responsible for specialty training in family medicine under the supervision of the Department of Family Medicine of the Faculty of Medicine in Ljubljana. The specialisation course is based on modern concepts of teaching, described in the EURACT Educational Agenda (9). The Agenda was conceived to define which specific competences can be expected from the discipline of general practice/family medicine, and to enable the harmonisation of the different learning programmes in Europe. The Educational Agenda represents the basis for preparing relevant training programmes that are mainly competence-driven (10).
The specialisation course has practical and theoretical elements, which run side by side during both the family practice component and the hospital-based component. Its total length is 48 months; 24 months take place in family medicine practices working with a direct mentor (the family practice component), and the other 24 months take place in different clinical wards (the hospital-based component). The family practice component is further divided into two parts, namely: the first part lasts 5 months and the second part 19 months, of which 4 months are spent in the primary care paediatric offices and 15 months in family medicine practices. During the family practice component, 20 two-day modules of theoretical teaching are organised on the specific knowledge, skills and attitudes of family medicine (e.g., the organisation of the practice, the quality of care, the management of patients with chronic diseases, and drug prescribing).
The hospital-based component of the specialisation course takes place at the Clinical Centre of Montenegro, which represents the teaching basis of the Faculty of Medicine, and in some general hospital wards. The clinical rotations are as follows: 6 months of internal medicine, 3 months of surgery and paediatrics, and 2 months of gynaecology and psychiatry, while other specialties, e.g., dermatology, orthopaedics, etc., have one month each. The detailed programme of the specialisation course has been published elsewhere (11).
Before the implementation of the training programme, several obstacles in organisation, negative public and professional opinion, and political indifference had to be overcome. Many problems were exacerbated by the fact that in Montenegro specialisation in family medicine is not mandatory (7).
In general, doctors were not interested in applying for specialty training for financial reasons – the salary of trainees was smaller than what they had been able to earn before, and even after they finished the programme, it would still only be similar to that of doctors who had not completed the specialisation course.
Doctors on the training course temporarily lost their autonomy – before entering the programme, they worked independently, but during the training, they had to work under the supervision of mentors in the mentors’ own practices. In addition, primary health care managers were not interested in sending their experienced GPs (most of them had at least five years of experience), as they lost part of their workforce and had to pay expenses.
The Faculty of Medicine did not fully support the development of the training course; they allocated only one lecturer (who was also involved in the teaching of another subject at the university) and two assistants. There was no Department of Family Medicine with administrative staff and devoted space, and financial resources for the coordination of the training course were very limited.
There was also no political will to legalise the specialisation in family medicine as obligatory for all GPs.
However, despite these many barriers, the specialisation course was carried out according to the programme, and in February and June 2017, trainees from the first generation completed the programme. Some of them have already passed the final examination. The evaluation of the programme is the first step in the quality improvement process to improve the process of this training. The aim of this article is to evaluate the implementation of specialty training in family medicine in Montenegro, regarding the content, structure and methods, by the first generation of trainees and the coordinator of the training.
All the 26 trainees in the first generation of the family medicine training, who began their training in 2013, were invited to participate in the study. Twenty-two (84.6%) of the trainees participated in the study, of whom 11 had already passed the final exam, while 10 trainees were prolonging their training (9 due to maternity leave) and one failed the exam.
A questionnaire was sent by post to all 26 eligible trainees in July and August 2017.
The questionnaire was developed based on the literature by researchers from both Faculties of Medicine involved in the training as educators. Specifically, the questionnaire was composed of one on the satisfaction of trainees with the training programme, prepared by the Irish College of General Practitioners (12), and the Job Evaluation Survey tool, a simple, validated tool to evaluate the satisfaction of trainees with their training (13). A consensus of all the researchers on the content of the questionnaire was reached, considering the aim of the study. Additionally, a former trainee who had recently passed the speciality exam was also involved in the preparation of the final version of the questionnaire.
The questionnaire consisted of the following combination of closed and open-ended questions: the demographic characteristics of the trainees, and questions related to the evaluation of the process of the training and the specialty exam. Evaluation of the training was made by using a Likert scale from 1 (very poor) to 5 (excellent) to assess the training in terms of usefulness, quality, learning methods, mentorship and organisation. The final questions asked the trainees how the training had influenced their work in the practice and how their expectations had been fulfilled.
A semi-structured interview was carried out by the two researchers (MPŠ and DP) with the coordinator of the training, to discover the obstacles in the implementation of the training and to evaluate the programme.
In the quantitative part of the analyses, descriptive statistical methods, with mean and standard deviation in the case of a normal distribution of data, and range and median in the case of a non-normal distribution of data, were used.
Simple thematic qualitative analyses with predefined themes were carried out: the assessment of the specialty training; the role of mentors; the organisation of the specialty training; the influence on the work in practice; and the fulfilment of expectations. The themes for analysis of the semi-structured interview with the coordinator were: teaching methods, assessment of knowledge, the final exam, mentors, and foreign advisors. Open coding of the text quotations was carried out separately by two researchers (DP and MPŠ); where there were differences between the researchers, the final decision was reached by a consensus. The sub-themes which emerged from the coding were checked as to whether they fitted into the predefined themes. If any new themes emerged, they were added to the predefined themes.
The basic demographical data of the 22 participating trainees are in Table 1. The trainees were from all regions of Montenegro and were predominantly female.
Description of the participating trainees.Characteristic N or Mean % or distribution of values, SD 19 86.4% 39.9 from 35-57, SD 5.0 years 35.7 from 31-52, SD 4.7 years 8.1 from 7.0-9.7, SD 0.8 6.3 from 2-20 years, SD 4.4 years
The trainees evaluated different aspects of training and the impact of the training on their daily practice. The evaluation of the training by Likert scale is set out in Table 2.
Evaluation of the training.Aspects of evaluation Lowest/highest rate (range) N or Mean 3-5 (2) 4.41 (0.73) 3-5 (2) 4.41 (0.67) 3-5 (2) 4.62 (0.59) 3-5 (2) 4.55 (0.74) 2-5 (3) 4.68 (0.72) 3-5 (2) 4.43 (0.75)
The trainees assessed the process of training with high scores. The highest mean was given for ‘organisation,’ but it also received the highest range – from poor to excellent. The teaching methods were also assessed very highly (between very good and excellent).
In each year of the training, the trainees had to take a written test, called the ‘progress test.’ It consisted of 100 multiple choice questions from a database of 1400 questions prepared for the final exam and gave the trainees feedback about the required and obtained knowledge. Table 3 represents the results of the progress test; 60 or more points were required to pass the final exam.
Results of the progress test.Test year N Minimum number of points Maximum number of points Mean (SD) The number (percentage) of trainees with at least 60 points (pass/fail criteria) 19 31 74 57.7 10 (52.6%) 21 63 94 80.4 21 (100%) 14 53 75 65.9 11 (78.6%) 17 56 84 71.7 15 (88.2%)
The percentage of trainees who obtained at least 60% of positive answers increased from the first to the last year of training, except for 2014, in which all the trainees successfully passed the progress test.
Of the 22 participants in the evaluation, 12 had already taken the final exam, which consisted of 120 multiple-choice questions, 12 OSCE stations (Objective Structured Clinical Examination), two MEQs (Multiple Essay Questions) and an oral exam. The multiple-choice questions assess the trainee’s knowledge, while the OSCE stations assess the trainee’s skills. The assessment criteria for the OSCE stations were predefined to reduce the subjectivity of the observer. The MEQs assessed the student’s ability to identify and resolve clinical problems by applying their existing knowledge. The answers to the MEQs were assessed by two independent assessors from the assessing committee, who arrived at a consensus.
Only those candidates who scored at least 60% of the points in the written exam (72 points), passed 9 out of the 12 OSCE stations, and satisfactorily answered both MEQs had the right to continue to the final oral exam.
Table 4 presents data on the grades in the different elements of the exam and in the final exam.
Grades in the different elements of the exam.Elements of the exam Pass criteria The number (%) of participants who passed this part of the exam Mean number points (N=12) Range, median 72 points (60%) 12 (100) 105.8 From 96 to 117, median 104 9 (75%) 12 (100) 11.8 From 10 to 12, median 12 Excellent, very good or good 12 (100) Pass 11 (91.7)
A comparison of the results of the progress test and of the final exam shows that additional learning was needed to succeed in the final exam. One candidate was not successful in the final exam.
The content analysis with predefined themes revealed 64 codes that were attributed to the following themes: the assessment of the specialty training, the role of the mentors, the organisation of the specialty training, the influence on the work in practice, and the fulfilment of expectations. Most of the themes were presented by positive and negative aspects. To protect the anonymity of the trainees, the quotations are only identified by the trainee number.
There were 14 different codes for positive assessment, and 6 for negative. The positive assessment stated that the specialisation course offered teaching modules and practice which were based on modern educational theory, improved existing and acquired new knowledge, and offered practical useful knowledge and skills necessary for everyday work. It helped the trainees not only to master the principles of modern medical trends, but also improved professionalism, increased self-confidence, and improved the scientific approach to everyday work using guidelines and evidence-based medicine (EBM).
Fulfilment of expectations was unanimously high, from enough/sufficient to completely fulfilled and ‘more than expected.’
Negative assessment was exclusively associated with the clinical components of rotations, where mentors were not adequately acquainted with the scope of clinical knowledge for FM, the lack of influence of the Department of Family Medicine on training in clinical rotations, and some clinical lectures being too theoretical. One opinion was also that the training programme was too ambitious for the existing working conditions.
To a great extent, the experience was very positive; the mentors were motivated, committed, enthusiastic, selfless, collaborative, good role models and friendly.
There were some negative comments regarding clinical mentors showing variable dedication and accessibility, and also that some were unprepared and insufficiently engaged, showing a lack of knowledge of the content and the specialisation course plan.
The trainees stated that the organisation was very good, offering continuity in learning with teaching organisations, and that the modules and practice were well-balanced throughout the course. Negative comments included that the practical and theoretical elements of the same field should be given simultaneously, and that more time should be spent in outpatient clinics. There were criticisms of the organisation of the clinical mentors and the quality of some lecturers at the beginning of the specialisation course.
The trainees stated that they had acquired a new approach to patients, new knowledge and skills, and more EBM and scientific reasoning. They also found the influence of the training in their changed attitudes towards work and medicine, specific methods of work (e.g., time as diagnostic criterion), and improved communication and self-confidence in their everyday work. The training influenced their organisation of teamwork, and the realisation of how important the satisfaction of both patients and health workers is as a quality criterion.
The exam was described as well-conceived, and appropriate for testing the necessary knowledge and skills for practical work. A human approach towards the trainees was mentioned. It was an important moment in the lives of the trainees. It was suggested that it should be organised over two days instead of one.
During the interview, the coordinator assessed the process of the training and the final exam positively, and expressed the main points for the success, described below.
Mixed teaching methods were introduced: fewer plenary lectures and more work in small groups; interactive learning and project work; and teaching of clinical skills on models. Small group work led by experienced group leaders fostered the changing of attitudes and changes in clinical practice. The professional integrity of trainees increased, and they were able to implement the new knowledge and attitudes in their practice.
Modern teaching and assessment methods, including formative assessment with progress tests, encouraged the upgrading of knowledge and continuity in learning.
The complex structure of the final exam gave a comprehensive assessment of the candidate.
The number of competent mentors and their distribution all over the country were not adequate to the needs of the training. There were some enthusiastic mentors without trainees and some mentors in the capital with more than three trainees at the same time. Nevertheless, the mentors, especially the main mentors, had an extremely important role in the whole process of the training; most of the trainees recognised a role model in their mentor.
The involvement of our Slovenian colleagues in the teaching process, their supervision of the programme, and their participation in the final exam set the whole process at a higher level and gave it a better reputation at the national and international level.
A strong will and efforts of the coordinator and team involved in the process of the training brought the project to a successful conclusion. The first generation of specialist family medicine trainees in Montenegro have successfully finished their specialty training.
The evaluation of the programme by the trainees and the coordinator showed that it did not only increase their knowledge and skills, but also increased their selfconfidence and improved professionalism and scientific approach in their everyday work, using guidelines and evidence-based medicine.
Training in family medicine, taking into account the EURACT Educational Agenda (9) and modern teaching methods, and also including constant feedback from mentors according to the ‘Slovenian model’ of training (14), was well accepted by the trainees in Montenegro. There is one important difference between the two programmes (namely, Slovenian and Montenegrin) - in Montenegro the trainees lose contact with general practice and with their main mentors for two years during their clinical rotations, because, at that time, all the teaching activities take place in hospitals.
The organisation of the training was in general assessed very highly. Most of the criticisms came from lower satisfaction with clinical rotations, mainly because the clinical mentors were not well acquainted with the training programme and goals.
During the training of the first group of Montenegro trainees, progress tests were used annually to evaluate the level of knowledge, and we found that the differences in knowledge reduced through the process of training (15). We did not use the OSCE method for formative purposes assessment because it has lower psychometric standards (16). Longitudinal and competence-based assessment were also found to be the currently preferred approach for FM specialty training in other studies, such as in a survey based on a convenient sample in five European countries (Denmark, Germany, Poland, the Netherlands and the United Kingdom) (17).
One of the challenges of teaching medical trainees is to choose an assessment method that is directed towards enhancing learning in addition to assessing clinical competence. Workplace-based assessments allow trainees to continually gather evidence of learning and formative feedback (18).
Feedback in workplace-based clinical settings often relies on expert trainers’ judgements of the directly observed trainee (15, 18). Close contact between the mentor and the trainee working in the same practice during the family practice component of the rotation enables constant feedback. According to the results of our study, the trainees assessed their mentor’s feedback as valuable and often took their mentors as a role model. However, this assessment was not structured, and it relied on the personal approach of the mentor. In the future, the feedback of mentors will need to be further developed and structured according to the developing competencies and progress of the trainees, so that it is useful for the trainees and feasible for the mentors at the same time (19, 20).
One of the important achievements of the training was the participants’ feelings that the training helped them to improve the quality of their work and patient safety. The trainees mentioned that they felt more competent in decision-making and felt more self-confident and less vulnerable to potential medical errors and complaints (21). A unique goal of the training programme was to give trainees an understanding of the holistic and generalist approach, on the one hand, and the usefulness of EBM, on the other, and to see possibilities for future development of the discipline (22, 23).
The use of OSCEs for the assessment of clinical skills in a standard setting has been shown to be normative in all high stake exams (16). The differences in clinical skills between the trainees were far smaller than the differences in the written part of the exam. One of the reasons for a good performance in the OSCEs was the limited number of OSCE stations and that the trainees could relatively easily gain the skills at least at a moderate level. In the future, the set of OSCE stations should be increased to enable the testing of various skills which a FM specialist must master. In the MEQs, and especially in the oral exam, we tested the level of critical thinking based on clinical knowledge. According to the findings of Ross and co-authors, there is a significant positive correlation between critical thinking skills and performance on knowledge tests (24). In addition, good critical thinking skills have been found to predict success in family medicine certification examinations. An assessment of critical thinking in the progress tests during the training may help to identify applicants more likely to be successful in the final certification exam (25).
Finally, an important emotional aspect of the exam was that it presented a positive experience, even ‘a beautiful moment in life,’ and gave the colleagues necessary confidence for their future professional careers and positive incentives for future learning.
Mentorship was shown to be extremely important in specialty training. Mentors have different roles, from organisational ones to very personal ones (being a friend); for most of the trainees, the mentor also became a role model. According to Hesketh et al., being a good mentor means being a good clinician, knowing the programme and goal of the training, but also having appropriate personal characteristics (26).These expectations have been found in other literature – the mentor’s ability to be a personal ‘role model’ and coaching is coupled with their clinical and coordinative work with family medicine trainees (27). From the trainees’ feedback it seems that the main mentors fulfilled all the expectations. This is very encouraging, especially if we consider that there was a severe shortage of appropriate main mentors, and that some family physicians were mentoring up to three trainees. The main shortcoming of the clinical mentors was their lack of knowledge of the content and specialisation plan. The clinical mentors did not have previous experience with mentoring family medicine trainees and sometimes had unrealistic expectations (e.g., they expected too much of the trainees’ clinical knowledge, and did not understand the role of the family medicine specialist in the system). Evidently, the problem is not only local, as we can find the same opinion in the literature, where Bulc et al. and Švab et al. emphasise that all teachers participating in the training of family medicine should be familiar with the basic characteristics of the discipline and theoretical framework of family medicine (28, 29). For the future, it seems there is a need to organise training for clinical mentors as well, so that they can become familiar with the aims, content, process and methods of teaching and giving feedback to the trainees (27). Further development of the programme and content for all clinical rotations is also necessary.
In Slovenia, which has almost 400 trainees at the moment, comments from the trainees are similar to those in Montenegro. To improve satisfaction with the clinical rotations and clinical mentorship, several regional coordinators have been implemented in the training programme. They represent a coordinative and pedagogical structure of working in the local environment, where most of the clinical rotations are performed, to make practical improvements in the training process (27, 30).
The recognition of family medicine as an academic discipline, and the support of the University and the Faculty of Medicine of Montenegro in establishing a Department of Family Medicine, would overcome many obstacles in the implementation of the programme. A department with professional personnel, an organisational structure and financial resources is necessary for a stable situation, which can offer conditions for improvement.
In this study, we assessed the process of implementation of speciality training in family medicine. However, our study has several limitations. We included trainees who had not yet passed the specialty exam; these trainees (10 out of 22) may have assessed the process of training more positively, in order to try to present themselves positively to their future examination committee.
Due to the reasons of feasibility (trainees living in different parts of the country, where some areas are not very well connected to the capital), we sent a questionnaire with open-ended questions to participants by post and asked them for written answers. Some of the participants gave us no answers or very general statements in reply to the open questions, mainly to the questions related to the organisation of the specialty course and the assessment of the exam.
As the study was not anonymous, the participants tended not to answer the open-ended questions or gave relatively general, not very critical answers. Other methods, e.g., a focus group study or more in-depth one-to-one interview with all the trainees, could have given us a more detailed and accurate view of the satisfaction of the trainees with the programme, the process of training and exam, but these two methods were not feasible on this occasion.
The assessment of quality is a complex process; many subjects can assess quality, including patients, colleagues, professional organisations, health care authorities and society in general, which assesses social acceptance. In our study, the evaluation of the implementation of the specialty training was made only by the trainees and the coordinator of training. The view of the coordinator of training is subjective and, from that point of view, biased.
New teaching methods, mentorship and formative assessment were introduced into the education process for the first time. The implementation of the evidence into the medical training, considering the feedback of the trainees and teachers, helps to change educational practice towards the goal. This study represents the first attempt to evaluate the process, results and satisfaction with the training programme. In the future, other methods should be implemented, such as a more structured evaluation of the progress of the trainee.