Evaluation of the training program for p16/ Ki-67 dual immunocytochemical staining interpretation for laboratory staff without experience in cervical cytology and immunocytochemistry

p16/Ki-67 dual immunocytochemical staining (DS) is considered easy to interpret if evaluators are properly trained. However, there is no consensus on what constitutes proper training. Authors have used different training approaches in studies investigating inter-observer reproducibility and accuracy of DS.1, 2, 3, 4, 5, 6 Most training protocols described in these studies consisted of initial and additional training. The initial training was provided by the manufacturer, however, it was not exactly the same in all cases except that it was completed by a proficiency test. The information on the initial training is sparse. Three of the above mentioned studies do not describe the initial training.1, 2, 4 In two studies, participants were shown 155 or 406 microscope-projected images, while in the third study participants examined a teaching set of slides.3 The number of cases in the teaching set is not mentioned. In all three studies the training was completed in one or two-half day sessions.3, 5, 6 Four authors described additional training which consisted of evaluating from 80 to 469 slides as well as reviewing and discussing cases with discrepant results.1, 2, 3,6 Agreement in DS interpretation among evaluators improved after additional training in all three studies (kappa range: 0.43‒0.73 compared to 0.50‒0.87).1, 2, 3,6 In the study of Wentzensen et al.2 agreement was evaluated only at the end of the study.

In our recently published study which assessed reproducibility of the DS test we described a training protocol which was designed to introduce DS in three Slovenian cytopathological laboratories participating in the national organized cervical cancer screening program.6 At the time we designed the protocol we found only one similar study by Waldstrom et al.1 The results of our study demonstrated that initial training by the manufacturer was not enough for achieving accurate results of DS interpretation. Furthermore, the manufacturer provides training when an institution is ready to implement the test. Later on, when we have the need to teach additional personnel, we have to have our own training protocol which will assure the students receive the necessary expertise.

On the basis of the results of our previous study, we proposed a training protocol for staff inexperienced in DS reading.6 In the present study, we aimed to test the proposed training protocol for DS interpretation on two students inexperienced in DS reading and to discern how improvement in DS evaluation influences the time needed for DS reading of one slide. An additional end point of the study was also monitoring the performance of the senior cytotechnologist involved in teaching DS interpretation to new personnel.

The results of our study confirmed that the training protocol we have used was adequate for teaching the interpretation of p16/Ki-67 DS. At the end of the training one student was competent for independent DS interpretation as evidenced by comparable accuracy results and by good agreement between her results and those of the reference. Towards the end of the training this student needed 5.2 min for evaluating one slide. The training protocol is suitable also for monitoring the performance of the teacher.

We do not have an explanation as to why the results of the other student showed a statistically significant difference in agreement and accuracy when compared to the results of the reference. Since environmental factors were the same for both students, it seems that internal factors played an important role in performance difference. It is well known that interpreting slides is partly a subjective method. Furthermore, in our previous study we demonstrated a number of reasons that contributed significantly to the difficulty of DS interpretation.6 Weak p16 staining, less preserved cell morphology and strong background staining were important drawbacks of decision making. Similar observations were also made by McMenamin et al.4 and Benevolo et al.5 An additional important information from our previous study was also the fact that 45% of cases which were marked as suspicious for p16/Ki-67 positivity had only one such cell.6 Therefore, this type of training is not suitable for every person without prior knowledge of DS evaluation. However, by monitoring student’s results, we can assess if additional experience to reach the necessary expertise is needed.

Most articles which describe agreement and accuracy of DS interpretation among observers briefly mention the training program they have used. Only the training programs in three studies had some similarities with our own.1, 2, 3 However, the initial training provided by the manufacturer is not described in detail in any of them. The secondary training can be compared to ours because it consisted of reviewing a certain number of slides (80, 150 and 469) and of discussing discrepant cases of the first viewing. In the second reading, Allia et al.3 mention evaluation of 350 new slides, while Wentzesen et al.2 used 480 new slides, however, not all reviewers red all of them. In the study of Waldstrom et al.1 they randomly selected 185 slides from set of 469 slides from the first reading. Only the study of Allia et al.3 included three evaluators inexperienced in cervical cytology (two medical students and one biologist) in addition to the four experienced ones.

Unfortunately, we can compare our results to those of Allia et al.3 only to a limited extent due to differences in methodology. Allia et al. compared agreement between results of four evaluators with experience in DS evaluation and between results of three evaluators without experience. Agreement improved after additional training in each group. In our study, the population of 118 women from the first arm of the study had a higher percentage of histologically confirmed CIN2+ compared to the population of 383 women in the second arm (58.5% vs. 20.1%). This difference in the prevalence of the disease has to be taken into account when comparing the results of primary and secondary evaluation, especially Kappa values and predictive values of a test. In the Allia’s et al. paper, the study population contained 14.7% of women with histological diagnoses of CIN2+.3 Since the populations of women in the two studies were not totally alike, we can compare only the end results indirectly. At the end of Allia’s et al. study, the specificity of DS for CIN2+ was 66.7% for the experienced evaluators, while the students reached a specificity of 60.5%, a difference of 5.2% points. On the basis of these data Allia et al. concluded that DS evaluation “can be performed even by staff not trained in the morphological interpretation of cytology” after a short training phase.3 At the end of our study the difference in specificity between the reference and the successful student was 1.6% points and 16.7% points for the unsuccessful student. Therefore, we agree with Allia et al. that it is possible to perform DS evaluation with personnel not experienced in cervical cytology, however, not after a short training period.

If we translate 97 hours needed for reviewing 501 slides into 12 days with an eight-hour working day, the whole training period in our study would last 19 days. The evaluation of slides after initial training, as well as partly after additional training, has to be considered part of the learning process. This is clearly demonstrated in the graph of continual monitoring of the students’ accuracy results. For the successful student the specificity continued rising during the evaluation of the first 118 slides. It continued to rise also for a period after the secondary training until approximately the time when the student evaluated roughly 350 slides altogether. At this point the accuracy results of the student were very similar to the results of the reference (same sensitivity, 87.0%) and slightly lower specificity (84.4 vs. 85.7%). The rest of the time used in the evaluation of the last 150 slides was necessary for monitoring the student’s performance. The necessity of such action has proven to be correct in the case of the other student. Her accuracy results came close to those of the reference after she evaluated approximately 250 slides, however, her performance started to decline afterwards.

For monitoring the students’ progress in DS evaluation it is advisable to use more than one criterion of successfulness. Using only agreement between the students and the reference will be reliable only when positive and negative predictive values of the reference results are high. However, these values depend partly on the percentage of CIN2+ cases within the population of women from which samples for DS evaluation originate. Therefore, comparing accuracy results between students and reference is beneficial because it will demonstrate more exactly where a particular student has difficulties in DS interpretation. For example, high sensitivity and low specificity point to the fact that a student is signing too many cases as positive. An additional measure of a student’s successfulness is also the time needed to evaluate one slide.

Since both students were inexperienced in DS evaluation it is reasonable that the time needed to evaluate one slide was progressively decreasing. The decrease was sharper towards the beginning and less pronounced latter on. In addition to the agreement and the accuracy results, time needed for evaluating one slide also showed the difference between the two students. The ultimately unsuccessful student reached faster the point after which her time per slide started to decrease very slowly, compared to the successful student. However, during the evaluation of the last 100 slides, the successful student needed significantly less average time per slide compared to the other student (5.2 min vs. 8.2 min). Only McMenamin et al. made a quick mention of the time needed per slide evaluation.4 They reported that their experienced DS evaluators needed less than 1 minute for evaluating a clearly positive slide and 3-4 min for more challenging ones. Their evaluation time per slide is shorter than in our study not only because our students were unexperienced but mainly because we used conventional smears while in the study of Mc Menamin et al. ThinPrep specimens were used.4

In addition to teaching DS interpretation to students, our training program was designed also to monitor the performance of the cytotechnologist involved in the teaching process. We believe that teacher monitoring is an important element of the training program which helps to assure the students will receive the best training. The intraobserver agreement was within the range of “very good” in both arms of our DS evaluation (OPA 100.0% and 98.7%, kappa 1.000 and 0.937, respectively). This result is even slightly better than the results obtained by McMenamin et al.4 where agreement for three cytotechnologists was 82.8% (0.65), 83.8% (0.67) and 94.9%, (0.91), respectively.

In conclusion we would like to say that teaching p16/Ki-67 interpretation should be a closely monitored process in which students’ results have to be compared to the reference results with known accuracy for CIN2+. The students’ progress has to be monitored using several criteria like agreement, accuracy and time needed for evaluating one slide. The monitoring process has to continue for a while after students reach satisfactory results in order to assure a continuous good performance. Monitoring of teacher’s performance is also advisable.