Which of the favorite optimal cut-off determination methods is preferable for the ordinal response data? A simulation study


The aim of this study is to investigate the performance of the optimal cut-off methods, which are generally used for the diagnostic tests with the continuous response, for the tests with the ordinal response. Diagnostic accuracy studies examine the ability of a diagnostic test to discriminate between the patients with and without the condition. For diagnostic tests with a continuous response, it is important in practice to calculate the optimal cut-off point that can differentiate patients and healthy individuals. There are many methods proposed in the literature to obtain the optimal cut-point value for continuous test results. The Youden index, the point closest-to-(0, 1) corner in the ROC plane approach, the concordance probability, and the minimum P-value approach are commonly used methods to determine optimal-cut-point. But the researches examining the performance of these methods in the setting of the ordinal response tests are lacking in the literature. So, we compared the mentioned optimal cut-off methods for the ordinal response data by the way of simulation design by considering the sample size and the balance of groups as simulation conditions. The sample sizes of the diseased and non-diseased group were set (50, 50), (100, 100), and (200, 200) for balanced design and (50, 100), (50, 150) and (50, 200) for unbalanced design. For each scenario, 1000 repeats were generated. The differences between the estimated and the true cut-off points (biases) were calculated. All these methods overestimated the true cut-off point, but the median biases of the methods were varying. For the unbalanced design, the same result was relevant but for the balanced design, the minimum P-value approach had a median bias as 0 while others have 1.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Andrich D. (1978). “A rating formulation for ordered response categories.” Psychometrika, 43(4): 561-573.

  • Habibzadeh F., Habibzadeh P., and Yadollahie M. (2016). “On determining the most appropriate test cut-off value: the case of tests with continuous results.” Biochemia Medica. 26(3):297–307.

  • Liu X. (2012). “Classification accuracy and cut point selection.” Stat Med. 31:2676–286.

  • Lopez-Raton M., Rodriguez-Alvarez M.X, Cadarso-Suarez C. and Gude-Sampedro F. (2014). “OptimalCutpoints: An R Package for Selecting Optimal Cutpoints in Diagnostic Tests.” Journal of Statistical Software 61(8):1-36.

  • Magis D. and Raiche G. (2012). “Random Generation of Response Patterns under Computerized Adaptive Testing with the R Package catR.” Journal of Statistical Software, 48(8), 1-31. doi:10.18637/jss.v048.i08

  • Miller R. and Siegmund D. (1982). “Maximally selected Chi square statistics.” Biometrics, 38:1011–1016.

  • Pepe MS. (2003). “The Statistical Evaluation of Medical Tests for Classification and Prediction.” Oxford University Press: New York.

  • Perkins N.J. and Schisterman E.F. (2006). “The inconsistency of optimal cutpoints obtained using two criteria based on the receiver operating characteristic curve.” Am. J. Epidemiol. 163:670–675.

  • R Core Team (2018). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.

  • Rota M. and Antolini L. (2014). “Finding the optimal cut-point for Gaussian and Gamma distributed biomarkers.” Computational Statistics & Data Analysis,69:1–14.

  • Youden WJ. (1950). “Index for rating diagnostic tests.” Cancer, 3:32–35.

  • Zhou X.H., Obuchowski N.A., and McClish D.K. (2011). “Statistical Methods in Diagnostic Medicine.” Wiley: New York


Journal + Issues