Multivariate approach to imposing additional constraints on the Benefit-of-the-Doubt model: The case of QS World University Rankings by Subject

Open access


Composite indexes have become a valuable asset for stakeholders as they provide ranks of entities and information upon which decisions are made. However, certain questions about their development procedure have been raised recently, especially regarding the weighting process. To tackle the observed issue, in this paper we propose a new multivariate approach for defining weights. Namely, the model based on the Data Envelopment Analysis (DEA), the Benefit-of-the-Doubt (BoD) model, has been used with significant success in the process of composite index development. On the other hand, the Composite I-distance Indicator (CIDI) methodology stands out as an impartial method for assigning weights to indicators. By combining these two approaches, some of the limitations of the original BoD model could be overcome. As a result, new entity-specific weights which maximize the value of the composite index can be proposed. As a case study, we analysed the Quacquarelli Symonds (QS) World University Rankings by Subject in the field of statistics and operational research. The obtained results, which are based on the data-driven weights, can provide new insights into the nature of the observed ranking. The approach presented here might provoke further research on the topic of composite index weights and on the university rankings by subject.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • 1. Bornmann L. Marx W. (2014). How to evaluate individual researchers working in the natural and life sciences meaningfully? A proposal of methods based on percentiles of citations. Scientometrics Vol. 98 No. 1 pp. 487-509.

  • 2. Charnes A. Cooper W.W. Rhodes E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research Vol. 2 No. 6 pp. 429-444.

  • 3. Cherchye L. Moesen W. Rogge N. Van Puyenbroeck T. (2007). An introduction to ‘benefit of the doubt’composite indicators. Social Indicators Research Vol. 82 No. 1 pp.111-145.

  • 4. Davenport T. H. Patil D. J. (2012). Data Scientist: The Sexiest Job of the 21st Century. Harvard Business Review 70 October 2012.

  • 5. Dobrota M. Bulajic M. Bornmann L. Jeremic V. (2016). A new approach to QS University Ranking using composite I-distance indicator: uncertainty and sensitivity analyses. Journal of the Association for Information Science and Technology Vol. 67 No. 1 pp. 200-211.

  • 6. Dobrota M. Jeremic V. (in press). Shedding the Light on the Stability of University Rankings in the ICT Field. IETE Technical Review.

  • 7. Dobrota M. Martic M. Bulajic M. Jeremic V. (2015). Two-phased composite I-distance indicator approach for evaluation of countries’ information development. Telecommunications Policy Vol. 39 No. 5 pp. 406-420.

  • 8. Federkeil G. (2015). Doing Field-based Rankings: Lessons Learned from U-Multirank and CHE-rankings in Subject and Discipline Related Rankings - a More Inclusive Approach to University Performance (IREG 2015)

  • 9. Intelligence Unit. (2015). QS World University Rankings by Subject. Available on: [9 January 2015]

  • 10. IREG. (2015). Rankings by Subject. Available on: [28 December 2015]

  • 11. Ivanovic B. (1977). Classification theory. Belgrade: Institute for Industrial Economics.

  • 12. Jeremic V. Bulajic M. Martic M. Radojicic Z. (2011). A fresh approach to evaluating the academic ranking of world universities. Scientometrics Vol. 87 No. 3 pp. 587-596.

  • 13. Jovanovic-Milenkovic M. Brajovic B. Milenkovic D. Vukmirovic D. Jeremic V. (2015). Beyond the equal-weight framework of the Networked Readiness Index a multilevel Idistance methodology. Information Development. In press.

  • 14. Marginson S. (2014). University rankings and social science. European Journal of Education Vol. 49 No. 1 pp. 45-59.

  • 15. Maričić M. Bulajić M. Dobrota M. (2016a). The alteration of U21 ranking methodology: from expert-driven to data-driven weighting scheme. Proceedings of the SYMORG 2016 Conference June 10-13 Zlatibor Serbia pp. 84-91.

  • 16. Maricic M. Bulajic M. Dobrota M Jeremic V. (2016b). Redesigning The Global Food Security Index: A Multivariate Composite I-Distance Indicator Approach. International Journal of Food and Agricultural Economics Vol. 4 No. 1 pp. 69-86.

  • 17. Maricic M. Kostic-Stankovic M. (2016). Towards an impartial Responsible Competitiveness Index: a twofold multivariate I-distance approach. Quality & Quantity Vol. 50 No. 1 pp. 103-120.

  • 18. Melyn W. Moesen W. (1991). Towards a synthetic indicator of macroeconomic performance: unequal weighting when limited information is available. Public Economics Research Paper 17 CES KU Leuven.

  • 19. Mizobuchi H. (2014). Measuring world better life frontier: a composite indicator for OECD better life index. Social Indicators Research Vol. 118 No. 3 pp. 987-1007.

  • 20. Nardo M. Saisana M. Saltelli A. Tarantola S. Hoffman A. Giovannini E. (2005). Handbook on constructing composite indicators: methodology and user guide. (No. 2005/3). OECD publishing

  • 21. Perišić A. (2015). Data-driven weights and restrictions in the construction of composite indicators. Croatian Operational Research Review Vol. 6 No. 1 pp. 29-42.

  • 22. QS. (2015a). QS World University Rankings by Subject: Methodology. Available on: university-rankings-subject-methodology [8 January 2016]

  • 23. QS. (2015b). QS World University Rankings by Subject 2015 - Statistics & Operational Research. Available at [Accessed 15 January 2016]

  • 24. Radojicic M. Savic G. Radovanovic S. Jeremic V. (2015). A novel bootstrap dba-dea approach in evaluating efficiency of banks. Scientific Bulletin" Mircea cel Batran" Naval Academy Vol. 18 No. 2 pp. 375-384.

  • 25. Rauhvargers A. (2013). Global university rankings and their impact: Report II. pp. 21-23. Brussels: European University Association.

  • 26. Rauhvargers A. (2014). Where are the global rankings leading us? An analysis of recent methodological changes and new developments. European Journal of Education Vol. 49 No. 1 pp. 29-44.

  • 27. Rogge N. (2012). Undesirable specialization in the construction of composite policy indicators: The Environmental Performance Index. Ecological indicators Vol. 23 pp.143-154.

  • 28. Saisana M. Saltelli A. (2014). JCR statistical audit of the WJP Rule of Law index 2014. In World Justice Project: The World Justice Project Rule of Law Index 2014 pp. 188-197

  • 29. Siwinski W. (2015). The era of rankings by subject is coming [Online]. Available on: [12 January 2016]

  • 30. UN. (2014). The Post ‐ 2015 Development Agenda: The Role of Statistical Community. Available on: [5 January 2016]

  • 31. Ziman J. (2000). Real Science. What it Is and What it Means. Cambridge University Press Cambridge.

  • 32. Zornic N. Bornmann L. Maricic M. Markovic A. Martic M. Jeremic V. (2015). Ranking institutions within a university based on their scientific performance: A percentile-based approach. El Profesional de la informacion Vol. 24 No. 5 pp. 551-566.

Journal information
Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 164 90 0
PDF Downloads 85 66 2