A code reviewer assignment model incorporating the competence differences and participant preferences

Open access


A good assignment of code reviewers can effectively utilize the intellectual resources, assure code quality and improve programmers’ skills in software development. However, little research on reviewer assignment of code review has been found. In this study, a code reviewer assignment model is created based on participants’ preference to reviewing assignment. With a constraint of the smallest size of a review group, the model is optimized to maximize review outcomes and avoid the negative impact of “mutual admiration society”. This study shows that the reviewer assignment strategies incorporating either the reviewers’ preferences or the authors’ preferences get much improvement than a random assignment. The strategy incorporating authors’ preference makes higher improvement than that incorporating reviewers’ preference. However, when the reviewers’ and authors’ preference matrixes are merged, the improvement becomes moderate. The study indicates that the majority of the participants have a strong wish to work with reviewers and authors having highest competence. If we want to satisfy the preference of both reviewers and authors at the same time, the overall improvement of learning outcomes may be not the best.

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • [1] Chen Y. Fan Z. P. A Method for proposal-reviewer assignment in proposal review based on the match degree of research discipline Chinese Journal of Management Science 19 2 2011 169-173 (in Chinese).

  • [2] Cook W. D. Golany B. Kress M. Penn M. Raviv T. Optimal allocation of proposals to reviewers to facilitate effective ranking Management Science 51 4 2005 655-661.

  • [3] Devito Da Cunha A. Greathead D. Does personality matter? An analysis of code-review ability Communications of the ACM 50 5 2007 109-112.

  • [4] Drexl M. Irnich S. Solving elementary shortest-path problems as mixed-integer programs OR spectrum 36 2 2014 281-296.

  • [5] Fagan E. Design and code inspections to reduce errors in program development IBM System Journal3 1976 182-211.

  • [6] Karimzadehgan M. Zhai C. X. Integer linear programming for constrained multi-aspect committee review assignment Information Processing and Management 48 4 2012 725-740.

  • [7] Li X. Using peer review to assess coding standards–a case study In Frontiers in education conference 36th annual San Diego CA USA 2006 9-14.

  • [8] Li X. Incorporating a code review process into the assessment In the 20th Annual Conference of the National Advisory Committee on Computing Qualifications Nelson New Zealand 2007 125-131.

  • [9] Li X. Watanabe T. Automatic paper-to-reviewer assignment based on the matching degree of the reviewers Procedia Computer Science 22 2013 633-642.

  • [10] Long C. Wong R. C. W. Peng Y. Ye L. On good and fair paper-reviewer assignment In IEEE 13th International Conference on Data Mining (ICDM’2013) 2013 1145-1150.

  • [11] Martello S. Pulleyblank W. R. Toth P. de Werra D. Balanced optimization problems Operations Research Letters 3 5 1984 275-278.

  • [12] Meyer B. Design and code reviews in the age of the internet Communications of the ACM 51 9 2008 66-71.

  • [13] Saaty T. L. How to make a decision: the analytic hierarchy process European journal of operational research 48 1 1990 9-26.

  • [14] Sun Y. H. Ma J. Fan Z. P. Wang J. A hybrid knowledge and model approach for reviewer assignment Expert Systems with Applications 34 2 2008 817-824.

  • [15] Tayal D. K. Saxena P. C. Sharma A. Khanna G. Gupta S. New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions Applied intelligence 40 1 2014 54-73.

  • [16] Topping K. Peer Assessment Between Students in Colleges and Universities Review of Educational Research 68 3 1998 249-276.

  • [17] Tsang E. W. K. Is this referee really my peer? A challenge to the peer-review process Journal of Management Inquiry 22 2 2013 166-171.

  • [18] Turner S. A. Peer review in CS2: the effects on attitudes engagement and conceptual learning Doctoral Dissertation of Virginia Polytechnic Institute and State University 2009 Retrieved from http://scholar.lib.vt.edu/theses/available/etd-08272009-003738/unrestricted/Turner_SA_D_2009.pdf

  • [19] Wang F. Shi N. Chen B. A comprehensive survey of the reviewer assignment problem International Journal of Information Technology and Decision Making 9 4 2010 645-668.

  • [20] Wang Y. Q. Li H. Feng Y. Q. Jiang Y. Liu Y. Assessment of programming language learning based on peer code review model: Implementation and experience report Computers & Education 59 2 2012 412-422.

  • [21] Wang Y. Q. Li H. Sun Y. N. Jiang Y. Yu J. Learning outcomes of programming language courses based on peer code review model In the 6th International Conference on Computer Science & Education August 3-5 SuperStar Virgo Singapore ThC 5.47 2011 751-754.

  • [22] Wang Y. Q. Li Y. J. Collins M. Liu P. J. Process improvement of peer code review and behavior analysis of its participants ACM SIGCSE Bulletin 40 1 2008 107-111.

  • [23] Wang Y. Q. Yang F. Liu P. J. Collins M. Quality assurance of peer code review process: A computer science based strategy Zhongshan Daxue Xuebao/Acta Scientiarum Natralium Universitatis Sunyatseni 46(suppl) 2007 116-120.

  • [24] Xu Y. H. Ma J. Sun Y. H. Hao G. Xu W. Zhao D. T. A decision support approach for assigning reviewers to proposals Expert Systems with Applications 37 10 2010 6948-6956.

Journal information
Impact Factor

CiteScore 2018: 0.61

SCImago Journal Rank (SJR) 2018: 0.152
Source Normalized Impact per Paper (SNIP) 2018: 0.463

Mathematical Citation Quotient (MCQ) 2017: 0.02

Cited By
All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 476 333 56
PDF Downloads 111 92 5