Practice and Challenge of International Peer Review: A Case Study of Research Evaluation of CAS Centers for Excellence

At present, China is a front-runner in the number of scientific outputs, but the quality remains to be improved. In order to respond to international competition and to satisfy domestic development needs, the orientation of China’s science and technology evaluation has changed from encouraging international publications to pursuing excellence and encouraging science and technology for promoting development. One of the policies is the introduction of international experts in science and technology evaluation when conditions permit. For instance, the Opinions of Deepening the Reform of Project Review, Talent Evaluation and Institutional Assessment (the “Sanping Opinions”) printed and released in 2018, proposes to “gradually introduce international peer review in some frontier and basic sciences areas in appropriate proportions” and “enhance performance evaluation of national science and technology plans and introduce international evaluation as needed”. In practice, some advanced institutions and universities in China have explored ways to implement international evaluation in various forms since 2000. The Chinese Academy of Sciences (CAS) took the lead in exploring and implementing international peer review at the institutional level. This initiative goes back to 2004 when a pilot international peer review of research institutes was carried out during Phase II of the Knowledge Innovation Project (Peng, 2013). In recent years, universities such as Shanghai Jiao Tong University (Zhang, 2019), Tsinghua University (Cheng, 2019), and Peking University have been exploring international evaluation with regard to the promotion of researchers; moreover, the Beijing Institute of Technology explored and implemented international evaluation of disciplinary (school or department) developments. In 2011, coinciding with the 25^th anniversary of the establishment of the National Natural Science Foundation of China (NSFC), an international peer review of the performance of science funding and management was successfully completed (Wu, Liang, & Zheng, 2012), which is the largest comprehensive science and technology performance evaluation so far in China. Internationally, world-known research institutes have explored and implemented various forms of international evaluation very early, for example, evaluation of research institutes and selection of academic leaders by Max Planck Society (Shuai, 2004) and evaluation of discipline-specific research quality by Research Excellence Framework (REF) (Xu, Liu, & Li, 2014). There is no doubt that international evaluation represents a new trend in science and technology evaluation (Marušić, Brezis, & Squazzoni, 2017). Is the time ripe to conduct international peer reviews in China? More importantly, how to design and conduct effective international peer reviews in China? This paper starts with these two critical policy questions about international peer review, and intends to find an answer through a case study of international evaluation practices of acceptance assessment of CAS Center for Excellences. Then, it will put forward considerations about China’s future international evaluations, in order to accumulate knowledge for the development of theories and methods related to such evaluations and provide a reference for future science and technology management exercises.

Question 1: Is the time ripe to conduct international peer reviews in China? First, it is meaningless to conduct an international peer review when the research level is too low. Over the forty years since its reform and opening-up, China has raised its influence and made achievements of world interest in science and technology. For instance, China maintains an annual average 10% growth of investment in science and technology, and ranked second in the world in the number of SCI papers published and in citations from 2007 to 2017 (ISTIC, 2019). In some basic and frontier research areas, China’s research is already at the international level, and it is meaningful and possible to conduct an international peer review in these areas. Second, can international peers understand the difference of research management forms and organizational forms of science research in China? Some may argue that international peers know little beyond a vague picture of the enormous and complicated Chinese S&T system. Therefore, they can hardly play effective roles and provide valuable opinions. This leads to Question 2, which is about the design of international peer reviews.

Question 2: How to design and conduct effective international peer reviews? From the point of a researcher manager’s view, international peer review provides useful suggestions by bringing international experience of peer panels into the picture. The challenge is to ensure that the expertise of evaluation panels is congruent with the research interests of the research groups (Rousseau, 2017). On the one hand, it is critical to well design the evaluation procedure and prepare sufficient evaluation materials such as the background introduction of the purpose, content, criteria, supporting documents (Gunashekar, Wooding, & Guthrie, 2017) of a particular evaluation. On the other hand, not always visible conflicts of interest may lead to toning down negative issues. In that case, the research manager needs to find out hidden meanings among words of excessive praise.

These two critical questions raised hot debates before conducting an international peer review of CAS Centers for Excellence. To some extent CAS’s practice provide a possible answer, which may provide useful insights to others.

In 2017, a total of 25 representative achievements, 31 subject leaders and 80 key members from eight Centers for Excellence were reviewed by international panels. From the feedback from CASHQ and the centers, the international peer review of CAS’s Centers of Excellence turned out to be satisfactory. Firstly, international review promoted the construction of four categories of institutions and the understanding of Center of Excellence. The entire review is a process for the leaders and researchers to “re-understand” the planning and key research fields of the Centers for Excellence. In the process of preparing evaluation materials, all the centers and their respective directors, subject leaders and researchers teased out the strategic orientation and major achievements of the research institutes as well as the relationship between various parts of the research work again, so as to deepen their understanding of the Centers of Excellence. Secondly, international peer review helps the development of CAS Centers of Excellence. A number of promising research work and distinguished researchers are identified, which provide valuable opinions for decision-making on resources allocations. Constructive opinions on the development of key research fields of the centers, especially their domestic and international status are pointed out, which are conducive for further improvement of the development plan of the centers. Thirdly, the philosophy of pursuit of research excellence is further promoted through the international peer review. During the on-site review, the international experts paid more attention to the quality and originality of research work rather than quantitative figures. Sometimes experts assess the current research status and level of researchers by asking them whether they know the best performers, competitors and their research content well in their respective research fields. These questions also help researchers to broaden their horizon, think deeply and understand their own advantages and disadvantages. Last but not least, there is no doubt that international reviews promote international exchanges and cooperation (DaiWai, 2019), which also provides an opportunity for the centers to promote themselves, advance the centers to the international forefront and expand international cooperation.

What is more, a questionnaire is conducted to collect opinions about the design of international peer view. Based on 42 valid questionnaires (86 questionnaires were issued, with a 49% recovery rate), the international experts believed that the international review provided a guarantee of independent and unbiased evaluation. According to their opinions, it is important to assess the Centers for Excellence by senior scientists from first rated international institutions as they can best review the status and assess the potential for future developments. Therefore, it is very important to collect evaluation opinions and suggestions through such an international review as a way to increase the quality and motivation of research work. From a personal perspective of experts, this review also provided them with a systematic understanding of CAS and its research institutes. Some experts said, “We also had exchanges and cooperation with some teams in the past, but that was just at the level of a research group or laboratory. This review provides comprehensive and systematic understanding of the overall situation of CAS.”

Moreover, each panel was required to give a score from 1 (the lowest) to 5 (the highest) in terms of panel member structure, preparation of evaluation materials, evaluation forms and standards, organization and implementation of review and finally the procedures and arrangements of review. It turns out that 83% of the international experts believed that the overall operation of international review is good or even excellent. In particular, the highest score is given to the organization and implementation of the international review, which was rated excellent or good by 90% of the international experts. The forms and standards of the international review received the lowest scores, but 79% of the international experts still rate them as excellent or good. In three other aspects, namely the procedures and arrangement of international review, preparation of materials and structure of expert group, 86%, 81% and 86% of the international experts rate them as excellent or good, as shown in Figure 2. In addition, the international experts made the following suggestions about the organization and implementation of international peer review: in terms of procedure design and arrangement, more time should be reserved for the preparatory meeting to gain a better understanding of the background. This also applies to one-on-one discussions. It is important to simplify assessment standards and forms and keep them consistent with international practice. In terms of the preparation of materials, it is necessary to collect more basic data such as personnel and funding information. Last, female scientists and more junior peer experts must be included.

Figure 2

Responding to the two critical questions proposed before, based on the example of CAS, we conclude that international peer review is effective in identifying problems in research activities, determining research level and status, and improving the development of research units with the help of international experts’ intelligence and experience. In order to gain maximal benefit from international panel members, the organizer must pay a lot of attention to the panel selection. On the one hand, the research level and reputation is the key factor when selecting appropriate experts. On the other hand, an expert is also expected to have strategic vision as well as rich evaluation experiences. In some cases, the authority and personal charm of the panel leader can lead to a smooth and rather pleasant experience. Particularly, ethnic Chinese experts play a great role in providing background information about the science and technology system in China. Other considerations during the on-site review such us informal communication with the panel leader in advance, and evaluation standards and language complying with international practices can improve the effectiveness and efficiency of international peer review as well.

In China, science and technology evaluation is at a critical stage. There is an urgent need for shifting the development path, that is, curbing the trend of excessive quantitative evaluation and developing a more quality-focused evaluation system to guarantee more breakthroughs in science and technology and transform from “quantity oriented to quality oriented” (Cheng, Li, & Xu, 2018). International peer review may provide a solution to address the problem of quantitative oriented research evaluation. In order to maximize the role of international peer review in research management, this paper puts forward the following points for discussion.

First, international peer review needs adaptation to specific circumstances to better play its role. International peer review is often applied in universities with which international panels are more familiar than with a large institute such as CAS. This shows that international peer review must adapt according to management needs and evaluation objectives. This also meets the current purpose of developing science and technology. In addition to serve the growth of scientific knowledge, science and technology has to contribute to economic and social development. Therefore, international peer review is required to evaluate not only the excellence and originality of research work, but also its contribution to economic and social progress. CAS has adopted international peer review in its practices, so that it can coordinate with management experts and better serve the needs of science and technology management.

Secondly, the range of applications and implementations of international peer review should be clear. China has issued relevant policies to encourage international peer review when appropriate, but the difficulty lies in how to define the range of applications and implementations of international peer review. Generally, international peer review is applicable to evaluate innovation and influence of non-confidential research work in frontier and basic scientific research areas, and to identify problems and make suggestions. Specifically, when referring to the evaluation of different subjects such as scientists, teams, organizations and projects, one needs to further explore which are suitable for international peer review.

Thirdly, theories and methods of international peer review need improvement. The advantage of international peer review lies in exerting the role of international intelligence and standards, and reducing the influences of quantitative indicators on experts. Given this, theories and methods of international peer review need urgent improvement. For example, how to avoid conflicts of interest while ensuring that selected experts are familiar with the evaluated objects. The use of evaluation results is another problem. If international experts are capable but choose not to provide honest and complete review opinions, international peer review will only play a weak role in promoting management work, allocation of resources and organizational adjustment. Therefore, efforts should be made to further explore and improve theories and methods concerning international peer review as well as normative and standard evaluation procedures (the Standard Evaluation Protocol proposed in Netherlands provides a good example (KNAW, VSNU, NOW, 2014)), in order to improve the efficiency and effect of international peer review and to support better decision-making.