At present, China is a front-runner in the number of scientific outputs, but the quality remains to be improved. In order to respond to international competition and to satisfy domestic development needs, the orientation of China’s science and technology evaluation has changed from encouraging international publications to pursuing excellence and encouraging science and technology for promoting development. One of the policies is the introduction of international experts in science and technology evaluation when conditions permit. For instance, the
These two critical questions raised hot debates before conducting an international peer review of CAS Centers for Excellence. To some extent CAS’s practice provide a possible answer, which may provide useful insights to others.
In the light of new national and societal requirements, CAS announced the
Basic feature of CAS four categories of research institutes.
Centers for Excellence | Innovation Institutes | Centers for Mega-science | Featured Institutes | |
---|---|---|---|---|
Research | • Focus on original | • Focus on economic development and | • Focus on building and running | • Focus on serving for sustainable |
orientation | innovation of science and | national security | large-scale science and | development of society |
technology | technology innovation platforms | |||
Research | • Basic and frontier sciences | • Important foundation and technological | • Design, construct and operate | • Indispensable research areas for |
areas | • Orientation towards major | direction for promoting industrial | international advanced | sustainable social development |
scientific problems. | upgrading, breaking through bottlenecks, | large-scale scientific devices | • Interdisciplinary study of natural | |
influencing or restricting national | • Comprehensive interdisciplinary | science and social science | ||
security | and cutting-edge sciences relying | • long-term observation and | ||
• High risk technological frontier | on large scientific devices | continuous accumulation related | ||
directions | basic research work | |||
Team | • Internationally or | • Strategic scientists, academic and | • High level teams in scheme | • Internationally recognized scientific |
composition | domestically recognized | technological leaders with broad vision | design, technology research and | and technology experts who also |
leading scientists | and good grasp of technological | development and engineering | understand national demand | |
• High-level academic | direction and organizational tackling key | organizations | • An integrated research team with | |
leaders and their support | problems | • Professional technical support | multiple subject background | |
teams | • A well-structured support team | and operational service team | ||
• A small group of excellent | consisting of scientific researchers, | • International and domestic | ||
researchers | engineers and industrialized personnel | multidisciplinary research teams | ||
Forms of | • Solve major scientific | • Breakthrough key technologies | • Technological services with open | • Providing scientific suggestions and |
research | problems | • Provide systematic integration solutions | sharing, efficient operation and | constructive solutions for |
output | • Open up new research | • Develop new technologies and standards | satisfactory users | macro-decision-making and |
areas | • Incubation of new industries and | • Major breakthroughs in science | sustainable development | |
• Invent significant | enterprises | and technology relying on the | • Forming new theories, methods, | |
scientific instruments | • Technological radiation produces | formation of large scientific | standards and tools in this field | |
• Break new ground in key | significant economic benefits | equipment | • Accumulate basic data and provide | |
experimental methods | • Important original innovation for | • Creating first-class scientists and | an open and shared analytical | |
• Train first-class scientists | national major strategic needs | engineers | technology platform | |
• Putt forward significant | • Create first-class strategic science and | • Propose suggestions on planning | • Creating first-class scientists, | |
prospective suggestions | technology experts and engineering | of large-scale scientific | strategic science and technology | |
technology specialists | equipment adopted by the state | experts and technical specialists | ||
Research | • Periodic evaluation every 5 years, annual monitoring and mid-term diagnosis assessment | |||
evaluation | • Focus on research quality | • Focus on goal accomplishment and | • Focus on the achievement of | • Focus on quality, benefit and impact |
system | and impact | adoption | construction objectives, | • Based on peers' and users' opinions |
• International peer review | • Based on peers' and users' opinions, and | operational efficiency and major | ||
market feedback | outputs | |||
• Based on peers' and users' | ||||
opinions |
The construction of CAS Centers for Excellence is intended to “set examples, promote progress and gather talented personnel”, namely, setting up the banner of China’s science and technology innovation in important areas, taking the lead in making major breakthroughs and gathering top experts, so as to build an internationally top-notch innovation platform. Therefore, its evaluation system is based on reviews from international peers. In accordance with the management requirements, CAS organized an international peer review of nine Centers for Excellence after their preparation period in 2016. Based on the experiences gathered in 2016, an acceptance assessment was conducted for another eight Centers of Excellence in 2017. The following sections focus on three issues related to the design of international peer review, namely “what to review, who shall review and how to review”, to analyze the practices of CAS Centers for Excellence from a methodological perspective.
In general, what to review directly depends on the purpose of the review. In the case of CAS, the acceptance assessment is required to implement the construction of four categories of institutions, systematically summarize and promote reform experiences, and study and solve existing problems in a timely manner. Based on this, the acceptance assessment focuses on the overall goals and the results achieved during the preparatory period of the Centers for Excellence, including their mission, orientation, research field layout, development objectives, domestic and international status, systems and mechanisms, and supporting conditions. At the same time, suggestions related to the advantages, disadvantages and future development of the Center are welcome. Generally speaking, the organizer should have a clear understanding about what international experts can and cannot review, to ensure that international experts can exert their professional knowledge and experience to the maximum and play a practical role in the review. In CAS’s case, international experts were required to review the status of the teams, including their quality, level and the rationality of the research structure, as well as the content and impact of major outputs (representative outstanding research outputs, normally no more than three items), which was exactly what international experts are familiar with and excel at. This is similar to current practice in international peer review across the world (De Wit, Van Heffen, & Verhoeven, 1998).
To some extent, the selection of international experts determines the quality of international peer review. In accordance with international practice, CAS’ Headquarters (in short: CASHQ) first established the principles for setting up international review panels, consisting of field and dedicated peer experts. Field experts, selected based on the positioning of the center, are expected to have an international vision, strong reputation and influence, combined with a profound insight into their research fields. They, moreover, have extensive contact with dedicated peer experts. In addition, they should have the ability to effectively organize peer panels for internal discussions, and have experiences in reviewing, such as holding management positions in large scientific research institutions or universities, or having served as management/review experts of major scientific research projects. The dedicated peer experts must have good judgment in key areas of the center and an in-depth understanding and knowledge of research fields, be actively involved in cutting-edge and pioneering scientific research, and preferably have experience in performing evaluation exercises. In order to understand domestic policies and the local environment for the development of science and technology, one or two ethnic Chinese experts who can comprehensively utilize their local and linguistic advantage are normally included in the panel group.
Taking the Max Planck Society of Germany as a major reference, potential panel members are first recommended by research institutes. These include field experts, peer review experts, and user experts for the main fields. Then the National Science Library of CAS conducted a data analysis on competency and activity of these experts in the preliminary list, which is further submitted to the administrative department of CAS for final decision-making (Fig. 1). In all, 86 international experts were invited for international review of eight Centers for Excellence, of which 53 were non-Chinese experts, accounting for 62%, and 33 were ethnic Chinese experts, accounting for 38%. Furthermore, 46 management experts from CASHQ participated in the review.
The preparation of evaluation materials and the design of evaluation procedures is the core part of international evaluation. In the case of CAS, given that the Centers for Excellence are oriented towards basic and frontier areas
To ensure that international experts can, within a limited period of time, fully understand the background and purpose of the review, as well as acquire information about the centers, while reducing the burden on the centers as much as possible, the centers and CAS Evaluation Research Center jointly prepared evaluation materials. Specifically, the Centers for Excellence are responsible for preparing a self-evaluation report (both in an English and a Chinese version), filling out the
Drawing on the general practice of international institutional evaluation, five major steps are designed and programmed, namely:
During the on-site evaluation, a set of evaluation forms are designed for collecting panel members’ opinions on strategic positioning and international status, the research level and impact of major research outputs, and the research level of the center’s teams. More specifically, each international peer is expected to give his/her judgement on the center’s international status based on its main research fields, the impact of original work, the impact of leading scientists and major breakthroughs or research outputs according to the principle of originality, significance and rigor. With regard to evaluation of teams, each panel member must identify the team member’s international research impact within their area. Any comments and suggestions on the center, the development of research work as well as the teams are welcome.
In 2017, a total of 25 representative achievements, 31 subject leaders and 80 key members from eight Centers for Excellence were reviewed by international panels. From the feedback from CASHQ and the centers, the international peer review of CAS’s Centers of Excellence turned out to be satisfactory. Firstly, international review promoted the construction of four categories of institutions and the understanding of Center of Excellence. The entire review is a process for the leaders and researchers to “re-understand” the planning and key research fields of the Centers for Excellence. In the process of preparing evaluation materials, all the centers and their respective directors, subject leaders and researchers teased out the strategic orientation and major achievements of the research institutes as well as the relationship between various parts of the research work again, so as to deepen their understanding of the Centers of Excellence. Secondly, international peer review helps the development of CAS Centers of Excellence. A number of promising research work and distinguished researchers are identified, which provide valuable opinions for decision-making on resources allocations. Constructive opinions on the development of key research fields of the centers, especially their domestic and international status are pointed out, which are conducive for further improvement of the development plan of the centers. Thirdly, the philosophy of pursuit of research excellence is further promoted through the international peer review. During the on-site review, the international experts paid more attention to the quality and originality of research work rather than quantitative figures. Sometimes experts assess the current research status and level of researchers by asking them whether they know the best performers, competitors and their research content well in their respective research fields. These questions also help researchers to broaden their horizon, think deeply and understand their own advantages and disadvantages. Last but not least, there is no doubt that international reviews promote international exchanges and cooperation (DaiWai, 2019), which also provides an opportunity for the centers to promote themselves, advance the centers to the international forefront and expand international cooperation.
What is more, a questionnaire is conducted to collect opinions about the design of international peer view. Based on 42 valid questionnaires (86 questionnaires were issued, with a 49% recovery rate), the international experts believed that the international review provided a guarantee of independent and unbiased evaluation. According to their opinions, it is important to assess the Centers for Excellence by senior scientists from first rated international institutions as they can best review the status and assess the potential for future developments. Therefore, it is very important to collect evaluation opinions and suggestions through such an international review as a way to increase the quality and motivation of research work. From a personal perspective of experts, this review also provided them with a systematic understanding of CAS and its research institutes. Some experts said, “We also had exchanges and cooperation with some teams in the past, but that was just at the level of a research group or laboratory. This review provides comprehensive and systematic understanding of the overall situation of CAS.”
Moreover, each panel was required to give a score from 1 (the lowest) to 5 (the highest) in terms of panel member structure, preparation of evaluation materials, evaluation forms and standards, organization and implementation of review and finally the procedures and arrangements of review. It turns out that 83% of the international experts believed that the overall operation of international review is good or even excellent. In particular, the highest score is given to the organization and implementation of the international review, which was rated excellent or good by 90% of the international experts. The forms and standards of the international review received the lowest scores, but 79% of the international experts still rate them as excellent or good. In three other aspects, namely the procedures and arrangement of international review, preparation of materials and structure of expert group, 86%, 81% and 86% of the international experts rate them as excellent or good, as shown in Figure 2. In addition, the international experts made the following suggestions about the organization and implementation of international peer review: in terms of procedure design and arrangement, more time should be reserved for the preparatory meeting to gain a better understanding of the background. This also applies to one-on-one discussions. It is important to simplify assessment standards and forms and keep them consistent with international practice. In terms of the preparation of materials, it is necessary to collect more basic data such as personnel and funding information. Last, female scientists and more junior peer experts must be included.
Responding to the two critical questions proposed before, based on the example of CAS, we conclude that international peer review is effective in identifying problems in research activities, determining research level and status, and improving the development of research units with the help of international experts’ intelligence and experience. In order to gain maximal benefit from international panel members, the organizer must pay a lot of attention to the panel selection. On the one hand, the research level and reputation is the key factor when selecting appropriate experts. On the other hand, an expert is also expected to have strategic vision as well as rich evaluation experiences. In some cases, the authority and personal charm of the panel leader can lead to a smooth and rather pleasant experience. Particularly, ethnic Chinese experts play a great role in providing background information about the science and technology system in China. Other considerations during the on-site review such us informal communication with the panel leader in advance, and evaluation standards and language complying with international practices can improve the effectiveness and efficiency of international peer review as well.
In China, science and technology evaluation is at a critical stage. There is an urgent need for shifting the development path, that is, curbing the trend of excessive quantitative evaluation and developing a more quality-focused evaluation system to guarantee more breakthroughs in science and technology and transform from “quantity oriented to quality oriented” (Cheng, Li, & Xu, 2018). International peer review may provide a solution to address the problem of quantitative oriented research evaluation. In order to maximize the role of international peer review in research management, this paper puts forward the following points for discussion.