Practice and Challenge of International Peer Review: A Case Study of Research Evaluation of CAS Centers for Excellence

Fang Xu 1 , 2  and Xiaoxuan Li 1 , 2
  • 1 Institutes of Science and Development, Chinese Academy of Sciences, Beijing, China
  • 2 School of Public Policy and Management, University of Chinese Academy of Sciences, Beijing, China
Fang Xu
  • Institutes of Science and Development, Chinese Academy of Sciences, Beijing, China
  • School of Public Policy and Management, University of Chinese Academy of Sciences, Beijing, China
  • Search for other articles:
  • degruyter.comGoogle Scholar
and Xiaoxuan Li
  • Corresponding author
  • Institutes of Science and Development, Chinese Academy of Sciences, Beijing, China
  • School of Public Policy and Management, University of Chinese Academy of Sciences, Beijing, China
  • Email
  • Search for other articles:
  • degruyter.comGoogle Scholar

Abstract

Purpose

The main goal of this paper is to show that international peer review can work in China’s context with satisfactory outcomes. Moreover, this paper also provides a reference for the practice of science and technology management.

Design/methodology/approach

This paper starts with a discussion of two critical questions about the significance and design of international peer review. A case study of international peer review of CAS Centers for Excellence is further analyzed.

Findings

International peer review may provide a solution to address the problem of quantitative oriented research evaluation in China. The case study of research evaluation of CAS Centers for Excellence shows that it is possible and feasible to conduct an international peer review in China’s context. When applying this approach to other scenarios, there are still many issues to consider including individualized design of international peer review combined with practical demands, and further improvement of theories and methods of international peer review.

Research limitation

1) Only the case of international peer review of CAS Centers for Excellence is analyzed; 2) A relatively small number of respondents were surveyed in the questionnaire.

Practical implications

The work presented in this study can be used as a reference for future studies.

Originality/value

Currently, there are no similarly detailed studies exploring the significance and methodology of international peer review in China.

1 Introduction

At present, China is a front-runner in the number of scientific outputs, but the quality remains to be improved. In order to respond to international competition and to satisfy domestic development needs, the orientation of China’s science and technology evaluation has changed from encouraging international publications to pursuing excellence and encouraging science and technology for promoting development. One of the policies is the introduction of international experts in science and technology evaluation when conditions permit. For instance, the Opinions of Deepening the Reform of Project Review, Talent Evaluation and Institutional Assessment (the “Sanping Opinions”) printed and released in 2018, proposes to “gradually introduce international peer review in some frontier and basic sciences areas in appropriate proportions” and “enhance performance evaluation of national science and technology plans and introduce international evaluation as needed”. In practice, some advanced institutions and universities in China have explored ways to implement international evaluation in various forms since 2000. The Chinese Academy of Sciences (CAS) took the lead in exploring and implementing international peer review at the institutional level. This initiative goes back to 2004 when a pilot international peer review of research institutes was carried out during Phase II of the Knowledge Innovation Project (Peng, 2013). In recent years, universities such as Shanghai Jiao Tong University (Zhang, 2019), Tsinghua University (Cheng, 2019), and Peking University have been exploring international evaluation with regard to the promotion of researchers; moreover, the Beijing Institute of Technology explored and implemented international evaluation of disciplinary (school or department) developments. In 2011, coinciding with the 25th anniversary of the establishment of the National Natural Science Foundation of China (NSFC), an international peer review of the performance of science funding and management was successfully completed (Wu, Liang, & Zheng, 2012), which is the largest comprehensive science and technology performance evaluation so far in China. Internationally, world-known research institutes have explored and implemented various forms of international evaluation very early, for example, evaluation of research institutes and selection of academic leaders by Max Planck Society (Shuai, 2004) and evaluation of discipline-specific research quality by Research Excellence Framework (REF) (Xu, Liu, & Li, 2014). There is no doubt that international evaluation represents a new trend in science and technology evaluation (Marušić, Brezis, & Squazzoni, 2017). Is the time ripe to conduct international peer reviews in China? More importantly, how to design and conduct effective international peer reviews in China? This paper starts with these two critical policy questions about international peer review, and intends to find an answer through a case study of international evaluation practices of acceptance assessment of CAS Center for Excellences. Then, it will put forward considerations about China’s future international evaluations, in order to accumulate knowledge for the development of theories and methods related to such evaluations and provide a reference for future science and technology management exercises.

2 Two critical questions about international peer review

Question 1: Is the time ripe to conduct international peer reviews in China? First, it is meaningless to conduct an international peer review when the research level is too low. Over the forty years since its reform and opening-up, China has raised its influence and made achievements of world interest in science and technology. For instance, China maintains an annual average 10% growth of investment in science and technology, and ranked second in the world in the number of SCI papers published and in citations from 2007 to 2017 (ISTIC, 2019). In some basic and frontier research areas, China’s research is already at the international level, and it is meaningful and possible to conduct an international peer review in these areas. Second, can international peers understand the difference of research management forms and organizational forms of science research in China? Some may argue that international peers know little beyond a vague picture of the enormous and complicated Chinese S&T system. Therefore, they can hardly play effective roles and provide valuable opinions. This leads to Question 2, which is about the design of international peer reviews.

Question 2: How to design and conduct effective international peer reviews? From the point of a researcher manager’s view, international peer review provides useful suggestions by bringing international experience of peer panels into the picture. The challenge is to ensure that the expertise of evaluation panels is congruent with the research interests of the research groups (Rousseau, 2017). On the one hand, it is critical to well design the evaluation procedure and prepare sufficient evaluation materials such as the background introduction of the purpose, content, criteria, supporting documents (Gunashekar, Wooding, & Guthrie, 2017) of a particular evaluation. On the other hand, not always visible conflicts of interest may lead to toning down negative issues. In that case, the research manager needs to find out hidden meanings among words of excessive praise.

These two critical questions raised hot debates before conducting an international peer review of CAS Centers for Excellence. To some extent CAS’s practice provide a possible answer, which may provide useful insights to others.

3 A case study of international peer review of CAS Centers for Excellence

3.1 Overview of CAS Centers for Excellence

In the light of new national and societal requirements, CAS announced the “Shuaixian Plan” in 2014 after examining and analyzing the existing foundation, strengths and weaknesses, and in response to the “four-first” requirement raised by President Xi Jinping. The Plan puts forward measures for a new classification of four categories of research institutes, namely: Centers for Excellence, Innovation Institutes, Centers for Mega-science and Featured Institutes. Table 1 shows the basic features of these four categories. Distinctions are based on different value orientations. An Innovation Institute is expected to meet major needs of national strategy and industrial development. Consequently, its evaluation system is based on reviews from the market and from users. Centers for Mega-science are focused on supporting sci-tech progress: its evaluation system is based on reviews from peers and users. Featured Institutes are expected to cultivate disciplinary characteristics, and its evaluation system is based on reviews from peers, relevant industry departments and local governments.

Table 1

Basic feature of CAS four categories of research institutes.

Centers for ExcellenceInnovation InstitutesCenters for Mega-scienceFeatured Institutes
Research• Focus on original• Focus on economic development and• Focus on building and running• Focus on serving for sustainable
orientationinnovation of science andnational securitylarge-scale science anddevelopment of society
technologytechnology innovation platforms
Research• Basic and frontier sciences• Important foundation and technological• Design, construct and operate• Indispensable research areas for
areas• Orientation towards majordirection for promoting industrialinternational advancedsustainable social development
scientific problems.upgrading, breaking through bottlenecks,large-scale scientific devices• Interdisciplinary study of natural
influencing or restricting national• Comprehensive interdisciplinaryscience and social science
securityand cutting-edge sciences relying• long-term observation and
• High risk technological frontieron large scientific devicescontinuous accumulation related
directionsbasic research work
Team• Internationally or• Strategic scientists, academic and• High level teams in scheme• Internationally recognized scientific
compositiondomestically recognizedtechnological leaders with broad visiondesign, technology research andand technology experts who also
leading scientistsand good grasp of technologicaldevelopment and engineeringunderstand national demand
• High-level academicdirection and organizational tackling keyorganizations• An integrated research team with
leaders and their supportproblems• Professional technical supportmultiple subject background
teams• A well-structured support teamand operational service team
• A small group of excellentconsisting of scientific researchers,• International and domestic
researchersengineers and industrialized personnelmultidisciplinary research teams
Forms of• Solve major scientific• Breakthrough key technologies• Technological services with open• Providing scientific suggestions and
researchproblems• Provide systematic integration solutionssharing, efficient operation andconstructive solutions for
output• Open up new research• Develop new technologies and standardssatisfactory usersmacro-decision-making and
areas• Incubation of new industries and• Major breakthroughs in sciencesustainable development
• Invent significantenterprisesand technology relying on the• Forming new theories, methods,
scientific instruments• Technological radiation producesformation of large scientificstandards and tools in this field
• Break new ground in keysignificant economic benefitsequipment• Accumulate basic data and provide
experimental methods• Important original innovation for• Creating first-class scientists andan open and shared analytical
• Train first-class scientistsnational major strategic needsengineerstechnology platform
• Putt forward significant• Create first-class strategic science and• Propose suggestions on planning• Creating first-class scientists,
prospective suggestionstechnology experts and engineeringof large-scale scientificstrategic science and technology
technology specialistsequipment adopted by the stateexperts and technical specialists
Research• Periodic evaluation every 5 years, annual monitoring and mid-term diagnosis assessment
evaluation• Focus on research quality• Focus on goal accomplishment and• Focus on the achievement of• Focus on quality, benefit and impact
systemand impactadoptionconstruction objectives,• Based on peers' and users' opinions
• International peer review• Based on peers' and users' opinions, andoperational efficiency and major
market feedbackoutputs
• Based on peers' and users'
opinions

The construction of CAS Centers for Excellence is intended to “set examples, promote progress and gather talented personnel”, namely, setting up the banner of China’s science and technology innovation in important areas, taking the lead in making major breakthroughs and gathering top experts, so as to build an internationally top-notch innovation platform. Therefore, its evaluation system is based on reviews from international peers. In accordance with the management requirements, CAS organized an international peer review of nine Centers for Excellence after their preparation period in 2016. Based on the experiences gathered in 2016, an acceptance assessment was conducted for another eight Centers of Excellence in 2017. The following sections focus on three issues related to the design of international peer review, namely “what to review, who shall review and how to review”, to analyze the practices of CAS Centers for Excellence from a methodological perspective.

3.2 What to review?

In general, what to review directly depends on the purpose of the review. In the case of CAS, the acceptance assessment is required to implement the construction of four categories of institutions, systematically summarize and promote reform experiences, and study and solve existing problems in a timely manner. Based on this, the acceptance assessment focuses on the overall goals and the results achieved during the preparatory period of the Centers for Excellence, including their mission, orientation, research field layout, development objectives, domestic and international status, systems and mechanisms, and supporting conditions. At the same time, suggestions related to the advantages, disadvantages and future development of the Center are welcome. Generally speaking, the organizer should have a clear understanding about what international experts can and cannot review, to ensure that international experts can exert their professional knowledge and experience to the maximum and play a practical role in the review. In CAS’s case, international experts were required to review the status of the teams, including their quality, level and the rationality of the research structure, as well as the content and impact of major outputs (representative outstanding research outputs, normally no more than three items), which was exactly what international experts are familiar with and excel at. This is similar to current practice in international peer review across the world (De Wit, Van Heffen, & Verhoeven, 1998).

3.3 Who to review?

To some extent, the selection of international experts determines the quality of international peer review. In accordance with international practice, CAS’ Headquarters (in short: CASHQ) first established the principles for setting up international review panels, consisting of field and dedicated peer experts. Field experts, selected based on the positioning of the center, are expected to have an international vision, strong reputation and influence, combined with a profound insight into their research fields. They, moreover, have extensive contact with dedicated peer experts. In addition, they should have the ability to effectively organize peer panels for internal discussions, and have experiences in reviewing, such as holding management positions in large scientific research institutions or universities, or having served as management/review experts of major scientific research projects. The dedicated peer experts must have good judgment in key areas of the center and an in-depth understanding and knowledge of research fields, be actively involved in cutting-edge and pioneering scientific research, and preferably have experience in performing evaluation exercises. In order to understand domestic policies and the local environment for the development of science and technology, one or two ethnic Chinese experts who can comprehensively utilize their local and linguistic advantage are normally included in the panel group.

Taking the Max Planck Society of Germany as a major reference, potential panel members are first recommended by research institutes. These include field experts, peer review experts, and user experts for the main fields. Then the National Science Library of CAS conducted a data analysis on competency and activity of these experts in the preliminary list, which is further submitted to the administrative department of CAS for final decision-making (Fig. 1). In all, 86 international experts were invited for international review of eight Centers for Excellence, of which 53 were non-Chinese experts, accounting for 62%, and 33 were ethnic Chinese experts, accounting for 38%. Furthermore, 46 management experts from CASHQ participated in the review.

Figure 1
Figure 1

International panel selection procedure in the case of CAS.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0013

3.4 How to review?

The preparation of evaluation materials and the design of evaluation procedures is the core part of international evaluation. In the case of CAS, given that the Centers for Excellence are oriented towards basic and frontier areas and aim to develop into incubators of major achievements, international experts are invited to participate in the on-site assessment (1–2 days) to evaluate the strategic positioning, major achievements and teams of the centers.

To ensure that international experts can, within a limited period of time, fully understand the background and purpose of the review, as well as acquire information about the centers, while reducing the burden on the centers as much as possible, the centers and CAS Evaluation Research Center jointly prepared evaluation materials. Specifically, the Centers for Excellence are responsible for preparing a self-evaluation report (both in an English and a Chinese version), filling out the Acceptance Self-evaluation Form, and providing a brief introduction and impact evidence of the major outputs (generally no more than three items), as well as an introduction to the leaders and academic support teams in all research fields. The Evaluation Research Center is responsible for preparing a data status report of the centers and other general materials such as a basic introduction to CAS, an introduction to international review, an expert manual and a work manual, to ensure that international experts have a comprehensive understanding of the current situation of CAS.

Drawing on the general practice of international institutional evaluation, five major steps are designed and programmed, namely: preparatory meeting, presentations of key research areas, group discussions, panel discussions, and panel feedback. The preparatory meeting is held to introduce background, purpose and requirements to the experts. It turned out that the success of the preparatory meeting determined to a large extent whether the review could produce the desired result. In particular, in order to improve the efficiency and effectiveness of international reviews, CASHQ and the center jointly held a pre-communication meeting with the panel leader to exchange opinions and answer questions on the overall situation of the review, which helped panel leader’s understanding of the review.

During the on-site evaluation, a set of evaluation forms are designed for collecting panel members’ opinions on strategic positioning and international status, the research level and impact of major research outputs, and the research level of the center’s teams. More specifically, each international peer is expected to give his/her judgement on the center’s international status based on its main research fields, the impact of original work, the impact of leading scientists and major breakthroughs or research outputs according to the principle of originality, significance and rigor. With regard to evaluation of teams, each panel member must identify the team member’s international research impact within their area. Any comments and suggestions on the center, the development of research work as well as the teams are welcome.

4 Feedback and effect of international peer review of CAS’s Centers of Excellence

In 2017, a total of 25 representative achievements, 31 subject leaders and 80 key members from eight Centers for Excellence were reviewed by international panels. From the feedback from CASHQ and the centers, the international peer review of CAS’s Centers of Excellence turned out to be satisfactory. Firstly, international review promoted the construction of four categories of institutions and the understanding of Center of Excellence. The entire review is a process for the leaders and researchers to “re-understand” the planning and key research fields of the Centers for Excellence. In the process of preparing evaluation materials, all the centers and their respective directors, subject leaders and researchers teased out the strategic orientation and major achievements of the research institutes as well as the relationship between various parts of the research work again, so as to deepen their understanding of the Centers of Excellence. Secondly, international peer review helps the development of CAS Centers of Excellence. A number of promising research work and distinguished researchers are identified, which provide valuable opinions for decision-making on resources allocations. Constructive opinions on the development of key research fields of the centers, especially their domestic and international status are pointed out, which are conducive for further improvement of the development plan of the centers. Thirdly, the philosophy of pursuit of research excellence is further promoted through the international peer review. During the on-site review, the international experts paid more attention to the quality and originality of research work rather than quantitative figures. Sometimes experts assess the current research status and level of researchers by asking them whether they know the best performers, competitors and their research content well in their respective research fields. These questions also help researchers to broaden their horizon, think deeply and understand their own advantages and disadvantages. Last but not least, there is no doubt that international reviews promote international exchanges and cooperation (DaiWai, 2019), which also provides an opportunity for the centers to promote themselves, advance the centers to the international forefront and expand international cooperation.

What is more, a questionnaire is conducted to collect opinions about the design of international peer view. Based on 42 valid questionnaires (86 questionnaires were issued, with a 49% recovery rate), the international experts believed that the international review provided a guarantee of independent and unbiased evaluation. According to their opinions, it is important to assess the Centers for Excellence by senior scientists from first rated international institutions as they can best review the status and assess the potential for future developments. Therefore, it is very important to collect evaluation opinions and suggestions through such an international review as a way to increase the quality and motivation of research work. From a personal perspective of experts, this review also provided them with a systematic understanding of CAS and its research institutes. Some experts said, “We also had exchanges and cooperation with some teams in the past, but that was just at the level of a research group or laboratory. This review provides comprehensive and systematic understanding of the overall situation of CAS.”

Moreover, each panel was required to give a score from 1 (the lowest) to 5 (the highest) in terms of panel member structure, preparation of evaluation materials, evaluation forms and standards, organization and implementation of review and finally the procedures and arrangements of review. It turns out that 83% of the international experts believed that the overall operation of international review is good or even excellent. In particular, the highest score is given to the organization and implementation of the international review, which was rated excellent or good by 90% of the international experts. The forms and standards of the international review received the lowest scores, but 79% of the international experts still rate them as excellent or good. In three other aspects, namely the procedures and arrangement of international review, preparation of materials and structure of expert group, 86%, 81% and 86% of the international experts rate them as excellent or good, as shown in Figure 2. In addition, the international experts made the following suggestions about the organization and implementation of international peer review: in terms of procedure design and arrangement, more time should be reserved for the preparatory meeting to gain a better understanding of the background. This also applies to one-on-one discussions. It is important to simplify assessment standards and forms and keep them consistent with international practice. In terms of the preparation of materials, it is necessary to collect more basic data such as personnel and funding information. Last, female scientists and more junior peer experts must be included.

Figure 2
Figure 2

Opinions from international experts.

Citation: Journal of Data and Information Science 4, 3; 10.2478/jdis-2019-0013

5 Conclusions and discussions

Responding to the two critical questions proposed before, based on the example of CAS, we conclude that international peer review is effective in identifying problems in research activities, determining research level and status, and improving the development of research units with the help of international experts’ intelligence and experience. In order to gain maximal benefit from international panel members, the organizer must pay a lot of attention to the panel selection. On the one hand, the research level and reputation is the key factor when selecting appropriate experts. On the other hand, an expert is also expected to have strategic vision as well as rich evaluation experiences. In some cases, the authority and personal charm of the panel leader can lead to a smooth and rather pleasant experience. Particularly, ethnic Chinese experts play a great role in providing background information about the science and technology system in China. Other considerations during the on-site review such us informal communication with the panel leader in advance, and evaluation standards and language complying with international practices can improve the effectiveness and efficiency of international peer review as well.

In China, science and technology evaluation is at a critical stage. There is an urgent need for shifting the development path, that is, curbing the trend of excessive quantitative evaluation and developing a more quality-focused evaluation system to guarantee more breakthroughs in science and technology and transform from “quantity oriented to quality oriented” (Cheng, Li, & Xu, 2018). International peer review may provide a solution to address the problem of quantitative oriented research evaluation. In order to maximize the role of international peer review in research management, this paper puts forward the following points for discussion.

First, international peer review needs adaptation to specific circumstances to better play its role. International peer review is often applied in universities with which international panels are more familiar than with a large institute such as CAS. This shows that international peer review must adapt according to management needs and evaluation objectives. This also meets the current purpose of developing science and technology. In addition to serve the growth of scientific knowledge, science and technology has to contribute to economic and social development. Therefore, international peer review is required to evaluate not only the excellence and originality of research work, but also its contribution to economic and social progress. CAS has adopted international peer review in its practices, so that it can coordinate with management experts and better serve the needs of science and technology management.

Secondly, the range of applications and implementations of international peer review should be clear. China has issued relevant policies to encourage international peer review when appropriate, but the difficulty lies in how to define the range of applications and implementations of international peer review. Generally, international peer review is applicable to evaluate innovation and influence of non-confidential research work in frontier and basic scientific research areas, and to identify problems and make suggestions. Specifically, when referring to the evaluation of different subjects such as scientists, teams, organizations and projects, one needs to further explore which are suitable for international peer review.

Thirdly, theories and methods of international peer review need improvement. The advantage of international peer review lies in exerting the role of international intelligence and standards, and reducing the influences of quantitative indicators on experts. Given this, theories and methods of international peer review need urgent improvement. For example, how to avoid conflicts of interest while ensuring that selected experts are familiar with the evaluated objects. The use of evaluation results is another problem. If international experts are capable but choose not to provide honest and complete review opinions, international peer review will only play a weak role in promoting management work, allocation of resources and organizational adjustment. Therefore, efforts should be made to further explore and improve theories and methods concerning international peer review as well as normative and standard evaluation procedures (the Standard Evaluation Protocol proposed in Netherlands provides a good example (KNAW, VSNU, NOW, 2014)), in order to improve the efficiency and effect of international peer review and to support better decision-making.

Author Contributions

Fang Xu (xufang@casisd.cn) conceived and designed the research work, contributed data or analysis tools, and wrote the paper. Xiaoxuan Li (xiaoxuan@casisd.cn) conceived and designed the research work, and performed the analysis.

Acknowledgment

The authors wish to acknowledge the review experts and journal editors, for their help in interpreting the significance of the results of this study and improving the quality of this manuscript.

References

  • Cheng, J.P., Li, X.X., & Xu, F. (2018). Science-evaluation reform on the road in China. National Science Review, 5(5), 605–605.

  • Cheng, X. (2019). Tenure-track system activates teachers. Department of Physics, Tsinghua University. Retrieved from http://www.tsinghua.edu.cn/publish/thunews/9914/2011/20111111084759369801776/20111111084759369801776_.html

  • DaiWai, M.O. (2019). Peer review is a collaborative process. Journal of Neuroscience Nursing, 51(3), 107–107.

  • De Wit, K., Van Heffen, O., & Verhoeven, J. (1998). Government, higher education and the national economy. Case study of Flanders. doi: 10.13140/2.1.1431.5526

  • Gunashekar, S., Wooding, S., & Guthrie, S. (2017). How do NIHR peer review panels use bibliometric information to support their decisions? Scientometrics, 112(3), 1813–1835.

  • ISTIC, Statistical results of Chinese scientific papers in 2017. Retrieved from https://www.istic.ac.cn/ScienceEvaluateArticalShow/tabid/679/Default.aspx?ArticleID=97570

  • KNAW, VSNU, NOW. (2014). Standard Evaluation Protocol 2015 – 2021, Protocol for Research Assessment in the Netherlands. Amsterdam: Royal Netherlands Academy of Arts and Sciences. Retrieved from https://www.knaw.nl/nl/actueel/publicaties/standard-evaluation-protocol-2015-2021

  • Marušić, A., Brezis, E., & Squazzoni, F. (2017). Scientometrics of peer review. Scientometrics, 113(1), 501–502.

  • Peng, D.L. (2013). The Chinese Academy of Sciences invited international experts to evaluate the Institute of History of Natural Sciences. Studies in the History of Natural Sciences, 32(3), 423–426.

  • Rousseau, R., Guns, R., Jakaria Rahman, A.I.M., & Engels, T. C.E. (2017). Measuring cognitive distance between publication portfolios. Journal of Informetrics, 11(2), 583–594.

  • Shuai, L.Y. (2004). Evaluation experience of max planck society. Science and Technology Policy and Development Strategy, (11), 27–33.

  • Wu, J., Liang, M., & Zheng, Y. (2012). International evaluation of NSFC’s performance by evidence-based design. Science Research Management, 33(6), 137–145.

  • Xu, F., Liu, W.B., & Li, X.X. (2014). The method and enlightenment of REF research influence evaluation in UK. Science of Science and Management of S.&.T., 35(7), 9–15.

  • Zhang, J. (2019). Thoughts on long-term international assessment in colleges and departments. Guangming Daily. Retrieved from http://plan.sjtu.edu.cn/info/1004/1004.htm

Footnotes

If the inline PDF is not rendering correctly, you can download the PDF file here.

  • Cheng, J.P., Li, X.X., & Xu, F. (2018). Science-evaluation reform on the road in China. National Science Review, 5(5), 605–605.

  • Cheng, X. (2019). Tenure-track system activates teachers. Department of Physics, Tsinghua University. Retrieved from http://www.tsinghua.edu.cn/publish/thunews/9914/2011/20111111084759369801776/20111111084759369801776_.html

  • DaiWai, M.O. (2019). Peer review is a collaborative process. Journal of Neuroscience Nursing, 51(3), 107–107.

  • De Wit, K., Van Heffen, O., & Verhoeven, J. (1998). Government, higher education and the national economy. Case study of Flanders. doi: 10.13140/2.1.1431.5526

  • Gunashekar, S., Wooding, S., & Guthrie, S. (2017). How do NIHR peer review panels use bibliometric information to support their decisions? Scientometrics, 112(3), 1813–1835.

  • ISTIC, Statistical results of Chinese scientific papers in 2017. Retrieved from https://www.istic.ac.cn/ScienceEvaluateArticalShow/tabid/679/Default.aspx?ArticleID=97570

  • KNAW, VSNU, NOW. (2014). Standard Evaluation Protocol 2015 – 2021, Protocol for Research Assessment in the Netherlands. Amsterdam: Royal Netherlands Academy of Arts and Sciences. Retrieved from https://www.knaw.nl/nl/actueel/publicaties/standard-evaluation-protocol-2015-2021

  • Marušić, A., Brezis, E., & Squazzoni, F. (2017). Scientometrics of peer review. Scientometrics, 113(1), 501–502.

  • Peng, D.L. (2013). The Chinese Academy of Sciences invited international experts to evaluate the Institute of History of Natural Sciences. Studies in the History of Natural Sciences, 32(3), 423–426.

  • Rousseau, R., Guns, R., Jakaria Rahman, A.I.M., & Engels, T. C.E. (2017). Measuring cognitive distance between publication portfolios. Journal of Informetrics, 11(2), 583–594.

  • Shuai, L.Y. (2004). Evaluation experience of max planck society. Science and Technology Policy and Development Strategy, (11), 27–33.

  • Wu, J., Liang, M., & Zheng, Y. (2012). International evaluation of NSFC’s performance by evidence-based design. Science Research Management, 33(6), 137–145.

  • Xu, F., Liu, W.B., & Li, X.X. (2014). The method and enlightenment of REF research influence evaluation in UK. Science of Science and Management of S.&.T., 35(7), 9–15.

  • Zhang, J. (2019). Thoughts on long-term international assessment in colleges and departments. Guangming Daily. Retrieved from http://plan.sjtu.edu.cn/info/1004/1004.htm

OPEN ACCESS

Journal + Issues

Search