The purpose of this article is to describe the development, components and properties of a publication indicator that the Ministry of Education and Culture in Finland uses for allocating direct core funding annually to universities. Since 2013, 13% of the core funding has been allocated on basis of publication indicator that, like the Norwegian model, is based on comprehensive national level publication data that is currently provided by the VIRTA publication information service. In 2015, the publication indicator was complemented with other components of the Norwegian model, namely, quality-weighted publication counts based on national Publication Forum authority list of the publication channels with ratings established by experts in the field. The funding model allocates around 1.6 billion euros annually to universities with the publication indicator annually distributing over 200 million euros. Besides the funding model, the indicator provides comparable data for monitoring the research performance of Finnish universities, fields and subunits. The indicator may also be used in the universities’ local funding models and research management systems, sometimes even at individual level evaluation. Positive and negative effects of the indicator have been extensively discussed and speculated. Since 2011, the Finnish universities’ productivity appears to have increased in terms of both quantity and quality of publications.
1 Background and motivation
University funding by government in Finland has been partly based on performance since the establishment in the 1990s of performance agreements, funding models and the KOTA-database for input and output statistics. The main policy goals have been to enhance efficiency, internationalisation, quality and impact of educational and research activities, gaining insights into the performance of the Finnish higher education institutions, as well as creating accountability and transparency regarding the use of public funds. Adaptation of the Norwegian model is embedded in the funding model that the Ministry of Education and Culture uses for allocating direct core funding annually to universities.
The funding model determines what performance indicators are used, and what is their share of core funding, in a fixed funding formula. The next years’ funding is calculated on the basis of three previous year’s performance outputs (average). Over the years, performance-based funding criteria have undergone several changes. In general, the share of core funding allocated on basis of indicators, and the weight of research indicators compared to education, has been increased. Publication statistics have been collected since 1994. The number of scientific publications related to the number of academic staff was introduced to the model in 2007–2012 as an indicator of research performance.
In the current model used since 2017, 41% of core funding is allocated on basis of indicators measuring performance in education, 33% is based on research performance indicators, and 28% on other policy considerations (Table 1). One of four research indicators is the quality-weighted count of scientific publications (share 13%). This indicator was adapted with components of the Norwegian model gradually in 2013–2015.
Universities’ funding model 2017–2020.
|Performance indicators||Share (%)|
|Other education and science policy objectives||28|
2 Organization of (re)design, implementation and operations
Development of a publication-based indicator for measuring research performance was discussed and planned since 2005 in several ministry working-groups and reports that investigated publication practices, information sources and international examples. In 2007, a KOTA working-group reviewed the performance-based research funding systems (PRFSs) used at the time in Australia, Flanders, Norway, Sweden and UK. Following the Norwegian model, the KOTA working-group recommended the development of a comprehensive national publication information system to improve the quality of publication data, as well as the development of a national authority list of publication channels to take quality of publications adequately into account in the funding model.
In 2009, the Finnish Council of University Rectors (now Universities Finland UNIFI) set up a working-group to develop the evaluation of quality of publications. Citation analysis, which is well established in science and medicine, was not chosen because of the inadequate coverage of the social sciences and humanities publications in the international databases (Web of Science and Scopus). Based on the Norwegian model adopted already in Denmark, also the UNIFI working-group recommended the development in Finland of a system based on a publication forum, in which peer-reviewed publication channels are identified and classified in quality levels by the experts in the field. Several advantages and disadvantages were identified with this model (Table 2).
UNIFI working-group’s SWOT analysis of a publication forum-based system.
3 Data sources and indicator design
There is a clear distinction of responsibilities with regard to the design of the publication indicator and the development of its different components: Federation of Finnish Learned Societies (TSV) produces together with the research community the publication channel authority list with level ratings, CSC – IT Center for Science Ltd. collects the comprehensive national level publication data from universities, while the funding model working-groups constituted by the Ministry Education and Culture decide if and how the publication data and channel ratings are used in the funding model indicator.
TSV, an independent organisation that represents the research community via its over 250 member societies, has been entrusted with the responsibility to produce the national authority list of peer-reviewed publication channels. Funding for this and other TSV activities comes from the Ministry. In 2010, TSV appointed a steering-group for a two-year project, which established a secretariat (2 persons fulltime) and appointed 23 expert panels covering all fields specifically for the task of evaluating the outlets. Panels have around 250 members from Finnish universities and public research institutes. The first authority list was published in the beginning of 2012, and after the project ended, the Publication Forum has become a fixed part of the TSV activities.
The most notable differences between the Norwegian and the Finnish authority lists concern the number of panels, the number of quality levels, the rating of national language channels, and the interval of updates. The rating has three levels for peer-reviewed publication series and book publishers: level 3 = top, level 2 = leading and level 1 = basic. There is also a level 0 for channels not qualifying as level 1. Both foreign and national journals/series, conferences and book publishers can be admitted to level 1 if they meet the following four criteria:
specialised in the publication of scientific or scholarly research outcomes
editorial board constituted by experts
entire manuscripts of scientific or scholarly articles or books subject to peer review
registered ISSN or ISBN number
As a main rule, however, even a publication channel meeting these criteria should not be included in Level 1 if they are local (mainly used by researchers of a single research organisation) or the quality and relevance to Finnish research community is questionable (e.g. predatory journals).
The authority list contains more than 30 000 publication series and book publishers, of which levels 2 and 3 include 10% of the most widely respected international journals and book publishers in different fields (table 3). In the social sciences and humanities fields, however, level 2 has also included since 2012 three book publishers and over 20 journals and book series publishing in the national languages (Finnish and Swedish). To balance the ratings across fields, in each expert-panel the publication channels selected for level 2 can represent in total up to 15%, and in level 3 up to 5%, of the world’s publications.
Publication Forum authority list of publication channels in 2018.
|Publication Forum level||Journals/series||Book publishers|
The authority list of publication channels is updated regularly, new channels being evaluated for addition to level 1 every year, and levels 2 and 3 are updated every four years. Also, the panels and the steering-group are appointed for four-year periods. Publication Forum ratings are openly available at the JUFO-portal, which the members of research community can also use for suggesting new additions and changes to the ratings.
CSC, a company partly owned by the state of Finland (70%) and Finnish higher education institutions (30%), has developed technical solutions for the national level integration of publication metadata from the universities since 2009. Finland faced the challenge that many universities had strongly invested in their own information systems, so it was not considered feasible to replace those systems with a national CRIS (such as CRISTIN in Norway). Integrating data from diverse local information systems has involved development of the data model, the publication type and field classifications, as well as the data collection guidelines and procedures to ensure quality and comparability of the publication data. Full publication data from 2011 to the present is currently stored in the VIRTA publication information service, an advanced solution launched in 2016.
In VIRTA, publications are divided in scientific, professional and general publications, and for the scientific publications separate publication types are used for the refereed and not-refereed articles in journals, conferences and books, monographs and edited work (Table 4). Publications are assigned to the publication types in universities at the time of registering of publications in the local information systems, from which these classifications are transferred with publication metadata to VIRTA. VIRTA stores records also for artistic and design activities, theses, patents and innovation announcements, as well as audiovisual material and ICT applications, but these have not been taken into account in the publication indicator.
Publication type classification in VIRTA publication information service.
|A1 Peer-reviewed journal article, original research|
|A2 Peer-reviewed journal article, review|
|A3 Peer-reviewed article or chapter in book|
|A4 Peer-reviewed article in conference proceedings|
|B1 Non-peer-reviewed journal article|
|B2 Non-peer-reviewed article or chapter in book|
|B3 Non-peer-reviewed article in conference proceedings|
|C1 Monograph (peer-reviewed)|
|C2 Edited book or special issue (peer-reviewed)|
|D1 Article in a trade journal|
|D2 Article in a professional book|
|D3 Article in professional conference proceedings|
|D4 Published development or research report or study|
|D5 Textbook, professional manual or guide|
|D6 Edited professional book|
|Publications for the general public|
|E1 Popularised article, newspaper article|
|E2 Popularised monograph|
|E3 Edited popularised book|
According to the data collection guidelines published by the Ministry of Education and Culture, scientific publications need to fulfil the following three conditions:
The publication must produce new information in relation to previous research data on the same subject.
The publication must be presented in a format that enables the verification of the research results and/or use of the research results in a new research, thus allowing other researchers to assess the research results and use them in their own work.
The publication channel for the publication specialises in publishing scientific research results, and it has an editorial staff consisting of experts in the field of science as well as a peer review practice.
A pre-publication peer-review, focusing on research quality, by independent experts in the field is required for inclusion of publications among the peer-reviewed publication types (A and C). To support the identification of peer-reviewed publications, TSV introduced in 2014 a national label for peer-reviewed publications inspired by the example of the GPRC-label in Flanders. Ten Finnish book publishers and 167 journals currently use the label to indicate articles and books that have undergone peer-review according to the label requirements.
The funding model is developed by working-groups appointed periodically by the Ministry to consider adjustments to the performance indicators and their share of the core funding. These groups have representatives from the Ministry, the higher education institutions, and the Academy of Finland (the main research funding agency). A funding model proposal is published in a report and sent out by the Ministry for consultation to a wide group of stakeholders. A decree concerning the universities’ core funding criteria is amended according to the new model approved by the Ministry.
The publication information now stored in VIRTA has been used as the data source of publication indicator since 2013. Weighting of publications according to the publication type was also introduced already in 2013, as monographs (C1) were counted with four times larger weight than articles (A1–A4, B 1–B3). The Publication Forum levels have been identified for all peer-reviewed articles and monographs published in 2011 or later. In the period 2013–2014, however, the funding model did not yet use the Publication Forum levels to differentiate publications according to quality. Instead, 9% of core funding was allocated on basis of an undifferentiated count of international refereed publications and 4% based on the other publications. The former category included publication types A1–A4 and C1 with country of publication other than Finland, and the latter category included all the other publications belonging to types A, B and C.
The funding model working-group originally planned that in 2015, international refereed publications would be replaced with the count of publications in Publication Forum level 2 and 3 channels, and other publications with publications in level 1 channels. Eventually, a weighting of publications by type and channel was introduced in 2015, and later adjusted for the period 2017–2020 (Tables 5 and 6). The Finnish publication indicator differs somewhat from the Norwegian model as it assigns similar weight to peer-reviewed articles in journals, books and proceedings, and takes into account with small weight not-refereed publications intended for scholarly, professional and general audiences (publication types D and E).
Weight of publications in the funding model 2015–2016.
|Publication type||Level 3||Level 2||Level 1||Level 0|
|Peer-reviewed monograph (C1)||12||12||6||4|
|Peer-reviewed article in journal (A1–2)||3||3||1.5||1|
|Peer-reviewed article in book (A3)||3||3||1.5||1|
|Peer-reviewed article in proceedings (A4)||3||3||1.5||1|
Weight of publications in the funding model since 2017–2020.
|Publication type||Level 3||Level 2||Level 1||Level 0|
|Peer-reviewed monograph (C1)||16||12||4||0.4|
|Peer-reviewed article in journal (A1–2)||4||3||1||0.1|
|Peer-reviewed article in book (A3)||4||3||1||0.1|
|Peer-reviewed article in proceedings (A4)||4||3||1||0.1|
|Peer-reviewed edited work (C2)||4||3||1||0.1|
|Not-peer-reviewed articles and edited work||0.1|
Institutional level whole-counts are used, meaning that co-publications with authors from several Finnish universities are counted more than once in the funding model. Fractionalization at institutional or author level has been discussed but not adopted in order not to discourage inter-university collaboration.
The Universities Act and the Polytechnics Act require all higher-education institutions, including 14 universities and 23 universities of applied sciences, to supply publication information to Ministry of Education and Culture. Also five university hospital districts, each consisting of several hospitals, and six public research institutes have agreed to provide their publication data to VIRTA. The publication information is openly available at the JUULI-portal for browsing and searching, and at the VIPUNEN-portal for statistics.
4 Funding implications
The funding model allocates around 1.6 billion euros annually to universities, and this core funding covers almost two-thirds of the universities’ budgets. The share of core funding allocated on basis of scientific publications has increased considerably, being 0.3% in 2007–2009, 1.7% in 2010–2012, and 13% since 2013. Since 2013, the publication indicator distributes annually over 200 million euros. In a report published in 2018 by Economic Policy Council on the university reform and financial Incentives, it was estimated that one publication point (equivalent e.g. to one peer-reviewed article in level 1 journal) is worth approximately 4200 euros to universities (Seuri & Vartiainen, 2018).
5 Other uses of the data
According to the UNIFI working-group, the development of the Publication Forum based publication indicator supported with national level publication data would serve three potential interests:
to provide quality indicator of publications for the universities’ funding model
to provide universities with comparable data for benchmarking with other universities, monitoring the publication profiles of units, and developing internal funding models
to increase the awareness of the researchers about quality of publication channels, thereby raising their level of ambition in publishing
The working-group also pointed out that Publication Forum could be used as information supporting research evaluation, for example in research assessment exercises the universities conduct themselves every few years, and the state of research in Finland reports produced by the Academy of Finland. Publication Forum ratings have indeed been used to supplement Web of Science based bibliometric analyses in some recent research assessment exercises.
Besides the funding model, the Ministry of Education and Culture uses the information based on the VIRTA publication data and the Publication Forum ratings for monitoring the research performance of the universities. Since 2010, the Ministry’s spending proposals submitted to the Ministry of Finance for the formulation of the state budget includes quantitative research performance targets for the universities collectively. Since 2016, targets have been set for the number of scientific level 2 and 3 publications per academic staff, as well as the international co-publications’ share of scientific level 1–3 publications. The same indicators are used by the Ministry as positioning statistics, openly available at the VIPUNEN-portal, for the performance agreements with individual universities.
The Ministry of Education and Culture has emphasized that the internal allocation of funds to faculties and departments should be based on universities’ own strategic priorities rather than replicate the funding model criteria. Nevertheless, universities are autonomous in using data and indicators in evaluation, development and management of their research activities. While the UNIFI working-group envisioned the potential use of the indicator in universities, appropriate local models and uses of the Publication Forum rating have not been defined. Already in 2012, the Publication Forum steering-group published a user-guide, in which it recommended not to use the rating in cross-field comparisons or in the evaluation of individual researchers.
In the fall 2015, TSV and UNIFI conducted a survey among university managers concerning the local use of the Publication Forum rating. The sample consisted of 10 rectors, 19 deans and 68 heads of department. Most Finnish universities make use of the channel rating for monitoring and developing their publishing activities, and in some universities the rating is used also for funding allocation to faculties and/or departments. The use of the ratings for evaluation purposes (recruitment, promotion, personal performance, bonuses) at the individual level is also attested, especially in social sciences and humanities. It is not clear, however, to what extent the rating supports, rather than supplants, qualitative expert assessment.
VIRTA publication data, including the Publication Forum ratings, has provided the source of information for bibliometric research and analyses concerning for example publications patterns and research productivity.
6 Experiences and effects
In Finland, as in the other Nordic countries using the Norwegian model, the positive and negative effects of the publication indicator have been discussed and speculated extensively. The effect of the publication indicator on research performance has been investigated in two recent reports, one evaluating the impact of the Universities Act reform in 2010 (2016), and another by Economic Policy Council assessing specifically the incentives created by the funding model (2018). In both reports, the development of Finnish universities publication activities was analyzed using the VIRTA publication data and Publication Forum levels. Both reports conclude that the research productivity has increased since 2011. Most notably, there has been a clear movement from level 0 channels toward level 1–3 channels. It seems that the publication channel rating has increased the Finnish research community’s awareness of quality of publication channels, even though the quality-weights were formally introduced to the funding model only in 2015.
The Economic Policy Council’s report also established that the average Publication Forum rating of peer-reviewed publications has increased in all universities, and more so in those universities where the average level was weaker in 2011. The differences in performance have decreased between universities, suggesting that the model has been most effective in those environments where full potential in productivity have not yet been reached. When related to the number of teaching and research personnel, it seems that the publication productivity of academic staff has increased but this has not happened at the expense of quality. But also the question is raised, to what extent the level ratings indicate increase in quality, as the selection of channels to level 2 and 3 can be influenced by local interests. The report also pointed out field specific differences in publication performance, making some fields seem more productive to universities.
Although it is regarded an advantage of the publication indicator that research community itself is entrusted the task of rating the publication channels, subjectivity of the expert-ratings is a common concern in Finland. The indicator has been developed for the purpose of providing comprehensive coverage of research outputs across fields, but equal treatment of different fields and specialties in level 2 and 3 nominations is much debated. The indicator is sensitive to the specificity of social sciences and humanities publishing culture, yet the funding models’ incentives are more often linked with decline rather than promotion of book and national language publishing. Other topics of discussion have included the effect of the publication indicator on interdisciplinarity and open access. Many of the concerns probably stem from the the publication indicator’s use for research management and evaluation purposes at universities, rather than from the indicators’ use in the universities’ funding model.
Description of the Finnish higher education system, performance agreements and funding model in English
Aarrevaara, T., Wahlfors, L., & Dobson, I. (2018). Higher Education Systems and Institutions, Finland. In Teixeira, Pedro et al. (eds.), Encyclopedia of International Higher Education Systems and Institutions, Springer 2018.
De Boer, H.F., Jongbloed, B., Benneworth, P., Cremonini, L., Kolster, R., Kottmann, A., Lemmens-Krug, K., & Vossensteyn, H. (2015). Performance-based funding and performance agreements in fourteen higher education systems—Report for the Ministry of Education, Culture and Science, Enschede: CHEPS, University of Twente. Retrieved from https://ris.utwente.nl/ws/portalfiles/portal/5139542/jongbloed+ea+performance-based-funding-and-performance-agreements-in-fourteen-higher-education-systems.pdf
Jonkers, K., & Zacharewicz, T. (2016). Research Performance Based Funding Systems: a Comparative Assessment. European Union. Retrieved from http://publications.jrc.ec.europa.eu/repository/bitstream/JRC101043/kj1a27837enn.pdf
Reports related to the development of the funding model and the publication indicator
Ministry of Education (2005). Yliopistojen tulosohjauksen kehittämistyöryhmä III (in English: Committee on the development of university performance management III). Helsinki: Opetusministeriö. Retrieved from https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/80321/tr24.pdf?sequence=1
Ministry of Education (2007). Bibliometristen aineistojen käytettävyys yliopistojen julkaisujen laadun ja tuottavuuden arvioinnissa (in English: Usability of bibliometric data in evaluation of quality and productivity of universities’ publications). Helsinki: OPetusministeriö. Retrieved from http://www.minedu.fi/export/sites/default/OPM/Julkaisut/2007/liitteet/opm02.pdf?lang=fi
Ministry of Education (2008). Julkaisukäytännöt eri tieteenaloilla (in English: Publishing patterns in different fields). Helsinki: Opetusministeriö. Retrieved from http://www.minedu.fi/export/sites/default/OPM/Julkaisut/2008/liitteet/opm33.pdf?lang=fi;
Ministry of Education (2008). TiedeKOTA-työryhmän raportti (In English: Report of the TiedeKOTA working-group). Helsinki: Opetusministeriö. Retrieved from http://docplayer.fi/14481463-Tiedekota-tyoryhman-raportti.html
Ministry of Education and Culture (2011). Laadukas, kansainvälinen, profiloitunut ja vaikuttava yliopisto – ehdotus yliopistojen rahoitusmalliksi vuodesta 2013 alkaen (In English: High-quality, profilised and effective international university - Proposal for a reform of the university financing model from 2013). Helsinki: Opetus- ja kulttuuriministeriö. Retrieved from http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75464/tr26.pdf?sequence=1&isAllowed=y
Ministry of Education and Culture (2014). Vahvemmat kannusteet koulutuksen ja tutkimuksen laadun vahvistamiselle: ehdotus yliopistojen rahoitusmallin tarkistamiseksi vuodesta 2015 alkaen (In English: Greater incentives for strengthening quality in education and research: A proposal for revising the funding model for universities as of 2015). Helsinki: Opetusja kulttuuriministeriö. Retrieved from https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75264/tr07.pdf?sequence=1
Ministry of Education and Culture (2015). Ehdotus yliopistojen rahoitusmalliksi 2017 alkaen (In English: Proposal for the funding model of universities as of 2017). Helsinki: Opetus- ja kulttuuriministeriö. Retrieved from https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75157/tr19.pdf?sequence=1
UNIFI (2010). Julkaisujen laadunarvioinnin kehittäminen (In English: Development of evaluation of publication quality). Retrieved from http://www.helsinki.fi/halvi/srno/raportit_ja_julkaisut/Julkaisujen_laadunarviointi.pdf
Ministry of education and culture: Retrieved from https://minedu.fi/en/steering-financing-and-agreements
Budget spending proposals submitted to the Ministry of Finance: Retrieved from http://budjetti.vm.fi/indox/index.jsp
Documents describing the development and properties of the publication channel rating
Auranen, O., & Pölönen, J. (2012). Classification of scientific publication channels: Final report of the Publication Forum project (2010–2012). Helsinki: Federation of Finnish Learned Societies. Retrieved from http://www.julkaisufoorumi.fi/sites/julkaisufoorumi.fi/files/publication_forum_project_final_report_0.pdf
Pölönen, J., & Ruth, A. S. (2015). Final report on 2014 review of ratings in Publication Forum. Helsinki: Federation of Finnish Learned Societies. Retrieved from http://www.julkaisufoorumi.fi/sites/julkaisufoorumi.fi/files/publication_forum_final_report_on_2014_review_of_ratings.pdf
Publication Forum Manual for Panels (2018). Retrieved from https://wiki.eduuni.fi/display/CSCJulkaisufoorumiportaali/Publication+Forum+Manual+for+Panels
User-guide (2018). Retrieved from http://www.julkaisufoorumi.fi/sites/julkaisufoorumi.fi/files/jufo_manual_2018v1.pdf
Publication Forum website: www.julkaisufoorumi.fi
JUFO-portal: Retrieved from http://jfp.csc.fi:8080/en/
Label for peer-reviewed publications: Retrieved from https://www.tsv.fi/en/services/label-for-peer-reviewed-scholarly-publications
Description of the properties of the national publication data
Sile, L., Guns, R., Sivertsen, G., & Engels, T. C. E. (2017). European Databases and Repositories for Social Sciences and Humanities Research Output. Antwerp: ECOOM & ENRESSH. Retrieved from https://doi.org/10.6084/m9.figshare.5172322.v2
Sile, L., Pölönen, J., Sivertsen, G., Guns, R., Engels, T., Arefievd, P., Duškováe, M., Faurbækf, L., Hollg, A., Kulczycki, E., Nelhans, G., Petr, M., Piskk, M., Soós, S., Stojanovskil, J., Stone, A., Šušolo, J., & Teitelbaump, R. (2018). Comprehensiveness of national bibliographic databases for social sciences and humanities: findings from a European survey, Research Evaluation doi: 10.1093/reseval/rvy016.
Puuska H. M., Guns, R., Pölönen, J., Sivertsen, G., Mañana-Rodríguez, J., & Engels, T.C.E. (2018). Proof of concept of a European database for social sciences and humanities publications: Description of the VIRTA-ENRESSH pilot. Helsinki: CSC & ENRESSH. Retrieved from https://doi.org/10.6084/m9.figshare.5993506
Publication data collection instructions for researchers (2018). Retrieved from https://wiki.eduuni.fi/display/cscsuorat/Julkaisutiedonkeruun+tutkijaohjeistukset?preview=/39984924/70200131/Publication%20data%20collection%20instructions%20for%20researchers%202018.pdf
VIPUNEN-portal for publication statistics. Retrieved from https://vipunen.fi/en-gb/
JUULI-portal for browsing publication information. Retrieved from http://www.juuli.fi/
Funding implications, other uses, effects and experiences
Auranen, O., Leino, Y., Poropudas, O., & Pölönen, J. (2013). Julkaisufoorumi-luokitus ja viittausindeksit tieteellisten julkaisujen laadun mittarina. Web of Science -aineistoon perustuva vertailu (In English: Publication Forum ratings and citation scores as indicators of quality of scientific publications. A comparison based on Web of Science data), TaSTI Työraportteja 8/2013. Retrieved from http://www.uta.fi/yky/tutkimus/tasti/Julkaisut/Sahkoinenkirjasto/Julkaisufoorumi-luokitus%20ja%20viittausindeksit%20tieteellisten%20julkaisujen%20laadun%20mittareina%2019.pdf
Auranen, O., & Pölönen, J. (2014). Julkaisufoorumi-luokitus ja kansallinen julkaiseminen (In English: Publication Forum rating and national publishing). In Muhonen Reetta, Puuska, Hanna-Mari (eds.) Tutkimuksen kansallinen tehtävä. Tampere: Vaystapaino, 153–175.
Hakala, J., & Roihuvuo, J. (2014). Research Assessment Exercise at The University of Tampere 2014: Final Report. Tampere: The University of Tampere. Retrieved from http://www.uta.fi/rae2014/index/UTA_RAE_2014_Final_Report(Web).pdf
Lyytinen, A., Muhonen, R., Koskinen, O., & Eskola, O. (2016). Ulkomaiset tutkijat ja julkaisuaktiivisuus (In English: Foreign researchers and their publishing frequency), Opetusja kulttuuriministeriön julkaisuja 2016:16. Retrieved from http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75337/okm16.pdf
Meriläinen, S. (2015). Julkaisufoorumin tasoluokitus: ovatko vaihtoehtoiset lähestymistavat ja poikkitieteellinen tutkimus uhattuina?: Puheenvuoro (In English: Publication Forum ratings: are alternative approaches and interdisciplinary research threatened?). Sukupuolentutkimus 28:3, 45–50. Retrieved from https://sunsseura.files.wordpress.com/2015/01/puheenvuoro_merilainen.pdf
Ministry of Education and Culture (2015). Suomalaisen yliopistotutkimuksen tuottavuus ja vaikuttavuus: opetus- ja kulttuuriministeriön profiiliryhmän raportti (In English: Productivity and impact of Finnish university research. Report by the profile working group of the Ministry of Education and Culture), Opetus- ja kulttuuriministeriön julkaisuja 2015:5. Retrieved from http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75132/okm05.pdf
Ministry of Education and Culture (2016). Yliopistolakiuudistuksen vaikutusten arviointi (In English: Impact evaluation of the Universities Act reform). Helsinki: Opetus- ja kulttuuriministeriö. Retrieved from https://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/75416/okm30.pdf?sequence=1
Muhonen, R., Eskola, O., Leino, Y., & Pölönen, J. (2016) Tutkimuksen monitieteisyys ja laatu (In English: The multidisciplinary nature and quality of research), Opetus- ja kulttuuriministeriön julkaisuja 2016:12. Retrieved from http://julkaisut.valtioneuvosto.fi/bitstream/handle/10024/64992/okm12.pdf?sequence=1&isAllowed=y
Pihlström, S. (2014). Academic publishing and interdisciplinarity: Finnish experiences. Human Affairs 24, 40–47. DOI: 10.2478/s13374-014-0204-2.
Poropudas, O. (2018). Miksi suomalaisen tutkimuksen tuottavuus kasvoi, mutta laatu heikentyi? (In English: Why the productivity of Finnish Research Increased but quality decreased?). Informaatiotutkimus 37:1, 5–16. Retrieved from https://journal.fi/inf/article/view/70169/31174
Puuska, H. M. (2014). Scholarly Publishing Patterns in Finland. Tampere: University of Tampere. Retrieved from https://tampub.uta.fi/bitstream/handle/10024/95381/978-951-44-9480-2.pdf?sequence= 1.
Pölönen, J., & Auranen, O. (2017). Julkaisupaine suomalaisessa tiedeyhteisössä (In English, Publishing pressure in Finnish research community). Informaatiotutkimus 36(2), 23–41. Retrieved from https://doi.org/10.23978/inf.65186
Pölönen, J., Auranen, O., Engels, T., & Kulczycki, E. (2018). Taking national language publications into account: the case of the Finnish performance-based research funding system. STI 2018 Conference Proceedings. Retrieved from https://openaccess.leidenuniv.nl/bitstream/handle/1887/65223/STI2018_paper_43.pdf?sequence=1
Pölönen, J., & Sivertsen, G. (2017). Experiences with the rating of publication channels for the Nordic Model. With a response to a proposal for automated ratings from Saarela et al. (2016). Presentation at the 22nd Nordic Workshop on Bibliometrics and Research Policy. Retrieved from https://figshare.com/articles/Experiences_with_the_rating_of_publication_channels_for_the_Nordic_Model_With_a_response_to_a_proposal_for_automated_ratings_from_Saarela_et_al_2016_/5624731
Pölönen, J., & Wahlfors, L. (2016). Local use of a national rating of publication channels in Finnish universities. Poster presented at the 21st Nordic Workshop on Bibliometrics and Research Policy. Retrieved from https://figshare.com/articles/Local_Use_of_a_National_Rating_of_Publication_Channels_in_Finnish_Universities_NWB_2016_poster_/4246541
Seuri, A., & Vartiainen, H. (2018). Yliopistojen rahoitus, kannustimet ja rakennekehitys (In English: University funding, incentives and structural development). Helsinki: Economic Policy Council. Retrieved from https://www.talouspolitiikanarviointineuvosto.fi/wordpress/wp-content/uploads/2018/01/Seuri_Vartiainen_2018-1.pdf
Sivula, A., Suominen, J., & Reunanen, M. (2015). A1 alkuperäisartikkeli tieteellisessä aikakauslehdessä. Uusien julkaisukäytänteiden omaksuminen ihmistieteissä 2000-luvulla (In English: A1 original research article in scientific journal. Embracing new publishing practices in social sciences and humanities in the 21st century). Kasvatus & Aika 9(3), 149–171. Retrieved from http://www.kasvatus-ja-aika.fi/dokumentit/a10_sivulaal_2910151558.pdf
Tähtinen, H., Rautava, P., & Happonen, R. P. (2014). EVO-pisteiden ja Julkaisufoorumin tasoluokituksen vertailu terveystutkimuksen arvioinnissa (In English: Comparison of EVO-points and Publication Forum level ratings for evaluation of health research). Suomen Lääkärilehti 69(10), 714–719.
Wahlfors, L., & Pölönen, J. (2018). Julkaisufoorumi-luokituksen käyttö yliopistoissa (In English: The use of Publication Forum ratings in universties. Hallinnon tutkimus 37(1), 7–21.
Wang, L., Vuolanto, P., & Muhonen, R. (2014). Bibliometrics in the research assessment exercise reports of Finnish universities and the relevant international perspectives, University of Tampere 2014. Retrieved from http://tampub.uta.fi/handle/10024/96266