Historical Bibliometrics Using Google Scholar: The Case of Roman Law, 1727–2016

Bibliometrics—the quantitative study of publications and citations—can be a useful resource for social sciences and humanities (SSH) research beyond its role in research evaluation and funding-schemes (Scharnhorst & Garfield, 2010). From a sociology of science perspective, it enables us to understand the size and growth of disciplines in terms of number of publications and researchers, as well as changes in dissemination and citation patterns. Additionally, bibliometrics can be used to study the intellectual base, development and exchange of ideas within and between disciplines, in order to enlighten and tackle discipline specific historiographical questions

This paper is an extended version of a poster presented in ISSI2019 conference: Pölönen, J. & Hammarfelt, B., “Historical bibliometrics using Google Scholar: the case of Roman law, 1500–2016”. In G. Catalano et al. (eds), Proceedings of the 17th International Conference of the International Society for Scientometrics and Informetrics, Vol II. Rome: Edizioni Efesto, pp. 2491–2492.

Predominately research in the field of bibliometrics focus on contemporary developments using datasets that rarely provide historical perspectives. Still, historical approaches are not unheard of, with de Solla Price's seminal studies of the growth of science being one key example of how developments over longer time periods can be studied (de Solla Price, 1986). Later studies using a longer time-frame have for example studied the distribution of citations over time (Larivière, Gingras, & Archambault, 2009), and production of papers per authors (Fanelli & Larivière, 2016). Few studies have however incorporated publications dating before the beginning of the 20^th century. Hence, the argument can be made that the field of bibliometrics has not fully explored the potential of what Hérubel (1999) calls “historical bibliometrics”. Especially there is a lack of studies that incorporates not only references to other scholarly publications, but which also considers primary sources in the form of historical documents (Colavizza, 2018). There have been several attempts of going beyond established databases in order to study for example Catalan literature (Ardanuy, Urbano, & Quintana, 2009), Swedish literature (Hammarfelt, 2012), and Venetian histography (Colavizza, 2018). Yet, the approaches and methods used are often time-consuming and not easily transferred to other contexts and materials.

Considering the lack of coverage in established citation databases, as well as the limitations of local and specific approaches we propose Google Scholar as a potentially valuable data source for historical studies using bibliometric data. Consequently, in this paper we study and discuss the potential that Google Scholar data has for studying the development of research fields from a historical perspective. By using the example of “Roman law” we demonstrate the potential of this approach, while at the same time highlighting challenges and difficulties when applying bibliometric methods to this kind of heterogeneous data. Our approach is a probing one, and rather than delivering a fixed recipe for how to conduct “historical bibliometrics” we hope to open a discussion on what such a, in our view promising, approach might have to offer in the future. Especially we find that this and similar initiatives might potentially provide a bridge between bibliometrics and the dynamic field of “digital humanities”.

A well-known problem when studying fields within the humanities is that well-structured international databases (Web of Science and Scopus) offer a limited coverage of publications and citations, especially in books and languages other than English (Nederhof, 2006). Studies comparing the international database coverage with comprehensive national publication data drawn from the institutional research information systems show that law is among the least covered fields (Kulczycki et al., 2018). This is because in law, articles in books and monographs, as well as national languages, play an important role in the dissemination of research results (Kulczycki et al., 2020; Pichonnaz, 2014; van Leeuwen, 2013).

In comparison Google Scholar (GS) indexes publication data from a wider range of documents available online, and a recent estimation suggest that it is the most comprehensive database, with more than 389 million records (Gusenbauer, 2018). However, from its launch in 2004 the quality and stability of data in GS has been questioned (Jacsó, 2010), and although the database has developed over the years it is still recommended to include other data sources in order to compare and confirm findings (Halevi, Moed, & Bar-Ilan, 2017). With these precautions in mind it offers great advantages when studying research in the social sciences and humanities, with its significantly broader coverage of non-English, and non-journals materials being particularly important (Halevi, Moed, & Bar-Ilan, 2017; Prins et al., 2016).

In general, studies of GS have focused on issues concerning coverage—both generally and in specific disciplines—and questions regarding data quality. The age of sources indexed has not received the same amount of attention despite temporal coverage being key question when using bibliometrics for the purpose of studying the development of fields and disciplines. An initial suggestions was that “GS does not perform well for older publications, as these publications and the sources that cite them have not yet been posted on the web” (Harzing & Van der Wal, 2008). Yet, if this assertion holds true remains an open question as the rapid digitalization of older materials makes an increasing amount of materials available for indexing by Google Scholar.

Roman law (RL) has constituted an international research field within the legal academia since the foundation of the modern Western university in Bologna in the 12^th century around the study of the Emperor Justinian's codification (Orestano, 1987; Stein, 1999). Even today, classical studies and Roman law are multilingual research fields (Scheidel, 2007). After Latin ceased to be the lingua franca, there are five international RL publishing languages: English, French, German, Italian, and Spanish. Yet, following the growth of national legal systems, an important part of the publishing activity in these languages has consisted of research and elementary publications addressed to national audiences. RL has also grown an increasingly interdisciplinary field, involving vide range of SSH researchers and approaches (Pölönen, 2016). In all, Roman law literature provides a good case for probing the historical and linguistic coverage of Google Scholar.

First, a longitudinal study of the size and growth of the field in terms of the number publications and authors in five major languages is attempted. The main challenge is the identification of Roman law publications. We propose to resolve this problem by limiting analysis to publications, in the title of which “Roman law” appears in one of the five languages studied. The rationale is that there probably are at any time only a few RL specialists in the field who never published an article or book including in its title “Roman law” in the language of the publication. This method will not yield a complete record of Roman law publications, as not all relevant publications have “Roman law” in their title, but it produces a subset of publications that most likely are relevant to our probing investigation.

Therefore, in order to create a basic dataset, all publication records including in the title words denoting “Roman law” in the current international publishing languages (English, French, German, Italian, or Spanish), published between years 1500 and 2016, were retrieved from Google Scholar in August 2017, in blocs not exceeding 1,000, using the Publish or Perish (PoP) interface (see Table 3 below). The publication records retrieved from GS with PoP were copied to Excel in RIS format, and processed with BibExcel tool-box developed by Olle Persson (Persson, Danell, & Schneider, 2009).

Secondly, this dataset of Roman law publications is analyzed to establish the number of publications and authors, differentiating between the five language groups, from 1500 to 2016. The growth of the field is estimated on basis of the development of the absolute number and average yearly number of publications, as well as the number of authors involved in producing them, in different periods.

Thirdly, bibliometric measurements are performed on the data to investigate its properties and consistency. These include the average number of publications per author (publication productivity), as well as the concentration of publications and citations.

The quality of data in GS must be considered in any study of this kind. Two or more authors with the same name can be confused as only one author, or small differences in name can result in one person being mistaken as two or more different authors. Thus, homonyms, and synonyms as well as misspellings and duplicates may cause errors in the data. Such an error was indeed discovered in the analysis, for example a monograph from 2006 was mistakenly indexed as being published in 1770. Estimations regarding the prevalence of such errors are difficult to make, and automated detection and correction of errors was not deemed feasible due to the heterogeneous nature of the data used in terms of publication types, languages and age of source. Manual checking of records would have required extensive resources, especially when checking so called “non-source items” e.g. publications that are not indexed by GS but referred to by publications indexed by GS. Thus, the cleaning and checking of data remains a major challenge in using Google Scholar as a source for historical bibliometrics, and methods for improving data quality is a central issue when developing this line of research.

By using Roman law as an example this study tested the possibilities of using Google Scholar data for historical analysis. The analysis reveals that Google Scholar can be used to find key publications and track citation over a long period of time. In this case we were able to find citations to works dating back to 1727. Moreover, the distribution of publications from different language areas over time allowed us to gain insights into larger developments within the field of Roman law. Our data shows a large surge of French Roman law publications in the 19^th century, with a possible explanation being specific requirements of the French legal education at this time. While one should be careful to infer to strong conclusions based on these findings—they may partly be due to the availability of digitised materials—we find that historical approaches might be used for further more detailed analysis of intellectual development. At the same time, it is obvious that studies of this kind need to be supported by experts in the field for findings to be contextualised and corroborated.

Obviously, the choice of GS also comes with a range of methodological challenges, and many of these has already been discussed in the literature. The data used is automatically collected, and therefore issues concerning inadequate data, duplicates, homonyms etc. are persistent. A further issue when working with a heterogeneous material in terms of languages is how to deal with different translations of the same work, and of course different editions of seminal texts.

An insight from working with this material is that “historical bibliometrics” comes with specific affordances for researchers, as they need to combine historical insights with bibliometric competencies. Working with a field such as Roman law necessities’ language skills that allows for accessing the material analysed. Moreover, when conducting bibliometric analysis of older materials there is a need for understanding the publication culture, or knowledge infrastructure, in which these materials were produced. Publication patterns, dissemination paths, and referencing practices (Grafton, 1999) are all important aspects to consider.

In this paper we have used data retrieved from GS. The possible next steps in developing the dataset and the method could include manual checking of the data quality, identification of duplicate records, mistakes in author attribution or publication year, and estimating the effect of the data cleaning to the results. It is also possible to investigate to what extent the GS dataset of “Roman law” publications overlaps with similar datasets retrieved from Web of Science, Scopus, and Microsoft Academic. These datasets, and the GS dataset, could be compared to a more complete field specific bibliography—L’année philologique—in which references to scholarly literature concerning classical studies, including Roman law, have been collected since 1888. More generally further inquiries into the specificities of historical citation patterns and distributions, including institutional and disciplinary affiliations of authors, could provide valuable insights on the social and intellectual history of research fields.

In all, we find Google Scholar to be a promising data source for historical bibliometrics: it is accessible, has broad coverage and as demonstrated it also has quite a historical depth which allows for the analysis of older materials. At the same time there are distinct disadvantages: the quality of data is low, and as the database is continuously updating it is hard to reproduce earlier searches and data collections. The historical coverage is moreover dependent on the continuous digitisation and availability on the internet of older materials, which may in turn have large influence of the coverage in specific fields. Still, the possibilities for historical bibliometrics will most likely increase as the digitisation of older materials progress. Hence, while the approach taken here is a probing one, with many difficulties till to solve, we find that employing Google Scholar data for historical studies of fields and disciplines is a promising path for the future, and it is likely that such a path might attract travellers among bibliometricians as well as historians and other digital humanists.