Measuring Societal Impact Is as Complex as ABC

For this study, I use the data from the bibliometric version the Web of Science core collection hosted at CWTS. It is a dedicated version of the Web of Science online for bibliometric studies, with additional features and improvements. The important add-on for this study is the publication-based classification (Waltman & van Eck, 2012) in which publications are clustered on the basis of their citation-based context into coherent groups (Traag, 2015; Traag, Waltman, & van Eck, 2019). Over 4000 clusters (CWTS classification, version 1813, available at www.leidenranking.com) provide a fine-grained structure of all sciences independent of the journals in which research papers are published. This structure enables a topic basic-based analysis and will be used to identify research communities, i.e., clusters of publications on a certain topic.

The VOSviewer map below shows the distribution of the 4,000 clusters. Each cluster is represented by a circle in the map. The position of a circle indicates its relatedness to all others, based on their mutual citation traffic. The denser the citation traffic, the closer two circles are. The size of a circle represents the number of publications included.

Fig. 1 visualizes a virtual landscape of all sciences (as far as covered by WoS) in two dimensions. The actual position of a clusters has no meaning, only the distances between them. Hence, one can rotate or flip the map in any direction. Only the distances matter. I added colors for clarity. These colors represent high level regions (disciplines or main fields) and are labelled as such.

Figure 1

Another add-on to the CWTS database is the direct link of WoS publications to Altmetric data and (PATSTAT) references in patents. For each paper we determine whether it has been mentioned on Twitter, in a news item, in a policy document, in a Wikipedia article etc or whether it was cited in a patent. I consider such mentions or citations as non-academic events or actions related to the research that was mentioned, cited etc. This information I will therefore use as signals of connectedness between research and society.

In the approach to characterize journals I apply a method, coined as area-based connectedness (ABC) to society (Noyons, 2018). This approach assesses the connectedness of the research covered based on the output of an actor, in this case a journal. This connectedness is captured by the communication signals we detect between research output and society. These signals are diverse and represent a variety of dimensions.

To determine connectedness of research areas or communities to society, we measure a set of signals. The individual signals represent different aspects, dimensions of this connectedness. For instance, (co-)authorship of industry in publications indicates technological application and/or commercial use of research. Mentions in policy documents of publications signifies societal or political interest or value. I will demonstrate that each signal has its own specific profile. The profiles relate to their distribution across the science landscape. Different signals connect different research areas.

Until now, such connectedness signals are measured at the level of actors. For instance, the share of output of actor A co-authored with industry would then be a an indication of the societal, economic, technological impact of A. Or the number of twitter mentions of articles by actor B would be an indication of B’s societal impact (Bornmann, 2013; Seppo & Lilles, 2012).

Such approaches have at least two flaws. Firstly, social media metrics are easily manipulated and therefore vulnerable for malpractices (Costas, Zahedi, & Wouters, 2015; Haustein et al., 2016). Moreover, such indicators may be affected by practical limitations. For instance, co-authorship with industry often depends on the willingness of the company involved to co-author. Similarly, mentions of scientific output in non-scholarly output often depends on the accessibility of scholarly literature. If a research paper is not available or too technical, the author of a policy report may decide to cite another publication on the same topic.

The ABC method measures the output of a journal using the character of research communities (areas) to which it belongs. A community is defined as a set of publications on the same topic, i.e. a cluster of publications from the CWTS publication-level classification.

Weighted by the number of publications a journal has within a community or area it “inherits” the character of the community/area. This character involves the connectedness to society as measured by the signals.

The approach may not meet all the criteria for proper journal indicators and/or responsible use thereof, but it does provide a broad range of the indicators to assess the role of journals which are less vulnerable for gaming and manipulation than the existing (mostly) one-dimension journal indicator.

The dimensions considered in the ABC method discussed in this paper are based on the signals available to measure societal connectedness. For journal indicators we implement the following dimensions with the according signals between parentheses:

In the remainder of this section I discuss the dimensions one by one. For each dimension I discuss the signal being used as well as their distribution across the science landscape.

The News dimension relates to research being relevant in the broadest sense for the general public. It is assessed by the proportion of publications in a research area mentioned in news items. These news items are collected and included in Altmetrics from a large set of sources. We detect around 2,500 news sources. The top 10 most common sources are:

These approximately 2,500 sources are consulted for mentions of publications. I use these mentions as signal for connectedness in the following way. For each area, the proportion of papers being mention in a news item is calculated. These proportions range from 0 to 50% across the 4,000 areas. Their distribution is visualized in Fig. 2.

Figure 2

In Fig. 2, I use a color coding indicating the proportion in such a way that the Blue areas have a lower proportion or even zero in Dark Blue, while the Yellow areas have a relatively high proportion.

Fig. 2 shows a preferred distribution of the news signal, primarily in the biomedical sciences, social sciences and life & earth sciences. In these fields, the areas are more likely to color Yellow. This does not mean that, for instance, all biomedical research is connected to society in the news dimension, but these fields are more connected to society than others.

The policy dimension relates to research being relevant to political issues. For this dimension I use the signal of mentions of research output in policy documents. These documents are retrieved from 56 sources. The top 10 most commonly used organizations are:

The distribution of proportions of papers mentioned in policy documents across the landscape ranges between 0 and 19% and is visualized in Fig. 3.

Figure 3

The policy signal shows a preferred position in the social sciences, life & earth sciences and biomedical sciences. This preference is different from the news dimension. Such differences support the fact that one should look at more than just one dimension when considering connectedness to society.

The dimension related to the interest from the private sector is assessed by the authorship of companies in publications. For this dimension I calculate the proportion of output in which the author affiliation is a company in an area, ranging between 0 and 30%. The distribution of this signal is shown in Fig. 4.

Figure 4

In Fig. 4, we see quite a different picture than in the other dimensions. Here, the preferred areas are in computer sciences, physical sciences, engineering and, to a lesser extent, in biomedical sciences.

The technology dimension relates to the support or enabling strength of research. The signal I use to monitor this is citations from patents, operationalized by measuring the proportion of publications being cited in patent as Non-Patent Literature references. The distribution of this signal, ranging between 0 and 40%, is shown in Fig. 5.

Figure 5

In this case the signal is primarily detected in biomedical sciences and the chemistry part of the physical sciences. On top of that, we find somewhat stronger signals in engineering.

The dimension of local interest relates to the extent to which research is relevant for local issues. Most likely, research with a local interest is published in a local language. This maybe any language so that I defined it as non-English. English is the most international, i.e. non-local, language. That does not imply, however, that English language papers cannot be aiming at local issues. This in particular is an example why the ABC method is an interesting approach. A journal is usually in one language only. Therefore, characterizing journals by the proportion of non-English papers would neglect journals in English with a local interest. By using the characterization of the areas in which a journal is covered, inheriting their character, we can identify English language journals with a local interest.

The proportion of papers not in English by area ranges between 0 and 83%. The areas with really high proportions are often small. The distribution over the landscape is shown in Fig. 6.

Figure 6

The preferred areas for non-English publications are in social sciences and clinical and health sciences.