Measuring and Visualizing Research Collaboration and Productivity

To help evaluate the performance of a large interdisciplinary, cross-institutional research project, we addressed a challenging complex of attributes. The five-year “innovative Urban Transitions and Aridregion Hydro-sustainability” (iUTAH) project was initiated in 2012 with support from the US National Science Foundation’s (NSF) “Established Program to Stimulate Competitive Research” (EPSCoR). The iUTAH research proposal mandated a comprehensive research collaboration assessment in Year 5 that would “…analyze the publication and citation patterns of researchers at all Utah institutions to identify the level of interconnectedness before iUTAH is awarded and in year 4 of the award [to] indicate the connections between institutions, disciplines, and individuals that were stimulated by the iUTAH activities.”

In the past, we and others have cast a variety of analytical and visualization tools to help analyze research attributes (Porter et al., 2010). Such measurement tools fall generally in the category of bibliometrics (De Bellis, 2009), with special note of:

Measures of citation diversity to help gauge interdisciplinary research knowledge interchange (Zhang, Rousseau, & Gläzel, 2016), including “Integration scores” (Porter et al., 2007; 2008), Rao-Stirling diversity (Rafols & Meyer, 2010; Stirling, 2007), and Diffusion scores (Carley & Porter, 2012; Garner, Porter, & Newman, 2014)

Measuring the interdisciplinarity of research outputs is problematic. Wagner et al. (2011) review the state of the art and conclude that indicators of interdisciplinarity are not well-validated. Integration of knowledge into research designs and processes entails cognitive processes, with additional complexities when that research involves a team (c.f., the rich discourse in conjunction with the Science of Team Science— http://www.scienceofteamscience.org/). Acknowledging such considerations, we have devised measures of research knowledge integration (interdisciplinarity) based on citation diversity, as noted just above, and apply those in this study.

“Diversity” is a key concept on which many measures focus (Stirling, 2007). Stirling’s three-component model that considers variety, balance, and disparity is compelling. It is equivalent to the empirical approach of measuring “Integration” devised in support of evaluation efforts for the US National Academies Keck Futures Initiative (Porter, Roessner, & Heberger, 2008). Rafols (2014) treats various measures of diversity and coherence. Of note are recent efforts to measure variety, balance, and disparity separately (Wang, Thijs, & Glänzel, 2014; Yegros-Yegros et al., 2010; Yegros-Yegros, Rafols, & d’Este, 2015). For this research assessment, the Integration scoring provided a suitable measure of interdisciplinarity.

Integration and Diffusion scoring, and the science overlay maps, use Web of Science Categories (WoSCs) as the basic unit of sub-discipline categorization (Leydesdorff & Bornmann, 2016). There are various approaches, and attendant issues (Glänzel & Schubert, 2003; Klavans & Boyack, 2009, 2016; Rafols & Leydesdorff, 2009), but the WoSCs offer suitable sub-disciplinary granularity to accord with a key National Academies report’s recommendations (National Academies, 2005). In our experience, granularity should match the study’s main objectives. Recently we have tuned journal assignments to focus on research knowledge interchange among Cognitive Sciences, Education Research, and associated Border Fields (Youtie et al., 2017). We have generated informative results that consolidated the 200+ WoSCs into four “meta-disciplines” to assess interest (citations) from the natural sciences to a US NSF social science program (Garner et al., 2013). Or, one can seek much finer grain, as Klavans and Boyack (2017) demonstrate using some 91,000 topics to predict grant funding prospects. For the iUTAH assessment, interest keyed on the extent to which Center participants collaborated across disciplines, for which the WoSCs provided a manageable categorization.

We have applied various research assessment tools in the course of different research assessments of US federal funding programs, particularly to gauge interdisciplinarity. These include Environmental Protection Agency Science to Achieve Results (STAR) projects (Porter et al., 2003), and NSF Research and Evaluation on Education in Science & Engineering (REESE—Porter et al., 2013), Human and Social Dynamics Program (HSD—Garner et al., 2013), and Research Coordination Networks (RCN—Porter, Garner, & Crowl, 2012; Garner et al., 2012).

What one cites in a paper depends on multiple factors, including relevance, disciplinary training, disciplinary norms, and awareness (c.f., De Bellis, 2009). For instance, in a past bibliometric analysis, we observed stark differences in influence of nanotechnology Environmental, Health, and Safety (EHS) research findings, as gauged by citation, within the larger nanotechnology research community (Youtie et al., 2011).

Turning to the assessment task at hand, we focus on connections among institutions, disciplines, and individuals that were possibly bolstered by iUTAH activities. iUTAH aims to strengthen science regarding management of the state’s water resources

http://iutahepscor.org/about.shtml

This paper reports on our efforts to combine and extend analytical and visualization tools to assess iUTAH’s contributions on multiple factors. We want to measure “connections” made across individuals, organizations, and disciplines, with express interest in variations by seniority (academic rank) and gender. We concentrate on research outputs and their impacts (in the form of citations); we do not address process data per se (i.e., no interviews or surveys of participants to document the nature of interactions).

We offer this study as a model of applying bibliometric tools to help assess the research outputs of a major center. It illustrates a quasi-experimental study design to provide practical comparisons that help gauge the effects of instituting the center. While one cannot generalize from a single such study, we offer it to suggest possible measurement approaches.

Figures 5 and 6 convey the most compelling results of our research assessment —a marked increase in collaboration attributable to participation in iUTAH activities. These charts contain four quadrants somewhat unequal in size and shape—separated by hand-drawn dashed lines), representing the institutions engaged—UU, USU, BYU, and PUIs. Nodes represent individual researchers, with larger nodes indicating more total publications (Google Scholar and WoS combined)

Network maps are made using VantagePoint software [www.theVantagePoint.com] and exported to Pajek [http://vlado.fmf.uni-lj.si/pub/networks/pajek/]. Nodes are positioned by dragging them to a location on the screen, in this case by the university the researcher is associated with. The resulting map is saved and used to re-create maps for both time periods. Colors are applied using partitions that are created using a text editor to create a Pajek-friendly format and loaded into Pajek. Images are then exported using SVG formats. Inkscape [https://inkscape.org/en/] is used to add labels and the final image is exported into png format.

Figure 5

Figure 6

Figure 5 maps co-authoring among the iUTAH researchers for 2010–2012 (This period precedes iUTAH activities that would affect research publication to any substantial degree). Interconnections across institutions are almost nonexistent.

Contrasting Figures 5 and 6, we note a substantial increase in networking within universities among many of the iUTAH participants (not all). This is pronounced for UU and USU, but less so for BYU and the PUIs. Even more dramatic is the large increase in connections between UU and USU colleagues in the After period.

Table 2 documents what is apparent in comparing Figures 5 and 6—marked increases in collaboration. This table tallies network statistics corresponding to Figures 5 and 6, as well as to Figures S-5, S-6, and S-7 (Supplemental Materials). Most striking is the confirmation of the visual patterns in Figures 5 and 6—iUTAH researchers increase their collaboration to a striking degree from Before to After.

Table 2

Networking statistics within Utah researcher cohorts.

This is clear by inspection, with increases running more than 4-fold—note Average Degree (up from 1.2 to 5.4), Density (up from 0.03 to 0.12), and various tallies of links (e.g., total co-authorship links among the iUTAH researchers increase from 27 to 122).

Likewise, Table 2 comparisons between iUTAH After and CG1 After show large differences here, as also shown in comparing Figures 6 and 8. For one measure, total links in iUTAH After are 122 vs. 4 for CG1 After—i.e., heavy vs. minimal co-authoring among the respective groups of 45 researchers. Average degree or density comparisons are similarly extreme.

Figure 7

Figure 8

Figures 5 and 6 also color code the five disciplinary groupings within iUTAH. Focusing for a moment just on the USU and UU researchers:

Of 20 USU faculty in our sample, most show multiple links to other researchers; only two show no collaborations with iUTAH colleagues on their publications in the After period.

Figures 7 and 8 provide the counterpart CG1 comparisons. [We choose to omit these analyses for CG2 as there is no reason to anticipate substantial collaboration among those individual researchers randomly assigned to a comparison group.] We see surprisingly little collaboration in CG1, considering that these are researchers associated with integrated environmental research centers. Since those are single university centers, we would not expect much UU-USU linkage. Also, BYU and the PUIs are not engaged in these centers. It is striking, since GCSC and EC were active in both periods, that the paucity of co-authoring sharply contrasts to the After period in which iUTAH was active (Figure 6).

Supplemental Figures S-6a and S-6b display iUTAH researchers by gender; we did not observe remarkable differences in group engagement of men versus women. Figures S-7a and S-7b are counterparts of Figures 5 and 6. They color-code rank instead of discipline. The impression is that USU has been more successful compared to other institutions in engaging their early-career faculty in research collaboration.

Figure S-6a

Figure S-6b

Figure S-7a

Figure S-7b

Substantively, we describe the publications, and citations to those publications, emanating from the researchers actively involved in the iUTAH project. We are expressly interested in the cross-disciplinary nature of the research, and show that it, indeed, engages many disciplines.

Moreover, we have run several quasi-experimental comparisons to compare those iUTAH research outputs to other environmental center participants (CG1) and to a matched set of individual researchers at Utah universities (CG2). These indicate that the iUTAH cohort participants are highly productive (equivalent to CG1; higher than CG2 researchers) prior to engaging in iUTAH, and that their publication activity appears to be increased by iUTAH involvement. CG1 publications in the 2010–2012 period attract somewhat more citations than the papers of researchers who join iUTAH (this is the Before period)

Again we note that the CG1 centers were in operation in the 2010–2012 period, so, in that sense, they are not “Before.”

We note certain limitations in these analyses. This is a quasi-experimental study, so comparisons are somewhat guarded in the absence of randomized control (which is not plausible in such a real case environment). Given the realities of composing an interdisciplinary research organization, such a design is certainly in order. [i.e., researchers are not amenable to being randomly assigned to join research centers or not.] We sought best available comparisons, but have noted limitations; for instance, the CG1 centers were operating during the period Before iUTAH began work. WoS coverage of publications in the After period are not as complete as for the Before period, due to lags in indexing. More serious, as discussed, citations accrued by those publications are more severely truncated for the After period. Such limitations are reasonably compensated by having the cross-group comparisons of iUTAH to CG1 and CG2. Tallying citations is problematic in several regards, some already noted (e.g., field differences). Fixed citation windows offer advantages vis-à-vis citations per year. Citation rates change over time and recent periods are apt to be less completely indexed by WoS.

Before vs. After citation comparisons are clouded by difficulties in adjusting citation propensities fairly (Zhang et al., 2017). We use cites per year since publication, but see advantages in the alternative of cites in 10 (or five) years postpublication. Hall et al. (2012) compared tobacco research center vs. individual grantee (National Institutes of Health R-01) publication rates. They too found that center researchers published more, but only after four years of project support (before that they lagged). Further assessment might want to compare citations received on a year-by-year basis. Gathering data on the year in which the citing document was published would enable such comparisons. We did not have those data; they require capturing information on each citing document individually. In contrast WoS readily provides consolidated “Times Cited” information within the abstract publication record, and that is what we used in this study.

One of us noted the apparent disconnect between collaboration and interdisciplinarity, as gauged by Integration scores. Presumably, cross-disciplinary collaboration offers a route to garner insights from the multiple disciplines represented; so Integration scores that measure diversity of references cited in a paper might be expected to increase. For our iUTAH researchers, collaboration within the 45 researchers escalates from Before to After, but Integration scores do not. As shown in the bottom row of Table 1, those scores do not change significantly. However, Integration scores do increase from Before to After for CG1 and CG2.

This lack of an apparent correlation between Integration scores of the papers and extent of within-group collaboration caused us to consider collaboration more broadly. While not the focus of this assessment that seeks to gauge change in within-group (the 45 iUTAH researchers) collaboration, one could examine collaboration generally. We introduce this here briefly to suggest future research potential in analyzing overall, as well as local, collaboration patterns. Table 3 offers some basic comparisons.

Table 3

Collaborative extent for all co-authoring on the Utah research group papers (indexed in Web of Science).

In brief, the After period papers of both center groups (iUTAH and CG1) show more authors per paper and more author affiliations (these are calculated at the organizational level; it was too difficult to measure departmental affiliations uniformly from the WoS records) per paper. The iUTAH increase in organizational affiliations is consistent with the expansion of cross-Utah organizational ties from Before to After (Table 2). Not an issue here, but we did run across articles with “mega” authoring (i.e., hundreds on a paper) in preliminary searches. Were those in one’s data set, they pose assessment challenges in discerning their degree of relationship (say to Center engagement) and statistical oddities.

To sum up, methodologically, we offer a multi-attribute suite of analytical and visual elements to help assess research outputs and impacts. Of particular interest are measures of research collaboration. We believe the combination of representations provides important, complementary perspectives. In particular, we find Figures 5 and 6 effective in capturing research networking attendant to interdisciplinary center activities. Without undue complexity, they graphically show changes from Before to After, and by comparing to counterpart figures for a comparison group, differences between iUTAH and CG1 here. In a presentation to an “All-Hands” meeting of iUTAH participants (July, 2017), they communicated effectively. These figures also show the nature of cross-organization and cross-disciplinary connections made among a group of researchers. [Supplemental figure variants do likewise for gender and ranks.]

The “take-away” from this study for others is a model of multiple measures of research outputs (publication characteristics) and impacts (citation characteristics). These could be adapted to meet assessment needs of other studies for which cross-disciplinarity and research collaboration are vital elements. In our view Figures 5–8, the co-author maps, are most novel—offering a concise way to communicate several facets of collaboration concisely. Perhaps this warrants consideration as a bibliometric assessment case study?