Article Category: Research Papers
Published Online: Apr 30, 2024
Page range: -
Received: Feb 06, 2024
Accepted: Apr 22, 2024
DOI: https://doi.org/10.2478/jdis-2024-0013
Keywords
© 2024 Mike Thelwall, published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Purpose
Assess whether ChatGPT 4.0 is accurate enough to perform research evaluations on journal articles to automate this time-consuming task.
Design/methodology/approach
Test the extent to which ChatGPT-4 can assess the quality of journal articles using a case study of the published scoring guidelines of the UK Research Excellence Framework (REF) 2021 to create a research evaluation ChatGPT. This was applied to 51 of my own articles and compared against my own quality judgements.
Findings
ChatGPT-4 can produce plausible document summaries and quality evaluation rationales that match the REF criteria. Its overall scores have weak correlations with my self-evaluation scores of the same documents (averaging r=0.281 over 15 iterations, with 8 being statistically significantly different from 0). In contrast, the average scores from the 15 iterations produced a statistically significant positive correlation of 0.509. Thus, averaging scores from multiple ChatGPT-4 rounds seems more effective than individual scores. The positive correlation may be due to ChatGPT being able to extract the author’s significance, rigour, and originality claims from inside each paper. If my weakest articles are removed, then the correlation with average scores (r=0.200) falls below statistical significance, suggesting that ChatGPT struggles to make fine-grained evaluations.
Research limitations
The data is self-evaluations of a convenience sample of articles from one academic in one field.
Practical implications
Overall, ChatGPT does not yet seem to be accurate enough to be trusted for any formal or informal research quality evaluation tasks. Research evaluators, including journal editors, should therefore take steps to control its use.
Originality/value
This is the first published attempt at post-publication expert review accuracy testing for ChatGPT.