Assessing the inter-coder reliability of the Body Type Dictionary (BTD)

Open access

Abstract

Computer-assisted content analysis has many advantages compared to a manual scoring system, provided that computerised dictionaries represent valid and reliable measures. This study aimed to assess the inter-coder reliability, alternate- form reliability and scoring consistency of the Body Type Dictionary (BTD) (Wilson 2006) based on Fisher and Cleveland’s (1956, 1958) manual body boundary scoring scheme. The results indicated an acceptable inter-coder agreement with barrier and penetration imagery in the sub-sample (N = 53) of manually coded Rorschach responses. Additionally manually coded scores showed an acceptable correlation with the computerised frequency counts, and thus indicating an alternate-form reliability. In the full data set (N = 526), barrier imagery in the Rorschach responses only correlated with the picture response test, showing low scoring consistency, which might disconfirm the notion of body boundary awareness representing a stable personality trait but instead it might be dependent on the level of cognitive dedifferentiation.

Ackerman, Steven J., Mark J. Hilsenroth, Amanda J. Clemence, Robin Weatherill and I. Christopher Fowler. 2001. Convergent validity of Rorschach and TAT scales of object relations. Journal of Personality Assessment 77: 295-306.

Anzieu, Didier. 1989. The skin ego. New Haven: Yale University Press.

Argamon, Shlomo and Shlomo Levitan. 2005. Measuring the usefulness of function words for authorship attribution. Proceedings of the 2005 ACH/ ALLC conference, June 2005. Victoria, Canada.

Artstein, Ron and Massimo Poesio. 2007. Inter-coder agreement for computational linguistics. Computational Linguistics 34: 555-596.

Bick, Esther. 1968. Experience of the skin in early object relations. International Journal of Psycho-Analysis 49: 484-486.

Buchanan, Tom. 2002. One assessment: Desirable or dangerous? Professional Psychology: Research and Practice 33: 148-154.

Buck, Lucien A. and Michael Barden. 1971. Body image scores and varieties of consciousness. Journal of Personality Assessment 35: 309-314.

Burris, Christopher T. and John K. Rempel. 2004. ‘It’s the end of the world as we know it’: Threat and the spatial-symbolic self. Journal of Personality and Social Psychology 86: 19-42.

Cariola, Laura A. 2012. A case study of primary process language and body boundary imagery in discourses of religious-mystical and psychotic altered states of consciousness. Empirical Text and Cultural Research - ETC 5: 36-61.

Chang, Cindy and James W. Pennebaker. 2007. The psychology of function words. In K. Fiedler (ed.). Social communication, 343-359. New York: Psychology Press.

Fisher, Seymour and Sidney E. Cleveland. 1956. Body-image boundaries and style of life. Journal of Abnormal and Social Psychology 52: 373-379.

Fisher, Seymour and Sidney E. Cleveland. 1958. Body image and personality. New York: Dover Publications.

Fisher, Seymour. 1986. Development and structure of the body image. Hillsdale: Lawrence Erlbaum.

Fleiss, Joseph L. 1981. Statistical methods for rates and proportions. New York: John Wiley.

Fox, John. 2005. The R commander: A basic statistics graphical user interface to R. Journal of Statistical Software 14: 1-42.

Friedman, Milton. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association 32: 675-701.

Garmer, Matthias, Jim Lemon, Ian Fellows and Suspendra Singh. 2012. Various coefficients of interrater reliability and agreement. Available at http://www.cran.r-project.org/web/packages/irr/irr.pdf. Last accessed on 19 January, 2014.

Gigerenzer, Gerd. 2004. Mindless statistics. The Journal of Socio-Economics 33: 587-606.

Guimon, Jose. 1997. Corporality and psychoses. In J. Guimon (ed.). The body in psychotherapy, 63-72. Basel: Karger.

Hogenraad, Robert, Claude Daubies, Yves Bestgen and Pierre Mahau. 2003. Une théorie et une méthode générale d’analyse textuelle assistée par ordinateur: Le systéme PROTAN (PROTocol ANalyzer). 32-bits version of November 10, 2003 by Pierre Mahau. Louvain-la-Neuve: Psychology Department, Catholic University of Louvain.

Jackson, Sherri. 2011. Research methods and statistics: A critical thinking approach. Belmont: Wadsworth.

Kolbe, Richard H. and Melissa S. Burnett. 1991. Content-analysis research: An examination of applications with directives for improving research reliability and objectivity. Journal of Consumer Research 18: 243-250.

Krippendorff, Klaus. 2004. Content analysis: An introduction to its methodology. Thousand Oaks: Sage. (First published 1980).

Lacy, Stephen and Daniel Riffle. 1996. Sampling error and selecting intercoder reliability samples for nominal content categories: Sins of omission and commission in mass communication quantitative research. Journalism & Mass Communication Quarterly 73: 969-973.

Lombard, Matthew, Jennifer Snyder-Duch and Cheryl Campanella Bracken. 2002. Content analysis in mass communication: Assessment and reporting of inter-coder reliability. Human Communication Research 28: 587-604.

Lombard, Matthew, Jennifer Snyder-Duch and Cheryl Bracken. 2010. Practical resources for assessing and reporting intercoder reliability in content analysis research projects. Available at http://matthewlombard.com/reliability/. Last accessed on 29 January, 2014.

Michalczyk, Alan E. and Lloyd A. Lewis. 1980. Significance alone is not enough. Journal of Medical Education 55: 835-838.

Morgan, Christina D. and Henry A. Murray. 1935. A method of investigating fantasies: The Thematic Apperception Test. Archives of Neurology and Psychiatry 34: 289-306.

Neuendorf, Kimberly A. 2002. The content analysis guidebook. London: Sage Publications.

Neundorf, Kimberly A. and Paul D. Skalsi. 2010. Extending the utility of content analysis via the scientific method. Manuscript in support of presentation to the Social Science Computing Workshop, University of Hawaii, Honolulu. Available at http://www.manoa.hawaii.edu/ccpv/workshops/KimberlyNeuendorf.pdf.

Ogden, Thomas H. 1989. The primitive edge of experience. Northvale: Jason Aronson. O’Neill, Regina M. 2005. Body image, body boundary and the Barrier and Penetration Rorschach scoring system. In R. F. Bornstein and J. M. Masling (eds.). Scoring the Rorschach: Seven validated systems, 159-189. London: Lawrence Erlbaum.

Orne, Martin T. 1962. On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American Psychologist 17: 776-783.

Passonneau, Rebecca. 2006. Measuring agreement on set-valued items (MASI) for semantic and pragmatic annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC). Genoa, Italy.

Potter, W. James and Deborah Levine-Donnerstein. 1999. Rethinking validity and reliability in content analysis. Journal of Applied Communication Research 27: 258-284.

R Development Core Team. 2011. R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing. Available at http://www.R-project.org. Last accessed on 19 January, 2014.

Rorschach, Hermann 1921. Psychodiagnostik. Leipzig: Ernst Bircher Verlag.

Rosenthal, Robert and Ralph L. Rosnow. 1984. Essentials of behavioural research: Methods and data analysis. London: McGraw-Hill.

Rourke, Liam, Terry Anderson, D. R. Garrison and Walter Archer. 2000. Methodology issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education 11: 8-22.

Saraceni, Carlo, Giuseppe Ruggeri and D. Filocamo. 1980. Studio sperimentale con il test di Rorschach sulle modificazioni dell’immagine corporea in ipnosi. Archivio di Psicologia, Neurologia e Psichiatria 41: 50-64.

Schmeidler, Gertrude and Lawrence LeShan. 1970. An aspect of body image related to ESP scores. Journal of the American Society for Psychical Research 64: 211-218.

Schmitt, Norbert and Bruce Dunham. 1999. Exploring native and non-native intuitions of Word frequency. Second Language Research 15: 389-411.

Schnurr, Paula P., Stanley D. Rosenberg, Thomas E. Oxman and Gary J. Tucker. 1986. A methodological note on content analysis: Estimates of reliability. Journal of Personality Assessment 50: 601-609.

Singletary, Michael W. 1994. Mass communication research: Contemporary methods and applications. New York: Longman. Spearman, Charles. 1904. The proof and measurement of association between two things. American Journal of Psychology 15: 72-101.

Tinsley, Howard. E. A. and David J. Weiss. 1975. Interrater reliability and agreement of subjective judgements. Journal of Counseling Psychology 22: 358-376.

Tustin, Frances. 1981. Autistic states in children. London: Routledge.

Weber, Robert P. 1990. Basic content analysis. London: Sage.

West, Alan N. 1991. Primary process content in the King James Bible: The five stages of Christian mysticism. Computers and the Humanities 25: 227-238.

Wilson, Andrew. 2006. The development and application of a content analysis dictionary for body boundary research. Literary and Linguistic Computing 21: 105-110.

Wilson, Andrew. 2008. Psychosomatic cycles and the liturgical year: A case study and framework for research. Gottingen: Cuvillier Verlag.

Wilson, Andrew. 2009. Barrier and penetration imagery in altered states of consciousness discourse: Replicating the five-stage model of Christian mysticism in the Bible. In W. Oleksy and P. Stalmaszczyk (eds.). Cognitive approaches to language and linguistic data: Studies in honor of Barbara Lewandowska-Tomaszczyk (Polish Studies in English Language and Literature 27), 357-372. Frankfurt am Main: Peter Lang.

Wilson, Andrew. 2011. The regressive imagery dictionary: A test of its concurrent validity in English, German, Latin and Portuguese. Literary and Linguistic Computing 26: 125-135.

Journal Information

Metrics

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 58 58 25
PDF Downloads 14 14 6