Usability Evaluation of E-Dunhuang Cultural Heritage Digital Library

Open access


Digital libraries have been strategic in preserving and making non-movable cultural heritage information accessible to everyone with network connections. In light of their cultural and historical importance in the ancient “Silk Road,” murals and stone caves in Dunhuang, a remote city in northwest China,have been digitized, and the first batch of digitized visual materials has been made available to the general public through the e-Dunhuang digital library since May 2016. The aim of this study was to systematically evaluate e-Dunhuang from users’ perspectives, through usability testing with nine user tasks in different complexity levels and in-depth interviews with regard to a set of criteria in user experience. The results of quantitative analysis confirmed the overall effectiveness of e-Dunhuang in supporting user task completion and demonstrated significant improvements in several criteria over an earlier panorama collection of Dunhuang caves. The results of qualitative analysis revealed in-depth reasons for why participants felt satisfied with some criteria but had concerns with other criteria. Based on the findings, suggestions are proposed for further improvement in e-Dunhuang. As e-Dunhuang is a representative repository of digitized visual materials of cultural heritage, this study offers insights and empirical findings on user-centered evaluation of cultural heritage digital libraries.


Digital libraries have been strategic in preserving and making non-movable cultural heritage information accessible to everyone with network connections. In light of their cultural and historical importance in the ancient “Silk Road,” murals and stone caves in Dunhuang, a remote city in northwest China,have been digitized, and the first batch of digitized visual materials has been made available to the general public through the e-Dunhuang digital library since May 2016. The aim of this study was to systematically evaluate e-Dunhuang from users’ perspectives, through usability testing with nine user tasks in different complexity levels and in-depth interviews with regard to a set of criteria in user experience. The results of quantitative analysis confirmed the overall effectiveness of e-Dunhuang in supporting user task completion and demonstrated significant improvements in several criteria over an earlier panorama collection of Dunhuang caves. The results of qualitative analysis revealed in-depth reasons for why participants felt satisfied with some criteria but had concerns with other criteria. Based on the findings, suggestions are proposed for further improvement in e-Dunhuang. As e-Dunhuang is a representative repository of digitized visual materials of cultural heritage, this study offers insights and empirical findings on user-centered evaluation of cultural heritage digital libraries.

1 Introduction

Studies of cultural heritages are invaluable in many disciplines including archeology, history, anthropology, linguistics, and religious studies. Among tangible cultural heritage (i.e., those with physical presence), many are non-movable such as monuments, architecture, and archeological sites. To observe non-movable heritages, people have to travel to the heritage site. Moreover, most non-movable heritages are exposed to the natural environment with limited protection, suffering from deformation and deterioration caused by weather or other physical or chemical reactions. Digitalizing these heritages and constructing digital libraries of the digitized materials are considered to be viable solutions to these problems (Evens & Hauttekeete, 2011;Piccialli & Chianese, 2017). Libraries, museums, and research organizations are currently developing collections of digitized heritages Digital Dunhuang is one of the largest digitizing cultural heritage projects in the world, involving scholars from many countries and several disciplines (Zhou, 2011; Wang et al., 2015).

Dunhuang is an ancient city in northwest China. Due to its strategic location, Dunhuang was an important hub on the “Silk Road” from about 300 A.D. to 1300 A.D. As it is located at the junction of the northern and southern Silk Roads, it was also one of the pivotal gateways between China and other parts of the world for nearly 1000 years. Embracing cultural and religious influences, a unique cultural heritage emerged mainly in the Buddhist community in ancient Dunhuang. Dunhuang is geographically proximate to Mogao Cave, a site inscribed on the World Heritage List since 1987, where hundreds of stone cave temples and murals were built in ancient China across 10 dynasties (from the 4th to the 14th century A.D.). Classic murals account for more than 4000 m2 depicting several themes during the time, including culture, religion, economics, and daily dresses. Dunhuang heritage is of exceptionally high value in historical, cultural, artistic, and religious terms.

1.1 The e-Dunhuang digital library

E-Dunhuang1 was released to the public by the Dunhuang Academy in China in May 2016 as an online, permanent archive of digital resources of Dunhuang heritage for public access. Aiming to conserve the cultural heritage by photographic techniques and digital modeling in three dimensions, the site includes high-resolution images, panorama views, and related descriptive information on the caves and classic murals in Dunhuang. Figures 13 illustrate the screenshots of the digital library with English annotations added by the authors.

Figure 1

Download Figure

Figure 1

Home page of e-Dunhuang.

Citation: Data and Information Management 2, 2; 10.2478/dim-2018-0008

Figure 2

Download Figure

Figure 2

Menu page filter conditions.

Citation: Data and Information Management 2, 2; 10.2478/dim-2018-0008

Figure 3

Download Figure

Figure 3

The page of Cave 285; retrieved from

Citation: Data and Information Management 2, 2; 10.2478/dim-2018-0008

E-Dunhuang is the first public-oriented, fully fledged digital library consisting of high-resolution digitized resources produced by the Digital Dunhuang project. A precursor of e-Dunhuang is a panorama digital collection accessible on the tourism resource website2 of Dunhuang Research Academy. With 360° panorama views of 14 of the Mogao Caves, the panorama digital collection allows public users to see high resolution images of murals and statues in the caves. However, this collection only provides minimum text labels and no text description, limiting the scope and depth of support for users. The panorama collection was recently evaluated by Hu and colleagues (Hu, Ho & Qiao, 2017a) in terms of user experience, which helped identify usability issues and suggested ways of improving the collection. The panorama views are also included in the e-Dunhuang digital library as one of its featured functions, while e-Dunhuang presents much richer metadata and descriptions, as well as more browsing and searching functions.

1.2 Research questions

To the best of our knowledge, e-Dunhuang has not been systematically evaluated from a user’s perspective. User experience, with its emphasis on human perception and user–system interactions, is considered a vital aspect of digital library evaluation. Through the process of interacting with the system, users’ awareness, preferences, emotions, and psychological behaviors can be revealed (Kuhlthau, 1991; ISO, 2009). Cognizant of this interactive process, this study conducted a user evaluation of e-Dunhuang through usability tests and in-depth interviews. Results will not only be instructive for further improvement in e-Dunhuang and the Digital Dunhuang project but also be insightful for other cultural heritage digitization projects worldwide. The following research questions are proposed:

  • To what extent does e-Dunhuang facilitate users conducting information-seeking tasks with different complexity levels?

  • To what extent does e-Dunhuang meet the criteria of usability evaluation?

The first question reveals how end users can fulfill typical information-seeking tasks with e-Dunhuang. Answers to the second question will shed light on detailed reasons for users’ evaluation, from which suggestions could be made to system designers on improving the user experience of e-Dunhuang. As e-Dunhuang is an example of a collection of digitized World Heritage, the methods and results are of value for other cultural heritage digital libraries.

2 Related Work

2.1 Usability testing of digital libraries

Usability of a digital resource refers to the following: 1) how the resource supports the user’s needs (Matusiak, 2012); 2) how easy the user perceives the resource is to use, the quality of the user experience (Kelly, 2014); and 3) how useful the resource is to the information needs of the user (Dalkiran et al., 2014; Zha et al., 2015). How well the user perceives the resource to meet the user’s needs and evaluation of the usability of a digital library can be conceived as an interplay of four components: users, task, tool, and environment (Jeng, 2009).

The process of evaluating usability in digital libraries can be conducted via experimental and analytical methods or a combination of experimental and analytical methodologies (Dalkiran et al. 2014). Experimental methods are based on gathering data from users, whereas analytical methods rely on expert judgment. Checklist, heuristic evaluations, and cognitive walkthroughs are examples of analytical methods (Dalkiran et al. 2014). Formal usability tests, interviews, focus groups, and questionnaires are examples of experimental methods. As one of the most fundamental and effective tools to evaluate services and information systems, usability testing has been widely adopted in different domains, including local, national, and international government websites (Dolan, 2014; Ataloglou & Economides, 2009), health information systems (Heradio Fernández-Amorós, Cabrerizo & Herrera-Viedma, 2012; Davis & Jiang, 2016), and mobile information applications (Cáliz, Martínez, Alamán, Terán & Cáliz, 2016). Usability testing is an important tool for the development, assessment, and evaluation of all digital library systems (Dalkiran, Aker, Öztemiz, Taşkin & Tunç, 2014; Li, 2013; Brahaj, Razum & Hoxha, 2013). It is the careful and systemic observation of “users interacting with the system,” either in person, or using screen capture software, or both (Comeaux, 2008: p. 459).

2.2 Cultural heritage digital libraries and user experience evaluation

In recent decades, digitization of cultural heritage sites has become a keenly studied topic (Evens & Hauttekeete, 2011) with pilot projects in existence since the late 1980s and early 1990s (Terras, 2011). Assessing and evaluating the extent to which a digital library satisfies the needs of its users are the most frequently used evaluation paradigms for digital libraries (Kelly, 2014; Todd-Diaz & O’Hare, 2014) as well as for digital museums and cultural heritage sites (e.g., Barbieri, Bruno & Muzzupappa, 2017; Artese, Ciocca & Gagliardi, 2017). User experience evaluation can not only reveal which aspects of these digital libraries are doing well and which could be further improved but also indicate what factors are important for users, providing insights for future studies on digital library users and digital library design, so that the effectiveness, functionality, and usability of the library are maximized (Picazo, Saz & Villalta, 2009).

The existing studies have evaluated users’ experience of digital libraries, museum, and cultural heritage sites (Kabassi, 2016) on a wide range of aspects, such as effectiveness, efficiency, learnability, or subjective satisfaction (Li, 2013). An evaluation framework often adopted by studies on digital libraries is the usability evaluation model devised by Jeng (2009). Jeng referred to ISO 9241-11 (2009), providing a usability evaluation framework that comprised elements of effectiveness, efficiency, and satisfaction (ease of use, organization of information, labeling, visual appearance, content, error correction). Liu (2003) proposed an interactive scale to help determine to what extent a website met the needs of users. Dimensions to determine the degree to which a user of a digital technology is interacting with a website include the following: active control, two-way communication, and synchronicity. The current study combined the interactivity dimensions of Liu’s work and Jeng’s usability evaluation framework to determine what elements of e-Dunhuang are working well for users and what elements can be improved.

3 Research Design and Methods

In this study, we adopted usability testing where each participant was given a set of predefined information-seeking tasks (Appendix) to complete. The time spent on completing each task and success rate (Kelly, 2009) were noted, which were the main measures of user effectiveness in evaluating interactive information retrieval systems. During the tests, thoughts and opinions of the participants were recorded with reference to the think-aloud protocol (Zhang, 2010). Following the usability testing, participants were invited to fill a short questionnaire rating the digital library against a set of evaluation criteria (Table 1) in a 5-point Likert scale, with 1 being the least agreeable value (e.g., difficult, unclear, and disagree) and 5 being the most agreeable (e.g., easy, clear, and agree). Afterward, the participants were individually interviewed with a set of open-ended questions on the basis of the criteria for sharing their experience and views in detail.

Table 1

Definitions and interview questions of evaluation criteria (derived from Jeng, 2005; Liu, 2003).

Ease of useDetermines whether the user finds the system easy to useWas the DL easy to use? Why or why not?
Organization of informationCaptures user’s satisfaction with the organization, layout, and structure of the systemHow well do you think the information is organized on the DL?
LabelingEvaluates whether the terminology of the system is easy to understand from users’ perspectives with clear labelingHow do you think the labels in the DL?
Visual appearanceRefers to the visual estheticism of the systemHow visually attractive do you think the DL is?
Error correctionDescribes whether the system’s design is prone to mistakes made by users and whether users can correct the mistakes easilyCan you recover from mistakes easily? Please give an example
ContentCorresponds to the accuracy and authority of the information provided on the systemWhat do you think about the accuracy of the information on the DL?
Active controlEvaluates users’ ability to control and voluntarily participate in the system while instrumentally influencing the communicationWhen exploring the DL, do you have a lot of control over what you can do? Why or why not?
Two-way communicationConsiders the bidirectional information flow when using the systemDo you think the DL allows two-way communication between you and the DL? Why or why not?
SynchronicityTests the speed of system’s response when it comes to interactionDo you think the DL is fast in responding to your requests? Please give an example

This study followed the framework established in the study of evaluating the Mogao Cave panorama collection (Hu et al., 2017a), where a set of nine user tasks were designed specifically for Dunhuang heritage, ranging from simple factual tasks to more complex tasks involving comparisons and judgments (Appendix). An advantage of adopting this framework was to be able to compare the results to those in the study by Hu et al (2017a).

The usability test, short questionnaire, and interviews were conducted in one single session for each participant ranging from 1 to 1.5 hours. All the think-aloud usability test sessions and interviews with participants were audio recorded and transcribed for analysis purposes. The participation of interviewees was voluntary with no remuneration provided. Data analysis included statistical tests and qualitative summary and quotations of the think-aloud and interview transcripts.

4 Results and Discussion

4.1 Participants

There were 16 participants (nine females) in this study, who were recruited through the researchers’ professional and social networks (i.e., convenience sampling). The sample size was recognized as effective for identifying usability issues for websites (Faulkner, 2003; Granić, Mitrović & Marangunić, 2011), but it was not intended for generalization of results. None of the participants were involved in the e-Dunhuang project. The age of the participants ranged from 22 to 32 years old, with 26 years being the mean age. Occupation-wise, three teachers (two in Chinese and one in History), one PhD student in library science, one assistant librarian, and one journalist, were regarded as professional users as their jobs involved cultural heritage information and they either sought for or managed cultural heritage information in their work. The remaining participants were from the general public, as they looked for cultural heritage information only out of personal interest. This group included five graduate students, two translators, two bankers, and one nurse. As the Dunhuang heritage was closely related to Buddhism and China, we asked the participants’ religious beliefs and cultural background. Most (13) of them had no religious beliefs, while two were Buddhists and one was Christian. A total of 15 participants identified themselves as being of mainland China background, while one was born and raised in Hong Kong. As the participants were all Chinese, the usability test and interviews were conducted in Mandarin.

4.2 Quantitative analysis of evaluation results

Usability tasks were completed with high success3 rates for most tasks except for task 6 which was a judgment task on identifying a cave with different styles of Flying Apsaras, a set of divine characters unique to Dunhuang caves. Completion time of all tasks and respective successful rates were analyzed as summarized in Table 2. Comparatively, tasks 1–5, which belonged to the categories of simple fact or comparison of facts (Appendix), yielded a success rate of 93.75%, while tasks 6–9, which involved judgments and comparison of judgments, yielded a much lower success rate of 78.13%. This showed that participants performed better in searching for simple facts and tackling simple comparisons when compared to making judgments. A Student’s t-test on task completion time between successful completions and unsuccessful ones indicated a significant difference (p = 0.002). Unsuccessful completions took much more time than successful ones, with mean completion times being 225.85 and 159.66 seconds, respectively. Among the successful completions, we compared the task completion time of tasks 1–5 and that of tasks 6–9, and the former was significantly shorter (141.31 seconds on average) than the latter (187.75 seconds on average; p = 0.002). This indicated that more complex tasks took significantly more time of the users to complete than simple tasks.

Table 2

Task completion statistics.

Average time (seconds)134.3143.464.7210.9147.4201.2227.4147.8238.2168.3

Success rates of the professional and public groups showed no significant difference by a nonparametric Mann–Whitney U-test (p = 0.951), while the task completion time differed with statistical significance. The professional group on average was faster than their general public counterparts (p = 0.036), with average times being 148.52 and 181.06 seconds, respectively. When looking at simple tasks (tasks 1–5) and complex tasks (tasks 6–9) separately, the difference remained for simple tasks (average time: 118.00 vs. 157.36 seconds, p = 0.07), but not for complex tasks (average time: 186.67 vs. 210.68 seconds, p = 0.197). This showed that the digital library supported professionals and public users equally effective in accomplishing information-seeking tasks, but professional users were more efficient in using the digital library for simple tasks than general users. For complex tasks, both groups of users were not very efficient.

Participants were asked to rate the website based on the aforementioned criteria, which consisted of two categories, users’ satisfaction and their interactivity with the digital library. The results are presented in Table 3. Users were mainly satisfied with the visual qualities and content presented in the digital library, but were less satisfied with the aspects of organization of information, error corrections, and ease of use.

Table 3

Users’ ratings in posttest questionnaire (in 5-point Likert scale).

CategoryCriterionCurrent studyHu et al. (2017a)
SatisfactionEase of use3.2533.23
Organization of information3.1833.43
Visual appearance4.1843.64
Error correction3.2533.94
InteractivityActive control3.3733.84
Two-way communication3.4341.92

The comparison of quantitative results from the study by Hu et al. (2017a) is also provided in Table 3, where the same evaluation framework was applied to evaluate the panorama collection of Mogao Cave. Results of Mann–Whitney U-test showed that visual appearance (p = 0.047), content (p = 0.006), and two-way communication (p < 0.001) showed significant improvement from user evaluation on the panorama collection. In particular, compared to the panorama collection, the textual description of each cave, more static images, and metadata (Fig. 3) are added in e-Dunhuang, where panoramic views of the caves are kept as one prominent feature. This greatly enhances the usability of e-Dunhuang, as static images impose much less cognitive load to users than panorama and do not require navigation with delicate mouse control (Hu et al., 2017a). Text information helps communicate descriptive information to users, such as cultural background and specifications of the caves and murals, which in turn helps users to understand the resources and feel they are in control.

4.3 Qualitative analysis of the usability test

Transcripts of the think-aloud sessions and follow-up interviews were analyzed to provide more qualitative evidence and reasons regarding users’ views on e-Dunhuang. The results according to different evaluation criteria are presented in the following sections.

4.3.1 Ease of use

Ease of use was one of the lowest rated criteria in this study. Concerns mainly focused on poor search functions. All participants, without exception, encountered difficulty using the search engine and complained about its effectiveness. The issues included the following:

  • Unchanged search results of various search terms: “Why is Cave 003 always listed as first search result no matter what I entered as keywords?” (participant 7)

  • Unclear rationale of hits: “I entered the keyword, and opened the first caves listed in the result page, turns out my keyword is not even included in the entire page!” (participant 5)

  • Inconsistency of style: “Why is the keyword sometimes highlighted in red, and sometimes not highlighted?” (participant 3)

Another related issue laid in the difficulty of discovering the advance search function:

“I need to go back to home page first, scroll down, click on ‘checking on more caves’ before I could access the advance search box.” (participant 4)

“If I had known they have all those search criteria designed in advance search, I wouldn’t have had to waste so much time using the simple search box and getting nowhere.” (participant 6)

Some participants also complained about the steep learning curve involved in using the panoramic functions, particularly due to the lack of instruction or guide. They needed to go through a period of trial and error, sometimes without success and sometimes missing useful functions:

“I can only understand the arrow keys and zoom in and out keys, what are other keys used for? Why is there no instruction?” (participant 6).

In addition, on the panoramic views, other participants were a little annoyed with the awkward control of the mouse:

“I don’t know what’s happening, why it spins so fast, I just use my mouse to drag left and right because I want to check out the paintings around, why it is so hard to control!” (participant 4).

Other participants also complained about the automatic spinning feature of the panoramic function:

“I feel quite dizzy that it spins all the time!” (participant 1)

“Why can’t I stop if from spinning?” (participant 8)

4.3.2 Organization of information

Another criterion with lower scores was “organization of information” where participants indicated that the layout of information was quite different from other conventional digital libraries with comprehensive text-based navigation menus (e.g., online catalogs of public libraries). Although visually appealing, it could be difficult to understand how to navigate the system. Some suggested to place the filter condition (Fig. 2) to the home page.

“It took me a while to figure out I can press in ‘More Caves’ to view the categories. The filter of conditions can be put on the main page.” (participant 5)

“As a new user, it’s very hard for me to find the categories, I feel it [filter condition] should be placed at an obvious position, better on the main page.” (participant 13)

Some participants also suggested more categorization criteria for organizing the caves, such as architecture features (participant 14), background stories (participant 16), and artistic styles (participant 7). In particular, the existing categories in “Cave Shape” were criticized as “too technical, less useful for general users than for experts” (participant 7). More intuitive ways to organize the resources can help cater users who have varying familiarity levels with the presented heritage.

4.3.3 Labeling

Several participants raised issues on the labeling system in e-Dunhuang, pointing out that the “cave,” and “wall painting” labels in the upper right corner of the pages (Fig. 13) might lead to potential problems, because the labels do not lead users to new pages as users would often expect:

“When I clicked on the ‘wall painting’ tag, it just led me to the lower portion of the main page, they are not actually in separate pages, which might cause problems because if I want to share a wall painting with my friend, I can’t, they could only open the link and see the main page.” (participant 4)

Some participants pointed out that the language used on some labels was somewhat hard to understand or awkward:

“These labels are all very factual, it won’t leave a mark on me after I finish browsing a certain cave, I wouldn’t remember what’s so different or unique about this cave.” (participant 11)

4.3.4 Visual appearance

Most participants commented that e-Dunhuang did a great job in creating a stunning visual experience. Praise mainly focused on the full-screen pictures on the main page and the high-resolution views provided by the images and panoramic functions. The Digital Dunhuang project has devoted considerable financial resources to create high-quality digital visual images using the latest digital technology (Zhou, 2011; Wang et al., 2015). It was thus not surprising that this criterion gained the highest score among all evaluation criteria. “Super cool” (participant 12), “high resolution, clear” (participant 14), and “striking” (participant 16) were just a few example comments from the participants. The design of the interface was also praised for its principle of simplicity and minimalism (Neilson, 1995).

“I think it is relatively simple, with clear quality of images.” (participant 15)

“The appearance is clear, clean, and concise.” (participant 12)

4.3.5 Error correction

The criterion of error correction was not well received by most participants who mainly referred to difficulties in using the panoramic functions:

“Sometimes I accidently clicked out of the cave and didn’t know how to find my way back inside.” (participant 1)

“Panoramic function should have a button to navigate back to the initial place.” (participant 13)

Some participants also pointed out that the advanced search function is not error proof:

“I don’t know how to un-choose one criterion after selecting it wrongly or when I want to “reset” my search criteria. My instinct is that there will be a “delete” key beside the selected criterion, turns out I need to click again on the original criterion. It took me a while trying to figure that out.” (participant 6)

4.3.6 Content

Generally speaking, participants mainly voiced the view that the content of E-Dunhuang is informative, but not attractive or interesting enough with the current plain text description. More introductory information on the background is needed.

“In general, the descriptions of caves are concise but emphasis should be placed on introducing historical incidents and stories.” (participant 15)

“I don’t think the information is attractive to me, they are just stating the simple fact of how many statues or how many artifacts are unearthed, I want to read more about why and what it all means.” (participant 6)

Some participants found that the content is not very comprehensible to the general public, who possess less prior knowledge on the historic, cultural, and artistic background of Dunhuang or even the Chinese characters.

“I have a hard time understanding what they are saying because they are written in somewhat old Chinese (before the establishment of modern PRC).” (participant 5)

“It feels like the overall description is quite good, enough for most people. The text description is boring without talking about the artistic characteristics and the stories behind. It won’t be memorable after reading it. It should be more detailed or designing mini stories and should show some comments below that are applicable to modern arts to inspire the public.” (participant 14)

Despite these issues, the criterion of “content” was rated significantly higher than that for the panorama collection evaluated in the study by Hu et al. (2017a). This can be attributed to the fact that the panorama collection did not provide any textual description. The descriptions in e-Dunhuang, although not perfect, are already a significant improvement and are highly welcomed by participants.

4.3.7 Interactivity

Overall, participants showed positive opinions toward “interactivity” of the digital library (with an average score of 3.60 of all the three criteria: active control, two-way communication and synchronicity). Among them, “twoway communication” was praised quite often, for the various indications of system status

“When the cursor becomes finger shape, I know the webpage that I am going to visit is selected.” (participant 13).

and the immersion experience of panorama function

“The panorama function is quite interactive and I kind of feel I am in it [the cave]” (participant 16).

Some participants suggested that the panorama function would be even better if there were full-screen and virtual reality features. In fact, those features were already built-in, but the buttons were not straightforward enough for participants to find them.

Considering the high resolution of the images, participants were quite satisfied by the speed of the digital library (and the network). The social sharing function (top of Fig. 3) included most commonly used social media platforms, which was praised by many participants. Yet, there were problems reported regarding the element of “active control.” Similar to “ease of use,” poor search functions and hard-to-grasp panoramic operations (despite very positive feelings of its visual affordance) induced frustration among participants.

5 Suggestions

Based on the evaluation results, the following suggestions for further improvement in the digital library were proposed as follows:

5.1 Search functions

The search function was the feature most often complained in the entire digital library (13 out of 16 participants). Owing to the relatively simple, minimalist style design of the interface, search was one of the few finding aids provided to users, making it crucial to user experience. The following suggestions hopefully can help improve the search facility in e-Dunhuang.

  • Make the link to “advance search” more obvious, to avoid users feeling at lost with only a simple search box on the home page

  • Improve the search algorithm and result display. The norm of end user facing search functions is that the search results should include the keywords searched and keywords should be highlighted in search result pages.

  • Adding more filters in advance search, such as the subject and specialty of caves/murals (e.g., the thousand Buddha motif). To implement this, a metadata schema must be developed to include these needed elements (Hu, Ng, Xia & Fu, 2017b). Furthermore, information needs of end users must be systematically collected from which new information access facilities such as labels and filters can be designed.

5.2 The panoramic functions

Panoramic functions of caves were mentioned by most participants as their favorite feature of the digital library (12 out of 16 participants). Despite the fact that users enjoyed the stunning visuals and high resolution for detailed examination, there were also a number of suggestions for creating a better user experience.

  • Add instructions to the buttons on the panorama tool bar. Many participants pointed out that they had no idea how to use the buttons (Fig. 4). Except for the familiar arrow and zoom keys, very few participants discovered the use of other buttons.

    Figure 4

    Download Figure

    Figure 4

    Tool bar of the panoramic function.

    Citation: Data and Information Management 2, 2; 10.2478/dim-2018-0008

  • Provide an option to control the automatic spinning of the view: While some participants felt it annoying, others said they enjoyed the spinning. Moreover, although all participants in this study did not have accessibility issues, the digital library needs to take accessibility into consideration. Users with visual/vertigo may feel uncomfortable with spinning views. Therefore, a control key would be helpful in facilitating users to make their own choices and to control the speed of the spinning.

  • Add more text descriptions to the visuals: Considering the targeted users of the digital library include the general public who may not have in-depth knowledge of Dunhuang heritage and want to learn more in addition to enjoying the pictures, text descriptions of most famous wall paintings or statues would be highly appreciated. To avoid text blocking image views, approaches of adding links, or incorporate hovering text can be applied.

  • Incorporate other digital technologies: Several participants suggested that they got bored after watching the spinning panorama for a while. Considering the common practices of cultural heritage museums, we suggested incorporating audio-guide function to create a more attractive appeal and feed more information together with pictures to users.

5.3 Organization of information and content

The layout of cave or mural pages could be improved to facilitate easier browsing and reading. Suggestions included adding more categories in filter conditions, listing out information in table-style format as well as adding more links to include more explanation of certain concepts, ideas, and background knowledge.

The use of traditional Chinese characters and ancientstyle wording certainly fits the context of Dunhuang, but considering the differing education backgrounds of users, we suggested the wordings to be adjusted to limit the use of jargon and make them more understandable to the general public.

5.4 Interactivity

Aside from the great sense of interactivity provided by the panoramic functions, users wished to see more engaging activities, and they wanted to have their voices and opinions heard. Suggestions included adding audio-guide functions (as mentioned in Section 5.2), an online forum for users to share their views, and a comparison function where users could easily examine and compare two or more different caves or murals.

5.5 Other suggestions

One participant tried to search for the digital library using Baidu (a popular search engine in mainland China), but was not able to find it in the search results, indicating the digital library may need better advertising and promotion. One participant was born and grew up in Hong Kong where the official written language is traditional Chinese (as opposed to mainland Chinese, which is simplified). He thus found it challenging reading simplified Chinese on the digital library pages as there was no translation function. In addition, the search function was unable to recognize traditional Chinese characters, which posed a problem for many users who did not have a mainland China education background. Therefore, a multilingual version of e-Dunhuang is suggested as a direction for future development.

6 Conclusions

This study evaluated the user experience of e-Dunhuang, the first fully fledged digital library for cultural heritage in Dunhuang. Usability tests on nine predesigned information-seeking tasks were conducted with 16 users from the professional and general public groups. In comparison to the usability test results of Mogao Cave panorama collection, e-Dunhuang improved significantly on three criteria: visual appearance, content and two-way communication. Qualitative analysis revealed in-depth reasons for why participants felt satisfied with some criteria but had concerns with other criteria. Overall, the results showed that e-Dunhuang is an attractive digital library, effectively showcasing cultural heritages of a rich period in Chinese and world history. It enabled users to experience in close proximity the charm and beauty of the cultural heritage of Dunhuang. Suggestions are proposed to address the usability issues identified in this study, aiming to further improve and promote educational, academic, and artistic experiences to more users.

This study is exploratory in nature. With a limited sample size, the research does not expect to be perceived as a generalized view of a large population toward e-Dunhuang. Nevertheless, this study provides insights into further improving the e-Dunhuang and contributes to user-centered evaluation of cultural heritage digital libraries.


Appendix: Usability Testing Tasks (modified from Hu et al [2017])

Simple fact tasks (to locate the given items or to identify specific information):

  • Task 1: Does the collection include any caves that are central pillared (which means there is pillar in the middle of the cave)?

  • Task 2: Please locate one item that is constructed in the northern Wei, Tang, and Yuan Dynasty.

  • Task 3: In the collection you are working with, how many caves can you find displaying in the e-Dunhuang DL?

  • Comparison of fact tasks (to conduct comparisons of simple facts or information):

  • Task 4: Please try to compare the central statues in Cave 285 and the first statues on the right in Cave 249 and see if the right hand of the first statues (the central one in Cave 285) is damaged.

  • Task 5: List two which are decorated with the Thousand Buddha motif judgment tasks (to first locate items and formulate opinions).

  • Task 6: You would like to know more about the artistic style (e.g., color and shape) of those Flying Apsaras (with an example picture shown) from the Mogao Cave arts. Name one cave which can provide you with such information?

  • Task 7: Other than Buddha stories and Chinese mythology, you would like to know more about how people lived in ancient China (e.g., their economic, military, political statues, and activities) from the wall paintings and statues. Which item(s) will you go for? Name one cave which can provide you with such information?

  • Task 8: Which of the following theme(s) best describe the paintings on the roof of Cave 249? A. War; B. Chinese mythology; C. Hunting; D. Music performing. Why do you think so?

  • Comparison of judgment task (to locate items, compare, and make judgments):

  • Task 9: Please identify and describe in general if there is/are any difference(s) between the artistic style of the cave arts found in that in the Early Tang and Yuan Dynasty?





A successful completion of task is defined as being able to find reasonable answers to the given task. Sometimes users may voluntarily give up the task, which could be counted as a failure. Alternatively, users may find requested information or answers that are obviously incorrect, particularly for factual tasks. For tasks involving subjective judgment, all answers were counted as correct.

Artese, M. T., Ciocca, G., & Gagliardi, I. (2017). Evaluating perceptual visual attributes in social and cultural heritage websites. Journal of Cultural Heritage, 26, 91–100.

Barbieri, L., Bruno, F., & Muzzupappa, M. (2017). Virtual museum system evaluation through user studies. Journal of Cultural Heritage, 26, 101–108.

Brahaj, A., Razum, M., & Hoxha, J. (2013). Defining digital library. In International Conference on Theory and Practice of Digital Libraries (pp. 23-28). Berlin, Heidelberg: Springer.

Cáliz, D., Martínez, L., Alamán, X., Terán, C., & Cáliz, R. (2016). Usability testing in mobile applications involving people with Down Syndrome: A literature review. Computer Science and Information Technology, 6(7), 1–11.

Comeaux, D. J. (2008). Usability studies and user-centered design in digital libraries. Journal of Web Librarianship, 2(2/3), 457–475.

Dalkiran, Ö., Aker, i., Öztemiz, S., Taşkin, Z., & Tunç, S. K. (2014). Usability Testing of Digital Libraries: The Experience of EPrints. Procedia: Social and Behavioral Sciences, 147, 535–543.

Davis, D., & Jiang, S. (2016). Usability testing of existing type 2 diabetes mellitus websites. International Journal of Medical Informatics, 92, 62–72.

Dolan, T. E. (2014). A six-dimensional strategic development tool for e-government effectiveness. In L. Anthopoulos & C. Reddick (Eds.), Government e-Strategic Planning and Management (pp. 105–124). New York: Springer.

Evens, T., & Hauttekette, L. (2011). Challenges of digital preservation for cultural heritage institutions. Journal of Librarianship and Information Science, 43(3), 157–165.

Faulkner, L. (2003). Beyond the five-user assumption: Benefits of increased sample sizes in usability testing. Behavior Research Methods, Instruments, & Computers, 35(3), 379–383.

Granić, A., Mitrović, I., & Marangunić, N. (2011). Exploring the usability of web portals: A Croatian case study. International Journal of Information Management, 11(4), 339–349.

Heradio, R., Fernández-Amorós, D., Cabrerizo, F. J., & Herrera-Viedma, E. (2012). A review of quality evaluation of digital libraries based on users’ perceptions. Journal of Information Science, 38(3), 269–283.

Hu, X., Ho, E. M. Y., & Qiao, C. (2017a). Digitizing Dunhuang cultural heritage: A user evaluation of Mogao Cave Panorama Digital Library. Journal of Data and Information Science, 2(3), 49–67.

Hu, X., Ng, J., Xia, S., & Fu, Y. K. Y. (2017b). Evaluating metadata schema for murals and stone cave temples: Towards digitizing cultural heritage. Proceedings of the 80th Annual Meeting of the Association for Information Science and Technology, pp.491-494.

ISO (2009). Ergonomics of human system interaction - Part 210: Human-centered design for interactive systems. International Organization for Standardization. ISO FDIS 9241-210:2009.

Jeng, J. (2009). Usability evaluation of digital library. In Y.-L. Theng, S. Foo, D. Goh, & J.-C. Na (Eds.), Digital Libraries: Design, Development, and Impact (pp. 278–286). Hershey, PA, USA & London: Information Science Reference.

Kabassi, K. (2016). Evaluating websites of museums: State of the art. Journal of Cultural Heritage, 24, 184–196. https://doi. org/10.1016/j.culher.2016.10.016

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and TrendsÒ in Information Retrieval, 3(1–2), 1-224.

Kelly, E. J. (2014). Assessment of digitized library and archives materials: A literature review. Journal of Web Librarianship, 8(4), 384–403.

Li, P. (2013). A cognitive task analysis approach for usability evaluation of digital libraries. In C. Cool (Ed.), Recent Developments in the Design, Construction, and Evaluation of Digital Libraries: Case Studies (pp. 164–177). Hershey, PA, USA: Information Science Reference.

Liu, Y. (2003). Developing a scale to measure the interactivity of websites. Journal of Advertising Research, 43(2), 207–216.

Matusiak, K. (2012). Perceptions of usability and usefulness of digital libraries. International Journal of Humanities and Arts Computing, 6(1-2), 133–147.

Neilsen, J. (1995). Ten Usability Heuristics for User Interface Design. Retrieved on September 10, 2017, from

Picazo, P. G., Saz, J. T., & Villalta, M. C. (2009). Digital libraries beyond cultural heritage information. In Y.-L. Theng, F. Schubert, D. Goh, & J.-C. Na (Eds.), Handbook of Research on Digital Libraries: Design, Development, and Impact (pp. 400–411). New York: IGI Global.

Piccialli, F., & Chianese, A. (2017). Cultural heritage and new technologies: Trends and challenges. Personal and Ubiquitous Computing, 21(2), 187–189.

Terras, M. M. (2011). The Rise of Digitization. In R. Rikowki (Ed.), Digitisation Perspectives (pp. 3–20). Rotterdam: Sense Publishers.

Todd-Diaz, A., & O’Hare, S. (2014). If you build it, will they come? A review of digital collection user studies. In S. K. Hastings (Ed.), Annual Review of Cultural Heritage Informatics: 20122013 (pp. 257–276). New York: Rowman & Littlefield.

Wang, X., Zhou, P., Wang, E., & Downie, J.S. (2015). Digital Dunhuang: Enhancing virtual explorations of the real Dunhuang Panel. Digital Humanities: Annual Conference of the Alliance of Digital Humanities Organizations. Retrieved on September 10, 2017, from Enhancing_Vir/DOWNIE_J__Stephen_Digital_Dunhuang__Enhancing_Virtual_E.html

Zha, X., Wang, W., Yan, Y., Zhang, J., & Zha, D. (2015). Understanding information seeking in digital libraries: Antecedents and consequences. Aslib Journal of Information Management, 67(6), 715–734.

Zhang, Y. (2010). Developing a holistic model for digital library evaluation. Journal of the American Society for Information Science and Technology, 61(1), 88–110.

Zhou, P. (2011). The development of a repository system of Dunhuang murals and digital resources for Dunhuang studies. In Proceedings of the International Conference on Cultural Heritages and Digitization (pp. 317-323). Dunhuang: Dunhuang Academy.

Journal Information



All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 689 689 29
PDF Downloads 164 164 16