Eye-tracking test design influences the relationship between gazing behaviour and evaluation decision

In recent research using eye-tracking technique, the relationship between consumer gazing behaviour and decision-making has been revealed. Several studies show that the chosen alternative is looked at longer and more often than other alternatives. Pieters and Warlop (1999) found that when making decisions, we often spend longer examining options that we ultimately choose rather than those we do not. Armel et al. (2008) reported that the relative amount of time that subjects are fixated on an item during a decision-making process increases the probability of choosing. In another study by Graham and Jeffery (2012), 203 participants viewed 64 food items on a computer screen equipped with an eye-tracking camera. Participants were made to simulate food purchasing decisions. They spent longer looking at labels for products that they ultimately decided to purchase compared with those they decided not to purchase. Jantathai et al. (2013) showed that fixation count and visit duration had a significantly positive correlation with choice rate. The latest study of van der Laan et al. (2015) found that the total fixation duration was determined by the decision goal and pre-existing preference. Participants selected either the product they wanted or the product they did not want. In both conditions, participants fixated longer on the product they chose, with the gaze bias being larger in the most wanted decision type. In brief, it has been clearly demonstrated that several gazing parameters are correlated with decisionmaking in eye-tracking tests.

Scientific literature has tried to explain the above findings by the relationship between visual attention, decision-making and cognitive processes. They found that eye movements are coextensive with cognition, and oculomotor processing is coextensive with cognitive processing (Liversedge et al., 2011). Hence, the cognitive process during decision-making might also be reflected by visual attention. As results, it is understandable, that correlations between consumer gazing behaviour and their final decision have been found. However, the question whether these correlations always exist or not, has not been systematically answered yet. If the answer is affirmative, it will open a great opportunity to predict consumer decisions by measuring their gazing behaviour. If the answer is negative, it will be necessary to know under which experimental conditions the correlations can be observed.

In eye-tracking tests, depending on the research purpose, each author designs his or her tasks in a different way. Design factors can be divided into two main groups corresponding to the two control mechanisms in decision-making: stimuli design factors and goal-driven attention factors. Bottom-up factors refer to attention captured by diverse visual stimuli design factors (e.g., number of images, complexity of images, colour, shape, and information level of images). Top-down factors refer to attention captured by task design factors (e.g., preference formation, decision goal, task instruction) (Orquin and Loose, 2013). On the one hand, the influencing mechanism of design factors is very similar to the decision-making process. On the other hand, design factors have been demonstrated to affect consumer gazing behaviour (Vu et al., 2016). Therefore, the questions arise, whether design factors are able to influence the correlation between consumer gazing behaviour and decision-making or not. And if yes, it is also important to answer the questions, which factors are important and how they do take effect? Following these considerations, our research questions have been worded in the following way:

Q1

By varying several design factors in an eye-tracking test, is there always a correlation between consumer gazing behaviour and decision-making?

One hundred and twenty-two persons participated in the study. They included students, staff and visitors of the University of Natural Resources and Life Sciences in Vienna (BOKU). However, data of only 100 participants (48 males and 52 females, aged from 18 to 53 years) were used, because they passed a calibration step and their eye-tracker recording quality was over 75%. The study was performed in agreement with the ethical guidelines for scientific research of BOKU. The participants were informed about the testing procedure and were also asked to give written informed consent before the test.

In this study, two groups of test design factors were investigated:

Type of evaluation: four types of evaluation were included (maximum choice, minimum choice, ranking, and rating). They reflected tasks with different levels of difficulty that are commonly used in consumer tests.

Fixation structure within an area of interest (AOI) is considered to be a sensitive measure of cognitive processing load (Russo and Dosher, 1983), so total fixation duration and fixation counts may reflect the way participants perceive information from an image. Moreover, visit duration, which is the time that eyes visit an AOI, may reflect, how long participants need to enter and observe each AOI before having decided. Thus, the three measures below are important for our study.

Total fixation duration: Sum of durations of all individual fixations within an area of interest (AOI) [second]

Food images were bought from a professional picture provider (Table 1). The selected images were similar in terms of visual attractiveness, meaning that no image attracted more attention than others to avoid unwanted influence of attractiveness on consumer gazing behaviour. This similarity was verified ex post by an analysis of variances on effect of images.

Table 1

Food images used in the study

Tabelle 1. Bilder der in der Studie verwendeten Speisen

By varying one factor and fixing the other in each experiment, we individually tested the effect of each test design factor. Participants were divided into four groups coded as Gr1, Gr2, Gr3, and Gr4, which were balanced in gender and mean age. Participants were asked to participate in two experiments, which took place during two days in succession (Table 2). After the 1^st experiment, ‘maximum choice’ was fixed for the next experiment.

Table 2

Experimental design

Tabelle 2. Versuchsdesign

During all experiments, participants first received instructions including the highlighted question on the screen. The experiment began with a red cross in the middle of the screen for three seconds to make the participants fix their gaze at a predefined point before looking at the test picture. The test picture was on the screen until the participant made his/her decision according to our question. When participants had made their decisions, they had to click the mouse button (the mouse cursor was invisible during decision making) to move on to the response page. There was no limit in decision time. The response pages depended on the type of evaluation and they were just for participants to give their answers. The mouse cursor was visible at the response page, but eye movements have not been recorded during that test phase (Figure 1).

Figure 1

A pre-test was conducted with ten consumers before the main experiment, to make sure that the procedure and its instruction were clear and easy to understand for untrained participants.

The Tobii-T60 eye tracker and Tobii studio software (version 3.0.5, Tobii Technology AB, Sweden) were used to gather and analyse data on the gazing behaviour of consumers. Test pictures were presented on a 17-inch-thin-film transistor LCD monitor with 1280 × 1024-pixel resolution. Participants were asked to sit at a distance approximately 65 cm from the Tobii T60 device sensor.

Areas of interest (AOI) were defined as single food images on the test picture. For each AOI, measures of consumers’ gazing behaviour were calculated. For each of the measured parameters, a two-way ANOVA was performed using the following model:

Gazingbehaviour=mean+maineffectforAOI+maineffectforGroup+error$$\begin{array}{} Gazing\, behaviour = mean + main\, effect \,for\, AOI + main \\ \qquad \qquad \qquad \quad effect \,for\, Group + error \end{array} $$

Effect of AOI would show if there were images more attractive than others or not.

In the maximum choice and the minimum choice test, consumer decision-making is presented as percentage of choosing. Chi-square test was conducted to examine the relationship between gazing parameters and choice rate. If there was a significant relationship, the high gazing group should possess higher percentage of choosing than other gazing groups, and vice versa.

In the ranking test, consumer decision-making is presented as ranking scores. A one-way ANOVA was performed as below to examine whether there are any significant differences in ranking scores between low, medium, and high gazing behaviour group.

Rankingscores=mean+maineffectforGazinggroup+errors$$\begin{array}{} Ranking\, scores = mean + main\, effect\, for\, Gazing\, group +\\ \qquad \qquad \qquad \qquad \qquad \quad \,\,\, errors \end{array} $$

In the rating task, consumer decision-making was presented as rated points. Again, a one-way ANOVA was performed to examine whether there are any differences in rating points between low, medium, and high gazing behaviour group or not.

Rating points = mean + main effect for Gazing group + errors

All statistical analyses were performed using Excel version 2010.

There were no significant differences between AOIs defined as food images on the testing picture, which shows that no food image was more attractive than others.

The p-value of chi-squared test indicated that the relationship between fixation duration and percentage of choosing was significant. Similarly, a significant relationship between fixation count/visit duration and choice rate was also found (Table 4 and Figure 2). The results indicate that if consumers looked at an item more often and for longer duration, it is more likely that the item was chosen.

Figure 2a, b, c, d

However, these significant relationships were not observed in the ranking and rating task. P-values of the one-way ANOVA for gazing behaviour and consumer’s ranking scores or consumer’ rating points were all clearly higher than 0.05. The ranking scores or rating points of participants having high values of fixation duration for the tested image were not significantly different from the scores and rating points of participants with low or medium fixation duration. Similar results were found with fixation counts and visit duration parameters (Table 4 and Figure 2).

Table 3

F- and p-values from the ANOVA performed for the effect of AOIs (food images)

Tabelle 3. F- und p-Werte der ANOVA bzgl. Wirkung von AOIs (Bilder der Speisen)

Table 4

p-values of the one-way ANOVA between gazing behaviour and consumer’s ranking scores or consumer’ rating points

Tabelle 4. p-Werte der Einfaktoriellen ANOVA bzgl. Blickverhalten und Ranking bzw. Bewertung durch die KonsumentInnen

This result showed that the relationship between consumer decision-making and gazing behaviour was influenced by Type of evaluation. Only in the maximum choice and minimum choice task, all the tested eye-tracking parameters (total fixation duration, fixation count, and visit duration) were found to correlate with consumer choice rate. Hence, we chose the maximum choice task for the next experiment.

Highly significant correlations between consumer choice and the three gazing behaviour parameters were found in groups of Deliciousness, Healthiness, and Familiarity. Food images with high fixation duration were chosen more often than images with low fixation duration. In the group with the question about Deliciousness, the choice rate of high fixation duration group was about four times higher than that of the medium fixation duration group. It was clear that consumers fixated on chosen images significantly longer and more often than the other images (Table 5 and Figure 3). However, in the group of Prices, this relationship between gazing and choosing could not be found. There were no significant differences in choice rate related to fixation duration, or fixation count or visit duration. The chi-squared (χ²) test of independence between gazing behaviour and consumer’s choices was not found to be significant at p < 0.05 (Table 5 and Figure 3).

Figure 3

Table 5

p-values of the chi-squared (χ2) test of independence between gazing behaviour and consumer’s choices

Tabelle 5. p-Werte des Chi-Quadrat-Test (χ2) auf Unabhangigkeit bzgl. Blickverhalten und Auswahl durch die KonsumentInnen

Hence, our study indicated that the relationship between choice and gazing behaviour is influenced by the factor ‘question content’. When the question content was varied, consumers fixated on chosen images significantly longer and more often than other images in the cases of question contents deliciousness, healthiness and familiarity, but when the question referred to price, there was no significant relationship found.

The presented data shows that the relationship between consumer decision-making and gazing behaviour, which might be the base for predicting choice based on gazing behaviour, is strongly influenced by the type of evaluation. A significant correlation was only found in the maximum and minimum choice tests (called choice test), but not in the ranking and rating test forms.

In this study, decisions including consumer choice, ranking orders, and rating scores depend mainly on preference formation and the healthiness of alternatives, because it was the decision that was the goal of all types of evaluation. Moreover, other factors that could influence consumer decision-making, such as attractiveness of alternatives, were limited by alternative selecting.

In the choice test, the decision-making process maximizes the coherence between alternatives and the choice problem. The choice is made when the decision maker reaches a threshold of sufficient internal coherence (Orquin and Loose, 2013). Hence, participants firstly scan alternatives and then the favoured alternative or most important attribute receives attention. The most informative and/or favourite alternative is the most attractive, while the second most informative alternative attracts less attention, and the least informative is the least attractive. In this case, the decision-making process and visual attention of consumers have the same tendency towards information with a higher coherence of decision goal and preference formation, therefore increasing the likelihood of decision makers fixating on the chosen alternative towards the end of the decision-making process.

In the ranking and rating test, participants were also influenced by decision goal and preference formation. This suggests that the more informative an image, the more the participants were attracted to it. However, ranking and rating tests were considered to be more difficult than the choice test (Vu et al., 2016) and the difficulty of a task is a consequence of the higher demand on working memory (Orquin and Loose, 2013). Hence, besides the time that participants were attracted by the information of the image itself, in ranking and rating tests, they need more time to find the appropriate decision option, such as comparing among alternatives based on relevant attributes, seeking experiences and preference in working memory. As a result, fixation duration and counts were not correlated with ranking orders or rating scores. Moreover, in previous studies, eye movements were found to be highly task dependent and linked to cognitive processing (Castelhano et al., 2009; Liversedge et al., 2011). The relationship between attention and working memory cannot be simply described as a linear function (Orquin and Loose, 2013). It is therefore understandable that eye movements cannot be correlated with ranking orders and rating scores, which depend mostly on the nature of alternatives.

Our results found a strong positive correlation between gazing behaviour and choice rates for three groups of question contents: Deliciousness, Healthiness, and Familiarity. Products that were high in gazing measures had a higher likelihood to get chosen and vice versa. The gazing behaviour parameters could be used to predict the choice rate to some degree. This result is in line with several previous studies (Krajbichet al., 2010; Atalay et al., 2012). These findings were explained by the effects of decision goals and preference formation on the gaze bias for the chosen alternative in the choice test. Firstly, the decision goal mechanism is a process of top-down attention that results from a specific task instruction (Yarbus, 1967). This means that the most informative alternative is also the most attractive one. For example, when the goal is to evaluate healthiness, health logos are more attractive (Orquin et al., 2012). In our study, different question contents played different decision goals: deliciousness, healthiness, price, and familiarity. The most informative alternative corresponding to the decision goal attracted the most attention. Therefore, we found a positive correlation in three groups of question content. Secondly, the influence of preference formation is an establishing process of preference for one of the alternatives in the test picture. The favoured alternative receives most attention (Orquin and Loose, 2013). In several cases, both decision goal and preference formation affect gazing parameters in the same direction. Hence, we observed a strong correlation between gazing and choosing in the three groups deliciousness, healthiness, and familiarity.

However, in the Prices group, no relationship between decision-making and gazing behaviour was found. This result may be explained by the fact that the effects of decision goal and preference formation might have counteracted each other. If participants were asked to choose the cheapest food (content of question or decision goal as well), they would pay attention to the cheapest product. However, the cheapest food was usually not their favourite one, so they paid attention to the chosen image because of the decision goal effect. But they did not pay too much attention because there was no effect of the preference formation. Conversely, images not chosen may attract attention because of their preference formation effect. As a result, there was no significant difference in choice rate among the high, medium, and low gazing behaviour groups.

In brief, if the tested question content is positively related with consumer preference formation, a strong correlation between gazing behaviour and decision-making will be found surely. However, if the tested question content is in opposition to consumer preference formation, this relationship will depend on how strong the decision goal effect is and how strong the preference formation effect is. The relation between decision goal effect and preference formation effect will finally decide, whether consumer gazing behaviour correlates with decision-making or not. If the decision goal effect is stronger than the effect of preference formation, a correlated relationship will be found.

To investigate each test design factor, we developed different tasks using different groups of participants. If participants carried out the first task by observing the testing picture, they should not see this picture again to do the other tasks. Knowing the test picture before the main test would strongly influence consumers’ gazing behaviour. Hence, we used four different groups of participants for four different tasks in each experiment. We accepted the bias between participants as error of the method.

A limited number of selected participants is another limitation of this work. We used only individuals from the staff and students from BOKU University, and this sample is not representative of any population. As our sample lacks representativeness, we should not overgeneralize and cannot make general statements about the relationship between decision making and gazing behaviour. However, the obtained results give valuable evidences.

Further work should be conducted to study the joint effect between test design factors. Moreover, other design factors such as content of images, information about test aims, experimental situation (real life vs. laboratory), and/or a story about the tested products should also be investigated. Further works with higher number of consumer representatives should be performed and welcomed.