Identification and Analysis of Multi-tasking Product Information Search Sessions with Query Logs

Previous research has identified two types of approaches for task identification: time splitting and query clustering (Lucchese et al., 2013). Query clustering is based on the content of the queries while time splitting uses contextual cues. Content-based methods to identify search tasks in Web search include comparisons of (1) similarities of two search queries, (2) URLs that the Web search engine returns (Glance, 2000), and (3) documents that the Web search engine returns (Raghavan & Sever, 1995). Similarity scores are calculated based on these three indexes to decide whether two queries belong to the same search task.

The two major methods used herein for comparing the relevance of these two search queries are (1) identifying word similarities in the queries and extracting the sets of the search terms from these two queries. Some useful indexes for this task include the Jaccard distance (Järvelin, Järvelin, & Järvelin, 2007), which calculates the ratio of the intersection and the union of the two search-term set and the Levenstein distance (Jones & Klinkner, 2008), and (2) comparison of the semantic relevance of the search terms by using the idea of vector space (Salton & Mcgill, 1986). For example, utilizing the semantic relation from Wiktionary and Wikipedia, Lucchese et al. (2011) calculated similarities between each search term and each source in the semantic network, and created a search term vector composed of the similarities between a search term and each source in the semantic network.

Usually the angle (cosine similarity) between two search query vectors is calculated as the index of the similarity between these two search queries. Lucchese et al. (2011) first processed the search log, including the removal of empty log records and stop words, as well as stemming and deleting sessions that last too long or include too many queries, which indicates it is likely produced by machines. Then they calculated the word and semantic similarities between queries using two methods to calculate the final similarity index. The first method is a weighted average of the word similarity and the semantic similarity, whereas the second method is to use a threshold. When the word similarity score is above the threshold, the final similarity index score equals the word similarity; when the word similarity score is lower than the threshold, the final similarity index is the greater value of the word similarity and the semantic similarity.

Information users often demonstrate multi-tasking behaviors in Web search. Spink et al. (2006) suggested that users generally produce multi-tasking sessions for two reasons. The first reason is that a user may have several search topics at the beginning of the search process, and the second reason is that although users may have only one search topic in the beginning of the search process, they may discover new search topics in relation to information needs while searching.

Numerous studies have examined the characteristics of multi-tasking search sessions, including the time involved in queries. For example, Spink, Ozmutlu, and Ozmutlu (2002) found that the length of search queries and the time costs in multitasking sessions are longer than those in mono-tasking sessions. Lin and Belkin (2005) also confirmed that the average number of search queries used in multitasking sessions is more than that in mono-tasking sessions. When Lucchese et al. (2011) analyzed the search logs of 307 search sessions and 1,424 queries from American On Line (AOL), they found that the average duration of each search session was about 15 minutes. The shortest session lasted for less than one minute and had only one or two queries, while the longest session lasted for about two and a half hours. There were on average 4.49 queries in one search session, where half of the sessions had fewer than five queries. The logs were divided into 554 search tasks, and the average number of queries per task was 2.57. On average, a session included 1.8 tasks. Within the total 307 sessions, there were 162 (52.8%) with only one search task, while the rest (47.2%) were multi-tasking search sessions. The number of queries in the multi-tasking sessions was 1,046, which accounts for 74.0% of total queries.

In another study of AltaVista (Spink et al., 2006), researchers found that among the 254 two-query sessions, 206 (81.1%) involved more than one task. There were 254 sessions that included two queries, 206 of which (81.1%) were multi-tasking sessions. There were 483 sessions that included more than two queries, 441 of which (91.3%) were multi-tasking sessions. In the multi-tasking sessions, there were on average 3.2 tasks per session.

Wang et al. (2013) analyzed the search logs collected from Bing.com, a dataset that includes 7,628 users, 37,547 sessions, and 114,723 queries. On average a user participated in 4.9 sessions and made 15.1 queries. There were 8,044 (77.9%) tasks that included only one query, 2,283 tasks (22.1%) that included more than one query, and 1,307 multi-session tasks. The average amount of tasks that a user performed was 7.2. Tasks that generally involved more than one query consisted of 2.8 sessions and 6.6 queries, where the task needed 491.1 minutes to finish.

In order to identify and analyze the characteristics of consumer multi-tasking product Web search, we performed a series of experiments on large-scale product search log records from taobao.com. The whole dataset includes browser clickthrough logs of 4,285 users with 81,759 sessions from taobao.com during the month of May, 2013. The whole dataset contains 1,410,960 records from 81,759 sessions (Yuan, 2014). Each record contains the following fields:

Uid: a uniqueuser code assigned to identify a user;

Figure 1 shows some sample log records.

Figure 1

The log data contains click-through activities of both consumers and shop owners, but we are only interested in the search and browsing activities of consumers. Since shop owners tend to be a lot more active in making purchases than average consumers, we removed users who had too many sessions as belonging to businesses. Figure 2 shows the distribution of the users by the number of sessions.

Figure 2

The x-axis in Figure 2 is the number of sessions, and the y-axis shows the number of users who had a particular number of sessions. The secondary axis of y-axis shows the accumulative percentage of the users. We remove users who belong to the upper 2.5% (with more than 33 sessions), those who were likely to be shop owners rather than regular consumers. The remaining log records include 2,910 users with 18,102 sessions and 47,387 queries. We use one fifth of this dataset for our experiment, which contains 582 users with 3,483 sessions and 8,949 queries. Table 1 shows sample records from these queries.

Table 1

Sample query records.

We use Rwordseg (Li, 2013) as the default dictionary and an additional dictionary containing terms from the Product Catalog acquired from Taobao API

http://open.taobao.com/

Hierarchical clustering stops, however, when the Jaccard indexes between the two clusters are lower than a given threshold. For each method (with the two dictionaries of default and product catalog), we experiment with threshold values ranging from 0.2–0.6 and plot the F-score results as discussed in Session 3.3. Figure 3 shows the results.

Figure 3

As Figure 3 shows, the performances of the clustering methods are most stable between thresholds 0.3 and 0.4. Therefore, we chose the following three thresholds for our later experiments: 0.3, 0.35, and 0.4.

To identify which combination of dictionary, method, and threshold works best for task identification, we created a gold standard with 10% of the experiment data (1,015 search log records) chosen at random. Two human coders examined the query terms and identified product search tasks separately. The coders were instructed to assign a task number to each query in a sequence, where the same task numbers are assigned to queries that belong to the same task. Table 2 presents part of the human task identification results.

Table 2

Sample of human task identifications.

As noted in Table 2, the two coders agreed on most of the queries, but for record #8 (gift card), Coder 1 considered it as a separate task than task #3 (mooncake), whereas Coder 2 considered it as the same task as task #3, making the agreement level for these two human identification results 91.97%. For the records that the two coders did not initially agree on, we asked the two coders to discuss and resolve their different interpretations. We then used the agreed-on identification result as the gold standard to assess different task identification methods used in this paper.

For each identification approach, we calculated standard recall and precision. Recall (R) is the percentage of correctly identified records in all manually identified tasks, and precision (P) is the percentage of correctly identified records in all identified records. Then we calculated the F-measure (F=2PRP+R)$\begin{array}{} \displaystyle (F=\frac {2PR}{P+R}) \end{array} $ to assess each task identification approach.

We experimented with several combinations of task identification methods, dictionaries, and thresholds. The results are shown in Table 3.

Table 3

Task identification results.

Results show that the combination of the clustering method with the maximum similarity score, default dictionary, and threshold 0.35 yields the highest F-measure. So we used this combination with all 8,949 queries in the dataset and identified 6,189 shopping tasks associated with these queries.

Then we analyzed the task characteristics based on the task identification results. Basic characteristics of the sessions and tasks are shown in Table 4.

Table 4

Basic characteristics of sessions and tasks.

On average, users issued 1.45 queries per task, with a maximum of 15 queries in one task. The average number of tasks is 1.78 per session, with a maximum of 41 tasks. The distribution of the sessions according to the number of task included in each session is shown in Table 5.

Table 5

Distribution of the sessions according to the number of task per session.

Of the 3,483 sessions, 2,140 (61.4%) contain only one task, and 38.6% are multitasking sessions. There are 748 (21.5%) two-task sessions and 292 (8.4%) three-task sessions. Only 98 (2.8%) sessions contain more than five tasks.