Examining Users on News Provider Web Sites: A Review of Methodology

Abstract

This project implemented and reviewed several methods to collect data about users’ information seeking behavior on news provider Web sites. While browsing news sites, participants exhibited a tendency toward a breadth-first search approach where they used the home page or a search results page as a hub to which they returned and then linked to other pages. Generally, they browsed before using search. Information seeking patterns were consistent within-user but varied somewhat across users. Most behaviors were characterized as visually scanning with users spending much time scrolling.

The methods used to identify information seeking behavior: (a) information seeking trails, (b) interaction variance, (c) Web pages recurrence, (d) URL frequency, (e) browse behavior identification, and (f) sequence analysis appear particularly useful for detailed analysis of browsing behavior. They afforded information about browsing directionality, complexity, and temporal order. A profile of user browsing behaviors was outlined in the project.

Practitioner’s Take Away

The analysis of information seeking behaviors yielded valuable data in several ways:

• Unlike other research, participants were found to engage in a considerable amount of scrolling. On news sites, users scrolled beyond the fold.
• Despite Web experience, browsers of news sites exhibited a breadth-first search approach.
• User first inclination was to browse rather than search regardless of task (directed or semi-directed) so news site should accommodate browsing.
• Behavior identification defined some common behaviors (see Table 4) that users perform while seeking information on news provider sites. With additional research, it is plausible that this method may provide a profile of user information seeking patterns on news sites.
• Behavior identification and sequence analysis provided data about information seeking behaviors and their sequence. For instance, upon entering news Web sites, users appear to make a brief initial overall visual scan of the page often followed by scrolling. They used the cursor purposefully to scan hyperlinks and when doing so the likelihood of a page change was high. When users scrolled up, the likelihood of a visual scan of text decreased and they were likely to move off the page.
• The multiple methods describe in this paper provide useful means for observing behaviors on news Web sites.

Introduction

More than fifty million Americans use the Internet daily to obtain online news (Horrigan, 2006). News provider services permeate the Web because the Web offers an efficient means of distributing news in multiple mediated forms that can also be personalized (Liang & Lai, 2002). A convergence of newspaper, television, and interactive (blogging) media is occurring on major news sites. Individuals can watch video on a newspaper site, such as usatoday.com, or read news articles on a television-oriented site, such as CNN.com, as well as participate in blogs about various topics. Coupled with this convergence is increased discussion of “citizen journalism” and proliferation of social media-type sites that deliver news more interactively, rather than being filtered and processed by editors. Blogsites are far more opinion-driven than television and newspaper sites, which tend to adhere to traditional journalistic precepts of objectivity, getting both sides of a story and not taking sides except in a specified columnist section. These trends, shaped by persistent and rapid technological change, will no doubt influence site usability as well as the ways users obtain news information.

Information Seeking

Herbert Simon (1996) noted that frequently designers of information systems wrongly perceive a design problem as information scarcity instead of attention scarcity. Consequently, they build systems that excel at supplying more and more information to people but in actuality what is needed are systems that filter out unimportant or irrelevant information. Marketers (and perhaps journalists) are recognizing that the amount of and quality of a consumer’s or reader’s attention is an important way to measure the success of a message. Attention scarcity is a serious concern for organizations using the Web as an information distribution channel. According to Nielsen and Loranger (2006), users spend one minute and forty-nine seconds visiting a Web site before they moved on, and they spend between 25 and 35 seconds on a Web site home page before leaving. While Web sites, including major news sites, continue to provide users more and more information in various media formats, one’s ability or willingness to attend to that information is limited. The preponderance of online information and the potential of attention deficiency are compounded by a proliferation of animated advertisements, media controls, live news feeds, and a host of other interface elements that compete for user attention and alter interactions.

The Web has grown into an information ecology of billions of documents and a multitude of users and thus understanding the complexity of interactions occurring within this system is an arduous scientific endeavor but one of much practical value (Chi, Pirolli, Chen, & Pitkow, 2001). By understanding user behaviors on Web sites, developers can redesign sites to improve usability, accommodate user needs and tasks more effectively, help users achieve their goals more efficiently, and improve Web designs and services overall (Chi, Pirolli, Chen, & Pitkow, 2001; Heer, & Chi, 2002, p. 243). Studying user information foraging behaviors may provide insight into new uses and incarnations of Web devices and applications (Sellen, Murphy, & Shaw, 2002). This is especially important for news provider Web sites where users are actively seeking content that changes on a moment-to-moment basis.

Information foraging theory examines information-gathering and sense-making strategies from an evolutionary ecological perspective. It attempts to understand adaptations of human information seeking strategies and technologies to the fluidity of information in the environment. According to Card et al. (2001), informational spaces of the Web are disseminated into patches such as a Web site or a Web page that segment into other patches to form an information hierarchy. A Web page can be thought of as an information patch that contains link descriptors, content, advertisements, sidebars and other informational regions. Users usually forage for information by navigating Web links (Chi, Pirolli, Chen, & Pitkow, 2001). During foraging, they visually scan and scroll pages and continuously deliberate about whether to stay within a patch, to navigate between patches, or to move on to new patches (Card et al., 2001). Information scent pertains to cues such as hyperlinks that users employ to assess whether or not to pursue a certain path to an information patch (Pirolli, 2006). Cues that have a high information scent provide users descriptive information about the content they will obtain if they pursue the path. Foraging for information can be difficult as users encounter a multitude of sites with varying designs.

Purpose

The informational content of news provider Web sites change daily. Interactive and animated advertisements are among the host of media elements that permeate them. There has been a convergence of journalist styles including television, newsprint, and blogs that influence, among other things, the structure and delivery of newsworthy information. These factors, along with the overall design of news sites, are likely to affect information seeking. It is important to examine users within this dynamic context to better understand their behavior and how they respond to and interact with the plethora of information. With improved understanding, Web sites can be redesigned to more effectively accommodate users (Chi, Pirolli, Chen, & Pitkow, 2001).

This project had two primary purposes. First, guided by previous research (e.g., Card et al., 2001; Nielsen & Loranger, 2006; Pirolli & Card, 1999; Tausher & Greenberg, 1997; White & Drucker, 2007), it employed several methods for observing users during information seeking on news Web sites and attempted to ascertain what, if anything, their behavior suggested about site design. Second, it reviewed the utility of these observational methods. While there are numerous approaches for collecting user behavior data such as analyzing server log click-streams, some may not adequately depict the full scope of user browsing (Card et al., 2001). Therefore, in the author’s view, a review of methodology was germane to the examination of Web behaviors.

Method

The following sections present information about participants, materials, procedures, and information seeking behaviors.

Participants

Graduate and undergraduate students from Duquesne University, Pennsylvania, USA participated in this project by locating information on news provider Web sites. The participant group was comprised of 31 individuals, 51% females and 48% males, with an average age of 23 years. They reported that they were experienced with Web browsing and obtaining news online. Most indicated that they read news Web sites each day or several times daily and that the Web was their preferred medium for obtaining news followed by television and then newspapers. For most participants (50%), their primary areas of the study were Journalism and Multimedia Arts, although some individuals majored in Advertising and Public Relations and English. All were proficient computer users.

Materials

The following six Web sites obtained from popular news organizations that have tradition in either television (e.g., CNN.com) or news print (e.g., New York Times) were chosen for this research:

• CNN: cnn.com
• Fox News: foxnews.com
• MSNBC: msnbc.msn.com
• Pittsburgh Post Gazette: post-gazette.com
• New York Times: nytimes.com
• USA Today: usatoday.com

The author chose these sites based on their orientation (television and news print) as well as popularity. The sites characterized prevailing online news outlets from which many people obtain the news.

Procedure

The following section presents the usability testing procedure.

Usability testing procedure

Marchionini (1995) observed that there are three general types of browsing: (a) directed occurs when browsing is focused, systematic, and aimed at a target; (b) semi-directed occurs when browsing is generally purposeful but not highly systematic and the target is less definite; (c) undirected occurs when browsing is not focused and no goal has been specified. In this project, all participants performed usability tests on the six news provider Web sites. They received an information seeking task that, using Marchionini’s classification, was either directed or semi-directed. For each of the six sites, those given the directed task had to locate two (top) news stories about an environmental issue and to browse to the end of the story. In addition, they had to find (a) when the story was last updated, (b) an author name, if one existed, and (c) an email link to email the story to someone. Participants given the semi-directed task viewed the same six sites and they had to (a) browse each of the sites freely, (b) locate stories of interest, and (c) browse to the end of each story.

Usability tests were conducted individually and the test order determined the task assignment. For example, participant one received the directed task, participant two received the semi-directed task, participant three received the directed task, and so on. In two instances, this ordering varied and the participants were given the semi-directed task instead of directed. The sequencing of the site presentations was counterbalanced to reduce order effects.

To capture participants’ thoughts while performing tasks, a moderator asked them to think aloud. Think aloud protocols are often used in usability testing because they provide valuable insight about user reactions, thoughts, and experiences. They have potential to afford much insight into the strategies participants use as well as their behavior during task performance (Jenkins, Corritore, & Wiedenbeck, 2001). Ericsson and Simon (1996) point out that verbal behavior is one form of behavior from which observations and analyses can be made.

During all tasks, a video camera connected to the participant’s computer recorded facial and verbal expressions. In addition, the Morae software program recorded all on-screen events (mouse clicks, keystrokes, screen changes) and integrated these data with the video and audio into a single digital file for later analysis. After each task, participants exited the browser and then opened the next site. The entire session lasted for approximately one hour.

In combination with a think-aloud protocol, the following measures and approaches were used to collect data:

• Time browsing: Average amount of time participants spent on each Web site.
• Mouse events: All mouse clicks were recorded and tallied.
• Search: All search events and the times at which the events occurred were recorded.
• Information seeking behaviors including: (a) information seeking trails, (b) interaction variance, (c) Web pages recurrence, (d) URL frequency, (e) behavior identification, and (f) the sequence of browsing behaviors.

The last category, “Information seeking behaviors,” included five approaches for observing behaviors, each of which is discussed more fully in the following section.

Information seeking behaviors

The following sections present the information seeking trails, the interaction variance, the Web page recurrence, the URL frequency as a function of distance, the behavior identification, and the sequential analysis.

Information seeking trails

White and Drucker’s (2007) presented a useful method for examining variance among information seeking tasks. Guided by their method, the author labeled browsing events on the information seeking trails as Browse (B), which is a view of a page that lies somewhere on the click path flowing away from the task starting point and Search (S), which is the use of a site’s search facility. Between these events there were two transition directions, forward (f) where the user clicked a hyperlink to a page not previously visited. This is illustrated in the Web Behavior graph (Card et al., 2001) in Figure 1 where the user moves from S1 to S2. The second transition direction is backward (b) where a user revisits a page on the trail, such as a move from S4 back to S3. Because users move forward in most events, (f) transitions were assumed and not labeled. Labeling results in a character string. For instance, the trails illustrated in Figure 1 can be represented as BBSBbS. In other words, there were two browse events, followed by Search and another Browse, and then a transition back followed by another Search.

Figure 1. Web behavior graph

Interaction variance

The extent to which interactions are consistent across information seeking tasks and across users has important implications for news provider Web sites. If, for example, an individual is consistent in how he or she browses but there is variability across users, it suggests that individual users are consistent in their search strategies but strategies vary from user to user. For instance, while person A’s browsing behavior may be consistent each time he or she seeks information, it may be quite distinct from how person B browses. Site developers might effectively accommodate both individuals by allowing them to tailor the placement of news content in ways that suit their unique browsing preferences.

The author employed White and Drucker’s (2007) method to examine interaction consistency by computing the Levenshtein Distance (LD) (1966) using character strings that represented browsing events (e.g., BBSBbS). The LD is the “…smallest number of insertions, deletions, and substitutions required to change one string into another” (National Institute of Standards and Technology, 2007). For example, when the character sequence of BBSB is compared to BBbB, the LD is 1, there is one substitution required to change one string into the other. By comparing the information seeking trails followed by users, it provided an estimate of interaction variance.

Web page recurrence

The content of news sites is usually wide and shallow, which possibly engenders browsing a wide scope of pages. However, Tauscher and Greenberg (1997) found that few pages (Uniform Resource Locator [URL]) are visited frequently and the majority of visited pages are revisits that often include search engines and home pages. They noted that a good predictor of a revisit is how recently a page was last visited. In this project, user data were analyzed to ascertain the number of unique and revisited pages. These data were then compiled for all users. The recurrence of page visits was calculated using the formula presented by Tauscher and Greenberg (1997):

R= Total URLs visited – Different URLs visited Total URLS visited

URL frequency as a function of distance

A second measure related to Web page visits is URL frequency as a function of distance. The distances of recurring URLs for all users were tallied. For example, if a user began a task by visiting CNN.com, browsed to three other pages, and then returned to CNN.com, a distance of 3 was recorded. CNN.com would be the first item in the list of visited pages (distance 0) and beginning with the first URL after CNN.com, the second occurrence of it would be at distance 3. This measure could show if a URL had an equal probability of recurring or if the distribution of visited pages is skewed by those recently visited.

Behavior identification

News site developers have vested interests in creating sites that support effective navigation and successful information foraging. Understanding how users interact with Web pages is vital to these endeavors and of much interest to developers (Miller, 2005). The author made a detailed analysis of the browsing behaviors of eight (4 males and 4 females) of the 31 participants in two of the information seeking tasks: CNN and the New York Times.

Miller (2005) indicates that a cognitive model of Web navigation should account for the visual scan of links, link assessment, selecting links, returning to previous paths, and attempting alternative navigation paths. While Miller’s work relates to a computation model of Web navigation, it served as a basis from which the author analyzed tasks. He reviewed the recordings of participants and noted frequently occurring behaviors (based on Miller’s work) such as when users visually scanned the screen or made link selections. From these observations, he formulated a physically-based coding scheme (Bakeman & Gottman, 1997) to represent the observed behavioral events.

Using Morae, the author studied participants performing each task and at the onset of a behavior event, placed a marker in the recording. For example, if a user began scrolling down, stopped and began using the cursor to scan a list of links, and then scrolled up, the author coded the video frame on which each one of the three events began. In this example, the events would be coded as the following: 1) scroll down – SRD, 2) cursor scan – CS, and 3) scroll up – SRU. The duration of an event ended with the onset of a subsequent event. All events, except the last behavior of the task, were followed by other events, permitting the author to use the onset time to discern event durations.

Sequential analysis

Using the aforementioned coding scheme to represent behavior events, the author calculated transitional probabilities of browsing behaviors for the eight participants. A transitional probability “…is the probability with which a particular target event occurred, relative to another given event” (Bakeman & Gottman, 1997, p. 95).

Results

This section presents findings from data analysis. The findings section is followed by a review of methodology.

General interaction patterns

The following sections present information about time browsing and mouse events.

Time browsing

On average, participants spent 4 minutes on tasks. Initial visits to site home pages lasted an average of 45 seconds (see Table 1). Those with the directed task spent an average of 3 seconds longer on initial visits to home pages than those with the semi-directed task.

Table 1. Average time spend on home page

	CNN	FOX	MSN	POST	NYT	USA
Mean	0:41:4	0:45:0	0:49:4	0:39:2	0:51:2	0:45:0
Max.	1:36:0	1:48:0	2:11:0	2:32:0	2:55:0	1:44:0
Min.	0:02:0	0:07:0	0:04:0	0:05:0	0:03:0	0:14:0

Mouse events

The news sites used in this study had much breadth with numerous links on the home page. During the tasks, the mean number of mouse clicks used to reach the target was 18, with those performing the directed task having slightly more clicks (M=19) than those with the semi-directed task (M=17) (see Table 2).

The Back button is used heavily during navigation and may account for 30% to 40% of navigation events (Catledge & Pitkow, 1995; Tauscher & Greenberg, 1997). Participants clicked Back 271 times across all tasks accounting for 8% of all clicks or navigation events. Participants performing the directed task click Back more (M=11) than those with the semi-directed task (M=7).

Table 2. Average number of mouse clicks by site

	CNN	FOX	MSN	POST	NYT	USA
Combined	17.5	16.2	16.3	19.2	17.8	20.0
Directed	19.4	17.9	17.07	21.1	19.2	20.7
Non Directed	16.0	14.8	15.8	17.7	16.6	19.5

Search

There were 82 searches or instances when participants used the site’s search facility. Individuals performing the directed task made 79 of the searches and those with the semi-directed task performed 3 searches. Most participants began by browsing and only after some attempt at browsing they used the search facility. On average it took participants 1 minute and 7 seconds before they abandoned browsing and performed a search.

Information seeking behaviors

The sections present the information seeking trails, the interaction variance, the Web page recurrence, the URL frequency as a function of distance, the behavior identification, and the sequential analysis.

Information seeking trails

The author labeled browsing events on the information seeking trails as Browse (B) and Search (S) and the transitions between them as forward (f) and back (b). The combination of events and transition resulted in four interaction events: (a) forward-to-search, (b) backward-to-search, (c) forward-to-browse, and (d) backward-to-browse. Figure 2 shows the proportion of the interaction events during all browsing tasks. The least popular interaction was forward-to-search; only 5% of interactions involved this operation. This finding is unlike White and Drucker’s (2007) research that reported 21% of interactions as forward-to-search. It is also noteworthy that only about 7% of the interactions involved backward-to-search and most involved browsing pages on the hyperlink trail.

Figure 2. Browsing events on the information seeking trails

Interaction variance

The Levenshtein Distance (LD) was computed so that every browsing event was compared to every other event followed by a user. Character strings representing each event were created and then compared. For example, two events for a user might consist of the following character strings:

Event 1 (CNN): BBSB

Event 2 (USA): BBbB

The LD (smallest number of substitutions needed to change one string into another) for these events is LD = (event 1, event 2) = 1. Similar to White and Drucker (2007), the author calculated the average distance for all events for a user. Each user had six events, one for each news Web site. The LD was calculated so that each event was compared to the other five. The fewest number of interactions is optimal to locate a target so the event with the smallest average distance from all events was selected as representative of the user’s interaction patterns. The averaged distance of this representative event was used to measure interaction variance so that if the representative event had a high-average distance from all other events for a user, then it was assumed that there was a high variance in the information search patterns of that individual. Conversely, a low-average distance would indicate a low variance for that person.

For all participants, the approximate within-user interaction variance was 1.0 whereas the interaction variance for all users across information seeking trials was 4.0 (SD = 1.3, Max = 6, Min = .2). The browsing events for an individual varied less than the browsing events across users.

Web page recurrence

Table 3 shows the recurrence rate for all users and for those having the directed and semi-directed task. The overall recurrence rate or the portion of pages revisited by users was 27%.

Table 3. Rate of recurrence

	Total pages	Unique pages	Recurrence rate
All users	1472	1069	27%
Directed	707	491	31%
Non directed	765	537	30%

URL frequency as a function of distance

Figure 3 shows the probability of a URL recurrence based on distance. The horizontal axis presents the distance URLs repeated and the vertical axis shows the proportion of the URL recurrence. As shown, there was a 19% probability that the current news Web site URL (distance 1) was a repeat of the previous URL and 3% and 2% probabilities that there were repeats at distances 2 and 3. As the distance increased from the current URL, the probability of the URL being revisited decreased. It should be noted that most (73%) URLs did not have a recurrence. Most participants (76%) with the semi-directed task revisited site home pages compared to roughly half (46%) of those with the directed tasks. However, most (71%) with the directed task revisited the search engine results page (SERP).

Figure 3. URL frequency as a function of distance

Behavior identification

The author examined behaviors of eight users in the context of a problem space (Newell, 1990; Newell & Simon, 1972), defined “…by a set of states, operators for moving between states, an initial state, a goal state, and a current state.” (Card et al., 2001, p. 498). The author identified four problem spaces: (a) link space where states consisted of a URL and moves were clicks of Web links; (b) search space where states consisted of the query strings typed into a search engine and associated hits; (c) visual scan space where states consisted of visible objects and points of regard and moves included reading, eye movements, and scrolling to the display screen elements; (d) interaction space where states consisted of URLs or windows and moves included actions that allowed users to control media (e.g., video, image) or enter information. The author defined the interaction space because it appeared to be unique from the others. It generally involved visual media (video or images) of a specific information domain, overlay windows, and interaction controls that afforded a horizontal rather than vertical orientation.

Figure 4 is a Web Behavior Graphs (WBG) that depicts the navigation structure for a user with the semi-directed task. Each box in the graph represents a state of the problem spaces. For instance, the first two boxes correspond to the visual scan space and the circular object (second row) corresponds to the search space. Arrows signify movements from one state to another and double vertical arrows indicate the return to a previous state. The diagram represents user navigation beginning in the upper left and proceeding to the lower right. From the WBG, one can readily see the complexity of the task and/or the degree to which the user pursued the desired information. Similar to other research (e.g., Jenkins, Corritore, & Wiedenbeck, 2001) where users return to a hub (home page or SERP), semi-directed users returned home and directed users returned to the SERP. In Figure 4 the six boxes on the left represent the hub or the user’s returns to the home page. The individual clicked links on the home page, navigated two or three pages out, and then returned home. Figure 5 depicts an individual with the directed task. He implemented search but failed, returned home but again was unsuccessful at locating a target link. He then retried search and was successful, returning to the SERP to view additional pages. This pattern of short steps away from a hub (home page or SERP) and back was evident in several participants’ browsing and is reflected in the data presented in Figure 3.

Figure 4. Web Behavior Graph, CNN semi-directed

Figure 5. Web Behavior Graph, CNN directed

Table 4 presents the observed behaviors categorized according to the aforementioned problem spaces (link, search, visual scan, and interaction). Across all eight participants, 1398 behavior events were coded for the two tasks (CNN and New York Times). Codes were tallied according to the problem spaces. Table 5 shows that 18% of the behaviors fell within the link space that included clicking Web links and Back. Most events (78%) fell within the visual scan space, while few were categorized as search (1%) or interaction (3%). In terms of the visual scan space, when all scroll-type events are combined, they constitute 39% of all behaviors. Cursor scans, in which the cursor was purposefully passed over points of interest, accounted for 20% of all behaviors, and visual scans, in which participants visually scanned the screen without fixating, made up 18% of behaviors.

Table 4. Problem spaces, behavior event codes, and descriptions

Code	Behavior	Description
Link space
CL	Click link
CI	Click image link
CL	Click link
B	Click Back button
PG	Page change
Search space
S1	Set search	Set search option to search local site or on the Web.
S	Type query in search
Visual scan space
VT	Visually scan text	Fixated on screen text or actually reads text aloud.
VS	Visually scan page	Looks at screen, does not fixate on any specific area, eye movements are large and cover large portions of screen.
SRU	Scrolls up	May look at screen, does not appear focused on screen and scrolls up.
SRD	Scroll down	May look at screen, does not appear focused on screen and scrolls down.
SRH	Scroll horizontally	May look at screen, does not appear focused on screen and scrolls horizontally.
CS	Cursor scan of links	Purposefully passes cursor over links within the page or in navigation areas.
CT	Cursor scan of text	Purposefully passes cursor over text while reading.
CA	Cursor scan of advertisement	Purposefully passes cursor over advertisement, usually while talking about advertisement.
CSP	Cursor scan of image	Purposefully passes cursor over image, usually while talking about image.
CSRD	Cursor scans links while scrolling down	Purposefully passes cursor over screen areas. Highly focused on the screen, may lean forward to look at screen, may verbalize that he or she is searching while scrolling down.
CSRU	Cursor scans links while scrolling up	Purposefully passes cursor over screen areas. Highly focused on the screen, may lean forward to look at screen, may verbalize that he or she is searching while scrolling up.
WV	Watch video	User watches video.
VV	Visually scan video popup window	Looks at video popup window, does not fixate on any specific area, eye movements are large and cover large portions of window.
VSRD	Visually scan page while scrolling down	Highly focused on screen, may lean forward to look at screen, may verbalize that he or she is searching while scrolling down.
VSRU	Visually scan page while scrolling up	Highly focused on screen, may lean forward to look at screen, may verbalize that he or she is searching while scrolling up.
Interaction space
TY	Typing in form
PH	Photo gallery popup	User clicks link to display a photo gallery window.
VD	Video popup	User clicks link to display a video window.
CP	Close popup window
MP	Minimize popup window
MX	Maximize popup window
CPN	Click next button in photo window
VC	Use video controls

 

Table 5. Browsing behavior by problem space

Space	Site (%)
	CNN	NYT	Combined
Link	18.2	18.2	18
Search	1.4	0.4	1
Visual scan	77	79	78
Interaction	4	2.2	3

To examine the amount of time participants spent scrolling as well as cursor scanning, relative to the total time computed for all events, the duration time for like-type behaviors were combined. For example, the time duration for a scroll up (SRU) was combined with scroll down (SRD), and the times for a cursor scan (CS) were combined with cursor text scan (CT). Participants spent 34% of their time in some form of scrolling, and 25% of their time using the cursor to scan points of interest. In addition, participants spent 17% of their time visually scanning the screen without fixating on a specific area and 7% of their time purposefully scanning or reading text.

Table 6 presents the percentage of frequently occurring codes by directed and semi-directed tasks. Participants with the semi-directed task had higher percentages for the majority of codes, most notably, CI (Click image link), SRD (Scroll down), CA (Cursor scan advertisement), and CSP (Cursor scan image). Individuals with the directed task had higher percentages for S1 and S (search), VT (Visually scan text), CT (Cursor scan text), and CSRD (Cursor scans links while scrolling down).

Table 6. Percent of codes by tasks

%	CI	S1	S	VT	VS	SRU	CS	CT	CA	CSP	CSRD	VSRD	VSRU
Directed	15	100	100	51	48	37	28	46	51	9	7	61	45	46
Non directed	85	0	0	49	53	63	72	54	49	91	93	39	55	54

Sequential analysis

Table 7 presents a transitional probability matrix for two-event behavior sequences. The author created a matrix of all the behavior codes (see Table 4) and for each code tallied the code that immediately followed. For instance, suppose a task contained three behavior events that were coded as 1A, 1B, and 2A. This task contains two two-event sequences, 1A to 1B and 1B to 2A. There is one incidence of 1B following 1A and one incidence of 2A following 1B. Thus, on the frequency matrix a 1 would be recorded in the cell where 1A (at lag 0) and 1B (at lag 1) intersect and where 1B (at lag 0) and 2A intersect (at lag 1). To compute the transitional probabilities, the frequency for a particular cell was divided by the total frequency for the row.

For clarity, Table 7 presents only the most relevant codes. It shows that immediately after a page change (PG) there was a 56% probability of a visual scan (VS), a 15% probability of visual scanning while scrolling down (VSRD), and about a 3% chance of clicking a link or Back (CL or B). Following a visual scan (VS), there was a 51% probability of a visual scan while scrolling down (VSRD). Given a preceding VSRD, there was a 35% percent probability of a visual scan (VS), and probabilities of 13% and 10% for visual scan of text (VT) and cursor scan of text (CT), respectively. This suggests that upon entering a page users briefly performed an overall visual scan of it and then scrolled down while being highly focused on the screen. Visual scans while scrolling were periodically interrupted by focused scans of text or reading or by events in which users employed the cursor to scan text.

Compared to visual scans when scrolling down (VSRD), visual scans when scrolling up (VSRU) had a slightly higher probability of being followed by cursor scan of links (CS) and cursor scan of text (CT) but a lower probability of being followed by a visual scan of text (VT). This suggests that users employed different information foraging strategies while scrolling up compared to down. While users employ the cursor during foraging in downward scrolls, they use it to an even greater extent when scrolling up. In addition, data in Table 7 suggest that scanning of text is less likely following visual scan when scrolling up (VSRU) events. It seems that after users scrolled down, they made an assessment about how likely they were to find the information. When they could not find it, they did not leave the page but choose to scroll up, possibly with minimal expectation of finding the information. It is plausible that during these upward scrolls, visual scanning became more generalized whereby users looked for salient features of the page. Finally, there was a 13% probability of a click link (CL) or Back (B) event following a VSRU, suggesting that users are more likely to move off the page after an upward scroll compared to scrolling down.

Navigation events (CL, CI, and B) were most likely to follow cursor scanning behaviors and scrolling up. More than half (52%) of all navigation events occurred after a cursor scan event. In addition, following a scroll up event (SRU), the probability of clicking a link, image link, or Back was 11%, 11%, and 15%, respectively, indicating that after users performed a scroll up event there was a good chance that they would leave the page.

Table 7. Transitional probabilities of browsing behavior events

LAG 0	LAG 1
	CL	B	PG	VT	VS	SRU	SRD	CS	CT	CA	CSP	CSRD	CSRU	VSRD	VSRU
CL	1.2	0.0	93.9	1.2	2.4	0.0	0.0	1.2	0.0	0.0	0.0	0.0	0.0	0.0	0.0
B	0.0	20.5	61.7	0.0	0.0	0.0	0.0	8.8	2.9	0.0	0.0	0.0	0.0	2.9	2.9
PG	2.5	2.5	0.8	2.5	56.4	1.7	2.5	6.8	7.6	0.0	0.8	0.0	0.0	15.3	0.0
VT	2.9	0.0	0.0	4.3	2.9	0.0	4.3	10.1	8.7	0.0	0.0	5.8	0.0	55.0	5.8
VS	3.6	1.6	0.0	2.4	0.4	3.6	2.4	13.4	6.1	0.0	0.0	0.8	0.0	51.2	13.4
SRU	10.8	15.2	0.0	2.1	19.5	0.0	13.0	10.8	8.7	0.0	0.0	0.0	2.1	6.5	0.0
SRD	0.0	0.0	0.0	4.0	16.0	56.0	0.0	8.0	4.0	4.0	0.0	0.0	0.0	4.0	4.0
CS	33.3	1.6	0.8	4.8	8.1	5.6	0.8	4.0	6.5	0.8	0.8	6.5	1.6	18.7	4.8
CT	12.0	4.0	2.0	1.0	6.0	1.0	3.0	8.0	8.0	1.0	0.0	2.0	0.0	44.0	7.0
CA	0.0	0.0	0.0	0.0	0.0	9.0	0.0	18.1	0.0	9.0	9.0	0.0	0.0	54.5	0.0
CSP	14.2	0.0	0.0	7.1	21.4	0.0	0.0	0.0	7.1	7.1	0.0	0.0	0.0	21.4	7.1
CSRD	15.7	0.0	0.0	5.2	26.3	5.2	0.0	15.7	10.5	0.0	5.2	0.0	15.7	0.0	0.0
CSRU	0.0	0.0	0.0	0.0	0.0	0.0	0.0	60.0	20.0	0.0	0.0	20.0	0.0	0.0	0.0
VSRD	0.3	0.0	0.0	13.1	35.4	6.6	0.0	8.68	10.4	1.3	2.4	0.0	0.0	1.3	20.1
VSRU	4.72	7.55	0.00	6.60	21.70	0.00	1.89	15.09	17.92	1.89	2.83	0.94	0.00	16.04	1.89

Methodology review

In this section, the following methods used to collect observational data are reviewed:

• recording of task performance while thinking aloud (with measures of time, mouse and page events, and search);
• identification of information seeking behavior including (a) information seeking trails; (b) interaction variance; (c) Web pages recurrence; (d) URL frequency; (e) Behavior identification; and (f) the sequence of browsing behaviors.

Recording of task performance while thinking aloud

The author found the think aloud protocol extremely useful in understanding user actions. It did not appear to impede task performance. Users were not hesitant about the method and none exhibited or expressed difficulties with it during task performance.

When reviewing the Morae recordings, participants often explained actions enabling the reviewer to make more informed observations. For example, when reading a column of text, a participant encountered an embedded ad that he immediately skipped to continue reading below it. However, at the bottom of the ad, the column of text shifted slightly causing the user to lose his place and to perceive the text below the ad as an entirely new story. He discontinued reading and moved off the site. Because he was thinking aloud, one could readily determine why he left the page and how the ad and the column shift affected behavior. In another instance, an ad covered the entire screen. While the user appeared to be reading the ad, it was clear from her verbalization that she was frustrated by it and was actively seeking a button to remove it. However, without her verbalization, her physical expression suggested that she was interested in reading the ad.

With the appropriate hardware and software, this approach to data collection is straightforward and yields a rich data set. The author used the Morae software from which it was easy to compile data about general behavior patterns, such as time on task and navigational events. The software provides a spreadsheet-like display of interactions and time. Events are linked to the recording so when an event is selected in the spreadsheet, the software accesses it in the recording enabling one to easily discern user actions. A disadvantage of this approach is that when making detailed observations about interactions, a researcher will likely have to sort through the data, which is time consuming. For example, with Morae, one can quickly identify the number of mouse clicks in a recording but it is more time consuming to identify when a specific button or link has been clicked.

Information seeking trails and interaction variance

White and Drucker’s (2007) method for examining variance among information seeking tasks was useful in three primary ways: (a) observing how a user’s information seeking varied across tasks as well as how information seeking varied among users; (b) labeling and describing navigational actions to visually depict how users traversed the informational space. While other methods provided a general sense of amount (number of mouse clicks), this method helped express directionality (as did Web Behavior Graphs) and temporal order. In other words, labeling depicted navigation direction and sequence; (c) creating Web Behavior Graphs enabled information seeking trails to be visualized. The author found that once the behaviors were labeled they could be used to check WBGs and, in some cases, assist in their development.

Using data exported from Morae, the author wrote a program that identified the information seek trails and labeled them. However, in some cases, because ads and superfluous page changes were logged, it was necessary to review the recording and to prepare the data, which was time consuming.

Web pages recurrence and URL frequency

In addition to the Web Behavior Graphs and data collected about information seeking trails, Web page recurrence and URL frequency were useful to identify patterns of browsing. As noted by other researchers (Catledge & Pitkow, 1995; Jenkins, Corritore, & Wiedenbeck, 2001; Tauscher & Greenberg, 1997a, 1997b), a breadth-first pattern was observed in which the users did not stray more than a few clicks beyond a hub (home or SERP). This pattern has been associated with Web novices but in this project, participants were experienced Web users. Calculating Web page recurrence and URL frequency helped make this pattern apparent and supported data presented in the WBG. The aforementioned program written by the author was used to calculate page recurrence and URL frequency. Again, it was necessary to review the recordings and to prepare the data, to weed out ads and superfluous page changes.

Behavior identification and sequence analysis

The behavior identification method provided an in-depth level of analysis of user actions. It helped reveal that users engaged in a multitude of actions that may go undetected using more conventional collection methods. Compared to the other methods, it enabled observation of a more comprehensive set of actions (e.g., scrolling, visual scans, cursor scans) and when combined with sequential analysis provided estimates for forecasting behavioral sequences. The primary disadvantages of this method were (a) the amount of time needed to identify and code behaviors. It required a meticulous review of the Morae recordings, which may be prohibitive with larger user groups; (b) while two-event sequences were analyzed in this project and provided useful information about behavior (behavior A is followed by B), complexities of browsing may require analysis of multi-event sequences.

Table 8 presents the methods used including Web Behavior Graphs (WBG). The table identifies the (a) level of analysis afforded by the method, as used in the project; (b) type of user action (e.g., mouse clicks) for which the method facilitated observation; (c) degree of preparation needed to prepared data; (d) time required to analyze data; (e) potential of method to forecast behavior; and (f) extent to which method afforded data visualization. For the behavior identification and sequence analysis method, the time for data preparation will increase dramatically higher numbers of participants. In addition, approaches used to label search trails and tally recurrence rate will determine the amount of time needed for data preparation. To reduce preparation, the author created a program that compiled search trails and recurrences from exported Morae files. However, a significant amount of time was still needed to prepare the files.

Morae offered many useful features for collecting and analyzing observational data. As the level of analysis became more focused, however, additional measures were needed. Given the value of recurrence rate, information seek trails, and WBGs, it would be helpful for programs like Morae to provide researchers capabilities to extract these data in order to pursue deeper levels of analyses as well as to visualize browsing patterns.

Table 8. Characteristics of methods.

Method	Level	Action	Data prep	Time	Forecast	Visualization
Task performance (think aloud)	Macro: General behavior	Mouse & page events, time	Little or none	Low	No	No
Information seeking trails	Micro: Browse pattern	Page events & sequence	Moderate or High	High	No	Yes
Recurrence/URL frequency	Macro: Browse pattern	Page events	Moderate or High	Moderate	No	No
Behavior ID and Sequence analysis	Micro: Specific behavior	Mouse, scroll, scan, pages, time, etc.	High	High	Yes	No
WBG	Micro: Browse pattern	Page events and sequences	High	High	No	Yes

Conclusion

It is important to point out that the observational findings presented here are not conclusive due to the limitations and scope of this project. The author employed the methods in specific ways and he looked at particular news Web sites with a unique population. Additional research is required to examine information-seeking behaviors across larger groups with a greater variety of news sites and for longer time durations.

General interactions

Using the Morae software, general interaction measures were easily obtained. On average, participants spent four minutes on each information task. The time duration appears long given that most people spend less than two minutes visiting a Web site. However, the information tasks and the controlled setting may have made participants persist when they would otherwise abandon the search. Nielsen and Loranger (2006) report that users spend 31 seconds on the home page and typically avoid scrolling. Participants stayed slightly longer (45 seconds) on home pages and based on a detailed analysis of eight participants, all users scrolled and spent 34% of their time scrolling. These findings may reflect the fact that the news sites used in the project (and many other news provider sites) have much breadth with numerous links on the home page, and pages are typically long requiring users to scroll.

Based on WBG, information seeking trails and recurrence rate analysis, participants exhibited a tendency toward a breadth-first search approach (Jenkins, Corritore, & Wiedenbeck, 2001) often associated with Web novices. Sixty-one percent of all users had at least one recurrence of the home page and 71% of directed users had one or more recurrence of SERP. Users appeared to use either home or the SERP as a hub from which they made short steps away and then returned. The finding is surprising given that Web novices typically exhibit this pattern but in this project participants were experienced at Web browsing. While speculative, the pattern of browsing may be indicative of information seeking on news sites whereby users identify a new story of interest on the home page, browse to the story, and then return home (or to the SERP) to find additional news stories of interest.

Browsing versus search

Some researchers cite users’ inclination to use search (Nielsen & Loranger, 2006) while others report that users choose to browse over search and they are more successful at finding the target content (Campagnoni & Erlich, 1989; Katz & Byrne, 2003; Miller, 2005). In this project, participants’ first inclination was to browse rather than search regardless of task (directed or semi-directed). In most cases, they resorted to search only after they were unsuccessful at locating information. With the prevalence and sophistication of search engines, it is surprising that users chose browsing over search. This suggest that when browsing news sites, users are likely to browse before attempting search and thus the design of sites should support user browsing.

Information seeking behaviors

During information seeking tasks, the most common type of interaction was forward-to-browse. When forward-to-browse and back-to-browse are combined they comprised over 88% of all interactions and search interactions comprised only 12%. This finding suggests that much interaction will involve browsing using hyperlinks and, to a considerable extent, users will be engaged with page content and navigating it with hyperlinks. In this regard, a noteworthy observation made by almost all participants related to interruptions caused by advertisements. Aside from expressing frustration with ads, participant browsing was at times diverted by them. Ads do interrupt browsing and reading, especially when embedded within text. A number of participants suggested that they should be positioned in a column on the right side of the page.

When behaviors are compared by tasks, differences appeared that may have implications for site developers. For instance, semi-directed users engaged in more scrolling and were more inclined to click images and scan advertisements. Conversely, directed users were inclined to search and to scan links while scrolling down. This is relevant because most participants complained about ads and felt they interrupted browsing. While preliminary, these findings suggest that semi-directed users are more inclined to look at ads. One can envision browsers that are intelligent enough to sense when a user is semi-directed and based on their behaviors, the browser displays, highlights, or repositions ads making them more prominent. Conversely, as the user becomes more directed, ads may dim or be redistributed on the page to facilitate information seeking. This is highly speculative and, from advertisers’ perspective, more research is needed to ascertain if users will look at ads that dynamically adjust to accommodate information foraging compared to those that seemingly intrude on browsing. Ads will likely remain part of the browsing experience. However, their ability to respond to user behaviors or needs may be mutually beneficial to users as well as advertisers.

Interaction variance is important because it has implications for Web personalization and predicting information retrieval (White & Drucker, 2007). Overall, participant information browsing events were consistent both within-user and across users. There was greater variance across users, which suggests that while an individual’s information foraging may have been consistent from one event to the next, it differed from the foraging behavior of other participants. While more research is needed, this intimates that users will vary in how they use online news sites and thus providing users the capability to personalize sites may better accommodate their browsing needs.

Methods

The think aloud protocol helped the observer to accurately identify user actions and to understand his or her rationale for executing an action. Measures of time and mouse clicks afforded general indicators about browsing but needed to be supplemented with other, generally more time consuming, means that provided information about browsing directionality, complexity, and temporal order. As the level of analysis became more granular, greater specificity was achieved using the information seek behavior methods described earlier.

Acknowledgements

The author wishes to thank Lavanya Jayarama for all her research assistance that was valuable to this project.

References

Bakeman, R., & Gottman, J. (1997). Observing Interaction: An introduction to sequential analysis. Cambridge: University Press.

Campagnoni, F. R., & Erlich, K. (1989). Information retrieval using a hypertext-based help system. Proceedings of the 12th annual international ACM SIGIR Conference on Research and development in information retrieval (pp. 212-220). ACM Press.

Card, S., Pirolli, P., Van Der Wage, M., Morrison, J., Reeder, R.W., Schraedley, P.K., & Boshart, J. (2001). Information scent as a driver of web behavior graphs: Results of a protocol analysis method for Web usability. Proc. CHI 2001, 498-505.

Chi, E. H., Pirolli, P., Chen, K., & Pitkow, J. (2001). Using information scent to model user information needs and actions and the Web. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Seattle, Washington, United States). CHI ’01 (pp. 490-497). New York, NY, USA. ACM.

Ericsson, K. A., & Simon, H. A. (1996). Protocol analysis: Verbal reports as data. Cambridge, Massachusetts: The MIT Press.

Gary M. Marchionini (1995). Information Seeking in Electronic Environments. Cambridge, England: Cambridge University Press

Heer, J., & Chi, E.H. (2002). Separating the swarm: Categorization methods for user sessions on the Web. Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves (pp. 243-250). Minneapolis, Minnesota, USA.

Horrigan, J. B. (2006). For many home broadband users, the Internet is a primary news source. Retrieved December 3, 2007, from http://www.pewinternet.org/PPF/r/178/report_display.asp

Jenkins, C., Corritore, C. L., & Wiedenbeck, S. (2003). Patterns of information seeking on the Web: A qualitative study of domain expertise and web expertise, IT&SOCIETY, 1(3), 64-89.

Katz, M. A., & Byrne, M. D. (2003). Effects of scent and breadth on use of site-specific search on e-commerce web sites. ACM Trans, Computer-Human Interaction, 10 (3), 198-220.

Levenshtein, V. (1966). Binary codes capable of correcting deletions, insertions and reversals. Soviet Physics Doklady, 10(8), 707-710.

Liang, T., & Lai, H. (2002). Discovering user interests from web browsing behavior: An application to internet news services. In Proceedings of the 35th Annual Hawaii international Conference on System Sciences (Hicss’02) Volume 7 (January 07-10, 2002). HICSS. 203. Washington, DC, USA. IEEE Computer Society.

Miller, C. S. (2005). Modeling Web Navigation: Methods and Challenges, Lecture Notes in Computer Science, Volume 3169, Nov 2005, pp. 37-52.

National Institute of Standard and Technology (2007). Levenshtein distance. Retrieved December 24, 2007, from http://www.nist.gov/dads/HTML/Levenshtein.html

Nielsen, J, & Loranger, H (2006). Prioritizing Web Usability. Berkeley, California: New Riders Publishing.

Newell, A. (1990). Unified Theories of Cognition. Cambridge, MA: Harvard University Press.

Newell, A., & Simon, H. A. (1972). Human Problem Solving. Englewood Cliffs, NJ: Prentice-Hall.

Pirolli, P. (2006). The use of proximal information scent to forage for distal content on the World Wide Web. In A. Kirlik (Ed.) Adaptive perspectives on human-technology interaction. (pp. 247-266). Cambridge: Oxford University Press.

Pirolli, P, & Card, S. (1999). Information foraging, Psychological Review, 106(4), pp. 643-675.

Sellen, A.J., Murphy, R., & Shaw, K. L. (2002). How knowledge workers use the Web. Proceedings of the SIGCHI conference on Human factors in computing systems: Changing our world, changing ourselves. (pp. 243-250) Minneapolis, Minnesota, USA.

Simon, H. A. (1996). The Sciences of the Artificial (3rd ed.). Cambridge, MA: The MIT Press.

Tauscher, L., & Greenberg, S. (1997a). How people revisit web pages: Empirical findings and implications for the design of history systems, International Journal of Human Computer Studies, 47, pp. 97-137.

Tauscher, L., & Greenberg, S. (1997b). Revisitation patterns in world wide web navigation. In Proceedings of the Conference on Human Factors in Computing, CHI’97. (pp. 399-406) New York, NY, USA. ACM.

White, R. W., & Drucker, S. M. (2007). Investigating behavioral variability in web search. In Proceedings of the 16th international Conference on World Wide Web (Banff, Alberta, Canada, May 08-12, 2007). WWW ’07. (pp. 21-30) New York, NY, USA. ACM.

Published in: in Volume 3, Issue 3,