SIGCHI Conference Paper Format

Notifications and Awareness:

A Field Study of Alert Usage and Preferences

Shamsi T. Iqbal and Eric Horvitz

Microsoft Research

One Microsoft Way, Redmond, WA 98052

{shamsi, horvitz}@microsoft.com

ABSTRACT

Desktop notifications are designed to provide awareness of

information while a user is attending to a primary task.

Unfortunately the awareness can come with the price of

disruption to the focal task. We review results of a field

study on the use and perceived value of email notifications

in the workplace. We recorded users’ interactions with

software applications for two weeks and studied how

notifications or their forced absence influenced users’ quest

for awareness of new email arrival, as well as the impact of

notifications on their overall task focus. Results showed

that users view notifications as a mechanism to provide

passive awareness rather than a trigger to switch tasks.

Turing off notifications cause some users to self interrupt

more to explicitly monitor email arrival, while others

appear to be able to better focus on their tasks. Users

acknowledge notifications as disruptive, yet opt for them

because of their perceived value in providing awareness.

Author Keywords

Task Switching, Notifications, Workplace, Interruption.

ACM Classification Keywords

H5.2. [Information interfaces and presentation]: User

Interfaces.

INTRODUCTION

To many, fast-paced multitasking defines the dominant

activity pattern of information workers [9, 10]. As

communication with collaborators continues to be an

integral part of today’s information workers’ tasks,

switching among productivity and communication

applications remains a common practice. Such switches are

often driven by the user’s own need to forage information

as per demands from the ongoing task, or after being

proactively alerted about arrival of new information.

Notifications, in the form of visual pop-ups or auditory cues

have been long viewed as an effective way of proactively

helping users maintain information awareness. While such

notifications hold great benefit in saving users the effort of

seeking information themselves, they unfortunately have

also been considered to be a large source of disruption to

ongoing tasks [2, 3, 7]. This assessment is due in part to the

fact that notifications often occur at inopportune moments

and lure users to process and respond to them [3, 7, 8].

Studies have shown that responding to notifications

impedes the efficient resumption of interrupted tasks and

negatively impacts task performance [1, 2]. A proposed

solution is to relinquish notifications and allow users to

seek for information at their own convenience. The

argument is that users will chose to switch attention from

ongoing tasks at moments that are more suitable for them,

rather than allowing a notification to divert their attention

unexpectedly during less opportune moments.

However, it is not clear how this shift in the balance

towards reducing unexpected disruption and increasing

effort in maintaining awareness may affect users and their

tasks. We pursue an answer to these fundamental questions:

How do notifications support users’ need for information

awareness? What effect does this need have on their focus

on tasks that are interrupted by arrival of notifications? This

work provides the initial steps in answering these questions.

Our goal is to understand how interruptions caused by

notifications influence users’ focus on ongoing tasks and to

contrast this with task focus if notifications are turned off.

We focus on notifications for email communications. We

conducted a field study with 20 users where we recorded

desktop interactions for two weeks, with the intervention of

disabling notifications during the second week. Results

showed that email notifications are primarily used as an

awareness mechanism, rather than a trigger to switch to

check email or perform other activities on the email client.

Approximately a quarter of the notifications users receive

result in an instant switch to the email client that had

initiated the notification and these accesses are typically

short in duration. We saw two behaviors, depending on the

user, after users followed a request to turn off notifications:

rates of accessing the email client went up for some users

and were reduced in others. Post-study surveys revealed

that while users are aware of the distractions that

notifications can cause, they highly value the awareness of

information provided. These results further highlight the

importance of balancing cost and benefits in delivering

notifications. We discuss design implications and directions

for future research.

Permission to make digital or hard copies of all or part of this work for

personal or classroom use is granted without fee provided that copies are

not made or distributed for profit or commercial advantage and that copies

bear this notice and the full citation on the first page. To copy otherwise,

or republish, to post on servers or to redistribute to lists, requires prior

specific permission and/or a fee.

CSCW 2010, February 6–10, 2010, Savannah, Georgia, USA.

STUDY

Our focus in this study was on the impact on ongoing tasks

and satisfaction of users’ information needs resulting from

notifications announcing arrival of new email.

Notifications appear regardless of user state, task switches

are generally not planned for, and as a result, require

unexpected suspension of the ongoing task and

occasionally, switch of work context. Specifically, we

sought answers to the following questions:

1. How disruptive are email notifications in their role of

providing information awareness to users? Does this

disruption reduce with notifications turned off?

2. Following a period of time with notifications turned

off, will users perceive the benefits of minimizing the

potential disruption they cause and leave them off?

Methodology

We conducted a two-week in situ field study collecting

desktop interaction data from computer users to answer

these questions. Microsoft Outlook was used as the email

client, a widely used application within our organization.

Outlook is used for a variety of tasks beyond email

management (e.g., calendaring activities and maintaining

to-do lists and contacts). Our interest was in the user’s

decision to switch to Outlook when they would receive an

email notification, but once they switched we did not

distinguish among the usage of Outlook. Outlook

notifications appear as a small modal window in the lower

right corner of the screen and persist for about 7 seconds

before fading away. Notifications are enabled by default,

but users can disable them if so desired. We only recruited

users who had Outlook notifications enabled.

Users were recruited through a random selection process

applied on the entire employee pool of our organization. 42

people signed up and eventually 20 users completed all

stages of the study (Managers=12, Developers=8). Users

were compensated with lunch coupons on completion. The

study also administered a pre- and post-survey to collect

self-reports on users’ Outlook usage behavior, preferences

and perceptions on notification usage.

Data Collection

Data was collected using a monitoring tool running as a

background process in users’ primary work machines. The

tool logged time-stamped names of applications in focus

and arrival of notifications. Logged data files were

periodically flushed to a central server and later processed

to be stored in an SQL database for future analysis.

For the first week, baseline data was collected without any

intervention. For the second week, users were instructed to

disable all notifications within Outlook. We collected data

in the no-notification condition for a week which allowed

users to settle into the new configuration. We assume that

the information needs of users in week 1 and week 2 of the

study were relatively stable. We have no reason to believe

that they changed, but there is opportunity to explicitly

control for such potential instability in future research.

RESULTS

Over 1682 hours of data was collected from the 20 users. In

the pre-survey, users were asked to provide a list of

applications that they used extensively as part of their

primary job description. These primary applications were

studied in the logs to evaluate task focus. For the purpose of

comparison, we examine time spent on Outlook and

primary applications, and reaction to notifications across

the two weeks.

Time spent on email and primary applications

In the baseline condition of notifications turned on, users

spent on average 30.5% (S.D. 12.1) of their active

computing time in Outlook, and 33.2% (S.D 18.0) of their

time interacting with other primary applications every

session. We define a session to be delimited by either the

logging on and off or by the unlocking and locking of a

machine. Turning off notifications did not yield significant

changes in these percentages; 31.7% (S.D. 13.7) of user

time was spent on Outlook, and 34.8% (S.D. 16.4) was

spent on other primary applications. This indicates that

Outlook occupies a significant percentage of users’ time,

and notifications do not appear to affect this percentage.

Reactions to Notifications

Users received on average 3 (S.D. 2.12) email notifications

from Outlook per hour, consistent with findings in [7, 8]. 4

users demonstrated no immediate responses (within less

than a minute) to notifications. The remaining 16 users

switched to Outlook solely as a result of a notification for

26.2% (S.D. 30.3) of the notifications. This shows that

majority of notifications do not cause users to immediately

suspend their ongoing task and switch to the source. Post

study surveys suggested that the information nuggets (e.g.

sender, subject) provided in the notification are sufficient

for users to garner the important parts of the message and

decide whether it merits immediate attention. As one user

stated : “I’m usually able to tell whether they’re worth viewing right

away from the title.” Users widely acknowledged the passive

awareness and ability to ignore messages or defer responses

as a valuable service of notifications – aptly summarized by

one user : “It’s just nice to know when new mail has arrived – even

if I don’t check outlook at that moment.”

Time spent on Outlook

When users switched to Outlook solely as a result of seeing

the notification, they spent on average 74.9s (S.D. 34.6) in

Outlook. This is significantly lower (F(1,15)=5.502,

p<0.04) than average time spent on Outlook (M=133.9s,

S.D. 106.1) when accessed without being prompted by a

notification. This suggests that Outlook switches triggered

by notifications are more opportunistic and users wish to

quickly return to their suspended tasks.

In the no-notifications condition users spent 149.9s (S.D.

123.4) on Outlook on every access. This was significantly

higher (F(1,15)=6.256, p<0.024) than with accesses

triggered by notifications, but not different than accesses

without notifications triggers in the baseline condition. It

appears that despite not having the awareness of new

Figure 1: Illustration of how Outlook access rates change with

notifications disabled for all users, normalized to an hour. The large

number of Outlook accesses represent users whose job roles require them

to be responsive to email.

information that notifications provide, users can still fulfill

their information needs without significant increases in the

time spent in Outlook when accessed independently.

Outlook Access Rates

In the no-notifications condition, users accessed Outlook at

a rate of 19.3 times/hour (S.D. 12.9), not significantly

different than the Outlook access rate with notifications

enabled (M=21.9, S.D. 17.8). However, the post study

survey showed that 13/18 respondents felt that with

notifications turned off their Outlook access rates had

changed. 7 users reported to have voluntarily switched to

Outlook more under the no-notifications condition and 6

users reported to have switched less. This prompted us to

explore whether the collected data also showed this

dichotomy in Outlook access rates across the experimental

conditions. Indeed, quantitatively one group (N=12)

demonstrated a decrease in their Outlook access rates

(M=40.7%, F(1,11)=15.73, p<0.002) while the other group

(N=8) showed an increase (M=130.6%, F(1,7)=8.04,

p<0.025). Figure 1 shows how Outlook access rates

changed for all 20 users from week 1 to week 2. It appears

that while some users were perhaps compensating for the

lack of awareness by accessing Outlook on their own at an

increased rate, for others absence of notifications allowed

them to be more focused on their ongoing tasks.

The ability to focus more by sacrificing automatic

awareness was not necessarily lauded by users. About

notification usage one user commented: “I will continue to use

them. I know I am more productive when I don't but I like keeping

abreast of my new email.” Another user stated: “ I found with my

notifications removed, that I would get more work done on whatever

particular task I was doing… unfortunately, because I had no instant

notifications, a lot of time would go by before I’d remember to check

my Outlook—then I would have a ton of email to catch up on…”

It appears that the lightweight and passive delivery

mechanism of notifications works well with users’ needs for

awareness, and the value of information provided often

compensates for the disruption to ongoing tasks.

Outlook Access Patterns

We determined two patterns in Outlook accesses: rapid,

almost cursory (duration <10s), and longer, spanning

multiple minutes. We hypothesized that the first type

indicates accesses in pursuit of awareness, where the user

would want to check email arrival status (or calendar

information) and then quickly return to their primary tasks,

while the other type of access represents more in depth

interaction, e.g., either planned or opportunistic reading and

replying to emails. There were no differences overall in

rapid and longer access across the experimental conditions.

However, for users whose Outlook accesses increased with

notifications disabled, their longer Outlook access rates

increased significantly (M=146.2%, F(1,7)=14.1, p<0.007).

Their rapid Outlook accesses did not increase. As one user

stated in the post survey, this could be because this group of

users now spent more time catching up with new

information in Outlook, as they no longer could be

passively aware of it via notifications.

For users whose Outlook accesses decreased, there was

both a significant decrease in longer Outlook accesses

(M=35.3%, F(1,11)=6.844, p<0.024) , and in rapid Outlook

accesses (M=43.3%, F(1,11)=11.9, p<0.005). For this class

of users, the absence of notifications appears to reduce the

footprint of Outlook in the space of their focal applications,

which, as users later stated, negatively impacts awareness.

Task Execution Behavior

For the first week with notifications enabled, users accessed

on average 15.3 (S.D. 10.4) applications, had 1.05

application switches/min (S.D. 0.55) and spent 416.5s (S.D.

399.7) on each application per switch. With no

notifications, users accessed 19.4 applications (S.D. 16.4),

switched applications 0.97 times/min (S.D. 0.5) and spent

250.2s (S.D. 311.7) on each application, none of which

were significantly different than in the baseline condition.

This suggests that user focus on ongoing tasks is not

significantly impacted with notifications turned off,

primarily because users reportedly are using notifications to

increase awareness and not as a trigger to switch. There was

no effect of job role in any of the dependent measures.

User Perception about Notifications

A basic goal of this study was to understand how users

perceived the use of notifications in their day-to-day

activities. Our post study surveys showed that users do

indeed acknowledge notifications to be often disruptive: “

[…] is very tempting to go and see the email when you receive a

notification”, but at the same time the lack of awareness

could often be costly: “There were too many times when I had

realized I hadn’t read mail in several hours and was way behind in

responding to international teams. Using the notifications I could

often gauge if something was important or not.”

When asked whether they would want to revert back to

using notifications, 17/18 respondents responded in the

affirmative. Users provided valuable feedback about how

notification delivery should consider a user’s current state

and perhaps render notifications only for important emails.

For example, one user stated:

“Yes. It can be distracting, but I also like the ability to work in

another program and monitor my inbox for important events.

I'd prefer if I could filter and determine which emails I wanted

to be notified of. This should be by person/group, but i also

think it'd be fantastic to set a "watch" on an email thread and

be notified whenever a new email in that thread appears.”

Such comments highlight users’ valuing awareness provided

by notifications. Perhaps a combination of how notifications

are currently being delivered and users’ natural propensity

to multitask is more responsible for the distraction that they

admittedly cause.

DISCUSSION

Our findings provide evidence of the non-invasive role that

notifications play on pursuit of awareness and task

execution behavior. That users continue to work on their

primary applications for three quarters of email

notifications indicates that users can effectively choose

which notifications to respond to. Users appear to find the

awareness aspect of notifications valuable and to maintain

that awareness, are willing to accept the potential

disruption. Even when users admitted to getting more work

done with notifications disabled, they also found it to be

counterproductive, e.g., having ”a ton of email to catch up with

and sift through.” This finding indicates that by reducing one

type of disruption, we may be introducing another. The

results suggest that design efforts should be focused on

balancing disruption and awareness.

Our results allude to a possible categorization of users, or to

users within different work contexts, by their pattern of

attention and disruption, based on their multitasking

behavior. Turning off notifications appears to have affected

users in different ways; some exhibited a greater need to

interrupt themselves to monitor information arrival while

others could remain more focused on their primary tasks.

This presents opportunities for investigating notification

design customized to different user types and their needs.

As one user mentioned: “It’s about discipline in using

technology.” Indeed, various levels of such discipline were

observed in the task execution behavior of the users.

Although we focused on how user behavior was changed by

the removal of notifications, we did not explore why this

change may have occurred. Feedback from users suggested

that consequences of missing urgent information, context of

user tasks and users simply not getting totally used to not

having notifications to remind them of new information

could be some of the reasons. Additional research is

required to explore these issues in more depth.

We believe that there is an opportunity to enhance the

controls on notification presentation to better serve users’

desire to stay aware. This study provides support to the

potential value of a trusted system that can reliably identify

subsets of incoming messages that a user would most like to

be aware of. Promising methods include the use of

machine learning to classify the urgency of messages,

smarter scheduling of presentation [4, 6], and rule-based

control of notifications in [5]. Users could specify

thresholds on urgency for notifications and provide

maximum bounds for deferral if they are currently focused

on a task [5]. Such methods could reduce distraction while

maintaining awareness of key developments. Future designs

of notifications could take topics and context (e.g.,

messages from people in meetings that will occur soon) into

consideration, providing even more insightful controls of

notifications.

CONCLUSION AND FUTURE WORK

We investigated the effects of email notifications and their

imposed absence on users’ task-execution patterns. Results

showed that users react to only about a quarter of all

notifications, and that user focus on primary tasks is largely

unaffected if notifications are disabled. Moreover, users

value the awareness provided by notifications and are

willing to incur some disruption to maintain that awareness.

Future work includes field studies of the use of notification

algorithms that take into account the timing and urgency of

information conveyed [4] and studying the influences of

such notifications on user focus of attention.

REFERENCES

1. Bailey, B.P. and Konstan, J.A. On the Need for Attention

Aware Systems: Measuring Effects of Interruption on Task

Performance, Error Rate, and Affective State. Journal of

Computers in Human Behavior, 22 (4). 709-732.

2. Cutrell, E., Czerwinski, M. and Horvitz, E., Notification,

Disruption and Memory: Effects of Messaging Interruptions

on Memory and Performance. in Proceedings of the IFIP

TC.13 International Conference on Human-Computer

Interaction, (Tokyo, Japan, 2001), 263-269.

3. Czerwinski, M., Horvitz, E. and Wilhite, S., A diary study of

task switching and interruptions. in Proceedings of the ACM

Conference on Human Factors in Computing Systems,

(2004), 175-182.

4. Horvitz, E., Apacible, J. and Subramani, M. Balancing

Awareness and Interruption: Investigation of Notification

Deferral Policies. Ardissono, L., Brna, P. and Mitrovic, A.

eds. User Modeling 2005, 10th International Conference,

Springer, Edinburgh, Scotland, UK, 2005, 433-437.

5. Horvitz, E., Jacobs, A. and Hovel, D. Attention-Sensitive

Alerting Conference Proceedings on Uncertainty in Artificial

Intelligence, 1999, 305-313.

6. Iqbal, S.T. and Bailey, B.P., Effects of Intelligent

Notification Management on Users and Their Tasks. in

Proceedings of the ACM Conference on Human Factors in

Computing Systems, (Florence, Italy, 2008), 93-102.

7. Iqbal, S.T. and Horvitz, E., Disruption and Recovery of

Computing Tasks: Field Study, Analysis and Directions. in

Proceedings of the ACM Conference on Human Factors in

Computing Systems, (San Jose, California, USA, 2007), 677-

686.

8. Mark, G., Gonzalez, V.M. and Harris, J. No task left behind?:

examining the nature of fragmented work in Proceeding of

the SIGCHI conference on Human factors in computing

systems ACM Press, Portland, Oregon, USA 2005, 321-330

9. Pashler, H. and Johnston, J.C. Attentional Limitations in

Dual Task Performance. in Pashler, H. ed., Psychology

Press/Erbaum (UK) Taylor & Francis, Hove, England, UK,

1998, 155-189.

10. Wickens, C.D. Multiple Resources and Performance

Prediction. Theoretical Issues in Ergonomic Science, 3 (2).

159-177.