Augmented Cognition
The Emergence of Augmented Cognition
Research relating to the concept of augmented cognition has been conducted for several decades in the cognitive science and HCI communities. There was a marked increase in papers describing this general area of research in the late 1990’s, including efforts to build and use models of attention in information display and notification systems. However, the phrase “Augmented Cognition” associated with this research did not find widespread use until the year 2000, culminating with both a Defense Advanced Research Project Agency (DARPA) Information Science and Technology (ISAT) Group study and workshop at the National Academy of Sciences on the field. Starting in the year 2002, there was an increase in the number of papers in the scientific literature detailing research specifically on this topic. This was due, in part, to the start of a DARPA research program in the area of Augmented Cognition in 2001, with a focus on challenges and opportunities with the real-time monitoring of cognitive state with physiological sensors. This substantial investment in these developing technologies helped bring together a research community and stimulated a set of thematically related projects on addressing cognitive bottlenecks via the monitoring of cognitive state. By 2003, the Augmented Cognition field extended well beyond the boundaries of those specific Department of Defense research projects, but that initial investment provided impetus for the infant field to begin to mature.
Early Investments in Related Work
Augmented Cognition is a field that does not reside in just one scientific discipline — it draws from areas such as neuroscience, biopsychology, cognitive psychology, human factors, information technology, and computer science. Each of these fields has itself undergone a substantial revolution over the past forty years that has allowed the problems/challenges raised by the researchers in this field to begin to be satisfactorily investigated. Although there are many individual research projects that contributed to the general development and direction of this field, several multimillion dollar efforts helped shape the foundation on which the current augmented cognition field is built.
Since the invention of the electronic computer, scientists and engineers have speculated about the unique relationship between humans and computers. Unlike mechanized tools, which were primarily devices for extending force and action, the computer became an entity with which humans forged an interactive relationship, particularly as computers came to permeate everyday life. In 1960, one of the great visionaries of intelligent computing, J.R. Licklider, wrote a paper on the “Man-Computer Symbiosis.” Dr. Licklider served as the Director of the Information Processing Techniques Office (IPTO) at the Department of Defense’s Advanced Research Projects Agency (ARPA) in the 1960’s. In this seminal paper, he stated “The hope is that, in not too many years, human brains and computing machines will be coupled together very tightly, and that the resulting partnership will think as no human brain has ever thought and process data in a way not approached by the information-handling machines we know today (Licklider, 1960).” Almost prophetic, this description of the symbiotic relationship between man and computer is one of the first explanations of what could be considered an augmented cognition computational system. Although research on this topic was not conducted during his tenure at ARPA in the 1960’s, Dr. Licklider championed the research which developed into the now burgeoning field of computer science, including creation of the ARPAnet (the first Internet). His research, vision and direction had a significant impact on both computer science and information technology and set the stage for the future field of augmented cognition.
In the early 1960’s researchers speculated that electrical signals emanating from the human brain in the form of electroencephalographic (EEG) recordings could be used as indicators of specific events in human cognitive processing. Several Department of Defense investments in detecting these signals and other measurements occurred through the Biocybernetics and Learning Strategies programs sponsored by ARPA in the 1970’s and 80’s. The earliest program was Biocybernetics, which tested the hypothesis that EEG activity might be able to control military devices and serve as indicators of user performance. In this program, biocybernetics was defined as a real time connection between the operator and computational via physiological signals recorded during specific tasks. Both the Biocybernetics and Learning Strategies efforts centered around the creation of closed loop feedback systems between operator and computer for the purposes of selection and training of personnel, display/control design, and online monitoring of operator status (although with slightly different military application domains between the two programs). In both programs, the real time identification of cognitive events was seen as critical to understanding the best methods for aiding military users rapidly and in a contextually appropriate way. However, at the time this research was begun, both computational systems and neuroscience were in their infancy, and the results of this research were not incorporated into production military systems. Augmented Cognition can be viewed as a descendant of these early programs.
A further investment in this type of work was through the Pilot’s Associate (PA) program sponsored by the Defense Advanced Research Agency (DARPA) in the 1980’s and early 1990’s. The goal of this program was to design an expert pilot aid for increased situational awareness and enhanced decision making. Pilot’s Associate was an integrated system of five components which incorporated AI (artificial intelligence) techniques and cognitive modeling to aid pilots in carrying out their missions. Unlike Biocybernetics, PA utilized cognitive modeling alone and did not incorporate any physiological monitoring. Cognitive modeling was the cornerstone of the Pilot Vehicle Interface, and although only one subcomponent of the entire Pilot’s Associate system, the PVI was tasked with critical role of managing all pilot interactions with the system. The main goal of the PVI was to infer the pilot’s intentions and communicate these intentions to the other components of the PA system. In addition to this central task it was also responsible for modeling pilot workload to adapt and configure the information displays in the cockpit, conveying workload information to the other subsystems, and compensating for pilot behavior that might result in an error. An example of this work was a PA program effort at NASA-Ames Research Center that explored the use of probabilistic models of a pilot’s goals and workload over time, based on multiple inputs and the use of the models to control the content and complexity of displays. Such models did not employ physiological measures of the pilot’s cognitive status. The best recommended courses of action were determined from inferences based on observations of the behavior of the aircraft and pilot. The Pilot’s Associate program contributed to setting the stage for Augmented Cognition that was to develop over the next decade.
Other background work includes efforts in the academic and private sectors, including the AUI (Attentional User Interface) project at Microsoft Research in the late 1990s which provided conceptual support to efforts in Augmented Cognition. In this work, methods were developed for building statistical models of attention and workload from data. Architectures were constructed, which demonstrated how these cognitive models could be integrated with real-time information from multiple sensors, including acoustical sensing, gaze and head tracking, and events representing interaction with computing systems to control the timing and communication medium of incoming notifications. AUI work that included psychological studies was performed to complement the systems and architectures work.
Foundations of Augmented Cognition
Based on these descriptions of earlier research efforts, the logical question arises - what sets Augmented Cognition apart from what has already been done? As mentioned in the section on related work, augmented cognition relies on many disciplines whose maturity is critical for its success. Although programs like Biocybernetics in the 1970’s had similar goals, they did not have access to the advanced computational power necessary for the processing of brain signals in real time, nor did researchers know enough about those signals to use them to control displays or machines. Likewise, the Pilot’s Associate program in the 1980’s shared many common aspirations of today’s Augmented Cognition, namely to develop adaptive interfaces for reduction of pilot workload. However, PA could only assess the status of the pilot from inferences and models based on the pilot’s overt behavior and the status of the aircraft. What distinguishes augmented cognition from previous efforts is its capitalization on advances in two main areas: behavioral/neural science and computer science.
At the start of the 21st century, researchers have an unparalleled understanding of human brain functioning. The depth of this knowledge is due to the development of new neuroscientific techniques funded through significant investments from the National Institute’s of Health (NIH) and other agencies during the 1990’s — a period now referred to as the ‘Decade of the Brain’. This billion dollar investment in the fields of neuroscience, cognitive science and biopsychology resulted in some of the greatest advances in our understanding of the human biological system in the 20th century. For example, using techniques like functional magnetic resonance imaging (fMRI), scientists were able to identify discrete three-dimensional regions of the human brain active during specific mental tasks. This opened up the field of cognitive psychology substantially (into the new field of cognitive neuroscience) and enabled researchers to test their theories of the human mind and associate previously observed human thoughts and behaviors with neural activity in specific brain regions. Additional investments from the Department of Defense and other agencies in the 21st Century allowed researchers to develop even more advanced sensors that would eventually be used in augmented cognition systems. Novel types of neurophysiological signals that are measurable non-invasively include electrical signals (using electroencephalography and event related potentials) and local cortical changes in blood oxygenation (BOLD), blood volume and changes in optical scattering directly due to neuronal firing (using Near Infrared (NIR) light). Most of these signals, unlike fMRI, can be collected from portable measurement systems in real time making them potentially available for everyday use. All augmented cognition systems do not necessarily contain advanced neurophysiological sensors, but the field of augmented cognition is broadened even further by their inclusion.
As a result of the “Decade of the Brain” researchers have an increased knowledge of the cognitive limitations that human’s face. The HCI field focuses on “the design, implementation and evaluation of interactive systems in the context of a user’s task and work (Dix et al., 1998).” However, researchers in this field can only work with the data and observations easily accessible to them, i.e., how people overtly behave while using the interfaces. Through the efforts in neuroscience, biopsychology and cognitive neuroscience described above we can now locate and measure activity from the brain regions that are actively involved in day to day information processing tasks. Researchers will have a greater understanding of the cognitive resources that humans possess and how many of these resources are available during a computationally based task, whether or not their computational systems will include advanced sensors. Once these cognitive resources are identified and their activity (or load) measured, designers of computational interfaces can begin to account for these limitations (and perhaps adapt to their status) in the design of new HCI systems.
Finally, without the advances in computer science and engineering, none of the neuroscientific developments listed would be possible, and the field of augmented cognition would certainly not be feasible. Over the past forty years society has experienced leaps in computation and algorithmic prowess. This has been due in part to the miniaturization of transistors and other silicon based components so that more computational power is available per square inch of hardware (see: Moore’s Law). This has allowed computers to shrink in size, until they have permeated the very fabrics people wear and even their surrounding environments. Computer code itself has become smaller and more flexible, with the emergence of agent based computing, JAVA and internet services. Thus there have been two computing advances that augmented cognition has benefited from, improvements to raw computational resources (CPUs, physical memory) and improvements in the languages and algorithms that make adaptive interfaces possible. There are many fields that have benefited from these advances as well, which in turn have fed into the Augmented Cognition community, and they include, but are not limited to, user modeling, speech recognition, computer vision, graphical user interfaces, multimodal interfaces, and computer learning/AI.