Virtual-Reality Movies Put a New Face on "User-Friendly"

BUFFALO, N.Y. -- A virtual-reality drama by University at Buffalo researchers -- aimed at transforming the movie-going experience -- is driving the development of increasingly "self-aware" computational agents that are able to improvise responses to the spontaneous actions of human users.

These improvisational computer agents are expected to influence the development of electronic devices of tomorrow, making them much more user-friendly because they will be able to respond to the idiosyncratic needs of each user.

The researchers' virtual-reality drama, The Trial The Trail, is a brand new type of dramatic entertainment, where instead of identifying with the protagonist, the audience becomes the protagonist.

The multidisciplinary team formed two years ago when Josephine Anstey, assistant professor in the Department of Media Study in the UB College of Arts and Sciences, was looking for ways to make VR dramas more believable. At the same time, Stuart C. Shapiro, Ph.D., professor in the Department of Computer Science and Engineering in the UB School of Engineering and Applied Sciences, was seeking applications to challenge the computerized cognitive agent called CASSIE that he and his colleagues had developed.

"We started thinking, 'What happens if you put a powerful artificial intelligence system -- which is what Stu has developed -- together with drama and stories?'" recalls Anstey. "The potential seems endless. You can get to the point where you have virtual characters that can believably respond to humans in real-time."

Instead of sitting in a darkened theater and passively watching a story unfold, human audience members in this interactive drama stand up, don gloves and headgear and become immersed virtually in the world of the characters on the screen.

Instead of using a joystick to compete against virtual characters as in a video game, the actions and utterances of human users determine how the virtual characters respond, based on an ever-growing "library" of actions and verbal communications with which the VR characters are endowed.

"Because of these attributes, we are creating a more intense psychological drama for users," said Anstey. "We are creating characters that are more similar to those you'd find in a novel."

The UB team fully explored these concepts in a paper presented last year entitled "Psycho-Drama in VR."

By necessity, added Shapiro, those characters are computational agents that must be capable of behaving in sophisticated and very human-like ways, attributes that also can help take "user-friendliness" for computers and other electronic devices to new heights.

"This is a step in the design and implementation of computer agents that are aware of themselves and their actions, as well as the environment they are in," he explained, "so this work is relevant to any application in which people interact with a device or system."

Shapiro referred to the team's approach toward its virtual characters as "cognitively realistic."

"We use a kind of computational 'self-perception' so that just the way that hearing people can pace their speech more effectively than deaf people, here's an agent that 'hears' computationally and can respond to what's happening," he explained. "The agent has some perception of itself and some level of self-awareness."

While other computer scientists are exploring multiple agent systems, he continued, this project is more demanding because the agents in the drama must be able to "perceive" themselves and then respond to the user.

So, as the human user proceeds through the drama, his or her actions are being recorded computationally over the Internet, interpreted psychologically and used to prompt the responses by the virtual characters.

In a sense, the computational agents in the drama must improvise around the human user, who is acting spontaneously without a script.

This, Anstey says, is a grand departure from the way that humans experience other types of computational dramas, such as video games.

"Very few computer games' characters have psychological lives that you can respond to," she says.

Because of this, the drama is different every time, a factor that the researchers say is both a more challenging and exciting type of entertainment, while also more computationally demanding.

In the UB drama, the main VR character is a dramatic representation of CASSIE, the cognitive agent, based on SNePS, Semantic Network Processing System, a knowledge-representation system developed over the past several decades by Shapiro and William Rapaport, Ph.D., UB associate professor of computer science, and scores of UB graduate students.

SNePS endows a computational agent with the ability to perform reasoning tasks, make inferences and do belief revision, where it can correct itself if it obtains additional information that indicates that it was misled initially.

"Cassie is one of the few intelligent agents that can process and communicate in natural language," said Shapiro. "You can talk to her in English."

He explained that one SNePS runs for each agent in the drama, and they are learning to communicate and interfere with one another.

"For computer-based fiction to really emerge, there has to be a lot of work using intelligent agents," said Anstey. "Anybody interested in the fiction and drama of the future can't ignore this. Writers will have to build these skills, will have to, in a way, think more like computer scientists."

In addition to Anstey and Shapiro, the UB team consists of David Pape, assistant professor in the Department of Media Study in the UB College of Arts and Sciences; Anthony Ekeh, a graduate student in computational linguistics; Michael Kandefer, a doctoral candidate, and Trupti Devdas Nayak, a graduate student, both in the Department of Computer Science and Engineering, and Orkan Telhan, a graduate student in media study.