Facilitating augmented learning using AI

ICDAR is a flagship conference series and gathering for researchers, scientists and practitioners in the document analysis community. It is endorsed by the International Association of Pattern Recognition.

“Increasingly, massive open online courses (MOOCs) and other learning portals are providing a variety of online educational resources to large audiences,” says Srirangaraj (Ranga) Setlur, principal research scientist at CUBS and co-advisor of the students. “Both of the winning papers address different approaches for using machine learning to make lecture videos more accessible for search and information retrieval by students.”

Xu received the Best Student Paper award for the overall conference. Her paper, entitled “Content Extraction from Lecture Videos via Speaker Action Classification based on Pose Information,” presents a novel method for summarizing the handwritten content from lecture videos. Instead of focusing on the handwriting itself, the method identifies speaker actions such as writing and erasing to extract key-frames from the video, ultimately extracting the content while using a smaller number of images than previous methods.

Kota’s paper, “Summarizing Lecture Videos by Key Handwritten Content Regions,” won the Best Paper Award at the Eighth International Workshop on Camera-Based Document Analysis and Recognition (CBDAR), held in conjunction with ICDAR. CBDAR is a satellite workshop that focuses on the analysis of camera captured documents and text and provides a forum for presenting up-to-date research, sharing experiences and stimulating discussions on future directions.

The paper presented a novel methodology for summarizing whiteboard lecture videos by extracting and presenting areas of key content. This research has the potential to help optimize search engines by allowing them to search info from lecture video content rather than just user provided tags.

Both projects were partially funded by the National Science Foundation as part of an ongoing $2.9 million grant from the Data Infrastructure Building Blocks (DIBBs) program. 

In addition to Xu, Kota and Setlur, others from the Department of Computer Science and Engineering at the conference were: Venu Govindaraju, SUNY Distinguished Professor, director of CUBS, UB’s vice president for research and economic development, and advisor of the two students; David Doermann, SUNY Empire Innovation professor and director of UB’s Artificial Intelligence Institute; and post-doctoral researcher Kenny Davila, who was also a co-author on the two papers.

“It was exciting to see more work in previously underemphasized areas such as infographics processing and extraction of knowledge from academic publications, blogs, educational videos and other multimedia sources at this year’s conference,” says Setlur. “A few years ago, in 2015, Venu had called for greater focus in these areas from the community during his IAPR/ICDAR Outstanding Achievements Award Keynote address.”

The CUBS group also organized the first ever competition on processing of CHART-Infographics at the conference, in collaboration with Adobe Research.

Setlur and Doermann were invited to serve as general chairs of the conference in the U.S. in 2023, along with Prem Natarajan of the University of Southern California. Govindaraju will be the Honorary Chair.

ICDAR took place in Sydney, Australia, from Sept. 20-25, 2019.

Govindaraju is the principal investigator on the NSF DIBBS program grant and Setlur is a co-investigator.