Annotating Video for Research Use
The first view of a vampire squid on high definition video can take your breath away. Fluttering fins propel the creature slowly through the midwater as its big blue eyes seem to fix on the camera. MBARI researchers have made many new discoveries while watching those breathtaking videos of deep-sea animals--discoveries that could never have occurred by studying pickled specimens from more traditional collecting methods. Video provides a different way of seeing—a moving record of the behavior and interactions of living creatures in their habitats, as well as details of color, pattern, size, and reproduction.
Even before ROV Ventana’s first dive in 1988, MBARI scientists recognized the value of preserving the visual record of our deep-sea dives. Early on, biologist Bruce Robison engaged MBARI engineers in planning an institutional video archive and processing the video as scientific data. Robison recalled, “With a brand new research institute, we wanted a user-friendly system for video in place before we started collecting data.” Engineer Dan Davis wrote software to link video time codes to other data being collected simultaneously by the ROV. In 1988, MBARI hired its first video lab technician to review all the video tapes and annotate which animals were sighted. But the data deluge became greater than one person could handle.
That original annotation process was very labor intensive and required typing in the name of each animal or object without a consistent list of organisms or any automated way to check spelling. As additional scientists began to use the ROV, the annotation became more extensive and detailed, incorporating descriptions of geological features on the seafloor and experimental equipment deployed by the ROV. But long hours of watching video and eye fatigue sometimes led to errors in spelling or misidentifications that reduced the value of the data.
Over the years, scientists and engineers inside and outside of the institute came up with various schemes and prototype systems for cataloguing MBARI's growing video archives. Nancy Jacobsen Stout and Brian Schlining led the effort to create the current system, known as the Video Annotation and Reference System (VARS). Their goal was to provide a robust, intuitive interface that would be distributed as open source code for further development and worldwide research use.
VARS consists of three interrelated software applications. The first application, the knowledge base, contains a list of organisms, geological features, tools and instruments with standardized spellings and definitions. This helps eliminate the old problem where a misspelled name made it impossible to find all the examples of a particular animal. The second application, an annotation program, allows a viewer of the video to quickly name and describe what is seen using names from the knowledge base. Each user can customize the program with buttons for names regularly used to describe a particular habitat. The person annotating can also use the program to take still images (frame grabs) from the video. The resulting annotations and frame grabs are stored in a database.
The third application allows users to search for items in the database of annotations. For example, a researcher can select “Vampyroteuthis” from a pull-down menu on the query GUI and within seconds review a spreadsheet of references to the vampire squid in the database, the detailed descriptions of each sighting, as well as information such as date, latitude, longitude, depth, salinity, temperature, light and oxygen levels of the water, and links to frame grab images.
In 2004, Schlining, the lead engineer on the VARS project, installed the new annotation application in MBARI’s video lab where Linda Kuhnz, Lonny Lundsten, Kyra Schlining, and Susan von Thun vetted it as they annotated ROV dive video tapes. After many iterations of intensive testing and correction of discovered “bugs” in the software, the annotation application was installed on the ships Point Lobos and Western Flyer for researchers to use during ROV dives.
The VARS system soon proved its worth on land and at sea. In 2005, with the system’s successful transition to regular research use at MBARI, the team stepped up plans to make it available to other institutions as open source code. Researchers came to MBARI from other institutions to learn how to install and use VARS to catalogue and analyze of their own video data. “It was challenging to document the procedures and prepare for the workshop, and then make sure all the workshop participants had the appropriate technical background and equipment,” Schlining said.
As a result of the dedicated teamwork from staff across the institute, we now have a functional annotation and reference system to create and access meaningful and accurate descriptions of video images for diverse research needs. The productivity of scientists and technicians both at sea and in the lab has improved. The ability to analyze video qualitatively and quantitatively has resulted in hundreds of publications, presentations, and educational programs based on MBARI video observations. Video laboratory supervisor, Nancy Jacobsen Stout acknowledges, “The response to VARS has been very promising and we now hope that the software system and the accompanying rich data set will be instrumental in preserving the scientific record of our deep-ocean environments.”
MBARI contributors to the Video Annotation project: Doug Au, Jim Barry, Holly Baum, Dave Caress, Lori Chaney, Andrew Chase, Neil Conner, Judith Connor, Dan Davis, Craig Dawe, Judy Donaldson, Duane Edgington, Kevin Gomes, John Graybeal, Bruce Gritton, Gene Guglielmo, Steve Haddock, Rich Henthorn, Linda Kuhnz, Steve Lowder, Lonny Lundsten, Mike McCann, Charles Paull, Jenny Paduan, Kim Reisenbichler, Bruce Robison, Paul Rogers, Todd Ruston, Brian Schlining, Kyra Schlining, Rob Sherlock, Rich Shramm, Nancy Jacobsen Stout, Susan von Thun, Kristine Walz, and Daniel Wilken.
- MBARI Video Annotation and Reference System (VARS) web page