The Video Annotation and Reference System (VARS) is a software interface and database system that provides tools for describing, cataloging, retrieving, and viewing the visual, descriptive, and associated data from MBARI’s deep-sea video archives.

MBARI uses high-resolution video equipment to record hundreds of remotely operated vehicle (ROV) dives per year. Since 1988, more than 30,000 hours of videotape have been archived, annotated, and maintained as a centralized institutional resource. This video library contains detailed observations of the biological, chemical, geological, and physical aspects of the Monterey Bay submarine canyon and other areas including the Arctic, Taiwan, Antarctic, Pacific Northwest, Northern California, Hawaii, and the Gulf of California.

Video is collected by cameras on MBARI ROVs and recorded to master (ProRes 422 HQ) and mezzanine (HEVC) video files in the shipboard control rooms. Using the VARS annotation application, researchers take frame grabs and enter observations as annotations at sea during the dives. Annotations are created and constrained using concepts which have been entered into the knowledge database (knowledgebase) and approved by the knowledge administrator. When the ships return to shore, the video files, annotations, frame grabs, and ancillary data are transferred to shore through sync processes and are archived on a high capacity, incrementally backed-up storage system. Video files are quality controlled and registered within VARS. Finally, ROV dives are annotated in more detail by MBARI Video Lab personnel, also using VARS. Alternatively, we have a workflow for using machine learning to generate annotations automatically using VARS-ML. Nightly processes merge video observations with ancillary data collected while the ROV is deployed so that for each video observation measured, physical data (latitude, longitude, depth, oxygen, etc.) is available and easily accessible. All of this information goes into the annotation database. The annotations and images are accessible through the VARS query interface, a newly developed web application.

Publications that use VARS data

The VARS database provides invaluable qualitative and quantitative data that have been used in hundreds of peer-reviewed publications. Some examples can be viewed here: VARS Publications.

VARS Knowledge database

The core of the VARS system is a knowledge database (knowledgebase) of over 4,400 biological, geological and technical terms used to describe deep-sea research conducted by the institute. This database is comprised of objects, or concepts, which are identifiable things that we observe — for example, species (Atolla wyvillei), geologic features (rock outcrop, gas vent, scarp), sampling devices and other equipment (suction sampler, larval incubation chamber), and marine debris (plastic bag, tire). The knowledgebase also contains associations, which serve as descriptors — such as colors, sizes, behaviors, sample numbers — that can be associated with the objects being observed.

Hierarchical relationships between objects are preserved within the knowledgebase allowing for consistent, rapid classification and description and complex querying of objects observed on video. For example, the hierarchical nature of the knowledgebase allows a user to search for all descendants of a particular taxon that has been observed in MBARI’s video archives (e.g., all species of Sebastes (rockfish)). Additionally, the knowledgebase allows for the easy display of exemplary images as well as the entry of descriptive information for each concept (species geographic and depth ranges, references, physical characteristics), thus the knowledgebase also serves as a marine identification reference guide available via the web which we call the Deep-Sea Guide.

VARS Annotation application

The annotation application references the knowledgebase and allows researchers on MBARI ships and at MBARI to make annotations of observations from ROV video, AUVs, and still images from MBARI’s growing fleet of vehicles. VARS annotation entries are constrained by the knowledgebase such that only concepts that have been entered into the knowledgebase and approved by the knowledgebase manager can be entered into the database. Associations that enhance video observations can be added and include such things as shape, position, color, behavior, sample number, etc.

VARS Query application

The VARS Query allows for easy searching of the observations stored in MBARI’s video annotation database. Complex queries can be made by constraining for concept, temporal, spatial, or physical parameters (for example: species, season, location, or depth). The VARS Query references the knowledgebase. Query results are delivered as a table and include the concept and associations, image links, physical and ancillary data, dive information, and tape information. The query results can be saved as tabular text data, Google Keyhole Markup Language (for geospatial viewing), or as images (where they exist).

Data characteristics disclaimer

Though they have proven to have great inherent and inferred scientific value, these data have not always been collected using procedures which render them consistent or quantitative. Biases have been introduced by observation techniques, camera settings, mission planning, and changing science terminology, among other factors. In no way should these data be viewed or analyzed as random, structured, or systematic observations. As with all databases, understanding how data were collected over time is necessary prior to using them in any analysis.

Contacts

For help accessing or using the VARS applications, email MBARI’s Video Lab team.

For reporting bugs or technical issues with VARS applications, email Brian Schlining (VARS software engineer).

To request or share questions regarding images, video, and related data for research and educational purposes, email Nancy Jacobsen Stout (Video Lab manager).