The Video Annotation and Reference System (VARS) is a software interface and database system that provides tools for describing, cataloging, retrieving, and viewing the visual, descriptive, and associated data from MBARI’s deep-sea video archives.

MBARI uses high-resolution video equipment to record hundreds of remotely operated vehicle (ROV) dives per year. Since 1988, more than 30,000 hours of video have been archived, annotated, and maintained as a centralized institutional resource. This video library contains detailed observations of the biological, chemical, geological, and physical aspects of the Monterey Bay submarine canyon and other areas including the Arctic, the Antarctic, Taiwan, the Pacific Northwest, Northern California, Hawaii, and the Gulf of California.

Video is collected by cameras on MBARI ROVs and recorded to master (ProRes 422 HQ) and mezzanine (HEVC) video files in the shipboard control rooms. Using the VARS Annotation application, researchers take frame grabs and enter observations as annotations at sea during the dives. Annotations are created and constrained using concepts which have been entered into the knowledge database (VARS Knowledgebase) and approved by the knowledge administrator. When the ships return to shore, the video files, annotations, frame grabs, and ancillary data are transferred to shore through sync processes and are archived on a high-capacity, incrementally backed-up storage system. Video files are quality controlled and registered within VARS. Finally, ROV dives are annotated in more detail by MBARI Video Lab personnel, also using VARS. Alternatively, we have a workflow for using machine learning to generate annotations automatically using VARS-ML. Nightly processes merge video observations with ancillary data collected while the ROV is deployed so that for each video observation measured, physical data (latitude, longitude, depth, oxygen, etc.) is available and easily accessible. All of this information goes into the annotation database. The annotations and images are accessible through the VARS Query interface, a newly developed web application.

Publications that use VARS data

The VARS database provides invaluable qualitative and quantitative data that have been used in hundreds of peer-reviewed publications. Examples can be explored on the VARS Publications page.

VARS Knowledgebase

The core of the VARS system is a knowledge database – the VARS Knowledgebase – of over 4,400 biological, geological and technical terms used to describe deep-sea research conducted by the institute. This database is comprised of objects, or concepts, which are identifiable things that we observe — for example, species (Atolla wyvillei), geologic features (rock outcrop, gas vent, scarp), sampling devices and other equipment (suction sampler, larval incubation chamber), and marine debris (plastic bag, tire). The Knowledgebase also contains associations, which serve as descriptors — such as colors, sizes, behaviors, sample numbers — that can be associated with the objects being observed.

Hierarchical relationships between objects are preserved within the Knowledgebase allowing for consistent, rapid classification and description and complex querying of objects observed on video. For example, the hierarchical nature of the Knowledgebase allows a user to search for all descendants of a particular taxon that has been observed in MBARI’s video archives (e.g., all species of Sebastes rockfish). Additionally, the Knowledgebase allows for the easy display of exemplary images as well as the entry of descriptive information for each concept (e.g.,species geographic and depth ranges, references, physical characteristics); thus, the Knowledgebase also serves as a web-based marine identification reference guide, which we call the MBARI Deep-Sea Guide.

VARS Annotation

The VARS Annotation application references the VARS Knowledgebase and allows researchers on MBARI ships or onshore to make annotations of observations from ROV video, AUVs, and still images from MBARI’s growing fleet of vehicles. Annotation entries are constrained by concept names in the structured Knowledgebase and have been reviewed and approved by an administrator. Associations that enhance video observations can be added and include such things as shape, position, color, behavior, sample number, etc.

VARS Query

The VARS Query application, which also references the Knowledgebase, allows for easy searching of the observations stored in the video annotation database. Complex queries can be made by constraining for concept and/or temporal, spatial, and physical parameters (for example: species, season, location, or depth). Query results are delivered as a table, on a georeferenced map view, and include the concept, associations, image links, direct access to the video sequence, environmental data, dive information, and other available metadata. The query results can be saved as tabular text data with an option to download affiliated images (where they exist).

Data characteristics disclaimer

Though they have proven to have great inherent and inferred scientific value, these data have not always been collected using procedures which render them consistent or quantitative. Biases have been introduced by observation techniques, camera settings, mission planning, and changing science terminology, among other factors. In no way should these data be viewed or analyzed as random, structured, or systematic observations. As with all databases, understanding how data were collected over time is necessary prior to using them in any analysis. For more information, please reference our Data Use Policy page or contact our team.

Contacts

For help accessing or using the VARS applications, email MBARI’s Video Lab team.

For reporting bugs or technical issues with VARS applications, email Brian Schlining (VARS software engineer).

To request or share questions regarding images, video, and related data for research and educational purposes, email Nancy Jacobsen Stout (Video Lab manager).