VIF2staging

"VIF2staging" is the first step of the process that creates the staging database. It is a perl script that extracts sample collection event information from video annotation files, merges them with navigation, frame-grabs, and other data, and creates a staging database. VIF2staging is designed to work on board the R/V Point Lobos (on either machine lobos or seaspray) after a dive has been completed, the VICKI annotation process finished, and a VIF file created. It has also been adjusted to work on shore on the video lab computer calypso after post-dive re-annotation.

User instructions for creating staging database:
» VIF2staging may be run on lobos or seaspray. This step occurs as part of the shipboard process. The staging database HTML file automatically will be written to host ariel. Log in as user 'vicki' and type:

VIF2staging <filename>
where <filename> is the name of the VIF file (eg., 1998023PV1.vif).

» VIF2staging may be run on host calypso as user 'vicki' by typing:

VIF2staging <filename>
where <filename> is the name of the VIF file (eg., 1998023PV1.vif).

The microwave connection to the ship must be good because the HTML files get written to host ariel. The VIF file must be in the same directory from which the command is typed.

» VIF2staging also may be run on calypso and the HTML file written on shore. This is recommended for post-dive processing so that the ship is not involved. In this example, the HTML file is written to the Sample Coordinator's internal web site (requires login as user 'paje') by typing:

/u/vicki/bin/VIF2staging <filename> /u/paje/mbari_html/stage/
Find the HTML file at http://mww.mbari.org/~paje/stage/.

The ship navigation file is read, and for each UTC time found in the previous step the ROV latitude, longitude, and depth is extracted. For each of these lookups, a check is made for temporal proximity. If any of the results are more than 10 minutes away from the sample time then a warning is indicated in the staging database web page. Depth was the variable reported and archived before January 1998, but pressure is reported subsequently. For dives after January 1998, pressure is converted to depth by the routine "sub calcDepth", which is presented in the appendix.

Finally, all of the relevant information (database table field/value pairs) that needs to be loaded into the Samples Database is compiled into an HTML form, the staging database, viewable inside a web browser by anyone on our Intranet. The action for the form is an Active Server Page (loadcoll.asp) that loads the data into the production Samples Database; it is restricted to the Sample Coordinator.

VIF2staging structure
The following diagram shows the calling hierarchy of the subroutines within the VIF2staging perl script:

Routines are called left-to-right and top down. Basically, the command line is processed, lookup files are read, annotation blocks are parsed with sample strings constructed for each sample, and finally the HTML file is written.

Routines in VIF2staging (brief description; see the source code for full details).

parse_cmd_ln()

Processes the command line arguments.

Takes the command line argument of a VIF file name. Accepts optional argument of an HTML directory.
Returns the basename of the VIF filename. The year, year-day part of this is used to identify and locate ancillary data files.

parse_ANNOTATE()

This function is specific to looking for information in the VIF file that pertains to samples.
It collects information between ANNOTATE blocks and saves it for the staging database (HTML file) before begining the next ANNOTATE block.
It creates a global variable hash of sample events keyed by Tape timecode.

Write_html()

Creates form for loading selected samples into database.
Uses CGI.pm methods for writing the HTML.
The data to be loaded into the database by loadcoll.asp is encoded into the HTML form as hidden variables. For each row in the staging database table there is a pipe ('|') sparated list of values that correspond to the field values of the CollectionEvent table. The CollectionEvent column names are also passed as a hidden variable named DBCOLS inside the form. Important: if the CollectionEvent table design is changed then @CollectionEventCols list in VIF2staging needs to be edited to reflect this. The reason it was done this way was so that loadcoll.asp could be simpler in that it it had everything it needed to construct the SQL insert statements: the column names and the values.

read_timecodelookup()

Logic is in here to check first for videologr file, which we started making on 1998120, then for RS20 Best file, then lastly for the backup .tc file that lives in current dir or in the framegrab directory on lepas. Much of this special handling was for old VIF file processing. All future processing will rely on the videologr files.

The videologr file is first looked for in /users/pllogger/data as either compressed or uncompressed. If not found then it looks for the compressed file in the shore-side archive location: "${ARCHIVEdir}/logger/$yyyy/$ship/${yyyyddd}videologr.dat.gz", where ${ARCHIVEdir} is '/hosts/barnacle/ARCHIVE'.

VIF2staging will die if a timecode lookup file cannot be found.

The tape timecode / epoch seconds lookup information is read into @Esecs and @timecode lists for use bu findUTC() in actually doing the lookups.

read_shipnavlogr()

Read the realtime ship nav logger file. (These data are now logged in videologr files.)

read_archivenav()

Read the Archived ROV nav file file. (These data are now logged in videologr files.)

ConstructSampleString()

Build pipe separated string to be passed to load step as hidden variable.

findUTC()

Look up epoch seconds from the @Esecs and @timecode lists (array). Do binary search on Timecode list to find record of closest time. Will do linear intrepolation if binary search tolerance of 6 seconds is not met.

findLocation()

Look up position from logged ship nav file. This depends on subroutine read_shipnavlogr() being called beforehand

image_cell()

Construct HTML for Image cell.

time_cell()

Construct HTML for Time cell.

location_cell()

Construct HTML for Location cell.

vickivims_cell()

Construct HTML for VICKI/VIMS cell.

comment_cell()

Construct HTML for Comment cell.

test_url()

Use Lynx to test for valid URL.

calcDepth()

Use Saunders & Fofonoff, DSR, 23, 109-111 (1976) method to calculate depth from pressure for a particular latitude. It passes in pressure and latitude from the ROV navigation file.

»Next: loadcoll.asp

Last updated: 10 February, 2005 by Jenny Paduan
Copyright © 1998 MBARI