A core suite of AUV science data processing software was developed in MATLAB for the August 2000 MUSE experiment during which two Odyssey IIb
AUVs, with different instrument configurations, were operated over a 12-day period. This software has been further developed since the MUSE experiment. Building on code developed for engineering review of AUV missions, this AUV science data processing software provides a basis for assembly and examination of AUV science observations:
- at sea to monitor data quality, direct maintenance of the instrument suite, and map environmental structure to guide deployment of other sampling resources
- post-cruise to optimize data quality and alignment, develop data presentation, and export processed data to other formats for greater accesibility (data files in ascii and netcdf; data representation in images).
AUV science data processing presents variable requirements that depend on the navigation system, the science payload characteristics, and the data acquisition software architecture. However, much of the AUV deployment strategy (e.g. series of missions that make up a survey) and many of the basic processing needs (calculation of derived variables, instrument alignment, mission concatenation into surveys, gridding and mapping) will remain the same. The AUV science data processing software is collected in a toolbox called auvsdp. This toolbox is intended to be updated and customized with changing AUV data processing needs.
John Ryan, MBARI, June 2001
Questions? Bug reports? firstname.lastname@example.org (831) 775-1978
Processing Steps for a typical survey (series of missions)
Processing occurs in three stages:
- Individual mission processing: This stage includes binning the navigation and science data to a common reference frame, calculating derived variables, and aligning instruments. Note that the lack of perfect constancy in instrument sampling frequency requires that all data first be binned to a fixed, constant-frequency reference frame (wrongly assuming a constant sampling frequency in the raw data for instrument alignment will result in cumulative error in the alignment).
- Multiple mission splicing: This stage includes ajdoining sequential missions into a mission set. Each set of missions is defined based on the science goal, e.g. a transect across a drifter track made in 4 consecutive 1-hour AUV missions. In the MUSE 2000 Odyssey AUV data set, error in geographic referencing comes from imprecision in individual mission [X,Y] endpoints (due to error in manual logging of the data necessary to compute AUV position) and from the assumption that the vehicle trajectory between the known start and end points was linear and at a constant horizontal rate. In the MUSE AUV data, these sources of error do not interfere with examination of environmental structure. However, georeferencing error for available navigation methods should be understood and minimized.
- Survey (mission set) plotting and export: This stage includes the gridding and plotting of [X,Y,Z] data into sections or volumes. This stage is highly dependent upon the sampling characteristics. To grid multiple-mission survey data into a vertical section, the approximation is made that the starting point of each consecutive mission is very close to the ending point of the previous mission. Although this introduces spatial bias in the mapped fields, for all daytime MUSE AUV data, the error again does not interfere with meaningful examination of environmental structure at the scales of interest. In this stage, the data can be exported to more widely accessible formats other than the matlab format in which processing occurs (ascii or netcdf data files; gif/jpeg image data representations).
More specific processing details follow, and Table 1 summarizes toolbox components:
- Setup: Modify auvpath.m according to your directory structure and place it in your standard matlab path. This program must be run from in MATLAB before each session of AUV science data processing. It defines the path to the essential toolboxes. It may also be an advantage to save the mission data as matlab files (if in the m-file format used for Odyssey).
- Process each mission separately:
- Copy the missiondata.m file to the mission data directory, and edit the mission endpoint navigation data. For the Odyssey IIb operations, endpoint navigation was from [ship position, ship heading, AUV bearing, and AUV range], and there were instances where incomplete recording of these data resulted in poor position fixes for individual missions. This method of defining X, Y georeferencing is specific to these operations, and your approach will depend on the navigation setup for the AUV you are working with.
- Odyssey IIb data were logged as ascii dumps bracketed in a small amount of matlab text to generate a data matrix. That is, the data were stored within an m-file that was run to load the data. These take very long to load, and it is useful to first save all data as MATLAB files for subsequent processing. This m-file ascii standard will probably change, but if your data are logged as an ascii dump m-file, run save_as_mat (after you have configured it to include the standard m-file suite, e.g. navigation, ctd, etc.).
- Modify process_auvmission.m as appropriate to your instrument suite and time lags for alignment, AUV navigation, and binning characteristics. From within the directory containing the data files, run process_auvmission.m. This will bin navigation and science data to a common time frame, align instruments, and provide a GUI for interactive data examination and quality control. Note that data cropping should retain the full X-Z domain sampled between the endpoints. Typically the salinity (and hence oxygen) during the early part of the AUV mission is bad (bubbles?) and must be separately cropped (don't crop all variables because a couple are noisy at the start). When complete, it will save the processed mission data to the current mission directory as mission_sd.mat.
- Concatenate contiguous (or approximately contiguous) missions of the survey, grid, plot, and export as appropriate.
- To begin mission concatenation into a survey, copy a template survey_assembly_setup.m file to the top level directory of that survey (above individual mission directories), and modify appropriately.
- Run assemble_survey from that same directory to produce a survey data file.
- For export of the assembled survey data, use xport_auv_survey.
- If the mission is a vertical section, you can grid and plot it using auv_section. Note that other gridding methods may be substituted (current code used griddata function of matlab that is based on Delaunay triangulation). The plotted vertical section can be exported with screen2jpeg.m
Table 1: auvsdp toolbox file listing
|PROCESSING SETUP |
|auvpath.m|| Define directory paths for AUV science data processing. |
|save_as_mat.m|| Save m-file format ascii data as matlab files. |
|INDIVIDUAL MISSION PROCESSING |
|process_auvmission.m|| Load navigation and science data, bin to common reference time frame, calculate derived variables, align variables according to lags (plumbing and instrument response), GUI-driven interactive data examination and quality control, save processed data. In addition to programs from the review and oceanographic toolboxes, this code calls numerous other m-files from the auvsdp toolbox (below).|
|missiondata.m|| Define mission endpoint navigation. This calls the program bearing.m. |
|bearing.m|| Calculates AUV position from ship position, ship heading, AUV bearing, and AUV range.|
|get_spatial.m|| Obtain X,Y,Z,T references (different) from surface and in-water navigation files. |
|bin_mission.m|| Bins navigation and science data to N-second averages. Currently uses median to reduce instrument noise.|
|calc_O2.m|| Calculate oxygen from 2 voltages of SeaBird Beckman-type oxygen sensor. With alignment for plumbing and instrument response lag, this produces nice oxygen fields, but note that a better oxygen sensor is now available from SeaBird.|
|O2_calibration.m|| Define sensor-specific oxygen sensor calibration.|
|calc_pH.m|| Calculate pH from the SeaBird SBE18 pH sensor voltages.|
|plot_all.m|| Plots all variables during interactive GUI quality control. |
|check_mission.m|| Grid and examine individual mission. |
|plot_auvpath.m || To check the AUV path after all individual missions have been processed.|
|SURVEY ASSEMBLY, GRIDDING, PLOTTING AND DATA/GRAPHICS EXPORT|
|survey_assembly_setup.m|| To set mission list and file names for survey definition. Modify and place in each survey directory (above the constituent mission directories) prior to using assemble_survey.|
|assemble_survey.m|| To concatenate consecutive missions (already processed with process_auvmission.m tree) into a contiguous survey. X-distance is computed for each mission (segment). For concatenation of missions and subsequent gridding to a section (auvsection.m), X-distance is assumed contiguous although there are small spatial offsets between consecutive missions.|
|xport_auv_survey.m|| To write processed matlab format data to ascii or netcdf.|
|auv_section.m|| To grid and map a vertical-section survey. Output to jpeg is standard.|
|Graphics- and mapping- related code |
| giffig(_wide).m || To set up a standard sized plotting window appropriate to monitor. |
| frame.m || Frame the current figure window for text placement. |
| llgrid.m || Set map axis tic marks and labels. |
| wclandfill.m || Mask land for mapping AUV X,Y path.|
| plot_iso.m|| Plot isobaths (shelf & canyon definition relative to survey).|
| screen2jpeg.m || Output current figure window to jpeg image. |
| mf.m || Multi-option 2-D median filtering for gridded fields. |
| mvavgmean.m || Moving average mean for smoothing depth rate to identify up and down cast; could be used in place of 1-D median filtering of variables via process_auvmission.m. |
| plan_yoyo.m || A simple tool for defining sampling times and horizontal resolutions for AUV surveys, as dependent upon vehicle velocity, pitch and sampling depth range.|
| myncpath.m || Modify and use to define the path to your MATLAB netcdf toolbox if you are gridding and mapping from the netcdf AUV survey data file. |
Toolboxes required in addition to auvsdp :
Brian Schlining wrote the code to convert processed survey data to netcdf, using standards to make the data widely accessible. The
DODS server installed at MBARI can easily serve these data in netcdf format. For browsing the data in this format, there is a free Java netcdf browse utility available online.