What's New in MB-System Version 5.3
The version 5.3.x releases of MB-System includes a number of changes and
improvements relative to the earlier version 5 releases. Two of these changes, a new "fast bathymetry" or fbt file format, and the implementation of file locking, will generate files that older versions of MB-System cannot read. Consequently, we recommend that any MB-System users that work together on common datasets upgrade simultaneously to release 5.3.1887 or higher. The particulars follow:
- Changes to the "fbt" or "fast bathymetry" files in the MB-System processing environment.
- We updated the definition of swath format MBF_LDEOIH (format 71),
which is used for the "fast bathymetry" or *.fbt files in MB-System
- The previous form of *.fbt files had a
serious limitation in that bathymetry from multibeams operated
near the seafloor in deep water did not represent the full
numerical resolution of those data.
- The updated format allows for
depth and distance resolution to 0.001 m even in the deep ocean.
- Old *.fbt files are read transparently, but newly written files
will not be readable by older versions of MB-System..
- File locking by data editing tools MBedit, MBeditvia, MBnavedit, MBclean, and the processing program MBprocess.
- A file locking mechanism has been implemented to allow multiple users to work on the same projects without interfering. The intent is for this mechansim to work on heterogeneous networks, which means that the data can be on non-Posix filesystems mounted on multiple computers running different operating systems.
- The file locking is implemented in a crude fashion involving the creation and deletion of "lock files". These are text files with a ".lck" suffix that are created in parallel with the associated "raw" swath files. If a lock file exists, the swath file has been locked by a program and cannot be opened for editing or processing by any other program. When a program completes its work on or with a swath file, it removes the lock by deleting the file.
- The lock files include text indicating the program that generated the lock, the time the lock was created, the purpose (e.g. bathymetry editing), which user generated the lock, and what machine that user was logged into.
- File locking is implemented for MBedit, MBeditviz, MBnavedit, and MBprocess.
- An example of the contents of a lock file is:
# File /Volumes/MappingAUVOps2011/20110525m1/20110525_202216.mb88
# Locked by user <caress> on cpu <diebold.shore.mbari.org> at <Fri May 27 09:45:42 2011>
Locking Program: MBedit
Locking User: caress
Locking CPU: diebold.shore.mbari.org
Locking Time: Fri May 27 09:45:42 2011
Locking Purpose ID: 2
Locking Purpose Description: Edit Bathymetry
- The primary negative consequence of file locking is the potential creation of orphaned lock files if a locking program crashes or is interrupted. The mbdatalist tool can now be used to detect lock files (-S option) in any of the files references through a datalist, or to remove any lock files (-Y option).
- MBedit and MBnavedit can now operate on datalists.
- Previously, these two editing tools opened one file at a time, specified from a file opening dialog accessed by clicking the <File> button.
- Now, <File> is a pull down menu with two options: <Open> and <File Selection List>. The first brings up the same file opening dialog as before. The second brings up a list dialog showing all of the files available for editing.
- If a user opens a single file for editing, that file will be added to an internal list of files available for editing, and then loaded.
- If a user opens a datalist, then all of the files referenced through the recursive datalist structure will be added to the internal list of files available for editing, and the first file will be loaded.
- The user can, by selecting the <File->File Selection List> menu item, display the internal list of available files. Selecting a file in this list will cause that file to be loaded for editing. If another file was already loaded, it will be closed out gracefully before the new file is loaded.
- The list of available files also indicates which files have been previously edited (so that ".esf" or ".nve" files exist) and which are currently locked by other programs and users.
- Pseudo-Parallel Processing With MBprocess
- Updating the processing of one or more surveys with MBprocess can be time consuming because each swath file must be processed in turn.
- Since the processing of each file depends only on the parameter and ancilliary files parallel to that file, the use of MBprocess is inherently parallelizable.
- The file locking mechanism described above allows users to simultaneously run multiple instances of MBprocess on the same datalist structure.
- Each MBprocess instance will parse through the datalist structure and attempt to process each swath file in turn.
- Swath files will be skipped if they are up-to-date, or if they are locked.
- The MBprocess instances can be on different computers, as long as the same filesystems are mounted.
- The benefit of this pseudo-parallel processing tends to be limited by the network throughput. For instance, at MBARI the swath data are served on CIFS or Samba filesystems over Gigabet ethernet. We find that processing runs go faster with up to six MBprocess instances running on up to three different computers, but that adding more than six processes causes the entire run to take longer.
- New data formats:
- HYSWEEP HSX multibeam data format (format id 201)
What Was New in MB-System Versions 5.1 and 5.2
The version 5.1.x and 5.2.z releases of MB-System included a number of changes and
improvements relative to the version 4 releases. The most significant changes
- A new approach to managing data processing.
- Many tools - one output file. In previous versions of MB-System, each
processing program read an input swath data file and produced an output
swath data file. This "serial" processing scheme generally produced
a large number of intermediate data files. MB-System version 5.0 features
the integration of the editing and analysis tools with a single program,
mbprocess, that outputs processed data files. The new "parallel"
processing scheme covers bathymetry data processing, but does not yet incorporate
the sidescan processing capabilities. All of the old tools and capabilities
are still part of the distribution.
- Recursive datalists. The lists of data files used by gridding and plotting
programs can now be recursive, making it simpler to manage data from many
- Automatic format identification. MB-System programs will now attempt
to automatically identify the swath data format based on the filename suffix.
- Extended inf files. Users can generate inf files by directing the output
of mbinfo to a file named by adding an ".inf" suffix to the swath
data file name. Several programs can parse inf files, if they exist, to
quickly obtain data locations or ranges. This feature speeds operations
such as gridding, mosaicing, and automated plotting.
- New command line tools.
- MBprocess. This new tool performs a variety of processing tasks and
produces a single output processed swath data file. The program mbprocess
can apply bathymetry edits from mbedit and mbclean, navigation edits from
mbnavedit, sound velocity profile changes from mbvelocitytool, and a variety
of other corrections.
- MBset. This new tool allows users to create and modify the parameter
files used to control the operation of mbprocess.
- MBdatalist. This new tool allows users to list the files referenced by
a recursive datalist structure. It can also be used to create the ancillary ".inf",
".fbt", and ".fnv" files for all of the data files referenced in a recursive datalist
- MBsvplist. This new tool lists water sound velocity profiles embedded
in swath data files, creating secondary files that
can be read into MBvelocitytool.
- MBareaclean. This new tool identifies and flags artifacts in swath sonar
bathymetry data within a specified area of interest. The
area is divided into a grid with square cells or bins, and the
data are grouped according to these bins. Once all
of data are read, statistical tests are applied
to the soundings within each bin.
- MBotps. This new tool uses the Oregon State Tidal Prediction Software (OTPSnc) package to calculate open ocean tidal models for bathymetry correction.
- MBextractsegy. This new tool extracts subbottom profiler data
from swath files to SEGY format files.
- MBsegyinfo. This new tool extracts SEGY data file information and statistics.
- MBsegylist. This new tool produces arbitrary ascii tables from SEGY data files.
- MBsegygrid. This new tool grids seismic and subbottom data from SEGY data files.
- MBsegypsd. This new tool calculates sonograms from SEGY data files.
- Improved bathymetry and navigation editors.
- MBedit and MBnavedit now swallow data files whole rather than reading
in limited size buffers.
- MBedit now outputs beam edit events rather than an entire swath file.
The edits are applied by MBprocess.
- MBnavedit now outputs the edited navigation rather than an entire swath
file. The edited navigation is merged using MBprocess.
- Both editors show the position of the currently displayed data within
the entire data file.
- MBnavedit has two navigation modeling modes relevant to swath data
collected using poorly navigated ROVs and towfishes. One mode applies a
dead reckoning model with interactively set drifts, and the other involves
inverting for an optimally smooth navigation by penalizing speed and acceleration.
- MBnavadjust. This new tool allows users to adjust poorly navigated
surveys by matching features in overlapping swathes. It is particularly
useful for processing surveys conducted from submerged platforms.
- New Visualization Based Tools
- MBgrdviz is a GMT grid 2D/3D visualization utility. MBgrdviz also allows the display of sonar navigation, sites, and routes, and interactive survey planning.
- MBeditviz is an interactive 3D visualization bathymetry editor and patch test tool.
- Support for Projected Coordinate Systems
- MB-System now incorporates the source code for the
PROJ.4 Cartographic Projections library, providing
support for (apparently) all commonly used geodetic
coordinate systems. PROJ.4 was developed by Gerald
Evenden (then of the USGS), and was obtained from the
- A large number of commonly used projected coordinate
systems (e.g. UTM) are defined in a file
(mbsystem/share/projections.dat) distributed with MB-System.
These include all of the standard UTM zones, all of
the standard state plate coordinate systems, and most
of the European Petroleum Survey Group (EPSG) coordinate
systems (also including UTM).
- MB-System can now handle swath data that is navigated
in a supported projected coordinate system. In particular,
data files that are navigated with UTM eastings
and northings instead of longitude and latitude can
now be plotted and processed with MB-System.
- The programs mbgrid and mbmosaic can now output grids
and mosaics in any of the projected coordinate systems
specified in mbsystem/share/projections.dat.
- The TIFF images generated with mbm_grdtiff and
mbgrdtiff now fully conform to the GeoTIFF standard,
providing that the source grids or mosaics were generated
using mbgrid or mbmosaic in either Geographic
coordinates, UTM coordinates, or any of the EPSG coordinate
systems specified in the projections.dat file.
This means, for instance, that GeoTIFF images generated
with mbgrdtiff will be properly georeferenced
when they are imported into ESRI ArcGIS or other GIS
- Restructuring the code.
- All of the C code now conforms to the ANSI C standard.
- The underlying input/output library (MBIO) has been substantially rewritten.
The structure has been streamlined, simplifying both future development
and support of the existing code. The MBIO API has been greatly modified.
- Handling of old Simrad multibeam data.
- Vendor format data from the old Simrad multibeams (pre-1997 sonars)
are now supported by a single format id (51) rather than a separate format
id for each sonar model. The old format id's are automatically aliased
to 51, so existing shellscripts will continue to work.
- MB-System no longer supports beam flagging in format 51 data. The
use of mbedit and mbclean on format 51 data will cause the flagged
beams to be irrevocably nulled. Previous versions of MB-System used
the highest bit in the depth values to represent beam flags because
no Simrad data seemed to use that bit. We have not obtained data
with depth values using the full bit-range, conflicting with the
old beam flagging scheme. We recommend that old Simrad data be translated
to the extended processing format (57) which contains proper beam
flags and supports all processing functions. Format 57 is also used for processing data from
the current Simrad multibeam sonars.
- Sidescan data from old Simrad multibeams (pre-1997 sonars) are now
handled in the same manner as data from the newer sonars (e.g. EM3000,
EM3000, EM120). The raw samples in the vendor data format are binned, averaged,
and interpolated into a 1024 pixel sidescan swath. This binned sidescan
is not saved in the vendor format, so (as above) it is recommended that the data be
translated to an extended format (57) that stores both bathymetry beam flags
and processed sidescan.
- Streamlining of MB-System Default Parameters.
- Prior to version 5.0, the MB-System defaults
set by mbdefaults included the format id, a control for
ping averaging, longitude and latitude bounds for windowing
by area, and begin and end times for windowing in time. These
values are no longer set in the .mbio_defaults file or controlled
by mbdefaults. As noted above, the format id is automatically
identified from the filename when possible. When filenames do not
match one of the recognized structures, users must specify the
format using the relevant programs -Fformat option.
The controls for ping averaging and windowing in time and space
are rarely used, and must now be explicitly set in command
- New Data Formats
- Furuno HS10 multibeam bathymetry is supported as format 171.
- SeaBeam 2120 multibeam data in the L3 Communications XSE format are
supported as format 94 (already used to support Elac Bottomchart MkII XSE
- Raw STN Atlas multibeam data generated by the upgraded Hydrosweep DS2
multibeam on the R/V Ewing are supported by read-only format 182.
Processing is supported using the augmented read-write format 183.
- The IFREMER netCDF multibeam archiving data format is supported
as format 75. Similarly, the IFREMER netCDF navigation
archiving data format is supported
as format 167.
- The STN Atlas processing data format SURF is supported as format 181. At
present, SURF is supported as a read-only format. This allows plotting and gridding
of the SURF data, but not processing. Writing or translating the SURF data to
allow processing will be supported in a later version.
- The Hawaii Mapping Research Group's MR1 format is supported as
format 64. This format is used to disseminate data from both the
HMRG interferometric sonars (e.g. MR1) and the WHOI DSL 120 deep-towed
inteferometric sonar. This format has been supported by including
the code for the HMRG library libmr1pr in the MB-System library. Thanks
to Roger Davis and HMRG for making the code available under the GPL.
- The Reson 7k format produced by the 7000 series Reson SeaBat multibeams
and the Reson 6046 datalogger is now supported as format 88. This format
can incoporate sidescan sonar and subbottom profiler data as well as the
- Third generation Simrad multibeams (EM122, EM302, EM710) are supported by formats 58 and 59.
- Imagenex and Odom DeltaT multibeams are supported by formats 191 and 192.
Last Updated: $Id: mbsystem_whatsnew.html 1917 2012-01-10 19:25:33Z caress $
to MB-System Home Page...