The Shore Side Data System (SSDS) is a data management system that stores both raw and processed data (and associated metadata) for a wide variety of oceanographic platforms and instruments. It lets users query for, visualize, and retrieve the managed data, which is stored in many different formats on distributed storage. Users can submit new and processed data sets to SSDS for others to access. The system rewards metadata definition during instrument or software development, and provides a wide array of metadata-aware services.
Origins and Requirements. The SSDS is a component of the Monterey Ocean Observing System--commonly called MOOS. MOOS is a long-term MBARI development effort to develop key infrastructure elements for national and international ocean observing systems. The initial MOOS Workshop Report cited the need for a reliable, cost-effective data management system. Subsequent SSDS requirements development established more clearly the specific goals and features of the system.
There are many data management systems, of course -- what makes SSDS different? Most major data management systems work within certain boundaries, dealing with specific data formats or conventions (e.g., netCDF COARDS), or a known and stable data set (WODC). But MBARI's diverse data management requirements mean SSDS must be able to ingest and serve:
- a large and ever-changing set of sensors, instruments, and data set formats
- varied data collection formats, ASCII and binary, simple and complex
- streaming, file-based, and human-entered data, from internal and external data systems, in near-real-time and offline
- reprocessed data, including quality control and domain-specific data transformations
- operational changes to observatory instrumentation, including calibrations, instrument changes, and platform changes
Design. The SSDS design emphasizes flexibility, scalability, expandability, and maintainability, anticipating the highly variable and growing research and operational demands on ocean observing systems. Because it is part of an Ocean Observing System, SSDS is particularly effective when coupled with a metadata-aware instrument infrastructure such as SIAM. However, it readily accommodates less systematic data sources, recognizing the critical role of external data and system interoperability in scientific data management. By applying an iterative methodology within a systems-oriented development approach, we have been able to tune the architecture for each new application while building toward long-term observing system goals.
Status and Plans. Although the SSDS status continues to be developmental, the system has been operating internally for 5 years and serves many research, operational, and developmental data products. It provides data management for several ocean observing platforms and data sets, including mooring data for the Center for Integrated Marine Technologies project. Provided services include archiving raw data from streams and files, notifying users of data arrival, and providing basic access to the data and related metadata.
For the future, the functional metadata schema (XML) and sophisticated metadata database provide a solid platform for data management and systems development. Pending SSDS services will simplify initial metadata entry (while standardizing metadata content) and provide more varied ways for users to access their data. The developers' goal is to make the software widely available on an open-source basis.