AOSN-III Data System Planning Map

Ensure success of ASAP

Create Observational Data (PIs)

Make Observations

SIO Gliders

WHOI Gliders

NPS Aircraft

MBARI AUV

MIT AUV

Ship observations

Process Raw Data

Provide QC flags (optional)

Make data available to central server

Push to central server: FTP, HTTP, media, or e-mail

Central server pulls

Maintain PI's FTP or HTTP Site

Use other FTP or HTTP Site

Support data pipeline creation

Provide example data so AOSN-III staff can create pipeline

Create own data pipeline metadata

Move data to central data server

Set up data server

Define metadata

Choose existing standard (Godin)

Modify to fit needs (Godin)

Agree on data push/pull interfaces/protocols (Godin will propose)

Write software

Data transfer

Push (detect new data on server) (AOSN-III Hire)

Pull (implements metadata settings) (AOSN-III Hire)

Versioning (SSDS programmers)

Translator (native -> common format) (Godin)

QC (simple -- no AI for now) (AOSN-III Hire)

Monitor data flow (SSDS programmers)

System Testing

Data transfer (e.g. speed) ( AOSN-III Hire)

Volume testing (storage requirements) (AOSN-III Hire)

Network testing (Nagios: AOSN-III Hire)

Archiving (Use existing software TBD)

Create test "Sandbox" region on server (Godin & AOSN-III Hire)

Create a mirror/backup (JPL - Peggy?)

Create archive repository (JPL? Godin?)

Establish data pipeline (fill out data flow chart)

Determine native format for data of each platform (PI's and/or AOSN-III Hire)

Establish metadata for each platform (PI's and/or AOSN-III Hire)

Test on "Sandbox"

Data conversion programs (see above)

Data transfer (e.g. speed) (see above)

Volume requirements) (see above)

Real-time operation

Data transfer

Push

Initiate Push (Data Generators)

Detect new data (see above)

Pull (see above)

Data processing

Versioning (see above)

Conversion to common format (see above)

QC (see above)

Monitor data flow (see above)

Network testing (see above)

Data center <-> data originators

Data center <-> modelers

Data flow monitor

Feed data into models for predictions & return model output

Establish data pipeline for models

Need list of data to be assimilated (JPL, Harvard, NRL?)

Decide on what model output should go to central data server (?)

(see Establish data pipeline, above)

Establish distributed model server for complete access (JPL?)

Real-time operation

Feed data to model (push or pull) (see below)

Move subset model output to central data center (push or pull) (see above)

Testing

Provide access to observational data and models to ASAP PIs

Complete MBARI LAS w MOQuA (Godin)

Queries

Graphic products

Platform-specific products

AUV sections

Glider sections

Mooring time-series

Data extraction (subject to usage constraints)

Establish distributed model output server (JPL?)

Enable real-time operational decision making

Provide state of system information

Last known position of observational assets (MOQuA query - Godin)

Data flow/work flow status (SSDS query - ?)

Observation state

Coverage evaluation (Zhang?)

Re-construction evaluations (Zhang, Davis?)

Model now-cast and forecast

HOPS (physics, chemistry, biology) (Harvard?)

ROMS (physics, chemistry, biology) (JPL?)

Icon (physics) (NRL, or MOQuA query?)

Model-data comparison

Comparisons with model interpolated to observation reference frame

Comparisons w unassimilated observations

(Performed at modeling sites?)

Adaptive sampling tools

Sensitivity analysis

ETKF (Sharan, Pierre?)

ESSE (Pierre?)

OSSEs/OSEs (Yi, Pierre?)

Path planning/asset allocation (Zhang, Bellingham)

Ancillary data (stored on server like other observations)

HF Radar (Paduan & Godin)

Wave forecasts (SIO -? & Godin)

Atmospheric forecasts (COAMPS) (Doyle & Godin)

Satellite (Dave Foley & Godin)

Mooring (M0, M1, M2) (Godin)

other ship observations (?)

Integrate with other programs

Collect Data from 2006 Field Programs (TBD)

AESOP

PLUS

LOCO?

Operate as node of larger distributed systems (Godin & AOSN-III Hire)

OOI

MMI

GCMD

Serve the public interest

Identify interest groups

Identify partners

Identify useful products

Create data flow pipeline to create products