Rules of thumb

My “rules-of-thumb” for choosing which regression to use are as follows:

For situations where the X-parameter is controlled, as in making-up standards for instrument calibration or doing laboratory experiments where only one variable is changed, then the standard model-I regressions are required.

When comparing two methods that measure the same quantity [e.g., DOC(per) vs DOC(htc)] then the major axis [lsqfitma] provides the most simple, easy to calculate and straight-forward solution.

  • One word of caution: the slope of major axis is sensitive to changes in scale (see: Kermack and Haldane, 1950, for an explanation).
  • If this is a concern (as in cases where the choice of measurement units is arbitrary), then the geometric mean regression [lsqfitgm] may be used.

For comparing two different measured parameters (like DOC vs TCO2, AOU, etc.) and especially when the parameters have different units or widely varying metrics, then the reduced major axis is the method of choice.

  • The reduced major axis method is both difficult and tedious to compute, so I recommend that the geometric mean regression [lsqfitgm] be used in this case as this gives the identical line (see Ricker, 1973, for the derivation). I have used the symmetrical limits for a model I regression to estimate the uncertainty in the slope and intercept of the geometric mean regression following Ricker’s (1973) treatment.
  • Sprent and Dolby (1980) have taken exception to the ad hoc use of the geometric mean regression in model II cases. They recommend the use of the least squares bisector [lsqbisec], the line that bisects the angle between the two model I regressions. Unfortunately, they did not present a statistical treatment for the estimation of the uncertainty limits for the least squares bisector slope, or intercept.

When the analytical methods for the different parameters have disparate uncertainties or experimental precision, then a weighted Model II  method may be required.

  • The method developed by York called “the least squares cubic” [lsqcubic] can be used in this case.
  • This method allows the uncertainty in each X and Y measurement be specified in order to appropriately weight the data.


Data repository
Data policy
What is happening in Monterey Bay today?
Central and Northern California Ocean Observing System
Chemical data
Ocean float data
Slough data
Mooring ISUS measurements
Southern Ocean Data
Mooring data
M1 Mooring Summary Data
M1 Asimet
M1 Download Info
M1 EMeter
M1 Fluorometer (CeNCOOS)
Molecular and genomics data
ESP Web Portal
Seafloor mapping
Upper ocean data
Spatial Temporal Oceanographic Query System (STOQS) Data
Image gallery
Video library
Previous seminars
David Packard Distinguished Lecturers
Research software
Video Annotation and Reference System
System Overview
Annotation Interface
Video Tape User Guide
Video File User Guide
Annotation Glossary
Query Interface
Basic User Guide
Advanced User Guide
Query Glossary
VARS Publications
Oceanographic Decision Support System
MB-System seafloor mapping software
How to Download and Install MB-System
MB-System Documentation
MB-System Announcements
MB-System Announcements (Archive)
MB-System FAQ
MB-System Discussion Lists
MB-System YouTube Tutorials
Matlab scripts: Linear regressions
Introduction to Model I and Model II linear regressions
A brief history of Model II regression analysis
Index of downloadable files
Summary of modifications
Regression rules of thumb
Results for Model I and Model II regressions
Graphs of the Model I and Model II regressions
Which regression: Model I or Model II?
Matlab scripts: Oceanographic calculations
Matlab scripts: Sound velocity
Visual Basic for Excel: Oceanographic calculations
Educational resources
MBARI Summer Internship Program
2017 Summer Interns Blog
Education and Research: Testing Hypotheses (EARTH)
EARTH workshops
2016—New Brunswick, NJ
2015—Newport, Oregon
2016 Satellite workshop—Pensacola, FL
2016 Satellite workshop—Beaufort, NC
EARTH resources
EARTH lesson plans
Lesson plans—published
Lesson plans—development
Lesson drafts—2015
Lesson drafts—2016 Pensacola
Adopt-A-Float Program
Center for Microbial Oceanography: Research and Education (C-MORE) Science Kits
Sample archive