Skip to content

Using the IMOS User Code Library with Python

Jump to bottom
Marty edited this page Jun 20, 2013 · 14 revisions

Table of Contents

1. Introduction

This document intends to present how to load IMOS NetCDF data into a Python environment, and offers some suggestions about how to use the data once loaded. All the examples below will use the netCDF4 Python module (http://code.google.com/p/netcdf4-python/)

The examples provided in this document only represent a tiny bit of the content of most of the NetCDF files. There are usually many more variables available in a NetCDF file, and therefore many other ways to display data.

1.1 Installation of the IMOS User Code Library (Python) and required packages

Each dataset-specific example below can be found in the demos folder as a separate Python script.

The code library can be checked out using a Git client, or be downloaded as a zip file : https://github.com/aodn/imos_user_code_library/archive/master.zip

The examples rely on the following Python packages, which need to be installed:

The examples below have been tested in Python 2.7.3, with numpy 1.6.1, matplotlib 1.1.1, and version 1.0.2 of netCDF4. Later versions of these packages will usually work. Earlier versions may work also.

1.2 Finding an IMOS NetCDF File

In order to find a dataset you are interested in, please refer to the portal help: http://portalhelp.aodn.org.au/Portal2_help/ . A HOWTO has been written to help any user in his way to find an IMOS NetCDF file.

For users who are already familiar with IMOS facilities and datasets, IMOS NetCDF files are also directly accessible via an OPeNDAP catalog at : http://thredds.aodn.org.au/thredds/catalog/IMOS/catalog.html

Once a NetCDF file has been chosen to work with (See http://portalhelp.aodn.org.au/Portal2_help/?q=node/112), the user needs to go to the 'OPeNDAP Dataset Access Form' page. The 'Data URL' located just above the 'Global Attributes' field is the URL which needs to be replaced in the examples which we'll present in the following sections.

The NetCDF file can also be downloaded to the user’s local machine using the HTTP Server option on the THREDDS options page. In this case, the user has to replace the 'Data URL' variable with the local address of the NetCDF file when using the function 'Dataset' in the following examples.

2. General Features of the netCDF4 module

The first step consists of opening a NetCDF file, whether this file is available locally or remotely on an OPeNDAP server.

Type in your Python command window (or script):

from netCDF4  import Dataset

aatams_URL = 'http://thredds.aodn.org.au/thredds/dodsC/IMOS/eMII/demos/AATAMS/marine_mammal_ctd-tag/2009_2011_ct64_Casey_Macquarie/ct64-M746-09/IMOS_AATAMS-SATTAG_TSP_20100205T043000Z_ct64-M746-09_END-20101029T071000Z_FV00.nc'
aatams_DATA = Dataset(aatams_URL)

This creates a netCDF Dataset object, through which you can access all the contents of the file.

2.1 Output structure

Please refer to the netCDF4 module documentation for a complete description of the Dataset object:: 'http://netcdf4-python.googlecode.com/svn/trunk/docs/netCDF4.Dataset-class.html' (or type help(Dataset) at the Python prompt).

2.2 Discover Metadata

In order to see all the global attributes and some other information about the file, type in your command window:

print aatams_DATA

The output will look something like this:

 root group (NETCDF3_64BIT file format): 
    project: Integrated Marine Observing System (IMOS) 
    conventions: IMOS-1.2 
    date_created: 2012-09-13T07:27:03Z 
    title: Temperature, Salinity and Depth profiles in near real time 
    institution: AATAMS 
    site: CTD Satellite Relay Data Logger 
    abstract: CTD Satellite Relay Data Loggers are used to explore how marine mammal behaviour relates to their oceanic environment. Loggers developped at the University of St Andrews Sea Mammal Research Unit transmit data in near real time via the Argo satellite system 
    source: SMRU CTD Satellite relay Data Logger on marine mammals 
…
    dimensions: obs, profiles 
    variables: TIME, LATITUDE, LONGITUDE, TEMP, PRES, PSAL, parentIndex, TIME_quality_control, LATITUDE_quality_control, LONGITUDE_quality_control, TEMP_quality_control, PRES_quality_control, PSAL_quality_control 
    groups:

Global attributes in the netCDF file become attributes of the Dataset object. A list of global attribute names is returned by the ncattrs() method of the object. The __dict__ attribute of the object is a dictionary of all netCDF attribute names and values.

# store the dataset's title in a local variable
title_str = aatams_DATA.title

# list all global attribute names
aatams_DATA.ncattrs()

# store the complete set of attributes in a dictionary (OrderedDict) object (similar to a standard Python dict, but
# maintains the order in which items are entered)
globalAttr = aatams_DATA.__dict__

# now you can also do (same effect as first command above)
title_str = globalAttr['title']

2.3 Discover Variables

To list all the variables available in the NetCDF file, type:

aatams_DATA.variables.keys()

Output:

[u'TIME', 
 u'LATITUDE', 
 u'LONGITUDE', 
 u'TEMP', 
 u'PRES', 
 u'PSAL', 
 u'parentIndex', 
 u'TIME_quality_control', 
 u'LATITUDE_quality_control', 
 u'LONGITUDE_quality_control', 
 u'TEMP_quality_control', 
 u'PRES_quality_control', 
 u'PSAL_quality_control']

(The 'u' means each variable name is represented by a Unicode string.)

Each variable is accessed via a Variable object, in a similar way to the Dataset object. To access the Temperature variable :

# netCDF4 Variable object
TEMP = aatams_DATA.variables['TEMP']

# now you can print the variable's attributes and other info
print TEMP

# access variable attributes, e.g. its standard_name
TEMP.standard_name

# extract the data values (as a numpy array)
TEMP[:]

# the variable's dimensions (as a tuple)
TEMP.dimensions

3. Dataset examples – Using the NetCDF Parser for Plotting

3.1 AATAMS – Animal Tagging and Monitoring - non QC'd data

The Australian Animal Tagging And Monitoring System (AATAMS) is a coordinated marine animal tagging project. CTD Satellite Relay Data Loggers are used to explore how marine mammal behaviour relates to their oceanic environment.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/AATAMS/marine_mammal_ctd-tag/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot all the animal's dives as a single profile time-series of temperature, measured by a CTD tag.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/aatams.py Illustration Example of a Temperature Profile Time-series from AATAMS data

Illustration Example of a Temperature profile from AATAMS data

3.2 ABOS – Deep Water Mooring

3.2.1 Southern Ocean Time-series - non QC'd data

The Southern Ocean Time Series (SOTS) sub-facility provides high temporal resolution observations in sub-Antarctic waters. Observations are broad and include measurements of physical, chemical and biogeochemical parameters from multiple deep-water moorings.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ABOS/SOTS/catalog.html

In the example below, the netCDF4 module is used to extract temperature data from a Pulse mooring instrument and print the abstract for the dataset. A temperature time series plot is produced using matplotlib.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/abos.py Illustration

3.3 ACORN – Ocean Radar - non QC'd data

The Australian Coastal Ocean Radar Network (ACORN) facility comprises a coordinated network of HF radars delivering real-time, non-quality controlled and delayed-mode, quality controlled surface current data into a national archive.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ACORN/catalog.html

Monthly aggregated files are also available in the following folders: monthly_gridded_1h-avg-current-map_QC monthly_gridded_1h-avg-current-map_non-QC

In the example below, we demonstrate how to use the netCDF4 module to plot velocity data for one time value only in a latitude / longitude grid.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/acorn.py

Illustration Example of a Sea Water Speed gridded data with a Velocity Field from ACORN data

3.4 ANFOG – Ocean Gliders - QC'd good data

The Australian National Facility for Ocean Gliders (ANFOG), with IMOS/NCRIS funding, deploys a fleet of eight gliders around Australia.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ANFOG/seaglider/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot salinity data as well as depth data in a same graph. Only the data points with a Quality Control flag greater than 1 (which means 'good data' , please refers to IMOS NetCDF User Manual for a description of the Quality Control, available at http://imos.org.au/facility_manuals.html)

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/anfog.py Illustration Example of Sea Water Time-series taken during a SeaGlider Dive. Filtered to plot good data only

3.5 ANMN – National Mooring Network - QC'd good data

The Australian National Mooring Network Facility is a series of national reference stations and regional moorings designed to monitor particular oceanographic phenomena in Australian coastal ocean waters.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/ANMN/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot the U current variable measured with an ADCP instrument (in Western Australia).

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/anmn_adcp.py

Illustration Example of a Sea Water Velocity plot from ADCP data

3.6 AUV – Autonomous Underwater Vehicle - non QC'd data

The IMOS Autonomous Underwater Vehicle (AUV) Facility operates an ocean going AUV called Sirius capable of undertaking high resolution, geo-referenced survey work.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/AUV/catalog.html

In the example below, the netCDF4 module is used to extract depth, temperature, and time data and then produce a multiple time-series plot showing the variation of water temperature with depth and time during the robot’s dive.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/auv.py

Illustration Example of a Temperature Time-series plot during an AUV dive

3.7 Argo – Argo Floats Program - non QC'd data

Argo floats have revolutionised our understanding of the broad scale structure of the oceans to 2000 m depth. In the past 10 years more high resolution hydrographic profiles have been provided by Argo floats then from the rest of the observing system put together. Each Argo float is identified by a unique identification number called a WMO ID.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/Argo/aggregated_datasets/catalog.html

In the examples below, we demonstrate how to use the netCDF4 module to plot Argo data from an aggregated file ( One file per year per basin : Atlantic, Indian, Pacific North, Pacific South).

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/Argo.py

Illustration Example of a Sea Water Temperature profile from an Argo float

Illustration Example of a Sea Water Temperature Time-series profile from an Argo float with its location over time

3.8 FAIMMS – Wireless Sensor Networks - QC'd good data

The IMOS Facility for Intelligent Monitoring of Marine Systems is a sensor network established in the Great Barrier Reef off the coast of Queensland, Australia. A 'sensor network' is an array of small, wirelessly interconnected sensors that collectively stream sensor data to a central data aggregation point. Sensor networks can be used to provide spatially dense bio-physical measurements in real-time.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/FAIMMS/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot a temperature time-series. Only data points which have a flag value equal to 1 are used (which means 'good data', please refers to IMOS NetCDF User Manual for a description of the Quality Control, available at http://imos.org.au/facility_manuals.html).

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/faimms.py

Illustration Example of a Sea Water Temperature at 5m depth on the Great Barrier Reaf from FAIMMS data

3.9 SOOP – Ship Of Opportunities

3.9.1 XBT - expandable bathythermographs - QC'd data

IMOS Ship of Opportunity Underway Expandable Bathythermographs (XBT) group is a research and data collection project working within the IMOS Ship of Opportunity Multi-Disciplinary Underway Network sub-facility.

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SOOP/SOOP-XBT/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot a XBT temperature profile.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/soop_xbt.py Illustration Example of Sea Water Temperature Profile from XBT data Illustration Example of Sea Water Temperature Time-series Profile from XBT data with the profiles location

3.10 SRS – Satellite Remote Sensing

3.10.1 Bio-Optical database – Pigment data

The bio-optical data base underpins the assessment of ocean colour products in the Australian region (e.g. chlorophyll a concentrations, phytoplankton species composition and primary production).

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/BioOptical/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot a Chlorophyll-a profile (High Performance Liquid Chromatography of pigments in discrete sea-water samples)

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/srs_BioOptical_pigment.py

Illustration Example of Pigment Data Profile from the BioOptical database dataset

3.10.2 Bio-Optical database – Absorption data

The bio-optical data base underpins the assessment of ocean colour products in the Australian region (e.g. chlorophyll a concentrations, phytoplankton species composition and primary production).

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/catalog/IMOS/SRS/BioOptical/catalog.html

In the example below, we demonstrate how to use the netCDF4 module to plot (1) the variation of Absorption coefficients of CDOM (gilvin) in discrete sea-water samples at different wavelengths and (2) the variation of absorption coefficients of CDOM at different wavelengths and different depths.

To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/srs_BioOptical_absorption.py

Illustration Example of Absorption Data plot from the BioOptical database dataset Illustration Example of Absorption Data at different depth from the BioOptical database dataset

3.10.3 GHRSST – L3P mosaic

Please refer to the SRS product Help page : http://portalhelp.aodn.org.au/Portal2_help/?q=node/149

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/GHRSST-SSTsubskin/

In the example below, we demonstrate how to use the netCDF4 module to plot the Sea Surface Temperature from a gridded data product.

Warning : this dataset can take a while to be loaded on your machine. To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/srs_l3p.py Illustration Example of Sea Surface Temperature plot from a L3P product centered on the GBR

3.10.4 GHRSST – L3S – multi swath, multi sensor, one day 3.10.4 – Geographic subset

Please refer to the SRS product Help page : http://portalhelp.aodn.org.au/Portal2_help/?q=node/149

NetCDF files can be found at : http://thredds.aodn.org.au/thredds/dodsC/IMOS/SRS/SRS-SST/L3S-01day/

In the example below, we demonstrate how to use the netCDF4 module to plot the Sea Surface Temperature from a gridded data product

Warning : this dataset can take a while to be loaded on your machine. To paste the code in your Python environment, please copy it from : https://github.com/aodn/imos_user_code_library/blob/master/Python/demos/srs_l3s.py Illustration Example of Sea Surface Temperature plot from a L3S product

Clone this wiki locally