ATOMIX - NetCDF file creation
Created by Cynthia Bluteau for SCOR WG160 members
Table of content
- Structure of a NetCDF file
- NetCDF matlab toolbox general layout
- Prepare your data into matlab structures
- Check variable & dimensions attributes databases
- Prepare global & group attributes via YAML files
- Check flag attribute via YAML files
- GUIs for checking nc file
-
Down navigation arrows provide more info for each topic!
Structure of a NetCDF file with groups
Variables & Dimensions also have attributes (metadata)
Global attributes: "Title", "Author", "Instrument", etc
Group attributes: "Processing_level", "Comment", any other notes about the group (e.g., Level1_raw)
Structure of ADV Level 1 Group
- Dimensions specify the size/shape of Variables
- Both have attributes (metadata)
Dimensions defined for the group level
Example for Level 1 (ADVs)
NetCDF Writing Toolbox
- This NetCDF toolbox relies on low-level netcdf functions in Matlab (feel free to use your own).
- YAML files contain boiler plate metadata information -- useful even if you use other NetCDF writing tools
Visual summary
NetCDF toolbox set-up
- Each group are stored in matlab structures
- Each fields corresponds to the desired short_name (e.g., PRES, TIME)
- Global and group metadata are stored in YAML files
- Dimensions are specified via the YAML file(s)
- Variables/dimensions attributes set from CSV database lookup using the short_names (i.e., fields in each matlab structure)
- Additional variable attributes can be set from YAML flag files
Prepare your data
Each group are stored in Matlab structures
Designating dimensions variable
- Define dimensions using YAML file of group attributes
- OR provide it manually in the group's Meta field along with any other attributes.
In both cases, the code uses variable sizes to assign the correct dimensions ...
Assigning dimensions at the variable level
If more than 1 dimension variable has the same size, then dimensions must be defined for EACH variable using the YAML flag files
Example of Variables & Dimensions Attributes
Check Variables & Dimensions Attributes
Attributes are stored in CSV databases, some are transfered from the wiki and/or atomix_netcdf.xlsx on teams.
These should remain fairly constant for everyone, but you can modify attributes by creating
- your custom attributes database
- Or by writing attributes for each variable via the YAML flag files
YAML files supply group and global attributes
This YAML tutorial explains more the format, which uses spaces (no tabs) to organise information
Prepare Global Attributes
- Look at Global attributes YAML example file
- Online YAML Tree View are useful for looking and ensuring format is correct
Prepare Group Attributes
Example YAML for supplying ADV's group attributes
Quality control flags
Variables with "_FLAGS" appended have extra attributes "flag_meanings" and "flag_values" as per CF standards. For example:
- XYZ_VEL contains velocity data
- XYZ_VEL_FLAGS contains 8-bit boolean flags to designate rejection reasons for velocity samples (low correlation, obstructions, etc)
See YAML file example for creating these attributes
Two CF-compliant flag types
| Boolean | Enumerated | |
|---|---|---|
| Values | 0-255 | 0-9 |
| DataType | 8 bytes | 10 integer |
| Enables multiple criteria/flags met at once | Only 1 criteria can be met |
YAML files specifies the additional flag attributes
Should remain fairly constant for a subgroup.
Running the tools, and checking your file
