isofit.radiative_transfer.luts

This is the netCDF4 implementation for handling ISOFIT LUT files. For previous implementations and research, please see https://github.com/isofit/isofit/tree/897062a3dcc64d5292d0d2efe7272db0809a6085/isofit/luts

Attributes

Logger

Classes

Keys
Create

Functions

findSlice(dim, val)	Creates a slice for selecting along a dimension such that a value is encompassed by
optimizedInterp(ds, strat)	Optimizes the interpolation step by subselecting along dimensions first then
sel(ds, dim[, lt, lte, gt, gte, encompass])	Subselects an xarray Dataset object using .sel
sub(→ xarray.Dataset)	Subsets a dataset object along a specific dimension in a few supported ways.
couple(ds[, inplace])	Calculates coupled terms on the input Dataset
load(→ xarray.Dataset)	Loads a LUT NetCDF
extractPoints(→ numpy.array)	Extracts the points and point name arrays
extractGrid(→ dict)	Extracts the LUT grid from a Dataset
saveDataset(→ None)	Handles saving an xarray.Dataset to a NetCDF file for ISOFIT. Will detect if the
cleanup(file)	Checks the ISOFIT status attribute on a LUT and removes the file if it is

Module Contents

Logger

class Keys

consts

onedim

alldim

class Create(file: str, wl: numpy.ndarray, grid: dict, attrs: dict = {}, consts: dict = {}, onedim: dict = {}, alldim: dict = {}, zeros: List[str] = [], compression: str = 'zlib', complevel: int = None)

file

wl

grid

hold = []

sizes

attrs

consts

onedim

alldim

compression = 'zlib'

complevel = None

initialize() → None

Initializes the LUT netCDF by prepopulating it with filler values.

pointIndices(point: numpy.ndarray) → List[int]

Get the indices of the point in the grid.

Parameters:

point (np.ndarray) – The coordinates of the point in the grid.

Returns:

Mapped point values to index positions.

Return type:

List[int]

queuePoint(point: numpy.ndarray, data: dict) → None

Queues a point and its data to the internal hold list which is used by the flush function to write these points to disk.

Parameters:

• point (np.ndarray) – The coordinates of the point in the grid.

• data (dict) – Data for this point to write.

flush(finalize: bool = False) → None

Flushes the (point, data) pairs held in the hold list to the LUT netCDF.

Parameters:

finalize (bool, default=False) – Calls the finalize function

writePoint(point: numpy.ndarray, data: dict) → None

Queues a point and immediately flushes to disk.

Parameters:

• point (np.ndarray) – The coordinates of the point in the grid.

• data (dict) – Data for this point to write.

setAttr(key: str, value: Any) → None

Sets an attribute in the netCDF

Parameters:

• key (str) – Key to set

• value (any) – Value to set

getAttr(key: str) → Any

Gets an attribute from the netCDF

Parameters:

key (str) – Key to get

Returns:

Retrieved attribute from netCDF, if it exists

Return type:

any | None

finalize()

Finalizes the netCDF by writing any remaining attributes to disk

__getitem__(key: str) → Any

Passthrough to __getitem__ on the underlying ‘ds’ attribute.

Parameters:

key (str) – The name of the item to retrieve.

Returns:

The value of the item retrieved from the ‘ds’ attribute.

Return type:

Any

__repr__() → str

findSlice(dim, val)

Creates a slice for selecting along a dimension such that a value is encompassed by the slice.

Parameters:

• dim (array) – Dimension array

• val (float, int) – Value to be encompassed

Returns:

Index slice to encompass the value

Return type:

slice

optimizedInterp(ds, strat)

Optimizes the interpolation step by subselecting along dimensions first then interpolating

Parameters:

strat (dict) – Interpolation stategies to perform in the form of {dim: interpolate_values}

Returns:

Interpolated dataset

Return type:

xr.Dataset

sel(ds, dim, lt=None, lte=None, gt=None, gte=None, encompass=True)

Subselects an xarray Dataset object using .sel

Parameters:

• ds (xarray.Dataset) – LUT dataset

• dim (str) – Dimension to work on

• lt (float, default=None) – Select along this dim coordinates that are valued less than this

• lte (float, default=None) – Select along this dim coordinates that are valued less than or equal to this

• gt (float, default=None) – Select along this dim coordinates that are valued greater than this

• gte (float, default=None) – Select along this dim coordinates that are valued greater than or equal to this

• encompass (bool, default=True) – Change the values of gte/lte such that these values are encompassed using the previous/next valid grid point

Returns:

ds – Subsetted dataset

Return type:

xarray.Dataset

sub(ds: xarray.Dataset, dim: str, strat) → xarray.Dataset

Subsets a dataset object along a specific dimension in a few supported ways.

Parameters:

• ds (xr.Dataset) – Dataset to operate on

• dim (str) – Name of the dimension to subset

• strat (float, int, list, dict, str, None) – Strategy to subset the given dimension with

Returns:

New subset of the input dataset

Return type:

xr.Dataset

couple(ds, inplace=True)

Calculates coupled terms on the input Dataset

Parameters:

• ds (xr.Dataset) – Dataset to process on

• inplace (bool, default=True) – Insert the coupled terms in-place to the original Dataset. If False, copy the Dataset first

Returns:

ds – Dataset with coupled terms

Return type:

xr.Dataset

load(file: str, subset: dict = None, dask: bool = False, mode: str = 'a', lock: bool = False, load: bool = True, coupling: str = 'after', **kwargs) → xarray.Dataset

Loads a LUT NetCDF Assumes to be a regular grid at this time (auto creates the point dim)

Parameters:

• file (str) – LUT file to load

• subset (dict, default={}) – Subset each dimension with a given strategy. Each dimension in the LUT file must be specified. See examples for more information

• dask (bool, default=False) – Use Dask on the backend of Xarray to lazy load the dataset. This enables out-of-core subsetting

• mode (str, default="a") – File mode to open with, must be set to append=”a” for coupled terms to be saved back

• lock (bool, default=False) – Set a lock on the input file

• load (bool, default=True) – Calls ds.load() at the end to cast from Dask arrays back into numpy arrays held in memory

• coupling (string, default="after") –

  Calculates coupling terms, if needed. This may be set one of four ways:

  ”before”

  Calculate before subsetting

  ”before-save”

  Before + save the coupled terms to the original input file

  ”after”

  Calculate after subsetting

  ”after-save”

  After + save to a new file (the original input file with the extension changed to “.coupled-subset.nc”)

Examples

>>> # Create a test file for the examples to load
>>> file = 'subsetting_example.nc'
>>> lut_dims = {
...     'AOT550': [0.001, 0.1009, 0.2008, 0.3007, 0.4006, 0.5005, 0.6004, 0.7003, 0.8002, 0.9001, 1.],
...     'H2OSTR': [0.2231, 0.4637, 0.7042, 0.9447, 1.1853, 1.4258, 1.6664, 1.9069, 2.1474, 2.388, 2.6285, 2.869, 3.1096, 3.3501],
...     'observer_zenith': [7.2155, 9.8900],
...     'surface_elevation_km': [0., 0.2361, 0.4721, 0.7082, 0.9442, 1.1803, 1.4164, 1.6524, 1.8885, 2.1245, 2.3606, 2.5966, 2.8327, 3.0688, 3.3048, 3.5409, 3.7769, 4.013],
...     'wl': range(285)
... }
>>> ds = xr.Dataset(coords=lut_dims)
>>> ds.to_netcdf(file)

>>> # Subset: Exact values along the dimension
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': [1.1853, 2.869],
...     'observer_zenith': None,
...     'surface_elevation_km': None,
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 792})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'H2OSTR': 2, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Interpolate H2OSTR to 1.5
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': {
...         'interp': 1.5
...     },
...     'observer_zenith': None,
...     'surface_elevation_km': None,
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 396})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: 1.1853 < H2OSTR < 2.869
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': {
...         'gt': 1.1853,
...         'lt': 2.869
...     },
...     'observer_zenith': None,
...     'surface_elevation_km': None,
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 2376})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'H2OSTR': 6, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: 1.1853 <= H2OSTR <= 2.869, encompassed
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': {
...         'gte': 1.1853,
...         'lte': 2.869
...     },
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 3960})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'H2OSTR': 10, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: 1.1853 <= H2OSTR <= 2.869, not encompassed
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': {
...         'gte': 1.1853,
...         'lte': 2.869,
...         'encompass': False
...     },
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 3168})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'H2OSTR': 8, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: Exact value, squeeze dimension
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': 2.869,
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 396})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: Using mean, squeeze dimension
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': 'mean',
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 396})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Subset: Using max, squeeze dimension
>>> subset = {
...     'AOT550': None,
...     'H2OSTR': 'max',
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 396})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Multiple subsets
>>> subset = {
...     'AOT550': [0.2008, 0.4006, 0.6004],
...     'H2OSTR': {
...         'gte': 1.1853,
...         'lte': 2.869
...     },
...     'observer_zenith': None,
...     'surface_elevation_km': None
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 1080})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 3, 'H2OSTR': 10, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

>>> # Multiple subsets
>>> subset = {
...     'AOT550': [0.2008, 0.4006, 0.6004],
...     'H2OSTR': {
...         'gte': 1.1853,
...         'lte': 2.869
...     },
...     'observer_zenith': 7.2155,
...     'surface_elevation_km': 'mean'
... }
>>> load(file, subset).dims
Frozen({'wl': 285, 'point': 30})
>>> load(file, subset).unstack().dims
Frozen({'AOT550': 3, 'H2OSTR': 10, 'wl': 285})

extractPoints(ds: xarray.Dataset, names: bool = False) → numpy.array

Extracts the points and point name arrays

Parameters:

• ds (xr.Dataset) – LUT Dataset object

• names (bool, default=False) – Return the names of the point coords as well

Returns:

Extracted points, plus names if requested

Return type:

points or (points, names)

extractGrid(ds: xarray.Dataset) → dict

Extracts the LUT grid from a Dataset

Parameters:

ds (xr.Dataset) – LUT Dataset object. Carried stacked: Dimensions wl, points

saveDataset(file: str, ds: xarray.Dataset) → None

Handles saving an xarray.Dataset to a NetCDF file for ISOFIT. Will detect if the point dim needs to be unstacked before saving (regular grids) or not (irregular)

Parameters:

• file (str) – Path to save the ds object to. This will be a NetCDF, recommended extension is .nc

• ds (xarray.Dataset) – Data object to save

cleanup(file)

Checks the ISOFIT status attribute on a LUT and removes the file if it is incomplete.

Parameters:

file (str) – Path to the file to check