isofit.radiative_transfer.luts
==============================

.. py:module:: isofit.radiative_transfer.luts

.. autoapi-nested-parse::

   This is the netCDF4 implementation for handling ISOFIT LUT files. For previous
   implementations and research, please see https://github.com/isofit/isofit/tree/897062a3dcc64d5292d0d2efe7272db0809a6085/isofit/luts



Attributes
----------

.. autoapisummary::

   isofit.radiative_transfer.luts.Logger


Classes
-------

.. autoapisummary::

   isofit.radiative_transfer.luts.Keys
   isofit.radiative_transfer.luts.Create


Functions
---------

.. autoapisummary::

   isofit.radiative_transfer.luts.findSlice
   isofit.radiative_transfer.luts.optimizedInterp
   isofit.radiative_transfer.luts.sel
   isofit.radiative_transfer.luts.sub
   isofit.radiative_transfer.luts.couple
   isofit.radiative_transfer.luts.load
   isofit.radiative_transfer.luts.extractPoints
   isofit.radiative_transfer.luts.extractGrid
   isofit.radiative_transfer.luts.saveDataset
   isofit.radiative_transfer.luts.cleanup


Module Contents
---------------

.. py:data:: Logger

.. py:class:: Keys

   .. py:attribute:: consts


   .. py:attribute:: onedim


   .. py:attribute:: alldim


.. py:class:: Create(file: str, wl: numpy.ndarray, grid: dict, attrs: dict = {}, consts: dict = {}, onedim: dict = {}, alldim: dict = {}, zeros: List[str] = [], compression: str = 'zlib', complevel: int = None)

   .. py:attribute:: file


   .. py:attribute:: wl


   .. py:attribute:: grid


   .. py:attribute:: hold
      :value: []



   .. py:attribute:: sizes


   .. py:attribute:: attrs


   .. py:attribute:: consts


   .. py:attribute:: onedim


   .. py:attribute:: alldim


   .. py:attribute:: compression
      :value: 'zlib'



   .. py:attribute:: complevel
      :value: None



   .. py:method:: initialize() -> None

      Initializes the LUT netCDF by prepopulating it with filler values.



   .. py:method:: pointIndices(point: numpy.ndarray) -> List[int]

      Get the indices of the point in the grid.

      :param point: The coordinates of the point in the grid.
      :type point: np.ndarray

      :returns: Mapped point values to index positions.
      :rtype: List[int]



   .. py:method:: queuePoint(point: numpy.ndarray, data: dict) -> None

      Queues a point and its data to the internal hold list which is used by the
      flush function to write these points to disk.

      :param point: The coordinates of the point in the grid.
      :type point: np.ndarray
      :param data: Data for this point to write.
      :type data: dict



   .. py:method:: flush(finalize: bool = False) -> None

      Flushes the (point, data) pairs held in the hold list to the LUT netCDF.

      :param finalize: Calls the `finalize` function
      :type finalize: bool, default=False



   .. py:method:: writePoint(point: numpy.ndarray, data: dict) -> None

      Queues a point and immediately flushes to disk.

      :param point: The coordinates of the point in the grid.
      :type point: np.ndarray
      :param data: Data for this point to write.
      :type data: dict



   .. py:method:: setAttr(key: str, value: Any) -> None

      Sets an attribute in the netCDF

      :param key: Key to set
      :type key: str
      :param value: Value to set
      :type value: any



   .. py:method:: getAttr(key: str) -> Any

      Gets an attribute from the netCDF

      :param key: Key to get
      :type key: str

      :returns: Retrieved attribute from netCDF, if it exists
      :rtype: any | None



   .. py:method:: finalize()

      Finalizes the netCDF by writing any remaining attributes to disk



   .. py:method:: __getitem__(key: str) -> Any

      Passthrough to __getitem__ on the underlying 'ds' attribute.

      :param key: The name of the item to retrieve.
      :type key: str

      :returns: The value of the item retrieved from the 'ds' attribute.
      :rtype: Any



   .. py:method:: __repr__() -> str


.. py:function:: findSlice(dim, val)

   Creates a slice for selecting along a dimension such that a value is encompassed by
   the slice.

   :param dim: Dimension array
   :type dim: array
   :param val: Value to be encompassed
   :type val: float, int

   :returns: Index slice to encompass the value
   :rtype: slice


.. py:function:: optimizedInterp(ds, strat)

   Optimizes the interpolation step by subselecting along dimensions first then
   interpolating

   :param strat: Interpolation stategies to perform in the form of {dim: interpolate_values}
   :type strat: dict

   :returns: Interpolated dataset
   :rtype: xr.Dataset


.. py:function:: sel(ds, dim, lt=None, lte=None, gt=None, gte=None, encompass=True)

   Subselects an xarray Dataset object using .sel

   :param ds: LUT dataset
   :type ds: xarray.Dataset
   :param dim: Dimension to work on
   :type dim: str
   :param lt: Select along this dim coordinates that are valued less than this
   :type lt: float, default=None
   :param lte: Select along this dim coordinates that are valued less than or equal to this
   :type lte: float, default=None
   :param gt: Select along this dim coordinates that are valued greater than this
   :type gt: float, default=None
   :param gte: Select along this dim coordinates that are valued greater than or equal to this
   :type gte: float, default=None
   :param encompass: Change the values of gte/lte such that these values are encompassed using the
                     previous/next valid grid point
   :type encompass: bool, default=True

   :returns: **ds** -- Subsetted dataset
   :rtype: xarray.Dataset


.. py:function:: sub(ds: xarray.Dataset, dim: str, strat) -> xarray.Dataset

   Subsets a dataset object along a specific dimension in a few supported ways.

   :param ds: Dataset to operate on
   :type ds: xr.Dataset
   :param dim: Name of the dimension to subset
   :type dim: str
   :param strat: Strategy to subset the given dimension with
   :type strat: float, int, list, dict, str, None

   :returns: New subset of the input dataset
   :rtype: xr.Dataset


.. py:function:: couple(ds, inplace=True)

   Calculates coupled terms on the input Dataset

   :param ds: Dataset to process on
   :type ds: xr.Dataset
   :param inplace: Insert the coupled terms in-place to the original Dataset. If False, copy the
                   Dataset first
   :type inplace: bool, default=True

   :returns: **ds** -- Dataset with coupled terms
   :rtype: xr.Dataset


.. py:function:: load(file: str, subset: dict = None, dask: bool = False, mode: str = 'a', lock: bool = False, load: bool = True, coupling: str = 'after', **kwargs) -> xarray.Dataset

   Loads a LUT NetCDF
   Assumes to be a regular grid at this time (auto creates the point dim)

   :param file: LUT file to load
   :type file: str
   :param subset: Subset each dimension with a given strategy. Each dimension in the LUT file
                  must be specified.
                  See examples for more information
   :type subset: dict, default={}
   :param dask: Use Dask on the backend of Xarray to lazy load the dataset. This enables
                out-of-core subsetting
   :type dask: bool, default=False
   :param mode: File mode to open with, must be set to append="a" for coupled terms to be
                saved back
   :type mode: str, default="a"
   :param lock: Set a lock on the input file
   :type lock: bool, default=False
   :param load: Calls ds.load() at the end to cast from Dask arrays back into numpy arrays held
                in memory
   :type load: bool, default=True
   :param coupling:
                    Calculates coupling terms, if needed. This may be set one of four ways:
                        "before"
                            Calculate before subsetting
                        "before-save"
                            Before + save the coupled terms to the original input file
                        "after"
                            Calculate after subsetting
                        "after-save"
                            After + save to a new file (the original input file with the extension
                            changed to ".coupled-subset.nc")
   :type coupling: string, default="after"

   .. rubric:: Examples

   >>> # Create a test file for the examples to load
   >>> file = 'subsetting_example.nc'
   >>> lut_dims = {
   ...     'AOT550': [0.001, 0.1009, 0.2008, 0.3007, 0.4006, 0.5005, 0.6004, 0.7003, 0.8002, 0.9001, 1.],
   ...     'H2OSTR': [0.2231, 0.4637, 0.7042, 0.9447, 1.1853, 1.4258, 1.6664, 1.9069, 2.1474, 2.388, 2.6285, 2.869, 3.1096, 3.3501],
   ...     'observer_zenith': [7.2155, 9.8900],
   ...     'surface_elevation_km': [0., 0.2361, 0.4721, 0.7082, 0.9442, 1.1803, 1.4164, 1.6524, 1.8885, 2.1245, 2.3606, 2.5966, 2.8327, 3.0688, 3.3048, 3.5409, 3.7769, 4.013],
   ...     'wl': range(285)
   ... }
   >>> ds = xr.Dataset(coords=lut_dims)
   >>> ds.to_netcdf(file)

   >>> # Subset: Exact values along the dimension
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': [1.1853, 2.869],
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None,
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 792})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'H2OSTR': 2, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Interpolate H2OSTR to 1.5
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': {
   ...         'interp': 1.5
   ...     },
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None,
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 396})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: 1.1853 < H2OSTR < 2.869
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': {
   ...         'gt': 1.1853,
   ...         'lt': 2.869
   ...     },
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None,
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 2376})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'H2OSTR': 6, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: 1.1853 <= H2OSTR <= 2.869, encompassed
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': {
   ...         'gte': 1.1853,
   ...         'lte': 2.869
   ...     },
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 3960})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'H2OSTR': 10, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: 1.1853 <= H2OSTR <= 2.869, not encompassed
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': {
   ...         'gte': 1.1853,
   ...         'lte': 2.869,
   ...         'encompass': False
   ...     },
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 3168})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'H2OSTR': 8, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: Exact value, squeeze dimension
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': 2.869,
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 396})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: Using mean, squeeze dimension
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': 'mean',
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 396})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Subset: Using max, squeeze dimension
   >>> subset = {
   ...     'AOT550': None,
   ...     'H2OSTR': 'max',
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 396})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 11, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Multiple subsets
   >>> subset = {
   ...     'AOT550': [0.2008, 0.4006, 0.6004],
   ...     'H2OSTR': {
   ...         'gte': 1.1853,
   ...         'lte': 2.869
   ...     },
   ...     'observer_zenith': None,
   ...     'surface_elevation_km': None
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 1080})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 3, 'H2OSTR': 10, 'observer_zenith': 2, 'surface_elevation_km': 18, 'wl': 285})

   >>> # Multiple subsets
   >>> subset = {
   ...     'AOT550': [0.2008, 0.4006, 0.6004],
   ...     'H2OSTR': {
   ...         'gte': 1.1853,
   ...         'lte': 2.869
   ...     },
   ...     'observer_zenith': 7.2155,
   ...     'surface_elevation_km': 'mean'
   ... }
   >>> load(file, subset).dims
   Frozen({'wl': 285, 'point': 30})
   >>> load(file, subset).unstack().dims
   Frozen({'AOT550': 3, 'H2OSTR': 10, 'wl': 285})


.. py:function:: extractPoints(ds: xarray.Dataset, names: bool = False) -> numpy.array

   Extracts the points and point name arrays

   :param ds: LUT Dataset object
   :type ds: xr.Dataset
   :param names: Return the names of the point coords as well
   :type names: bool, default=False

   :returns: Extracted points, plus names if requested
   :rtype: points or (points, names)


.. py:function:: extractGrid(ds: xarray.Dataset) -> dict

   Extracts the LUT grid from a Dataset

   :param ds: LUT Dataset object. Carried stacked: Dimensions wl, points
   :type ds: xr.Dataset


.. py:function:: saveDataset(file: str, ds: xarray.Dataset) -> None

   Handles saving an xarray.Dataset to a NetCDF file for ISOFIT. Will detect if the
   point dim needs to be unstacked before saving (regular grids) or not (irregular)

   :param file: Path to save the `ds` object to. This will be a NetCDF, recommended extension
                is `.nc`
   :type file: str
   :param ds: Data object to save
   :type ds: xarray.Dataset


.. py:function:: cleanup(file)

   Checks the ``ISOFIT status`` attribute on a LUT and removes the file if it is
   incomplete.

   :param file: Path to the file to check
   :type file: str