isofit.utils.template_construction

Classes

Pathnames	Class to determine and hold the large number of relative and absolute paths that are needed for isofit and
SerialEncoder	Encoder for json to help ensure json objects can be passed to the workflow manager.
LUTConfig	A look up table class, containing default grid options. All properties may be overridden with the optional
SerialEncoder	Encoder for json to help ensure json objects can be passed to the workflow manager.

Functions

check_surface_model(→ str)	Checks and rebuilds surface model if needed.
build_presolve_config(→ None)	Write an isofit config file for a presolve, with limited info.
build_main_config(→ None)	Write an isofit config file for the main solve, using the specified pathnames and all given info
get_lut_subset(vals)	Populate lut_names for the appropriate style of subsetting
write_modtran_template(atmosphere_type, fid, ...[, ...])	Write a MODTRAN template file for use by isofit look up tables
load_climatology(config_path, latitude, longitude, ...)	Load climatology data, based on location and configuration
calc_modtran_max_water(→ float)	MODTRAN may put a ceiling on "legal" H2O concentrations. This function calculates that ceiling. The intended
define_surface_types(tsip, rdnfile, obsfile, ...)
copy_file_subset(matching_indices, pathnames)	Copy over subsets of given files to new locations
get_metadata_from_obs(obs_file, lut_params, ...)	Get metadata needed for complete runs from the observation file
get_metadata_from_loc(loc_file, lut_params, ...)	Get metadata needed for complete runs from the location file (bands long, lat, elev).
reassemble_cube(matching_indices, paths)	Copy over subsets of given files to new locations
sensor_name_to_dt(sensor, fid)
get_wavelengths(envi_file, wavelength_path, numpy.array)	Get wavelengths and FWHM from the header of an ENVI file
write_wavelength_file(filename, wl, fwhm)	Write a wavelength file in isofit-expected format
make_surface_config(paths[, surface_category, ...])	Constructs the surface component of the config

Module Contents

class Pathnames(input_radiance, input_loc, input_obs, surface_class_file, surface_path, working_directory, ray_temp_dir, sensor='NA-*', copy_input_files=False, modtran_path=None, rdn_factors_path=None, model_discrepancy_path=None, aerosol_climatology_path=None, channelized_uncertainty_path=None, interpolate_inplace=False, skyview_factor=None, subs: bool = False, classify_multisurface: bool = False)

Class to determine and hold the large number of relative and absolute paths that are needed for isofit and MODTRAN configuration files.

aerosol_climatology = None

input_radiance_file

input_loc_file

input_obs_file

working_directory = b'.'

full_lut_directory = b'.'

surface_path

lut_h2o_directory = b'.'

config_directory = b'.'

data_directory = b'.'

input_data_directory = b'.'

output_directory = b'.'

rfl_working_path = b'.'

uncert_working_path = b'.'

lbl_working_path = b'.'

state_working_path = b'.'

surface_template_path = b'.'

surface_working_paths

channelized_uncertainty_working_path = b'.'

model_discrepancy_working_path = b'.'

svf_subs_path = b'.'

rdn_subs_path = b'.'

obs_subs_path = b'.'

loc_subs_path = b'.'

rfl_subs_path = b'.'

atm_coeff_path = b'.'

state_subs_path = b'.'

uncert_subs_path = b'.'

h2o_subs_path = b'.'

wavelength_path = b'.'

modtran_template_path = b'.'

h2o_template_path = b'.'

isofit_full_config_path = b'.'

h2o_config_path = b'.'

sixs_path

earth_sun_distance_path = ''

irradiance_file = ''

aerosol_tpl_path = ''

rdn_factors_path = None

ray_temp_dir

make_directories()

Build required subdirectories inside working_directory

stage_files()

Stage data files by copying into working directory

class SerialEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: json.JSONEncoder

Encoder for json to help ensure json objects can be passed to the workflow manager.

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

class LUTConfig(lut_config_file: str = None, emulator: str = None, no_min_lut_spacing: bool = False, atmosphere_type='ATM_MIDLAT_SUMMER')

A look up table class, containing default grid options. All properties may be overridden with the optional

input configuration file path

Parameters:

• lut_config_file – configuration file to override default values

• emulator – emulator used - will modify required points appropriately

• no_min_lut_spacing – span all LUT dimensions with at least 2 points

elevation_spacing = 0.25

elevation_spacing_min = 0.2

h2o_spacing = 0.25

h2o_spacing_min = 0.03

h2o_min = 0.2

h2o_range = [0.2, 5]

to_sensor_zenith_spacing = 10

to_sensor_zenith_spacing_min = 2

to_sun_zenith_spacing = 10

to_sun_zenith_spacing_min = 2

relative_azimuth_spacing = 60

relative_azimuth_spacing_min = 25

aerosol_0_spacing = 0

aerosol_0_spacing_min = 0

aerosol_1_spacing = 0

aerosol_1_spacing_min = 0

aerosol_2_spacing = 0.1

aerosol_2_spacing_min = 0

aerosol_0_range

aerosol_1_range

aerosol_2_range

aot_550_range

aot_550_spacing = 0

aot_550_spacing_min = 0

co2_range = [380, 440]

co2_spacing = 60

co2_spacing_min = 60

no_min_lut_spacing = False

get_grid_with_data(data_input: numpy.array, spacing: float, min_spacing: float)

get_grid(minval: float, maxval: float, spacing: float, min_spacing: float)

class SerialEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)

Bases: json.JSONEncoder

Encoder for json to help ensure json objects can be passed to the workflow manager.

default(obj)

Implement this method in a subclass such that it returns a serializable object for o, or calls the base implementation (to raise a TypeError).

For example, to support arbitrary iterators, you could implement default like this:

def default(self, o):
    try:
        iterable = iter(o)
    except TypeError:
        pass
    else:
        return list(iterable)
    # Let the base class default method raise the TypeError
    return JSONEncoder.default(self, o)

check_surface_model(surface_path: str, output_model_path: str = None, wl: numpy.array = [], surface_wavelength_path: str = '', surface_category: str = 'multicomponent_surface', multisurface: bool = False) → str

Checks and rebuilds surface model if needed.

TODO - Could be extended to allow for both dir and file surface_path

inputs. The dir inputs could accomdate complex surfaces where different surface priors might be useful. This extension would be relatively easy. Wrap this in a check for surface_path vs. surface_path_dir. Each file in the dir can undergo the same check coded here.

Parameters:

• surface_path – path to surface model or config dict

• wl – instrument center wavelengths

• surface_wavelength_path – path to wavelength file

build_presolve_config(paths: Pathnames, h2o_lut_grid: numpy.array, n_cores: int = -1, use_superpixels: bool = False, surface_category='multicomponent_surface', emulator_base: str = None, uncorrelated_radiometric_uncertainty: float = 0.0, dn_uncertainty_file: str = None, segmentation_size: int = 400, debug: bool = False, inversion_windows=[[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]], prebuilt_lut_path: str = None, multipart_transmittance: bool = False) → None

Write an isofit config file for a presolve, with limited info.

Parameters:

• paths – object containing references to all relevant file locations

• h2o_lut_grid – the water vapor look up table grid isofit should use for this solve

• n_cores – number of cores to use in processing

• use_superpixels – flag whether or not to use superpixels for the solution

• surface_category – type of surface to use

• emulator_base – the basename of the emulator, if used

• uncorrelated_radiometric_uncertainty – uncorrelated radiometric uncertainty parameter for isofit

• dn_uncertainty_file – Path to a linearity .mat file to augment S matrix with linearity uncertainty

• segmentation_size – image segmentation size if empirical line is used

• debug – flag to enable debug_mode in the config.implementation

• prebuilt_lut_path – lut path to use; if none, presolve config will create a new file

• multipart_transmittance – flag to indicate whether a 4-component transmittance model is to be used

build_main_config(paths: Pathnames, lut_params: LUTConfig, h2o_lut_grid: numpy.array = None, elevation_lut_grid: numpy.array = None, to_sensor_zenith_lut_grid: numpy.array = None, to_sun_zenith_lut_grid: numpy.array = None, relative_azimuth_lut_grid: numpy.array = None, mean_latitude: float = None, mean_longitude: float = None, dt: datetime.datetime = None, use_superpixels: bool = True, n_cores: int = -1, surface_category='multicomponent_surface', emulator_base: str = None, uncorrelated_radiometric_uncertainty: float = 0.0, dn_uncertainty_file: str = None, multiple_restarts: bool = False, segmentation_size=400, pressure_elevation: bool = False, debug: bool = False, inversion_windows=[[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]], prebuilt_lut_path: str = None, multipart_transmittance: bool = False, surface_mapping: dict = None, retrieve_co2: bool = False) → None

Write an isofit config file for the main solve, using the specified pathnames and all given info

Parameters:

• paths – object containing references to all relevant file locations

• lut_params – configuration parameters for the lut grid

• h2o_lut_grid – the water vapor look up table grid isofit should use for this solve

• elevation_lut_grid – the ground elevation look up table grid isofit should use for this solve

• to_sensor_zenith_lut_grid – the to-sensor zenith angle look up table grid isofit should use for this solve

• to_sun_zenith_lut_grid – the to-sun zenith angle look up table grid isofit should use for this solve

• relative_azimuth_lut_grid – the relative to-sun azimuth angle look up table grid isofit should use for this solve

• mean_latitude – the latitude isofit should use for this solve

• mean_longitude – the longitude isofit should use for this solve

• dt – the datetime object corresponding to this flightline to use for this solve

• use_superpixels – flag whether or not to use superpixels for the solution

• n_cores – the number of cores to use during processing

• surface_category – type of surface to use

• emulator_base – the basename of the emulator, if used

• uncorrelated_radiometric_uncertainty – uncorrelated radiometric uncertainty parameter for isofit

• dn_uncertainty_file – Path to a linearity .mat file to augment S matrix with linearity uncertainty

• multiple_restarts – if true, use multiple restarts

• segmentation_size – image segmentation size if empirical line is used

• pressure_elevation – if true, retrieve pressure elevation

• debug – if true, run ISOFIT in debug mode

• multipart_transmittance – flag to indicate whether a 4-component transmittance model is to be used

• surface_mapping – optional object to pass mapping between surface class and surface model

• retrieve_co2 – flag to include CO2 in lut and retrieval

get_lut_subset(vals)

Populate lut_names for the appropriate style of subsetting

Parameters:

vals – the values to use for subsetting

write_modtran_template(atmosphere_type: str, fid: str, altitude_km: float, dayofyear: int, to_sensor_azimuth: float, to_sensor_zenith: float, to_sun_zenith: float, relative_azimuth: float, gmtime: float, elevation_km: float, output_file: str, ihaze_type: str = 'AER_RURAL')

Write a MODTRAN template file for use by isofit look up tables

Parameters:

• atmosphere_type – label for the type of atmospheric profile to use in modtran

• fid – flight line id (name)

• altitude_km – altitude of the sensor in km

• dayofyear – the current day of the given year

• to_sensor_azimuth – azimuth view angle to the sensor, in degrees

• to_sensor_zenith – sensor/observer zenith angle, in degrees

• to_sun_zenith – final altitude solar zenith angle (0→180°)

• relative_azimuth – final altitude relative solar azimuth (0→360°)

• gmtime – greenwich mean time

• elevation_km – elevation of the land surface in km

• output_file – location to write the modtran template file to

• ihaze_type – type of extinction and default meteorological range for the boundary-layer aerosol model

load_climatology(config_path: str, latitude: float, longitude: float, acquisition_datetime: datetime.datetime, lut_params: LUTConfig)

Load climatology data, based on location and configuration

Parameters:

• config_path – path to the base configuration directory for isofit

• latitude – latitude to set for the segment (mean of acquisition suggested)

• longitude – latitude to set for the segment (mean of acquisition suggested)

• acquisition_datetime – datetime to use for the segment( mean of acquisition suggested)

• lut_params – parameters to use to define lut grid

:Returns

tuple containing:

aerosol_state_vector - A dictionary that defines the aerosol state vectors for isofit aerosol_lut_grid - A dictionary of the aerosol lookup table (lut) grid to be explored aerosol_model_path - A path to the location of the aerosol model to use with MODTRAN.

calc_modtran_max_water(paths: Pathnames) → float

MODTRAN may put a ceiling on “legal” H2O concentrations. This function calculates that ceiling. The intended

use is to make sure the LUT does not contain useless gridpoints above it.

Parameters:

paths – object containing references to all relevant file locations

Returns:

max_water - maximum MODTRAN H2OSTR value for provided obs conditions

define_surface_types(tsip: dict, rdnfile: str, obsfile: str, out_class_path: str, wl: numpy.array, fwhm: numpy.array)

copy_file_subset(matching_indices: numpy.array, pathnames: List)

Copy over subsets of given files to new locations

Parameters:

• matching_indices (np.array) – indices to select from (y dimension) from source dataset

• pathnames (List) – list of tuples (input_filename, output_filename) to read/write to/from

get_metadata_from_obs(obs_file: str, lut_params: LUTConfig, trim_lines: int = 5, max_flight_duration_h: int = 8, nodata_value: float = -9999)

Get metadata needed for complete runs from the observation file (bands: path length, to-sensor azimuth, to-sensor zenith, to-sun azimuth, to-sun zenith, phase, slope, aspect, cosine i, UTC time).

Parameters:

• obs_file – file name to pull data from

• lut_params – parameters to use to define lut grid

• trim_lines – number of lines to ignore at beginning and end of file (good if lines contain values that are erroneous but not nodata

• max_flight_duration_h – maximum length of the current acquisition, used to check if we’ve lapped a UTC day

• nodata_value – value to ignore from location file

Returns:

tuple containing:

h_m_s - list of the mean-time hour, minute, and second within the line increment_day - indicator of whether the UTC day has been changed since the beginning of the line time mean_path_km - mean distance between sensor and ground in km for good data mean_to_sensor_azimuth - mean to-sensor azimuth for good data mean_to_sensor_zenith - mean to-sensor zenith for good data mean_to_sun_azimuth - mean to-sun-azimuth for good data mean_to_sun_zenith - mean to-sun zenith for good data mean_relative_azimuth - mean relative to-sun azimuth for good data valid - boolean array indicating which pixels were NOT nodata to_sensor_zenith_lut_grid - the to-sensor zenith look up table grid for good data to_sun_zenith_lut_grid - the to-sun zenith look up table grid for good data relative_azimuth_lut_grid - the relative to-sun azimuth look up table grid for good data

get_metadata_from_loc(loc_file: str, lut_params: LUTConfig, trim_lines: int = 5, nodata_value: float = -9999, pressure_elevation: bool = False)

Get metadata needed for complete runs from the location file (bands long, lat, elev).

Parameters:

• loc_file – file name to pull data from

• lut_params – parameters to use to define lut grid

• trim_lines – number of lines to ignore at beginning and end of file (good if lines contain values that are erroneous but not nodata

• nodata_value – value to ignore from location file

• pressure_elevation – retrieve pressure elevation (requires expanded ranges)

Returns:

tuple containing:

mean_latitude - mean latitude of good values from the location file mean_longitude - mean latitude of good values from the location file mean_elevation_km - mean ground estimate of good values from the location file elevation_lut_grid - the elevation look up table, based on globals and values from location file

reassemble_cube(matching_indices: numpy.array, paths: Pathnames)

Copy over subsets of given files to new locations

Parameters:

• matching_indices (np.array) – indices to select from (y dimension) from source dataset

• paths (Pathnames) – output file array set

sensor_name_to_dt(sensor: str, fid: str)

get_wavelengths(envi_file: str, wavelength_path: str = None)

Get wavelengths and FWHM from the header of an ENVI file

Parameters:

• envi_file – path to the ENVI file to read wavelengths from

• wavelength_path – optional path to a file containing wavelengths and FWHM

Returns:

wl - array of wavelengths in nm fwhm - array of full width at half maximum in nm

Return type:

tuple containing

write_wavelength_file(filename, wl, fwhm)

Write a wavelength file in isofit-expected format Units can be either nm or microns, but should be the same

Parameters:

• filename – path to the file to write

• wl – array of wavelengths

• fwhm – array of full width at half maximum

Returns:

None

make_surface_config(paths: Pathnames, surface_category='multicomponent_surface', pressure_elevation=None, elevation_lut_grid=[], surface_mapping: dict = None, use_superpixels=False)

Constructs the surface component of the config :param paths: Pathnames object with all key values passed from apply_oe :param surface_category: Base surface category

Returns:

Dictionary with all surface parameters and file

locations.

Return type:

surface_config_dict