isofit.utils.template_construction ================================== .. py:module:: isofit.utils.template_construction Classes ------- .. autoapisummary:: isofit.utils.template_construction.Pathnames isofit.utils.template_construction.SerialEncoder isofit.utils.template_construction.LUTConfig isofit.utils.template_construction.SerialEncoder Functions --------- .. autoapisummary:: isofit.utils.template_construction.check_surface_model isofit.utils.template_construction.build_presolve_config isofit.utils.template_construction.build_main_config isofit.utils.template_construction.get_lut_subset isofit.utils.template_construction.write_modtran_template isofit.utils.template_construction.load_climatology isofit.utils.template_construction.calc_modtran_max_water isofit.utils.template_construction.define_surface_types isofit.utils.template_construction.copy_file_subset isofit.utils.template_construction.get_metadata_from_obs isofit.utils.template_construction.get_metadata_from_loc isofit.utils.template_construction.reassemble_cube isofit.utils.template_construction.sensor_name_to_dt isofit.utils.template_construction.get_wavelengths isofit.utils.template_construction.write_wavelength_file isofit.utils.template_construction.make_surface_config Module Contents --------------- .. py:class:: Pathnames(input_radiance, input_loc, input_obs, surface_class_file, surface_path, working_directory, ray_temp_dir, sensor='NA-*', copy_input_files=False, modtran_path=None, rdn_factors_path=None, model_discrepancy_path=None, aerosol_climatology_path=None, channelized_uncertainty_path=None, interpolate_inplace=False, skyview_factor=None, subs: bool = False, classify_multisurface: bool = False) Class to determine and hold the large number of relative and absolute paths that are needed for isofit and MODTRAN configuration files. .. py:attribute:: aerosol_climatology :value: None .. py:attribute:: input_radiance_file .. py:attribute:: input_loc_file .. py:attribute:: input_obs_file .. py:attribute:: working_directory :value: b'.' .. py:attribute:: full_lut_directory :value: b'.' .. py:attribute:: surface_path .. py:attribute:: lut_h2o_directory :value: b'.' .. py:attribute:: config_directory :value: b'.' .. py:attribute:: data_directory :value: b'.' .. py:attribute:: input_data_directory :value: b'.' .. py:attribute:: output_directory :value: b'.' .. py:attribute:: rfl_working_path :value: b'.' .. py:attribute:: uncert_working_path :value: b'.' .. py:attribute:: lbl_working_path :value: b'.' .. py:attribute:: state_working_path :value: b'.' .. py:attribute:: surface_template_path :value: b'.' .. py:attribute:: surface_working_paths .. py:attribute:: channelized_uncertainty_working_path :value: b'.' .. py:attribute:: model_discrepancy_working_path :value: b'.' .. py:attribute:: svf_subs_path :value: b'.' .. py:attribute:: rdn_subs_path :value: b'.' .. py:attribute:: obs_subs_path :value: b'.' .. py:attribute:: loc_subs_path :value: b'.' .. py:attribute:: rfl_subs_path :value: b'.' .. py:attribute:: atm_coeff_path :value: b'.' .. py:attribute:: state_subs_path :value: b'.' .. py:attribute:: uncert_subs_path :value: b'.' .. py:attribute:: h2o_subs_path :value: b'.' .. py:attribute:: wavelength_path :value: b'.' .. py:attribute:: modtran_template_path :value: b'.' .. py:attribute:: h2o_template_path :value: b'.' .. py:attribute:: isofit_full_config_path :value: b'.' .. py:attribute:: h2o_config_path :value: b'.' .. py:attribute:: sixs_path .. py:attribute:: earth_sun_distance_path :value: '' .. py:attribute:: irradiance_file :value: '' .. py:attribute:: aerosol_tpl_path :value: '' .. py:attribute:: rdn_factors_path :value: None .. py:attribute:: ray_temp_dir .. py:method:: make_directories() Build required subdirectories inside working_directory .. py:method:: stage_files() Stage data files by copying into working directory .. py:class:: SerialEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) Bases: :py:obj:`json.JSONEncoder` Encoder for json to help ensure json objects can be passed to the workflow manager. .. py:method:: default(obj) Implement this method in a subclass such that it returns a serializable object for ``o``, or calls the base implementation (to raise a ``TypeError``). For example, to support arbitrary iterators, you could implement default like this:: def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o) .. py:class:: LUTConfig(lut_config_file: str = None, emulator: str = None, no_min_lut_spacing: bool = False, atmosphere_type='ATM_MIDLAT_SUMMER') A look up table class, containing default grid options. All properties may be overridden with the optional input configuration file path :param lut_config_file: configuration file to override default values :param emulator: emulator used - will modify required points appropriately :param no_min_lut_spacing: span all LUT dimensions with at least 2 points .. py:attribute:: elevation_spacing :value: 0.25 .. py:attribute:: elevation_spacing_min :value: 0.2 .. py:attribute:: h2o_spacing :value: 0.25 .. py:attribute:: h2o_spacing_min :value: 0.03 .. py:attribute:: h2o_min :value: 0.2 .. py:attribute:: h2o_range :value: [0.2, 5] .. py:attribute:: to_sensor_zenith_spacing :value: 10 .. py:attribute:: to_sensor_zenith_spacing_min :value: 2 .. py:attribute:: to_sun_zenith_spacing :value: 10 .. py:attribute:: to_sun_zenith_spacing_min :value: 2 .. py:attribute:: relative_azimuth_spacing :value: 60 .. py:attribute:: relative_azimuth_spacing_min :value: 25 .. py:attribute:: aerosol_0_spacing :value: 0 .. py:attribute:: aerosol_0_spacing_min :value: 0 .. py:attribute:: aerosol_1_spacing :value: 0 .. py:attribute:: aerosol_1_spacing_min :value: 0 .. py:attribute:: aerosol_2_spacing :value: 0.1 .. py:attribute:: aerosol_2_spacing_min :value: 0 .. py:attribute:: aerosol_0_range .. py:attribute:: aerosol_1_range .. py:attribute:: aerosol_2_range .. py:attribute:: aot_550_range .. py:attribute:: aot_550_spacing :value: 0 .. py:attribute:: aot_550_spacing_min :value: 0 .. py:attribute:: co2_range :value: [380, 440] .. py:attribute:: co2_spacing :value: 60 .. py:attribute:: co2_spacing_min :value: 60 .. py:attribute:: no_min_lut_spacing :value: False .. py:method:: get_grid_with_data(data_input: numpy.array, spacing: float, min_spacing: float) .. py:method:: get_grid(minval: float, maxval: float, spacing: float, min_spacing: float) .. py:class:: SerialEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None) Bases: :py:obj:`json.JSONEncoder` Encoder for json to help ensure json objects can be passed to the workflow manager. .. py:method:: default(obj) Implement this method in a subclass such that it returns a serializable object for ``o``, or calls the base implementation (to raise a ``TypeError``). For example, to support arbitrary iterators, you could implement default like this:: def default(self, o): try: iterable = iter(o) except TypeError: pass else: return list(iterable) # Let the base class default method raise the TypeError return JSONEncoder.default(self, o) .. py:function:: check_surface_model(surface_path: str, output_model_path: str = None, wl: numpy.array = [], surface_wavelength_path: str = '', surface_category: str = 'multicomponent_surface', multisurface: bool = False) -> str Checks and rebuilds surface model if needed. TODO - Could be extended to allow for both dir and file surface_path inputs. The dir inputs could accomdate complex surfaces where different surface priors might be useful. This extension would be relatively easy. Wrap this in a check for surface_path vs. surface_path_dir. Each file in the dir can undergo the same check coded here. :param surface_path: path to surface model or config dict :param wl: instrument center wavelengths :param surface_wavelength_path: path to wavelength file .. py:function:: build_presolve_config(paths: Pathnames, h2o_lut_grid: numpy.array, n_cores: int = -1, use_superpixels: bool = False, surface_category='multicomponent_surface', emulator_base: str = None, uncorrelated_radiometric_uncertainty: float = 0.0, dn_uncertainty_file: str = None, segmentation_size: int = 400, debug: bool = False, inversion_windows=[[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]], prebuilt_lut_path: str = None, multipart_transmittance: bool = False) -> None Write an isofit config file for a presolve, with limited info. :param paths: object containing references to all relevant file locations :param h2o_lut_grid: the water vapor look up table grid isofit should use for this solve :param n_cores: number of cores to use in processing :param use_superpixels: flag whether or not to use superpixels for the solution :param surface_category: type of surface to use :param emulator_base: the basename of the emulator, if used :param uncorrelated_radiometric_uncertainty: uncorrelated radiometric uncertainty parameter for isofit :param dn_uncertainty_file: Path to a linearity .mat file to augment S matrix with linearity uncertainty :param segmentation_size: image segmentation size if empirical line is used :param debug: flag to enable debug_mode in the config.implementation :param prebuilt_lut_path: lut path to use; if none, presolve config will create a new file :param multipart_transmittance: flag to indicate whether a 4-component transmittance model is to be used .. py:function:: build_main_config(paths: Pathnames, lut_params: LUTConfig, h2o_lut_grid: numpy.array = None, elevation_lut_grid: numpy.array = None, to_sensor_zenith_lut_grid: numpy.array = None, to_sun_zenith_lut_grid: numpy.array = None, relative_azimuth_lut_grid: numpy.array = None, mean_latitude: float = None, mean_longitude: float = None, dt: datetime.datetime = None, use_superpixels: bool = True, n_cores: int = -1, surface_category='multicomponent_surface', emulator_base: str = None, uncorrelated_radiometric_uncertainty: float = 0.0, dn_uncertainty_file: str = None, multiple_restarts: bool = False, segmentation_size=400, pressure_elevation: bool = False, debug: bool = False, inversion_windows=[[350.0, 1360.0], [1410, 1800.0], [1970.0, 2500.0]], prebuilt_lut_path: str = None, multipart_transmittance: bool = False, surface_mapping: dict = None, retrieve_co2: bool = False) -> None Write an isofit config file for the main solve, using the specified pathnames and all given info :param paths: object containing references to all relevant file locations :param lut_params: configuration parameters for the lut grid :param h2o_lut_grid: the water vapor look up table grid isofit should use for this solve :param elevation_lut_grid: the ground elevation look up table grid isofit should use for this solve :param to_sensor_zenith_lut_grid: the to-sensor zenith angle look up table grid isofit should use for this solve :param to_sun_zenith_lut_grid: the to-sun zenith angle look up table grid isofit should use for this solve :param relative_azimuth_lut_grid: the relative to-sun azimuth angle look up table grid isofit should use for this solve :param mean_latitude: the latitude isofit should use for this solve :param mean_longitude: the longitude isofit should use for this solve :param dt: the datetime object corresponding to this flightline to use for this solve :param use_superpixels: flag whether or not to use superpixels for the solution :param n_cores: the number of cores to use during processing :param surface_category: type of surface to use :param emulator_base: the basename of the emulator, if used :param uncorrelated_radiometric_uncertainty: uncorrelated radiometric uncertainty parameter for isofit :param dn_uncertainty_file: Path to a linearity .mat file to augment S matrix with linearity uncertainty :param multiple_restarts: if true, use multiple restarts :param segmentation_size: image segmentation size if empirical line is used :param pressure_elevation: if true, retrieve pressure elevation :param debug: if true, run ISOFIT in debug mode :param multipart_transmittance: flag to indicate whether a 4-component transmittance model is to be used :param surface_mapping: optional object to pass mapping between surface class and surface model :param retrieve_co2: flag to include CO2 in lut and retrieval .. py:function:: get_lut_subset(vals) Populate lut_names for the appropriate style of subsetting :param vals: the values to use for subsetting .. py:function:: write_modtran_template(atmosphere_type: str, fid: str, altitude_km: float, dayofyear: int, to_sensor_azimuth: float, to_sensor_zenith: float, to_sun_zenith: float, relative_azimuth: float, gmtime: float, elevation_km: float, output_file: str, ihaze_type: str = 'AER_RURAL') Write a MODTRAN template file for use by isofit look up tables :param atmosphere_type: label for the type of atmospheric profile to use in modtran :param fid: flight line id (name) :param altitude_km: altitude of the sensor in km :param dayofyear: the current day of the given year :param to_sensor_azimuth: azimuth view angle to the sensor, in degrees :param to_sensor_zenith: sensor/observer zenith angle, in degrees :param to_sun_zenith: final altitude solar zenith angle (0→180°) :param relative_azimuth: final altitude relative solar azimuth (0→360°) :param gmtime: greenwich mean time :param elevation_km: elevation of the land surface in km :param output_file: location to write the modtran template file to :param ihaze_type: type of extinction and default meteorological range for the boundary-layer aerosol model .. py:function:: load_climatology(config_path: str, latitude: float, longitude: float, acquisition_datetime: datetime.datetime, lut_params: LUTConfig) Load climatology data, based on location and configuration :param config_path: path to the base configuration directory for isofit :param latitude: latitude to set for the segment (mean of acquisition suggested) :param longitude: latitude to set for the segment (mean of acquisition suggested) :param acquisition_datetime: datetime to use for the segment( mean of acquisition suggested) :param lut_params: parameters to use to define lut grid :Returns tuple containing: aerosol_state_vector - A dictionary that defines the aerosol state vectors for isofit aerosol_lut_grid - A dictionary of the aerosol lookup table (lut) grid to be explored aerosol_model_path - A path to the location of the aerosol model to use with MODTRAN. .. py:function:: calc_modtran_max_water(paths: Pathnames) -> float MODTRAN may put a ceiling on "legal" H2O concentrations. This function calculates that ceiling. The intended use is to make sure the LUT does not contain useless gridpoints above it. :param paths: object containing references to all relevant file locations :returns: max_water - maximum MODTRAN H2OSTR value for provided obs conditions .. py:function:: define_surface_types(tsip: dict, rdnfile: str, obsfile: str, out_class_path: str, wl: numpy.array, fwhm: numpy.array) .. py:function:: copy_file_subset(matching_indices: numpy.array, pathnames: List) Copy over subsets of given files to new locations :param matching_indices: indices to select from (y dimension) from source dataset :type matching_indices: np.array :param pathnames: list of tuples (input_filename, output_filename) to read/write to/from :type pathnames: List .. py:function:: get_metadata_from_obs(obs_file: str, lut_params: LUTConfig, trim_lines: int = 5, max_flight_duration_h: int = 8, nodata_value: float = -9999) -> (List, bool, float, float, float, numpy.array, List, List) Get metadata needed for complete runs from the observation file (bands: path length, to-sensor azimuth, to-sensor zenith, to-sun azimuth, to-sun zenith, phase, slope, aspect, cosine i, UTC time). :param obs_file: file name to pull data from :param lut_params: parameters to use to define lut grid :param trim_lines: number of lines to ignore at beginning and end of file (good if lines contain values that are erroneous but not nodata :param max_flight_duration_h: maximum length of the current acquisition, used to check if we've lapped a UTC day :param nodata_value: value to ignore from location file :Returns: tuple containing: h_m_s - list of the mean-time hour, minute, and second within the line increment_day - indicator of whether the UTC day has been changed since the beginning of the line time mean_path_km - mean distance between sensor and ground in km for good data mean_to_sensor_azimuth - mean to-sensor azimuth for good data mean_to_sensor_zenith - mean to-sensor zenith for good data mean_to_sun_azimuth - mean to-sun-azimuth for good data mean_to_sun_zenith - mean to-sun zenith for good data mean_relative_azimuth - mean relative to-sun azimuth for good data valid - boolean array indicating which pixels were NOT nodata to_sensor_zenith_lut_grid - the to-sensor zenith look up table grid for good data to_sun_zenith_lut_grid - the to-sun zenith look up table grid for good data relative_azimuth_lut_grid - the relative to-sun azimuth look up table grid for good data .. py:function:: get_metadata_from_loc(loc_file: str, lut_params: LUTConfig, trim_lines: int = 5, nodata_value: float = -9999, pressure_elevation: bool = False) -> (float, float, float, numpy.array) Get metadata needed for complete runs from the location file (bands long, lat, elev). :param loc_file: file name to pull data from :param lut_params: parameters to use to define lut grid :param trim_lines: number of lines to ignore at beginning and end of file (good if lines contain values that are erroneous but not nodata :param nodata_value: value to ignore from location file :param pressure_elevation: retrieve pressure elevation (requires expanded ranges) :Returns: tuple containing: mean_latitude - mean latitude of good values from the location file mean_longitude - mean latitude of good values from the location file mean_elevation_km - mean ground estimate of good values from the location file elevation_lut_grid - the elevation look up table, based on globals and values from location file .. py:function:: reassemble_cube(matching_indices: numpy.array, paths: Pathnames) Copy over subsets of given files to new locations :param matching_indices: indices to select from (y dimension) from source dataset :type matching_indices: np.array :param paths: output file array set :type paths: Pathnames .. py:function:: sensor_name_to_dt(sensor: str, fid: str) .. py:function:: get_wavelengths(envi_file: str, wavelength_path: str = None) -> (numpy.array, numpy.array) Get wavelengths and FWHM from the header of an ENVI file :param envi_file: path to the ENVI file to read wavelengths from :param wavelength_path: optional path to a file containing wavelengths and FWHM :returns: wl - array of wavelengths in nm fwhm - array of full width at half maximum in nm :rtype: tuple containing .. py:function:: write_wavelength_file(filename, wl, fwhm) Write a wavelength file in isofit-expected format Units can be either nm or microns, but should be the same :param filename: path to the file to write :param wl: array of wavelengths :param fwhm: array of full width at half maximum :returns: None .. py:function:: make_surface_config(paths: Pathnames, surface_category='multicomponent_surface', pressure_elevation=None, elevation_lut_grid=[], surface_mapping: dict = None, use_superpixels=False) Constructs the surface component of the config :param paths: Pathnames object with all key values passed from apply_oe :param surface_category: Base surface category :returns: Dictionary with all surface parameters and file locations. :rtype: surface_config_dict