isofit.debug.resource_tracker ============================= .. py:module:: isofit.debug.resource_tracker Classes ------- .. autoapisummary:: isofit.debug.resource_tracker.ResourceTracker isofit.debug.resource_tracker.FileResources Functions --------- .. autoapisummary:: isofit.debug.resource_tracker.stream Module Contents --------------- .. py:class:: ResourceTracker(callback: Callable[[dict], None], interval: float = 2, units: tuple = ('GB', 1024**3), cores: Union[int, Literal['all']] = None, round: Union[bool, int] = 2, summarize: bool = True, allow_unsafe: bool = False) Tracks system resources :param callback: Function to call on each resource refresh. Signature must accept: callable(dict) -> None the first call will contain the non-changing values: used_cores : int Number of CPU cores in use. See the 'cores' parameter for more total_cores : int Total number of cores available on the system via os.cpu_count() sys_mem_total : float Total memory of the system mem_unit : str Unit label that the memory values are in mem_value : float The value used to convert the bytes to the mem_unit. This may be used to reverse the conversion poll_interval : float Resource polling interval timestamp : float The start timestamp of the resource tracker via time.time() all calls afterwards will consist of: pid : int Main process ID name : str Main process name mem : float Main process memory used mem_total : float Approximate total memory of the process (actual + shared) mem_actual : float Approximate private memory in use by the process mem_shared : float Approximate shared memory cpu : float Main process CPU percentage over the interval sys_cpu_per_core : list[float] Per-core usage percentage over the interval timestamp : float Timestamp of the resource record via time.time() status : str Main process status, eg. 'running', 'sleeping' children : list[dict] Information of child processes: pid : int Child process ID name : str Child process name mem_total : float Approximate total memory of the child process (actual + shared) mem_actual : float Approximate private memory in use by the child process mem_shared : float Approximate shared memory cpu : float Child process CPU percentage over the interval status : str Child process status, eg. 'running', 'sleeping' if summarize is enabled, these will also be included: mem_app_total : float Approximate total memory of the main process + children mem_app_actual : float Approximate total private memory over all processes mem_app_shared_avg : float Approximate average shared memory over all processes mem_used : float Memory in use by the system mem_avail : float Remaining available memory, defined as free + reclaimable cpu_avg : float Average CPU percentage calculated as: sum(main + children) / cores sys_cpu : float System-wide CPU percentage over the interval :type callback: Callable[[dict], None] :param interval: Interval frequency in seconds to check resources Must be greater than 0. Values less than 0.1 risk high CPU usage and skewing polled results The CPU usage is calculated as the percentage of CPU used over this interval :type interval: int | float, default=2 :param units: Units to convert the memory values to. Must be in the form of (str, float) where the float is used to divide the bytes values that psutil returns Some possible conversions: - ('b', 1/8) # Convert to bits (multiply by 8) - ('B', 1) # No conversion, leave as the default bytes - ('KB', 1024) # Kilobytes - ('MB', 1024**2) # Megabytes - ('GB', 1024**3) # Gigabytes, default :type units: tuple[str, float], default=("GB", 1024**3) :param cores: Number of cores being used by the source program. This is used for calculating the average CPU percentage. Can be passed 'all' to retrieve the os.cpu_count() :type cores: int | 'all', default=1 :param round: Round the memory variables to this many decimals. Set to False or 0 to disable True will be set to 1 :type round: int | bool, default=2 :param summarize: Includes summary statistics such as the sum of all children :type summarize: bool, default=True :param allow_unsafe: Bypasses the exception and allows unsafe interval values (less than 0.1) Not recommended :type allow_unsafe: bool, default=False .. rubric:: Examples Basic usage:: import time from isofit.debug.resource_tracker import ResourceTracker def myFunction(info: dict): print(info) rt = ResourceTracker(myFunction) rt.start() time.sleep(10) rt.stop() .. py:attribute:: info .. py:attribute:: thread :value: None .. py:attribute:: callback .. py:attribute:: interval :value: 2 .. py:attribute:: cores :value: None .. py:attribute:: round :value: 2 .. py:attribute:: summarize :value: True .. py:method:: _track() System resource tracker intended to be set in a thread .. py:method:: start() Starts the _track function in a thread .. py:method:: is_running() Checks if there is a thread running .. py:method:: stop() Sets the stop event to kill any running threads .. py:class:: FileResources(file: str, /, reset: bool = False, **kwargs) Bases: :py:obj:`ResourceTracker` Subclass of ResourceTracker that writes the polled resources to a json list file :param file: Path to a JSONL file to log resource information to :type file: str :param reset: If the file exists, reset it :type reset: bool, default=False .. rubric:: Examples Basic usage:: import time from isofit.debug.resource_tracker import FileResources fr = ResourceTracker("resources.jsonl", reset=True) fr.start() time.sleep(10) fr.stop() .. seealso:: :obj:`ResourceTracker` .. py:attribute:: file .. py:method:: write(info: dict) -> None Writes the resource information as a JSON object per line :param info: A dictionary containing resource information to log :type info: dict .. py:function:: stream(file: str, sleep: float = 0.2) -> dict Generator that yields parsed JSONL objects from a growing json file produced by FileResources :param file: Path to the JSONL file being written to :type file: str :param sleep: How long to wait (in seconds) between polling for new lines :type sleep: float, default=0.2 :Yields: *dict* -- Parsed JSONL object from each line