15 Manage System Processes in OML4Py Using psutil

If you find that your Python process is consuming too many of your machine's resources, or causing your machine to crash, you can both monitor and set limits on the resources Python is using.

To manage resource consumption by OML4Py, you will use the Python package psutil.

Python Package: psutil

The Python package psutil is a cross-platform library that lets you retrieve information about your system and running processes. With psutil, you can:
  • Monitor how much CPU and memory your program or system is using.
  • Control or limit how many resources your Python code can use.
This makes psutil especially useful for monitoring and managing your Python applications.
  • psutil.Process.rlimit: Use psutil.Process.rlimit to get or set the resource limits of your current Python process. For more information, see psutil.Process.rlimit.

  • psutil.RLIMIT_*: Resource types all start with psutil.RLIMIT_. For more information, see constants. Each limit is controlled by a soft limit and a hard limit tuple. For example:

    psutil.RLIMIT_AS:
    • Controls the maximum amount of virtual memory (RAM) your process can use.
    • The default is often unlimited (-1, or psutil.RLIM_INFINITY).
    • You can change this to a lower value to keep your program from using too much memory.
  • Resource Planning for OML4Py
    Resource planning for OML4Py depends on a combination of factors, including the number of users, the type and amount of data, the operations you perform (such as model building and scoring), and available computational resources.
  • Examples of Managing System Process
    This section describes how to monitor and manage the resources used by OML4Py Python user-defined functions (UDFs). It demonstrates how to retrieve the current process, measure resource utilization before and after UDF execution, and use the psutil library to configure resource limits for the current process.