• Factors Influencing Resources

  Resources for OML4Py depends on

  • Number of Concurrent Users: More users running simultaneous workloads increase resource demands.
  • Type and Mix of Data: Different data types and varied data mixes can change processing requirements.
  • Operations in Use: Activities such as data preparation, in-database model building, and scoring, as well as user-defined functions (UDF) via embedded Python, impact resources
  • Algorithm and Data Complexity: Even when using the same algorithm on similarly sized datasets, varying data patterns and algorithm complexity can lead to significant differences in execution times.

• Determining Resource Requirements

  When estimating resources, consider:

  • Resource Requirements of Code and Data Volume: Evaluate your Python code’s memory and processing needs, along with the size of your datasets (rows and columns).

  • Available Computational Resources: Understand your environment—whether you’re using dedicated or shared systems, and assess available RAM, CPU, or—for Autonomous Database—the number of ECPUs and service level.

  • Parallel vs. Serial Embedded Python Execution: Parallel execution generally offers greater scalability by processing data in manageable chunks rather than all at once.

  • Execution Scheduling: Consider running resource-intensive workflows in batch mode or during off-peak hours.

  • Database Resource Availability: For in-database algorithms, monitor PGA/SGA memory and tablespace; for Autonomous Database, check available ECPUs and autoscaling settings.

• Embedded Python Execution

  For embedded Python execution:

  • Data Fit in Memory: All processed data must fit into the memory of the Python engine. Exceeding the memory or processing capabilities can result in failures or degraded performance.

  • Parallel Processing Interfaces: OML4Py supports parallel user-defined functions across your data:

    • oml.row_apply: Executes functions in parallel on single rows or row chunks.

    • oml.group_apply: Enables group-based parallel execution using a grouping column.

    • oml.index_apply: Simulates parallel processing by executing scripts a specified number of times.

    Note:

    Embedded Python execution does not automatically modify third-party code for parallelism or scalability. Each data group or chunk processed must fit entirely into a single Python engine.

• Monitoring and Setting Resource Limits

  You can monitor and control resource usage for your user-defined functions using Python’s psutil library:

  • System Monitoring: psutil helps track CPU and memory utilization of running processes.

  • Resource Limit Management:psutil.Process.rlimit allows you to get or set process resource limits, using constants beginning with `psutil.RLIMIT_`.

  • Soft Limit and Hard Limit: Each limit is a tuple of a soft and a hard value.

  • Example: Use psutil.RLIMIT_AS to restrict the maximum virtual memory a process can use. The default is unlimited (-1), but you can lower this value to prevent Python programs from over-consuming memory.

Effective resource management requires careful analysis of user concurrency, data size and type, processing complexity, available computational resources, and method of execution. Actively monitoring system usage with tools like psutil helps maintain stability and prevent resource overuse.