Monitoring, Tuning, and Troubleshooting Overview

Getting Started: Your System Stack

Before jumping into specific monitoring, tuning, and troubleshooting topics for Converged Application Server, you should step back and consider your entire system stack, from the lowest level components to the highest:

At the base of your system stack is the underlying hardware, or, in a virtualized environment, your virtual machine or hypervisor.
The next step up is your operating system.
Sitting on top of the operating system is your Java Virtual Machine (JVM).
Running within the JVM, is your Converged Application Server, based itself upon the WebLogic application server, and, within that, your Session Initiation Protocol (SIP) applications themselves.

While you may be most concerned about the performance of your SIP applications, you need to make sure that all levels of your system stack are optimized and running at peak performance. For instance, if your operating system is misconfigured, no amount of WebLogic tuning can improve the reliability of your SIP applications. Likewise, system hardware defects will stop everything dead in its tracks.

Once you know that the base of your system stack is stable, that you've got ample disk space and RAM, that your operating system is patched and not running any extraneous services, only then should you proceed up the stack with tuning recommendations. With a stable base, you can be certain that performance issues, should they occur, are localized to a particular top level software component.

The following sections will go through monitoring, tuning, and troubleshooting considerations for each level of the system stack, from lowest level to highest.

Hardware/VM Monitoring, Tuning, and Troubleshooting

Hardware or virtual machine monitoring and tuning is entirely dependent on your environment. Depending upon your requirements, some things to keep in mind include:

General climate control monitoring for physical servers including temperature and humidity
Temperature monitoring for CPU and power supplies
Enclosure fan speed monitoring
Hardware reliability elements such as error correcting RAM and RAID configurations as well as manageable network interface cards

Operating System and CPU Monitoring, Tuning, and Troubleshooting

Converged Application Server is certified to run on either Oracle Linux or Solaris operating systems, and there are many resources available covering monitoring, tuning and troubleshooting topics.

For both Oracle Linux and Solaris, Oracle provides OSWatcher Black Box which is useful for OS and CPU monitoring. Oracle OSWatcher Black Box (OSWbb) collects and archives operating system and network metrics that you can use to diagnose performance issues. OSWbb operates as a set of background processes on the server and gathers data on a regular basis, invoking such Unix utilities as vmstat, netstat, iostat, and top. For information on installing and using OSWbb, see "Working with OSWatcher Black Box" in Oracle Linux Monitoring and Tuning the System. Note that OSWbb is also compatible with Solaris.

For Linux tuning and troubleshooting, see Monitoring the System and Optimizing Performance.

Operating System Tuning Recommendations

In addition to tuning guidelines provided by your operating system documentation, you can use these guidelines to improve your Converged Application Server system performance.

Use the latest compatible OS version and network adapters.

Enlarge the maximum file descriptor and the number of user processes by editing /etc/security/limits.conf and adding entries like the following:

#<domain>   <type>   <item>   <value>
{user}      soft     nofile   8192
{user}      hard     nofile   80000
{user}      soft     nproc    8192
{user}      hard     nproc    80000

Tune the network for high throughput. For example, in /etc/sysctl.conf, increasing the socket buffer sizes. For example:
```
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 87380 134217728
net.ipv4.tcp_wmem = 4096 87380 134217728
net.core.netdev_max_backlog = 30000
net.ipv4.ip_local_port_range = 1024 65500
```
If you make any changes to sysctl.conf, you need to run sysctl -p for the changes to take effect.
By default, Coherence sends many, smaller packages instead of larger ones. In /etc/sysctl.conf, you can increase the UDP MTU size limit. For example:
```
# Enable jumbo frames
ifconfig eno1 mtu 9000 up
```
The default value is 1500 bytes.
By default, CPU scaling does not take advantage of all available processing power. Run the following command at the command line:
```
echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null
```
Check the CPU scaling with grep "cpu MHz" /proc/cpuinfo.
Configure network interrupt hashing. Run the following command at the command line:
```
ethtool -N eno1 rx-flow-hash udp4 sdfn
```
Check the CPU scaling with ethtool -n eno1 rx-flow-hash udp4.
Make sure all your Converged Application Server servers are connected to the NTP server and are synchronized.

JVM Monitoring, Tuning and Troubleshooting

Since Converged Application Server itself runs within a JVM, you need to make sure that the JVM is correctly tuned and that you monitor it for any issues. For more information, see Monitoring, Tuning, and Troubleshooting the JVM. In that chapter, you'll find the following information:

Converged Application Server Monitoring, Tuning, and Troubleshooting

Next, you can attend to your Converged Application Server environment.

You can use Simple Networking Management Protocol (SNMP) to monitor your Converged Application Server environment. For information on enabling and using SNMP see Configuring Converged Application Server SNMP. In that chapter you'll find the following information:

Converged Application Server debugging and tuning topics are covered in Converged Application Server Debugging and Tuning, including the following topics:

Converged Application Server provides a monitoring console for SIP applications as well as flexible overload protection facilities which let you intercept and deal with SIP application performance issues before they threaten the stability of your environment. For more information, see Converged Application Server Monitoring and Overload Protection, which covers the following topics:

Converged Application Server offers flexible logging configuration which is helpful when debugging SIP application issues. See Logging SIP Requests and Responses, which covers the following topics:

For troubleshooting general WebLogic messages, see BEA-000001 to BEA-2163006 in Oracle Fusion Middleware Error Messages.