Coherence recommends keeping heap sizes at 1-8GB per JVM. However, larger heap sizes, up to 20GB, are suitable for some applications where the simplified management of fewer, larger JVMs outweighs the performance benefits of many smaller JVMs. Using multiple cache servers allows a single computer to achieve higher capacities. With Oracle's JVMs, heap sizes beyond 8GB are reasonable, though GC tuning is still advisable to minimize long GC pauses. See Introduction in Java Platform, Standard Edition HotSpot Virtual Machine Garbage Collection Tuning Guide. It is also advisable to run with fixed sized heaps as this generally lowers GC times. See JVM Tuning.

When available Coherence uses the highly concurrent AtomicLong class, which allows concurrent atomic updates to long values without requiring synchronization.

It is suggested to run in server mode to ensure that the stable and highly concurrent version can be used. To run the JVM in server mode include the -server option on the Java command line.

JVMs that experience an OutOfMemoryError can be left in an indeterministic state which can have adverse effects on a cluster. We recommend configuring JVMs to exit upon encountering an OutOfMemoryError instead of allowing the JVM to attempt recovery. Here is the parameter to configure this setting on Sun JVMs:

UNIX:

-XX:OnOutOfMemoryError="kill -9 %p"

Windows:

-XX:OnOutOfMemoryError="taskkill /F /PID %p"

Additionally, it is recommended to configure the JVM to generate a heap dump if an OutOfMemoryError is thrown to assist the investigation into the root cause for the error. Use the following flag to enable this feature on the Sun JVM:

-XX:+HeapDumpOnOutOfMemoryError

JVMs that experience an OutOfMemoryError can be left in an indeterministic state which can have adverse effects on a cluster. We recommend configuring JVMs to exit upon encountering an OutOfMemoryError instead of allowing the JVM to attempt recovery. Here is the parameter to configure this setting on IBM JVMs:

UNIX:

-Xdump:tool:events=throw,filter=java/lang/OutOfMemoryError,exec="kill -9 %pid"

Windows:

-Xdump:tool:events=throw,filter=java/lang/OutOfMemoryError,exec="taskkill /F /PID %pid"

IBM does not recommend fixed size heaps for JVMs. In many cases, it is recommended to use the default for -Xms (in other words, omit this setting and only set -Xmx). See this link for more details:

http://www.ibm.com/developerworks/java/jdk/diagnosis/

It is recommended to configure the JVM to generate a heap dump if an OutOfMemoryError is thrown to assist the investigation into the root cause for the error. IBM JVMs generate a heap dump on OutOfMemoryError by default; no further configuration is required.

Linux has several high resolution timesources to choose from, the fastest TSC (Time Stamp Counter) unfortunately is not always reliable. Linux chooses TSC by default, and during startup checks for inconsistencies, if found it switches to a slower safe timesource. The slower time sources can be 10 to 30 times more expensive to query then the TSC timesource, and may have a measurable impact on Coherence performance. For more details on TSC, see

https://lwn.net/Articles/209101/

Coherence and the underlying JVM are not aware of the timesource which the operating system is using. It is suggested that you check your system logs (/var/log/dmesg) to verify that the following is not present.

kernel: Losing too many ticks!
kernel: TSC cannot be used as a timesource.
kernel: Possible reasons for this are:
kernel:   You're running with Speedstep,
kernel:   You don't have DMA enabled for your hard disk (see hdparm),
kernel:   Incorrect TSC synchronization on an SMP system (see dmesg).
kernel: Falling back to a sane timesource now.

As the log messages suggest, this can be caused by a variable rate CPU (SpeedStep), having DMA disabled, or incorrect TSC synchronization on multi CPU computers. If present, it is suggested that you work with your system administrator to identify the cause and allow the TSC timesource to be used.

The default Windows configuration is not optimized for background processes, heavy network loads, and network interruptions. This may be addressed by running the optimize.reg script included in the Coherence installation's bin directory. See Operating System Tuning.

1. Run Registry Editor (regedit)
2. Locate the following registry key

   HKLM\System\CurrentControlSet\Services\Tcpip\Parameters
   

3. Add or reset the following new DWORD value

   Name: DisableDHCPMediaSense
   Value: 1 (boolean)

4. Reboot

OS X defaults to running multicast over IPv6 rather than IPv4. If you run in a mixed IPv6/IPv4 environment, configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line.

Generally, Coherence prefers 2MB or higher buffers, but for OS X this may result in unexpectedly high kernel CPU time, which in turn reduces throughput. For OS X, the suggested buffers size is 768KB, though your own tuning may find a better size. See Configuring the Size of the Packet Buffers in Developing Applications with Oracle Coherence.

When deploying Coherence into environments where the default character set is EBCDIC rather than ASCII, ensure that Coherence configuration files which are loaded from JAR files or off of the classpath are in ASCII format. Configuration files loaded directly from the file system should be stored in the systems native format of EBCDIC.

Under some circumstances, Coherence cluster nodes that run within the same logical partition (LPAR) on z/OS on IBM zSeries cannot communicate with each other. (This problem does not occur on the zSeries when running on Linux.)

The root cause is that z/OS may bind the MulticastSocket that Coherence uses to an automatically-assigned port, but Coherence requires the use of a specific port in order for cluster discovery to operate correctly. (Coherence does explicitly initialize the java.net.MulitcastSocket to use the necessary port, but that information appears to be ignored on z/OS when there is an instance of Coherence running within that same LPAR.)

The solution is to run only one instance of Coherence within a z/OS LPAR; if multiple instances are required, each instance of Coherence should be run in a separate z/OS LPAR. Alternatively, well known addresses may be used. See Using Well Known Addresses in Developing Applications with Oracle Coherence.

AIX 5.2 and above default to running multicast over IPv6 rather than IPv4. If you run in a mixed IPv6/IPv4 environment, configure your JVMs to explicitly use IPv4. This can be done by setting the java.net.preferIPv4Stack system property to true on the Java command line. See the IBM 32-bit SDK for AIX User Guide for details.

Virtualization adds another layer to your network topology and it must be properly configured to support multicast networking. See Configuring Multicast Communication in Developing Applications with Oracle Coherence.

It is less likely that a process running in a virtualized operating system can fully use gigabit Ethernet. This is not specific to Coherence and is visible on most network intensive virtualized applications.

Additional configuration is required to ensure that cache entry backups reside on physically separate hardware. See Specifying a Cluster Member's Identity in Developing Applications with Oracle Coherence.

Some Cisco switches may run out of buffer space and exhibit frequent multi-second communication pauses under heavy packet load some. These communication pauses can be identified by a series of Coherence log messages referencing communication delays with multiple nodes which cannot be attributed to local or remote GCs.

Experienced a 4172 ms communication delay (probable remote GC) with Member(Id=7, Timestamp=2008-09-15 12:15:47.511, Address=xxx.xxx.x.xx:8089, MachineId=13838); 320 packets rescheduled, PauseRate=0.31, Threshold=512

The Cisco 6500 series support configuring the amount of buffer space available to each Ethernet port or ASIC. In high load applications it may be necessary to increase the default buffer space. This can be accomplished by executing:

fabric buffer-reserve high

See Cisco's documentation for additional details on this setting.

Cisco's default switch configuration does not support proper routing of multicast packets between switches due to the use of IGMP snooping. See the Cisco's documentation regarding the issue and solutions.

Some Cisco switches have shown difficulty in maintaining multicast group membership resulting in existing multicast group members being silently removed from the multicast group. This cause a partial communication disconnect for the associated Coherence node(s) and they are forced to leave and rejoin the cluster. This type of outage can most often be identified by the following Coherence log messages indicating that a partial communication problem has been detected.

A potential network configuration problem has been detected. A packet has failed to be delivered (or acknowledged) after 60 seconds, although other packets were acknowledged by the same cluster member (Member(Id=3, Timestamp=Sat Sept 13 12:02:54 EST 2008, Address=192.168.1.101, Port=8088, MachineId=48991)) to this member (Member(Id=1, Timestamp=Sat Sept 13 11:51:11 EST 2008, Address=192.168.1.101, Port=8088, MachineId=49002)) as recently as 5 seconds ago.

To confirm the issue, use the same multicast address and port as the running cluster. If the issue affects a multicast test node, its logs show that it suddenly stopped receiving multicast test messages. See Performing a Multicast Connectivity Test.

The following test logs show the issue:

Test Node 192.168.1.100:

Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:23 GMT 2008: Received test packet 76 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:23 GMT 2008: Sent packet 85. Sun Sept 14 16:44:23 GMT 2008: Received test packet 85 from self. 
Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:25 GMT 2008: Received test packet 77 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:25 GMT 2008: Sent packet 86. 
Sun Sept 14 16:44:25 GMT 2008: Received test packet 86 from self. Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:27 GMT 2008: Received test packet 78 from ip=/192.168.1.101, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:27 GMT 2008: Sent packet 87. 
Sun Sept 14 16:44:27 GMT 2008: Received test packet 87 from self. 
Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:29 GMT 2008: Sent packet 88. 
Sun Sept 14 16:44:29 GMT 2008: Received test packet 88 from self.
 Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:31 GMT 2008: Sent packet 89. 
Sun Sept 14 16:44:31 GMT 2008: Received test packet 89 from self. 
Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ??? 
Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 182.168.1.100: ???

Test Node 192.168.1.101:

Sun Sept 14 16:44:22 GMT 2008: Sent packet 76.
Sun Sept 14 16:44:22 GMT 2008: Received test packet 76 from self. 
Sun Sept 14 16:44:22 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:22 GMT 2008: Received test packet 85 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:23 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? Sun Sept 14 16:44:24 GMT 2008: Sent packet 77.
Sun Sept 14 16:44:24 GMT 2008: Received test packet 77 from self. 
Sun Sept 14 16:44:24 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:24 GMT 2008: Received test packet 86 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:25 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:26 GMT 2008: Sent packet 78.Sun Sept 14 16:44:26 GMT 2008: Received test packet 78 from self. 
Sun Sept 14 16:44:26 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:26 GMT 2008: Received test packet 87 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:27 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:28 GMT 2008: Sent packet 79.
Sun Sept 14 16:44:28 GMT 2008: Received test packet 79 from self. 
Sun Sept 14 16:44:28 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:28 GMT 2008: Received test packet 88 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:29 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:30 GMT 2008: Sent packet 80.
Sun Sept 14 16:44:30 GMT 2008: Received test packet 80 from self. 
Sun Sept 14 16:44:30 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:30 GMT 2008: Received test packet 89 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:31 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:32 GMT 2008: Sent packet 81.Sun Sept 14 16:44:32 GMT 2008: Received test packet 81 from self. 
Sun Sept 14 16:44:32 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:32 GMT 2008: Received test packet 90 from ip=/192.168.1.100, group=/224.3.2.0:32367, ttl=4. 
Sun Sept 14 16:44:33 GMT 2008: Received 83 bytes from a Coherence cluster node at 192.168.1.100: ??? 
Sun Sept 14 16:44:34 GMT 2008: Sent packet 82.

The Cisco 6500 series router may become overloaded if too many packets with a time-to-live (TTL) value of 1 are received. In addition, a low TTL setting may overload single group members. Set the Coherence multicast TTL setting to at least the size of the multicast domain (127 or 255) and make sure that clusters do not use overlapping groups. See Specifying the Multicast Time-to-Live in Developing Applications with Oracle Coherence.

Foundry switches have shown to exhibit difficulty in handing multicast traffic. When deploying on with Foundry switches, ensure that all computers that are part of the Coherence cluster can communicate over multicast. See Performing a Multicast Connectivity Test.

If you encounter issues with multicast you may consider changing your configuration to unicast-only by using the well-known-addresses feature. See Using Well Known Addresses in Developing Applications with Oracle Coherence.

A typical deployment on a BladeCenter may include blades with two NICs where one is used for administration purposes and the other for cluster traffic. By default, the MAC addresses assigned to the blades of a BladeCenter are uniform enough that the first NIC generally has an even MAC address and the second has an odd MAC address. If the BladeCenter's uplink to a central switch also has an even number of channels, then layer 2 (MAC based) load balancing may prevent one set of NICs from making full use of the available uplink bandwidth as they are all bound to either even or odd channels. This issue arises due to the assumption in the switch that MAC addresses are essentially random, which in BladeCenter's is untrue. Remedies to this situation include:

• Use layer 3 (IP based) load balancing (if the IP addresses do not follow the same even/odd pattern).

  • This setting must be applied across all switches carrying cluster traffic.

• Randomize the MAC address assignments by swapping them between the first and second NIC on alternating computers.

  • Linux enables you to change a NIC's MAC address using the ifconfig command.

  • For other operating systems custom tools may be available to perform the same task.