1. Oracle Server X9-2L Service Manual
  2. Servicing the DIMMs (CRU)

9 Servicing the DIMMs (CRU)

This section describes how to service memory modules (DIMMs). DIMMs are customer-replaceable units (CRUs) that require you to power off the server. For more information about CRUs, see Illustrated Parts Breakdown and Customer-Replaceable Units.

Caution:

These procedures require that you handle components that are sensitive to electrostatic discharge. This sensitivity can cause the components to fail. To avoid damage, ensure that you follow antistatic practices as described in Electrostatic Discharge Safety.

Caution:

Ensure that all power is removed from the server before removing or installing DIMMs, or damage to the DIMMs might occur. You must disconnect all power cables from the system before performing these procedures.

The following topics and procedures provide information to assist you when replacing a DIMM or upgrading DIMMs:

Servicing Processors (FRU)

DIMM and Processor Physical Layout

Each processor, P0 and P1, has sixteen DIMM slots organized into eight memory channels. Each memory channel contains two DIMM slots: a black DIMM slot (channel slot 0) and a white DIMM slot (channel slot 1).

The physical layout of the DIMMs and processor(s) is shown in the following figure. When viewing the server from the front, processor 0 (P0) is on the left.

Memory Channels DIMM Slot 0 (Black) DIMM Slot 1 (White)

0

D13

D12

1

D15

D14

2

D9

D8

3

D11

D10

4

D2

D3

5

D0

D1

6

D6

D7

7

D4

D5


Figure showing the DIMM and processor layout.

Note:

In single-processor systems, the DIMM slots associated with processor 1 (P1) are nonfunctional and should not be populated with DIMMs.

DIMM Population Scenarios

There are two scenarios in which you are required to populate DIMMs:

  • A DIMM fails and needs to be replaced.

    In this scenario, you can use the Fault Remind button to determine the failed DIMM, then remove the failed DIMM and replace it. To ensure that system performance is maintained, you must replace the failed DIMM with a DIMM of the same size (in gigabytes) and type (quad-rank or dual-rank). In this scenario, you should not change the DIMM configuration.

  • You have purchased new DIMMs and you want to use them to upgrade the server memory.

    In this scenario, you must adhere to the DIMM population rules and follow the recommended DIMM population order for optimal system performance.

DIMM Population Rules

The population rules for adding DIMMs to the server are as follows:

  • The server supports 64-GB and 32-GB dual-rank Registered DIMMs (RDIMMs).

  • Each memory channel is composed of a black slot (channel slot 0) and a white slot (channel slot 1). Populate the black slot first, and then populate the white slot.

    Note:

    The black slot for each channel must be populated first because it is considered furthest away from the processor.

  • The server supports either one DIMM per channel (1DPC) or two DIMMs per channel (2DPC).

  • The server operates properly with a minimum of one DIMM installed per processor.

  • The server does not support lockstep memory mode, which is also known as double device data correction, or Extended ECC.

  • Mixing of single-rank and dual-rank DIMMs in Oracle Server X9-2L is not supported.

Populating DIMMs for Optimal System Performance

Optimal performance is generally achieved by populating the DIMMs so that the memory is symmetrical, or balanced. Symmetry is achieved by adhering to the following guidelines:

  • In single-processor systems, populate DIMMs of the same size in multiples of eight.

  • In dual-processor systems, populate DIMMs of the same size in multiples of sixteen.

  • Populate the DIMM slots in the order described in the following sections, which provide an example of how to populate the DIMM slots to achieve optimal system performance.

Note:

Not all possible configurations are provided in this section.

Populating DIMMs in Single-Processor Systems for Optimal System Performance

In single-processor systems, install DIMMs only into DIMM slots associated with processor 0 (P0). Starting with slot P0 D0, first fill the black slots, and then fill the white slots, as shown in the following figure.


Figure showing the DIMM population order for single-processor systems.

The following table describes the proper order in which to populate DIMMs in a single-processor system using the numbered callouts in the above figure, and the DIMM slot labels (D0 through D15).

Population Order Processor/DIMM Slot

Populate black slots first in the following order:

  1. P0/D0

  2. P0/D15

  3. P0/D4

  4. P0/D11

  5. P0/D2

  6. P0/D13

  7. P0/D6

  8. P0/D9

After black slots have been populated, populate white slots in the following order:

  1. P0/D1

  2. P0/D14

  3. P0/D3

  4. P0/D12

  5. P0/D5

  6. P0/D10

  7. P0/D7

  8. P0/D8

Populating DIMMs in Dual-Processor Systems for Optimal System Performance

In dual-processor systems, populate DIMMs into DIMM slots starting with processor 0 (P0) D7, alternating between slots associated with processor 0 (P0) and matching slots for processor 1 (P1). Fill the black slots, and then the white slots, as shown in the following figure.


Figure showing the DIMM population order for dual-processor systems.

The following table describes the proper order in which to install DIMMs in a dual-processor system using the numbered callouts in the above figure, and the DIMM slot labels (D0 through D15).

Population Order Processor/DIMM Slot

Populate black slots first in the following order:

  1. P0/D0

  2. P1/D0

  3. P0/D15

  4. P1/D15

  5. P0/D4

  6. P1/D4

  7. P0/D11

  8. P1/D11

  9. P0/D2

  10. P1/D2

  11. P0/D13

  12. P1/D13

  13. P0/D6

  14. P1/D6

  15. P0/D9

  16. P1/D9

After black slots have been populated, populate white slots in the following order:

  1. P0/D1

  2. P1/D1

  3. P0/D14

  4. P1/D14

  5. P0/D3

  6. P1/D3

  7. P0/D12

  8. P1/D12

  9. P0/D5

  10. P1/D5

  11. P0/D10

  12. P1/D10

  13. P0/D7

  14. P1/D7

  15. P0/D8

  16. P1/D8

The DIMM population for each processor (P0 and P1) must be identical.

Table 9-1 DIMM Memory Slot Population Requirements

DIMMs per CPU Channel Population Order DIMM Memory Slots CPU-0/1

16 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  X  X  X  X  X  X  X  X  X  X   X   X   X   X   X

12 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  X  X  X  X  X  E  E  E  E  X   X   X   X   X   X

8 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  E  X  E  X  E  X  E  E  X  E   X   E   X   E   X

6 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  E  X  E  X  E  E  E  E  E  E   X   E   X   E   X

4 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  E  E  E  X  E  E  E  E  E  E   X   E   E   E   X

2 DIMMs per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  E  E  E  E  E  E  E  E  E  E   E   E   E   E   E

1 DIMM per CPU

Number

Installed

 
D0 D1 D2 D3 D4 D5 D6 D7 D8 D9 D10 D11 D12 D13 D14 D15
X  E  E  E  E  E  E  E  E  E  E   E   E   E   E   E

DIMM Operating Speeds

The maximum supported memory speed is 3200 MT/s. However, not all system configurations support operation at this speed. The maximum attainable memory speed is limited by the maximum speed supported by the specific type of processor. All memory installed in the system operates at the same speed, or frequency.

DIMM Rank Classification Labels

DIMMs come in a variety of ranks: dual or quad. Each DIMM is shipped with a label identifying its rank classification. The following table identifies the label corresponding to each DIMM rank classification:

Table 9-2 DIMM Rank Classification Labels

Rank Classification Label

Quad-rank LRDIMM

4Rx4

Dual-rank RDIMM

2Rx4

Inconsistencies Between DIMM Fault Indicators and the BIOS Isolation of Failed DIMMs

When a single DIMM is marked as failed by Oracle ILOM (for example, fault.memory.intel.dimm.training-failed is listed in the service processor Oracle ILOM event log), BIOS might disable the entire memory channel that contains the failed DIMM, up to two DIMMs. As a result, none of the memory installed in the disabled channel is available to the operating system. However, when the Fault Remind button is pressed, only the fault status indicator (LED) associated with the failed DIMM lights. The fault LEDs for the other DIMMs in the memory channel remain off. Therefore, you can correctly identify the failed DIMM using the lit LED.

Using the Server Fault Remind Button

When the server Fault Remind button [1] is pressed, an LED located next to the Fault Remind button lights green to indicate that there is sufficient voltage present in the fault remind circuit to light any fault LEDs that were lit due to a component failure. If this LED does not light when you press the Fault Remind button, it is likely that the capacitor powering the fault remind circuit has lost its charge. This can happen if the Fault Remind button is pressed for several minutes with fault LEDs lit or if power has been removed from the server for more than 15 minutes.

The following figure shows the location of the Fault Remind button.


Figure showing the location of the Fault Remind Button.

Identify and Remove a DIMM

  1. Prepare the server for service.
    1. Power off the server, and disconnect the power cords from the server power supplies.
    2. Extend the server into the maintenance position.
    3. Attach an antistatic wrist strap to your wrist and then to a metal area on the chassis.
    4. Remove the server top cover.
    5. Remove the air baffle.
  2. Identify and note the location of the failed DIMM by pressing the Fault Remind button on the motherboard.

    See Using the Server Fault Remind Button.

    Failed DIMMs are identified with a corresponding amber LED on the motherboard.

    • If the DIMM Fault LED is off, then the DIMM is operating properly.

    • If the DIMM Fault LED is on (amber), then the DIMM is failed and must be replaced [1].


    Figure showing a failed DIMM.
  3. To remove the failed DIMM, do the following:
    1. Rotate both DIMM slot ejectors outward as far as they go.

      The DIMM is partially ejected from the slot.

    2. Carefully lift the DIMM straight up to remove it from the slot.

      Figure showing a memory DIMM being removed from the server.
  4. Replace each failed DIMM with either another DIMM of the same rank size (quad-rank or dual-rank) or leave the slot empty.

    For DIMM replacement instructions, see Install a DIMM.

Install a DIMM

  1. Unpack the replacement DIMM and place it on an antistatic mat.
  2. Ensure that the replacement DIMM matches the size, type, and rank of the DIMM it is replacing.

    Caution:

    You must not replace a dual-rank DIMM with a quad-rank DIMM and vice versa. If you violate this rule, the performance of the server might be adversely affected. For DIMM slot population rules, see DIMM Population Rules.
  3. Install a DIMM.
    1. Ensure that the ejector tabs are in the open position.
    2. Align the notch in the replacement DIMM with the connector key in the connector slot.

      The notch ensures that the DIMM is oriented correctly.

    3. Push the DIMM into the connector slot until the ejector tabs lock the DIMM in place.

      If the DIMM does not easily seat into the connector slot, verify that the notch in the DIMM is aligned with the connector key in the connector slot.

      Caution:

      If the notch is not aligned, damage to the DIMM might occur.

      Figure showing a memory DIMM being installed into the sever.
  4. Repeat Step 3 until all replacement DIMMs are installed.
  5. Return the server to operation:
    1. Install the air baffle.
    2. Install the server top cover.
    3. Return the server to the normal rack position.
    4. Reconnect the power cords and data cables to the server.
    5. Power on the server.
    6. Verify that the power supply AC OK LED is lit.
  6. (Optional) Use Oracle ILOM to clear server DIMM faults.

    DIMM faults are automatically cleared after a new DIMM has been installed. If you need to manually clear DIMM faults, refer to Oracle Integrated Lights Out Manager (ILOM) 5.0 Documentation.

    Note:

    If you want to move a failed DIMM to a new DIMM slot or re-seat a DIMM for troubleshooting, you must first manually clear the associated DIMM fault. Otherwise, the DIMM fault might follow the DIMM to the new slot location or reoccur in the current slot location, causing a false DIMM fault condition.