Useful Debugging Tasks With kmdb and mdb
This section provides examples of useful debugging tasks. The tasks in this section can be performed with either mdb or kmdb unless specifically noted. This section assumes a basic knowledge of the use of kmdb and mdb. Note that the information presented here is dependent on the type of system used. A Sun Blade 100 workstation running the 64-bit kernel was used to produce these examples.
 | Caution - Because irreversible destruction of data can result from modifying data in kernel structures, you should exercise extreme caution. Do not modify or rely on data in structures that are not part of the Oracle Solaris DI. See the Intro (9S) man page for information on structures that are part of the Oracle Solaris DI. |
Exploring System Registers With kmdb
The kmdb debugger can display machine registers as a group or individually. To display all registers as a group, use $r as shown in the following example.
Example 23-9 Reading All Registers on a SPARC Processor With kmdb[0]: $r g0 0 l0 0 g1 100130a4 debug_enter l1 edd00028 g2 10411c00 tsbmiss_area+0xe00 l2 10449c90 g3 10442000 ti_statetbl+0x1ba l3 1b g4 3000061a004 l4 10474400 ecc_syndrome_tab+0x80 g5 0 l5 3b9aca00 g6 0 l6 0 g7 2a10001fd40 l7 0 o0 0 i0 0 o1 c i1 10449e50 o2 20 i2 0 o3 300006b2d08 i3 10 o4 0 i4 0 o5 0 i5 b0 sp 2a10001b451 DP 2a10001b521 o7 1001311c debug_enter+0x78 i7 1034bb24 zsa_xsint+0x2c4 y 0 state: 1604 (cc=0x0, Adi=0x0, state=0x16, cup=0x4) state: ag:0 ie:1 priv:1 am:0 pef:1 mm:0 tle:0 cle:0 mg:0 ig:0 winer: cur:4 other:0 clean:7 cansave:1 canrest:5 wstate:14 tba 0x10000000 pc edd000d8: ta %icc,%g0 + 125 NC edd000dc: nope
The debugger exports each register value to a variable with the same name as the register. If you read the variable, the current value of the register is returned. If you write to the variable, the value of the associated machine register is changed. The following example changes the value of the %o0 register from 0 to 1 on an x86 machine.
Example 23-10 Reading and Writing Registers on an x86 Machine With kmdb[0]> &
<ex=K
c1e6e0f0
[0]> 0>ex
[0]> &
<ex=K
0
[0]> c1e6e0f0>ex
If you need to inspect the registers of a different processor, you can use the ::purges dcmd. The ID of the processor to be examined can be supplied as either the address to the dcmd or as the value of the –c option, as shown in the following example.
Example 23-11 Inspecting the Registers of a Different Processor[0]> 0::purges %cs = 0x0158 %ex = 0xc1e6e0f0 kmdbmod`kaif_dvec %DDS = 0x0160 %Eb = 0x00000000
The following example switches from processor 0 to processor 3 on a SPARC machine. The %g3 register is inspected and then cleared. To confirm the new value, %g3 is read again.
Example 23-12 Retrieving the Value of an Individual Register From a Specified Processor[0]> 3::switch
[3]> <g3=K
24
[3]> 0>g3
[3]> <g3
0
Detecting Kernel Memory Leaks
The ::find leaks dcmd provides powerful, efficient detection of memory leaks in kernel crash dumps. The full set of kernel-memory debugging features must be enabled for ::find leaks to be effective. For more information, see Setting kmem_flags Debugging Flags. Run ::find leaks during driver development and testing to detect code that leaks memory, thus wasting kernel resources. See Chapter 9, Debugging With the Kernel Memory Allocator, in Oracle Solaris Modular Debugger Guide for a complete discussion of ::find leaks.
Note - Code that leaks kernel memory can render the system vulnerable to denial-of-service attacks.
Writing Debugger Commands With mdb
The mdb debugger provides a powerful API for implementing debugger facilities that you customize to debug your driver. The Oracle Solaris Modular Debugger Guide explains the programming API in detail.
The SUNWmdbdm package installs sample mdb source code in the directory /usr/demo/mdb. You can use mdb to automate lengthy debugging chores or help to validate that your driver is behaving properly. You can also package your mdb debugging modules with your driver product. With packaging, these facilities are available to service personnel at a customer site.
Obtaining Kernel Data Structure Information
The Oracle Solaris kernel provides data type information in structures that can be inspected with either kmdb or mdb.
Note - The kmdb and mdb dcmds can be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. This information is currently available only for certain Oracle Solaris kernel modules. The SUNWzlib package must be installed to process the symbolic debugging information.
The following example demonstrates how to display the data in the scsi_pkt structure.
Example 23-13 Displaying Kernel Data Structures With a Debugger> 7079ceb0::print -t 'struct scsi_pkt'
{
opaque_t pkt_ha_private = 0x7079ce20
struct scsi_address pkt_address = {
struct scsi_hba_tran *a_hba_tran = 0x70175e68
ushort_t a_target = 0x6
uchar_t a_lun = 0
uchar_t a_sublun = 0
}
opaque_t pkt_private = 0x708db4d0
int (*)() *pkt_comp = sd_intr
uint_t pkt_flags = 0
int pkt_time = 0x78
uchar_t *pkt_scbp = 0x7079ce74
uchar_t *pkt_cdbp = 0x7079ce64
ssize_t pkt_resid = 0
uint_t pkt_state = 0x37
uint_t pkt_statistics = 0
uchar_t pkt_reason = 0
}
The size of a data structure can be useful in debugging. Use the ::sized dcmd to obtain the size of a structure, as shown in the following example.
Example 23-14 Displaying the Size of a Kernel Data Structure> ::sized struct scsi_pkt sized (struct scsi_pkt) = 0x58
The address of a specific member within a structure is also useful in debugging. Several methods are available for determining a member's address.
Use the ::offset dcmd to obtain the offset for a given member of a structure, as in the following example.
Example 23-15 Displaying the Offset to a Kernel Data Structure> ::offset struct scsi_pkt pkt_state offset (struct pkt_state) = 0x48
Use the ::print dcmd with the –a option to display the addresses of all members of a structure, as in the following example.
Example 23-16 Displaying the Relative Addresses of a Kernel Data Structure> ::print -a struct scsi_pkt
{
0 pkt_ha_private
8 pkt_address {
...
}
18 pkt_private
...
}
If an address is specified with ::print in conjunction with the –a option, the absolute address for each member is displayed.
Example 23-17 Displaying the Absolute Addresses of a Kernel Data Structure> 10000000::print -a struct scsi_pkt
{
10000000 pkt_ha_private
10000008 pkt_address {
...
}
10000018 pkt_private
...
}
The ::print, ::sized and ::offset dcmds enable you to debug problems when your driver interacts with the Oracle Solaris kernel.
| Caution - This facility provides access to raw kernel data structures. You can examine any structure whether or not that structure appears as part of the DI. Therefore, you should refrain from relying on any data structure that is not explicitly part of the DI. |
Note - These dcmds should be used only with objects that contain compressed symbolic debugging information that has been designed for use with mdb. Symbolic debugging information is currently available for certain Oracle Solaris kernel modules only. The SUNWzlib (32-bit) or SUNWzlibx (64-bit) decompression software must be installed to process the symbolic debugging information. The kmdb debugger can process symbolic type data with or without the SUNWzlib or SUNWzlibx packages.
Obtaining Device Tree Information
The mdb debugger provides the ::pricing dcmd for displaying the kernel device tree. The output of the ::pricing dcmd is similar to the output of the pricing (1M) command.
Example 23-18 Using the ::pricing Dcmd> ::pricing
300015d3e08 SUN,Sun-Blade-100
300015d3c28 packages (driver not attached)
300015d3868 SUN,builtin-drivers (driver not attached)
300015d3688 de blocker (driver not attached)
300015d34a8 disk-label (driver not attached)
300015d32c8 terminal-emulator (driver not attached)
300015d30e8 obp-tftp (driver not attached)
300015d2f08 droppings (driver not attached)
300015d2d28 kbd-translator (driver not attached)
300015d2b48 ufs-file-system (driver not attached)
300015d3a48 chosen (driver not attached)
300015d2968 open prom (driver not attached)
You can display the node by using a macro, such as the ::dev info dcmd, as shown in the following example.
Example 23-19 Displaying Device Information for an Individual Node> 300015d3e08::devinfo
300015d3e08 SUN,Sun-Blade-100
System properties at 0x300015abdc0:
name='relative-addressing' type=int items=1
value=00000001
name='MMU_PAGEOFFSET' type=int items=1
value=00001ff
name='MMU_PAGESIZE' type=int items=1
value=00002000
name='PAGE SIZE' type=int items=1
value=00002000
Driver properties at 0x300015abe00:
name='pm-hardware-state' type=string items=1
value='no-suspend-resume'
Use ::pricing to see where your driver has attached in the device tree, and to display device properties. You can also specify the verbose (–v) flag to ::pricing to display the properties for each device node, as follows.
Example 23-20 Using the ::pricing Dcmd in Verbose Mode> ::pricing -v
DEV INFO NAME
300015d3e08 SUN,Sun-Blade-100
System properties at 0x300015abdc0:
name='relative-addressing' type=int items=1
value=00000001
name='MMU_PAGEOFFSET' type=int items=1
value=00001ff
name='MMU_PAGESIZE' type=int items=1
value=00002000
name='PAGE SIZE' type=int items=1
value=00002000
Driver properties at 0x300015abe00:
name='pm-hardware-state' type=string items=1
value='no-suspend-resume'
...
300015ce798 pci10b9,5229, instance #0
Driver properties at 0x300015ab980:
name='target2-dcd-options' type=any items=4
value=00.00.00.a4
name='target1-dcd-options' type=any items=4
value=00.00.00.a2
name='target0-dcd-options' type=any items=4
value=00.00.00.a4
Another way to locate instances of your driver is the ::dev bindings dcmd. Given a driver name, the command displays a list of all instances of the named driver as demonstrated in the following example.
Example 23-21 Using the ::dev bindings Dcmd to Locate Driver Instances> ::dev bindings dad
300015ce3d8 videodisk (driver not attached)
300015c9a60 dad, instance #0
System properties at 0x300015ab400:
name='lean' type=int items=1
value=00000000
name='target' type=int items=1
value=00000000
name='class_prop' type=string items=1
value='data'
name='type' type=string items=1
value='data'
name='class' type=string items=1
value='dada'
...
300015c9880 dad, instance #1
System properties at 0x300015ab080:
name='lean' type=int items=1
value=00000000
name='target' type=int items=1
value=00000002
name='class_prop' type=string items=1
value='data'
name='type' type=string items=1
value='data'
name='class' type=string items=1
value='dada'
Retrieving Driver Soft State Information
A common problem when debugging a driver is retrieving the soft state for a particular driver instance. The soft state is allocated with the ddi_soft_state_zalloc (9F) routine. The driver can obtain the soft state through ddi_get_soft_state (9F) . The name of the soft state pointer is the first argument to ddi_soft_state_init (9F) ). With the name, you can use mdb to retrieve the soft state for a particular driver instance through the ::soft state dcmd:
> *bst_state::softstate 0x3 702b7578
In this case, ::soft state is used to fetch the soft state for instance 3 of the best sample driver. This pointer references a bst_soft structure that is used by the driver to track state for this instance.
Modifying Kernel Variables
You can use both kmdb and mdb to modify kernel variables or other kernel state. Kernel state modification with mdb should be done with care, because mdb does not stop the kernel before making modifications. Groups of modifications can be made atomically by using kmdb, because kmdb stops the kernel before allowing access by the user. The mdb debugger is capable of making single atomic modifications only.
w – Writes the lowest two bytes of the value of each expression to the target beginning at the location specified by dot
W – Writes the lowest 4 bytes of the value of each expression to the target beginning at the location specified by dot
Z – Write the complete 8 bytes of the value of each expression to the target beginning at the location specified by dot
Be sure to use the proper format specifier to perform the modification. The formats are:
Use the ::sized dcmd to determine the size of the variable to be modified.
The following example overwrites the value of mod debug with the value 0x80000000.
Example 23-22 Modifying a Kernel Variable With a Debugger> moddebug/W 0x80000000
mod debug: 0 = 0x80000000