You define DTrace probes in a .d source file, which is then used when compiling and linking your application. First, select an appropriate name for your user application provider. The provider name that you choose is appended with the process identifier for each process that is executing your application code. For example, if you chose the provider name myserv for a web server that was executing as process ID 1203, the DTrace provider name that corresponds to this process would be myserv1203. In a .d source file, you would add a provider definition similar to the one that is shown the following example:

provider myserv
{
  ...
};

Next, add a definition for each probe and the corresponding arguments. The following example defines the two probes that are discussed in Choosing the Probe Points. The first probe has two arguments, both of type char *. The second probe has no arguments. The D compiler converts two consecutive underscores (__) to a dash (-) in the probe name:

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

You can add stability attributes to your provider definition so that consumers of your probes understand the likelihood of change in future versions of your application. See DTrace Stability Features for more information on DTrace stability attributes.

The following example illustrates how stability attributes are defined:

#pragma D attributes Evolving/Evolving/Common provider myserv provider
#pragma D attributes Private/Private/Unknown provider myserv module
#pragma D attributes Private/Private/Unknown provider myserv function
#pragma D attributes Evolving/Evolving/Common provider myserv name
#pragma D attributes Evolving/Evolving/Common provider myserv args

provider myserv
{
  probe query__receive(char *, char *);
  probe query__respond();
};

After you have defined your probes in a .d file, you then need to augment your source code to indicate the locations that should trigger your probes. Consider the following example C application source code:

void main_look(void)
{
  ...
  query = wait_for_new_query();
  process_query(query);
  ...
}

To add probes to an application, use the -h option to the dtrace command, which generates a header file based on the probe definitions. For example, the following command generates the header file myserv.h, which contains macro definitions corresponding to the probe definitions in myserv.d:

# dtrace -h -s myserv.d

This method is recommended, as the coding is easier to implement and understand. The method is also compatible with both C and C++. In addition, because the generated macros depend on the types that you define in the provider definition, the compiler can perform type checking on them.

For example, you can add a probe site by using the MYSERV_QUERY_RECEIVE macro that dtrace -h defines in myserv.h:

#include "myserv.h"
...
void main_look(void)
{
  ...
  query = wait_for_new_query();
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);
  process_query(query);
  ...
}

In the previous example, the name of the macro encodes both the provider name and the probe name.

The computational overhead of a DTrace probe is usually equivalent to a few no-op instructions. However, setting up probe arguments can be expensive, particularly in the case of dynamic languages, where the code has to determine the name of a class or the method at runtime.

In addition to the probe macro, the dtrace -h command creates an is-enabled probe macro for each probe that you specify in the provider definition. To ensure that your program computes the arguments to a DTrace probe only when required, you can use the is-enabled probe test to verify whether the probe is currently enabled, for example:

if (MYSERV_QUERY_RECEIVE_ENABLED())
  MYSERV_QUERY_RECEIVE(query->clientname, query->msg);

If the probe arguments are computationally expensive to calculate, the slight overhead that is incurred by performing the is-enabled probe test is more than offset when the probe is not enabled.

You must augment the build process for your application to include the DTrace provider and probe definitions. A typical build process takes each source file and compiles it to create a corresponding object file. The compiled object files are then linked to each other to create the finished application binary, as shown in the following example:

src1.o: src1.c
    gcc -c src1.c

src2.o: src2.c
    gcc -c src2.c

myserv: src1.o src2.o
    gcc -o myserv src1.o src2.o

If you included DTrace probe definitions in your application, you need to add appropriate Makefile rules to your build process to execute the dtrace command.

The dtrace command post-processes the object files that are created by the preceding compiler commands and generates the object file myserv.o from myserv.d and the other object files. The -G option is used to link provider and probe definitions with a user application.

The -Wl,--export-dynamic link options to gcc are required to support symbol lookup in a stripped executable at runtime, for example, by running ustack().

If you inserted probes in the source code by using the macros that were defined in a header file created by dtrace -h, you need to include that command in the Makefile:

myserv.h: myserv.d
    dtrace -h -s myserv.d

src1.o: src1.c myserv.h
    gcc -c src1.c

src2.o: src2.c myserv.h
    gcc -c src2.c

myserv.o: myserv.d src1.o src2.o
    dtrace -G -s myserv.d src1.o src2.o

myserv: myserv.o
    gcc -Wl,--export-dynamic,--strip-all -o myserv myserv.o src1.o src2.o

The rules in the Makefile take into account the dependency of the header file on the probe definition.

The DTrace helper device (/dev/dtrace/helper) enables a user-space application that contains USDT probes to send probe provider information to DTrace.

If the program that is to be traced is run by a user other than root, change the mode of the DTrace helper device to allow the user to record tracing information:

# chmod 666 /dev/dtrace/helper

Alternatively, if the acl package is installed on your system, you can use an ACL rule to limit access to a specific user, as shown in the following example:

# setfacl -m u:guest:rw /dev/dtrace/helper
# ls -l /dev/dtrace
total 0
crw-rw----  1 root root 10, 16 Sep 26 10:38 dtrace
crw-rw----+ 1 root root 10, 17 Sep 26 10:38 helper
drwxr-xr-x  2 root root     80 Sep 26 10:38 provider
# getfacl /dev/dtrace/helper
getfacl: Removing leading '/' from absolute path names
# file: dev/dtrace/helper
# owner: root
# group: root
user::rw-
user:guest:rw-
group::rw-
mask::rw-
other::---

For example, for a myserv process with a PID of 1173, the full name of the query-receive probe would be myserv1173:myserv:main_look:query-receive.

The following simple example shows how to invoke a traced process from dtrace:

# dtrace -c ./myserv -qs /dev/stdin <<EOF
                     $target:::query-receive
    {
      printf("%s:%s:%s:%s %s %s\n", probeprov, probemod, probefunc, probename,
                                    stringof(args[0]), stringof(args[1]));
    }
  $target:::query-respond
    {
      printf("%s:%s:%s:%s\n", probeprov, probemod, probefunc, probename);
    }
EOF

myserv1173:myserv:main_look:query-receive foo1 msg1
myserv1173:myserv:process_query:query-respond
myserv1173:myserv:main_look:query-receive bar2 msg1
myserv1173:myserv:process_query:query-respond
...

dtrace: failed to compile script /dev/stdin: line 7:
printf( ) argument #5 is incompatible with conversion #4 prototype:
	conversion: %s
	 prototype: char [] or string (or use stringof)
	  argument: char *

In file included from src1.c:5:
myserv.h:39: warning: parameter names (without types) in function declaration

The following script illustrates the complete process of instrumenting, compiling and tracing a simple user-space program. Save it in a file named testscript:

#!/bin/bash

# Define the probes
cat > prov.d <<EOF
provider myprog
{
  probe dbquery__entry(char *);
  probe dbquery__result(int);
};
EOF

# Create the C program
cat > test.c <<EOF
#include <stdio.h>
#include "prov.h"

int
main(void)
{
        char *query = "select value from table where name = 'foo'";
        /* If the dbquery-entry probe is enabled, trigger it */
        if (MYPROG_DBQUERY_ENTRY_ENABLED())
                MYPROG_DBQUERY_ENTRY(query);
        /* Pretend to run query and obtain result */
        sleep(1);
        int result = 42;
        /* If the dbquery-result probe is enabled, trigger it */
        if (MYPROG_DBQUERY_RESULT_ENABLED())
                MYPROG_DBQUERY_RESULT(result);
        return (0);
}
EOF



test.o: test.c prov.h
  gcc -c test.c

prov.o: prov.d test.o
  dtrace -G -s prov.d test.o

test: prov.o
  gcc -o test prov.o test.o
EOF

# Make the executable
make test

# Trace the program
dtrace -c ./test -qs /dev/stdin <<EOF
myprog\$target:::dbquery-entry
{
        self->ts = timestamp;
        printf("Query = %s\n", stringof(args[0]));
}

myprog\$target:::dbquery-result
{
        printf("Query time = %d microseconds; Result = %d\n",
            (timestamp - self->ts) / 1000, args[0]);
}
EOF

The output from running this script shows the compilation steps, as well as the results of tracing the program:

# chmod +x testscript
# ./testscript
dtrace -h -s prov.d
gcc -c test.c
dtrace -G -s prov.d test.o
gcc -o test prov.o test.o
Query = select value from table where name = 'foo'
Query time = 1000481 microseconds; Result = 42