DPROBES

dprobes − IBM Dynamic Probe Facility

dprobes {--help|-h}

dprobes {--version|-v}

dprobes {--insert|-i <probe-program-file>}

[-D key = value]
[--sym|-s <symbol-file>]
[--group|-u = <group_list>]
[--type|-t = <type_list>]
[--merge|-m | --replace|-c]
[--dont-verify-opcodes]
[--stop-cpus]
[--log|-l {[ PROCNAME [ PID [ UID [ SSESP
[ CSEIP [ PSW [ UPSW [ TSC ]]]]]]]] | ALL} ]

[--unformatted-output|-f]
[--log-target { KLOG | COM1 | COM2 | LTT | EVL}]

dprobes {--remove|-r}

{--name <loadable_module_name>| --all|-a}

dprobes {--getvars|-g}

{--name <loadable_module_name>| --all|-a}

[--index|-i=index1-index2]

[--local | --global] [--reset]

dprobes {--query|-q}

{--name <loadable_module_name>| --all|-a}

[--extended|-x]

dprobes {--build-ppdf|-b <rpn_file_name> }

[--outfile|-o <ppdf_file_name> ]
[-D key = value]
[--sym|-s <symbol_file_name> ]

dprobes {--apply-ppdf|-p <ppdf_file_name> }

[--group|-u = <group_list> ]

[--type|-t = <type_list> ]
[--merge|-m | --replace|-c]
[--dont-verify-opcodes]
[--log|-l <[ PROCNAME [ PID [ UID [ SSESP
[ CSEIP [ PSW [ UPSW [ TSC ]]]]]]]] | ALL} ]
[--unformatted-output|-f]
[--log-target < KLOG | COM1 | COM2 | LTT | EVL>]

The dprobes facility can be used to insert software probes dynamically into executing code modules. When a probe is fired, a user written probe-handler is executed. The probe-handler is a program written in an assembly-like language, based on the Reverse Polish Notation (RPN). Instructions are provided to enable the probe-handler to access all the hardware registers, system data structures and memory.

This is equivalent to inserting a breakpoint, monitoring various context specific values, dumping out certain data areas when the breakpoint is hit. So, in short, it is a dynamic debug session that runs without disrupting the code being debugged and with only a very small runtime performance hit. The dprobes facility is particularly useful for diagnosing problems that show up rarely in frequently hit areas of code.

DProbes also allows the probing storage accesses, implemented using the debug registers available on Intel x86 processors. These probes, called the watchpoint probes, can be specified to be fired on a specific type of memory access (execute|write|readwrite|io). Watchpoints are not currently available for S/390.

Another interesting aspect of the Dynamic Probes facility is that the probes are effective globally and they interoperate well with other debuggers. This means that probes placed in an executable module will be applied on all instances of that executable, under the context of all processes.
For eg.:
− if probes are placed in an executable program, they will be active in all instances of that program
− if probes are placed in a shared library, they will be active under the context of all processes using that shared library
− the probes will be effective even on those instances of the processes that may be running under the control of a debugger

Some of the unique aspects of the Dynamic Probes facility are:
− Probes can be placed almost anywhere
− Probes can be placed in any executable code, including the kernel, even in interrupt handlers, kernel modules etc.
− Read access to all the hardware registers and write access to most of them.
− Read/write access to any area in the virtual address space that is currently resident in the physical memory.
− Probes placed on an executable program or shared library are active globally under the context of all processes executing it
− Probes can be placed on programs that are being run under a debugger
− DProbes can be essentially used as an enabler of other debugging facilities like kernel debugger, trace facility and crash dump. Kernel debugger and crash dump facility can be invoked from DProbes by using exit n rpn instruction. Data collected from DProbes can be directed to trace buffer.
− DProbes can be used to monitor specific type of memory accesses using the watchpoint probes.

At the heart of the mechanism is an interpreter which executes a "probe program" when a probe fires. A probe program is registered per executable module and contains the probe handlers to be run for each probe placed in that module.

Probes may be registered before a process is created or a module is loaded.

Refer to the man page of dprobes.lang(8) for the details of the probe program file format and language to be used for writing probe handlers.

This facility can be used by root only.

This command works in the following basic operating modes:

insert:
dprobes --insert probe-program-file [insert-options]

remove:
dprobes --remove [remove-options]

getvars:
dprobes --getvars [getvars-options]

querying:
dprobes --query [query-options]

building ppdf:
dprobes --build-ppdf probe-program-file [ppdf-options]

applying ppdf:
dprobes --apply-ppdf probe-definition-file [apply-options]

miscellaneous:
dprobes --help
dprobes --version

The general form of a dprobes insert command is:

dprobes --insert <probe-program-file>

[-D key = value]
[--sym <symbol-file>]
[--group = <group_list>]
[--type = <type_list>]
[--merge | --replace]
[--dont-verify-opcodes]
[--stop-cpus]
[--log < PROCNAME [ PID [ UID [ SSESP
[ CSEIP [ PSW [ UPSW [ TSC ]]]]]]]] | ALL} ]

[--unformatted-output]
[--log-target <KLOG | COM1 | COM2 | LTT | EVL>]

-i
--insert
This command will insert the probes specified in the given probe program. Probes can be placed in the Linux kernel and kernel modules too.
Note : Probes cannot be inserted on a file which is currently being written to and attempts to write to a file which has probes inserted will fail.

-D key = value
This sub-option can be used for command line substitution for the Probe Program file. The key specifies the string in the probe program file which is to be replaced by value. Refer to the dprobes.lang(8) man page for details of substitution.

-u
--group = <group_name>[,<group_name>[,<group_name>, ...]]
Only the probes having a group field that matches one of the group names specified in the list above will be applied in the specified module. If this option is not specified then the group field is not checked when applying probe points. The group names specified on the command line should be present in the groupdef field in the Probe Program File Header (See dprobes.lang(8) )

-t
--type = <type_name>[,<type_name>[,<type_name>, ...]]
Only the probes having a type field that matches one of the type names specified in the list above will be applied in the specified module.If this option is not specified then the type field is not checked when applying probe points. The type names specified on the command line should be present in the typedef field in the Probe Program File Header (See dprobes.lang(8) )

Both group and type are optional on the command line. If nothing is specified then all the probes present in the ppdf file will be applied. If both --group AND --type are present then only the probes belonging to both one of the groups <group_name> as well as one of the types <type_name> will be applied.
These options are used to selectively insert a subset of the probes present in the specified RPN file,

-s
--sym = symbol-file (full path-name should be given)
If the probe locations are specified symbolically in the probe program file, this option can be used to specify the file used to resolve the symbols. For probes placed on the kernel, this file must be the System.map file. For all other module types, the symbols can be resolved from the corresponding object file, shared library object, or executable, provided that symbol information hasn’t been stripped from those binaries.

-m
--merge
This option can be used to add new probe points to the module which already has some probe points in it, without having to remove and reinsert. If there are any existing probes for the loadable module specified by the probe program file, they will be merged with the new probes. If a new probe point is specified at the same location where an old one is present, then the old probe point will be removed and the new one will be effective. Since this is done internally by removing and reinserting, there is a possibility to miss probe hits in the interval between removing the old points and inserting the merged points. Also the existing probe points will have their hit count reset when merging is done.

-c
--replace
This option can be used to replace the set of probe points for a module by a new set of points by using a new probe program file.

--dont-verify-opcodes
This option specifies that opcode= statement is optional for probe points when their location is specified by a symbol. If the probe point is specified by a symbol, we can be sure that it is on an instruction boundary, in which case opcode verification becomes optional.

--stop-cpus
[Not supported in this version]
Applicable only under an SMP environment, this option can be used to temporarily halt other processors while any of the probe handlers in this file are being executed. This option is provided in case the user wants to ensure that system variables are not changed while the probe handler is running.

-l
--log <PROCNAME [PID [UID [SSESP [CSEIP [PSW [UPSW [TSC]]]]]]]] | ALL>
This switch specifies any additional data that should be added to each log record generated by the probes in this program.

PROCNAME - log the process name for each probe

PID - log the pid of the process for which the

probe was hit

UID - log the uid of the process for which the

probe was hit

CSEIP - log caller’s CS:EIP (IA32)

SSESP - log caller’s SS:ESP (IA32)

PSW - log caller’s PSW (S/390)

UPSW - log caller’s user mode PSW. If the probe point is in the

kernel, then the psw of the program that made the call is stored.

Otherwise acts the same as PSW (S/390)

TSC - log Pentium high resolution Time Stamp Counter (IA32),

log the STCK value (S/390),

or ignore if not supported by the hardware

ALL - log all special data

By default, none of these will be added to the log record.

-f
--unformatted-output
This option is introduced to support the old logging format. Applicable only for logging the record to the kernel log buffers or with default logging. This option specifies that the log messages will be in old format.

--log-target <KLOG | COM1 | COM2 | LTT | EVL>

After a probe is successfully executed, the logs generated by the probe handler will be written to the target specified by this option.

KLOG:
Log records will be written to the kernel log buffer. This is equivalent to a printk() statement in the kernel. This may show up on the console if console_loglevel is low enough. It may also get saved in the /var/log/messages file if syslogd is running. The format of log records is similar to the dump format <offset> : <hexadecimal words> : <ascii characters>

COM1:
Log will be directed to the serial port COM1 (IA32)

COM2:
Log will be directed to the serial port COM2 (IA32)

The serial port logging options can be used to collect the log generated by one machine in another machine using serial connections.

LTT:
Log generated by DProbes can be redirected to a Trace facility given by Opersys, The Linux Trace Toolkit(LTT). This is possible only if the kernel supports LTT. DProbes interoperates with LTT version 0.9.4pre4 or later.

EVL:
Log generated by DProbes can be logged using the posix_log_write function. This is possible only if the kernel supports POSIX Event Logging.

The general form of a dprobes command for removing the probes is:

dprobes --remove

{--name=loadable_module_name | --all}

-r

--remove

dprobes remove command.

One of the following options must be specified.

-n

--name

If the loadable module name is specified, this command will remove all the probes inserted for this module.

-a

--all

This option removes all the probes in the system.

The general form of this dprobes command is:

dprobes --getvars {[--name <loadable_module_name>] |--all}

[--index = index1-index2]

[--local | global]

[--reset]

-g

--getvars

dprobes command for displaying the value of variables.

--name = loadable_module_name

This option will list the values of variables of the dynamic probes inserted for the specified loadable module.

-e
--index = index1-index2

This option will retrieve the dynamic probe variables from index1 to index2. If this option is not specified the default is to retrieve all variables.

--reset

This option will zero out the variables specified

--local

This option will retrieve the values of local variables.

--global

This option will return the values of global variables.

--all

This option will return values of all the variables presently being used in all probe programs.

The general form of a dprobes query command is:

dprobes --query {--name = loadable_module_name | --all}

[--extended]

-q

--query

dprobes query command.

-a

--all

This option will list all the dynamic probe definitions, including those that are not currently active. If this option is not specified, only those probes that are currently active and inserted will be listed.

--name = loadable_module_name

This option will list all the dynamic probe definitions inserted for the specified loadable module i.e. a executable, shared object,kernel module etc.

--extended
Will provide the following details for each probe point listed:

− probe point number with in the loadable module,

− major and minor codes for the probe point,

− address of the probe point location,

− opcode at the probe point location,

− probe point status,

− local variables.

The probe point states could be one of the following:

COMPILED not yet applied as the module is not in memory

ACTIVE applied successfully and is active

MISMATCH not applied due to opcode mismatch

INVALID OFFSET not applied due to invalid offset.

DISABLED disabled as probe is hit max-hit number of times

REMOVED permanently, due to recursion or due to explicit remove RPN command

WATCHPOINT RANGE INVALID The range of memory specified for the watchpoint probe is inconsistent with the ranges allowed by the processor.

This builds a pre-built probe definition file (PPDF) , which is essentially the filed version of the package built by the build-ppdf function. The value in providing this function is that probe definitions can be pre-built from a Makefile and later inserted using the dprobes command apply-ppdf function. This would allow a module to be installed along with its PPDF so that is debug-ready, without being a special built with the debugging code present. Pre-building is made necessary because the probe location may be expressed symbolically using symbols from the module’s symbol table.

The general form of a dprobes ppdf command is:

dprobes --build-ppdf | -b <rpn_file_name>

[--outfile | -o <ppdf_file_name>]
[--sym | -s <symbol_file>]
[-D key = value]

-b = rpn_filename

--build-ppdf = rpn_filename
The dprobes buildppdf command. This command generates a file called pre-built probe definition file which is a binary file consisting of all the details of an rpn file and the group and type definitions present in the rpn file. This file can be used to selectively apply probes of some particular group or type or both.

-o = ppdf_file_name

--outfile = ppdf_file_name
This option is used to specify an output ppdf filename instead of the default which is rpn_filename.rpn.ppdf

-s = symbol_file

--sym = symbol_file
Same as the --sym option in the dprobes --insert command.

-D key = value
Same as the -D option in the dprobes --insert command.

This command works in the same way as insert command and thus it accepts all the options accepted by the insert except -D and --sym suboptions.
See INSERT OPTIONS for details of the command line options of the dprobes insert command which can be used in --apply-ppdf also.

The general form of a dprobes applyppdf command is:

dprobes --apply-ppdf | -p <ppdf_file_name>

[--group = <group_list>]
[--type = <type_list>]
[--merge | --replace]
[--log <[ PROCNAME [ PID [ UID [ SSESP
[ CSEIP [ PSW [ UPSW [ TSC ]]]]]]]] | ALL >]
[--unformatted-output|-f]
[--log-target < KLOG | COM1 | COM2 | LTT | EVL>]

--apply-ppdf ppdf_file_name
The dprobes applyppdf command. This command is used for applying the probe using a pre-built probe definition file rather than using an rpn file. This can be used to selectively apply probes of some particular group or type using the --group and --type options.

Please refer to the man page of dprobes.lang(8) for the syntax and details of how to write a probe program file.

The following are the restrictions in this version of dprobes:

Short options cannot be clubbed together
For example:

dprobes -qax is invalid

dprobes -q -a -x is valid

The following options are not supported:

Insert Options: --stop-cpus

If the name of any file contains any character other than [a-z,A-Z] then the name should be specified in double-quotes and double-quotes should be given with escape sequence in the --name and --insert options.

For Example:

If the file name is 3c59x.o then it should be specified in the following manner:
\\"3c59x.o\\" on the command line.

Send all the bug-reports to:

DProbes mailing list dprobes@www-124.ibm.com

IBM Corporation

DProbes version: 3.6.5
man pages last modified on: 16 June 2004

Dynamic Probes is licensed under GNU General Public License version 2 or later.

Copyright (c) International Business Machines Corp., 2000