Dynamic Probes Program File Format and Language Reference |
The Dynamic Probes (dprobes) facility can be used to insert software probes dynamically into executing code modules. When a probe is fired, a user written probe-handler is executed. The probe-handler is a program written in an assembly-like language, based on the Reverse Polish Notation (RPN). Dynamic Probes also allows probes to be fired on specific type of memory accesses(execute|write|read or write|io). This is made possible by using the debug registers available on Intel x86 processors. These probes are called watchpoint probes. Watchpoint probes are not currently available for S/390. A probe program file specifies one or more probes for an executable program. It consists of a file header followed by a set of probe definitions. A probe definition contains the probe’s identity and a series of instructions that make up the probe handler. The assembly-like language in which the probe handlers are written provides: |
- arithmetic/logical/bitwise operations |
Special instructions are provided for a probe handler to: - access CPU registers - access user/kernel memory contents - access key system variables - access IO ports - log data into a buffer - exit to other debug facilities RPN instructions work on a circular stack of 1024 elements. The top of the stack is the only element accessible at any time. Typical instruction to place a data item on to the top of the stack is by a push instruction. Similarly, a pop instructions a retrieves the data item on the top of the stack. The stack elements are of fixed size (machine word length — typically sizeof(long)), and instructions always operate on an integral number of elements. The stack pointer will always be aligned on a machine-word-length boundary. Thus, even if a byte is pushed on to the stack, it will be padded to size of machine word length and placed on the stack. In addition to RPN program stack, RPN program interpreter also provides a call stack. Call stack is primarily present to hold the return address of the call rpn instruction. RPN language supports Exception Handling and Stack Tracing. Several kinds of faults can occur when rpn instructions are executed. RPN program interpreter generates different exceptions based on the type of fault. Programmer has the option of catching the exceptions and try to recover from faults by coding exception handlers. Programmer has the option of masking/unmasking different exceptions. Some exceptions are non-maskable. Maskable exceptions are not generated if they are masked. Non-maskable exceptions terminate the probe program interpretation if they are masked. When an exception is generated, programmer has the option to store probe program execution state (RPN stack entries, probe point details, call chain etc) in Stack trace buffer. This is called stack tracing. Stack trace buffer data is logged along with the normal dprobes log(see man page dprobes(8) ) when the probe program terminates. Programmer can also store the execution state at arbitrary time during probe program execution by using an rpn instruction (explained later). Exception handling and stack tracing are primarily intended to be used by the High-Level Language (HLL) DProbes Compiler, dpcc. The HLL compiler allows probe programs to be written in a C-like language, which will then be compiled into the RPN language. Refer to the section RPN Exception Handling and Stack Tracing for a complete explanation. The dprobes interpreter provides some internal registers which the probe program can make use of. There are rpn instructions provided to access/modify these. Top of the Stack Pointer register(TSP) This register always contains the pointer to top of the RPN stack. This register gets modified on every push and pop rpn instruction. Stack Base Pointer register(SBP) This register is intended to be a pointer to arbitrary locations on RPN stack. SBP is stored/restored on every call/return. Programmer can use SBP as pointer to the base of RPN stack during the call. Log Pointer register(LP) Holds the value of the current dprobes log buffer pointer. It is initialized to the beginning of the user log area in the log buffer. Note that a few bytes in the beginning of the log buffer are used to store the log header elements. Previous Log Pointer register(PLP) PLP will be maintained as a lagging value of LP. This will normally be the same value as LP when a logging instruction starts execution. PLP is updated to LP only if logging instruction succeeds. Using PLP, programmer has the option of backing out the partial log in the log buffer if logging fails. |
Probe Program file consists of the following sections: |
* File Header |
* Probe Point Definitions (one or more) |
- Probe Point Header |
- RPN Instructions |
Comments can be specified anywhere in the probe program file. Comments start with two backslashes(//) and extend up to the end of the line. Blank lines are allowed anywhere in the probe program file. File header and each probe point header consist of a number of statements of the key = value format. Numeric values may be specified in decimal or hexadecimal. Hexadecimal values have to be prefixed with 0x , as in the C language. Except the value of name= statement in the file header, everything else is case-insensitive. The value of name= has to be case-sensitive as the executable file names in Linux are case-sensitive. All the sections are defined below in detail. PROBE PROGRAM FILE HEADER name = <modulename> |
name = testapp (valid) |
modtype = <typename> vars = <n> gvars = <n> groupdef = <group_name>[ <group_name>[
<group_name> ...]] typedef = <type_name>[ <type_name>[
<type_name> ...]] major = <n> id = <n> logmax = <n> jmpmax = <n> logonfault = <yes|no> PROBE-POINT DEFINITION Probe-Point Header: |
Probe point header consists of statements of the key = value format. The valid statements are explained below. These can be specified in any order, with the exception that offset = <n> must appear first. opcode = <n> and offset = <n> statements are mandatory, remaining statements are optional. In case of kernel mode probe program files, instead of offset = <n> one can specify address = statement (explained below) to specify the probe location by its absolute linear address. offset = <n> address = <n1>[:<n2>] This statement is mutually exclusive with the offset = <n> statement. Only one of them should be present. Instead of specifying the probe point by offset, this statement can be used to specify it directly by its absolute linear address. However, this statement can also take symbolic expressions. watchpoint = <x|w|rw|io> This classifies the probe as watchpoint type and also specifies the type of watchpoint probe. This statement can be present only in the kernel mode probe program files. When this statement is not present, probes are meant to be of breakpoint type. (Currently not implemented on S/390) x: Watchpoint probes will be hit on instruction fetch from the address specified in ’address =’ statement. Here the valid address range is 1 byte. w: Watchpoint probes will be hit if any write is made to the address range specified in ’address =’ statement. Valid address ranges are 1, 2 and 4 bytes. rw: Watchpoint probes will be hit if any read or write occurs in the address range specified. Valid address ranges are 1, 2 and 4 bytes. io: Watchpoint probes will be hit if any io read or write happens in the address range specified. Valid address ranges are 1, 2 and 4 bytes. Probes won’t be applied if the address range specified is not consistent with the address range specified by the processor. Note: If watchpoint probes are to be put on user address , the user address should be specified in a kernel mode probe program file. Also note that user mode watchpoints are global. i.e., the are active across all process contexts. See Example 7 in COMPLETE EXAMPLES OF RPN for typical use of watchpoint probes. opcode = <n> Opcode specification is mandatory only for breakpoint probes. For watchpoint probes, it may not be specified. Note that only the first byte of the instruction on which a probe is desired is specified in this statement. If the instruction has a prefix, the prefix byte has to be specified here. There are a few instructions on which probes cannot be placed. They are: |
int3, int, into, wait (IA32, X86-64) |
svc, lpsw (S/390) |
and floating point instructions when FPU is emulated. |
WARNING: The onus is on the user to correctly specify the offset and the opcode statements. Incorrect use may cause probes to be inserted in the middle of an instruction which could lead to a system crash and data corruption. While Dprobes interoperates well with other debuggers, if a breakpoint is placed using another debugger at the same location where a probe already exists, the breakpoint will be ineffective. Although probes on floating point instructions are not allowed, when the FPU is emulated, we are not explicitly disallowing probes on FXSAVE and FXSTOR instructions. Putting probes on these instructions when FPU is emulated might result in unexpected behavior. group = <group_name> type = <type_name> Note: See Example 6. in COMPLETE EXAMPLES OF RPN for usage of the keywords group, type, groupdef and typedef. minor = <n> ignore = <n> maxhits = <n> logonfault = <yes|no> INSTRUCTION SET label: optional string used to specify "jmp" targets. It has to start with a non-numeric character and delimited by a ’:’ and can consist of any alpha-numeric characters. An RPN instruction has to follow a label. operator: any of the instructions listed below. opN: any operands required for the instruction. Note that TOS in the following description refers to the contents of the top of the stack on which RPN instructions operate. The Instructions in this language are classified under various categories as follows: RPN Execution Group: jmp <label> jlt <label> jle <label> jgt <label> jge <label> jz <label> jnz <label> Loop <label> Note: If the total number of branches (jumps+loops) taken during one run of the probe handler exceeds the jmpmax value specified in the program file header, probe handler will generate a JMP_MAX exception or will terminate if this exception is masked. call <procname> ret Format of Procedure: |
proc <procname> |
callk abort exit exit <n> |
1 = SGI kdb 2 = SGI crash dump 3 = Core Dump |
Any other value of <n> will result in an INVALID_OPERAND exception being generated. remove nop Logging Group setmin <minor> setmin setmaj <major> setmaj Note that the instructions to temporarily override major and minor codes are provided to enable users to easily identify of certain log records in the log buffer. log str If the log is successful, the logged string is prefixed by a token byte of 1 and a word indicating the length of the string logged. This data is intended to be used by any applications that may try to format the data collected in the log buffer. If the logging fails due to an invalid memory reference, the fault record is logged. Then an INVALID_ADDR exception is generated if this exception is unmasked, or the probe handler is terminated if it is masked. The fault record consists of a token byte of -1(0xff), an unsigned short containing the length (4 or 8) of the faulting address, and the faulting address itself. This instruction is intended to be used to log null-terminated strings at a known address to the log buffer. The length is only used to specify the upper limit on the string length. An example scenario would be (say we know that address of the string is in ECX) |
push 256 |
log arf log ars log mrf If the logging fails due to invalid memory reference, a fault record is logged and an INVALID_ADDR exception is generated. If this exception is masked, the probe handler is terminated. The fault record consists of a token byte of -1(0xff), an unsigned short containing the length (4 or 8) of the faulting address, and the faulting address itself. This instruction is intended to be used to log any data structures at a known address to the log buffer. An example scenario to log, say, the first 40 bytes of the current task structure would be: |
push 40 |
log mrs log <count> log b, <count> Note that the above logging instructions will generate a LOG_OVERFLOW exception if the size of the log exceeds the size specified on the logmax= statement in the probe program file header. The exception is generated only if it is unmasked. The partial log is retained. log Note: With this instruction, the actual log is preceded by a 3-byte prefix having a token byte of 7 and a word indicating number of RPN elements logged. If logging fails due to log buffer overflow, a LOG_OVERFLOW exception is generated (only if this exception is unmasked). The partial log is retained. Local Variable Group The following instructions are provided to manipulate variables. Note that the local here means that the variables defined in a probe program are not accessible from other probe programs. However, variables defined in a probe program are accessible to all the probe handlers in that program, as mentioned before. Variables are identified by their index. An index must be in the range 0 to n where n is the number specified on the vars= statement in the probe program file header. The instructions that manipulate local variables obtain the index of the variable to operate on either as an immediate operand in the instruction or from the RPN stack (in which case the value on the top of the stack is considered as the index). When index specified as an immediate operand, if its value is not the allowed range, then a compilation error will be returned when the probe program is being applied. push lv, <index> push lv pop lv, <index> pop lv move lv, <index> move lv inc lv, <index> inc lv dec lv, <index> dec lv log lv Note: For all the above instructions accessing local variables, an INVALID_OPERAND exception is generated if the index is exceeds the range specified by vars = statement in the file header. If this exception is masked, the probe handling is terminated. Global Variable Group. Global variables can be accessed across all the probe programs that are active. Probe programs are expected to use global variables in a cooperative manner. Variables are identified by their index. An index must be in the range 0 to n where n is the number specified on the gvars= statement in the probe program file header. The instructions that manipulate global variables obtain the index of the variable to operate on either as an immediate operand in the instruction or from the RPN stack (in which case the value on the top of the stack is considered as the index). When index specified as an immediate operand, if its value is not the allowed range, then a compilation error will be returned when the probe program is being applied. push gv, <index> push gv pop gv, <index> pop gv move gv, <index> move gv inc gv, <index> inc gv dec gv, <index> dec gv log gv Note: For all the above instructions accessing global variables, an INVALID_OPERAND exception is generated if the index is exceeds the range specified by gvars = statement in the file header. If this exception is masked, the probe handling is terminated. Arithmetic & Logic Group add sub mul div idiv Note: The division instructions generate DIVIDE_BY_ZERO exception if divisor is zero. If this exception is masked, the probe handling is terminated. neg and or xor rol <count> rol ror <count> ror shl <count> shl shr <count> shr pbl pbl <n> pbr pbr <n> Note: xchg dup <count> dup ros <count> Register Group Inside the probe handler, access is provided to two different kinds of register contexts, the user context registers and current context registers. The current context registers refer to the state of the registers at the time of the probe hit. User context registers refer to the state of the registers in the current process context just before it switched to kernel mode. For probes in user space code, user and current context registers are same. In the following instructions, r indicates registers in the current context and u indicates registers in the user context. Please refer to Appendix A for names of the registers and which registers are valid in these contexts. push r, <name> push u, <name> pop r, <name> pop u, <name> Writing to user and current context registers (code and stack segment, instruction pointer, stack pointer), or to control and debug registers, is not permitted. Also writing to gdtr, ldtr, tr, idtr is not permitted. Registers fs and gs are context independent. Processor registers are directly used when fs and gs are referenced. WARNING Changing the contents of registers in the probe handlers is dangerous and can cause system crashes and data corruption. Use these features with caution and only when you know exactly what you are doing. Data Group push <value> push b, <val> push <symbol> push mem, u8 push mem, u16 push mem, u32 push mem, u64 pop mem, u8 pop mem, u16 pop mem, u32 pop mem, u64 push bif All these instructions providing segmented addressing mode have been removed in dprobes v3.4.0. push bis All these instructions providing segmented addressing mode have been removed in dprobes v3.4.0. Note: The instructions which access memory will generate an INVALID_ADDR exception if the memory accessed is invalid or is currently paged out. If these exceptions are masked, the probe handling is terminated. System Variable Group push pid push procid push task Pushes the address of the current task (system variable current) onto the stack. IO Group (IA32, X86-64) The following instructions can be used to read and write to IO ports from the probe handlers. push io, u8 push io, u16 push io, u32 pop io, u8 pop io, u16 pop io, u32 Address Verification The following instructions can be used to verify the validity of an address before actually accessing it. If the given address is valid, a zero will be pushed onto the RPN stack. Otherwise, 1 will be pushed to indicate invalid address. Note that the execution of a probe handler can be terminated if it attempts to access any invalid address. vfyr vfa vfyrw vsa Miscellaneous seg2lin cnvrt sxf cnvrt sxd |
The RPN Compiler works in two passes. In the first pass
it does all the pre-processing for the conditional and
unconditional directives and in the second pass it actually
compiles the RPN files. All the conditional directives &
macros can be specified in an RPN file in the same way as in
C language. |
#define <variable> <value> #include <RPN filename> #ifdef <some-condn> #elseif <other-condn> #endif |
The RPN compiler allows for substitution of variables from the command line as provided by the GNU C Compiler(gcc). With the -D option in the command line, the keys to be substituted by values can be specified as follows: dprobes -i <rpn-file> -D key = value |
key is the string present in the RPN file and value is the value (string/numeral) to be substituted for key. |
Several RPN instructions generate faults during execution. RPN program interpreter throws exceptions when it encounters RPN instruction faults. Interpreter provides infrastructure/instructions for catching the exceptions, handling them, rethrowing the exceptions and propagating them up the call chain. When the interpreter throws an exception, it puts the exception information on the RPN stack. The top of the RPN stack will look like this when an exception occurs: |
Exception code identifies the type of exception. This depends on the type of RPN instruction fault. The width of the exception code is equal to the RPN stack width. For IA32, X86-64, and S/390, the exception code is of the form: |
The u-field is for user use and does not affect the meaning of the exception to the interpreter when the exception is re-thrown. For a non-user exception, the u-field is initialized to zero. The x-field actually defines the exception. The values indicated by parameter 1 and parameter 2 depends on the type of exception. The following table gives a list of exceptions, associated parameter meanings and the type of fault causing the exception. |
Masking/Unmasking Exceptions Exceptions can be selectively masked/unmasked by using the probe header control statement excpt_mask. Masked exceptions are not generated. Presently only LOG_OVERFLOW and USER_EX are maskable exceptions. All others are non-maskable. By default excpt_mask is set to 0x0FFF so that these exceptions are masked. If a masked non-maskable exception occurs, probe handling will be terminated. Bits in excpt_mask corresponds to the bits in x-field of the exception code. An exception type can be masked by resetting it’s corresponding bit in excpt_mask. Exception related instructions sx <handler> / ux push x rx Exception handling - some rules 1. Exceptions handlers cannot nest. That is if there are nested sx/ux pairs within a subroutine scope, only the innermost one will be effective. 2. A raised (generated) exception remains handled once a handler is called. 3. On raising an exception, the handler is implicitly reset. That is, a handler to catch an exception in a group of instructions will be effective only for the 1st exception generated by the group. The same handler will not be called for subsequent exceptions generated by the group. 4. Exceptions propagate up the call chain until a handler is found. That is, if an exception occurs in a sub-routine which has not set any exception handlers, the exception will automatically propagate up to the calling routine. During this, an implicit return is made to the calling routine. If the exception is not handled anywhere till the outermost routine, the probe handling is terminated. 5. The program interpreter remembers the last generated exception. STACK TRACING Interpreter provides a stack trace buffer to store stack trace information. Stack trace buffer data is logged along with the dprobes log data using dprobes (major, minor) = (0,0) This data can be used to debug the rpn program. The size of the stack trace buffer is governed by a file header control statement excpt_log. Default size of excpt_log is 1024 bytes. Stack trace information can be filled up in two ways: implicitly when an exception is generated, or explicitly using the log st instruction. log st purge st autostacktrace = [yes|no] This is a file header control statement. If set to yes stack trace information is collected whenever an exception is generated. Stack trace information has the following
format The header consists of Stack trace records Stack trace record contains information about the
execution state of the probe point. It has the
following format: The number of call stack frame records depends on the number of calls (nested) made at the time log st is executed (implicitly or explicitly) Format of call stack frame record The 3-byte prefixes associated with RPN stack entries, lv entries and gv entries indicate the number of RPN stack entries, local variables and global variables logged and have token bytes of 4, 5 and 6 respectively. Prior to the collection of stack trace information, interpreter needs to be told about the range of RPN stack, lv, and gv entries to be logged as part of stack trace. This is achieved by the following instructions. trace pv trace lv trace gv If the interpreter is not told about the range and index (i.e., if the above instructions are not present in a call scope), it logs 0 number of RPN stack, lv and gv entries, while building the call stack frame record for this call scope. So if any subroutine wants the RPN stack, lv and gv entries to be logged as part of its call stack, it should code the above three instructions. Interpreter provides Stack Trace Pointer register which will always point to the stack trace buffer from where the next log starts. Instructions using interpreter internal registers. push stp pop stp push lp pop lp push plp save sbp restore sbp save tsp restore tsp push sbp pop sbp copy sbp push sbp, n pop sbp, n copy sbp, n push tsp pop tsp copy tsp push tsp, n pop tsp, n copy tsp, n |
The comments in RPN Language are specified by
//. 1) Log EAX and EBX registers: push r, eax |
push u, ebx |
log 2 // logs two elements from the top of stack 2) Log the values after exchanging the two elements on the stack: push 0x02 |
push 0xff |
xchg |
log 2 // logs two elements after exchanging 3) Operations on a local variable (the var = keyword should be present in the RPN file header): push 0xffffffff 4) Arithmetic operations: push 0x02 5) Logs the cpuid data after pushing it on to the stack : push 0x0 6) Example depicting the use of jumps and calls: push 0xff |
In the following examples offset and opcodes must be verified accordingly before running any of these on a Linux system. 1) This rpn program puts a probe on an user application. name = appln //appln is the module on which
//probe is put. offset = main //probe is on function main. push u, cs //push user context code segment
register. Whenever the application appln is executed the probe is hit and the log shows the values of cs and ds. 2) This rpn file puts probe on kernel offset = do_fork //probe on do_fork push pid //push pid 2) Putting a probe on the network device driver and logging memory and register values.Traces the entry points to the various driver functions and logs the data. // The module 3c59x (Ethernet module) is loaded at boot // time and for testing the probe, network services are // restarted. #define func1 vortex_open name =
"/lib/modules/2.2.12-20/net/3c59x.o" offset = func1 // After preprocessing
"func1" is // replaced by
"vortex_open" push r, eip // push the address of the instruction
// pointer offset = vortex_start_xmit push 0x10 offset = vortex_close 3) Monitor the number of calls to do_fork name = vmlinux offset = do_fork inc lv, 0x01 // increments the local variable //
at index 1 4) Dump the system registers whenever swap_out function is called. name = vmlinux offset = swap_out push r, eax // dump the current registers 5) Putting probe on an interrupt handler "do_general_protection" by keeping some maxhits and logging some information. name = vmlinux offset = do_general_protection 6) Putting probes on the entry and exit points of some of the most frequently used functions in the kernel like do_fork, kmalloc and do_page_fault. This example makes use of the group and type keywords in the probe program file header and groupdef and typedef keywords in the RPN file header. This probe program can be used to selectively apply probes of same group and same type. name = vmlinux offset = do_fork offset = do_fork + 1987 offset = kmalloc offset = kmalloc + 344 offset = ext2_file_write offset = ext2_file_write + 1583 offset = do_page_fault offset = do_page_fault + 813 // Exit point of
do_page_fault The above RPN file can be built and applied in the
following ways: dprobes --build-ppdf example.rpn -o example.ppdf |
Generates ppdf file example.ppdf. |
dprobes --apply-ppdf example.ppdf |
Applies all the probes present in example.ppdf. |
dprobes --apply-ppdf example.ppdf --group process --type enter |
Applies probes belonging to the group process and type enter. |
dprobes --apply-ppdf example.ppdf --group = filesys |
Applies all probes belonging to the group filesys. |
dprobes --apply-ppdf example.ppdf --type = leave |
Applies all probes belonging to type leave. |
dprobes --apply-ppdf example.ppdf --group process memory --type leave |
Applies all probes belonging to group process, type leave and group memory, type leave. |
7) Illustration of watchpoint probes a) This examples puts probe on a kernel data symbol and watches it whenever a 4 byte write occurs to it. name = vmlinux address = nr_free_pages:nr_free_pages+3 push nr_free_pages b) This example shows how to watch a user space address. name = vmlinux address = 0x0804040c:0x0804040f // User data
address |
List of user and current registers for X86-64 allowed/not allowed for push and pop operations. y -- allowed, n -- not allowed. |
Floating Point Registers fr0 - fr7 Floating point data registers (Each register is 80bit wide, hence occupies > 1 rpn stack elements) fcw Control register fsw Status register ftw Tag word register fop Operand (data) pointer register fip Instruction pointer register fdp Data segment register Streaming SIMD Extension (SSE) Registers xmm0 - xmm15 SSE Floating Point Registers each 128 bits wide, occupying 2 rpn stack elements mxcsr SSE Floating Point Control/Status Register 64 bits wide |
List of user and current registers for INTEL allowed/not allowed for push and pop operations. y -- allowed, n -- not allowed. |
Floating Point Registers fr0 - fr7 Floating point data registers (Each register is 80bit wide, hence occupies 2.5 rpn stack elements) fcw Control register fsw Status register ftw Tag word register fip Instruction pointer register fcs Code segment register fdp Data (operand) pointer register fds Data segment register Inserting probes on floating point instructions when the FPU is emulated is not allowed. If FPU is emulated, accessing FPU registers will give a default value of zero. Streaming SIMD Extension (SSE) Registers xmm0 - xmm7 SSE Floating Point Registers each 128 bits wide, occupying 4 rpn stack elements mxcsr SSE Floating Point Control/Status Register 32 bits wide |
List of user and current registers for S/390(32bit) and zSeries(64bit) Lallowed/not allowed for push and pop operations. y -- allowed, n -- not allowed. |
Inserting probes on floating point instructions when the FPU is emulated is not allowed. If FPU is emulated, accessing FPU registers will give a default value of zero. FPR0-15 Floating Point Registers each 64 bits wide, occupying 2 rpn stack elements FPC Floating Point Control Register 32 bits wide |
List of user and current registers for ppc and ppc64 allowed/not allowed for push and pop operations. y -- allowed, n -- not allowed. |
Some methods to get the opcode present at any offset. For any user executable, objdump with -D option can be used to obtain the complete disassembly of the executable. Disassembled code can then be used to obtain the opcode. For example, the disassembled section of the function main of a user application looks like this: 080483d0 <main>: 80483d0: 55 pushl %ebp 80483d1: 89 e5 movl %esp,%ebp 80483d3: 83 ec 0c subl $0xc,%esp . . . If probe has to be put on main then opcode is 0x55. For S/390 code the method is the same, but the object code looks a bit different: 004007ac <main>: 4007ac: 90 af f0 28 stm %r10,%r15,40(%r15) 4007b0: a7 d5 00 0a bras %r13,4007c4 <main+0x18> . . . If probe has to be put on main then the opcode for the probe definition is 0x90af (S/390 has to use two byte opcodes in the probe point definition). The above method can also be used for kernel probes by disassembling and obtaining the opcode from vmlinux. Application debuggers or Kernel debuggers can also be used for the purpose. |
IBM Corporation |
DProbes version: 3.6.5 |
Dynamic Probes is licensed under GNU General Public License version 2 or later. Copyright (c) International Business Machines Corp., 2000 |