linux程序性能分析工具stap,SystemTap Linux下的万能观测工具

SystemTapLinuxchuba@taobao.comhttp://yufeng.info2010/11/18

AgendaSystemTap

SystemTapAccording to http://sourceware.org/systemtap/SystemTap provides free software (GPL) infrastructure tosimplify the gathering of information about the runningLinux system. This assists diagnosis of a performance orfunctional problem. SystemTap eliminates the need forthe developer to go through the tedious and disruptiveinstrument, recompile, install, and reboot sequence thatmay be otherwise required to collect data.

SystemTap1. write or choose a script describing what you want to observe 2. stap translates it into a kernel module 3. stap loads the module and communicates with it4. just wait for your data

#stap -uv test.stp Pass 1: parsed user script and 74 library script(s) using 86868virt/20488res/1792shr kb, in 190usr/20sys/209real ms. Pass 2: analyzed script: 1 probe(s), 0 function(s), 0 embed(s), 0 global(s) using 87264virt/21148res/1976shr kb, in 10usr/0sys/7real ms. Pass 3: translated to C into “/tmp/stapz2iv97/stap_aef621603e006af62084b361e0a0c981_553.c” using 87264virt/21332res/2144shr kb, in 0usr/0sys/0real ms. Pass 4: compiled C into “stap_aef621603e006af62084b361e0a0c981_553.ko” in 1230usr/160sys/1384real ms. Pass 5: starting run. Pass 5: run completed in 10usr/20sys/12331real ms.

SystemTap SystemTap is all about executing certain actions when hitting certain probe points.syscall.readwhen entering read() system callsyscall.close.returnwhen returning from the close() system callmodule(“floppy”).function(“*”)when entering any function from the “floppy” modulekernel.function(“*@net/socket.c”).returnwhen returning from any function in le net/socket.ckernel.statement(“*@kernel/sched.c:2917”)when hitting line 2917 of le kernel/sched.c

timer.ms(200) every 200 millisecondsprocess(“/bin/ls”).function(“*”) when entering any function in /bin/ls (not its libraries orsyscalls)process(“/lib/libc.so.6”).function(“*malloc*”)when entering any glibc function which has “malloc” in its namekernel.function(“*exit*”).returnwhen returningfrom any kernel function which has “exit” in its nameRTFM for more (man stapprobes).

SystemTapmostly C-style syntax with a feeling of awkbuiltin associative arraysbuiltin aggregates of statistical datavery easy to collect data and do statistics on it (average, min, max, count,. . . )many helper functions (builtin and in tapsets)RTFM: SystemTap Language Reference shipped with SystemTap (langref.pdf)

Performances and safetylanguage-level safety featuresno pointersno unbounded loopstype inferenceyou can also write probe handlers in C (with -g) but don’t complain if you break stuffruntime safety featuresstap enforces maximum run time for each probe handlervarious concurrency constraints are enforcedoverload processing (don’t allow stap to take up all the CPU time)many things can be overriden manually if you really wantsee SAFETY AND SECURITY section of stap(1)The overhead depends a lot of what you are trying to do but in general stap will try to stop you from doing something stupid (butthen you can still force it to do it).

Some helper functions you’ll see a lotpid() which process is this? uid() which user is running this? execname() what is the name of this process? tid() which thread is this? gettimeofday_s() epoch time in seconds probefunc() what function are we in? print_backtrace() figure out how we ended up here There are many many more. RTFM (man stapfuncs) and explore /usr/share/systemtap/tapset/.

Some cool stap options-x trace only speci ed PID (only for userland probing) -c run given command and only trace it and its children (will still trace all threads for kernel probes) -L list probe points matching given pattern along with available variables-d load given module debuginfo to help with symbol resolution in backtraces -g embed C code in stap script unsafe, dangerous and fun

AgendaSystemTap

RequirementsSystemTaputraceLinuxRedhatpackage-debuginfo on RPM distrosgcc-g -gdwarf-2 -g3staproot

SystemTapRHEL5U4:rpm -i kernel-debuginfo-common-2.6.18-164.el5.x86_64.rpm rpm -i kernel-debuginfo-2.6.18-164.el5.x86_64.rpm5U4SystemTap0.971.3:./configure prefix=/usr && make && make install# stap topsys.stp SYSCALL COUNT read 48 fcntl 42 … fstat 1 ————————————————————–

AgendaSystemTap

Example: Listing: exec.stpprobe syscall.exec*{ printf(“exec %s %s\n”, execname(), argstr) }$ stap -L ‘syscall.exec*’ syscall.execve name:string filename:string args:string argstr:string $filename:char* $argv:char** $envp:char** $regs:struct pt_regs*# stap exec.stp exec sshd /usr/sbin/sshd “-R” exec sshd /bin/bash

: Listing: sigkill.stpprobe signal.send{ if(sig_name == “SIGKILL”) printf(“%s was sent to %s (pid:%d) by %s uid :%d\n”, sig_name, pid_name , sig_pid, execname(), uid()) }# kill -9 `pgrep top`#stap sigkill.stp SIGKILL was sent to top (pid:19281) by bash uid :50920

Example tac.c: #include #include #include char* haha = “wahaha\n”;char* read_line(FILE* fp, char* buf, size_t len){ return fgets(buf, len, fp);}char* reverse_line(char* line, size_t l){ char *s = line, *e = s + l – sizeof(“\n”), t; while(s < e) {t =*s, *s = *e, *e = t; s++, e–; } return line;}void write_line(char* line){ fputs(line, stdout);}

Example tac.c continued : int main(int argc, char * argv[]){ char buf[4096], *line; FILE* fp = stdin; if(argc != 1 ) {fp = fopen(argv[1], “r”);} if(fp == NULL){fprintf(stdout, “usage: %s filename\n”, argv[0]);return -1;} while((line = read_line(fp, buf, sizeof(buf)))){ line = reverse_line(line, strlen(line)); write_line(line); } if(argc != 1) fclose(fp); return 0; }

tac# # gcc -g -gdwarf-2 -g3 tac.c# # stap -L ‘process(“a.out”).function(“*”)’ process(“/tmp/a.out”).function(“main@/tmp/tac.c:25”) $argc:int $argv:char** $buf:char[] $line:char* $fp:FILE* process(“/tmp/a.out”).function(“read_line@/tmp/tac.c:7”) $fp:FILE* $buf:char* $len:size_t process(“/tmp/a.out”).function(“reverse_line@/tmp/tac.c:11”) $line:char* $l:size_t $s:char* $e:char* $t:char process(“/tmp/a.out”).function(“write_line@/tmp/tac.c:21”) $line:char*

Example 1: function get_argv_1:long(argv:long) %{ /* pure */ THIS->__retvalue =(long) ((char**)THIS->argv)[1]; %}probe process(“a.out”).function(“main”){ filename = “stdin”; if($argc > 1) { filename = user_string(get_argv_1($argv)); } println(filename); }

Example 1 continued:#echo “hi”|./a.out #./a.out tac.c#stap -gu ./ex1.stp :) stdin tac.c

Example 2: callgraph for anythingfunction trace(entry_p, extra) { %( $# > 1 %? if (tid() in trace) %) printf(“%s%s%s %s\n”, thread_indent (entry_p), (entry_p>0?”->”:”

Example 2 continued: #echo “hi”|./a.out #sudo stap ./ex2.stp ‘process(“a.out”).function(“*”)’ :) 0 a.out(18123):->main argc=0x1 argv=0x7fff351ee0c8 30 a.out(18123): ->readline fp=0x3f7bb516a0 buf=0x7fff351ecfd0 len=0x1000 590 a.out(18123): reverse_line line=0x7fff351ecfd0 l=0x3 625 a.out(18123): write_line line=0x7fff351ecfd0 731 a.out(18123): readline fp=0x3f7bb516a0 buf=0x7fff351ecfd0 len=0x1000 762 a.out(18123):

Example 3:global line_lenprobe process(“a.out”).statement(“reverse_line@tac.c+1”){ line_len

Example 3 continued:# ls -al|./a.out#./ex3.stp :)value |————————————————– count

Example 4: global t, call_timeprobe process(“a.out”).function(“reverse_line”){ t = gettimeofday_ns() }probe process(“a.out”).function(“reverse_line”).return{ call_time

Example 4 continued :# ls -al|./a.out#./ex4.stp :) avg reverse_line execute time: 6651 ns

Example 5: probe process(@1).function(@2){ print_ubacktrace(); exit(); }

Example 5 continued:# ls -al|./a.out# stap ./ex5.stp ‘./a.out’ ‘*_line’ :) 0x40066d : reverse_line+0xc/0x61 [a.out] 0x40078f : main+0xaf/0x100 [a.out] 0x3bd441d994 [libc-2.5.so+0x1d994/0x357000]

Example 6: global linefunction alert_line(line:long) %{ /* pure */ strcpy((char*)THIS->line, “abcdefg\n”); %}probe process(“a.out”).function(“reverse_line”){ line = user_string($line); }probe process(“a.out”).function(“reverse_line”).return{ if(isinstr(line, “tac”)) $return = $haha; else if (isinstr(line, “hello”)) alert_line($return); }

Example 6 continued:# stap ./ex6.stp#echo tac|./a.out wahaha#echo hello|./a.out abcdefg#echo world|./a.out dlrow

AgendaSystemTap

Emacs Systemtap mode systemtap-mode.el: http://coderepos.org/share/browser/lang/elisp/systemtap-mode/systemtap-mode.el?format=txt .emacs (autoload ‘systemtap-mode “systemtap-mode”) (add-to-list ‘auto-mode-alist ‘(“\.stp$” . systemtap-mode))

http://sourceware.org/systemtap/langref/http://sourceware.org/systemtap/tapsets/http://baike.corp.taobao.com/images/d/df/Systemtap-haxogreen-2010072301.pdfhttp://sourceware.org/systemtap/wiki/AddingUserSpaceProbingToAppshttp://github.com/posulliv/staphttp://www.slideshare.net/posullivan/monitoring-mysql-with-dtracesystemtap

AgendaSystemTap

SystemTap is often described as “DTrace for Linux”.OProfile takes sample every $N CPU cycles so you can try to fi gureout what each CPU is spending its time on.SystemTap

Any question