BPF filtering tests can sometime fail. Running the test in verbose mode
shows the following:
$ sudo perf test 42
42: BPF filter :
42.1: Basic BPF filtering : FAILED!
42.2: BPF pinning : Skip
42.3: BPF prologue generation : Skip
$ perf --version
perf version 4.18.0-425.3.1.el8.ppc64le
$ sudo perf test -v 42
42: BPF filter :
42.1: Basic BPF filtering :
--- start ---
test child forked, pid 711060
...
bpf: config 'func=do_epoll_wait' is ok
Looking at the vmlinux_path (8 entries long)
Using /usr/lib/debug/lib/modules/4.18.0-425.3.1.el8.ppc64le/vmlinux for symbols
Open Debuginfo file: /usr/lib/debug/.build-id/81/56f5a07f92ccb62c5600ba0e4aacfb5f3a7534.debug
Try to find probe point from debuginfo.
Matched function: do_epoll_wait [4ef8cb0]
found inline addr: 0xc00000000061dbe4
Probe point found: __se_compat_sys_epoll_pwait+196
found inline addr: 0xc00000000061d9f4
Probe point found: __se_sys_epoll_pwait+196
found inline addr: 0xc00000000061d824
Probe point found: __se_sys_epoll_wait+36
Found 3 probe_trace_events.
Opening /sys/kernel/tracing//kprobe_events write=1
...
BPF filter result incorrect, expected 56, got 56 samples
test child finished with -1
---- end ----
BPF filter subtest 1: FAILED!
The statement above about the result being incorrect looks weird, and it
is due to that particular perf build missing commit 3e11300cdf
("perf test: Fix bpf test sample mismatch reporting"). In reality, due
to commit 4b04e0decd ("perf test: Fix basic bpf filtering test"),
perf expects there to be 56*3 samples.
However, the number of samples we receive is going to be dependent on
where the probes are installed, which is dependent on where
do_epoll_wait gets inlined. On s390x, it looks like probes at all the
inlined locations are hit. But, that is not the case on ppc64le.
Fix this by switching the test to instead use the syscall tracepoint.
This ensures that we will only ever install a single event enabling us
to reliably determine the sample count.
Reported-by: Disha Goel <disgoel@linux.vnet.ibm.com>
Signed-off-by: Naveen N. Rao <naveen.n.rao@linux.vnet.ibm.com>
Cc: Kajol Jain <kjain@linux.ibm.com>
Cc: bpf@vger.kernel.org
Link: http://lore.kernel.org/lkml/20230123083224.276404-1-naveen.n.rao@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The libbpf is switching off support for legacy map definitions [1],
which will break the perf llvm tests.
Moving the base source map definition to BTF-defined, so we need
to use -g compile option for to add debug/BTF info.
[1] https://lore.kernel.org/bpf/20220627211527.2245459-1-andrii@kernel.org/
Signed-off-by: Jiri Olsa <jolsa@kernel.org>
Acked-by: Andrii Nakryiko <andrii@kernel.org>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Ian Rogers <irogers@google.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: bpf@vger.kernel.org
Link: https://lore.kernel.org/r/20220704152721.352046-1-jolsa@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: https://lkml.kernel.org/n/tip-p0kg493z2m8qizjbdefzip1i@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since e145242ea0 ("syscalls/core, syscalls/x86: Clean up syscall stub
naming convention") changed the main syscall function for 'epoll_pwait'
to something other than the expected 'SyS_epoll_pwait the' 'perf test
BPF' entries started failing, fix it by using something called from the
main syscall function instead, 'epoll_wait', which should keep this test
working in older kernels too.
Before:
# perf test BPF
40: BPF filter :
40.1: Basic BPF filtering : FAILED!
40.2: BPF pinning : Skip
40.3: BPF prologue generation : Skip
40.4: BPF relocation checker : Skip
If we use -v for that test we see the problem:
Probe point 'SyS_epoll_pwait' not found.
After:
# perf test BPF
40: BPF filter :
40.1: Basic BPF filtering : Ok
40.2: BPF pinning : Ok
40.3: BPF prologue generation : Ok
40.4: BPF relocation checker : Ok
#
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Dominik Brodowski <linux@dominikbrodowski.net>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Steven Rostedt <rostedt@goodmis.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: https://lkml.kernel.org/r/tip-y24nmn70cs2am8jh4i344dng@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The 'perf test bpf' was hooking a eBPF program on the SyS_epoll_wait()
kernel function, that was what the epoll_wait() glibc function ended up
calling, but since at least glibc 2.26, the one that comes with, for
instance, Fedora 27, glibc ends up calling SyS_epoll_pwait() when
epoll_wait() is used, causing this 'perf test' entry to fail.
So switch to using epoll_pwait() and hook the eBPF program to the
SyS_epoll_pwait() kernel function to make it work on a wider range of
glibc and kernel versions.
Tested-by: Wang Nan <wangnan0@huawei.com>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Link: https://lkml.kernel.org/n/tip-zynvquy63er8s5mrgsz65pto@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Something made the sys_epoll_wait() function alias not to be found in
the vmlinux DWARF info, being found only in /proc/kallsyms, which made
the BPF perf tests to fail:
[root@jouet ~]# perf test BPF
37: Test BPF filter :
37.1: Test basic BPF filtering : FAILED!
37.2: Test BPF prologue generation : Skip
37.3: Test BPF relocation checker : Skip
[root@jouet ~]#
Using -v we can see it is failing to find DWARF info for the probed function,
sys_epoll_wait, which we can find in /proc/kallsyms but not in vmlinux with
CONFIG_DEBUG_INFO:
[root@jouet ~]# grep -w sys_epoll_wait /proc/kallsyms
ffffffffbd295b50 T sys_epoll_wait
[root@jouet ~]#
[root@jouet ~]# readelf -wi /lib/modules/4.7.0+/build/vmlinux | grep -w sys_epoll_wait
[root@jouet ~]#
If we try to use perf probe:
[root@jouet ~]# perf probe sys_epoll_wait
Failed to find debug information for address ffffffffbd295b50
Probe point 'sys_epoll_wait' not found.
Error: Failed to add events.
[root@jouet ~]#
It all works if we use SyS_epoll_wait, that is just an alias to the probed
function:
[root@jouet ~]# grep -i sys_epoll_wait /proc/kallsyms
ffffffffbd295b50 T SyS_epoll_wait
ffffffffbd295b50 T sys_epoll_wait
[root@jouet ~]#
So use it:
[root@jouet ~]# perf test BPF
37: Test BPF filter :
37.1: Test basic BPF filtering : Ok
37.2: Test BPF prologue generation : Ok
37.3: Test BPF relocation checker : Ok
[root@jouet ~]#
Further info:
[root@jouet ~]# gcc --version
gcc (GCC) 6.1.1 20160621 (Red Hat 6.1.1-3)
[acme@jouet linux]$ cat /etc/fedora-release
Fedora release 24 (Twenty Four)
Investigation as to why it fails is still underway, but it was always
going from sys_epoll_wait to SyS_epoll_wait when looking up the DWARF
info in vmlinux, and this is what is breaking now.
Switching to use SyS_epoll_wait allows this test to proceed and test the
BPF code it was designed for, so lets have this in to allow passing this
test while we fix the root cause.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Alexei Starovoitov <alexei.starovoitov@gmail.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-7hekjp0bodwjbb419sl2b55h@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The prototype for epoll_wait() is available in older distros, so use it
instead of epoll_pwait() (removing the last NULL arg, the sigmask,
makes it the same thing anyway) to avoid breaking the build.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/n/tip-pwiwizloxt0jujy8em80qut3@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch replaces the original toy BPF program with the previously
introduced bpf-script-example.c. Dynamically embeddeding it into
'llvm-src-base.c'.
The newly introduced BPF program attaches a BPF program to
'sys_epoll_pwait()'. perf itself never use that syscall, so further test
can verify their result with it. The program would generate 1 sample in
every 2 calls of epoll_pwait() system call.
Since the resulting BPF object is useful per se for further tests,
test_llvm__fetch_bpf_obj() is introduced for creating BPF objects from
source. The LLVM test was rewritten to use it.
Committer note:
Running it:
[root@zoo wb]# perf test -v LLVM
35: Test LLVM searching and compiling :
--- start ---
test child forked, pid 17740
Kernel build dir is set to /lib/modules/4.3.0-rc1+/build
set env: KBUILD_DIR=/lib/modules/4.3.0-rc1+/build
unset env: KBUILD_OPTS
include option is set to -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.9.2/include -I/home/git/linux/arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated -I/home/git/linux/include -Iinclude -I/home/git/linux/arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I/home/git/linux/include/uapi -Iinclude/generated/uapi -include /home/git/linux/include/linux/kconfig.h
set env: NR_CPUS=4
set env: LINUX_VERSION_CODE=0x40300
set env: CLANG_EXEC=/usr/libexec/icecc/bin/clang
set env: CLANG_OPTIONS=-xc
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/4.9.2/include -I/home/git/linux/arch/x86/include -Iarch/x86/include/generated/uapi -Iarch/x86/include/generated -I/home/git/linux/include -Iinclude -I/home/git/linux/arch/x86/include/uapi -Iarch/x86/include/generated/uapi -I/home/git/linux/include/uapi -Iinclude/generated/uapi -include /home/git/linux/include/linux/kconfig.h
set env: WORKING_DIR=/lib/modules/4.3.0-rc1+/build
set env: CLANG_SOURCE=-
llvm compiling command template: echo '/*
* bpf-script-example.c
* Test basic LLVM building
*/
#ifndef LINUX_VERSION_CODE
# error Need LINUX_VERSION_CODE
# error Example: for 4.2 kernel, put 'clang-opt="-DLINUX_VERSION_CODE=0x40200" into llvm section of ~/.perfconfig'
#endif
#define BPF_ANY 0
#define BPF_MAP_TYPE_ARRAY 2
#define BPF_FUNC_map_lookup_elem 1
#define BPF_FUNC_map_update_elem 2
static void *(*bpf_map_lookup_elem)(void *map, void *key) =
(void *) BPF_FUNC_map_lookup_elem;
static void *(*bpf_map_update_elem)(void *map, void *key, void *value, int flags) =
(void *) BPF_FUNC_map_update_elem;
struct bpf_map_def {
unsigned int type;
unsigned int key_size;
unsigned int value_size;
unsigned int max_entries;
};
#define SEC(NAME) __attribute__((section(NAME), used))
struct bpf_map_def SEC("maps") flip_table = {
.type = BPF_MAP_TYPE_ARRAY,
.key_size = sizeof(int),
.value_size = sizeof(int),
.max_entries = 1,
};
SEC("func=sys_epoll_pwait")
int bpf_func__sys_epoll_pwait(void *ctx)
{
int ind =0;
int *flag = bpf_map_lookup_elem(&flip_table, &ind);
int new_flag;
if (!flag)
return 0;
/* flip flag and store back */
new_flag = !*flag;
bpf_map_update_elem(&flip_table, &ind, &new_flag, BPF_ANY);
return new_flag;
}
char _license[] SEC("license") = "GPL";
int _version SEC("version") = LINUX_VERSION_CODE;
' | $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf -O2 -o -
test child finished with 0
---- end ----
Test LLVM searching and compiling: Ok
[root@zoo wb]#
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1446817783-86722-6-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This patch provides infrastructure for passing source files to --event
directly using:
# perf record --event bpf-file.c command
This patch does following works:
1) Allow passing '.c' file to '--event'. parse_events_load_bpf() is
expanded to allow caller tell it whether the passed file is source
file or object.
2) llvm__compile_bpf() is called to compile the '.c' file, the result
is saved into memory. Use bpf_object__open_buffer() to load the
in-memory object.
Introduces a bpf-script-example.c so we can manually test it:
# perf record --clang-opt "-DLINUX_VERSION_CODE=0x40200" --event ./bpf-script-example.c sleep 1
Note that '--clang-opt' must put before '--event'.
Futher patches will merge it into a testcase so can be tested automatically.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Alexei Starovoitov <ast@plumgrid.com>
Cc: Brendan Gregg <brendan.d.gregg@gmail.com>
Cc: Daniel Borkmann <daniel@iogearbox.net>
Cc: David Ahern <dsahern@gmail.com>
Cc: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kaixu Xia <xiakaixu@huawei.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1444826502-49291-10-git-send-email-wangnan0@huawei.com
Signed-off-by: He Kuang <hekuang@huawei.com>
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
