diff options
| author | Stanislav Fomichev <sdf@google.com> | 2021-01-15 08:34:59 -0800 |
|---|---|---|
| committer | Alexei Starovoitov <ast@kernel.org> | 2021-01-20 14:23:00 -0800 |
| commit | 9cacf81f8161111db25f98e78a7a0e32ae142b3f (patch) | |
| tree | f20ef5d2014e204f3084ebbea54515427fdef43e /tools/testing/selftests/bpf/progs/sockopt_sk.c | |
| parent | 13ca51d5eb358edcb673afccb48c3440b9fda21b (diff) | |
| download | linux-stable-9cacf81f8161111db25f98e78a7a0e32ae142b3f.tar.gz linux-stable-9cacf81f8161111db25f98e78a7a0e32ae142b3f.tar.bz2 linux-stable-9cacf81f8161111db25f98e78a7a0e32ae142b3f.zip | |
bpf: Remove extra lock_sock for TCP_ZEROCOPY_RECEIVE
Add custom implementation of getsockopt hook for TCP_ZEROCOPY_RECEIVE.
We skip generic hooks for TCP_ZEROCOPY_RECEIVE and have a custom
call in do_tcp_getsockopt using the on-stack data. This removes
3% overhead for locking/unlocking the socket.
Without this patch:
3.38% 0.07% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt
|
--3.30%--__cgroup_bpf_run_filter_getsockopt
|
--0.81%--__kmalloc
With the patch applied:
0.52% 0.12% tcp_mmap [kernel.kallsyms] [k] __cgroup_bpf_run_filter_getsockopt_kern
Note, exporting uapi/tcp.h requires removing netinet/tcp.h
from test_progs.h because those headers have confliciting
definitions.
Signed-off-by: Stanislav Fomichev <sdf@google.com>
Signed-off-by: Alexei Starovoitov <ast@kernel.org>
Acked-by: Martin KaFai Lau <kafai@fb.com>
Link: https://lore.kernel.org/bpf/20210115163501.805133-2-sdf@google.com
Diffstat (limited to 'tools/testing/selftests/bpf/progs/sockopt_sk.c')
| -rw-r--r-- | tools/testing/selftests/bpf/progs/sockopt_sk.c | 23 |
1 files changed, 21 insertions, 2 deletions
diff --git a/tools/testing/selftests/bpf/progs/sockopt_sk.c b/tools/testing/selftests/bpf/progs/sockopt_sk.c index 712df7b49cb1..d3597f81e6e9 100644 --- a/tools/testing/selftests/bpf/progs/sockopt_sk.c +++ b/tools/testing/selftests/bpf/progs/sockopt_sk.c @@ -1,8 +1,8 @@ // SPDX-License-Identifier: GPL-2.0 #include <string.h> -#include <netinet/in.h> -#include <netinet/tcp.h> +#include <linux/tcp.h> #include <linux/bpf.h> +#include <netinet/in.h> #include <bpf/bpf_helpers.h> char _license[] SEC("license") = "GPL"; @@ -12,6 +12,10 @@ __u32 _version SEC("version") = 1; #define PAGE_SIZE 4096 #endif +#ifndef SOL_TCP +#define SOL_TCP IPPROTO_TCP +#endif + #define SOL_CUSTOM 0xdeadbeef struct sockopt_sk { @@ -57,6 +61,21 @@ int _getsockopt(struct bpf_sockopt *ctx) return 1; } + if (ctx->level == SOL_TCP && ctx->optname == TCP_ZEROCOPY_RECEIVE) { + /* Verify that TCP_ZEROCOPY_RECEIVE triggers. + * It has a custom implementation for performance + * reasons. + */ + + if (optval + sizeof(struct tcp_zerocopy_receive) > optval_end) + return 0; /* EPERM, bounds check */ + + if (((struct tcp_zerocopy_receive *)optval)->address != 0) + return 0; /* EPERM, unexpected data */ + + return 1; + } + if (ctx->level == SOL_IP && ctx->optname == IP_FREEBIND) { if (optval + 1 > optval_end) return 0; /* EPERM, bounds check */ |