Skip to content

example/rdma_performance 地址消毒偶发heap-use-after-free #3273

Description

@daming6

Describe the bug
在两个shell窗口分别执行如下命令:
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset -c 192-231 ./server -use_rdma 0 -thread_num 40

ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset -c 232-271 ./client -thread_num 40 -queue_depth 40 -attachment_size 37136 -use_rdma 0 -connection_type pooled

加上ASAN_OPTIONS=detect_leaks=0 ,client并发总数据量(-thread_num * -queue_depth * -attachment_size)较大,压力中等时仍能偶发出现heap-use-after-free,说明是有效问题
具体的地址消毒函数栈报错如下:

==180482==ERROR: AddressSanitizer: heap-use-after-free on address 0xffffb44092c8 at pc 0x000000425e68 bp 0xffffab7fe9e0 sp 0xffffab7fe9d0
READ of size 4 at 0xffffb44092c8 thread T10 (brpc_wkr:0-2)
#0 0x425e64 in PerformanceTest::HandleResponse(PerformanceTest::RespClosure*) (/home/ci/brpc/pkgs/example/rdma_performance/client+0x425e64)
#1 0x4115a0 in brpc::internal::FunctionClosure1PerformanceTest::RespClosure*::Run() (/home/ci/brpc/pkgs/example/rdma_performance/client+0x4115a0)
#2 0xffffba89d9dc in brpc::Controller::EndRPC(brpc::Controller::CompletionInfo const&) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6ad9dc)
#3 0xffffba8a1f88 in brpc::Controller::OnVersionedRPCReturned(brpc::Controller::CompletionInfo const&, bool, int) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6b1f88)
#4 0xffffbab9b650 in brpc::policy::ProcessRpcResponse(brpc::InputMessageBase*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x9ab650)
#5 0xffffba909358 in brpc::ProcessInputMessage(void*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x719358)
#6 0xffffba90ff40 in brpc::InputMessenger::OnNewMessages(brpc::Socket*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x71ff40)
#7 0xffffbaa2db78 in brpc::Socket::ProcessEvent(void*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x83db78)
#8 0xffffba717c98 in bthread::TaskGroup::task_runner(long) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x527c98)
#9 0xffffba68629c in bthread_make_fcontext (/home/ci/brpc/pkgs/lib/libbrpc.so+0x49629c)

0xffffb44092c8 is located 24 bytes inside of 72-byte region [0xffffb44092b0,0xffffb44092f8)
freed by thread T6 (brpc_wkr:0-5) here:
#0 0xffffbc443f8c in operator delete(void*, unsigned long) (/usr/lib64/libasan.so.8.0.0+0xb3f8c)
#1 0x40da44 in DeleteTest(void*) (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40da44)
#2 0xffffba717c98 in bthread::TaskGroup::task_runner(long) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x527c98)
#3 0xffffba68629c in bthread_make_fcontext (/home/ci/brpc/pkgs/lib/libbrpc.so+0x49629c)

previously allocated by thread T0 here:
#0 0xffffbc442ecc in operator new(unsigned long) (/usr/lib64/libasan.so.8.0.0+0xb2ecc)
#1 0x40e490 in Test(int, int) (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40e490)
#2 0x40abe8 in main (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40abe8)
#3 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#4 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#5 0x40b82c in _start (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)

Thread T10 (brpc_wkr:0-2) created by T0 here:
#0 0xffffbc3db790 in __interceptor_pthread_create (/usr/lib64/libasan.so.8.0.0+0x4b790)
#1 0xffffba6cb2ec in bthread::TaskControl::init(int) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x4db2ec)
#2 0xffffba67c570 in bthread::get_or_new_task_control() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x48c570)
#3 0xffffba679ad4 in bthread_start_background (/home/ci/brpc/pkgs/lib/libbrpc.so+0x489ad4)
#4 0xffffba8e995c in brpc::GlobalInitializeOrDieImpl() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f995c)
#5 0xffffb8b04b68 (/usr/lib64/libc.so.6+0x84b68)
#6 0xffffba8e4d10 in brpc::GlobalInitializeOrDie() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f4d10)
#7 0xffffba9c98e0 in brpc::Server::InitializeOnce() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7d98e0)
#8 0xffffba9e6658 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7f6658)
#9 0xffffba9ec5ec in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fc5ec)
#10 0xffffba9ecb34 in brpc::Server::Start(int, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fcb34)
#11 0xffffba9ed114 in brpc::StartDummyServerAt(int, brpc::ProfilerLinker) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fd114)
#12 0x40a98c in main (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40a98c)
#13 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#14 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#15 0x40b82c in _start (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)

Thread T6 (brpc_wkr:0-5) created by T0 here:
#0 0xffffbc3db790 in __interceptor_pthread_create (/usr/lib64/libasan.so.8.0.0+0x4b790)
#1 0xffffba6cb2ec in bthread::TaskControl::init(int) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x4db2ec)
#2 0xffffba67c570 in bthread::get_or_new_task_control() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x48c570)
#3 0xffffba679ad4 in bthread_start_background (/home/ci/brpc/pkgs/lib/libbrpc.so+0x489ad4)
#4 0xffffba8e995c in brpc::GlobalInitializeOrDieImpl() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f995c)
#5 0xffffb8b04b68 (/usr/lib64/libc.so.6+0x84b68)
#6 0xffffba8e4d10 in brpc::GlobalInitializeOrDie() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x6f4d10)
#7 0xffffba9c98e0 in brpc::Server::InitializeOnce() (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7d98e0)
#8 0xffffba9e6658 in brpc::Server::StartInternal(butil::EndPoint const&, brpc::PortRange const&, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7f6658)
#9 0xffffba9ec5ec in brpc::Server::Start(butil::EndPoint const&, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fc5ec)
#10 0xffffba9ecb34 in brpc::Server::Start(int, brpc::ServerOptions const*) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fcb34)
#11 0xffffba9ed114 in brpc::StartDummyServerAt(int, brpc::ProfilerLinker) (/home/ci/brpc/pkgs/lib/libbrpc.so+0x7fd114)
#12 0x40a98c in main (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40a98c)
#13 0xffffb8aa76c0 (/usr/lib64/libc.so.6+0x276c0)
#14 0xffffb8aa77a4 in __libc_start_main (/usr/lib64/libc.so.6+0x277a4)
#15 0x40b82c in _start (/home/ci/brpc/pkgs/example/rdma_performance/client+0x40b82c)

SUMMARY: AddressSanitizer: heap-use-after-free (/home/ci/brpc/pkgs/example/rdma_performance/client+0x425e64) in PerformanceTest::HandleResponse(PerformanceTest::RespClosure*)
Shadow bytes around the buggy address:
0x200ff6881200: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881210: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881220: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881230: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x200ff6881240: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
=>0x200ff6881250: fd fa fa fa fa fa fd fd fd[fd]fd fd fd fd fd fa
0x200ff6881260: fa fa fa fa fd fd fd fd fd fd fd fd fd fa fa fa
0x200ff6881270: fa fa fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x200ff6881280: fd fd fd fd fd fd fd fd fd fa fa fa fa fa fd fd
0x200ff6881290: fd fd fd fd fd fd fd fa fa fa fa fa fd fd fd fd
0x200ff68812a0: fd fd fd fd fd fa fa fa fa fa fd fd fd fd fd fd
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
==180482==ABORTING

To Reproduce
在两个shell窗口分别执行如下命令:
ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset -c 192-231 ./server -use_rdma 0 -thread_num 40

ASAN_OPTIONS=detect_leaks=0 LD_PRELOAD=/usr/lib64/libasan.so.8.0.0 taskset -c 232-271 ./client -thread_num 40 -queue_depth 40 -attachment_size 37136 -use_rdma 0 -connection_type pooled

加上ASAN_OPTIONS=detect_leaks=0 ,压力中等时仍能偶发出现heap-use-after-free,说明是有效问题

Expected behavior
预期example/rdma_performance -use_rdma 0 默认走TCP协议应该不会出现多线程之间heap-use-after-free问题,可能是example/rdma_performance/server client为了性能,线程之间的保护有点欠缺

Versions
OS: openEuler 24.03 (LTS-SP2)
Compiler: gcc 12.3.1
brpc: 1.16
protobuf: protobuf-25.1-12.oe2403sp2.aarch64

Additional context/screenshots

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions