runtime error

[xzlxcs]

when I run “./bin/botsort_tracking_example ../config/tracker.ini ../config/gmc.ini ../config/reid.ini ../assets/osnet_x0_25_market1501.onnx ../examples/data/MOT20-01.mp4 ../examples/data/det/det.txt ../output/ “ in container(botsort-trt),it print:
Initializing ReID model
Trying to get input dims
Read input dims
Input dims: 1 3 256 128
ERROR: 3: [runtime.cpp::~Runtime::346] Error Code 3: API Usage Error (Parameter check failed at: runtime/rt/runtime.cpp::~Runtime::346, condition: mEngineCounter.use_count() == 1. Destroying a runtime before destroying deserialized engines created by the runtime leads to undefined behavior.
)
GMC disabled
[BoTSORT::track] - Average over 100 frames: 20.5359 ms (48.6953 FPS)
[BoTSORT::track] - Average over 100 frames: 20.7878 ms (48.1051 FPS)
[BoTSORT::track] - Average over 100 frames: 22.8051 ms (43.8498 FPS)
[BoTSORT::track] - Average over 100 frames: 21.2747 ms (47.0041 FPS)
Average tracker FPS: 50.2271
Average processing time per frame (ms): 19.9096

[AiVerisimilitude]

@viplix3 I took a look at it to see if config changes were the issue, and it seems to be an issue with how the engine is loaded where the runtime goes out of scope.

TensorRT_InferenceEngine.cpp

bool inference_backend::TensorRTInferenceEngine::_deserialize_engine(
...
// Runtime in created witin the scope of deserialize_engine -> destroyed at the end of this function call
std::unique_ptr<nvinfer1::IRuntime> runtime{
            nvinfer1::createInferRuntime(*_logger)};

// Engine is a member variable so it has the lifetime of the object rather than function
_engine = makeUnique(runtime->deserializeCudaEngine(
            trt_model_stream.data(), trt_model_stream.size()));
...

// After this return, the runtime will be destroyed triggering the check that appears in the output
return true;

I reckon it should be fine to put the runtime as a member as well and just take care of the order of destruction in the destructor but I am not as familiar with this part.

@xzlxcs Thanks for the report :)

[viplix3]

Thanks a lot for the detailed analysis, @AiVerisimilitude!

Apologies for the delay, @xzlxcs — I’ve been caught up with another side project outside of work hours.

I’m hoping to get back to BotSort in a week or two and start actively working on the open issues, along with some much-needed refactoring and overall improvements.

[xzlxcs]

@viplix3 According to your comments,I did some modifications,and then the runtime error disappeared,thanks.my modifications are:
class TensorRTInferenceEngine
{
private:
...

cudaStream_t _cuda_stream{nullptr};

...

TRTUniquePtr<nvinfer1::IRuntime> _runtime_xzl{nullptr};

...
}

inference_backend::TensorRTInferenceEngine::~TensorRTInferenceEngine()
{
// 释放执行上下文
_context.reset();

// 释放引擎
_engine.reset();

for (auto &buffer: _buffers)
    cudaFree(buffer);

// cudaStreamDestroy(_cuda_stream);
//由xzl添加
if (_cuda_stream != nullptr) {
    cudaError_t err = cudaStreamDestroy(_cuda_stream);
    if (err != cudaSuccess) {
        _logger->log(nvinfer1::ILogger::Severity::kERROR,
            std::string("Failed to destroy CUDA stream: " + std::string(cudaGetErrorString(err))).c_str());
    }
    _cuda_stream = nullptr; // 销毁后标记为null
}

// 最后释放 IRuntime
_runtime_xzl.reset();

}

bool inference_backend::TensorRTInferenceEngine::_deserialize_engine(
const std::string &engine_path)
{
// Reference: https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#perform_inference_c
std::ifstream engine_file(engine_path, std::ios::binary);
if (!engine_file)
{
_logger->log(nvinfer1::ILogger::Severity::kERROR,
std::string("Failed to open engine file").c_str());
return false;
}

// Read engine file
std::vector<char> trt_model_stream;
size_t size{0};
engine_file.seekg(0, std::ios::end);
size = engine_file.tellg();
engine_file.seekg(0, std::ios::beg);
trt_model_stream.resize(size);
engine_file.read(trt_model_stream.data(), size);
engine_file.close();

// Deserialize engine
// std::unique_ptr<nvinfer1::IRuntime> runtime{
//         nvinfer1::createInferRuntime(*_logger)};
// _engine = makeUnique(runtime->deserializeCudaEngine(
//         trt_model_stream.data(), trt_model_stream.size()));
// if (!_engine)
// {
//     _logger->log(nvinfer1::ILogger::Severity::kERROR,
//                  std::string("Failed to deserialize engine").c_str());
//     return false;
// }
// std::cout<<"kk"<<std::endl;
_runtime_xzl = makeUnique(nvinfer1::createInferRuntime(*_logger));
_engine = makeUnique(_runtime_xzl->deserializeCudaEngine(
        trt_model_stream.data(), trt_model_stream.size()));
if (!_engine)
{
    _logger->log(nvinfer1::ILogger::Severity::kERROR,
                 std::string("Failed to deserialize engine").c_str());
    return false;
}

// Create execution context
_context = makeUnique(_engine->createExecutionContext());
// std::cout<<std::string("Failed to create execution context").c_str()<<std::endl;
if (!_context)
{
    _logger->log(nvinfer1::ILogger::Severity::kERROR,
                 std::string("Failed to create execution context").c_str());
    return false;
}

return true;

}

[viplix3]

Thanks to @AiVerisimilitude!
@xzlxcs, if you don't mind, can you please raise a PR for these changes?

[xzlxcs]

Thanks to @AiVerisimilitude! @xzlxcs, if you don't mind, can you please raise a PR for these changes?

sorry ,I am not familiar with github,so i don't know how to raise a PR and I even don't know what it means.can you explain it for me?

[AiVerisimilitude]

@xzlxcs A pull request (PR) is a way to propose changes to a codebase by submitting your updates for review before they are merged into the main branch.

You can generally find documentation online but here is the github help page: Creating a pull request from a fork

[viplix3]

@xzlxcs @AiVerisimilitude solved by codex in this PR