qat engine coredump in ASYNC_get_wait_ctx()

[wanlebing]

HI all：

I met a coredump with the backtrace below. The application calls ASYNC_start_job, and the crash occurs when RSA_private_encrypt returns 0.

#0  0x000000000096ff30 in ASYNC_get_wait_ctx ()
#1  0x00007f57352194a4 in qat_wake_job (job=<optimized out>, jobStatus=2) at qat_events.c:306
#2  0x00007f5734e33138 in LacPke_MsgCallback () from //opt/QAT/build/libqat_s.so
#3  0x00007f5734e58d13 in adf_user_notify_msgs_poll () from //opt/QAT/build/libqat_s.so
#4  0x00007f5734e5216c in adf_pollRing () from //opt/QAT/build/libqat_s.so
#5  0x00007f5734e525da in icp_adf_pollInstance () from //opt/QAT/build/libqat_s.so
#6  0x00007f5734e4cc2d in icp_sal_CyPollInstance () from //opt/QAT/build/libqat_s.so
#7  0x00007f573521c9a4 in qat_timer_poll_func (ih=<optimized out>) at qat_hw_polling.c:200
#8  0x00007f5739c081ca in start_thread () from /lib64/libpthread.so.0
#9  0x00007f57372398d3 in clone () from /lib64/libc.so.6

It appears the root cause is that ASYNC_start_job did not effectively start a valid ASYNC job at the application layer, but the underlying layer still generated a QAT asynchronous task. Consequently, qat_wake_job retrieved an invalid ASYNC job, leading to the core dump.

I wonder if this is a known issue, any help would be appreciated, thanks.

[Yogaraj-Alamenda]

Async_jobs are created and maintained by application and QAT Engine checks ASYNC_get_current_job() and if it has non-NULL pointer, it gets into Async Flow. That said if there is no valid ASync Job started then I would expect failure at qat_pause_job() but here it is failing during wake job which is strange. Can you please check what is happening during qat_pause_job() ?

[wanlebing]

Async_jobs are created and maintained by application and QAT Engine checks ASYNC_get_current_job() and if it has non-NULL pointer, it gets into Async Flow. That said if there is no valid ASync Job started then I would expect failure at qat_pause_job() but here it is failing during wake job which is strange. Can you please check what is happening during qat_pause_job() ?

sorry for the delay, here's our implementation:

firstly, in the application, external calls trigger internal execution:

ASYNC_start_job(&job->asyncjob, job->waitctx, &job_ret,
                               async_job_func, &args, sizeof(struct async_args)));
RSA_private_encrypt is called within async_job_func.

The call to RSA_private_encrypt within async_job_func succeeds, this is the expected scenario:

1. The call to ASYNC_start_job is expected to return ASYNC_PAUSE and initiate an asynchronous task within the QAT engine.
2. Once the QAT engine polls and completes the task, it notifies the application layer via a callback.
3. The application layer then invokes ASYNC_start_job(&job->asyncjob, job->waitctx, &ret, NULL, NULL, 0); again; this returns ASYNC_FINISH, at which point the asyncjob is freed by OpenSSL.

The call to RSA_private_encrypt within async_job_func fails, which is an abnormal scenario:

1. The call to async_job_func fails, and ASYNC_start_job immediately returns ASYNC_FINISH, Consequently, the asyncjob is freed by OpenSSL.
2. However, the QAT engine's asynchronous task is still generated. When the QAT engine polls for job completion and triggers the callback to notify the application layer, it cannot retrieve a valid job. This results in a core dump during qat_wake_job.

From what we can observe, that seems to be the case:

1. ASYNC_start_job initiates a job and triggers a QAT asynchronous task. The underlying execution begins, and the job context is switched out.
2. When the job switches back to execute the function, it fails, and the job is subsequently freed by OpenSSL.
3. qat_wake_job cannot retrieve the valid job, resulting in a core dump.

Thanks again.

[venkatesh6911]

@wanlebing I will check and come back soon.