Use asyncio.to_thread to avoid DeadlockError while sending exception to Sentry#267
Use asyncio.to_thread to avoid DeadlockError while sending exception to Sentry#267Natim wants to merge 1 commit intotemporalio:mainfrom
asyncio.to_thread to avoid DeadlockError while sending exception to Sentry#267Conversation
|
|
|
Well, well |
| if not workflow.unsafe.is_replaying(): | ||
| with workflow.unsafe.sandbox_unrestricted(): | ||
| scope.capture_exception() | ||
| await asyncio.to_thread(scope.capture_exception, e) |
There was a problem hiding this comment.
This is not legal code from a workflow, and as you saw, is not supported by our event loop (by intention). Two things here...
First, Sentry is on the verge of a 3.x SDK which will include async support (per getsentry/sentry-python#2824 (comment)). They appear to have released a few alphas already.
Second, not sure we should assume they want to use the default executor and you're going to have to use an intentionally-undocumented feature called "unsafe extern" here (we use this for a similar purpose for OTel at https://github.com/temporalio/sdk-python/blob/main/temporalio/contrib/opentelemetry.py). I would suggest doing the following:
- Have the constructor of
SentryInterceptoraccept an optionalThreadPoolExecutor, and capture the current event loop as a private attribute on the class - Have a
def _capture_exception(self, scope: sentry_sdk.Scope, exception: Exception)that calls the loop'srun_in_executorwith the known thread pool executor for scope'scapture_exception - Pass the
SentryInterceptorto_SentryActivityInboundInterceptor's interceptor so it can use this - On
workflow_interceptor_class, callinput.unsafe_extern_functions.update("__temporal_sentry_capture_exception", self._capture_exception) - In
_SentryWorkflowInterceptorinstead of what's called today, calltemporalio.workflow.extern_functions()["__temporal_sentry_capture_exception"](scope, e)
But would understand if there isn't value in doing all of this if it's just going to be outdated shortly with v3 SDK
What was changed
Make the capture_exception call async in Sentry intercepor
Why?
Use
asyncio.to_threadto avoidDeadlockErrorwhile sending exception to SentryChecklist
Refs feat: add example using Sentry V2 SDK #140
How was this tested:
In production _DeadlockError was raised
Any docs updates needed?
No