feat: Add a script for debugging Memgraph using ephemeral containers#203
Merged
feat: Add a script for debugging Memgraph using ephemeral containers#203
Conversation
Member
|
If you haven't alrady started, make sure to update/expand https://memgraph.com/docs/database-management/debugging |
antejavor
approved these changes
Feb 26, 2026
11 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Add a convenience script
scripts/debug-memgraph.shthat attaches a GDB debug container to a running Memgraph HA pod usingkubectl debugephemeral containers. The scriptauto-detects the target container name (
memgraph-dataormemgraph-coordinator) from the pod name, creates a temporary custom profile to override the pod's non-rootsecurityContext, and attaches GDB with-ex continueso the process keeps running until a crash signal is caught. Also adds GDB debugging instructions tocharts/memgraph-high-availability/templates/NOTES.txt.Why
When debugging Memgraph crashes caused by specific queries in HA deployments, developers need to attach GDB to the running process. Restarting pods is undesirable because
Memgraph recovery can be slow for large datasets. This script wraps the
kubectl debugworkflow and handles the non-trivial security context override needed because Memgraphpods run as non-root (uid 101) with
runAsNonRoot: true.How
securityContext(runAsUser: 101,runAsNonRoot: true) prevents ephemeral containers from running as root, which blocks bothapt-get installandptrace. The script generates a temporary--customprofile JSON that setsrunAsUser: 0on the ephemeral container only, used together with--profile=sysadminfor fullSYS_PTRACEcapability.-ex continueso the Memgraph process is not paused — it keeps running and GDB only breaks on crash signals (SIGSEGV, SIGABRT, etc.).kubectl debug --targethandles PID namespace sharing without requiringshareProcessNamespaceon the podspec.
Files changed:
scripts/debug-memgraph.sh— New convenience wrapper scriptTesting
pgrep -x memgraphfinds the process PID through shared PID namespacekubectl exec <pod> -c memgraph-data -- kill -SIGSEGV <PID>Notes for reviewers
kubectl1.32+ (the--customflag forkubectl debugwent GA in 1.32). Older kubectl versions will fail with an unknown flag error.apt-get install gdbstep inside the ephemeral container adds ~30s latency before GDB attaches. A follow-up could add a pre-built debug image with GDB already installed,or a gdbserver sidecar approach for instant attach.
trap ... EXIT.