Sentry Storage Space Issue - Large Volume Usage

[yyzhou0811]

Self-Hosted Version

25.11.1

CPU Architecture

x86_64

Docker Version

28.1.1

Docker Compose Version

2.37.1

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade from self-hosted-25.11.0 to self-hosted-25.11.1

Steps to Reproduce

Current Storage Usage

• sentry-seaweedfs: 1.653TB (extremely large)
• sentry-kafka: 1.706TB (also very large)
• Total storage: ~3.4TB

⚙️ Current Configuration

SENTRY_EVENT_RETENTION_DAYS=7
KAFKA_LOG_RETENTION_HOURS=7
KAFKA_LOG_RETENTION_BYTES=53687091200  # 50GB
KAFKA_LOG_SEGMENT_BYTES=1073741824      # 1GB
KAFKA_LOG_CLEANUP_POLICY=delete
KAFKA_LOG_CLEANER_ENABLE=true

1. SeaweedFS Issues

From volume status analysis:
- Total volumes: 806 volumes (570/571 active, 1 free)
- Read-only volumes: 383 volumes marked as read_only
- Deleted data pending cleanup: 968GB
- Only 1 free volume remaining - Critical capacity issue


Configuration Issues
- Despite 7-day retention, storage continues to grow
- SeaweedFS garbage collection not working effectively
- Kafka logs accumulating despite retention policy

Troubleshooting Steps Taken
SeaweedFS Cleanup Attempts
- ✅ Attempted collection.deleteEmpty nodestore - Command not available
- ✅ Attempted volume.fix -collection=nodestore - Command not available
- ✅ Verified SeaweedFS 3.96 version compatibility issues
- ❌ Unable to identify correct cleanup commands for this version

Configuration Verification
- ✅ Confirmed SENTRY_EVENT_RETENTION_DAYS=7 is set
- ✅ Verified Kafka retention policies are configured
- ❌ Retention policies not effectively reducing storage

### Expected Result

With 7-day retention, storage usage should stabilize around a much smaller size (few hundred GB
max), not continue growing to 3.4TB. The system should automatically clean up disk space by
running garbage collection, processing read-only volumes, cleaning old Kafka logs, and providing
 manual cleanup commands when needed

### Actual Result

<img width="754" height="72" alt="Image" src="https://github.com/user-attachments/assets/1f49dafe-4c3e-41ed-bb11-d59d7576b9a6" />

<img width="680" height="75" alt="Image" src="https://github.com/user-attachments/assets/cda54526-d37f-4016-aaab-98a0f026bedf" />

<img width="1247" height="223" alt="Image" src="https://github.com/user-attachments/assets/7adc8d2b-29c2-4cad-aa21-f45a7ee1d7f2" />

### Event ID

_No response_

[LordSimal]

I am not a sentry dev, but I can give you some info on what this tells me from my experience using self hosted sentry over the years.

High seaweed FS usage (previously this was in a postgres table) means, that a lot of "nodestorage" is being utilized.
Node Storage is used in a buch of different places, but primarily for me it filled up quickly when I had a lot of "performance monitoring" active - meaning spans and transactions for the Insights area in sentry.

You can find out which projects send you a 💩 load of transactions and spans via going into your Settings => Stats & Usage page and checking each project in each category over the last e.g. 14 days.

If you find projects, which have millions of entries in there, I'd recommend adjusting your sample rate to not send all events to sentry, but only e.g.10% or whatever you feel comfortable.

---

Why kafka is so huge for you I can't tell you. My kafka volume has 22 GB and my seaweedFS volume has 130GB

You can try to look at your kafka cluster via setting up a web UI, but I'd say its not really a kafka problem and more a problem of the sheer amount of data you have to process.

There is also of course https://develop.sentry.dev/self-hosted/troubleshooting/kafka/#reducing-disk-usage but again I don't think thats your real solution.

[yyzhou0811]

I am not a sentry dev, but I can give you some info on what this tells me from my experience using self hosted sentry over the years.

High seaweed FS usage (previously this was in a postgres table) means, that a lot of "nodestorage" is being utilized. Node Storage is used in a buch of different places, but primarily for me it filled up quickly when I had a lot of "performance monitoring" active - meaning spans and transactions for the Insights area in sentry.

You can find out which projects send you a 💩 load of transactions and spans via going into your Settings => Stats & Usage page and checking each project in each category over the last e.g. 14 days.

If you find projects, which have millions of entries in there, I'd recommend adjusting your sample rate to not send all events to sentry, but only e.g.10% or whatever you feel comfortable.

Why kafka is so huge for you I can't tell you. My kafka volume has 22 GB and my seaweedFS volume has 130GB

You can try to look at your kafka cluster via setting up a web UI, but I'd say its not really a kafka problem and more a problem of the sheer amount of data you have to process.

There is also of course https://develop.sentry.dev/self-hosted/troubleshooting/kafka/#reducing-disk-usage but again I don't think thats your real solution.

Thank you so much for the detailed explanation! This is incredibly helpful.