[BUG] Memory leak in Rust Durable Object eviction

[lukevalenta]

Is there an existing issue for this?

• I have searched the existing issues

What version of workers-rs are you using?

0.5.0

What version of wrangler are you using?

4.10.0

Describe the bug

In Rust Workers, memory allocated for a Durable Object is not freed upon eviction. I'm able to reproduce locally with miniflare, and seem to have hit this same issue in a production worker (cloudflare/azul#11).

Steps To Reproduce

With the following files:

cat Cargo.toml
[package]
name = "memory-leak"
version = "0.1.0"
edition = "2021"

[package.metadata.release]
release = false

# https://github.com/rustwasm/wasm-pack/issues/1247
[package.metadata.wasm-pack.profile.release]
wasm-opt = false

[lib]
crate-type = ["cdylib"]

[dependencies]
worker = { version = "0.5.0" }

cat wrangler.jsonc
{
	"name": "memory-leak",
    "main": "build/worker/shim.mjs",
	"build": {
		"command": "cargo install -q worker-build && worker-build --release"
	},
	"compatibility_date": "2025-04-08",
	"migrations": [
		{
			"new_sqlite_classes": [
				"MyDurableObject"
			],
			"tag": "v1"
		}
	],
	"durable_objects": {
		"bindings": [
			{
				"class_name": "MyDurableObject",
				"name": "MY_DURABLE_OBJECT"
			}
		]
	},
	"observability": {
		"enabled": true
	}
}

cat src/lib.rs
use wasm_bindgen::prelude::*;
#[allow(clippy::wildcard_imports)]
use worker::*;

#[event(fetch, respond_with_errors)]
async fn main(_req: Request, env: Env, _ctx: Context) -> Result<Response> {
    let ns = env.durable_object("MY_DURABLE_OBJECT")?;
    let id = ns.id_from_name("foo")?;
    let stub = id.get_stub()?;
    stub.fetch_with_str("http://example.com").await
}

#[durable_object]
struct MyDurableObject {
    count: u64,
    _buffer: Vec<u8>,
}

#[durable_object]
impl DurableObject for MyDurableObject {
    fn new(state: State, _env: Env) -> Self {
        Self {
            count: 0,
            _buffer: Vec::with_capacity(100_000_000),
        }
    }
    #[allow(clippy::unused_async)]
    async fn fetch(&mut self, mut _req: Request) -> Result<Response> {
        self.count += 1;
        Response::ok(format!("hello {}", self.count))
    }
}

Then, run npx wrangler dev and hit the d key to open devtools. Now, take a heap memory snapshot, and then run the following in another terminal window to repeatedly re-initialize the DO (after it is evicted after 10s of inactivity):

while true; do curl http://localhost:8787; sleep 10; done

Take memory snapshots after every request and we can see it's continuously increasing by 100MB:

Now, with a JavaScript worker, we can see that there is no leak:

cat package.json
{
	"name": "memory-leak-js",
	"version": "0.0.0",
	"private": true,
	"scripts": {
		"deploy": "wrangler deploy",
		"dev": "wrangler dev",
		"start": "wrangler dev"
	},
	"devDependencies": {
		"wrangler": "^4.9.1"
	}
}
cat wrangler.jsonc
{
	"$schema": "node_modules/wrangler/config-schema.json",
	"name": "memory-leak-js",
	"main": "src/index.js",
	"compatibility_date": "2025-04-08",
	"migrations": [
		{
			"new_sqlite_classes": [
				"MyDurableObject"
			],
			"tag": "v1"
		}
	],
	"durable_objects": {
		"bindings": [
			{
				"class_name": "MyDurableObject",
				"name": "MY_DURABLE_OBJECT"
			}
		]
	},
	"observability": {
		"enabled": true
	}
}
cat src/index.js
import { DurableObject } from "cloudflare:workers";

export class MyDurableObject extends DurableObject {
	constructor(ctx, env) {
		super(ctx, env);
		this.count = 0;
		this.buffer = new Uint8Array(100_000_000)
	}

	async fetch() {
		this.count += 1
		return new Response('hello ' + this.count);
	}
}

export default {
	async fetch(request, env, ctx) {
    console.log('fetch');
		const id = env.MY_DURABLE_OBJECT.idFromName("foo");
		const stub = env.MY_DURABLE_OBJECT.get(id);
		return await stub.fetch("http://example.com");
	},
};

In devtools, I see this memory usage. The final snapshot is after I allowed the DO to be fully evicted (killing the script from earlier):

[lukevalenta]

Turns out this is a known bug, and should be solved soon when support for FinalizationRegistry is added.

See #653 for one possible workaround, although it didn't seem to help in the example above.

[lukevalenta]

Following up, it doesn't look like Finalization Registry support changed anything here. I'm still able to reproduce this with worker 0.6.5 and with compatibility date set to today: https://github.com/lukevalenta/workers-rs/tree/lvalenta/do-alarm-memory-leak/do-heap-memory-leak.

[PeterMHammond]

Following up, it doesn't look like Finalization Registry support changed anything here. I'm still able to reproduce this with worker 0.6.5 and with compatibility date set to today: https://github.com/lukevalenta/workers-rs/tree/lvalenta/do-alarm-memory-leak/do-heap-memory-leak.

Thank you for the extensive writeup and details @lukevalenta
Question: Have you confirmed this memory leak occurs when deployed to Cloudflare production, or only locally with miniflare?
Reason: I've encountered several DO anomalies that only manifest locally but work fine in production, so I'm curious if this is another case of local/production divergence.

If you've verified this in production (e.g., with the cloudflare/azul#11 worker you mentioned), that's definitely critical to know. - Thanks in advance.

[lukevalenta]

Hi @PeterMHammond yes, I've confirmed that this happens in a deployed worker, although you need to extend the timeout a little longer than 10s to get the Durable Object to hibernate (15s seems to work). Repro code is at https://github.com/lukevalenta/workers-rs/tree/lvalenta/do-alarm-memory-leak/do-heap-memory-leak and the deployed worker at https://do-heap-memory-leak.luke-valenta.workers.dev/.

I've also found a slower memory leak that I can trigger by spinning an alarm in a 1s loop: https://github.com/lukevalenta/workers-rs/tree/lvalenta/do-alarm-memory-leak/do-alarm-memory-leak. I've been able to confirm both via internal metrics and with npx wrangler dev and the devtools memory profile (see heap usage increasing in image below):

I'm following up with the team internally on both of these.

[PeterMHammond]

Understood @lukevalenta - thank you very much for the confirmation.
I use durable object alarms extensively so this is a big concern as we're going live 10/25 with our new implementation and if I understand this correctly it could have very bad impact on our billing?

[joshthoward]

Hey @PeterMHammond. EM for Durable Objects here. From our pricing page, we state that:

Duration billing charges for the 128 MB of memory your Durable Object is allocated, regardless of actual usage. If your account creates many instances of a single Durable Object class, Durable Objects may run in the same isolate on the same physical machine and share the 128 MB of memory. These Durable Objects are still billed as if they are allocated a full 128 MB of memory.

The presence of a memory leak is something that we take seriously, but the impact to customers is that the available memory to an application will decrease over time, not that customers bills will grow. Once the available 128 MB of memory is exhausted, the isolate will be condemned and your application will get a fresh isolate on a subsequent request.

[PeterMHammond]

Once the available 128 MB of memory is exhausted, the isolate will be condemned and your application will get a fresh isolate on a subsequent request.

Always love when my isolate is condemned - what does that mean for a practical experience?

Using a very bad example of a clock ticker - sending SSE updates to the client each second.
I assume it would mean that an active alarm that is updating my UX would just stop functioning when it gets "condemned" - with the user having no way to know why or how to restore it, with the end user thinking my super awesome clock is frozen and my billion dollar dream of selling my AI awesome clock is dead (had to put AI in it to sell the investors)?

[lukevalenta]

@PeterMHammond speaking from experience as a DO user and not authoritatively, when the isolate hits its memory limit the Durable Object will be re-initialized (and new called again) within a very short period of time. Durable Objects get evicted for all sorts of reasons (e.g., the metal its running on needs to be rebooted), and this is mostly transparent to users, but in-flight requests might be cancelled, and alarms might be slightly delayed.

From the docs:

Alarms have guaranteed at-least-once execution and are retried automatically when the alarm() handler throws.

Retries are performed using exponential backoff starting at a 2 second delay from the first failure with up to 6 retries allowed.