Avro reader memory leak

[Declow]

Apache Iceberg version

version = "0.9.1"

Please describe the bug 🐞

It seems like there is a memory leak in the avro/reader.py
I have a long running service that keeps crashing. I tried to replicate the issue locally and it seems it also has this issue.

The following code creates a Memory catalog and generates some random data for ingestion into iceberg.

from pyiceberg.catalog.memory import InMemoryCatalog
import tracemalloc
from datetime import datetime, timezone
import polars as pl

def generate_df():
    df = pl.DataFrame(
        {
            "event_type": ["playback"] * 1000,
            "event_origin": ["origin1"] * 1000,
            "event_send_at": [datetime.now(timezone.utc)] * 1000,
            "event_saved_at": [datetime.now(timezone.utc)] * 1000,
            "data": [
                {
                    "calendarKey": "calendarKey",
                    "id": str(i),
                    "referenceId": f"ref-{i}",
                }
                for i in range(1000)
            ],
        }
    )
    return df

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")
table = iceberg_table = catalog.create_table(
    "default.leak", schema=df.to_arrow().schema, location="/tmp/iceberg/leak"
)

df = pl.DataFrame()

tracemalloc.start()
for i in range(1000):
    df = generate_df()
    df.write_iceberg(table, mode="append")
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics("lineno")
    for stat in top_stats[:10]:
        print(stat)

Slowly but steadily the outputs for the avro reader memory size increases

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:330: size=370 KiB, count=3782, average=100 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:190: size=222 KiB, count=1891, average=120 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:133: size=184 KiB, count=5673, average=33 B

After some more writes the output looks like this

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:330: size=420 KiB, count=4290, average=100 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:190: size=251 KiB, count=2145, average=120 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:133: size=208 KiB, count=6435, average=33 B

If we take a look at the AvroFile class it uses the enter and exit dunder methods. The enter method assigns the reader to a variable on the instance but it seems like the different reader classes sticks around.
https://github.com/apache/iceberg-python/blob/main/pyiceberg/avro/file.py#L192

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Declow]

I think I found the root cause.
If we only print the avro/reader

    for i in range(1000):
        df = generate_df()
        df.write_iceberg(table, mode="append")
        snapshot = tracemalloc.take_snapshot()
        top_stats = snapshot.statistics("lineno")
        print(i)
        for stat in top_stats:
            if "avro/reader" in str(stat):
                print(stat)

In manifest.py a cache is set https://github.com/apache/iceberg-python/blob/main/pyiceberg/manifest.py#L878
If I set this to 1 the following will always increase but by a very tiny amount 200 byte per write.

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:322: size=1464 B, count=23, average=64 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:372: size=728 B, count=13, average=56 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:295: size=620 B, count=1, average=620 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:453: size=590 B, count=1, average=590 B

The higher the max cache value the faster the memory increases.

If I remove the cache altogether the memory will be released correctly no matter the amount of iterations.

[sungwy]

Hi @Declow — thanks for reporting this issue. If your observation is correct, it’s definitely a serious matter, and we should address it promptly. I appreciate your initiative in digging into the root cause.

/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:322: size=1464 B, count=23, average=64 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:372: size=728 B, count=13, average=56 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:295: size=620 B, count=1, average=620 B
/Users/dits/git/play-recommendation-input-consumer/.venv/lib/python3.11/site-packages/pyiceberg/avro/reader.py:453: size=590 B, count=1, average=590 B

Even with the cache set to 1, it still stores a tuple of ManifestFile objects, which grows with each new commit. Since each ManifestFile has the base overhead of a Python object instance plus some attributes (e.g., file path, partition), an increase of around ~100 B to ~500 B per commit seems reasonable.

Did you notice any memory usage growth beyond this range that might indicate a more substantial leak?

[Declow]

For my toy example here it was not that bad. When I used our production setup with a lot more manifest files the service went from using 800MB RAM to reaching it max of 1556MB and crashing within 5 minutes. It was using almost 10MB for each commit maybe more.
When I fixed the issue it on the service by removing the cache everything was stable.

[sungwy]

I'm wondering if that just maybe the size of a large manifest list * 128. How many manifests are you materializing in a single manifest-list path? Do you have an estimate of the memory used for a single return of _manifests, i.e. tuple(read_manifest_list(file)) for your Iceberg table?

[kevinjqliu]

Thanks for creating the issue. The repro script is super helpful.

This is my current hypothesis:
I think what you're observing is the side affect of the write pattern and its interaction with the _manifests LRU cache.

append the dataframe here will always create a new manifest entry since we use the "fast append" strategy.
This means each write will generate a new manifest list. _manifests, with its LRU cache, will cache the new manifest list's path along with the tuple of ManifestFile objects.
Each cached manifest list path will have overlapping ManifestFile objects since we're constantly adding new manifest files through append.
This means in a LRU cache of size 128, we cache 128 manifest file paths. the first cached item will have 1 tuple of ManifestFile, and the 128th cached item will have 128 tuples of ManifestFile. Most of these ManifestFile are referring to the same underlying manifest file but in a python new object.

# cache key: value, str: tuple[ManifestFile]
1: (ManifestFile1)
2: (ManifestFile1, ManifestFile2)
3: (ManifestFile1, ManifestFile2, ManifestFile3)
...
n: (ManifestFile1, ..., ManifestFilen-1, ManifestFilen)

Each of these ManifestFile objects will refer to the data read using the avro reader.
This is the top allocation, even above the avro reader objects. I believe these are just ManifestFile objects

/Users/kevinliu/repos/iceberg-python/pyiceberg/typedef.py:183: size=1146 KiB, count=12750, average=92 B

Heres a modified repro script:

from pyiceberg.catalog.memory import InMemoryCatalog
import tracemalloc
from datetime import datetime, timezone
import polars as pl

def generate_df():
    df = pl.DataFrame(
        {
            "event_type": ["playback"] * 1000,
            "event_origin": ["origin1"] * 1000,
            "event_send_at": [datetime.now(timezone.utc)] * 1000,
            "event_saved_at": [datetime.now(timezone.utc)] * 1000,
            "data": [
                {
                    "calendarKey": "calendarKey",
                    "id": str(i),
                    "referenceId": f"ref-{i}",
                }
                for i in range(1000)
            ],
        }
    )
    return df

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")

df = generate_df()
catalog = InMemoryCatalog("default", warehouse="/tmp/iceberg")
catalog.create_namespace("default")
table = iceberg_table = catalog.create_table(
    "default.leak", schema=df.to_arrow().schema, location="/tmp/iceberg/leak"
)

df = pl.DataFrame()
import gc, objgraph

from pyiceberg.manifest import _manifests

def get_max_value_size(manifest_cache):
    max_key, max_size = max(
        ((key, len(value)) for key, value in manifest_cache.items()),
        key=lambda item: item[1],
        default=(None, -1)
    )
    return max_key, max_size


tracemalloc.start()
for i in range(1000):
    df = generate_df()
    df.write_iceberg(table, mode="append")

    # debug
    manifest_cache = _manifests.cache
    print("ManifestFile instances alive:", objgraph.count("ManifestFile"))
    print(f"size: {len(manifest_cache)}, keys: {len(manifest_cache)}, values: {len(manifest_cache)}")
    max_key, max_size = get_max_value_size(manifest_cache)
    print(f"Max size of the cache value tuple: {max_size}, key: {max_key}")
    # gc.collect()
    snapshot = tracemalloc.take_snapshot()
    top_stats = snapshot.statistics("lineno")
    for stat in top_stats[:10]:
        print(stat)

    print()

Let me know if this makes sense. I'd like to hear what others think

[Declow]

@kevinjqliu It makes sense that the list of manifest files keeps increasing as the write continues but it is not root cause.

As long as the function is cached the AvroFileHeader is never GC'ed. These objects are created each time a AvroFile is read inside of read_manifest_list.
If you run the script with any cache size it will continue to increase the memory usage.
/avro/reader.py:330: size=116 KiB, count=1190, average=100 B
If you run the script for more than 50 iterations.

[kris-gaudel]

Flame graph and memory graph after running the script with 50 iterations

[kris-gaudel]

After 100 iterations

[andormarkus]

We're experiencing this exact same memory leak in production AWS Lambda functions using pyiceberg. Our data shows the same avro/reader.py line growth:

Memory progression across Lambda invocations (same 50MB file):

• 1st run: reader.py:330: 52.8 MiB
• 2nd run: reader.py:330: 88.0 MiB
• 3rd run: reader.py:330: 106 MiB

Memory grows from 1.4GB to 2GB+ after just 5 invocations processing identical files. Despite aggressive cleanup attempts with gc.collect() and libc.malloc_trim(0), the avro reader allocations persist between Lambda executions.

Our logs confirm @Declow's findings. This is causing production Lambda functions to hit memory limits and crash.

Happy to test any proposed fixes in our production environment with concurrent Lambda executions.

[kevinjqliu]

thanks for all the great discussion here. I want to also mention #2004, where we're looking to replace the cython avro reader with one from iceberg-rust. this might help but i think the main issue is still with the cache.

As long as the function is cached the AvroFileHeader is never GC'ed. These objects are created each time a AvroFile is read inside of read_manifest_list.

this is also what im seeing. but it looks like it we manually clear the cache, gc will clean up the objects

@kris-gaudel 's graph shows that at least this is bounded... :)

@andormarkus could you try changing the cache size or manually clear the cache here?

iceberg-python/pyiceberg/manifest.py

Line 878 in 19efd2d

@cached(cache=LRUCache(maxsize=128), key=lambda io, manifest_list: hashkey(manifest_list))

[andormarkus]

I have implemented manual clearing of the _manifests cache but the memory leak persists in AWS Lambda.

Environment: AWS Lambda, PyIceberg with Glue catalog

Cache Clearing Implementation:

def cleanup(self):
    """Clear PyIceberg cachetools LRUCache to prevent memory leaks"""
    try:
        # Clear the specific cachetools LRUCache that's causing the memory leak
        import pyiceberg.manifest as manifest_module
        
        # The _manifests function has @cached(cache=LRUCache(maxsize=128))
        if hasattr(manifest_module, '_manifests'):
            manifests_func = getattr(manifest_module, '_manifests')
            # cachetools @cached decorator stores cache in the function
            if hasattr(manifests_func, 'cache'):
                cache_size = len(manifests_func.cache)
                manifests_func.cache.clear()
                logger.info(f"Cleared PyIceberg manifest cache ({cache_size} entries)")
                        
    except Exception as e:
        logger.warning(f"Could not clear PyIceberg cache: {e}")
    
    # Clean up objects
    if hasattr(self, 'catalog'):
        del self.catalog
    if hasattr(self, 'client'):
        del self.client
    gc.collect()

Results: Memory still grows from 1438MB → 1858MB across invocations. The manifest cache clearing had minimal impact.

[Declow]

Well there is a trend up. This is after 300. It does not seem bounded.

@andormarkus Clearing the cache does help

But even if I remove the cache completely it still increases. The caching just makes it way worse.

[kris-gaudel]

I was running memray on 1000 iterations and the usage increases over time significantly (detailed via the logs). I couldn't generate a flame graph bc of the binary size being too large

iceberg-python/pyiceberg/avro/file.py

Line 192 in a33cd38

"""Perform cleanup when exiting the scope of a 'with' statement."""