Issue in multiple appends in one transaction

[dor-bernstein]

Apache Iceberg version

None

Please describe the bug 🐞

Hey,
I have a large arrow table that I want to append to a partitioned iceberg table.
I'm working locally with dockers and I'm using the tabulario/iceberg-rest:1.6.0 as my rest catalog.
To avoid OOMs, I'm splitting the arrow table into chunks. When using regular appends everything works as expected. However, I want to append all data in a single transaction. I have this code that does that:

            with table.transaction() as tx:
                for offset in range(0, data.num_rows, MAX_APPEND_CHUNK_SIZE):
                    data_slice = data.slice(offset, MAX_APPEND_CHUNK_SIZE)
                    logger.info(f'Writing batch of {data_slice.num_rows} with offset {offset} to table {table.name()}')
                    tx.append(data_slice)
                tx.commit_transaction()

The table is empty and was created in a different task.
I get the following error - CommitFailedException: Requirement failed: branch main was created concurrently.
When retrying I get this error pyiceberg.exceptions.CommitFailedException: CommitFailedException: Requirement failed: branch main has changed: expected id 4547037169132709864 != 132570956257248456.

Any help would be appreciated,
Thanks!

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[Fokko]

Hey @dor-bernstein Thanks for raising this. Do you have the full stack trace by any chance?

I see in the code that you probably commit twice:

with table.transaction() as tx:
    for offset in range(0, data.num_rows, MAX_APPEND_CHUNK_SIZE):
        data_slice = data.slice(offset, MAX_APPEND_CHUNK_SIZE)
        logger.info(f'Writing batch of {data_slice.num_rows} with offset {offset} to table {table.name()}')
        tx.append(data_slice)
    tx.commit_transaction() <-- this will commit
<-- Leaving the context manager will also trigger a commit

Please let me know, it would be great to get this fixed 🚀

[dor-bernstein]

@Fokko Can I leave the context manager without triggering a commit? I prefer to be more explicit

[Fokko]

Hey @dor-bernstein If there are any updates left, it will still try to process them.

#1961 will also allow you to commit within the context manager. Once the PR is in, the code above will pass without any error. Let me know what you think

[dor-bernstein]

@Fokko looks good, thanks for handling this so quickly!

[tom-at-rewbi]

is the intended behavior for multiple commits within a single transaction that pending appends are uploaded without completing the transaction?