Spark (4.0, 3.5): Set data file sort_order_id in manifest for writes from Spark

[jbewing]

This PR updates all writes from Spark 4.0 & 3.5 to set the sort_order_id of data files when applicable per the Iceberg Table Spec. I've opened this PR to:

1. Gauge interest in this and validate my approach
2. If the approach is good, make a decision on whether to backport these changes to Spark 3.4. This patch doesn't apply super cleanly to 3.4 as it stands right now, so I don't want to invest effort into it unless others would find value from it.
3. As a follow up to those PRs, implement the ability to then report this ordering information when applicable to Spark to better inform the query planner of potential optimizations. This is rather useful when used in conjunction with SPJ and I have this running on an Iceberg fork with success

This is a successor to my initial PR #13636 which has since been closed for being stale (but I've revived it here and ported the changes forwards to spark 4.0). I've re-opened this PR as of late there has been some increased interest in this:

• A community member has expressed interest in reviving my older PR Spark 3.5: Set data file sort_order_id in manifest when writing/compacting data files #13636 (comment)
• Other community members have reported this same bug on the issue list To persist sort order in new manifest files after it's been updated #14531

So it appears that there is value to these changes being upstreamed instead of confined to a fork.

Testing

I've added tests for newer added utility functions and updated existing tests that write data files and compact data files in a sorted manner to verify that we're setting the sort_order_id entry in the manifests to the correct value. Additionally, I've used this patch on an internal fork and verified that it correctly sets this field during compaction and normal writes.

Issue: #13634

[jbewing]

I'm not in love with this approach, but I went through a couple of iterations of this and plumbing the intended Iceberg SortOrder along through with the Spark SortOrder ended up being the cleanest in my opinion (open to any suggestions of course). I initially had an approach which would attempt to re-guess the Iceberg SortOrder from a Spark SortOrder. It worked pretty well, but was pretty naive in my opinion—the algorithm was to essentially reverse the Iceberg -> Spark SortOrder transformation as best as possible and then see if things matched up. It gets complicated when you start to consider Spark Orderings for stuff like say a positionDeltaUpdateMergeOrdering.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[anuragmantri]

Thanks for this PR @jbewing. I was working on similar optimizations and this PR is a pre-requisite for reporting sort orders to Spark. Overall this looks good to me.

I have used this PR as a base for implementing SupportsReportOrdering API here. Please take a look at that PR as well when you get a chance.

I would like more reviewers to take a look. Tagging @aokolnychyi who is an expert in this area.

[RussellSpitzer]

I'm very interested in this, but could you explain the changes in the PR in a summary? It looks like the distribution and ordering code is being redone?

Also I would recommend just doing the changes in 1 Spark version for review purposes, we can backport to anything applicable afterwards.