`DataScan` `count` method does not respect limit

[jayceslesar]

Apache Iceberg version

0.9.1 (latest release)

Please describe the bug 🐞

When calling count() on a DataScan, limit is not respected. Seems trivial but if I set a limit of 5 I expect 5 or less rows back, at least with a scan-like implementation

The underlying ArrowScan does not get passed the limit param

https://github.com/apache/iceberg-python/blob/main/pyiceberg/table/__init__.py#L1940

This results in scans taking longer due to not respecting the limit.

The fix will involve more than just passing the limit to the ArrowScan

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time

[tusharchou]

Oh this seems important, i will groom this issue

[rashampreet-singh]

HI @jayceslesar I would like to work on this issue.

[tushar-choudhary-tc]

hi @rashampreet-singh how can I help you get unblocked

[tushar-choudhary-tc]

#1388

Here is the PR that where I had contributed this function.

[Fokko]

What kind of use-case would this serve? If the count is less than the number of rows in the table, then it always will be count == limit?

[kaustuvnandy]

Hi Fokko, tried to raise a small PR to solve for this.
Should I add another test case to re-create this? Can you please help me here!

[jayceslesar]

@Fokko I my use case is a short circuit to test for data existence

[jayceslesar]

although I suppose I can achieve this by simply performing a normal table scan with a limit...

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.