Split client.py by endpoint family — 3,842 LOC vs 2,000 hard ceiling (PR #516 NB-2)

[padak]

Summary

src/keboola_agent_cli/client.py is 3,842 LOC on main (3,962 on the feat/512-table-snapshots branch, which adds the snapshot methods) against CONTRIBUTING.md's 2,000-line hard ceiling for client.py / manage_client.py. The guide is explicit: at the hard ceiling "splitting is required before merging more functionality into it." We are ~1.9× over and the file grows with every new endpoint family.

CONTRIBUTING.md already prescribes the remedy verbatim:

client.py mixing multiple Keboola subsystems (Storage, Queue, Sandboxes, Manage proxy, AI, encryption, …) → split by endpoint family, e.g. client/storage.py, client/queue.py, client/sandboxes.py. Keep BaseHttpClient shared.

This issue documents the current shape and proposes a concrete, incremental, behavior-preserving split. Flagged as NB-2 in the PR #516 review (non-blocking there — pre-existing systemic debt, not introduced by that PR — deferred to this follow-up).

Current shape

• client.py: 3,842 LOC (main). One class, KeboolaClient(BaseHttpClient), spans lines 154–3437 (~3,280 LOC, 134 methods), followed by ~525 LOC of module-level cloud-transfer / query-error helper functions and small helper classes.
• BaseHttpClient already lives separately in http_base.py (315 LOC) — the shared retry/backoff base is already factored out, so this split only concerns KeboolaClient and its module-level helpers.

Endpoint-family inventory

Measured by summing each method's span (next-def − this-def) and grouping by subsystem (change_sharing_type folded into storage-sharing; _prepare_sliced_download into storage-files):

Endpoint family	approx LOC	#methods	Proposed home
storage — buckets / tables / snapshots / sharing	~1,168	36	client/storage_tables.py
components / configs (config + row CRUD, metadata)	~545	19	client/configs.py
module-level helpers (cloud up/download, _CloudDownloader, query-error parsing, URL safety)	~525	18	client/_transfer.py
storage files (upload/download/tag, sliced download)	~310	10	client/storage_files.py
jobs (Queue API)	~302	9	client/queue.py
tokens / auth / project / features	~196	8	client/tokens.py
infra / plumbing (__init__, _request, sub-client mgmt, base-URL props, _wait_for_storage_job, context mgr)	~167	15	client/_core.py (stays)
branches (dev-branch + branch metadata)	~135	7	client/branches.py
stream (delegates to StreamClient)	~123	7	client/stream.py
query service (workspace SQL)	~96	4	client/query.py
workspaces / sandboxes	~67	5	client/workspaces.py
search (global_search)	~58	1	client/misc.py
sync-actions (run_sync_action)	~43	1	client/misc.py
oauth (get_oauth_url)	~37	1	client/misc.py
encrypt (encrypt_values)	~27	1	client/misc.py

The storage family alone (~1.5k LOC across tables + files) is bigger than the entire hard ceiling for a client file, so it should be sub-split (tables vs files).

Hard constraint: the public surface must not move

Two contracts pin the shape of any split:

1. from ...client import KeboolaClient must keep working. 20 modules import from client. The full name surface imported across src/ + tests/ (with counts) is:

   name	uses	kind
   KeboolaClient	38	class
   _extract_cloud_error_code	4	helper (tests)
   _collect_inline_results	3	helper (lib.py + tests)
   _assert_safe_download_url	3	helper (security tests)
   _iter_poll_intervals	2	helper (tests)
   _unwrap_bigquery_error	1	helper
   _extract_query_job_error	1	helper
   _build_abs_upload_url	1	helper
   InlineQueryResult	1	dataclass

2. The SDK .raw contract. keboola_agent_cli.Client.raw (in lib.py) returns the underlying KeboolaClient and callers invoke Storage/Queue methods on it directly (client.raw.list_tables(...), .upload_file(...), …). KeboolaClient must therefore remain one class exposing every method with identical signatures — a delegating sub-client design (client.storage.list_tables()) would break .raw and is out of scope.

⟹ Mixin composition, not sub-client delegation. Keep KeboolaClient a single class whose methods are physically split across mixin modules and recombined by inheritance. Method resolution, signatures, .raw, and every call site stay byte-identical; only the file layout changes.

Proposed layout

client/
  __init__.py        # re-exports the 9 names above → `from .client import X` unchanged
  _core.py           # class _CoreClient(BaseHttpClient): __init__, _request/_queue_request/
                     #   _query_request/_encrypt_request/_sync_actions_request, base-URL props,
                     #   _wait_for_storage_job, close, __enter__/__exit__  (the shared plumbing)
  _transfer.py       # module-level cloud up/download + _CloudDownloader/_IterBytesReader +
                     #   query-error parsers + URL-safety helpers (pure funcs/classes, no self)
  storage_tables.py  # class _StorageTablesMixin(_CoreClient)
  storage_files.py   # class _StorageFilesMixin(_CoreClient)
  configs.py         # class _ConfigsMixin(_CoreClient)
  queue.py           # class _QueueMixin(_CoreClient)
  branches.py        # class _BranchesMixin(_CoreClient)
  tokens.py          # class _TokensMixin(_CoreClient)
  workspaces.py      # class _WorkspacesMixin(_CoreClient)
  query.py           # class _QueryMixin(_CoreClient)
  stream.py          # class _StreamMixin(_CoreClient)
  misc.py            # class _MiscMixin(_CoreClient): search / oauth / encrypt / sync-actions
  _client.py         # class KeboolaClient(_StorageTablesMixin, _StorageFilesMixin, _ConfigsMixin,
                     #   _QueueMixin, _BranchesMixin, _TokensMixin, _WorkspacesMixin, _QueryMixin,
                     #   _StreamMixin, _MiscMixin): pass

• Each mixin inherits the shared _CoreClient base so the plumbing it calls (self._request, self._queue_request, self._wait_for_storage_job, self.stack_url, …) is statically visible to ty — no # type: ignore. The common base is a cooperative diamond; _CoreClient.__init__ runs once. (This is the key detail that keeps make typecheck clean — a bare mixin that references self._request without a typed base would fail ty.)
• __init__.py re-exports KeboolaClient + the 8 helper names, so no importer changes.

Incremental plan (each step independently mergeable, make check green)

The point of splitting by family is that each move is mechanical and isolated. Suggested order, easiest/highest-LOC-relief first:

1. Convert client.py → client/ package; move the ~525 LOC of module-level helpers to _transfer.py. Pure functions/classes with no self — the safest, biggest single reduction. __init__.py re-exports everything. Zero behavior change. (Drops the file well below where the next few extractions matter.)
2. Extract _core.py (plumbing) and stand up the empty KeboolaClient(_CoreClient) composition — still one class, all methods temporarily still on _CoreClient; then peel families out one PR at a time.
3. Small families first: stream, branches, tokens, query, workspaces, misc (each 40–200 LOC, trivially isolated).
4. Mid families: configs (~545), queue (~302).
5. Storage last, sub-split: storage_tables.py (~1,168) + storage_files.py (~310).

After each PR: KeboolaClient composes the same set of methods, .raw is untouched, and make check (incl. ty and the full suite) stays green. No new tests are required for a pure move, but existing test_* files that import helpers by name (_assert_safe_download_url, _iter_poll_intervals, …) validate the re-export surface for free.

Acceptance

• client.py → client/ package; every module ≤ the 2,000-LOC hard ceiling (target: each family module ≤ ~800 soft ceiling where practical).
• from keboola_agent_cli.client import KeboolaClient (+ the 8 helpers) unchanged; Client.raw unchanged.
• KeboolaClient remains a single class; no method signature changes.
• make check green at every intermediate PR; ty clean with no new ignores.

---

Filed as the NB-2 follow-up from PR #516 review. Inventory numbers are grep-measured against main (2026-07-22).

[padak]

Implemented in #524 (branch refactor/520-client-split, base main).

The split follows the issue's 5-step plan exactly, one clean commit per step, each leaving make check green (4,648 passed):

1. client.py → client/ package; module-level helpers → _transfer.py
2. extract _CoreClient plumbing → _core.py
3. peel small families (stream, branches, tokens, query, workspaces, misc)
4. peel configs + queue
5. peel storage_tables + storage_files; finalize the KeboolaClient(<10 mixins>, _CoreClient) composition; monolith gone

Every module is now under the 2,000-LOC hard ceiling (largest: storage_tables.py at 1,164). KeboolaClient stays one class (mixin composition), the 9-name import surface + .raw are unchanged, and it's a provable pure move — 124/124 methods present, 121 byte-identical, 3 with only a documented ty-required annotation change (__enter__ → Self; two self: "KeboolaClient" typed-mixin annotations for the only cross-family calls). No new ty diagnostics, no # type: ignore.