[Python] Support tree-model TsFiles in TsFileDataFrame

[Young-Leo]

Summary

This PR teaches the Python TsFileDataFrame to read tree-model TsFiles
(in addition to the existing table-model support) and, in the same series of
commits, slims the underlying dataset index so it scales to wide / sparse
schemas without paying for phantom (device, field) cells.

The user-facing dataset surface (__len__, list_timeseries, __getitem__,
.loc, aligned reads) is unchanged for both models — tree-model files
just become loadable through the same API.

What changed

Tree-model support

• Detect the model kind at reader open. An empty table-schema map ⇒ tree
  model; otherwise table model.
• For tree files, synthesize one virtual TableEntry:
  • table name = the shared root segment of every device path
  • tag columns = _col_1 .. _col_N (one positional column per remaining
    path segment, padded with None for shorter devices)
  • fields = union of measurements across all devices
• Per-device measurement ownership is preserved by registering only the
  (device_id, field_idx) pairs that are actually written on disk in
  series_stats_by_ref, so the dataset never advertises phantom series.
• Tree-model reads route through query_table_on_tree with client-side
  device filtering. This works around two cwrapper limitations on the
  current native build (query_tree_by_row rejects multi-segment device
  paths, and successive query_table_on_tree calls leak duplicate col_*
  columns); both are documented inline at the cwrapper boundary.
• Tree-mode rendering: drop the leading table column, use _col_i
  headers, print None tag cells as "None", and surface the model
  marker in the repr header.
• Mixing table-model and tree-model files in one load set is rejected
  with a clear error.

Dataset index slim-down

• SeriesStats becomes a NamedTuple (~120 B vs. the previous ~360 B
  per-series dict).
• _DerivedCache removed; lookups computed lazily on top of existing
  state.
• Per-reader device_refs: List[List[DeviceRef]] collapsed into a
  pre-aggregated device_time_bounds: List[Tuple[Optional[int], Optional[int]]], so _query_aligned reads bounds in O(1).
• Drop the redundant series_ref_set (use series_ref_map keys).
• Phase 6: unify table/tree semantics so the table-model branch no
  longer pads series_stats_by_ref with empty placeholders for
  schema-declared but never-written cells. The dataset view is now
  strictly “real devices × real fields” in both models.
• Cleanup: rename _LogicalIndex → _DataFrameCatalog, shorten the five
  internal field names (devices / device_index / device_time_bounds
  / series / series_shards), inline the now-trivial
  iter_owned_series_refs wrapper.

New public API

• TsFileDataFrame.model — read-only model marker ("table" or "tree").
• TsFileDataFrame.list_timeseries_metadata() — per-series metadata as a
  flat tabular view (works identically for both models).

Compatibility

• No changes to the existing dataset surface. Existing user code that
  loads table-model TsFiles continues to work without modification.
• No changes to the on-disk format, the cwrapper, or the C++/Java sides.
• SeriesStats integer fields tighten from Optional[int] to int. The
  surrounding get_series_info_by_ref still exposes them as the existing
  dict shape, so callers do not see an API change.

Memory impact

Two benches were used because the wins land in different shapes of
schema.

Bench A — 30 k devices × 1 field, full density

Step	Tracked size	Δ
baseline	81.20 MB	—
SeriesStats NamedTuple	70.67 MB	−10.53 MB
_DerivedCache removal	59.82 MB	−10.85 MB
device_time_bounds aggregation	56.40 MB	−3.42 MB
series_ref_set removal	54.40 MB	−2.00 MB
drop phantom (device, field) cells	54.40 MB	0

End-to-end: 81.20 → 54.40 MB (−33 %). Dropping phantom cells brings
nothing here because every device writes the single declared field;
there are no skipped cells to prune.

Bench B — 5 k devices × 5 fields × 60 % density (sparse / wide)

Component	Before	After
series_ref_map	15.93 MB	9.55 MB
series_stats_by_ref	7.53 MB	4.51 MB
tracked total	26.50 MB	16.30 MB

Dropping phantom cells alone brings −38 % on this fixture; the
sparser and wider the schema, the larger the win.

Testing

• python -m pytest python/tests/test_tsfile_dataset.py → 41 / 41 pass.
• Four new tree-model tests cover: metadata + repr layout, single-series
  read, multi-field aligned read, list_timeseries_metadata column
  shape, and the mixed-model load rejection.
• One new sparse-schema test (test_dataset_omits_table_model_phantom_series_for_skipped_cells)
  proves Tablet-skipped cells stay out of list_timeseries, __len__,
  series_ref_map, and that tsdf[skipped_path] raises KeyError.

[JackieTien97]

Code Review: [Python] Support tree-model TsFiles in TsFileDataFrame

Overall this is a well-structured PR with solid test coverage. The tree-model integration, phantom-series elimination, and index slim-down are all clean and well-motivated. I have a few concerns — one potential correctness issue, one performance concern, and several nits. See inline comments.

[JackieTien97]

MODEL_TABLE and MODEL_TREE are defined identically in both reader.py:37-38 and dataframe.py:54-55. The dataframe module already imports from reader (from .reader import ...). Consider defining these constants once in reader.py (or metadata.py) and importing them in dataframe.py to avoid drift.

The test file imports MODEL_TABLE from reader, so both definitions are actively used — easy to get them out of sync in the future.

[JackieTien97]

The filter condition changed from checking timeline_statistic.has_statistic to checking statistic.has_statistic. This is a semantic shift: the old code filtered based on the timeline (timestamp axis), while the new code filters on the value statistic.

Are there cases where statistic.has_statistic is False but timeline_statistic.has_statistic is True, or vice versa? If the two can diverge, this changes which series are visible. The docstring says "iff its native statistic block is populated" but the old behavior was gating on timeline_statistic.

Also, line 388 now reads timeline_statistic unconditionally after the statistic check passes. If a measurement has statistic.has_statistic=True but timeline_statistic.has_statistic=False (or timeline_statistic is None), lines 393-395 would raise. Worth a defensive check or a note explaining why this can't happen.

[JackieTien97]

Both _read_series_by_row_tree and _read_arrow_tree do a full query_table_on_tree scan over the entire file and then client-side filter by target_path_segments. For a file with many devices, this means reading every row to extract data for a single device.

This is acknowledged in the PR description as a cwrapper limitation workaround, which is fine for now — but worth flagging for future readers: this is O(total_rows) per device, so an aligned read across N devices in the same file becomes O(N × total_rows). The comments at the call-site document the "why" well; a brief # O(total_rows) — see PR #816 for cwrapper limitations here would also help future profilers find the hot path.

[JackieTien97]

MetadataCatalog.series_count (metadata.py:148) still computes the cross-product of all (device, field) combos, but the reader's series_count property (line 111) now uses len(series_stats_by_ref) which counts only physically-present series.

For sparse schemas these two diverge. Tests at test_tsfile_dataset.py:688 and :789 still call reader.catalog.series_count and happen to pass because those fixtures are fully dense, but this is a latent inconsistency.

Consider either: (a) updating MetadataCatalog.series_count to also return len(self.series_stats_by_ref), or (b) deprecating it in favor of the reader property.

[JackieTien97]

_from_subset creates a new _DataFrameCatalog but does not propagate tables_with_sparse_tag_values or sparse_device_indices_by_compressed_path. If a view (created by slicing or boolean indexing) later calls _resolve_series_name on a series whose device has sparse tags, the lookup will fail because the compressed-path index is empty.

This was also missing in the old _LogicalIndex code, so it's a pre-existing gap — but since this PR refactors _from_subset, it's worth noting. A one-line fix:

tables_with_sparse_tag_values=parent._index.tables_with_sparse_tag_values,
sparse_device_indices_by_compressed_path=parent._index.sparse_device_indices_by_compressed_path,

[JackieTien97]

_build_field_stats is now called lazily on every _build_series_info invocation instead of being precomputed once in _DerivedCache. This means:

• list_timeseries_metadata() calls _build_series_info for every series, each of which calls _build_field_stats. For N series across K shards, this is O(N×K) reader calls.
• __repr__ and __getitem__(column_name) also go through _build_series_info.

The old approach precomputed these stats once at load time. The PR description frames this as removing _DerivedCache, but the trade-off is that repeated access patterns (e.g., interactive exploration calling repr() then list_timeseries_metadata() then column access) now recompute stats from scratch each time.

For typical usage with moderate series counts this is fine. For wide schemas (the PR's own benchmark cites 5k devices × 5 fields), it could add up. Worth noting in a comment or considering a @functools.lru_cache on the hot path.

[JackieTien97]

The "stale col_ columns from prior queries" workaround (all_col_indices[-expected_path_len:]) appears in both _read_series_by_row_tree (line 556) and _read_arrow_tree (line 747). This is a fragile assumption about cwrapper internal state.

If the cwrapper behavior changes (e.g., fixed to not leak duplicate columns, or leaks in a different pattern), this slicing logic will silently break. Consider at minimum an assertion:

assert len(col_indices) == expected_path_len, (
    f"Expected {expected_path_len} col_ columns, got {len(col_indices)} "
    f"(total {len(all_col_indices)})"
)

This would fail fast if the assumption is violated rather than silently returning wrong data.

[JackieTien97]

Nit: PEP 8 expects two blank lines between top-level methods in a class. _repr_header ends at line 1059 and __repr__ starts at line 1060 — missing a blank line separator.

[JackieTien97]

The test suite covers single tree-model files, but there's no test for loading multiple tree-model files into one TsFileDataFrame. The multi-file merge path (via _register_reader + device_time_bounds aggregation) has tricky edge cases for tree-model:

• Two files with the same device but different measurement subsets — do we get the union of fields? Do time bounds merge correctly?
• Two files with different max_depth — does the tag padding still work correctly?

These are the same merge paths that table-model tests cover, but tree-model's synthetic table construction introduces new failure modes around the union-fields and padding logic.

[JackieTien97]

This validation rejects tree files with multiple root segments (root_name != current_root). In practice, IoTDB tree-model files virtually always use root as the single root. However, the synthetic table uses the root segment as the table_name, so if a file ever had devices under multiple roots, the virtual table would be ambiguous.

The error message is clear and this is the right behavior — just noting that I'd expect this to never actually fire in practice. If it does, it signals a seriously malformed file rather than a normal use case.