Skip to content

feat: enrich IndexContent with describe_indices metadata #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
wjones127 merged 3 commits into from
Jun 9, 2026
Merged

feat: enrich IndexContent with describe_indices metadata #349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
13f71b9
feat: enrich IndexContent with describe_indices metadata
wjones127 Jun 4, 2026
26dac2f
docs: clarify num_indexed_rows and created_at descriptions
wjones127 Jun 4, 2026
c53bb5f
add num_unindexed_rows
wjones127 Jun 9, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/src/namespace/operations/models/AddVirtualColumnEntry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
| Name | Type | Description | Notes |
|------------ | ------------- | ------------- | -------------|
|**inputColumns** | **List<String>** | List of input column names for the virtual column | |
|**dataType** | **Object** | Data type of the virtual column using JSON representation | |
|**outputs** | [**List<AddVirtualColumnOutputEntry>**](AddVirtualColumnOutputEntry.md) | Output columns produced by the virtual column UDF | |
|**image** | **String** | Docker image to use for the UDF | |
|**udf** | **String** | Base64 encoded pickled UDF | |
|**udfName** | **String** | Name of the UDF | |
Expand Down
17 changes: 17 additions & 0 deletions docs/src/namespace/operations/models/AddVirtualColumnOutputEntry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@


# AddVirtualColumnOutputEntry


## Properties

| Name | Type | Description | Notes |
|------------ | ------------- | ------------- | -------------|
|**column** | **String** | Physical output column name | |
|**structField** | **String** | Field name in the UDF output struct | |
|**dataType** | **Object** | Data type of the output column using JSON representation | |
|**nullable** | **Boolean** | Whether the output column is nullable | |
|**metadata** | **Map<String, String>** | User-supplied output field metadata (string key-value pairs) | [optional] |



9 changes: 9 additions & 0 deletions docs/src/namespace/operations/models/IndexContent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,15 @@
|**indexUuid** | **String** | Unique identifier for the index | |
|**columns** | **List<String>** | Columns covered by this index | |
|**status** | **String** | Current status of the index | |
|**indexType** | **String** | Friendly index type, e.g. IVF_PQ, BTREE. Unknown if no plugin recognizes the index. | [optional] |
|**typeUrl** | **String** | Protobuf type URL, a precise type identifier for the index. | [optional] |
|**numIndexedRows** | **Long** | Number of live rows covered by the index. This does not count rows that are in the index but have since been deleted. | [optional] |
|**numUnindexedRows** | **Long** | Number of rows that are not indexed. | [optional] |
|**sizeBytes** | **Long** | Total index size in bytes across all segments. Null for indices predating file-size tracking. | [optional] |
|**numSegments** | **Integer** | Number of index deltas/segments. | [optional] |
|**createdAt** | **OffsetDateTime** | Creation time for indexes. Null for legacy indices. | [optional] |
|**indexVersion** | **Integer** | On-disk index format version. | [optional] |
|**indexDetails** | **String** | Opaque, type-specific JSON with additional index details. For vector indices this carries metric/distance type, partitioning, and HNSW/PQ/SQ/RQ parameters. | [optional] |



52 changes: 52 additions & 0 deletions docs/src/spec.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3219,6 +3219,58 @@ components:
status:
type: string
description: Current status of the index
index_type:
type: string
description: >-
Friendly index type, e.g. IVF_PQ, BTREE. Unknown if no plugin
recognizes the index.
type_url:
type: string
description: Protobuf type URL, a precise type identifier for the index.
num_indexed_rows:
type: integer
format: int64
minimum: 0
description: >-
Number of live rows covered by the index. This does not count
rows that are in the index but have since been deleted.
num_unindexed_rows:
type: integer
format: int64
minimum: 0
description: >-
Number of rows that are not indexed.
size_bytes:
type: integer
format: int64
minimum: 0
nullable: true
description: >-
Total index size in bytes across all segments. Null for indices
predating file-size tracking.
num_segments:
type: integer
format: int32
minimum: 0
description: Number of index deltas/segments.
Comment on lines +3251 to +3255

@wjones127 wjones127 Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: I'm undecided whether we should replace this with a list of segments. Open to thoughts on that.

@wjones127 wjones127 Jun 4, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thought is that we can always add segment-level data layer. Not much harm in having the count and the list in here, even if it would be somewhat redundant.

created_at:
type: string
format: date-time
nullable: true
description: >-
Creation time for indexes. Null for legacy indices.
index_version:
type: integer
format: int32
minimum: 0
description: On-disk index format version.
index_details:
type: string
nullable: true
description: >-
Opaque, type-specific JSON with additional index details. For vector
indices this carries metric/distance type, partitioning, and
HNSW/PQ/SQ/RQ parameters.

DescribeTableIndexStatsRequest:
type: object
Expand Down
73 changes: 73 additions & 0 deletions java/lance-namespace-apache-client/api/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4469,17 +4469,35 @@ components:
example:
indexes:
- index_uuid: index_uuid
size_bytes: 0
columns:
- columns
- columns
created_at: 2000-01-23T04:56:07.000+00:00
index_details: index_details
num_indexed_rows: 0
num_segments: 0
index_type: index_type
index_version: 0
num_unindexed_rows: 0
index_name: index_name
status: status
type_url: type_url
- index_uuid: index_uuid
size_bytes: 0
columns:
- columns
- columns
created_at: 2000-01-23T04:56:07.000+00:00
index_details: index_details
num_indexed_rows: 0
num_segments: 0
index_type: index_type
index_version: 0
num_unindexed_rows: 0
index_name: index_name
status: status
type_url: type_url
page_token: page_token
properties:
indexes:
Expand Down Expand Up @@ -4511,11 +4529,20 @@ components:
IndexContent:
example:
index_uuid: index_uuid
size_bytes: 0
columns:
- columns
- columns
created_at: 2000-01-23T04:56:07.000+00:00
index_details: index_details
num_indexed_rows: 0
num_segments: 0
index_type: index_type
index_version: 0
num_unindexed_rows: 0
index_name: index_name
status: status
type_url: type_url
properties:
index_name:
description: Name of the index
Expand All @@ -4531,6 +4558,52 @@ components:
status:
description: Current status of the index
type: string
index_type:
description: "Friendly index type, e.g. IVF_PQ, BTREE. Unknown if no plugin\
\ recognizes the index."
type: string
type_url:
description: "Protobuf type URL, a precise type identifier for the index."
type: string
num_indexed_rows:
description: Number of live rows covered by the index. This does not count
rows that are in the index but have since been deleted.
format: int64
minimum: 0
type: integer
num_unindexed_rows:
description: Number of rows that are not indexed.
format: int64
minimum: 0
type: integer
size_bytes:
description: Total index size in bytes across all segments. Null for indices
predating file-size tracking.
format: int64
minimum: 0
type: integer
nullable: true
num_segments:
description: Number of index deltas/segments.
format: int32
minimum: 0
type: integer
created_at:
description: Creation time for indexes. Null for legacy indices.
format: date-time
type: string
nullable: true
index_version:
description: On-disk index format version.
format: int32
minimum: 0
type: integer
index_details:
description: "Opaque, type-specific JSON with additional index details.\
\ For vector indices this carries metric/distance type, partitioning,\
\ and HNSW/PQ/SQ/RQ parameters."
type: string
nullable: true
required:
- columns
- index_name
Expand Down
9 changes: 9 additions & 0 deletions java/lance-namespace-apache-client/docs/IndexContent.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,15 @@
|**indexUuid** | **String** | Unique identifier for the index | |
|**columns** | **List<String>** | Columns covered by this index | |
|**status** | **String** | Current status of the index | |
|**indexType** | **String** | Friendly index type, e.g. IVF_PQ, BTREE. Unknown if no plugin recognizes the index. | [optional] |
|**typeUrl** | **String** | Protobuf type URL, a precise type identifier for the index. | [optional] |
|**numIndexedRows** | **Long** | Number of live rows covered by the index. This does not count rows that are in the index but have since been deleted. | [optional] |
|**numUnindexedRows** | **Long** | Number of rows that are not indexed. | [optional] |
|**sizeBytes** | **Long** | Total index size in bytes across all segments. Null for indices predating file-size tracking. | [optional] |
|**numSegments** | **Integer** | Number of index deltas/segments. | [optional] |
|**createdAt** | **OffsetDateTime** | Creation time for indexes. Null for legacy indices. | [optional] |
|**indexVersion** | **Integer** | On-disk index format version. | [optional] |
|**indexDetails** | **String** | Opaque, type-specific JSON with additional index details. For vector indices this carries metric/distance type, partitioning, and HNSW/PQ/SQ/RQ parameters. | [optional] |



Loading
Loading