[FSTORE-1972] Defining byok connector should create onlinefs instance

docs/setup_installation/admin/onlinefs.md

@@ -0,0 +1,155 @@
+---
+description: Administrator guide for configuring the Hopsworks-managed OnlineFS service that ingests data into the online feature store.
+---
+
+# OnlineFS Service
+
+The Online Feature Store service (OnlineFS in short) consumes feature data from Kafka and writes it to the online feature store (RonDB).
+This guide covers the cluster-level OnlineFS instance, which consumes from the Kafka cluster embedded in Hopsworks.
+For the per-project OnlineFS instances used with an external Kafka cluster, see the [external Kafka cluster guide][online-feature-store-service-configuration].
+
+## Helm-managed vs. Hopsworks-managed
+
+The cluster-level OnlineFS instance can be deployed in two ways, and only one is active at a time:
+
+- **Deployed by the `onlinefs` Helm chart.**
+  This is the traditional deployment method and continues to be fully supported.
+  If the `onlinefs` chart is installed, Hopsworks detects the resulting deployment and leaves it alone entirely: Hopsworks never creates, modifies or removes it, and the configuration described below is unavailable — configure the instance through the Helm chart's own values instead.
+- **Deployed and managed by Hopsworks itself.**
+  If no `onlinefs` Helm release is present, Hopsworks deploys and reconciles the cluster-level instance automatically, using the configuration described below.
+
+Because of this precedence, installing the `onlinefs` Helm chart on a cluster where Hopsworks was previously managing the instance (or vice versa) is safe: whichever one is Helm-managed always wins, so the two never run at the same time.
+
+Cluster Settings → `OnlineFS Service` (and the `/admin/onlinefs/status` REST endpoint) shows which mode is currently active.
+
+## The Hopsworks-managed deployment
+
+When there is no Helm-managed instance, Hopsworks deploys the cluster-level OnlineFS instance itself, on by default.
+It can be paused with the `Enabled` toggle in its configuration (see below).
+
+Hopsworks creates and reconciles all resources the service needs in the Hopsworks namespace: the deployment, its configuration, the metrics service scraped by Prometheus, the TLS material used to connect to the internal Kafka cluster, and the API key the service uses to call the Hopsworks REST API.
+Configuration changes roll the OnlineFS pods automatically.
+A periodic reconciler additionally converges the deployment against the stored configuration, so manual changes to the Kubernetes resources are reverted; the same reconciler also re-checks whether a Helm-managed instance has appeared or disappeared, and defers to it or takes over accordingly.
+
+## Configuration {#onlinefs-configuration}
+
+The cluster-level OnlineFS instance is configured from the admin UI under `Cluster Settings` → `OnlineFS Service`, or through the `/admin/onlinefs/config` REST endpoint.
+This is only available while Hopsworks is managing the cluster-level instance itself (see above); while a Helm-managed instance is active, these settings have no effect.
+Empty fields use the platform defaults.
+
+<p align="center">
+  <figure>
+    <img src="../../../assets/images/guides/onlinefs/cluster_onlinefs.png" alt="Cluster-level OnlineFS Service configuration in Cluster Settings">
+    <figcaption>Cluster-level OnlineFS Service configuration in Cluster Settings</figcaption>
+  </figure>
+</p>
+
+The same set of settings is available for the per-project OnlineFS instances deployed when using an external Kafka cluster, under `Project Settings` → `OnlineFS Service`.
+Unlike the cluster-level instance, which is enabled by default, per-project instances are opt-in: `Enabled` defaults to off and a Data Owner must turn it on before the instance is deployed.
+See the [external Kafka cluster guide][online-feature-store-service-configuration] for details.
+
+<p align="center">
+  <figure>
+    <img src="../../../assets/images/guides/onlinefs/project_onlinefs.png" alt="Per-project OnlineFS Service configuration in Project Settings">
+    <figcaption>Per-project OnlineFS Service configuration in Project Settings</figcaption>
+  </figure>
+</p>
+
+| Field | Description |
+| --- | --- |
+| Enabled | Turning this off removes the OnlineFS deployment and pauses all ingestion; the configuration and consumer offsets are kept. Defaults to on for the cluster-level instance, off for per-project instances. |
+| VectorDB ingestion | Also ingest embeddings into the vector database (OpenSearch). |
+| Replicas | Number of OnlineFS pods (0-10). Instances coordinate through a shared Kafka consumer group. |
+| CPU/Memory request and limit | Kubernetes resource quantities for the OnlineFS container, e.g. `500m` or `2Gi`. |
+| Service settings | Free-form overrides for `onlinefs-site.xml` as `section.key` entries, e.g. `service.threadNumber`, `rondb.batchSize` or `kafkaConsumer.topicPattern`. See the [reference table](#service-settings-reference) below for every available key. |
+| Kafka consumer properties | Free-form Kafka client properties appended to the consumer configuration, e.g. `max.poll.records`. |
+| Kafka producer properties | Free-form Kafka client properties appended to the notification producer configuration, e.g. `batch.size`. |
+| Kafka vectorDB consumer properties | Free-form Kafka client properties for the vectorDB consumer; only used when vectorDB ingestion is enabled. |
+
+The override entries take precedence over the values generated by Hopsworks, so every OnlineFS setting is configurable.
+
+### Service settings reference
+
+`Service settings` entries are `section.key` pairs written into `onlinefs-site.xml`. Below is every key OnlineFS reads, grouped by section, with its default value.
+
+**`service`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `service.threadNumber` | `10` | Number of threads used for reporting metadata/status back to Hopsworks and for sending CDC notifications. |
+| `service.ronDbThreadNumber` | `10` | Number of Kafka consumer threads ingesting into RonDB. |
+| `service.vectorDbThreadNumber` | `5` | Number of Kafka consumer threads ingesting into the vector database (OpenSearch); only relevant when VectorDB ingestion is enabled. |
+| `service.reportingQueueSizePerThread` | `100` | Queue size per reporting/notification thread. |
+| `service.getSessionRetrySleepMs` | `100` | Milliseconds to sleep between retries when acquiring a RonDB (ClusterJ) session. |
+| `service.maxBlacklistSize` | `100` | Max size of the blacklist cache used by the Hopsworks helper. |
+| `service.maxFeatureGroupCacheSize` | `1000` | Max entries kept in the feature group metadata cache. |
+| `service.maxFeatureStoreCacheSize` | `1000` | Max entries kept in the feature store metadata cache. |
+| `service.maxFeatureViewCacheSize` | `1000` | Max entries kept in the feature view metadata cache. |
+| `service.featureGroupCacheExpire` | `30` | Feature group cache entry expiry, in minutes. |
+| `service.featureStoreCacheExpire` | `30` | Feature store cache entry expiry, in minutes. |
+| `service.featureViewCacheExpire` | `10` | Feature view cache entry expiry, in minutes. |
+| `service.region` | _(none)_ | Cloud region reported in CDC notification payloads, identifying where this OnlineFS instance runs. |
+| `service.pauseRetrySleepMs` | `5000` | Milliseconds to sleep between attempts while ingestion is paused waiting for a transient failure (e.g. a RonDB or Hopsworks upgrade) to resolve. |
+
+**`rondb`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `rondb.connectionString` | `127.0.0.1:1186` | RonDB (NDB) cluster connect string. |
+| `rondb.batchSize` | `1000` | Max batch size when committing rows to RonDB. |
+| `rondb.maxRetries` | `3` | Max retries for a RonDB commit operation. |
+| `rondb.poolSize` | `1` | ClusterJ connection pool size. |
+| `rondb.maxCachedSessions` | `20` | Max ClusterJ sessions cached per connection. |
+| `rondb.reconnectTimeout` | `5` | ClusterJ connection reconnect timeout, in seconds. |
+| `rondb.maxTransactions` | `1024` | Max concurrent ClusterJ transactions. |
+| `rondb.maxCachedInstances` | `1024` | Max ClusterJ cached instances. |
+| `rondb.useDynamicObjectCache` | `true` | Whether ClusterJ uses a dynamic object cache. |
+| `rondb.useSessionCache` | `true` | Whether ClusterJ uses a session cache. |
+
+**`kafka`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `kafka.propertiesFile` | `onlinefs-kafka.properties` | Kafka client properties file used for the RonDB ingestion consumer. |
+| `kafka.propertiesFileVectorDb` | `onlinefs-kafka-vector-db.properties` | Kafka client properties file used for the VectorDB ingestion consumer. |
+| `kafka.propertiesFileNotification` | `producer.properties` | Kafka client properties file used for the CDC notification producer. |
+
+**`kafkaConsumer`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `kafkaConsumer.topicPattern` | `.*_onlinefs` | Regex pattern used to subscribe to Kafka topics. |
+| `kafkaConsumer.topicList` | _(empty)_ | Comma-separated list of explicit topics to consume, in addition to the topic pattern. |
+| `kafkaConsumer.pollTimeoutMs` | `1000` | Kafka consumer poll timeout, in milliseconds. |
+
+**`hopsworks`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `hopsworks.url` | `https://hopsworks.glassfish.service.consul:8182` | Base URL of the Hopsworks REST API. |
+| `hopsworks.trustStoreLocation` | `trustStore.jks` | Path to the trust store used for TLS to Hopsworks (and to OpenSearch). |
+| `hopsworks.tokenLocation` | `token` | Path to the file containing the Hopsworks API key used by this instance. |
+
+**`opensearch`** (VectorDB ingestion)
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `opensearch.host` | `https://elastic.service.consul:9200` | OpenSearch endpoint URL. |
+| `opensearch.userName` | `onlinefs` | OpenSearch auth username. |
+| `opensearch.password` | `onlinefs` | OpenSearch auth password. |
+| `opensearch.batchSize` | `1000` | Max batch size when committing embeddings to OpenSearch. |
+| `opensearch.maxRetries` | `10` | Max retries for an OpenSearch commit operation. |
+
+**`metrics`**
+
+| Key | Default | Description |
+| --- | --- | --- |
+| `metrics.port` | `12800` | Port for the Prometheus metrics HTTP server. |
+
+The Hopsworks-managed deployment provisions `hopsworks.*` and `opensearch.*` automatically (URL, trust store, token, OpenSearch endpoint and credentials); overriding them is rarely needed outside of troubleshooting.
+
+## VectorDB ingestion
+
+When VectorDB ingestion is enabled, OnlineFS runs an additional set of consumers that write embedding features to the vector database (OpenSearch).
+The OpenSearch endpoint and credentials are provisioned automatically from the cluster.
+The VectorDB consumers use their own Kafka consumer group and track their own offsets, so enabling VectorDB ingestion on an existing deployment starts consuming from the earliest available offset.

docs/setup_installation/on_prem/external_kafka_cluster.md

@@ -21,33 +21,22 @@ hopsworks:
 
 ### Online Feature Store service configuration
 
-In addition to the configuration changes above, you should also configure the Online Feature Store service (OnlineFS in short) to connect to the external Kafka cluster.
-This can be achieved by provisioning the necessary credentials for OnlineFS to subscribe and consume messages from Kafka topics used by the Hopsworks feature store.
+With the external Kafka cluster integration enabled and a project's `kafka_connector` data source configured (see below), Hopsworks can deploy a dedicated Online Feature Store service (OnlineFS in short) instance in that project's namespace.
+Unlike the cluster-level instance, which is deployed by default, a per-project instance is opt-in: a Data Owner must explicitly enable it under `Project Settings` → `OnlineFS Service` (`Enabled` toggle) before it is deployed and starts ingesting.
+Once enabled, the instance consumes the project's feature store topics from the external Kafka cluster using the connection settings, credentials, and options of the data source, and it is updated automatically whenever the data source changes.
+Deleting the data source removes the instance again.
 
-OnlineFs can be configured to use these credentials by adding the following configurations to the cluster definition used to deploy Hopsworks:
+Each project instance only subscribes to its own project's topics, so multiple projects can safely share the same external Kafka cluster.
+Additional Kafka client settings, such as SASL authentication options, can be provided as additional options on the data source and are applied to both the producers and the OnlineFS consumers.
 
-```yaml
-  onlinefs:
-    config_dir: "/home/ubuntu/cluster-definitions/byok"
-    kafka_consumers:
-      topic_list: "comma separated list of kafka topics to subscribe to"
-```
-
-In particular, the `onlinefs/config_dir` should contain the credentials necessary for the Kafka consumers to authenticate.
-Additionally the directory should contain a file name `onlinefs-kafka.properties` with the Kafka consumer configuration.
-The following is an example of the `onlinefs-kafka.properties` file:
-
-```properties
-bootstrap.servers=cluster_identifier.us-east-2.aws.confluent.cloud:9092
-security.protocol=SASL_SSL
-sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password";
-sasl.mechanism=PLAIN
-```
+Project members with the Data Owner role can enable and further tune their project's OnlineFS instance under `Project Settings` → `OnlineFS Service`, including pausing ingestion, changing the number of replicas and resources, enabling VectorDB ingestion, and overriding any OnlineFS service or Kafka client setting.
+The available settings are the same as for the cluster-level instance, documented in the [OnlineFS service guide][onlinefs-configuration].
+Per-project instances are always managed by Hopsworks, independently of whether the cluster-level instance is Helm-managed or Hopsworks-managed.
 
 !!! note "Hopsworks will not provision topics"
     Please note that when using an external Kafka cluster, Hopsworks will not provision the topics for the different projects.
     Users are responsible for provisioning the necessary topics and configure the projects accordingly (see next section).
-    Users should also specify the list of topics OnlineFS should subscribe to by providing the `onlinefs/kafka_consumers/topic_list` option in the cluster definition.
+    The OnlineFS instance discovers matching topics automatically; if a feature group uses a custom topic name that does not follow the default naming scheme, the subscription can be widened with a `kafkaConsumer.topicPattern` service setting override in `Project Settings` → `OnlineFS Service`.
 
 ### Project configuration
 

mkdocs.yml

@@ -269,6 +269,7 @@ nav:
           - User Management: setup_installation/admin/user.md
           - Project Management: setup_installation/admin/project.md
           - Configure Alerts: setup_installation/admin/alert.md
+          - OnlineFS Service: setup_installation/admin/onlinefs.md
           - IAM Role Chaining: setup_installation/admin/roleChaining.md
           - Configure Project Mapping: setup_installation/admin/configure-project-mapping.md
           - Airflow 3 operator notes: setup_installation/admin/airflow3.md