This repo is part of the Data Platform project. The Data Platform consists of services for capturing and providing access to data captured from a particle accelerator facility. The data-platform repo provides a project overview and links to the various project componnents, as well as an installer for running the latest version.
This repo, dp-support includes a set of utilities for managing the processes comprising the Data Platform ecosystem.
The Data Platform ecosystem consists of the following components:
- the service implementations in the dp-service repo
- MongoDB, used for persistence by the Data Platform
- (optionally) An Envoy proxy, for sending http1 traffic from the Data Platform web application to the http2-based gRPC service implementations
- (optionally) An Apache web server, for serving files generated by the export operation.
The dp-support repo includes tools for managing each of these components. Use of the tools is optional. They can be used as is (by downloading the installer from the data-platform repo), or can be treated as a starting place for a custom installation.
Note that Docker installation with command-line access to "docker" is a prerequisite to use the docker-based scripts. Docker is not required for a production installation, but can be a useful way to deploy demo, test, development, and QA systems.
The utilities for each component are described in more detail below.
The data-platform repo includes a Quick Start section for quickly getting the Data Platform up and running, with example usage of the scripts in dp-support to manage the ecosystem.
MongoDB is used for Data Platform persistence. The current reference version is 8.0.6. For details about installing MongoDB as a locally installed package or as a docker container deployment, see the data-platform repo documentation. The tools for managing MongoDB include:
One set of scripts is used for managing a Linux MongoDB installation using the "systemctl" utility. These are simple wrapper scripts:
- mongodb-systemctl-start: Starts standard MongoDB installation using systemctl.
- mongodb-systemctl-stop: Stops MongoDB.
- mongodb-systemctl-status: Checks MongoDB status.
- mongodb-systemctl-enable: Enables MongoDB auto-start after reboot.
- mongodb-systemctl-disable: Disables MongoDB auto-start after reboot.
The second set of scripts is for managing MongoDB deployed as a Docker container. To use these scripts, docker must be installed on the host with command-line access to the "docker" command. The following scripts are included:
- mongodb-docker-create: Creates MongoDB Docker container.
- mongodb-docker-start: Starts the MongoDB Docker container.
- mongodb-docker-stop: Stops the MongoDB Docker container.
- mongodb-docker-remove: Removes the MongoDB Docker container. BEWARE THIS WILL DELETE ALL DATA IN THE DATABASE.
- mongodb-docker-shell: Runs mongosh shell against the (running) MongoDB Docker container.
Compass is a GUI application for navigating a MongoDB database. This repo contains a simple wrapper script for starting the application, passing the default MongoDB connection string (MongoDB user/password = "admin") on start up:
mongodb://admin:admin@localhost:27017
This script will need to be modified as appropriate if the user/password are customized.
- mongodb-compass-start: Starts the MongoDB Compass application, passing the default connection string on the command line.
The Envoy proxy is an optional part of the Data Platform ecosystem. It is needed to run the web application, for converting the http1 traffic from the JavaScript browser application into http2 for interacting with the Data Platform services.
Scripts are provided to support deployment of the Envoy proxy as a Docker container, including:
- envoy-docker-create: Creates Envoy Docker container.
- envoy-docker-start: Starts the Envoy Docker container.
- envoy-docker-stop: Stops the Envoy Docker container.
- envoy-docker-remove: Removes the Envoy Docker container.
These scripts use config files "envoy.yaml" and "envoy.mac.yaml" in the "dp-support/config" directory.
The Apache web server is an optional part of the Data Platform ecosystem. It can be used to provide access to files generated by the export operation.
Scripts are provided to support deployment of the Apache server as a Docker container, including:
- apache-docker-create: Creates Apache Docker container.
- apache-docker-start: Starts the Apache Docker container.
- apache-docker-stop: Stops the Apache Docker container.
- apache-docker-remove: Removes the Apache Docker container.
The bin directory contains scripts for managing the Data Platform server applications, including the Ingestion and Query Services. These scripts use a set of lower level scripts in the same directory for starting/stopping processes and checking their status: util-pm-start, util-pm-stop, and util-pm-status, respectively.
- server-ingest-start: Starts the ingestion server application using the util-pm-start script.
- server-ingest-stop: Stops the running ingestion server application using the util-pm-stop script.
- server-ingest-status: Checks the status of the ingestion server application using util-pm-status.
- server-query-start: Starts the query server application using the util-pm-start script.
- server-query-stop: Stops the running query server application using the util-pm-stop script.
- server-query-status: Checks the status of the query server application using util-pm-status.
- server-annotation-start: Starts the annotation server application using the util-pm-start script.
- server-annotation-stop: Stops the running annotation server application using the util-pm-stop script.
- server-annotation-status: Checks the status of the annotation server application using util-pm-status.
- server-ingestion-stream-start: Starts the ingestion stream server application using the util-pm-start script.
- server-ingestion-stream-stop: Stops the running ingestion stream server application using the util-pm-stop script.
- server-ingestion-stream-status: Checks the status of the ingestion stream server application using util-pm-status.
The data-platform installation directory includes a "docker" subdirectory that contains tools for running Data Platform services and applications using Docker.
The "docker" directory in the installation contains a "docker-compose" subdirectory that includes configuration files for using "docker compose" to run Data Platform services. The following scripts are provided:
This docker-compose configuration runs MongoDB as a Docker container, and starts the Data Platform services including Ingestion, Query, Annotation, and Ingestion Stream. This is a simple way to run the Data Platform and supports using regular Java client applications and Dockerized client applications interacting with the services. To start the ecosystem, use the following command:
docker compose -f ~/data-platform/docker/docker-compose/dp-ecosystem/docker-compose.yml -p dp-ecosystem up -d
To run a Dockerized Java client that generates some test data and sends it to the Ingestion Service for archival, use the following command:
~/data-platform/bin/app-run-docker-test-data-generator
This docker-compose configuration runs MongoDB as a Docker container, starts two instances of the Data Platform Ingestion Service (in performance benchmark mode), and runs the Envoy proxy as a Docker container configured to be a round-robin load balancer for the Ingestion Service instances.
The main purpose of this docker-compose configuration is to demonstrate the static configuration using Envoy as a Data Platform ingestion load balancer. To start the load balancer configuration, use the following command:
docker compose -f ~/data-platform/docker/docker-compose/ingestion-load-balancer/docker-compose.yml -p ingestion-load-balancer up -d
To run a Dockerized Java client that runs the Ingestion Service performance benchmark against the load balancer, use the following command:
~/data-platform/bin/app-run-docker-lb-ingestion-benchmark
This sends ingestion requests to the load balancer on port 8080, which dispatches the requests to the two Ingestion Service instances using a round-robin policy.
The data-platform installation directory includes a "kubernetes" subdirectory that contains tools for running Data Platform services and applications using Kubernetes. Inside that directory is a "dp-ecosystem" subdirectory that contains kubernetes config files for running the full Data Platform ecosystem.
The file "deployment-script" illustrates the steps needed to run the system using kubernetes. Because it uses Minikube to run Kubernetes on a local machine, it is not suitable for production. However, the configuration files used by the script are a good starting place for running the Data Platform services on a Kubernetes cluster.
The script loads Docker images for each of the Data Platform services into minikube so they can be used in the deployment configuration files for running each service. It creates a "dp-ecosystem" namespace, and then applies the configuration files creating the Kubernetes deployment and service for each element of the ecosystem including MongoDB and the Data Platform Ingestion, Query, Annotation, and Ingestion Stream services.
The configuration initially starts 2 replicas for each Data Platform service, and uses Kubernetes Horizontal Pod Autscaler (HPA) to manage horizontal scaling. The "hpa.yaml" file specifies rules for scaling each service by CPU and memory utilization.
The bin directory contains scripts for running Data Platform gui applications.
- app-run-desktop-app: Starts the desktop GUI application.
The Data Platform includes applications for running Ingestion and Query Service performance benchmarks. The benchmarks require running non-standard Ingestion and Query servers that override the default port and database name, and then a client application for running scenarios and measuring performance.
The bin directory includes bash scripts for running the benchmark server and client applications.
- server-benchmark-ingest-start: Starts the benchmark ingestion server application using the util-pm-start script.
- server-benchmark-ingest-stop: Stops the running benchmark ingestion server application using the util-pm-stop script.
- server-benchmark-ingest-status: Checks the status of the benchmark ingestion server application using util-pm-status.
- app-run-ingestion-benchmark: Runs the data platform ingestion service performance benchmark application against the running benchmark ingestion server. Displays an error if the server is not running. Uploads one minute's data for 4,000 data sources sampled at 1 KHz.
- app-run-ingestion-bytes-benchmark: Runs the same ingestion scenario as the regular ingestion benchmark, but uses the byte data mechanism for sending SerializedDataColumns instead of regular DataColumns in the ingestion requests.
- server-benchmark-query-start: Starts the benchmark query server application using the util-pm-start script.
- server-benchmark-query-stop: Stops the running benchmark query server application using the util-pm-stop script.
- server-benchmark-query-status: Checks the status of the benchmark query server application using util-pm-status.
- app-run-query-benchmark: Runs the query service performance benchmark application using the server-streaming time-series data query API. Loads a set of data and exercises the query mechanism against it.
- app-run-query-bytes-benchmark: Runs the same query scenario as the regular query benchmark, but uses the byte data mechanism for receiving SerializedDataColumns instead of regular DataColumns in the query result.
The Data Platform includes a utility for generating sample data for use in web application development and demo purposes. The application generates sample data for one minute and 4,000 signals each sampled at 1 KHz. The data generator uses the standard "dp" database and data is created with a fixed date/time starting at "2023-10-31T15:51:00.000+00:00". Happy Halloween...
- app-run-test-data-generator: Runs the sample data generator against the running ingestion server application.