feat: add support for opensearch containers #30
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Opensearch | ||
|
|
||
| Enable Opensearch via the property `opensearch.enabled`. Opensearch is available on port `9200`. | ||
|
|
||
| _note that `elasticsearch.enabled` must be `false` for the Opensearch container to start_ | ||
|
|
||
| ## Related properties | ||
|
|
||
| | Property | Usage | Default | | ||
| |--------------------------------------|-----------------------------------------------------------------|---------------| | ||
| | opensearch.enabled | Whether a Docker Opensearch container should be started. | `false` | | ||
| | opensearch.image.tag | The image tag of the Opensearch Docker container to use. | `2.15.0` | | ||
| | opensearch.password | The Opensearch password to use. | | | ||
| | opensearch.cluster.name | The name of the Opensearch cluster. | `opensearch` | | ||
| | opensearch.discovery.type | Whether to form a single node or multi node Opensearch cluster. | `single-node` | | ||
| | opensearch.container.logging.enabled | Whether to output the Opensearch Docker logs to the console. | `false` | | ||
|
|
||
| The container base URL can be obtained using the `OpensearchCtfClient`: | ||
|
|
||
| ``` | ||
| import dev.lydtech.component.framework.client.opensearch.OpensearchCtfClient; | ||
|
|
||
| String baseUrl = OpensearchCtfClient.getInstance().getBaseUrl(); | ||
| ``` | ||
|
|
||
| The `org.opensearch.client.opensearch` can be obtained which can then be used to query the dockerised | ||
| Opensearch. Note this method is overloaded, also taking a `JsonpMapper`. | ||
|
|
||
| ``` | ||
| OpenSearchClient osClient = OpensearchCtfClient.getInstance().getOpensearchClient(); | ||
| GetResponse<Item> getResponse = osClient.get(s -> s | ||
| .index("item") | ||
| .id(location), Item.class); | ||
| ``` |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,50 @@ | ||
| package dev.lydtech.component.framework.client.opensearch; | ||
|
|
||
| import java.util.Optional; | ||
| import lombok.extern.slf4j.Slf4j; | ||
| import org.apache.http.HttpHost; | ||
| import org.opensearch.client.RestClient; | ||
| import org.opensearch.client.json.JsonpMapper; | ||
| import org.opensearch.client.json.jackson.JacksonJsonpMapper; | ||
| import org.opensearch.client.opensearch.OpenSearchClient; | ||
| import org.opensearch.client.transport.OpenSearchTransport; | ||
| import org.opensearch.client.transport.rest_client.RestClientTransport; | ||
|
|
||
| @Slf4j | ||
| public class OpensearchCtfClient { | ||
|
|
||
| private static OpensearchCtfClient instance; | ||
| private static String baseUrl; | ||
|
|
||
| private OpensearchCtfClient() { | ||
| String host = Optional.ofNullable(System.getProperty("docker.host")) | ||
| .orElse("localhost"); | ||
| String port = Optional.ofNullable(System.getProperty("opensearch.mapped.port")) | ||
| .orElseThrow(() -> new RuntimeException("opensearch.mapped.port property not found")); | ||
| baseUrl = "http://" + host + ":" + port; | ||
| log.info("Opensearch base URL is: " + baseUrl); | ||
| } | ||
|
|
||
| public synchronized static OpensearchCtfClient getInstance() { | ||
| if(instance==null) { | ||
| instance = new OpensearchCtfClient(); | ||
| } | ||
| return instance; | ||
| } | ||
|
|
||
| public String getBaseUrl() { | ||
| return baseUrl; | ||
| } | ||
|
|
||
| public OpenSearchClient getOpensearchClient() { | ||
| return getOpensearchClient(new JacksonJsonpMapper()); | ||
| } | ||
|
|
||
|
|
||
| public OpenSearchClient getOpensearchClient(JsonpMapper mapper) { | ||
| RestClient restClient = RestClient.builder(HttpHost.create(baseUrl)).build(); | ||
| OpenSearchTransport transport = new RestClientTransport(restClient, mapper); | ||
| return new OpenSearchClient(transport); | ||
| } | ||
|
|
||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -24,6 +24,7 @@ | |
| import org.apache.kafka.clients.admin.NewTopic; | ||
| import org.apache.kafka.common.config.SaslConfigs; | ||
| import org.apache.kafka.common.security.plain.PlainLoginModule; | ||
| import org.opensearch.testcontainers.OpensearchContainer; | ||
| import org.slf4j.LoggerFactory; | ||
| import org.testcontainers.containers.BindMode; | ||
| import org.testcontainers.containers.GenericContainer; | ||
|
|
@@ -48,6 +49,7 @@ | |
| import static dev.lydtech.component.framework.resource.Resource.KAFKA_SCHEMA_REGISTRY; | ||
| import static dev.lydtech.component.framework.resource.Resource.LOCALSTACK; | ||
| import static dev.lydtech.component.framework.resource.Resource.MONGODB; | ||
| import static dev.lydtech.component.framework.resource.Resource.OPENSEARCH; | ||
| import static dev.lydtech.component.framework.resource.Resource.POSTGRES; | ||
| import static dev.lydtech.component.framework.resource.Resource.MARIADB; | ||
| import static dev.lydtech.component.framework.resource.Resource.WIREMOCK; | ||
|
|
@@ -73,6 +75,7 @@ public final class TestcontainersManager { | |
| private GenericContainer conduktorContainer; | ||
| private GenericContainer conduktorGatewayContainer; | ||
| private GenericContainer elasticSearchContainer; | ||
| private GenericContainer openSearchContainer; | ||
| private GenericContainer ambarContainer; | ||
|
|
||
| private TestcontainersManager(){} | ||
|
|
@@ -184,6 +187,9 @@ private void createContainers() { | |
| if (ELASTICSEARCH_ENABLED) { | ||
| elasticSearchContainer = createElasticsearchContainer(); | ||
| } | ||
| if (OPENSEARCH_ENABLED && !ELASTICSEARCH_ENABLED) { | ||
| openSearchContainer = createOpensearchContainer(); | ||
| } | ||
| if(AMBAR_ENABLED) { | ||
| ambarContainer = createAmbarContainer(); | ||
| } | ||
|
|
@@ -248,6 +254,9 @@ private void startContainers() { | |
| if(ELASTICSEARCH_ENABLED) { | ||
| elasticSearchContainer.start(); | ||
| } | ||
| if(OPENSEARCH_ENABLED) { | ||
| openSearchContainer.start(); | ||
| } | ||
| if(AMBAR_ENABLED) { | ||
| ambarContainer.start(); | ||
| } | ||
|
|
@@ -743,6 +752,28 @@ private GenericContainer createElasticsearchContainer() { | |
| return container; | ||
| } | ||
|
|
||
| private GenericContainer createOpensearchContainer() { | ||
| String containerName = OPENSEARCH.toString(); | ||
| DockerImageName opensearchImage = DockerImageName.parse("opensearchproject/opensearch" + ":" + OPENSEARCH_IMAGE_TAG); | ||
| OpensearchContainer container = new OpensearchContainer(opensearchImage.withTag(OPENSEARCH_IMAGE_TAG)) | ||
| .withNetwork(network) | ||
| .withNetworkAliases(containerName) | ||
| .withEnv("cluster.name", OPENSEARCH_CLUSTER_NAME) | ||
| .withEnv("discovery.type", OPENSEARCH_DISCOVERY_TYPE) | ||
| .withEnv("DISABLE_INSTALL_DEMO_CONFIG", "true") | ||
| .withEnv("DISABLE_SECURITY_PLUGIN", "true") | ||
| .withEnv("cluster.routing.allocation.disk.threshold_enabled", "false") | ||
|
Contributor
Author
There was a problem hiding this comment. avoiding tests failing to create indexes - especially on CI platforms |
||
| .withReuse(true) | ||
| .withCreateContainerCmdModifier(cmd -> { | ||
| String containerCmdModifier = CONTAINER_APPEND_GROUP_ID ?CONTAINER_NAME_PREFIX + "-" + containerName + "-" + CONTAINER_GROUP_ID :CONTAINER_NAME_PREFIX + "-" + containerName; | ||
| cmd.withName(containerCmdModifier); | ||
| }); | ||
| if(OPENSEARCH_CONTAINER_LOGGING_ENABLED) { | ||
| container.withLogConsumer(getLogConsumer(containerName)); | ||
| } | ||
| return container; | ||
| } | ||
|
|
||
| private GenericContainer createAmbarContainer() { | ||
| String containerName = AMBAR.toString(); | ||
| GenericContainer container = new GenericContainer<>("ambarltd/emulator:" + AMBAR_IMAGE_TAG) | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -14,6 +14,7 @@ public enum Resource { | |
| LOCALSTACK, | ||
| MONGODB, | ||
| ELASTICSEARCH, | ||
| OPENSEARCH, | ||
| MARIADB, | ||
| AMBAR; | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are these new dependencies actually needed?