Skip to content

[AMORO-3989] Amoro's support for the domestic database Damang #3992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
linjianchang wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[AMORO-3989] Amoro's support for the domestic database Damang #3992

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions amoro-ams/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,12 @@
<artifactId>derby</artifactId>
</dependency>

<dependency>
<groupId>com.dameng</groupId>
<artifactId>DmJdbcDriver18</artifactId>
<scope>provided</scope>
</dependency>
Copy link
Contributor

@xxubai xxubai Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be provided

Copy link
Author

@linjianchang linjianchang Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be provided

企业微信截图_17659411772400 I see that the JDBC driver packages for PostgreSQL and Derby are not provided

Copy link
Contributor

@xxubai xxubai Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think Dameng needs to be included in the release. Similar to MySQL, users who need it can add the JDBC dependency manually, rather than having it as a standard component.

Copy link
Author

@linjianchang linjianchang Dec 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t think Dameng needs to be included in the release. Similar to MySQL, users who need it can add the JDBC dependency manually, rather than having it as a standard component.

Aleady modified!

Copy link
Author

@linjianchang linjianchang Dec 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xxubai Thanks for your review. Do you have any other suggestions


<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
Expand Down Expand Up @@ -564,6 +570,7 @@
<exclude>derby/**</exclude>
<exclude>mysql/**</exclude>
<exclude>postgres/**</exclude>
<exclude>dameng/**</exclude>
</excludes>
</configuration>
</plugin>
Expand Down
1 change: 1 addition & 0 deletions amoro-ams/src/main/java/org/apache/amoro/server/AmoroManagementConf.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,7 @@ public class AmoroManagementConf {
public static final String DB_TYPE_DERBY = "derby";
public static final String DB_TYPE_MYSQL = "mysql";
public static final String DB_TYPE_POSTGRES = "postgres";
public static final String DB_TYPE_DAMENG = "dameng";

// terminal config
public static final List<String> TERMINAL_BACKEND_VALUES =
Expand Down
6 changes: 4 additions & 2 deletions amoro-ams/src/main/java/org/apache/amoro/server/AmoroManagementConfValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,17 @@ public static void validateConfig(Configurations configurations) {
AmsUtil.lookForBindHost(configurations.getString(AmoroManagementConf.SERVER_EXPOSE_HOST));
configurations.setString(AmoroManagementConf.SERVER_EXPOSE_HOST, inetAddress.getHostAddress());

// mysql or postgres config
// mysql or postgres or dameng config
if (AmoroManagementConf.DB_TYPE_MYSQL.equalsIgnoreCase(
configurations.getString(AmoroManagementConf.DB_TYPE))
|| AmoroManagementConf.DB_TYPE_POSTGRES.equalsIgnoreCase(
configurations.getString(AmoroManagementConf.DB_TYPE))
|| AmoroManagementConf.DB_TYPE_DAMENG.equalsIgnoreCase(
configurations.getString(AmoroManagementConf.DB_TYPE))) {
if ("".equals(configurations.getString(AmoroManagementConf.DB_PASSWORD))
|| "".equals(configurations.getString(AmoroManagementConf.DB_USER_NAME))) {
throw new IllegalArgumentException(
"username and password must be configured if the database type is mysql or postgres");
"username and password must be configured if the database type is mysql or postgres or dameng ");
}
}

Expand Down
9 changes: 8 additions & 1 deletion amoro-ams/src/main/java/org/apache/amoro/server/persistence/DataSourceFactory.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,14 +47,16 @@ public class DataSourceFactory {
private static final String DERBY_INIT_SQL_SCRIPT = "derby/ams-derby-init.sql";
private static final String MYSQL_INIT_SQL_SCRIPT = "mysql/ams-mysql-init.sql";
private static final String POSTGRES_INIT_SQL_SCRIPT = "postgres/ams-postgres-init.sql";
private static final String DAMENG_INIT_SQL_SCRIPT = "dameng/ams-dameng-init.sql";

public static DataSource createDataSource(Configurations config) {
BasicDataSource dataSource = new BasicDataSource();
dataSource.setUrl(config.getString(AmoroManagementConf.DB_CONNECTION_URL));
dataSource.setDriverClassName(config.getString(AmoroManagementConf.DB_DRIVER_CLASS_NAME));
String dbType = config.getString(AmoroManagementConf.DB_TYPE);
if (AmoroManagementConf.DB_TYPE_MYSQL.equals(dbType)
|| AmoroManagementConf.DB_TYPE_POSTGRES.equals(dbType)) {
|| AmoroManagementConf.DB_TYPE_POSTGRES.equals(dbType)
|| AmoroManagementConf.DB_TYPE_DAMENG.equals(dbType)) {
dataSource.setUsername(config.getString(AmoroManagementConf.DB_USER_NAME));
dataSource.setPassword(config.getString(AmoroManagementConf.DB_PASSWORD));
}
Expand Down Expand Up @@ -104,6 +106,9 @@ private static void createTablesIfNeed(DataSource ds, Configurations config) {
String.format(
"SELECT 1 FROM information_schema.tables WHERE table_schema = %s AND table_name = '%s'",
"current_schema()", "catalog_metadata");
} else if (AmoroManagementConf.DB_TYPE_DAMENG.equals(dbTypeConfig)) {
query =
String.format("SELECT 1 FROM user_tables WHERE table_name = '%s'", "catalog_metadata");
}
LOG.info("Start check table creation, using query: {}", query);
try (ResultSet rs = statement.executeQuery(query)) {
Expand Down Expand Up @@ -131,6 +136,8 @@ private static URI getInitSqlScriptPath(String type) throws URISyntaxException {
scriptPath = DERBY_INIT_SQL_SCRIPT;
} else if (type.equals(AmoroManagementConf.DB_TYPE_POSTGRES)) {
scriptPath = POSTGRES_INIT_SQL_SCRIPT;
} else if (type.equals(AmoroManagementConf.DB_TYPE_DAMENG)) {
scriptPath = DAMENG_INIT_SQL_SCRIPT;
}
URL scriptUrl = ClassLoader.getSystemResource(scriptPath);
if (scriptUrl == null) {
Expand Down
2 changes: 2 additions & 0 deletions amoro-ams/src/main/java/org/apache/amoro/server/persistence/SqlSessionFactoryProvider.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ public void init(DataSource dataSource) throws SQLException {
properties.setProperty("MySQL", "mysql");
properties.setProperty("PostgreSQL", "postgres");
properties.setProperty("Derby", "derby");
properties.setProperty("DM DBMS", "dameng");
provider.setProperties(properties);
configuration.setDatabaseId(provider.getDatabaseId(dataSource));
if (sqlSessionFactory == null) {
Expand All @@ -101,6 +102,7 @@ private void registerDialectAliases() {
registerDialectAlias("postgres", PostgreSqlDialect.class);
registerDialectAlias("mysql", MySqlDialect.class);
registerDialectAlias("derby", SqlServer2012Dialect.class);
registerDialectAlias("dameng", MySqlDialect.class);
}

public SqlSessionFactory get() {
Expand Down
1 change: 1 addition & 0 deletions amoro-ams/src/main/java/org/apache/amoro/server/persistence/mapper/TableRuntimeMapper.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,6 +107,7 @@ public interface TableRuntimeMapper {
+ "<bind name=\"isMySQL\" value=\"_databaseId == 'mysql'\" />"
+ "<bind name=\"isPostgreSQL\" value=\"_databaseId == 'postgres'\" />"
+ "<bind name=\"isDerby\" value=\"_databaseId == 'derby'\" />"
+ "<bind name=\"isDerby\" value=\"_databaseId == 'dameng'\" />"
+ "SELECT r.table_id, group_name, status_code, status_code_update_time, table_config, table_summary, bucket_id FROM "
+ TABLE_NAME
+ " r JOIN table_identifier i "
Expand Down
2 changes: 1 addition & 1 deletion amoro-ams/src/test/java/org/apache/amoro/server/TestAmoroManagementConfValidator.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ public void testValidateServerExposeHost() {
}

public static Stream<Arguments> testValidateDBConfig() {
return Stream.of(Arguments.of("mysql"), Arguments.of("postgres"));
return Stream.of(Arguments.of("mysql"), Arguments.of("postgres"), Arguments.of("dameng"));
}

@ParameterizedTest
Expand Down
2 changes: 1 addition & 1 deletion charts/amoro/templates/amoro-deployment.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ spec: {{/* TODO If Support Replica can be use more than 1 */}}
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: [ "/entrypoint.sh", "ams" ]
env:
{{- if or (eq .Values.amoroConf.database.type "mysql") (eq .Values.amoroConf.database.type "postgres") }}
{{- if or (eq .Values.amoroConf.database.type "mysql") (eq .Values.amoroConf.database.type "postgres") (eq .Values.amoroConf.database.type "dameng") }}
- name: "AMS_DATABASE_USERNAME"
valueFrom:
secretKeyRef:
Expand Down
2 changes: 1 addition & 1 deletion charts/amoro/templates/amoro-secret.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ data:
AdminUsername: {{ .Values.amoroConf.ams.adminUsername | b64enc | quote }}
AdminPassword: {{ .Values.amoroConf.ams.adminPassword | b64enc | quote }}

{{- if or (eq .Values.amoroConf.database.type "mysql") (eq .Values.amoroConf.database.type "postgres") }}
{{- if or (eq .Values.amoroConf.database.type "mysql") (eq .Values.amoroConf.database.type "postgres") (eq .Values.amoroConf.database.type "dameng") }}
# use helm set database username and password
DatabaseUsername: {{ .Values.amoroConf.database.username | b64enc | quote }}
DatabasePassword: {{ .Values.amoroConf.database.password | b64enc | quote }}
Expand Down
5 changes: 5 additions & 0 deletions dist/src/main/assemblies/bin.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,11 @@
<outputDirectory>conf/derby</outputDirectory>
<fileMode>0644</fileMode>
</fileSet>
<fileSet>
<directory>../amoro-ams/src/main/resources/dameng</directory>
<outputDirectory>conf/dameng</outputDirectory>
<fileMode>0644</fileMode>
</fileSet>
<fileSet>
<directory>../amoro-ams/target/amoro-ams-dependency/lib</directory>
<outputDirectory>lib/</outputDirectory>
Expand Down
5 changes: 5 additions & 0 deletions dist/src/main/assemblies/release-bin.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,11 @@
<outputDirectory>conf/derby</outputDirectory>
<fileMode>0644</fileMode>
</fileSet>
<fileSet>
<directory>../amoro-ams/src/main/resources/dameng</directory>
<outputDirectory>conf/dameng</outputDirectory>
<fileMode>0644</fileMode>
</fileSet>
<fileSet>
<directory>../amoro-ams/target/amoro-ams-dependency/lib</directory>
<outputDirectory>lib/</outputDirectory>
Expand Down
3 changes: 3 additions & 0 deletions docker/amoro/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ WORKDIR ${AMORO_HOME}
RUN cd ${AMORO_HOME}/lib \
&& wget ${MAVEN_MIRROR}/mysql/mysql-connector-java/8.0.30/mysql-connector-java-8.0.30.jar

RUN cd ${AMORO_HOME}/lib \
&& wget ${MAVEN_MIRROR}/com/dameng/DmJdbcDriver18/8.1.2.141/DmJdbcDriver18-8.1.2.141.jar

ENTRYPOINT ["/entrypoint.sh"]
CMD ["help"]

7 changes: 7 additions & 0 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,7 @@
<snakeyaml.version>2.2</snakeyaml.version>
<mybatis.version>3.5.15</mybatis.version>
<postgres-jdbc.version>42.7.2</postgres-jdbc.version>
<dameng-jdbc.version>8.1.2.141</dameng-jdbc.version>
<derby-jdbc.version>10.14.2.0</derby-jdbc.version>
<commons-dbcp2.version>2.9.0</commons-dbcp2.version>
<netty.version>4.1.128.Final</netty.version>
Expand Down Expand Up @@ -787,6 +788,12 @@
<version>${mysql-jdbc.version}</version>
</dependency>

<dependency>
<groupId>com.dameng</groupId>
<artifactId>DmJdbcDriver18</artifactId>
<version>${dameng-jdbc.version}</version>
</dependency>

<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-dbcp2</artifactId>
Expand Down