Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 21 additions & 2 deletions tidb-cloud/naming-conventions-for-data-import.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,28 @@ If the SQL file is exported through TiDB Dumpling with the default configuration

## File pattern

If the source data file of CSV or Parquet does not conform to the naming convention, you can manually map the source data file to the target table using a file name pattern. This feature does not support Aurora Snapshot and SQL data files.
If a CSV or Parquet source file does not conform to the naming convention, you can manually map the source file to a target table using a file name pattern.

In the import wizard, on the **Destination Mapping** step, deselect **Use TiDB file naming conventions for automatic mapping**, and then fill in the **Source**, **Target Database**, and **Target Table** fields. The **Source** field accepts a file name pattern that supports the `*` and `?` wildcards.
<CustomContent plan="starter,essential,dedicated">

Manual file-pattern mapping does not support Aurora Snapshot or SQL data files.

</CustomContent>
<CustomContent plan="premium">

Manual file-pattern mapping does not support SQL data files.

</CustomContent>

In the mapping step of the import wizard, deselect **Use TiDB file naming conventions for automatic mapping**, and then fill in the **Source**, **Target Database**, and **Target Table** fields. The **Source** field accepts a file name pattern relative to the source URI and supports the `*` and `?` wildcards.

<CustomContent plan="premium">

> **Note:**
>
> For Parquet files exported from an Aurora Snapshot, manual mapping applies only the source patterns that you configure. It does not infer a complete snapshot mapping or create the target schema. Create the target databases and tables before the import, add a mapping for each target table, and verify that the pre-check scans the expected number of data files and maps each source pattern to the intended target table.

</CustomContent>

- For CSV files, see [Step 4. Import CSV files to TiDB Cloud](/tidb-cloud/import-csv-files.md#step-4-import-csv-files-to-tidb-cloud).
- For Parquet files, see [Step 4. Import Parquet files to TiDB Cloud](/tidb-cloud/import-parquet-files.md#step-4-import-parquet-files-to-tidb-cloud).
68 changes: 26 additions & 42 deletions tidb-cloud/premium/import-csv-files-premium.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ To ensure data consistency, {{{ .premium }}} allows importing CSV files into emp
>
> - To achieve better performance, it is recommended to limit the size of each compressed file to 100 MiB.
> - The Snappy compressed file must be in the [official Snappy format](https://github.com/google/snappy). Other variants of Snappy compression are not supported.
> - For uncompressed files, if you cannot update the CSV filenames according to the preceding rules in some cases (for example, the CSV file links are also used by your other programs), you can keep the filenames unchanged and use the **Mapping Settings** in [Step 4](#step-4-import-csv-files) to import your source data to a single target table.
> - For uncompressed files, if you cannot update the CSV filenames according to the preceding rules in some cases (for example, the CSV file links are also used by your other programs), you can keep the filenames unchanged and configure manual source-to-target mappings in [Step 4](#step-4-import-csv-files).

## Step 2. Create the target table schemas

Expand Down Expand Up @@ -105,9 +105,7 @@ To import the CSV files to {{{ .premium }}}, take the following steps:
3. On the **Import Data from Cloud Storage** page, provide the following information:

- **Storage Provider**: select **Amazon S3**.
- **Source Files URI**:
- When importing one file, enter the source file URI in the following format `s3://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `s3://sampledata/ingest/TableName.01.csv`.
- When importing multiple files, enter the source folder URI in the following format `s3://[bucket_name]/[data_source_folder]/`. For example, `s3://sampledata/ingest/`.
- **Source Files URI**: enter the URI of the top-level folder that contains the source files. For example, `s3://sampledata/ingest/`.
- **Credential**: you can use either an AWS Role ARN or an AWS access key to access your bucket. For more information, see [Configure Amazon S3 access](/tidb-cloud/configure-external-storage-access.md#configure-amazon-s3-access).
- **AWS Role ARN**: enter the AWS Role ARN value. If you need to create a new role, click **Click here to create a new one with AWS CloudFormation** and follow the guided steps to launch the provided template, acknowledge the IAM warning, create the stack, and copy the generated ARN back into {{{ .premium }}}.
- **AWS Access Key**: enter the AWS access key ID and AWS secret access key.
Expand All @@ -116,29 +114,23 @@ To import the CSV files to {{{ .premium }}}, take the following steps:

4. Click **Next**.

5. In the **Source Files Mapping** section, {{{ .premium }}} scans the bucket and proposes mappings between the source files and destination tables.
5. In the **Source Files Mapping** section, specify how source files are mapped to target tables.

When a directory is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.
The **Use [TiDB file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.

> **Note:**
>
> When a single file is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is not displayed, and {{{ .premium }}} automatically populates the **Source** field with the file name. In this case, you only need to select the target database and table for data import.
- To use automatic mapping, leave the option selected. {{{ .premium }}} applies the [TiDB file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables.

- Leave automatic mapping enabled to apply the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables. Keep **CSV** selected as the data format.
- To configure mappings manually, deselect the automatic mapping option, and then configure a mapping for each target table:

- **Advanced options**: expand the panel to view the `Ignore compatibility checks (advanced)` toggle. Leave it disabled unless you intentionally want to bypass schema compatibility validation.
- **Source**: enter a source file pattern relative to the **Source Files URI**. The pattern supports `*` and `?` wildcards. For example, `my-data*.csv` matches all CSV files whose names start with `my-data`.
- **Target Database** and **Target Table**: enter the target database and table for the matched files.
- To add another mapping, click **+**.

<!-- future feature -->
> **Note:**
>
> Manual mapping is coming soon. When the toggle becomes available, clear the automatic mapping option and configure the mapping manually:
>
> - **Source**: enter a filename pattern such as `TableName.01.csv`. Wildcards `*` and `?` are supported (for example, `my-data*.csv`).
> - **Target Database** and **Target Table**: choose the destination objects for the matched files.
6. Click **Next** to run the pre-check. Review the scan results and verify the source files and target tables.

6. {{{ .premium }}} automatically scans the source path. Review the scan results, check the data files found and corresponding target tables, and then click **Start Import**.
7. Click **Start Import**.

7. When the import progress shows **Completed**, check the imported tables.
8. When the import progress shows **Completed**, check the imported tables.

</div>

Expand All @@ -159,38 +151,30 @@ To import the CSV files to {{{ .premium }}}, take the following steps:
3. On the **Import Data from Cloud Storage** page, provide the following information:

- **Storage Provider**: select **Alibaba Cloud OSS**.
- **Source Files URI**:
- When importing one file, enter the source file URI in the following format `oss://[bucket_name]/[data_source_folder]/[file_name].csv`. For example, `oss://sampledata/ingest/TableName.01.csv`.
- When importing multiple files, enter the source folder URI in the following format `oss://[bucket_name]/[data_source_folder]/`. For example, `oss://sampledata/ingest/`.
- **Source Files URI**: enter the URI of the top-level folder that contains the source files. For example, `oss://sampledata/ingest/`.
- **Credential**: you can use an AccessKey pair to access your bucket. For more information, see [Configure Alibaba Cloud Object Storage Service (OSS) access](/tidb-cloud/configure-external-storage-access.md#configure-alibaba-cloud-object-storage-service-oss-access).
- **Test Bucket Access**: click this button after the credentials are in place to confirm that {{{ .premium }}} can reach the bucket.
- **Target Connection**: provide the TiDB username and password that will run the import. Optionally, click **Test Connection** to validate the credentials.

4. Click **Next**.

5. In the **Source Files Mapping** section, {{{ .premium }}} scans the bucket and proposes mappings between the source files and destination tables.
5. In the **Source Files Mapping** section, specify how source files are mapped to target tables.

When a directory is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.
The **Use [TiDB file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is selected by default.

> **Note:**
>
> When a single file is specified in **Source Files URI**, the **Use [File naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping** option is not displayed, and {{{ .premium }}} automatically populates the **Source** field with the file name. In this case, you only need to select the target database and table for data import.
- To use automatic mapping, leave the option selected. {{{ .premium }}} applies the [TiDB file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables.

- Leave automatic mapping enabled to apply the [file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) to your source files and target tables. Keep **CSV** selected as the data format.
- To configure mappings manually, deselect the automatic mapping option, and then configure a mapping for each target table:

- **Advanced options**: expand the panel to view the `Ignore compatibility checks (advanced)` toggle. Leave it disabled unless you intentionally want to bypass schema compatibility validation.
- **Source**: enter a source file pattern relative to the **Source Files URI**. The pattern supports `*` and `?` wildcards. For example, `my-data*.csv` matches all CSV files whose names start with `my-data`.
- **Target Database** and **Target Table**: enter the target database and table for the matched files.
- To add another mapping, click **+**.

<!-- future feature -->
> **Note:**
>
> Manual mapping is coming soon. When the toggle becomes available, clear the automatic mapping option and configure the mapping manually:
>
> - **Source**: enter a filename pattern such as `TableName.01.csv`. Wildcards `*` and `?` are supported (for example, `my-data*.csv`).
> - **Target Database** and **Target Table**: choose the destination objects for the matched files.
6. Click **Next** to run the pre-check. Review the scan results and verify the source files and target tables.

6. {{{ .premium }}} automatically scans the source path. Review the scan results, check the data files found and corresponding target tables, and then click **Start Import**.
7. Click **Start Import**.

7. When the import progress shows **Completed**, check the imported tables.
8. When the import progress shows **Completed**, check the imported tables.
Comment thread
coderabbitai[bot] marked this conversation as resolved.

</div>

Expand All @@ -209,10 +193,10 @@ If you get an importing error, do the following:

### Resolve warnings during data import

After clicking **Start Import**, if you see a warning message such as `can't find the corresponding source files`, resolve this by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or using **Advanced Settings** to make changes.
If the pre-check shows a warning such as `can't find the corresponding source files`, resolve it by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or returning to **Source Files Mapping** and configuring manual mappings.

After resolving these issues, you need to import the data again.
After resolving the issues, run the pre-check again.

### Zero rows in the imported tables

After the import progress shows **Completed**, check the imported tables. If the number of rows is zero, it means no data files matched the Bucket URI that you entered. In this case, resolve this issue by providing the correct source file, renaming the existing one according to [Naming Conventions for Data Import](/tidb-cloud/naming-conventions-for-data-import.md), or using **Advanced Settings** to make changes. After that, import those tables again.
After the import progress shows **Completed**, check the imported tables. If the number of rows is zero, verify that the **Source Files URI** is correct and that the source files either follow the [TiDB file naming conventions](/tidb-cloud/naming-conventions-for-data-import.md) for automatic mapping or match the manual source patterns. Correct the URI, rename the files, or configure manual mappings, and then import the tables again.
11 changes: 6 additions & 5 deletions tidb-cloud/premium/import-from-s3-premium.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,7 @@ This document describes how to import CSV files from Amazon Simple Storage Servi
## Limitations

- To ensure data consistency, {{{ .premium }}} allows importing CSV files into empty tables only. If the target table already contains data, import into a staging table and then copy the rows using the `INSERT ... SELECT` statement.
- During the public preview, the user interface currently supports Amazon S3 as the only storage provider. Support for additional providers will be added in future releases.
- Each import job maps a single source pattern to one destination table.
- Each source pattern maps to one target table. You can add multiple mappings to an import job.

## Step 1. Prepare the CSV files

Expand Down Expand Up @@ -50,12 +49,14 @@ The wizard includes a helper link labeled **Click here to create a new one with
2. In the left navigation pane, click **Data** > **Import**, and choose **Import data from Cloud Storage**.
3. In the **Source Connection** dialog:
- Set **Storage Provider** to **Amazon S3**.
- Enter the **Source Files URI** for a single file (`s3://bucket/path/file.csv`) or for a folder (`s3://bucket/path/`).
- Enter the **Source Files URI** for the top-level folder that contains the source files, for example, `s3://bucket/path/`.
- Choose **AWS Role ARN** or **AWS Access Key** and provide the credentials.
- Click **Test Bucket Access** to validate connectivity. <!--Todo-- Known preview issue: the button returns to the idle state without a success toast.-->
- Click **Test Bucket Access** to validate connectivity.

4. Click **Next** and provide the TiDB SQL username and password for the import job. Optionally, test the connection.
5. Review the automatically generated source-to-target mapping. Disable automatic mapping if you need to define custom patterns and destination tables.
5. Configure the source-to-target mapping:
- To use automatic mapping, leave **Use TiDB file naming conventions for automatic mapping** selected.
- To configure mappings manually, deselect the automatic mapping option. For each target table, enter a source file pattern relative to the **Source Files URI**, and then enter the target database and table. The source pattern supports `*` and `?` wildcards. To add another mapping, click **+**.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
6. Click **Next** to run the pre-check. Resolve any warnings about missing files or incompatible schemas.
7. Click **Start Import** to launch the job group.
8. Monitor the job statuses until they show **Completed**, then verify the imported data in TiDB Cloud.
Expand Down