diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index 337967ea..3864dc1e 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -32,12 +32,7 @@ ** {zdm-automation-repo-url}/releases[{zdm-automation} release notes] .{sstable-sideloader} -* xref:sideloader:sideloader-overview.adoc[] -* xref:sideloader:prepare-sideloader.adoc[] -* xref:sideloader:migrate-sideloader.adoc[] -* xref:sideloader:stop-restart-sideloader.adoc[] -* xref:sideloader:cleanup-sideloader.adoc[] -* xref:sideloader:troubleshoot-sideloader.adoc[] +* xref:astra-db-serverless:sideloader:sideloader-overview.adoc[] .{cass-migrator} * {cass-migrator-repo-url}[About {cass-migrator-short}] diff --git a/modules/ROOT/pages/components.adoc b/modules/ROOT/pages/components.adoc index a904184d..31d0b5fb 100644 --- a/modules/ROOT/pages/components.adoc +++ b/modules/ROOT/pages/components.adoc @@ -154,7 +154,7 @@ You can use these tools alone or with {zdm-proxy}. {sstable-sideloader} is a service running in {astra-db} that imports data from snapshots of your existing {cass-short}-based cluster. This tool is exclusively for migrations that move data to {astra-db}. -For more information, see xref:sideloader:sideloader-overview.adoc[]. +For more information, see xref:astra-db-serverless:sideloader:sideloader-overview.adoc[]. === {cass-migrator} diff --git a/modules/ROOT/pages/create-target.adoc b/modules/ROOT/pages/create-target.adoc index eb3f09e6..d5fd3a70 100644 --- a/modules/ROOT/pages/create-target.adoc +++ b/modules/ROOT/pages/create-target.adoc @@ -66,7 +66,7 @@ As a best practice, omit xref:astra-db-serverless:cql:develop-with-cql.adoc#unsu * {astra-db} doesn't support Materialized Views (MVs) and certain types of indexes. You must adjust your data model and application logic to discard or replace these structures before beginning your migration. For more information, see xref:astra-db-serverless:cql:develop-with-cql.adoc#limitations-on-cql-for-astra-db[Limitations on CQL for {astra-db}]. -* If you plan to use {sstable-sideloader} for xref:ROOT:migrate-and-validate-data.adoc[Phase 2], see the xref:sideloader:migrate-sideloader.adoc#record-schema[target database configuration requirements for migrating data with {sstable-sideloader}]. +* If you plan to use {sstable-sideloader} for xref:ROOT:migrate-and-validate-data.adoc[Phase 2], see the xref:astra-db-serverless:sideloader:migrate-sideloader.adoc#record-schema[target database configuration requirements for migrating data with {sstable-sideloader}]. == Migrate to {hcd-short}, {dse-short}, or open-source {cass-reg} diff --git a/modules/ROOT/pages/feasibility-checklists.adoc b/modules/ROOT/pages/feasibility-checklists.adoc index c0648c58..23606252 100644 --- a/modules/ROOT/pages/feasibility-checklists.adoc +++ b/modules/ROOT/pages/feasibility-checklists.adoc @@ -346,7 +346,7 @@ include::ROOT:partial$multi-region-migrations.adoc[] To configure {zdm-proxy} for a multi-datacenter migration, see the xref:ROOT:deployment-infrastructure.adoc[{zdm-proxy} infrastructure guidelines for multi-datacenter clusters]. -For more guidance on migrations to {astra}, see the xref:sideloader:prepare-sideloader.adoc#additional-preparation-for-specific-migration-scenarios[{sstable-sideloader} preparations for specific migration scenarios]. +For more guidance on migrations to {astra}, see the xref:astra-db-serverless:sideloader:prepare-sideloader.adoc#additional-preparation-for-specific-migration-scenarios[{sstable-sideloader} preparations for specific migration scenarios]. == Next steps diff --git a/modules/ROOT/pages/index.adoc b/modules/ROOT/pages/index.adoc index 6777e3b7..cd10ffc5 100644 --- a/modules/ROOT/pages/index.adoc +++ b/modules/ROOT/pages/index.adoc @@ -24,7 +24,7 @@ @@ -60,7 +60,7 @@ svg::sideloader:astra-migration-toolkit.svg[role="absolute bottom-1/2 translate-

{sstable-sideloader} is a service running in {astra-db} that directly imports data from snapshots of your existing {cass-short}-based cluster.

- xref:sideloader:sideloader-overview.adoc[Get started with {sstable-sideloader}] + xref:astra-db-serverless:sideloader:sideloader-overview.adoc[Get started with {sstable-sideloader}]
diff --git a/modules/ROOT/pages/migrate-and-validate-data.adoc b/modules/ROOT/pages/migrate-and-validate-data.adoc index 7a61328d..76e2d859 100644 --- a/modules/ROOT/pages/migrate-and-validate-data.adoc +++ b/modules/ROOT/pages/migrate-and-validate-data.adoc @@ -27,10 +27,10 @@ For compatible origin clusters, see xref:ROOT:astra-migration-paths.adoc[]. You can use {sstable-sideloader} alone or with {zdm-proxy}. -For more information and instructions, see xref:sideloader:sideloader-overview.adoc[]. +For more information and instructions, see xref:astra-db-serverless:sideloader:sideloader-overview.adoc[]. .Use {sstable-sideloader} with {zdm-proxy} -svg::sideloader:astra-migration-toolkit.svg[] +svg::astra-db-serverless:sideloader:astra-migration-toolkit.svg[] == {cass-migrator} diff --git a/modules/sideloader/images/astra-migration-toolkit.svg b/modules/sideloader/images/astra-migration-toolkit.svg deleted file mode 100644 index 074b7aed..00000000 --- a/modules/sideloader/images/astra-migration-toolkit.svg +++ /dev/null @@ -1,99 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/modules/sideloader/images/cql-console-create-identical-schema.png b/modules/sideloader/images/cql-console-create-identical-schema.png deleted file mode 100644 index bb14216e..00000000 Binary files a/modules/sideloader/images/cql-console-create-identical-schema.png and /dev/null differ diff --git a/modules/sideloader/images/data-importer-workflow.svg b/modules/sideloader/images/data-importer-workflow.svg deleted file mode 100644 index 98141dc0..00000000 --- a/modules/sideloader/images/data-importer-workflow.svg +++ /dev/null @@ -1,72 +0,0 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - diff --git a/modules/sideloader/pages/cleanup-sideloader.adoc b/modules/sideloader/pages/cleanup-sideloader.adoc deleted file mode 100644 index 4ed5ed40..00000000 --- a/modules/sideloader/pages/cleanup-sideloader.adoc +++ /dev/null @@ -1,92 +0,0 @@ -= Clean up {sstable-sideloader} migrations -:description: {sstable-sideloader} has an automatic cleanup process. - -{description} -You can also manually start or reschedule a cleanup. - -The cleanup process deletes all SSTable snapshots from the migration directory, revokes any unexpired upload credentials, and then closes the migration. - -Each migration ID has its own cleanup schedule, and the cleanup process deletes only the files and credentials associated with the specific migration ID that is being cleaned up. -Cleaning up one migration doesn't affect other migrations associated with the same database. - -== Idle timeout and automatic cleanup - -A migration becomes idle if it is _not_ in `Initializing` or `ImportInProgress` status. -If a migration remains continuously idle for one week, it hits the idle timeout and triggers the automatic migration cleanup process. - -A migration's idle timer starts when you initialize the migration, and it automatically restarts when you xref:sideloader:migrate-sideloader.adoc#import-data[import data]. - -The idle time _doesn't_ restart when you upload snapshots or take any other action besides importing data. - -You can override the idle timer by manually starting or scheduling a cleanup. -However, you cannot permanently prevent the cleanup process. - -[WARNING] -==== -{company} recommends that you <> if your migration could be idle for several days. -This includes time spent completely idle, as well as time required to upload snapshots or import data to the target database. - -For mutli-terabyte and cross-region migrations, it can take several days to upload snapshots or import data. -Make sure you <> to avoid automatic cleanup. -==== - -== Manually start a cleanup - -. Use the {devops-api} to immediately start the cleanup process for a migration: -+ -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/cleanup \ - | jq . ----- -+ -The cleanup process never runs on migrations in `ImportInProgress` status. -If the request fails due to `ImportInProgress`, you must either wait for the import process to end, xref:sideloader:stop-restart-sideloader.adoc#abort-migration[abort the migration], or <>. - -. Wait a few minutes, and then check the migration status: -+ -include::sideloader:partial$check-status.adoc[] -+ -While the cleanup is running, the migration status is `CleaningUpFiles`. -When complete, the migration status is `Closed`. - -[#reschedule-a-cleanup] -== Reschedule a cleanup - -[IMPORTANT] -==== -If you reschedule a cleanup, the cleanup timer doesn't reset when you import data. - -Keep a record of your rescheduled cleanups so you can reschedule them again, if necessary. - -For example, you might need to reschedule a cleanup if your migration needs more time. -Alternatively, if your migration is complete, you might reschedule the cleanup to minimize storage costs for the migration bucket. -==== - -You can use the {devops-api} to schedule a migration cleanup for a specific date and time: - -[source,bash,subs="+quotes"] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/cleanup \ - ?option.cleanupTime=**CLEANUP_TIME** \ - | jq . ----- - -Replace the following: - -* Set your `dbID` and `migrationID` environment variables according to the migration that you want to reschedule. -+ -This endpoint overrides the idle timeout for the specified migration ID only. - -* Replace `*CLEANUP_TIME*` with the date and time that you want the cleanup process to run. -You must use https://en.wikipedia.org/wiki/ISO_8601[ISO 8601] format (`YYYY-MM-DDTHH:MM:SSZ`), such as `option.cleanupTime=2025-03-31T14:30Z`. -+ -Setting a cleanup time in the past immediately starts the cleanup process. - -At your scheduled time, the cleanup process runs on the specified migration ID. -The cleanup process never runs on migrations in `ImportInProgress` status. -If the migration is in `ImportInProgress` at the scheduled cleanup time, the cleanup process will start when the migration's status changes. \ No newline at end of file diff --git a/modules/sideloader/pages/migrate-sideloader.adoc b/modules/sideloader/pages/migrate-sideloader.adoc deleted file mode 100644 index 26639ac4..00000000 --- a/modules/sideloader/pages/migrate-sideloader.adoc +++ /dev/null @@ -1,731 +0,0 @@ -= Migrate data with {sstable-sideloader} -:description: You can use {sstable-sideloader} to migrate data to {astra-db} from {cass-reg}, {dse}, or {hcd}. -:loop-var: pass:[${i}] - -{description} - -== Prerequisites - -Before you use {sstable-sideloader} for a migration, xref:sideloader:sideloader-overview.adoc[learn about the {sstable-sideloader} process] and xref:sideloader:prepare-sideloader.adoc[prepare your environments for {sstable-sideloader}]. - -[#create-snapshots] -== Create snapshots - -On _each node_ in your origin cluster, use `nodetool` to create a backup of the data that you want to migrate, including all keyspaces and CQL tables that you want to migrate. - -=== Prepare to create snapshots - -. Due to {sstable-sideloader} limitations related to materialized views, secondary indexes, and encrypted data, you might need to modify the data model on your origin cluster to prepare for the migration. -For more information, see xref:sideloader:prepare-sideloader.adoc#origin-cluster-requirements[Origin cluster requirements]. - -. Optional: Before you create snapshots, consider running `xref:dse:managing:tools/nodetool/cleanup.adoc[nodetool cleanup]` to remove data that no longer belongs to your nodes. -This command is particularly useful after adding more nodes to a cluster because it helps ensure that each node only contains the data that it is responsible for, according to the current cluster configuration and partitioning scheme. -+ -If you run `nodetool cleanup` before you take a snapshot, you can ensure that the snapshot only includes relevant data, potentially reducing the size of the snapshot. -Smaller snapshots can lead to lower overall migration times and lower network transfer costs. -+ -However, take adequate precautions before you run this command because the cleanup operations can introduce additional load on your origin cluster. - -=== Run nodetool snapshot - -Use `xref:dse:managing:tools/nodetool/snapshot.adoc[nodetool snapshot]` to create snapshots for the tables that you want to migrate. - -Don't create snapshots of system tables or tables that you don't want to migrate. -The migration can fail if you attempt to migrate snapshots that don't have a matching schema in the target database. -{sstable-sideloader} ignores system keyspaces. - -The structure of the `nodetool snapshot` command depends on the keyspaces and tables that you want to migrate. - -==== Snapshot all keyspaces - -Create a snapshot of all tables in all keyspaces: - -[source,bash,subs="+quotes"] ----- -nodetool snapshot -t *SNAPSHOT_NAME* ----- - -Replace the following: - -* *`SNAPSHOT_NAME`*: A descriptive name for the snapshot. -Use the same snapshot name for each node's snapshot; this makes it easier to programmatically upload the snapshots to the migration directory. - -==== Snapshot specific keyspaces - -Create a snapshot of all tables in one or more specified keyspaces: - -.Snapshot one keyspace -[source,bash,subs="+quotes"] ----- -nodetool snapshot -t *SNAPSHOT_NAME* *KEYSPACE_NAME* ----- - -.Snapshot multiple keyspaces -[source,bash,subs="+quotes"] ----- -nodetool snapshot -t *SNAPSHOT_NAME* *KEYSPACE_NAME_1* *KEYSPACE_NAME_2* ----- - -Replace the following: - -* *`SNAPSHOT_NAME`*: A descriptive name for the snapshot. -Use the same snapshot name for each node's snapshot; this makes it easier to programmatically upload the snapshots to the migration directory. - -* *`KEYSPACE_NAME`*: The name of the keyspace that you want to migrate. -+ -To snapshot multiple keyspaces, pass a space-separated list of keyspace names. -For example, `customer_data product_data purchase_history` specifies three keyspaces. - -==== Snapshot specific tables - -Create a snapshot of one or more specified tables: - -.Snapshot one table -[source,bash,subs="+quotes"] ----- -nodetool snapshot -kt *KEYSPACE_NAME*.*TABLE_NAME* -t *SNAPSHOT_NAME* ----- - -.Snapshot multiple tables -[source,bash,subs="+quotes"] ----- -nodetool snapshot -kt *KEYSPACE_NAME_1*.*TABLE_NAME_A* *KEYSPACE_NAME_1*.*TABLE_NAME_B* *KEYSPACE_NAME_2*.*TABLE_NAME_X* -t *SNAPSHOT_NAME* ----- - -Replace the following: - -* *`KEYSPACE_NAME.TABLE_NAME`*: The name of the table that you want to migrate and the keyspace that it belongs to, separated by a period. -For example, `product_data.appliances` specifies the `appliances` table in the `product_data` keyspace. -+ -To snapshot multiple tables, pass a space-separated list of keyspace-table pairs. -For example, `product_data.appliances purchase_history.nevada purchase_history.wisconsin` specifies the `appliances` table in the `product_data` keyspace and the `nevada` and `wisconsin` tables in the `purchase_history` keyspace. - -* *`SNAPSHOT_NAME`*: A descriptive name for the snapshot. -Use the same snapshot name for each node's snapshot; this makes it easier to programmatically upload the snapshots to the migration directory. - -=== Verify snapshot creation with nodetool listsnapshots - -Use `xref:6.9@dse:managing:tools/nodetool/list-snapshots.adoc[nodetool listsnapshots]` to verify that the snapshots were created: - -[source,bash] ----- -nodetool listsnapshots ----- - -[IMPORTANT] -==== -Snapshots have a specific directory structure, such as `*KEYSPACE_NAME*/*TABLE_NAME*/snapshots/*SNAPSHOT_NAME*/...`. -{sstable-sideloader} relies on this fixed structure to properly interpret the SSTable components. -**Don't modify the snapshot's directory structure; this can cause your migration to fail.** -==== - -=== Optional: Use `for` loops for snapshot creation and validation - -If the nodes in your origin cluster are named in a predictable way (for example, `dse0`, `dse1`, `dse2`, etc.), you can use a `for` loop to simplify snapshot creation. -For example: - -Use a `for` loop to snapshot all keyspaces:: -To snapshot all keyspaces on each node, append the `nodetool` command to your `for` loop: -+ -[source,bash,subs="+quotes"] ----- -for i in 0 1 2; do ssh dse${i} nodetool snapshot -t *SNAPSHOT_NAME*; done ----- - -Use a `for` loop to snapshot specific keyspaces:: -To snapshot one keyspace on each node, append the `nodetool` command to your `for` loop: -+ -[source,bash,subs="+quotes"] ----- -for i in 0 1 2; do ssh dse${i} nodetool snapshot -t *SNAPSHOT_NAME* *KEYSPACE_NAME*; done ----- -+ -To snapshot multiple specific keyspaces on each node, use commas (not spaces) to separate the keyspace names: -+ -[source,bash,subs="+quotes"] ----- -for i in 0 1 2; do ssh dse${i} nodetool snapshot -t *SNAPSHOT_NAME* *KEYSPACE_NAME_1*,*KEYSPACE_NAME_2*; done ----- - -Use a `for` loop to snapshot specific tables:: -To snapshot one table on each node, append the `nodetool` command to your `for` loop: -+ -[source,bash,subs="+quotes"] ----- -for i in 0 1 2; do ssh dse${i} nodetool snapshot -kt *KEYSPACE_NAME*.*TABLE_NAME* -t *SNAPSHOT_NAME*; done ----- -+ -To snapshot multiple specific tables on each node, use commas (not spaces) to separate the keyspace-table pairs: -+ -[source,bash,subs="+quotes"] ----- -for i in 0 1 2; do ssh dse${i} nodetool snapshot -kt *KEYSPACE_NAME_1*.*TABLE_NAME_A*,*KEYSPACE_NAME_1*.*TABLE_NAME_B* -t *SNAPSHOT_NAME*; done ----- - -You can use the same `for` loop structure to verify that each snapshot was successfully created: - -[source,bash] ----- -for i in 0 1 2; do ssh dse${i} nodetool listsnapshots; done ----- - -[#record-schema] -== Configure the target database - -To prepare your target database for the migration, you must record the schema for each table in your origin cluster that you want to migrate, re-create these schemas in your target database, and then set environment variables required to connect to your database. - -[WARNING] -==== -For the migration to succeed, your target database must meet the schema requirements described in this section. -Additionally, your snapshots must contain compatible data and directories, as described in xref:sideloader:prepare-sideloader.adoc#origin-cluster-requirements[Origin cluster requirements] and xref:sideloader:migrate-sideloader.adoc#create-snapshots[Create snapshots]. -For example, {astra-db} doesn't support materialized views, and {sstable-sideloader} cannot migrate encrypted data. - -However, indexes don't need to match. -You can define indexes in your target database independently from the origin cluster because {sstable-sideloader} ignores Storage Attached Indexes (SAI) defined on the origin cluster. -During the migration, {sstable-sideloader} automatically populates any SAI defined in your target database, even if those SAI weren't present in your origin cluster. -==== - -. Get the following schema properties for _each table_ that you want to migrate: -+ -* Exact keyspace name. -* Exact table name. -* Exact column names, data types, and the order in which they appear in the table creation DDL. -* Exact primary key definition as defined in your origin cluster, including the partition key, clustering columns, and ascending/descending ordering clauses. -You must define partition key columns and clustering columns in the exact order that they are defined on your origin cluster. -+ -To retrieve schema properties, you can run the `xref:astra@cql:cqlsh:cqlsh-commands/describe-keyspace.adoc[DESCRIBE KEYSPACE]` command on your origin cluster: -+ -[source,cql,subs="+quotes"] ----- -DESCRIBE *KEYSPACE_NAME*; ----- -+ -Replace *`KEYSPACE_NAME`* with the name of the keyspace that contains the tables you want to migrate, -such as `DESCRIBE smart_home;`. -+ -Then, get the schema properties from the result: -+ -[source,cql] ----- -CREATE TABLE smart_home.sensor_readings ( - device_id UUID, - room_id UUID, - reading_type TEXT, - reading_value DOUBLE, - reading_timestamp TIMESTAMP, - PRIMARY KEY (device_id, room_id, reading_timestamp) -) WITH CLUSTERING ORDER BY (room_id ASC, reading_timestamp DESC); ----- - -. Re-create the schemas in your target database: -+ -.. In the {astra-ui-link} navigation menu, click *Databases*, and then click the name of your {astra-db} database. -.. xref:astra-db-serverless:databases:manage-keyspaces.adoc#keyspaces[Create a keyspace] with the exact same name as your origin cluster's keyspace. -.. In your database's xref:astra-db-serverless:cql:develop-with-cql.adoc#connect-to-the-cql-shell[{cql-console}], create tables with the exact same names and schemas as your origin cluster. -+ -image::sideloader:cql-console-create-identical-schema.png[] -+ -{astra-db} rejects or ignores some table properties, such as compaction strategy. -See xref:astra-db-serverless:databases:database-limits.adoc[] for more information. - -. In your terminal, set environment variables for your target database: -+ -[source,bash,subs="+quotes"] ----- -export dbID=*DATABASE_ID* -export token=*APPLICATION_TOKEN* ----- -+ -Replace the following: -+ --- -* *`DATABASE_ID`*: The xref:astra-db-serverless:databases:create-database.adoc#get-db-id[database ID] of your target {astra-db} database. -* *`APPLICATION_TOKEN`*: An xref:astra-db-serverless:administration:manage-application-tokens.adoc[application token] with a role that has the required permissions for {sstable-sideloader}, which are {create-db-permission} and {view-db-permission}. -You can use a built-in role, such as the {database-administrator-role} role, or a xref:astra-db-serverless:administration:custom-roles.adoc[custom role] with the required permissions. --- -+ -[TIP] -==== -Later, you will add another environment variable for the migration ID. - -The curl commands in this guide assume that you have set environment variables for token, database ID, and migration ID. -Running the commands without these environment variables causes error messages like `Moved Permanently` and `404 page not found`. - -Additionally, the curl command use https://jqlang.github.io/jq/[jq] to format the JSON responses. -If you don't have jq installed, remove `| jq .` from the end of each command. -==== - -[#initialize-migration] -== Initialize the migration - -Use the {devops-api} to initialize the migration and get your migration directory path and credentials. - -To learn more about the initialization process, see xref:sideloader:sideloader-overview.adoc[About {sstable-sideloader}: Initialize a migration]. - -The initialization process can take several minutes to complete, especially if the migration bucket doesn't already exist. - -=== Get a migration ID - -. In your terminal, use the {devops-api} to initialize the data migration: -+ -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/initialize \ - | jq . ----- - -. Get the `migrationID` from the response: -+ -[source,json] ----- -{ - "migrationID": "272eac1d-df8e-4d1b-a7c6-71d5af232182", - "dbID": "b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d", - "status": "Initializing", - "progressInfo": "", - "uploadBucketDir": "", - "uploadCredentials": { - "name": "", - "keys": null, - "credentialExpiration": null - }, - "expectedCleanupTime": "2025-03-04T15:14:38Z" -} ----- -+ -The `migrationID` is a unique identifier (UUID) for the migration. -+ -The response also includes the migration `status`. -You will refer to this status multiple times throughout the migration process. - -. Assign the migration ID to an environment variable: -+ -[source,bash,subs="+quotes"] ----- -export migrationID=*MIGRATION_ID* ----- -+ -Replace *`MIGRATION_ID`* with the `migrationID` returned by the `initialize` endpoint. - -=== Check the migration status to verify initialization - -. Check the migration status: -+ -include::sideloader:partial$check-status.adoc[] - -. Check the `status` field in the response: -+ -* `"status": "ReceivingFiles"`: Initialization is complete and your upload credentials are available. -Proceed to the next step. -* `"status": "Initializing"`: The migration is still initializing. -Wait a few minutes before you check the status again. - -=== Get migration directory path and upload credentials - -Get your migration directory path and upload credentials from the response. -You need these values to xref:sideloader:migrate-sideloader.adoc#upload-snapshots-to-migration-directory[upload snapshots to the migration directory]. - -==== Get AWS credentials from MigrationStatus - -Securely store the `uploadBucketDir`, `accessKeyID`, `secretAccessKey`, and `sessionToken` from the response: - -[source,json] ----- -{ - "migrationID": "272eac1d-df8e-4d1b-a7c6-71d5af232182", - "dbID": "b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d", - "status": "ReceivingFiles", - "progressInfo": "", - "uploadBucketDir": "s3://ds-mig-b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d/272eac1d-df8e-4d1b-a7c6-71d5af232182/sstables/", - "uploadCredentials": { - "name": "sessionToken", - "keys": { - "accessKeyID": "ASXXXXXXXXXXXXXXXXXX", - "secretAccessKey": "2XXXXXXXXXXXXXXXWqcdV519ZubYbyfuNxbZg1Rw", - "sessionToken": "XXXXXXXXXX" - }, - "credentialExpiration": "2024-01-18T19:45:09Z", - "hint": "\nexport AWS_ACCESS_KEY_ID=ASXXXXXXXXXXXXXXXXXX\nexport AWS_SECRET_ACCESS_KEY=2XXXXXXXXXXXXXXXWqcdV519ZubYbyfuNxbZg1Rw\nexport AWS_SESSION_TOKEN=XXXXXXXXXXXXXX\n" - }, - "expectedCleanupTime": "2024-01-25T15:14:38Z" -} ----- - -`uploadBucketDir` is the migration directory URL. -Note the trailing slash. - -`uploadCredentials` contains the AWS credentials that authorize uploads to the migration directory, namely `accessKeyID`, `secretAccessKey`, and `sessionToken`. - -[IMPORTANT] -==== -The `sessionToken` expires after one hour. -If your total migration takes longer than one hour, xref:sideloader:troubleshoot-sideloader.adoc#get-new-upload-credentials[generate new credentials], and then xref:sideloader:stop-restart-sideloader.adoc[resume the migration] with the fresh credentials. - -If you use automation to handle {sstable-sideloader} migrations, you might need to script a xref:sideloader:stop-restart-sideloader.adoc[pause] every hour so you can generate new credentials without unexpectedly interrupting the migration. -==== - -==== Get Google Cloud credentials from MigrationStatus - -. Find the `uploadBucketDir` and the `uploadCredentials` in the response: -+ -[source,json] ----- -{ - "migrationID": "272eac1d-df8e-4d1b-a7c6-71d5af232182", - "dbID": "b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d", - "status": "ReceivingFiles", - "progressInfo": "", - "uploadBucketDir": "gs://ds-mig-b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d/272eac1d-df8e-4d1b-a7c6-71d5af232182/sstables/", - "uploadCredentials": { - "name": "TYPE_GOOGLE_CREDENTIALS_FILE", - "keys": { - "file": "CREDENTIALS_FILE" - }, - "credentialExpiration": "2024-08-07T18:51:39Z" - }, - "expectedCleanupTime": "2024-08-14T15:14:38Z" -} ----- -+ -`uploadBucketDir` is the migration directory URL. -Note the trailing slash. -+ -`uploadCredentials` contains a base64-encoded file containing Google Cloud credentials that authorize uploads to the migration directory. - -. Pipe the Google Cloud credentials `file` to a `creds.json` file: -+ -[source,bash] ----- -curl -X GET \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID} \ - | jq -r '.uploadCredentials.keys.file' \ - | base64 -d > creds.json ----- - -. Securely store the `uploadBucketDir` and `creds.json`. - -==== Get Azure credentials from MigrationStatus - -Securely store the `uploadBucketDir` and `urlSignature` from the response: - -[source,json] ----- -{ - "migrationID": "456ca4a9-0551-46c4-b8bb-90fcd136a0c3", - "dbID": "ccefd141-8fda-4e4d-a746-a102a96657bc", - "status": "ReceivingFiles", - "progressInfo": "", - "uploadBucketDir": "https://muztx5cqmp3jhe3j2guebksz.blob.core.windows.net/mig-upload-456ca4a9-0551-46c4-b8bb-90fcd136a0c3/sstables/", - "uploadCredentials": { - "name": "URL signature", - "keys": { - "url": "https://UPLOAD_BUCKET_DIR/?si=AZURE_SAS_TOKEN", - "urlSignature": "si=AZURE_SAS_TOKEN" - }, - "credentialExpiration": "2025-04-02T15:14:31Z" - }, - "expectedCleanupTime": "2025-03-04T15:14:38Z" -} ----- - -`uploadBucketDir` is the migration directory URL. -Note the trailing slash. - -`uploadCredentials` contains `url` and `urlSignature` keys that represent an https://learn.microsoft.com/en-us/azure/ai-services/translator/document-translation/how-to-guides/create-sas-tokens[Azure Shared Access Signature (SAS) token]. -You need the `urlSignature` to upload snapshots to the migration directory. -In the preceding example, these strings are truncated for readability. - -[#upload-snapshots-to-migration-directory] -== Upload snapshots to the migration directory - -//TODO: ENV VARS: A variable for MIGRATION_DIR would simplify these steps slightly. Env vars for all the values except the ones that change each time (Node name, snapshot name) would be most efficient. - -Use your cloud provider's CLI and your upload credentials to upload snapshots for _each origin node_ into the migration directory. - -[IMPORTANT] -==== -Be aware of the following requirements for the upload commands: - -* You must include the asterisk (`*`) character as shown in the commands, otherwise the commands won't work properly. - -* With the exception of the leading `://` in the migration directory path, your paths must _not_ include double slashes (`//`). - -* Use the CLI that corresponds with your target database's cloud provider. -For more information, see xref:sideloader:prepare-sideloader.adoc[]. - -* These commands assume that you installed the cloud provider's CLI on the nodes in your origin cluster. -For more information, see xref:sideloader:prepare-sideloader.adoc[]. - -* You might need to modify these commands depending on your environment, node names, directory structures, and other variables. -==== - -=== Upload snapshots to AWS - -. Set environment variables for the AWS credentials that were generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration]: -+ -[source,bash,subs="+quotes"] ----- -export AWS_ACCESS_KEY_ID=**ACCESS_KEY_ID** -export AWS_SECRET_ACCESS_KEY=**SECRET_ACCESS_KEY** -export AWS_SESSION_TOKEN=**SESSION_TOKEN** ----- - -. Use the AWS CLI to upload one snapshot from one node into the migration directory: -+ -[source,bash,subs="+quotes,attributes"] ----- -du -sh **CASSANDRA_DATA_DIR**/**KEYSPACE_NAME**/{asterisk}/snapshots/{asterisk}**SNAPSHOT_NAME**{asterisk}; \ -aws s3 sync --only-show-errors --exclude '{asterisk}' --include '{asterisk}/snapshots/**SNAPSHOT_NAME**{asterisk}' **CASSANDRA_DATA_DIR**/ **MIGRATION_DIR****NODE_NAME** ----- -+ -Replace the following: -+ --- -include::sideloader:partial$command-placeholders-common.adoc[] --- -+ -.Example: Upload a snapshot with AWS CLI -[source,bash] ----- -# Set environment variables -export AWS_ACCESS_KEY_ID=XXXXXXXX -export AWS_SECRET_ACCESS_KEY=XXXXXXXXXX -export AWS_SESSION_TOKEN=XXXXXXXXXX - -# Upload "sensor_readings" snapshot from "dse0" node -du -sh /var/lib/cassandra/data/smart_home/*/snapshots/*sensor_readings*; \ -aws s3 sync --only-show-errors --exclude '*' --include '*/snapshots/sensor_readings*' /var/lib/cassandra/data/ s3://ds-mig-b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d/272eac1d-df8e-4d1b-a7c6-71d5af232182/sstables/dse0 ----- - -. Monitor upload progress: -+ -.. Use the AWS CLI to get a list of cloud storage keys for the files that have been successfully uploaded to the migration directory: -+ -[source,bash,subs="+quotes"] ----- -aws s3 ls --human-readable --summarize --recursive *MIGRATION_DIR* ----- -+ -Replace *`MIGRATION_DIR`* with the `uploadBucketDir` that was generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration]. -+ -.. Compare the returned list against the files in your snapshot directory. -When the lists match, the upload is complete. -+ -You can _potentially_ increase upload speeds by adjusting the `max_concurrent_requests`, `multipart_threshold`, and `multipart_chunksize` parameters in your https://docs.aws.amazon.com/cli/latest/topic/s3-config.html[AWS CLI S3 configuration]. -However, upload time primarily depends on the snapshot size, network throughput from your origin cluster to the migration bucket, and whether the origin cluster and migration bucket are in the same region. - -. Repeat the upload process for each snapshot (*`SNAPSHOT_NAME`*) and node (*`NODE_NAME`*) in your origin cluster. -+ -If your credentials expire, see xref:sideloader:troubleshoot-sideloader.adoc#get-new-upload-credentials[Get new upload credentials]. -+ -.Use a `for` loop to simplify snapshot uploads -[TIP] -==== -If the nodes in your origin cluster have predictable names (for example, `dse0`, `dse1`, and `dse2`), then you can use a `for` loop to streamline the execution of the upload commands. -For example: - -[source,bash,subs="+quotes,attributes"] ----- -# Set environment variables -export AWS_ACCESS_KEY_ID=**ACCESS_KEY_ID** -export AWS_SECRET_ACCESS_KEY=**SECRET_ACCESS_KEY** -export AWS_SESSION_TOKEN=**SESSION_TOKEN** - -# Loop over the sync command for all nodes -for i in 0 1 2; do ssh dse{loop-var} \ -"du -sh **CASSANDRA_DATA_DIR**/**KEYSPACE_NAME**/{asterisk}/snapshots/{asterisk}**SNAPSHOT_NAME**{asterisk}; \ -aws s3 sync --only-show-errors --exclude '{asterisk}' --include '{asterisk}/snapshots/**SNAPSHOT_NAME**{asterisk}' **CASSANDRA_DATA_DIR**/ **MIGRATION_DIR**dse{loop-var}" & done ----- -==== - -include::sideloader:partial$staged-snapshots-need-import-ph.adoc[] - -include::sideloader:partial$idle-migration-directories-note.adoc[] - -=== Upload snapshots to Google Cloud Storage - -. Authenticate to Google Cloud with the `creds.json` file that you created when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration]: -+ -[source,bash,subs="+quotes,attributes"] ----- -gcloud auth activate-service-account --key-file=creds.json ----- -+ -If necessary, modify the `--key-file` path to match the location of your `creds.json` file, such as `--key-file=~/.gcloud_credentials/creds.json`. -+ -You can also use `gcloud auth login --cred-file creds.json`. - -. Use `gsutil` to upload one snapshot from one node into the migration directory: -+ -[source,bash,subs="+quotes,attributes"] ----- -gsutil -m rsync -r -d **CASSANDRA_DATA_DIR**/**KEYSPACE_NAME**/{asterisk}{asterisk}/snapshots/**SNAPSHOT_NAME**/ **MIGRATION_DIR****NODE_NAME**/ ----- -+ -Replace the following: -+ --- -include::sideloader:partial$command-placeholders-common.adoc[] --- -+ -.Example: Upload a snapshot with gcloud and gsutil -[source,bash,subs="attributes"] ----- -# Authenticate -gcloud auth activate-service-account --key-file=creds.json - -# Upload "sensor_readings" snapshot from "dse0" node -gsutil -m rsync -r -d /var/lib/cassandra/data/smart_home/{asterisk}{asterisk}/snapshots/sensor_readings/ gs://ds-mig-b7e7761f-6f7f-4116-81a5-e8eefcf0cc1d/272eac1d-df8e-4d1b-a7c6-71d5af232182/sstables/dse0 ----- - -. Monitor upload progress: -+ -.. Use `gsutil` to get a list of objects that have been successfully uploaded to the migration directory: -+ -[source,bash,subs="+quotes"] ----- -gsutil ls -r *MIGRATION_DIR* ----- -+ -Replace *`MIGRATION_DIR`* with the `uploadBucketDir` that was generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration]. -+ -.. Compare the returned list against the files in your snapshot directory. -When the lists match, the upload is complete. -+ -The `https://cloud.google.com/storage/docs/gsutil/commands/rsync#description[-m]` flag in `gsutil -m rsync` enables parallel synchronization, which can improve upload speed. -However, upload time primarily depends on the snapshot size, network throughput from your origin cluster to the migration bucket, and whether the origin cluster and migration bucket are in the same region. - -. Repeat the upload process for each snapshot (*`SNAPSHOT_NAME`*) and node (*`NODE_NAME`*) in your origin cluster. -+ -.Use a `for` loop to simplify snapshot uploads -[TIP] -==== -If the nodes in your origin cluster have predictable names (for example, `dse0`, `dse1`, and `dse2`), then you can use a `for` loop to streamline the execution of the `gsutil rsync` commands. -For example: - -[source,bash,subs="+quotes,attributes"] ----- -for i in 0 1 2; do ssh dse{loop-var} \ -du -sh **CASSANDRA_DATA_DIR**/**KEYSPACE_NAME**/{asterisk}/snapshots/{asterisk}**SNAPSHOT_NAME**{asterisk}; \ -gsutil -m rsync -r -d **CASSANDRA_DATA_DIR**/**KEYSPACE_NAME**/{asterisk}{asterisk}/snapshots/**SNAPSHOT_NAME**/ **MIGRATION_DIR**dse{loop-var} & done ----- -==== - -include::sideloader:partial$staged-snapshots-need-import-ph.adoc[] - -include::sideloader:partial$idle-migration-directories-note.adoc[] - -=== Upload snapshots to Azure - -. Set environment variables for the following values: -+ --- -* *`AZURE_SAS_TOKEN`*: The `urlSignature` key that was generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration]. -* *`CASSANDRA_DATA_DIR`*: The absolute file system path to where {cass-short} data is stored on the node, including the trailing slash. -For example, `/var/lib/cassandra/data/`. -* *`SNAPSHOT_NAME`*: The name of the xref:sideloader:migrate-sideloader.adoc#create-snapshots[snapshot backup] that you created with `nodetool snapshot`. -* *`MIGRATION_DIR`*: The entire `uploadBucketDir` value that was generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration], including the trailing slash. -* *`NODE_NAME`*: The host name of the node that your snapshots are from. -It is important to use the specific node name to ensure that each node has a unique directory in the migration bucket. --- -+ -[source,bash,subs="+quotes"] ----- -export AZURE_SAS_TOKEN="**AZURE_CREDENTIALS_URL**" -export CASSANDRA_DATA_DIR="**CASSANDRA_DATA_DIR**" -export SNAPSHOT_NAME="**SNAPSHOT_NAME**" -export MIGRATION_DIR="**MIGRATION_DIR**" -export NODE_NAME="**NODE_NAME**" ----- - -. Use the Azure CLI to upload one snapshot from one node into the migration directory: -+ -[source,bash] ----- -for dir in $(find "$CASSANDRA_DATA_DIR" -type d -path "*/snapshots/${SNAPSHOT_NAME}*"); do - REL_PATH="${dir#"$CASSANDRA_DATA_DIR"}" # Remove the base path - DEST_PATH="${MIGRATION_DIR}${NODE_NAME}/${REL_PATH}/?${AZURE_SAS_TOKEN}" - - azcopy sync "$dir" "$DEST_PATH" --recursive -done ----- - -. Monitor upload progress: -+ -.. Use the Azure CLI to get the curent contents of the migration directory: -+ -[source,bash] ----- -azcopy list ${MIGRATION_DIR}?${AZURE_SAS_TOKEN} ----- -+ -.. Compare the returned list against the files in your snapshot directory. -When the lists match, the upload is complete. -+ -Upload time primarily depends on the snapshot size, network throughput from your origin cluster to the migration bucket, and whether the origin cluster and migration bucket are in the same region. - -. Repeat the upload process for each snapshot and node in your origin cluster. -Be sure to change the `SNAPSHOT_NAME` and `NODE_NAME` environment variables as needed. - -include::sideloader:partial$staged-snapshots-need-import-ph.adoc[] - -include::sideloader:partial$idle-migration-directories-note.adoc[] - -[#import-data] -== Import data - -After you completely upload snapshots for each origin node, import the data into your target database. - -Data import is a multi-step operation that requires complete success. -If one step fails, then the entire import operation stops and the migration fails. -//Does all data fail to import or is it possible to have a partial import? - -To learn more about the data import process, see xref:sideloader:sideloader-overview.adoc[About {sstable-sideloader}: Import data]. - -[WARNING] -==== -* Before you start the import process, make sure all snapshots are completely uploaded. -For commands to monitor upload progress and compare uploaded data against the original snapshots, see xref:sideloader:migrate-sideloader.adoc#upload-snapshots-to-migration-directory[Upload snapshots to the migration directory]. - -* If necessary, you can xref:sideloader:stop-restart-sideloader.adoc[pause or abort the migration] during the import process. -include::sideloader:partial$no-return.adoc[] -==== - -. Use the {devops-api} to launch the data import: -+ -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/launch \ - | jq . ----- -+ -Although this call returns immediately, the import process takes time. - -. Check the migration status periodically: -+ -include::sideloader:partial$check-status.adoc[] - -. Check the `status` field in the response: -+ -* `"status": "ImportInProgress"`: The data is still being imported. -Wait a few minutes before you check the status again. -* `"status": "MigrationDone"`: The import is complete, and you can proceed to <>. - -. If the migration takes more than a few days, xref:sideloader:cleanup-sideloader.adoc#reschedule-a-cleanup[manually reschedule the cleanup] to avoid automatic cleanup. - -. If the migration fails, see xref:sideloader:troubleshoot-sideloader.adoc[]. - -[#validate-the-migrated-data] -== Validate the migrated data - -include::sideloader:partial$validate.adoc[] - -== See also - -* xref:sideloader:cleanup-sideloader.adoc[] -* xref:sideloader:troubleshoot-sideloader.adoc[] \ No newline at end of file diff --git a/modules/sideloader/pages/prepare-sideloader.adoc b/modules/sideloader/pages/prepare-sideloader.adoc deleted file mode 100644 index f06a37fb..00000000 --- a/modules/sideloader/pages/prepare-sideloader.adoc +++ /dev/null @@ -1,304 +0,0 @@ -= Prepare to use {sstable-sideloader} -:description: Before you use {sstable-sideloader}, review the requirements and prepare your target database, origin cluster, and administration server. - -{description} - -Due to the nature of the {sstable-sideloader} process and the tools involved, you need to be familiar with using the command line, including the following: - -* Installing and using CLI tools -* Issuing curl commands -* Basic scripting -* Modifying example commands to fit your environment -* Security best practices - -[IMPORTANT] -==== -The {sstable-sideloader} process uses authentication credentials to write to the migration directory and your database. - -Make sure you understand how to securely store and use sensitive credentials when working on the command line. -==== - -== Target {astra-db} database requirements - -The following requirements, recommendations, and limitations apply to the target {astra-db} database. -Review all of these to ensure that your database is compatible with {sstable-sideloader}. - -=== {astra} plan requirement - -Your {astra} organization must be on an xref:astra-db-serverless:administration:subscription-plans.adoc[*Enterprise* plan]. - -{sstable-sideloader} is a premium feature that incurs costs based on usage. -This includes the total amount (GB) of data processed as part of the {sstable-sideloader} workload, and the amount of data stored in the migration bucket is metered at the standard {astra-db} storage rate. - -[TIP] -==== -Migration directories are automatically cleaned up after one week of idle time. - -To minimize costs, you can xref:sideloader:cleanup-sideloader.adoc[manually clean up migration directories] when you no longer need them. -==== - -=== Database type requirement - -Your target database must be an {astra-db} Serverless database. -{sstable-sideloader} isn't compatible with {db-classic} databases. - -If you haven't done so already, xref:astra-db-serverless:databases:create-database.adoc[create an {astra-db} database]. -{sstable-sideloader} supports both {db-serverless} and {db-serverless-vector} databases. -Both types support CQL tables. -However, you must create a {db-serverless-vector} database if you want to use the xref:astra-db-serverless:api-reference:dataapiclient.adoc[{data-api}] or xref:astra-db-serverless:databases:manage-collections.adoc[dynamic-schema collections]. - -To call the {sstable-sideloader} endpoints, you need the xref:astra-db-serverless:databases:create-database.adoc#get-db-id[database ID] and an xref:astra-db-serverless:administration:manage-application-tokens.adoc[{astra} application token] with a sufficiently privileged role. -The required permissions for {sstable-sideloader} are {create-db-permission} and {view-db-permission}. -You can use a built-in role, such as the {database-administrator-role} role, or a xref:astra-db-serverless:administration:custom-roles.adoc[custom role] with the required permissions. - -=== PCU group requirement - -Your target database must be in a xref:astra-db-serverless:administration:provisioned-capacity-units.adoc[Provisioned Capacity Unit (PCU) group]. -You can use either a flexible capacity PCU group or a committed capacity PCU group, depending on your long-term needs and other PCU group usage. - -==== Use a flexible capacity PCU group (Recommended) - -Because {sstable-sideloader} operations are typically short-term, resource-intensive events, you can xref:astra-db-serverless:administration:create-pcu.adoc#flexible-capacity[create a flexible capacity PCU group] exclusively to support your target database during the migration. - -After the migration, you can move your target database out of the flexible capacity PCU group, and then park or delete the group. -Don't park the PCU group during the {sstable-sideloader} process because databases in a parked PCU group are hibernated and unavailable for use. - -{company} recommends the following configurations for flexible capacity PCU groups for {sstable-sideloader} migrations: - -* **Recommended configuration if the target database is a {db-serverless} database**: -+ -** **Minimum capacity**: One or more, depending on the scale of the migration. -** **Maximum capacity**: Greater than the minimum by several units to allow autoscaling during resource intensive stages of the migration. -+ -For non-trivial migrations, consider setting the maximum to 10. -For extremely large migrations, contact your {company} account representative or {support-url}[IBM Support] to request more than 10 units to support your migration. - -* **Recommended configuration if the target database is a {db-serverless-vector} database**: -+ -By default, {db-serverless-vector} databases can have no more than one unit per PCU group. -For any non-trivial migration, contact your {company} account representative or {support-url}[IBM Support] for assistance configuring a PCU group for your target {db-serverless-vector} database. - -==== Use a committed capacity PCU group - -For a long-term PCU group option, you can use a xref:astra-db-serverless:administration:create-pcu.adoc#committed-capacity[committed capacity PCU group] for your target database. -This could be your database's permanent PCU group assignment, or it could be a long-lived PCU group that you use for many migrations over time, adding and removing databases from the group as needed. - -[IMPORTANT] -==== -The {sstable-sideloader} process can be extremely resource intensive. -If there are any other databases in the same PCU group, the migration process can affect their performance due to resource contention. - -To avoid interfering with other databases in the same PCU group, {company} recommends isolating the database during the migration using either a single-database committed capacity PCU group or a flexible capacity PCU group. -==== - -{company} recommends the following configurations for committed capacity PCU groups for {sstable-sideloader} migrations: - -* **Recommended configuration if the target database is a {db-serverless} database**: -+ --- -** **Reserved capacity**: One or more, depending on the PCU group's normal, long-term workload requirements. -+ -This is the amount of long-term capacity that you want the group to have after the migration is complete. - -** **Minimum capacity**: Equal to or greater than the reserved capacity. -+ -If the minimum is greater than the reserved capacity, the surplus capacity is prepared in advance, and there is no autoscaling required to access that capacity. - -** **Maximum capacity**: Greater than the minimum by several units to allow autoscaling during resource intensive stages of the migration. -+ -For non-trivial migrations, consider setting the maximum to 10. -For extremely large migrations, contact your {company} account representative or {support-url}[IBM Support] to request more than 10 units to support your migration. --- -+ -After the migration, you can reduce the minimum and maximum capacity down to the levels required for normal database operations. - -* **Recommended configuration if the target database is a {db-serverless-vector} database**: -+ -By default, {db-serverless-vector} databases can have no more than one unit per PCU group. -For any non-trivial migration, contact your {company} account representative or {support-url}[IBM Support] for assistance configuring a PCU group for your target {db-serverless-vector} database. - -[#origin-cluster-requirements] -== Origin cluster requirements - -The following requirements, recommendations, and limitations apply to origin clusters. -Review all of these to ensure that your cluster is compatible with {sstable-sideloader}. - -=== Cluster infrastructure - -* Your origin cluster can be hosted on premises or on any cloud provider. - -* Your origin cluster must run a supported database version: -+ -** {dse-short} 5.1 or later -** {hcd-short} 1.1 or later -** {cass-reg} 3.11 through 4.x -+ -{cass-short} 5.0 tables aren't supported unless the cluster is configured to explicitly use the {cass-short} 4.x storage format (`storage_compatibility_mode: CASSANDRA_4`). - -* Your origin cluster must use the default https://cassandra.apache.org/doc/stable/cassandra/configuration/cass_yaml_file.html#partitioner[partitioner], `Murmur3Partitioner`. -+ -Older partitioners, such as `RandomPartitioner`, `ByteOrderedPartitioner`, and `OrderPreservingPartitioner`, are not supported. - -=== Cloud provider CLI - -To upload snapshots directly from the origin cluster, you must install your cloud provider's CLI on each node in the origin cluster. - -The tool you install depends on the region where your target {astra-db} database is deployed: - -* AWS: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html[Install AWS CLI] -* Google Cloud: https://cloud.google.com/sdk/docs/install-sdk[Install gcloud] and https://cloud.google.com/storage/docs/gsutil_install[install gsutil] -* Microsoft Azure: https://learn.microsoft.com/en-us/cli/azure/install-azure-cli[Install Azure CLI] - -Alternatively, you can upload copies of the snapshots from a separate staging server that has the CLI installed, and you must coordinate this through the administration server. -However, this process _isn't_ covered in this guide. -The CLI commands in this guide assume you have installed your cloud provider's CLI on the nodes in the origin cluster. -If you choose the alternative option, you must modify the commands accordingly for your environment. - -=== Incompatible data - -* *{astra-db} doesn't support materialized views*: You must replace these with SAI or an alternative data model design. - -* *{sstable-sideloader} doesn't support encrypted data*: If your origin cluster uses xref:6.9@dse:securing:transparent-data-encryption.adoc[{dse-short} Transparent Data Encryption], be aware that {sstable-sideloader} cannot migrate these SSTables. -+ -If you have a mix of encrypted and unencrypted data, you can use {sstable-sideloader} to migrate the unencrypted data. -After the initial migration, you can use another strategy to move the encrypted data, such as {cass-migrator-repo-url}[{cass-migrator}] or a manual export and reupload. - -* *{sstable-sideloader} doesn't support secondary indexes*: If you don't remove or replace these in your origin cluster, {sstable-sideloader} ignores these directories when importing the data to your {astra-db} database. - -== Administration server requirements - -You need a server where you can run the {sstable-sideloader} commands. - -Your administration server must have SSH access to each node in your origin cluster. - -{company} recommends that you install the following additional software on your administration server: - -* {cass-migrator-repo-url}[{cass-migrator-short}] to validate imported data and, with {zdm-proxy}, reconcile it with the origin cluster. -* https://jqlang.github.io/jq/[jq] to format JSON responses from the {astra} {devops-api}. -The {devops-api} commands in this guide use this tool. - -[#additional-preparation-for-specific-migration-scenarios] -== Additional preparation for specific migration scenarios - -The following information can help you prepare for specific migration scenarios, including multi-region migrations and multiple migrations to the same database. - -[#minimum-migration-scope] -=== Minimum migration scope - -To minimize data reconciliation issues, the recommended minimum migration scope is one CQL table across all nodes. - -This means that a single migration process, from start to finish, should encapsulate the data for one CQL table as it exists on all of your origin nodes. -For example, if you are migrating one table, you need to upload snapshots of all SSTables from all nodes for that CQL table. - -Avoid breaking one table into multiple migrations because migrating a subset of SSTables for one CQL table will likely result in data loss, corruption, or resurrection of previously deleted data. - -Each migration is performed separately, and each migration has no awareness of prior migrations. -This means that data from later migrations can be incorrectly applied to the table. -For example, if your first migration includes tombstones, that data could be resurrected if it is present in a subsequent migration from another node. - -In contrast, if you use a single large migration to migrate all SSTables for a CQL table across all nodes, {astra-db} can reconcile the data across all nodes, ensuring that your migration is accurate and complete. - -=== Multi-region migrations - -include::ROOT:partial$multi-region-migrations.adoc[] - -=== Multi-node migrations - -You can migrate data from any number of nodes in your origin cluster to the same target database or multiple target databases. - -When you xref:sideloader:migrate-sideloader.adoc[migrate data with {sstable-sideloader}], there is no difference in the core process when migrating from one node or multiple nodes. -The following steps summarize the process and considerations for migrating multiple nodes. - -==== Migrate multiple nodes to one database - -. On your origin cluster, make sure your data is valid and ready to migrate, as explained in <>. - -. From your origin cluster, create snapshots for all of the nodes that you want to migrate. -+ -Run `nodetool snapshot` as many times as necessary to capture all of your nodes. -+ -For important warnings about multi-node migrations, see <>. - -. On your target database, replicate the schemas for all tables that you want to migrate. -+ -This is critical for a successful migration. -If the schemas don't match, the migration fails. -+ -You don't need to make any changes based on the number of nodes, as long as the keyspaces and table schemas are replicated in the target database. - -. Initialize the migration to prompt {sstable-sideloader} to create a migration bucket for your target database. - -. Upload all of your node snapshots to the migration bucket. - -. Use {sstable-sideloader} to import the data to your target database. -+ -{sstable-sideloader} imports snapshots from the migration bucket to your target database based on the matching schemas. -The number of node snapshots that you uploaded to the migration bucket doesn't determine the success of the import. -The success of the import depends primarily on the validity of the schemas and the data in the snapshots. - -. After the import, validate the migrated data to ensure that it matches the data in the origin cluster. -For example, you can {cass-migrator-repo-url}?tab=readme-ov-file#steps-for-data-validation[run {cass-migrator-short} in validation mode]. - -==== Migrate multiple nodes to multiple databases - -Orchestrating concurrent migrations from multiple nodes to multiple target databases can be complex. - -Consider focusing on one target database at a time, or create a migration plan to track origin nodes, target databases, migration bucket credentials, and timelines for each migration. - -. On your origin cluster, make sure your data is valid and ready to migrate, as explained in <>. - -. From your origin cluster, create snapshots for all of the nodes that you want to migrate. -+ -Run `nodetool snapshot` as many times as necessary to capture all of your nodes. -+ -For important warnings about multi-node migrations, see <>. - -. On each of your target databases, replicate the schemas for the tables that you want to migrate to each database. -+ -This is critical for a successful migration. -If the schemas don't match, the migration fails. -+ -You don't need to make any changes based on the number of nodes, as long as the keyspaces and table schemas are replicated in the target databases. -+ -If you want to migrate the same data to multiple databases, you must re-create the schemas in each of those databases. -{sstable-sideloader} requires a schema to be present in the target database in order to migrate data. - -. For each target database, initialize a migration to prompt {sstable-sideloader} to create migration buckets for each database. -+ -At minimum, you must initialize one migration for each database. - -. Upload the node snapshots to their corresponding migration buckets. - -. Use {sstable-sideloader} to import the data to your target databases. -+ -You can import data to multiple databases at once, but each import event must be triggered separately using the unique migration ID. -+ -{sstable-sideloader} imports snapshots from the migration bucket to your target database based on the matching schemas. -The number of node snapshots that you uploaded to the migration bucket doesn't determine the success of the import. -The success of the import depends primarily on the validity of the schemas and the data in the snapshots.\ - -. After the import, validate the migrated data to ensure that it matches the data in the origin cluster. -For example, you can {cass-migrator-repo-url}?tab=readme-ov-file#steps-for-data-validation[run {cass-migrator-short} in validation mode]. - -=== Multiple migrations to the same database - -When you initialize a migration with {sstable-sideloader}, a unique migration ID is generated for that specific migration workflow. -For each migration ID, there is a unique migration directory and migration directory credentials. - -If you initialize multiple migrations for the same database, you generate multiple migration IDs, each with its own migration directory and credentials. - -This can be useful for breaking large migrations into smaller batches. -For example, if you have 100 snapshots, you could initialize 10 migrations, and then upload 10 different snapshots to each migration directory. -However, don't break one CQL table into multiple migrations, as explained in <>. - -You can upload snapshots to multiple migration directories at once. -However, when you reach the import phase of the migration, {sstable-sideloader} can import from only one migration directory at a time per database. -For example, if you have 10 migration IDs for the same database, you must run 10 separate import actions. -Each import must completely finish before starting the next import. - -After all of the imports are complete, validate the migrated data in your target database to ensure that it matches the data in the origin cluster. -For example, you can {cass-migrator-repo-url}?tab=readme-ov-file#steps-for-data-validation[run {cass-migrator-short} in validation mode]. - -== Next steps - -* xref:sideloader:migrate-sideloader.adoc[] \ No newline at end of file diff --git a/modules/sideloader/pages/sideloader-overview.adoc b/modules/sideloader/pages/sideloader-overview.adoc deleted file mode 100644 index 491b5ea4..00000000 --- a/modules/sideloader/pages/sideloader-overview.adoc +++ /dev/null @@ -1,184 +0,0 @@ -= About {sstable-sideloader} -:page-aliases: data-importer:data-importer-overview.adoc, astra-db-serverless:sideloader:sideloader-overview.adoc -:description: {sstable-sideloader} lets you migrate data from an {cass-reg} or {dse} cluster into {astra-db} without impacting the origin cluster or your {astra-db} Serverless database. - -{sstable-sideloader} is a service running in {astra-db} that directly imports data from snapshot backups that you've uploaded to {astra-db} from an existing {cass-reg}, {dse}, or {hcd} cluster. - -Because it imports data directly, {sstable-sideloader} can offer several advantages over CQL-based tools like xref:dsbulk:overview:dsbulk-about.adoc[{dsbulk}] and {cass-migrator-repo-url}[{cass-migrator}], including faster, more cost-effective data loading, and minimal performance impacts on your origin cluster and target database. - -== {sstable-sideloader} concepts - -Origin, origin cluster:: -In the context of {sstable-sideloader}, this refers to your existing {cass-short}, {dse-short}, or {hcd-short} cluster. - -Target, target database:: -In the context of {sstable-sideloader}, this refers to the {astra-db} Serverless database where you will migrate your data. - -Administration server:: -A server where you run the migration commands, including CLI commands and {astra} {devops-api} calls. -It must have SSH access to each node in your origin cluster. - -Migration:: -A workflow that you initiate within {sstable-sideloader} that encompasses the lifecycle of uploading and importing snapshot backups of a specific set of keyspaces or CQL tables. -+ -This process produces artifacts and parameters including migration buckets, migration IDs, migration directories, and upload credentials. -You use these components throughout the migration workflow. - -[#sideloader-process] -== The {sstable-sideloader} process - -Transferring data with {sstable-sideloader} is a multi-phase process. -Before you use {sstable-sideloader}, learn about the events, outcomes, warnings, and requirements of each phase: - -=== Prepare your infrastructure - -There are requirements for using {sstable-sideloader} that you must consider before you start a migration. -Additionally, you must take steps to prepare your target database, origin cluster, and administration server before you begin the migration. - -For more information, see xref:sideloader:prepare-sideloader.adoc[]. - -=== Create snapshot backups - -{sstable-sideloader} uses snapshot backup files to import SSTable data from your existing origin cluster. -Each snapshot for each node in the origin cluster must include all the keyspaces and individual CQL tables that you want to migrate. - -These snapshots are ideal for database migrations because creating snapshots has a negligible performance impact on the origin cluster, and the snapshots preserve metadata like `writetime` and `ttl` values. - -When using {sstable-sideloader} with {zdm-proxy}, {cass-short}'s last-write-wins semantics ensure that new, real-time writes accurately take precedence over historical writes. -Last-write-wins compares the `writetime` of conflicting records, and then retains the most recent write. -For example, if a new write occurs in your target database with a `writetime` of `2023-10-01T12:05:00Z`, and then {sstable-sideloader} migrates a record against the same row with a `writetime` of `2023-10-01T12:00:00Z`, the target database retains the data from the new write because it has the most recent `writetime`. - -For more information, see xref:sideloader:migrate-sideloader.adoc#create-snapshots[Migrate data with {sstable-sideloader}: Create snapshots]. - -=== Prepare the target database - -Because snapshots don't store schema definitions, you must pre-configure the schema definition in your target {astra-db} database so that it matches the origin cluster's schema. - -For the migration to succeed, the schema in your target database must align with the schema in the origin cluster. -However, you might need to modify your schema or data model to be compatible with {astra-db}. - -For specific requirements and more information, see xref:sideloader:migrate-sideloader.adoc#record-schema[Migrate data with {sstable-sideloader}: Configure the target database]. - -[#initialize-a-migration] -=== Initialize a migration - -After you create snapshots on the origin cluster and pre-configure the schema on the target database, use the {astra} {devops-api} to initialize the migration. - -.{sstable-sideloader} moves data from the migration bucket to {astra-db}. -svg::sideloader:data-importer-workflow.svg[] - -When you initialize a migration, {sstable-sideloader} does the following: - -. Creates a secure migration bucket. -+ -The migration bucket is only created during the first initialization. -All subsequent migrations use different directories in the same migration bucket. -+ -{company} owns the migration bucket, and it is located within the {astra} perimeter. - -. Generates a migration ID that is unique to the new migration. - -. Creates a migration directory within the migration bucket that is unique to the new migration. -+ -The migration directory is also referred to as the `uploadBucketDir`. -In the next phase of the migration process, you will upload your snapshots to this migration directory. - -. Generates upload credentials that grant read/write access to the migration directory. -+ -The credentials are formatted according to the cloud provider where your target database is deployed. - -For instructions and more information, see xref:sideloader:migrate-sideloader.adoc#initialize-migration[Migrate data with {sstable-sideloader}: Initialize the migration]. - -=== Upload snapshots - -When initialization is complete, use your cloud provider's CLI to xref:sideloader:migrate-sideloader.adoc#upload-snapshots-to-migration-directory[upload your snapshots to the migration directory]. - -To upload snapshots directly from the origin cluster, you must install your cloud provider's CLI on each node in the origin cluster. -While it is possible to orchestrate this process through a staging server, the commands given in this documentation assume you are uploading snapshots directly from the origin cluster. - -The time required to upload the snapshots depends on the size of your dataset and the network throughput between the origin cluster and the migration bucket: - -[cols="10,30,60"] -|=== -|Speed |Migration type |Description - -|Fastest -|Inter-datacenter -|All else equal, snapshots take the least time to upload when the origin cluster is in the same cloud provider and region as the target database. - -|Fast -|Cross-datacenter, co-located -|Uploads are slower by default when they must exit the local datacenter. -The delay increases relative to the physical distance between the datacenters. - -For example, all else equal, uploading from AWS `us-east-1` (Dulles, VA, USA) to AWS `ca-central-1` (Montréal, QC, Canada) is faster than uploading from `us-east-1` to `us-west-2` (The Dalles, OR, USA) because Oregon is significantly further from Virginia than Montréal. - -|Variable -|Cross-provider, co-located -|If the target database is in a different cloud provider than the origin cluster, the upload can be slower as the data passes from one provider's infrastructure to another. - -This is considered a cross-datacenter transfer, and the delay increases relative to the physical distance between the datacenters. - -|Slowest -|Transoceanic -|The slowest uploads happen when the data must travel over transoceanic cables. -If the data must also change cloud providers, there can be additional delays. - -In this case, consider creating your target database in a co-located datacenter, and then xref:astra-db-serverless:databases:manage-regions.adoc[deploy your database to other regions] after the migration. -|=== - -[#import-data] -=== Import data - -After uploading the snapshots to the migration directory, use the {devops-api} to start the data import process. - -During the import process, {sstable-sideloader} does the following: - -. Revokes access to the migration directory. -+ -You cannot read or write to the migration directory after starting the data import process. - -. Discovers all uploaded SSTables in the migration directory, and then groups them into approximately same-sized subsets. - -. Runs validation checks on each subset. - -. Converts all SSTables of each subset. - -. Disables new compactions on the target database. -+ -[WARNING] -==== -This is the last point at which you can xref:sideloader:stop-restart-sideloader.adoc#abort-migration[abort the migration]. - -Once {sstable-sideloader} begins to import SSTable metadata (the next step), you cannot stop the migration. -==== - -. Imports metadata from each SSTable. -+ -If the dataset contains tombstones, any read operations on the target database can return inconsistent results during this step. -Since compaction is disabled, there is no risk of permanent inconsistencies. -However, in the context of xref:ROOT:introduction.adoc[{zdm}], it's important that the {zdm-short} proxy continues to read from the origin cluster. - -. Re-enables compactions on the {astra-db} Serverless database. - -Each step must finish successfully. -If one step fails, the import operation stops and no data is imported into your target database. - -If all steps finish successfully, the migration is complete and you can access the imported data in your target database. - -For instructions and more information, see xref:sideloader:migrate-sideloader.adoc#import-data[Migrate data with {sstable-sideloader}: Import data] - -=== Validate imported data - -include::sideloader:partial$validate.adoc[] - -== Use {sstable-sideloader} with {zdm-proxy} - -If you need to migrate a live database, you can use {sstable-sideloader} instead of {dsbulk-short} or {cass-migrator-short} during of xref:ROOT:migrate-and-validate-data.adoc[Phase 2 of {zdm}]. - -.Use {sstable-sideloader} with {zdm-proxy} -svg::sideloader:astra-migration-toolkit.svg[] - -== Next steps - -* xref:sideloader:prepare-sideloader.adoc[] \ No newline at end of file diff --git a/modules/sideloader/pages/stop-restart-sideloader.adoc b/modules/sideloader/pages/stop-restart-sideloader.adoc deleted file mode 100644 index e4625b00..00000000 --- a/modules/sideloader/pages/stop-restart-sideloader.adoc +++ /dev/null @@ -1,66 +0,0 @@ -= Stop or restart an {sstable-sideloader} migration -:description: If necessary, you can pause, cancel, or restart an {sstable-sideloader} migration. - -{description} - -== Pause a migration - -Use the {devops-api} to pause a migration: - -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/pause \ - | jq . ----- - -A paused migration retains its current state and progress. - -Any in-progress jobs will complete, but no new jobs will start. - -=== Resume a migration - -Resume a previously paused migration from the point at which it was paused: - -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/resume \ - | jq . ----- - -You can only resume an active migration that has been paused. -Running this command against migrations in other statuses, such as idle migrations that were automatically cleaned up, has no effect. - -[#abort-migration] -== Abort a migration - -Abort a migration only if you want to abandon it completely. - -. Abort a migration and remove all migration progress: -+ -[source,bash] ----- -curl -X POST \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID}/abort \ - | jq . ----- -+ -include::sideloader:partial$no-return.adoc[] -For more information about what happens during each phase of a migration and the point of no return, see xref:sideloader:sideloader-overview.adoc[]. - -. Wait a few minutes, and then check the migration status to confirm that the migration stopped: -+ -include::sideloader:partial$check-status.adoc[] - -== Retry a failed migration - -For information about reattempting a failed migration, see xref:sideloader:troubleshoot-sideloader.adoc[]. - -== See also - -* xref:sideloader:cleanup-sideloader.adoc[] -* xref:sideloader:troubleshoot-sideloader.adoc[] \ No newline at end of file diff --git a/modules/sideloader/pages/troubleshoot-sideloader.adoc b/modules/sideloader/pages/troubleshoot-sideloader.adoc deleted file mode 100644 index 1b3ff5fd..00000000 --- a/modules/sideloader/pages/troubleshoot-sideloader.adoc +++ /dev/null @@ -1,100 +0,0 @@ -= Troubleshoot {sstable-sideloader} -:description: Get help with {sstable-sideloader} - -Learn how to troubleshoot common {sstable-sideloader} issues. - -== Migration status is outdated - -You can use the {devops-api} to check the migration status at any time. -However, it can take a few minutes for the {devops-api} to reflect status changes during a migration. -Immediately calling the {devops-api} after starting a new phase of the migration might not return the actual current status. - -[#get-new-upload-credentials] -== Get new upload credentials - -//TODO: Does checking the migration status always generate new creds or only if they are expired? - -If your credentials expire, do the following: - -. Use the `MigrationStatus` endpoint to generate new credentials: -+ -include::sideloader:partial$check-status.adoc[] - -. Continue the migration with the fresh credentials. -+ -If you set environment variables for your credentials, be sure to update those values. - -== Retry a failed migration - -If a migration fails, there are two ways that you can reattempt the migration. -The option you use depends on the type of error that occurred. - -If you are able to resolve the cause of the failure without modifying the migration directory contents, you can relaunch the migration using the data already present in the migration directory. -Otherwise, you must abandon the failed migration and restart the entire migration process from the beginning. - -The two most common errors are as follows: - -* *Schema discrepancies*: There is a mismatch between the origin and target schemas. -To resolve this error, you can <>. - -* *Invalid data in migration directory*: The data uploaded to the migration directory is invalid or improperly formatted. -Common causes include data corruption, incomplete upload due to a timeout, malformed file paths, and the presence of invalid data. -+ -When this type of failure occurs, you must abandon the failed migration and restart the entire migration process. -For more information, see <>. - -[#relaunch] -=== Relaunch a failed migration - -. Check the migration status for an error message related to the failure: -+ -include::sideloader:partial$check-status.adoc[] - -. If possible, resolve the issue described in the error message. -+ -For example, if there is a problem with the schema in the target database, make sure that your schemas align, as described in xref:sideloader:migrate-sideloader.adoc#record-schema[Configure the target database]. - -. Repeat the `launch` command that you used to xref:sideloader:migrate-sideloader.adoc#import-data[import the data], and continue the migration process from there. - -If the migration fails again, see <>. - -//// -Future: -=== Reset failed migration endpoint -TODO: Add to this page and stop-restart page. - -"resetting" a failed/aborted migration: -- Call a new endpoint called reset, which removes all the metadata and restores their write and read access to the migration directory. -- By doing this they accept that the progress will be wiped and that they will be charged for the failed attempt in accordance with our pricing rules. -- Amend the data in the migration directory as needed. -- Call relaunch to re-execute the migration from scratch. -//// - -[#restart] -=== Restart a failed migration - -When a migration fails due to a problem with the data uploaded to the migration directory, you must completely restart the migration. - -This is because you cannot change the data in the migration directory after you upload it. -For example, if your snapshots contain corrupt data, you have to restart the migration with new snapshots and a new migration directory. - -. Review the xref:sideloader:prepare-sideloader.adoc#origin-cluster-requirements[origin cluster requirements] to ensure that your snapshot doesn't contain invalid data, including materialized views and encrypted data. - -. If necessary, xref:sideloader:migrate-sideloader.adoc#create-snapshots[create new snapshots] of any invalid snapshots. -+ -If your snapshots don't appear to contain invalid data, continue to the next step. - -. If necessary, xref:sideloader:migrate-sideloader.adoc#record-schema[reprepare the target database]. -There are two reasons you might need to do this: -+ -** The origin and target schemas don't match. -** The migration reached a point that some data was loaded to the target database. -This is unlikely, but, if this happens, you must xref:astra-db-serverless:databases:manage-collections.adoc#delete-a-table-in-the-astra-portal[drop the table] from your target database, and then re-create the table in the target database. -+ -In this case, if the migration _didn't_ fail due to a problem with the snapshot data, you can potentially reuse the existing snapshots for the new migration. - -. Repeat the remainder of the migration process from xref:sideloader:migrate-sideloader.adoc#initialize-migration[Initialize the migration]. -+ -This starts a fresh migration with a new migration directory, a migration ID, and upload credentials. - -. If the migration fails again and you are unable to determine the cause of the failure, contact {support-url}[IBM Support]. \ No newline at end of file diff --git a/modules/sideloader/partials/check-status.adoc b/modules/sideloader/partials/check-status.adoc deleted file mode 100644 index 74428f06..00000000 --- a/modules/sideloader/partials/check-status.adoc +++ /dev/null @@ -1,11 +0,0 @@ -[source,bash] ----- -curl -X GET \ - -H "Authorization: Bearer ${token}" \ - https://api.astra.datastax.com/v2/databases/${dbID}/migrations/${migrationID} \ - | jq . ----- -+ -A successful response contains a `MigrationStatus` object. -It can take a few minutes for the {devops-api} to reflect status changes during a migration. -Immediately calling this endpoint after starting a new phase of the migration might not return the actual current status. \ No newline at end of file diff --git a/modules/sideloader/partials/command-placeholders-common.adoc b/modules/sideloader/partials/command-placeholders-common.adoc deleted file mode 100644 index 69ccfdd2..00000000 --- a/modules/sideloader/partials/command-placeholders-common.adoc +++ /dev/null @@ -1,7 +0,0 @@ -* *`CASSANDRA_DATA_DIR`*: The absolute file system path to where {cass-short} data is stored on the node. -For example, `/var/lib/cassandra/data`. -* *`KEYSPACE_NAME`*: The name of the keyspace that contains the tables you want to migrate. -* *`SNAPSHOT_NAME`*: The name of the xref:sideloader:migrate-sideloader.adoc#create-snapshots[snapshot backup] that you created with `nodetool snapshot`. -* *`MIGRATION_DIR`*: The entire `uploadBucketDir` value that was generated when you xref:sideloader:migrate-sideloader.adoc#initialize-migration[initialized the migration], including the trailing slash. -* *`NODE_NAME`*: The host name of the node that your snapshots are from. -It is important to use the specific node name to ensure that each node has a unique directory in the migration bucket. \ No newline at end of file diff --git a/modules/sideloader/partials/idle-migration-directories-note.adoc b/modules/sideloader/partials/idle-migration-directories-note.adoc deleted file mode 100644 index bd9983b5..00000000 --- a/modules/sideloader/partials/idle-migration-directories-note.adoc +++ /dev/null @@ -1,10 +0,0 @@ -.Idle migration directories are evicted -[WARNING] -==== -As an added security measure, migrations that remain continuously idle for one week are subject to xref:sideloader:cleanup-sideloader.adoc[automatic cleanup], which deletes all associated snapshots, revokes any unexpired upload credentials, and then closes the migration. - -{company} recommends that you xref:sideloader:cleanup-sideloader.adoc#reschedule-a-cleanup[manually reschedule the cleanup] if you don't plan to launch the migration within one week or if you need several days to upload snapshots or import data. - -For large migrations, it can take several days to upload snapshots and import data. -Make sure you xref:sideloader:cleanup-sideloader.adoc#reschedule-a-cleanup[manually reschedule the cleanup] to avoid automatic cleanup. -==== \ No newline at end of file diff --git a/modules/sideloader/partials/no-return.adoc b/modules/sideloader/partials/no-return.adoc deleted file mode 100644 index 77631bc0..00000000 --- a/modules/sideloader/partials/no-return.adoc +++ /dev/null @@ -1,2 +0,0 @@ -You can abort a migration up until the point at which {sstable-sideloader} starts importing SSTable metadata. -After this point, you must wait for the migration to finish, and then you can use the {cql-shell} (`cqlsh`) to drop the keyspace/table in your target database before repeating the entire migration procedure. \ No newline at end of file diff --git a/modules/sideloader/partials/staged-snapshots-need-import-ph.adoc b/modules/sideloader/partials/staged-snapshots-need-import-ph.adoc deleted file mode 100644 index f66d47df..00000000 --- a/modules/sideloader/partials/staged-snapshots-need-import-ph.adoc +++ /dev/null @@ -1,2 +0,0 @@ -Uploaded snapshots are staged in the migration directory, but the data is not yet written to the target database. -After uploading snapshots, you must xref:sideloader:migrate-sideloader.adoc#import-data[import the data] to finish the migration. \ No newline at end of file diff --git a/modules/sideloader/partials/validate.adoc b/modules/sideloader/partials/validate.adoc deleted file mode 100644 index df391ab1..00000000 --- a/modules/sideloader/partials/validate.adoc +++ /dev/null @@ -1,4 +0,0 @@ -After the migration is complete, you can query the migrated data using the xref:astra-db-serverless:cql:develop-with-cql.adoc#connect-to-the-cql-shell[{cql-shell}] (`cqlsh`) or xref:astra-db-serverless:api-reference:row-methods/find-many.adoc[{data-api}]. - -You can {cass-migrator-repo-url}?tab=readme-ov-file#steps-for-data-validation[run {cass-migrator-short} in validation mode] for more thorough validation. -{cass-migrator-short} also offers an AutoCorrect mode to reconcile any differences that it detects. \ No newline at end of file