Mysql 9.7 support

[svetasmirnova]

• The contributed code is licensed under GPL v2.0
• Contributor Licence Agreement (CLA) is signed
• util/update-modules has been ran
• Documentation updated
• Test suite update

[Copilot]

Pull request overview

Adds MySQL 9.7 compatibility across Percona Toolkit by updating sandbox/test infrastructure and adapting checksum/replication/explain-handling logic and fixtures to MySQL 9.7 behavior changes.

Changes:

• Update checksum/hash and replication-related logic/tests for MySQL 9.7 (SHA2 usage, replication status sourcing, binlog_format handling).
• Adjust EXPLAIN usage/output expectations (TRADITIONAL formatting) and collection scripts to keep parsing stable.
• Add/extend MySQL 9.7 sandbox configuration and introduce/refresh sample outputs/fixtures for 9.7.

Reviewed changes

Copilot reviewed 108 out of 122 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
t/pt-variable-advisor/advice.t	Wipe-clean sandbox at end of test.
t/pt-table-sync/wait.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/triggers.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/traces.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/sync_using_channels.t	Adjust channel/GTID setup behavior for 9.7; tweak DSN passed to main().
t/pt-table-sync/ssl.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/safety_checks.t	Replace manual replication reset with sandbox restart.
t/pt-table-sync/issue_627.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/issue_560.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/filters.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/bugs.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-sync/basics.t	Add replica2 usage + binlog_format save/restore around test.
t/pt-table-checksum/throttle.t	Add binlog_format save/restore around test.
t/pt-table-checksum/samples/static-chunk-size-results-9.7.txt	New expected sample output for MySQL 9.7.
t/pt-table-checksum/samples/default-results-9.7.txt	New expected sample output for MySQL 9.7.
t/pt-table-checksum/run_time.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/resume.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/replication_filters.t	Ensure binlog_format STATEMENT after sandbox restarts; add replica binlog_format handling.
t/pt-table-checksum/pt-226.t	Add replica2 + binlog_format save/restore; adjust setup.
t/pt-table-checksum/pt-2250.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/pt-225.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/pt-2052.t	Add replica2 + binlog_format save/restore; adapt to 9.7.
t/pt-table-checksum/pt-136.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/pt-1059.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/progress.t	Add binlog_format save/restore around test.
t/pt-table-checksum/privs.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/plugin.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/issue_602.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/issue_388.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/issue_1651002.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/issue_1592608.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/issue_1485195.t	Add replica2 + binlog_format save/restore; adjust extra table accounting.
t/pt-table-checksum/ignore_columns.t	Add replica2 + binlog_format save/restore; target replica1 explicitly.
t/pt-table-checksum/error_handling.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/create_replicate_table.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/chunk_size.t	Add replica2 + binlog_format save/restore.
t/pt-table-checksum/chunk_index.t	Add replica1/2 + binlog_format save/restore.
t/pt-table-checksum/char_chunking.t	Add replica1/2 + binlog_format save/restore.
t/pt-table-checksum/bugs.t	Add binlog_format save/restore; ensure replica binlog_format after restart.
t/pt-table-checksum/basics.t	Add binlog_format save/restore around test.
t/pt-stalk/samples/ps.sql	New sample for prepared statement / sleep.
t/pt-stalk/pt-stalk.t	Adjust retention-size test diagnostics; adapt channel setup for 9.7.
t/pt-stalk/collections.t	New test validating pt-stalk collection outputs.
t/pt-show-grants/pt-2190.t	Relax HEX password matching to work across versions.
t/pt-replica-restart/until.t	Adapt error retrieval for 9.7 via performance_schema; adjust waits/until-pos logic.
t/pt-replica-restart/ssl.t	Adapt error retrieval for 9.7 via performance_schema.
t/pt-replica-restart/pt-slave-restart.t	Skip deprecated pt-slave-restart on MySQL 9.7 (non-MariaDB).
t/pt-replica-restart/pt-replica-restart.t	Skip/adjust error-text tests for 9.7; update recurse path and error-number usage.
t/pt-replica-restart/gtid.t	Adapt error retrieval for 9.7 via performance_schema.
t/pt-online-schema-change/slave_lag.t	Replace RESET usage with CHANGE ... _DELAY=0 for replica delay handling.
t/pt-online-schema-change/samples/stats-execute-9.7.txt	New expected output sample for 9.7.
t/pt-online-schema-change/samples/pt-1455_reset_replica.sql	Remove RESET REPLICA from sample script.
t/pt-online-schema-change/samples/pt-1455_replica.sql	Remove RESET REPLICA from sample script.
t/pt-online-schema-change/pt-2355.t	Replace RESET usage with CHANGE ... _DELAY=0 for delay handling.
t/pt-online-schema-change/pt-229.t	Add replica DBH + binlog_format save/restore; ensure replicas caught up after load.
t/pt-online-schema-change/pt-2241.t	Replace RESET usage with CHANGE ... _DELAY=0 for delay handling.
t/pt-online-schema-change/pt-2168.t	Replace RESET usage with CHANGE ... _DELAY=0 for delay handling.
t/pt-online-schema-change/pt-2168-cancel.t	Replace RESET usage with CHANGE ... _DELAY=0 for delay handling.
t/pt-online-schema-change/pt-1760.t	Use channel-specific RESET for newer versions; keep older reset path conditional.
t/pt-online-schema-change/pt-1717-resume.t	Replace RESET usage with CHANGE ... _DELAY=0 for delay handling.
t/pt-online-schema-change/pt-1455.t	Remove source RESET usage tied to RESET REPLICA removal.
t/pt-online-schema-change/basics.t	Add 9.7-specific stats expected output selection logic.
t/pt-mysql-summary/roles.t	New test for roles reporting in pt-mysql-summary output.
t/pt-fk-error-logger/ssl.t	For 9.7 enable innodb_native_foreign_keys via cnf include + restart, then restore.
t/pt-fk-error-logger/basics.t	Same as ssl.t: cnf include + restart for 9.7, then restore.
t/pt-config-diff/basics.t	Add diag() on failure for easier debugging.
t/pt-archiver/standard_options.t	Wait for replicas after loading fixture data.
t/pt-archiver/replica_lag.t	Replace RESET usage with CHANGE ... _DELAY=0 cleanup.
t/pt-archiver/archive_using_channels.t	Skip gtid_on.sql loading on 9.7+ in channel setup path.
t/lib/TableSyncer.t	Skip SHA1-dependent tests on 9.7; add SHA2 variants/expectations.
t/lib/TableSyncChunk.t	Switch checksum function to SHA2 on 9.7; add/skip expectations accordingly.
t/lib/TableChunker.t	Skip zero-date interval chunking test on 9.7.
t/lib/TableChecksum.t	Add SHA2 expectations; skip MD5 expectations on 9.7; extend coverage for widths/types.
t/lib/samples/SchemaIterator/mysql-user-ddl-9.7.txt	New MySQL 9.7 DDL fixture.
t/lib/RowChecksum.t	Add SHA2 expectations; skip MD5 expectations on 9.7; verify widths/types.
t/lib/Percona/Toolkit.t	Allow tool versions like x.y.z-1; improve diagnostics on mismatch.
t/lib/OptionParser.t	Update expected DSN parsing structure (new key).
t/lib/MySQLConfig.t	Relax MySQL version regex to accept 9.x.
t/lib/MasterSlave.t	Adjust skip text and 9.7 channel setup; add diagnostics for failure.
t/lib/KeySize.t	Force TRADITIONAL EXPLAIN format in test query expectations.
t/lib/IndexLength.t	Add VersionParser import (now flagged as unused by review).
t/lib/HTTP/Micro.t	Improve diagnostics when Micro/Tiny content mismatch.
t/lib/bash/report_mysql_info.sh	Extend plan; handle uutils sort/collation differences; add 9.7 InnoDB formatting sample test.
sandbox/start-sandbox	Allow explicit GTID=0 behavior; adjust gtid_mode config generation.
sandbox/servers/9.7/my.sandbox.cnf	New sandbox defaults for MySQL 9.7.
sandbox/replica_channels.sql	Remove GTID-mode toggling statements from channel setup script.
sandbox/replica_channels_t.sql	Remove GTID-mode toggling statements from channel setup script.
lib/TableUsage.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/TableSyncNibble.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/TableParser.pm	Use TRADITIONAL EXPLAIN format in possible-key detection.
lib/TableChunker.pm	Switch to TRADITIONAL EXPLAIN and SHA2-related adjustments; char chunking map creation change.
lib/TableChecksum.pm	Add SHA2 support and parameterization; update hash selection/width calculations.
lib/Sandbox.pm	Use SHA2 for wait_for_replicas ping; ignore mysql.gtid_executed in verify_test_data.
lib/RowChecksum.pm	Add SHA2 support and parameterization for row checksums.
lib/QueryReportFormatter.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/OobNibbleIterator.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/NibbleIterator.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/KeySize.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/IndexLength.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/HTTP/Micro.pm	Update copyright year range.
lib/ExplainAnalyzer.pm	Use TRADITIONAL EXPLAIN format for 9.7 compatibility.
lib/bash/report_mysql_info.sh	Add redo-log-capacity fallback; fix roles query output mode; includes a comment typo (flagged).
lib/bash/collect.sh	Switch to mysql -E mode instead of embedding \G; adjust retention-size du parsing for uutils.
lib/bash/collect_mysql_info.sh	Switch roles query to non-\G output + -E mode.
bin/pt-table-usage	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-table-sync	Propagate TRADITIONAL EXPLAIN + SHA2/char-chunking changes to packaged script.
bin/pt-table-checksum	Propagate TRADITIONAL EXPLAIN + SHA2 changes to packaged script.
bin/pt-stalk	Propagate collection output formatting changes (+ -E mode) to packaged script.
bin/pt-replica-restart	Adjust GTID/worker-thread handling and docs for 9.7 assumptions.
bin/pt-query-digest	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-online-schema-change	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-mysql-summary	Adjust roles query output formatting and redo-log-capacity fallback.
bin/pt-kill	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-index-usage	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-heartbeat	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-find	Propagate TRADITIONAL EXPLAIN changes to packaged script.
bin/pt-duplicate-key-checker	Propagate TRADITIONAL EXPLAIN changes to packaged script.

Comments suppressed due to low confidence (2)

t/lib/IndexLength.t:22

• use VersionParser; is imported but never referenced in this test file. This adds noise and can hide real unused-import issues.
  lib/OobNibbleIterator.pm:103
• Same as the previous EXPLAIN: /*! FORMAT=TRADITIONAL */ is executable and not version-gated. To align with other changes in this PR, gate it with 90700.

      my $explain_past_upper_sql
         = "EXPLAIN /*! FORMAT=TRADITIONAL */ SELECT "
         . ($args{past_select}
            || join(', ', map { $q->quote($_) } @{$self->{sql}->{columns}}))
         . " FROM "  . $self->{sql}->{from}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 108 out of 122 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

lib/QueryReportFormatter.pm:1330

• $sth->execute() is called twice: once inside the eval/if branch and again unconditionally. This runs EXPLAIN twice and makes error handling harder to reason about. Remove the unconditional second execute call.

             $self->{no_partitions} = 1;
             $sth = $dbh->prepare("EXPLAIN /*!90700 FORMAT=TRADITIONAL */ $query");
             $sth->execute();
         }
         $sth->execute();
         my $i = 1;

lib/TableParser.pm:283

• This uses an unversioned /*! ... */ comment for FORMAT=TRADITIONAL, which may be treated as executable SQL on older MySQL versions and can break the documented MySQL 5.0+ support. Make it a versioned comment so only MySQL 9.7+ sees FORMAT=TRADITIONAL.

   my $sql = 'EXPLAIN /*! FORMAT=TRADITIONAL */ SELECT * FROM ' . $quoter->quote($database, $table)
      . ' WHERE ' . $where;

lib/Sandbox.pm:352

• wait_for_replicas() now uses SHA2(RAND(), 256) to generate the ping token. Percona Toolkit documents support for MySQL 5.0+, but SHA2() is not available on older MySQL versions, so this can break sandbox/test synchronization. Use a version-agnostic function like UUID() for the ping value.

   my $source_dbh = $self->get_dbh_for($args{source} || 'source');
   my $replica2_dbh = $self->get_dbh_for($args{replica}  || 'replica2');
   my ($ping) = $source_dbh->selectrow_array("SELECT SHA2(RAND(), 256)");
   $source_dbh->do("UPDATE percona_test.sentinel SET ping='$ping' WHERE id=1 /* wait_for_replicas */");

lib/TableChunker.pm:607

• Temporarily enabling $dbh->{AutoCommit} should be exception-safe and should not leave AutoCommit altered if CREATE TEMPORARY TABLE fails. The current code restores AutoCommit only on the success path. Wrapping the CREATE in a small scope with local AutoCommit ensures it is always restored.

      # We need to enable autocommit here, so CREATE TEMPORARY TABLE
      # does not fail when GTIDS are enabled
      my $old_autocommit = $dbh->{AutoCommit};
      if ( !$dbh->{AutoCommit} ) {
         $dbh->{AutoCommit} = 1;

t/lib/Percona/Toolkit.t:29

• The version regex only allows a single digit after the dash (e.g. 3.1.0-1). Tool versions like 3.1.0-12 won't be captured correctly, causing false failures. Use -\d+ for the suffix.

[Copilot]

Pull request overview

Copilot reviewed 110 out of 124 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (2)

sandbox/start-sandbox:126

• GTID is compared with -eq, but some callers pass non-numeric values (e.g. GTID=ON_PERMISSIVE). This will raise a shell error ("integer expression expected") and break sandbox startup.
  t/pt-replica-restart/until.t:278
• In the MySQL 9.7 branch, $mysqlbinlog can remain undef if it isn't found, and $until_pos_from_relay can remain undef if the relay-log pattern isn't matched. With use warnings FATAL => 'all', this will turn into a fatal warning or confusing shell error.