-
-
Notifications
You must be signed in to change notification settings - Fork 1.9k
MDEV-31956 SSD based InnoDB buffer pool extension #4510
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 11.8
Are you sure you want to change the base?
Changes from all commits
1948000
64eb035
3ac1611
c78f5ac
3eedbcf
4918d15
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| --innodb-buffer-pool-size=21M --innodb-extended-buffer-pool-size=1M | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,169 @@ | ||
| --source include/have_innodb.inc | ||
| --source include/have_debug.inc | ||
| --source include/have_debug_sync.inc | ||
|
Comment on lines
+1
to
+3
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Could we have a non-debug version of this test as well?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, we can't. We have to inject debug code to have ability to test it. |
||
| --source include/count_sessions.inc | ||
| --source ../encryption/include/have_file_key_management_plugin.inc | ||
| #--source include/innodb_page_size.inc | ||
|
|
||
| --let $encrypted_row_compressed=6 | ||
| --let $unencrypted_row_compressed=5 | ||
| --let $unencrypted_uncompressed=4 | ||
| --let $encrypted_uncompressed=3 | ||
| --let $unencrypted_page_compressed=2 | ||
| --let $encrypted_page_compressed=1 | ||
| --let $i = $encrypted_row_compressed | ||
|
|
||
| --let $page_size=`SELECT @@GLOBAL.innodb_page_size` | ||
| if ($page_size != 16384) { | ||
| --let $i=$unencrypted_uncompressed | ||
| } | ||
|
|
||
| --connect (prevent_purge,localhost,root) | ||
| START TRANSACTION WITH CONSISTENT SNAPSHOT; | ||
|
|
||
| --connection default | ||
|
|
||
| --let $DATADIR = `select @@datadir` | ||
| # Set ext buffer pool file size and check it was created, check it's size | ||
| --list_files $DATADIR ext_buffer_pool | ||
|
|
||
| --disable_query_log | ||
| --error 0,ER_UNKNOWN_SYSTEM_VARIABLE | ||
| SET @old_innodb_limit_optimistic_insert_debug = @@innodb_limit_optimistic_insert_debug; | ||
| SET @old_debug_dbug = @@debug_dbug; | ||
| --enable_query_log | ||
|
|
||
| --error 0,ER_UNKNOWN_SYSTEM_VARIABLE | ||
| SET GLOBAL innodb_limit_optimistic_insert_debug = 3; | ||
| SET GLOBAL DEBUG_DBUG='+d,ib_ext_bp_count_io_only_for_t'; | ||
|
|
||
| while($i) { | ||
|
|
||
| SET GLOBAL DEBUG_DBUG='+d,ib_ext_bp_disable_LRU_eviction_for_t'; | ||
| if ($i == $unencrypted_uncompressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for unencrypted uncompressed table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0; | ||
| } | ||
| if ($i == $encrypted_uncompressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for encrypted uncompressed table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0 encrypted=yes encryption_key_id=1; | ||
| } | ||
| if ($i == $unencrypted_page_compressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for unencrypted PAGE_COMPRESSED=1 table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0 PAGE_COMPRESSED=1; | ||
| } | ||
| if ($i == $unencrypted_row_compressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for unencrypted ROW_FORMAT=COMPRESSED table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0 ROW_FORMAT=COMPRESSED KEY_BLOCK_SIZE=1; | ||
| } | ||
| if ($i == $encrypted_page_compressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for encrypted PAGE_COMPRESSED=1 table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0 PAGE_COMPRESSED=1 encrypted=yes | ||
| encryption_key_id=1; | ||
| } | ||
| if ($i == $encrypted_row_compressed) { | ||
| --echo ################################################################### | ||
| --echo # Testing for encrypted ROW_FORMAT=COMPRESSED table # | ||
| --echo ################################################################### | ||
| CREATE TABLE t ( | ||
| `a` INT NOT NULL, | ||
| PRIMARY KEY (`a`) | ||
| ) ENGINE=InnoDB STATS_PERSISTENT=0 ROW_FORMAT=COMPRESSED encrypted=yes | ||
| encryption_key_id=1; | ||
| } | ||
|
|
||
| SELECT variable_value INTO @prev_flushed_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_PAGES_FLUSHED'; | ||
| SELECT NUMBER_PAGES_WRITTEN_TO_EXTERNAL_BUFFER_POOL INTO @prev_written_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
| SELECT variable_value INTO @prev_reads_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_READS'; | ||
| SELECT NUMBER_PAGES_READ_FROM_EXTERNAL_BUFFER_POOL INTO @prev_reads_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
|
|
||
| --eval SET @start_val = $i*100 | ||
| INSERT INTO t SET a = @start_val+1; | ||
| INSERT INTO t SET a = @start_val+2; | ||
| INSERT INTO t SET a = @start_val+3; | ||
| INSERT INTO t SET a = @start_val+4; | ||
| INSERT INTO t SET a = @start_val+5; | ||
| INSERT INTO t SET a = @start_val+6; | ||
| INSERT INTO t SET a = @start_val+7; | ||
| INSERT INTO t SET a = @start_val+8; | ||
| INSERT INTO t SET a = @start_val+9; | ||
| INSERT INTO t SET a = @start_val+10; | ||
| INSERT INTO t SET a = @start_val+11; | ||
| INSERT INTO t SET a = @start_val+12; | ||
|
|
||
| SET GLOBAL DEBUG_DBUG='-d,ib_ext_bp_disable_LRU_eviction_for_t'; | ||
| SET GLOBAL innodb_force_LRU_eviction = TRUE; | ||
|
|
||
| let $wait_condition = | ||
| SELECT (variable_value-@prev_flushed_gs) >= 9 | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_PAGES_FLUSHED'; | ||
| --source include/wait_condition.inc | ||
|
|
||
| SELECT variable_value-@prev_flushed_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_PAGES_FLUSHED'; | ||
| SELECT NUMBER_PAGES_WRITTEN_TO_EXTERNAL_BUFFER_POOL-@prev_written_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
| SELECT variable_value-@prev_reads_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_READS'; | ||
| SELECT NUMBER_PAGES_READ_FROM_EXTERNAL_BUFFER_POOL-@prev_reads_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
|
|
||
| SELECT * FROM t; | ||
|
|
||
| SELECT variable_value-@prev_flushed_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_PAGES_FLUSHED'; | ||
| SELECT NUMBER_PAGES_WRITTEN_TO_EXTERNAL_BUFFER_POOL-@prev_written_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
| SELECT variable_value-@prev_reads_gs | ||
| FROM information_schema.global_status | ||
| WHERE variable_name LIKE 'INNODB_EXT_BUFFER_POOL_READS'; | ||
| SELECT NUMBER_PAGES_READ_FROM_EXTERNAL_BUFFER_POOL-@prev_reads_ps | ||
| FROM INFORMATION_SCHEMA.INNODB_BUFFER_POOL_STATS; | ||
|
|
||
| DROP TABLE t; | ||
| --dec $i | ||
| } | ||
|
|
||
| --disable_query_log | ||
| SET GLOBAL DEBUG_DBUG=@old_debug_dbug; | ||
| --error 0,ER_UNKNOWN_SYSTEM_VARIABLE | ||
| SET GLOBAL innodb_limit_optimistic_insert_debug = @old_innodb_limit_optimistic_insert_debug; | ||
| --enable_query_log | ||
|
|
||
| --disconnect prevent_purge | ||
| --source include/wait_until_count_sessions.inc | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -32,5 +32,7 @@ INNODB_BUFFER_POOL_STATS CREATE TEMPORARY TABLE `INNODB_BUFFER_POOL_STATS` ( | |
| `LRU_IO_TOTAL` bigint(21) unsigned NOT NULL, | ||
| `LRU_IO_CURRENT` bigint(21) unsigned NOT NULL, | ||
| `UNCOMPRESS_TOTAL` bigint(21) unsigned NOT NULL, | ||
| `UNCOMPRESS_CURRENT` bigint(21) unsigned NOT NULL | ||
| `UNCOMPRESS_CURRENT` bigint(21) unsigned NOT NULL, | ||
| `NUMBER_PAGES_WRITTEN_TO_EXTERNAL_BUFFER_POOL` bigint(21) unsigned NOT NULL, | ||
| `NUMBER_PAGES_READ_FROM_EXTERNAL_BUFFER_POOL` bigint(21) unsigned NOT NULL | ||
|
Comment on lines
-35
to
+37
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The test |
||
| ) ENGINE=MEMORY DEFAULT CHARSET=utf8mb3 COLLATE=utf8mb3_general_ci | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Typically, the local SSD is larger than the RAM, and therefore it would be more interesting to test a scenario where the buffer pool is smaller than the extended buffer pool. I understand that our current CI setup is not suitable for any performance testing. But, currently we only cover this functionality in debug-instrumented executables.
Would it be possible to write some sort of a test, or extend an existing one (such as
encryption.innochecksum) with a variant that would use the minimalinnodb_buffer_pool_sizeand a largerinnodb_extended_buffer_pool_size? Currently, that test is running withinnodb_buffer_pool_size=64M.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The test flushes small amount of pages into extended buffer pool file, that's why there was not any reason to set a big file size. Probably it makes sense to develop separate test, we can set the minimal
innodb_buffer_pool_sizeand a largerinnodb_extended_buffer_pool_size, but what exactly are we going to test in that separate test?