[IMP] util.explode_query_range#435
Conversation
Randomize the generated queries. As we split queries based on their ids, it's more likely than consecutive queries (i.e. `id BETWEEN 101 AND 200` and `id BETWEEN 201 AND 300`) will hit the same page when being executed by two concurrent PG processes, blocking themself with read and write locks. By shuffling the queries, we avoid such problem.
|
upgradeci retry with always only crm |
|
|
| query = _explode_format(query.replace("%", "%%"), parallel_filter=parallel_filter) | ||
|
|
||
| return [ | ||
| queries = [ |
There was a problem hiding this comment.
| queries = [ | |
| n_ids = len(ids) - 1 | |
| modulo = n_ids // 3 | |
| return [ | |
| cr.mogrify(query, {"lower-bound": ids[i], "upper-bound": ids[i + 1] - 1}).decode() | |
| for i in sorted(range(n_ids), key=lambda x: (x % modulo, x)) | |
| ] |
Alternative idea to make it deterministic. Though slightly less efficient.
Or if we want to go random, why don't we just shuffle the numbers?
| queries = [ | |
| n_ids = len(ids) - 1 | |
| return [ | |
| cr.mogrify(query, {"lower-bound": ids[i], "upper-bound": ids[i + 1] - 1}).decode() | |
| for i in random.sample(range(n_ids), n_ids) | |
| ] |
| queries = [ | ||
| cr.mogrify(query, {"lower-bound": ids[i], "upper-bound": ids[i + 1] - 1}).decode() for i in range(len(ids) - 1) | ||
| ] | ||
| random.shuffle(queries) |
There was a problem hiding this comment.
I think patching this method is not ideal. We should keep it returning the queries in the stable order. We should instead patch the executing part. I'd add a parameter to parallel_execute that signals whether the queries are chunked in consecutive chunks and so we may benefit from a "better" ("randomized" to keep it simple) dispatcher. explode_execute could set the param to True. We could then set it in places where we know it helps for older scripts as well. In most cases just switching to explode_execute would be enough.
There was a problem hiding this comment.
Note that by altering the order at execution time we expose ourselves to variations in running time for big tables that are scattered on disk. I cannot say how "significant", but this makes me think that using a random strategy may haunt us later.
4 participants
Randomize the generated queries.
As we split queries based on their ids, it's more likely than consecutive queries (i.e.
id BETWEEN 101 AND 200andid BETWEEN 201 AND 300) will hit the same page when being executed by two concurrent PG processes, blocking themself with read and write locks. By shuffling the queries, we avoid such problem.