-
Notifications
You must be signed in to change notification settings - Fork 3.2k
feat: Improve progress reporting during identifier conversion #23186
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Changes from all commits
418b3a8
52a7b8a
3735a86
1cc3457
cb60003
cfecc6c
51af437
8bec621
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -64,12 +64,19 @@ def has_problematic_projects? | |||||||||||||||||||||
| def form_id = "wp-identifier-settings-form" | ||||||||||||||||||||||
|
|
||||||||||||||||||||||
| def in_progress_banner_message | ||||||||||||||||||||||
| key = if ProjectIdentifiers::IdentifierAutofix.reversion_in_progress? | ||||||||||||||||||||||
| "admin.settings.work_packages_identifier.in_progress.reverting_banner_message" | ||||||||||||||||||||||
| else | ||||||||||||||||||||||
| "admin.settings.work_packages_identifier.in_progress.converting_banner_message" | ||||||||||||||||||||||
| end | ||||||||||||||||||||||
| I18n.t(key) | ||||||||||||||||||||||
| if ProjectIdentifiers::IdentifierAutofix.reversion_in_progress? | ||||||||||||||||||||||
| pending_count = Project.with_non_classic_identifier.count | ||||||||||||||||||||||
| I18n.t("admin.settings.work_packages_identifier.in_progress.classic_conversion_status_message", | ||||||||||||||||||||||
| count: pending_count) | ||||||||||||||||||||||
| else | ||||||||||||||||||||||
| pending_count = ProjectIdentifiers::PendingProjectsFinder.project_ids.size | ||||||||||||||||||||||
| scope = "admin.settings.work_packages_identifier.in_progress.semantic_conversion_status_message" | ||||||||||||||||||||||
| if pending_count.zero? | ||||||||||||||||||||||
| I18n.t("#{scope}.zero") | ||||||||||||||||||||||
| else | ||||||||||||||||||||||
| I18n.t(scope, count: pending_count) | ||||||||||||||||||||||
| end | ||||||||||||||||||||||
|
Comment on lines
+74
to
+78
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍎 I18n already routes
|
||||||||||||||||||||||
| if pending_count.zero? | |
| I18n.t("#{scope}.zero") | |
| else | |
| I18n.t(scope, count: pending_count) | |
| end | |
| if pending_count.zero? | |
| I18n.t("#{scope}.zero") | |
| else | |
| I18n.t(scope, count: pending_count) | |
| end |
→ becomes just I18n.t(scope, count: pending_count).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting, I thought that automatic routing doesn't actually work and 0 gets direct to other. Will re-check.
| Original file line number | Diff line number | Diff line change | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -33,9 +33,14 @@ module Projects::Identifier | |||||||||||
|
|
||||||||||||
| CLASSIC_IDENTIFIER_MAX_LENGTH = 100 | ||||||||||||
| SEMANTIC_IDENTIFIER_MAX_LENGTH = 10 | ||||||||||||
| # Classic identifier format: lowercase letters, digits, hyphens, underscores — but not all-numeric. | ||||||||||||
| CLASSIC_IDENTIFIER_FORMAT = /\A(?!\d+\z)[a-z0-9\-_]+\z/ | ||||||||||||
|
|
||||||||||||
| # Classic format validation regexes: | ||||||||||||
| # Simple character set meant only for SQL matching. No anchoring or anti-all-numeric guard. | ||||||||||||
| CLASSIC_FORMAT_CHARS = /[a-z0-9\-_]+/ | ||||||||||||
|
Comment on lines
+38
to
+39
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍏 Comment reads as if the SQL is unanchored
The constant itself isn't anchored, but the only caller (line 126) adds |
||||||||||||
| # Anchored form with an anti-all-numeric guard. | ||||||||||||
| CLASSIC_FORMAT = /\A(?!\d+\z)#{CLASSIC_FORMAT_CHARS}\z/ | ||||||||||||
|
akabiru marked this conversation as resolved.
|
||||||||||||
|
|
||||||||||||
| # Semantic format validation regex: | ||||||||||||
| # Unanchored shape of a semantic project identifier ("PROJ", "MY_PROJECT_1"). | ||||||||||||
| # Composed into `WorkPackage::SemanticIdentifier::SEMANTIC_ID_PATTERN`. | ||||||||||||
| SEMANTIC_FORMAT = /[A-Z][A-Z0-9_]*/ | ||||||||||||
|
|
@@ -114,7 +119,11 @@ def raw_values = pluck(:slug) | |||||||||||
|
|
||||||||||||
| class_methods do | ||||||||||||
| def classic_identifier_format?(str) | ||||||||||||
| str.match?(CLASSIC_IDENTIFIER_FORMAT) | ||||||||||||
| str.match?(CLASSIC_FORMAT) | ||||||||||||
| end | ||||||||||||
|
|
||||||||||||
| def with_non_classic_identifier | ||||||||||||
| where("identifier !~ ?", "^#{CLASSIC_FORMAT_CHARS.source}$") | ||||||||||||
|
Comment on lines
+125
to
+126
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍎
|
||||||||||||
| def with_non_classic_identifier | |
| where("identifier !~ ?", "^#{CLASSIC_FORMAT_CHARS.source}$") | |
| def with_non_classic_identifier | |
| where("identifier !~ ?", '^(?!\\d+$)[a-z0-9_-]+$') | |
| end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the lookahead doesn't work for SQL in this query 👀 and if nothing else, it's a bit faster without it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But, interestingly enough, we may want to include all-numeric IDs here for projects created before 2021.
(Though, that will require adding a little bit of extra error handling for the back-conversion, because conversion back to 123 would yield a validation error.)
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -449,8 +449,13 @@ en: | |||||||||||||||||||
| checkbox_label: I understand that this will permanently change all work package IDs | ||||||||||||||||||||
| success_banner: Successfully updated work package identifier format. | ||||||||||||||||||||
| in_progress: | ||||||||||||||||||||
| converting_banner_message: Project identifiers are currently being converted to semantic format. This may take some time. | ||||||||||||||||||||
| reverting_banner_message: Project identifiers are currently being reverted to classic format. This may take some time. | ||||||||||||||||||||
| semantic_conversion_status_message: | ||||||||||||||||||||
| zero: Finalizing conversion to semantic format... | ||||||||||||||||||||
| one: Project identifiers are being converted to semantic format. 1 project remaining. | ||||||||||||||||||||
| other: Project identifiers are being converted to semantic format. %{count} projects remaining. | ||||||||||||||||||||
| classic_conversion_status_message: | ||||||||||||||||||||
| one: Project identifiers are being converted to classic format. 1 project remaining. | ||||||||||||||||||||
| other: Project identifiers are being converted to classic format. %{count} projects remaining. | ||||||||||||||||||||
| workflows: | ||||||||||||||||||||
| tabs: | ||||||||||||||||||||
|
Comment on lines
+456
to
460
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍎 Add
|
||||||||||||||||||||
| classic_conversion_status_message: | |
| one: Project identifiers are being converted to classic format. 1 project remaining. | |
| other: Project identifiers are being converted to classic format. %{count} projects remaining. | |
| workflows: | |
| tabs: | |
| classic_conversion_status_message: | |
| zero: Finalizing conversion to classic format... | |
| one: Project identifiers are being converted to classic format. 1 project remaining. | |
| other: Project identifiers are being converted to classic format. %{count} projects remaining. |
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
|
|
@@ -53,9 +53,46 @@ def render_component(component) | |||||
| context "when state is :change_in_progress" do | ||||||
| let(:state) { :change_in_progress } | ||||||
|
|
||||||
| it "renders the in-progress spinner message" do | ||||||
| before do | ||||||
| allow(ProjectIdentifiers::IdentifierAutofix).to receive(:reversion_in_progress?).and_return(false) | ||||||
| allow(ProjectIdentifiers::PendingProjectsFinder).to receive(:project_ids).and_return(Set.new(1..7)) | ||||||
| end | ||||||
|
|
||||||
| it "renders the converting banner with the pending project count" do | ||||||
| render_component(component) | ||||||
| expect(page).to have_text("Project identifiers are currently being converted to semantic format.") | ||||||
| expect(page).to have_text("7 projects remaining") | ||||||
| end | ||||||
|
|
||||||
| context "with 1 project remaining" do | ||||||
| before { allow(ProjectIdentifiers::PendingProjectsFinder).to receive(:project_ids).and_return(Set[42]) } | ||||||
|
|
||||||
| it "uses the singular form" do | ||||||
| render_component(component) | ||||||
| expect(page).to have_text("1 project remaining") | ||||||
| end | ||||||
| end | ||||||
|
|
||||||
| context "with 0 projects remaining" do | ||||||
| before { allow(ProjectIdentifiers::PendingProjectsFinder).to receive(:project_ids).and_return(Set.new) } | ||||||
|
|
||||||
| it "renders the finalizing message" do | ||||||
| render_component(component) | ||||||
| expect(page).to have_text("Finalizing conversion to semantic format") | ||||||
| expect(page).to have_no_text("projects remaining") | ||||||
| end | ||||||
| end | ||||||
|
|
||||||
| context "when reversion is in progress" do | ||||||
| before do | ||||||
| allow(ProjectIdentifiers::IdentifierAutofix).to receive(:reversion_in_progress?).and_return(true) | ||||||
| allow(Project).to receive(:with_non_classic_identifier).and_return(double(count: 3)) | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍊 Stubbing
|
||||||
| allow(Project).to receive(:with_non_classic_identifier).and_return(double(count: 3)) | |
| create_list(:project, 3, identifier: "PROJ#{_1}") |
(And drop the allow(Project).to receive(:with_non_classic_identifier)… stub.)
The forward-direction stub on PendingProjectsFinder.project_ids (lines 58, 67, 76) is fine — it's a module-level pure function, not an AR scope.
| Original file line number | Diff line number | Diff line change | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -36,6 +36,36 @@ | |||||||||
|
|
||||||||||
| current_user { user } | ||||||||||
|
|
||||||||||
| describe "GET #status" do | ||||||||||
| let(:component_target) { "work-packages-admin-settings-identifier-settings-form-component" } | ||||||||||
|
|
||||||||||
| context "when a migration job is in progress" do | ||||||||||
| before do | ||||||||||
| allow(ProjectIdentifiers::IdentifierAutofix).to receive(:job_in_progress?).and_return(true) | ||||||||||
| end | ||||||||||
|
|
||||||||||
| it "returns 200 with a turbo stream replacing the form component in change_in_progress state" do | ||||||||||
| get :status, format: :turbo_stream | ||||||||||
|
|
||||||||||
| expect(response).to have_http_status(:ok) | ||||||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🍊 Both
|
||||||||||
| expect(response).to have_http_status(:ok) | |
| expect(response).to have_http_status(:ok) | |
| expect(response).to have_turbo_stream(action: "replace", target: component_target) | |
| expect(response.body).to include("projects remaining").or include("Finalizing conversion") |
Mirror with Successfully updated work package identifier format. in the completed example (line 60-62).
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure the proposal here on adopting a cache is valid- as some projects will take less than 10s and afaik, cache is an opt-in; not guaranteed to be present. But the indexing point is sth worth exploring?
🍊
PendingProjectsFinder.project_idsis three unindexed queries every 5sPendingProjectsFinder.project_ids(app/services/project_identifiers/pending_projects_finder.rb:40-58— not in this PR) materializes three Sets per call:ProblematicIdentifiers#scope.ids— 4 OR'd regex predicates onprojects.identifier(no functional index; the file's own perf note acknowledges this).WorkPackage.unsequenced.distinct.pluck(:project_id)— scan filtered onsequence_number IS NULL.WorkPackage.non_semantic.distinct.pluck(:project_id)—JOIN projects+ per-rowwork_packages.identifier IS DISTINCT FROM projects.identifier || '-' || sequence_number::text. No index supports the concat.This runs every 5s (
poll_for_changes_interval_value: 5000below) for every admin polling the page, against the samework_packagesrows the conversion job is mutating. On a large instance that's seconds of DB time per poll while the job is contending for the same rows.A monotonically-decreasing counter doesn't need second-level freshness — short cache window is enough:
Alternative: add
PendingProjectsFinder.countthat aggregates the three buckets in one SQL union — the caller doesn't actually need the IDs, only the size.