Expand WordPress Intelligence support to other locales

[kean]

What

Fixes CMM-762: Excerpts are created in the system language not the content language and CMM-798: Add support for other locales

I smoke-tested it by generating a post in Spanish and testing that the excerpts are also in Spanish. It should work for any other scenarios as well.

How

• Move the Intelligence-related code to a new WordPressModule and use it as a namespace (rename some of the features)
• Add an extensive lists of tests primarily focusing on locales. The tests are non-deterministic and slow, so they are not intended to be run on every change (for now). They are designed for semi-manually verifying the changes to prompts and other parts of WordPressIntelligence.

It's a large PR, but the majority of the changes are in unit tests. The main changes to look for is in prompts.

[kean]

It's technically redundant; just some more defensive code.

[wpmobilebot]

📲 You can test the changes from this Pull Request in Jetpack by scanning the QR code below to install the corresponding build.

	App Name	Jetpack
	Configuration	Release-Alpha
	Build Number	30202
	Version	PR #25034
	Bundle ID	com.jetpack.alpha
	Commit	a45969e
	Installation URL	2uidh380oa8l8

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[wpmobilebot]

📲 You can test the changes from this Pull Request in WordPress by scanning the QR code below to install the corresponding build.

	App Name	WordPress
	Configuration	Release-Alpha
	Build Number	30202
	Version	PR #25034
	Bundle ID	org.wordpress.alpha
	Commit	a45969e
	Installation URL	7d77ftbbpq2to

Automatticians: You can use our internal self-serve MC tool to give yourself access to those builds if needed.

[crazytonyli]

Nitpick: I think the prompt can be more direct here: "You MUST generate the summary in the (locale.identifier) locale.", and remove the makeLocaleInstructions() above?

[crazytonyli]

I'd expect the generated summary to be in the post's language, rather than the device's language. What do you think?

[crazytonyli]

Oh, sorry, I missed that there are updates to other prompts too.

[crazytonyli]

The post summary generation is not updated in this PR. Probably because that went to a separate code path: LanguageModelHelper.makeGenerateExcerptPrompt.

[crazytonyli]

I used a news article in Chinese for testing. The post tags generation in the "Post Settings" page does not work for me.

Here are some debug prints:

IntelligenceService.suggestTags executed in 2771.9709873199463 ms
Printing description of instructions:
The person's locale is en_NZ.
You are helping a WordPress user add tags to a post or a page.
**Parameters**
- POST_CONTENT: contents of the post (HTML or plain text)
- SITE_TAGS: case-sensitive comma-separated list of the existing tags used elsewhere on the site (not always relevant to the post)
- EXISTING_POST_TAGS: tags already added to the post
**Steps**
- 1. Identify the specific formatting pattern used (e.g., lowercase with underscores, capitalized words with spaces, etc)
- 2. Identify the language used in SITE_TAGS and POST_CONTENT
- 3. Generate a list of ten most relevant suggested tags based on POST_CONTENT and SITE_TAGS relevant to the content.
**Requirements**
- You MUST generate tags in the same language as SITE_TAGS and POST_CONTENT
- Tags MUST match the formatting pattern and language of existing tags
- Do not include any tags from EXISTING_POST_TAGS
- If there are no relevant suggestions, returns an empty list
- Do not produce any output other than the final list of tag
Printing description of prompt:
Suggest up to ten tags for a post.
POST_CONTENT: '''
<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>
<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>
<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>
<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>
<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>
<p>12月1日（周一）下午，香港媒体披露关靖丰离开警署的画面，称他获准保释候查。</p>
<p>目前，警方以涉嫌误杀拘捕13人，各人来自以下公司，包括：大判工程（总包商）、工程顾问、二判（分包商）搭棚及外墙工程；调查贪污的廉政公署拘捕12人，包括：工程顾问、承建商及搭棚判头。</p>
<p>现任大埔区区议员黄碧娇12月1日就宏福苑大火发声明称，将亲自向香港警务处报案，要求立即立案调查事件是否涉及现届法团渎职、隐瞒安全风险等严重罪行， 同时向廉政公署（ICAC）要求立案调查，彻查现届法团是否存在贪污渎职，导致消防设备长期失修多年未被执管。黄曾担任宏福苑旧法团顾问，去年9月卸任。</p>
'''
SITE_TAGS: 'high, on, rf, jonnie, hobbs, hello, an, great, bob, he'll'
EXISTING_POST_TAGS: ''
Printing description of response:
(FoundationModels.LanguageModelSession.Response<WordPressShared.SuggestedTagsResult>) response = {
  userPrompt = "Suggest up to ten tags for a post.\n\nPOST_CONTENT: \'\'\'\n<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>\n<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>\n<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>\n<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>\n<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>\n<p>12月1日（周一）下午，香港媒体披露\u{e5}"...
  duration = 2.7653398340000002
  feedbackAttachment = nil
  content = {
    tags = 10 values {
      [0] = "high"
      [1] = "on"
      [2] = "rf"
      [3] = "jonnie"
      [4] = "hobbs"
      [5] = "hello"
      [6] = "an"
      [7] = "great"
      [8] = "bob"
      [9] = "he\'ll"
    }
  }
  rawContent = {
    value = {
      storage = object {
        object = {
          properties = 1 key/value pair {
            [0] = {
              key = "tags"
              value = {
                storage = array {
                  array = 10 values {
                    [0] = {
                      storage = string (string = "high")
                      isComplete = true
                    }
                    [1] = {
                      storage = string (string = "on")
                      isComplete = true
                    }
                    [2] = {
                      storage = string (string = "rf")
                      isComplete = true
                    }
                    [3] = {
                      storage = string (string = "jonnie")
                      isComplete = true
                    }
                    [4] = {
                      storage = string (string = "hobbs")
                      isComplete = true
                    }
                    [5] = {
                      storage = string (string = "hello")
                      isComplete = true
                    }
                    [6] = {
                      storage = string (string = "an")
                      isComplete = true
                    }
                    [7] = {
                      storage = string (string = "great")
                      isComplete = true
                    }
                    [8] = {
                      storage = string (string = "bob")
                      isComplete = true
                    }
                    [9] = {
                      storage = string (string = "he\'ll")
                      isComplete = true
                    }
                  }
                }
                isComplete = true
              }
            }
          }
          order = nil
        }
      }
      isComplete = true
    }
    id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
  }
  transcriptEntries = 1 value {
    [2] = response {
      response = {
        id = "13EB559F-C1B9-4309-A3AB-B93B5C2C9E24"
        assetIDs = 3 values {
          [0] = "com.apple.fm.language.instruct_3b.fm_api_generic_12.0.0.13.101733,0"
          [1] = "com.apple.fm.language.instruct_3b.tokenizer_12.0.0.13.202233,0"
          [2] = "com.apple.fm.language.instruct_3b.fm_api_generic.draft_12.0.81307.13.202252,0"
        }
        segments = 1 value {
          [0] = structure {
            structure = {
              id = "0"
              source = "SuggestedTagsResult"
              content = {
                value = {
                  storage = object {
                    object = {
                      properties = 1 key/value pair {
                        [0] = {
                          key = "tags"
                          value = {
                            storage = array {
                              array = 10 values {
                                [0] = {
                                  storage = string (string = "high")
                                  isComplete = true
                                }
                                [1] = {
                                  storage = string (string = "on")
                                  isComplete = true
                                }
                                [2] = {
                                  storage = string (string = "rf")
                                  isComplete = true
                                }
                                [3] = {
                                  storage = string (string = "jonnie")
                                  isComplete = true
                                }
                                [4] = {
                                  storage = string (string = "hobbs")
                                  isComplete = true
                                }
                                [5] = {
                                  storage = string (string = "hello")
                                  isComplete = true
                                }
                                [6] = {
                                  storage = string (string = "an")
                                  isComplete = true
                                }
                                [7] = {
                                  storage = string (string = "great")
                                  isComplete = true
                                }
                                [8] = {
                                  storage = string (string = "bob")
                                  isComplete = true
                                }
                                [9] = {
                                  storage = string (string = "he\'ll")
                                  isComplete = true
                                }
                              }
                            }
                            isComplete = true
                          }
                        }
                      }
                      order = nil
                    }
                  }
                  isComplete = true
                }
                id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
              }
              rawValue = "{\"tags\": [\"high\", \"on\", \"rf\", \"jonnie\", \"hobbs\", \"hello\", \"an\", \"great\", \"bob\", \"he\'ll\"]}"
            }
          }
        }
      }
    }
  }
}
Printing description of response:
(FoundationModels.LanguageModelSession.Response<WordPressShared.SuggestedTagsResult>) response = {
  userPrompt = "Suggest up to ten tags for a post.\n\nPOST_CONTENT: \'\'\'\n<p>香港警方表示，周一（12月1日）新发现5具大埔宏福苑火灾遗体，死亡人数增至151人。警方称有些遗体已烧化灰烬，难以辨认。</p>\n<p>中国国务院港澳办网站消息，港澳办主任夏宝龙今日在深圳听取大埔火灾有关情况的详细汇报。</p>\n<p>火灾后有市民成立“大埔宏福苑火灾关注组”发起联署，提出四大诉求，包括持续支援受灾居民，成立独立调查委员会，全面彻查潜利益输送，以及全力追究监管疏忽，问责政府官员。</p>\n<p>其中一名联署发起人关靖丰11月29日被传因涉嫌煽动被香港警方国安处拘捕；翌日，再有消息传出两名男女被捕，其一人为前屯门区议员张锦雄，另一名女子则为义工。</p>\n<p>警方未有交代拘捕行动，但回覆BBC中文称，警方采取任何行动，均会按实际情况，依法处理。</p>\n<p>12月1日（周一）下午，香港媒体披露\u{e5}"...
  duration = 2.7653398340000002
  feedbackAttachment = nil
  content = {
    tags = 10 values {
      [0] = "high"
      [1] = "on"
      [2] = "rf"
      [3] = "jonnie"
      [4] = "hobbs"
      [5] = "hello"
      [6] = "an"
      [7] = "great"
      [8] = "bob"
      [9] = "he\'ll"
    }
  }
  rawContent = {
    value = {
      storage = object {
        object = {
          properties = 1 key/value pair {
            [0] = {
              key = "tags"
              value = {
                storage = array {
                  array = 10 values {
                    [0] = {
                      storage = string (string = "high")
                      isComplete = true
                    }
                    [1] = {
                      storage = string (string = "on")
                      isComplete = true
                    }
                    [2] = {
                      storage = string (string = "rf")
                      isComplete = true
                    }
                    [3] = {
                      storage = string (string = "jonnie")
                      isComplete = true
                    }
                    [4] = {
                      storage = string (string = "hobbs")
                      isComplete = true
                    }
                    [5] = {
                      storage = string (string = "hello")
                      isComplete = true
                    }
                    [6] = {
                      storage = string (string = "an")
                      isComplete = true
                    }
                    [7] = {
                      storage = string (string = "great")
                      isComplete = true
                    }
                    [8] = {
                      storage = string (string = "bob")
                      isComplete = true
                    }
                    [9] = {
                      storage = string (string = "he\'ll")
                      isComplete = true
                    }
                  }
                }
                isComplete = true
              }
            }
          }
          order = nil
        }
      }
      isComplete = true
    }
    id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
  }
  transcriptEntries = 1 value {
    [2] = response {
      response = {
        id = "13EB559F-C1B9-4309-A3AB-B93B5C2C9E24"
        assetIDs = 3 values {
          [0] = "com.apple.fm.language.instruct_3b.fm_api_generic_12.0.0.13.101733,0"
          [1] = "com.apple.fm.language.instruct_3b.tokenizer_12.0.0.13.202233,0"
          [2] = "com.apple.fm.language.instruct_3b.fm_api_generic.draft_12.0.81307.13.202252,0"
        }
        segments = 1 value {
          [0] = structure {
            structure = {
              id = "0"
              source = "SuggestedTagsResult"
              content = {
                value = {
                  storage = object {
                    object = {
                      properties = 1 key/value pair {
                        [0] = {
                          key = "tags"
                          value = {
                            storage = array {
                              array = 10 values {
                                [0] = {
                                  storage = string (string = "high")
                                  isComplete = true
                                }
                                [1] = {
                                  storage = string (string = "on")
                                  isComplete = true
                                }
                                [2] = {
                                  storage = string (string = "rf")
                                  isComplete = true
                                }
                                [3] = {
                                  storage = string (string = "jonnie")
                                  isComplete = true
                                }
                                [4] = {
                                  storage = string (string = "hobbs")
                                  isComplete = true
                                }
                                [5] = {
                                  storage = string (string = "hello")
                                  isComplete = true
                                }
                                [6] = {
                                  storage = string (string = "an")
                                  isComplete = true
                                }
                                [7] = {
                                  storage = string (string = "great")
                                  isComplete = true
                                }
                                [8] = {
                                  storage = string (string = "bob")
                                  isComplete = true
                                }
                                [9] = {
                                  storage = string (string = "he\'ll")
                                  isComplete = true
                                }
                              }
                            }
                            isComplete = true
                          }
                        }
                      }
                      order = nil
                    }
                  }
                  isComplete = true
                }
                id = (value = "50F02B8F-D6C4-4483-9491-9C1D3BA5B9BF")
              }
              rawValue = "{\"tags\": [\"high\", \"on\", \"rf\", \"jonnie\", \"hobbs\", \"hello\", \"an\", \"great\", \"bob\", \"he\'ll\"]}"
            }
          }
        }
      }
    }
  }
}

[dangermattic]

	2 Warnings
⚠️	Modules/Package.swift was changed without updating its corresponding Package.resolved. If the change includes adding, removing, or editing a dependency please resolve the Swift packages as appropriate to your project setup (e.g. in Xcode or by running swift package resolve). If the change to the Package.swift did not modify dependencies, ignoring this warning should be safe, but we recommend double checking and running the package resolution just in case. .
⚠️	This PR is larger than 500 lines of changes. Please consider splitting it into smaller PRs for easier and faster reviews.

Generated by 🚫 Danger

[kean]

I added a significant amount of automated tests, improved the prompts to get it to produce the correct language output at a much higher rate, and updated the PR description.

I also have another upcoming PR with additional LLM-based evaluation for automation.

[kean]

There are a couple of SwiftLint issues which I will be addressing before merge.

[crazytonyli]

It seems it does not work with Chinese posts.

->	->	->

[kean]

I've tested a few different options and thanks to the unit tests identified the one that seems to work the best. I included NLLanguageRecognizer in the solution to detect the target language before invoking the LLM: c3076f2.

I skipped other checks for testing it:

@available(iOS 26, *)
    @Test(arguments: ExcerptTestCaseParameters.nonEnglishCases)
    func excerptGenerationNonEnglish(parameters: ExcerptTestCaseParameters) async throws {
        _ = try await runExcerptTest(parameters: parameters, skip: [.skipWordCountCheck, .skipDiversityCheck])
    }

The language tests are not all passing. Previously, there were still a bit inconsistent for some of the supported languages.

[kean]

I updated tag and summary generation to follow the same example. I plan to continue iterating on the general quality of the responses in separate PR and in the scope of #25059. The localization support now seem to be up to acceptable quality level.

[crazytonyli]

I can still reproduce the issue in my comment above.

[kean]

Tony, can you make sure you have the latest code, please? It verifying it before publishing and it worked, and it does in unit tests covering multiple locales. I wouldn't expect it to be 100% reliable.

In addition to looking for issues, could you please review the PR first and see if there something that needs to change code-wise? There are quite a lot of changes, but mostly moved. I'd be happy to address any feedback.

[sonarqubecloud]

Quality Gate passed

Issues
8 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[crazytonyli]

The excerpt works for me now. But the suggested tags are still unrelated to the post content.

Also, maybe we don't need to give the option to generate excerpts if the content is super short? It'd be pretty tricky for LLM to generate accurate excerpts if the post content is short.

[crazytonyli]

Is this function async because detectLanguage can be slow?

[crazytonyli]

Does this prompt work with giving LLM the detectLanguage result?

[crazytonyli]

BTW, I think it's okay to generate a summary in the device language in this context. The summary is for the customer, not website visitors.

[crazytonyli]

We probably want the generated summary to be in the device language. The summary is for the current user after all.

I guess the AI summary feature can be used as a quick way to translate and summarize if I don't understand the original post's language.

[kean]

For some reason, GitHub won't let me reply to my own PR comments. Here's what I posted so far:

in the device language in this context. The summary is for the customer, not website visitors.

I was just thinking about it – do we want to translate the post or not and whether it should be the same thing as "summarize". I would argue it should be two separate features. I do plan to add translation, starting with comments CMM-744: On-device translation for comments and then post. It is not in the scope of this PR. I'd start with summarizing without translation the way it does now.

Is this function async because detectLanguage can be slow?

Yes, it to make detectLanguage and everything else run in the background.

[kean]

The excerpt works for me now. But the suggested tags are still unrelated to the post content.

I did a bit more testing, and I confirm this is a potential issues, but it's not directly related to localization. I opened CMM-1073: Improve quality of suggested tags to track it. The issue is that the prompt prioritizes current site tags, their language, and format – which is generally a good thing for real sites. If you have a test site with a bunch of tags unrelated to the posts, it will not work as well, which is also expected. However, I think it still needs improvement as it should suggest tags only if there is a high level of confidence they match the text. Having said that, I'm not completely sure because you may be tagging based on some other criteria, so showing some suggestions can be better than showing none. I think the app might need do a little of work in the background to try and find relationships between your posts and tags before making suggestions.

Also, maybe we don't need to give the option to generate excerpts if the content is super short? It'd be pretty tricky for LLM to generate accurate excerpts if the post content is short.

It is an existing ticket CMM-763: If the post content is too short, the results are not helpful. I don't know how to define "short", so I haven't looked into it yet. It doesn't seem like an issue that should be addressed as nothing bad happens if you do it and it's an unlikely scenario a user would use generation in the first place. For tags, I'll look into it in the scope of the previous issue I linked.