feat: Redirecting MDB feeds to TDG

[cka-y]

Summary:
This pull request introduces a new mechanism for redirecting duplicate MDB feeds to their corresponding TDG-imported feeds, along with comprehensive tests and a supporting database constraint update. The main changes include the implementation of a new handler for updating redirects based on a CSV mapping, integration of this handler into the task executor, and the addition of tests to ensure correctness. Additionally, a database migration enforces non-null operational status for feeds.

Redirects and Data Import Improvements:

• Added a new handler (update_tdg_redirects_handler) in update_tdg_redirects.py to process a CSV mapping of MDB to TDG feeds, ensuring database redirects are created or updated as needed, with support for dry-run and batch commits.
• Registered the new "mdb_to_tdg_redirect" task in main.py, making the handler available as a callable task with a description. [1] [2]

Testing:

• Introduced a new test suite (test_redirect_mdb_feeds.py) covering various scenarios for the redirect logic, including creation, missing feeds, existing redirects, and CSV load failures.

Database Migration:

• Added a new Liquibase migration (fix_operation_status_constraint.sql) to set all null operational_status values in the feed table to 'published' and enforce the column as non-null, referenced in the main changelog. [1] [2]

Please make sure these boxes are checked before submitting your pull request - thanks!

• Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
• Add or update any needed documentation to the repo
• Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
• Linked all relevant issues
• Include screenshot(s) showing how this pull request works and fixes the issue(s)

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.

[qcdyx]

LGTM

[davidgamez]

LGTM!

[cka-y]

Post release we can set the TDG redirects by running the task-executor-prod function with parameters:

{
  "task": "mdb_to_tdg_redirect",
  "payload": {
    "dry_run": false
  }
}

I suggest running the function after TDG feeds as set as published to avoid redirecting to a wip feed that is not accessible to users.

[davidgamez]

LGTM!

[davidgamez]

👍

[davidgamez]

👍