refactor: migrate framework subpackages to the nested layout#65
Draft
avinash2692 wants to merge 17 commits into
Draft
refactor: migrate framework subpackages to the nested layout#65avinash2692 wants to merge 17 commits into
avinash2692 wants to merge 17 commits into
Conversation
Adds a workflow that listens for a repository_dispatch event (event_type=mellea-released) and bumps every contribs pyproject.toml's version + mellea>= constraint to the released version. Refreshes every uv.lock and opens a PR. Uses the default GITHUB_TOKEN. The helper script leaves == exact pins alone — subpackages opting into exact pins own their own bumps — and is idempotent. The mellea-side dispatcher follows in a separate PR. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Replaces the previous behavior of bumping both `[project] version` and `mellea>=` lines. The receiver now bumps only `version`; each subpackage owns its `mellea>=` floor and only raises it when CI proves something below it breaks. The script also errors out on `mellea` dependency lines that lack an explicit version constraint — bare `mellea`, `mellea[extras]` without operator, and `mellea @ git+...` are rejected. The receiver cannot reason about those forms and silently skipping them would let incompatibilities through. Acceptable forms remain `mellea>=X.Y.Z` and `mellea==X.Y.Z` (with or without extras). The current contribs repo uses these exclusively; the new check is a forward-looking guard. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Replaces the single bump PR with one PR per subpackage so owners can merge independently — slow movers no longer block fast movers, and CI signals compatibility per package. PRs open in dependency tiers: - Tier 1: _integration_core (consumed by frameworks). - Tier 2: dspy, langchain, crewai, tools (depend on tier 1). - Tier 3: legal-reqs, python-imports, grounding-context (leaves). The workflow re-runs when a sync-mellea-* PR merges, advancing to the next tier automatically. Subpackages whose bump PR doesn't merge before the next coordinated contribs release are left at the old version and ship in the following release. open_per_package_bump_prs.py owns the tier detection and PR opening; the workflow itself just wires the trigger events (repository_dispatch, pull_request closed-merged on sync-mellea-* branches, manual dispatch). Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
The previous pattern `"mellea(\[[^\]]+\])?(?P<spec>[^"]*)"` matched sibling distributions like `mellea-contribs-integration-core` and `mellea-tools` because anything after `mellea` was eaten by the optional extras group + spec capture. The receiver then treated those sibling lines as malformed mellea deps and errored out. A negative lookahead `(?![a-zA-Z0-9_-])` after `mellea` rules out the prefix-of-a-longer-name case while still allowing `mellea>=`, `mellea==`, `mellea[extras]>=`, and bare `mellea`. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Adds an empty meta-package at the repo root with no runtime dependencies and a dev-only [dependency-groups] for the tooling that runs across the whole repo (ruff, mypy, actionlint-py, cookiecutter, pyyaml, pre-commit, pytest). `uv sync` at the root installs only those tools; subpackages stay independent. [project.optional-dependencies] is left empty for now and gets populated incrementally as subpackages migrate. At release time, release.yml rewrites those entries to versioned GitHub Release URLs at build time. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Generates a subpackage scaffold with the namespace package physically on disk: `<subpkg>/mellea_contribs/<name>/<core mirror>/...`. The wheel layout matches the source layout, no hatch sources remap needed; this is the only shape that works with editable installs (uv sync) when the import path is `mellea_contribs.<name>.<...>`. Includes the empty mirror skeleton (stdlib, backends, formatters, helpers, core), a stub module, smoke test, basic example, OWNERS, README, and a pyproject with a [tool.mellea-contribs.ci] block. The pre-gen hook validates snake_case name and rejects unknown core_path values against core_paths.json — a snapshot of valid mellea core paths regenerated by the upstream release sync. The post-gen hook creates the inner directory chain under mellea_contribs/<name>/, writes a hello() stub, and runs `uv lock` best-effort. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Adds a validator that walks every subpackage at the repo root and asserts the structural contract: required files (pyproject.toml, OWNERS, README.md), required dirs (tests/, examples/), non-empty OWNERS, [tool.hatch.build.targets.wheel].packages = ["mellea_contribs"], the namespace dir mellea_contribs/<name>/ with __init__.py, only known mirrors under it, and every nested mirror dir resolving to a known dotted path in cookiecutter/core_paths.json. Subpackages whose dependencies list `mellea` without an explicit version constraint (bare `mellea`, `mellea[extras]` without operator, `mellea @ git+...`) are rejected — the receiver workflow can't reason about those forms and silently skipping them would let incompatibilities through. Distribution-name uniqueness is checked across the whole repo. A grandfather list at .github/scripts/grandfather_legacy.json holds the six legacy subpackages under mellea_contribs/ that pre-date the restructure; each migration shrinks the list. The workflow runs on every PR and on push to main. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Replaces the old ci.yml with a scoped-discovery + matrix-dispatch pair. The new ci.yml walks the diff against base, classifies the PR (docs-only, cookiecutter-only, root-pyproject, workflow, subpackage, union, or stacked-PR), and dispatches package-ci.yml per touched subpackage. Stacked PRs (base_ref != main) promote to all packages so downstream branches see the cumulative effect of their parent. package-ci.yml is a workflow_call template that reads each subpackage's [tool.mellea-contribs.ci] table for skip_ollama, timeout_minutes, and python_versions, replacing the hardcoded path-string special-cases the previous ci.yml had to carry. The previous ci.yml is preserved as legacy-ci.yml, scoped to mellea_contribs/** so it stops firing once those paths are removed. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Adds a daily smoke workflow that runs each opted-in subpackage's tests against mellea@main, plus an auto-issue bot that opens a tracking issue after two consecutive reds, comments on recovery, and applies a 21-day archival timeline driven by the contribs-broken label. The smoke matrix in .github/smoke-matrix.json starts empty so the smoke job is skipped until the first subpackage opts in. The bot ships in two modes: an in-memory fake used by the unit tests (14 tests covering the failure threshold, recovery, and the day-7 / 14 / 21 milestones) and a PyGithub-backed real mode that persists state on a bot-managed branch. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Adds a section explaining the daily smoke job's gating (.github/smoke-matrix.json), the auto-issue bot's two-consecutive-reds threshold, the contribs-broken label as the source of truth for the 21-day archival timeline, and the day-7 / 14 / 21 milestones. Also shows how to run the bot locally against the in-memory fake. The rest of RELEASING.md still describes the per-subpackage tag release flow that's slated for replacement when the coordinated release pipeline lands; this commit only documents the smoke-bot lifecycle that the foundation work just introduced. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
…low config/ The validate-structure rule that requires explicit mellea>= or mellea== constraints was matching sibling distribution names like mellea-contribs-integration-core and mellea-tools because they start with "mellea". Same prefix-matching bug class as the receiver script's regex; fix is the analogous disambiguation: a hyphen, underscore, letter, or digit immediately after `mellea` means we're looking at a sibling dist, not the upstream package. Also adds `config` to META_DIRS so subpackages that ship YAML configs alongside their code don't trip the unexpected-top-level-directory check. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Moves the legacy mellea_contribs/mellea-integration-core subpackage to
the new top-level _integration_core/ directory with the disk-matches-
wheel namespace layout (mellea_contribs/_integration_core/{core,stdlib,
backends,formatters,helpers}/). Renames the distribution to
mellea-contribs-integration-core, pins mellea>=0.3.2 explicitly, and
rewrites every import + unittest.mock patch target in the test suite
to the new dotted path. The package's top-level __init__.py re-exports
the core symbols so call sites can keep using
`from mellea_contribs._integration_core import X`.
Drops mellea_contribs/mellea-integration-core (and its stale
MIGRATION_GUIDE.md) outright. The grandfather list shrinks by one;
.github/smoke-matrix.json gets its first entry so the daily mellea@main
smoke job picks the package up.
Assisted-by: Claude Opus 4.7 (1M context)
Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Moves mellea_contribs/dspy_backend (legacy src layout) to dspy/ at the
repo root with the disk-matches-wheel namespace layout
(mellea_contribs/dspy/{core,stdlib,backends,formatters,helpers}/). The
distribution renames from mellea-dspy to mellea-contribs-dspy, depends
on mellea-contribs-integration-core via [tool.uv.sources] pointing at
../_integration_core, and re-exports MelleaLM, MelleaBestOfN,
MelleaRefine, and create_reward_fn through the package init so the
public API stays accessible at the top level.
Tests and examples come along with imports rewritten from mellea_dspy
to mellea_contribs.dspy and from mellea_integration to
mellea_contribs._integration_core. Drops the legacy GETTING_STARTED.md
and the per-subpackage uv.lock; a fresh lock is generated under the new
layout.
Assisted-by: Claude Opus 4.7 (1M context)
Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Moves mellea_contribs/crewai_backend to crewai/ at the repo root with the disk-matches-wheel namespace layout. The distribution renames from mellea-crewai to mellea-contribs-crewai and depends on mellea-contribs-integration-core via [tool.uv.sources] pointing at ../_integration_core. Public API (MelleaLLM, CrewAIMessageConverter, CrewAIToolConverter, create_guardrail, create_guardrails) is preserved through re-exports. Tests and examples come along with imports rewritten. The pytest marker list shrinks from 12 custom markers (ollama/llm/slow/requires_gpu/etc.) to the 4 standard tiers; legacy markers map to integration/e2e where applicable. CI timeout is set to 90 minutes via [tool.mellea-contribs.ci] because crewai test runs are heavy. requires-python is capped at <3.14 because crewai's transitive dep chromadb pulls in a pydantic v1 module that's incompatible with Python 3.14+. Lift the cap once chromadb supports 3.14. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Moves mellea_contribs/langchain_backend to langchain/ at the repo root with the disk-matches-wheel namespace layout. The distribution renames from mellea-langchain to mellea-contribs-langchain and depends on mellea-contribs-integration-core via [tool.uv.sources] pointing at ../_integration_core. Public API (MelleaChatModel, MelleaGuardrail, MelleaOutputParser, ValidationResult, LangChainMessageConverter, LangChainToolConverter) is preserved through re-exports. Tests and examples come along with imports rewritten from mellea_langchain to mellea_contribs.langchain and from mellea_integration to mellea_contribs._integration_core. A stale sys.path.insert hack in the tools example is removed (no longer needed under the new layout). Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Renames the legacy mellea_contribs/tools_package to agent-utilities/ at the repo root, distributed as mellea-contribs-agent-utilities. The generic `tools` package name (which would collide with any other tools-named package on install) is replaced with the more meaningful mellea_contribs.agent_utilities namespace. The three modules — top_k, double_round_robin, and benchdrift_runner — live under mellea_contribs/agent_utilities/core/. Swaps the build backend from pdm.backend to hatchling for parity with the rest of contribs, drops the per-subpackage [tool.semantic_release] config, renames test/ to tests/, lists the actual authors (@shubhiasthana for the selectors, @shailja-thakur for benchdrift) in OWNERS, and bumps the mellea floor from ==0.3.2 to >=0.3.2 (matching the other framework subpackages). The benchdrift_runner is intentionally not re-exported from the package init — it depends on the optional [robustness] extra (a git-source benchdrift install) that not every consumer wants. Import it explicitly: `from mellea_contribs.agent_utilities.core.benchdrift_runner import …`. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
Adds dspy, crewai, langchain, and agent-utilities to the daily smoke matrix so the mellea@main smoke job picks them up after merge. Drops their legacy paths from the grandfather list; only mellea_contribs/ __init__.py and mellea_contribs/reqlib_package remain for the next migration cycle. Assisted-by: Claude Opus 4.7 (1M context) Signed-off-by: Avinash Balakrishnan <avinash.bala@us.ibm.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Migrates five subpackages out of
mellea_contribs/into top-leveldirectories on the new
<subpkg>/mellea_contribs/<name>/<core mirror>/layout. The release-pipeline overhaul (legacy CI removal, new
release.yml) lands in a follow-up commit on this branch before merge.The
reqlib_packagemigration is also pending and will land in thisbranch.
After this PR plus the pending follow-ups, the legacy
mellea_contribs/flat directory is gone except for
__init__.py(which the final cleanupin this branch removes).
Migrated subpackages
_integration_core/— distributionmellea-contribs-integration-core(wasmellea-integration-core)dspy/— distributionmellea-contribs-dspy(wasmellea-dspy)crewai/—mellea-contribs-crewai(wasmellea-crewai);requires-python = ">=3.11,<3.14"because chromadb is incompatible with 3.14+langchain/—mellea-contribs-langchain(wasmellea-langchain)agent-utilities/—mellea-contribs-agent-utilities(wasmellea-tools); package name made specific instead of baretools.benchdrift_runneris intentionally not re-exported from the package init (depends on the optional[robustness]extra).Each migrated subpackage:
[tool.hatch.build.targets.wheel] packages = ["mellea_contribs"](no sources remap; disk matches wheel)mellea_contribs.<name>.__init__.pymellea-contribs-integration-corevia[tool.uv.sources] path = "../_integration_core"(where applicable —agent-utilitiesis standalone)mellea>=constraints (no bare names per the validate-structure rule)[tool.mellea-contribs.ci]flags (crewaisetstimeout_minutes = 90)Pending on this branch
reqlib_packagetoreqlib/as a single subpackage with optional[legal]/[python-imports]/[grounding-context]extrasrelease.ymlmellea_contribs/__init__.py(final flat-dir cleanup)Verify locally
Known caveats
crewaicannot test on Python 3.14 (chromadb pydantic v1 incompatibility — capped viarequires-python).agent-utilitiesintegration tests require a running mellea session and[robustness]extra (with the benchdrift git source).dspyhas one flaky integration test on first run (test_aforward_with_requirements) — runs against an LLM; pass rate depends on the model.