Skip to content

Add agentic training data synthesizer#1287

Merged
tushuhei merged 3 commits into
mainfrom
design_agentic_training_pipeline
Jul 20, 2026
Merged

Add agentic training data synthesizer#1287
tushuhei merged 3 commits into
mainfrom
design_agentic_training_pipeline

Conversation

@tushuhei

@tushuhei tushuhei commented Jul 17, 2026

Copy link
Copy Markdown
Member

This PR introduces a three-stage agentic synthesis architecture to dynamically generate, align, and polish candidate sentences to supplement the training dataset.
This pipeline is highly experimental and only focused on Japanese phrase-based segmentation for now.

Pipeline Workflow

  1. Intent Understanding Agent (extract_intent_target & parse_direct_input): Parses direct CLI patterns (--input="いよいよ/はじまる") or GitHub bug report descriptions (--issue=468) to identify targeted segments (expected_split) and confirm reproduction against live production models (ja.json).
  2. Example Generator Agent (generate_oversample_candidates): Synthesizes varied natural language corpus sentences strictly enclosing the targeted break intervals.
  3. Boundary Alignment & Linguistic Expert Review:
    • align_to_base_parser_splits: Retains existing parser segmentations across surrounding outside context to preserve baseline unigram training integrity while strictly changing only the target interval.
    • prune_linguistic_anomalies: Audits syntax fluency and polishes improper boundary fusions around independent vocabulary words without splitting across enclosing conversational quotation brackets (「」).

Usage

# Direct targeted string segmentation
uv run scripts/synthesize_samples.py --input="いよいよ/はじまる" --lang ja --outfile staging_ja.txt

# Automated bug parameters derived straight from a public issue report
uv run scripts/synthesize_samples.py --issue=468 --lang ja --outfile staging_468.txt

@tushuhei
tushuhei requested a review from kojiishi July 17, 2026 23:34

@kojiishi kojiishi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tushuhei
tushuhei merged commit 2f81069 into main Jul 20, 2026
43 checks passed
@tushuhei
tushuhei deleted the design_agentic_training_pipeline branch July 20, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants