DSPy Integration

[Ryzhtus]

Related Issues

• fixes DSPy Integration #1635

Proposed Changes:

This DSPy integration provides generation components:

DSPyGenerator is the internal base class that holds all the core logic: DSPy LM initialization, module creation (Predict/ChainOfThought/ReAct), serialization, and the run(prompt: str) -> {"replies": List[str]} execution path

DSPyChatGenerator — a Haystack @component for quick prototyping with DSPy signatures. Accepts List[ChatMessage], returns List[ChatMessage]. Wraps DSPy modules (Predict, ChainOfThought, ReAct) behind Haystack's chat generator and generator interface.

DSPyChatGenerator just simply extends DSPyGenerator to follow Haystack's ChatGenerator.

I also want to add in either this PR or the next one another component which I would like to call DSPyProgramRunner. The difference will be that DSPyProgramRunner @component loads and executes pre-compiled/optimized DSPy programs in production pipelines. So it can accept dspy.Module for initialization

How did you test it?

• unit tests

Checklist

• I have read the contributors guidelines and the code of conduct
• I have updated the related issue with new insights and changes
• I added unit tests and updated the docstrings
• I've used one of the conventional commit types for my PR title: fix:, feat:, build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

[anakin87]

Hey @Ryzhtus, thanks a lot for the contribution.

Since the original issue is quite open about how the integration should be structured, I'd prefer to align on the high-level design before implementing it:

• How do you envision the overall integration?
• Which components to introduce?
• A draft usage example would also help.

Feel free to share your thoughts in #1635.

I haven't reviewed the code, but one early note: if possible, let's avoid introducing new Generators. We are switching to Chat Generators only, which are more flexible, and we may eventually deprecate legacy Generators.

[Ryzhtus]

@anakin87 Hi, sure, let's move our discussion to #1635. I will leave my ideas about the integration and what I've already submitted there