Chatlas/inspect integration

[gshotwell]

Love the new to_solver method, but I'm not totally clear how it works with structured data.

[cpsievert]

Hey, thanks! We didn't really consider this case for the 1st pass, but I think we could support it by adding a data_model param to to_solver().

[gshotwell]

After using chatlas and inspect for a real project, I think the thing that I really want is to have the integration between chatlas and inspect-ai go the other way. So instead of turning a chat object into a solver we would allow the chat objects to take a solver. The basic idea is that a user might want to do eval-first development where they develop and validate the model using inspect-ai tools and then deploy it as a chatlas object.

I'm not familiar enough with the internals of either library to assess the difficulty of doing this, but it would be awesome if it worked. Also there are more inspect users than chatlas users so from a social perspective it makes sense to think of adding chatlas to the inspect workflow rather than adding inspect to the chatlas workflow.

[cpsievert]

#264 makes it possible to output structured data.

have the integration between chatlas and inspect-ai go the other way.

I haven't thought about this deeply, but my gut reaction is that it'd be a lot of work and based on shaky heuristics.