LLM cleanup canceled

[roblourens]

This is one kind of issue that I keep running into, I have all this raw text and it doesn't get cleaned up at all. There is one cleanup request which was canceled, I don't know why

[meganrogge]

It probably took too long and timed out.

[meganrogge]

What's the longest you'd want to wait for this to come back?

[roblourens]

Long enough that it usually works, max on the order of 5s? Having some progress UX treatment would help. Plus, I'm wondering whether anyone is still looking into updating to a modern small model. 4o-mini is years old and I know we can get better performance. And I don't know whether anyone has really pushed CAPI on the perf of this model because it's not usually on the hot path.

Wispr Flow had a blog post where they said that they aim for 200ms.