works with openai and anthropic

[andyllegrand]

Used litellm to implement streaming with openai api calls. Could not get this to work with anthropic but the original anthropic code is still present and supported.

Streaming can be a little finicky, in order to get the formatted text to work each time a new chunk arrives the previously printed text is erased and replaced with the new text. Sometimes a line is not erased properly, but i don't think this is a big issue.

[andyllegrand]

Also I'm not sure if the way i delete lines from the terminal will work properly on windows, please test this merging

[erkinalp]

Review Summary

The streaming implementation is a valuable addition that improves the user experience. Here are the key findings and recommendations:

Windows Compatibility ⚠️

The terminal line clearing implementation needs cross-platform handling:

def clear_lines(num_lines):
    for _ in range(num_lines):
        sys.stdout.write('\033[F')
        sys.stdout.write('\033[K')
    sys.stdout.flush()

Recommendation: Use colorama for Windows compatibility.

Memory Management 🔄

Current implementation needs bounds:

full_res = []
async for chunk in response:
    full_res.append(chunk)

Recommendations:

• Add chunk limits
• Implement cleanup on errors
• Add timeout handling

Response Printer 📝

Suggested improvements:

• Buffer size limits
• Token state tracking
• Error recovery
• Line length constraints

Progress Indication 🔄

Unify progress indication:

• Add connection feedback
• Consistent status updates
• Maintain critical operation spinners

Anthropic Integration 🔧

Current implementation:

res_content = ''
for content_item in response.content:
    if isinstance(content_item, TextBlock):
        res_content += content_item.text

Recommendations:

• More efficient content handling
• Investigate streaming options
• Consider simulated streaming

The changes are well-structured and improve the user experience. With the suggested enhancements for cross-platform support, memory management, and consistent progress indication, it will be even more robust.

Approving with suggestions for improvements.

---

_This review was conducted by Devin and revised by Erkin Alp. View the Devin run