Repository navigation
feat: add LiteLLM AI gateway integration - #803
RheagalFire wants to merge 2 commits into
Conversation
elijahbenizzy
left a comment
There was a problem hiding this comment.
Thanks! Any chance you could add an E2E example?
|
This PR has been inactive for 16 days after receiving review feedback. Please mark it as ready for review when you have addressed the comments. |
Add examples/integrations/litellm/ with application.py demonstrating both LiteLLMAction (non-streaming) and LiteLLMStreamingAction, following the same structure as the existing Bedrock integration example.
|
@elijahbenizzy added e2e example |
|
This PR has had no activity for 95 days. Feel free to reopen when you are ready to continue. |
elijahbenizzy
left a comment
There was a problem hiding this comment.
Reopening -- you did what I asked and then the stale bot got to it before I did; sorry. Ran both paths with litellm's mock_response, works. Two things before merge, both mirroring what bedrock ships with:
- Tests:
tests/integrations/test_burr_litellm.pycovering kwargs building (api_key/temperature/max_tokens skipped when None, extra_kwargs merge),run_and_updatewriting response+usage,stream_runaccumulation + the finalcompletechunk, andupdate()being a no-op otherwise.mock_response=gives you real-shaped responses with no network -- no MagicMock needed. Plus atest-litellmjob in python-package.yml liketest-bedrock, since the main job won't have litellm installed. - Docs:
docs/reference/integrations/litellm.rst+ toctree entry in index.rst + a line in install.rst -- same shape as bedrock.rst.
Nits, take or leave: response.usage.model_dump() over dict(...); apache-burr[litellm] in requirements.txt.
Elijah
Adds LiteLLM AI gateway integration, giving Burr users access to 100+ LLM providers (OpenAI, Anthropic, Google, Azure, AWS Bedrock, Ollama, Groq, Mistral, and more) through a single Action class.
Changes
burr/integrations/litellm.py- NewLiteLLMAction(non-streaming) andLiteLLMStreamingAction(streaming) following the same_Core+_Base+ Action pattern asburr/integrations/bedrock.py. Useslitellm.completion()withdrop_params=Truefor cross-provider compatibility.pyproject.toml- Added[litellm]optional dependency group (burr[litellm])The codebase already references LiteLLM in TODO comments (
core/parallelism.py,docs/concepts/parallelism.rst) and uses LiteLLM pricing data for cost tracking (telemetry/ui/scripts/token_costs.py). This PR ships the actual integration.How I tested this
Live E2E via Anthropic Claude:
Pin resolution verified:
pip install -e '.[litellm]'resolves cleanly.Notes
_BedrockCore/_BedrockBase/BedrockAction/BedrockStreamingAction)litellmis an optional extra, base install unaffecteddrop_params=Truesilently drops provider-unsupported kwargs so the same action works across OpenAI, Anthropic, Gemini, etc.Checklist