Skip to content
Hebbian-RoboticsPublic

Latest commit

 

History

19 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Hebbian Robot-Use Agents (RUA)

Build your own robot-use agents. Give an LLM a goal, camera views and robot tools, and let it observe, act and adjust. We're open sourcing the parts of our stack that make that loop easy to build. More coming soon.

Included

For a physical SO-101, follow the source quickstart to calibrate the arm with so101-calibrate and start RUA with --profile auto.

Start an agent project with uv:

uv init my-robot-agent
cd my-robot-agent
uv add "hebbian-rua[openai,anthropic,gemini]"

Already have a uv project? Start with uv add. Choose just the extras you need. With the published RUA 0.1.2 wheel, use an explicit profile and LeRobot calibration to connect a physical arm. Prebuilt wheels support Linux x86-64 (glibc 2.39+) and macOS 14+ on Apple Silicon. On Windows, use WSL2.

See getting started and the Python examples to try it, and the example applications for example toy projects.

See connecting a simulator to try in simulation. To watch your cameras, follow the LiveKit web preview setup.

Build an agent

RUA supplies robot tools, camera observations and execution. You bring the model and the loop.

Here's an example agent loop. The complete example includes the connection, lease and arm setup, plus loops for Claude and Gemini:

from hebbian_rua.client.llm.openai import OpenAIRobotTools

robot_tools = OpenAIRobotTools(robot)
messages = [{"role": "user", "content": "Wave hello"}]
while True:
    messages.append(await robot_tools.observe())
    response = await client.responses.create(
        model=model,
        instructions=instructions,
        tools=robot_tools.tools,
        input=messages,
    )
    turn = await robot_tools.execute(response)
    messages.extend(turn.messages)
    if turn.should_stop or not turn.results:
        break

The model chooses tools, RUA executes them, and fresh camera frames and robot state come back for the next turn. You can change the prompt, model and loop. Use AnthropicRobotTools from hebbian_rua.client.llm.anthropic or GeminiRobotTools from hebbian_rua.client.llm.gemini with the corresponding SDK. See the Python examples. Choose what reaches the model with output formatters.

RUA compiles action sequences automatically. The Robot Use Compiler checks and translates the requested actions. The Robot Use Runtime prepares each step against live robot state and admits it for execution. Advanced users can compile and inspect before running. See how execution works.

Hardware setup, camera tools and recording utilities have their own guides: SO-101, LiveKit Cameras and MCAP Tools. RUA handles their integration with your robot and agent.

Coming soon

  • TypeScript and Rust SDKs
  • CLIs
  • Browser viewer and manual controls
  • More agents
  • More examples and integrations
  • Recording playback and analysis
  • Policy and training tools

Research and related projects

Some reading and projects around robot-use agents:

Contributing and license

See contributing. Code is MIT; models and dependencies keep their own licenses. See third-party notices.

Releases

Packages

Contributors

Languages