Nilo

Nilo is an open backend platform for autonomous social robots with voice, vision, memory, personality, and embodied behavior.

What Nilo is

Nilo Server is the brain that a small social robot talks to. Today it is a production-grade voice session server: a robot (or any compatible device) opens a WebSocket, streams Opus audio, and the server runs voice activity detection, speech recognition, an LLM with tool calling, and text-to-speech, streaming audio back. Tools reach the device through MCP, so the robot's own hardware capabilities are exposed to the agent as callable tools.

Around that core, Nilo is growing a robot domain layer — world state, behaviour, personality, an action executor and a safety policy — so the same server can drive an expressive robot rather than a smart speaker. The direction is described in docs/robot-architecture.md.

Current status

Implemented (in main/nilo-server/, verified against the code):

• WebSocket device sessions with a JSON control channel and binary Opus audio (core/connection.py)

• A Nilo-owned device protocol (robot/protocol/): WebSocket on /nilo/v1/, OTA on /nilo/ota/; routes come from a registry, not from hard-coded paths, and unknown paths are rejected

• HTTP OTA bootstrap that hands devices their WebSocket URL, auth token and firmware updates from data/bin/

• Audio pipeline: Silero VAD, streaming and batch ASR, sentence-chunked streaming TTS, barge-in (abort)

• Provider abstraction with 17 ASR, 21 TTS, 17 LLM, 3 vision-LLM, 4 memory and 3 intent configurations (docs/providers.md)

• Unified tool system: server plugins, device IoT descriptors, device MCP tools, remote MCP endpoints, server-side MCP servers (docs/mcp.md)

• Vision: /mcp/vision/explain runs a vision-LLM over a device camera frame

• Configuration layering (config.yaml → user file → NILO_* environment), HMAC device tokens, structured logging

• Robot domain layer (main/nilo-server/robot/): typed models for identity, capabilities, connection, telemetry and world state; an async registry of connected robots that survives reconnects; a state-store abstraction; a bounded in-process event bus; and capability discovery over the device MCP tool channel (docs/robot-domain.md)

• Test suite, lint (Ruff), strict typing for robot/ (mypy), Docker image and Compose file

• Robot simulator (main/nilo-server/robot/simulator/): a complete simulated robot that connects over the real WebSocket route, publishes 16 hardware capabilities as device MCP tools, moves over time, senses a configurable room, renders camera frames, reports telemetry as MCP notifications, injects failures and plays scenarios (docs/robot-simulator.md)

• Device telemetry ingestion (main/nilo-server/robot/telemetry.py): notifications/* frames become world state and typed events instead of a log line

• Robot action layer (main/nilo-server/robot/actions/): ten semantic actions with an explicit lifecycle, a priority queue, resource claims over DRIVE / HEAD / LIFT / DISPLAY / AUDIO / CAMERA so two actions cannot command the same hardware, and await robot.move(...) on top (docs/robot-actions.md)

• Robot world model and behaviour engine (main/nilo-server/robot/state/world.py, main/nilo-server/robot/behavior/): tracked entities with a decay schedule, and deterministic utility-scored autonomy — sixteen behaviours, four autonomy modes, seeded randomness, and python -m robot.behavior explain to ask why it chose what it chose. No LLM is involved in deciding what the robot does (docs/behavior-system.md)

• Robot personality and expressive animation (main/nilo-server/robot/personality/, main/nilo-server/robot/animation/): six stable traits and seven internal control variables that decay towards trait-derived baselines, plus animations defined as YAML data — adding one is adding a file (docs/robot-personality.md, docs/robot-animation.md)

• Robot vision (main/nilo-server/robot/vision/): a snapshot pipeline — capture, decode, detect, track, world model, events — behind five replaceable provider protocols, with normalized 0.0-1.0 coordinates, face identity that keeps embeddings in one place, frame retention that defaults to keeping nothing, and latency metrics. OpenCV and YOLO are optional and lazily imported; the defaults need nothing installed (docs/robot-vision.md)

• Robot memory (main/nilo-server/robot/memory/): four stores — working, episodic, semantic and person — over SQLite with migrations behind an interface Postgres can implement, optional embeddings (the robot works without them), token-budgeted retrieval that explains every item it returns, consolidation that tracks provenance and never lets a model overwrite a fact it is less sure about, and real delete operations (docs/robot-memory.md)

• Robot management API and development dashboard (main/nilo-server/robot/api/): its own aiohttp app, its own port, its own token. Inspect state, world, behaviour scores, memory, tools and the conversation; drive the robot; stream everything as server-sent events; and a single-file dashboard with a simulator panel that injects people, obstacles, cliffs, touch and low battery through the same doors real perception uses. Control endpoints are gated three ways — a token, a loopback rule and a rate limit — and every one of them submits a typed action to the same executor and the same safety policy as a behaviour's own command (docs/robot-api.md)

• Robot safety policy (main/nilo-server/robot/safety/): deterministic admission against configurable limits and live sensor state, typed rejections (never silent clamping), a sticky emergency-stop latch, and a supervisory watchdog on its own thread. A policy filter, not a guarantee — firmware owns every guarantee (docs/safety-model.md)

• Robot agent (main/nilo-server/robot/agent/): the LLM seam. Fourteen semantic tools with strict argument validation and live safety bounds in their schemas, four permission classes behind a configurable policy, a runtime context with no telemetry history in it, interruptible conversation, one arbitration path for proactive speech, and a fallback that keeps the robot working when the model is not there. It wraps the existing provider architecture rather than adding a second one, and every test runs against a scripted model (docs/robot-agent.md)

• Robot voice loop (main/nilo-server/robot/voice/): microphone to speaker with a robot in the middle. Five audio states with checked transitions and an event per change, barge-in that keeps the conversation, one arbitrated mouth with four speech priorities, and a face that follows the conversation without twitching on every transition. The inherited VAD, ASR, TTS and Opus pipeline is reused unchanged; the seam is two calls, both of which never raise (docs/robot-voice.md)

• End-to-end integration and observability: ten scenarios that run a real server, a real socket and a real simulated robot with local fakes for ASR, TTS, LLM and vision — startup, conversation, a voice-commanded move, an autonomous greeting, boredom, a cliff, a low battery, a barge-in, a dead model and a reconnect — plus metrics and one correlation id from an utterance to the device's answer (docs/observability.md, docs/testing.md)

Planned: the physical robot firmware. See docs/robot-roadmap.md. What actually runs today, with the test that proves each line: PROJECT_STATUS.md.

Architecture

robot / device ──WebSocket + Opus──▶ nilo-server
                                      ├─ inherited infrastructure (core/)
                                      │    session handler · VAD → ASR → LLM → TTS · tool system · MCP · OTA · vision
                                      └─ Nilo robotics layers (robot/)
                                           protocol registry ✓ · robot registry ✓ · state ✓ · events ✓ · capabilities ✓
                                           telemetry ✓ · simulator ✓ · actions · safety · behaviour · world map · memory

The inherited infrastructure is a stable, well-tested voice pipeline; Nilo adds robotics on top of it through a few explicit seams instead of rewriting it. Details: docs/architecture.md (what runs today) and docs/robot-architecture.md (where it is going).

Quick start

Requirements: Python 3.12, ffmpeg on the PATH, an LLM API key (any OpenAI-compatible endpoint). On Linux install libopus0; macOS and Windows use the bundled Opus libraries in libs/.

git clone https://github.com/AA-Box/Nilo.git nilo
cd nilo/main/nilo-server
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

Download the local speech-recognition model (FunASR SenseVoiceSmall, ~900 MB) — or pick a cloud ASR in the next step and skip this:

modelscope download iic/SenseVoiceSmall model.pt --local-dir models/SenseVoiceSmall

Create your configuration file. It overrides keys from config.yaml; everything else keeps its default:

mkdir -p data && cat > data/.config.yaml <<'YAML'
selected_module:
  LLM: MyLLM
LLM:
  MyLLM:
    type: openai
    url: https://api.openai.com/v1/
    model_name: gpt-4o-mini
    api_key: sk-your-key
YAML

Start the server:

python app.py

The log prints the OTA and WebSocket endpoints. Point a device at http://<host>:8003/nilo/ota/ and it receives the WebSocket URL to connect to, or check the routes yourself:

python ../../scripts/smoke_check.py

More: docs/getting-started.md.

Configuration

Three layers, later wins: main/nilo-server/config.yaml (shipped defaults, fully commented) → the user file (data/.config.yaml, or the file named by NILO_CONFIG) → environment variables NILO_SERVER_HOST, NILO_SERVER_PORT, NILO_HTTP_PORT, NILO_LOG_LEVEL. Providers are chosen in selected_module; each provider block's type names the adapter module. Full reference: docs/configuration.md.

Supported AI providers

Derived from main/nilo-server/core/providers/:

Kind	Adapters
ASR	FunASR (local), FunASR server, sherpa-onnx (local), Vosk (local), OpenAI-compatible (incl. Groq), Doubao, Doubao streaming, Tencent, Aliyun, Aliyun streaming, Aliyun Bailian streaming, Baidu, Xunfei streaming, Qwen3 ASR Flash
TTS	Edge TTS, OpenAI-compatible, Doubao, Huoshan double-stream, Aliyun, Aliyun streaming, Aliyun Bailian streaming, Tencent, Minimax, SiliconFlow CosyVoice, Fish Speech, GPT-SoVITS v2/v3, Index TTS, PaddleSpeech, Xunfei streaming, Coze CN, custom HTTP
LLM	OpenAI-compatible (OpenAI, DeepSeek, Doubao, ChatGLM, LM Studio, Volces gateway…), Ollama, Gemini, Dify, Coze, FastGPT, Langflow, Xinference, Aliyun Bailian apps, Home Assistant conversation
Vision LLM	OpenAI-compatible (ChatGLM-4V, Qwen-VL, Xunfei Spark)
VAD	Silero
Memory	none, local short-term summary, mem0ai, PowerMem, report-only
Intent	function calling, LLM intent classifier, none

Module and class names per adapter: docs/providers.md.

Robot architecture

Robot
├── Perception (audio ✓, vision, sensors)
├── World Model
├── Memory
├── Behavior Engine
├── Agent (LLM + tools ✓)
├── Action Executor ✓
├── Safety ✓ (policy only — firmware owns every guarantee)
└── Device Protocol ✓

The rule that shapes everything: the LLM never controls motors. It requests semantic actions — move, turn, look_at, follow, play_animation, stop — with bounded parameters; the backend admits or rejects each one against a safety policy; the robot's firmware executes trajectories and owns acceleration limits, collision and cliff avoidance, watchdogs and emergency stop.

Backend safety is a second layer only. It rejects and supervises; it never guarantees. Cliff protection, the motor watchdog, current and thermal protection, physical motion bounds, acceleration constraints and a local emergency stop must be implemented independently in firmware, because this process can be killed or stalled mid-motion. Do not connect a machine that can hurt someone and rely on the backend to stop it. See docs/robot-actions.md, docs/robot-architecture.md and docs/safety-model.md.

Development

cd main/nilo-server
pip install -r requirements-dev.txt     # test/lint tooling + the slice the tests need
ruff check .                            # lint
mypy                                    # strict types for robot/
pytest -q                               # tests (heavy ones skip without the full requirements)

Nilo-specific code goes under main/nilo-server/robot/; inherited code under core/ is edited as little as possible. Layout, conventions and where things go: docs/development.md.

Testing

pytest -q in main/nilo-server (the robot, integration and end-to-end suites included); make test, make e2e, make lint, make typecheck from the repository root; scripts/smoke_check.py against a running server. make e2e also writes main/nilo-server/tmp/e2e-report.md: every state transition of all ten scenarios, taken from the running system's own event bus. CI runs lint + mypy, the suite on Python 3.12 with full and with dev-only dependencies, the scenarios on their own, and validates the Compose file. Details: docs/testing.md.

Docker

echo "$(openssl rand -hex 32)" > secrets/nilo_admin_token
docker compose up                       # development: builds from the tree, no model needed
docker compose --profile demo up        # ...and a simulated robot talking to it

make docker-build                       # ghcr.io/aa-box/nilo-server:base and :latest, locally
cd main/nilo-server && docker compose up -d   # deployment: the published image

The Compose file mounts ./data (your config) and the FunASR model file, exposes 8000 (WebSocket) and 8003 (HTTP) and takes NILO_* variables. Deployment options: docs/deployment.md.

Roadmap

Protocol registry and migration (done) → robot skeleton, event bus, simulator (done) → device registry and session adapter (done) → action system and safety policy → semantic action vocabulary → vision and world model → behaviour engine and personality → management API, memory, multi-robot → hardware bring-up. Acceptance criteria per phase: docs/robot-roadmap.md.

Documentation

The architecture on one page, with every flow and every failure mode, is docs/architecture-final.md.

Start at docs/getting-started.md for the server, or docs/robot-getting-started.md to go from git clone to a simulated robot that talks, moves and decides for itself. The full tree: architecture, configuration, development, deployment, protocol, MCP, audio, providers, robot architecture, robot simulator, safety, testing, upstream provenance, branding, migration — all under docs/.

Origins and attribution

Nilo began from open-source infrastructure from xinnan-tech/xiaozhi-esp32-server and has since been adapted toward a general autonomous robotics backend; it no longer shares that project's device protocol or endpoints. The engineering record of the origin is in docs/upstream.md. Nilo is inspired by social robots such as Cozmo but is not affiliated with Anki, Digital Dream Labs or any upstream project.

Licensed under the MIT License; see LICENSE, which retains the original copyright notice.