Back to projects
In progress

Claude-Powered Home Robot

A physical robot built from scratch — voice, eyes, smart home control, and semantic memory

Hero project — fully functional, running in my house

AI Python Hardware Claude Raspberry Pi FastAPI React

How it works

Sancho — Claude-powered home robot Voice in and out run fully offline on the Pi. FastAPI is the hub: it owns the hardware, dispatches tools, and streams state to the dashboard. Any Python file dropped in the tools directory is auto-registered and becomes something Claude can call. DeviceCoreIntelligenceActuation audio expression WebSocket tool-use loop dispatch recall · store Voice I/O faster-whisper · STTPiper · TTS Animated eyes expressive display React dashboard live chat · remote FastAPI core tool dispatchWebSocket streaming Claude SDK Sonnet — reasoning Tool registry drop in a .py file →auto-registered Semantic memory ChromaDB + Postgreson the homelab Home Assistant lights · devices WhatsApp · Web messages · search
Voice in and out run fully offline on the Pi. FastAPI is the hub: it owns the hardware, dispatches tools, and streams state to the dashboard. Any Python file dropped in the tools directory is auto-registered and becomes something Claude can call.

Built a fully autonomous physical robot from scratch at home. Named Sancho (after Sancho Panza from Don Quixote) — a grounded, practical companion to the AI. Powered by the Claude SDK with voice interaction, animated eyes, smart home control, and persistent semantic memory. Not a demo. It lives in my house and actually runs.

What it does

  • Animated eyes — expressive display that reacts to conversation state
  • Voice I/O — faster-whisper for local speech-to-text, Piper TTS for offline voice synthesis (no cloud, no latency)
  • Real-time chat dashboard — WebSocket-based React + Vite interface for monitoring and interacting with the robot remotely
  • Smart home control — natively integrated with Home Assistant to control lights, devices, and automations
  • WhatsApp — send and receive messages through the robot
  • Web search — live information retrieval when needed
  • Music playback — on-device audio control
  • Semantic memory — ChromaDB + PostgreSQL hosted on homelab for persistent context across sessions
  • Tool auto-discovery — any Python file dropped in the tools directory is automatically registered and available to Claude

Tech stack

  • Hardware: Raspberry Pi 5, camera module, sensors, animated display, Osoyoo Flexirover chassis, reverse-engineered Roomba battery
  • AI: Claude Sonnet (Anthropic SDK), local models via homelab for lightweight inference
  • Backend: FastAPI — handles all hardware interfacing, tool dispatch, and WebSocket streaming
  • Frontend: React + Vite — real-time chat and robot status dashboard
  • Memory: ChromaDB for semantic vector search, PostgreSQL for structured session data
  • Local AI: faster-whisper (STT), Piper (TTS) — fully offline voice pipeline

Why I built it

The challenge of connecting software intelligence to physical action — with real constraints. Not a cloud demo. Every feature has to deal with hardware failure, latency, battery life, and the messiness of real environments. Sancho is where I push against the limits of what the Claude SDK can do when the stakes are physical.

← Back to projects