I open-sourced Sienna — my own AI music player with a personality. Ask her for "rainy day jazz" or "songs like this one" and she DJs a continuous YouTube mix that keeps playing, track after track, with minimal repeats, ducking the volume under her own voice when she has something to say and picking the music back up when she's done. And "her" is the right word: Sienna is a push-to-talk desk companion — a little ESP32 device with a mic, a speaker, a camera, and LEDs, paired with a server-side agent that has persistent memory and a personality she edits herself. The music player is my favorite part of her, and the part with the most engineering hiding under a simple promise: the mix never stops, and it never gets stuck in a loop of the same twenty songs.
A Jukebox That Never Stops and Rarely Repeats
The brief for Sienna's jukebox was one sentence: behave like a good radio DJ with infinite shelves. You name an artist, a song, or just a vibe; she searches YouTube, shuffles many results into a queue, and autoplays track after track (via yt-dlp and ffmpeg, decoded server-side) until you tell her to stop. Everything else in the design falls out of taking "until you tell her to stop" literally. A track that fails to decode gets skipped. An empty search retries forever on an exponential backoff. A burst of failures forces a fresh live search that bypasses — and repairs — the warm cache. Her speaking a reply mid-song doesn't kill the session; the mix waits out her voice and resumes. The loop only ends on an explicit stop.
The "minimal repeats" half is the more interesting problem, because YouTube actively fights you on it. The same song exists as dozens of uploads under slightly different titles, so a naive history filter does nothing. Sienna records every track that actually played and excludes the last 250 of them from every future queue — matching on video ID, on normalized title, and on the title's parenthetical-stripped base, so "Song (Instrumental Mix)" and a re-upload of "Song" both count as the same song. Queues are re-checked at play time too, catching the second upload of something that slipped into the same batch under a different ID.
When the Mix Runs Dry, Ask a Better Question
Play one artist long enough and the no-repeat window eventually swallows everything the search can find. The obvious move — search the same query again — just returns the same exhausted pool. So instead of repeating itself, the jukebox escalates. First it tries a deeper search, three times the normal size. If that's still dry, it asks a language model to brainstorm about ten alternative queries in the same spirit — related artists, sub-genres, adjacent vibes and eras — then shuffles them and tries each until one surfaces a song that hasn't been played. If the model call fails, static query variants keep things moving, because the one unforgivable failure mode for a music player is silence while a retry loop thinks. This is my favorite kind of LLM integration: not a chatbot bolted onto a product, but a model quietly answering the one question a search index can't — "what should I listen to next, given all of this?"
Single-song requests get their own trick. Ask for one specific track and it carries a hidden continuation — a "songs like this" search. The moment that song ends or you skip past it, the session graduates: the continuation becomes the query, and "next" means "something like it," not another upload of the same thing. The whole loop rests on a monotonic generation token — every control action bumps it, and the running loop re-checks it after every await — so a stale loop unwinding from a preempted track can never fight the one that replaced it. The music also deliberately runs outside the agent's run mutex: the mix must outlive the conversational turn that started it.
The Personality Behind the Turntable
What makes Sienna feel like a companion rather than an appliance is that the DJ remembers you. Conversations, saved facts, and her own personality live in MongoDB and are recalled into every prompt — and the personality is self-editable: after a quiet delay she reflects on recent interactions and may save a memory, refine how she talks, or (if you've enabled autonomy) decide to speak up on her own. Tell her once that you like your jazz instrumental and your mornings quiet, and that context is simply there the next time you hold the button.
The rest of her senses round out the desk-companion premise: a camera tool for looking around, a light sensor, once-a-day ambient behaviors gated behind waking hours and an autonomy toggle. You talk to her by holding a physical push-to-talk button — no wake word, no always-open mic — speech goes through ElevenLabs Scribe transcription to the agent (a Claude or Gemini tool-use loop), and her reply streams back as expressive TTS, sentence-chunked for fast first audio — the same first-audio latency lesson as Murder Mystery's suspects. The agent drives everything through tools: the jukebox, the LEDs, timers, volume, the camera. Asking for music and asking her to dim the lights are the same kind of sentence.
The Hardware on the Desk
The device itself is deliberately hackable: a Freenove ESP32-S3 camera board, an I2S microphone, a small amplifier and speaker, an LDR, a couple of LEDs, and a pushbutton, with full-duplex 16 kHz audio. The firmware stays intentionally thin — a non-blocking I/O client speaking a small JSON-and-binary-frames protocol over a WebSocket — while the server does all the heavy lifting: transcription, TTS, the agent loop, music decoding, volume. That split is why the music features live entirely server-side, and why the firmware fits a single cooperative loop(). A web dashboard handles provisioning over Web Bluetooth and gives you live camera, audio, and LED panels plus a card view of her messages, memories, and personality history.
Every integration is optional and degrades gracefully: ElevenLabs for voice, Anthropic or Gemini for reasoning, MongoDB for memory. Bring the keys you have; the rest of the feature set steps aside cleanly rather than crashing.
Build Your Own
Sienna is open source at github.com/keysforthewin/sienna — server, firmware, pin map, and an annotated .env.example that doubles as the feature list. If you've ever wanted a music player that takes requests out loud, digs deeper instead of repeating itself, and occasionally has opinions about what you're listening to, the parts cost less than a nice dinner and the code is all there. The README's quick start goes from docker compose up to a provisioned device in an evening.
For the patterns underneath — agent tool loops, streaming voice pipelines, graceful-degradation architecture — the field notes on building conversational AI agents and AI-powered web applications are the long-form background.