Boris Smus

interaction engineering

Alloy: a local-first AI workbench

AI model vendors usually ship their own desktop apps, and aside from coding agents, these tend to be the ways in which I interface with AIs the most. But these apps are sticky by design: each vendor wants you to stay with them and continue to use them. As a result, I tend to have very lumpy usage patterns, focusing on one model at a time to the detriment of the others. This leads to an incomplete understanding of the ever-shifting AI landscape, with models constantly leapfrogging one another on the leaderboards.

To remedy this, I built my own web-based desktop app which is model agnostic. Beyond easy access to Gemini, Grok, Claude, and GPT, I had a few additional goals in mind:

  1. Local-first: all conversations are stored as plaintext
  2. Build an orchestrator to gain intuition around agents, tools, skills, etc,
  3. Play around with new interaction patterns beyond the usual chat-and-wait

The result of this is Alloy, which I began in December and have used daily since then. With it, I've been able to more easily see how good each model is at using tools, agent skills, etc. Alloy also serves as a personal playground for experimenting with new interaction patterns: freewriting with a quiet AI collaborator, delegating to sub-agents mid-conversation, and setting up triggers that monitor the world on your behalf.

Alloy simple screenshot

Continued →

Book: The Score by C. Thi Nguyen

Notes →

Book: In Patagonia by Bruce Chatwin

Notes →

Book: Children of Time by Adrian Tchaikovsky

Notes →

Local e-ink handwriting recognition with on-device VLMs

For decades, I've carried a Field Notes notebook and a pen. I mainly used them to capture ideas on the go, but it would also be great to sketch out a diagram, or to journal a little bit, especially while traveling.

Fast note-taking via the iPhone's action button has alleviated a lot of my need for quickly capturing ideas. But nothing can replace pen and paper for long form stream-of-consciousness writing or diagramming.

I wanted to give my writing a digital life alongside the rest of my notes. So a year ago, I bought an A5 e-ink writer called Supernote. I really like it so far: it's a good size, input latency is reasonable, and the overall writing experience is fine. The device provides real-time text recognition on-device and a modest cloud syncing service. I've been using their unofficial API to sync notes and bring them into my Obsidian inbox. But then something happened...

Continued →

Book: The Crisis of Narration by Byung-Chul Han

Notes →

Book: Livewired by David Eagleman

Notes →

Book: Monkey by Arthur Waley

Notes →

The pursuit of frictionless capture

The most fruitful moments for contemplation are often the least conducive to capture. I've been working on reducing friction in capturing thoughts, feelings, and ideas wherever I may be and whatever I may be doing. At the same time, I want to stay in the moment and not get distracted by the act of capturing itself.

In addition, I want to retain control over all of this data. It should be stored in human-friendly plaintext format. It should touch as few servers as possible and come to rest on a surface that I control.

I am excited about all three of the capture methods I'm about to share with you. Three questions to whet your appetite:

  1. What if you could freewrite with pen on paper and have the salient bits magically show up in my digital note corpus?
  2. What if you could use your locked smartphone to type a note without ever being distracted by its contents?
  3. What if you could dictate notes while walking, running, or riding a bike without even requiring a smartphone?

To motivate this, watch this quick summary of the scenarios above, and read on for some tech details for how it all works.

Continued →

Book: Hard to Be a God by The Strugatskys

Notes →

Book: Piranesi by Susanne Clark

Notes →

Book: Secret of our Success by Joseph Henrich

Notes →

Book: Math from Three to Seven by A. K. Zvonkin

Notes →

Book: Abundance by Ezra Klein and Derek Thompson

Notes →

Book: VALIS by Phillip K. Dick

Notes →

Book: The Maniac by Benjamín Labatut

Notes →

Large and sometimes Oppressive Language Models (LOLs)

Authoritarian governments have latched onto open-source LLMs like Llama to craft their own models, complete with censorship rules. In places like Russia and China, these censors manifest as an abrupt cutoff, where a stream of text is replaced with an uncanny canned response like "I'd better keep quiet". In this post I'm interested in probing for subtler, more insidious manipulations potentially present in language models controlled by authoritarian governments. How might these warped filters influence entire populations? Are they already shaping our collective understanding in ways we barely notice?

Continued →

Book: A Peace to End All Peace by David Fromkin

Notes →

Horsing around with invention.cards

This is the first in a series of vignettes based on observations I captured while creating invention.cards, a visual chronology of science and discovery. In this series I hope to explore bits of the history of science and technology I found fascinating ("that's funny...") while reading and digesting Asimov's chronology and also examine the limitations of source material. First, let's take a really thin slice of human ingenuity: horse-related inventions and discoveries.

Continued →