Architecting Semantic Retrieval for Reader and Editorial Discovery
I designed and implemented a system that allows readers to search for poetry by emotional intent, translating subjective human states into machine-understandable vectors and back into meaningful discovery.
WHAT
It’s a full-stack semantic retrieval system for San Antonio Review that translates emotionally complex queries into structured embeddings, retrieves relevant poems from a vector database, and generates thoughtful reading recommendations.
The system combines:
-design of a controlled vocabulary of emotions and themes
-embedding-based indexing
-vector similarity search (Pinecone)
-human-in-the-loop poetry classifications
-layered safety constraints
WHY
Editors previously spent a lot of time relying on memory, browsing through published poetry, or relying on traditional search to curate packages for the site and new issues. They needed a richer way to find poetry that didn’t rely on tagging and keywords.
Readers found their way to poetry through social media and editorial curation. That’s fine for new content, but poetry doesn’t expire. We want readers to explore the back catalog and fall in love with new authors. To do that, they needed better ways to engage.
IMPACT
We’re increasing reader engagement by helping them connect with poetry in a more meaningful way, through emotional vectors. And the publication builds a stronger relationship with them because they trust us to connect them with content that matters to them.
Editors have a more efficient curation tool. It takes less time to put together new packages for the site, for new issues, and for social media. SAR is able to continue to operate as a lean organization without sacrificing editorial quality, and we free up editors to do other things, like contributor outreach and working with writers to enhance pieces.
LINKS TO WORK
Live product at San Antonio Review: https://sanantonioreview.org/poetry/
GitHub Repository:
https://github.com/mistycripps
DOCUMENTATION
This is design documentation that describes the bot persona, how the system works at a high and detailed level, and the data design. I talk about these in more detail in the articles in the the Read About the Process section below.
READ ABOUT THE PROCESS
Mood as a tool for poetry discovery
Poetry discovery is difficult for the same reasons content discovery has always been difficult: We publish a lot of poetry at San Antonio Review, enough that it gets lost in the shuffle. Readers find poems through social media, search engines, the homepage, or browsing by issue. That’s fine for new work, but poetry is evergreen. A poem published three years ago is as relevant as one published last week. We don’t have the editorial resources to constantly resurface older work, and “most popular” lists that go stale have little value for us. The real problem is that poetry resists simple…
Keep readingHow I Designed Poetry Embeddings for Semantic Search: Combining Human Judgment with LLM Classification
My dog died recently. It was sudden, but not unexpected. He was 15 and suffering from advanced kidney disease. When I was considering approaches to structuring and classifying data for the Poetry Explorer, I began to think about scenarios like this. I want to be able to find poetry by saying, “My best fuzzy friend went over the rainbow bridge last week. I’m heartbroken, but also relieved that he is no longer suffering.” Because that is how I felt. Conflicted. Ambivalent. Sad, but also a little relieved that he wasn’t miserable anymore. This is the kind of complexity I want…
Keep readingDesigning the Character of Poetry Explorer: Voice, Boundaries, and Safety
Now that I had the data schema designed, I started thinking about how the Poetry Explorer would interact with users. This was the most challenging part of the project. I’ve spent years creating personas, but always as representations of system users, not as representations of the system itself. And they represented people, not non-human entities. Poetry Explorer isn’t a corporate customer service bot. It isn’t a proxy for San Antonio Review. It isn’t transactional in the “I want to buy tickets to a concert” sense. So what is it? It isn’t the writing that’s difficult; it’s designing a character that’s…
Keep readingHow Do You Go From Design Vision to Shipped Product Without Dedicated Engineering Resources?
I designed the Poetry Explorer; now it was time to build it. As I’ve covered before, I’m a product designer, not a developer. I’m not completely lacking in technical skills. I can read Python well enough to understand what’s happening. I can write HTML/CSS/JavaScript/PHP to various degrees, but I can’t architect a production-grade backend from scratch without assistance or implement vector search from scratch. So I didn’t write anything from scratch. I used LLMs, docs, and developer community resources to help me and iterated until it worked. This wasn’t because I’m trying to become a developer. That wasn’t my goal.…
Keep reading
VAQUITA DESIGN 