Ruby AI: Interview with Carmine Paolino, Creator of RubyLLM

In this special interview with Carmine Paolino, the creator of RubyLLM, we look at the library’s current state and future, RubyLLM’s advanced capabilities in production applications, and Ruby’s prime position to be the go-to language for AI app development.

• Introducing RubyLLM

• Getting Started with RubyLLM

• Interview with Carmine Paolino

  • RubyLLM

  • The Future of RubyLLM

  • Chat with Work

  • RubyLLM in Production

  • The Ruby AI Community

  • Closing Thoughts

• Additional Resources

• RubyLLM Usage Survey

Introducing RubyLLM

RubyLLM has taken the Ruby community by storm, nearing 3.5 million downloads and consistently being one of the top trending Ruby libraries on Github. Over the past six months, RubyLLM progressed from version 1.0 to version 1.8.2, and has been a staple in the Ruby AI Newsletter since the first edition. Carmine Paolino, the creator of RubyLLM, along with more than 50 contributors, have shipped updates at a lighting pace, building incredible documentation, features, and a vibrant ecosystem that allows Rubyists to easily integrate powerful artificial intelligence capabilities into their applications.

RubyLLM is a Ruby gem for integrating modern Large Language Models in Ruby and Rails applications, providing a single programming interface to build AI chat, vision, audio, image, and document analysis software, with full support for vector embeddings, tool calling, and streaming responses. The complete set of features provide nearly everything you need to take advantage of the full potential of advanced generative AI models:

• Chat: Conversational AI with RubyLLM.chat

• Vision: Analyze images and videos

• Audio: Transcribe and understand speech

• Documents: Extract from PDFs, CSVs, JSON, any file type

• Image generation: Create images with RubyLLM.paint

• Embeddings: Generate embeddings with RubyLLM.embed

• Moderation: Content safety with RubyLLM.moderate

• Tools: Let AI call your Ruby methods

• Structured output: JSON schemas that just work

• Streaming: Real-time responses with blocks

• Rails: ActiveRecord integration with acts_as_chat

• Async: Fiber-based concurrency

• Model registry: 500+ models with capability detection and pricing

• Providers: OpenAI, Anthropic, Gemini, VertexAI, Bedrock, DeepSeek, Mistral, Ollama, OpenRouter, Perplexity, GPUStack, and any OpenAI-compatible API

RubyLLM’s creation has fostered a vibrant community, powering over 100 open source applications, 20 Ruby gems, numerous startups, and a growing ecosystem of extensions including:

RubyLLM::MCP - Created by Patrick Vice, RubyLLM::MCP is a Ruby client for the Model Context Protocol (MCP), designed to work seamlessly with RubyLLM. The gem enables Ruby applications to connect to MCP servers and use their tools, resources and prompts as part of LLM conversations. [Documentation]

RubyLLM::Schema - Created by Daniel Friis, RubyLLM::Schema is a Ruby DSL for creating JSON schemas with a clean, Rails-inspired API, allowing you to define structured data schemas for LLM function calling or structured outputs.

RubyLLM::Template - Also created by Daniel Friis, RubyLLM::Template provides a sytem for organizing your prompts in folders with separate files for system, user, assistant, and schema messages, with ERB support for context variables and Ruby logic.

AI Agents - Created by Chatwoot, AI Agents is an agent framework on top of RubyLLM that enables developers to create sophisticated multi-agent AI workflows. The SDK allows you to build specialized AI agents with multi-agent orchestration that can collaborate, use tools, and seamlessly transfer conversations with shared context to solve complex tasks.

PromptEngine - Created by Avi Flombaum, PromptEngine is a Rails Engine for managing your LLM prompts without having to hardcode them in your application files, providing version control, A/B testing, and a user interface to optimize your prompts without having to deploy code changes.

Getting Started with RubyLLM

With the latest version of RubyLLM, it has never been easier to build and deploy an AI-powered application in minutes. First generate a new Rails application:

rails new chatbot

Add the RubyLLM gem to your Gemfile:

gem "ruby_llm"

Update your bundled gems:

bundle install

Run the RubyLLM installation generator in your application’s directory:

rails generate ruby_llm:install

Run the migrations installed by the RubyLLM generator:

rails db:migrate

Add the openai_api_key to your Rails’ credentials:

rails credentials:edit --environment=development

Load the Rails console and verify that the installation is working:

chat = RubyLLM.chat.ask('Hello!')
puts chat.content
#=> "Hello! How can I assist you today?"

If successful, you can now run the new Chat UI generator:

rails generate ruby_llm:chat_ui

Run your development server, visit http://localhost:3000/chats, start a new chat, and watch the response stream in!

In less than five minutes, you have a working AI chatbot with RubyLLM. Incredible!

Interview with Carmine Paolino

This week I had the privilege of interviewing Carmine Paolino, the creator of RubyLLM, a Ruby and Rails library that provides a unified interface for working with Large Language Models. RubyLLM allows you to build powerful interactive chatbots, multimodal agentic workflows, and intelligent information retrieval applications with a simple Ruby DSL. Carmine is also the founder of Chat with Work, an AI-powered assistant that connects to your company's knowledge bases to answer questions and instantly find relevant documents. Previously, he was the cofounder of Freshflow, an AI solution for retailers to reduce food waste, along with running the Floppy Disco artist collective, DJing as Crimson Lake, and a hobbyist photographer. With 20 years of experience in information technology, artificial intelligence, and data science, Carmine is constantly bringing his deep domain expertise to the Ruby AI ecosystem for the benefit of everyone.

So here it is! Presenting Carmine on:

RubyLLM

Congratulations on RubyLLM being Ruby’s top trending library on Github and surpassing 3 million downloads on RubyGems! For those that aren’t familiar, what can you tell us about RubyLLM? What makes RubyLLM special? What was the inspiration and motivation behind RubyLLM?

Carmine: Thank you so much! RubyLLM is the "One Beautiful API, One Person, One Machine" for AI development in Ruby. What makes it special is that it provides a single, elegant interface for working with every major LLM provider - whether you're using OpenAI's GPT, Anthropic's Claude, Google's Gemini, or even your local Ollama models.

The inspiration came from my frustration while building Chat with Work. I looked at the existing libraries and they were all broken in fundamental ways. They require many lines of code just to say "Hello" to an LLM - and that's from their first tutorial! The official SDKs aren't much better. While we live in a world where we need to switch to the latest models quickly, each provider has its own client with different APIs, different response formats, and different conventions. It's exhausting!

Now that AI development has shifted from training models to using APIs, and therefore to product development, we need clean abstractions at the right level, not complexity theater. So I built RubyLLM following five core principles: Simple should be simple, complex should be possible; Models and providers are commodities; Convention over configuration; Progressive disclosure; and One API, One Person, One Machine.

With RubyLLM, you can say hello in one line: RubyLLM.chat.ask "Hello!". But you can also build complex multi-agent systems, switch between models mid-conversation, analyze any file type (images, PDFs, videos, audio), generate structured output with schemas, and use tools to let AI call your Ruby code. We support 11+ providers and 620+ models, all through one consistent, beautiful API. And we do it with just three dependencies: Faraday, Zeitwerk, and Marcel. No bloat, no complexity theater - just clean Ruby code that makes sense.

The Future of RubyLLM

What is your vision for RubyLLM? What upcoming features are you most excited about, and are there any aspirational capabilities you hope to see in the library in the future? How can the community help?

Carmine: I want to make RubyLLM the best library for building LLM-based applications in any language. I actually think we're already there - there's nothing quite like RubyLLM in Python, JavaScript, or any other ecosystem in terms of developer experience and beauty of the API. But we can go much further.

I don't want to make promises about exact timelines, but here's what I'm working on:

First, I want to streamline the Tool interface even further. Next, I’d like to add a proper Agent interface that encapsulates autonomous agents in a class for people who prefer that interface to progressive disclosure. I'm also looking at changing some of the .with_ methods to be more intuitive, and aligning our callback methods (.on_) with ActiveRecord's naming scheme using before_ and after_.

For Rails integration, I want to go even deeper with convention over configuration. Imagine tool calls and tool results being tightly integrated into views, so you can build AI features the same way you build CRUD apps.

I'm also planning to record a Rails 8-style demo video showing how you can go from rails new to a production AI app in minutes - similar to DHH's famous blog-in-15-minutes video that launched Rails.

I'd also love to work on a book about building AI applications with RubyLLM and Rails as soon as I’ll have the time. There's so much to share about RubyLLM, prompt engineering, tool design, and architectural patterns that I've learned and still learning from building Chat with Work.

On the technical side, we are going to have more provider features - prompt caching for Anthropic, tighter integration with thinking models like o1 and o3, OpenAI's new Responses API for transcription, OpenTelemetry support for observability, and evaluation frameworks. We're also adding new providers like xAI and Azure.

The community can help by triaging issues, responding to discussions, writing documentation, fixing bugs, and contributing features - as long as they fit the RubyLLM philosophy. The best contributions are the ones that capture the style and philosophy of the library. They understand that we're not trying to be LangChain - we're trying to be the Rails of AI development. After my EuRuKo talk drops, which has a whole section on philosophy, I'll post a dedicated philosophy guide.

Chat with Work

What led you to create Chat with Work? What have you learned about the current state of AI from building the company and where do you see the industry heading? What does the tech stack look like, and how do you leverage RubyLLM to ship AI-powered features?

Carmine: Last year at Freshflow, I had an employee ask me about details from a client meeting. I searched everywhere in our Notion but couldn't find it. Turns out the information was buried in a long thread in our #sales Slack channel - they only knew about it because they happened to be following that conversation. That was my lightbulb moment: we need a better way to search and interact with workplace knowledge, and LLMs can help a lot with that.

But Chat with Work evolved far beyond just search. It's become more like Claude Code for your workplace. Yes, it searches across all your tools, but it also helps you write new documents with context from your company's knowledge, compose emails that match your tone and style, draft Slack messages with the right context, and even generate reports pulling from multiple sources. It's an AI assistant that truly understands your work context.

The tech stack is intentionally simple and powerful: Rails 8 at the core, RubyLLM for all AI interactions, the full Async stack (Async::Cable for WebSockets, Async::Job for background processing, Falcon as the web server), Kamal for deployment, PostgreSQL with pgvector for embeddings, all running on Hetzner VPSs. Our entire monitoring, logging, and backup infrastructure is open source and self-hosted. It's incredible what you can build as one person with one machine when you reject complexity and cloud provider lock-in. I'm not spending a fortune on AWS or Azure - just solid, simple infrastructure that works.

Building Chat with Work has taught me so much about the current state of AI. The market is absolutely booming, and it's incredibly competitive with new products launching daily. New techniques get adopted at lightning speed - when Claude Code came out a few months ago, it sparked a major pivot in how I thought about the product.

But the biggest lesson is that AI development has fundamentally shifted. It's no longer about training models - that's reserved for companies with millions to spend on a single training run. For the rest of us, AI is about building products people love using APIs. And who better than the Ruby community knows how to build products people love? We built Basecamp, GitHub, Shopify, Airbnb's frontend - we know how to ship.

RubyLLM in Production

What is your favorite use of RubyLLM in Chat with Work? What are the best examples of RubyLLM use cases and integrations you’ve seen so far?

Carmine: In Chat with Work, it's all about excellent tool design and crafting precise prompts. The way I've structured tools is similar to how Claude Code works, but instead of operating on your local files, it operates on your Google Drive, Slack, Notion, and other work accounts. Each tool has carefully crafted descriptions that help the LLM understand not just what it can do, but when and why to use it.

The beauty of RubyLLM is that complex workflows become simple. For example, Chat with Work can search across multiple data sources in parallel, synthesize the results, and generate a report - all in a few lines of Ruby code using our Async support. Try doing that elegantly with any other library or programming language!

In the wild, I've seen absolutely incredible uses of RubyLLM. One team is using it to run city government simulations. Another built an AI nutritionist. Another is generating personalized lesson content for K-12 students. One of my favorites is a company that went from zero to proof-of-concept in ONE day, and had it in beta testing within a week. Others told me "RubyLLM is pretty much the 'devise' of this generation."

From the survey I conducted, I found companies using RubyLLM in production with 100,000+ users. When I added up just the companies that responded, over half a million people are using RubyLLM-powered features today! We're seeing it at startups like Zipchat and Startup Jobs, established companies like Chatwoot, and even creative agencies. The breadth is amazing.

The Ruby AI Community

You’ve written about how Asynchronous Ruby is the future of Ruby AI, do you have anything to add to that? Is there anything about Ruby’s async capabilities or the Ruby language that needs to improve to support AI development? What AI tooling, technologies, or libraries would you like to see built by the Ruby AI community?

Carmine: Async is criminally underrated in the Ruby world, but it's absolutely essential for AI applications. Here's why: LLM communication is long running and I/O bound. When you're waiting for tokens from an API, your CPU is doing nothing.

With traditional worker-based job queues, if you have 100 concurrent chats, you need 100 workers. Your 101st user has to wait. Each worker is sitting there doing nothing while waiting for the next token. It's wasteful and doesn't scale.

With Async::Job, those 100 chats become 100 fibers that efficiently share resources. Your 101st user doesn’t have to wait because the worker pool gets resized automatically. Because of this and how efficient are Fibers at scheduling and I/O bound operations, you can handle thousands of concurrent conversations on a single machine. No connection pool exhaustion (within reason!), no worker limits, just efficient resource usage.

I got the feeling people are still scared to try async. Let me be crystal clear: if you're building LLM applications and using a background job queue with max workers, you're shooting yourself in the foot. Use Async::Job (with Redis for production).

I'd love to see or make an async worker pool added to Solid Queue. Since Solid Queue already has Mission Control and is integrated into Rails, this would make async mainstream. I've talked with Rosa Gutiérrez about this, and it seems technically feasible. This could be a game-changer for Rails scalability.

We need more support for Samuel Williams' work on async - he's doing incredibly important work for Ruby's future. I'd also love to see more appreciation for Andrew Kane's data science gems. He's single-handedly created torch.rb, polars-ruby, and dozens of others. These gems alone won't convince the ML world to switch from Python, but RubyLLM plus Rails could convince developers to try Ruby for AI apps, and then these other gems would make them stay forever.

Closing Thoughts

What skill sets and disciplines do you think Rubyists should pursue to level up their careers and expertise in the age of AI? Is there anything else exciting you would like to share?

Carmine: Start by actually using AI tools as a power user. Play with ChatGPT, Claude, and Gemini. Understand their personalities, their strengths, their quirks. Then implement one feature in your app with RubyLLM - I guarantee you'll get hooked and want to add more.

Learn to treat each model as having its own personality. Get good at crafting system prompts and designing tools. The difference between a mediocre AI feature and a great one often comes down to prompt engineering and tool design.

Ignore most of the "get rich quick" or "here's 10 magic prompts" content out there. It's noise. Build your expertise by testing things yourself and studying patterns in successful AI apps you actually enjoy using. Focus on what makes your product great, not on following AI hype cycles.

The beautiful thing is you don't need to study the underlying math anymore - that was my generation's burden when I was doing ML research. You don't need to understand transformers or attention mechanisms in detail. You need to first understand your users and build great products.

I'm incredibly excited about what's coming. I'll be giving the keynote at SF Ruby 2025 and RubyConf Thailand 2026! There are other conferences I'll be speaking at that I'll announce soon. My EuRuKo 2025 talk will be online shortly, and I think it'll fire up a lot of people about Ruby's potential in AI.

I'm also planning to record a demo video in the style of Rails' famous screencasts, showing how to build a complete AI application from scratch. I hope it'll convince developers that Ruby isn't just relevant for AI apps - we have the best tools in the industry for building them. With Rails, RubyLLM, and the Async stack, we can build things that would require entire teams in other ecosystems.

This is our moment. Python owned AI when it was about training models. But now that AI development is about building great products with APIs, Ruby has a real shot at becoming the go-to language for AI app development. Go ship some AI apps!

Additional Resources

RubyLLM is a Ruby gem for integrating modern Large Language Models in Ruby and Rails applications, providing a single programming interface to build AI chat, vision, audio, image, and document analysis software, with full support for vector embeddings, tool calling, and streaming responses. The gem features complete documentation, a growing ecosystem, an LLM model registry, and the source code is available on Github.

Carmine Paolino is the creator of RubyLLM and regularly posts updates about library developments on his personal website, LinkedIn, and X/Twitter account.

Chat with Work is an AI workplace assistant that searches across all your work tools simultaneously. It connects to platforms like Slack, Google Drive, and internal knowledge stores to answer questions in plain language, eliminating time wasted searching for information across scattered apps. Ask questions in plain language and get answers from all your work tools at once.

Euruko (European Ruby Conference) is an annual conference about the Ruby programming language with an informal atmosphere and lots of opportunities to listen, learn, and talk with developers from all around the world. The conference included a talk by Carmine Paolino on RubyLLM: Making AI Development Beautiful Again, and will be available online shortly.

RubyConf Tailand is a two-day conference in Bangkok, Thailand featuring 16 speakers sharing real-world Ruby experiences, best practices, and cutting-edge techniques on the Ruby programming language. The conference will feature a keynote presentation by Carmine Paolino on RubyLLM.

The San Francisco Ruby Conference is a three-day event (November 19th to the 21st) celebrating Ruby's ecosystem and future, featuring talks on open source/AI tooling, Ruby at scale, CTO panels, and startup demos, organized by the SF Ruby Meetup group. Day 2 of the conference will begin with a keynote presentation by Carmine Paolino on RubyLLM.

RubyLLM Usage Survey

Are you using RubyLLM in production or an open source application? Please take a moment to help Carmine make the library better by completing a short usage survey.

That’s all for now! Stay tuned for the next edition of The Ruby AI Newsletter. Be sure to reach out if you have any stories, content, jobs, or events you would like to see featured.