It doesn’t have to be a Chatbot

It usually starts the same way. In a leadership meeting, someone said: “Let’s use artificial intelligence!” Heads nod, enthusiasm builds, and before you know it, the room reaches its default conclusion: “Sure – we’ll build a chatbot.” This instinct is understandable. Big language models are powerful, ubiquitous, and wonderful. They promise intuitive access to global knowledge and functionality.

The team drives away and construction begins. Soon, it will be demo time. A polished chat interface appears, accompanied by confident arguments about why it’s different this time. However, at this stage, it usually does not reach real users in real situations, and the evaluation is biased and optimistic. Inevitably, an audience member will ask a custom question, which will irritate the bot. The developers promise to fix this issue, but in most cases, the underlying issue is systemic.

Once a chatbot gets going, initial optimism is often met with user frustration. Here, things get a little personal because over the past weeks, I’ve had to spend some time talking to different chat programs. I tend to delay engaging with providers until the situation becomes unsustainable, and some of these cases have piled up. Smiley chat tools became my last hope before calling the eternal hotline, but:

• After logging into my car insurance company’s website, I asked to explain an unannounced rate increase, only to realize that the chatbot didn’t have access to my pricing data. All you can provide is a hotline number. Oh.
• After a last-minute flight cancellation, I asked the airline’s chatbot why. I politely apologized, since the check-out time was already in the past, it couldn’t help. However, the floor was open for discussion of all other topics.
• On a telecom site, I asked why my cell phone plan suddenly expired. The chatbot responded confidently that it could not comment on contractual matters and referred me to the FAQ. As expected, these were long but irrelevant.

These interactions did not bring me closer to a solution and left me on the other end of the euphoria. Chatbots felt like UFOs. Sitting there, they consumed real estate, response time, and attention, but didn’t add any value.

Let’s move beyond the debate about whether these are intentional dark patterns. The truth is that legacy systems as mentioned above carry a heavy burden of entropy. They come with a lot of unique data, knowledge and context. And the moment you try to combine it with a general purpose MBA, you make two worlds collide. The model needs to understand the context of your product so it can think usefully about your industry. Engineering the right context requires skill and time for continuous evaluation and iteration. Before you get to this point, your data should be ready, but in most organizations, data is scrambled, fragmented, or missing.

In this article, I will summarize the ideas from my book The art of developing artificial intelligence products And my last conversation in Google AI Web Summit and share a more organic and gradual approach to integrating AI into existing products.

Use smaller models for incremental, low-risk AI integration

“When implementing AI, I see more organizations failing by starting too big rather than starting too small.” ( Andrew Ng).

AI integration takes time:

• Your technical team needs to prepare the data and know what technologies and tools are available.
• You need to prototype and iterate to find the sweet spots of AI value in your product and market.
• Users need to calibrate their confidence when moving to new probabilistic experiences.

To adapt to these learning curves, you shouldn’t rush to expose AI — especially open chat functionality — to your users. AI introduces uncertainty and errors into the experience, which most people don’t like.

One effective way to accelerate your AI journey in an architectural context is to use small language models (SLMs), which typically range from a few hundred million to a few billion parameters. It can flexibly integrate with your existing product data and infrastructure, rather than adding additional technology overhead.

How are SLMs trained?

Most SLMs are derived from larger models through Distillation of knowledge. In this setting, the senior model serves as the teacher and the junior model serves as the student. For example, Google’s Gemini was a teacher Gemma 2 and Gemma 3 while Meta’s Llama Behemoth trained her A flock of 4 smaller llama models. Just as a human teacher condenses years of study into clear explanations and structured lessons, a large model distills its vast parameter space into a smaller, denser representation that a student can comprehend. The result is a compact model that retains much of the parameter efficiency but operates with much fewer parameters and significantly lower computational cost.

Using SLMs

One of the main advantages of SLM is the flexibility of its deployment. Unlike LLMs which are often used through external APIs, smaller models can run locally, either on your organization’s infrastructure or directly on the user’s device:

• Local publishing: You can host SLMs on your own servers or within your own cloud environment, while maintaining full control over data, latency, and compliance. This setup is ideal for enterprise applications where sensitive information or regulatory restrictions make third-party APIs impractical.

📈 Local publishing also provides you with flexible opportunities for fine-tuning You collect more data and need to respond to increasing user expectations.

• Publishing to the device via the browserModern browsers have built-in AI capabilities that you can rely on. For example, Chrome integrates Gemini Nano via Embedded AI APIswhile Microsoft Edge includes Phi-4 (see Quick API documentation). Running forms directly in the browser enables low-latency, privacy-preserving use cases such as smart text suggestions, form autofill, or contextual help.

If you would like to learn more about the technical aspects of SLM management, here are some useful resources:

Now let’s move on and see what you can create with SLMs to provide user value and make consistent progress in your AI integration.

Product opportunities for SLMs

SLM shines in focused, well-defined tasks where the context and data are already known — the types of use cases that live deep within existing products. You can think of them as specialized, embedded intelligence rather than general purpose assistants. Let’s review the main opportunity sets they open up in the brown field, as shown in the following opportunity tree.

1. Better product analytics

Before revealing AI features to users, find ways to improve your product from within. Most products already generate a constant stream of unstructured text — supporting chats, help requests, and in-app comments. SLMs can analyze this data in real-time and reveal insights that inform product decisions and immediate user experience. Here are some examples:

• Flag and guide support conversations As they occur, direct technical issues to the appropriate teams.
• Mark volatility signals during the sessionprompting timely interventions.
• Suggest relevant content or actions Based on the user’s current context.
• Detect frequent friction points While the user is still in the flow, not retroactively weeks later.

These internal enablers keep risk low while adding value and giving your team time to learn. It strengthens your database and prepares you for more straightforward, user-facing AI features in the future.

2. Eliminate friction

Next, take a step back and review the UX debt you already have. In the brown space, most products aren’t exactly a designer’s dream. They were designed within the technical and architectural constraints of their time. With AI, we now have the opportunity to lift some of these limitations, reduce friction and create faster, more intuitive experiences.

A good example of this is smart filters on search-based websites like Booking.com. Traditionally, these pages use long lists of checkboxes and categories that attempt to cover every possible user preference. They are cumbersome to design and use, and in the end, many users are unable to find a setting that interests them.

Language-based filtering changes this. Instead of navigating a complex classification, users simply type what they want (for example, “pet-friendly hotels near the beach”), and the form translates it into a structured query behind the scenes.

More broadly, find areas in your product where users need to apply your internal logic — categories, structures, or terms — and replace that with natural language interaction. When users can express their intent directly, you remove a layer of cognitive friction and make the product smarter and friendlier.

3. Increase

With your user experience in disarray, it’s time to think about augmentation — adding small, useful AI capabilities to your product. Instead of reinventing the core experience, look at what users are already doing with your product — the side quests, workarounds, or external tools they rely on to get to their goal. Can focused AI models help them do this faster or smarter?

For example, a travel app could integrate a contextual trip notes generator that summarizes itinerary details or crafts messages for participating travelers. The productivity tool can include a meeting summary builder that summarizes discussions or action items from text notes, without sending the data to the cloud.

These features grow organically from real user behavior and expand the context of your product rather than redefining it.

4. Personalization

Successful personalization is the Holy Grail of AI. It flips the traditional dynamic: Instead of requiring users to learn and adapt to your product, your product now adapts to them like a fitting glove.

When you’re starting out, stay away from ambition – you don’t need a completely adaptable assistant. Instead, make small, low-risk adjustments to what users see, how information is worded, or which options appear first. At the content level, AI can adapt tone and style, such as using concise wording for experts and more explanatory wording for newcomers. At the expertise level, it can create adaptable interfaces. For example, a project management tool can display the most relevant actions (“Create Task,” “Share Update,” “Create Summary”) based on the user’s previous workflow.

⚠️ When personalization goes wrong, it quickly erodes trust. Users feel like they’ve traded their personal data for an experience that doesn’t feel better. Therefore, only offer personalization when your data is ready to support it.

Why does “small” win over time?

Every successful AI feature — whether it’s an analytics enhancement, an easy UX touchpoint, or a custom step in a larger pipeline — strengthens your data foundation and builds your team’s iteration power and AI literacy. It also lays the foundation for larger, more complex applications later on. When your “small” features work reliably, they become reusable components in larger workflows or modular agent systems (see Nvidia research Small language models are the future of agentic AI).

To summarize:

✅ Start small – Preferring gradual improvement over disruption.

✅ Quick experience — Smaller models mean lower cost and faster feedback loops.

✅ Be careful — start internally; Deliver user-facing AI once validated.

✅ Build your repetition muscles – Steady and compound progress outperforms key projects.

Originally published in https://jannalipenkova.substack.com.