Evaluating AI chatbots with LangSmith

Did you set up your chatbot once and never check again whether it still gives good answers? That is exactly the problem many businesses underestimate. LLM models change constantly: new versions, new strengths, sometimes surprising weaknesses. What worked flawlessly yesterday can break with the next model update, without anyone noticing.

In this article we show how we at ServasBot systematically evaluate every change to the chatbot. As our tool we use LangSmith, a platform for observing, testing and improving LLM applications. You get a step-by-step guide with real Python code you can follow along with directly.

What is LangSmith?

LangSmith is an observability and evaluation platform for LLM applications, built by LangChain. It offers:

• Tracing: A complete record of every step in your LLM pipeline. Inputs, outputs, intermediate steps, latency and token usage.
• Datasets and experiments: Curated test datasets on which you can benchmark different models and prompts.
• Evaluators: Automatic scoring of response quality, rule-based or via LLM-as-a-judge.
• Annotation Queues: Structured human feedback on chatbot responses.
• Online evaluations: Continuous quality monitoring on production traces.

The typical workflow looks like this:

1. Collect traces in production
2. Route problematic responses to annotation queues for manual review
3. Add reviewed examples to datasets
4. Run experiments and compare models or prompts
5. Roll out improvements to production and confirm them with online evaluations

This cycle is the core of running a chatbot professionally, and exactly what happens behind the scenes at ServasBot when we continuously optimise a chatbot for a customer.

Step 1: Setting up LangSmith

Create an account at smith.langchain.com (via GitHub, Discord or email). In the Settings area, generate an API key. Important: the key is shown only once, so store it safely.

A word on the model used in the examples. ServasBot runs on Mistral by default, a European provider, with OpenAI available as an option. Because Mistral offers an OpenAI-compatible endpoint, the same client code works for both providers, with only base_url and the model name changing. That is exactly what makes the model comparison in step 5 so straightforward later on.

pip install -U langsmith openai

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="ls_..."
export LANGSMITH_PROJECT="servasbot-evaluation"
export MISTRAL_API_KEY="..."
export OPENAI_API_KEY="sk-..."

Step 2: Enabling tracing

Tracing is the foundation of everything. LangSmith records every step of your LLM pipeline, from the incoming user input to the final answer, including all intermediate steps such as tool calls or retrieval steps.

import os

from openai import OpenAI
from langsmith.wrappers import wrap_openai
from langsmith import traceable

# Mistral via the OpenAI-compatible endpoint.
# wrap_openai logs every call to LangSmith automatically.
mistral = wrap_openai(OpenAI(
    base_url="https://api.mistral.ai/v1",
    api_key=os.environ["MISTRAL_API_KEY"],
))

SYSTEM_PROMPT = (
    "You are a friendly customer service assistant "
    "for an Austrian small business. Answer questions "
    "briefly, precisely and in German. "
    "If you are not sure, say so openly."
)

@traceable  # The function is recorded as a complete trace
def servasbot_answer(question: str, model: str = "mistral-medium-latest") -> str:
    response = mistral.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
    )
    return response.choices[0].message.content

if __name__ == "__main__":
    print(servasbot_answer("Was sind eure Öffnungszeiten?"))

After running this, the trace appears automatically in LangSmith in the servasbot-evaluation project. You see the exact system prompt, the user's question, the model's answer, as well as latency and token usage.

If you prefer the native Mistral SDK, the @traceable decorator gets you there too. LangSmith then expects the provider and model as metadata, for example @traceable(run_type="llm", metadata={"ls_provider": "mistral", "ls_model_name": "mistral-medium-latest"}).

A trace contains spans, meaning hierarchical execution steps. For a simple chatbot there are two: the outer servasbot_answer function and the actual model call. For more complex applications such as RAG chatbots with a knowledge base, the retrieval step and the documents it found are added, which makes all the difference when debugging.

Multi-tenant: separating traces by customer and industry

Anyone running a chatbot service for multiple customers (as ServasBot does for driving schools, tourism businesses and car dealerships) needs tenant isolation in the traces. Without metadata, you later end up with one large pile of traces, with no way to spot industry-specific failure patterns.

The solution: attach tenant_id and industry as metadata to every trace.

from langsmith import traceable

@traceable(metadata={"tenant_id": tenant_id, "industry": industry})
def chat_handler(message: str, tenant_id: str, industry: str):
    # ... chatbot logic
    return servasbot_answer(message)

Alternatively, metadata can also be set at runtime via RunnableConfig if the values are only known once the request comes in. With this tagging discipline in place, you can later define filters such as “tourism traces with negative user feedback only” or “all driving-school traces from the past month”. That is exactly what makes online evaluation worthwhile in multi-tenant setups.

Step 3: Building the Golden Dataset

A Golden Dataset is a collection of test questions with expected answers. It is the benchmark against which every model and prompt variant is measured.

How many examples do you need? Even 10 to 50 well-chosen examples deliver enormous value. Coverage matters more than quantity: typical enquiries, edge cases and common sources of error.

3a. Creating a dataset in the LangSmith GUI

The simplest way in is straight through the LangSmith interface, no code required:

1. Expand Datasets & Testing in the left-hand navigation
2. Click “+ New Dataset”, give it a name (e.g. ServasBot-Goldstandard-v1) and enter a description
3. Open the dataset, click “+ Add Example” for each test case: type in the Input (the user's question) and the Output (the expected reference answer)
4. Save, done

3b. The more powerful way: “Add to Dataset” straight from a trace

In practice, a Golden Dataset rarely grows through manual typing alone. The faster route: you spot an interesting or faulty trace in production and pull it in directly:

1. Open the trace in the trace view (e.g. a real customer enquiry)
2. Click “Add to Dataset” in the top right
3. Select the target dataset (ServasBot-Goldstandard-v1)
4. Correct the reference answer if needed, one click, no code

This workflow is especially valuable because real user questions make the dataset more robust over time than made-up ones do. At ServasBot, examples from production traces flow automatically into the review queue and from there into the dataset.

3c. Bulk import via code (optional, one-off)

For the initial seeding from an existing FAQ database or a CSV source, the Python route is practical. It runs once and can be checked into Git:

from langsmith import Client

client = Client()

dataset_name = "ServasBot-Goldstandard-v1"
dataset = client.create_dataset(
    dataset_name,
    description="Goldstandard-Testfälle für ServasBot KMU-Chatbots",
)

client.create_examples(
    dataset_id=dataset.id,
    examples=[
        {
            "inputs": {"question": "Was sind eure Öffnungszeiten?"},
            "outputs": {"answer": "Wir sind Mo bis Fr von 8 bis 18 Uhr für Sie da."},
        },
        {
            "inputs": {"question": "Wie kann ich einen Termin vereinbaren?"},
            "outputs": {"answer": "Telefonisch, per E-Mail oder über unser Kontaktformular."},
        },
        {
            "inputs": {"question": "Liefert ihr auch nach Wien?"},
            "outputs": {"answer": "Ja, wir liefern österreichweit, auch nach Wien."},
        },
        {
            "inputs": {"question": "Kannst du mir Aktien empfehlen?"},
            "outputs": {"answer": "Das ist nicht mein Fachgebiet. Ich helfe bei Fragen zu unserem Unternehmen."},
        },
        {
            "inputs": {"question": "Sprecht ihr auch Englisch?"},
            "outputs": {"answer": "Yes, we also speak English. Feel free to ask in English."},
        },
    ],
)

After the initial seeding via code, the dataset is typically maintained through the GUI, using the “Add to Dataset” button from traces and the “+ Add Example” dialog. That is the real day-to-day workflow.

Categories for a solid Golden Dataset

Step 4: Defining evaluators

Evaluators are functions that automatically score chatbot responses. LangSmith supports two main approaches: code-based (deterministic) and LLM-as-a-judge (semantic).

Code-based evaluators

Fast and 100% reproducible, ideal for structural checks:

def antwort_nicht_leer(outputs: dict, reference_outputs: dict) -> bool:
    """Checks whether a response exists at all."""
    return bool(outputs.get("response", "").strip())

def antwort_nicht_zu_lang(outputs: dict, reference_outputs: dict) -> bool:
    """Response should be at most twice as long as the reference answer."""
    actual = len(outputs.get("response", ""))
    reference = len(reference_outputs.get("answer", ""))
    return actual < 2 * reference if reference > 0 else actual < 500

def keine_halluzinierten_links(outputs: dict, reference_outputs: dict) -> bool:
    """Checks that the response contains no URLs."""
    import re
    response = outputs.get("response", "")
    return not bool(re.search(r'https?://', response))

LLM-as-a-judge (semantic)

An LLM scores the quality of the response against a rubric prompt. Similar to a human reviewer, but scalable. In the MT-Bench study by Zheng et al. (NeurIPS 2023), GPT-4 as a judge reached over 80% agreement with human raters. That is exactly the level of agreement also observed between humans themselves.

import openai
from langsmith.wrappers import wrap_openai

eval_client = wrap_openai(openai.OpenAI())

def korrektheit(inputs: dict, outputs: dict, reference_outputs: dict) -> bool:
    """LLM-as-a-judge: is the response factually correct?"""
    prompt = f"""You are a quality reviewer for chatbot responses.

User's question:
{inputs['question']}

Expected answer (gold standard):
{reference_outputs['answer']}

Actual chatbot response:
{outputs['response']}

Is the actual response factually correct and helpful in the sense of the expected answer?
Reply with ONLY: CORRECT or INCORRECT"""

    result = eval_client.chat.completions.create(
        model="gpt-4o-mini",
        temperature=0,
        messages=[{"role": "user", "content": prompt}],
    ).choices[0].message.content.strip()

    return result == "CORRECT"

In the LangSmith UI: Alternatively, you can configure the same LLM-as-judge directly in the interface. Evaluators → “+ New Evaluator” → template LLM-as-Judge → adjust the prompt template (with placeholders for inputs, outputs, reference_outputs) → choose a model → define the output schema (e.g. boolean correct). No Python needed, the scoring runs inside LangSmith.

Golden rules for good evaluators

1. Prefer binary or low-granularity scores. “Correct/incorrect” is more reliable than a 1-to-10 scale.
2. Use chain-of-thought in the evaluator prompt. Explanations improve consistency.
3. Build in few-shot examples. Show the LLM judge concrete examples of good and bad.
4. Calibrate against human feedback. Collect manual corrections and compare them with the evaluator.
5. Use a separate model for evaluation. The same model that generated the response should not be the one scoring it.

Step 5: Experiments and model comparison

Now for the core of it: testing different models against the same Golden Dataset. Because both Mistral and OpenAI can be addressed through an OpenAI-compatible client, the two target functions differ only in base_url and model name.

import os

from openai import OpenAI
from langsmith.wrappers import wrap_openai
from langsmith import evaluate

mistral = wrap_openai(OpenAI(
    base_url="https://api.mistral.ai/v1",
    api_key=os.environ["MISTRAL_API_KEY"],
))
openai_client = wrap_openai(OpenAI())  # uses OPENAI_API_KEY

def _answer(client, model: str, question: str) -> dict:
    response = client.chat.completions.create(
        model=model,
        temperature=0,
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": question},
        ],
    )
    return {"response": response.choices[0].message.content}

def servasbot_mistral(inputs: dict) -> dict:
    return _answer(mistral, "mistral-medium-latest", inputs["question"])

def servasbot_gpt(inputs: dict) -> dict:
    return _answer(openai_client, "gpt-4o-mini", inputs["question"])

EVALUATORS = [antwort_nicht_leer, antwort_nicht_zu_lang, korrektheit]

# Experiment 1: the current production setup as the baseline
evaluate(
    servasbot_mistral,
    data="ServasBot-Goldstandard-v1",
    evaluators=EVALUATORS,
    experiment_prefix="mistral-medium-baseline",
)

# Experiment 2: the candidate, against the same dataset
evaluate(
    servasbot_gpt,
    data="ServasBot-Goldstandard-v1",
    evaluators=EVALUATORS,
    experiment_prefix="gpt-4o-mini-kandidat",
)

Comparing experiments: In LangSmith you select multiple experiments and click Compare. The side-by-side comparison view shows the responses of both models next to each other for every test case, with the evaluator scores colour-coded underneath.

• 🟢 Green: improvement over the baseline
• 🔴 Red: regression, meaning a decline
• Grey: no significant change

The point of this view: an average across all test cases can improve while one important individual case breaks. The row view shows exactly that, an aggregated score hides it.

Why re-evaluate with every LLM update?

This is the central question for anyone running chatbots in production. The short answer: because LLM updates are not guaranteed improvements, they are changes.

The classic example: the silent regression

A chatbot correctly answers questions in English. After a model update, it suddenly always answers in English, even when the user asks in German. Without systematic evaluation, this only comes to light once customers complain. With a Golden Dataset and automated experiments in LangSmith, it comes to light before the update goes live.

Most small-business chatbots run without evaluation. You can tell, because six months in they answer worse than on the day they went live. Nobody notices, simply because nobody measures. Customers rarely complain, they just go to the competitor.

The trap of the “-latest” aliases

Providers offer convenient aliases: mistral-medium-latest, mistral-small-latest and the like. If those sit in your production code, you get new model versions automatically, without a single deploy. That is convenient and exactly why it is risky, because your chatbot's behaviour can change while you have touched nothing.

The clean approach: pin the versioned model ID in production (Mistral issues dated identifiers such as mistral-medium-2604 for this) and only move to a new version after an experiment on the Golden Dataset. Then you decide when the behaviour changes.

When do you need to re-evaluate?

• With every model update within the same provider
• With a model change (e.g. OpenAI → Anthropic Claude, GPT → Mistral), see switching chatbot providers
• With prompt changes (a new system prompt, new instructions)
• With changes to the knowledge base (new company information, changed prices)
• With new use cases (a new area, a new language)
• Regularly in production (at least monthly)

At ServasBot, evaluation is not a one-off event, it is part of the ongoing service. Every change to the chatbot system is tested against the Golden Dataset before it goes live for a customer.

This effort is precisely why many chatbot providers do not do evaluation at all. The sale is over quickly, keeping the quality up for years costs time permanently.

Step 6: Online evaluation in production

Offline evaluation is important, but it has a gap: it only tests the questions you have already thought of. Real users are more creative. That is why you need online evaluation, meaning automatic scoring on live traffic.

Online evaluators are configured directly in the LangSmith UI:

1. On the left, Tracing Projects → select your project
2. Top right, “+ New” → “New Evaluator”
3. Name the evaluator (e.g. “Production tone check”)
4. Set a filter: all traces, only those with negative user feedback, or specific metadata tags
5. Configure the sampling rate (e.g. 0.2 for 20% of traces, to control cost)
6. Define the evaluator logic: LLM-as-judge or custom code

What do we monitor in production at ServasBot?

• Hallucination check: Does the bot answer with facts that are not in its knowledge base?
• Out-of-scope detection: Does the bot try to answer questions it is not equipped for?
• Tone quality: Does the bot stay friendly and professional?
• Response length: Do responses suddenly become too long or too short?
• Language detection: Does the bot answer in the right language?

Step 7: Annotation queues for human feedback

Not everything can be judged by an algorithm. For fine nuances (is the answer really helpful? Does the tone fit the brand?) you need human feedback. LangSmith offers Annotation Queues for this: traces are placed in a queue where they can be reviewed in a structured way.

Particularly powerful: Automation Rules can send traces into the queue automatically, for example all traces with negative user feedback, all with a poor online evaluator score, or a random 5% sample of all traces. The reviewed traces can then be pulled into the Golden Dataset, so the dataset grows with real production cases.

In the LangSmith UI: On the left, Annotation Queues → “+ New Queue” → define a name and rubric items (e.g. Correctness as pass/fail, Tone quality on a 1–5 scale, Notes as free text). Then open the queue: you see one trace after another with the chatbot's response and the rubric fields to fill in. Click through, score, next.

Step 8: Prompt versioning with the LangSmith Prompt Hub

System prompts are the heart of every LLM chatbot. If you change them without versioning, you lose traceability. The Prompt Hub in LangSmith solves that:

• Every change to the prompt is stored as a new commit
• You can set tags: production, staging, v2-test
• Your code always references the tag, not a hardcoded version
• You move the tag to a newer commit without changing the app code

from langsmith import Client

ls_client = Client()

# Pull the prompt (always the currently tagged production version)
prompt = ls_client.pull_prompt("servasbot/kmu-chatbot:production")

# Use it in your code
messages = prompt.invoke({
    "company_name": "Muster GmbH",
    "opening_hours": "Mon-Fri 8am-6pm",
}).to_messages()

Step 9: The LangSmith CLI and skills for coding agents

In 2026, LangChain released two new tools that make it possible to work with traces directly from the terminal and from coding agents like Claude Code: the LangSmith CLI and three LangSmith skills (trace, dataset, evaluator). With these, a coding agent no longer just has access to the code, it has access to actual behaviour in production.

The effect is measurable. According to LangChain's own evaluation, Claude Code's success rate on LangSmith tasks jumps from 17% to 92% once the skills are installed. Note: this is an in-house benchmark from the vendor, not an independent study. The order of magnitude is plausible, though, because an agent that can read traces debugs problems against reality, while an agent without trace access is guessing.

Installation

# Install the LangSmith CLI
curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh

# Authenticate with your LangSmith API key
langsmith auth login

# Install the skills globally for Claude Code
npx skills add langchain-ai/langsmith-skills \
    --agent claude-code --skill '*' --yes --global

Three skills get pulled: trace, dataset, evaluator. That gives Claude Code direct terminal access to LangSmith from within the repo. A typical use case: “Look at the last 50 traces with negative feedback, identify the most common failure patterns, propose correction examples for the Golden Dataset.”

At ServasBot, this is the bridge between tracing data and automated code optimisation. The agent sees what the bot actually answers, not just what is in the code.

Practical roadmap: an evaluation setup in 4 phases

Theory is nice, but how do you actually set this up? Here is the order I recommend for a new multi-tenant setup:

Phase 1: Install the CLI and skills (≈ 30 minutes)

See step 9. This prepares the coding agent to work with real trace data.

Phase 2: Tracing audit and metadata patches (1 to 2 hours)

Check existing coverage: are all relevant functions decorated with @traceable? Are tenant_id and industry being passed as metadata? If not, retrofit and deploy. Without clean metadata, every later step is worthless.

Phase 3: Set up online evaluators (≈ 1 hour)

In the LangSmith UI, under Tracing Projects → Rules, set up three evaluators:

• GDPR compliance (LLM-as-judge, 100% sample, filter environment=production): checks whether responses handle personal data correctly and that no cross-tenant leaks occur.
• Industry tone (LLM-as-judge, 20% sample, grouped by industry): checks tone of voice per industry (formal for car dealerships, friendly for tourism, matter-of-fact for driving schools).
• Tool correctness (code-based, 100% sample): deterministically validates the schema and parameters of every tool call.

Important: filter all evaluators to environment=production. Otherwise dev traffic eats into the budget.

Phase 4: Annotation queue (≈ 30 minutes to set up)

Set up a queue with the following filter:

feedback.thumbs_down = true OR GDPR_Compliance_Score < 0.7

Workflow: every few days, go through 10 to 20 traces and label the correct answer. These become the ground truth for the offline eval suite. Reviewer profile: a domain expert first (or myself, per tenant), optionally the customer for industry-specific feedback.

Follow-up (not day one)

• Build an offline eval suite per tenant from annotated traces (rule of thumb: 50 annotated traces per tenant for a first usable dataset)
• Integrate a CI/CD gate into deployment (the eval suite must be green before release)
• Run an insights agent over production traces to cluster intent per industry

The complete evaluation cycle

In summary, quality assurance at ServasBot runs in this cycle:

1. Production: traces are collected during normal operation.
2. Online evaluation: notable answers are flagged automatically.
3. Annotation queue: a human reviews and corrects the flagged cases.
4. Golden Dataset: the reviewed examples become the new benchmark.
5. Offline experiment: a new model or prompt runs against that benchmark.
6. Comparison and decision: a regression means the change is discarded, an improvement means it ships.
7. Rollout: online evaluation confirms in production that the improvement holds. Then the round starts again.

In numbers: initial setup (Golden Dataset, evaluators, first experiments) takes roughly 1 to 2 days per chatbot. Ongoing operation: 2 to 4 hours per week for trace review, working the queue, new dataset entries and experiments after every model or prompt update.

This cycle is the difference between a chatbot that was set up once and a chatbot that keeps getting better. And it is day-to-day business, not project work.

Anyone trying to do this alongside their day-to-day operations does not keep it up for long.

Bonus: CI/CD integration for developer teams

For teams with CI/CD, the evaluation cycle can be integrated directly into the pipeline. A simple pytest example:

# test_chatbot_quality.py
from langsmith import evaluate

def test_korrektheit_mindest_score():
    """Chatbot must reach at least 80% correctness on the gold standard."""
    results = evaluate(
        servasbot_mistral,
        data="ServasBot-Goldstandard-v1",
        evaluators=[korrektheit],
        experiment_prefix="ci-test",
    )

    # ExperimentResults is iterable and every row carries the results of
    # all evaluators. Filter by key so that additional evaluators do not
    # skew the number.
    scores = [
        result.score
        for row in results
        for result in row["evaluation_results"]["results"]
        if result.key == "korrektheit" and result.score is not None
    ]

    assert scores, "No scores returned, is the evaluator running?"

    avg_score = sum(scores) / len(scores)
    assert avg_score >= 0.8, (
        f"Correctness only {avg_score:.1%}, at least 80% required!"
    )

If a model update pushes correctness below 80%, the CI build fails and the rollout is stopped. No regression reaches production without being seen.

What you can do right now

Whether you are building your first chatbot or want to improve an existing one, these steps help right away:

1. Create a LangSmith account and enable tracing (15 minutes)
2. Create 10 to 20 Golden Dataset examples for your use case
3. Implement a simple code evaluator (e.g. response not empty)
4. Run a first experiment on the dataset
5. At the next model update: run a second experiment and compare

Sources and further reading

• Zheng et al. (NeurIPS 2023): Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — the peer-reviewed primary source for the over-80% agreement between GPT-4 as a judge and human raters.
• LangChain: The agent improvement loop starts with a trace — a vendor blog post, useful for the workflow concepts.
• LangChain: LangSmith CLI & Skills — the source of the 17%-to-92% figure. An in-house benchmark from the vendor, not independently validated.
• LangSmith: Trace Mistral applications — the official way to instrument Mistral calls.
• LangSmith documentation — the official API and UI documentation.

Related articles

• Why your AI chatbot needs ongoing maintenance
• What an AI chatbot really costs
• GDPR and AI: what SMEs must know
• EU hosting vs. US cloud
• Switching chatbot providers: what if the AI gets pricier?