Building an Enterprise AI Observability Framework

Conventional application monitoring tells you a request took four seconds and returned HTTP 200. For an LLM or retrieval-augmented generation (RAG) application, that is the least informative part of the story. The answer may have been confidently wrong, grounded in the wrong document, unsafe, or far more expensive than last week because a prompt change doubled the context. None of that moves an error rate.

This article sets out a framework for observing LLM and RAG applications in an enterprise: the signals to capture, an architecture that combines application tracing with an AI gateway, a Delta table model for telemetry, cost attribution, offline and online evaluation, and governance. Each section states the pattern, then shows how it lands on Databricks and MLflow using OpenTelemetry-compatible formats, so the design stays portable.

Why LLM applications need more than APM

Latency, throughput and error rates still matter. They are no longer sufficient:

• Non-determinism. The same input can produce different outputs, so a regression is a shift in a distribution, detected by scoring samples over time rather than checking one response.
• Quality is a first-class signal. A fluent, wrong answer is a successful request to every infrastructure metric. Quality must be measured, with scores attached to the traces that carry latency and cost.
• Cost per request varies. Spend follows tokens, which depend on context size, retrieved documents, output length and, in agentic flows, the number of model and tool calls. Averages hide the expensive tail.
• Failures are semantic. Retrieval returns stale chunks, a template drops a variable, a tool gets the wrong arguments, a guardrail redacts the fact the user needed. Every component looks healthy and the request still fails; only the whole request, as one trace, shows why.

The signals to capture

Model every request as a trace, with child spans for retrieval, reranking, prompt construction, each model call, each tool execution and each guardrail check. The OpenTelemetry GenAI semantic conventions provide the shared vocabulary: operation names such as chat, embeddings, retrieval and execute_tool, and attributes such as gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens and gen_ai.conversation.id. They are still at Development status and now live in their own repository, so names change between versions. Pin the version you emit and map names in one place.

Record tokens, not cost: input, output and, where reported, cached input tokens, which are priced differently. Compute cost downstream from an effective-dated price table, so a price change is a data update rather than a redeploy. Capture duration per span and, when streaming, time to first token, which is what users feel. For retrieval, log document ids, scores and k: enough to debug retrieval and run groundedness judges without copying document text into every trace. Store guardrail decisions (policy, stage, action) and user feedback as data linked to the trace id.

A reference architecture

There are two telemetry sources because each sees something the other cannot. The application sees context: retrieved documents, how the prompt was assembled, the user, the session, the prompt version. The gateway sees every model call, including calls from applications nobody instrumented, and is the natural place for authentication, rate limits, usage tracking and payload logging.

Instrumentation in the application

MLflow Tracing instruments Python with a decorator and a context manager, and offers automatic tracing for common LLM SDKs and agent frameworks. It exports to OpenTelemetry collectors, and an MLflow server accepts OTLP traces. The example instruments a RAG call by hand to show what each span should carry. It stores traces in Unity Catalog, calls any OpenAI-compatible endpoint (gateway route, serving endpoint or provider), and leaves search_index as a stand-in for your vector store client.

import os
import time
import uuid

import mlflow
from mlflow.entities import Document, SpanType
from mlflow.entities.trace_location import UnityCatalog
from openai import OpenAI

mlflow.set_tracking_uri("databricks")
mlflow.set_experiment(
    experiment_name="/Shared/support-assistant",
    trace_location=UnityCatalog(
        catalog_name="ai_ops", schema_name="traces", table_prefix="support_assistant"
    ),
)

llm = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ.get("LLM_MODEL", "support-chat")
PROMPT_URI = "prompts:/ai_ops.prompts.support_answer@production"
APP_TAGS = {
    "app_id": "support-assistant",
    "team": "customer-ops",
    "use_case": "support_answers",
    "environment": "prod",
}


def search_index(query: str, k: int) -> list[dict]:
    """Your vector store client: returns dicts with id, text, uri and score."""
    raise NotImplementedError


@mlflow.trace(span_type=SpanType.RETRIEVER)
def retrieve(question: str, k: int = 5) -> list[Document]:
    hits = search_index(question, k)
    mlflow.get_current_active_span().set_attributes({
        "retrieval.k": k,
        "retrieval.returned": len(hits),
        "retrieval.top_score": max((h["score"] for h in hits), default=0.0),
    })
    return [
        Document(id=h["id"], page_content=h["text"],
                 metadata={"doc_uri": h["uri"], "score": h["score"]})
        for h in hits
    ]


@mlflow.trace(name="build_prompt", span_type="PROMPT")
def build_messages(question: str, docs: list[Document], prompt) -> list[dict]:
    context = "\n\n".join(f"[{d.id}] {d.page_content}" for d in docs)
    return [
        {"role": "system", "content": prompt.format(context=context)},
        {"role": "user", "content": question},
    ]


def generate(messages: list[dict]) -> str:
    with mlflow.start_span(name="chat", span_type=SpanType.CHAT_MODEL) as span:
        span.set_inputs({"model": MODEL, "messages": messages})
        started = time.perf_counter()
        first_chunk_at, parts, usage = None, [], None
        stream = llm.chat.completions.create(
            model=MODEL,
            messages=messages,
            temperature=0.2,
            stream=True,
            stream_options={"include_usage": True},  # usage arrives in the last chunk
        )
        for chunk in stream:
            if chunk.usage is not None:
                usage = chunk.usage
            if chunk.choices and chunk.choices[0].delta.content:
                if first_chunk_at is None:
                    first_chunk_at = time.perf_counter()
                parts.append(chunk.choices[0].delta.content)

        span.set_attribute("gen_ai.request.model", MODEL)
        if first_chunk_at is not None:  # seconds, as in the GenAI conventions
            span.set_attribute(
                "gen_ai.response.time_to_first_chunk", first_chunk_at - started
            )
        if usage is not None:  # MLflow rolls this key up into trace totals
            span.set_attribute("mlflow.chat.tokenUsage", {
                "input_tokens": usage.prompt_tokens,
                "output_tokens": usage.completion_tokens,
                "total_tokens": usage.total_tokens,
            })
        answer = "".join(parts)
        span.set_outputs(answer)
        return answer


@mlflow.trace(name="support_answer", span_type=SpanType.CHAIN)
def answer_question(question: str, user_key: str, session_id: str):
    """Returns the answer and the trace id, so the UI can attach feedback."""
    prompt = mlflow.genai.load_prompt(PROMPT_URI)
    mlflow.update_current_trace(
        client_request_id=str(uuid.uuid4()),
        user=user_key,  # a pseudonymous id, never an email address
        session_id=session_id,
        tags={**APP_TAGS, "prompt_version": f"{prompt.name}/{prompt.version}"},
    )
    docs = retrieve(question)
    answer = generate(build_messages(question, docs, prompt))
    return answer, mlflow.get_active_trace_id()

The details that carry most of the value:

• Span types matter. MLflow’s retrieval judges look for RETRIEVER spans whose outputs are documents, and usage recorded under mlflow.chat.tokenUsage (MLflow 3.2 and later) rolls up into trace totals.
• Time to first token is measured in the client, the only place that sees the stream. The include_usage option returns token counts on the final chunk; check that your endpoint supports it.
• Versions. user and session_id on update_current_trace need MLflow 3.11 or later (earlier 3.x releases use the mlflow.trace.user and mlflow.trace.session metadata keys), and Unity Catalog trace storage needs MLflow 3.14 or later with explicit MODIFY and SELECT grants on the trace tables.
• Inputs and outputs are recorded by default, so the question, retrieved text and answer land in the trace. That is a governance decision, covered below.
• Feedback attaches to the trace through mlflow.log_feedback, using the returned trace id and an AssessmentSource of type HUMAN.

Outside Databricks, or alongside an existing OpenTelemetry estate, the same traces can go to a collector:

# Export MLflow traces to an OpenTelemetry collector over OTLP/HTTP.
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://otel-collector.internal:4318/v1/traces"
export OTEL_EXPORTER_OTLP_TRACES_PROTOCOL="http/protobuf"
# Emit gen_ai.* attributes rather than MLflow's own keys (MLflow 3.11 and later).
export MLFLOW_ENABLE_OTEL_GENAI_SEMCONV="true"
# Head-based sampling of new traces; 1.0 keeps every trace.
export MLFLOW_TRACE_SAMPLING_RATIO="0.25"

With an OTLP endpoint configured, traces go to the collector instead of the MLflow tracking server. A dual-export setting sends them to both; its variable name differs between the MLflow and Databricks documentation, so check it for your version.

The AI gateway

Route every model call through a gateway. It is the one place that can enforce authentication and rate limits (queries or tokens per minute, per user or group), apply guardrails such as sensitive-data detection, fail over between model destinations, and log usage for every caller. On Databricks this is Unity Gateway (formerly Mosaic AI Gateway; the earlier gateway for serving endpoints is now documented as legacy). Usage lands in the system.ai_gateway.usage system table, and payloads can be logged to inference tables in Unity Catalog. Feature status moves quickly, so check what is generally available in your region.

Landing telemetry in Delta tables

Binding an MLflow experiment to a Unity Catalog schema stores traces as OpenTelemetry-format Delta tables (<prefix>_otel_spans, <prefix>_otel_logs, <prefix>_otel_metrics and <prefix>_otel_annotations), which Databricks recommends for new and production workloads. Applications that already emit OpenTelemetry can export spans to the workspace OTLP endpoint, naming the target table in a request header. Gateway usage and inference tables arrive in Unity Catalog on their own; inference tables can take up to an hour to receive rows and skip payloads larger than 10 MiB. These raw tables are the system of record; a scheduled job builds the small curated model that dashboards, alerts and cost reports query.

A telemetry data model

Three curated tables cover most questions: one row per request, one row per assessment, and an effective-dated price list.

CREATE SCHEMA IF NOT EXISTS ai_ops.telemetry;

CREATE TABLE IF NOT EXISTS ai_ops.telemetry.llm_requests (
  request_id          STRING    NOT NULL COMMENT 'Client request id, also sent as a tag',
  trace_id            STRING    COMMENT 'Trace id in the raw trace tables',
  event_time          TIMESTAMP NOT NULL,
  app_id              STRING    NOT NULL,
  app_version         STRING,
  environment         STRING    COMMENT 'dev, staging or prod',
  team                STRING    NOT NULL,
  use_case            STRING    NOT NULL,
  user_key            STRING    COMMENT 'Pseudonymous user id, never an email',
  session_id          STRING,
  prompt_name         STRING,
  prompt_version      STRING,
  model_endpoint      STRING,
  model_name          STRING,
  status              STRING    COMMENT 'ok, error or blocked',
  error_type          STRING,
  latency_ms          BIGINT,
  ttft_ms             BIGINT    COMMENT 'Time to first token; streamed calls only',
  input_tokens        BIGINT    COMMENT 'All input tokens, including cached',
  cached_input_tokens BIGINT,
  output_tokens       BIGINT,
  retrieval_k         INT,
  retrieval_top_score DOUBLE,
  guardrail_events    ARRAY<STRUCT<policy: STRING, stage: STRING, action: STRING>>,
  payload_sampled     BOOLEAN   COMMENT 'True when redacted payloads were kept',
  request_payload     VARIANT   COMMENT 'Redacted; set only when sampled',
  response_payload    VARIANT   COMMENT 'Redacted; set only when sampled'
)
CLUSTER BY (app_id, event_time)
COMMENT 'One row per end-user request, curated from traces and gateway logs';

CREATE TABLE IF NOT EXISTS ai_ops.telemetry.llm_assessments (
  request_id    STRING    NOT NULL,
  trace_id      STRING,
  assessed_at   TIMESTAMP NOT NULL,
  name          STRING    NOT NULL COMMENT 'correctness, groundedness, user_helpful',
  source_type   STRING    NOT NULL COMMENT 'LLM_JUDGE, HUMAN or CODE',
  source_id     STRING    COMMENT 'Judge and version, reviewer group or check',
  value_numeric DOUBLE    COMMENT 'Pass or fail as 1 or 0, or a score',
  value_label   STRING,
  rationale     STRING
)
CLUSTER BY (name, assessed_at);

CREATE TABLE IF NOT EXISTS ai_ops.telemetry.model_prices (
  model_name            STRING        NOT NULL,
  effective_from        TIMESTAMP     NOT NULL,
  effective_to          TIMESTAMP     COMMENT 'NULL for the current price',
  currency              STRING        NOT NULL,
  input_per_mtok        DECIMAL(12,6) NOT NULL COMMENT 'Per million input tokens',
  cached_input_per_mtok DECIMAL(12,6),
  output_per_mtok       DECIMAL(12,6) NOT NULL
);

The design choices behind it:

• The grain is the user request, so cost per team, p95 latency and error rate become single-table aggregations.
• Metrics are typed columns; payloads are optional. Payloads sit in nullable VARIANT columns, populated only for sampled, redacted requests. Where governance requires stricter separation, move them to a separate table with narrower grants.
• Guardrail events are an array of structs: few per request, always read with it.
• Assessments are long, not wide. A new judge is a new name, not a schema change, and judges, human reviews and code checks share one table, mirroring MLflow’s assessment sources.
• Prices are effective-dated, like system.billing.list_prices, so historic cost stays correct when prices change.
• Liquid clustering on application and time matches the dominant filters.

Token usage and cost attribution

Attribution keys must be on every call before any dashboard is useful: application, team, use case and environment. Set them twice, as request tags on gateway calls (they land in the request_tags map of the usage table) and as trace tags in the application. Untagged traffic is a number with an owner.

-- Estimated daily model spend by team, use case and model, last 30 days.
WITH priced AS (
  SELECT
    r.event_time, r.team, r.use_case, r.model_name,
    r.input_tokens, r.output_tokens,
    p.model_name IS NULL AS unpriced,
    ( (r.input_tokens - COALESCE(r.cached_input_tokens, 0)) * p.input_per_mtok
      + COALESCE(r.cached_input_tokens, 0)
          * COALESCE(p.cached_input_per_mtok, p.input_per_mtok)
      + r.output_tokens * p.output_per_mtok
    ) / 1000000 AS est_cost
  FROM ai_ops.telemetry.llm_requests AS r
  LEFT JOIN ai_ops.telemetry.model_prices AS p
    ON  p.model_name = r.model_name
    AND r.event_time >= p.effective_from
    AND (p.effective_to IS NULL OR r.event_time < p.effective_to)
  WHERE r.event_time >= current_timestamp() - INTERVAL 30 DAYS
    AND r.environment = 'prod'
)
SELECT
  CAST(event_time AS DATE)               AS day,
  team,
  use_case,
  model_name,
  COUNT(*)                               AS requests,
  SUM(input_tokens + output_tokens)      AS tokens,
  ROUND(SUM(est_cost), 2)                AS est_cost_total,
  ROUND(SUM(est_cost) / COUNT(*), 5)     AS est_cost_per_request,
  count_if(unpriced)                     AS unpriced_requests
FROM priced
GROUP BY ALL
ORDER BY day DESC, est_cost_total DESC;

unpriced_requests stops a model missing from the price table from silently vanishing from the report. The gateway’s usage table is the independent check, because it counts every call, including those from applications that never wrote a trace:

-- Gateway view of the same traffic, last 7 days.
SELECT
  CAST(event_time AS DATE)                      AS day,
  COALESCE(request_tags['app_id'], 'untagged')  AS app_id,
  destination_model,
  COUNT(*)                                      AS requests,
  SUM(input_tokens)                             AS input_tokens,
  SUM(output_tokens)                            AS output_tokens,
  percentile_approx(latency_ms, 0.95)           AS p95_latency_ms,
  percentile_approx(time_to_first_byte_ms, 0.5) AS p50_ttfb_ms,
  count_if(status_code >= 400)                  AS failed_requests
FROM system.ai_gateway.usage
WHERE event_time >= current_timestamp() - INTERVAL 7 DAYS
GROUP BY ALL
ORDER BY day DESC, input_tokens DESC;

Offline and online evaluation

Evaluation turns quality from an opinion into a time series. It runs in three loops that share judges and data.

Offline: regression suites in CI

Keep a curated evaluation set of a few hundred representative and adversarial questions with expected facts, versioned with the application. Run it on every change to a prompt, model, retrieval setting or chunking strategy, and fail the build when scores drop below agreed thresholds. mlflow.genai.evaluate runs the application over the dataset, records a trace per row and applies scorers: built-in LLM judges such as Correctness, RetrievalGroundedness and RelevanceToQuery, plain-language guideline judges, and code checks.

"""CI gate: fail the build when answer quality regresses."""
import sys

import mlflow
from mlflow.genai.scorers import Correctness, Guidelines, RetrievalGroundedness

from support_assistant.app import answer_question

EVAL_ROWS = [
    {
        "inputs": {"question": "How do I rotate a service API key?"},
        "expectations": {
            "expected_facts": [
                "Keys are rotated from the service settings page.",
                "The previous key stays valid until it is revoked.",
            ]
        },
    },
    # ... a few hundred curated rows, versioned with the application
]

# Illustrative thresholds: agree real values with the product owner.
THRESHOLDS = {"correctness/mean": 0.85, "retrieval_groundedness/mean": 0.90}


def predict_fn(question: str) -> str:
    answer, _ = answer_question(question, user_key="ci", session_id="ci-run")
    return answer


mlflow.set_experiment("/Shared/support-assistant-ci")
results = mlflow.genai.evaluate(
    data=EVAL_ROWS,
    predict_fn=predict_fn,
    scorers=[
        Correctness(),
        RetrievalGroundedness(),
        Guidelines(
            name="cites_sources",
            guidelines="The answer cites at least one source id in square brackets.",
        ),
    ],
)

print(results.metrics)  # confirm the metric names for your MLflow version
failed = {
    name: results.metrics.get(name)
    for name, floor in THRESHOLDS.items()
    if (results.metrics.get(name) or 0.0) < floor
}
if failed:
    print(f"Quality gate failed: {failed}")
    sys.exit(1)

A missing metric counts as a failure on purpose: a renamed scorer should break the build, not silently pass it.

Online: judges on sampled production traffic

Offline sets drift away from what users actually ask. Online evaluation runs judges on a sample of production traces and attaches the results as assessments, so quality sits next to latency and cost. On Databricks, production monitoring registers scorers against an experiment with a sampling rate. It is in Beta at the time of writing, and traces stored in Unity Catalog also need a SQL warehouse configured for the monitoring job; check the current documentation.

import mlflow
from mlflow.genai.scorers import RetrievalGroundedness, Safety, ScorerSamplingConfig

mlflow.set_experiment("/Shared/support-assistant")

# Sample rates are illustrative: cheap, critical checks high; costly judges low.
safety = Safety().register(name="safety")
safety = safety.start(sampling_config=ScorerSamplingConfig(sample_rate=1.0))

grounded = RetrievalGroundedness().register(name="groundedness")
grounded = grounded.start(sampling_config=ScorerSamplingConfig(sample_rate=0.1))

Judges consume tokens too, so include their spend in the cost model.

Human review

LLM judges are models with their own error rate. Route a regular sample of traces, plus everything with negative feedback or a failing judge score, to subject-matter reviewers, and track agreement between reviewers and judges. When agreement drops, fix the judge before trusting its trend. Reviewed traces with corrected answers are also the best source of new evaluation rows. With assessments in one table, a regression alert is a short query:

-- Judges whose pass rate yesterday fell below their trailing level.
WITH daily AS (
  SELECT
    r.app_id,
    a.name                       AS assessment,
    CAST(a.assessed_at AS DATE)  AS day,
    AVG(a.value_numeric)         AS mean_value,
    COUNT(*)                     AS n
  FROM ai_ops.telemetry.llm_assessments AS a
  JOIN ai_ops.telemetry.llm_requests AS r ON r.request_id = a.request_id
  WHERE a.source_type = 'LLM_JUDGE'
    AND a.assessed_at >= date_sub(current_date(), 30)
  GROUP BY ALL
),
scored AS (
  SELECT
    *,
    AVG(mean_value) OVER (
      PARTITION BY app_id, assessment ORDER BY day
      ROWS BETWEEN 14 PRECEDING AND 1 PRECEDING
    ) AS trailing_mean
  FROM daily
)
SELECT app_id, assessment, day, n,
       ROUND(mean_value, 3) AS pass_rate, ROUND(trailing_mean, 3) AS trailing_rate
FROM scored
WHERE day = date_sub(current_date(), 1)
  AND n >= 50                            -- illustrative minimum sample size
  AND mean_value < trailing_mean - 0.05  -- illustrative tolerance
ORDER BY mean_value - trailing_mean;

Governance

Prompts carry user input and retrieved enterprise content, which makes LLM telemetry some of the most sensitive data a platform holds.

PII in prompts and traces. Assume trace tables contain personal and confidential data unless you have engineered otherwise. Redact before logging where you can, at the gateway with a sensitive-data guardrail or in the application before a span records its inputs. Keep payloads out of the metrics columns, and protect the payloads you keep with column masks or separate tables.

Retention and deletion. Set retention per data class: payloads for days or weeks, metrics and assessments for as long as trend analysis needs. Plan deletion before launch. Deleting individual traces is not supported for traces stored in Unity Catalog, so deletion obligations are far easier to meet if user-identifying data never enters the raw trace tables and curated tables carry pseudonymous keys. In Delta tables, DELETE is logical until VACUUM removes the files, and with deletion vectors enabled you need REORG TABLE ... APPLY (PURGE) first.

Access control and audit. Most people need metrics; few need payloads. Anyone with access to the Unity Catalog trace tables can read every trace in them, so separate schemas by sensitivity rather than relying on experiment permissions. Use row filters and column masks on curated tables, and review system.access.audit for payload access. The Unity Catalog governance article covers the grant model.

Prompt and model versioning. Every trace should identify the prompt version, model, retrieval index and application version behind it, or a quality change cannot be attributed. MLflow’s prompt registry stores immutable versions with movable aliases such as production; on Databricks, prompts are Unity Catalog objects with three-level names. Load by alias, record the resolved version on the trace, and promote by moving the alias after the CI suite passes.

An adoption path

Build observability in the order it pays back:

1. Gateway first. Route model traffic through the gateway, require attribution tags, and turn on usage tracking and rate limits. Cost visibility arrives without changing application logic beyond the endpoint URL.
2. Trace the most important application, with retrieval, prompt and model spans, user and session ids and version tags.
3. Curate and report: the curated and price tables, and a first dashboard of cost, latency and errors by application and team.
4. Evaluate: the offline suite in CI, then online judges on sampled traffic, then human review.
5. Harden governance (retention, masks, access reviews, audit) before the next applications arrive.

Sample traces, not metrics. Gateway usage records are cheap and should cover all traffic. At high volume, MLFLOW_TRACE_SAMPLING_RATIO or sampling_ratio_override on the root function sets a head-based trace sampling rate. Head-based sampling decides when a trace starts, so it cannot know which requests will fail or draw negative feedback. If you need every error trace, apply tail-based rules in an OpenTelemetry collector, or write a compact record for every request regardless of sampling, as the llm_requests table does.

For help designing or implementing this on a production estate, see AI engineering and observability.

Checklist

• One trace per request, with typed spans for retrieval, prompts, model calls, tools and guardrails.
• Input, output and cached tokens on model spans, plus time to first token when streaming.
• Cost from an effective-dated price table, reconciled monthly against actual charges.
• Application, team, use case and environment tags on every call, at the gateway and in the trace.
• All model traffic through a gateway with usage tracking and rate limits.
• Traces in governed Delta tables, curated into request, assessment and price tables.
• An offline suite gating every change; online judges on samples, calibrated by human review.
• Payloads redacted, sampled, access-controlled and retained for a defined period.
• Prompt, model, index and application versions on every trace.

References

• OpenTelemetry semantic conventions for generative AI
• MLflow manual tracing: decorators and spans
• MLflow span concepts and span types
• MLflow token usage and cost tracking
• Export MLflow traces with OpenTelemetry
• MLflow built-in LLM judges
• Store MLflow traces in Unity Catalog
• Monitor GenAI applications in production on Databricks
• Unity Gateway usage tracking and the usage system table
• Unity Gateway inference tables