OpenTelemetry Tracing

Overview

Upsonic instruments every agent run with OpenTelemetry spans following the GenAI semantic conventions. Every LLM call, pipeline step, tool execution, and agent run is traced automatically — giving you complete visibility into what your agent did, how long it took, and how much it cost.

Install the optional dependencies:

uv pip install "upsonic[otel]"
# pip install "upsonic[otel]"

What You Get

When tracing is enabled, each agent run produces a span hierarchy:

agent.run (SERVER)
  └─ pipeline.execute (INTERNAL)
       ├─ pipeline.step.ModelExecutionStep (INTERNAL)
       │    └─ chat (CLIENT) — LLM call with GenAI attributes
       ├─ pipeline.step.ToolExecutionStep (INTERNAL)
       │    └─ tool.execute (INTERNAL) — per tool call
       └─ pipeline.step.MemorySaveStep (INTERNAL)

Attributes on spans include:

gen_ai.request.model, gen_ai.response.model — model used
gen_ai.usage.input_tokens, gen_ai.usage.output_tokens — token counts
gen_ai.request.temperature, gen_ai.request.max_tokens — request parameters
gen_ai.response.id, gen_ai.response.finish_reasons — response metadata
upsonic.total_cost, upsonic.execution_time — cost and duration
upsonic.model_execution_time, upsonic.framework_overhead_time — timing breakdown
upsonic.time_to_first_token — streaming latency
upsonic.tool_call_count — number of tool calls
upsonic.tool.name, upsonic.tool.execution_time, upsonic.tool.success — per-tool details
upsonic.input, upsonic.output — task input/output
upsonic.run_id, upsonic.status — run identification and status
upsonic.agent.name, upsonic.agent.model — agent metadata
upsonic.usage.total_tokens, upsonic.usage.cache_read_tokens, upsonic.usage.cache_write_tokens — detailed token usage
upsonic.pipeline.total_steps, upsonic.pipeline.executed_steps — pipeline progress
upsonic.step.name, upsonic.step.status, upsonic.step.execution_time — per-step details
user.id, session.id — user and session tracking

Usage

Environment Variable — Zero Code Change

Set UPSONIC_OTEL_ENABLED=true and every Agent is automatically instrumented — no code changes needed.

export UPSONIC_OTEL_ENABLED=true
export UPSONIC_OTEL_ENDPOINT=http://localhost:4317

from upsonic import Agent

# Automatically instrumented — no instrument= parameter needed
agent = Agent("anthropic/claude-sonnet-4-6")
agent.print_do("What is 2 + 2?")

When Upsonic is imported, it detects UPSONIC_OTEL_ENABLED and calls Agent.instrument_all() with a DefaultTracingProvider under the hood. All agents created in the process are instrumented automatically.

UPSONIC_OTEL_ENDPOINT must point to a running OTLP collector (Jaeger, Grafana Tempo, etc.). If no collector is reachable, you’ll see a ConnectionError in stderr at process exit when the exporter tries to flush spans. This is harmless — the agent runs normally, spans are simply dropped. See Troubleshooting below.

Quick Start — `instrument=True`

Creates a DefaultTracingProvider from environment variables automatically.

from upsonic import Agent

agent = Agent("anthropic/claude-sonnet-4-6", instrument=True)
agent.print_do("What is 2 + 2?")

DefaultTracingProvider — Any OTLP Backend

Send traces to Jaeger, Grafana Tempo, Datadog, or any OTLP-compatible collector.

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("Summarize this report.")

DefaultTracingProvider — Full Configuration

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(
    endpoint="http://localhost:4317",
    service_name="payment-agent",
    sample_rate=0.25,
    include_content=False,
)

agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("What is 2 + 2?")

Global Instrumentation — `Agent.instrument_all()`

Instruments every Agent created after the call, without passing instrument each time.

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
Agent.instrument_all(provider)

agent1 = Agent("anthropic/claude-sonnet-4-6")   # instrumented
agent2 = Agent("anthropic/claude-sonnet-4-6")   # also instrumented

agent1.print_do("What is 2 + 2?")
agent2.print_do("What is 3 + 3?")

Agent.instrument_all(False)        # disable

Session & User Tracking

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")

agent = Agent(
    "anthropic/claude-sonnet-4-6",
    instrument=provider,
    session_id="conversation-123",
    user_id="user@example.com",
)
agent.print_do("What is 2 + 2?")

Streaming

Tracing works identically with streaming — no extra setup.

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317")
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)

for chunk in agent.stream("Count to 5", events=False):
    print(chunk, end="")

Hide Sensitive Content

from upsonic import Agent
from upsonic.integrations.tracing import DefaultTracingProvider

provider = DefaultTracingProvider(endpoint="http://localhost:4317", include_content=False)
agent = Agent("anthropic/claude-sonnet-4-6", instrument=provider)
agent.print_do("Process SSN: 123-45-6789")
# Prompts and responses will NOT appear in traces

Advanced — Raw InstrumentationSettings

For testing or custom setups, pass InstrumentationSettings directly.

from upsonic import Agent
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.sdk.metrics import MeterProvider
from upsonic.models.instrumented import InstrumentationSettings

exporter = InMemorySpanExporter()
tp = TracerProvider()
tp.add_span_processor(SimpleSpanProcessor(exporter))

settings = InstrumentationSettings(
    tracer_provider=tp,
    meter_provider=MeterProvider(),
)

agent = Agent("anthropic/claude-sonnet-4-6", instrument=settings)
agent.print_do("Hello!")

for span in exporter.get_finished_spans():
    print(span.name, span.attributes)

Environment Variables

Env Var	Description	Default
`UPSONIC_OTEL_ENABLED`	Set `true` to auto-instrument all agents on import	—
`UPSONIC_OTEL_ENDPOINT`	OTLP collector endpoint	`http://localhost:4317`
`UPSONIC_OTEL_SERVICE_NAME`	`service.name` resource attribute	`upsonic`
`UPSONIC_OTEL_SAMPLE_RATE`	Sampling rate (0.0–1.0)	`1.0`
`UPSONIC_OTEL_HEADERS`	Comma-separated `key=value` headers	—

Setting UPSONIC_OTEL_ENABLED=true triggers automatic OpenTelemetry setup on import and calls Agent.instrument_all().

Lifecycle

All providers set flush_on_exit=True by default — pending spans are flushed automatically when the process exits. You do not need to call shutdown() manually in normal usage. Call shutdown() explicitly only if you need to flush spans mid-process (e.g., in tests or before a graceful restart).

Troubleshooting

ConnectionError at process exit

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4317):
Max retries exceeded ... Connection refused

This is harmless. It means the OTLP exporter tried to send spans but no collector is running at the configured endpoint. Your agent executed normally — only the trace export failed. To fix it, either:

Run a collector — start Jaeger, Grafana Tempo, or any OTLP-compatible backend:

docker run -d --name jaeger -p 4317:4317 -p 16686:16686 jaegertracing/all-in-one:latest

Use Langfuse instead — no local collector needed, traces go to the cloud:

from upsonic.integrations.langfuse import Langfuse
agent = Agent("anthropic/claude-sonnet-4-6", instrument=Langfuse())

Don’t use tracing — simply don’t set instrument or UPSONIC_OTEL_ENABLED.

Langfuse Integration

Send traces to Langfuse for LLM-specific dashboards, cost tracking, and prompt management.

PromptLayer Integration

Log agent runs and evaluations to PromptLayer for prompt management, versioning, and observability.

​Overview

​What You Get

​Usage

​Environment Variable — Zero Code Change

​Quick Start — instrument=True

​DefaultTracingProvider — Any OTLP Backend

​DefaultTracingProvider — Full Configuration

​Global Instrumentation — Agent.instrument_all()

​Session & User Tracking

​Streaming

​Hide Sensitive Content

​Advanced — Raw InstrumentationSettings

​Environment Variables

​Lifecycle

​Troubleshooting

​ConnectionError at process exit

Langfuse Integration

PromptLayer Integration

Overview

What You Get

Usage

Environment Variable — Zero Code Change

Quick Start — `instrument=True`

DefaultTracingProvider — Any OTLP Backend

DefaultTracingProvider — Full Configuration

Global Instrumentation — `Agent.instrument_all()`

Session & User Tracking

Streaming

Hide Sensitive Content

Advanced — Raw InstrumentationSettings

Environment Variables

Lifecycle

Troubleshooting

ConnectionError at process exit