Telemetry pipelines (OTLP & Kafka)

Agentcy treats telemetry — logs, metrics, traces, audit events — as just another data source. The same realm, target_table, run history, and Explore query surface that holds your knowledge graph also holds your operational signals. Agents can correlate "the deploy that broke checkout" with "the GitHub PR that landed three minutes earlier" because both live in the same Context Engine.

TL;DR

Protocol	Status	Where it lands
OTLP gRPC logs (port 4317)	✅ Native	otel_logs table in your kyma database
OTLP gRPC traces / metrics	🟡 On the kyma roadmap	—
HTTP webhook (NDJSON)	✅ Native	Whatever target_table your pipeline declares
Kafka topics	✅ Native (env-configured)	Topic→table mapping in KYMA_KAFKA_TOPICS
Vector / Fluent Bit / Logstash → HTTP	✅ Native (via Webhook pipeline)	Whatever target_table your pipeline declares

OTLP traffic on Cloud lands in one otel_logs table per Agentcy instance. Each customer instance has its own kyma + its own otel_logs, so multi-tenant isolation works at the deployment boundary, not the pipeline boundary. (If you need per-pipeline routing on a shared kyma instance, see the section Per-pipeline OTLP routing at the bottom.)

Path 1: Native OTLP gRPC (the easiest path)

Your Agentcy instance exposes a TCP-proxied OTLP gRPC endpoint. Point any OpenTelemetry SDK or Collector at it and logs flow into the same Context Engine your knowledge graph uses.

Find your endpoint

Look at Explore → Pipelines — the OTLP endpoint URL is shown at the top of the page. For Agentcy Cloud:

turntable.proxy.rlwy.net:39547   # production console.agentcylabs.com

For self-hosted, expose port 4317 on your kyma container (set KYMA_OTLP_ADDR=0.0.0.0:4317) and use whatever address routes there.

Send from an OTel SDK

Standard OTLP SDK env, no Collector needed:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://turntable.proxy.rlwy.net:39547
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_SERVICE_NAME=checkout
# Auth header — kyma reads bearer tokens via OTEL_EXPORTER_OTLP_HEADERS:
export OTEL_EXPORTER_OTLP_HEADERS="authorization=Bearer ${KYMA_TOKEN}"

Logs from your app appear in the otel_logs table within seconds.

Send from an OTel Collector

# otel-collector.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 1s
    send_batch_size: 500

exporters:
  otlp/agentcy:
    endpoint: turntable.proxy.rlwy.net:39547
    tls:
      insecure: true   # the Railway TCP proxy front-ends raw TCP; kyma terminates plaintext gRPC
    headers:
      authorization: "Bearer ${KYMA_TOKEN}"

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/agentcy]

Query

otel_logs
| where service_name == "checkout" and severity_text == "ERROR"
| summarize n = count() by bin(timestamp, 1m)

The agent's tool catalog can ask "what's been failing on checkout in the last hour" and join the result against the rest of your knowledge graph (deploys, recent PRs, runbooks).

Path 2: HTTP Webhook (any shape, with realm + target_table)

For non-OTel data shapes — application events, business metrics, audit log entries, third-party webhooks — use a Webhook pipeline. NDJSON over POST, full schema-evolve, picks its own target_table and realm.

Use this when:

• You want each kind of event in its own table
• You want HMAC signing
• You want per-shape run history in the Pipelines UI
• Your data isn't shaped like OTLP signals

Path 3: Kafka topic ingest

kyma-ingest-kafka consumes NDJSON from one or more topics into kyma tables. Topic→table routing is via env at the kyma instance:

KYMA_KAFKA_ENABLED=1
KYMA_KAFKA_BOOTSTRAP_SERVERS=broker.svc:9092
KYMA_KAFKA_GROUP_ID=agentcy-ingest
KYMA_KAFKA_TOPICS=app-events:default.context_events,audit:default.context_events

Re-deploy kyma after adding/removing topics. At-least-once delivery; offsets commit after the catalog snapshot commits, so worst-case duplicates on a kyma crash within the same batch.

For one-off Kafka → Agentcy without restarting kyma, point Vector at your Kafka cluster and let it forward to a Webhook pipeline (Path 4 below) — that path requires zero engine config.

Path 4: Vector → Webhook (universal log/event router)

For shops that aren't yet OTel-native, Vector accepts every input under the sun (file, syslog, journald, Kafka, Kinesis, Pub/Sub, …) and emits NDJSON to Agentcy:

[sources.k8s_logs]
type = "kubernetes_logs"

[sources.kafka_events]
type = "kafka"
bootstrap_servers = "broker.svc:9092"
topics = ["app-events", "audit-events"]
group_id = "agentcy-ingest"
decoding.codec = "json"

[transforms.shape]
type = "remap"
inputs = ["k8s_logs", "kafka_events"]
source = '''
  .label = "Event"
  .source = "vector"
  .at = .timestamp
'''

[sinks.agentcy]
type = "http"
inputs = ["shape"]
uri = "https://your.agentcy.dev/api/v1/context/ingest/webhook/ctx_3a9b…"
method = "post"
encoding.codec = "ndjson"
framing.method = "newline_delimited"
batch.timeout_secs = 1
batch.max_events = 1000
buffer.type = "disk"
buffer.max_size = 1073741824   # 1 GiB at-least-once durability

This is the path most production deployments take when they need a single shipper for OTLP + Kafka + arbitrary other sources behind one place to set retries / disk buffering / TLS / batching.

Per-pipeline OTLP routing

Today the OTLP gRPC port writes everything into otel_logs in the kyma instance's KYMA_OTLP_DATABASE. There's no per-pipeline filtering on the receive side. For Cloud customers this is fine because each customer has their own kyma instance — your otel_logs is yours.

If you need to multiplex multiple isolated pipelines onto a single kyma instance:

• Option A — different OTEL_RESOURCE_ATTRIBUTES per app, then query by attributes_json.
• Option B — different services targeting different OTLP endpoints (one TCP proxy per kyma instance per tenant).
• Option C — wait for the per-pipeline x-pipeline-token header support, which is on the kyma roadmap.

Reference: signal shapes

All examples assume target_table = "context_events". Suggested record shapes:

{"label":"Log","at":"2026-04-29T12:00:00Z","service":"checkout","severity":"ERROR","message":"db timeout","trace_id":"0af7651916cd…"}
{"label":"Metric","at":"2026-04-29T12:00:00Z","service":"checkout","metric":"http_5xx","value":3,"unit":"count"}
{"label":"Span","at":"2026-04-29T12:00:00Z","service":"checkout","name":"db.query","duration_ms":214,"trace_id":"0af7651916cd…"}

The Explore Query tab can now do:

context_events
| where label in ("Log","Span") and trace_id == "0af7651916cd…"
| order by at asc

— one trace, all signals, joined.

Next

• HTTP Webhook how-to
• Filedrop how-to
• Concept: Context Engine