Is your feature request related to a problem? Please describe.
Teams that produce schema-aware Kafka payloads (for example Avro or Protobuf with Confluent Schema Registry) often depend on extra components outside Fluent Bit so records match consumer expectations: correct serialized bytes plus Confluent wire framing (including schema id where required).
Fluent Bit’s Kafka output uses librdkafka, which does not implement Schema Registry or registry-driven serialization. There is no built-in way to resolve schemas from a central Schema Registry over HTTP(S), cache them, and emit registry-compatible messages through the same Kafka output users already configure.
Describe the solution you'd like
Extend out_kafka with Schema Registry consumption:
HTTP(S) client — Fetch schema metadata from SR by schema id and/or subject + version.
Local cache — Cache schema material and ids; configurable limits and defined behavior on miss, SR errors, and staleness.
Encode + frame — Serialize records to Avro, with Confluent-compatible payloads for librdkafka. Avro-first delivery is acceptable if it reduces risk.
Config — SR base URL, TLS, authentication (should be covered by common http?), and subject rules for value (and optionally key). See this
We should evaluate libserdes (C library for this matter) before implementing this completely in-house.
Describe alternatives you've considered
External serialization (sidecar, processor, or custom filter) that talks to Schema Registry and passes opaque bytes to Kafka — adds deployment and operational overheads.
Manual schema and id configuration in Fluent Bit where supported — works for static schemas; not very useful when schemas are owned and versioned centrally in Schema Registry.
Additional context
Registry: Confluent Schema Registry; output: Kafka via out_kafka. Only for AVRO
Is your feature request related to a problem? Please describe.
Teams that produce schema-aware Kafka payloads (for example Avro or Protobuf with Confluent Schema Registry) often depend on extra components outside Fluent Bit so records match consumer expectations: correct serialized bytes plus Confluent wire framing (including schema id where required).
Fluent Bit’s Kafka output uses librdkafka, which does not implement Schema Registry or registry-driven serialization. There is no built-in way to resolve schemas from a central Schema Registry over HTTP(S), cache them, and emit registry-compatible messages through the same Kafka output users already configure.
Describe the solution you'd like
Extend out_kafka with Schema Registry consumption:
HTTP(S) client — Fetch schema metadata from SR by schema id and/or subject + version.
Local cache — Cache schema material and ids; configurable limits and defined behavior on miss, SR errors, and staleness.
Encode + frame — Serialize records to Avro, with Confluent-compatible payloads for librdkafka. Avro-first delivery is acceptable if it reduces risk.
Config — SR base URL, TLS, authentication (should be covered by common http?), and subject rules for value (and optionally key). See this
We should evaluate libserdes (C library for this matter) before implementing this completely in-house.
Describe alternatives you've considered
External serialization (sidecar, processor, or custom filter) that talks to Schema Registry and passes opaque bytes to Kafka — adds deployment and operational overheads.
Manual schema and id configuration in Fluent Bit where supported — works for static schemas; not very useful when schemas are owned and versioned centrally in Schema Registry.
Additional context
Registry: Confluent Schema Registry; output: Kafka via out_kafka. Only for AVRO