Confluent Schema Registry and Kafka
AvroSharp.Confluent gives Confluent.Kafka producers and consumers Avro serializers on AvroSharp, with Confluent Schema Registry. The same serializers work with registries that have its API, such as Redpanda, Karapace and Apicurio. It replaces Confluent's own Avro serializer with no Apache.Avro, the same configuration keys and the same message bytes; the differences are listed under Moving from Confluent's Avro serializer. For the registry, it uses Confluent's own code, through Confluent's serializer base classes:
- subject name strategies;
- registration;
- schema ID strategies;
- references;
- rules.
For KafkaFlow, AvroSharp.KafkaFlow puts these serializers in KafkaFlow's middleware.
On this page:
- Use
- Settings
- Moving from Confluent's Avro serializer
- Moving from Chr.Avro
- Behavior to know
- Metrics and tracing
- Performance
- Data contract rules
The Confluent sample produces and consumes through a real broker and registry. It reads the same messages three ways: as the type that wrote them, as generic values, and as a newer version of the type. Its compose file starts Redpanda: run docker compose up -d --wait, then dotnet run, in samples/Confluent.
dotnet add package AvroSharp.Confluent
dotnet add package AvroSharp.Generators
AvroSharp.Confluent depends on Confluent.SchemaRegistry 2.14.0 or later (below 3.0), and is released with AvroSharp, at the same version.
Use
using AvroSharp.Confluent;
using Confluent.Kafka;
using Confluent.SchemaRegistry;
using var registry = new CachedSchemaRegistryClient(new SchemaRegistryConfig { Url = "http://localhost:8081" });
// Order is generated from Order.avsc, or is a partial class marked [AvroSerializable].
using var producer = new ProducerBuilder<string, Order>(new ProducerConfig { BootstrapServers = "localhost:9092" })
.SetAvroSharpKeySerializer(registry)
.SetAvroSharpValueSerializer(registry)
.Build();
await producer.ProduceAsync("orders", new Message<string, Order> { Key = order.Id.ToString(), Value = order });
using var consumer = new ConsumerBuilder<string, Order>(new ConsumerConfig { BootstrapServers = "localhost:9092", GroupId = "billing" })
.SetAvroSharpKeyDeserializer(registry)
.SetAvroSharpValueDeserializer(registry)
.Build();
The serializers can also be created directly: new AvroSharpSerializer<Order>(registry, config) and new AvroSharpDeserializer<Order>(registry, config).
- Both synchronous and asynchronous. Each implements Confluent.Kafka's synchronous interface (
ISerializer<T>,IDeserializer<T>) as well as the asynchronous one, soproducer.Produceworks, and a consumer takes the deserializer as it is. - The producer builder has a
SetValueSerializerfor each interface, so pass the serializer as one of them, such as(ISerializer<Order>)serializer.AvroSharpSerdeExtensionssets the synchronous ones, which serveProduceAsynctoo. - No task once the schema is known. The synchronous path waits for the registry the first time a topic's schema is written or a schema ID is read; after that it reads and writes without a task. Settings that can run rules asynchronously (
use.latest.version,use.latest.with.metadata,use.schema.id, and schemas with rules) wait on every message.
Generic records. For code that has schemas rather than types, AvroSharpGeneric:
var serializer = AvroSharpGeneric.CreateSerializer(registry, schema); // writes AvroValue values of the schema
var records = AvroSharpGeneric.CreateSerializer(registry); // writes each record with its own schema
var deserializer = AvroSharpGeneric.CreateDeserializer(registry); // each message as its writer's schema
var asV2 = AvroSharpGeneric.CreateDeserializer(registry, readerSchema: v2); // or resolved to one schema
On Confluent.Kafka's builders, SetAvroSharpGenericValueSerializer and SetAvroSharpGenericValueDeserializer set them, with or without a schema.
Several record types on one topic. A topic that carries several record types, with the TopicRecord or Record subject name strategy, or a top-level union of records under one subject, takes one serializer and one deserializer of all of them. T is object, or a base type or interface the types share:
Type[] events = [typeof(OrderPlaced), typeof(OrderShipped)];
var serializer = new AvroSharpSerializer<object>(registry, events, new AvroSharpSerializerConfig { SubjectNameStrategy = SubjectNameStrategy.TopicRecord });
var deserializer = new AvroSharpDeserializer<object>(registry, events);
- Writing: each value is written as its own type, with its own record's schema, under its own subject (
orders-shop.OrderPlaced,orders-shop.OrderShipped). Confluent'sAvroSerializer<ISpecificRecord>writes the same bytes. A value of a type that isn't listed fails. - Reading: each message is read as the type of its writer's record, by the record's full name or one of the type's aliases, and resolved to that type's schema as usual. For a union, it is read as the type of the branch the message holds. A record no type has fails.
.NET Standard and .NET Framework. On .NET 5 and later, a generated type registers itself in AvroTypes when its assembly loads, and the constructors above find it. On other targets, either:
- call
AvroTypes.Register(Order.AvroTypeInfo)once at startup; - or pass the type's info:
new AvroSharpSerializer<Order>(registry, Order.AvroTypeInfo).
Settings
AvroSharpSerializerConfig and AvroSharpDeserializerConfig take the keys of Confluent's AvroSerializerConfig and AvroDeserializerConfig. They can be built from key-value pairs, which they copy, such as a dictionary or a configuration section:
// appsettings.json: { "Kafka": { "Serializer": { "avro.serializer.auto.register.schemas": "false" } } }
var config = new AvroSharpSerializerConfig(configuration.GetSection("Kafka:Serializer").AsEnumerable(makePathsRelative: true)!);
A section's own entry has no value, and is left out.
| Serializer | Key | Default |
|---|---|---|
AutoRegisterSchemas |
avro.serializer.auto.register.schemas |
true |
NormalizeSchemas |
avro.serializer.normalize.schemas |
false |
UseSchemaId |
avro.serializer.use.schema.id |
none |
UseSchemaGuid |
avro.serializer.use.schema.guid |
none; Java's use.schema.guid |
UseLatestVersion |
avro.serializer.use.latest.version |
false; can't be combined with auto-registration |
UseLatestWithMetadata |
avro.serializer.use.latest.with.metadata |
none |
SubjectNameStrategy |
avro.serializer.subject.name.strategy |
Associated: the topic's association, or else Topic |
SchemaIdStrategy |
avro.serializer.schema.id.strategy |
Prefix |
BufferBytes |
avro.serializer.buffer.bytes |
1024 |
RemoveJavaProperties |
avro.serializer.remove.java.properties |
false; Java's avro.remove.java.properties |
TombstoneBehavior |
avro.serializer.tombstone.behavior |
None, Confluent's behavior; AvroSharp's own key |
LatestCompatibilityStrict |
avro.serializer.latest.compatibility.strict |
true; Java's latest.compatibility.strict |
IdCompatibilityStrict |
avro.serializer.id.compatibility.strict |
true; Java's id.compatibility.strict |
| Deserializer | Key | Default |
|---|---|---|
UseLatestVersion |
avro.deserializer.use.latest.version |
false |
UseLatestWithMetadata |
avro.deserializer.use.latest.with.metadata |
none |
SubjectNameStrategy |
avro.deserializer.subject.name.strategy |
Associated |
SchemaIdStrategy |
avro.deserializer.schema.id.strategy |
Dual: a header, or else the prefix |
FailOnTrailingData |
avro.deserializer.fail.on.trailing.data |
false; Java's avro.fail.on.trailing.data |
TombstoneBehavior |
avro.deserializer.tombstone.behavior |
None, Confluent's behavior; AvroSharp's own key |
Two keys come from Java's serializers, which Confluent's .NET serializer doesn't have:
RemoveJavaPropertiesremoves the propertyavro.java.stringfrom the schema before it is registered or looked up, as Java's serializer does. Java's generated classes add it, so with the option on everywhere, the same.avscregisters as one version from Java and .NET.FailOnTrailingDatafails a message that has bytes left after the value, a sign of a wrong schema or a corrupt message. Without it, they are ignored, as Confluent's .NET deserializer does.
Subjects and pinned schemas. For subject names that SubjectNameStrategy can't express, such as a schema context or a team's prefix, give the serializer or deserializer a function:
var serializer = new AvroSharpSerializer<Order>(registry)
{
CustomSubjectNameStrategy = (context, recordType) => Task.FromResult($":.billing:{context.Topic}-value"),
};
To write with one version of a subject, whatever the topic, pin it. The registry is asked for the version's schema ID once:
var builder = await new ProducerBuilder<string, Order>(producerConfig)
.SetAvroSharpValueSerializerAsync(registry, subject: "orders-value", version: 3);
UseSchemaId pins a schema by its ID, and UseSchemaGuid by its GUID, under the subject the strategy names. A pinned schema that differs from your type's is written as described under behavior to know, with IdCompatibilityStrict.
Moving from Confluent's Avro serializer
Confluent.SchemaRegistry.Serdes.Avro uses Apache.Avro and its avrogen classes.
- Generate the types with AvroSharp.Generators.
- Moving gradually? Set
<AvroSharpApacheCompatible>true</AvroSharpApacheCompatible>, and the generated classes also implement Apache'sISpecificRecord. The same class then works with both serializers while you move one producer or consumer at a time. This repository's tests use such classes, and the Apache.Avro compatibility mode has the details. - Moving at once? Use AvroSharp's own types, without Apache.Avro.
- Moving gradually? Set
- Replace the serializers:
AvroSerializer<T>→AvroSharpSerializer<T>;AvroDeserializer<T>→AvroSharpDeserializer<T>;AvroSerializerConfig→AvroSharpSerializerConfig, with the same keys.
- Nothing changes on the wire or in the registry:
- the same subjects;
- the same message bytes, schema ID included. The schema's JSON may be formatted differently, but registries treat equivalent schemas as one;
- a top-level
bytesvalue is still the message body itself, as Confluent's serializers (.NET and Java) write it. Bytes with a logical type, such as a decimal, keep Avro's length prefix, as Java's serializer writes aBigDecimal.
- What differs:
- Writing with another schema: with
use.latest.version,use.latest.with.metadataoruse.schema.id, values are written in that schema, as Java writes them, where Confluent's specific serializer writes the type's own bytes under its ID (behavior to know). - Generic records:
AvroSharpGeneric.CreateSerializer(registry)writes each record with its own schema, as Confluent's generic serializer does; it writes records only. With a schema, it writes values of that schema, primitives included. - Synchronous too: both serializers also implement Confluent.Kafka's synchronous interfaces, which Confluent's don't. So passing one straight to
SetValueSerializerneeds a cast; theSetAvroSharp…extensions don't.
- Writing with another schema: with
Moving from Chr.Avro
Chr.Avro.Confluent maps your classes to schemas by reflection when the serializer is built. AvroSharp writes that code at compile time instead:
- Your types: add
[AvroSerializable], andpartial, to the classes you serialize, or generate them from the subject's.avscfiles. - Builder methods:
SetAvroValueSerializer(registry, AutomaticRegistrationBehavior.Always)→SetAvroSharpValueSerializer(registry);SetAvroValueDeserializer(registry)→SetAvroSharpValueDeserializer(registry).
- Registration: Chr.Avro doesn't register schemas by default, but AvroSharp does, as Confluent's serializer does. To keep Chr.Avro's behavior, set
AutoRegisterSchemas = false, together withUseLatestVersion = trueif you wrote with the subject's latest schema. - Names: Chr.Avro matches a schema's fields to your members loosely. AvroSharp names fields after your members, as written:
- set
[AvroName]on a member whose field is named differently; - or set
[assembly: AvroSerializableDefaults(FieldNames = AvroNaming.CamelCase)]for camelCase fields.
- set
Behavior to know
- Native AOT. CI publishes an application that uses it with Native AOT and runs it (the add-ons smoke test). AvroSharp.Confluent has no trim or AOT warnings. Confluent's
CachedSchemaRegistryClientreads the registry's REST API with Newtonsoft.Json, which has trim and AOT warnings of its own; they come from Confluent's client, not from the serializers. - Tombstones. A
nullvalue, or a nullAvroValue, is written as a message with no body, which Kafka calls a tombstone. Reading one givesnull, or a nullAvroValue. A value type such asintcan't be null, so reading a tombstone into one throws, as in Confluent's deserializer.- Confluent.Kafka's
Nullhas the schema"null", as in Confluent's serializer:AvroSharpSerializer<Null>writes a message with the schema ID and an empty body, not a tombstone. TombstoneBehavior.Strict, as in Chr.Avro, makes sure a tombstone can't be mistaken for a value. The serializer or deserializer is refused when it's created if the type can't be null (a value type, rather thanint?), or if the schema can hold null ("null", or a union with"null"). The deserializer checks each writer's schema as it meets it. A key with no value throws, since keys have no tombstones.
- Confluent.Kafka's
use.latest.version,use.latest.with.metadataanduse.schema.id. The message carries that schema's ID, so it is written in that schema, as Java's serializer and Confluent's generic serializer write it. Confluent's specific serializer doesn't: it writes your type's bytes under the other ID. This is decided once per schema ID:- When the two schemas encode alike (the same Parsing Canonical Form and logical types), your type's bytes are written as they are.
- Otherwise the value is resolved into that schema, as a reader of your type's data would read it, and written in it. Fields the schema lacks are dropped, and fields it adds get their defaults. This costs a second encoding per message.
- With
LatestCompatibilityStrict(orIdCompatibilityStrict, foruse.schema.id), the default, that schema must be able to read your type's, by the compatibility check; the error names every reason it can't. Off, values are written whenever the schemas resolve, and a value the schema can't hold, such as a union branch it lacks, fails when it is written. - Other logical types, or another decimal precision or scale, are always refused: the encoded numbers would mean other values.
- A top-level
bytesvalue is the message body alone, so it is written only with a schema that encodes alike.
- Configuration keys. A key the serializer doesn't know is an error, as in Confluent's serializers, so a misspelled key isn't ignored. Keys under
rules.andsubject.name.strategy.are passed through to Confluent's code. - The
Nonesubject name strategy isn't supported, with or withoutuse.schema.id: Confluent's client needs a subject to look a schema ID up in, and Confluent's serializer fails the same way. The serializer says so. - The
RecordandTopicRecordstrategies name the subject after a record schema, as Confluent's .NET and Java serializers do. Any other schema, such as a primitive, has no record name, and serializing it fails with an error that says so.
Metrics and tracing
The serializers publish metrics through a Meter, and the registry calls as activities through an ActivitySource. Both are named AvroSharp.Confluent. Nothing is measured unless a listener subscribes, so they cost nothing otherwise. With OpenTelemetry:
builder.Services.AddOpenTelemetry()
.WithMetrics(metrics => metrics.AddMeter("AvroSharp.Confluent"))
.WithTracing(tracing => tracing.AddSource("AvroSharp.Confluent"));
| Instrument | Type | Tags |
|---|---|---|
avrosharp.confluent.serde.duration (s) |
histogram: the time to write or read a message, registry calls included | avrosharp.serde (serializer, deserializer), messaging.destination.name (the topic), avrosharp.component (key, value), error.type when it failed |
avrosharp.confluent.serde.errors |
counter: messages that failed | as above |
avrosharp.confluent.rule.failures |
counter: messages whose data contract rules failed (a RuleException), among the errors |
as above |
avrosharp.confluent.registry.calls |
counter: calls to the registry client when the serializer's own caches miss; Confluent's client may still answer from its cache | avrosharp.operation (register, lookup, get_schema, migrations), avrosharp.subject |
Each registry call is also an activity, schema_registry <operation>, with the same tags.
Performance
After the first message of a topic or schema ID, the serializers don't go back to the registry, and they skip Confluent's per-message lookups. A message writes into a buffer the thread keeps, so a write allocates only the message, and a read only the value (plus the task, on the asynchronous path). With use.latest.version, the latest schema is still asked for on every message, as Confluent's serde does, but the writer's schema and the check for migrations are looked up once.
bench/AvroSharp.Confluent.Benchmarks compares AvroSharp.Confluent with Confluent's Avro serializer and Chr.Avro.Confluent on an in-memory registry: dotnet run -c Release -f net10.0 --project bench/AvroSharp.Confluent.Benchmarks. The results are in benchmarks.
Data contract rules
Rules run in Confluent's executors (Confluent.SchemaRegistry.Rules and Confluent.SchemaRegistry.Encryption), which you register as with Confluent's serializer.
| Rule | Generated or [AvroSerializable] types |
Generic values (AvroValue) |
|---|---|---|
CEL conditions and transforms (CEL) |
Run, by the Avro field names | Run, by the Avro field names |
Payload encryption (ENCRYPT_PAYLOAD) |
Runs | Runs |
Field rules: field-level encryption (ENCRYPT, CSFLE) and CEL_FIELD |
Run, on the tagged fields | Run, on the tagged fields |
Migration rules (UPGRADE, DOWNGRADE, such as JSONATA) |
Run, on Avro JSON, then read as the type | Run, on Avro JSON |
CEL names the Avro fields, as rules written for Java's and Confluent's serializers do: message.customer_name, not the C# property message.CustomerName. Confluent's CEL executor reads a value by its Avro field names only as one of Apache.Avro's records. So when a subject's domain rules include CEL rules, the rules get the value decoded from its encoding as Apache.Avro's generic model: a GenericRecord, with GenericEnum, lists and dictionaries inside, as Confluent's generic serializer gives them.
- Apache.Avro comes with Confluent.SchemaRegistry.Rules, which has the CEL executor. AvroSharp.Confluent doesn't reference it; it finds it at run time.
- A condition that passes writes the value's own encoding, so the bytes are the same as without the rule.
- A transform returns the value to write (or, on reads, to read): a record of the subject's schema in Apache.Avro's generic model. A rule that returns anything else, such as a CEL map literal, fails with
InvalidOperationException. - Other domain rules, with no CEL rule among them, get your value itself.
Field rules run as in Confluent's serde, on the fields whose tags the rule names (every field, for a rule without tags): a field's tags are its confluent:tags property and the schema metadata's tags for its path.
- Fields are found by walking the value in the rules' schema, the latest one (or the writer's, on reads): records, arrays, maps and union branches, as Confluent's
AvroUtilswalks them. Messages written with field-level encryption by Confluent's .NET serde read here, and it reads what AvroSharp writes (tested with the same registry and keys). - The transform gets the field's .NET value: a
string, abyte[]for bytes and fixed, a number, abool, an enum's symbol, ornull. Field-level encryption takes strings and bytes, as in Confluent's serde. CEL_FIELDexpressions can usevalue,name,fullName,typeNameandtags. Itsmessagevariable, the record that holds the field, isn't readable by CEL here.- Without field rules, nothing is walked.
Migration rules run as in Confluent's and Java's deserializers, with use.latest.version or use.latest.with.metadata, between the writer's version and that one: UPGRADE rules to a newer version, DOWNGRADE rules to an older one.
The rules see the value as Avro JSON, in the shape of Apache.Avro's
JsonEncoder: unions wrapped by their branch's full name ({"string": "x"}), bytes and fixed as ISO-8859-1 strings, enums as their symbols, and logical types as their underlying values. Expressions written for Confluent's or Java's serializers work unchanged.What they return is read as the target version's schema, so it must be a value of that schema; a generated type then reads it as its own schema.
The rules are looked up once per writer version and target version; the migration itself runs on each message.
A top-level
bytesschema isn't migrated: its message is the bytes alone.Trimming and Native AOT: Confluent.SchemaRegistry.Rules and Apache.Avro use reflection and aren't marked trim-compatible, and CEL rules aren't tested under Native AOT. Without CEL rules, none of this is used.