Integrations

AvroSharp works with message brokers and schema registries at two levels. The AvroSharp package itself writes and reads each registry's wire framing, with no dependency on a registry client. Add-on packages plug AvroSharp into the clients and frameworks applications already use: Confluent.Kafka, KafkaFlow, Azure Schema Registry and AWS Glue Schema Registry, listed below. They're tracked in #77, and feature requests for others are welcome.

In AvroSharp today

  • Registry wire framing for Confluent (4-byte IDs, and GUIDs with the __value_schema_id header), Apicurio (4- and 8-byte IDs) and AWS Glue (with or without zlib): AvroRegistryFraming, AvroRegistryMessage and AvroRegistryMessageReader, described in schema registries. Confluent's framing is byte-identical to Confluent's own serializer.
  • Schema references across subjects (AvroSchemaParser.AddNamedSchemas), and schema JSON (AvroSchema.ToJson) that is Java's Schema.toString() byte for byte, so registering AvroSharp's text finds the version a Java client registered.
  • Field transforms over the generic model (AvroValueTransformer), with each field's path and properties: the building block for data-quality rules, field-level encryption and masking.
  • Container files for anything that writes .avro files, such as Azure Event Hubs Capture: AvroFileReader, with every codec in AvroSharp.Codecs.
  • Serializers by type for frameworks that are handed a T or a Type, such as a Kafka serializer or AWS Lambda Powertools' Deserialize(byte[], Type): AvroTypes gives the schema and the read and write functions of every generated type, [AvroSerializable] ones included, and of the primitives string, int, long, float, double, bool and byte[], without reflection and on every target.

With Confluent.Kafka, use AvroSharp.Confluent below. Without it, an application registers its schema with Confluent's registry client, and produces and consumes byte[] values framed by AvroRegistryMessage and read by AvroRegistryMessageReader with a resolver (IAvroSchemaIdResolver) that fetches schemas by ID.

Add-on packages

Package For Docs
AvroSharp.Confluent Confluent.Kafka with Confluent Schema Registry, and registries with its API (Redpanda, Karapace, Apicurio) Guide
AvroSharp.KafkaFlow KafkaFlow producers and consumers, on AvroSharp.Confluent Guide
AvroSharp.Azure.SchemaRegistry Azure Schema Registry with Event Hubs and Service Bus (MessageContent) Guide
AvroSharp.Aws.Glue AWS Glue Schema Registry, fully managed, on every platform Guide
AvroSharp.Aws.Glue.Kafka Confluent.Kafka serializers for AWS Glue Schema Registry, on AvroSharp.Aws.Glue Guide

AvroSharp.Confluent has a guide of its own, and a sample. It is built on Confluent's own serializer base classes, so subject name strategies, auto-registration, use.latest.version, schema ID strategies and rules behave as they do with Confluent's serializer; field rules and migration rules aren't supported yet (#186, #187). It depends on Confluent.SchemaRegistry only, not on Apache.Avro. Its messages are byte-identical to Confluent's Avro serializer's, and both read each other's. The prototype serialized 8.7× and deserialized 6.5× faster, allocating 39% and 23% as much; the package's benchmarks are #197.

AvroSharp.KafkaFlow (its guide) is serializer middleware for KafkaFlow on AvroSharp.Confluent, in place of KafkaFlow's Confluent Avro serializer. It writes the same bytes, reads several record types from one topic, and finds the types without reflection when given them.

AvroSharp.Azure.SchemaRegistry (its guide) writes and reads MessageContent (and so EventData and ServiceBusMessage) as Microsoft's SchemaRegistryAvroSerializer does: the Avro body, and the schema ID in the content type (avro/binary+<id>). It uses Azure.Data.SchemaRegistry's SchemaRegistryClient.

AvroSharp.Aws.Glue (its guide) writes and reads AWS Glue Schema Registry's wire format (already in AvroSharp as AvroRegistryFraming.AwsGlue), and calls Glue through AWSSDK.Glue. Its Confluent.Kafka serializers are in AvroSharp.Aws.Glue.Kafka. AWS's own .NET package is a native build for Linux only; this one runs everywhere.

Other frameworks that wrap Confluent's serializers, such as Streamiz, Silverback and MassTransit, will be covered by AvroSharp.Confluent and documentation rather than packages of their own. Apache Iceberg manifests are planned after 1.0. Pulsar, AWS Lambda Powertools and CloudEvents are candidates that haven't been evaluated yet.

The add-on packages live in this repository and are released with AvroSharp, all at the same version (#77). Each depends on a tested range of its third-party library (#83). For example, Confluent changed its serializer base classes in a minor release (2.14.0), so AvroSharp.Confluent depends on [2.14.0, 3.0.0), and its tests run against both ends.

The ecosystem spike has the measurements, the findings for each candidate, and why the others were left out.

Edit this page