Container files
An Avro object container file (.avro) holds many records of one schema, in compressed blocks, with the schema in its header. On this page you write page-view events to a file, read them back without knowing the schema, compress the file, store your own metadata in its header, and append to and concatenate files. The examples use the generic data model, so they need no code generation; the full program is the ContainerFiles sample.
The types are in the AvroSharp.Containers namespace. The records are GenericRecord values of a PageView schema with a timestamp, a user ID, a URL, an optional referrer, a country and a duration. The sample makes 5,000 of them.
Write a file
AvroFileWriter.CreateGeneric takes a stream and the schema, and returns an AvroFileWriter<AvroValue>. You write records one at a time. The writer groups them into blocks, writes the header with the schema, and writes a sync marker after each block. Disposing the writer writes the last block.
The writer closes the stream when it is disposed, unless AvroFileWriterOptions.LeaveOpen is set, as it is here to keep the MemoryStream:
using var file = new MemoryStream();
using (var writer = AvroFileWriter.CreateGeneric(file, schema, new AvroFileWriterOptions { LeaveOpen = true }))
{
foreach (var pageView in pageViews)
{
writer.Write(pageView);
}
}
byte[] uncompressed = file.ToArray();
Read it back
AvroFileReader.OpenGeneric reads the header when it opens the file. The file carries the schema it was written with, so the reader needs no schema: WriterSchema holds it, and Codec the codec. ReadAll returns the records as AvroValues, one block at a time:
List<GenericRecord> read;
using (var reader = AvroFileReader.OpenGeneric(new MemoryStream(uncompressed)))
{
var written = (RecordSchema)reader.WriterSchema;
Console.WriteLine($"Writer schema: {written.FullName} with {written.Fields.Count} fields, codec: {reader.Codec.Name}");
read = reader.ReadAll().Select(value => value.AsRecord()).ToList();
}
Choose a codec
AvroFileWriterOptions.Codec compresses each block. The default is null, no compression. DeflateCodec is built in; the AvroSharp.Codecs package adds ZstandardCodec, snappy, bzip2 and xz. WriteFile is a helper in the sample that writes the records as above, with the options it is given:
byte[] deflated = WriteFile(schema, pageViews, new AvroFileWriterOptions { Codec = DeflateCodec.Default });
byte[] zstandard = WriteFile(schema, pageViews, new AvroFileWriterOptions { Codec = ZstandardCodec.Default });
Console.WriteLine($"Sizes: null {uncompressed.Length:N0} bytes, deflate {deflated.Length:N0}, zstandard {zstandard.Length:N0}");
For these 5,000 records it prints null 325,523 bytes, deflate 54,947, zstandard 56,449. The ratio depends on the data, so measure with your own.
A reader knows only null and deflate unless you give it more codecs. A file's codec isn't known until the file is opened, so a reader of files from elsewhere takes them all, with AvroCodecs.All in AvroFileReaderOptions.Codecs:
var readerOptions = new AvroFileReaderOptions { Codecs = AvroCodecs.All };
List<GenericRecord> decompressed;
using (var reader = AvroFileReader.OpenGeneric(new MemoryStream(zstandard), options: readerOptions))
{
decompressed = reader.ReadAll().Select(value => value.AsRecord()).ToList();
}
Store metadata in the header
The header is a map of string keys to byte values. AvroFileWriterOptions.Metadata adds your own entries, such as where the file came from. Keys that start with avro. belong to the specification, and the writer rejects them. On the reader, TryGetMetadataString decodes an entry as UTF-8, and Metadata holds all of them, including avro.schema and avro.codec:
var metadata = new Dictionary<string, ReadOnlyMemory<byte>>(StringComparer.Ordinal)
{
["source"] = Encoding.UTF8.GetBytes("web-frontend"),
["export.date"] = Encoding.UTF8.GetBytes("2026-09-01"),
};
byte[] withMetadata = WriteFile(schema, pageViews, new AvroFileWriterOptions { Codec = DeflateCodec.Default, Metadata = metadata });
string? source;
using (var reader = AvroFileReader.OpenGeneric(new MemoryStream(withMetadata)))
{
reader.TryGetMetadataString("source", out source);
Console.WriteLine($"Metadata: source = {source}, keys = {string.Join(", ", reader.Metadata.Keys.Order(StringComparer.Ordinal))}");
}
Read asynchronously
OpenGenericAsync and ReadAllAsync read the stream without blocking, for files on a network share or in blob storage. Each block is decoded synchronously once it is in memory. The writer has WriteAsync and DisposeAsync too.
long totalDuration = 0;
await using (var reader = await AvroFileReader.OpenGenericAsync(new MemoryStream(zstandard), options: readerOptions))
{
await foreach (var value in reader.ReadAllAsync())
{
totalDuration += value.AsRecord()["duration_ms"].AsInt32();
}
}
Append and concatenate
AvroFileWriter.OpenAppendGeneric opens an existing file to add records at its end, as log files grow: it reads the header for the schema, the codec and the sync marker, and the file reads as one afterwards. The stream must be readable, writable and seekable. AvroFile.Concat joins files of one schema block by block, without decoding the records: blocks of the first file's codec are copied as they are, and the others recompressed, so the uncompressed file and the deflate one here become one uncompressed file:
var log = new MemoryStream();
log.Write(uncompressed);
using (var writer = AvroFileWriter.OpenAppendGeneric(log, options: new AvroFileWriterOptions { LeaveOpen = true }))
{
writer.Write(pageViews[0]);
}
var combined = new MemoryStream();
AvroFile.Concat(combined, [new MemoryStream(log.ToArray()), new MemoryStream(deflated)]);
long combinedCount;
using (var reader = AvroFileReader.OpenGeneric(new MemoryStream(combined.ToArray())))
{
combinedCount = reader.ReadAll().LongCount();
Console.WriteLine($"Appended and concatenated: {combinedCount:N0} page views, codec {reader.Codec.Name}");
}
Both are built on blocks as stored: TryReadBlock reads one, and WriteBlock writes it to another file, recompressing it when the codecs differ. The avrosharp tool does the same from the command line, with file concat and file recodec.
Read as a newer schema version
Pass a reader schema, and the reader resolves each record from the file's schema to it. Here version 2 drops referrer and adds a device field with a default, so every record read has device set to "unknown":
GenericRecord first;
using (var reader = AvroFileReader.OpenGeneric(new MemoryStream(uncompressed), readerSchema: schemaV2))
{
first = reader.ReadAll().First().AsRecord();
Console.WriteLine($"Read as v2: {first}");
}
The rules for what may change between versions are in schema evolution.
Generated C# types read and write container files with AvroFileWriter.Create<T> and AvroFileReader.Open<T>, without the generic model: see the GeneratedTypes sample.
Next: getting started, schema evolution, JSON, logical types, and code generation.