Fast and compact object serialization in .NET

I want to use object serialization to communicate over the network between a Mono server and Silverlight clients. It is pretty important that serialization is space efficient and pretty fast, as the server is going to host multiple real time games.

What technique should I use? The BinaryFormatter adds a lot of overhead to serialized classes (Version, culture, class name, property names, etc.) that is not required within this application.

What can I do to make this more space efficient?

You can use Protocol Buffers. I'm changing all my serialization code from BinaryFormatter with compression to Protocol Buffers and obtaining very good results. It's more efficient in both time and space.

There are two .NET implementations by Jon Skeet and Marc Gravell.

Update: Official .NET implementation can be found here.

I have some benchmarks for the leading .NET serializers available based on the Northwind dataset.

@marcgravell binary protobuf-net is the fastest implementations benchmarked that is about 7x faster than Microsoft fastest serializer available (the XML DataContractSerializer) in the BCL.

I also maintain some open-source high-performance .NET text serializers as well:

• JSV TypeSerializer a compact, clean, JSON+CSV-like format that's 3.1x quicker than the DataContractSerializer
• as well as a JsonSerializer that's 2.6x quicker.

As the author, I would invite you to try protobuf-net; it ships with binaries for both Mono 2.0 and Silverlight 2.0, and is fast and efficient. If you have any problems whatsoever, just drop me an e-mail (see my Stack Overflow profile); support is free.

Jon's version (see the earlier accepted answer) is also very good, but IMO the protobuf-net version is more idiomatic for C# - Jon's would be ideal if you were talking C# to Java, so you could have a similar API at both ends.

I had a similar problem, although I'm just using .NET. I wanted to send data over the Internet as quickly and easily as possible. I didn't find anything that would be optimized enough, so I made my own serializer, named NetSerializer.

NetSerializer has its limitations, but they didn't affect my use case. And I haven't done benchmarks for a while, but it was much much faster than anything else I found.

I haven't tried it on Mono or Silverlight. I'd bet it works on Mono, but I'm not sure what the level of support is for DynamicMethods on Silverlight.

You could try using JSON. It's not as bandwidth efficient as Protocol Buffers, but it would be a lot easier to monitor messages with tools like Wireshark, which helps a lot when debugging problems. .NET 3.5 comes with a JSON serializer.

You could pass the data through a DeflateStream or GZipStream to compress it prior to transmission. These classes live in the System.IO.Compression namespace.

I had a very similar problem - saving to a file. But the following can also be used over a network as it was actually designed for remoting.

The solution is to use Simon Hewitt's library - see Optimizing Serialization in .NET - part 2.

Part 1 of the article states (the bold is my emphasis): "... If you've ever used .NET remoting for large amounts of data, you will have found that there are problems with scalability. For small amounts of data, it works well enough, but larger amounts take a lot of CPU and memory, generate massive amounts of data for transmission, and can fail with Out Of Memory exceptions. There is also a big problem with the time taken to actually perform the serialization - large amounts of data can make it unfeasible for use in apps ...."

I got a similar result for my particular application, 40 times faster saving and 20 times faster loading (from minutes to seconds). The size of the serialised data was also much reduced. I don't remember exactly, but it was at least 2-3 times.

It is quite easy to get started. However there is one gotcha: only use .NET serialisation for the very highest level datastructure (to get serialisation/deserialisation started) and then call the serialisation/deserialisation functions directly for the fields in the highest level datastructure. Otherwise there will not be any speed-up... For instance, if a particular data structure (say Generic.List) is not supported by the library then .NET serialisation will used instead and this is a no-no. Instead serialise the list in client code (or similar). For an example see near "'This is our own encoding." in the same function as listed below.

For reference: code from my application - see near "Note: this is the only place where we use the built-in .NET ...".

You can try BOIS which focuses on packed data size and provides the best packing so far. (I haven't seen better optimization yet.)

https://github.com/salarcode/Bois