blog.s11n.net

s11n for JSON

Hello, fans of C++ and JSON! (Everyone else can stop reading here.)

Recently i started another pet project, the nosjob library, which is a C++ library for creating and consuming JSON data. It provides a set of JSON-specified value types, like String, Boolean, Object, and Array, which can be combined to create data structures of near-arbitrary complexity. The library can output such structures to JSON form and parse such structures from JSON input.

One particularly interesting aspect of it is that it contains a bare-bones variant of libs11n with which it can de/serialize near-arbitrary client types from/to JSON.

For example, in its simplest form:

std::map<int,double> map;
for( int i = 0; i < 5; ++i ) {
    map[i] = i * 1.1;
}
nosjob::s11n::save( map, std::cout );

(Strangely enough, a std::map actually serializes to a JSON Array, instead of a JSON Object, in order to be able to support map keys of arbitrary type (JSON Object keys are always strings).)

More detailed examples can be found here.

Like libs11n, it uses a two-step serialization process: first serialize the “client data” to a JSON-compatible data structure, then format that JSON data structure to a given output stream (or output iterator). For deserialization, it it parses JSON input (from a stream or input iterator), builds up a tree of JSON-like value objects, and then feeds that data to the client-defined deserialization routine for final processing. The client provides a pair of conversion functors for converting their type to/from JSON-compatible form, and nosjob does the rest.

The serialization support does not rival libs11n’s in terms of flexibility and features, but it’s not supposed to.

If you’re (still) looking for easy-to-use JSON support for your C++ application, then nosjob might be interesting for you.

Happy Hacking!
Bob Ross is The Man!

—– stephan

PS: one might ask why not simply build this support into libs11n? By the time the nosjob code got to a point where i could add an s11n API for it, the effort and architectural overhead of including it into libs11n seemed too high. There are also disparities in the libs11n and JSON data models which would be awkward to consolidate (e.g. in s11n key/value pairs are only used for basic data types, where JSON allows keys to reference arbitrary other data types). Adding the s11n-like API to the library was a trivial, straightforward task, however: less than 50 lines of code, not including the built-in JSON-to/from-Native conversions which the nosjob::s11n bits use as the default de/serialization implementations (i had already written them for other reasons, which is why they don’t count as “effort” for this purpose).

Tip: speeding up s11n client compile times

As i’ve written in the past few blog entries, as part of the QBoard project i’ve been working on s11n proxies for serializing a variety of Qt types. That application has helped stress a lesson which i actually learned a long time ago, but had forgotten what an impact it makes. It potentially affects all s11n clients, so i thought i’d share it…

Two days ago, QBoard needed about 70 seconds to compile using parallel compilation on a dual-processor machine. After refactoring the s11n support a bit (as explained below), it now takes about 40 seconds. The extra 30 seconds were due 100% to libs11n, and we’ll see why below…

Any user of libs11n is probably familiar with the registration of s11n proxies. In its simplest form it looks like this:

#include <s11n.net/s11n/proxy/pod/int.hpp>

That registers the “int” type as a first-class Serializable. More commonly, for client-side types a registration looks like:

#define S11N_TYPE QVariant
#define S11N_TYPE_NAME "QVariant"
#define S11N_SERIALIZE_FUNCTOR s11n::qt::QVariant_s11n
#include <s11n.net/s11n/reg_s11n_traits.hpp>

That registers a proxy for QVariant.

That registration process does several things:

• Sets up a class template specialization for the client-specified type, such that calls to s11n::de/serialize(…,ThatType) will go through the appropriate API (in the above case, the QVariant_s11n functor).
• Sets up a classloader registration, so that deserializing of pointers can be done polymorphically. (That is actually the only reason in the world why libs11n has to associate a class name with types. For types which never get deserialized by pointer, this registration is never used!)
• There’s something else here which is slipping my mind at the moment.

In any case, the registration process is not (internally) 100% trivial, though the client has only to include a header file.

A registration must be visible to all client code which attempts to de/serialize that type, or else de/serialization will not work (that’s not true for some special cases, actually). This normally means including it in a common header file, and that’s where we hit our problem…

A registration creates several template specializations which the client will never see, but which nonetheless must be created by the compiler in order for s11n to do its work. That create happens in every client file which includes the registration, and can take a measurable amount of time per registered type (some experiments have shown a bit over 1 second per registered type!).

In the case of QBoard, we currently have registrations for almost 30 Qt types. Until a couple days ago, those registrations were all in a single header file which was included by all clients which wanted to de/serialize Qt types.

In the interest of a cleaner build tree, i split up the monolithic header into one header per serialized type, and then went back and updated the client code to only include the headers they needed. After doing so, the total build time was cut from about 70 seconds to about 40 seconds. The 30 extra seconds were taken up solely with the compilation of the s11n registration code.

So, the lesson is: try to give your s11n registrations the lowest visibility possible. Everywhere they are visible, they will be compiled, and where they are compiled they will take an observable amount of time to do so. In some cases it is possible to encapsulate all s11n-related calls into a small number of implementation files, invisible to the majority of the client code. The speed benefits of doing can be worth it. Consider, for example, if you save 30 seconds per build (as QBoard did), and it takes 30 minutes of work to refactor the registrations, then it will only take 60 compilation runs to win that time back. If you follow this lesson from the start, as opposed to refactoring later, you’ll get even more time benefits (because you obviously save the refactoring time).

Have fun, and happy hacking!

Storing arbitrary Serializables in QVariant objects

As part of the QBoard project i’ve been creating s11n bindings for Qt types which the app wants to serialize. The most interesting s11n work there has been the handling of Qt’s QVariant type. A QVariant object can hold a copy of various different types of data, e.g. QString, QPoint, and most of the other basic Qt data types. This is used heavily in the QObject Properties and QtScript APIs, and one can add support for their own types. One of the limitations is that QVariants always copy, instead of referencing, and this makes it not so useful for larger types or passing shared state around. Because it deeply copies, polymorphism goes out the window. There’s a way around that, though…

S11n supports polymorphic de/serialization if the Serializable type in question is set up for it (it’s not hard to do). When a Serializables is serialized, the data is not written directly to a stream, but is stored in intermediary objects called S11nNodes (basically a DOM tree of  serialized data). S11nNodes are themselves copyable. Since they carry all the info they need for polymorphic deserialization (and can copy it freely), S11nNodes themselves need not be polymorphic. So…

All we had to do was write a small wrapper type to integrate the S11nNode type with QVariant’s conventions. We can use that wrapper to serialize any SerializableType into a QVariant object. That looks like:

    #include "S11nQt.h"
    ...
    QVariant var( s11n::qt::VariantS11n(mySerializable) );

To deserialize a SerializableType object from the QVariant, we convert the QVariant to a VariantS11n and then deserialize the data through that object:

if( var.canConvert<s11n::qt::VariantS11n>() ) {
  s11n::qt::VariantS11n sv( myVariant.value<s11n::qt::VariantS11n>() );
  MySerializable * my = sv.deserialize<MySerializable>();
  ...
}

Note that S11nNodes can contain arbitrarily large amounts of data, and are therefor inherently not efficient to copy. However, VariantS11n employs reference counting and copy-on-write to keep the copying to a minimum. This makes const copy operations O(1) and non-const operations O(N), where N is the size of the S11nNode’s data (i.e. the number and size of the node’s properties, values, and child nodes).

The S11nQt code is part of the QBoard project, and can be found here:

http://code.google.com/p/qboard/wiki/S11nQt

Happy hacking!

Qt 4.3/4.4 and s11n

As part of a new hobby project of mine (QBoard), i’ve been writing utility code for serializing Qt-based types. In an shameless attempt to show a good use for libs11n, i’ve extracted that code into its own distribution. It may be helpful to coders working on Qt-based projects.

Because i don’t have access to the development copy of the s11n web site at the moment, the code is distributed via the QBoard project:

http://code.google.com/p/qboard/wiki/S11nQt

See that page for information about what Qt types are supported and to download a copy.

s11n and C++0x

A couple weeks ago i came across PEGTL (http://code.google.com/p/pegtl/) and that got me really interested in some of the new C++0x features. gcc 4.3 has beta support for 0x, and i’ve decided that i will start adding some optional parts to libs11n 1.3.x. While i’m not yet entirely sure that variadic templates will be useful in s11n, i’ve put together some code which shows some of the potential:

#include "s11n.net/s11n/s11nlite.hpp"
#include "s11n.net/s11n/s11n_debuggering_macros.hpp"
#include "s11n.net/s11n/proxy/pod/int.hpp"
#include "s11n.net/s11n/proxy/pod/double.hpp"
#include "s11n.net/s11n/proxy/pod/string.hpp"
#include "s11n.net/s11n/0x/ohex.hpp"

int main(int argc, char *argv[])
{
    int vi = 42;
    double vd = 42.42;
    std::string vs(”i’m a std::string”);
    try
    {
	s11nlite::node_type node;
	bool ret = s11n::cpp0x::serialize_v(node, vi, vd, vs);
        // ^^^ we can pass any number of Serializables here
	if( ret )
	{
	    s11nlite::save( node, std::cout );
	}
    }
    catch(std::exception const & ex)
    {
	CERR < < "EXCEPTION: " << ex.what() << '\n';
	return 1;
    }
    return 0;
}

That outputs:

#SerialTree 1
s11n_node class=s11n::s11n_node
{
        int class=int
        {
                v 42
        }
        double class=double
        {
                v 42.42
        }
        string class=string
        {
                v i'm a std::string
        }
}

This approach has some serious limitations (mainly involving naming of the children serialized this way), but as i get more comfortable with 0x i hope to find new ways to use it to empower s11n more. Your ideas are of course welcomed…

[A short while later...]

i’ve refined the above to de/serialize_group(), which can be used like this:

int main(int argc, char *argv[])
{
    int vi = 42;
    double vd = 42.42;
    std::string vs(”i’m a std::string”);
    try
    {
	s11nlite::node_type node;
	bool ret = s11n::cpp0x::serialize_group(node, “pods”, vi, vd, vs);
	if( ret )
	{
	    s11nlite::save( node, std::cout );
	}
	else
	{
	    throw s11n::s11n_exception( S11N_SOURCEINFO, “serialize_group() failed” );
	}

	int di = -1;
	double dd = -1;
	std::string ds = “error”;
	ret = s11n::cpp0x::deserialize_group(node, “pods”, di, dd, ds);
	if( ret )
	{
	    COUT < < "Deserialized group: ("<< di <<", "<< dd <<", ["<< ds <<"])\n";
	}
	else
	{
	    throw s11n::s11n_exception( S11N_SOURCEINFO, "deserialize_group() failed" );
	}
    }
    catch(std::exception const & ex)
    {
	CERR << "EXCEPTION: " << ex.what() << '\n';
	return 1;
    }
    return 0;
}

In addition to the output shown above, it shows us the results of the deserialization:

0x.cpp:36 : Deserialized group: (42, 42.42, [i'm a std::string])

[And a while later ...]

These variadics are cool. We can now easily serialize multiple child objects more flexibly than the above example:

serialize_subnodes( node, "myInt", di, "myDouble", dd, "myString", ds );

That takes pairs of (string,Serializable) and serializes each object into a subnode with the given name.
We can then deserialize with:

int myi; double myd; std::string mystr;
deserialize_subnodes( node, "myInt", myi, "myDouble", myd, "myString", mystr );

And the order of the (string,Serializable) pairs passed to deserialize_subnodes() is independent of the order passed to serialize_subnodes(). The first functions shown above do not have that property - they require the args in the same order. With deserialize_subnodes() we also have the option of partial deserialization - fetching a subset of the serialized data.

New s11n APT/.deb repository

At long last i’ve been tinkering with the .deb packaging tools. After some degree of effort i’ve been able to generate .deb packages for Ubuntu Gutsy and Nexenta/GnuSolaris.

Earlier today i set up an APT repository for distributing s11n via the APT tools. It’s over at apt.s11n.net - just browse over there to get the instructions (e.g. the list of APT repo URLs).

The repo currently has s11n 1.3.0 for Ubuntu Gutsy and Nexenta/GnuSolaris. The plan is to create packages for the latest s11n 1.2.x, and possibly add separate packages for the docs (API + manual).

Contributions of .deb packages for platforms i don’t have access to is always welcomed. Just get in touch with me via http://s11n.net/home/stephan/.

Update 26 Apr 2008: i’ve gone ahead and released 1.2.6, including .debs for Gutsy and GnuSolaris.

libs11n .deb packages for *buntu and GnuSolaris

The first .deb packages for libs11n 1.3, targeting i386 *buntu/Debian systems and i386 Nexenta Gnu/Solaris have been released: http://s11n.net/download/. Contributions of new precompiled binaries/packages for other systems are always welcomed!

These releases contain the headers, the DLLs, and the s11nconvert binary. They don’t contain the doxygen API docs nor the library manual - those might be shipped in their own packages someday.

libs11n on OpenSolaris

Yesterday i came across NexentaOS, which is an OpenSolaris kernel in an environment hosting the GNU toolset and the Debian apt tools (which are, IMO, the best package management tools out there). Nice stuff, and Nexenta is currently the only OpenSolaris distro which i would personally consider installing (mainly because of its GNU tools base and the apt tools).

Anyway… it turns out that libs11n will compile as-is on NexentaOS (which uses an antiquated gcc 4.0.x). i guess it’s about time to start building Debian-format source packages for s11n.