This morning i wrote a small C application which can be used to fetch data from sqlite3 and MySQLv5 databases and convert it to JSON. While it was originally written as a test app for my C JSON library, it certainly has other uses.

It can be found here:

http://fossil.wanderinghorse.net/repos/cson/index.cgi/wiki/select-to-json

It “should” compile fine out of the box on any Linux-like system with gcc, bash, and GNU Make. Building on other platforms requires either converting the included Makefile logic to the platform’s build tool, or linking it against the “amalgamation build” of the underlying JSON library. The code is portable ANSI C-89 and should not require any platform-specific fixes.

Happy Hacking!

—– stephan beal

A few months ago i started writing cson (pronounced “season”), a C library for generating and consuming JSON data using an OO interface (as opposed to a printf()-like API). The past few days i’ve extended it to support a mini-framework for writing JSON-only CGI applications. It’s biggest claim to fame is that it can use HTTP cookies (or a client-provided session ID) to manage persistent application sessions, which are stored (of course) as JSON. It currently supports using files, sqlite3, or mysql5 for session storage, and can easily be extended to support new back-ends. (So far i have been unable to find any other C/C++ CGI libraries which support persistent sessions.)

If you just happen to be one of the 3 or 4 remaining C hackers who like to write CGI apps, and you are also a big fan of JSON, the code might be of interest to you:

http://fossil.wanderinghorse.net/repos/cson/index.cgi/wiki/cson_cgi

The JSON-based session support is actually independent of the CGI bits, and can be used without HTTP/CGI:

http://fossil.wanderinghorse.net/repos/cson/index.cgi/wiki/cson_session

Here’s a demo:

http://fossil.wanderinghorse.net/demos/cson/

Happy Hacking!

—– stephan beal

i just have to ask…

As part of the cpdo project i would really like to start work on a database driver for Oracle (based on ocilib, which i quite like and is well documented). My main limiting factor here is access to a Unix/Unix-ish machine with the necessary Oracle bits.

If there is anyone out there who could give me (unprivileged!) SSH access to such a machine, you would be doing me a great service. My requirements would be:

• Some form of Linux, BSD, Solaris, etc.
• A decent C compiler and related build tools.
• Access to link to libclntsh.so (some admins lock that away in a dir only readable by the oracle user and/or dba group).
• A decent editor: xemacs (preferred) or emacs. (Note that xemacs works fine without X11, and i would not need to tunnel X11 over ssh.)
• Access to a single Oracle database with minimal space requirements (maybe a meg or two, at the very most).
• i think that i need Oracle v10 or higher, but to be honest i’m not certain.
• Far under 1% (amortized) of your total CPU capacity, with most of that coming from the C compiler and the linker.
• Initially i would need a total of probably 36-48 hours of login time, and would of course like to have further access to improve upon the driver over time (but i would settle for the initial block if that’s all i can get).

i would of course Do No Evil (or even Mischief) on your machine - my sole purpose would be to develop and test this driver. i would of course also sign any disclaimers, waivers, etc. which you would require, as long as copyrights to the source code developed there are not transfered from me (the code will remain free and open source, like the should be).

Is there someone out there who could help a brother out? If so, please get in touch (contact info is at http://wanderinghorse.net/home/stephan/).

i actually have tried to get Oracle running, but after spending a whole day trying, and never getting past the installer, i gave up. (Oracle for Linux requires truly ancient system libraries which have not been seen on my PCs since about the time of Bill Clinton’s presidency.)

Thanks once again for reading, and Happy Hacking!

—– stephan beal

Inspired by a recent project at work, where i had to write a DB abstraction layer for wrapping MySQL and Oracle, i’ve started a new database abstraction library for C:

http://fossil.wanderinghorse.net/repos/cpdo/

cpdo gets its name from the PHP PDO API, after which it is modeled.

It comes with a driver for sqlite3. Work started on a MySQL driver, but their prepared statements API is essentially unusable from such an abstraction layer (as best as i can determine, anyway), so i gave up. (Update: and later got it working - see below.)

The original wrapper (part of a commercially-supported add-on for the Nagios monitoring framework) only needed “plain SQL” support, not prepared statements, and thus worked fine with MySQL and Oracle. (We replaced their usage of libdbi so that they can support Oracle.) With cpdo, i decided to “implement it properly” and internally use only prepared statements for all SQL. This, however, this appears to be essentially impossible using the MySQL API, so we’re left with an abstraction layer which abstracts only sqlite3 for the time being.

If there are any MySQL-loving C gurus out there with too much energy on their hands who would like to help implement the MySQL binding (or for some other database), please get in touch. i’ve implemented the driver skeleton and maybe 1/3rd of the code (e.g. connection, disconnection, and the first step of preparing statements), but can’t proceed because the MySQL prepared statements API is so insane. Update 2011-Jan-31 @ 00:17: The MySQL driver is now basically working. There’s more testing to be done, in particular with BLOBs and large string fields, but it appears to work. Thankfully, i found a useful example (most aren’t!) of how to use the MySQL in the PHP source tree (for their mysqli bindings), and that made all the difference.

Happy Hacking!

—– stephan beal

(If you have no idea what a “v8 hacker” is then this post will mean nothing to you.)

i would like to announce a new v8-based project…

v8::convert is a header-only utility library for Google v8 with a very small focus:

• Converting between v8 and native values, including client-defined types.
• Converting C/C++ functions/methods to v8::InvocationCallback implementations.

With those two features we have almost everything we need for most class-binding purposes. The the ability to convert functions to have v8::InvocationCallback signatures has proven especially useful, and allows us to plug in in many C-standard or 3rd-party APIs with ease. Conversions are compile-time typesafe where possible and runtime conversion failures fail gracefully (e.g. by throwing a JS-side exception).

The core components are in no way project-specific, and can be used in conjunction with arbitrary other function/class binding mechanisms. Since the implementation is header-only, it’s particularly easy to include into other projects.

The API is trivial to use. The client API contains a good number of classes and functions, but most client code needs only one or m/ore of the 5 main function templates: CastToJS(), CastFromJS(), FunctionToInvocationCallback(), MethodToInvocationCallback(), and ConstMethodToInvocationCallback().

The missing parts are:

- Binding JS accessors to native values/functions. i have the code in v8-juice but need to port it to this API’s conventions.
- A class binding mechanism which takes care of the WeakPointer bits. There is such a mechanism in v8::convert, but it is not considered a core component because hackers tend to each have their own favourite class binding mechanism.

The code is largely a spin-off of the core-most v8-juice components, but the function-binding parts were re-implemented from scratch so that we can use function-pointer-style template parameters (which, IMO, are more readable than the v8-juice’s approach). Additionally, instead of inheriting v8-juice’s huge collection of weirdly-named function-binding templates, clients now need only 3 templates to bind functions (one each for free functions, const- and non-const member functions).

The idea of doing a header-only spin-off has been brewing for some time, but i was recently inspired to finally do the work by two other v8-users list members:

- James Pike: when i saw how vu8 takes advantage of function-pointer-style template parameters, i just had to have that feature for myself.

- Fuzzy Spoon: the separation of const- and non-const member handling was prompted by Fuzzy’s report about MSVC not being able to select the proper template specializations when the the member functions differ only in constness.

v8::convert is only about 3 nights old, but already has at least as much API documentation as v8 itself. There is an overview and example code here:

http://code.google.com/p/v8-juice/wiki/V8Convert

The source code is at:

http://code.google.com/p/v8-juice/source/browse/#svn/convert

It includes a standalone demo application and a demo v8-juice plugin.

Happy Hacking!
—– stephan beal

We’re all aware of the “dangers” of adding comments to implementation code, such as:

int x = 32; // default value was decided by committee...

In practice, comments tend become disassociated from their code, especially when more than one person works on the same code. For example, the following is relatively common in real-world code:

// default value was decided by committee...
int y = 7;
int x = 32;

With that change, we’re not really sure which default value the comment is referring to.

Recently i stumbled across a commenting approach which has helped me avoid this type of problem:

int x = 32 /*default value was ...*/;

Note that the comment is inside the statement, before the semicolon. The side-effect of that is that coders who edit that line are much more likely to have to deal with the comment. For longer comments, it helps to split up the lines:

int x = 32
  /* default value was ...
     on 20101214 ... by ....
   */
;

Again, the comment is inserted as part of the statement. If someone wants to move that line, they’ll be forced to acknowledge that “something unusual” is happening, and are likely to read the comment to figure out what’s standing in their way.

It’s a simple to apply approach which i’ve started using in all my code, and i nowadays i recommend it as standard-practice over adding comments in the more conventional forms:

// before the code:
int x = ...;

int y = ...;
// ^^^^ after the code, with carets meaning "look up to the previous line"

int z = ...; // after the semicolon

Happy Hacking!

—- stephan beal

PS: there are people out there who despise any form of comments, and i was once told (on a popular open source project) that having comments in my code contributions made an “unprofessional” impression. You may disagree that comments are useful/necessary/whatever in code, and that’s fine. Let’s just agree to disagree.

Update on 2010.12.20: some of you will simply laugh and say, “my programming language of choice doesn’t use semicolons” or “my programming language uses semicolons as comment markers!” To you i say, “how sad!”

While i tend to write absurd amounts of technical documentation, mostly in the form of API docs and other code documentation, it’s been several years since i’ve published a technical article.

For those 5 or 6 remaining aspiring C programmers in the world, i’ve got a new article which demonstrates a useful 999(shit, my left shift key just quit working…)… ahem… demonstrates a useful (in my opinion) model for implementing object-oriented data structures and APIs in C. DAMMIT, AND NOW MY ENTER KEY BROKE!

(thank god for the numeric pad’s enter key…)

Since my keyboard just died, i’ll keep this short… the article is here:

http://wanderinghorse.net/computing/papers/

happy hacking1

9you never truly appreciate the left shift key until it’s gone…0

—– stephan beal

Update:

LOL! (i’ve now got my backup keyboard…) At the very same time my keyboard broke i began being affected by this new Google Chrome bug:

http://code.google.com/p/chromium/issues/detail?id=66071

(Short version: mouse wheel cannot scroll up, but can scroll down.)

i thought, because my keyboard and mouse (in Chrome, anyway) weren’t working, that maybe my USB bus was dying.

For a couple years now i’ve been maintaining a project called whio, the wanderinghorse.net I/O library for C. In short, it’s a collection of input/output APIs for working with sequential streams and random-access devices. It allows one to wrap a variety of lower-level I/O APIs (e.g. standard FILE handles or file descriptors), or implement custom ones, behind a uniform interface. Over the years it has grown to include a good deal of functionality.

i’ve spent a good deal of time looking for and investigating storage-based hashtables for C over the years (B-trees as well), and have found depressingly few of them (and none which i could drop in and re-use without modification). The few i did find were limited to using C-standard FILE handles as storage, and FILE handles don’t always suit my I/O needs. The whio_dev API has several different device implementations, not just for files, and i wanted a hashtable with that flexibility. So of course i had to write one myself ;).

A few days ago i added a new API to whio, called whio_ht, which is an on-storage hashtable.

A very brief summary of its properties:

• Insertion, searching, and removal are all amortized O(1). This means that the time it takes to insert, search for, and remove records is essentially independent of the number of records in the hashtable.
• Constant memory usage of less than 500 bytes. (There is one exceptional case involving “really long” keys, but it is not expected to that most clients will fall into that category.)
• Can use any storage back-end for which we have a whio_dev implementation. e.g. it can be used with a memory buffer device or embedded as a pseudofile within a whio_epfs container.
• Data file format is platform-neutral. Encoding of keys/values, if necessary, is left to the client.
• Supports arbitrary key types, provided the user can supply hashing and comparison functions for them. The library includes routines for hashing/comparing strings and the standard fixed-sized integer types (from stdint.h).

If you’re still looking for a storage-based hashtable API in C, whio_ht might just be of interest to you.

Happy Hacking!

—– stephan beal

Recently i started another pet project, the nosjob library, which is a C++ library for creating and consuming JSON data. It provides a set of JSON-specified value types, like String, Boolean, Object, and Array, which can be combined to create data structures of near-arbitrary complexity. The library can output such structures to JSON form and parse such structures from JSON input.

One particularly interesting aspect of it is that it contains a bare-bones variant of libs11n with which it can de/serialize near-arbitrary client types from/to JSON.

For example, in its simplest form:

std::map<int,double> map;
for( int i = 0; i < 5; ++i ) {
    map[i] = i * 1.1;
}
nosjob::s11n::save( map, std::cout );

(Strangely enough, a std::map actually serializes to a JSON Array, instead of a JSON Object, in order to be able to support map keys of arbitrary type (JSON Object keys are always strings).)

More detailed examples can be found here.

Like libs11n, it uses a two-step serialization process: first serialize the “client data” to a JSON-compatible data structure, then format that JSON data structure to a given output stream (or output iterator). For deserialization, it it parses JSON input (from a stream or input iterator), builds up a tree of JSON-like value objects, and then feeds that data to the client-defined deserialization routine for final processing. The client provides a pair of conversion functors for converting their type to/from JSON-compatible form, and nosjob does the rest.

The serialization support does not rival libs11n’s in terms of flexibility and features, but it’s not supposed to.

If you’re (still) looking for easy-to-use JSON support for your C++ application, then nosjob might be interesting for you.

Happy Hacking!
Bob Ross is The Man!

—– stephan

PS: one might ask why not simply build this support into libs11n? By the time the nosjob code got to a point where i could add an s11n API for it, the effort and architectural overhead of including it into libs11n seemed too high. There are also disparities in the libs11n and JSON data models which would be awkward to consolidate (e.g. in s11n key/value pairs are only used for basic data types, where JSON allows keys to reference arbitrary other data types). Adding the s11n-like API to the library was a trivial, straightforward task, however: less than 50 lines of code, not including the built-in JSON-to/from-Native conversions which the nosjob::s11n bits use as the default de/serialization implementations (i had already written them for other reasons, which is why they don’t count as “effort” for this purpose).