Sunday, August 3, 2014

From zero to LDP: a tale of our 5 day journey/sprint to create an LDP impl from scratch in Node.js

Goals:

Highest priority, produce a Node.js® based W3C Linked Data Platform (LDP) implementation.  I will discuss some of the goals later (and in subsequent posts).

Result:
Had a BasicContainer and RDFSource implementation at the end of the week, passing all the automated test cases from the test suite.  Guessing we spent < 40 hours total on the work.  40 sounds like a lot, though we had a bit of learning curve in areas.  And it is 'live' at: http://ldpjs.mybluemix.net/ (assuming our autodeploys continue to be green).

Some background on us that did the development, well just Sam Padgett and Steve Speicher.  To be fair, we know a fair amount about the LDP spec already and have already done a reference implementation in Java in Eclipse Lyo.  Neither of us have done anything in Node.js other than a “Hello Node” sample.  Sam on the other hand, is an experienced JavaScript developer and Steve has stumbled his way through a few applications.

Day 1, we started the effort on Monday July 21.  We had sparkle in our eyes and two thermos full of coffee.  Then we were in meetings for a while and caught up on LDP test suite items.  Well, we did do some initial project structure setup to share the code and auto deploys to Bluemix (more on that later), read a little bit to determine a general direction we were ready to head.  After day 1, we had a place to store code, a sample app checked in, using express.js and starting to play with rdfstore.js API.

Day 2, day started off with the promise of all day code slinging.  We only got about 1/2 a day in, due to other OSLC and Lyo normal operations (WG meetings, code reviews, …).  We may a good dent in a simple GET/PUT working with rdfstore.js.  Though we were struggling to make a few things to work with rdfstore.js and Steve’s newbie callback hell was not improving his drinking problems (he says he doesn’t have a problem, he has it figured out).

Day 3, again about 1/2 a day of hacking…some progress with some of the BasicContainer features and various formats supported.

Day 4, realization that we should reconsider using rdfstore.js to store and parse RDF data.  The needs we have with LDP are quite simple.  We looked at mongoDB model and what we were doing, and looked at a simple format for JSON we dealt with using N3 library.  It was fairly straightforward to do, greatly simplified our dependencies and removed a couple barriers we were hitting with rdfstore.js.  We ended taking the graph-centric approach, where each JSON document in mongo is an RDF graph.  This approach, and drawbacks, is outlined over in dotNetRdf's blog post.

Day 5, complete transition to mongoDB, handle all GET/HEAD/OPTIONS/POST/DELETE, got viz service working, all tests green (well Sam did all/most of the hard LDP work, Steve was the “idea man” and “bounce ideas off man”).

Days 6-10 we were able to add JSON-LD support and full support for all variations of ldp:DirectContainers.
What next?  We’d like to solidify the impl more, a little more testing, and do non-RDFSource.  We’ve talked about building a sample app as well, something like a poor-mans bug tracker, photo album or address book.  Oh yes, and making the source code available.  That wasn't an initial priority just we didn't know how much would be worth sharing, we'll be going through approval processes to make it available.

Be on the look out for an upcoming blog post that talks about our experiences with DevOps Services and Bluemix.