Web API maturity model

Web APIs (REST-based HTTP services with simple payloads, typically JSON) have proven to be a very simple and effective way for applications to consume capabilities provided by apps or expose some capabilities themselves. As we've experienced, there are various ways these APIs are defined from satisfying a large audience based on a single server (Twitter) to something more general that handles various kinds of resources and operations.

The reality of many of these Web APIs, their purpose and needs have evolved from their early inception and then evolving into their latest 3.1 version with additional bells and whistles, deprecating some baggage along the way.

So the title of this post is "Web API maturity model", so what is this and why is it needed?  Perhaps maturity model isn't the right title for it, let me describe the aspects of it and give some examples.  This looks a bit beyond Martin Fowler's article about this to go beyond his levels.

Basically it is a way to talk about how Web APIs are development and the various needs as the customer needs evolve and therefore the APIs evolve to meet those needs.

Maturity levels (loosely sketched out as):

1. REST+JSON (barely docs) - this is barebones, your app defines a simple Web API.  The goals at this level are not to make it rock solid and cover ever edge case but instead to focus on supporting an important scenario quickly, getting feedback on usage and evolve from there (if needed).  You rely on a sample, an incomplete wiki page or reading the source to figure out how best to use it.
2. REST+JSON (better docs, source) - a better web page dedicated to describing the API, samples and some source.  At this point you are promoting it a bit more as "the API" for your service.
3. REST+JSON (LD, docs, source) - needing a way to describe the payloads more consistently between various endpoints, could reuse existing vocabularies such as schema.org.  App developers may even want to share some of their vocabularies or capabilities
4. LDP (REST+JSON-LD&Turtle+containers) - needing a way to have a consistent pattern for interacting with "buckets" of like resources
5. Domain extensions (OSLC, etc) - needs around app development, systems engineering, ... which could create new vocabularies or capabilities themselves

In fact, these layers of maturity can be cleanly applied on one another without disrupting previous apps that consume the API.

Some may say "ya, dumbass this is just how we work", which they are right (at least not the dumbass part).  We often start small, find what works and evolve from it.  Though it is good sometimes to be clear and intentional in that is how you plan to operate as some look building integration APIs in a way that starts in the middle or end, instead of at level 1.  Though learning about various techniques and technologies in other levels helps guide developers to upgrade their mental toolbox of skills.

The levels I highlighted are just examples, incomplete and could even be viewed that there are different axis that might emerge.  Related to that is understanding in what scenarios stepping up the levels, or over to another axis, may make sense.  I plan to explore some more of these ideas, including how best to define, present, collaborate, evolve, enable all this. 

As always, interested to hear feedback (I prefer agreement but accept other forms as well)

Considering the right solution to your integration problem

So often we often get tied up in a specific subject or technology area it feels like we see this is the everything it is just to help people think different.
only game in town.  It reminds me of the Jim Carrey film "Yes man" (by the way, I relate all my work to Jim Carrey films -- please no "Dumb and Dumber "references ;-) where he takes part in a program to change his life by answering "Yes" to any question or opportunity.  He learns, it doesn't apply to all situations but just trying to force people to see a different perspective by opening up to a new way of approaching situations.

I relate my work in OSLC and Linked Data work in a similar way.  It is not the only solution but thinking this way, apart from some traditional ways of thinking about integrations, helps to find alternative solutions that could end up being more loosely-coupled, scale better and are more resilient to upgrades.  Other benefits are that it allows the data to stay where it can be properly protected via access-control and updated as needed.

Often people and companies are so passionate about a technology or solution, they often answer the customer's question before they ever fully hear what the problem is.  There are varying degrees of this, of course if your job is to sell a product then that is then it is hard to imagine that you'd recommend an alternative solution.  Though I think if you worked with the customer to determine it isn't the right fit or the trade-offs, they would be more willing to continue to attempt to business with you.

There are so many factors on deciding what is the right integration solution for your current integration problem.  It would be fantastic if I could clearly define a concise decision tree that fit on a cocktail napkin and handled 90% of the cases...unfortunately, it is not that easy (and a group of integration consultants might hunt me down).  I've worked with a number of customers to identify possible solutions to either simple 1-to-1 integration problems to defining a corporate integration architecture/approach and plan.

Here's some factors that I typically consider to drive a recommendation:

• problem statement & user stories, including any other constraints
• # of tools
• anticipated growth or consolidation
• integration technology already available in tools landscape
• ability to develop integrations
• timeframe and ownership

As I come up with more, I will add to the list but I'd be interested to hear what other considerations people have when tackling integration problems.  I'd like to elaborate more on each of these points and weight them compared to each other as well.

Creating Containers in LDP, it is just this easy

I've heard a number of times that it isn't very clear how one creates a Linked Data Platform (LDP) Container.

Let's start with an example of how to create a simple resource (say LDP RDF Source).  You need to know a URL that accepts POST of the media type you want.
Here's the POST request:

1:  POST /netWorth/nw1/liabilities/ HTTP/1.1  
2:  Host: example.org  
3:  Accept: text/turtle  
4:  Content-Type: text/turtle  
5:  Content-Length: 63  
6:    
7:  @prefix dcterms: <http://purl.org/dc/terms/>.
8:  @prefix o: <http://example.org/ontology>.
9:
10:  <> a <http://example.org/ontology#Liability>;
11:       dcterms:title "Home loan";
12:       o:value 200000.00 .
13:    # plus any other properties that the domain says liabilities have  

(Example "borrowed" from LDP spec example)

s Very simple, just POST some Turtle to a URL.

Now let's look at what it would look like to create a Container:

1:  POST /netWorth/nw1/liabilities/ HTTP/1.1  
2:  Host: example.org  
3:  Accept: text/turtle  
4:  Content-Type: text/turtle  
5:  Content-Length: 72  
6:  Link: <http://www.w3.org/ns/ldp#BasicContainer>; rel="type"  
7:  
8:  @prefix dcterms: <http://purl.org/dc/terms/>.
9:  @prefix o: <http://example.org/ontology>.  
10:
11:  <> a <http://www.w3.org/ns/ldp#BasicContainer> ;
12:       dcterms:title "Home loans" ;
13:       o:limit 500000.00 .   
14:    # plus any other properties  

That's it, just POSTing some content just like before. I added a Link header (line #6) on the POST request to be explicitly clear that I want this newly created resource to behave as an ldp:BasicContainer. Note that I wanted to be expressive so I included the type triple on line 11, though the spec doesn't require this as the Link header as the spec-defined way to clearly assign the desired behavior.

There are various other ways that ldp:Containers can come into existence, most of those reasons are dependent on the server's application-specific rules.  For example, an application might allow the creation of a resource of say "Customer", the application logic may determine to create a number of supporting Containers for the new Customer resource, such as which assets and liabilities they have.

OSLC Specification Update (OASIS TCs, open-services.net WGs), 3rd Quarter 2014 report

I regularly give an overall specification, Technical Committee and Working Group update for the OSLC Steering Committee.  I thought it would be useful to share it more broadly as well.  Intended to be a brief high-level update, perhaps quarterly. (if I missed anything or misspoke, let me know I'll get it fixed up)

OASIS OSLC Core TC

• Participation from PTC, Mentor Graphics, Tasktop, Software AG, Sodius, IBM and recently joined Dassault
• Various updates around template specification, specification authoring tools using ReSpec, Core 3.0 overview, resource preview stabling and starting delegated dialogs.
• Received "attachments" specification from CCM TC as Core draft
• Updates on naming policies and usage of http://open-services.net for namespace

OASIS OSLC Change and Configuration Management TC

• Participation from IBM, PTC, Mentor Graphics, Boeing
• Various updates to Configuration specification based on review feedback
• Splitting out Change Management 3.0 spec into separate capabilities: States, Severity/Priority, Resource types, Attachments (to core)

OASIS OSLC Automation TC

• Participation from IBM, Mentor Graphics
• Contribution and review of scenarios for such things as: Automation in Systems Engineering and Model transformation

OASIS OSLC PROMCODE TC

• Participation from Fujitsu, NEC, IBM, Nanzan University
• Refinement of overall model.  Working to define how best to leverage Estimation and Measurement work, specifically around usage in ScopeItem and measurement units (see minutes).
• An initial draft of the vocabulary and shapes has been created.

OSLC Core WG

• Updates to Tracked Resource Set 2.0 and Indexable Linked Data Provider guidance to align with changes made in: LDP, LDP-Paging and LD-Patch

OSLC Automation WG

• Closing down work on Automation 2.1 and Actions, incorporating review feedback and preparing to transfer to OASIS TC

From zero to LDP: a tale of our 5 day journey/sprint to create an LDP impl from scratch in Node.js

Goals:

Highest priority, produce a Node.js® based W3C Linked Data Platform (LDP) implementation.  I will discuss some of the goals later (and in subsequent posts).

Result:
Had a BasicContainer and RDFSource implementation at the end of the week, passing all the automated test cases from the test suite.  Guessing we spent < 40 hours total on the work.  40 sounds like a lot, though we had a bit of learning curve in areas.  And it is 'live' at: http://ldpjs.mybluemix.net/ (assuming our autodeploys continue to be green).

Some background on us that did the development, well just Sam Padgett and Steve Speicher.  To be fair, we know a fair amount about the LDP spec already and have already done a reference implementation in Java in Eclipse Lyo.  Neither of us have done anything in Node.js other than a “Hello Node” sample.  Sam on the other hand, is an experienced JavaScript developer and Steve has stumbled his way through a few applications.

Day 1, we started the effort on Monday July 21.  We had sparkle in our eyes and two thermos full of coffee.  Then we were in meetings for a while and caught up on LDP test suite items.  Well, we did do some initial project structure setup to share the code and auto deploys to Bluemix (more on that later), read a little bit to determine a general direction we were ready to head.  After day 1, we had a place to store code, a sample app checked in, using express.js and starting to play with rdfstore.js API.

Day 2, day started off with the promise of all day code slinging.  We only got about 1/2 a day in, due to other OSLC and Lyo normal operations (WG meetings, code reviews, …).  We may a good dent in a simple GET/PUT working with rdfstore.js.  Though we were struggling to make a few things to work with rdfstore.js and Steve’s newbie callback hell was not improving his drinking problems (he says he doesn’t have a problem, he has it figured out).

Day 3, again about 1/2 a day of hacking…some progress with some of the BasicContainer features and various formats supported.

Day 4, realization that we should reconsider using rdfstore.js to store and parse RDF data.  The needs we have with LDP are quite simple.  We looked at mongoDB model and what we were doing, and looked at a simple format for JSON we dealt with using N3 library.  It was fairly straightforward to do, greatly simplified our dependencies and removed a couple barriers we were hitting with rdfstore.js.  We ended taking the graph-centric approach, where each JSON document in mongo is an RDF graph.  This approach, and drawbacks, is outlined over in dotNetRdf's blog post.

Day 5, complete transition to mongoDB, handle all GET/HEAD/OPTIONS/POST/DELETE, got viz service working, all tests green (well Sam did all/most of the hard LDP work, Steve was the “idea man” and “bounce ideas off man”).

Days 6-10 we were able to add JSON-LD support and full support for all variations of ldp:DirectContainers.
What next?  We’d like to solidify the impl more, a little more testing, and do non-RDFSource.  We’ve talked about building a sample app as well, something like a poor-mans bug tracker, photo album or address book.  Oh yes, and making the source code available.  That wasn't an initial priority just we didn't know how much would be worth sharing, we'll be going through approval processes to make it available.

Be on the look out for an upcoming blog post that talks about our experiences with DevOps Services and Bluemix.

Trip Report - INCOSE IS 2014 Systems Engineering meets (needs) open integrations + 2025 vision

I was fortunate enough to be able to attend the INCOSE International Symposium (IS) 2014, an event I've never been to before.  I've been spending more and more time over the years with Systems Engineering, it was good to learn more, share a bit more what it going on with OSLC and related topics, catch up with some friends and help some implementors lay out a plan.

One of the things that struck me as interesting is how the speakers and attendees referred to OSLC.  I'm used to seeing so many presentations over the years defining it, spelling out what the acronym means, etc.  At the IS, there was none of that.  It was just referred to by name, as everyone knows clearly what it is.  I didn't hear anyone asking or taking a note to look it up later.  OSLC was often referred to as an area which showed great promise for SE tool interoperability: as a protocol to exchange data, a way to define a minimal data model at web scale and simple ways of doing UI integration.

We had an impromptu meet up at lunch for OSLC, we in fact had too many people at the table (and yes I was the only IBMer).  It included people from PTC, Atego, JPL, Deere, Koneksys, Eclipse Foundation. Great discussion to share peoples interest, share what things are in motion and look for a way to coordinate all the activity going on in all the different places: INCOSE TII, OASIS OSLC, OMG OSLC4MBSE and more.  Looking forward to following up with this group and seeing how it advances.

I was able to give an overview and update on OSLC to an audience the represented many industries: automotive (2), air & space (2) and large machinery.

An interesting piece of work that I received when I registered what INCOSE's System's Engineering Vision for 2025, specifically these items:

• Foundations and Standards (p. 20) 
  "This systems engineering body of knowledge today is documented in a broad array of standards, handbooks, academic literature, and web-resources, focusing on a variety of domains. A concerted effort is being made to continually improve, update and further organize this body of knowledge. "
• Current Systems Engineering Practices and Challenges (p. 20-21) practice areas of "Modeling, Simulation, and Visualization", "Design Traceability by Model-Based Systems Engineering" which highlight the growing needs around improved tools and tool interoperability.
• Leveraging Technology for Systems Engineering Tools (p. 30)
  Discusses the need to move towards a set of tools that allow for: "high fidelity simulation, immersive technologies to support data visualization, semantic web technologies to support data integration, search, and reasoning, and communication technologies to support collaboration. Systems engineering tools will benefit from internet-based connectivity and knowledge representation to readily exchange information with related fields."

I'm hoping to make it to Boston around September 10th to run an OSLC workshop for the INCOSE community, stay tuned.

Rational User Conference (aka IBM Innovate) Take #10

Last week I attended my 10th (yes I said one-zero or tenth) Rational User Conference (aka IBM Innovate, aka Rational Software Developer User Conference, aka Rational Software Conference).  It is also the 5th time I have attended while talking about OSLC.  Hard to believe that Mik Kersten and myself did the first ever OSLC presentation back in 2009.  It has been interesting to be part of the transition from people hearing "O S L C" and having no idea, to today where most attendees not only know what it is, they are actively working to build integrations using OSLC, encouraging their other tool suppliers to support it and active in various OSLC activities such as specification working groups or general community promotion.   I has transitioned from an unknown new concept, to the way we do integrations.  By "we", I'm not just talking about Rational, I'm talking about attendees there that were talking how how they are using OSLC such as Airbus, NEC, ...

Though, still many people have a hard time saying or spelling OSLC right (it is a tough one)...most commonly is OSCL.  If only we pushed to rename it back in 2010 to something like I proposed as SLIC, that would have been...well "slick".  I digress.

This year, I arrived a couple days before the official conference started as it was a good opportunity for those of us very active in OSLC to get together for some face-to-face discussions on OSLC strategy.  This was spearheaded by the Steering Committee (SC).  Out of these early discussions (which have been a continuation of ongoing thoughts by the community and SC), came the idea of an organizing and higher-level concept of "Integration Patterns".  I threw together a page to articulate the thoughts, propose a way forward and start to gather interest.  This was discussed at couple other times during the week, such as the OSLC SC discussion at the Wednesday's Birds of a Feather session, which was well received from the attendees.

Sunday afternoon held the Open Technology Summit, where various leaders in open technologies shared how various efforts have help drive business efficiencies and improve overall time and quality of delivery around such things as: OpenStack, OSLC, CloudFoundry, Apache Cordova, ...

I led a panel discussion titled "Best practices on implementing integrated tools" with panelist with a wide and vast set of experience (I hope to share the recording once I receive it)

Innovate2014 Panel - Best Practices on Implementing Integrations from Steve Speicher

After 5 years, Mik and I were reunited as we talked about "Lifecycle Tool Integration through Open Interfaces" (though Mik and I have been talking/collaborating this whole time, it wasn't like a band breakup and then reunion)

Innovate2014 Better Integrations Through Open Interfaces from Steve Speicher

There were many other great conversations, learning how customers are looking to build out their own OSLC implementations by either evolving their own in house tools or looking to build adapters for 3rd party tools.  The demand continues to grow and look forward to continue to helping them succeed by making their integrations happen.

As with many of these conferences, especially ones that you've gone to 10 times, it is great to catch up with many good friends I've made over the years.  Now on to make sure we continue to deliver value and have some cool things to show and talk about next year (oh and at next week's EclipseCon France event and end of June's INCOSE conference).

Trip Report - OSLC Connect @ ALM Forum in Seattle March 30 - April 4

I recently attended an event in Seattle called ALM Forum, used to go by a slightly different name and purpose in past years (ALM Summit).

Quick Summary

Overall I thought it was well worth the time and would recommend going back next year.  Since my primary purpose for being there was to promote OSLC and get a better understanding of adoption problems.  So I think we covered those fairly well.

Event by Event Summary

I had to opportunity to attend many of the sessions and events, I'll touch on a majority of them but some with less significant information to share I probably just omitted for brevity.

Monday

Spent most of the day meeting up with some customers and OSLC advocates.

OASIS OSLC Booth in Exhibit Hall

Booth setup, this was a first.  It was a good opportunity for the OASIS OSLC Member Section to leverage funds from OASIS membership to contribute to sponsorship of the event and have this booth in the exhibit hall.  When we could, Sean Kennedy and I would staff the booth.

OSLC Happy Hour

This was a good social event Monday evening which we had about 10% of the conference attendees (not bad) attend.  Collected some information via some surveys, met some new people, finally me some face-to-face and many good conversations on issues and successes with integrations.

#integration #HappyHour @ALMForum - join us at Fountain Wine Bar w/ @DavidJWest & @sspeiche pic.twitter.com/xW7mRMpSM8
— Sean Kennedy (@seanpk8) March 31, 2014

Tuesday

Breakout Session: Better Integrations through Open Interfaces

This was my session on the Integration track, which had good attendance.

Better integrations through open interfaces from Steve Speicher

Wednesday

ALM for the Internet of Things by Ravit Danino (Director, Applications Product Management HP Software & Solutions)

Ravit gave a good overview of the key challenges and opportunities for ALM and PLM tool integrations, highlighting needs for standards-based integration between vast set of tools and suppliers that will be used.

Thursday

PROMCODE: An Open Platform for Large-Scale Contracted Software Delivery in Software Supply Chains - Mikio Aoyama

Professor Aoyama gave an excellent presentation on the challenges of large-scale efforts and how OSLC is being used to combat those challenges.

Lightning Sessions

Challenges and Opportunities in ALM-PLM integration - Michael Azoff

Michael touched on large opportunity for ALM-PLM integrations, seeing that there is still a large gap between the disciplines.  He also observed that OSLC was the place where many PLM vendors were turning towards solving some of the integration challenges and spoke of the positive outlook there.

Integration Principles and Reality - Ludmila Ohlsson

Ludmilla summarized their work at Ericsson and the vision around open standards-based integrations based on leading standards such as OSLC and how to broaden the adoption to more tools.

Considerations with event driven solutions in a Linked Data / OSLC world

I've often been asked about OSLC's plans to support some technology that allows for an event driven model.  Often what this request comes down to is that customers would like to have an open way to subscribe to certain events from a tool and then have that tool (or intermediary) notify them when their criteria has been met.  Once I drill down in the use case a bit further it is often the case that have some clear logic they want to run on the notification receiver side.

For example, let's look at the scenario between a bug tracking system a development team uses and a ticketing system the operations team uses to track customer reported problems.  We've already established how we leverage OSLC to easily relate the ticket from ops to dev.  Though the ops team has a process by which they modify the ticket's status to indicate a fix from development is ready.  The ops team has a number of tools, scripts and reports that run against the ticking system for these 'fix ready' tickets.

There are many ways this problem could be solved, let's take a look at some:

Event driven - this would require new software to be written on both the ticking and bug tracking tool end (perhaps with some eventing software) to make this work.

Polling or cross tool query - there really is no need to promote the state of the ticket to 'fix ready', the ticketing system could just look at (fetch directly) the status of the linked bug.  This would require processes to change on the ops side to not be driven based on that specific state of the ticket but the combined view.

• A variant of this is to just have the ticketing system poll the bug tracking tool (either when ops person is viewing the ticket or an agent) and set the status of the ticket to 'fix ready'.

Manual - dev sends ops an email/IM/txt, ops loads dashboard with sees bugs from dev that have a fix ready, or NCIS-like where ops shows up to dev with giant slushy asking if and when their reported bug will get fixed.

I think we all have seen the manual way of working, which I don't think I need to highlight what is so not fantastic about this.  I touched briefly on the the impact of the two other automated ways to solved this.  Each requires change --  though in the case of polling, only the consumer needs to change (assuming linking is already there).  Here's a quick summary of considerations with each approach:

• Event driven
• Need to manage subscriptions
• Need to process notifications (which could include processing subscription rules and handling authentication)
• May require a 3rd party bus (message broker) tool.  This comes with its own costs to acquire and maintain
• Administration to handle failed notifications (authentication, firewall, server down, etc)
• Easy model for consumers to just subscribe
• Polling / query
• Responses could be cached, polling requests to origin could be intercepted by caching servers and therefore taking the load off the backend tool
• Linked-to tools never needs to know about external apps

Some of these considerations are of course potentially offset if your organization already has an ESB deployed, so that cost may have already been sunk and the investment in administration expertise has been made.  One aspect of event driven solutions is often a problem of mismatch of models and conflicts of change.  For example, if the configuration (state model) of one of the tools changes it may make it so the ESB can't deliver and process the event to the desired end point.  Also if there is a desired to use this approach to synchronize data, in 1-to-1 or many-to-many, then conflicts will arise and the authority (or master) of the data is perhaps lost.

There are many other factors to consider.  It is often best to sit down and look at the topology of tools today, expected view into the future, and scenarios to see what is right for you.  I'm not saying one is better than the other and there is a clear answer for every integration question that will be asked.  I wanted to elaborate on some of the considerations when looking at each alternative.  Some day OSLC may define or endorse a RESTful event driven approach that meets the scenarios provided by the community.  After all that is what drives the work and those the work in the various groups agree with that, that is what get done.

Supporting Accept-Post in JAX-RS applications

Recently in the W3C Linked Data Platform working group we did a survey of the various discovery (or also referred to as affordances) there are for various methods or actions a client may want to introspect or learn from error responses.  One specific scenario is the case of which resource formats (content-types) are accepted by a server when a client wants to send the representation of a resource using POST with the intent of giving birth to a new resource.  The current approaches rely on trial-and-error or some out-of-band knowledge the client application has.  The trial-and-error approach relies on the client sending content of the content-type it believes the server accepts, if the server does accept it and successfully processes the request, it will send back a friendly 201 (maybe 202 or other 200-range) status code.  If the server doesn't like the content-type the client is giving it, it can kindly reply with a 415 (Unsupported Media Type).  Well the client knows what doesn't work but has to guess what might.  Let me introduce you to Accept-Post which is being proposed as a way for a server to tell a client what content-types it prefers.  Accept-Post is somewhat like the Accept header but more closely matches the newer (and less supported) Accept-Patch header.

Ok, that is a enough about the motivation and usage.  I thought I'd share the few lines of Java code needed to support this in JAX-RS 2.0 based implementations.  Since I want the HTTP response header Accept-Post to be returned for a variety of use scenarios such as: when JAX-RS returns a 415, on OPTIONS and HEAD requests, and so on, I decided to always return the header.  To do this, I implemented the ContainerResponseFilter with a simple Class and filter() method as:

import javax.ws.rs.container.ContainerRequestContext;
import javax.ws.rs.container.ContainerResponseContext;
import javax.ws.rs.container.ContainerResponseFilter;
import javax.ws.rs.ext.Provider;

public class AcceptPostResponseFilter 
       implements ContainerResponseFilter {
   @Override
   public void filter(ContainerRequestContext requestContext,
                      ContainerResponseContext responseContext) 
                      throws IOException {
      responseContext.getHeaders().putSingle(
         "Accept-Post", 
         "text/turtle", "application/ld+json", "image/png");
   }
}

That is about it, except of course you needs to register this filter with your JAX-RS Application, such as:

import java.util.HashSet;
import java.util.Set;
import javax.ws.rs.core.Application;

public class MyApplication extends Application {
   @Override
   public Set<Class<?>> getClasses() {
      Set<Class<?>> classes = new HashSet<Class<?>>();
      classes.add(AcceptPostResponseFilter.class);
      return classes;
   }
}

I've made this change for the in-progress LDP reference implementation occurring at Eclipse Lyo.

Similar approaches to other web server implementations or configurations make implementing Accept-Post quite trivial as well. Feel free to provide feedback on the IETF working draft for Accept-Post as well.

OSLC Resource Models - pictures are worth a thousand words

Pardon the metaphor but seems quite accurate here that in order to scale, OSLC working groups (WGs) operate as near-independent groups.  These WGs all produced their piece of the overall picture of resources, their properties and relationships to other types of resources.  The resource models (sometimes referred to as data models) were driven based on a key set of integration scenarios and defined following guidance produced by the Core WG.  The Core WG itself even has various models to support a number of cases such as: describing service providers, resource shapes, discussions, and other commonly used terms.  With all these pieces often laying around in separate specifications (Wiki pages, vocabulary documents, etc) it can be quite helpful to pull these all together...especially using visualization of the resource models.

This first figure is an example of a diagram from the perspective of an OSLC Change Management Change Request definition.

I'll go into a bit more detail about this model a bit later.

In an attempt to simply view these resource models, I started with some work that Scott Rich had done, along with some approaches I had experimented with using Rational Software Architect (RSA)

To keep things very simple, I'll highlight some guidelines on how to develop this model in a UML modeling tool:

• This is just a picture (for now), semantics are not clearly defined and they are not those of OO design.
• All properties are modeled as an  'Attribute', they are just visualized in the diagram as an association (since property values/objects in RDF are nothing special).
• Each domain, which has its own vocabulary document, is a separate package.  Also give each domain/package its own color
• No special profile is used (I attempted to use OWL profile from OMG).
• Even though there isn't an example restriction on the resource types (range) of properties, an explicit expected Class is still set.  A diagram with everything pointing to rdf:Resource wouldn't be too interesting.  Note to self: create a stereotype/visualization to make this obvious.

Ideally (and I know my modeling geek friends are going to like this) we can transform to/from some other descriptive form (OSLC Resource Shapes + RDFSchema).

The current model has been shared in the Eclipse Lyo project and additional examples are highlighted over on the OSLC Core wiki page.  I tucked the RSA model file into an Eclipse project titled org.eclipse.lyo.model which you can find in expected git location.  For those that use some tool other than RSA, I have also provided the .uml file.  I'd be interested to hear if anyone has another tool (and/or approach) to modeling these.  I'll try to advance in my spare time, including improving the diagrams.

OSLC's motivation on the design of the LDP Container

It is often that when some concepts in specifications exist, it is hard to tie together the learning from past experiences and what had been considered prior.  Sometimes preserving this and providing the background is too much filler for some already long documents, so I will use this opportunity to pull together a series of activities.  Some of these activities can be found in OSLC 1.0 specs and elsewhere.

All these approaches were motivated with a set of requirements which are quite similar:

• Common patterns for dealing with multi-valued properties
• These multi-valued properties often become large
• Easy way to get all the properties
• Easy way to add new property values

Below are a sample of approaches

OSLC CM 1.0 - @collref

In CM 1.0, when these patterns first evolved and we were working on applying REST we came up with a concept of @collref which was an XML attribute that held the value of the URL of the collection where you could POST to add new members and GET on to get a list of all members.  This approach was developed based on some legacy XML and pragmatic need to solve a narrow problem.  It would for a good number of cases but was a bit special and non-standard.

OSLC CM 2.0 - multi-value props

In CM 2.0 and Core 2.0, we took a more RDF-view of REST and simply treated these as multi-valued properties or in RDF terminology they were same-subject and same-predicate triples.  They were just like any other property on a HTTP (OSLC) resource, using GET to fetch the representation of the state of the resource, POST to create it and PUT to update.  This approach worked well in that it didn't require any new invention but was limited by what existed, it would be good if we could layer on semantics without losing the simple usage of RDF to define the resources.

OSLC CM 3.0 - multi-value props + LDP Container

In drafts of CM 3.0 and Core 3.0, we look to build off a concept called Linked Data Platform Container defined by Linked Data Platform 1.0.  This isn't far from what we've done in 2.0 except it is based on a W3C specification and has some additional semantics, namely layering on the ldp:Container semantics.

Other motivation

Another key scenario motivating similar requires is dealing with things such as query results or filtering long lists of resources and where to POST to create new ones.  In Core 1.0 and 2.0, we used concepts such as query capability and creation factory to meet these needs.  As we see with ldp:Container, this provides that same function while we can annotation it to provide some of the additional needs for knowing what kinds of resources a server will accept in a container it hosts and things like Web UI dialogs that can be used to select and create resources.

Applying LDP concepts to bug tracker

In this post I'll explore how some products that support bug tracking, such as Rational Team Concert, model their internal "Work Item" resource given W3C Linked Data Platform and OSLC concepts.  This will also highlight how various capabilities are needed to solve some key problems in a simple way.  I will be "paraphrasing" what the actual runtime resource will look like to help with human consumption of the examples (as the real-world ones can sometimes have additional concepts and capabilities that don't apply to the simple part of the model we are focused on here).

Let's take one of the most simple examples of what one might find, let's say a bug (aka "Defect") that has a few properties but no associated attachments or child bugs.

Representation of bug http://example.org/bugs/13:

@prefix dc:  <http://purl.org/dc/terms/> .
@prefix ocm: <http://open-services.net/ns/cm#> . 

<> a ocm:Defect ;
   dc:title "Bug in rendering large jpgs" ;
   dc:identifier "13" .

Now we'll explore how add a couple screen shots to the bug. Using this information I have with this resource, I'm not sure how I do that (assuming I am a Linked Data client). I could just attempt to PUT replace the contents and add a ocm:attachment statement referring to the screenshot. Depending on the server, it may reject it for a couple of reasons, such as: it doesn't known anything about ocm:attachment, or it has restrictions on the object (where the attachment is physically stored), or simply PUT updates not allowed for ocm:attachment.
To help with this problem, we can associate an ldp:Container with this bug resource to assist with this. So we expand our example, with out modifying any of the original model, just adding a few more statements.

@prefix dc:  <http://purl.org/dc/terms/> .
@prefix ldp: <http://w3.org/ns/ldp#> .
@prefix ocm: <http://open-services.net/ns/cm#> . 

<> a ocm:Defect ;
   dc:title "Bug in rendering large jpgs" ;
   dc:identifier "13" .

<attachments> a ldp:Container ;
   ldp:membershipPredicate ocm:attachment ;
   ldp:membershipSubject <>.

This tells my client now that we have an ldp:Container associated with the bug, since the ldp:membershipSubject connects this container to the bug. I can inspect also ldp:membershipPredicate to know for which same-subject and same-predicate pairing I can use this container to assist with managing and navigating them.
Now I have a URL http://examples.org/bugs/13/attachments where I can POST a screenshot to create an attachment and associate it with bug 13. Let's look at what the POST looks like:

Request:

POST http://example.com/bugs/13/attachments
Slug: screenshot1.png
Content-Type: image/png
Content-Length: 18124

[binary content]

Response:

HTTP/1.1 201 CREATED
Location: http://example.com/bugs/13/attachments/3
Content-Length: 0

Now that the attachment has been created, we can fetch bug 13 again to see what we have.

@prefix dc:  <http://purl.org/dc/terms/> .
@prefix ldp: <http://w3.org/ns/ldp#> .
@prefix ocm: <http://open-services.net/ns/cm#> . 

<> a ocm:Defect ;
   dc:title "Bug in rendering large jpgs" ;
   dc:identifier "13" ;
   ocm:attachment <attachments/3> .

<attachments> a ldp:Container ;
   ldp:membershipPredicate ocm:attachment ;
   ldp:membershipSubject <>.

We now see that there is an ocm:attachment statement associated with bug 13. This statement was added by the server when it processed the POST request to create a new resource (attachment) and then added it as a member of the attachments associated with the bug.
We can also see that this list can grow to be quite large. Experience has shown, that end users of bug trackers need to attach a number of documents, images, logs, etc. with bugs. This need also comes from a number of other capabilities such as having nested bugs or tasks. To illustrate, let's assume our bug tracking server has been upgrade or now exposes children resource within bug resources (or has had children added by other means). Let's take a look at bug 13 again:

@prefix dc:  <http://purl.org/dc/terms/> .
@prefix ldp: <http://w3.org/ns/ldp#> .
@prefix ocm: <http://open-services.net/ns/cm#> . 

<> a ocm:Defect ;
   dc:title "Bug in rendering large jpgs" ;
   dc:identifier "13" ;
   ocm:attachment <attachments/3>, <attachments/14> .
   ocm:child <../47>, <../2> .

<attachments> a ldp:Container ;
   dc:title "Attachment container for bug 13" ;
   ldp:membershipPredicate ocm:attachment ;
   ldp:membershipSubject <>.

<children> a ldp:Container ;
   dc:title "Children for bug 13" ;
   ldp:membershipPredicate ocm:child ;
   ldp:membershipSubject <>.

As you can see, the bug model stays very simple with statements about bug 13 being made directly about it using simple RDF concepts where the statements are of the form [bug13, predicate, object|literal]. We can repeat this pattern and use it in many other forms, such as a container of all the bugs the serve knows about, which I plan to illustrate in other posts.
Disclaimer: some of the predicates used in these examples show they are in the OSLC-CM namespace but have not been approved by the WG. This was done just for simple illustration.

Making progress towards an OSLC-CM 3.0

Been a while I have talked about a specific domain working group and what has been going on.  Though the work for a post-2.0 Change Management specification has been evolving for some time, it is coming along nicely.  I've been focusing much of my energy on getting the core-est part of Core (W3C Linked Data Platform and Core 3.0) rolling, while keeping an eye and hand involved with CM.

To summarize the OSLC-CM WG has looked at a number of prioritized scenarios:

• IDE Task federation and management
• Transitioning states
• Programmatically accessing and uploading attachments
• Querying across multiple states in a single request
• Additional semantics on regards to priority, severity and additional types of resources

We currently have draft specification text for most of the scenarios above.  We are planning to enter into convergence soon and start to do some implementation to validate the specifications.

Special thanks to Sam Padgett for leading the OSLC-CM efforts, planning meetings, keeping minutes, keeping track of issues and pushing forward with the draft specification.  Looking forward to getting some of these approaches implemented and specification wrapped up.

Trip Report from EclipseCon 2013

I had a chance, along with Michael Fiedler,  to attend the US EclipseCon 2013 in Boston, MA in late March.   In addition to presenting a BOF and a session on OSLC, Linked Data and Lyo, we were able to attend several sessions, panel discussions and BOFs.   It was also a great chance to meet with members of the OSLC and Lyo communities to discuss the challenges of tool integration.

General Impressions from the Conference

It was cool to see and hear OSLC and Eclipse Lyo in a variety of sessions.  There were good active discussions with the Eclipse Mylyn team regarding their recent m4 proposal, including how OSLC and Lyo play a key part.  It was good to explore ways we could work together with Mylyn, Orion, Hudson and more.  We did do a lot of education and exploration with a number of attendees that had no or limited knowledge of what was going on within OSLC and Lyo.  Various panels, keynotes and sessions often called out the need for tool vendors to be able to collaborate on open interfaces....we asked them: "have you heard of OSLC?"  There were a number of people making the case for OSLC-based integrations very easy.

Here is a summary of the two sessions we presented at EclipseCon:

1) Lifecycle Tool Integrations: Linked Data, OSLC and Eclipse Lyo
Session type: Birds of a Feather

The participants ranged from experienced OSLC implementers interested in contributing to Eclipse Lyo to those new to OSLC looking for learning resources. 

The discussion covered the re-vamped tutorial on open-services.net, using the Lyo OSLC4J Bugzilla adapter as a learning resource and some general OSLC and linked data integration philosophies.  Topics included:

• a description of an OSLC4J implementation in progress which exposes EMF Models as OSLC resources - being considered for contribution to Lyo
• the experiences of a developer who has implemented several OSLC integrations to enable tools to participate in an ALM system
• The OSLC CM 1.0 integrations developed for the Mantis bug tracker and FusionForge
• Status of Eclipse Lyo - what is new in 1.1 and what is coming
• One OSLC implementer strongly recommended developing integrations using 2.0 of OSLC.  He also pointed out authentication is a primary painpoint for the work he has done.

2) Leveraging W3C Linked Data, OSLC, and Open Source for Loosely Coupled Application Integrations
Session type: ALM Connect track

EclipseCon 2013 Learn and share about integrations using Eclipse Lyo, OSLC and Linked Data from sspeiche

This session was a 35 minute tour through tool integration, problems with previous approaches and new approaches using linked data and OSLC.

This was followed by an overview of open source projects relevant to this space (Jena, Wink, Clerezza, Lyo etc) and a brief demo of some potential integrations between Eclipse Orion and a Change Management tool (Bugzilla) and an Automation tool (Lyo automation reference implementation).   

There was very little time left for questions at the end, but there were some good ones which pushed the session into overtime: 

• Interest in the concept of delegated UIs and the responsibilities of the tool hosting them - described the OSLC concepts in some detail
• Interest in the concept of UI previews and compact representations.   How these resources can be used to link to full representations was described.
• Some clarifying questions were raised by a few folks about the details of the Bugzilla integration demonstrated.   Explained it is the live Eclipse Bugzilla instance, but that it could have been any OSLC CM provider.

Special thanks to Michael Fiedler for both authoring some of this content of this blog post, as well as building the demo, presenting and too many things to list here.

W3C Linked Data Platform WG 2nd Face to Face meeting March 13-15 Boston

I just returned from the 2nd face to face meeting for the W3C Linked Data Platform (LDP) working group located at MIT in Cambridge, MA. There were about 14 attendees in all, a good number but perhaps low for a WG that has 50 total registered. Being the 2nd face to face meeting of a relatively new working group, meant this was the first meeting where we could dig in our heals and start tackling issues. We reviewed our deliveries: a must deliverable of the spec and a number of supporting documents: use case and requirements, access control guidance, possible primer, deployment guide (need a better name) and test suite.

We prioritized the issues we wanted to discuss early on the first day to make sure we got all the "big hitter" issues out on the table. I added to this list some of the key OSLC items such as binary resources and metadata, patch and more. One thing we learned as we got into the next day, there still was some confusion on the model. I surfaced ISSUE-59 to hopefully help simplify the model.

Here are the detailed meeting minutes from: day 1, day 2 and day 3.

Some of the key issues discussed were:

ISSUE-15 Sharing binary resources and metadata
The WG came to the following resolution after much discussion about concerns of different behavior of POST'ing RDF formats which create LDP-Rs and which response location headers should used.

Resolution: Assuming the existing qualifications that POST is optional, and supporting "binary" media types is optional: The expected response to a post for create that creates a non-LDPR is a 201, with location header whose URI I identifies the resource whose representation matches the POST input, and the response SHOULD include a Link header rel="meta" href= another URI P identifying the resource whose state is the server-managed properties. The URIs I and P MAY be distinct, but this is not required. When the object of a membership triple (I) is DELETEd, the server MUST automatically deletes any related resource P that it created previously

ISSUE-17 Recommended PATCH document (and 27 & 45) 

There were a number of PATCH related issues, like 27 whether POST could be used to tunnel PATCH but WG decided just to use PATCH. Also there is 45, which suggested the need to use POST'ing of RDF formats to LDP-Rs to append/add to existing resources. The WG decided that this would be fairly different than what is done with POSTing to LDP-Cs which creates a new resource and adds membership triples, plus we have a solution to append by using PATCH-insert. For 17, the original proposal was to use the Talis changeset format but due to a variety of reason this was withdrawn (don't recall the problems with Talis changeset). Instead the WG decided to pursue its own simple PATCH document that leveraged existing quad formats such as TriG. The WG has an action to define the minimal and work with issues and proposals to expand on it.

Other issues we discussed were on our list to watch as of course the resolution could impact us but the issues listed were not identified by OSLC community members as a high priority.  Overall it was very encouraging to see the WG make progress on such key issues.  It is also a little disappointing we didn't make more progress in other areas.  It is most likely that we won't enter into "Last Call" until after our 3rd face to face which is being planned for mid-June.

OSLC's relationship with Semantic Web technologies

The technical basis for accessing data through OSLC has its roots in Linked Data, which has its origins with Semantic Web technologies.  Some worry about the costs of supporting this for some simple integration scenarios, though OSLC only depends on a small amount of it.  For those that bought into the full stack of Semantic Web technologies for various domain solutions realize there is tremendous value in what it can provide but that it also comes at some cost.  The cost is in aspects of Semantic Web such as reasoners, inferencing, search engines, RDF and specialized repositories for dealing with these things.

OSLC takes a minimal incremental approach to depend on only what is needed to satisfy the integration scenarios.  So far that has led us to leverage a simple standard way of describing resources using RDF.  That is about where the Semantic Web technology dependency ends.  We leverage a few terms out of RDF Schema to help with defining our own vocabulary terms but do not go beyond that as it might imply that clients would need to process inference rules against the resource representations they receive to learn more.

Since a primary goal of OSLC is to not reinvent but leverage standards based approaches that meet our requirements, I can see cases where it might be good to adopt some more Semantic Web technologies.  Though to be clear, for tools to get value out of OSLC-based integrations, only some RDF syntax readers and writers are needed.  There is no need for tools to have be rehost or rewrite them onto a new technology base, they can simple adapt their solution with a simple fasade or update their existing REST APIs to provide this support.

W3C Linked Data Platform WG -- 1st Face to Face Meeting

I recently attended the 2012 W3C Technical Plenary / Advisory Committee (TPAC) in Lyon, France from October 29 to November 2.

RDF WG: Monday and Tuesday 

I attended the RDF F2F Working Group face to face meeting as an observer since I'm not an member of that WG though of course interested in a number of topics.  The RDF working group is nearing the end of its charter (Jan 2013) and making good progress on a number of its deliverables such as Turtle, JSON, TriG, n-triple, n-quad serialization formats.  There are a number of key issues that are of great interest such as ISSUE-77 marking rdf:Seq as archaic (this if particular interest to work at OSLC). There was a breakthrough in direction with named graphs by resolving to not define detailed semantics on identification of graphs, instead just allowing simply names/labels for graphs.

RDF WG Minutes: Day 1 and Day 2

TPAC Plenary Day: Wednesday

Jeff Jaffe CEO of W3C kicked off the day with a brief keynote and then logistics for breakout sessions.

I attended these sessions:

• Social Web: lots of interest in linked data and social web.
• Government Linked Data: discussion and update on work going on with existing WG, there is need for some guidelines and have need for LDP work
• Linked Data for Web Developers: a fair amount of discussion around JSON, including a "white-boarding" session by Tim Berners-Lee talking about some approaches using JSON.  There was also discussion of a need for some schema-like need for RDF data, which I explained we have similar need and have some experience with OSLC Resource Shapes.

Linked Data Platform (LDP): Thursday and Friday 

Of course this was the primary reason I was here.  Agenda covered a range of topics as it was our first face to face meeting. Discussed Use Case and Requirements, Primer, Test Suite, Implementations, next F2F and of course spec issues.  Tim Berners-Lee attended both days and was very active in discussing the issues, he had strong views on how some of LDP should be shaped.

Probably the biggest change is the resolution around ISSUE-25 where the ldp:Container follows strong composition model instead of weaker aggregation.  There was discussion for a common set of needs around paging and ordering of contents.

LDP WG Minutes: Day 1 and Day 2

Overall Impressions

It is clear there are many W3C members that are very passionate and believers of Linked Data, including Tim Berners-Lee.  Discussions with many shown they are interested in keeping LDP simple and willing to jump in and add support to their products early.

Trip Report from JavaOne 2012

Trip Report: JavaOne 2012 October 1-4, San Francisco 

I attended my first JavaOne conference, where I presented and attended a number of sessions and events.

Leveraging W3C Linked Data for Loosely Coupled Application Integrations

Tuesday Oct 2, 8:30

Track: Enterprise Service Architectures and the Cloud

There were about 90 people who preregistered for the session (see attachment) but seemed like there was a bit less than that at the actual session (50). Since they don’t serve the free coffee until after the 8:30 sessions, I felt like the attendees seemed very engaged by their responses and didn’t see anyone sleeping. The best was at the end where I had about 10 people approach me. Someone stated he was from a major news media company who asked how they could get involved in W3C Linked Data Platform (LDP) work and I highlighted some work at BBC that he might be interested in as well. Also a development manager from automative sector, asked how they could get involved as he was very interested in requirements traceability. Good news was that I could connect him with what other efforts in automative that were already occurring. There were a few others from healthcare and life sciences sector that said they were interested as well, handing out my card to a number of folks.

Linking the Java Application Lifecycle with Hudson

Wednesday Oct 3 4:30

Track: Development Tools

I was not on the original agenda to speak here but Mik Kersten CEO of Tasktop asked that I join him to explain Linked Data and OSLC, and give an overview of current activities. This session was also co-presented by Gregory Stachnick, Oracle Senior Director, Product Management. Greg is responsible for Oracle’s Eclipse-based tools strategy. Good overall flow considering some late changes, story around linking the lifecycle and need for standard integrations was good.

Some other activities:

Tuesday night Eclipse community meetup, good to talk with some Eclipse community members and talk about Lyo futures.

A number of sessions on JAX-RS, REST, WebSockets, Cloud, NoSQL, Hudson, CI, e4, ...

It was interesting to see the overall interest in REST, WebSockets, NoSQL and big data.

Overall very useful trip, look forward to trying to get back in 2013.

Quick Introduction to Linked Data and OSLC

I thought it would be useful to explain both Linked Data and OSLC using a simple buildup example.  Hoping to also show where Linked Data ends and OSLC picks up.  This was taken from a presentation that I did at Innovate 2012 and received better than bad feedback.