Think Link, not Sync when doing tool integrations

Often when working with ALM tool development teams, partners and customers there is a need to go through an evolutionary re-thinking of how tool interoperability should work.  Often the time when discussing how OSLC can help them, it often starts as a discussion how they can throw out their current 10's of product specific connectors to have a single OSLC API to synchronize data with their tool.   Though there are cases where pulling data, not bi-directional synchronization, has value for purposes of efficient data warehouse access for reporting solutions.  I challenge these tool integrators and vendors to re-think their integrations to instead provide "just enough" information about the other tool being integrated with.  This "just enough" could be as simple as two things: 1) a link to resource owned by another tool and 2) knowing the semantics of that link (OSLC).


There are many advantages of leaving the data within that tool that owns it and providing a link to it:

• The tool that owns it knows best the rules to govern changes to it, including auditing support
• The tool that owns it, knows best how to control access to data.  By copying the data, often access controls needs to be replicated as well (as best possible)
• State-models of resources across tools often don't align
• There are ways to expose this data in other tools, without replicating it

It is sometimes convenient to have a cloned/cached copy of data in a local tool, OSLC does not prohibit it this and can support this.

Update:  Also see interesting and related IBM DeveloperWorks article "Stop copying, start linking"

OSLC reference implementations and test suites

There have been a number of implementation efforts underway for various OSLC specifications.
I've heard many requirements including...

for Service Providers:

• A hosted reference implementation that can react to client consumer requests, to ensure consistent behavior across implementations
  
• Make the source code available for download
• Allow contributions to the source
• The language it is written in is less important, though some tend towards JEE based
• A client testsuite that can give level
• Samples that highlight key integration scenarios
• A framework in which to quickly enable new implementations
  

for Consumers:

• A hosted reference implementation that can react to client consumer requests
• A reference service provider that can provide feedback on consumer implementations (testsuite)
  
• Provide a variety of samples
  
• Java client samples and/or SDK
  
• Command-line or Perl based samples and/or SDK
  
• HTML/Javascript samples and/or SDK

Some current efforts underway:

• Eclipse Mylyn (client Java API)
  
• Helios (client and service provider PHP)
  

Some thoughts on technology basis for service provider reference implementation:

• Apache Wink - REST framework
• Jena
  - RDF/XML, Turtle parsers and generators
  - Apply custom rules for RDF/XML and Turtle
  - Add JSON support
  - Simple storage
  - Extended to support ResourceShapes
  - Query - mapping of query syntax - oslc.where/select
  - Resource subsets - oslc.properties
• OAuth - Provider only
  (need good consumer example)
• Service Discovery
  - Various models combinations of Catalogs and ServiceProviders
  
• Web UI
  - Simple example/demo HTML/JS
  - Prefill
  - Via a draft resource creation
  - Via direct prefill and redirect
  - UI Preview

Dave Johnson posted some thoughts as well here. Which are fairly close to this as well.

Feedback and additional requirements as needed.

CM 1.0 Simple Query Syntax, how we got here

Often I get asked about what how it was decided what was included into the CM 1.0 Simple Query Syntax, so I figured I'd take a couple of minutes to summarize some key points. The goals for the 1.0 were quite simple: produce a simple syntax that supports our scenarios and that will work with SQL and SPARQL based back-ends. One of the most obvious omissions from the query syntax is the "or " operator, or UNION if you are from SPARQL. This was intentional since the CM WG realized that all scenarios could be accomplished by having support for the "in" operator. By doing this, we avoided the complexity of having to support grouping of terms with parenthesis.

The type of query we saw as being key to support in 1.0 were queries such as: Show me all open change requests assigned to a specific user
?oslc_cm.query= owner="bob" and status in ["submitted","working"]

Beyond that, most other operators are quite common. Another primary scenario was the ability to retrieve a set of change requests that have been modified since a given date. Which is supported by both having the Dublin Core usage of dc:modified and comparison operators such as:
?oslc_cm.query=dc:modified>="12-02-2008T18:42:30"

This has been proven to be very powerful and easy to map to service provider search capabilities. At some point there may need to be support for a full query syntax service, which could be a unique URL end point to post a query to. We'll have to wait and see how this plays out over time.

Using Selective Properties (aka Partial Fetch)

Often is an issue when working with retrieving resources and their properties is getting the desired information efficiently. By definition of the CM 1.0 specification, when requesting a change request resource without any parameters to refine the list of properties you should retrieve ALL properties that resource has. Take for example this simple request:

GET http://example.com/bugs/2314
Accept: application/x-oslc-cm-change-request+xml

This will result in the change request identified by the URL to be retrieved:

<oslc_cm:ChangeRequest 
  rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
  dc="http://purl.org/dc/terms/" 
  oslc_cm="http://open-services.net/xmlns/cm/1.0/"
  xmlns="http://myserver/xmlns"
  about="http://example.com/bugs/2314">

  <dc:title> Provide import </dc:title>
  <dc:identifier> 2314 </dc:identifier>
  <dc:type> http://myserver/mycmapp/types/Enhancement </dc:type>
  <dc:description>
    Implement the system's import capabilities.
  </dc:description>
  <dc:subject> import, blocker </dc:subject>
  <dc:creator resource="http://example.com/users/aadams" />
  <dc:modified> 2008-09-16T08:42:11.265Z </dc:modified>
  <owner resource="http://example.com/users/john">
  <priority>High</priority>
  <severity>High</severity>
  <status>Working</status>
</oslc_cm:ChangeRequest>

This is useful when a consumer doesn't know which properties to request but can provide more information than is needed (both that the provider needs to generate and the consumer needs to process). To limit the amount of properties returned, the concept of selective properties can be used. If the consumer is only interested in the owner and the status, the request could be formulated such as:

GET http://example.com/bugs/2314?oslc_cm.properties=owner,status
Accept: application/x-oslc-cm-change-request+xml

Resulting in:

<oslc_cm:ChangeRequest 
  dc="http://purl.org/dc/terms/"
  oslc_cm="http://open-services.net/xmlns/cm/1.0/"
  xmlns="http://myserver/xmlns"
  about="http://example.com/bugs/2314">

  <owner resource="http://example.com/users/john" />
  <status>Working</status>
</oslc_cm:ChangeRequest>

Perhaps we are interested in the owner's name and email address, we can expand the owner entry and select only those properties to be returned such as:

GET http://example.com/bugs/2314?oslc_cm.properties=owner{fullname,email},status
Accept: application/x-oslc-cm-change-request+xml

Resulting in:

<oslc_cm:ChangeRequest 
  rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#" 
  dc="http://purl.org/dc/terms/"
  oslc_cm="http://open-services.net/xmlns/cm/1.0/"
  xmlns="http://myserver/xmlns"
  about="http://example.com/bugs/2314">

  <owner>
    <User rdf:about="http://example.com/users/john">
      <fullname>John Doe</fullname>
      <email>jdoe@myco</email>
    </User>
  <status>Working</status>
</oslc_cm:ChangeRequest>

This technique is used in CM 1.0 not only for selective retrieval of properties on a change request resource, it is used for partial updates of resources as well as controlling the content on query responses.

OSLC in the wild

Figured I'd take a moment to highlight some recent public references and press around OSLC involvement (thanks to Scott Bosworth, I "borrowed" most of this material from him)

White paper (The Case for Open Services), John Wiegand

Podcast (OSLC Bears First Fruits), Steve Abrams, Carl Zetie, along with Mik Kersten

Presentation: (ALM Integration in a Web 2.0 World), Steve Abrams

Presentation: (Restful Work Items: Opening up Collaborative ALM), Steve Speicher with Mik Kersten

Blog (Let's Try Something Different), Carl Zetie

eBook (Scaling Agile with C/ALM), Carolyn Pampino, Erich Gamma, John Wiegand

Interview (Slashdot: Jazz Technical Lead Erich Gamma answers your questions), Erich Gamma

Website (open-services.net), Scott Bosworth

Jazz.net blog (OSLC and Rational Team Concert), Patrick Streule

Jazz.net article (How to consume the Rational Team Concert change management services), Martin Aeschlimann

IDC, Melinda Ballou (IBM Implements Open ALM Spec), John Wiegand, Scott Bosworth

Release (IBM, OSLC Promote Interoperability Across the Software Lifecycle), Scott Bosworth, Robyn Gold, Martin Nally, Mik Kersten, Mary Rose Greenough

Application Development Trends, John Waters (IBM Implements Open ALM Spec), Scott Bosworth

Information Week, Charles Babcock (IBM Sharpens Rational Tools), Scott Bosworth

InfoWorld, Paul Krill (IBM Hails ALM Standards Participation), Scott Bosworth, Steve Abrams

Software Development Times, David Rubenstein (IBM Suppor ts Open CM Initiative in Tools), Scott Bosworth, Steve Abrams

InternetNews.com, Alex Goldman (IBM Rational supports Open Services with new software), Scott Bosworth, Martin Nally, Mik Kersten

PC Week Russia (IBM Rational tools support the specification OSLC), original article

Delegated Resource Creation and Form Factories

One of the most debated topics during the development of the OSLC-CM 1.0 specifications was around resource creation. Many different approaches were discussed. First there were some traditional methods, where the consumer learns about what properties are needed, their types, constraints, etc and then appropriately builds a request body to POST to a given resource factory URI. Another approach is to agree on what a simple resource should look like, then have a resource factory URI that will accept these and in most cases will respond with a 201-Created and a URI for the new resource.

One of the most powerful features of the OSLC-CM 1.0 spec is the ability to delegate UIs for certain functions. Most (if not all) CM applications have a complex Web UI form used to collect all the required, dependent and typed fields; apply some validation, then commit the changes to its repository. We can leverage that already working fine Web UI form, apply some rules to it and then leverage it within other contexts. Applying these rules, or techniques, we can communicate across web applications that run within web browsers to communicate when the resource is created.

One other challenge to this approach it how to prefill a form, a key aspect of scenarios in CM integrations. It is not feasible to do more URL path building, which is fragile and has size limits. So we created a discoverable URL for posting some resource data and the result would be a URL to a prefilled form. This follows the some model as creating resources in REST, though in this case the resource is a form.

RESTful authentication, some more thinking

Unfortunately, I haven't had too much time to dedicate to this but have had a few discussions and a bit more time to think about it.

It seems to me that form-based auth for RESTful web services should be forbidden. There are clear standards on how to deal with authentication programmatically over HTTP: basic, digest, OAuth. Since the cost to implement basic-auth and utilize https is relatively small, it seems that adding more options (beyond basic, digest and oauth) will only further compound interoperability issues with client consumers.

More discussions and research should be done, though my current thinking would only be to add a statement to the OSLC-CM 1.0 specs stating that form-based auth is NOT recommended.