Handling conflicts in media type headers

If you read my previous entry discussing versioning of RESTful services, I pointed out how we can leverage media types to support incompatible versioning of RESTful services. I think this is an excellent solution, but it can lead to an issue for service implementors

Imagine you have a service which has more than one version. Let's say it's a hotel reservation system, just to pick something familiar. We've picked media types for our versions of the service, and adopted the convention that in the absence of any media types we will use "Version 1" of the service. Assume our media types are "application/hotel-json" and "application/hotel-json-v2".

If you support operations for updating a reservation using PUT, you can run into a situation where you have both a Content-Type header (specifying the content of the entity body being used to update the reservation) and an Accept header specifying the kind or kinds of entities the client can accept in the response. This might be just fine, if the two versions of the service can be intermixed between the sent entity and the returned entity. But the two versions of the service might not mix. For example, an update sent using version one of the updating entity might not contain a value required for version two of the service.

In this situation what the client did is an error, so we clearly need to return a status code in the 400 range. But which is the most appropriate code? A quick look at the 4XX codes shows two promising candidates:

406 - Not Acceptable
415 - Unsupported Media Type

But when we read the description of 415 in RFC 2616 we see it isn't appropriate:

The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.

The Accept header is specifying a media type the service understands for the resource, it's just the combination of provided entity media type and requested return media type that is a problem. Happily, status code 406 fits perfectly:

The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.

Interestingly, the RFC makes a comment about this very situation, and suggests that returning an entity type different than those specified in the Accept header might be preferable to a 406:

Note: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.

In this particular scenario (where a required value isn't provided), one way to return a result would be to "downgrade" to a version 1 response in hopes that works for the client. There are potential issues with this, if the semantics of applying a version 1 update differ from those that would have occurred with a version 2 update. And if the client can't interpret the returned version 1 entity, they may end up in an inconsistent state. But it's a good idea to keep in mind and make work whenever possible.

Versioning REST services

I read a blog entry yesterday from Ganesh musing on whether RESTful web services need versioning. In his posting, Ganesh suggests putting some kind of version number into the body of a request. He suggests that this be done even for a GET or DELETE.

While I suppose this could work, it feels awkward. Putting a version number into an entity body for a GET, which would typically not have an entity body, is awkward and a fair bit of work. Things get even more complicated in the case of a PUT or POST where the entity body is some binary format like an image format. In this case, the Content-Type can't just be image/jpeg (for example) since your putting some kind of version information before the image data.

Even in the case of textual data such as XML, putting version information into the entity body requires that our XML format support a version number somewhere (probably as an attribute of the top-level element in the entity body?). This may be feasible, but it may not be feasible if we are trying to use a data format that is already defined. We could start hacking and just throw a line at the beginning of the entity body that indicates the version with the understanding that the 'normal' payload starts after this line. But this is hackish, and makes our Content-Type delcarations wrong (we aren't adhering to the content type definition since we've thrown that version information at the front).

After reading Ganesh's entry, I thought about versioning for a while and it seemed to me that the right way to handle versioning is via media types (using, for example, the Accept and Content-Type headers). This works well because the first version of our service can be delivered when no media-type headers are specified. Then, if a client wants to take advantage of a newer, incompatible version of our service, they simply specify this using media types in the Accept and Content-Type headers.

I could spend a fair bit of time describing this in more detail, but I did some googling and found a great blog entry (written back in May of 2008) by Peter Williams where he describes things concisely and clearly. It's short and sweet and well worth a read.

Back from reading his posting? There are two issues I thought of which Peter didn't directly address. One is specific to versioning, and one is a more general issue that can arise. The versioning related issue is this:

What if your service model changes to much that the set of resources you expose has to remove one of the resources in your original version of the service?

In this case, you make operations to that URI path with an Accept header and/or Content-Type header for later versions of your service return 404 (or similar error code) since that type of resource doesn't exist with that version of the service. While Peter didn't talk specifically about this, this approach meshes well with his description and I suspect it is the same answer he would give if asked.

A related but more general problem can occur if your service receives a request with both an Accept and a Content-Type header and the values are incompatible. I'll address that in my next entry.

[Updated 2010-06-15 11:25 a.m.]

I should mention that you shouldn't create new media types unless you have to. If your service deals with ATOM or RSS, use the version number for those to distinguish different versions of your service if at all possible. In his thesis, Roy Fielding states that creating new media types is not RESTful. And that is true in the sense that custom media types reduce the likelihood that clients will know how to interpret your representations. But there is a tension here between accurately representing incompatible versions of a service and using a pre-existing media type. There's no perfect answer, just a continuum with different trade-offs for different kinds of applications.

I spoke at the Gateway Software Symposium (NFJS St. Louis) last weekend. After my RESTful Web Services with JAX-RS talk, one of the attendees asked me how to discover the proper URLs when starting to interact with a RESTful service. The impetus for the question came from my statement that many RESTful services don't have large specifications (such as the WSDL files for "Big" web services).

I explained that well-designed RESTful services are "well-connected". By this, I mean they adhere to the idea of "hypermedia as the engine of application state", or HATEOAS. If your service adheres to HATEOAS, your service is accessible and navigable given just a single URL. How can this be?

Recall that in RESTful services, we use URIs/URLs to connect various related resources. To be well-connected (I'll use this term in place of to HATEOAS mostly because the acronym looks odd to some people) and be RESTful, representations of resources should contain URIs/URLs to related resources. To make this clearer, let's look at a specific example. As we look at this example, we'll make note of the aspects of the API which couldn't be discovered by examining the results of searches or resource retrieval.

In my talks on JAX-RS, I use a simple music service. In this service, each artist has a name and some number of music albums they have published. Each album as a name and some number of tracks on that album. Here is a simple UML diagram of the objects:

We now come to the first piece of information we can't discover for ourselves. We need to know a starting URL. With the example service, it's simply a host name and a port, as in: "http://localhost:3131".

When we run the music service and hit it with a browser, we see a spartan web page:

While this certainly allows us to do some (mildly) interesting things, how does it help us discover the API for this RESTful service? The answer lies in examining the underlying HTML. When we do that, we see this:

At first, there might not seem to be much here. But let's look at the form for searching for artists:

From this we can see that we can search for artists whose name "is" (IS), "starts with" (STARTS), "contains" (CONTAINS), or "ends with" (ENDS) the text specified by 'searchString'. If you don't live and work with HTML forms and their resulting queries every day, this still might not be obvious. So we can always go ahead and do a search. Let's search for artists whose name begins with 'F':

Clicking the 'Search' button does two things for us:

1) It performs the search and gives us the results:

2) It shows us the proper format for executing queries against the service (in the browser's address text box):




Looking at this, we know that we can query for artists by specifying a searchType and a searchString if we search under /artists.

My example music service supports more than HTML for representations. It also supports an XML format and a JSON format. In order to use the XML format, we need to specify that we accept a content type of "application/music-xml" (a content type I made up for this example). This is the second piece of information the service itself did not tell us. However, we could simply query the service and get back an HTML representation of the search results so it's not strictly necessary. The XML format is a more compact representation than the HTML, but the same information is contained in the HTML version.

If we use the curl command line utility to query:

curl --header "Accept: application/music-xml" "http://localhost:3131/artists?searchType=STARTS&searchString=F"

We get XML results:

Because this service is designed with API discovery in mind, we see that it reminds us of the search that was performed:

< artists uri="/artists?" searchString="F" searchType="STARTS">

In these results we also see an example of how to build well-connected services. Notice that each search result lists not only the name of the artist found, but also the URI of the resource we would need to retrieve in order to get a representation for that artist. For example, the artist "Fish" lists a URI of /artists/id/149. If we go to the music service in the browser and query for this URI (a URL of http://localhost:3131/artists/id/149), we get information about the artist named Fish:

Looking at the HTML, we see:

And if we query for the XML representation using curl:

curl --header "Accept: application/music-xml" "http://localhost:3131/artists/id/149"

We see:

Notice that the information about each album contains a URI we can use to retrieve a representation of that album, just like the search results listed a URI for each artist that was found. These are examples of how we build a well-connected RESTful service. By making sure that each HTML page (or XML result) contains URIs for related resources, users of our services can discover the proper URIs/URLs to use when invoking our service without a large specification.

In part two, I'll explore this notion further, looking at how our self-describing service allows us to discover how to create and delete resources as well as search for and retrieve them.

A RESTful web service testbed

It's time to build a more complete RESTful service example. In this web service, we'll aim to perform all the common activities of a real service. This is often referred to by the acronym CRUD, which stands for Create, Read, Update, and Delete for all of the common operations associated with persistent data. Since we're dealing with a RESTful web service, we'll also throw in multiple representations for a resource, and connectedness to make it easy for clients to navigate through the service resources.

So, what should our service do? My friend and colleague, Mark Volkmann, introduced the eample of a database of music information. In this example, the service service contains information about several kinds of resources:

• Music artists (Artists)
• Albums
• Songs

This is a nice example because it is rich enough to expose many of the problems that need to be solved by RESTful web service without becoming so large that it's unwieldy to explore. The domain model is immediately familiar, so we can focus on the technologies and not the model.

With this service we can explore not only traditional browser-based HTML, but XML and even an Ajax client using JSON. We can also use it as testbed for things like caching, security, clustering/failover, and composition of services.

RESTful clients with the Jersey Client API

[Note: This is one in a series of posts about JAX-RS. You may wish to start at the beginning]

Jersey, the reference implementation of the JAX-RS specification, is a framework that makes it very easy to implement RESTful web services. But there is more to Jersey than just the server side. In this post I will spend a little time exploring the Jersey client library.

Depending upon how you design your RESTful service, you may or may not want to have separate URLs for separate representations of the same resource. This presents a problem when trying to test with a browser. There is no way to tell popular browsers that you want a text/html representation or an image/jpeg representation. The browser has a list of preferred media types, but none that I'm aware of allow you to customize this (either in general or for a particular request). Even more importantly, we need to be able to build solid unit tests for our services. The Jersey client framework provides a good solution to this problem. It is designed to allow developers of RESTful web services to write good unit tests, but is more general purpose than that. It can also be used to write RESTful client applications.

There is an excellent tutorial on the Jersey client API, which you should download and read if you plan to use it. But I will give you a taste of the API here.

The Jersey client API is very clean, and requires a minimum of fuss to use. As an example, let's write a unit test which tests our web service serving up information about fighter planes. First, we write the code to set up and tear down our service implementation. This code is the same as that in our Main class before, just split up between the setup and tear-down methods.

public class Test3b {

private SelectorThread selector;

@org.junit.Before
public void createService() throws IOException {
   Map initParams = new HashMap();
   initParams.put("com.sun.jersey.config.property.packages", "net.gilstraps.server");
   selector = GrizzlyWebContainerFactory.create("http://localhost:9998/", initParams);
}

@org.junit.After
public void DestroyService() {
   selector.stopEndpoint();
   selector = null;
}

The only difference in this case is we don't read from standard in to shut down the service, since we always want to shut it down at the end of the unit test.

Now, we can test that the text/html representation returned by invoking the service is what we expect. First, we do a bit of hoop jumping to read in a copy of the HTML we expect to receive:

@org.junit.Test
public void testF22Html() {
try {
   File expected = new File("f22.html");
   long fileSize = expected.length();
   if (fileSize > Integer.MAX_VALUE) {
       throw new IllegalArgumentException("File it larger than a StringWriter can hold");
   }
   int size = (int) fileSize;
   BufferedReader r = new BufferedReader(new FileReader(expected), size);
   char[] chars = new char[size];
   int readChars = r.read(chars);
   if (readChars != size) {
       throw new RuntimeException("Failed to read all chars of the expected result html file");
   }
   final String expectedText = new String(chars);

At this point, the variable expectedText contains what we should receive back from our request. Making the request is quite straightforward. First, we create a JAX-RS client:

Client client = new Client();

These clients are 'heavyweight' objects (taking time to create and using significant resources). As such, in a production client we would want to create this client once and use it many times. For the sake of independent unit tests, we will go ahead and create a Client object for each test.

Next, we specify the resource we want to retrieve using a WebResource object:

WebResource f22 = client.resource("http://localhost:9998/planes/f22/f22.html");

And then we ask the client to retrieve the resource for us, specifying that we want a text/html representation (MediaType.TEXT_HTML_TYPE) and specifying that we want to get back a ClientResponse object:

ClientResponse response =
f22.accept(MediaType.TEXT_HTML_TYPE).get(ClientResponse.class);

This code is using the builder pattern, where we build up our request through a chain of method calls. In this case, the chain is only two calls long. First, we call the accept method to specify the media types we will accept in the response (text/html), then we call the get method to actually retrieve the resource. It is possible to chain together more calls to specify other characteristics of either the request or the expected response (for more information, see the whitepaper on the Jersey client API).

Now that we have gotten a representation of the resource in the form of an HTTP response, we can then retrieve the HTML entity contained within that response as a string:

String returnedHTML = response.getEntity(String.class);
And finally, since this is a unit test, we make sure that what we got back matches what we expected:
assertEquals("Expected and actual HTML don't match"
expectedText, returnedHTML);
The ClientResponse object is useful if you want to look at other characteristics of the response, such as the returned headers. In this case, we could just as well have asked for the string from the WebResource directly. To do so, we would replace these two lines:
ClientResponse response =
f22.accept(MediaType.TEXT_HTML_TYPE).get(ClientResponse.class);
String returnedHTML = response.getEntity(String.class);
With this one: 
String returnedHTML = asHTML.accept(MediaType.TEXT_HTML_TYPE).get(String.class);
Testing for retrieval of the JPEG representation of an image is almost identical. The only differences are that we read in the image file as an array of bytes, ask for the response entity as an array of bytes, and compare the two as arrays of bytes. Here is the entire test method:

@org.junit.Test
public void testF22JPEG() {
  try {
      // Read in our expected result
      File imageFile = new File("f22.jpg");
      long fileSize = imageFile.length();
      if (fileSize > Integer.MAX_VALUE) {
          throw new IllegalArgumentException("File it larger than a StringWriter can hold");
      }
      int size = (int) fileSize;
      byte[] expectedBytes = new byte[size];
      BufferedInputStream bis = new BufferedInputStream(new FileInputStream(imageFile), size);
      int bytesRead = bis.read(expectedBytes);
      assertEquals(size, bytesRead);

      // Request the representation
      Client client = new Client();
      WebResource f22 = client.resource("http://localhost:9998/planes/f22/f22.jpg");
      ClientResponse response = f22.accept(MediaType.WILDCARD).get(ClientResponse.class);
      MultivaluedMap headers = response.getHeaders();
      for ( String key : headers.keySet() ) {
          System.out.println( key + "=" + headers.get(key) );
      }
      byte[] returnedBytes = new byte[0];
      returnedBytes = response.getEntity(returnedBytes.getClass());

      // Compare the two sets of bytes to make sure they match
      assertEquals(size, returnedBytes.length);
      assertTrue(Arrays.equals(expectedBytes,returnedBytes));
  }
  catch (FileNotFoundException e) {
      AssertionError ae = new AssertionError("File containing expected HTML not found!");
      ae.initCause(e);
      throw ae;
  }
  catch (IOException e) {
      AssertionError ae = new AssertionError("Problems reading expected text!");
      ae.initCause(e);
      throw ae;
  }
}
Just like the core of Jersey, the client library enables clear, short code that clearly expresses what the developer is trying to do. This is a marked departure from many other web frameworks and APIs. It is refreshing to be able to write web services clients in such a terse, yet clear style.

[Note: I plan to continue working with Jersey and continue posting about it. But I'm going to take a few weeks to look into some other topics first. I'll make sure to update the list of JAX-RS and Jersey posts as I add more]

JAX-RS Response Entity Providers

[Note: This is one in a series of posts about JAX-RS. You may wish to start at the beginning]

If you read my previous entry about returning more than one representation of a resource, you saw this sample code for a RESTful F-22 fighter plane resource:

@Path("/planes/f22/")

public class F22 {

@GET @Produces("text/html") @Path( "f22.html" )

public File getHTMLRepresentation() throws IOException {

return new File( "f22.html");

}

@GET @Produces("image/jpeg") @Path( "f22.jpg" )

public File getImageRepresentation() {

return new File( "f22.jpg");

}

}

These methods might seem a bit odd at first sight. The getHTMLRepresentation method has a @Produces annotation which indicates the method produces a MIME type of "text/html", but the return type of the Java method is java.io.File. Similarly, the getImageRepresentation method produces "image/jpeg" but the Java method also returns a File.

This apparent disconnect is resolved by a JAX-RS feature called entity providers. The specification states that entity providers "supply mapping services between representations and their associated Java types". The job of entity providers is to take HTTP request entities and turn them into Java objects on the request side, and take Java objects and turn them into response entities on the result side. Entity providers which convert request entities to Java types implement the MessageBodyReader interface and must be marked with the @Provider annotation. Similarly, entity providers which convert Java return types to response entities implement the MessageBodyWriter interface and are marked with the @Provider annotation (see Figure 1).

Figure 1: Conversion between HTTP entities and Java objects

JAX-RS mandates a set of built-in entity providers covering common entity types such as strings, input streams, java.io.Reader objects, java.io.File objects, and a number of other sophisticated types of conversions.

In this case, Jersey takes the returned File object and uses it’s contents as the response entity which it marks as “text/html”. This mapping from Java types to HTTP entities can be quite powerful, as you don’t have to write code to read the contents of files and return them as byte arrays, Strings, etc. It also offers a JAX-RS implementation the opportunity to implement handling of some types much more efficiently. For example, a JAX-RS implementation might choose to use Java’s NIO facilities to read the contents of a file for performance reasons.

This notion of entity providers is a crucial aid to the POJO (Plain Old Java Objects) style of JAX-RS applications, and avoids what would otherwise be a great deal of boilerplate code in the application performing these conversions.

[Next: RESTful clients with the Jersey Client API]

Exploring RESTful web services with JAX-RS and Jersey.

I've been reading about RESTful web services for a while now. The best book on the subject I've found is RESTful Web Services by Leonard Richardson and Sam Ruby. I've been reading this book for what seems like forever (a function of my life and available time, not the book). I'm getting close to done and have begun using Jersey, the reference implementation of the JAX-RS specification to experiment with building RESTful web services.

I really like Jersey. A big reason for that is because Jersey makes it straightforward to stand up a standalone RESTful service. And once you've done it once, standing up additional services is easy. Deploying services without a heavyweight container makes me happy. Containers were created in the early days of Java when Java's limitations made them necessary. Since then, they have taken on a life of their own. Yet, in today's world I think the container has outlived almost all of its purpose. It's disadvantages now outweigh the few remaining benefits.

In the coming months, I plan to explore how to build, deploy, replicate, secure, and evolve industrial-class services without a container. I think I will succeed, but who knows? Maybe I'll learn a bunch due to abject failures. Either way, it should be fun.

My next post will describe how you download all the various pieces needed to stand up a Jersey-based REST service. There aren't very many, and it doesn't take long. Stay tuned.

[Next: Setting up Jersey for Use]