View from the Fringe

Pardon me, but I believe there's been a mistake...

2011-03-10T09:18:00.000-06:00

I logged into my Twitter account this morning after having not looked at it for quite some time. Imagine my surprise when I looked at the list of people following me and I see this:

While it's flattering to entertain the idea that the UK Prime Minister cares about my tweets, somehow I don't think so.

I've gotten my unintended humor for the day and the day has barely begun...

Jargon, REST, and Reuse

2011-03-01T11:53:00.000-06:00

Benjamin Carlyle has a great post about jargon in REST and how it relates to media types and reuse. RESTs uniform API is great but he succinctly makes a compelling case for why we also need to focus on reusing media types whenever possible to really enable reuse across services and over time.

It's well worth your time. Go read it now.

Handling conflicts in media type headers

2010-06-15T21:22:00.000-05:00

If you read my previous entry discussing versioning of RESTful services, I pointed out how we can leverage media types to support incompatible versioning of RESTful services. I think this is an excellent solution, but it can lead to an issue for service implementors

Imagine you have a service which has more than one version. Let's say it's a hotel reservation system, just to pick something familiar. We've picked media types for our versions of the service, and adopted the convention that in the absence of any media types we will use "Version 1" of the service. Assume our media types are "application/hotel-json" and "application/hotel-json-v2".

If you support operations for updating a reservation using PUT, you can run into a situation where you have both a Content-Type header (specifying the content of the entity body being used to update the reservation) and an Accept header specifying the kind or kinds of entities the client can accept in the response. This might be just fine, if the two versions of the service can be intermixed between the sent entity and the returned entity. But the two versions of the service might not mix. For example, an update sent using version one of the updating entity might not contain a value required for version two of the service.

In this situation what the client did is an error, so we clearly need to return a status code in the 400 range. But which is the most appropriate code? A quick look at the 4XX codes shows two promising candidates:

406 - Not Acceptable
415 - Unsupported Media Type

But when we read the description of 415 in RFC 2616 we see it isn't appropriate:

The server is refusing to service the request because the entity of the request is in a format not supported by the requested resource for the requested method.

The Accept header is specifying a media type the service understands for the resource, it's just the combination of provided entity media type and requested return media type that is a problem. Happily, status code 406 fits perfectly:

The resource identified by the request is only capable of generating response entities which have content characteristics not acceptable according to the accept headers sent in the request.

Interestingly, the RFC makes a comment about this very situation, and suggests that returning an entity type different than those specified in the Accept header might be preferable to a 406:

Note: HTTP/1.1 servers are allowed to return responses which are not acceptable according to the accept headers sent in the request. In some cases, this may even be preferable to sending a 406 response. User agents are encouraged to inspect the headers of an incoming response to determine if it is acceptable.

In this particular scenario (where a required value isn't provided), one way to return a result would be to "downgrade" to a version 1 response in hopes that works for the client. There are potential issues with this, if the semantics of applying a version 1 update differ from those that would have occurred with a version 2 update. And if the client can't interpret the returned version 1 entity, they may end up in an inconsistent state. But it's a good idea to keep in mind and make work whenever possible.

Versioning REST services

2010-06-15T10:04:00.001-05:00

I read a blog entry yesterday from Ganesh musing on whether RESTful web services need versioning. In his posting, Ganesh suggests putting some kind of version number into the body of a request. He suggests that this be done even for a GET or DELETE.

While I suppose this could work, it feels awkward. Putting a version number into an entity body for a GET, which would typically not have an entity body, is awkward and a fair bit of work. Things get even more complicated in the case of a PUT or POST where the entity body is some binary format like an image format. In this case, the Content-Type can't just be image/jpeg (for example) since your putting some kind of version information before the image data.

Even in the case of textual data such as XML, putting version information into the entity body requires that our XML format support a version number somewhere (probably as an attribute of the top-level element in the entity body?). This may be feasible, but it may not be feasible if we are trying to use a data format that is already defined. We could start hacking and just throw a line at the beginning of the entity body that indicates the version with the understanding that the 'normal' payload starts after this line. But this is hackish, and makes our Content-Type delcarations wrong (we aren't adhering to the content type definition since we've thrown that version information at the front).

After reading Ganesh's entry, I thought about versioning for a while and it seemed to me that the right way to handle versioning is via media types (using, for example, the Accept and Content-Type headers). This works well because the first version of our service can be delivered when no media-type headers are specified. Then, if a client wants to take advantage of a newer, incompatible version of our service, they simply specify this using media types in the Accept and Content-Type headers.

I could spend a fair bit of time describing this in more detail, but I did some googling and found a great blog entry (written back in May of 2008) by Peter Williams where he describes things concisely and clearly. It's short and sweet and well worth a read.

Back from reading his posting? There are two issues I thought of which Peter didn't directly address. One is specific to versioning, and one is a more general issue that can arise. The versioning related issue is this:

What if your service model changes to much that the set of resources you expose has to remove one of the resources in your original version of the service?

In this case, you make operations to that URI path with an Accept header and/or Content-Type header for later versions of your service return 404 (or similar error code) since that type of resource doesn't exist with that version of the service. While Peter didn't talk specifically about this, this approach meshes well with his description and I suspect it is the same answer he would give if asked.

A related but more general problem can occur if your service receives a request with both an Accept and a Content-Type header and the values are incompatible. I'll address that in my next entry.

[Updated 2010-06-15 11:25 a.m.]

I should mention that you shouldn't create new media types unless you have to. If your service deals with ATOM or RSS, use the version number for those to distinguish different versions of your service if at all possible. In his thesis, Roy Fielding states that creating new media types is not RESTful. And that is true in the sense that custom media types reduce the likelihood that clients will know how to interpret your representations. But there is a tension here between accurately representing incompatible versions of a service and using a pre-existing media type. There's no perfect answer, just a continuum with different trade-offs for different kinds of applications.

2010-05-31T18:51:00.001-05:00

I spoke at the Gateway Software Symposium (NFJS St. Louis) last weekend. After my RESTful Web Services with JAX-RS talk, one of the attendees asked me how to discover the proper URLs when starting to interact with a RESTful service. The impetus for the question came from my statement that many RESTful services don't have large specifications (such as the WSDL files for "Big" web services).

I explained that well-designed RESTful services are "well-connected". By this, I mean they adhere to the idea of "hypermedia as the engine of application state", or HATEOAS. If your service adheres to HATEOAS, your service is accessible and navigable given just a single URL. How can this be?

Recall that in RESTful services, we use URIs/URLs to connect various related resources. To be well-connected (I'll use this term in place of to HATEOAS mostly because the acronym looks odd to some people) and be RESTful, representations of resources should contain URIs/URLs to related resources. To make this clearer, let's look at a specific example. As we look at this example, we'll make note of the aspects of the API which couldn't be discovered by examining the results of searches or resource retrieval.

In my talks on JAX-RS, I use a simple music service. In this service, each artist has a name and some number of music albums they have published. Each album as a name and some number of tracks on that album. Here is a simple UML diagram of the objects:

We now come to the first piece of information we can't discover for ourselves. We need to know a starting URL. With the example service, it's simply a host name and a port, as in: "http://localhost:3131".

When we run the music service and hit it with a browser, we see a spartan web page:

While this certainly allows us to do some (mildly) interesting things, how does it help us discover the API for this RESTful service? The answer lies in examining the underlying HTML. When we do that, we see this:

At first, there might not seem to be much here. But let's look at the form for searching for artists:

From this we can see that we can search for artists whose name "is" (IS), "starts with" (STARTS), "contains" (CONTAINS), or "ends with" (ENDS) the text specified by 'searchString'. If you don't live and work with HTML forms and their resulting queries every day, this still might not be obvious. So we can always go ahead and do a search. Let's search for artists whose name begins with 'F':

Clicking the 'Search' button does two things for us:

1) It performs the search and gives us the results:

2) It shows us the proper format for executing queries against the service (in the browser's address text box):

Looking at this, we know that we can query for artists by specifying a searchType and a searchString if we search under /artists.

My example music service supports more than HTML for representations. It also supports an XML format and a JSON format. In order to use the XML format, we need to specify that we accept a content type of "application/music-xml" (a content type I made up for this example). This is the second piece of information the service itself did not tell us. However, we could simply query the service and get back an HTML representation of the search results so it's not strictly necessary. The XML format is a more compact representation than the HTML, but the same information is contained in the HTML version.

If we use the curl command line utility to query:

curl --header "Accept: application/music-xml" "http://localhost:3131/artists?searchType=STARTS&searchString=F"

We get XML results:

Because this service is designed with API discovery in mind, we see that it reminds us of the search that was performed:

< artists uri="/artists?" searchString="F" searchType="STARTS">

In these results we also see an example of how to build well-connected services. Notice that each search result lists not only the name of the artist found, but also the URI of the resource we would need to retrieve in order to get a representation for that artist. For example, the artist "Fish" lists a URI of /artists/id/149. If we go to the music service in the browser and query for this URI (a URL of http://localhost:3131/artists/id/149), we get information about the artist named Fish:

Looking at the HTML, we see:

And if we query for the XML representation using curl:

curl --header "Accept: application/music-xml" "http://localhost:3131/artists/id/149"

We see:

Notice that the information about each album contains a URI we can use to retrieve a representation of that album, just like the search results listed a URI for each artist that was found. These are examples of how we build a well-connected RESTful service. By making sure that each HTML page (or XML result) contains URIs for related resources, users of our services can discover the proper URIs/URLs to use when invoking our service without a large specification.

In part two, I'll explore this notion further, looking at how our self-describing service allows us to discover how to create and delete resources as well as search for and retrieve them.

Gateway Software Symposium 2010 (NFJS St. Louis)

2010-05-22T14:51:00.001-05:00

I'm attending (and speaking at) the NFJS conference here in St. Louis this weekend. The quality of the talks is excellent, and the speakers clearly know their stuff.

The two talks I'll be delivering on Sunday are:

I hope if you're attending the conference you come to the talks to learn and participate.

Some of my fellow OCI-ers are also presenting:

Sessions I've attended include:

Open Source Debugging Tools for java - Matthew McCullough
The Busy Java Developer's Guide to Advanced Collections - Ted Neward
Software Architecture for the Cloud - Michael Nygard
Programming Scala - Venkat Subramaniam
Emergent Design - Neal Ford
The Busy Java Developer's Guide to Concurrency Part 1 and Part 2 - Ted Neward

All good stuff.

URI Fragments

2010-03-28T09:56:00.000-05:00

I learned something interesting when reading the Architecture of the World Wide Web, Volume One. It turns out that URI fragments (the part of a URI after a '#' character) are not interpreted as part of a URI:

Note that the HTML implementation in Emma's browser did not need to understand the syntax or semantics of the SVG fragment (nor does the SVG implementation have to understand HTML, WebCGM, RDF ... fragment syntax or semantics; it merely had to recognize the # delimiter from the URI syntax [URI] and remove the fragment when accessing the resource). This orthogonality (§5.1) is an important feature of Web architecture; it is what enabled Emma's browser to provide a useful service without requiring an upgrade.

The semantics of a fragment identifier are defined by the set of representations that might result from a retrieval action on the primary resource. The fragment's format and resolution are therefore dependent on the type of a potentially retrieved representation, even though such a retrieval is only performed if the URI is dereferenced. If no such representation exists, then the semantics of the fragment are considered unknown and, effectively, unconstrained. Fragment identifier semantics are orthogonal to URI schemes and thus cannot be redefined by URI scheme specifications.

Interpretation of the fragment identifier is performed solely by the agent that dereferences a URI; the fragment identifier is not passed to other systems during the process of retrieval. This means that some intermediaries in Web architecture (such as proxies) have no interaction with fragment identifiers and that redirection (in HTTP [RFC2616], for example) does not account for fragments.

While I had an intuitive understanding of how browsers work with URI fragments in HTML documents (retrieve the document, find the fragment, display the document starting at the fragment), I hadn't considered the semantic split, nor the implications of URI fragments being applied to other kinds of representations.

I find the handling of fragments interesting in a number of ways. For one thing, it means that as new content types become part of the web, the creators of those content types are free to map URI fragments into that content type. So in HTML the format of URI fragments is generally a textual name that appears in the html. But in a 3D modeling format, it might take the form of [position,orientation,scale] to define a location from which the model is being viewed, the direction the camera is facing, and the scaling factor. That's nice because it allows URI fragments to be structured in a manner most appropriate for the kind of representation being retrieved.

One possibly surprising consequence of this split is that URI fragments are not considered in URI resolution activities such as interacting with proxies or redirection. It also means that you shouldn't try to use URI's with fragments as if they represented actual resources, since the web isn't allowed to cache individual fragments and no semantic interpretation is allowed from the '#' character onward.

Why you should care about adhering to the Architecture of the World Wide Web

2010-03-24T23:42:00.000-05:00

I was reading more of the Architecture of the World Wide Web, Volume One", and RFC 2616 and the definition of "safe" methods:

9.1.1 Safe Methods

Implementors should be aware that the software represents the user in their interactions over the Internet, and should be careful to allow the user to be aware of any actions they might take which may have an unexpected significance to themselves or others.

In particular, the convention has been established that the GET and HEAD methods SHOULD NOT have the significance of taking an action other than retrieval. These methods ought to be considered "safe". This allows user agents to represent other methods, such as POST, PUT and DELETE, in a special way, so that the user is made aware of the fact that a possibly unsafe action is being requested.

Naturally, it is not possible to ensure that the server does not generate side-effects as a result of performing a GET request; in fact, some dynamic resources consider that a feature. The important distinction here is that the user did not request the side-effects, so therefore cannot be held accountable for them.

This matches quite nicely with my intuition about how the web works (at least the normal web). That might seem like something trivial, but in fact it is something profound.

Unfortunately, many, many web services violate the principle of safe operations. For example, they frequently have all interactions occur via GET, and use query parameters or headers or other conventions to stipulate whether the result is a resource retrieval, creation, update, or delete. This is (unfortunately) only one example of aberrant service design.

At first, this might not seem like such a problem. But as you start to think about consuming such services or providing support for services you've built, you realize that the conventions of HTTP and the architecture of the web represent a sort of "lingua franca". Rather than making your services do more, you are making your services easier for (potential) consumers of your service to understand and use. If you adhere to the architecture of the web, your services don't do any more than they would have done otherwise. Rather (and this is crucial) you have implemented your services in such a way that users of your services can very easily and rapidly understand their features, capabilities, and limitations. That is a huge advantage in today's fast-moving technology world.

Parsing "The Architecture of the World Wide Web, Volume 1"; URI allocation

2010-03-08T23:10:00.004-06:00

As I continue to read Architecture of the World Wide Web, Volume I, I keep running into material that is completely outside of what I would have expected, yet valuable.

For example, Section 2.2.2 talks about URI allocation. Since URIs are supposed to identify a single resource, it becomes important to make sure that the social organizations which allocate and assign URIs are organized so that they don't allocate the same URI to refer to more than one resource. In other words, we want to make sure that we give organizations authority to assign URIs that don't overlap, so that different organizations don't assign the same URI to different resources (sort of like giving the same Social Security number or driver's license number or bank account number to two different people).

This sort of material may sound obvious when we read it. But it is frequently not obvious to everyone involved in building, deploying, managing, and evolving software systems. In fact, I think failure to make these sorts of issues clear at the architectural and administrative levels is quite possibly the single greatest cause of problems in managing software systems in the real world.

Principals of the Web

2010-03-02T23:15:00.000-06:00

As I noted in a previous entry, I've been reading Architecture of the World Wide Web, Volume One and am finding it a great read. For example, take this little gem:

Constraint: URIs Identify a Single Resource

Assign distinct URIs to distinct resources.

In a nutshell the authors have made it clear that a URI should refer to a particular resource. And just a bit further on they point out that URI's can be aliases for a single resource:

Just as one might wish to refer to a person by different names (by full name, first name only, sports nickname, romantic nickname, and so forth), Web architecture allows the association of more than one URI with a resource. URIs that identify the same resource are called URI aliases. The section on URI aliases (§2.3.1) discusses some of the potential costs of creating multiple URIs for the same resource.

They even offer thoughts on the performance consequences of aliases.

Ya gotta love it... :-)

The Architecture of the World Wide Web - Volume 1

2010-03-01T00:18:00.003-06:00

As part of my foray into RESTful services, I've been reading The Architecture of the World Wide Web - Volume 1 and find it refreshingly informative. For example:

The choice of syntax for global identifiers is somewhat arbitrary; it is their global scope that is important. The Uniform Resource Identifier, [URI], has been successfully deployed since the creation of the Web. There are substantial benefits to participating in the existing network of URIs, including linking, bookmarking, caching, and indexing by search engines, and there are substantial costs to creating a new identification system that has the same properties as URIs.

And:

A resource should have an associated URI if another party might reasonably want to create a hypertext link to it, make or refute assertions about it, retrieve or cache a representation of it, include all or part of it by reference into another representation, annotate it, or perform other operations on it. Software developers should expect that sharing URIs across applications will be useful, even if that utility is not initially evident.

There is so much packed into each of these brief statements, and they are in equivalent of the first 10 pages of the document.

I find it both amazing and sad that this document was published in 2004 yet I've found very few references to it in the six years since it's publication. Maybe I just haven't been looking in the right places?

I will share additional passages that I find enlightening in the days (and weeks?) to come.

GWT and SmartGWT

2010-02-27T15:30:00.003-06:00

I'm doing some work with GWT 2.0 and also with SmartGWT. I like both toolkits, but I'm in the midst of a very steep learning curve (meaning I'm learning a lot quickly). While I'm making good progress in becoming proficient, I'm finding that SmartGWT suffers from the same problem that many open source products with commercial support options suffer from: weak documentation.

It's completely understandable and I don't blame the developer(s) of SmartGWT. When you are working on an open source project and also trying to provide commercial support as a means of revenue, it's hard to find the time to produce good documentation. And the truth is that if you provide a good product with documentation good enough that developers don't need your services, you have just put yourself out of business.

Such is the nature of open source projects that have commercial support models as their primary financing model. There's nothing that can be done about it (except find a philanthropist who will fund the project; and philanthropists of any sort are a rare breed these days, not to mention those interested in the the obscurity of open source software).

So I'll keep climbing the learning curve with the documentation as is, and take good notes for the future when I've stopped working with these technologies and come back to them.

Oh, and if you know any philanthropists looking to fund open source projects, I've got a project or two of my own I can suggest...

Guice 2.0 - tasty

2010-02-10T17:38:00.006-06:00

I'm finally getting a chance to do some work with Guice 2.0. I don't know if I just couldn't wrap my head around Guice in the past and I've finally "gotten" it, or if Guice 2.0 provides a more approachable API. Either way, I'm finding it great to use.

I've been mildly whiny about Guice in the past, stating that it wasn't completely statically typed (which is true, since it's possible you'll ask for a resource at runtime that isn't available because it wasn't configured). But even with that small limitation, I'm finding Guice 2.0 to be far better than Spring for dependency injection. The code is much smaller, much more type-safe, not XML (a big plus), and much more strongly type-checked.

If you haven't checked out Guice, or if you tried Guice 1.0 but haven't tried out 2.0, you should give 2.0 a serious look.

Now I need to integrate Guice with Jersey (the JAX-RS reference implementation)...

JAX-WS tarpit

2010-02-08T14:21:00.007-06:00

I'm currently doing some work with an open source framework (to be left unnamed) built upon JAX-WS (using SOAP, SAML, and HTTPS). I'm making headway working with it, but talk about a tar pit. You go into the code and it's almost impossible to get out. Every time you gain a bit of understanding, some other anti-pattern slaps you until you see starts and you're back to global searches with google and looking through the code to make further progress.

Violations of the DRY principle are rampant. The generated code has magic constants sprinkled liberally throughout. Generated classes have default constructors and public getters and setters for all fields despite the fact that some fields are actually required; no hashCode; no equals; no ability to determine if one of these data objects is valid or not. Turning the WSDL into code generates vast numbers of classes (reminding me of the terrible mapping from CORBA IDL to Java). The generated code has almost no helpful comments (despite the fact that generating at least some reasonable back-references in generated code is easy precisely because you are generating code). And we've only delved a bit into the SAML aspects of things; I expect that to be another can of worms entirely.

I've heard that the SOAP/WS-* specifications were co-opted by certain large companies and made so complex it's nearly impossible to work with them except with the very big IDEs-with-god-complexes those vendors sell. Based on these experiences and previous ones working with some SOA frameworks, I can believe it. If that isn't the reason for their incredible opaqueness and anti-patterns, I'm afraid to find out who thought all of this was a good idea.

So while I'm making progress and getting things done, the entire experience makes me want to go wash my hands and update my resume.

If you know of good ways to work with this stuff short of plunking down vast quantities of money for some IDE that will sort-of/mostly/kind-of hide all the complexity (until the moment things break and you really need to understand what is going on), I'd love to hear about them...

Java Annotations have Become Pixie Dust

2010-02-02T11:35:00.001-06:00

I was giving a talk about RESTful services using JAX-RS and Jersey recently and was asked why I had used Mark Volkmann's WAX for generating HTML and XML. The person asking the question pointed out that Jersey has integration with JAXB.

There were two answers to that question. One answer is that I am leery of anything which automatically converts my Java objects into a serialized format (bitten by Java's object serialization in the past). Incompatible object changes can be difficult or impossible to reconcile in a backward-compatible manner.

But the main answer I gave got some chuckles and further questions. I explained I was trying to avoid too much "pixie dust". In the example code, I was already using the Java Persistence API (JPA) and JAX-RS and their associated annotations. If I had not been careful, there would have been annotations for Spring and JAXB as well. All of these annotations are small in the code but have very large effects. Those innocent looking annotations subject my poor unsuspecting code to some very big (and some would argue bad) frameworks. Understanding how these frameworks interact is not only hard, but those interactions change as the frameworks change (possibly resulting in the system breaking with no code changes).

I have real misgivings about the number of annotation-based technologies that should be applied to any one project. Each annotation you use represents some amount of code you don't have to write. And that is, of course, a good thing from a development perspective. But every annotation you use represents 'pixie dust', behavior which is hidden from you and generally performed at runtime. That means that when things go wrong, there isn't any code to look at. It also means that small differences between configurations can produce large changes in behavior. That's a very bad thing for testing, production deployment, production maintenance, etc.

I've been thinking about this issue for some time*, so I was pleasantly surprised to find Stephen Schmidt's post admonishing us to Be careful with magical code. His post is not specific to annotations (he calls out aspect-oriented programming, for example - I agree that AOP is another kind of pixie dust). And he points out some examples of the "pixie dust" phenomenon. While I don't agree with his 'magic containment' example, it's a good post. You should read it.

As a rule of thumb, I think two kinds of pixie dust is the maximum to sprinkle on a project. So think hard and choose wisely when picking which ones to use: the more kinds of pixie dust you sprinkle, the harder it will be for you and others to understand and troubleshoot things, now and especially in the future.

*Thanks to Mike Easter for planting the idea of talking about the state of annotations in Java

Taxonomy of Technical Blog Posts

2010-01-08T17:15:00.007-06:00

I categorize technical blog postings into a taxonomy:

Type I: Describing how to use some kind of technology, your own or someone else's

Type II: Describing how to overcome some limitation, bug, or quirk of technology

Type III: Whining about failures to get one or more technologies to work (together)

Type IV: Crowing about getting one or more technologies to work (together) - often a follow-up to a Type III posting

Type V: Indulging in a post that really doesn't belong in a technical blog.

For a classic example of a Type II posting, see:

http://codetojoy.blogspot.com/2010/01/grails-tip-for-internet-connections.html

A RESTful web service testbed

2010-01-05T22:24:00.001-06:00

It's time to build a more complete RESTful service example. In this web service, we'll aim to perform all the common activities of a real service. This is often referred to by the acronym CRUD, which stands for Create, Read, Update, and Delete for all of the common operations associated with persistent data. Since we're dealing with a RESTful web service, we'll also throw in multiple representations for a resource, and connectedness to make it easy for clients to navigate through the service resources.

So, what should our service do? My friend and colleague, Mark Volkmann, introduced the eample of a database of music information. In this example, the service service contains information about several kinds of resources:

Music artists (Artists)
Albums
Songs

This is a nice example because it is rich enough to expose many of the problems that need to be solved by RESTful web service without becoming so large that it's unwieldy to explore. The domain model is immediately familiar, so we can focus on the technologies and not the model.

With this service we can explore not only traditional browser-based HTML, but XML and even an Ajax client using JSON. We can also use it as testbed for things like caching, security, clustering/failover, and composition of services.

JAX-RS and Jersey talk materials

2009-12-07T23:10:00.003-06:00

I recently gave a talk on JAX-RS, JSR 311, and Jersey at the St. Louis Java User's Group website. You can find the presentation and the sample code on their website.

Curl has a bug with redirects?

2009-11-13T10:25:00.015-06:00

I gave a presentation at the JUG last night about JAX-RS and Jersey. The talk went well, and it was a great crowd with a lot of participation (I'll post more once I have the materials cleaned up and they are posted to the JUG website).

But one thing that didn't go right last night was when I was using the 'curl' command line tool to create new artists in the RESTful music database service I was using as an example.

The curl man page on my MacBook Pro says:

When curl follows a redirect and the request is not a plain GET (for example POST or PUT), it will do the following request with a GET if the HTTP response was 301, 302, or 303. If the response code was any other 3xx code, curl will re-send the following request using the same unmodified method.

Yet, when I was using it, it wasn't doing that. For example when trying to create artist 'Jane':

brian@Widget: curl -v -H "Accept: text/html" -X POST -L "http://localhost:3131/artists/Jane"
* About to connect() to localhost port 3131 (#0)
*   Trying ::1... Connection refused
*   Trying fe80::1... Connection refused
*   Trying 127.0.0.1... connected
* Connected to localhost (127.0.0.1) port 3131 (#0)
> POST /artists/Jane HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: localhost:3131
> Accept: text/html
> 
< HTTP/1.1 303 See Other
< server: grizzly/1.9.10
< Location: http://localhost:3131/artists/id/313
< Content-Type: text/plain; charset=iso-8859-1
< Content-Length: 0
< Date: Fri, 13 Nov 2009 05:24:12 GMT
< 
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:3131/artists/id/313'
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 3131 (#0)
> POST /artists/id/313 HTTP/1.1
> User-Agent: curl/7.16.3 (powerpc-apple-darwin9.0) libcurl/7.16.3 OpenSSL/0.9.7l zlib/1.2.3
> Host: localhost:3131
> Accept: text/html
> 
< HTTP/1.1 405 Method Not Allowed
< server: grizzly/1.9.10
< Allow: DELETE,OPTIONS,HEAD,GET
< Content-Type: text/plain; charset=iso-8859-1
< Content-Length: 0
< Date: Fri, 13 Nov 2009 05:24:12 GMT
< 

When it followed the redirect and issued the follow-up request, it went to the right place:

...
< HTTP/1.1 303 See Other
< server: grizzly/1.9.10
< Location: http://localhost:3131/artists/id/313
...
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:3131/artists/id/313'

But it didn't switch to GET:

> POST /artists/id/313 HTTP/1.1

As a result, I got a 405 error code because curl tried to do a POST to the redirected URL rather than doing a GET as is documented.

I did some further experiments last evening after the talk and confirmed this behavior. I even tried switching the return code to a 302 just to see if it would make a difference and it doesn't.

How does your curl behave? Does it switch from POST to GET when following a redirect using the -L option?

I'd appreciate it if anyone who knows anything about this could comment. Is curl doing the wrong thing, or is the man page wrong and curl is no longer supposed to do this?

RESTful clients with the Jersey Client API

2009-07-21T22:00:00.023-05:00

[Note: This is one in a series of posts about JAX-RS. You may wish to start at the beginning]

Jersey, the reference implementation of the JAX-RS specification, is a framework that makes it very easy to implement RESTful web services. But there is more to Jersey than just the server side. In this post I will spend a little time exploring the Jersey client library.

Depending upon how you design your RESTful service, you may or may not want to have separate URLs for separate representations of the same resource. This presents a problem when trying to test with a browser. There is no way to tell popular browsers that you want a text/html representation or an image/jpeg representation. The browser has a list of preferred media types, but none that I'm aware of allow you to customize this (either in general or for a particular request). Even more importantly, we need to be able to build solid unit tests for our services. The Jersey client framework provides a good solution to this problem. It is designed to allow developers of RESTful web services to write good unit tests, but is more general purpose than that. It can also be used to write RESTful client applications.

There is an excellent tutorial on the Jersey client API, which you should download and read if you plan to use it. But I will give you a taste of the API here.

The Jersey client API is very clean, and requires a minimum of fuss to use. As an example, let's write a unit test which tests our web service serving up information about fighter planes. First, we write the code to set up and tear down our service implementation. This code is the same as that in our Main class before, just split up between the setup and tear-down methods.

public class Test3b {

private SelectorThread selector;

@org.junit.Before
public void createService() throws IOException {
   Map initParams = new HashMap();
   initParams.put("com.sun.jersey.config.property.packages", "net.gilstraps.server");
   selector = GrizzlyWebContainerFactory.create("http://localhost:9998/", initParams);
}

@org.junit.After
public void DestroyService() {
   selector.stopEndpoint();
   selector = null;
}

The only difference in this case is we don't read from standard in to shut down the service, since we always want to shut it down at the end of the unit test.

Now, we can test that the text/html representation returned by invoking the service is what we expect. First, we do a bit of hoop jumping to read in a copy of the HTML we expect to receive:

@org.junit.Test
public void testF22Html() {
try {
   File expected = new File("f22.html");
   long fileSize = expected.length();
   if (fileSize > Integer.MAX_VALUE) {
       throw new IllegalArgumentException("File it larger than a StringWriter can hold");
   }
   int size = (int) fileSize;
   BufferedReader r = new BufferedReader(new FileReader(expected), size);
   char[] chars = new char[size];
   int readChars = r.read(chars);
   if (readChars != size) {
       throw new RuntimeException("Failed to read all chars of the expected result html file");
   }
   final String expectedText = new String(chars);

At this point, the variable expectedText contains what we should receive back from our request. Making the request is quite straightforward. First, we create a JAX-RS client:

Client client = new Client();

These clients are 'heavyweight' objects (taking time to create and using significant resources). As such, in a production client we would want to create this client once and use it many times. For the sake of independent unit tests, we will go ahead and create a Client object for each test.

Next, we specify the resource we want to retrieve using a WebResource object:

WebResource f22 = client.resource("http://localhost:9998/planes/f22/f22.html");

And then we ask the client to retrieve the resource for us, specifying that we want a text/html representation (MediaType.TEXT_HTML_TYPE) and specifying that we want to get back a ClientResponse object:

ClientResponse response =
f22.accept(MediaType.TEXT_HTML_TYPE).get(ClientResponse.class);

This code is using the builder pattern, where we build up our request through a chain of method calls. In this case, the chain is only two calls long. First, we call the accept method to specify the media types we will accept in the response (text/html), then we call the get method to actually retrieve the resource. It is possible to chain together more calls to specify other characteristics of either the request or the expected response (for more information, see the whitepaper on the Jersey client API).

Now that we have gotten a representation of the resource in the form of an HTTP response, we can then retrieve the HTML entity contained within that response as a string:

String returnedHTML = response.getEntity(String.class);
And finally, since this is a unit test, we make sure that what we got back matches what we expected:
assertEquals("Expected and actual HTML don't match"
expectedText, returnedHTML);
The ClientResponse object is useful if you want to look at other characteristics of the response, such as the returned headers. In this case, we could just as well have asked for the string from the WebResource directly. To do so, we would replace these two lines:
ClientResponse response =
f22.accept(MediaType.TEXT_HTML_TYPE).get(ClientResponse.class);
String returnedHTML = response.getEntity(String.class);
With this one: 
String returnedHTML = asHTML.accept(MediaType.TEXT_HTML_TYPE).get(String.class);
Testing for retrieval of the JPEG representation of an image is almost identical. The only differences are that we read in the image file as an array of bytes, ask for the response entity as an array of bytes, and compare the two as arrays of bytes. Here is the entire test method:

@org.junit.Test
public void testF22JPEG() {
  try {
      // Read in our expected result
      File imageFile = new File("f22.jpg");
      long fileSize = imageFile.length();
      if (fileSize > Integer.MAX_VALUE) {
          throw new IllegalArgumentException("File it larger than a StringWriter can hold");
      }
      int size = (int) fileSize;
      byte[] expectedBytes = new byte[size];
      BufferedInputStream bis = new BufferedInputStream(new FileInputStream(imageFile), size);
      int bytesRead = bis.read(expectedBytes);
      assertEquals(size, bytesRead);

      // Request the representation
      Client client = new Client();
      WebResource f22 = client.resource("http://localhost:9998/planes/f22/f22.jpg");
      ClientResponse response = f22.accept(MediaType.WILDCARD).get(ClientResponse.class);
      MultivaluedMap headers = response.getHeaders();
      for ( String key : headers.keySet() ) {
          System.out.println( key + "=" + headers.get(key) );
      }
      byte[] returnedBytes = new byte[0];
      returnedBytes = response.getEntity(returnedBytes.getClass());

      // Compare the two sets of bytes to make sure they match
      assertEquals(size, returnedBytes.length);
      assertTrue(Arrays.equals(expectedBytes,returnedBytes));
  }
  catch (FileNotFoundException e) {
      AssertionError ae = new AssertionError("File containing expected HTML not found!");
      ae.initCause(e);
      throw ae;
  }
  catch (IOException e) {
      AssertionError ae = new AssertionError("Problems reading expected text!");
      ae.initCause(e);
      throw ae;
  }
}
Just like the core of Jersey, the client library enables clear, short code that clearly expresses what the developer is trying to do. This is a marked departure from many other web frameworks and APIs. It is refreshing to be able to write web services clients in such a terse, yet clear style.

[Note: I plan to continue working with Jersey and continue posting about it. But I'm going to take a few weeks to look into some other topics first. I'll make sure to update the list of JAX-RS and Jersey posts as I add more]

JAX-RS Response Entity Providers

2009-07-09T22:12:00.001-05:00

[Note: This is one in a series of posts about JAX-RS. You may wish to start at the beginning]

If you read my previous entry about returning more than one representation of a resource, you saw this sample code for a RESTful F-22 fighter plane resource:

@Path("/planes/f22/")

public class F22 {

@GET @Produces("text/html") @Path( "f22.html" )

public File getHTMLRepresentation() throws IOException {

return new File( "f22.html");

}

@GET @Produces("image/jpeg") @Path( "f22.jpg" )

public File getImageRepresentation() {

return new File( "f22.jpg");

}

These methods might seem a bit odd at first sight. The getHTMLRepresentation method has a @Produces annotation which indicates the method produces a MIME type of "text/html", but the return type of the Java method is java.io.File. Similarly, the getImageRepresentation method produces "image/jpeg" but the Java method also returns a File.

This apparent disconnect is resolved by a JAX-RS feature called entity providers. The specification states that entity providers "supply mapping services between representations and their associated Java types". The job of entity providers is to take HTTP request entities and turn them into Java objects on the request side, and take Java objects and turn them into response entities on the result side. Entity providers which convert request entities to Java types implement the MessageBodyReader interface and must be marked with the @Provider annotation. Similarly, entity providers which convert Java return types to response entities implement the MessageBodyWriter interface and are marked with the @Provider annotation (see Figure 1).

Figure 1: Conversion between HTTP entities and Java objects

JAX-RS mandates a set of built-in entity providers covering common entity types such as strings, input streams, java.io.Reader objects, java.io.File objects, and a number of other sophisticated types of conversions.

In this case, Jersey takes the returned File object and uses it’s contents as the response entity which it marks as “text/html”. This mapping from Java types to HTTP entities can be quite powerful, as you don’t have to write code to read the contents of files and return them as byte arrays, Strings, etc. It also offers a JAX-RS implementation the opportunity to implement handling of some types much more efficiently. For example, a JAX-RS implementation might choose to use Java’s NIO facilities to read the contents of a file for performance reasons.

This notion of entity providers is a crucial aid to the POJO (Plain Old Java Objects) style of JAX-RS applications, and avoids what would otherwise be a great deal of boilerplate code in the application performing these conversions.

[Next: RESTful clients with the Jersey Client API]

Multiple representations of a REST Resource using JAX-RS

2009-07-08T14:56:00.008-05:00

[Note: I have a post that provides an in-order list of all the JAX-RS and Jersey-related posts I've written. You may want to start at the beginning if you haven't already read the previous posts]

One of the principles of RESTful web services is that a client makes a request for a representation of a resource. The resource itself is an abstract notion, just as an object-oriented model of some real world object is an abstraction. But in the case of a RESTful web service, it becomes possible to return different kinds of information based upon what the client requests.

As a simple example, imagine we are aficionados of military planes and decide to create a web service which provides non-secret information (and perhaps speculation) about various military aircraft (thanks to my friend Bruce for this idea). In the REST world, our resource might be "F-22". In this case the resource is the United States F-22 fighter plane. But there are many possible representations of an F-22. We might provide a summary description of the plane, or an image of the plane, or a CAD schematic of the plane, or any of a number of other representations. This basic notion that a single resource may have more than representation can be demonstrated with a simple object that can provide either an HTML text description of the F-22 or a simple graphic drawing of the plane.

Perhaps the simplest way to provide more than one representation for a given resource is to provide two different methods in a Java class with different MIME types in their @Produces annotations. In our example, assume we have a class named F22 which provides representations of an F-22. It can have a method that returns HTML and a different method that returns a JPEG image. This is very easy to do with JAX-RS:


@Path("/planes/f22")
public class F22 {
@GET @Produces("text/html")
public String getHTMLRepresentation() throws IOException {
    File f = new File( "F22.html");
    BufferedReader br = new BufferedReader(new FileReader(f));
    StringWriter sw = new StringWriter();
    for( String s = br.readLine(); s != null ; s = br.readLine() ) {
        sw.write( s );
        sw.write( '\n' );
    }
    return sw.toString();
}

@GET @Produces("image/jpeg")
public File getImageRepresentation() {
    return new File( "F22.jpg");
}

}

We now run into a problem using a web browser for a testing client. The browser is going to specify in it’s HTTP request the formats it prefers for a response. The browser doesn’t allow us to specify the type of representation we want. So, do we get back the HTML representation or the JPEG image? The answer depends upon the particular web browser you are using. Firefox 3.5 on a Mac and Windows and Safari on a Mac return the HTML form:

IE 6 on Windows and IE 8 on Windows return the JPEG image:

There are other ways to return more than on representation. For example, can have a single method which returns a JAX-RS Response object. That Response can have different contents depending upon the requested representation. I may explore these in future posts.

[Next: JAX-RS Response Entity Providers]

JAX-RS and Jersey posts

2009-06-12T21:14:00.007-05:00

I'm writing a series of posts on JAX-RS and Jersey (the reference implementation for JAX-RS). I'll be updating this post to provide links to all the others. This post gives you "one stop shopping" for all the related posts.

JAX-RS Hello World

2009-06-08T22:24:00.013-05:00

[NOTE: All of my examples will use standalone services (not housed inside a container). The integration between Jersey and the Grizzly HTTP server makes this a snap]

So, what is the simplest, smallest hello world we can implement with JAX-RS? The Jersey site provides an example and it's surprisingly small. We will start with an example almost identical to the hello world example from the Jersey documentation and grow from there.

If we ignore the package statements and imports, there are two files of about ten lines each. The first one is our main class that starts up the service and generally gets things going. Later examples will show why this is it's own class separate from the rest of a service's implementation. Here's the class in its entirety:


package net.gilstraps.server;

import com.sun.grizzly.http.SelectorThread;
import com.sun.jersey.api.container.grizzly.GrizzlyWebContainerFactory;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;

public class Main {

   public static void main(String[] args) throws IOException {
       Map<String,String> initParams = new HashMap<String, String>();
       initParams.put( "com.sun.jersey.config.property.packages", "net.gilstraps.server" );
       SelectorThread selector = GrizzlyWebContainerFactory.create( "http://localhost:9998/", initParams );
       //noinspection ResultOfMethodCallIgnored
       System.in.read();
       threadSelector.stopEndpoint();
       System.exit(0);
   }

}

The code here is pretty easy to understand, so let's look at it. These two lines are a crucial part of configuring Jersey:


       Map<String,String> initParams = new HashMap<String, String>();
       initParams.put( "com.sun.jersey.config.property.packages", "net.gilstraps.server" );

With these lines, we tell Jersey which java packages should be examined at runtime to find classes which implement RESTful resources. We've specified that Jersey should look at the "net.gilstraps.server" package and examine the classes in that package to find ones which have JAX-RS annotations on them. Those annotations tell a JAX-RS implementation what resources each class serves up to clients and what URI's those resources have. There are also annotations for specifying which HTTP methods match up to which Java methods. We'll look at these shortly.

The next line actually instantiates the Grizzly HTTP server, telling it the base URI for all the URL's it should support and providing it with initialization parameters:


       SelectorThread selector = GrizzlyWebContainerFactory.create( "http://localhost:9998/", initParams );

The call returns us a SelectorThread which represents the control object for the HTTP server. With the string "http://localhost:9998/", we are telling the service to use port 9998 of the host named 'localhost'. Grizzly has built-in knowledge of Jersey and our inclusion of the "com.sun.jersey.config.property.packages" parameter causes Jersey to used to provide the web service.

The remaining lines provide a simple means to shut down the service gracefully. The program reads input from System.in and then shuts things down cleanly. So if you run the program on a command line you can get it to shut down without killing it:


       System.in.read();
       threadSelector.stopEndpoint();
       System.exit(0);

So, where is our cliched "hello world" resource? There is one more class used to implement our service, also quite small. Here it is:


package net.gilstraps.server;

import javax.ws.rs.Produces;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.swing.text.html.HTML;

@Path("/helloworld")
public class HelloWorldResource {
   @GET @Produces("text/plain")
   public String getClichedMessage() {
       return "Hello Stage 1";
   }
}

This class is almost a POJO (plain old Java object). It doesn't implement a particular interface, or extend a class. It has a very simple method ('getClichedMessage') that returns a regular Java string. How can this implement a hello world web service? The key to making it a JAX-RS resource class is the annotations. Let's examine each one:

@Path

The '@Path' annotation attached to the class itself tells a JAX-RS what URI or set of URIs this class serves. In this case, we are specifying just the URI of "/helloworld". Thus, with the base URI of "http://localhost:9998" we would access this resource via the URI "http://localhost:9998/helloworld".

@GET

The @GET annotation on a method tells JAX-RS that the Java method implements HTTP's GET verb (the verb your browser uses every time you visit a web page). So JAX-RS now knows that a client doing a GET of "http://localhost:9998/helloworld" should result in the 'getClichedMessage' method being invoked.

@Produces

The @Produces annotation tells JAX-RS that the MIME type of the resource returned by 'getClichedMessage' is "text/plan", which is the MIME type for a plain text document. We've aimed for a minimal service here, so we're going for the simplest type of resource we can. It's also possible to return other kinds of entities as the result of a request, and it's even possible to use separate conversion classes to convert different kinds of Java objects into particular kinds of response entities (like a JPEG image, for example). But this service just returns a text document.

At this point, JAX-RS knows everything it needs to know to support a client doing a GET on the resource identified by the URI "http://localhost:9998/helloworld". When we compile and run our service, it doesn't output anything at first. But if we open a browser and go to the URL, we see some output from Grizzly/Jersey:


Jun 12, 2009 9:02:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
 net.gilstraps.server
Jun 12, 2009 9:02:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Root resource classes found:
 class net.gilstraps.server.HelloWorldResource
Jun 12, 2009 9:02:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Provider classes found:

From this we can see that Grizzly brings Jersey in to handle the request we made with the browser. Jersey then scans for classes which serve up REST resources. It tells us that it found our class:


Jun 12, 2009 9:02:51 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Root resource classes found:
 class net.gilstraps.server.HelloWorldResource

Even more interesting, the browser where we made the GET request gets the string that was returned from 'getClichedMessage' as the result of it's GET request:

And that's a simple 'hello world' implemented using JAX-RS with Jersey and Grizzly.

[Next: Multiple representations of a REST Resource using JAX-RS ]

Setting up Jersey for use

2009-06-02T21:06:00.003-05:00

If you use Maven you can use it to download Jersey and keep it up to date. There are links to the POM files on the getting started page. In particular, you need the jersey-server module, and the grizzly-servlet-webserver module. If you want to use WADL with JDK 1.5 you need the jaxb-impl module. If you are not using Maven, you need only a few JARs:

grizzly-servlet-webserver.jar
jersey-server.jar
jersey-core.jar
jsr311-api.jar
asm.jar

[NOTE: these projects are under active development, so you may find newer versions of these jars available]

As with using Jersey via Maven, if you want to use WADL with JDK 1.5, you need some additional jars:

jaxb-impl.jar
jaxb-api.jar
activation.jar
stax-api.jar

We will ignore WADL for now, which limits the required jars to just the first five. But in addition to the binaries, I prefer to have the sources and javadoc available, which makes the total number of jars 15 (a binary, source, and javadoc jar for each).

Once you have the JARs, configure your classpath or your IDE to include them in your project. You're then ready to write the obligatory 'hello world' program.

[Next: JAX-RS Hello World]

Exploring RESTful web services with JAX-RS and Jersey.

2009-05-27T13:29:00.006-05:00

I've been reading about RESTful web services for a while now. The best book on the subject I've found is RESTful Web Services by Leonard Richardson and Sam Ruby. I've been reading this book for what seems like forever (a function of my life and available time, not the book). I'm getting close to done and have begun using Jersey, the reference implementation of the JAX-RS specification to experiment with building RESTful web services.

I really like Jersey. A big reason for that is because Jersey makes it straightforward to stand up a standalone RESTful service. And once you've done it once, standing up additional services is easy. Deploying services without a heavyweight container makes me happy. Containers were created in the early days of Java when Java's limitations made them necessary. Since then, they have taken on a life of their own. Yet, in today's world I think the container has outlived almost all of its purpose. It's disadvantages now outweigh the few remaining benefits.

In the coming months, I plan to explore how to build, deploy, replicate, secure, and evolve industrial-class services without a container. I think I will succeed, but who knows? Maybe I'll learn a bunch due to abject failures. Either way, it should be fun.

My next post will describe how you download all the various pieces needed to stand up a Jersey-based REST service. There aren't very many, and it doesn't take long. Stay tuned.

[Next: Setting up Jersey for Use]

Everybody Needs Cron

2008-11-12T08:39:00.003-06:00

I was looking at one of my daily emails from javablogs.com when I saw one entitled "Nobody should need cron". This struck me as rather naive, so I went over to read Odd Thesis's entry: Nobody Should Need Cron, hoping to learn what I've been doing wrong all these years.

I posted a comment on the blog which hasn't shown up yet. I'll recreate it here.

Trying to recreate cron in Java/Ruby is asking for trouble. Any real-world application will have activities which are hard or impossible to implement in Java or Ruby. They will be OS-specific, or low-level, or be needed to monitor the very container you are trying to run them from. They often need to be extremely robust (more robust than Java apps can be) in the face of very unusual conditions on the box (huge load, almost zero memory available, etc). Cron and OS-specific tools are well-suited to these situations. Java and Ruby are not.

I'm a software engineer who has been working with Java since 1.0 beta, and I find it a great development language (I'm less experienced with Ruby but it is nice also). But they are simply inappropriate replacements for cron.

I understand the desire to stay in the productive, known world of Java or Ruby. And for periodic tasks "within" the application, some sort of cron-like facility is a nice feature. But robust software systems that are resilient and handle 'unexpected' (i.e. difficult to handle) situations without simply falling over require developers to step outside the virtual machine and deal with the OS and hardware (and databases and filesystems and other software systems) that the virtual machine depends upon. For those watchdog or maintenance activities that have to occur outside the virtual machine, cron (or it's OS-specific equivalent) are indespensible.

If you really think you can build a non-trivial system without cron, you've failed to identify or are ignoring requirements which you must satisfy to make your system robust.

Interface Chaining

2008-09-11T19:00:00.006-05:00

I was recently talking with my friend Mark Volkmann about an application framework he was developing. The framework, named WAX, is a very nice Java API for writing out XML. Despite it's small size, it's much nicer than any other XML writing API I've ever seen and you should go check it out.

As part of the API, Mark implementing method chaining. This allows for very compact code that is still quite easy to read. For example, you could produce this XML:


<?xml version="1.0" encoding="UTF-8"?>
<car year="2008">
  <model>Prius</model>
</car>

with just this code::


WAX wax = new WAX(WAX.Version.V1_0);
    wax.start("car").attr("year", 2008)
        .start("model" ).text( "Prius").close();

Much of the compactness comes from the fact that each method called returns the WAX object, allowing the developer to chain method calls together.

As we were discussing the code, the topic of incorrect ordering of calls came up. For example, what happens if you try to call the attr method to create an attribute after already creating some text inside the element with the text method?

Because WAX is a streaming API, you can't 'go back' and add an attribute in the start element tag after having already 'moved on' to putting text in the body of the element. For example, what if I tried to create an attribute to the model of the car after putting text in it, like this:


WAX wax = new WAX(WAX.Version.V1_0);
    wax.start("car").attr("year", 2008)
        .start("model" ).text( "Prius").attr( "trim", "GT" ).close();

WAX detects this situation and throws an exception at runtime, which it should do. But I'm a fan of static type checking. I love having tools do as much checking for me as possible. So I suggested to Mark that he create a set of interfaces. Each interface would contain only the methods that make sense at the current point in the output stream. This would allow the compiler to catch invalid method invocations. Even better, modern IDEs like IDEA would flag the method invocation visually even before compilation.

For example, if we arrange for the text method to return an interface which does not have an attr method, then the compiler will catch it and the IDE will flag the error visually. For example, in IDEA, the error is flagged like this:

Now we get the compactness of runtime exceptions (which Mark really likes) and static checking (which I really like). I did some looking around and couldn't find a design pattern that really fits. A number of people suggested that this was an example of a domain-specific language (DSL). But to my thinking, something that deserves the moniker of 'language' should be more involved than simply constructing some interfaces so that static checking is applied.

But whatever you call it, I think it's a useful design pattern than can be applied in other situations to make for clean, compact, yet statically checked APIs.

Create inbox items in OmniFocus from the dashboard

2008-07-17T12:31:00.014-05:00

I am a very heavy user of OmniFocus. It has many ways to enter information. For example, you can use the "Quick Entry Window", which is great for creating an item with a project, context, and optional description.

But sometimes I am not in OmniFocus and don't have any text selected. This means there is nothing for OmniFocus to treat as a clipping. As a result it won't bring up the quick entry window.

So, I wrote my first Dashboard widget. It has a text area where you type the text for your item and a "Send It" button. When you click the button or hit enter in the text area, a new item is sent to your inbox.

I submittted it to Apple for inclusion in their download area for dashboard widgets download site. Until it shows up there, you can get it directly from here.

[Updated 2008-07-17 21:05]

Someone pointed out that I had forgotten that there is a separate keystroke to bring up the quick entry window without having any text selected. This eliminates much of the purpose of this widget unfortunately, unless you spend a lot of time in the dashboard anyway. At least it was a good learning exercise... :-)

Groovy Type Handling is Broken

2008-05-05T14:20:00.004-05:00

Here's a very simple Java program snippet:


List<Integer> list = new ArrayList<Integer>();
Collection<Integer> col = list;
list.add( 2 );
list.add( 1 );
list.add( 0 );
System.out.println( list );
col.remove( 0 );

Run this and you'll get what you'd expect. Before using the col reference to remove the integer/Integer with value 0 (zero), you see:


[2, 1, 0]

Then, after calling remove on the collection, we see the integer/Integer with value zero is gone (that's the one we said to remove):


[2, 1]

Groovy says that it allows you to use types and will honor them at runtime. But it doesn't.

Take the same snippet of code and run it as a Groovy script. Remember that the Collection interface has a remove method, and that method expects you to pass it an object which it will remove from the collection. So we would expect to see the same result as in Java. But we don't. Instead, we get this:


[2, 1, 0]
[1, 0]

Groovy has removed the Integer with a value of 2! It did this because of Groovy's 'multi-methods'. Groovy looked at the real type of col, which is an ArrayList. It then found a remove method that takes an integer/Integer. That method removes the Nth item from the list. It used that method and removed the zero-th item (the one with a value of 2).

But remember, I invoked the remove. method through the Collection interface. The collection interface does not have a method taking an integer/Integer but it does have a remove method taking the object to be removed!

This is a truly awful result. It removes our ability to reason about how code will behave by looking at the types involved. In order to know how this will behave, I cannot make any use of the type information; in fact the type information is downright misleading. Even worse, as the type system evolves and new classes are introduced, new methods with entirely different signatures may get invoked.

This approach to handling types is essentially broken. The claim that Groovy honors types (even the weakened form of honoring it by doing casts at runtime) is flat out wrong.

Groovy needs to either start honoring type information, or get rid of it.

Insidious Coupling

2008-04-24T16:08:00.004-05:00

This will get to the topic of code coupling soon enough, but bear with me for a bit of background.

I like to use the 'fail fast' approach to my code. This means that invalid inputs cause a failure in the code as quickly as possible. Perhaps the simplest example of this in the JDK is the collections classes, whose iterators fail as soon as they detect that the underlying collection object has been modified. This allows them to avoid synchronization costs while also detecting unsafe usage.

In my own code, the most common situation where I find fast-fail code is in constructors. If an argument the constructor cannot be null (either because it will be used in the constructor, saved to a field, or both), I check that the object reference is not null. For example:


public class Foo {
 // ...
 public Foo( Bar b ) {
  // Check that b is not null
  // rest of constructor...
 }
}

I started out performing this check with an if statement:


public class Foo {
 // ...
 public Foo( final Bar b ) {
  if ( b == null ) {
   throw new IllegalArgumentException( "Bar b cannot be null" );
  }
  // rest of constructor...
 }
}

This gets us a pretty understandable stack trace:


Exception in thread "main" java.lang.IllegalArgumentException: Bar b cannot be null
 at org.sandbox.Foo.(Foo.java:10)
 at org.sandbox.SimpleDoesStuff.doIt(SimpleDoesStuff.java:11)
 at org.sandbox.Baz.main(Baz.java:10)

But this Java code is pretty verbose, especially if you have multiple arguments to check. After that I tried creating a static method named assureNotNull which would perform the check:


public class Foo {
 // ...
 public Foo( final Bar b ) {
  assureNotNull( b );
  // rest of constructor...
 }
 //...
 private static void assureNotNull( final Object o ) throws IllegalArgumentException {
  if ( o == null ) {
   throw new IllegalArgumentException( "Object cannot be null" );
  }
 }
}

This has the unfortunate consequence that it becomes harder to track down the nature of the problem.


Exception in thread "main" java.lang.IllegalArgumentException: Object cannot be null
 at org.sandbox.Foo.assureNotNull(Foo.java:14)
 at org.sandbox.Foo.(Foo.java:9)
 at org.sandbox.SimpleDoesStuff.doIt(SimpleDoesStuff.java:11)
 at org.sandbox.Baz.main(Baz.java:10)

The stack trace on the exception shows assureNotNull as the source of the exception, which while strictly true doesn't clearly show that the real problem was with someone passing a null Bar to the Foo constructor. This can be mitigated to some degree by taking an description as argument:


public class Foo {
 // ...
 public Foo( final Bar b ) {
  assureNotNull( b, "Bar b cannot be null" );
  // rest of constructor...
 }
 //...
 private static void assureNotNull( final Object o, final String msg ) throws IllegalArgumentException {
  if ( o == null ) {
   throw new IllegalArgumentException( msg );
  }
 }
}

This provides a somewhat more helpful exception message but the method generating the exception is still 'off by one':


Exception in thread "main" java.lang.IllegalArgumentException: Bar b cannot be null
 at org.sandbox.Foo.assureNotNull(Foo.java:14)
 at org.sandbox.Foo.(Foo.java:9)
 at org.sandbox.SimpleDoesStuff.doIt(SimpleDoesStuff.java:11)
 at org.sandbox.Baz.main(Baz.java:10)

And even with this semi-fix to help find the cause of the problem, we're still left with recreating this method in every class. From here the logical next step is to centralize the method. We can do this with an Assertions class:


public class Assertions {
 // ...
 public  static void assureNotNull( final Object o, final String msg ) throws IllegalArgumentException {
  if ( o == null ) {
   throw new IllegalArgumentException( msg );
  }
 }
 // ...
}

Now, our Foo constructor becomes simpler:


import org.acme.Assertions;

public class Foo {
 // ...
 public Foo( final Bar b ) {
  Assertions.assureNotNull( b, "Bar b cannot be null" );
  // rest of constructor...
 }
 //...
}

And with Java 5's static imports, we can even simplify it further:


import static org.acme.Assertions.assureNotNull;

public class Foo {
 // ...
 public Foo( final Bar b ) {
  assureNotNull( b, "Bar b cannot be null" );
  // rest of constructor...
 }
 //...
}

But we still have the wrong method as the first listed in the stack trace:


Exception in thread "main" java.lang.IllegalArgumentException: Bar b cannot be null
 at org.acme.Assertions.assureNotNull(Assertions.java:9)
 at org.sandbox.Foo.(Foo.java:7)
 at org.sandbox.SimpleDoesStuff.doIt(SimpleDoesStuff.java:11)
 at org.sandbox.Baz.main(Baz.java:10)

Since all of us Java developers are big on frameworks, we might even take it to the 'next level'. Why not centralize the Assertions in some common framework? If the Assertions class were in a common framework, the thinking goes, then we'd all be able to use it everywhere. That consistency would really help us read unfamiliar code.

Ignoring the issue of getting any set of developers of size greater than one to agree on anything, especially a basic assertions framework used everywhere, we've just introduced insidious code coupling where it really doesn't belong. In our desire to achieve clarity, brevity, and avoid redundancy, we've coupled code together. If we have a common class in our framework, all the classes in the framework are now coupled to it and indirectly to each other. If we move the class to a generic framework, we externalize the coupling, making it even broader. This coupling introduces many hazards for versioning and incompatibility problems that come with the independent evolution of code.

So, what choice are we left with? Well, we could always depend upon the runtime itself to generate an exception for us. We can do this quite briefly, with less redundancy of code than the other approaches, and with zero coupling:


public class Foo {
 // ...
 public Foo(Bar b) {
  b.getClass(); // Assure not null
 }
}

This code is quite short, and generates a NullPointerException with the top element pointing to exactly the line of code where the problem occurred:


Exception in thread "main" java.lang.NullPointerException
 at org.sandbox.Foo.(Foo.java:7)
 at org.sandbox.SimpleDoesStuff.doIt(SimpleDoesStuff.java:11)
 at org.sandbox.Baz.main(Baz.java:10)

This code also avoids nearly all redundancy, and I think after seeing it once it becomes quite clear. As a bonus, it's easy to create a macro in any modern IDE to help in generating the code.

Sometimes, the right solution to a problem is not abstracting code into a separate method/class/framework.

Something else Apple Gets Right

2008-03-07T18:53:00.005-06:00

Eric Burke made a reference to the work Apple puts into making their applications simple but capable. He also gave Google as an example. That item produced quite a few comments and Eric followed up with more details. Again, the discussion wasn't just about Apple but the company was again an example of hard work making things simple and functional. I agree with his comments, and remembered another example of Apple's efforts toward elegant designs.

For as many years as I can remember, the Apple website has had simple URLs to get to various parts of the website. For example:

http://www.apple.com/developer - the developer site

http://www.apple.com/mac - information about Macs

http://www.apple.com/iphone - information about the iPhone

http://www.apple.com/ipod - information about the iPod

http://www.apple.com/store - the online Apple store and information about the retail stores

Yes, other sites do this now. But Apple was doing this many years ago, long before competitor's sites.

This seemingly simple idea makes it much easier to navigate their web site for the vast majority of people going to the site. It allows people to get more quickly to the area they want. Without this, the task becomes much more involved:

Visit the home page

Parse the layout

Find the navigation links

Find the right link - if it's not there guess where to drill down

Get to where you wanted to go

An alternate syntax for Java closures

2008-02-13T10:42:00.004-06:00

I'm no expert on the details of the closures proposals. I do think if we're going to put closures into the language, we need to make them complete, full-fledged closures. Generics are a mess precisely because the semantics and features were compromised for the goal of getting it into the language.

But one thing I do know about the BGGA closures proposal is that I don't like the syntax. Take the example in Weiqi Gao's recent Java Quiz:


public class Fib {
  private static {int=>int} fib = { int n =>
      n==0 ? 0 :
      n==1 ? 1 :
      fib.invoke(n-1) + fib.invoke(n-2)
  };

  public static void main(String[] args) {
    System.out.println(fib.invoke(6));
  }
}

To me, this is just hard to read. The whole "=>" thing seems like a weak attempt to create a new lexical character and is too visually similar to ==, >>, etc. I also don't like the 'invoke' syntax. I was discussing this with Weiqi the other day and came up with a syntax that I like better.


public class Fib {
  private static {int # int} fib = { # int n #
      n==0 ? 0 :
      n==1 ? 1 :
      #fib(n-1) + #fib(n-2)
  };

  public static void main(String[] args) {
    System.out.println(#fib(6));
  }
}

Note that you could also invoke fib like this:


System.out.println(#Fib.fib(6));

I wonder what other people think of this alternative?

The Wall of Erasure

2007-11-12T22:08:00.000-06:00

I mentioned the term "Wall of Erasure" in my previous post about Iterator adapters. My friend Weiqi Gao described it best in his blog entry where he said 'Every really cool idea involving Java generics will invariably lead to insurmountable difficulties the root cause of which is erasure'.

I just got a chance to read Eric Burke's latest blog entries, and he too has hit the Wall of Erasure (some commentors are suggesting it's not "all" erasure's fault, but that's a red herring).

Unit Testing Simple Code

2007-11-05T08:56:00.001-06:00

I just read Alex Miller's blog entry where he states quite nicely why unit testing even very simple code is not dumb. He posted because of Eric Burke's excellent discussion of the matter.

I really don't have much more to add, except to add one more voice to those who state that unit testing seemingly trivial code is important. I will say that I've actually caught silly mistakes (like failing to set a field to the correct value in a constructor) with simple unit tests. Also, writing tests for such code lets me spend more time thinking about the hard parts of the problem and less time examining 'simple' code to make sure it is correct.

Java 5+ Iterator Adapters

2007-11-04T21:28:00.001-06:00

I've been working with Java Generics more lately, and have run smack into the "Wall of Erasure". I'm referring to the fact that generics disappear in the bytecode. As you probably know, generics were a targeted feature at a time when Sun was attempting to avoid changing the Java Virtual Machine (JVM). They really wanted to maintain backward compatibility. So, this compromise was put forth to maintain backward compatibility. In a classic case of irony, other changes forced a change the JVM in the same release that gave us generices, but the compromised version of generics stayed. So we are stuck with half an implementation of generics.

Recently, I've been doing work with interfaces and extending interfaces, as in:


public interface Superclass {
    // ...
}

public interface Subclass extends Superclass {
    // ...
}

I have classes which implement a third interface:


public interface DoesStuff {
    Iterator<Superclass> iterator();
    // ...
}

You would think that an implementation of DoesStuff could return an Iterator for the iterator method:


public class MyClass implements DoesStuff {
    // ...
    private Set<Subclass> myThings = new HashSet<Subclass>();
    // ...
    public Iterator<Superclass> iterator() {
        return myThings.iterator();
    }
    // ...
}

Unfortunately, the Wall of Erasure means we can't do this. Try it for yourself if you don't believe me.

So, what to do? Happily, there is a simple answer. We create an adapter that adapts the subclass iterator to the superclass iterator:


/**
 * Adapts a derived class iterator to a base class.
 */
public class IteratorAdapter<B, D extends B> implements Iterator<B> {
    private Iterator<D> d;

    public IteratorAdapter(Iterator<D> aD) {
        d = aD;
    }

    public boolean hasNext() {
        return d.hasNext();
    }

    public B next() {
        return d.next();
    }

    public void remove() {
        d.remove();
    }
}

The sad thing is that this shouldn't be necessary. It's trivial. It shouldn't have to be written by developers. The simplicity of this is a clear sign that we need to tear down the Wall of Erasure.

Will it happen any time soon? I hope so. Though I'm hearing talk that "erasing erasure" is probably off the table for Java 7. That would be a truly sad thing.

Making BufferedReader Iterable

2007-10-26T09:44:00.000-05:00

I was reading recently about a issues around reading an entire file into a string. I pretty much never do that, since I like my code to be robust even in the face of very large files. But it did get me thinking about reading in lines of a file one at a time. I do have reason to do that on occasion.

Originally, I might have done it something like this:


File file = ...
BufferedReader br = new BufferedReader(new FileReader(file));
String line = br.readLine();
while ( line != null ) {
    // Do something with line
    line = br.readLine();
}

That's not bad, but I've been working with Ruby lately and really like the terse yet easy-to-read nature of the code. We can improve the above code with a for loop:


File file = ...
BufferedReader br = new BufferedReader(new FileReader(file));
for (String line = br.readLine(); line != null; line = br.readLine()) {
    // Do something with line
}

Better, but still not great. Now, what if the BufferedReader were Iterable? It isn't, but we can wrap an Iterable around it. First, let's look at usage:


File file = ...
BufferedReaderIterable bri = new BufferedReaderIterable(new BufferedReader(new FileReader(file)));
for (String read : bri) {
    // Do something with the line
}

Very nice. So, all we need is to implement BufferedReaderIterable. Happily, it's quite simple:


public class BufferedReaderIterable implements Iterable<String> {

    private Iterator<String> i;

    public BufferedReaderIterable( BufferedReader br ) {
        i = new BufferedReaderIterator( br );
    }
    public Iterator iterator() {
        return i;
    }

    private class BufferedReaderIterator implements Iterator<String> {
        private BufferedReader br;
        private java.lang.String line;

        public BufferedReaderIterator( BufferedReader aBR ) {
            (br = aBR).getClass();
            advance();
        }

        public boolean hasNext() {
            return line != null;
        }

        public String next() {
            String retval = line;
            advance();
            return retval;
        }

        public void remove() {
            throw new UnsupportedOperationException("Remove not supported on BufferedReader iteration.");
        }

        private void advance() {
            try {
                line = br.readLine();
            }
            catch (IOException e) { /* TODO */}
        }
    }
}

It would be nice if we could make the construction cleaner. What if BufferedReaderIterable had a constructor that took a File?



public class BufferedReaderIterable implements Iterable<String> {

    private BufferedReader mine;
    private Iterator<String> i;

    public BufferedReaderIterable( BufferedReader br ) {
        i = new BufferedReaderIterator( br );
    }

    public BufferedReaderIterable( File f ) throws FileNotFoundException {
        mine = new BufferedReader( new FileReader( f ) );
        i = new BufferedReaderIterator( mine );
    }
    public Iterator<String> iterator() {
        return i;
    }

    private class BufferedReaderIterator implements Iterator<String> {
        private BufferedReader br;
        private String line;

        public BufferedReaderIterator( BufferedReader aBR ) {
            (br = aBR).getClass();
            advance();
        }

        public boolean hasNext() {
            return line != null;
        }

        public String next() {
            String retval = line;
            advance();
            return retval;
        }

        public void remove() {
            throw new UnsupportedOperationException("Remove not supported on BufferedReader iteration.");
        }

        private void advance() {
            try {
                line = br.readLine();
            }
            catch (IOException e) { /* TODO */}
            if ( line == null && mine != null ) {
                try {
                    mine.close();
                }
                catch (IOException e) { /* Ignore - probably should log an error */ }
                mine = null;
            }
        }
    }
}

Now, usage is even cleaner:


File file = ...
BufferedReaderIterable bri = new BufferedReaderIterable(file);
for (String read : bri) {
    // Do something with the line
}

[Editorial Note: Updated to put the template angle brackets back in after Eric Burke noticed Blogger was eating them]

Eric Burke suppressing a desire to get a Mac?

2007-09-18T12:05:00.000-05:00

My friend and colleague, Eric Burke, recently altered his blog to show the date of the posting with a calendar 'icon' rendered by a CSS style-sheet. While I appreciate his frustration with the inconsistent handling of CSS by different browsers, that is not what caught my eye. Here is his graphic:

And here is the icon for iCal (the calendar program on Mac OS X):

What do you think? Is he is suppressing a desire to get a Mac?

Gilstrap Estimation Curves

2007-08-08T23:10:00.001-05:00

I am constantly producing estimates. How long will this project take? How long will that feature take? How long will it take us to test things? And my favorite: How long will it take to fix the bugs we haven't found?

Other than the last, nonsense question, these are legitimate issues to address in the process of software development. Unfortunately, our industry (and some others) tends to be very simplistic in their estimates.

There are many ways to go wrong in estimating. Perhaps the the most common is attempting to pretend that people are 'resources' and that everyone is interchangeable. But that's a topic for another day. Today, I want to look at a way of estimating that is simple enough to use yet isn't one dimensional.

One of the biggest problems with estimating is trying to incorporate some notion of risk in the estimate. Is your estimate iron-clad, where you are 99.99999% sure you'll meet the deadline (and consequently there's a chance you can beat it), or will you be working flat-out trying to meet the date with only a limited chance of making it? This is one of the most common problems in projects: one party gives an aggressive estimate and another party assumes it's conservative. We need a better way to communicate the likelihood of completing a task in a given amount of time.

To capture this idea, we need to represent the work with more than a single number, and we need to incorporate risk. Happily, this lends itself to a two-dimensional graph, with effort (time) as the x-axis and likelihood of completion (what I'll call "confidence") as the y-axis, like this:

The line represents your confidence, at the time you make the estimate, that you will solve the problem with the effort specified. Depending on the scope of the work, the effort line might be measured in hours, days, weeks, months, or even years. The basic idea is that it will take some period of time to understand the unknowns of a problem, which is the early, low-confidence part of the curve. Once the basic problem is understood work proceeds apace, and confidence goes up rapidly. The tail of the curve represents the fact that things might go well and we could get done earlier than expected. Let's take an example.

Assume you are asked to implement a new kind of cache for your system. It involves a "least recently used" (LRU) strategy along with a maximum age for cache entries. Instead of saying "it'll take 10 man-days of effort", as if it can not take only 9 or would never take 11, you can graph it, like this:

This says that you think there's about a 2% chance of completing the work in 1 man-day, a 20% chance in four man-days, a 90% chance you'll get done in 8 days, etc. You are not saying you'll be done in a specific number of man-days. Instead, you provide an estimate of the likelihood you'll complete the work in a particular number of man-days. This is useful because it helps provide some insight into the overall risk you see in the work.

This is not to say that all problems follow this curve. Some may in fact be so difficult that they never approach 100%. For example, if you thought there was only a 50% chance you would actually be able to come up with a solution to a problem, you might have a graph like this:

Similarly, if you think the problem is well-understood, your graph might look like this:

At first this might seem odd. But if you really understand the problem you'll be able to estimate the required effort very accurately. This means there's very little chance of getting done faster than your estimate, but it also means there's very little chance you'll run long.

Perhaps this is a well-known thing in project management circles, but I've never seen anything like it in over twenty years in industry. I plan to use these estimation curves to better communicate my estimates to others.

RemoteIterator, episode 2

2007-07-31T10:31:00.000-05:00

Last time, I lamented the lack of a standard RemoteIterator interface in Java. I went on to show a version that showed up not long ago in Jackrabbit (an implementation of JSR 170) and the version I've recreated several times in the past. Let's take a quick look at them again.

Jackrabbit's:


public interface RemoteIterator extends Remote {

    long getSize() throws RemoteException;

    void skip(long items) throws NoSuchElementException, RemoteException;

    Object[] nextObjects() throws IllegalArgumentException, RemoteException;

}

The RemoteIterator interface I've recreated a number of times over the years:


public interface RemoteIterator<T> extends Remote {

    public boolean hasMore() throws RemoteException;

    public List<T> next( int preferredBatchSize ) throws RemoteException;

}

Let's go through the methods individually:

long getSize() throws RemoteException

This seems problematic to me. Perhaps in Jackrabbit, it is reasonable to assume that you will always know the size of the collection to be iterated. But in many situations, this is not the case. For example, if the iterator is ultimately coming from a database cursor and the collection is large, we want to avoid having to run two SQL queries or store the entire collection in memory (which may be prohibitively expensive). I can see a derived interface, perhaps named BoundedRemoteIterator or something, but not in this interface.

void skip(long items) throws NoSuchElementException, RemoteException

I love this idea. Wish I'd thought of it myself. A trivial implementation would do whatever next( items ), would do and simply not return the items. A smart implementation can save time and space by not actually generating the objects (since they won't be returned). The only change I would make is to have a return value of the number of items skipped and get rid of the NoSuchElementException. If I ask to skip 50 items and only 29 more exist, I would get a return of 29 and no exception. Admittedly, if the interface included a getSize operation, throwing NoSuchElementException is less strange. But even in that case, it seems too rigid to consider skipping more items than are left an 'exception'.

boolean hasMore() throws RemoteException

At first, you might think this is a wasteful operation. Why make a call to find out if there are more items? This results in 'extra' remote calls instead of simply asking for the next item. However, if we imagine the items themselves as quite large (e.g. multi-megapixel images), we start to see a use for this API. If we were keeping the getSize operation, this might not be necessary, but again, getSize() requires state tracking on the client side that can substantially complicate the client code.

Object[] nextObjects() throws IllegalArgumentException, RemoteException

This is similar to my next operation discussed below, except (1) it throws IllegalArgumentException, (2) returns an array of objects instead of a List of a specific type, and (3) does not take a preferred size.

(1) I'm guessing that the getSize operation is intended to be called and then the caller is expected to know when all items in the iterator are consumed. This creates a problem if the client consumes different parts of the iterator in different parts of the code. Now, they have to pass around 2 or 3 items (remote iterator, number of items consumed so far, and possibly the total size) to do this. This is clunky to say the least.

(2) Jackrabbit may have been specified before 1.5 was in wide use, or may feel the need for backward compatibility, hence the use of an array of Object. The downside to this is the loss of type-safety and loss of an API that can be optimized. See the discussion of next below.

(3) See the discussion of next below for the rationale for a preferred batch size.

List<T> next( int preferredBatchSize ) throws RemoteException

By returning a List<T>, we not only introduce type information/safety, we also allow for further optimization. The List returned could be a simple implementation such as LinkedList or ArrayList. Or, it could be a smart List that collaborates with the RemoteIterator to delay retrieving its contents or retrieve the contents in separate threads. There are lots of possibilities introduced by avoiding raw Java arrays.

Allowing the client to pass a preferred batch size provides many optimization opportunities. For example, if displaying the items in a GUI, they can ask for 'page-sized' numbers of items. By making the preferred size a suggestion, it allows an implementation leeway to ignore requests that are deemed unmanageably small or large for the kind of information being returned.

So, where does this leave us? Our RemoteIterator now looks like this:


import java.rmi.Remote;
import java.rmi.RemoteException;
import java.util.List;

/**
 * Interface for a remote iterator that returns entries of a particular type.
 * @see java.util.Iterator
 * @see java.rmi.Remote
 */
public interface RemoteIterator<T> extends Remote {

    /**
     * Determine if there are more items to be iterated across
     * @return true if there are more items to be
     * iterated, false otherwise.
     * @throws RemoteException if a problem occurs with
     * communication or on the remote server.
     */
    boolean hasMore() throws RemoteException;

    /**
     * Skip some number of items.
     * @param items the number of items to skip at maximum. If there are fewer
     * items than this left, all remaining items are skipped.
     * @return the number of itmes actually skipped. This will equal <code>items</code>
     * unless there were not that many items left in the iteration.
     * @throws RemoteException if a problem occurs with
     * communication or on the remote server.
     */
    int skip(long items) throws RemoteException;

    /**
     * Get some number of items
     * @param preferredBatchSize a suggested number of items to return. The implementation
     * is not required to honor this request if it will prove too difficult.
     * @return a List of the next items in the iteration, in iteration order.
     * @throws RemoteException if a problem occurs with
     * communication or on the remote server.
     */
    List<T> next( int preferredBatchSize ) throws RemoteException;

}

I'm still wondering why this isn't part of the JDK.

RemoteIterator, where art thou?

2007-07-19T21:56:00.000-05:00

I've written my fair share of remote iterators over the years. The concept is simple enough: provide an interface for iterating over a set of items that come from a remote source. The harder part is making a good abstraction that can still perform well.

Even so, it surprises me that, as far as I can tell, there hasn't been a standardized form of this. There's been discussion for a long time, with my earliest memories stemming from the JINI mailing lists back in 2001. Yet there's nothing in the core as of Java 6. There are random postings on various groups on the subject through the years, but no actual API that I can find.

Recently, I was teaching my course in Java performance tuning and discussing remote iterators, when a student mentioned that Jackrabbit (an implementation of JSR 170) has an RMI remoting API called Jackrabbit JCR-RMI which includes a RemoteIterator. As of version 1.0.1, it looks like this:


public interface RemoteIterator extends Remote {

    long getSize() throws RemoteException;

    void skip(long items) throws NoSuchElementException, RemoteException;

    Object[] nextObjects() throws IllegalArgumentException, RemoteException;

}

Here is the RemoteIterator interface I've recreated a number of times over the years:


public interface RemoteIterator extends Remote {

    public boolean hasMore() throws RemoteException;

    public List next( int preferredBatchSize ) throws RemoteException;

}

These two APIs are strikingly similar. They both extend Remote, they both provide a means to get some number of the next items to be retrieved (mine with generics and theirs without), and they are both quite small. Yet they are also strikingly different: JackRabbit's API has both getSize and skip while mine doesn't.

Next time, I'll describe what I see as the major differences and what I like about each. I'll also compare the client-side front-end for remote iterators from Jackrabbit and the one I've created. In the mean time, consider it a challenge to comment about everything I was going to bring up and more.

Random Acts of Blog

2007-07-02T15:20:00.000-05:00

I've been subscribed to Brian Coyner's blog for some time. I had this URL:

feed://beanman.wordpress.com/atom.xml

Recently, I was quite surprised when Brian seemed to stop blogging about software and began blogging about bands I had no idea he liked. But I chalked it up to a sudden burst of music interest.

Then, today, I tried to go to his blog and got this:

It turns out to be a blog feed for People Magazine's StyleWatch. WTF? A bit of digging and it would seem that the feed above no longer works. But instead of giving me an error, I'm directed to a random wordpress.com blog page instead. Talk about non-intuitive!

Freaky....

Friday Java Quiz

2007-06-28T22:09:00.001-05:00

My friend, Weiqi Gao, frequently posts a Friday Java Quiz . I ran into a situation today that seems like a good one.

Suppose you have the following two classes:


public class Test {
    public static void main(String[] args) {
        System.err.println( Foo.class.getName() );
        System.err.println( "Testing, 1, 2, 3..." );
        new Foo();
    }
}


public class Foo {
    static {
        System.err.println( "Foo here." );
    }
    public Foo() {
        System.err.println( "New Foo!" );
    }
}

Without running this program, do you know what the output will be?

Kyle Cordes on tools

2007-05-17T10:10:00.000-05:00

Kyle Cordes wants people to view a presentation by Linus Torvalds discussing distributed version control tools. I plan to watch the talk soon, but a comment Kyle made in his blog entry really rang true. It's something I've been saying (less eloquently) for a long time:

"I have heard it said that these are all 'just tools' which don’t matter, you simply use whatever the local management felt like buying. That is wrong: making better tool choices will make your project better (cheaper, faster, more fun, etc.), making worse tool choices will make your project worse (more expensive, slower, painful, higher turnover, etc.)"

I've been tagged

2007-04-06T21:10:00.000-05:00

There is apparently a blog-tag game going around, and Jeff Brown tagged me.

I practice Yang-style short form T'ai Chi Ch'uan.

I've been having fun lately making home-made no knead bread.

I enjoy woodworking. I'm currently tuning my latest large tool purchase. It took me about a year to save enough to get my bandsaw. I even build things.

I've used a Mac since the original 128k version introduced in 1984.

Perhaps because I didn't drink much as a teenager, I never developed a taste for beer. I do enjoy single malt scotch.

Now I'm tagging Weiqi, Brad, Dave, Brian, and Melben.

Too much C++

2007-03-20T00:03:00.000-05:00

(with apologies to Monty Python)

A customer enters a software shop.

Mr. Praline: 'Ello, I wish to register a complaint.

The owner does not respond.

Mr. Praline: 'Ello, Miss?

Owner: What do you mean "miss"?

Mr. Praline: I'm sorry, I have a cold. I wish to make a complaint!

Owner: We're closin' for lunch.

Mr. Praline: Never mind that, my lad. I wish to complain about this programming language what I purchased not half an hour ago from this very boutique.

Owner: Oh yes, the, uh, the Danish C++...What's,uh...What's wrong with it?

Mr. Praline: I'll tell you what's wrong with it, my lad. 'E's crap, that's what's wrong with it!

Owner: No, no, 'e's uh,...it's resting.

Mr. Praline: Look, matey, I know a crap language when I see one, and I'm looking at one right now.

Owner: No no it's not crap, it's, it's restin'! Remarkable language, the Danish C++, idn'it, ay? Beautiful syntax!

Mr. Praline: The syntax don't enter into it. It's stone crap.

Owner: Nononono, no, no! 'E's resting!

Mr. Praline: All right then, if it's restin', I'll wake him up! (shouting at the cage) 'Ello, Mister Plymorphic Language! I've got a lovely fresh computer for you if you show...

owner hits the cage

Owner: There, he moved!

Mr. Praline: No, he didn't, that was you hitting the cage!

Owner: I never!!

Mr. Praline: Yes, you did!

Owner: I never, never did anything...

Mr. Praline: (yelling and hitting the cage repeatedly) 'ELLO POLYMORPH!!!!! Testing! Testing! Testing! Testing! This is your nine o'clock alarm call!

Takes language out of the cage and thumps its head on the counter. Throws it up in the air and watches it plummet to the floor.

Mr. Praline: Now that's what I call a crap language.

Owner: No, no.....No, 'e's stunned!

Mr. Praline: STUNNED?!?

Owner: Yeah! You stunned him, just as he was wakin' up! Danish C++'s stun easily, major.

Mr. Praline: Um...now look...now look, mate, I've definitely 'ad enough of this. That language is definitely deceased, and when I purchased it not 'alf an hour ago, you assured me that its total lack of movement was due to it bein' tired and shagged out following a prolonged standards meeting.

Owner: Well, it's...it's, ah...probably pining for a common runtime.

Mr. Praline: PININ' for a COMMON RUNTIME?!?!?!? What kind of talk is that?, look, why did he fall flat on his back the moment I got 'im home?

Owner: The Danish C++ prefers keepin' on it's back! Remarkable language, id'nit, squire? Lovely syntax!

Mr. Praline: Look, I took the liberty of examining that language when I got it home, and I discovered the only reason that it had been sitting on its perch in the first place was that it had been NAILED there.

pause

Owner: Well, o'course it was nailed there! If I hadn't nailed that language down, it would have nuzzled up to those computers, broke 'em apart with its beak, and VOOM! Feeweeweewee!

Mr. Praline: "VOOM"?!? Mate, this language wouldn't "voom" if you put four million volts through it! 'E's bleedin' demised!

Owner: No no! 'E's pining!

Mr. Praline: 'E's not pinin'! 'E's passed on! This language is no more! He has ceased to be! 'E's expired and gone to meet 'is maker! 'E's a stiff! Bereft of life, 'e rests in peace! If you hadn't nailed 'im to the perch 'e'd be pushing up the daisies! 'Is processes are now 'istory! 'E's off the map! 'E's kicked the bucket, 'e's shuffled off 'is mortal coil, run down the curtain and joined the bleedin' choir invisibile!! THIS IS AN EX-LANGUAGE!!

pause

Owner: Well, I'd better replace it, then. (he takes a quick peek behind the counter) Sorry squire, I've had a look 'round the back of the shop, and uh, we're right out of languages.

Mr. Praline: I see. I see, I get the picture.

Owner: I got Java.

pause

Mr. Praline: Pray, does it support pointers?

Owner: Nnnnot really.

Mr. Praline: WELL IT'S HARDLY A BLOODY REPLACEMENT, IS IT?!!???!!?

Owner: N-no, I guess not. (gets ashamed, looks at his feet)

Mr. Praline: Well.

pause

Owner: (quietly) D'you.... d'you want to come back to my place an' dereference some pointers?

Mr. Praline: (looks around) Yeah, all right, sure.

The ever-leaky abstractions of Microsoft

2007-03-15T14:51:00.000-05:00

I was using Windows XP inside Parallels today and got a message from Windows that I had never seen before:

How on earth could you design an operating system that could detect there wasn't enough virtual memory, expand the virtual memory pool automatically, but would deny applications requests for virtual memory at the same time?

C++ Unit testing frameworks: do they all suck?

2007-03-01T10:57:00.000-06:00

Well, I've been doing some work with C++ lately and I quickly ran into the need for a unit testing framework. I found an excellent article reviewing the state of C++ unit testing frameworks. My first surprise was how many of them there are. This article alone reviewed six of them. Even if those are all of the 'serious contenders', that's about 3 or 4 frameworks too many.

I was also surprised at how primitive the C++ unit testing frameworks feel. I realized after a while that the lack of dynamic/reflective behavior in C++ is part (or even all) of the issue. Since reflection is not built in, many of the grungy tasks that unit testing frameworks in Java automate for us can't be automated in a C++ unit testing framework. What a shame. We're left using frameworks that feel awkard and leave a lot to be desired.

Then I got to thinking: a number of the C++ wizards around here talk about the power of C++ template programming. It seems to me it ought to be possible to clean up and automate what would normally be done in Java with reflection using templates in C++. I wonder why this hasn't been done. I don't know templates well enough to do it myself, but I hope some wiz at C++ templates does this sometime soon.

For now, I'll keep trudging along with CppUnit, and maybe try out CxxTest.

Adding a JNI Library template to Xcode

2007-02-01T18:55:00.000-06:00

I've been doing some work with Xcode, Apple's IDE, recently. I wanted to use it for doing Java JNI work, since I'm doing some of that, but found the normal tutorial assumes you are building an entire application with Xcode. Well, Xcode pales in comparison to IDEA for working with Java.

So I wondered if I could use IDEA for working with the Java code and Xcode for managing the JNI bindings and the C++ code that goes with it. After a bit of playing around (primarily pulling bits and pieces from Apple's mini-tutorial on JNI with Xcode), I figured out how to build a JNI library that I can use easily from IDEA.

That got me to thinking about the four or five steps I would have to do each time I created a new one of these projects (which I expect to be doing for a while). I vaguely remembered a friend talking about customizing Xcode by creating new templates. So I set out to create one for building JNI libraries.

I googled around a bit and found out that Xcode keeps its templates in:


    /Library/Application Support/Apple/Developer Tools

Looking there, I see a folder called "Project Templates", and sure enough inside are all the templates for Xcode projects.

There are several that look promising, but what's the difference between a "BSD Dynamic Library", a "C++ Dynamic Library", and a "C++ Standard Dynamic Library"? Something based on the "C++ Standard Dynamic Library" seems right.

So, I create a test project and make a few changes to produce a JNI library:

Change "Executable Extension" to be "jnilib"
Change "Header Search Paths" to include "$(SDK)/System/Library/Frameworks/JavaVM.framework/Headers"
Delete the Carbon framework (why was it there in the first place?)

Now, save the project, make sure it builds (which it does) and create a simple Java HelloWorld with a native method that calls our Test class.

Having verified that this sort of project can be used to produce a JNI Library, it's time to create the custom project template. First I copy the "C++ Standard Dynamic Library template" to "JNI C++ Standard Dynamic Library". Now I need to find the right places to insert the additional items that were changed (and how to remove the Carbon framework). Looking at the template directory, I see a number of files. But most of these are just boilerplate for source files in the project ('.pch', '.h', '.cp'). I think I can safely ignore these files at least for now (though I made note of them; if there prove to be certain common files in my JNI libraries moving forward, I can come back and try to tweak the project template to have different boilerplates).

The key 'file' appears to be CppShared.xcodeproj. Right-clicking on it shows it is really a 'package' and I take a look inside.

When I do so, I see two files: "TemplateInfo.plist" and "project.pbxproj". The '.plist' file has some interesting items about template files and lists of files upon which to perform macro expansion. Interestingly, the list of files matches the list of boilerplate files almost exactly. For now, I'm leaving these alone (again making note in case I want to change the set of boilerplate files in the project). There is also a Description item, which shows up in the "New Project..." window in Xcode. After tweaking that to make it better describe this new kind of project I'm creating, I move on to the "project.pbxproj" file.

The "project.pbxproj" file is semi-text and semi-binary. So it's time to be careful and make a backup before experimenting. The first section of interest is this:


/* Begin PBXFileReference section */
                08FB77AAFE841565C02AAC07 /* Carbon.framework */ = {isa = ...
                32BAE0B70371A74B00C91783 /* «PROJECTNAME»_Prefix.pch */ = {isa = ...
                50149BD909E781A5002DEE6A /* «PROJECTNAME».h */ = {isa = ...
                5073E0C409E734A800EC74B6 /* «PROJECTNAME».cp */ = {isa = ...
                5073E0C609E734A800EC74B6 /* «PROJECTNAME»Proj.xcconfig */ = {isa = ...
                5073E0C709E734A800EC74B6 /* «PROJECTNAME»Target.xcconfig */ = {isa = ...
                50B2938909F016FC00694E55 /* «PROJECTNAME»Priv.h */ = {isa = ...
                D2AAC09D05546B4700DB518D /* lib«PROJECTNAME».dylib */ = {isa = 
PBXFileReference; explicitFileType = "compiled.mach-o.dylib"; 
includeInIndex = 0; path = "lib«PROJECTNAME».dylib"; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */

I ditch the Carbon framework reference (since I'm doing JNI) by deleting the line that references Carbon. If I want to use Carbon in a JNI library, I can always add it later. It turns out there are a bunch of other references to Carbon in the template, and I have to remove those as well.

Now, I see the line that makes reference to "lib«PROJECTNAME».dylib". I want the library to end with '.jnilib' since that is required by Mac OS X. So I patch this:


                D2AAC09D05546B4700DB518D /* lib«PROJECTNAME».jnilib */ = {isa = 
PBXFileReference; explicitFileType = "compiled.mach-o.dylib"; 
includeInIndex = 0; path = "lib«PROJECTNAME».jnilib"; sourceTree = BUILT_PRODUCTS_DIR; };
/* End PBXFileReference section */

Now, I want the header search paths to include the location of the Java headers, and the executable extension to be '.jnilib'. Searching the test project's '.pbxproj' file, I find them in the Debug and Release build sections. However, there are two pairs of these, one pair for the library target and one pair for the overall project. I need to change the ones for the overall project:


                1DEB916508733D950010E9CD /* Debug */ = {
                        isa = XCBuildConfiguration;
                        baseConfigurationReference = 5073E0C609E734A800EC74B6 /* «PROJECTNAME»Proj.xcconfig */;
                        buildSettings = {
                                COPY_PHASE_STRIP = NO;
                                DEBUG_INFORMATION_FORMAT = dwarf;
                                EXECUTABLE_EXTENSION = jnilib;
                                GCC_ENABLE_FIX_AND_CONTINUE = YES;
                                GCC_OPTIMIZATION_LEVEL = 0;
                                HEADER_SEARCH_PATHS = "$(SDK)/System/Library/Frameworks/JavaVM.framework/Headers";
                                ZERO_LINK = YES;
                        };
                        name = Debug;
                };
                1DEB916608733D950010E9CD /* Release */ = {
                        isa = XCBuildConfiguration;
                        baseConfigurationReference = 5073E0C609E734A800EC74B6 /* «PROJECTNAME»Proj.xcconfig */;
                        buildSettings = {
                                ARCHS = (
                                        ppc,
                                        i386,
                                );
                                DEAD_CODE_STRIPPING = YES;
                                DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
                                EXECUTABLE_EXTENSION = jnilib;
                                HEADER_SEARCH_PATHS = "$(SDK)/System/Library/Frameworks/JavaVM.framework/Headers";
                                PRESERVE_DEAD_CODE_INITS_AND_TERMS = YES;
                                SEPARATE_STRIP = YES;
                        };
                        name = Release;
                };

I was encouraged when creating a new project didn't fail. Looking at the default setup, everything seems to be in order (no Carbon framework, all the boilerplate files are there, etc.). A few minutes to add in the JNI binding function and change the Java program to use the new JNI library and I have a successful test.

I'm sure there are many other customizations I could make if I only knew about them. I also wish there were a better tool than a text editor for editing these files (hopefully there is and someone will point it out to me).

Learning Cocoa

2007-01-21T23:05:00.000-06:00

If you have any interest in the Cocoa frameworks for Mac OS X, you might be interested a blog entry about learning how to program Cocoa. It's on my other blog, Walking the Fringe, because it's more exploratory than the material I like to put on this blog.

Configuring logging in your Spring configuration file

2007-01-19T17:22:00.000-06:00

I detest multiple configuration files. Generally, I can't avoid having them, but whenever I can find a way to reduce the number of configuration files, the better.

So, when I need to tailor the logging levels in my application, I have a choice: edit the default logging file (possibly effecting other applications if that copy of the JDK is shared as is typical), or create a separate logging configuration file and point logging to it (in the case of the JDK's logging, this means setting the java.util.logging.config.file property).

When using Spring, I'd really like to put my custom logging levels in my Spring configuration file, so I don't have yet another file that has to travel with my application. There doesn't appear to be any way built into Spring to do this, but by adding a tiny Java class to the application (one that is reusable for each application), we gain the ability to do this.

The idea is to create a bean at the top of your Spring configuration file that will set your custom logging levels. For example:


    <bean id="loggingConfigurer"
          class="com.acme.logging.LoggingConfigurer"
          lazy-init="false">
        <constructor-arg>
            <props>
                <prop key="org.springframework">WARNING</prop>
                <prop key="org.hibernate">WARNING</prop>
            </props>
        </constructor-arg>
    </bean>

The LoggingConfigurer class is trivial:


package com.acme.logging;

import java.util.Iterator;
import java.util.Properties;
import java.util.logging.Level;
import java.util.logging.Logger;

/**
 * Configures logging by being constructed. 
 * Especially useful when created as a bean from a Spring configuration file.
 */
public class LoggingConfigurer {

  public LoggingConfigurer(Properties props) {
    Iterator i = props.keySet().iterator();
    while (i.hasNext()) {
      String loggerName = (String) i.next();
      String levelName = props.getProperty(loggerName);
      try {
        Level level = Level.parse(levelName);
        Logger l = Logger.getLogger(loggerName);
        l.setLevel(level);
      }
      catch ( IllegalArgumentException e ) {
        System.err.println( "WARNING: Unable to parse '" 
           + levelName + "' as a java.util.Level for logger " 
           + loggerName + "; ignoring..." );
      }
    }
  }
}

Now, when Spring is initialized, very early on in the process, it encounters the entry for our loggingConfigurer bean. Since the bean has lazy initialization turned off, it is immediately created and configured. That sets our logging levels to what we want.

Admittedly, this isn't perfect. For example, these few lines of logging show up when launching a Spring and Hibernate application (prior to our bean turning the logging level to WARNING):


Jan 19, 2007 3:44:41 PM org.springframework.core.CollectionFactory 
INFO: JDK 1.4+ collections available
Jan 19, 2007 3:44:41 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [baseContext.xml]
Jan 19, 2007 3:44:41 PM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
INFO: Loading XML bean definitions from class path resource [unitTestingContext.xml]
Jan 19, 2007 3:44:41 PM org.springframework.context.support.AbstractRefreshableApplicationContext refreshBeanFactory
INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext;hashCode=2490106]: org.springframework.beans.factory.support.DefaultListableBeanFactory defining beans [loggingConfigurer,hibernateStatsService,mbeanServer,exporter,persistence,setPersistenceManager,customEditorConfigurer,mySessionFactory]; root of BeanFactory hierarchy
Jan 19, 2007 3:44:41 PM org.springframework.context.support.AbstractApplicationContext refresh
INFO: 8 beans defined in application context [org.springframework.context.support.ClassPathXmlApplicationContext;hashCode=2490106]
Jan 19, 2007 3:44:41 PM org.springframework.context.support.AbstractApplicationContext initMessageSource
INFO: Unable to locate MessageSource with name 'messageSource': using default [org.springframework.context.support.DelegatingMessageSource@a9a32c]
Jan 19, 2007 3:44:41 PM org.springframework.context.support.AbstractApplicationContext initApplicationEventMulticaster
INFO: Unable to locate ApplicationEventMulticaster with name 'applicationEventMulticaster': using default [org.springframework.context.event.SimpleApplicationEventMulticaster@2153fe]
Jan 19, 2007 3:44:41 PM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
INFO: Pre-instantiating singletons in factory [org.springframework.beans.factory.support.DefaultListableBeanFactory defining beans [loggingConfigurer,hibernateStatsService,mbeanServer,exporter,persistence,setPersistenceManager,customEditorConfigurer,mySessionFactory]; root of BeanFactory hierarchy]

But this is a great improvement over the huge amounts of output from Spring and (especially) Hibernate that show up otherwise.

I'm sure a similar approach can be used with log4j. If anyone adapts this to log4j I'd love to know how that works.

Simplified Database Management in Spring-based Hibernate Unit Tests

2007-01-17T22:39:00.000-06:00

I've been working with Hibernate recently, and needed a way to implement good unit tests. The goal of these unit tests is to make sure the Java code works in conjunction with the Hibernate mapping files. I found an excellent article on using HSQLDB to do unit testing of Hibernate classes on The Server Side.

Unfortunately, one of the drawbacks of the approach described is that it requires you to write some custom JDBC code to clear out your tables after each unit test. From the article:


    statement.executeUpdate("delete from Batting");
    statement.executeUpdate("delete from Fielding");
    statement.executeUpdate("delete from Pitching");
    statement.executeUpdate("delete from Player");
    connection.commit();

Another possible option for handling this scenario if you are using Hibernate 3 is to use it's bulk update feature. I haven't used this feature yet, but apparently one wrinkle is that bulk deletes don't seem to cause cascade deletes. This requires that you remember to update your code each time you add a new table to those used by your application, or end up with problems in your unit tests due to data left over from the last test method.

But in my opinion there's a better way: leverage the SchemaExport utility of Hibernate to automaticallly wipe out and recreate all your tables for you. The basic idea is that the setUp method of your unit test calls the SchemaExport utility to wipe out the existing tables and recreate them. This leaves you with the proper tables and a clean slate upon which to create exactly the objects needed to run your test. This simplifies your unit testing and avoids having to maintain both the mapping files and the bulk delete code in tandem.

To accomplish this, you need to make sure your Hibernate mapping files specify enough information for Hibernate's SchemaExport utility to generate and run database-specific DDL to recreate your tables, indexes, foreign keys, etc. For example, you might have a mapping file for a SimplePerson class that looks like this:


<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD/EN"
        "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="com.acme.project">
    <class name="SimplePerson" table="PERSONS">
        <id name="id" type="int">
            <column name="CODE" not-null="true"/>
            <generator class="native"/>
        </id>
        <property name="firstName" column="FIRST_NAME" not-null="true"/>
        <property name="lastName" column="LAST_NAME" not-null="true"/>
        <property name="contactEmail" column="CONTACT_EMAIL_ADDRESS"/>
        <property name="disabled" type="boolean">
            <column name="DISABLE" default="0" not-null="true"/>
        </property>
    </class>
</hibernate-mapping>

You need to make sure you have information about the SQL types of your mapped columns so Hibernate knows how to generate the tables, like this:


<?xml version="1.0"?>
<!DOCTYPE hibernate-mapping PUBLIC "-//Hibernate/Hibernate Mapping DTD/EN"
        "http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping package="com.acme.project">
    <class name="SimplePerson" table="PERSONS">
        <id name="id" type="int">
            <column name="CODE" sql-type="integer" not-null="true"/>
            <generator class="native"/>
        </id>
        <property name="firstName" column="FIRST_NAME" not-null="true"/>
        <property name="lastName" column="LAST_NAME" not-null="true"/>
        <property name="contactEmail" column="CONTACT_EMAIL_ADDRESS"/>
        <property name="disabled" type="boolean">
            <column name="DISABLE" sql-type="int" default="0" not-null="true"/>
        </property>
    </class>
</hibernate-mapping>

The sql-type attributes are what tell Hibernate how to generate DDL for your table. Once you have this, you can invoke the SchemaExport utility on your in-memory HSQLDB database and it will wipe out the old tables and their contents and recreate them empty.

Now our only catch is invoking SchemaExport when using the Spring framework. If you're using Spring outside of any other container, you might use a LocalSessionFactoryBean to do your Hibernate mapping work, like this:


    <bean id="mySessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
        <property name="mappingResources">
            <list>
                <value>com/acme/project/SimplePerson.hbm.xml</value>
                <!-- ... -->
            </list>
        </property>
        <property name="configLocation" value="hibernateConfig.xml"/>
    </bean>

This is great, except for one thing: you need access to the Hibernate Configuration object in order to invoke SchemaExport and this is not easily available (from what I can find). There is nice workaround for this problem, though. If you use the ampersand ('&') character in front of the name of a bean, you can get at the actual Spring object instead of the bean it creates. For example, the LocalSessionFactoryBean has a public getConfiguration method will get us the Hibernate Configuration object we want. So we can implement a setUp method that contains code like this:


        LocalSessionFactoryBean l = (LocalSessionFactoryBean) factory.getBean("&mySessionFactory");
        Configuration cfg = l.getConfiguration();
        SchemaExport schemaExport = new SchemaExport(cfg);
        schemaExport.create(false, true);

Now, every test method in your unit test will have a clean database with all the right tables, indexes, etc.

One side benefit of this is that there's a good chance you will make the tests run faster than when connecting to an external database. But, most importantly, by avoiding dependencies on an external database you improve the reliability of your Hibernate mapping unit tests.

Hibernate: Simple Classes Aren't So Simple

2007-01-10T14:13:00.000-06:00

I was fighting with Hibernate the other day, trying to figure out why it was giving me a query error. After much too long, I decided to implement unit tests for my measly two classes, which I thought were too simple to really need unit tests.

Almost immediately, I found bugs in one of the two persistent classes I was working with that were confusing Hibernate and leading it to generate the error message that was confusing me.

I checked the Hibernate best practices page and my lesson isn't listed. It may sound obvious, but here it is:

While your classes may be simple, Hibernate is not. Always implement a unit test to verify that the simple parts of your persistent objects are mapped correctly by Hibernate before moving on to more complex issues like inheritance, relationships, containment, etc.

Open Source Router

2006-09-11T14:36:00.000-05:00

As part of a project to move my noisy Power Mac G4 to the basement (see my blog entery Silence is Golden), I recently bought a LinkSys WRT45GL. This is a 4-port ethernet and WiFi (802.11g) router which is hooked up to my DSL line and provides a firewall and port forwarding for the G4. I know that Kevin Heifner got this same router and immediately installed Thibor's HyperWRT version of the firmware. I don't use Vonage, and Skype has been working fine for me, so I didn't go so far as to replace the stock firmware (yet).

This router has the distinction of being built on top of a Linux kernel (interestingly, it was only after someone figured out that the firmware was based upon Linux that the firmware and technical details were released: the wikipedia for the WRT45GL).

I don't know if I'll ever replace the stock firmware with one of the open source versions, but it's nice to have the option. I doubt I'll try to do anything about it unless I run into a problem caused by the current firmware that isn't addressed by a stock updated from Cisco and is solved by some open source version with very high stability. I don't mind spending time setting these sorts of things up, but I do not want to maintain them on a regular basis.

By the way, I was amazed at the level of detail about the WRT45GL available from wikipedia. There's another blog topic for some other time.

Silence is Golden

2006-09-10T19:39:00.000-05:00

I recently got a new 20" iMac (yes, recently enough to be saddened that they just came out with a 24" model). Since I got the new machine, I've been working to make a place in the basement where I could move the six year old Power Mac G4). The G4 is the mail server for the family domain and will eventually host a family web site.

I finally got that project completed today. It involved running ethernet cables to the attic and basement (solid masonry homes substantially decrease the range of WiFi), putting a shield along the stringers for the basement stairs (to prevent dust, crud, and dropped objects from falling onto the computer), and getting power to the location.

The actual move of the computer was anti-climactic, which is a good thing. Shut it down, disconnect the wires, move the UPS, move the computer, monitor and keyboard, and hook everything back up. A small hiccup with the outlet having a loose wire was quickly solved and everything 'just worked'. Of course, this was after about two months of on-and-off work to get networking cables run and tested, get the dirt shield created, etc. I ran into a problem with attenuation of the Ethernet signal that required me to install a 5-port ethernet switch in the basement. Just goes to show you that 'cat-5' wiring isn't always really 'cat-5' (whether it was not up to spec to begin with, or whether I crimped something and created a problem will probably never be known).

With the G4's (notorioiusly loud) cooling fans no longer generating dBs, the study where the iMac is located is now soothingly quiet. Tonight, while I was fighting with the PC the kids use to play computer games, it was noisy again (the PCs fans are even louder than the G4s). But after I shut that off, I sat in front of the iMac and enjoyed the quiet. In fact, it was so quiet I could hear the sound of a cricket outside the study window trying to foster a next generation before the end of summer.

As for the girl's PC, configuring it to use a WiFi card and to use the printer attached to the iMac was a blunt reminder of just how crappy PC hardware and software really is. But that's a story for another day.

Now, I have to figure out how to convince my wife to let me buy a 24" iMac and give the 20" iMac to the girls. Then we'd be PC-free once again.

Incenting behavior: why IS strives for mediocrity

2004-09-16T23:10:00.000-05:00

There is a frequent complaint I hear from people working in the Information Systems (IS) departments of corporations: the IS department doesn't seem to care about producing good software, and in some cases doesn't even seem to care about producing software at all.

At first, this behavior would seem counter-intuitive, since the purpose of an IS department is to see to the computing needs of its company. But if we remind ourselves that people and organizations do what they are incented to do, things become clear. Before you go arguing that there are altruistic individuals in all organizations, including IS, I agree that such individuals exist. But as in all social situations, the number of altruistic individuals is not enough to change the overall behavior of the group.

To understand why mediocrity ends up as the goal of IS, it's important to realize that the IS department is a cost center, a red line on the balance sheet. Because of this, the IS department can only look good by (a) not screwing up and (b) by lowering it's average cost to get things done. At first, this suggests that IS departments will aim for the best and cheapest solution to any given computing problem. In order to understand why this isn't true, we have to remember that IS departments are run by individuals and individuals do what they are incented to do. We don't care about the individuals who are happy in their current position, as they will simply try to deal with everything thrown at them without losing their jobs. We need to focus our attention on the member of IS who has aspirations.

A staff member who has aspirations has to achieve uncommon success to get promoted (unless they have family ties or blackmail material; and while both of these occur, they are rare enough that we can essentially ignore them). Such a person can never make the company money (remember, IS is a cost center), so they need to stand out in other ways. They could try to get more done with the same amount of time and money, and frequently individual managers do this. But the truth is that there is a maximum productivity that can be achieved for a given budget. And getting more done in the short run doesn't help. Anyone who bases their promotions on getting more done with less has to reach new heights of achievement each time they seek the next promotion. While some exceptional individuals may succeed this way, there is an upper bound on what a single technical person can achieve, and so a limit to how far they can rise and how quickly. For a manager, there is a path of lesser resistance: grow bigger.

Despite some obvious flaws in the logic, the accepted wisdom in IS departments is that a manager with a larger team is doing more work and is therefore more capable than a manager with a smaller team. This creates an obvious path to promotion: build your team as large as possible while avoiding major failures. Individuals who do this while dodging the obstacles thrown at them by others seeking promotions (or those who manage to sabotage their competitors) get promoted. The net effect of this is IS departments run by people who believe the best way to succeed is to grow the IS department as large as possible.

As a result of this "bigger is better" mentality, we end up with mediocrity. If the staff is too large and too unskilled, there will be too many things done poorly and we will end up with major failures. If the staff is too small, no matter how talented, upper management will view us as less important precisely because the group is small. So the real goal of IS mangers with aspirations is to grow as large as possible with minimally qualified staff.

Of course, a good manager has to make sure they are in fact getting things done that help the company. So there is an additional incentive to make sure that work done furthers the goals of the company. But for an individual trying to get promoted, this goal is secondary and only need be minimally achieved.

Unfortunately, the only way to solve this problem is to get upper management which actively discourages the view that "bigger is better". This is extraordinarily hard to achieve, since CIOs are people who have risen to the top by succeeding at the "bigger is better" game. And so the wheel goes round and round with no end in sight.

If you want to work with better than mediocre people, count yourself lucky if you live in a place where software companies exist (since the developers there are producing the lifeblood of the company). If, like most computing professionals, you live in an area where IS departments are the only real option, don't hold your breath waiting for things to change.