Bob Lee Achieves Agile Execution by Going Smaller, Faster, Tighter, and More Open
If you blinked, you might have missed it. JSR 330, Dependency Injection for Java, completed the shortest development cycle of any Java Specification Request on record within the Java Community Process (JCP) program, a feat that looked only theoretically imaginable on paper until Bob Lee proved it could be done. As co-Spec Lead of JSR 330, Bob wasn't focusing specifically on agility when he pushed the envelope. "While I'm glad JSR 330 executed quickly, I'm even prouder of our specification's quality. I think we raised the bar," he says.
How is it possible to achieve both speed and superiority? Doesn't something have to give? In fact, Bob gave a lot to make this happen, and other Spec Leads can learn a lot by retracing his steps. He attributes the short execution time and enhanced quality to a number of factors, including a crack team of Experts, an aggressive schedule that hacked the normal process, transparency, limited scope, and a willingness to collaborate, compromise, question assumptions, and think outside the box.
Bob and his co-Spec Lead Rod Johnson hand-picked representatives from across the entire dependency injection community to include in the JSR 330 Expert Group. To enhance rapid development, he kept the group small. Having a publicly-readable mailing list reduced the pressure to include all comers in the official Expert Group. Excluding the Spec Leads, only one expert from each organization was allowed to participate.
The mailing list configuration followed the precedent of JSR 294 led by Alex Buckley. All Expert Group members were required to actively participate. Everyone else was invited to join the observer list. The Expert Group list forwarded automatically to the observer list. While the two lists operate as one most of the time, the separation came in handy when the it came to contentious issues. Bob says, "It's easier to achieve consensus among 16 people than it is 80, the number of people on our observer list."
Bob is not shy about sharing his ideas and getting feedback from others. Before kicking off JSR 330, Bob shared the proposal with his co-Spec Lead Rod Johnson, Java Enterprise Edition (EE) 6 Spec Leads Bill Shannon and Roberto Chinnici, and other dependency injection stakeholders. Bill and Roberto wanted to adopt JSR 330 as a replacement for the dependency injection functionality already specified in Java EE 6, but they didn't want to postpone the planned Java EE 6 release date. To meet their deadline, JSR 330 had to complete in slightly less than 5 months. "After a little creative thinking, JCP Chair Patrick Curran, Bill Shannon, and I decided it was theoretically possible to complete a JSR in four and a half months. We'd have to blaze some new trails to pull it off. There'd be no room for error," says Bob. He welcomed the challenge.
At first Bob thought he could just skip the Early Draft Review and Public Review stages and jump straight to the Proposed Final Draft. He reasoned that the JCP Process Document didn't explicitly require those stages. Moreover, if an Executive Committee (EC) member had disagreed with that interpretation of the process, they could always have voted no. That plan would have enabled Bob's Expert Group to go through the process in just one and a half months. "Unfortunately, Patrick didn't go for it. In hindsight, it was a bit ambitious," Bob admits.
Ultimately, with plenty of help from the dedicated JCP Program Management Office (PMO) staff, JSR 330 pushed the boundaries of the existing process and completed in four and a half months without breaking or changing any rules. Traditionally, Expert Groups have followed the process synchronously by completing stage A, updating the spec with feedback gathered during stage A, submitting the updated spec for Stage B, then completing stage B.
In contrast, JSR 330 executed concurrently by exploiting the fact that after submitting a draft specification, it can take several days for the draft to appear on the web site, after which there is a fixed-length review period that typically ends with a vote. With this in mind, Bob's Expert Group submitted a draft spec for stage B well before stage A completed. In this way, the review period for stage B started immediately after stage A completed! "JSR 330 finished in the absolute minimum time allowed by the current process. We spent the entire four and a half month duration in back-to-back review periods," says Bob.
The risk in this kind of approach is clear, but Bob knew it was well worth taking in light of the reward: inclusion in the Java EE 6 spec. If, after submitting the draft for stage B, stage A raised an issue that had to be addressed before stage B could complete, the Expert Group would have had to resubmit the draft for stage B, thereby pushing the release date out. By limiting the scope of the spec and operating with transparency, they mitigated this risk. "We never had to resubmit, and everything proceeded as planned," says Bob.
"We couldn't have hacked the process without 100 percent transparency," Bob says. With traditional Java Specification Requests (JSRs), the community may not even see a spec until a review period starts. With JSR 330, the community followed the Expert Group discussions and specification changes in real time through the publicly readable Expert Group mailing list, while the specification was maintained in public source control. Any issues that could potentially come out of a review period were made known, recorded in a publicly accessible issue tracker, and addressed well ahead of time, making it safe to submit the draft for the subsequent stage early. Using email and the issue tracker instead of group-wide conference calls also ensured a searchable record and allowed people to work asynchronously on their own schedules.
To operate with agility, Bob says issue trackers are "indispensable," and he likes Google Code's issue tracker best for its high power-to-weight ratio. While email threads sometimes overload and bog down with too many topics, an issue tracker allows a moderator to break issues down and organize the threads. An issue tracker also helps maintain momentum by clearly indicating when an issue is closed, a definitive signal to move on. An Expert who thinks an issue needs more work can reopen it, and and the group will come back to participate in the renewed discussion. When it became apparent that the spec needed a change, they fixed it and published a new draft almost immediately. As a morale builder, nothing beats watching the list of open issues shrink as the JSR improves and progresses toward closure.
When it comes to transparency, Bob is an evangelist. He says, "Some Expert Group members resist public mailing lists because they feel like they'll have to watch what they say, they don't want to confuse the community with half-baked specs, and they don't want to invite a flood of premature reviews. I believe the benefits of transparency far outweigh the theoretical drawbacks. Nothing cleanses like sunshine. Having a public record of Expert Group discussions discourages bureaucracy and bikeshedding. It provides a valuable learning resource for aspiring Experts who may participate in future JSRs. It engenders trust from the community. When you publish the discussion leading up to a decision, there's little room to question the motives behind that decision."
Transparency also helps different Expert Groups collaborate. The Java EE 6 JSR was able to integrate JSR 330 even though the two JSRs completed around the same time. In contrast, the JSR 330 Expert Group ran into difficulties supporting another JSR because Bob's group couldn't see up-to-date revisions of the other group's specification. Bob says, "The lack of transparency left the door open to inconsistencies, duplicated functionality between the specs, and wasted time."
A growing number of Spec Leads are keeping their specification in source control, and there are several benefits to doing so. From his own experience, Bob explains, "I've served in other Expert Groups where the spec lead would send out revision after revision of a 100+ page PDF. It was impossible for me to keep up. The quality of the specification suffered." With source control, the Experts and the community can easily see exactly what changed. The Experts can modify the spec directly, and the community can submit code reviews and even patches if they are members of the JCP program.
Not everything should be discussed in the open, however. An agile Spec Lead knows where to draw the line, and usually that line is at potential controversy. "When approaching a contentious issue, I find it's best to build consensus incrementally through a series of small discussions. Dropping a contentious issue on a large group with little direction or preparation is a recipe for failure," Bob says.
"When in doubt, leave it out" is Joshua Bloch's mantra, which the JSR 330 Expert Group followed resolutely. JSR 330 included an aggressively reduced subset of possible features. It's generally faster to develop a spec that is more limited in scope. More important to the Expert Group, though, was the reality that dependency injector configuration is still a subject of debate. Standardizing that configuration would stifle the rapidly ongoing innovation and burden the community with a snapshot of today's still-fledgling state of the art. Bob says, "We left injector configuration out and focused on the much less controversial annotations. Spring and Guice's annotations already had very similar semantics. Writing the spec was a simple matter of pulling the best ideas from each framework."
Timing is one of the biggest challenges faced by any standards organization. Too often, experts start the specification process prematurely, leaving a trail of unadopted specifications and, in cases where the experts need to restart specs almost from scratch in order to overcome flaws, duplicated effort, and worse yet, duplicated functionality.
The community seems to have the perception that JSRs can't duplicate functionality, but we've seen a few cases within the JCP program -- situations like EJB persistence versus JDO -- where such duplication has occurred. Inevitably, overlap like this stirs up conflict, and motivates everyone to resolve to disallow such duplication in the future. Bob observes that the reasons for preventing duplicate functionality are good, but one natural consequence is that the Java standardization process becomes "a bit of a land grab," with competitors vying to stake their claim first, whether the timing is right or not. Starting a JSR before the Spec Lead has a clear vision of the preferred results "leads to design by committee, and committees design messes," he says. A better outcome that could come from disallowing duplication would be for stakeholders to think long term work together for the good of the community and the industry as a whole.
Bob and the Experts behind JSR 330 have deliberately followed the latter path. "We could have standardized dependency injection years ago if we had picked any one of the many competing, now superceded approaches. Instead, we waited patiently for a natural consensus to emerge, we cordoned off a useful, non-controversial set of functionality, and we quickly standardized it, leaving the community in an enviable long term position," Bob says. Little to no innovation occurred in the JSR itself, so the JSR proceeded predictably and rapidly.
According to Bob, open source enables the kind of patient, long-term observation that yields a singular best solution in a given area. In the past, when tools and libraries were mostly closed source, standard APIs protected users for the possibility that a vendor might go under or otherwise fail to meet a user's needs. Developers who coded to a standard could just switch to a different vendor. Conversely, not coding to standards exposed developers to unjustifiably high risk. In the case of dependency injection, almost every proprietary implementation that preceded JSR 330 was open source. This greatly reduced the pressure to standardize. If a user ran into a problem with an open source dependency injector, they couldn't easily switch to a different vendor, but they could at least fix the problem themselves or pay someone to fix it for them.
Without a willingness to compromise and endeavor for the greater good, JSR 330 would still be in limbo. For example, the Expert Group and Sun almost reached an impasse over, of all things, the specification license. The Expert Group wanted to use the same, well-known Apache license that we used for the Reference Implementation (RI) and Technology Compatibility Kit (TCK). Bob says, "I pushed hard because I felt that future JSRs, not just JSR 330, should be able to use the Apache license for their specifications." However, Sun claimed that the JSPA prohibited the Expert Group from licensing their intellectual property (IP) to any implementation that hadn't yet passed the TCK, citing section 5.B of the JSPA.
Bob had a different spin: "We interpreted the same JSPA clause to say we must license our IP to any implementation that passed the TCK, but we could also license our IP more freely." As open source developers, the Expert Group wants to encourage wide adoption of the JSR and has no interest in making money directly from it. Moreover, Bob's thinking went, "We have no intention of suing incompatible implementations for IP infringement, so why sacrifice simplicity to explicitly reserve that right?"
There was more to the discussion, but in the end, to keep the schedule, the JSR 330 Expert Group relented and accepted a dual licensing approach. Users can get a copy of the JSR 330 spec under a standard Sun license from the JCP web site, or they can get the Apache-licensed spec at http://code.google.com/p/atinject.
The Expert Group used Javadoc for the entire spec. The high level, narrative parts of the spec went in the package description. Javadoc's new features in Java 5 make it easier than ever to embed code samples and reference different parts of the spec and even external specs. "This enabled us to treat the spec like code and attack it similarly to how we would any programming problem. We compiled our spec from day one. As an API designer, an API implementor, and an API user, I strongly prefer Javadocs over PDF, in part because Javadoc has so much native tool support," says Bob.
Bob notes that Expert Groups that like to publish Javadocs before a JSR is final should use a proprietary package name or put a big red "DRAFT" banner across the top of the Javadocs. He says, "Users have no other way to differentiate between an in-progress API and a final one. I've actually seen users adopt in-progress JSRs thinking they'd been finalized."
Because the current JCP process requires final licensing terms early on, there was little to do by the time Final Release rolled around. The JSR 330 Expert Group simply removed the "DRAFT" warning from the Javadocs and shipped. And celebrated.
As for operating with agility, Bob concludes, "You might say I'm agile but not Agile(TM). Good design comes from good designers, not processes. If you want to become a good API designer, read Effective Java, not a book about agility. Identify APIs you admire. Follow their designers. Before you accept API design advice, look at who it comes from; judge experts by their designs."