Java EE Module Configuration Editors Draft Proposal

I am working on a proposal for Java EE Module Configuration Editors project. The goal is to create a new project under WTP Java EE Tools project and build there editors for web.xml, application.xml, ejb-jar.xml and possibly all the rest of the module configuration files. The editors would be built using Sapphire and work across various versions of Java EE spec. For Java EE 5 or newer, the editors would be able to also edit Java annotations instead of the XML configuration file from the same interface.

Draft Proposal

While I am working on the getting the new project created to hold this work, I’ve started implementing the web.xml editor in Sapphire repository as a sample. You can get a feel for what’s there now and what the final result will look like in the screencast that I put together.

Screencast

To take the editor for a spin yourself, install everything from the following repository. Don't try to go to a newer build as this plugin has been removed from the Sapphire build in preparation for the 0.3 release.

https://hudson.eclipse.org/hudson/job/sapphire-0.3.x/408/artifact/build/repository/

To access the source code, point your Eclipse at the following location in CVS. Note that you will need WTP + Sapphire in your target platform and Sapphire SDK in your dev eclipse. The latter part is very important as the Sapphire SDK contains the code generate necessary to build this plugin.

Source Code

If you are interested in this work, leave a comment with your name on this blog post or send a note to wtp-dev mailing list so that I can add you to the list of interested parties in the project proposal. If you are interested in helping to implement these editors, state as much as well and we will discuss.

Inconvenient process? Let’s fix it.

Some of you may have noticed the debate happening regarding proper entry expectations for WTP incubator project following Holger’s veto of a committer election. Holger is acting well within the power granted to him by Eclipse Development Process (EDP), but is it a right and proper action?

Every committer on a project has the veto power in an election. By extension, any entry criteria for a project (whether written or unwritten) is nothing more than a social convention. The reality is that every committer can choose to levy their own personal expectations. Most of the time it’s not a problem, except when it is.

Here are some quotes from this particular event:

“I found only some bug reports but not a single code contribution from any of the four nominated persons. Please attach the planned code contribution to a bug report. I'd like to vote for each of the nominated persons as soon as I know that the code is readable and covered by JUnit tests.”

“Have [snip] been asked if they like to become committers as individuals (and not only as employees of SAP)? Are these authors of the code or what is their motivation to maintain and enhance these editors?”

In a regular project with established code base, established team and well-defined scope, you can argue that giving every committer veto power over elections is appropriate. After all, there is an established code base to protect. The same considerations do not apply when a new component is proposed in an incubator.

The WTP incubator project was started with the intention to provide a low entry barrier playground for people to come and experiment on new ideas while gaining experience and proving their merit to committers on the core projects that will eventually be asked to admit matured functions. Incubators make sense because they provide a quicker way to get started than a separate project proposal. Unfortunately incubators have to rely on a social convention that existing committers act in a welcoming fashion to newcomers. Most of the time that happens, except when it doesn’t.

I would posit that there is no legitimate purpose served by holding a committer election when a new component is proposed for an incubator. The situation is supposed to be very similar to new project creation and we don’t hold elections there. The party proposing a project gets to designate a group of individuals to be the initial committers without anyone questioning their credentials or motives. A similar process is needed to make incubators work better.

The last revision of EDP has formalized the concept of a persistent incubator. I propose that we build on those revisions and amend EDP to remove the committer vote requirement for incubator projects when a new component is being proposed. The project’s PMC would still have the oversight and ability to decline a new component proposal. This change would also fix the rather awkward problem of having to have “seeder” committers when creating incubator projects.

Note that my suggestion is for persistent incubator projects rather than normal projects during incubation phase. I am also not suggesting that we remove committer vote entirely from incubators. Anyone wishing to join existing effort already underway in the incubator should still be subject to committer vote.

Thoughts?

PS.1 : This situation has served to highlight a process problem and it is the process that I seek to improve. I have no beef with Holger. I am sure he is acting on what he believes in.

PS.2 : I am further confident that this particular storm will blow over, Holger’s objections will be met, another election held, etc. That doesn’t mean we shouldn’t try to improve the process so that such situations do not happen again and we continue to have vibrant incubator projects at Eclipse.

Update: At Wayne’s request I created a bug to track this proposed improvement to Eclipse Development Process.

Common Servers View for Eclipse

There is a cool collaboration happening right now between WTP and DTP to build a shared Servers view that will replace the separate WTP's Servers view and DTP's Data Source Explorer view. The new view is built using the Common Navigator framework and will make it easy for other Eclipse Projects to contribute content. One of the important goals is to reduce clutter on user's workbench by collapsing many individual views into one. If you are interested in this effort, you should check out Bug 252239. There has also been some relevant discussion on Bug 245013 and Bug 247934.

So, I am looking at you... CVS Tooling, Subversive, DSDP Target Management, SOA Tools, etc. You know who you are. Come join the party.

Eclipse Project Declaration : Faceted Project Framework

I am blogging today to raise awareness of a project declaration that might have gone unnoticed in the inboxes of Eclipse committers and other members. The goal of the Faceted Project Framework is to provide a re-usable system that facilitates treating Eclipse projects as composed of units of functionality (called facets) that can be easily added or removed by users. The initial code contribution will come from a mature and a rather successful component in the Eclipse Web Tools Platform (WTP), but the ultimate goal for creating this independent project is to encourage broader adoption in contexts beyond that of WTP. I expect this project will evolve substantially as others bring their use cases to the table.

Anyone who has ever wondered if there was a better solution that a multitude of "Enable function X" menu items or has ever thought that users should be able to add and remove natures without hacking the .project file should get involved. I am looking for both potential contributors and potential consumers. If you have use cases or just random thoughts on this subject, I encourage you to jump in and get involved at the newly-created newsgroup.

Project Proposal: http://www.eclipse.org/proposals/fproj/

Newsgroup: http://www.eclipse.org/newsportal/thread.php?group=eclipse.fproj

Creating API, Lessons Learned

The subject of how to properly create and declare API is frequently debated at various communities within the Eclipse ecosystem. One of the thorniest issues is the disagreement over how to treat the so-called "provisional API", basically the API that either has not yet received sufficient feedback or has known issues that cannot be addressed prior to the release. There is a lively debate going on right now on this very subject on the mailing list of the Eclipse E4 project (the effort to build the next generation Eclipse platform - sometimes referred to as Eclipse 4.0), so I thought I should jump in and share our experience in this area at the Eclipse Web Tools Platform (WTP) project in hopes that others don't repeat the mistakes that were made.

It's important to note that opinions expressed here are mine alone. Other involved parties may not agree. Names and other identifying information is withheld to protect the guilty.

History

The start of WTP project was pretty rough. Large code contributions had to be rationalized in the context of a platform that is supposed to be extensible for many external adopters. One of the challenges was the belief by many of the committers from the company that made the code contribution that the APIs are good as is because they existed like that for a long time inside that company's commercial products and are therefore proven. Aligned against that was the growing feedback from new adopters who were saying that the APIs were insufficient and in some cases just plain wrong. Creating good APIs that are flexible enough to address variety of adopter usecases takes a very long time, but unfortunately time was running out. Major companies involved in WTP were pressuring the project to make a release so that they can build commercial products with it. After much debate, a compromise was reached. WTP was going to make a release, but we were not going to declare any API as stable. Everything will be labeled as provisional.

Sounds good in theory, right? What went wrong is that the concept of provisional API was not concretely defined as part of the initial agreement. Everyone (from committers to adopters) ended up with their own ideas about the meaning of the concept. The first release happened and WTP team went to work on the next release trying to improve the APIs based on the growing feedback. That's when the fireworks really started. Certain adopters were not particularly happy that WTP was continuously breaking them despite the fact that they were leveraging code clearly placed in internal packages or otherwise marked as provisional. Granted, adopters didn't have much choice if they wanted to build products on top of WTP, but that's what you get when a project is starting out. None of that seemed to matter and eventually WTP PMC bowed under the pressure by instituting a very restrictive code change policy. An "adopter usage scan tool" was created that could be used by WTP adopters to scan their code base for references to WTP code and send these reports back to WTP where they would be collected and used as a reference for determine whether a change is allowed or not. This new policy effectively negated the original promise that was made with regard to provisional API. The new contract covered everything including code previously designated as provisional and purely internal code. Instead of committers promoting API, anything that an adopter touched (as represented by these reports) effectively became API.

Work on improving API essentially ground to a halt. It just became too expensive to fix many of the larger problems. Technically, a committer could seek PMC approval to break code referenced in adopter scans, but exceptions were rarely granted. The argument that was frequently made is that by making a proposed change, many lines of code in adopter products would be effected. It's "cheaper" for committers to not make the change in question or at least make it in a way that's completely backwards compatible. I will leave it as an exercise for the reader to see the fallacy of that argument.

The end result is that WTP was left with large amounts of "in progress" API code in random internal packages that was effectively frozen because it became too expensive for committers to continue to work on this API within the imposed constraints. In many cases, providing the requisite backwards compatibility would have effectively doubled the amount of work. Some improvements that were easy to make in additive fashion continued to be made over the next few releases, but real progress essentially stopped.

Finally, last year a group of committers was convened to try to improve the situation by proposing a new API policy for WTP. The end result formally defined provisional API and started the process of phasing out the flawed adopter usage scans policy. As someone who was involved in drafting the new API policy, I can tell you that I still see many flaws in it, but it is an effort to take a step in the right direction. Only time will tell for sure.

Thoughts on API Creation

The following is a collection of my somewhat random thoughts on API creation and the related processes.

• Ability to declare provisional API is an essential step in API creation. You cannot be sure that the API is right until you received sufficient and diverse feedback. It is impossible to attain that level of feedback with one release cycle. Most external adopters will not start looking at a release until it's close to being finished. They will not start building products on it until even later. The best you will get early on is "yeah, that looks about right", which is not good enough.
• Placing provisional API in an internal package (such as internal.provisonal convention sometimes used by Eclipse Platform ) creates unnecessary churn for adopters and committers. Consider the case where provisional API turns out to be 90% correct. The advantage of the internal.provisional approach is that you don't have to separately define expectations for provisional code (it gets treated as internal by virtue of the package name), but I would argue that it's worth taking the time to define a separate contract for provisional API since allowing provisional API in non-internal packages results in less work for both adopters and committers.
• It's important to have a good system for determining whether API is ready to be declared as fully supported (not provisional any more). Leaving the decision completely in the hands of committers or even project leads can lead to problems since people are inherently biased towards their own code. Some things to check when deciding if API is ready to be declared are level of documentation, unit test coverage, presence of outstanding API issues in bugzilla and level (as well as diversity) of adopter feedback. I prefer a system where a committer nominates the API for declaration and there is a process where other committers and adopters can raise objections.
• It's important to carefully balance the needs of committers working on the API and adopters consuming the API. It's a mistake to only look at the problem from the perspective of resource expenditure. For any successful platform, there will always be far fewer resources working on the platform than consuming the platform. Trying to add too much protection for platform adopters can inhibit innovation in the platform and ultimately hurt those same adopters.