Wednesday, May 30, 2007

Work in progress....

In between other things to be done, I have been working on the project throughout the last days to push it forward.
In spite of the fact that we ran it down to the infamous but typical one-man-show again (yeah, I'll happily ignore any form of excuse, so son't spend time making one up; and no, I don't have any expectations any longer), following things have been added/done/achieved recently:

  • we have new hardware (thanks to Mario for the ware and the time and effort it took to set it up) for the testbed

  • we have an up-to-date build running on the testbed

  • I have fixed some initial policy problems, and I am working on a better policy initialization

  • I have added a bank service, so users may have accounts and we have the old system in place (eg. awards for posting, casino runs against your account)

  • I have been working on the documentation, generating a standard specification document using ANT and our OBR repository descriptor

  • I fixed a bunch of bugs

  • I have been working on the EMPP client, specifically the XHTML/CSS styled output panel using Flying Saucer R7. I posted on this one some while ago, but actually it was followed by some work with the FS people (thanks especially to Peter) and a modified but leakless R7 based implementation

  • I have been working on the import of the jkara fora, which is progressing slowly, because there are all kind of traps with the contents (e.g. KML, Control Characters, Entities, etc). I am also not familiar with the jkara database, so I am making up the best mapping on the go.


Guess that pretty much summarizes it.

Wednesday, May 23, 2007

Customizable IM frontend with XHTML+CSS rendering using Flying Saucer

Joshua Marinacci wrote in 2004:


I really think that a complete XHTML renderer is a vital component of any modern programming toolkit.


I cannot do more but agree with him completely, much more so three years later.

I am not talking about trying to load native components into a Java application, but rather an all Java XHTML + CSS renderer.

Unfortunately, I also have to agree with the two questions from the same blog entry:


Why are there so few renderers, and almost none that are opensource? Is it really that hard?


Fortunately, he set out to answer these questions and made the product open source:
Flying Saucer - The 100% Java XHTML/CSS2 Renderer

This entry is a little bit of "Flying Saucer" evangelism, I wanted to share some interesting use case with you and some tuning tricks to make it do quite impressive things.

The base description of the use case I had at hand is relatively simple:
Create a panel that can display instant messages, in realtime, layouting them from an XHTML+CSS source that makes the look completely customizable. Messages arrive async and should be inserted into the actual document.

I first searched the mailing lists for some hints, but all I found was somehow not very promising:


Everytime a new chat message is added we update the Document, the panel blanks out, re-renders, and then we scroll back to the bottom where the new message is added.



So I set out to try this myself, and I realized there were actually two performance problems:

  1. the setup time of the XHTMLPanel when loading the basic XHTML document:

    <?xml version="1.0" encoding="utf-8"?>
    <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
    <html xmlns="http://www.w3.org/1999/xhtml">
    <head>
    <link rel="stylesheet" href="style.css" media="all" />
    </head>
    <body>
    <div id="content">
    </div>
    </body>
    </html>


  2. The time to insert message divs (with some structure; e.g. header, body etc.) in the content div.



And then, there is a related problem, which is basically the fact that we want to display emoticons using images, translating them and probably allowing different sets. Now without spoiling everything, here an image to give you an idea what we are talking about:



1. Setup time and solving them with some Java URL Tricks


The initialization of the XHTMLPanel was a weird puzzle from the very beginning. In fact, it was kind of unpredictable how much time it was taking, but most of the time it was simply too much. Now, after guessing a while I was working without a network connection and I suddenly realized what the problem was:
http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd
Hacking some debug statements into the flying saucer source revealed it clearly, a complete set of documents was loaded over the network, the DTD and the Entities.

Probably I have taken a strange route, but it resolved the problem and it was at the same time an elegant solution for the task with the emoticons.

Instead of hacking the flying saucer code I remembered some URL acrobatics that had helped me out in various situations during my time with Java: custom URL handler implementations. Now, instead of getting into details, I will just point you to a document that does a very good job on this topic. With a little bit of code you will end up with a custom handler and be able to use something like the following URL to replace the DTD:
resource:net/coalevo/empp/client/resources/xhtml1-strict.dtd

All the resources can be embedded into your JAR, which makes it easy to deploy. Finally, this approach allows you to plug in a simple cache (most of the time a LRU map as provided by the Java class library is just fine), and get max performance for repetitive resources....like, yes, I suppose you guessed it: emoticons! :) It can't get simpler than that, and if your resource loader is well built, you may just swap the images on the fly....:

<img src="resource:net/coalevo/empp/client/resources/images/emoticons/smile.png" />

Before this works however, you need to null out the base uri:

SharedContext ctx = m_HTMLPane.getSharedContext();
ctx.getUac().setBaseURL(null);

UPDATE: I have tweaked R6 NaiveUserAgent.java to allow this. At the beginning of the public String resolveURI(String uri) method, I added:

if(baseURL == null || baseURL.length()==0) {
return uri;
}


2. Inserting Messages


With the first problem and the additional task solved, what was left was to insert messages into the preloaded document.
First step: We need to parse the previously prepared XHTML base document into a Document. That's an easy part, flying saucer actually offers some utility to do this (see the FAQ). I preferred to grab a StringReader and wrap it into a new InputSource, which in turn can be used by DocumentBuilder.parse().

new InputSource(new StringReader(documentContents));

However, this isn't a good idea for the actual messages, less if we need performance. More later.

Second Step: we need to insert the message at the correct node. Actually this turns out to be rather easy to achieve:

m_InsertNode = m_Document.getElementById("content");


Third Step: Parsing messages from a String quickly into the actual document.
It's a bit more tricky. One needs to know some XML acrobatics... :) I came up with a helper class that solves the problem: a document fragment insertion utility class that parses the incoming message strings directly as new nodes into the document.

package net.coalevo.empp.client.impl;

import org.w3c.dom.Document;
import org.w3c.dom.Element;
import org.w3c.dom.Node;
import org.xml.sax.Attributes;
import org.xml.sax.SAXException;
import org.xml.sax.helpers.DefaultHandler;

import javax.xml.parsers.ParserConfigurationException;
import javax.xml.parsers.SAXParser;
import javax.xml.parsers.SAXParserFactory;
import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;

class DocumentFragmentInserter {


protected final SAXParser m_Parser;
protected Document m_Document;
protected Node m_InsertNode;
protected Handler m_DocumentHandler;

public DocumentFragmentInserter()
throws SAXException, ParserConfigurationException {
this(SAXParserFactory.newInstance().newSAXParser());
}//DocumentFragmentInserter

public DocumentFragmentInserter(SAXParser parser) {
this.m_Parser = parser;
}//DocumentFragementBuilder

/**
* Treats the xml parameter as a well-formed XML document to be inserted
* into the given document.
*/
public void parse(Document doc, Node insert, String xml)
throws SAXException {
try {
parse(doc, insert, new ByteArrayInputStream(xml.getBytes("UTF-8")));
}
catch (IOException ex) {
}
}//parse

/**
* Treats the xml parameter as a well-formed XML document to be inserted
* into m_Document.
*/
public void parse(Document doc, Node insert, InputStream xml)
throws SAXException, IOException {
m_Document = doc;
m_InsertNode = insert;

m_Parser.parse(xml, getDocumentHandler());
}//parse

protected DefaultHandler getDocumentHandler() {
if (m_DocumentHandler == null) {
m_DocumentHandler = new Handler();
}
return m_DocumentHandler;
}//getDocumentHandler

protected class Handler extends DefaultHandler {

protected Node m_Current = m_InsertNode;

public void characters(char [] ch, int start, int length) {
m_Current.appendChild
(m_Document.createTextNode(new String(ch, start, length)));
}//characters

public void endElement(String uri, String localName, String qName)
throws SAXException {
m_Current = m_Current.getParentNode();
}//endElement

public void processingInstruction(String target, String data) {
m_Current.appendChild
(m_Document.createProcessingInstruction(target, data));
}//processingInstruction

public void startElement(String uri, String localName, String qName, Attributes attributes)
throws SAXException {
//System.out.println(qName);
Element e = m_Document.createElement(qName);
m_Current.appendChild(e);
m_Current = e;
int nAtts = attributes.getLength();
for (int i = 0; i < nAtts; i++) {
String qn = attributes.getQName(i);
String val = attributes.getValue(i);
e.setAttribute(qn, val);
if("id".equals(qn)) {
e.setIdAttribute(qn,true);
}
}
}//startElement

}//class Handler

}//DocumentFragmentInserter


The code for inserting a message now looks similar to this:

m_DocumentInserter.parse(m_Document, m_InsertNode, msg);
m_XHTMLPane.relayout();


Clean. Works. Everything looks good, except....

Fourth Step:
Now, we really don't want an infinite number of messages being displayed, because it may simply kill performance after a while. Fortunately it's realtively easy to also remove a child node message from the Document. So all you actually need is a counter, a max history variable and a conditional that removes the oldest node (e.g. the first child of our insert node) from the moment the max number is reached.


m_DocumentInserter.parse(m_Document, m_InsertNode, msg);

if(m_MessageCount >= m_MaxHistory) {
m_InsertNode.removeChild(m_InsertNode.getFirstChild());
} else {
m_MessageCount++;
}

m_XHTMLPane.relayout();


Sixth Step:
Finally we want the window to scroll properly to the bottom. It took me quite a while to find, but a solution that seems to work is described here, as ReverseScrollBar.

Now, altogether it works out nicely. On my PowerBook G4 Alu (1.67Ghz) it runs smoothly, no blanking out of the panel and almost no delay for displaying messages. I think that this should at least give you some pointers and ensure you that it is possible. If you are stuck, don't hesitate to contact me, also, it is likely that the code will soon be in the Coalevo SVN repository.

Monday, May 21, 2007

......GO!!!!!

We are up and running.

Have fun, behave.

Sunday, May 20, 2007

Ready...Set.....

Everything is set for a new testbed, which will probably be online somewhen this week. First (maybe) there will be a small hardware upgrade :) Then, new versions of some components of Coalevo are ready to be deployed. Here a small summary:

  • New Equinox version

  • Updated TelnetD version, with some terminal update improvements

  • New MailService that will provide mailing of confirmation emails

  • New UserdataService with update, review and confirmation mechanisms in place

  • New MessagingService with support for offline messages

  • New TextService with a whiteboard model for transformations

  • All components initial Java5 only updates

  • New and updated shell access commands for new functionality

  • A bunch of fixes and improvements


We are slowly but steadily advancing towards something useful....

Sunday, May 13, 2007

Ant me out....generating OBR bundle repository descriptors

Last week while trying to update our documentation on the Coalevo project I ran into a relatively simple problem. Modularity in Design means that we have a lot of components, also at a deployment level.
A possible answer within the OSGi context is probably a well managed Bundle Repository. Well managed means that it should have up-to-date machine and human read-able information available about what is inside the repository.

Now, basically your repository will be a directory on some harddisk, and the information you need is already in the bundles. The task may be visualized as follows:



Building and Ant Task


I have been working with Apache Ant as a build tool since a long time, and I still prefer it over other possibilities, because it's really versatile. Usually all I needed was provided right out of the box or by some task you could find searching, but this time it didn't seem to be the case. Thus, I decided to get some hands on excercise in writing the task myself.

I found all I needed to get started quickly in Writing Ant Tasks. The rest I put together from the API documentation and experience with Java (which came in handy for a very specific problem I will discuss later).

There are about four basic elements to writing an Ant task:

  1. the right imports

  2. extending Task and knowing which members are useful

  3. understanding how attributes can be set to make things configurable

  4. understanding how to use your task in a build


Lets attack them one by one.

The right imports


Actually all I needed to import where the most basic classes from Ant that you will use in almost any Task you write:

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
import org.apache.tools.ant.Project;

That's the exception to be thrown if something goes wrong (you can nest the cause), the Task class that we will extend, and the Project class, that will provide us some basic functionality and access to build properties (if required).

Extending Task


I decided to call the class GenerateRepositoryDescriptor, so all it needs is a proper class definition
and to implement the execute() method that will execute your task implementation when the task is
called from a build file.

public class GenerateRepositoryDescriptor extends Task {

public void execute()
throws BuildException {
}//execute

}//class GenerateRepositoryDescriptor

That's actually all, not really complicated :)
Probably the most important member you will need is project that is a Project (the one that gives you access to project properties and basic functionality).

Enabling build attributes


Next you may need some attributes that make your task configurable. Again this is really simple. All you need are standard Java bean modifiers. In my example, I decided that I need three attributes:

  1. dir: the directory that holds the Bundle JARs

  2. outfile: the path for the repository descriptor file (XML/OBR)

  3. name: the repository name to be used in the descriptor


In code, this translates to (javadoc omitted):

public void setDir(String dir) {
m_Repository = project.resolveFile(dir);
}//setDir

public void setOutfile(String outfile) {
m_RepositoryDescriptor = project.resolveFile(outfile);
}//setOutfile

public void setName(String name) {
m_RepositoryName = name;
}//setName

Pretty straightforward I would say. The only special thing in the code is the fact that we use the project member of the task to resolve filenames (see the API docs for more details).

Using the task in a build


This requires two essential steps:

  1. Packaging the task into a jar file and dropping it into your ANT_HOME/lib directory.

  2. Defining the task and calling it in you build file:

    <taskdef name="genrepdesc" classname="net.coalevo.ant.GenerateRepositoryDescriptor" />
    [...]
    <target name="generate" description="Generates the bundle repository descriptor (OBR/XML).">
    <genrepdesc dir="jars" outfile="repository.xml" name="Coalevo Bundle Repository" />
    </target>



Try it out with some simple log statements, it works just perfect:

public void execute()
throws BuildException {
project.log(this, m_Repository.toString(),Project.MSG_INFO);
project.log(this, m_RepositoryDescriptor.toString(),Project.MSG_INFO);
project.log(this, m_RepositoryName,Project.MSG_INFO);
}//execute


Doing the real work


Now, what is left is the logic to process the bundle JARs and write the repository descriptor. I won't reproduce all of the details here, rather I will concentrate on two important elements:

  1. Extracting the Manifest information from a JAR

  2. Handling OSGi R4 localized bundles


Extracting Manifest Information


The simplest way to do this is to resolve each bundle:

//list all jar files in the repository
String[] jars = m_Repository.list(new FilenameFilter() {
public boolean accept(File dir,String name) {
return (m_Repository.equals(dir) && name.endsWith(".jar"));
}//accept
});

//iterate over them
for(int i = 0; i < jars.length; i++) {
generateDescriptor(new File(m_Repository,jars[i]),pw);
}

and then access it programmatically through the java.util.jar API:

protected void generateDescriptor(File f,PrintWriter pw) throws Exception {
[...]
//Get Manifest
JarFile jf = new JarFile(f);
Manifest mf = jf.getManifest();
Attributes attr = mf.getMainAttributes();
[...]
}//generateDescriptor

All it needs then is to resolve the right attribute values, normally using
Object o = attr.getValue(key);
where the keys point to the OSGi related attributes we are interested in (e.g. Bundle-Name, Bundle-Version etc.) when generating the repository descriptor. However, there is a small pitfall here. OSGi R4 defines a new mechanism for localization, that allows some of these attributes to be localized. It makes sense, given that it is information that we pass to the user, however, it complicates our task a little bit.

Handling localized Manifest values


The basic knowledge required are two elements from the OSGi R4 specifications:

  1. Localized variables are prefixed with a %

  2. Localized strings are to be stored in property files in the OSGI-INF/l10n directory, and should be resolved similar to the way it works for java.util.PropertyResourceBundle with the basename bundle


With this information and some Java experience the problem can be resolved quickly:

//1. Prepare a URLClassLoader that has access to the JAR file
URL[] urls = new URL[1];
urls[0] = new URL("jar:" + f.toURL() + "!/");
URLClassLoader cl = new URLClassLoader(urls, Thread.currentThread().getContextClassLoader());

//2. Use a resource bundle, but pass our custom class loader
ResourceBundle rb = ResourceBundle.getBundle("/OSGI-INF/l10n/bundle",Locale.getDefault(),cl);

All that is missing is to add a routine that automates resolving if possible/required

public String getValue(ResourceBundle rb, Attributes attr, String key) {
//1. Try to get Attribute value
Object o = attr.getValue(key);
if(o==null) {
return "unspecified";
}
//2. transform
String val = o.toString();
if(val.startsWith("%") && rb !=null) {
val = rb.getString(val.substring(1));
}
return val;
}//getValue

and use
getValue(rb,attr,"Bundle-Name")
instead of
att.getValue("Bundle-Name").

Transformation


All that is left is a word on transformation. I won't start with details about XSL, if you need something to get started from you may use the one we use for the coalevo bundle repository (available from the same directory as repository.xsl;also demonstrates the result of all the above). However, I just wanted to show you how you can use Ant to generate a HTML version at the same time:

<xslt in="repository.xml" out="index.html" style="repository.xsl" />

That's it ! :)

Saturday, May 12, 2007

Foundation Bundle Docs added

Added the first snapshot of the updated foundation bundle documentation.
Note that the bundle specification document is generated, I will follow up with some more information on this little detail ;)

Monday, May 7, 2007

Coalevo site updated

Since today we have some new look and feel on the Coalevo site. Looking forward to enhance and extend it over the next few days, so you can actually find something, and probably get started helping us out a little bit {author's wishful thinking... :)}

Stay tuned...and feedback is welcome.