Essentially, there are three key points that make it possible:
First, the ProcessBuilder, that helps to create and run a process from within the JVM
Second, handling the Streams and pumping the I/O between the JVM and the process
Third, tricking Linux into running nano as if a terminal is available
There is no big deal about the first part, actually its relatively easy to create a ProcessBuilder instance with a command, add environment variables and start a process:
//Prepare command
Listl = new ArrayList ();
l.add(System.getProperty(NANO_SYS_PROP));
l.add("-R");
if(m_Shell.getSession().isRebindDelete()) {
l.add("-d");
}
l.add(m_File.getAbsolutePath());
ProcessBuilder pb = new ProcessBuilder(l);
//Prepare environment
Mapenv = pb.environment();
env.put("COLUMNS", Integer.toString(m_Shell.getShellIO().getColumns()));
env.put("LINES", Integer.toString(m_Shell.getShellIO().getRows()));
//Start Process
m_Process = pb.start();
The second part is a little bit trickier. There are several approaches to this, but given that there is already one thread per user connection running, we decided to use this thread to pump the streams and take an approach (kudos for this one to gnodet from the Apache Mina project).
//Prepare connection streams
m_Input = m_Shell.getShellIO().getTerminalIO().getBaseIO().getInputStream();
m_Output = m_Shell.getShellIO().getTerminalIO().getBaseIO().getOutputStream();
//Acquire process streams
m_ProcessInput = m_Process.getInputStream();
m_ProcessOutput = m_Process.getOutputStream();
m_ProcessError = m_Process.getErrorStream();
//start pumping
pumpStreams();
Here is how the pumping mechanism works:
private boolean pumpStream(InputStream in, OutputStream out, byte[] buffer) throws IOException {
if (in.available() > 0) {
int len = in.read(buffer);
if (len > 0) {
out.write(buffer, 0, len);
out.flush();
return true;
}
}
return false;
}//pumpStream
private void pumpStreams() {
try {
for (; ;) {
if (!isAlive()) {
return;
}
if (pumpStream(m_Input, m_ProcessOutput, m_Buffer)) {
continue;
}
if (pumpStream(m_ProcessInput, m_Output, m_Buffer)) {
continue;
}
if (pumpStream(m_ProcessError, m_Output, m_Buffer)) {
continue;
}
// Sleep a bit. This is not very good, as it consumes CPU, but the
// input streams are not selectable for nio, and any other blocking
// method would consume at least two threads
Thread.sleep(1);
}
} catch (Exception e) {
m_Process.destroy();
}
}//pumpStreams
The third part is actually not necessary on Darwin/OS X, where the process from the JVM seems to be already providing a terminal. However, it is necessary on Linux for standard JVMs. Kudos for this trick goes to Mitch; its essentially running nano through a local relay called socat:
#/bin/bash
export LANG=en_US.utf8
socat - EXEC:"/home/coalevo/nano $*",pty,stderr
Last but not least, the nano we are using has been slightly hacked, and is always started in a restricted mode (-R flag).
For the complete picture, we suggest to check out the EmbeddedNano source code.

