tales on the TrimPath

Back a few weeks ago, Kevin Hoyt and I got together to chat about AIR and TrimPath Junction (thanks for the lunch, Kevin!). In less than 20 minutes, Kevin got the Junction-based Next Action web application running on AIR. And, it ran while offline, too!

The AIR runtime apparently maps and transforms the graphical output of an internal Safari engine into a AIR/Flash-controlled viewport. Wild. I wonder if it goes through a matrix transform? Down with the reign of the rectangles-only GUI. Long live the trapezoid, parallelogram and other oppressed quadrilaterals.

Since it’s just Safari underneath, Junction automatically defaults to using the in-memory TrimQuery database engine, because the Google Gears RDBMS isn’t available.

Kevin and AIR team, let me throw in my vote for getting synchronous RDBMS API into AIR!! Once there’s a synchronous API, it’ll be easy to write a database adapter for Junction to hook it up to AIR’s Sqlite RDBMS.

A free idea for the hacker with lots of free time…

Open source is great.

But, make/configure/dependency hell is not.

Perhaps Nested VM can ride to the rescue? Especially as it continues to mature? Let’s bet, for a second, that it does…

Perhaps there’s value (ahem, startup value) in building a “Nested (VM) Forge”, along the lines of SourceForge but without the hell…

The idea: automate the following: web crawl and grab every piece of open-source application you can find. Run it through Nested VM. Post the resulting jar files in a SourceForge-like site. Throw it up on S3. Encourage users to rate, vote, comment, tag, find/report/fix bugs.

Instead of apt-get, create a nvm-get.

Besides advertising, charge for consulting services, Nested VM expertise, and for ‘optimized versions’ of popular software where you rewrite out the slow 20% code that causes 80% of the performance trouble. Turn on Java’s Security Manager and add policies. Charge for better integration into Java, providing sane API’s to access the legacy software. Is there a long tail of legacy software?

For us end-users, no more version dependency / make / configure pain. It would be a library of pain-free open-source software that runs on every platform, executing in a JVM-enabled security sandbox. Like a virutalized OS but at the user-level still.

Hire the Nested VM guys and flip it to Sun or someone else later.

Anyways, back to work.

Junction version 1.1.18 is now ready for download. Also, Next Action’s latest version, 0.94, adds a brand new Calender view along with some other minor improvements and fixes.

Some of the changes in Junction 1.1.18 include…

• jsconsole — similar to the “script console” command-line tool of Ruby on Rails, I added a command-line tool for Junction, so you can explore your Junction application in a Rhino-based, interactive read-eval-print loop. Also, this will serve as a basis for a ‘headless’ testing framework.
• the “local” ajax pattern is now supported. This was driven by the new Calender feature in Next Action, which had an AJAX-based drag/drop interface for rearranging Action due dates. AJAX requests will be automatically serviced by the client-side, in-browser app server, if it’s available; otherwise, the Junction client system will default to sending the AJAX request remotely to the server-side app server.
• optimizations to make translation/localization functions faster, with translated string caching
• removed some more HOP (Helma Object Publisher) dependencies, especially to support the new jsconsole command-line tool
• server support for binary static public file data, via the readBinaryFile() function in junctionCommon.js — this improvement lets Junction support application features like logos and toolbar icons.
• documentation fixes and additions, especially for the modelInit() function
• cleaned up some TrimPath initialization code

Also, for Next Action, you can get version 0.94 and automatically upgrade to it from the Next Action about page. Some of the changes in 0.94.

• from user feedback: added a more friendly and descripted “start” page
• from user feedback: added a monthly Calender view, so you can easily see your Actions due in any month. You can also now rearrange your Action due dates by simply dragging and dropping your Actions around the Calender.
• fixed a Context bug, so now sub-sub-Contexts (and deeper) display correctly. Previously, only sub-Contexts dispalyed correctly
• also, from user feedback: cleaned up and simplified the Dashboard look and feel
• added tooltips for Projects
• fixed some tooltip bugs, involving strings with special characters, like quotes
• fixed the “Add a new Action” sidebar form to always show up
• added a list of completed Projects to the Completed view, so you can now revisit both your completed Actions and your completed Projects and delete them
• fixed a “Today” button bug on the Project edit form
• improved some UI look and feel styling for Internet Explorer users
• and, my favorite — finally added a Next Action logo!

Thanks for all the feedback and ideas — please keep them coming!

Jack Herrington wrote an introductory article on Junction framework for the O’Reilly ONLamp.com site, at
http://www.onlamp.com/pub/a/onlamp/2007/08/30/introducing-trimpath-junction.html

I’m listed as an co-author, but I only provided the contactApp sample code to Jack. He did all the hard work. Great stuff and thanks Jack!

– Steve

The newest Junction, version 1.1.16, is available for download. Some of the changes include…

• big client-side speed increase during data synchronization.
• a new and improved modelInit() syntax, based on suggestions by Brian Moschel. Now, you can initialize Models without using a ‘with’ block. For example…

  // The Milestone model...
  function Milestone() {}
  
  modelInit('Milestone');
  
  Milestone.belongsTo('Creator',  { modelName : 'Person' });
  Milestone.belongsTo('Domain',   { modelName : 'Org' });
  Milestone.belongsTo('Project');
  Milestone.belongsTo('Assignee', { modelName : 'Person' });
  
  Milestone.hasMany('Messages');
  Milestone.hasMany('TodoLists');
  
  Milestone.findRecentByProject = function(projectId, date) {
      return Milestone.find('all', {
        conditions: [ "Milestone.project_id = ? AND Milestone.created_at > ?",
                      projectId, date ] });
  }
  

  Old code based on modelFor() and ‘with’ still works, but modelInit() is the new way of declaring your Models.

• optional declarative migrations, again based on a conversation with Brian Moschel. Lots of database migrations are simple, and in the spirit of Don’t Repeat Yourself, there’s a new alternative syntax to defining a migration. For example, here’s the new declarative way…

  { def: [
      [ 'standard_table', 'Task',
        [ 'name',         'varchar(100)' ],
        [ 'notes',        'varchar(4000)' ],
        [ 'project_id',   'integer' ],
        [ 'completed_at', 'datetime' ] ],
      [ 'standard_table', 'Project',
        [ 'name',         'varchar(100)' ],
        [ 'notes',        'varchar(4000)' ],
        [ 'due_at',       'datetime' ],
        [ 'completed_at', 'datetime' ],
        [ 'color',        'varchar(100)' ] ]
    ]
  }
  

  You can compare that to the original way, where you would supply matching up() and down() functions for each schema migration, and where you would call the DDL functions directly…

  { up: function() {
      createStandardTable('Task',
        column('name',         'varchar(100)'),
        column('notes',        'varchar(4000)'),
        column('project_id',   'integer'),
        column('completed_at', 'datetime')
      );
      createStandardTable('Project',
        column('name',         'varchar(100)'),
        column('notes',        'varchar(4000)'),
        column('due_at',       'datetime'),
        column('completed_at', 'datetime'),
        column('color',        'varchar(100)')
      );
    },
    down: function() {
      dropTable('Task');
      dropTable('Project');
    }
  }
  

  One big win with the new declarative syntax is better support for tools, as the ‘def’ item is easily parsable. You can mix and match up(), down(), and def items, too, for full flexibility.

• refactored out the View template helper functions into a separate junctionHelpers.js file. And, the View helper functions are now available from outside of a template, so you could call them from your Controller, Model, or Helper code. You can now also register your own extra helper functions, too, opening up the possibilities for external libraries and plug-ins.
• cleaned up documentation to prefer camelCase() over under_score_names(), at the behest of Justin Meyer. The functions with the under_score_names() are still there but undocumented, for backwards compatibility. So, new code should be written to use camelCase() names, which is more JavaScript-y.
• some doc bugs fixed.

Meanwhile, Next Action also gets an upgrade, to version 0.92, with some minor bug fixes. New users can get it from the Next Action about page, and existing users can go there too to get the automatic upgrade.

Thanks to a big code contribution from Brian Moschel from Jupiter IT, we have a new release to Junction version 1.1.14 right off the heels of the last one. The biggest change is new features for TrimQuery — the 100% JavaScript SQL engine.

Here’s Brian’s message about the improvements…

The following are the major things I added or improved:

Left Outer Join

SQL fragments for the ON clause of the left outer join. So now you can create more complex sql fragments to match the left outer join, such as:

DELETE User, Thing FROM Thing LEFT OUTER JOIN User ON Thing.created_by = User.id OR Thing.updated_by = User.id WHERE Thing.id = '1'

Insert

Simple Insert clauses with one record inserted per query:

INSERT INTO Event (id, date, invoice_id) VALUES (1, '2003-1-1', 43)

Table.* Requirement

This is no longer the required syntax. Now you can do queries like:

SELECT * FROM Invoice

Update

Update accepts a where clause as well as the field assignments, so you can perform queries like:

UPDATE User SET name='Frank Thomas' WHERE User.id = '3'

Delete

Delete accepts anything that a SELECT query accepts, for example:


DELETE User, Thing FROM Thing LEFT OUTER JOIN User ON Thing.created_by = User.id OR Thing.updated_by = User.id WHERE Thing.id = '1'


Like

Like is supported in Where clauses:


SELECT * FROM User WHERE User.name LIKE 'P%'


Type Support

It appeared although you had types in some of your examples, they weren’t implemented in TrimQuery as of yet. I added support for ‘Date’ and ‘Number’ types. Any tables set up to use these types for fields will attempt to parse incoming data in INSERT or UPDATE queries and convert them to numbers or javascript dates. This is useful for sorting and where clauses. Data returned from SELECT queries are reconverted back to their original form. So now queries such as:


SELECT * FROM Event ORDER BY Event.date ASC



SELECT * FROM Event WHERE Event.date >= '2004-1-1' AND Event.date < = '2005-4-5'



SELECT * FROM Event WHERE Event.invoice_id > 0 AND Event.invoice_id < = 25


With_table Option

For SELECT queries, the third parameter in the filter is a hash of parameters. Setting with_table: true in that hash will return the data in the format table_name+’.’+field_name as the keys in the resulting hash. This is useful for SELECT queries containing LEFT OUTER JOINs or other joins where there will be multiple tables represented in the results. For this query:

SELECT Thing.*, User.* FROM Thing LEFT OUTER JOIN User ON WHERE Thing.created_by = User.id OR Thing.updated_by = User.id

Calling filter with that option:

rs = stmt.filter(tableData, null, {with_table: true});

The results will contain a hash resembling:

{[Thing.id: 1, User.id: 2, User.name: ‘Pete’], [Thing.id: …]}

All of this is covered in the unit tests that I included.

Thanks Brian and Justin from Jupiter IT, who are building a very interesting online IDE and app marketplace called Scaffold.

You can also just get the latest version TrimQuery by itself, or standalone, from the download page.

Oh, and Next Action got a bug fix, too, bringing that to 0.90.

Another week, another release, bringing us to Junction 1.1.12 and Next Action 0.88.

In Next Action 0.88, here are some of the changes…

• added a data export and import feature, in the JSON data format. You can use this for backups, or for moving your Next Action data between computers. Thanks Dan Zurg!
• Action list display now uses alternating row colors, for increased readability
• fixed a bug for repeating Actions not working right, especially with the Dashboard — thanks again Dan Zurg!
• scrubbed the codebase to prevent JS injection attacks, filtering all user-entered data on display
• improved the description of the concept of Contexts for first timers
• fixed a tooltip bug not displaying the right Project
• more db structural/schema work in preparation for an upcoming Contact management feature

Additionally, some other infrastructural tweaks made it into Junction 1.1.10…

• some refactorings of template rendering code to fix a scaffold issue
• preventing direct template execution if a Controller action method is missing

Enjoy your Junction!

You can download it all at…

http://code.google.com/p/trimpath/downloads/list

And, new Next Action users can just visit the Start link that’s off of…

http://code.google.com/p/trimpath/wiki/NextAction

For existing Next Action users, just visit the same Start link, and you’ll automatically upgrade.

Thanks to debugging info and traces provided by Jan Kokoska, the latest fixes in Junction 1.1.10 allows Junction to run on Linux. Thank you, Jan.

Additionally, some other infrastructural tweaks made it into Junction 1.1.10…

• fixed a find_by_sql() bug
• db.getSchema() looks only at database tables now, ignoring other database objects (indexes, etc)
• exposed a res.escape() and res.h() functions to help prevent JavaScript injection attacks

Sync away!

You can download it at…

http://code.google.com/p/trimpath/downloads/list

Junction 1.1.8 and Next Action 0.86 are now available for usage and download. You can get the source package at…

trimpath-junction-1.1.8.tar.gz.

And, for Next Action, Personal Edition users, you can do the easy updgrade: just point your browser at the Next Action intro page to get the instant, personal edition of Next Action.

Here are some of the improvement highlights.

For Next Action:

• Performance improvements with database indexes.
• Caching of Project and Context queries so we’re not hitting the db so much for improved performance.
• Bunch of bug fixes.
• Added ‘two weeks’ to the Active At drop-down — so you can more easily wimp out and delay an Actions by two weeks (hey, works for me).
• Added ‘next two weeks’ to the Dashboard filter — so you see your active Actions out to the next two weeks.
• Fixed the Statistics query so it works on Google Gears RDBMS.
• Some minor styling/UI tweaks.
• Choosing a preferred locale/language is now moved to a Settings page, and is correctly remembered across sessions.
• Prep work and refactoring to get ready for the next set of features.

For Junction:

• The perceived performance for Firefox/Mozilla is now greatly enhanced, with an optimization (layoutRender) that puts it on par with IE and Safari. Junction-based applications now look really snappy on Firefox.
• Bunch of performance fixes all over (concating Strings faster, etc)
• Added more Junction docs for database migrations.
• File caching of start files handles UTF-8 charset correctly now.

Enjoy!

Junction 1.1.6 and Next Action 0.84 are now available for usage and download. They’re all part of the open source trimpath-junction-1.1.6.tar.gz download package.

Or, for Next Action users who really don’t care about Junction, just point your browser at: the Next Action intro page to get the instant, personal edition of Next Action.

Here are some of the change highlights.

For Next Action:

• Fixed a few Next Action bugs (thanks Dan).
• Next Action no longer wastefully sends sync messages when the user is an anonymous demo user.
• Translations for Next Action, first one is Vietnamese (thanks Thuan). When running Next Action, click on the ‘about’ link to switch languages between English and Vietnamese.
• Listing your Contexts on the right hand sidebar in Next Action for easy access.

For Junction:

• Added more Junction docs.
• Added an onBeforeSync() callback event which Junction invokes before it synchronizes a table or Model.
• The refactoring work for true multi-user login/logout work is done.
• Upgraded to hop (Helma Object Publisher) 1.6.0 final release version.
• Handling utf-8 file reading correctly
• New translations infrastructure, with get/setLocale() and “hello”.t()
• Updated redirect_to() to have 2 call signatures, to be more Rails-like.
• Refactored so that server-side user session management, synchronization algorithm and database connectivity are independently plug-in’able.

Enjoy!

Been starting more docs, such as on synchronization and the server-side files in Junction…

I’ve finally pushed out the latest Junction and the Next Action releases. It’s been two years since the last release, due to quite a detour for me into a couple other projects.

But, now, Junction is getting the attention it needs.

The biggest addition overall is the brand new Junction server

The biggest client-side change is Google Gears support.

The release has grown, from <150 KB back in 2005, to 5.3 MB today (see the download page for the comparison), mostly due to the server-side code.

Next up, more documentation, more sample apps…

As part of reinvigorating Junction, I had been dusting off my own, clean slate implementation of a Rhino-based server. It was similar to many of the ideas behind Rhinola, mod_js, and others that seem to be popping up in the news recently — just slap a JS interepreter (Rhino, in my case, not SpiderMonkey) together with a web server. Simple.

But, I’ve now canned this side project.

Instead, I’ve now ported the Junction server-side to use the open-source Helma Object Publisher (Hop) codebase. Mainly, it is because Hop wins the production-ready, battle-scar test, and it’s architected pretty well.

The fit ain’t perfect, though. I’m using Hop in unintended ways, mostly as a pseudo-Fast-CGI for Rhino. But I’ll just grit my teeth about that right now while happily making progress towards the relaunch of Junction. Other friction points: Helma ’skin’ templates are not expressive enough for what I wanted to do; Helma prototype directories didn’t work with private variable closure tricks. Anyways, I worked around those and other bumps, and tallying up the pros and cons, I believe I’ll be able to get to nirvana much faster with the Hop headstart than without.

For longtime Helma folks, the Junction server-side runs as just another Helma app. But this Helma app is actually managing its own collection of Junction-based apps. At this point, I’ve changed nothing in the core Helma, instead going through its public API’s and extension points. The idea is to allow for easy Helma upgrades, and in case I need to switch to something else. You never know.

Hey, the Hop/Helma team has even implemented its own cron-job-like framework, comes with embedded Jetty, has start/shutdown hooks, 3rd party extension points, a GUI debugger, log rotation & archived-log-compression, and has even submitted patches to the core Rhino engine from nearly 6+ years of in-the-wild usage. You can even run it behind Apache, or in your own Tomcat or Servlet server, if you wish. Can’t beat that for a head start.

One benefit of a platform like Junction is that you have several application runtime deployment options. You can mix and match among these scenarios…

1. If your end-user has Google Gears installed, they can execute the web-app primarily within the web-browser. Here, the client-side RDBMS is an asynchronously replicated cache of records from the main server-side RDBMS. The big benefit is increased UI responsiveness. And, the web-app can go offline and survive down or flaky network connections. By flaky, I’m thinking iPhone on AT&T, or 200 attendees trying to share a conference WiFi hotspot.

2. If your end-user does not have Google Gears, no problem. They can still execute the web-app completely within the web-browser, for increased UI responsiveness. And, the web-app can still survive flaky network connections. Here, the client-side RDBMS is not persistent, just a memory-only SQL engine. The downside with a memory-only DB is you can’t truly go offline where you power-cycle, or close and restart your web-browser.

3. The last option is where your application code executes on the server side, running on a traditional-style, server-side web application server. It communicates directly with the full, main RDBMS that is the source of truth. While not as snappy as the browser-hosted web-app, at least the user is working with the full set of data, not a cached subset, minimizing outdated views of information. Also, the pages served up by the web application server are crawlable, if you wish, by your favorite search engines.

Here’s a table…

		  Runtime Environment...
                  1) in-browser   2) in-browser       3) web app-server
                     with Gears      with memory-DB      

Feature...
---------------------------------------------------------------------------------
fast/snappy UI -  Y               Y                   N 

good for flaky -  Y               Y                   N
network

can go offline -  Y               N                   N

full data set  -  N               N                   Y

crawlable      -  N               N                   Y

For options 1 and 2, by the way, there’s an interesting, shared sub-scenario…

I call this the ‘Personal Edition‘ mode. You still execute the web-app within the web browser, getting a nice, snappy UI. However, you don’t let the end-user synchronize with the server-side RDBMS. So, effectively, the end-user is just running a private, single-computer application with a private RDBMS.

This is an interesting pathway to demo or show off your web-application, perhaps as the first, introductory step in a tiered menu of application service plans. Customers can run the Personal Edition of your web application at a cheap, introductory rate (e.g., <$15/month). Later, if they want to sync between computers, such as between their home PC and the office computer, or if they want to share data within a team of folks, they can upgrade to Team Edition or better, for a higher charge, which enables data synchronization.

The frosting to Personal Edition mode is that your server resource usage will be very light for all the tire-kickers and demo users, because all the runtime activity is happening primarily on the end-user’s computer. Only the higher-tiered, higher-paying customers will consume large server resources. That’s good for frugal startups.

It’s been in the air for the several years. Yet, finally, it’s 2007, and the idea of bringing Ruby-on-Rails concepts to JavaScript now has some real traction.

There’s not only my own (Steve Yen’s) TrimPath Junction project, but Steve Yegge’s also been cooking up a RoR port to JS/Rhino. Too many Steve Y.’s here!

One might claim a backwards chain of causality here that passes right through Google. Steve (Yegge) works for Google, and his blog post provides many insights as to why the Google environment got him onto his Rhino on Rails project. And, I revitalized Junction because of Google Gears.

The Helma folks, though, have been doing convention-over-configuration web MVC apps for JavaScript since 2000. But, nobody knew it. I’ll have more to say about Helma in another post.

Other recent activity along the same spirit of taking Rails/Ruby ideas to JavaScript include…

• Jester
• Active Support for JavaScript
• RubyJS

I’m still contemplating about how those last 2 projects might play with Junction, but one thing that jumps to mind is compiling a Rails app so that it can run in-browser, against a Google Gears RDBMS.

By the way, here’s a 3 point recipe to execute Ruby on the Java VM, that features a true bug-for-bug level of compatibility. Even Rails, with its metadata-heavy, open class method_missing magic, should execute exactly the same here…

• Step 1: Download Ruby source and gcc for the MIPS target
• Step 2: Run Nested VM compiler
• Step 3: Wait 2 Moore’s Law cycles for the resulting program to run as fast as the original Ruby version, and while waiting, sing your praises to Turing.

Back a few years ago when I first created and named Junction, it was a play on being a JavaScript clone of Ruby on Rails, thinking that junctions are related to the rails, somehow.

And, for you Saturday Morning Schoolhouse Rock’ers, there’s that snappy ditty with the railway engineer…

♫ conjunction junction ♫
♫ what’s your function? ♫
♫ hooking up views and models and controllers ♫

Or, something like that.

Now that Google has brought us Google Gears, the Junction name seems even more apt. Take a look at this Google diagram discussing offline application architecture…

TrimPath Junction wants to be your Switch!

I’m now defrosting the Junction codebase and this blog. Chipping off the old bit rot, refactoring and documenting…

Why?

The missing piece has arrived.

The final piece of the puzzle is Google Gears. Gears now makes the 3 year old (gah!) vision of Junction now practical by adding a client-side RDBMS and offline capabilities into the mix.

Sure, anybody could have written a browser extension that exposes something like sqlite. Thought about doing it myself for 2 microseconds, but it would have been futile — nobody would have used it. Who else besides one of the big GYMA (goog, yhoo, msft, adobe) could provide the brand name and distribution to get end users to actually download and trust yet another browser plug-in/extension? And, it’s open source. Thank you Google Gears team!

Now, I’ve been dusting off the old Junction framework, porting it to Gears. Stay tuned…

Slides from my talk: Web Apps Without The Web at the March 2006 Emerging Technology conference. Hmm, powerpoint has a really ugly Save As Web Page — oh well, content over style.

Looks like the pre-emininent wiki startups, Socialtext and Jot Spot, have become neighbors. Not just Palo Alto city neighbors, but block party neighbors.

Wonder if they got open wifi?

Talk about an old new thing: modern DHTML is my new WINAPI hell. Instead of a tree of HWND’s, I’ve got a tree of DOM elements to bang my head against.

It’s just like early Windows API’s and MFC programming. To Gary (my manager): yep, I tell ya it works great on Windows 3.1, but not on Windows 95, or Windows 98, or vice versa. But, I got this ugly terrible hack to work around it.

Well, with these fancy Web 2.0 Ajax DHTML apps, I can now report that it now works great on Firefox, but not on IE, or vice versa.

Head bonk against keyboard.