link

In his blog today, Andrew Grumet waxes nostalgic about AOLServer. I used AOLServer with ArsDigita Community System 3, and it was clearly the finest environment for building database-backed web applications that I've ever used. It took me a little while to get TCL, but eventually I understood that everything is a string—really, really, everything.

This site you're reading, however, isn't programmed in ACS or its excellent successor OpenACS. It's programmed in Apache::PageKit, which sits on top of Apache's mod_perl. This is the same environment I chose for the Williams Students Online (WSO) website.

PageKit works well for WSO for two reasons: it's simple, and it's Perl. Simplicity is key because as a student organization, WSO has high turnover and it must be easy for new people to contribute. Perl is good because it's a language that most young geeks know or aspire to learn, and can learn quickly thanks to many good online sources. Also, it has an insanely great repository of reusable modules.

I had hoped to use a Java framework at first, but these were so brutally complex and huge I couldn't even set them up, let alone train other students with little programming experience to hack them. That failure brought up two priorities: the web framework had to be simple enough that I could explain it to someone with a little programming experience in 5 minutes and they would get it, and the language had to be something that didn't make such a person recoil in horror.

ACS and OpenACS are wonderful systems but they fail on both counts. To someone who is well educated about relational databases and understands the implications of programming multi-user web/db apps, ACS is beautiful and makes excellent sense; but WSO had only one such person—me. To someone who has just taken a Java data structures course and has never developed any software for users, ACS is unfathomably huge and the strange languages of TCL and SQL are peculiar to mysterious. Experienced geeks who haven't done DB apps, of which WSO had a handful, can usually see SQL as something potentially useful for real-world stuff but tend to view TCL as a passe fad. You need these experienced folks to lead interesting projects, and you want them to be comfortable, so you tend to avoid suggesting any language that elicits a bitter beer face.

PageKit really is a simple and wonderful framework. The essential feature is that your website is split into a Model (program code) and a View (HTML pages). When a URL is hit, PageKit looks for a corresponding subroutine in your Model code. That subroutine is called, and is responsible for setting some values. The View is then rendered; the HTML in each page is mixed with special tags that the server processes and replaces with the values your Model code produced. The Model code is nice, object-oriented Perl, and the View templates are HTML with about 5 special template tags that do IF blocks, loops, and stick values set by the model into your document. It's a pretty good framework, and though it doesn't have the sophisticated user management, package system, or array of software that OpenACS has, it's much much easier to learn and extend.

But Andrew did bring up one point where Perl/PageKit is simply uglier than the AOLServer API: database calls with loops. One of the most common tasks in generating a dynamic web page is executing a SQL query and looping over the results. AOLServer offers the beautiful db_foreach construction: 

 set title "Late Night With Conan O'Brien"

db_foreach get_matches {
select description, tvchannel, when_start, when_stop
from xmltv_programmes
where title = :title
} {
do_something_with $title $description $tvchannel
do_something_else_with $when_start $when_stop
}

In PageKit right now, you'd write this as: 

 my $title = "Late Night With Conan O'Brien";
my $sth = $model->dbh->prepare('
select description, tvchannel, when_start,
when_stop from xmltv_programmes where title = ? ');
$sth->execute($title);
while(my $row = $sth->fetchrow_hashref)) {
do_something_with($row->{title}, $row->{description},
$row->{tvchannel});
do_something_else_with($row->{when_start},
$row->{when_stop});
}

Now, this isn't 2 to 8 times as much code as AOLServer; it is only 2 extra lines and a few characters. But all the differences are mere scaffolding, and we should be able to abstract that scaffolding and make the essential feature—running a SQL query and looping over the results—absolutely central. The TCL does some scary stuff though: it takes the columns returned from the query and creates variables in the loop body's namespace with the same names. It also does something really, really useful that Perl's DBI (generic database interface) lacks: its :title syntax specifies that the title variable's value should be used (bound) in the query at that point. In Perl, this is a two-step process: you specify ? in the query where you want a value to be bound, and then when you execute the query, the arguments you pass are filled into the corresponding question marks. The big problem with the Perl approach (used also in Java and Microsoft ADO, and probably others) is that you can easily have a dozen ?s in a SQL statement and it's a pain to match them up in order. We should be able to use names like AOLServer. Andrew probably didn't intend his post this way, but I'm looking at it as a challenge: can I make my environment as good?

Answering the Challenge

Can I answer this challenge? Let me try to state it first:

  1. Make it possible to write Perl that has the same effect and structure as the AOLServer db_foreach statement, without extraneous code.
  2. Package this code as a module so that it's easy for any programmer to use the new construct.
  3. Slight adjustments to accomodate Perl syntax and conventions are OK, but any programmer who knows AOLServer's db_foreach should be able to instantly recognize and accurately understand the corresponding Perl.

Sketch of a Solution

I'll take the following template as my goal. I'd like to have working Perl that looks like this: 

 my $title = "Late Night With Conan O'Brien";
$dbh->db_foreach 'get_matches' q{
select description, tvchannel, when_start, when_stop
from xmltv_programmes where title = :title
} {
do_something_with($title, $description, $tvchannel);
do_something_else_with($when_start, $when_stop);
}

Can I do it? Will Perl let me meddle with the language syntax enough to make this kind of crazy business compilable? This post will be continued tonight…

Update 2/27/04: I didn't get to continue this post the night I wrote it. Since then, I've mostly hacked the code I want, but I haven't had a chance to write it up. Meanwhile, Jay has written some code that basically does the looping for you. He also calls poking around in the symbol table to find existing variables "disgusting", a point which I'm both inclined to agree with and find rather irrelevant. As a teaser, I have Perl code that looks and works almost exactly like the TCL code, including grabbing existing variables and support for named bind variables.