Oliver Steele

JavaScript Gradient Roundrects adds gradient roundrects to an HTML page, without images. It uses the WHATWG canvas tag if it’s available. Otherwise it uses a stack of divs, whose heights are adaptively chosen according to the height of the graded element, the color components, and the radius curvature. There’s a demo here.

I also wrote a JavaScript CSS parser that lets you attach gradients to an element without writing code. You do this by including CSS inside a div tag whose class is ’style’. View the source of the demo page for an example.

Last week for the first time I did some serious browser JavaScript programming. I put the following tools to good use, but ran against limits with each of them:
* fvLogger
is terrific, but doesn’t include an evaluator. You have to reload your page each time you want to query a new value.
* Rhino is great for pure logic, but you can’t use it with anything that use a browser API. In fact, you can’t use it with anything that uses anything that uses a browser API. This means, for example, that you can’t use it with a library that uses Prototype, without writing some mock objects first.
* The JavaScript Shell is pretty amazing, but I wanted something a bit lighter weight (within the same window), and that works in Safari.

What I came up with is inline-console.js. This file adds an output console, and a text field with an “Eval” button, to the bottom of your application. It also defines some logging functions — info, debug, warn, and error — that append text to the console. (If you include fvlogger, it will use it instead.)

The point of this is to be as lightweight as possible. Add
script type=”text/javascript” src=”inline-console.js” (appropriately tagged) to the document head, and the script will take care of adding the UI.

For more fun, include readable.js after inline-console.js. Then {a: 1} will print as {a: 1} instead of [object Object].

Here’s an example. Try entering some JavaScript expressions, such as 2*3, Math.sqrt(2), or document.body, and then pressing “Eval”. (Click here to open the example in a separate window, where you can view source.)

Files:
* inline-console.js — adds the inline console
* readable.js — adds readable representations for JavaScript objects (optional)
* fvlogger — a nicer console UI with more control over which log levels are displayed (optional)

My other JavaScript libraries are here.

One problem with JavaScript development is that the string representation of a value doesn’t tell you much about the value. For example, [null], [undefined], and '' all display as the empty string. [1,2}, [[1,2]], and [[1],[2]] all display as 1,2 (and so does "1,2"). And ({a: 1}), ({b: 2}), and new MySwankyNewObject() all display as [object Object].

If you use an IDE for development, this may not be a problem. Probably the IDE has its own string representation; even if it doesn’t, you can generally drill into objects by clicking on them. This doesn’t help those us of who prefer REPL development or printf-style debugging. When you display a debugging value (to the browser status line, to the alert() dialog, or to the Rhino console), you’d like some indication of what it actually is. And JavaScript doesn’t generally tell you, at least when the value is more complex than a string, number, or boolean.

Hence, readable.js. Readable adds a Readable class that can stringify a JavaScript value readably, for debugging purposes. Readable.toReadable([1,'', null, [3, 4]]) evaluates to [1, '', null, [3, 4]], not 1,,,3,4. And so on.

To make it easier to use the Readable class, Readable comes with a couple of hooks. First of all, it defines defines info, warn, error, and debug functions¹ that display their arguments to the user. In Rhino, these functions call through to print. In a browser, they use alert() — unless fvlogger has been loaded first, in which case they use it instead[]. You can also replace Readable.log(level, message) or
Readable.display(message) to add your own behavior; for example,
to display the message in the status line, or AJAX it up to the server.

Secondly, Readable can add toString methods to Array.prototype and Object.prototype. Do this, and evaluating an expression in Rhino writes a readable representation to the console, without your having to wrap it in info(<var>...</var>) or Readable.toString(<var>...</var>). Doing this has the consequence that iterating over the properties of an Object or Array will yield an extra one (toString), so this is off by default. But define READABLE_REPLACE_TOSTRING before loading the file, or invoke Readable.replaceToString() after loading it, and you’ll get this behavior.

Files:
* readable.js
* documentation

Update: Fixed for Internet Explorer.

¹ The reason there’s more than one function is that this is intended to be consistent with fvlogger. It’s also handy to be able to search your sources for one logging function, and not the other.

² One advantage of including Readable even if you’re already using fvlogger is that now info([1,2]) prints something different from info([[1],[2]]). Another is that Readable extends the fvlogger functions with variadicity: info(key, '->', value) works now. (Without Readable, it’s equivalent to info(key), except that value is also evaluated for effect.) Finally, you can use Readable to extend Rhino with the same logging API. I use this to write modules — such as paths and beziers — that I test with Rhino and integrate into a UI in the browser.

The two times that I’ve used the WHATWG canvas element recently, I’ve wanted a canvas with string rendering. The most recent time that I’ve used the OpenLaszlo drawview class (which has substantially the same API), I’ve wanted string rendering too.

The graph in reAnimator is a drawview, but with text labels for the edges. And the graph and parse tree in the Graph and Parse tabs of reMatch both use WHATWG canvas for lines, but text for labels. (These tabs are only visible in Firefox, for now.)

TextCanvas.js implements the canvas context extended with labels, for DHTML. And “textdrawview.lzx” implements drawview extended with labels. They share the same API, so that I can write graphics libraries (such as graph drawing) that work with both DHTML and OpenLaszlo. That API is described here.

The first example below is an OpenLaszlo application that uses textdrawview; view source here.

If you’re using Firefox, you can also view the DHTML example. This uses TextCanvas; open it in a separate page here.

Files:
* textcanvas.js — DHTML implementation
* textcanvas-example.html — DHTML example (shown running above)
* textdrawview.lzx — OpenLaszlo implementation
* textdrawview-example.lzx — OpenLaszlo example (shown running above)
* textcanvas-api — API documentation

Some example code:

// <div id="canvasContainer"></div>
var container = document.getElementById('canvasContainer');
var textCanvasController = new TextCanvasController(container);
var ctx = textCanvasController.getContext('2d');
ctx.moveTo(0, 0);
ctx.lineTo(100, 100);
ctx.stringStyle.color = 'red';
ctx.drawString(0, 0, "red");
ctx.stringStyle.color = 'blue';
ctx.drawString(100, 100, "blue");

The OpenLaszlo application below demonstrates animation along a line, a quadratic Bezier, and a cubic Bezier (the top three paths). It also demonstrates (the bottom path) animation along a path composed of multiple segments.

Drag the slider back and forth to display the point on each path at t=slider.value/100, or click the “Animate” button to animate t from 0 to 1.

I wrote this in order to animate the state markers along the edges of the graph in reAnimator. The GraphViz dot tool, which I’m using for graph layout, generates cubic beziers, so I had to write code to render and evaluate them.

If you’re writing for FireFox a Browser that supports the canvas element, you can use the same code with the new HTML canvas element. Click on “Start Animation” to animate the points on the canvas below. And click here to open the HTML applet in its own window.

Files:
* bezier.js — measurement, interpolation, sampling, and subdivision for arbitrary-order Beziers
* path.js — measurement, interpolation, and sampling for paths composed of multiple lines and Beziers
* bezier-demo.js — the code to draw the paths in the demo. This is platform-agnostic, and works in OpenLaszlo (bezier-demo.lzx) and HTML (bezier-demo.html).
* bezier-demo.lzx — the OpenLaszlo demo
* bezier-demo.html — the HTML demo
* drawview-patches.js — patches to the OpenLaszlo LzDrawView. This includes an implementation of LzDrawView.bezierCurveTo.

A couple of caveats:
* This animates along the Bezier parameterization, not the path length. This was good enough for my application, but you could get animation that speeds up and slows down, depending on how you choose your control points.
* The demo code is written for brevity, not speed. In particular, it re-renders the background each time, which involves some expensive math to render the cubic. In reAnimator, I placed the animated elements on a separate view in front of the background. I didn’t do that here because I wanted to share code between the OpenLaszlo and HTML demos, and the techniques for doing overlays were too different.Using a separate overlay for the background didn’t actually speed this up. The hot spot is the parametric position calculation.

¹ I think the reason the code doesn’t work in Safari is that Bezier.draw uses the Function.apply method to apply methods on the graphics context, such as quadraticCurveTo, to argument lists. It looks like Safari doesn’t implement apply when the function is a native method. I didn’t try to work around this because in cases where I’ve actually tried to use canvas (such as the “Parse” and “Graph” tab in reWork), there were other problems with Safari anyway.

Update: The inline examples weren’t showing up. Thanks to Bret Victor for both finding and diagnosing the problem. (I was linking to my laptop, osteele.dev. The post looked fine from my laptop!)

reAnimator got me interested in writing something that would let you use regular expressions. That something is reWork. This web page has a couple of fields where you can type in a regular expression and a string to match it against, and see the results update as you type. It also displays the code to perform the match in some of the languages (JavaScript, PHP, Python, and Ruby) that I use with regular expressions.

reWork limited to the features of the JavaScript regex engine. In particular, it’s missing dotall (/.../s), because JavaScript is. I actually figured out a hack to implement dotall anyway, but this will have to wait for another day.

I suspect that it still has bugs involving scanning and splitting on regular expressions that match the empty string, and browsers that aren’t Safari and Firefox 1.5, but I’m publishing this now in the hope that it will be useful anyway.

JSON for OpenLaszlo is a JSON library for OpenLaszlo.

I wrote this in order to implement my regular expression visualizer.

There’s a live example below. Clicking on a button requests some JSON text from the server and parses it on the client. The source code to the example is here.

(When it runs off my web site, the debugger in the example displays a warning about not being able to connect to the LPS server. This means that the debugger can’t evaluate expressions, as it could if you were running it off the SDK. I’m just using the debugger here to print inspectable representations of the JSON parse results, and the warning doesn’t affect this.)

Rationale

OpenLaszlo implements most of JavaScript 1.5 (ECMAScript 3), but it’s missing regular expressions and throw/catch, so it can’t run JSON in JavaScript. And the OpenLaszlo compiler doesn’t (yet) implement the proposed JavaScript 2.0 (ECMAScript 4) extensions such as class and type declarations, so JSON in ActionScript doesn’t work either. Hence, this implementation, which doesn’t require either regular expressions or JavaScript 2.0 extensions.

It’s open source, of course. (MIT License.)

Here’s something I’ve wanted for a long time. So I finally built it. reAnimator is a tool for visualizing how regular expression engines use finite-state automata to match regular regular expression patterns against text.

This is intended to demonstrate the implementation of regular expressions. If you want to learn how to use them instead, I recommend these references instead:
* Regular-Expressions.info
* A.M. Kuchling’s Regular Expression HOWTO
* Steve Mansour A Tao of Regular Expressions
* Jeffrey Friedl’s Mastering Regular Expressions
* The Regular Expression Library’s list of resources
* New My own reWork online regular expression workbench.

The User Interface

The screen has these areas:

The Pattern

The “pattern” shows the regular expression. Click on it to set another regular expression to match against.

The Input

The “input” is the string that is matched. As you type into the input string, the color of this string indicates whether it’s a complete match (green), a partial match (blue), or a non-match (red).

The Graphs

There are two graphs, which each display a finite-state automaton (FSA) that corresponds to the regular expression in the “pattern” area. As you type into the “input” area, the graphs also update, to display the state of the match.

A deterministic FSA (a DFA) is like a board game, with a counter that is moved according to the successive letters of the input string. The counter starts at the initial state (the leftmost circle with the arrow from off the board). Each consecutive letter of the input string tells where to move the counter to next. If the counter ends up in a terminal state (a double circle), there was a match.

A nondeterministic FSA is the same except that when there’s more than one legal move, you take them both.

The nondeterministic FSA bears the most direct resemblance to the regular expression. In exchange for this simplicity in its construction, it’s more complex to evaluate: instead of keeping track of just one counter, you have to keep track of a set of them. (This is an instance of the compile-time versus execution-time trade-offs that computer science is rife with.)

If you want to learn more about finite-state automata, the wikipedia entry has some useful information, but this also seems like a good place to plug my father-in-law’s book.

Implementation details

The front end is written in OpenLaszlo, and compiled to Flash. It uses AJAX and JSON to request the FSAs and the graphs.

The back end is written in Python and C. The Python part is my PyFSA library, plus a bit of glue to turn various JavaScript objects and graph files into JSON strings. There’s also a cache so that my shared server doesn’t get quite so stressed even if the site becomes popular. (Having the front end logic on the client should help here too.)

The C portion is the wonderful GraphViz. PyFSA creates a .dot file for each FSA, and uses GraphViz to lay out the graph. The server parses the annotated .dot file into a JavaScript object, and uses JSON to download the resulting graph to the client.

An OpenLaszlo class on the client interprets the JavaScript graph description into a sequence of drawing instructions. It also saves the node positions, so that it can animate against them.

Update: Some of the support libraries are now available as open source. See JSON for OpenLaszlo, JavaScript Beziers, and Canvas with Text. OpenLaszlo and PyFSA were available previously.

Implementation choices

Client vs. server: There’s no reason that this couldn’t be a client-side-only application. I just happened to have PyFSA lying around, and didn’t feel like porting it to JavaScript. My goal for this was one day of implementation time, and I didn’t think porting PyFSA would fit into this. (It ended up taking three days anyway, because I forgot about implementing cubic beziers and JSON, and because I got carried away and added animation.)

CGI vs. FastCGI: I’ve been doing most of my server-side programming with either PHP or FastCGI and Ruby, so that pages don’t take so long to serve. This is the first Python service I’ve deployed since I started using FastCGI, and I was planning to use it. But Python doesn’t include FastCGI in its library, and the fact that there were four different third-party libraries with different APIs, none of them endorsed, and that the latest PEP to mention FastCGI was deferred five years ago, made me unwilling to take on the project of evaluating them.I’m glad to hear that I was completely off base about Python and FastCGI. See Phillip J. Eby’s comment below. I stuck with CGI, which is in the standard library.

OpenLaszlo vs. DHTML: It would be just as easy to draw the graph itself using the canvas class in DHTML. I balked at doing the animation and user interface in DHTML, though. (There’s little touches like laying the graphs out horizontally only if there’s enough room, which were only a few lines of declarative code in OpenLaszlo.) And then it wouldn’t have worked on as many browsers. I decided to wait until OpenLaszlo compiles to DHTML, for a DHTML version of this.

PyFSA vs. ââ�¬Â¦: Higher-performance © implementations of FSA minimization and determinization exist. I went with my own because it’s the only one I know of that has the option of preserving source location information across transformations. I use source location information minimally in the interface, and might add more.

Credits

Thanks to Margaret Minsky and Gary Drescher for commenting on a draft of the application. I used Patrick Logan’s json-py for server-side JSON. The credits for GraphViz are here. Thanks to my former colleages at Laszlo Systems, Inc. for helping create the OpenLaszlo platform. I adapted Philip J. Scheider’s code for subdividing cubic beziers; this is a compact implementation of de Casteljau’s algorithm for Algol-like languages. Thanks to Guido and company for Python. Lastly, since people always ask, I drew the architecture diagram with Omigraffle (gee I wish I got a commission!) — which I like because I’m not much of a designer, and diagrams I draw with it are passable without much work.

Update: I changed the name to reAnimator. (It was reMatch.) Thanks to Apache RewriteRule for letting me do this without breaking the old URLs!

Recently I reviewed the programming languages I’ve used over the 28 years¹ of my programming career. The result is shown in the chart below. (Click on the image to see it full size.)

There are some obvious trends here². The languages are mostly getting higher level. There are a few “survivors”: languages that I’ve used over the the course of a decade, although discontiguously: C/C++, Common Lisp, and Java. Java has replaced C (except for a stint around 2000 where I went back to low-level graphics programming at Alphamask), and the scripting languages have taken over from Common Lisp — they’re slower, but they’re terse, have better libraries, and are easier to deploy.

It didn’t surprise me that there are so many languages. For one thing, I like languages, and I enjoy learning how to program by learning new ones. For another thing, times changes and programming languages get better. You’d have to be crazy to program in 1970’s BASIC these days. (That’s BASIC, not Visual Basic. The language I cut my teeth on didn’t have functions or structures.) And there just isn’t much call for Z80 assembly any more.

But why so many languages at one time?

The reason is that different languages are good at different things. When I first learned to program, I used BASIC for everything. Then I started using assembly for code that had to be fast, but I stuck with BASIC for doing my homework. (In tenth grade I wrote programs to multiply and invert matrices, so I wouldn’t be bored silly doing it by hand. It took longer, but I had more fun.)

This is what the second chart shows. Again, click on the thumbnail to see the full size version. And beware! — the colors don’t mean the same thing as in the chart above. (Sorry.)

I use four kinds of languages: utility languages, for writing tools and solving problems; application languages, for writing distributable integrated applications; system programming languages, for implementing operating systems and low-level libraries; and a fourth category of languages, which I’ll call “stretch languages”, for learning how to program. (I also dabble with bash for shell programming and R for statistics, but I don’t really think of the way I use them as programming.)

A utility language is useful for solving small problems and for exploring problem domains. It’s what you write your ten-minute program in. If you can use your utility language as your file manipulation language and your build language, all the better. (For a while I did a lot of programming in C and then Java, and I tried to use make and ant for these. Now that I’m using Ruby as my utility language, I’m a lot happier using Rake instead.)

An application language is for building larger, more full-feature programs that other people can use. (This is my own use of the term. Think “desktop application”, or “web application”.) An application language often has requirements for speed, OS integration, and deployability, that a utility language does not.

(“Application” for me used to mean desktop applications. Recently I’ve only written web applications — although I did write a MacOS app a few months ago, in order to learn about the MacOS toolbox. That’s why the “Desktop Application” category in the second chart trails off, to be replaced by languages for writing web applications, and browser-based clients that connect to them.)

Lastly, the point of a stretch language is to teach me new ways to think about programming. This is the opposite of a utility language, although a stretch language can turn into a utility language after I absorb its concepts. A utility language makes it easy to express programs that I have in mind; it gets out of the way, so I only have to think about the problem domain, and not how to program. A stretch language can make it difficult to do even simple things, because I’m still learning the concepts that are necessary to use it idiomatically or, sometimes, to use it at all.

Sometimes a stretch language becomes a utility language or an application language. This was the case with Common Lisp, which took me a while to understand, but for a while was my most productive language.

More often, the concepts that I learn from a stretch language are helpful in using other languages, even if I have to express these concepts by writing the equivalent of macro-expanded code or a VM. For example, I’ve written a continuation-based pattern matcher in Common Lisp and a logic program in Java, even though neither language supports those respective features. Designing the programs as though the features existed made a hard problem tractable.

And finally, learning concepts from one language makes it easier for me to recognize them and start using them when they’re available in a more mainstream language or one with better deployment characteristics. Those of us who were familiar with advise and the CLOS MOP had a leg up on understanding how to use AOP when it came around to Java. The modern round of scripting languages (Python 2.x and Ruby) have MOPs — if you’re familiar with them from Smalltalk, you know how to take advantage of them in Python and Ruby too. And if I ever have to use C++ again, at least I’ll know from Haskell to look for template implementations of combinator libraries.

Here are the languages that I’ve learned from over the years, and what I’ve learned from each one³. Some of the languages that aren’t in my list would be perfectly respectable stretch languages for someone else — I just happened to be familiar with their concepts by the time I got to them. For example, Python is probably a fine way to learn about OO (and my son learned a lot of OO concepts from Logo Microworlds), and I could have learned metaobject programming from Ruby if I hadn’t already learned it from Smalltalk and Common Lisp.

• BASIC: basic programming
• Z80: ADTs; recursion
• Prolog: logic variables; abstracting over control flow
• CLU: exception handling; coroutines; parameterized data types
• Smalltalk: OOP
• Common Lisp: functional objects; meta-object programming
• Java: concurrency
• Eiffel: design by contract
• Haskell: lazy evaluation; type classes
• LZX (OpenLaszlo): constraint-based programming

Right now my stretch language is Haskell. When I write in Haskell, I feel like a beginning programmer again. I can’t use it when I’m in a hurry, and I don’t use it when I’m more interested in the problem domain than in the craft of programming. Reading Haskell is like reading poetry (I’m very slow at it, and I can’t skim because the new-concept density is very high), and writing Haskell is like writing poetry (something else that takes me forever). As opposed to Python and Ruby, which are more prosaic, and Enterprise Java, which is more like a tax form.

¹ I first learned program in seventh grade, from the TRS-80 Level I Basic reference. That makes me both (1) almost 30, and (2) innumerate.

² In this entire essay, I’m quantifying over the languages I use and the way I use them. YMMV.

³ This list includes more languages than the charts do because it includes languages that I’ve never used commercially. I’ve only ever used Prolog and CLU for college projects, for example, but I learned a lot from them anyway.

Last week I wrote a couple of tools to keep track of subversion checkins:

The Subversion Log Viewer is a master-detail list of recent subversion revisions. It’s based on the OpenLaszlo contactlist example. The nicest feature is really an afterthought: at the last moment, I added faces for authors; I think this makes projects a lot friendlier. Right now it only adds the faces to the OpenLaszlo log; let me know if you’re interested in using this for your own project, and I’ll make a public API for adding faces to a repository.

The Subversion iCalendar Gateway transcodes subversion logs into iCalendar files, that you can subscribe to with Apple iCal or Mozilla Sunbird. I find it useful for a projects that I want to check in on occasionally. Unlike an RSS feed, it gives you a sense of the activity level and the change frequency, at least if you’re a spatial person like me.

Both of these point at the OpenLaszlo log by default, but they’ve got a UI for putting in any subversion repository (http: or svn: protocol only), and generating a permalink for that repository.

One caveat: It takes a long time to request a complete subversion log, so the iCalendar gateway only requests the first 500 revisions the first time you (or anyone) view a given calendar, and then the next time anyone or refreshes the same calendar, it catches up to the present 500 revisions at time.