Oliver Steele

If everyone in America makes an extra-large Thanksgiving dinner so that they can feed guests the next day, isn’t this a pyramid scheme?

The developer world is divided into two camps. Language mavens wax rhapsodic about the power of higher-level programming — first-class functions, staged programming, AOP, MOPs, and reflection. Tool mavens are skilled at the use of integrated build and debug tools, integrated documentation, code completion, refactoring, and code comprehension. Language mavens tend to use a text editor such as emacs or vim — these editors are more likely to work for new languages. Tool mavens tend to use IDEs such as Visual Studio, Eclipse, or IntelliJ, that integrate a variety of development tools¹.

New languages, such as Laszlo and Groovy, and new language extensions, such as AOP, are typically available for text-editor-only development before they have support within an IDE². After a while, if the language or extension is successful, the tools catch up. This isn’t just because it’s harder to implement a tool than just a language. It’s because language expertise and tool expertise are to a certain extent alternatives, that each reinforce themselves to the exclusion of the other. Here’s why.

Language-Maven Perspective

If you’ve spent most of your time learning about languages and how to use them, your picture of the world may look like this:

In this diagram, the choice of language can make a great deal of difference to your productivity, because you know how to apply each language feature to a variety of situations. The choice of IDE, on the other hand, doesn’t matter much, because you’re mostly using the IDE for its text editor — maybe with a little bit of compilation support thrown in, but not for the whole set of modern IDE features that a modern tool maven uses.

Tool-Maven Perspective

Conversely, if you’ve spent most of your time mastering the development tools of your trade, you understand what it’s like to get into a flow state with an integrated editor and debugger, and to go at a program with refactoring and code comprehension tools at your disposal. Your picture of the world may look like this:

An IDE presents huge advantages relative to a simple text editor, for you. The choice of programming language, on the other hand, doesn’t matter that much — you’re mostly working with the same old classes and methods, statements and expressions, in any of them, and the real development power comes from the IDE.

Two Camps

Why can’t one be a language maven and a tool maven? It’s hard. One reason is that developers have limited time, especially for learning new skills. You can use any given block of time to master language features, or to master development tools. Each of these choices contributes to a positive feedback cycle, but they’re competing cycles, so they create divided camps.

Taking the time to learn language features puts you in a position to appreciate the features of new languages. You can then adopt a language before tools support it, because you don’t rely on the tools for your productivity anyway. And it’s worth adopting these tools early, because their features are valuable to you — making use of them is where your expertise lies.

Invest the time in mastering development tools, on the other hand, and you won’t want to give these tools up to try a new language — the development tools are a major part of your productivity. For you, a new language doesn’t have many advantages over other languages anyway, because you haven’t studied how to use language features to make you productive.

This means that the more you invest in language features, the more they benefit you, to the exclusion of tool features — and vice versa. And this is what creates the two camps, with two perspectives on the relative merits of language features and tool support:

Language Buffet

At any given time, the editor-only developer is in a position to choose from a wider selection of languages — because, at any given time, more languages have a compiler and a runtime, than a compiler and a runtime and a language-aware editor and and debugger and etc.. The diagram below shows what this looks like for a number of languages at different points in their evolution. Within each language, the line at the end of the red transition marks the point at which it has acquired enough features to be useful for some class of programming. The line at the end of the purple transition marks the point at which tool support (beyond a compiler and runtime) exists for it too. At any time, more languages are programmable (the red squares) than have tool support (the blue squares).

One consequence of the greater language selection available to the editor-only developer is it typically includes languages that are more powerful. So while one reason that the “Language” block is bigger in the “langauge maven” illustration above (reproduced below) is that the language maven dedicates more time to learning how to leverage language features, the other reason is that the language maven may have a more powerful language to work with, because there are more languages available to her.

The Language Developers Dilemma

In fact, the most powerful languages may initially have the least powerful tool support. The reason for this is that the language developer, like the language adopter, has to make a choice: whether to dedicate limited development resources towards language features, or towards tool support³.

Consider the diagram below. This diagram represents three languages, whose respective developers have chosen to concentrate initially on language features (the top arrow), on tool support (the left arrow), or (the middle arrow) a middle-of-the-road strategy that emphasizes them both equally. The arrows are roughly the same length, representing roughly equal amounts of effort.

Although the arrows are equal length, they achieve different levels of language features and tool support. The “language-first” strategy of language development achieves a greater degree of language features, as indicated by the red tics across the bottom, but a lesser degree of tool support, as indicated by the blue tics across the right. And conversely for the tool-first strategy.

Of course, if a language succeeds, it will eventually get tool support. So in the long run both the language and the tool mavens will use the popular languages, such as Java. That’s where I’m hoping we’re going with Laszlo.

The Perils of the Pioneer

All this makes the language approach sound comparable to the tool approach; it’s just a matter of which skills to learn. But this isn’t always true. We could argue about whether understanding closures or a source debugger makes one more productive. But what isn’t arguable is that there are penalties, beyond the lack of tools, to adopting a language early.

One is that the language may not have staying power. Another is that, just as tools lag initial development, so do other ecosystem artifacts such as books, articles, newsgroups, and, in the enterprise world, the ability to hire developers who know a language, or to be hired on the basis of knowing one.

There are contexts where none of these disadvantages matters. If you learn languages easily, finish projects quickly, and move on to new projects frequently, it doesn’t matter whether a language is going to succeed, so long as it helps you with your current project. If you learn languages from reading source code or reference manuals, you don’t need the books and articles that will follow later. And if you’re working with a small team (perhaps as small as one) of elite developers, you aren’t hiring them for what they already know anyway.

Which isn’t to say that you can’t be an elite tool maven too, it’s just that the conditions for success are different. You can be the only one on a team to use Eclipse. If you’re the only one on your team using Haskell, something is probably wrong.

Comments

Bob Congdon points out Lisp and Smalltalk as counterexamples to the “language developer’s dilemma”: both are powerful languages with powerful development environments.

Lisp was a powerful language first, and had a powerful environment second: it only appears to be an exception because it has been around so much longer than any other extant language and has had time to acquire both. Smalltalk, however, is a genuine exception. Smalltalk might be viewed as a step backwards in language features from Simula 67, and a simultaneous step forwards in tool support — but this is holding the language to a higher standard than any other.

Congdon also points out that it is possible to both a language maven and a tool maven. True in theory, just as it’s possible in principle to become both a concert pianist and mathematician, but in practice there may not be enough hours in the day to do both.

And Harry Fuecks explains some of what I was trying to say better than I did, as well as adding points of his own.

Notes

¹ A sophisticated emacs user, who uses something like psgml-mode or jde for every single language they edit, is arguably in both camps. I haven’t met many of those, though. The emacs users I know use it configure it as an HTML-editor or a T_EX-editor, say, but use it as a text editor for everything else.

² C#, with Visual Studio.NET, was an exception — at least, if you’re outside Microsoft. Microsoft is an exception to almost everything.

³ And this is why Microsoft is an exception: because its development resources are effectively unlmited.

Last month, Laszlo Systems released the Laszlo platform as Open Source. The intent was to to remove the bottleneck in the virtuous cycle of software development.

Today IBM alphaWorks released IDE for Laszlo. This is an Eclipse-based IDE for creating, editing, debugging, and testing applications written in Laszlo.

Eclipse is an impressive platform, and the IDE for Laszlo is an impressive effort. This page demonstrates many of its features, with screenshots for each. (It was written in LZX, using IDE4Laszlo.) Kudos to the developers!

From http://alphaworks.ibm.com/tech/ide4laszlo :

IDE for Laszlo is a technology preview of an Eclipse-based development environment for creating, editing, debugging, and testing applications based on the LZX declarative mark-up language.

Laszlo is based on LZX, which is an XML and JavaScript description language similar in spirit to XUL (XML User interface Language) and XAML (”Longhorn” mark-up language by Microsoft®). The Laszlo Platform is an open-source platform for the development and delivery of Rich Internet applications (see http://www.openlaszlo.org for more information) where the LZX XML mark-up is used to create the user interfaces.

The IDE consists of a set of plug-ins that allow creation and testing of Laszlo applications, all within the Eclipse platform. These applications can then be deployed and run on a Web server. IDE for Laszlo also provides a rich editing environment for the LZX mark-up language. Its editing features include XML- and script-based content assistance, XML syntax highlighting, and XML code formatting.

In addition, IDE for Laszlo allows the developer to preview the resulting application without deployment, within the Eclipse environment. It supports markers for reflecting compilation and syntactical errors. When development is complete, the applications created can then be deployed and run.

Decisions involve tradeoffs. Time at work subtracts from time with your family; money saved for the future subtracts from goods and services now; many food choices trade off among taste, convenience, price, and nutrition.

Some tradeoffs in computer science are the cost to update versus the cost to search, and execution time versus memory consumption. Tradeoffs in software development include implementation time versus execution time, and compilation time versus object code quality. Tradeoff in project management include resource pool size versus communications overhead, and cost versus time versus quality. These tradeoffs are some of the tradeoffs, respectively, in such tasks as choosing a data structure, algorithm, or cache size; choosing a programming language, a compiler and compiler settings; and choosing a project team size and personnel.

Often the factors involved in a decision are more complicated that A versus B. More time at work makes me happier, and more time with my family makes me happier too, but the contributions are different and non-linear. To some extent, I can sidestep the tradeoff between work time and family time by sharing my work with my family and vice versa, but this has different effects on different aspects of work: I make better decisions when I talk over the less technical aspects of what I do, but I can only achieve the flow state necessary for sustained technical work when I’m working alone. Family collaboration facilitates decision making and inhibits flow state; effective decision making and flow state both facilitate work enjoyment, which facilitates happiness.

This is a degree of complexity that it’s hard to keep in your head, and keeping it in your head makes it hard reflect on it. (The more of your system bus you’re using to refresh volatile memory, the less is available for computation.) Hence the constraint diagram.

A constraint diagram represents the factors in a decision. Lines between nodes represent interactions between factors. Arrowheads represent facilitation (increasing A increases B), and circles represent inhibition (increasing A increases B).

(The terms and symbols are from neural mapping. Substitute “promotion” for “facilitation”, and this is also the language of gene expression.)

A tradeoff is a choice between two mutually inhibitory factors. If increasing A decreases B, and vice versa, then a decision about the settings of A and B involves trading off between them.

Let’s look at a classical project management tradeoff: the three-way tradeoff among cost, time, and quality. There are a couple of ways to look at this tradeoff. One is by listing “cost” and “time” as factors which facilitate to quality:

The advantage of getting this onto paper is that it makes it easier to unpack the factors. Time vs. cost vs. quality is a simplistic way of looking at this. The diagram below illustrates the way that time, cost, and quality interact. The project manager controls time directly, but not cost: cost is an emergent property of staffing. Borrowing from the language of statistics, time and staffing are independent factors (they’re under the decision maker’s control), and everything else is a dependent factor. Cost and quality are the goal factors: they’re what the decision maker is trying to optimize. I’ve indicated independent factors and goal factors typographically in this diagram and the ones that follow.

(This model, which is still very simplified, is of a non-capital-intensive project that’s managed to a date, such as a software project with a hard launch date. Building an airplane or writing incrementally deployed software would have different independent factors.)

To make the model more precise, we can zoom in on, and expand, the relation between effort and quality. Effort can be spent either on the feature set, or on the quality (usability, extensibility, low defect rate) of the substrates and of individual features. Both of these factors contribute to the overall product quality:

Returning to the example at the beginning of this posting, of the family-time vs. work-time tradeoff, the following diagram represents graphically the factors that I described in text above. Like the time-cost-quality diagram above, it’s still simplistic (my family is happier when they participate in my decisions, and this makes me happier too, and also increases the amount of time they’re willing to let me spend away from home, which increases the time I have to get into a flow state), but it’s already an order of magnitude more than the degree of complexity that I can comfortably hold in my head.

Over the next few weeks, I’d like to explore the use of constraint diagrams to represent some tradeoffs in software engineering, especially where these tradeoffs intersect my other interests: software development, productivity, and the use of abstractions and high-level languages.

As of today, the Laszlo platform for building rich internet applications is open source. This includes everything: the server software, the client software, the examples, the documentation, the language — the whole platform. Like Mozilla, this is open source with a corporate sponsor; and like Mozilla, it’s honest-to-goodness open source — no dual licensing, no poison pill. It uses the Common Public License, listed on OpenSource.org.

OpenLaszlo.org has the source distribution for our new release, LPS 2.2, which also includes support for SOAP and XML-RPC, and over 500 new pages of documentation. For developing Laszlo applications, as opposed to hacking on the source to the Laszlo compiler and runtime, I recommend the binary distribution instead, which comes with installers for MacOS, Linux, and Windows. (You don’t have to actually write any code to see some neat stuff in the standard installation.) If you want to see some examples of the kinds of applications you can write, take a look at the customer showcase, the demos, and at MyLaszlo.com. If you want to dive into the source code, look at Laszlo Explorer and the Developers Guide.

Today is part one: the source code is available, the license is free. Part two is to open up our development process, including our source repository and bug tracking systems, so that you don’t have to be at Laszlo Systems (the company) to see what’s going with Laszlo (the open source project). Currently we’re in send-mail-to-the-dev-list mode for questions, and send-us-a-patch mode for contributions — about on a par with some of my other open source projects, but we can use those corporate sponsorship $$ to do better.

I’ve worked on open source projects before, and I’ve worked for companies before. One of these gets your software into the hands of those with the time to figure it out without training, and lets you work with great people who are able to find an excuse to use it in their work or free time. The other pays the rent, backs your product up with training and support, and lets you work with great people who need to pay their rent. I’m looking forward to doing both at the same time.

(By the way, if you want to work on Laszlo and fit the first category, sign up for the dev list. If you fit the second, send us a resume.)

Today my office is an airplane. I’m visiting the home office in San Francisco for the week. I get to remind everyone that those of us in the Boston office are real people (insert your favorite joke here), and come back with enough to understand what’s behind the email and phone conferences for another month.

One of the geek games you can play on an airplane is stretching out the battery life of your computer. I have enough batteries to last me through a six-hour flight now, but old habits die hard.

Since I save my files every minute or two, spinning down the hard disk isn’t an option. (I tend to use programs that communicate use the file system to communicate. And I don’t want to be in a position to lose more than a few minutes of work anyway.) I don’t usually use a CD or DVD player, so I’m already optimizing there. The CPU that I’m using steps down to 800MHz when the plug is out, so that’s taken care of for me. That leaves screen brightness.

The problem with turning down the screen brightness is that this also turns down my reading speed. I’ve noticed that I read a lot faster when I’m reading higher-contrast text. Since I already use black text on a white background, the contrast is proportional to the screen brightness (as long as the overhead light is off).

This effect matters most when I’m reading English. When I’m reading math or code, or writing anything, my word recognition time, which is the part of the pipeline modulated by the contrast, isn’t the bottleneck anyway — in those cases I’m limited by comprehension time, or by other central abilities. So I used to turn the brightness up when I’m reading, and turn it down the rest of the time.

Let’s call this the contrast modulation strategy. It trades off reading speed against battery life, based on the current activity (reading versus other work). The contrast modulation strategy works if I’m spending long stretches of time reading, or not-reading. It’s a pain if I’m going back and forth — referring to one document while I’m writing another, for instance.

Here’s today’s discovery: my reading speed is also proportional to the text size. I just ran a few informal comparisons, and setting my document zoom to 200% at the lowest brightness works just as well as turning the brightness all the way up. (With a large font, it doesn’t seem to matter how bright the screen is, at least within the range that my screen adjusts to.)

With larger text (a larger font size, or a greater zoom), I have to scroll more, but when I’m reading English I don’t need to see more than a few paragraphs for continuity anyway. When I’m reading code I want to see as many lines as possible (this one reason I prefer concise programming languages, so I can see several different levels of structure without having commit as much to my mental buffers so that I can scroll) and technical documents always seem to refer to tables and figures a page away, which makes for a lot of scrolling, anyway. So I want big fonts for English that I’m reading, and small fonts for everything else. But font size, unlike screen brightness, is something you can associate with a specific document.

Let’s call this the text size modulation strategy. The text size modulation strategy trades reading speed off against the amount of the document that’s simultaneously visible. The text size modulation strategy takes decouples the trade-off between reading speed and battery life and, unlike the contrast modulation strategy, it’s automatic once you zoom your documents, no matter how many times you switch between them.

Miles told me about the computers at his elementary school:

They’re running anti-virus software, but they’ve installed a virus! It’s called Novell. It makes the computer boot slowly, it does a lot of stuff while it’s booting, and then you can’t log on.

Patrick Roberts has written a fascinating post on Responsive User Interfaces. It’s easy to make a specific application responsive, through careful coding and by limiting what the user can do. An architecture for doing this, with arbitrary functionality, is one of the holy grails of GUI frameworks. Patrick’s post is a step towards this.

Patrick defines responsive as “the UI never locks up and provides at least partial results ASAP, not that every operation is completed instantaneously”. His design uses a UI thread that runs performs fast operations, and pushes slow operations onto a queue. A background thread runs these operations in priority order.

In his second post, Patrick refines the queue to handle superseded job items. This is important in a UI, where, for example, scrolling a list item out of the visible region of the list makes it unnecessary to figure out how to display it. It requires some finesse, because a naive implementation never gets around to displaying any of the items that the user is scrolling through. (A lot of work went into solving this particular case in the Laszlo implementation of lists.)

Patrick’s queue is just a step towards the responsiveness grail, and not the grail itself, because it still requires hand-coding to decide whether to perform each action immediately or place it on the queue, and to factor long-lasting tasks into subtasks that can display incremental progress — although maybe MVC usually takes care of the latter?

This design reminds me of David Allen’s Getting Things Done system. GTD is summarized here (PDF) and here. 43 Folders is an excellent site about how to use the system.

GTD can be thought of as a really sophisticated priority queue, and a decision procedure for deciding what to execute immediately and what to place in the queue. Each item in the queue has metadata: whether it has multiple steps, whether it requires additional information, its priority, external deadlines, and the context that the item requires: people, place, time, and energy. (GTD includes other storage structures too, and uses very different terms. I’m playing up the queue, and assimilating it to my own system, in order to make my point here.) When time is available, an item is selected from the queue according to the match between the current context and the item context, and then according to relative properties such as item priorities and external deadlines. For instance, I do my busywork tasks late in the day or when I’m otherwise feeling not at my sharpest; there’s some tasks I only do when I’m in California or when I’ve got a coworker on the phone; and there’s some that I’m only willing to start when I know that I’ve got a few uninterrupted hours in front of me. Beyond that, I choose according to whether there’s an external commitment and then by how important they are.

Here’s the idea I’m toying with: Use a queue like the GTD queue, and items with metadata, to schedule actions an application framework. The idea of matching task context with current context brings together some of the context-dependent actions that a computer performs. For example, indexing is a lower-priority task, but one that doesn’t require the user and doesn’t require an available time commitment to be worth starting; this is like my busywork. Synching, checking mail, and sending queued messages require a network connection; this is like my being in California or on the phone with a coworker. I also like the idea of adding metadata to an action, and using this to schedule it. In the GUI frameworks that I’m familiar with, the only metadata associated with an action is how to undo it (and what to call it in the Undo menu item). If the metadata also includes some of the same kinds of information that tasks in a project plan included, such as resources, dependencies, deadlines, and time estimates, then a GTD-style decision procedure can be used to move actions between a GUI thread, a background thread, and a Roberts-style queue.

There’s an obvious connection to blackboard systems and to AI planners such as Soar here. In fact, we can take a leaf from AI to solve the problem of how to supersede tasks, as in the scrolling case. The idea is that tasks have supertasks, and cancelling a supertask cancels its subtasks. (This is like a goal tree in planning; a process thread or child process in an operating system; and a project milestone or a milestone task in project management.) In the scrolling case, the task of rendering a list item is a subtask of displaying the visible content of the list. Once the list has scrolled again, the extension of the visible content has changed, and any subtasks of the task of displaying the content of the old position should be cancelled. (A smart system would pool the tasks, so that it could restart them or leave them running if they’re also subtasks of a new task — in the scrolling case, if the new position includes the same list item.)

Although the Roberts queue and the GTD system are similar, they serve different purposes. The Roberts queue was designed to optimize responsiveness — the time between the user event that initiates the action, and the initial feedback that the action is proceding. The GTD system was designed to optimize productivity — the throughput of the system, and the portion of that throughput dedicated to important items. Responsiveness and productivity are duals. A responsive application is easier to use productively, because it reduces input stuttering and it allows for tighter OODA cycles. And a productive person has an easier time being responsive, because (1) their backlog is smaller, and (2) they can tell you what their backlog is, and whether the task of responding to you is in it.

Python really is becoming lisp. With the type/class unification, decorators, and generator expressions, it’s jumped from 80 percent of Common Lisp + CLOS to 90 percent¹, and for web tasks the web programming libraries often make up the difference.

I’ve listed below some of the functional and metaprogramming recipes and packages that have shown up in PyPI and the Python Cookbook over the past few weeks. [As of Memorial Day weekend, when I first wrote this. I didn't post it until a week later. My"real job":http://laszlosystems.com is heating up :-]

The point is not that these recipes exist. There’s plenty of clever Perl preprocessor hacks; there’s some pretty amazing C++ template metaprogramming libraries ; and I bet if you scoured the web you could find this many recipes for Java reflection. (Well, I don’t bet very much on the Java part. I’ll buy you an ice cream at Bart’s. First taker only.)

The point is that these are flowing in fast and furious, and that they don’t take much code. The recipes below are from just two weeks. There’s half a dozen functional programming recipes, and a dozen MOP recipes; many of them are one-pagers. There were some real wizards in the Lisp community, but I don’t recall anything like this pace of development on top of the CLOS MOP.

Two weeks of MOP and reflection recipes:
* Overriding new for attribute initialization
* Named tuple items: here and here
* Cacheable value objects
* Thread-safe caching
* “Simpler super syntax”:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286195
* Constants
* Prolog syntax
* Named Singletons
* Handler design pattern
* Monostate design pattern
* Simplified Mutable Instances

Two weeks of functional programming recipes:
* izip inverse
* Batched iterables
* Windowed iterables
* Grouped iterables
* Coroutines
* Cartesian product

Web programming in August:
* The Whole Web in a Dictionary
* Webcleaner, a filtering HTTP proxy
* rxrdf, an RDF application stack
* milter, interface to sendmail milter API
* itools, uri, resources, handlers, i18n, and workflow tools
* libgmail, a binding for Google’s Gmail service
* naja, a download manager and a website grabber
* IRC bots: pyfibot and supybot
* linkchecker, not sure what this does

And hotswap finally adds object evolution (what “live edit”).

¹ Percentage gains are subjective and are based on simulated exploratory programming conditions. Actual productivity will vary with hardware speed, workplace environment, develpment habits, and problem complexity. Results reported to SEI indicate that the majority of projects with these estimates will achieve between 71% and 123% of Lisp productivity in single-person projects and between 83% and 159% in team projects.

I gave Miles a set of Logo programming problems:
* sv 3 draws a square divided vertically into three columns
* sh 4 draws a square divided horizontally into four rows
* svn 3 4 draws a square with three columns and four rows

(These are going to build towards some work with fractions, but he won’t know that unless he reads my web site. Hi, Miles!)

The first thing he did was place a slider and a button on the screen. The slider ranges from 1 to 10, and the button calls sv@ with the value of the slider. He used these to test the program while he wrote @sv, to quickly try it on different arguments without typing. When he added sh@ he added another button, and so on for @svn.

This looked to me like some sort of hybrid between test-driven development (with a unit testing framework or FIT), and using the command line. It’s more parameterizable than unit tests, but easier to fit into a development cycle than the command line.

What was really interesting, though, was that Extract Method wasn’t a new concept. The initial implementation of sh@ was copy-pasted from @sv, and svn was copy-pasted from both of them. I started on my DRY lecture — “see how this part of sh@ is the same as this part of @sv” — and he jumped the gun. “Oh, cool, you can use aliases?!” he exclaimed, before I even modified any code.

The analogy is between files in a directory and callees in a method. If directory could transclude its contents — list one of its children’s contents as its own — then the analogy would be exact.