Python really is becoming lisp. With the type/class unification, decorators, and generator expressions, it’s jumped from 80 percent of Common Lisp + CLOS to 90 percent1, and for web tasks the web programming libraries often make up the difference.
I’ve listed below some of the functional and metaprogramming recipes and packages that have shown up in PyPI and the Python Cookbook over the past few weeks. [As of Memorial Day weekend, when I first wrote this. I didn't post it until a week later. My"real job":http://laszlosystems.com is heating up :-]
The point is not that these recipes exist. There’s plenty of clever Perl preprocessor hacks; there’s some pretty amazing C++ template metaprogramming libraries ; and I bet if you scoured the web you could find this many recipes for Java reflection. (Well, I don’t bet very much on the Java part. I’ll buy you an ice cream at Bart’s. First taker only.)
The point is that these are flowing in fast and furious, and that they don’t take much code. The recipes below are from just two weeks. There’s half a dozen functional programming recipes, and a dozen MOP recipes; many of them are one-pagers. There were some real wizards in the Lisp community, but I don’t recall anything like this pace of development on top of the CLOS MOP.
Two weeks of MOP and reflection recipes:
* Overriding new for attribute initialization
* Named tuple items: here and here
* Cacheable value objects
* Thread-safe caching
* “Simpler super syntax”:http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286195
* Constants
* Prolog syntax
* Named Singletons
* Handler design pattern
* Monostate design pattern
* Simplified Mutable Instances
Two weeks of functional programming recipes:
* izip inverse
* Batched iterables
* Windowed iterables
* Grouped iterables
* Coroutines
* Cartesian product
Web programming in August:
* The Whole Web in a Dictionary
* Webcleaner, a filtering HTTP proxy
* rxrdf, an RDF application stack
* milter, interface to sendmail milter API
* itools, uri, resources, handlers, i18n, and workflow tools
* libgmail, a binding for Google’s Gmail service
* naja, a download manager and a website grabber
* IRC bots: pyfibot and supybot
* linkchecker, not sure what this does 
And hotswap finally adds object evolution (what “live edit”).
1 Percentage gains are subjective and are based on simulated exploratory programming conditions. Actual productivity will vary with hardware speed, workplace environment, develpment habits, and problem complexity. Results reported to SEI indicate that the majority of projects with these estimates will achieve between 71% and 123% of Lisp productivity in single-person projects and between 83% and 159% in team projects.
Oliver,
You’re way smarter than me on topics like this. But its been bothering me a little to see all these new features become part of python. While I think overall they have done a good job of adding features while maintaining the cleanliness of the language, it seems like now we are starting to get too many possible paradigms in one coding environment. Power in a language is usually always good for writing code but not always for reading it. C++’s biggest weakness imho is that there are so many different ways to (u|abu)se templates, references and the like that source code from different projects doesn’t even seem like its written in the same language.
At Cyan we had our orignal build scripts written in perl and we replaced them all with python when we realized that no one had a clue how they worked. Even the people who had never used python could figure out the python code.
So I worry that to the extent python can work like (c/ada/modula) AND (lisp/scheme/etc) we might end up with code that is more obscure.
What do you think?
Python is great – a wonderful language and environment.
But it seems to me that until it has macros it will
never really be as nice as Lisp.
Hi, Brice, and thanks!
Your comment was very interesting, and raised a lot of issues — especially the difference between readability and writability, and the difference (which you hint at) between features and paradigms. Those are worthy of a longer answer; I’m not going to try at all in this comment.
Two points:
1. Features increase the range of readability. Each feature makes it possible to write both more-readable and less-readable programs than one could write without it. Consider Python
_&2d;new__. A program with__new__strewn throughout is less readable than it otherwise would be. A program that uses a named tuple class (which can be implemented with a single__new__) can be more readable than one that doesn’t — especially if the one that doesn’t uses what is basically a source-level expansion of the named tuple implementation, each place a named tuple would be appropriate.Consider virtual functions (or method overriding, in an OO language). Virtual functions make it easier to write readable programs, because they move the switch statements or function pointers that would otherwise be necessary out of the problem domain code. (In fact, virtual functions are a way of moving this dispatch code out of the program source entirely, to the language implementation). But virtual functions also make it easier to write unreadable programs — it can be very hard to find which implementation of a method is called under which circumstances.
I’d rather have the possibility of working with readable programs than the certainty of avoiding unreadable programs, but it depends on who you’re working with and where the code you read is coming from.
2. “source code from different projects doesn’t seem like it’s written in the same language”: Maybe this is a good thing! Consider the case where the best language for one project is Prolog and the best language for another project is Haskell. Then there’s two choices: Use completely different languages for different projects, or using the wrong language for some projects because it’s the right language for others. Language features give you a third choice: use different libraries and extensions for different projects. It’s not clear that seeming like different projects are in different languages is as bad as the alternatives: being in different languages, or being in the wrong language!
I think it comes down to having the right number of tools. Each one you add to your toolbox has to be worth the weight of carrying it around. I wouldn’t want to give up on virtual functions, operator overloading or constructor/destructors in c++. But that doesn’t mean that each new tool we add makes the situation better. I think templates are great for making generic collection classes but I have also seen them used to make code that is obscure and almost impossible to debug through. Sure, that’s the fault of the coder and not the tool. Guns don’t kill people, People do. However, I think you have to be careful to avoid a point where you have allowed too many different tools into your project for the members of the team to use and play nicely with. Establishing best practices for using any tool/feature can take a significant amount of time in any group. Sometimes I joke that all the productivity gain from using things like C++ is lost in the time spent debating about how to use these features. Of course, might be making an argument for smaller teams as much as for holding the line on language features…
[In order to tone down the amount of hostile commentary, I've decided to require email addresses for comments, and I've edited out the ad hominem attacks in the following comment. If the author would like to resubmit the comment with an email address, I'll replace the edited version by the original.]
[Ad-hominem comment deleted.] Just because Python has a lambda operator, it does /not/ mean it supports functional programming.
And why do you think little recipes are impressive?
I think that something like creating a free CLIM, now /that’s/ impressive.
Python midshare is huge compared to Lisp. [Ad-hominem comment deleted.] It means O’Reilly supports you, but so what?
Please go see what the Common Lisp community is doing at http://www.cliki.net.
Besides, sorry, but no CL macros. No code that reads code in the language code was written. Data is code – code is data. That’s Lisp.
Brice: I think you’ve hit the nail on the head with your comment that different tools are appropriate for different teams (and for different projects, and different contexts — I wouldn’t have written a rendering engine in Python, or PyWordnet in assembly).
Still, it would be nice if the different tools for different teams could be hosted in one language.
If you start with a superset of the features you need, subsetting it requires policies, conventions, and, ideally, tool support — something like Checkstyle or Macker that can be run as a supplement to the compiler to create a composite compiler that compiles a subset of the full language, or like the language devels in DrScheme.
If you start with a subset of the features you need, you end up writing what would have been the compiler expansion of these features into your source code, using a precompiler, or implementing these features with metaobject programming. But you can only do the last if the language has metaobject programming features to begin with, which reduces this problem to the previous case, where you want the language to have these features, but you want to restrict their use, or the use of the features they create, to certain libraries. Something like Macker for language features seems like it would be appropriate here.
Chris Dean and JJC: What sorts of things do you use macros for in Lisp?
I used to use macros for a several different things in Lisp. Many of them I switched to closures or multimethods for even when I still did most of my work in Lisp. Other uses of macros (for example, reader macros) rarely turned out to be worth the trouble of maintaining or integrating with other libraries.
The use of macros that I found the most powerful was to define new top-level defining forms, but metaprogramming works pretty well for this, and so does having a literal syntax for finite maps (or “dicts” or “hashes”. I found Lisp’s MAKE-HASHTABLE a poor substitute for this.
The other use of macros in my work was to define new binding forms such as with-open-file or with-graphics-context. Python doesn’t have a good replacement for this.
JJC: “Just because Python has a lambda operator, it does /not/ mean it supports functional programming.” What did you have in mind? (BTW, Python has had lambda for years. What it’s added lately are Haskell-style list comprehensions, generators, and libraries such as itertools.)
wow! nice article. thanks for sharing.
I agree, python is becoming more like lisp every day. I think that python is a great platform but it needs some things to work better for ME. Everyone has their own needs and uses.