Oliver Steele

Have you ever written a function that looks like this?

function requestProductDetails(id, k) {
  var value = gProductDetailsCache[id];
  if (value)
    k(value)
  else
    ajax.get('/product/'+id, function(data) {
      gProductDetailsCache[id] = data;
      k(data);
    });
}

requestProductDetails calls its callback with the product details, which are stored in a cache. Since it might need to request this information from the server, it has to “return” it by passing it to a callback; in order to present a uniform API whether or not the product is cached, it “returns” the data this way whether it came from the cache or not.

requestProductDetails is intended to be used this way:

requestProductDetails(id, function(details) {
  infoPanel.setDetails(id, details);
});
infoPanel.setName(id, gProductNames[id]);

(I gave infoPanel a somewhat silly API in order to demonstrate a point. The general pattern is that there’s some computation in the callback, and some other computation after the call.)

There’s a subtle problem in this code, which is that two different code paths run through it. In the cached case, infoPanel.setDetails is called before infoPanel.setName. In the uncached case (the first time through), it’s the other way around. If there’s a bug that causes setDetails to work only after setName has been called, you may well miss it during casual testing, because it will only trigger the second time you trigger the code — and once it does trigger, it will appear intermittently (especially if you have a more sophisticated cache), and be darned difficult to find.

I recommend this implementation of requestProductDetails instead. It makes the inside of the function more complex — and the setTimeout looks gratuitous — but it makes its outside simpler : requestProductDetailss callers are much easier to debug.

function requestProductDetails(id, k) {
  var value = gProductDetailsCache[id];
  if (value)
    setTimeout(function() { k(value) }, 10);
  else
    ajax.get('/product/'+id, function(data) {
      gProductDetailsCache[id] = data;
      k(data);
    });
}

The general principle here is if a function sometimes has to call its callback asynchronously, always call it asynchronously, to minimize the number of possible code paths through the application.

Here’s a potential JSSpec spec for Sequentially.trickle.map:

describe('Sequentially.trickle.map', {
  'should apply to all the elements': function() {
    Sequentially.trickle.map(
      ['a', 'b', 'c'],
       function(x) { return x + 1 },
       1,
       function(result) {
         value_of(result.join(',')).should_be('a1,b1,c1');
       });
    });
  }
});

This doesn’t work. The problem is that Sequentially.trickle.map is asynchronous (it defers most of its computation — including the invocation of the callback — via setTimeout). This means that should_be isn’t called until after the spec has returned. If it succeeds, this isn’t a problem, but if it fails, JSSpec can’t associate it with the failing spec — worse, JSSpec will have already have marked it successful.

Here’s the version that I actually used:

describe('Sequentially.trickle.map', {
  'should apply to all the elements': function() {
    withMockTimers(function() {
      Sequentially.trickle.map(
        ['a', 'b', 'c'],
         function(x) { return x + 1 },
         1,
         function(result) {
           value_of(result.join(',')).should_be('a1,b1,c1');
         });
      });
    }
});

withMockTimers temporarily replaces setTimeout and friends with its own deferred execution system, so that it can make sure to call them all before it returns. Get it here.

Limitations

This approach has its limits. It doesn’t mock new Date to pretend that more time has passed, so whether it works or not will depend on how your code uses the timers (if it keeps an interval running or re-submitting a timeout until an amount of time measured on new Date has passed, it will probably get a “script running slowly” error). And, I don’t know how kosher it is to replace setTimeout — this let me test against Firefox 2.0 and Safari 3.1; I haven’t tried on Opera and MSIE. Nonetheless, it got me what I wanted here — unit tests for the new methods in Sequentially.

(I’ve got a more involved implementation that patches the test suite to run callbacks within an emulation of the dynamic scope of the original test function, but it’s tricky, and I haven’t got it integrated with JSSpec — or any other browser JavaScript implementation — yet.)

What’s wrong with this function? (Hint: it’s meant to execute periodically on a JavaScript page.)

function updateExpirationText() {
  var now = new Date;
  products.forEach(function(item) {
    var expiresDate = item.expiresDate || Date.parse(item.expires),
        remaining = expiresDate - now,
        text = remaining < 0 ? 'expired' : msToDuration(remaining);
    $('item-' + item.id + ' .time-remaining').text(remaining);
  });
}

It’s a trick question. Maybe nothing’s wrong. But if products can get very long, or if the msToDuration is very slow, you’ve locked up the UI for a long time. At best, this makes for sluggish response; at worst, the page that contains this will trigger a “script running slowly” error, and the user will likely abort all the JavaScript on the page.

If this computation only needs to run once, and when (or before) the page loads, you can do it on the server. But often a computation depends on some aspect of the client state, that isn’t known when the page is requested. In this example, the computation depends on the current time (and the current time keeps changing). In another case, the computation might depend upon the values of some controls or other widgets on the page — if we’ve gone all AJAXy, and want to show the user an instant response, even if that means some client-side computation.

Here’s an alternative to the function above, that doesn’t lock up the page. It uses Sequentially.trickle.forEach, a new function in Sequentially. This function walks its second argument over some span of the first argument — up until 250ms has passed, in this case — and then sleeps for a frame (via setTimeout) before waking up to walk over the next span, until all is done. This gives time back to the browser (and to other setTimeout and setInterval threads), and avoids the “script running slowly” error. Note the one-line change: "products.forEach("” becomes "Sequentially.trickle.forEach(products,".

function updateExpirationText() {
  var now = new Date;
  Sequentially.trickle.forEach(products, function(item) {
    var expiresDate = item.expiresDate || Date.parse(item.expires),
        remaining = expiresDate - now,
        text = remaining < 0 ? 'expired' : msToDuration(remaining);
    $('item-' + item.id + ' .time-remaining').text(remaining);
  }, 250);
}

Sometimes you need to run some code after the iteration is done. In other words, sometimes you need to transform a function that looks like this:

  var startTime = new Date;
  array.forEach(function(item) { ... });
  console.info(new Date - startTime, 'elapsed');

(Here, the code that runs after the iteration just reports how long the iteration took.)

You can do that with a continuation function (or callback), the same as you would with an AJAX request:

  var startTime = new Date;
  Sequentially.trickle.forEach(array, function(item) { ... }, 250, k);
  function k() {
    console.info(new Date - startTime, 'elapsed');
  }

JavaScript being lexically scoped, you can refer to all the same variables from a nested function (@k@).

There’s a Sequentially.trickle.map too. Since the return value can’t contain the function application results yet when the function returns, you have to get them back from the callback. Before (the synchronous version):

  var results = array.map(function(item) { ... });
  console.info('Results:', results);

and after (asynchronous):

  var startTime = new Date;
  Sequentially.trickle.map(array, function(item) { ... }, 250, k);
  function k(results) {
    console.info('Results:', results);
  }

The DB Content Rails plugin adds tasks to save and restore database content.

Usage

-- dump the development database to db/archive/development-content.sql.gz
rake db:content:dump

-- load the dumped database, and apply any necessary migrations
$ rake db:content:load

-- dump the production database to db/archive/production-content.sql.gz
$ RAILS_ENV=production rake db:content:dump

-- save the development database to db/archive/{timestamp}.sql.gz
$ rake db:content:save

-- save the (compressed) database to my-data.sql.gz
$ rake db:content:save FILE=my-data.sql.gz

-- save the (uncompressed) database to my-data.sql
$ rake db:content:save FILE=my-data.sql

-- load the database from my-data.sql
$ rake db:content:load FILE=my-data.sql

Tasks

rake db:content:archive

Saves a timestamped database to db/archive/{timestamp}.sql.gz.

rake db:content:dump

Dumps the database to FILE or db/{RAILS_ENV}-content.sql.gz. If FILE ends in .gz, the file is compressed.

rake db:content:load

Loads the database from FILE or db/{RAILS_ENV}-content.sql.gz, and migrates it to the current schema version. If FILE ends in .gz, the file is piped through gunzip.

Installation

git clone git://github.com/osteele/db_content.git vendor/plugins/db_content

If you’re running off Edge Rails (or, presumably, Rails > 2.0.2), you should be able to do this instead:

script/plugin install git://github.com/osteele/db_content

Limitations

The plugin works only with the MySQL databases. (It adds methods to the Mysql adaptor; see the source.) The gzip option probably only works on *nix (MacOS, Linux, etc.).

JCON (the JavaScript Conformance gem) tests JSON values against ECMAScript 4.0-style type definitions
(PDF) such as string?, (int, boolean), or [string, (int, boolean), {x:double, y:double}?].

Usage

type = JCON::parse "[string, int]"
type.contains?(['a', 1])     # => true
type.contains?(['a', 'b'])   # => false
type.contains?(['a', 1, 2])  # => true

JCON also defines an RSpec matcher, conforms_to_js:

[1, 'xyzzy'].should conform_to_js('[int, string]')
[1, 2, 'xyzzy'].should_not conform_to_js('[int, string]')  # 2 isn't a string
{:x => 1}.should conform_to_js('{x: int}')

Use JCON together with the JavaScript Fu Rails plugin to test the argument values to functions in generated JavaScript:

# this will succeed if e.g. response contains a script tag that includes
#   fn("id", {x:1, y:2}, true)
response.should call_js('fn') do |args|
  args[0].should conform_to_js('string')
  args[1].should conform_to_js('{x:int, y:int}')
  args[2].should conform_to_js('boolean')
  # or:
  args.should conform_to_js('[string, {x:int, y:int}, boolean]')
end

Whence

Github for the sources.

Rubyforge for docs.

gem install jcon to install.

License and version

MIT License, of course.

JCON is at version 0.1 because it’s just a few days old and I had to guess about the ECMAScript 4.0 type syntax from the examples in the overview. I can’t imagine that I got everything right.

Three small libraries, that I carry with me from project to project:

Fluently — Construction Kit for Chainable Methods

With Fluently, you can do this:

    var o = Fluently.make(function(define) {
      define('fn1', function() {console.info('called fn1')});
      define('fn2', function() {console.info('called fn2')});
      define('fn3', function() {return 3});
    });

to define an object with chained methods, that can be invoked thus:

  o.fn1().fn2() // calls fn1 and then fn2
  o.fn2().fn1() // calls fn2 and then fn1
  o.fn1().fn3() // returns 3 (an explicit 'return' breaks the chain)

You can also define modifiers, and aliases:

    var o = Fluently.make(function(define) {
      define('fn1', function() {console.info('called fn1')});
      define('fn2', function() {console.info('called fn2')});
      define.empty('and');
      define.alias('fn3', 'fn1');
      define.modifier('not');
    });
 
  o.fn3(); // same as o.fn1()
  o.fn1().and.fn2() // same as o.fn1().fn2()
  o.fn1().and.not.fn2() // options.not is set when fn2 is called

I used this to build a mock and spec construction kit. I don’t use Fluently to define the mocks; I use it to define the methods that define the mocks. Doing all this in one library made my head hurt, so I factored this part of it out.

Git Fluently from here.

MOP JS

MOP JS defines utilities for JavaScript metaprogramming. You don’t think you need it until you try asynchronous programming, where some methods don’t have enough information to operate until the response to another method’s asynchronous request have returned.

  MOP.delegate(target, propertyName, methods)

For each name in methods, defines a method on target with this
name, that delegates to the method of the propertyName property
of target with the same name.

  new MOP.MethodReplacer(object, methods)

When a new MethodReplacer is constructed, it replaces each method
on object by the method in methods with the same key value, if
such a method exists. A MethodReplacer has a single method,
restore, which restores each method to its pre-replacement
value.

  new MOP.QueueBall(object, methodNames)

When a new QueueBall is constructed, it replaces each method named
by methodNames with a method that enqueues the method call (the
name of the method and its arguments). A QueueBall has a single
method, replayMethodCalls, which plays back the method calls and
restores the methods.

  MOP.withMethodOverridesCallback(object, methods, fn)

Calls fn on object, within a dynamic scope within which the
methods in methods have temporarily replaced the like-named
methods on object. The scope is terminated by the argument to
the call to fn; this argument should be treated as a
continuation, and restores the methods.

  MOP.withDeferredMethods(object, methodNames, fn)

Calls fn on object, within a dynamic scope within which the
methods in methodNames have been enqueued. The scope is
terminated by the argument to the call to fn; this argument
should be treated as a continuation, and ends the queue, replaying
the methods.

See the specs for examples; git MOP JS from here.

Collections JS

Finally, the Collections library defines framework-independent JavaScript collection methods, for use in browser JavaScript and in ActionScript / OpenLaszlo. There are many libraries like this; this one is mine.

The Array and String methods extend the class prototype; the Hash methods use a proxying wrapper to avoid prototype pollution. The methods with the same names as the ECMAScript 1.6+ extensions have the same spec as those; the ones with the same name as prototype extensions have the same spec as those in the Prototype library; and there’s a few odds and ends such as String#capitalize.

I use this when I don’t want the overhead of Prototype, or want to use these functions in an environment that Prototype doesn’t run on, such as OpenLaszlo. It has some overlap with Functional, but isn’t nearly so radical — this can be an advantage.

Git Collections JS from here.

JavaScript Fu extends Rails with a few facilities to better integrate JavaScript into Rails development:

1. The notes and statistics rake tasks compass JavaScript files in the public/javascript directory:

$ rake notes
public/javascripts/controls.js:
  * [782] [TODO] improve sanity check

$ rake stats
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
[...]
| JavaScript           |  7287 |  6322 |       0 |       0 |   0 |     0 |
[...]

2. The call_js RSpec matcher asserts that a string or response contains a script tag, that contains JavaScript that calls the named function or method:

response.should call_js('fn')
response.should call_js('fn(true)')
response.should call_js('gApp.setup')

If you pass a block to call_js, it’s called back with the argument list, parsed as though it were a JSON array:

# matches <script>fn(1, 'aString', {x:10,y:20})< /script>
response.should call_js('fn') do |args|
  args[0].should == 1
  args[1].should == 'aString'
  args[2].should == {:x => 10, :y => 20}
end

Use this with jcon to test for type conformance, using ECMAScript 4.0 type definitions. (Well, you can’t use it with jcon yet, because I haven’t released it — this is just a teaser. But you can peek.)

response.should call_js('fn') do |args|
  args[0].should conform_to_js('[Array, (int, boolean)]')
  args[1].should conform_to_js('{x: double, y: double}')
  # or just:
  args.should conform_to_js('[[Array, (int, boolean)], {x: double, y: double}]')
end

3. The page.onload page generator method generates code that executes the content
of the block upon the completion of page load:

page.onload do
  page.call alert', 'page loaded!'
end

These lines generate one of these (depending on whether the jRails plugin has been loaded):

Event.observe("window", "load", function() { alert("page loaded!"); });

$(document).ready(function() { alert("page loaded!"); });

Gitting It

JavaScript Fu is hosted on git. If you have git installed, you can clone it into your Rails directory thus:

git clone git://github.com/osteele/javascript_fu.git vendor/plugins/javascript_fu

If you’re running off Edge Rails (or, presumably, Rails > 2.0.2), you should be able to do this instead:

script/plugin install git://github.com/osteele/javascript_fu

Otherwise, you can simply download the tarball from here.

Update: changed the conform_to_js example so that it actually works with the (albeit unreleased) plugin..

I’ve updated my OpenLaszlo utility grab-bag to make browser <-> applet communication even easier. How easy?

Proxies

Put this in your browser JavaScript:

var gObject = {
  f: function() { console.info('gObject.f', arguments) },
  g: function() { console.info('gObject.g', arguments) }
};

And this in an OpenLaszlo applet:

var gObject = FlashBridge.createRemoteProxy('gObject', ['f', 'g']);
gObject.f(1, 2);
gObject.g(3);

When you run the applet code, it prints this to the browser console:

gObject.f [1,2]
gObject.g [3]

That’s right, Flash is invoking the function calls, but they’re executing in the browser.

Now switch these around — put the first block in the applet, and the second block in the browser JavaScript — and it still runs the same way, except that it’s the browser that invokes the functions, and they run in the applet (and print to the OpenLaszlo debug console, if the applet was compiled with debugging on).

(By the way, the full sources for the examples are here.)

Return Values

Callbacks, or continuations for return values, make it easy for the applet to operate on the return value from a call into the browser, even though these calls are asynchronous.

Put this in the browser:

var gService = {
  add: function(a, b) {
    logCall('gBrowserObject.add', arguments);
    return a+b;
  },
  error: function(msg) {
    logCall('gBrowserObject.error', arguments);
    throw msg;
  }
};

And this in the applet:

gBrowserObject.add(1, 2).onreturn(function(value) {
  console.info('1 + 2 -> ' + value);
});
gBrowserObject.error('error msg').onexception(function(value) {
  console.info('error !> ' + value);
});

The argument to onreturn is called (asynchronously) with the return value. The argument to onexception is called with the message from the exception, if an exception occurred.

Callbacks, unlike proxies, only work one direction — for calls from the applet to the browser. That’s not for a technical reason — I’ve just only needed it one direction so far.

Call Storage

Browser code can call into the applet even if the applet hasn’t initialized yet, and vice versa.

To implement this, each side of the bridge stores calls (and return value handlers) in a mailbox until it hears back that the other side has loaded. Once this happens, the mailboxes are flushed and the remote call methods switch to direct invocation.

This works around a couple of race conditions. First, the applet won’t generally have run its initialization code by the time the browser receives its load event, so a naive implementation of the bridge wouldn’t allow the browser to make calls into the applet until the browser had heard back that the applet had loaded — which is hard to detect. (It isn’t enough to wait for the object’s onload event, because this can trigger before the first frame of the movie plays, so the applet may still not have initialized enough to receive messages.) Conversely, depending on your page organization and initialization raindance, the applet might load before page side has registered — so the applet couldn’t call into the page until an unknown time.

Security Implications

FlashBridge, by default, allows the browser to call anything sitting in the applet, and vice versa. This increases the attack surface of your application, because it allows an embedded Flash applet to invoke any part of it. This means that an XSS can tunnel through your applet to gain access to any site with a crossdomain.xml file that allows your applet to connect to it — something that XSS on a pure JavaScript page can’t do.

It you prefer not to audit your application against this, you can call FlashBridge.secure to prevent it from accepting arbitrary calls, and then FlashBridge.register to register callins.

There’s no lockdown facility in the other direction — to lock down the browser JavaScript against calls from the Flash application. That’s because it’s trivial for a Flash application to invoke arbitrary JavaScript in the browser context — in fact, that’s how the applet -> browser communication is implemented, and if that were secured at the FlashBridge layer, the vulnerability would still be accessible one layer down.

Gitting It

All this is in the LzOsUtils project on GitHub, with examples here. Download it via the Download button, clone it via git clone git://github.com/osteele/lzosutils.git, or add it as a submodule to an existing git repo via git add submodule git://github.com/osteele/lzosutils.git.

FlashBridge is written for OpenLaszlo, but would probably run in straight Flash too. And it uses my own funky alternative to ExternalInterface for calling from Flash to the browser (since the built-in API is seriously broken), but it could be ported to run on top of Dojo or something pretty easily.