Songbird is an iTunes-style music player that contains an integrated web browser. Songbird detects rich media on pages you visit and integrates them into itself in a number of ways: making them navigable through a “web playlist” reflecting the current page, remembering them in a “web media history” that you can browse and search, and, finally, providing a bridge between the music player and the web browser itself that allows web pages to interact with the playing song and the user’s music library.

Why does Songbird matter?

Songbird is to iTunes what Firefox is to Internet Explorer: it provides an open source alternative to an application built as part of a vendor’s extremely successful lock-in strategy. And, following in Firefox’s footsteps, Songbird aims to introduce a raft of technical innovations into a stagnant space.

The bird. Shown here in a rare non-farting moment.

While iTunes (and especially the iTunes store) revolutionized the music industry on first release, its rate of innovation is restricted by Apple’s ability to imagine and implement new features and, even more tightly, by the constricting legal ties the company has woven with the media conglomerates that provide its content. Songbird, on the other hand, is trying to offer an open platform for anyone to build their own innovative applications or competing business models. If they succeed, these lowered boundaries to participation will mean that Songbird’s feature count and available content options will zoom past iTunes’, inevitably bringing along an ever-growing number of users and, eventually, breaking Apple’s monopoly and forcing them to scramble to compete — just as Firefox has done with Internet Explorer.

What can I do with it?

So, in what ways is Songbird open? What can you do with it as a developer? Well, just like Firefox, Songbird is open in a number of ways. First, the code for the player itself is available for download and licensed under the GPL. This means you can modify it, use it in your own applications, print it on your school binder, whatever you want. Secondly, Songbird has a plug-in architecture against which you can develop. The Songbird team calls applications written on this architecture Add-ons and they have access to everything from the player’s appearance (for building skins) to the actual data and UI of the application. Finally, and most importantly for us here, Songbird provides web pages with a javascript API that allows them access to the app’s sophisticated playback interface and, with their permission, the user’s media library and custom playlists. Since this last is by far the easiest to get started with, I’ll spend the rest of this post focusing specifically on it.

Setting up a Songbird Javascript Development Environment

One of the hardest parts of starting development in any new environment is getting all the pieces in place to successfully write and run your first line of code. This is especially true in complex runtime environments like javascript hosted in a web page running against APIs in a rapidly developing client. Here are the steps I followed to get there:

• Download a nightly build. At the time of this writing, 0.6 was the latest hotness, but things are moving rapidly.
• Get Firebug lite. Like most Javascript developers, I’ve become increasingly dependent on Firebug since its introduction. Being able to test out code in its interactive shell and navigate the resulting objects is priceless. Heck, even just being able to log information to the console rather than having to dismiss an endless parade of alert statements is worth the price of admission alone. Unfortunately, because of Songbird’s rapid pace of development, there isn’t currently a version of Firebug that is compatible with 0.6 (if you come from the future, visit Firebug’s page on the Songbird add-on site to see if things have changed). Thankfully, there’s another version of Firebug that can reside in the DOM of any given page. It doesn’t have all the niceties we’d want (like object navigation) and it’s access to the Songbird-specific javascript APIs we’ll be spending our time with can be a bit…err…intermittent, but at least it provides a console for logging and some ability to inspect our objects. Once you’ve downloaded firebug, stick the directory somewhere in your project and create an HTML file with the following in the head tag:
  
<script language="javascript" type="text/javascript"
src="/path/to/firebug/firebug.js"></script>

  You should also set the debug attribute in your opening html tag to “true”. Now, when you open your page in Songbird, it’ll have a mini version of Firebug running at the bottom of its window. You can log to it and interact with objects (though limits and quirks will abound).

Songbird’s Model of the World

Songbird’s viewer window explained (full-size version).

The purpose of the Songbird webpage API is to provide the javascript running on a particular page with access to the media-player side of the application. Through it your page can track changes in the currently playing song, manage the “web playlist” (a small iTunes-style song browser that shows up underneath the current web page), and access song listings and metadata for the user’s local music library, including reading and creating local playlists — you can even download songs directly to the user’s computer. Pretty much the only thing you can’t do is delete files.

Songbird presents three main abstractions for working with this functionality: Libraries, Media Lists, and Media Items. Let’s start at the bottom with Media Items and work our way up.

Media Items == Songs

A Media Item is a single song or video in your Songbird library (at this point, I’m just speculating about the video part, though the generic name is certainly tantalizing and I know that Songbird does have some form of video support; however, I’ve only worked with songs, and from here on I’ll assume that a Media Item is really a song). Media Items provide access to a raft of metadata such as “trackName”, “artistName”, “duration”, etc. See the full list of Media Item properties for everything that Songbird makes available. You can read these attributes and set them. Interestingly, metadata properties are namespaced so you can create your own solely for use by your application, or even attempt to get a community of developers to standardize around new attributes for a particular domain.

One important thing about Media Items is that they don’t necessarily have to correspond to a song with a locally available media file. Just as your iTunes library can contain songs whose actual mp3 files are elsewhere (like on an external backup drive), Media Items can be associated with files that are not currently available or with no file at all. Unlike iTunes, Media Items in Songbird can also be associated URLs that point at online mp3s.

Media Lists == Playlists

The next step up the ladder is Media Lists. These, as you might expect, are collections of Media Items, i.e. playlists. They have a name which is a handle for finding them (and also what shows up in the app’s sidebar if you add them to the Main Library). You can find out which Media Items belong to a Media List and you can add and remove them. Media Lists are ordered stacks, so there’s some functionality for managing the position, uniqueness, and suchlike of individual Media Items and for iterating through the full set.

Libraries

At the top of this whole shebang are Libraries. Libraries represent the permissions and interaction scope associated with a Media List or group of Media Lists, i.e. in which part of the application they’ll be visible and which chunks of code will have access to them. There are three types of Libraries in Songbird — the Main Library, the Web Library, and the Site Library — and one other entity (the “Web Playlist” mentioned above) that sits in the same spot as a Library and behaves almost exactly like a Library, but isn’t technically a Library. Things will get clearer quickly if we talk about the purpose of each type of Library, so let’s dive right into that.

Libraries: the Main Library

First up, we’ve got the Main Library. The Main Library contains all the Media Items on the user’s computer. Remember, since Songbird can provision audio for Media Items from public URLs and can include local Media Items for which the file is not currently available (the Missing Backup Drive scenario), this is not limited to files actually “physically” present. Reasons to talk to the Main Library include: wanting to create a new playlist on the user’s computer that will be available after they browse away from your site, wanting to see what songs the user already has in order to make content recommendations, getting a list of all the local playlists a user has created and what songs are in them, etc. By default websites are blocked from accessing the Main Library at all unless the user has changed their “Web Integration” preferences to allow general access or to allow it for your website in particular. If you attempt to access the Main Library of a user who has limited access to their Main Library, Songbird will prompt them to permit or deny the action; the javascript API includes callbacks to make your code aware of the user’s response to this prompt.

Libraries: the Web Library

Next up, we’ve got the Web Library. The Web Library contains all the Media Items Songbird has extracted from the webpages the user has visited. When you navigate to a URL that contains links to music files, Songbird discovers these links and gives you an iTunes style playlist representation of them at the bottom of your window (the Web Playlist). The Web Library is everything that’s ever shown up in this playlist. It’s every song from every webpage the user has ever visited within Songbird. This list of songs is made available to the user as the “Web Media History” and is an incredibly useful feature — if you spend any significant amount of time reading mp3 blogs or browsing band sites your Web Library will rapidly become populated with most of the music you’ve ever heard about, even in passing. This makes it an extremely rich collection to search against and idly browse. I can only imagine all the cool apps you could build if you had access to the entire mp3 blog browsing history of your users…obviously recommendations (for both music and blogs) come to mind, but you could probably also build some kind of very kick ass client-resident distributed spidering operation, or something else extremely neat that I can’t think of off the top of my head. As you’d expect, the Web Library is under exactly the same access restrictions as the Main Library.

Libraries: the Web Playlist

Now we start to move into the Libraries that belong more to our individual websites than to the users themselves and over which we therefore have more control. The first and most visible of these is the Web Playlist. In the normal course of events (i.e. while browsing sites that aren’t cool enough to use the Songbird API), the Web Player simply reflects any mp3s or other media files Songbird happened to find on the page. However, with the API we can control just about every aspect of the Web Playlist from the tracks that populate it to what pieces of data show up in each of its columns. I mentioned above that the Web Playist is not, strictly speaking, a Library. This means that it doesn’t offer quite the same API as other Libraries — it only contains a single Media List; it doesn’t have its own access controls separated from those of the Media List assigned to it; etc. — however, I include it in this list because, in the javascript, it lives at the root of the ‘songbird’ namespace as a sibling to the other Libraries and it offers an API that is largely (though not entirely) parallel to the others.

Libraries: Site Libraries

Lastly, we come to Site Libraries. Unlike the other Libraries we’ve covered so far, Site Libraries don’t directly control the display of tracks to the user. Instead, they are simply an application-specific namespace for storing Media Lists. In other words, they let your javascript build Media Lists that are available only to users visiting particular pages on your domain (controlled via a scoping mechanism much like the traditional robots.txt syntax). Once you’ve got a Media List safely inside your own private Site Library, you can then display it in a Web Playlist, save it to the user’s Main Library, or simply keep it invisible and use it to swap out the currently playing song. In fact, if you don’t actually use a Site Library to manipulate any part of Songbird’s UI (for example if they’re just a convenient way to store your app’s internal data), you don’t even need the user’s permission to create them. Site Libraries are a little harder to grok than the other varieties since they lack visible representation, but you’ll quickly get used to working with them as they’re the main place you’ll be storing all the Media Lists you build. If you’re still confused as you come away from this paragraph, you can just think of Site Libraries as a private place to store the Media Lists you compose without having to display them to the user in any way.

Some code samples

Let’s make this whirlwind conceptual tour more concrete by looking at some examples. I’ll show you the code to accomplish some basic tasks and then explain it in terms of the concepts I’ve outlined above.

Putting songs in the Web Playlist


songbird.setSiteScope("", "/");
var library = songbird.siteLibrary;
var myMediaItem = songbird.siteLibrary.createMediaItem("http://www.example.com/track.mp3");
myMediaList.add(myMediaItem);
songbird.webPlaylist.mediaList = myMediaList;


I cribbed this example from the Songbird wiki. It simply creates a media item from an mp3 URL, assigns the item to a media list that is scoped to the domain of the current page, and tells songbird to display the media list in the web playlist. The song won’t actually appear anywhere visible to the user until the last line of code in this sample. Once the web playlist receives the media item, it’ll scan the URL for metadata and fill in the track display based on what it finds.

Set a property on a media item


var myMediaItem = songbird.siteLibrary.createMediaItem("http://www.example.com/track.mp3");
myMediaItem.setProperty( "http://songbirdnest.com/data/1.0#artistName", "The Rolling Stones" );


If you set a property before adding the media item to a list and displaying it, Songbird will trust your metadata assignment and not override it based on the results it finds on the attached mp3. There are parallel methods for getting the values of any already set properties. You can set properties on media lists in the same way.

Download a media items’ file to the user’s computer


var library = songbird.siteLibrary("", "");
var mediaItem = library.createMediaItem("http://path/to/item.mp3");
songbird.downloadItem(mediaItem);


In addition to providing access to the various libraries, the songbird object allows you to do a whole raft of useful things such as download an mp3 from the web into a local media item (as shown here), getting access to information about the currently playing track, controlling which track is playing, etc.

Get all the local playlists


var mediaLists = songbird.mainLibrary.getPlaylists();
var results = [];
while(mediaLists.hasMoreElements()) {
var mediaList = mediaLists.getNext();
results.push(mediaList);
}
return results;
};


This code asks the main library for a list of its playlists. When that first line of code runs, requesting access to the main library for the first time, Songbird will prompt the user for permission. In order to iterate through the results of getPlaylists, we have to call the special enumeration helper, hasMorePlaylists. This is a common pattern with collections of Songbird objects, there’s often custom enumeration code that be somewhat counter-intuitive compared to standard array iteration.

Additional Resources

These examples only scratch the surface of what you can do with songbird. For more examples and more documentation, visit the Songbird webpage-api docs and the Songbird wiki entry on Webpage API integration.

The Songbird team hangs out at #songbird on irc.mozilla.org. They are quite friendly and open to helping out if you’re confused or run into issues. I’d also like to thank them specifically for answering my large raft of questions as I was learning this API. This guide wouldn’t have been possible without them.

Tagged: songbird, music, player, javascript, api, open, source

An intro to routing and controllers.

I wrote recently about getting started with Merb. Yesterday, after having refer to my own post to remember how it all worked, I started on a new Merb app as part of the Great Grabb.it Rewrite of 2008. I very quickly ran into another area of Merb that’s a bit thin on documentation: Controllers. After some googling, some fiddling, and some brainstorming in the #merb irc channel I managed to figure out enough of the basics to get things up and running.

Right now, Merb’s sweet spot seems to be as an API server, which is exactly how we’re planning on using it. This usage means that while there might be a little bit of model integration for access control, most of the job is in routing URLs and data formatting. That means we’ll be spending most of our effort in config/router.rb and in our controllers. Specifically, we’ll need to know a lot about how Merb provides responses in various formats depending on the request url, i.e. it’s equivalent to Rails’ respond_to method. Let’s start by laying out our requirements and then I’ll show you how I translated them into Merbish.

For this app, I needed to take URLs within a namespace and route them to a single controller setting various params based on the rest of the request path. In other words, I wanted to provide URLs like:

http://myapp.com/music/artist.jspf

and



where the first url would give you a playlist of all of an artist’s tracks in JSPF (Javascript Shareable Playlist Format) and the second would provide an mp3 of the song with the given title by the given artist.

So, the first thing I needed was a series of routes that would recognize all URLS starting with /music and then grab the path after the first slash into an :artist param and the path after the second slash into a :title param. Finally, it would need to look for a format after the dot and pass that through as well if available.

It turns out the best way to do this in Merb is with nested routes, thusly (inside of the Merb::Router.prepare block in config/router.rb):

r.match('/music') do |music|
music.match("/:artist.:format").to(:controller => 'music', :action => 'artist')
music.match("/:artist/:title.:format").to(:controller => 'music', :action => 'track')
end


This is pretty self explanatory if you’re familiar with Rails routes, though it is probably worth noting that an early version of this I tried which eschewed the nesting in favor of including the /music part of the path in each route separately seemed to have problems detecting the format correctly (though it is more than likely the cause was driver error on my part rather than anything that Merb might do differently in that situation). There’s pretty thorough documentation on Merb routing if you need more specific help and that’s definitely not the case for the Controller APIs, so let’s tackle those right away.

Now, how do we consume those URLs in our application? We need a controller, thusly:

class Music < Application
provides :jspf
def track
display Track.new params[:artist], params[:title]
end
def artist
display Artist.new params[:artist]
end
end

There’s two main things going on here that distinguish Merb’s controllers from Rails’: the ‘provides’ class method and the ‘display’ method we’re calling in each action. These two methods work together to let us return our data in the right format on every request. And, while they might look confusingly different from the familiar Rails respond_to block at first, they turn out to make things much cleaner, eliminating the need to redeclare common formats over and over in each controller method.

So, here’s how it works: provides sets up a list of formats that we’re willing to consume and then display makes sure that we provide the right one for each request. When a request comes in ending in .jspf, display calls to_jspf on the object that gets passed to it (in this case new tracks and artists). If we wanted to provide javascript, xml, html, etc. we’d just add those symbols to provides’ arguments and then make sure the objects we passed to display could respond to to_xml, to_html, to_etc with the right data (there’s an additional option for html of using a template in the views directory that Merb can automatically detect; that case works just like Rails; also: html seems always to be installed as a format regardless of whether or not you pass :html to provides).

While this should be enough to get you up and running writing Merb controllers, there are a few other details worth noting. First of all, if you want to provide alternative formats besides, js, html, xml, and the usual suspects (like we do here with jspf), you’ll need to register a mime-type. We configured that by adding a single line to config/init.rb:

Merb.add_mime_type(:jspf, :to_jspf, %w[application/jspf+json])

There’s lots of other additional information about the request available to your controller actions as well, such as content_type. Secondly there’s other options besides display for returning content from your action. Calling “render” will attempt to automatically detect a template with the right format in the appropriate views directory. And you can always just render a raw string. Again, more details are available here.

There’s the beginnings of a great open source book being written by some folks on GitHub: Life On The Edge With Merb, DataMapper & RSpec.They’ve done a pretty solid job covering the basics of app initialization, modeling with the DataMapper ORM, and testing with RSpec so far, but their section on Controllers is blank so far. If someone wants to pick up the thin thread I’ve laid down here and run with it, that would be a great place for more documentation to accumulate.

Tagged: merb, ruby, router, controller

A few months back Chad Fowler asked me to coordinate the lightning talks for this year’s RailsConf. It turned out to be a rolicking good time! We had 36 speakers in three sessions over three days — more than half the number of official talks! Topics ranged from announcements of new plugins (man, there are a lot of these suckers out there!) to demos of cool sites (Shared Copy certainly blew some minds) to rants and harangues (Ryan Davis never disappoints in this regard).

Below, I’ve attempted to summarize each of the talks and provide appropriate links where I could find them. The results are based on my sketchy notes while keeping time (and preparing my own talk) and so are definitely less than definitive. If you spoke and I got some of your details wrong, drop me a comment and I’ll make corrections. If any speakers want to send me links to their slides, I can add those as well.

Thanks to everyone who participated — audience and speakers alike — and thanks to the RailsConf orgnaizers for letting me put this together.

Friday

• Jon Garrin – Gatekepeer Plugin
• Max Baumann – Varnish as a Page Caching Alternative
• Prabode Weebadde – Paging: Pro-Paging
• Four Hewes – “Designers: Will you respect me in the morning?”
• Scott Chacon – Git + Lighthouse
• Chew Choon Keat – Shared Copy
• Luke Francl – Fan Chatter Stadium
• Gleb Arshinov – Keyboard Shortcuts for the web
• Ryan Davis – Write less code

Saturday

• Max Warnock – RDF framework
• Josh Goebel – Pastie
• Greg Borenstein – RAD: Ruby Arduino Development
• Matt Conway – Rubber for managing EC2 with Capistrano
• David Lowenfels – Scrum Ninja
• Jeremy Seitz/Mike Hill – geo search w/sphinx (PrimoSpot.com: find parking spots in NY)
• Rich Cavanaugh – acts_as_reasonable
• Ryan Jarvinen – Earfl: Voice Flickr
• Brian Chamberlin – RailsBrain ajax docs
• Eric Mill – ohnomymoney.com: Wesabe integration
• Matthew Deiters – Content Management DSL, DSL Engine

Sunday

• Dan Chak – Active Resource Kool-Aid
• David James – Method Trails
• Michael Slater – buildingwebapps.com
• Noel Rappin – iPhone on Rails
• Robert Thau – Permissioning in ActiveRecord
• Loren Johnson – Resful Workflow Plugin
• David Woodward – REST with Rails and ejabberd
• Jon Dahl – How Not to Present at RailsConf
• Brian Durand – Performance Plugins
• Jacob Dunphy – KABLAME!! (check a git or svn repo to find out who didn’t write tests)
• Ben Mabey – RSpec Story TM Bundle
• Chad Wooley – Geminstaller
• Aaron Campos – Automator
• Joannou Ng – NSTableViewFTW
• Mark Anderson – Jive Pages Plugin

Tagged: RailsConf, 2008, ruby, rails, conference, lightning+talks

Spent a while today getting up and running with Merb, the minimalist modular alternative Ruby framework from Ezra and the good people at Engine Yard. Merb has been in a bit of chaos these recent months as it’s gone through a major reworking to acheive a whole new level of performance as well as honest-to-goodness modularity, including choosing your own ORM and templating system. I’ve been watching Merb’s development for some time waiting for it to get to a level of stability that looked safe enough to dive in; their most recent release, 0.9.2, combined with pressing needs in a few exciting new Grabb.it features, made today the day.

My first day with Merb has been mostly great, but the one thing I found really sorely missing was a tutorial on how to get started. In all fairness, the Merb team promises left and right that copious documentation will be coming as they settle down to 1.0. In the meantime, I thought I’d pitch in for any brave early adopters out there with this Guide to Installing Merb and ActiveRecord.

Acquire and Install the Source

So, my goal here was to get from 0 (no Merb on my machine whatsoever), to running a hello world for accessing the database from an existing Rails project using ActiveRecord from within Merb. The first step was to acquire the source. One of the downsides to Merb’s modular architecture is the complexity involved with installing it (again, all relevant disclaimers here about how the merb team will, I’m sure, be working to simplify and improve the process as they reach 1.0). At least for now, you actually have to get your hands on three different packages: merb-core (the base of the framework, a requirement), merb-more (has more advanced features like the command for actually creating a new app), and merb-plugins (this is where things like ORMs and templating systems live). Let’s do that:

$ git clone git://github.com/wycats/merb-core.git
$ git clone git://github.com/wycats/merb-more.git
$ git clone git://github.com/wycats/merb-plugins.git

That’ll get us the bleeding edge trunk version (about which more here).

Now that we’ve got the pieces, we need to install them, thusly:

$ cd merb-core ; rake install ; cd ..
$ cd merb-more ; rake install ; cd ..
$ cd merb-plugins/merb_activerecord; rake install; cd ..


Create a New Project and Configure it for Active Record

We’ve got all the pieces; it’s time to create our project and get it setup to use ActiveRecord. Go to the place you want to create your app and do this:

$ merb-gen app my_app

This is equivalent to the ‘rails’ command and will create your project directoy with most of what you need. But since a basic project in Merb is assumed to be simpler than a basic project in Rails, you’ll quickly notice that you don’t have a models directory. Since we’re actually going to need a model if we want to connect up to a database with ActiveRecord, go ahead and create that directory and create a file inside if for your model just as you would in Rails, for example my_resourece.rb, which could look like this:

class MyResource < ActiveRecord::Base
end


We’ll probably want a controller as well, so create a new file: controllers/my_resources.rb:

class MyResources < Application
def show
r = MyResource.find :first
render r.some_method
end
end

Notice that all Merb controllers inherit from Application just like Rails controllers inherit from Application::Controller. The naming choice there is kind of interesting because it reveals Merb’s controller-centric history and philosophy (remember that the framework doesn’t assume that we need models by default; it turns out there’s a lot you can do with just controllers).

Since we’re using ActiveRecord, we’ll obviously need to tell Merb that we want it to go ahead and actually load AR as our ORM. Go into config/init.rb in your project and uncomment the line that says “use_orm :activerecord”.

We’re almost there! These last few steps will feel familiar from setting up a Rails app: letting the framework know about our route and database configuration. To set up your route, open up config/router.rb and, inside the ‘prepare’ block, add a line like this:

r.resources :my_resources

Merb’s routes work pretty much like Rails’s, but with a few more advanced features some of which are explained in the comments at the top of that file. If you need something other than standard RESTful routing, read those.

Finally, all we’ve got to do is configure database access and we’ll be ready to roll. In Merb this looks exactly like Rails, in fact, I simply copied the database.yml file over from the Rails project that usually manages the db I wanted to access, dropped it in config/databse.yml and it worked straight out of the box.

Ok! If you’ve made it this far, then you’re probably more than ready for the big reveal. In you project directory do this:

$ merb

The server will start and you’ll get a few log messages in your terminal that look like this:

$ merb
~ Loaded DEVELOPMENT Environment...
~ loading gem 'merb_activerecord' from ...
~ loading gem 'activerecord' from ...
~ Connecting to database...
~ Compiling routes...
~ Using 'share-nothing' cookie sessions (4kb limit per client)
~ Using Mongrel adapter

Once that settles down, browse to http://localhost:4000 and you should see the Merb welcome screen. Go to your expected url to see the result of your handiwork: i.e. http://localhost:4000/my_resources/7

If this works, you’re officially up and running with Merb and ActiveRecord. If not, you’ll see one of Merb’s very stylish error screens and it’ll be time to go to #merb on irc.freenode.net where all the friendly merbfolk hang out and are more than willing to help.

Tagged: merb, ruby, web, framework, install, 0.9.2

Ok, I’ll say it: I think JavaScript is cool. One of my favorite effects of the move to the modern AJAX-oriented web application architecture has been the opportunity to move ever more functionality into the client. At Grabb.it, we like to say, “Anything you can implement in JavaScript is free.” Instead of running on our servers, the JavaScript portion of our app runs on a distributed grid of thousands of machines maintained for us by our users. Also, despite the reputation given it by the Browser Wars, JavaSript is incredibly fun to develop in: it’s lightweight and extremely flexible in a unique way that somehow forces you to constantly keep your code very closely tied to the data it’s manipulating.

The one big downside to JavaScript is its runtime environment. Not only does code running in the browser confront a Gordian Knot of browser compatibility problems, but it’s also irretreviably isolated from interoperating with other application code. While javascript libraries (like the inestimable jQuery) are increasingly proving the Alexander’s sword of the browser compatibility Knot, the issue of lack of application interoperability is only just beginning to get serious. As JavaScript’s innate advantages lure more and more application code into the browser, the question will be unavoidable: How do you get modules implemented in JavaScript to interact with those built in other languages that live in more traditional environments? How do you avoid duplicating all functionality that you put into the JavaScript portion of the application so that you can call it from outside the browser?

This week, trying to solve exactly these types of problems, I discovered a tantalizing avenue towards addressing some of these questions: browser automation from the command line and from scripting languages. Here was my situation.

As part of an upcoming Grabbit project, I’ve built a a highly interactive data browser for our customers. The JavaScript running on that page makes a series of JSON GET requests to gather all of the necessary information to compose its display and it makes a few AJAX POST requests to report back to the server on certain bits of status. But now, I wanted to trigger those POSTs programatically on a schedule rather than waiting for customers to trigger them. The dilemma is that I’d already written this relatively sophisticated JavaScript application that makes all the necessary requests, implements the business logic, and knows how to POST in the data. I had two options: redo all of that work again in my server-side application (ick!) or figure out a way to trigger this JavaScript code by automating its runtime enviornment (the browser).

After a half day’s research, here’s what I discovered: there’s a Firefox extension that allows other applications to establish JavaScript shell connections to a running Mozilla process via TCP/IP. It’s called JSSH. Once you’ve got JSSH installed and running in Firefox, you can open a telnet connection to the browser that allows you to automate it using JavaScript commands to do things like load new pages or even manipulate the DOM on pages you’ve loaded. You can then automate this interaction using any scripting language with a telnet library. For the remainder of this post, I’ll provide step-by-step instructions for running JSSH and for automating it with Ruby.

Install JSSH

The easiest way to install JSSH is to download the JSSH.xpi and open it with Firefox which will offer to install the extension (if you’re interested in compiling Firefox with it from scratch or installing an existing binary, you should read these instructions).

Start Firefox with JSSH

Once you’ve got a copy of Firefox with JSSH installed, you’ll need to run it. You can do this by providing the correct options when launching Firefox from the command line. On Mac OS X, that looks like this:

/Applications/Firefox.app/Contents/MacOS/firefox -jssh &

The “&” at the end of that line will background your command so it doesn’t take over your terminal session.

Telnet into the JavaScript Shell

Once Firefox is running, we can use telnet to log into JSSH like so:

$ telnet localhost 9997
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
Welcome to the Mozilla JavaScript Shell!
>


Load a URL from JSSH

Now that we’re in, we can tell Firefox to load pages for us, thusly:

var w0 = getWindows()[0]
var browser = w0.getBrowser()
browser.loadURI("http://www.urbanhonking.com/ideasfordozens")

And that’s it! If the JavaScript application I wanted to run lived at “http://urbanhonking.com/ideasfordozens”, we’d be done. That command would load the page and Firefox would interpret and run the JavaScript it found there.

Now, all we’ve got left to do is make it so that we can trigger this process from our application code. So, we’ll…

Automate the Process with Ruby

Like any good scripting language, Ruby has a telnet library, which means that once we’ve got an instance of Firefox running with JSSH enabled, we can talk to it from Ruby whenever we want. Here’s an example script that logs into the telnet shell and loads a series of URLs one at a time:

require 'net/telnet'
my_urls = ["http://urbanhonking.com/ideasfordozens", "http://atduskmusic.com", "http://grabb.it", "http://pdxpopnow.com"]
# start telnet session with the Firefox javascript shell and setup browser object
puts "starting telnet session"
firefox = Net::Telnet::new("Host" => "localhost", "Port" => 9997)
firefox.cmd "var w0 = getWindows()[0]"
firefox.cmd "var browser = w0.getBrowser()"
# load each page
my_urls.each do |url|
puts "loading...#{url}"
firefox.cmd "browser.loadURI('#{url}')"
sleep 10 # so that the browser has time to load even if the page is slow
end
firefox.close


Further Research: Screen Scraping JavaScript Heavy Sites

What else might this rickety bridge we’ve built to the JavaSript runtime environment be good for? One thing that immediately occurs to me is: screen scraping for sites with a lot of JavaScript. Another side effect of the rise of rich JavaScript applications has been to create intractable problems for people trying to do screen scraping. If the data you want is not in the page’s HTML when you request it in the first place but is only written in later when the page’s JavaScript runs then traditional spidering and screen scraping techiques will fail to find it. Freebase, the open database application built by Danny Hillis and his team, for example, uses a highly dynamic interface for presenting its data that is almost entirely based in JavaScript. Or, on the low-brow side, MySpace uses JavaScript throughout the forms in its interface to help with date picking and such. If you wanted to scrape or automate interaction with either of these sites, you’d need access to a runtime environment that could execute JavaScript.

I haven’t really tackled this problem with JSSH, but I do have some leads. For example, here’s how you get the html of the document:

> browser.contentDocument
[object XPCNativeWrapper [object HTMLDocument]]
> domDumpFull(domNode(browser.contentDocument))
<HTML><HEAD><META content="text/html...

If you want to explore this avenue further, one of the best places to look is Firewatir, a project to add Firefox support to the WATIR browser testing framework. They do lots of click-by-click automation and checking for results, so I’m sure they’ve figured out approaches for a lot of what you’d confront when screen scraping. The JSSH documentation itself is useful and clear but not the most in depth.

Happy automating! Let me know what you discover…

Tagged: ruby, firefox, jssh, javascript, automation, browser, ajax, json

Of course, I’ve been making them for awhile now, e.g. Largehearted Goat and The NY Times Explains the Ratings. At first, creating a new SSS is a satisfying experience. You have a whacky idea and withing a few hours, you’ve registered a domain, written some simple code, and put something up. But, over time, each one gets to be more and more of a burden. My ideas, at least, have involved sites that need to be constantly updated with new information over time. Since I’ve always implemented these sites with simple ruby scripts that run on my local machine and then upload the static finished versions of the sites, this has meant keeping an eye on unreliable cron jobs and, sometimes, hand maintenance. And, over the years, I’ve wondered if there was a better solution.

Today, I took the first steps towards finding one. It turns out that good-ole CGI scripts — so foreign to those of us who’s main experience has taken place in the age of sophisticated web frameworks like Rails — make a great basis for SSS development. What follows is a basic introduction to writing and running CGI scripts with Ruby. I’ll focus on Dreamhost as a deployment target since it’s the service I have access to for this kind of thing and also the issues that arise there are probably not that dissimilar from what comes up with the other shared hosting services that are the natural habitat of SSSes.

Step One: Get Ruby and Rubygems up and running

If you plan on keeping your SSSes extremely spartan simple, you might be able to skip this step. Dreamhost accounts come with an old version of Ruby already installed. If you don’t need to install any custom gems for your scripts or control anything else about your ruby environment, you can skip straight down to the step two. However, if you plan on doing anything the least bit more sophisticated — from installing one individual gem all the way up to using Rails itself — you’ve got a bit of work to do beforey you can get started on the fun part.

Being far from an expert on unix build process and dependencies, I followed Nate Clark’s excellent instructions for building Ruby and Rubygems on Dreamhost with just a few discrepancies. My biggest note is that the version of the Ruby source to which he links is out-of-date. I changed his line for download the Ruby source to:

$ wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p111.tar.gz

But, don’t trust me! The most recent version of Ruby changes all the time and you can always find the most recent version of Ruby at ruby-lang.org. The one other thing worth noting that I discovered was that in any of the compilations make may die by timeout (Dreamhost kills processes that last longer than a given interval, a major pain point we’ll be returning to in a moment when we try to install individual gems). It won’t always be clear that a timeout was at fault so if you see a ‘configure’, ‘make’, or ‘make install’ command die mysteriously, just go ahead and try again.

Step Two: CGI Hello World with Ruby

Now that we’ve got Ruby and Rubygems installed, the first thing we want to do is to just run a basic ‘hello world’ CGI script to make sure that we’ve got things configured correctly. Here are the steps:

1. Log into dreamhost over ssh.
2. Navigate to the desired directory within your web space.
3. Create a new file, e.g. test.cgi.
4. Fill it with the following content (if you skipped Step One and are using the build-in DH Ruby, the first ‘shebang’ line should read: “#!/usr/bin/ruby” instead of what is shown):
   
#!/usr/bin/env ruby
require "cgi"
cgi = CGI.new("html3")
cgi.out("text/plain"){
"Hello World #{Time.now}"
}

5. Change the permissions of that file by running:
   
$ chmod 775 test.cgi


That should do it. Navigate to that path in your browser and you should see output that looks something like this:

Hello World Sat Mar 01 19:04:53 -0800 2008


Step Three: Install Gems by Hand

Now, if you plan on doing anything more interesting than displaying the date, the odds are you’re going to want to take advantage of Ruby’s great and growing weatlh of libraries made available as Gems. Unfortunately, the RubyGems server system running at RubyForge has lately become incredibly slow. In order to find packages and detect dependecies, the default ‘gem install’ command has to communicate with that server and so becomes subject to the ruthless Dreamhost killing of long-running processes. In my experience, trying to install gems on Dreamhost will universally end in frustration:

$ gem install feedalizer
Bulk updating Gem source index for: http://gems.rubyforge.org
Killed

What to do?

Fortunately, in addition to using the RubyForge server to search for gems, it’s possible to install them directly from .gem files if you can find them for your desired packages. For example, when building a recent SSS I wanted to use Feedalizer — a great library that scrapes web pages and automatically creates RSS feeds from the resulting content — so I hunted down the .gem file via the RubyForge website, downloaded it, attempted local installation, and then (when that failed) chased down the sources for the dependencies (in this case just Hpricot) to follow the same process. To give you a taste of things here were the gory details:

Installing Feedalizer:

$ wget http://rubyforge.org/frs/download.php/13797/feedalizer-0.1.0.gem
$ gem install feedalizer-0.1.0.gem

which threw an error complaining about the need for the Hpricot dependency, so:

$ wget http://code.whytheluckystiff.net/dist/hpricot-0.5.140.tgz
$ tar xzf hpricot-0.5.140.tgz
$ cd hpricot-0.5.140
$ rake gem
$ gem install pkg/hpricot-0.5.gem

The ‘rake gem’ step there is necessary because _why only makes the source directory available for direct download rather than an explicit .gem file. After this completes successefully, return to the ‘gem install’ step for Feedalizer above (be sure to the return to the directory into which you dowmloaded the Feedalizer .gem before running it).

Much bumpier than the smooth ride normally provided by ‘gem install’, but it’ll get the job done. You could clamber your way down similar rutted paths for most any other gem you needed for you script.

And that should be more than enough to get you started. If you run into any troubles, dive into the comments here and on the other blog posts linked for help. Also, if anyone has any experience using lightweight persistence strategies in this context, I’d love to hear about them, espcially if they’re file-system only; a lot of my scripts requiring saving records and I tend to use a rickety YAML-based system for them that could stand improving.

Note: Thanks to The Pug Automatic for inspiring me to start down this path in the first place.

Tagged: ruby, cgi, dreamhost, rubygems, sss, single+serving+sites

Inevitably, my list reflects my interests and bias just as Brogan’s does his. The posts I’d like to see are largely aimed at people like me: people who picked up Ruby in the last couple of years because of Rails and have got the language and framework pretty handily under control but lack the deep rooting in technical fundamentals and culture that comes from having gone to school for this stuff or having had a long career in it.

The theme is: “I know Ruby. Now what?”

Like Brogan, I’ll ask that if you write on one of these topics link back here or come back to comment so I can find your post and gradually turn this list into something of a directory of this kind of info. Now, without further ado, here’s my list:

1. My First C Extension
2. A Set Theory Primer for Relational Database Users
3. What I Get Out of My Local Ruby Users’ Group
4. All About Postgres Indexes
5. Ruby Was My First Language, Here’s My Second
6. The Five Most Useful Things I Learned in Computer Science
7. My First Contribution to an Open Source Project
8. Participating Actively on Mailing Lists
9. Participating Actively on IRC
10. Understanding and Using Threads
11. Keeping Up to Date with New Developments to the Libraries You Care About
12. How to Write Good API Documentation
13. An Intro to Code Research, Or: Is There A Library for That?
14. An Intro to Queues, Pools, Runners, and Inter-Process Communication
15. Unicode Once and for All
16. Timezones Once and for All
17. How Do Migrations Actually Work?
18. Basic System Administration for Developers
19. Domain-based Programming in Javascript
20. Instrumenting My Rails App
21. Approaches to Processing Large Data Sets
22. A Developer’s Guide to Deployment
23. Bootstrapping Rails Development for the Absolute Beginner
24. Hey Look, I Did Something Useful with LISP!
25. Approaching Your Idols: How to Start Conversations with Gray Beards and Gurus
26. My Painless Gem Integration System
27. Using Your Own Documentation
28. Building a Rails App from Someone Else’s Excel Spreadsheet
29. Useful Ruby Outside of Rails
30. Ruby as PHP Replacement
31. Writing Simple CGI Scripts with Ruby
32. Finding and Fixing Memory Leaks in Rails, Or: Why Are My Mongrels So Big?
33. Writing and Managing Long-Running Processes
34. SMS Integration in Rails
35. What I Learned from Working with Statically Typed Languages
36. My First Profiling Session
37. Big Team Tools and Small Teams, Or: Why Is My Trac Empty?
38. Rules of Thumb for Performant Ruby Code
39. A Survey of Persistence Strategies Beyond Relational Databases
40. Living with a Large Schema
41. My First Cocoa Program
42. Writing and Distributing Rails Apps for Desktop Installation
43. Domain Registration and Hosting for Rails Apps
44. Setting Up Subdomains and Pointing Them at Rails Apps
45. My First Apache Configuration
46. My First Nginx Configuration
47. Is It Worth Releasing?: When to Open Source Your Work
48. An Introduction to GUI Programming
49. Lessons from Java for Someone Who’s Never Written Any
50. Practical Tips for Learning Protocols and Reading Specs

Tagged: ruby, programming, blog, list, education, brogan

For mere mortal developers like me, the most attractive one of these three is the second: Git’s distributed architecture. In practice, the fact that Git is distributed means that you can separate the act of committing your changes from the act of publishing them. With Subversion, and other centralized systems, the only way to get your changes under version control is to push them out to the central server from which all of your collaborators pull. So, if you’ve got a set of changes that make a logical atomic commit, but break the build in a major way, you can’t commit them without disrupting the work of others. Git, on the other hand, lets you make commits on your local machine without pushing any changes out to anyone. The result is that you can make a series of commits as you go that leave the app in whatever state you want — even including creating your own experimental branches for trying out ideas, a process Git makes almost trivially easy — and then when you’re ready to push your changes out into the wider world, you can do so without anyone having to see the mess you made along the way. (A cool side effect of this architecture is that you can commit even if you don’t have network access, for example if you’re on a plane, which is where I wrote most of the code I’m about to tell you about…)

Anyway, as soon as I started using Git myself, I noticed the lack of a tool I use so often I’d stopped even noticing it was there: jchris’s svn_tools plugin. It’s a series of rake tasks for controlling Subversion. It provides a bunch of tasks to do common things like add all existing unrecognized files all the way up to bootstrapping a brand new Rails project into a new checkout. And Git definitely has some bumpy edges that would be nice to abstract out in this same way (like the need to type “git commit -a” for almost every commit and the need to manually tell Git about each new file individually when they’re first being added). So, I set out to write a version of the same library for git, one that could provide some utilities that would speed up everyday activities and, eventually, some more complex macros for doing things like bootstrapping a new Rails project.

I didn’t manage to accomplish that second goal, but I made enough progress towards the first, that I thought it would be useful to throw the project out there to see if one of those feverishly twittering Rubyists busy trying out Git might want to pick it up and run it into the end zone. So, without further ado here it is:

git_tools_0.1.zip

The main tasks of interest are:

• rake git:add

  – adds all files that need a manual git add

• rake git:commit

  – commits with the -a flag thrown so your new content will actually get committed

• rake git:ignore

  – removes the given files (or all files matching a quotation-surrounded pattern if given) from version control

For those looking to develop the library further, a full framework for parsing Git’s response messages is already in place as well as a system for executing Git commands (the design of which is yoinked from the svn_tools project). Bits here and there may be a touch wonky (for example, if you send a pattern to rake git:ignore, git raises an error even though it completes the task successfully).

A note on contributing: The archive of git_tools is itself a Git repository. One of the nice things about Git is that the files containing the versioning info are very compact and portable, so it’s easy to ship around copies of a full repository with its entire history. If you make changes or improvements you’d like to see incorporated, just send me an archived version of the updated repository and I’ll merge them in.

If you’re intrigued and would like to learn more about Git I’d recommend the
excellent tutorial intro to Git on Kernel.org and this how-to on building Git on OS X.

Happy Gitting!

Tagged: git, svn, cvs, ruby, rake, linux

I’m proud to announce the release of my very first gem: THUMBNAIL. It’s a Ruby wrapper for the Amazon Web Services Alexa Site Thumbnail Service, which lets you automatically download thumbnails of any website or dynamically embed them in your own pages for an incredibly small fee ($0.20 per 1000 images). And it has a badass logo (see above)!

You can get it from RubyForge thusly:

$ sudo gem install thumbnail


You’ve got to do some bureaucratic overhead at Amazon before you can play (more about that on the THUMBNAIL homepage), but once you do it’s just as easy as pie to download pix of sites from around the web like so:

require 'rubygems'
require 'thumbnail'
require 'open-uri'
t = Thumbnail::Client.new :access_key_id => YOUR_ACCESS_KEY_ID,
:secret_access_key => YOUR_SECRET_ACCESS_KEY
url = t.get("www.urbanhonking.com")[:thumbnail][:url]
File.open("urho.jpg", "w") { |f| f.write open(url).read }

where YOUR_ACCESS_KEY_ID and YOUR_SECRET_ACCESS_KEY are things you get from Amazon when signing up for the service. Running such code would make you the proud owner of a local copy of an image like this:

THUMBNAIL will also build you a url you can include in any webpage that will redirect to the site thumbnail you want like so:

require 'thumbnail'
t = Thumbnail::Client.new :access_key_id => YOUR_ACCESS_KEY_ID,
:secret_access_key => YOUR_SECRET_ACCESS_KEY,
:action => :redirect
url = t.get("www.twitter.com")
#=> http://ast.amazonaws.com/?Action=Redirect&AWSAccessKeyId=YOUR_ACCESS_KEY_ID
&Signature=sdhfiawrkjw3h9bncoa8ue&Timestamp=2007-06-14T09:09:18.000Z&Url=www.tw
itter.com&Size=Large


I’m working on a Rails plugin that will provide a view helper so you can easily do this from any template, but for now you’ll have to be satisfied with the additional sample code available on the THUMBNAIL RDoc.

There are all kinds of other details available at the THUMBNAIL homepage. And the code is, of course, available to all for free under the MIT License. So, Go! Get your thumbs dirty! Enjoy!

Tagged: thumbnail, ruby, aws, amazon, web, service, alexa, api, rubyforge, gem

One of the best is the way you get started. Hpricot is a toolkit for parsing and manipulating XHTML. So, obviously enough, just about every time you invoke it, you’re going to want to pass it an XHTML document so it can, you know, prep it for parsing and manipulation. And how do you do that? What’s the syntax?

Hpricot(my_document)

That’s it. There’s no “Hpricot::Base.new(my_document).parse” nonsense, or any of the other more or less torturous common options. Not a single character of syntax is wasted.

But, if you’re a mere Ruby mortal, like me, you’re probably looking at that code and going: ‘Huh?’ Isn’t Hpricot a constant? It’s capitalized. But it’s taking an argument like a method. How is that even valid Ruby? How can the parser tell if it’s a constant or a method?

Well, it turns out that there’s no rule against having capitalized method names; the parser can tell it’s a method because it’s got an argument. And that’s all that’s required for it to be sent off to method- instead of constant-dispatch (as Chris pointed out, this is one advantage of not having Ruby be “turtles all the way down”; Smalltalk couldn’t do this).

Beyond providing fodder for a Language Nerd Attack, though, what’s the upshot? How’s this fact help the man on the street? Well: there’s nothing actually sophisticated going on here. So: you can do it too.

Here’s an admittedly contrived (and useless) example:

class Dogger
def initialize
puts "dog"
end
end
def Dogger()
Dogger.new
end

a simple class definition followed by a simple method invoking it.

Which leaves us with the ability to write two snippets of code that, while they may look nearly the same, do very different things:

>> Dogger
=> Dogger
>> Dogger()
dog
=> #<Dogger:0x15d2478>

and that is exactly from where _why’s use of this little quirk derives its leverage. This trick makes you feel like you’re invoking a constructor or calling some other kind of class method when you are, in fact, doing nothing of the sort. Just as our Dogger() method above needn’t have done anything remotely related to the Dogger class, _why could have named his method Clown() or ChunkyBacon() while still calling Hpricot.parse(input, opts) inside it (which is exactly what Hpricot() does).

But his chosen usage is particularly inspired. In one fell swoop, he gives his whole complex feature-ful library a single welcoming point of entry. You need never concern yourself with the internal machinery; just heave a document over the transom and let the library figure out what to do with it. And this is the wider lesson of _why: real power comes from combining the playfulness (better: the insouciance) needed to probe, question, and even bend the limits of the language with the discipline and aesthetic sense required to use what you find not to obfuscate and confuse, but to write elegant and, above all, more humane code.

I mean, Hpricot would definitely not be a better library if that method was called ChunkyBacon(). Right?

Tagged: ruby, why, dispatch, hpricot, syntax, chunkybacon