Ideas For Dozens

Celia Hirschman from KCRW's On the Beat

In the most recent On the Beat, Celia Hirschman advocates for the Musician's Union to take a more active stand for musicians' rights online. In a world where only 42% of internet users say they pay for music, she argues, the Union has a responsibility to advocate for musicians' "right to be paid for their efforts anytime their music is played". She notes the failure of legislative attempts to extract revenue for musicians from peer-to-peer file sharing, mp3 blogs, and CD burning and lays the blame at the feet of the Union as the primary organization that represents musicians.

I agree with half of this argument. The Musician's Union has been embarrassingly lax in fighting the real battles that matter for artists in the modern music distribution landscape. What has the Musician's Union done to stop the labels from cheating artists out of revenue from iTunes downloads? How about to stop them wasting their shrinking budgets on glamorous perks and inflated short-run corporate profits rather than developing new artists with the possibility of long and successful careers? Or what about to force them to hire people who can figure out new business models that fit the new technology?

None of these battles receive the slightest mention in Hirscman's critique of the Union. She ignores the losses artists have suffered by being forced into vastly inequitable relationships with labels and instead highlights the ways in which the industry has failed to fully extract revenue from new forms of music consumption and fandom.

This wrong-headed focus on enforcement over transformation is symptomatic of the problems plaguing the record industry as a whole. Hirschman's catalogue of Union failings closely resembles a punchlist of RIAA complaints and talking points. Focusing on missed revenue from new forms of fandom while the music industry's entire distribution and profit model sinks into irrelevance is like getting upset over a spilled glass of champagne on the deck of the Titanic.

This mistake derives directly from the principle that Hirschman articulates in the piece: that artists "have the right to be paid for their efforts anytime their music is played." This same idea was proposed recently by Peter Kirn at Create Digital Music:

Recorded music has value to consumers. And, in business, if something has value somewhere, it's a business.

This is a core principle of the industry's thinking right now and it is obviously, palpably false. In middle school, when a friend played me Sebadoh off of a walkman that he'd smuggled to school, transforming me in a single moment into a lifelong fan of indie rock, was that "theft"? Should Sebadoh have gotten paid? When I worked at a local patisserie I used to play my favorite CDs throughout my shifts and would often write the band names and album titles down for intrigued customers. Should the shop have had to track and regulate everything we played so they could pay royalties to the artists?

This last is something of a trick question since enforcement of this kind of public performance royalty is something for which Hirschman specifically lauds rights-enforcers like ASCAP:

If you walk into a restaurant, nightclub or boutique and hear music playing, chances are very good a performance-rights organization have demanded compensation. These rights societies literally go door-to-door to insure their members get paid for music

Hirschman could not have this issue more wrong. ASCAP contacted the patisserie while I worked there saying they'd observed us playing music controlled by their members and we had to either cut it out, sign up for a very expensive pay service they were offering, or face a lawsuit. The business owners felt like they'd been shaken down by the mafia. Their response was to stop playing ASCAP music altogether and instead to put together a library of local music which we had explicit permission from the artists to play without royalties. Maintaining this library was a lot of work and we rapidly fell back into the old system of playing whatever we wanted, including ASCAP music, without permission, but now in an environment of greater fear and resentment. I would bet that a similar story holds for most places you actually visit beyond corporate chains: if you walk in and hear good music playing it is either local or in explicit defiance of an ASCAP threat.

And this story is a parable of what's wrong with focusing on enforcement. Enforcement alienates consumers and tastemakers. It tarnishes the reputations of artists and the organizations that should represent them. It forces natural music consumption and sharing patterns underground. Possibly worst, enforcement distracts artists and the industry itself from solving the huge existential issues that they face.

There is one piece of information from Hirschman's piece, however, that does hold out hope for the music industry if they do ever overcome these distractions and decide to face the real challenge of transformation: 42% of internet users pay for music. That's an enormous number. How many internet users pay for news? Or search? Or social networks? I don't hear anyone in these businesses complaining that their industry is in decline. That's an enormous number and it reflects the incredible amount of passion that exists for music online. If the industry can't find a way to transform that passion into a functional profitable business, it won't be because because the users outfoxed their enforcements efforts. They'll have only themselves to blame.

Tagged: music, industry, on the beat, Celia Hirschman, kcrw, enforcement, piracy, riaa

On sunday night, I released version 0.2.2 of the Ruby Arduino Development gem. The main focus of this update is compatibility with version 0011 of the Arduino software tools (follow that link for the release notes). While 0010 may still work with this new version of RAD, 0011 is now the default and upgrading is highly encourages. Download it here: Arduino 0011

.

This release also includes a patch to fix a problem with type-compatibility between RAD's var system and some Arduino function return values courtesy of David Michael. David also gave what looks like a great talk about RAD at NYC.rb: Ruby Meets Worlds, which includes a really nice example of using the observer pattern to communicate with an Arduino over a serial port that portends really well for one of the exciting things we've been talking about on the RAD Google Group: an interactive serial console.

Tagged: ruby, arduino, rad, 0011, computerkraft, console, serial

"The enemy isn't Microsoft. The enemy is non-use"

— Matt Asay, The Ten Commandments of Open Source

For the past month, watching Hillary Clinton and Barack Obama — two relatively noble politicians who largely want the same good things for this country — say and do silly things to each other over the most superficial of disagreements in an atmosphere of ever-increasing animosity, I've been experiencing a disgust that was strangely familiar. All month I struggled to place where I'd felt this particular variety of nausea before, what other phenomenon oozes this same poison. And then I figured it out: it's the old Open Source vs. Evil Corporation flamewar. It's Slashdot all over again.

I won't alienate 40.4 percent of you by telling you which side I think is which since the key point is this: when two groups with basically the same objectives spend their best energy beating on each other those objectives are guaranteed to stay largely unaccomplished.

And just like every day that Obama and Clinton spend attacking each other leaves less hope for those good goals the two candidates share, every flamewar, every easy attack on Microsoft, every knee jerk reaction against a company that charges for its software leaves less attention available to solve the problems we all share of data interoperability, scaling, user friendliness, etc.

It's a fine thing to stand up for important values like openness and freedom, and, gosh knows, we have real disagreements on some topics that deserve to be debated vigorously, but the constant mutual harassment and posturing actually make these debates harder to resolve.

Both sides are guilty of this and we pay a measurable cost for it.

On the Open Source side, we tend to rule out technologies that reek of big corporate support even when they might be the best choice for the job and on the corporate side they tend to be too cautious, holding off on new ideas until they're blessed by the vendor of choice.

For example, I recently had a good old fashioned geek out with Rory Blyth, a college friend who went on to work at Microsoft. We spent a few hours having two of the oldest, least productive, and most fun coding conversations around: static vs. dynamic typing and SOAP vs. REST, as well as one new one that's shaping up to acquire a similar status: relational vs. document databases. While we seem to have an unbridgeable spiritual divide on the first topic, we each managed to convince each other, at least a little bit, on the last two. Rory did a strong job arguing in favor of the value of SOAP APIs for client implementors because of the strong level of automation and framework/IDE support that they enable and convincing me that at least some of my RESTful bias is inherited unquestioned from my community. Likewise, I managed to overcome Rory's profound SQL Server loyalty to convince him that relational databases, which were designed to perform original relational algebra in order to resolve a query with an unknown answer, are a bad architectural fit for most web applications which simply use them for object persistence. He left eager to check out CouchDB and I left resolved to explore Ruby's SOAP options.

This is exactly the kind of outcome that gets crowded out of most online discourse by the flamewars and name calling. The animosity and ugliness pushes us deeper into our own camps, just as moderate politicians tend to head to the extremes of their own party during a hard fought primary.

Well, the primary's over. I hereby declare it to be the general election. There's a world out there with real problems. From global warming to the growing international food crisis to the war on free speech, the world faces serious challenges that can only be overcome by radically great technology, the license it's released under be damned.

Tagged: microsoft, open, source, politics, rory blyth, soap, rest, couchdb, flamewar

It's been an active few weeks in the world of RAD! First off, RAD 0.2.1 was released last week which includes a bunch of small features and bug-fixes:

• added first significant documentation
• fixed require 'yaml' bug in makefile.rb
• added examples directory to the website
• applied Brian Riley's SWSerLCDpa patch
• experimental libraries system in vendor/libraries
• enhancements to the rad command: can set project config.

The documentation and examples should be lots of help for those of you are just getting started. If you've got successful sketches, send them to me and I'll add them!

I setup a Ruby Arduino Development Google Group for discussion amongst the growing group of contributors and users of the project. We've already got some fun stuff brewing there including someone working on a serial console, so drop by if you've got questions or want to participate.

Also, big thanks to Brian Riley of Really Bare Bones Arduino supplier Wulfden.org. His patch here for SWSerLCDpa support hopefully marks the start of a great collaboration to get a bunch of important Arduino libraries ported to RAD including OneWire and I2C. Plus, he sent me a totally awesome data logging kit to play with!

Last, but definitely not least, I presented about RAD at Dorkbotpdx 0x01 last week. I was fortunate enough to be in the good company of Ward Cunningham (who gave an amazing talk about his biologically-inspired Cybords project) and many others. Jared from Dorkbotpdx shot video of the event which is now online. I did two demos, the basic blinky-LED hello world and a more advanced assembler sketch that drove a 7-segment LED and serial output (my segment starts at about 4:46):

Watch the rest of the videos from the event here: Dorkbotpdx 0x01 on Vimeo.

Tagged: arduino, ruby, rad, dorkbotpdx, wulfden, electronics, physical, computing, portland, oregon

Spent a while today getting up and running with Merb, the minimalist modular alternative Ruby framework from Ezra and the good people at Engine Yard. Merb has been in a bit of chaos these recent months as it's gone through a major reworking to acheive a whole new level of performance as well as honest-to-goodness modularity, including choosing your own ORM and templating system. I've been watching Merb's development for some time waiting for it to get to a level of stability that looked safe enough to dive in; their most recent release, 0.9.2, combined with pressing needs in a few exciting new Grabb.it features, made today the day.

My first day with Merb has been mostly great, but the one thing I found really sorely missing was a tutorial on how to get started. In all fairness, the Merb team promises left and right that copious documentation will be coming as they settle down to 1.0. In the meantime, I thought I'd pitch in for any brave early adopters out there with this Guide to Installing Merb and ActiveRecord.

Acquire and Install the Source

So, my goal here was to get from 0 (no Merb on my machine whatsoever), to running a hello world for accessing the database from an existing Rails project using ActiveRecord from within Merb. The first step was to acquire the source. One of the downsides to Merb's modular architecture is the complexity involved with installing it (again, all relevant disclaimers here about how the merb team will, I'm sure, be working to simplify and improve the process as they reach 1.0). At least for now, you actually have to get your hands on three different packages: merb-core (the base of the framework, a requirement), merb-more (has more advanced features like the command for actually creating a new app), and merb-plugins (this is where things like ORMs and templating systems live). Let's do that: $ git clone git://github.com/wycats/merb-core.git $ git clone git://github.com/wycats/merb-more.git $ git clone git://github.com/wycats/merb-plugins.git That'll get us the bleeding edge trunk version (about which more here).

Now that we've got the pieces, we need to install them, thusly: $ cd merb-core ; rake install ; cd .. $ cd merb-more ; rake install ; cd .. $ cd merb-plugins/merb_activerecord; rake install; cd ..

Create a New Project and Configure it for Active Record

We've got all the pieces; it's time to create our project and get it setup to use ActiveRecord. Go to the place you want to create your app and do this: $ merb-gen app my_app This is equivalent to the 'rails' command and will create your project directoy with most of what you need. But since a basic project in Merb is assumed to be simpler than a basic project in Rails, you'll quickly notice that you don't have a models directory. Since we're actually going to need a model if we want to connect up to a database with ActiveRecord, go ahead and create that directory and create a file inside if for your model just as you would in Rails, for example my_resourece.rb, which could look like this: class MyResource < ActiveRecord::Base end

We'll probably want a controller as well, so create a new file: controllers/my_resources.rb: class MyResources < Application def show r = MyResource.find :first render r.some_method end end Notice that all Merb controllers inherit from Application just like Rails controllers inherit from Application::Controller. The naming choice there is kind of interesting because it reveals Merb's controller-centric history and philosophy (remember that the framework doesn't assume that we need models by default; it turns out there's a lot you can do with just controllers).

Since we're using ActiveRecord, we'll obviously need to tell Merb that we want it to go ahead and actually load AR as our ORM. Go into config/init.rb in your project and uncomment the line that says "use_orm :activerecord".

We're almost there! These last few steps will feel familiar from setting up a Rails app: letting the framework know about our route and database configuration. To set up your route, open up config/router.rb and, inside the 'prepare' block, add a line like this: r.resources :my_resources Merb's routes work pretty much like Rails's, but with a few more advanced features some of which are explained in the comments at the top of that file. If you need something other than standard RESTful routing, read those.

Finally, all we've got to do is configure database access and we'll be ready to roll. In Merb this looks exactly like Rails, in fact, I simply copied the database.yml file over from the Rails project that usually manages the db I wanted to access, dropped it in config/databse.yml and it worked straight out of the box.

Ok! If you've made it this far, then you're probably more than ready for the big reveal. In you project directory do this: $ merb The server will start and you'll get a few log messages in your terminal that look like this: $ merb ~ Loaded DEVELOPMENT Environment... ~ loading gem 'merb_activerecord' from ... ~ loading gem 'activerecord' from ... ~ Connecting to database... ~ Compiling routes... ~ Using 'share-nothing' cookie sessions (4kb limit per client) ~ Using Mongrel adapter Once that settles down, browse to http://localhost:4000 and you should see the Merb welcome screen. Go to your expected url to see the result of your handiwork: i.e. http://localhost:4000/my_resources/7

If this works, you're officially up and running with Merb and ActiveRecord. If not, you'll see one of Merb's very stylish error screens and it'll be time to go to #merb on irc.freenode.net where all the friendly merbfolk hang out and are more than willing to help.

Tagged: merb, ruby, web, framework, install, 0.9.2

I'm proud to announce a new release of the Ruby Arduino Development gem (RAD)! This release brings two major new features (software serial serial support and inline assembler), a major revamping of the hardware serial library, and a raft of other small features and bug fixes. I've got lots of details below, but first I wanted to thank Scott Windsor who contributed the software serial support as well as all the other people who wrote in with bug reports, ideas, and comments, and of course all the members of the RAD core team. It's exciting to see RAD starting to support physical computing scenarios I've never had the hardware to work with myself.

Inline Assembler

I'm starting with this new feature not because it's necessarily the most import or most broadly useful, but just because I think it's the sexiest (which probably says something sick about me, but there you go)! RAD now offers a class method to all sub-classes of ArduinoSketch for writing routines in assembly language. You give the method three arguments: the name of the routine, its signature, and the assembly code you'd like to consitute its body. Here's an example script showing the method in action: class AssemblerTest < ArduinoSketch serial_begin def loop serial_println product(10,4) end assembler( :product, "int product(int a, int b);", <<-CODE product: mov r18,r24 ; move a to another register ldi r24,0 ; clear running sum, used to coalesce product ldi r25,0 ; sum = 0 .loop: tst r18 ; is a = 0? if so, we're done breq .end mov r19,r18 ; copy a andi r19,1 ; is a % 2 == 0 breq .skip add r24,r22 ; add b to sum adc r25,r23 .skip: lsr r18 ; divide a by 2 clc rol r22 ; multiply b by 2 rol r23 rjmp .loop .end: ret .size product, .-product CODE ) end As you can see, the body of the assembly code is simply passed as a string to the 'assembler' method. In this case, all I'm doing is implementing multiplication of two integers. RAD takes your assembly code, adds some architecture-specific headers and other boilerplate, writes it to a source file, and makes sure that the compiler knows to compile it into object code and link it into your project during the build process.

I'm an asm newbie and I found that the hardest part of getting started learning the stuff was bootstrapping an environment where you could actually run your code and see the results. Hopefully this addition to RAD will smooth that process for people in a similar situation. Now all you've got to do is put your assembly code in an ArduinoSketch model like the one above, upload it to your Arduino, and read the board's serial output (for example by attaching to it with screen). You can be writing (and debugging) your first assembler routines in seconds.

It's worth noting that this feature wouldn't have been possible without Reed College math professor Jim Fix teaching me the basics of AVR assembler and the avr-gcc build process. Thanks, Jim!

Software Serial Support

This feature came in as a patch from Scott Windsor. Scott was working on hooking up his Boarduino (an Arduino-compatible clone from Lady Ada) to a GPS module and a Serial LCD both of which require serial connections. Thankfully, Arduino provides a SoftwareSerial library for doing serial connection on any of the board's digital i/o pins. Scott's patch gives you really convenient access to this functionality, thusly: class MySketch < ArduinoSketch output_pin 13, :as => :led software_serial 6, 7, :as => :gps serial_begin def loop digitalWrite(led, true) serial_print(gps.read) end end As you can see, he configures ports 6 and 7 as a software seria connetion called "gps" and then he simply says "gps.read" to read the data his device is sending over that connection. Scott also implemented "print" and "println" methods for outputting to serial-connected devices such as his LCD, which work in the same way.

I was especially psyched to see this patch since I haven't gotten to work with any devices that communicate over software serial yet, which had meant that I couldn't really test and add this functionality myself. I'm planning to get my hands on some of that soon and so I'm excited to play with the clean simple API Scott's provided here.

Fixed Hardware Serial Support

In previous releases of RAD, hardware serial support had been, uh, perfunctory. The trouble stemmed from the wide variety of functionality hidden behind Arduino's elegant serial API. Serial.read(), Serial.print(), and their brethren do subtle things to act correctly when called with different arguments, from integers to individual characters to full strings (or arrays of characters, as C would have it). The result was that RAD's versions of these methods would sometimes do strange things like returning ASCII-value integers when sent characters. That should no longer be happening. For example if your run the following sketch: class SlowTypewriter < ArduinoSketch serial_begin def loop serial_print serial_read end end and attach to the serial output with screen, you'll see the characters you type successfully roundtripping to the Arduino and coming back with their encoding preserved. It's the world's slowest typewriter!

A Panoply of Small Features, Tweaks and Bug Fixes

Including, but not limited to:

• Made 'rake make:upload' default to skipping the prompt for reseting the board. The older NG boards that require the reset are getting rare and I recently modded mine to no longer require it. The reset is still available as an option by editing the "physical_reset" option in config/hardware.yml.
• Added support for HIGH/LOW and ON/OFF constants. This is a nice feature of the Arduino API that I hadn't been able to sneak throug the RubyToC translation stage until now.
• Support for variables that are not local to the loop method. It's a common desire to be able to initialize variables outside of the loop method of a sketch so that they will not be constantly re-initialized with each run, for example to define environmental constants or initialize counters. Since RAD replaces the Arduino setup functions and global declarations with ArduinoSketch class methods (such as output_pin), this wasn't previously possible. As of this release, you can use the 'vars' method to initialize variables in this scope thusly: class VarTest < ArduinoSketch vars :a => :int, :b => 7, :c => "hello" def loop a = 1 serial_print a serial_print b serial_println c end end This will produce output like:

  17hello
  17hello
  17hello
  17hello
  [...]
  

  As you can see, the vars method takes a hash where each key is the name of the variable you'd like to initialize and the value and be either a type (rendered as a symbol, i.e. ":int"), or an actual value if you want the var to be initialized to something (i.e. "hello").
• Changed default Arduino location to be /Applications/arduino-00010, which seems to be where most people (now including me) have it.
• Plus too many small bug fixes to list here!

What's Next?

There's all kinds of things big and small on the immediate roadmap for RAD. There's the administrative: putting the RAD source on Git Hub, adding sorely needed documentation to the core ArduinoSketch methods, organizing an email list for the growing group of people wanting to be kept up-to-date on the project, etc. There's the lofty: I've got the beginnings of a plan for putting building a testing and simulation framework on top of RAD that would let you run your sketch against a software version of the Arduino hardware while developing (with a visualization/interactive app built on Shoes!), etc.

And then there's your contribution: send me your patches and RAD can become whatever your want it to! As always, if you've got any questions, ideas for contributions, or run into any troubles running RAD, feel free to email me: greg [dot] borenstein [at] gmail [dot] com or comment here; I'd especially love to see/hear what projects people are building with RAD. Happy hacking!

Tagged: rad, ruby, arduino, c, assembly, reed, gem, release, physical computing, framework

This post explains how to control Firefox from the command line with Telnet and Ruby. After presenting some context to explain why I think this hack represents an important area of concern in contemporary web application development, I'll show how to execute it with actual install directions and code samples.

Ok, I'll say it: I think JavaScript is cool. One of my favorite effects of the move to the modern AJAX-oriented web application architecture has been the opportunity to move ever more functionality into the client. At Grabb.it, we like to say, "Anything you can implement in JavaScript is free." Instead of running on our servers, the JavaScript portion of our app runs on a distributed grid of thousands of machines maintained for us by our users. Also, despite the reputation given it by the Browser Wars, JavaSript is incredibly fun to develop in: it's lightweight and extremely flexible in a unique way that somehow forces you to constantly keep your code very closely tied to the data it's manipulating.

The one big downside to JavaScript is its runtime environment. Not only does code running in the browser confront a Gordian Knot of browser compatibility problems, but it's also irretreviably isolated from interoperating with other application code. While javascript libraries (like the inestimable jQuery) are increasingly proving the Alexander's sword of the browser compatibility Knot, the issue of lack of application interoperability is only just beginning to get serious. As JavaScript's innate advantages lure more and more application code into the browser, the question will be unavoidable: How do you get modules implemented in JavaScript to interact with those built in other languages that live in more traditional environments? How do you avoid duplicating all functionality that you put into the JavaScript portion of the application so that you can call it from outside the browser?

This week, trying to solve exactly these types of problems, I discovered a tantalizing avenue towards addressing some of these questions: browser automation from the command line and from scripting languages. Here was my situation.

As part of an upcoming Grabbit project, I've built a a highly interactive data browser for our customers. The JavaScript running on that page makes a series of JSON GET requests to gather all of the necessary information to compose its display and it makes a few AJAX POST requests to report back to the server on certain bits of status. But now, I wanted to trigger those POSTs programatically on a schedule rather than waiting for customers to trigger them. The dilemma is that I'd already written this relatively sophisticated JavaScript application that makes all the necessary requests, implements the business logic, and knows how to POST in the data. I had two options: redo all of that work again in my server-side application (ick!) or figure out a way to trigger this JavaScript code by automating its runtime enviornment (the browser).

After a half day's research, here's what I discovered: there's a Firefox extension that allows other applications to establish JavaScript shell connections to a running Mozilla process via TCP/IP. It's called JSSH. Once you've got JSSH installed and running in Firefox, you can open a telnet connection to the browser that allows you to automate it using JavaScript commands to do things like load new pages or even manipulate the DOM on pages you've loaded. You can then automate this interaction using any scripting language with a telnet library. For the remainder of this post, I'll provide step-by-step instructions for running JSSH and for automating it with Ruby.

Install JSSH

The easiest way to install JSSH is to download the JSSH.xpi and open it with Firefox which will offer to install the extension (if you're interested in compiling Firefox with it from scratch or installing an existing binary, you should read these instructions).

Start Firefox with JSSH

Once you've got a copy of Firefox with JSSH installed, you'll need to run it. You can do this by providing the correct options when launching Firefox from the command line. On Mac OS X, that looks like this: /Applications/Firefox.app/Contents/MacOS/firefox -jssh & The "&" at the end of that line will background your command so it doesn't take over your terminal session.

Telnet into the JavaScript Shell

Once Firefox is running, we can use telnet to log into JSSH like so: $ telnet localhost 9997 Trying ::1... telnet: connect to address ::1: Connection refused Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. Welcome to the Mozilla JavaScript Shell! >

Load a URL from JSSH

Now that we're in, we can tell Firefox to load pages for us, thusly: var w0 = getWindows()[0] var browser = w0.getBrowser() browser.loadURI("http://www.urbanhonking.com/ideasfordozens") And that's it! If the JavaScript application I wanted to run lived at "http://urbanhonking.com/ideasfordozens", we'd be done. That command would load the page and Firefox would interpret and run the JavaScript it found there.

Now, all we've got left to do is make it so that we can trigger this process from our application code. So, we'll...

Automate the Process with Ruby

Like any good scripting language, Ruby has a telnet library, which means that once we've got an instance of Firefox running with JSSH enabled, we can talk to it from Ruby whenever we want. Here's an example script that logs into the telnet shell and loads a series of URLs one at a time: require 'net/telnet' my_urls = ["http://urbanhonking.com/ideasfordozens", "http://atduskmusic.com", "http://grabb.it", "http://pdxpopnow.com"] # start telnet session with the Firefox javascript shell and setup browser object puts "starting telnet session" firefox = Net::Telnet::new("Host" => "localhost", "Port" => 9997) firefox.cmd "var w0 = getWindows()[0]" firefox.cmd "var browser = w0.getBrowser()" # load each page my_urls.each do |url| puts "loading...#{url}" firefox.cmd "browser.loadURI('#{url}')" sleep 10 # so that the browser has time to load even if the page is slow end firefox.close

Further Research: Screen Scraping JavaScript Heavy Sites

What else might this rickety bridge we've built to the JavaSript runtime environment be good for? One thing that immediately occurs to me is: screen scraping for sites with a lot of JavaScript. Another side effect of the rise of rich JavaScript applications has been to create intractable problems for people trying to do screen scraping. If the data you want is not in the page's HTML when you request it in the first place but is only written in later when the page's JavaScript runs then traditional spidering and screen scraping techiques will fail to find it. Freebase, the open database application built by Danny Hillis and his team, for example, uses a highly dynamic interface for presenting its data that is almost entirely based in JavaScript. Or, on the low-brow side, MySpace uses JavaScript throughout the forms in its interface to help with date picking and such. If you wanted to scrape or automate interaction with either of these sites, you'd need access to a runtime environment that could execute JavaScript.

I haven't really tackled this problem with JSSH, but I do have some leads. For example, here's how you get the html of the document: > browser.contentDocument [object XPCNativeWrapper [object HTMLDocument]] > domDumpFull(domNode(browser.contentDocument)) <HTML><HEAD><META content="text/html... If you want to explore this avenue further, one of the best places to look is Firewatir, a project to add Firefox support to the WATIR browser testing framework. They do lots of click-by-click automation and checking for results, so I'm sure they've figured out approaches for a lot of what you'd confront when screen scraping. The JSSH documentation itself is useful and clear but not the most in depth.

Happy automating! Let me know what you discover...

Tagged: ruby, firefox, jssh, javascript, automation, browser, ajax, json

There's been a lot of hullabaloo lately about Single Serving Sites. Stimulated by the inexplicable runaway success of Barack Obama is Your New Bicylce, these simple sites that provide a small dollop of amusement (isitchristmas.com) or utility (istwitterdown.com) have become all the rage.

Of course, I've been making them for awhile now, e.g. Largehearted Goat and The NY Times Explains the Ratings. At first, creating a new SSS is a satisfying experience. You have a whacky idea and withing a few hours, you've registered a domain, written some simple code, and put something up. But, over time, each one gets to be more and more of a burden. My ideas, at least, have involved sites that need to be constantly updated with new information over time. Since I've always implemented these sites with simple ruby scripts that run on my local machine and then upload the static finished versions of the sites, this has meant keeping an eye on unreliable cron jobs and, sometimes, hand maintenance. And, over the years, I've wondered if there was a better solution.

Today, I took the first steps towards finding one. It turns out that good-ole CGI scripts — so foreign to those of us who's main experience has taken place in the age of sophisticated web frameworks like Rails — make a great basis for SSS development. What follows is a basic introduction to writing and running CGI scripts with Ruby. I'll focus on Dreamhost as a deployment target since it's the service I have access to for this kind of thing and also the issues that arise there are probably not that dissimilar from what comes up with the other shared hosting services that are the natural habitat of SSSes.

Step One: Get Ruby and Rubygems up and running

If you plan on keeping your SSSes extremely spartan simple, you might be able to skip this step. Dreamhost accounts come with an old version of Ruby already installed. If you don't need to install any custom gems for your scripts or control anything else about your ruby environment, you can skip straight down to the step two. However, if you plan on doing anything the least bit more sophisticated — from installing one individual gem all the way up to using Rails itself — you've got a bit of work to do beforey you can get started on the fun part.

Being far from an expert on unix build process and dependencies, I followed Nate Clark's excellent instructions for building Ruby and Rubygems on Dreamhost with just a few discrepancies. My biggest note is that the version of the Ruby source to which he links is out-of-date. I changed his line for download the Ruby source to: $ wget ftp://ftp.ruby-lang.org/pub/ruby/1.8/ruby-1.8.6-p111.tar.gz But, don't trust me! The most recent version of Ruby changes all the time and you can always find the most recent version of Ruby at ruby-lang.org. The one other thing worth noting that I discovered was that in any of the compilations make may die by timeout (Dreamhost kills processes that last longer than a given interval, a major pain point we'll be returning to in a moment when we try to install individual gems). It won't always be clear that a timeout was at fault so if you see a 'configure', 'make', or 'make install' command die mysteriously, just go ahead and try again.

Step Two: CGI Hello World with Ruby

Now that we've got Ruby and Rubygems installed, the first thing we want to do is to just run a basic 'hello world' CGI script to make sure that we've got things configured correctly. Here are the steps:

1. Log into dreamhost over ssh.
2. Navigate to the desired directory within your web space.
3. Create a new file, e.g. test.cgi.
4. Fill it with the following content (if you skipped Step One and are using the build-in DH Ruby, the first 'shebang' line should read: "#!/usr/bin/ruby" instead of what is shown): #!/usr/bin/env ruby require "cgi" cgi = CGI.new("html3") cgi.out("text/plain"){ "Hello World #{Time.now}" }
5. Change the permissions of that file by running: $ chmod 775 test.cgi

That should do it. Navigate to that path in your browser and you should see output that looks something like this: Hello World Sat Mar 01 19:04:53 -0800 2008

Step Three: Install Gems by Hand

Now, if you plan on doing anything more interesting than displaying the date, the odds are you're going to want to take advantage of Ruby's great and growing weatlh of libraries made available as Gems. Unfortunately, the RubyGems server system running at RubyForge has lately become incredibly slow. In order to find packages and detect dependecies, the default 'gem install' command has to communicate with that server and so becomes subject to the ruthless Dreamhost killing of long-running processes. In my experience, trying to install gems on Dreamhost will universally end in frustration: $ gem install feedalizer Bulk updating Gem source index for: http://gems.rubyforge.org Killed What to do?

Fortunately, in addition to using the RubyForge server to search for gems, it's possible to install them directly from .gem files if you can find them for your desired packages. For example, when building a recent SSS I wanted to use Feedalizer — a great library that scrapes web pages and automatically creates RSS feeds from the resulting content — so I hunted down the .gem file via the RubyForge website, downloaded it, attempted local installation, and then (when that failed) chased down the sources for the dependencies (in this case just Hpricot) to follow the same process. To give you a taste of things here were the gory details:

Installing Feedalizer: $ wget http://rubyforge.org/frs/download.php/13797/feedalizer-0.1.0.gem $ gem install feedalizer-0.1.0.gem which threw an error complaining about the need for the Hpricot dependency, so: $ wget http://code.whytheluckystiff.net/dist/hpricot-0.5.140.tgz $ tar xzf hpricot-0.5.140.tgz $ cd hpricot-0.5.140 $ rake gem $ gem install pkg/hpricot-0.5.gem The 'rake gem' step there is necessary because _why only makes the source directory available for direct download rather than an explicit .gem file. After this completes successefully, return to the 'gem install' step for Feedalizer above (be sure to the return to the directory into which you dowmloaded the Feedalizer .gem before running it).

Much bumpier than the smooth ride normally provided by 'gem install', but it'll get the job done. You could clamber your way down similar rutted paths for most any other gem you needed for you script.

And that should be more than enough to get you started. If you run into any troubles, dive into the comments here and on the other blog posts linked for help. Also, if anyone has any experience using lightweight persistence strategies in this context, I'd love to hear about them, espcially if they're file-system only; a lot of my scripts requiring saving records and I tend to use a rickety YAML-based system for them that could stand improving.

Note: Thanks to The Pug Automatic for inspiring me to start down this path in the first place.

Tagged: ruby, cgi, dreamhost, rubygems, sss, single+serving+sites

Welcome to the third in my series of tutorials on getting the most out of I Want Sandy, the "virtual personal assitant" from Portland company Values of N. In the last two installments, I covered how to communicate with Sandy on the go using SMS and Twitter and outlined a Sandy-specific version of the Quicksilver append trick. This time, I'm going to show you how I use Safari's new 'web snippets' feature so that I always have quick access to all the stuff I've stored with Sandy.

I know what you're saying: 'Web snippets? Isn't that a Dashboard feature? Personally, I'd be perfectly happy if Dashboard never launched again!' I know, I know; before figuring out this little trick I felt similarly. But, believe it or not the combination of Sandy and web snippets has actually made Dashboard useful to me for the first time since its inception. So put your well-deserved Dashboard skepticism aside for a moment and listen up.

Now that you've been using my first two tips for awhile, Sandy's starting to know about a lot of your appointments and all your other little tidbits, but what's the best way for her to give you easy access to them? Sure, you can query Sandy for individual bits of information when you're on the go, but there are some things — like your calendar and TODO list — that you've just got to have close at hand in full. Creating web snippets out of Sandy's monthly and tag views is the best way I've found to accomplish this.

A 'web snippet' is simply a chunk of a web page that you select from Safari. Safari then creates a Dashboard widget that will be constantly kept up to date with the current content of that little bit of the web. What we want to do is to create web snippets out of the pages on Sandy that have our most relevant information. For my part, I've settled on the Monthly view and the view for my @computer tag (more about that in a minute). Here's how I got those created:

1. Login to IWantSandy.com/home.
2. Click on Daily Digest > This Month (the 'This Month' link under the Daily Digest drop down).
3. Click the web snippet icon as shown above.
4. Drag the highlight box around the yellow pad part of the screen containing your reminders. Sometimes, the box will lock onto individual sub-portions of the page making selection easier, but you can also drag its corners around as well.
5. Click the "Add" button in the top right.
6. Dashboard will launch. You're done.

Now, you'll have a custom-made widget with all of your reminders for the month available anytime you invoke Dashboard.

So, how do I use this? And what's so great about it? Couldn't I just visit Sandy's perfectly nice website anytime I wanted to view my stuff?

I've made two Sandy-related widgets: one for my appointments for the month and one for my items tagged @computer. I use the @computer tag in a GTD-ish manner to specify TODO items that I can accomplish at my computer (click on the image below to see a larger version):

Each of these uses demonstrates a different big advantage of accessing Sandy from Dashboard. The key is: the AJAX bits of Sandy's interface still work here, so I can edit items and mark them as done directly from these widgets.

For example, when I complete an item from my @computer TODO list, I simply click the 'x' next to its entry in the relevant widget and the item gracefully fades away never to return:

Similarly, I can edit details of appointments if they change, simply by clicking on an item from my monthly calendar. The first click pops up a box with some more details:

Clicking 'Edit' brings up a form where you can change everything from the date to the description to when you want Sandy to send your reminder:

Each of these web snippets really starts to feel like a tiny little custom desktop client for a single specific bit of Sandy's functionality.

One of the big lessons I've learned from my attempts at doing GTD has been the importance of putting your inboxes and other TODOs directly in your path, someplace you'll stumble over them in the course of your normal daily operations. And these little applications are a really lightweight way to do just that, eliminating nearly all the friction involved in maintaining and making reference to my lists and reducing the opportunity for fiddling to a minimum.

Tagged: sandy, gtd, safari, web+snippets

Today I'm proud to celebrate the third anniversary of Ideas for Dozens. In the the three years of this blog's existence, I've written 197 entries (or about 1 post every 5 and a half days) using three different blog engines (after a brief flirtation with Blogger I moved to a Blosxom install for more control before joining Urban Honking and learning about the many mysteries of Moveable Type).

In that time, I've covered topics from art to Ruby on Rails to a series of eccentric projects. And you all have responded, writting 286 comments for an average 1.45 comments per entry. My posts have been picked up by BoingBoing, 43 Folders, and many others.

Anyway, I've got lots of exciting stuff in the works including a series on the basics of microcontroller architecture and assembler programming, a few ideas for phsycial/ubiquitous computing projects, and a few other non-tech projects starting up as well.

So stay tuned and keep those comments coming…

Tagged: idfdz, meta, thrid, anniversary