learns_to use Expect for Easy Automation

One of the great hopes you might have in beginning to learn about technology and computers is that they will save you time and effort. This is such an obvious expectation that it almost goes without saying, but, in my experience, it is rarely fulfilled and really unrelated to the true joy of technological learning. That joy comes in gaining whole new abilities, not in slightly improving existing capacities. I’ve was motivated to learn what I have about the web and programming because I wanted to publish my thoughts and my music for anyone in the world to read and hear and there was simply no other feasible way for me to do that. As my technical capacity has grown, I’ve come up with new ideas for things I wanted to do and make that I had never even known were possible. And now these ideas themselves drive me deeper into the technology in order to realize them.

Given this dynamic, I was a little shocked recently to come across Expect. For once, here’s a command line utility that offers a staggering productivity increase without the attendant black hole of necessary technical mastery.

Expect is a tool for automating interactions with other programs. Expect scripts allow you to start up a program and then have the computer act use it in your stead. In your Expect script, you write out a dialogue for the interaction, e.g. ‘if the program says that, respond with this,’ and then the script holds up your side of the ‘conversation’ with the program, providing feedback, entering inputs, making simple decisions.

Why is this useful? With Expect, you can write scripts that fire off relatively complex interactions with a single command, so you don’t have to remember all the individual sub-steps. Or, even sexier, you can automate multi-stage tasks you’ve previously had to do by hand so that you can trigger them with cron so you never have to think about them ever again.

This may sound fuzzy and abstract so far, but Expect scripts actually turn out to be a cinch to write. As proof, I’ll show you the simple script I worked up last night to automate my daily “production process” for Largehearted Goat. In my original post on the subject, I mentioned that the code behind Largehearted Goat required “just a little hand holding.” Here’s what was involved: (1) run the ruby script which reads the Largehearted Boy RSS feed, finds the Goats, and rewrites the html, (2) sftp into my web hosting and copy the new html file over the existing one being served up to Largehearted Goat. And here’s Expect script I worked up to get it all done (paths and passwords have been changed to protect the innocent):


#!/usr/local/bin/expect -f
spawn ruby /path/to/goat/script/goats.rb
expect eof
spawn sftp mylogin@myhost.com
expect -exact "Password:"
send "MySecretPassword
"
expect "sftp>"
send "put /path/to/my/new/html/file/goat.html path/to/my/online/goat/directory/
"
expect "sftp>"
send "exit
"
expect eof


So, here’s how this works. The first line is just a necessary invocation to allow the Expect utility to read a set of commands from a file. The “spawn” command tells expect to start up a process, in the case of the second line, there, I’m running my ruby script. Already here, we have a big advantage over some other shell scripting choices available out there. Step (2), which I described above, only works properly if my ruby script has already been run. Otherwise, it would send the old version of the html up to the web and www.largeheartedgoat.com wouldn’t change. Expect makes it incredibly easy to wait for the completion of that script. All we have to say is “expect eof” (for End Of File). That line tells Expect to wait for control to be returned to it from the previous process that it spawned before proceeding on.

Once the ruby script is done running, then it’s time to go ahead and ftp it the new html file into place. Since my host requires ssh for login, I’ve got to use SFTP (Secure File Transfer Protocol), which I invoke with the next spawn line. From here on in, all I’m really doing is alternating prompts I “expect” to see from SFTP with commands I want to “send” to it. One of the best things about Expect is that if any of these “expect” conditions aren’t met, the script won’t just go ahead with the rest of the interaction running roughshod over your files, but will instead shut down without taking further action.

So, yeah, it’s pretty easy. If you can do this task once by hand using SFTP, you can write this Expect interaction no problem. The only clever thing going on is the use of the new line when submitting a command, like so:


send "MySecretPassword
"


Think of this line break as hitting ‘return’ in order to actually submit the command.

Now, once your script is written, all you’ve got to do is make it executable by running ‘chmod x’ on it and then actually call it like


$ expect my_new_script


You should see all of the normal output of your commands scroll by in the terminal. And once you’ve got it working, you can check out this great crontab tutorial to set it up to run automatically!

I’ve only barely scratched the surface here of what Expect can do. It’s a real programming language, allowing branching based on the response of the program you’re interacting with and a full vocabulary for logic and variables, etc. But even with just this limited Expect vocabulary, I bet you can save yourself a ton of time. Is there a simple process like this that you have to do everyday? Automate it. Is there a complicated interaction you only have to do every once in a long while whose commands you always forget and so have to spend an hour re-googling? Next time you do it, capture it in an Expect script, save it somewhere and then just run it when you need it. Could you spend a long time fiddling with all the different options, improving your Expect chops? Sure you could. But why would you? This one’s easy. This one’s for getting things done.

Tagged: expect, shell, scripting, osx, mac, terminal, automation, crontab, sftp, ruby, largehearted, goat