Ohio LinuxFest

I’m recovering from the 2010 Ohio LinuxFest. Between talking about Python to Linux sysadmins for a solid day, the evening gatherings, the hallway track, and everything else, I very nearly lost my voice. But that was a small price to pay.

For my “Python for Linux System Admistrators” tutorials everything went quite well. There was a minor delay waiting for a projector, but otherwise the venue was fine and I had a good turnout – some 20-25 people who (in addition to being sysadmins) were a joy to teach. By the end of the day, everyone was tired, but I was pretty happy that we’d covered a core of Python that would a) let them get going on their own and b) was particularly relevant to their jobs. If anyone is interested some materials for the course are here. The slides are just as I used them, meaning that in some cases they are just prompts to remind me of where I’m going, rather than a standalone resource. But anyway, they might be of some use.

In addition, the bookstore sold a few copies of Quick Python Book, 2e, I caught a few good talks, and more important, I got to talk to several interesting people – maddog, Jorge Castro and Amber Graner from the Ubuntu community, Linux scribe Brian Profitt, Robert Blackwell from the Perl community (I promise I’ll be nicer when I talk about Perl from now on), and several others, not to mention a rather inebriated Buckeye fan and even my high school girlfriend. How can you beat that?

Python for Linux at OLF

I’m in the final (but not as final as I would like) stages of preparing for my day-long tutorial at Ohio LinuxFest. OLF, as we call it, is a great event, with some good keynotes, interesting talks, and even maddog. Not to mention first rate tutorials, such as, oh… “Python for Linux System Administration”.

The morning session I’ll spend on basics – writing scripts that illustrate control flow, lists, dictionaries, strings, etc. from the point of view some basic sysadmin scenarios. I’ll also introduce the basics of the subprocess module to call other Linux tools.

Then in the afternoon session, we’ll look at some more involved tasks, like traversing files systems, regular expresssions, daemons, using the network, etc.

I’m looking forward to it – I think it will be a blast.

So if anyone has any cool intersections between Python and Linux sysadmin you wouldn’t mind me stealing, or any other suggestions or words of wisdom, by all means let me know.

IndyPy Meetup and Python 3

This past Thursday, I gave a talk to the Indianapolis Python Meetup. It was, in a word, a blast. The venue was “The Boardroom” at Scotty’s Brewhouse, which was really nice – a big table, with centrally located connections to a 42″ flat panel monitor on the wall. And Scotty’s also has good burgers and beer, so it has to be the most comfortable place I’ve ever given a talk.

There were 14 of us, and both the Python knowledge and interest level were high. The guys from sixfeetup.com provide a certain amount of support, particularly co-organizers Calvin Hendryx-Parker and self-designated PEP-8 nazi Clayton Parker. I talked about Python 3, hitting both the features that I consider improvements to the language and the changes that cause the most pain in porting old code.

The talk was well received, the burgers and beers were great, and I was delighted to find such a vibrant group of Pyhtonistas in Indy. I hope to return.

Seven Languages, Seven Weeks – Io

I’ve fallen behind, but not given up, on my seven week tour of different languages as I read Bruce Tate’s Seven Languages in Seven Weeks: A Pragmatic Guide to Learning Programming Languages. The second language in the tour is Io, a prototype-based language created by Steve Dekorte. Io is not mainstream, and perhaps not yet fully production ready, but it is interesting. In fact, part of my delay has been in digesting the very different paradigm that Io presents.

In a language like Io there are no classes – to create an instance, you clone something – either Object or something else that has ultimately been created by cloning Object or something else that has been created by cloning Object or something else… well, you get the idea. And everything about an object can be overridden, since those attributes are “slots” that can be arbitrarily assigned.

In Io everything is handled as messages passed to objects, and the way that any object handles any message (a slot, again) can be overridden. So in Io, ‘hello world’ becomes:

“Hello, world” print

That is, send the print message to the “Hello, world” object. And what’s in the print slot can be redefined. Once you get in the habit of looking at things that way a lot of options open up. Add to that some pretty slick concurrency handling and Io starts to look pretty intriguing.

I can’t describe Io in the small space of a blog post, so I’m not going to try. And in fact that would be exactly my criticism of Seven Languages. For such a different (from the procedural/OO norm) language, the text gets the reader in to deep water very fast. The thinness of explanation in the text quickly lead me to the web, which is sort of what the author intended. The problem is that looking at Io material on the web quickly became more absorbing than the text (what’s this? Python bindings for Io?) and it was only belatedly that I came back and played with the books’ examples.

And since I carped about it in an earlier post, I have to mention that I found it a little irritating that the author had to describe overriding an object’s forward method (the mechanism for searching prototypes to find a method) as being equivalent to Ruby’s method_missing, which seems to be a bit of a stretch (and a bit Ruby-centric). Io isn’t Ruby, and while it can be made to do something equivalent, it simply doesn’t have the same features.

On the other hand, I found the choice of Io to be a good one, and completely in line with the intention of the book. I didn’t really have any feel for prototype-base languages before, and now I feel I do.

One of the comments on my post about the Ruby chapter suggested that if I was upset about Ruby, Io would give me a heart attack. In fact, my heart rate ended up being quite low. I had quite a bit of fun playing around with Io and figuring out how to create a DSL to produce XML. I think the reason is Io’s consistency and simplicity. EVERYTHING is objects with slots being passed messages, and that’s pretty much it. Once you accept that, Io behaves as you would expect.

While I’m not entirely sure that Io is ready for production use, I can definitely see implementing a DSL in it, and I may well be be back to experiment more.

Python and recovering Firefox sessions

First let me confess, I am a tab-aholic…

I tend to have several Firefox windows open at once, with several tabs in each. And I leave them open for… well… weeks. And usually that’s fine – it restores everything if I have to reboot or restart Firefox, and all is well. Except when it doesn’t.

Sometimes, for whatever reason, the sessions aren’t restored, and all of those tabs, as in “hey, I wasn’t quite done with that!”, are lost.

When this happened to me last night, I took a look in my Firefox profile directory and discovered something about the way Firefox stores it’s sessions. I’m sure it’s well-known to many, but it was news to me.

When I looked at sessionstore.bak (the automatically created backup of sessionstore.js) I saw something like:

({"windows":[{"tabs":[{"entries":[{"url":"http://stoa.canterburyschool.org:9080
/stoa/courses/Faculty/TechnologyGuides/VoIPandSkypeinForeignLanguage
/document_edit_form","title":": VoIP and Skype in Foreign Language","ID":2528274837}...

Hmmmm... I know that's Javascript, but it also looks a lot like a Python dictionary. Why not see if Python could eval() it to a dictionary? My first try failed on the unknown identifiers "true" and "false", but after defining true and false as equal to Python's True and False, it worked. Now I had a nested set of dictionaries containing all of my previous session's info. Not bad.

[Edit: Not being a regular JSON user, and since the file extension was .js, not .json, I never considered that this might be JSON. It turns that it's not QUITE JSON and the Python JSON library won't decode it (without at least stripping off the leading and trailing parentheses). Check out this discussion for more info.]

From there it was easy to create some code that would grab the URL's from an unrecoverable session and put them into an HTML page for easy access later.

Warning: this code uses eval() on arbitrary strings loaded from an arbitrary file, so it is not secure - use with great caution. If it wipes your hard drive, sells your house and absconds with the money, and kicks your dog on the way out, you have been warned.

In any case, below is my session recovery code, which I've already used a couple of times.

---

import sys
""" this code is released to the public domain and meant only for illustration. It is not
    warranted as being safe or suitable for any particular purpose at all.

    in particular, this program evaluates (and potentially executes) arbitrary code,
    so use with caution.
"""

def get_session(infilename, outfilename):
    true = True
    false = False
    sessionstring = open(infilename).read()
    session = eval(sessionstring)

    outfile = open(outfilename, "w")
    for window in session['windows']:
        outfile.write("---------------Window-----------------<p>\n")
        outfile.write("<ul>\n")
        for tab in window['tabs']:
          outfile.write( '<li><a href="%s" \="">%s</a> - %s</li> \n' %
                         (tab['entries'][0]['url'],tab['entries'][0]['url'],
                          tab['entries'][0]['title']))
        outfile.write("</ul>\n")
    outfile.close()

if __name__ == '__main__':
    infile = sys.argv[1]
    if len(sys.argv) == 3:
        outfile = sys.argv[2]
    else:
        outfile = "sessionsave.html"

    get_session(infile, outfile)

Seven Languages, Seven Weeks – Ruby

I just finished the first stop on my seven language tour. Ruby was in some ways an easy start, although not as easy as I would have thought. While Ruby is enough like Python to seem familiar, I was still surprised by some of the differences I found.

The author of the book, Bruce Tate, is a Rubyist as I understand it, and he was definitely pushing metaprogramming as one of Ruby’s cool features. And in that department (and in others) Ruby definitely has some cool tricks.

After some easy build-ups, including arrays and hashes (pretty much equivalent to lists and dictionaries in Python), a sort of slick implementation of a Roman numeral class, and using inject to cumulatively operate on all of the elements of an array, the final problem to be solved in Ruby is creating a module which creates a class to read a CSV file. The file is opened automatically when an object of the class is instantiated, assuming a filename derived from the class name. That is, if the class is ‘ActAsCsv’, it will open a file ‘actascsv.txt’.

The first line of the CSV file is assumed to be a header containing the names of the fields, and one is supposed to create a CsvRow class which supplies accessor methods corresponding to each field. So if the first line contains “one, two, three”, you should be able to access the third field of any row with ‘row.three’.

A few things in this exercise left me feeling uncomfortable. First of all, the automatic opening of a file based on the class name, struck me more a parlor trick to illustrate Ruby’s capabilities for introspection, than a practical strategy. While I can do the same thing in Python, I can’t think of a single instance, even in simple scripts, where I would be happy linking all instances of a class to a single file, or even constraining what file is opened. I try to write my classes to handle arbitrary file names instead. I may be making too much of this, but as a teacher I’ve come to really dislike teaching examples which make you say “huh?” right at the beginning.

I’ve always advocated to my programming students that they write programs in such a way that runtime errors are obvious. If an error matters, it should be as obvious as possible, so that you can find and fix it. so I was surprised and a little uneasy to discover that Ruby seems to be taking the opposite approach – if a field or method or array slot is missing, just return nil and carry on. That’s right, no index out of bounds exceptions… ever.

In fact, Ruby takes things one step further. Objects have a overridable method ‘method_missing’ which gets called whenever you try to access a method that the class doesn’t actually have. The default version raises a NoMethodError exception, but once you override it, the behavior is up to you. This is powerful – if you construct your method correctly you can do some interesting things, including dynamically supplying field accessors for rows in CSV file. OTOH, if you don’t construct it right, if you don’t remember to raise a NoMethodError for the truly unexpected values, you run the risk of masking important information.

In my solution, the contents of the row are a hash in the CsvRow, and the method_missing implementation of the accessors is really just syntactic sugar hiding a key, value hash lookup. Add to that the fact that Ruby returns a nil rather than raising an exception if a key isn’t found, and my code can’t tell the difference between an empty field and a missing one. I freely admit my solution may not be the “Ruby way”, but I would argue it’s at least plausible.

So while I’m glad I had the chance to play with a bit of Ruby, and I’m still liking the books approach, I end up with two issues, one regarding Ruby and the other with the example problem. The first is what I’ve just mentioned – I was guided into using Ruby in a way that I think conceals potential bugs, and Ruby was complicit in that.

The second issue is with the exercise. While I’ll certainly concede that Ruby has some syntactic flexibility that Python doesn’t, this problem doesn’t really present a convincing use case. I don’t intend to make my reading of this book into just a ‘so what, I can do that in Python’ response, but in this case it does seem to me that a Pythonic version takes care of the problem more efficiently. If I’m OK with using dictionary access rather than accessor methods – saying ‘row["one"]‘ rather than ‘row.one’ – this example is a non-problem in Python, thanks to the csv module in the standard library.

[Edit: Yes, I know that you can override __getattr__ in Python to provide the same functionality. See the comments for suggested versions. My point was that Ruby's method_missing approach was what made me uneasy. And it's more inviting - in Ruby you would override the method_missing method of a class derived from hash, and nothing else. In Python, I would argue, the surgery is a bit more serious.]

Easy as (Cherry)Py

or “Refactoring a tiny web app in 2 hours”

Last fall I wrote a little web app (less than 500 lines) that ran on the wsgi_ref server that comes with Python.

Yes, I know wsgi_ref isn’t exactly intended for production apps, but I had my reasons. First of all, it was a very bare-bones report extractor/generator aimed at fewer than half a dozen users and I didn’t want to mess with any third party libraries, installation, configuration, etc. Also, after using wsgi_ref in some examples in The Quick Python Book, I was curious as to how useful it might be in a real situation.

It worked okay, and the people concerned were quite happy with their new web-based report extraction system. Of course, as often happens, the more I thought about it (and maintained the code), the less happy I was.

For one thing, the code was a little ugly… well, as I think about it, given it’s small size, it was a LOT ugly. URL’s had be mapped to code explicitly, and the the whole thing was no where near as modular nor as readable as it should have been.

So as I thought about refactoring the app, I decided to see how much work it would be to swap wsgi_ref for CherryPy. I’ve always had a soft spot in my heart for CherryPy – I used it for my very first web app some 7 or 8 years ago, and at PyCon 2009 my sprint time was spent on the very early stages of converting it to Python 3.x.

And as it happened my thoughts of refactoring pretty much coincided with Bob Brewer’s talk about “The Zen of CherryPy” at PyCon 2010. I recommend watching the video of his talk when you have the chance, but I’ll give you the first two points – “Common tasks should be fast and easy” and “Doing nothing should be faster and easier”. That matched my app perfectly, since, as Yogi might have said, 90% of my app was doing common tasks, and the other 90% was doing nothing.

So I was looking for three things: first, how easy is it to port an app from the wsgi_ref server to a server like CherryPy? and second, would it help me improve my code? and finally, does CherryPy follow the “Zen” as Bob presented it? Or to put all three in another way, is it worth the effort to use a server package like CherryPy for such a tiny web app?

Shortly after lunch on Friday (March 19) I grabbed CherryPy and installed it on our vTiger server. Following the tutorial, I shortly had a little demo app working. The next step was to move my main web code into a class. After that, each one of the if statements explicitly mapping a URL was recast as an appropriately named method.

Fortunately, I was smart enough to put the code that actually queries the database and massages the data into the desired report into a separate module, so it remained untouched. After adjusting some parameters and making sure things were correctly exposed, the  main report URL’s were working fine.

I had one entry form, and its HTML template needed a few tweaks. Reading the values from the form post (a common task) was dead easy – they just became parameters to the form action method. I also needed to change the response headers in one method to send a PDF instead of HTML, and that was a simple dictionary access of cherrypy.response. Again, a common task, easily done.

Finally, I needed to adjust the port number and make a couple of other minor changes to the configuration – a quick call to cherrypy.config.update took care of that. As to things like sessions, authentication, compression, etc, I didn’t want them, so I didn’t do anything. Doing nothing to do nothing works for me.

As to the answers to my questions, let me put it this way – my code is now much more modular and readable, and I got the whole refactoring done in time to make a three o’clock appointment.

CherryPy. Definitely worth it.

Quick Python Book, 2e Signing @pycon Friday 3:30 pm

This is just a quick note before I had to the airport to catch my flight to Atlanta for PyCon.

Michael Foord (IronPython in Action) and I (Quick Python Book, 2nd Ed.) will be representing Manning Publications and signing books in the bookstore in the expo hall during the afternoon break on Friday. Come by and say hi!

Get the most out of PyCon – VOLUNTEER

PyCon Atlanta is now less than 2 weeks away, and things are coming together. My big concern, the poster session, is pretty much set to go. Transportation, check. Hotel, check. Conference registration, time off from work, talks I want to catch, tentative open space plans: check, check, check, and check.

Yesterday I added the final piece. Volunteer to be a session runner, check.

As everyone knows, PyCon is a community run conference, and most of us like it that way. That means that the cost is lower,  the speakers and events are what we want and it’s one of the best conferences around. But it also means volunteers. Volunteers to plan and organize it, volunteers to help with registration and setup, and volunteers to help run the sessions. This year PyCon will have 5 tracks and 95 talks and will need 40 session chairs and 40 session runners.

Session chairs are the ones who introduce speakers, keep things on time and moving. Session runners are the ones who make sure that the presenters’ equipment works and that they get to the right room on time. Between the two of them, PyCon’s session staff do more to make sure that a presentation is a success than any other conference I’ve spoken at.

But there’s more. The session chairs and session runners are volunteers. And there aren’t enough of them right now. Hmmm.

So here’s my invitation – sign up to be a session chair or session runner and help PyCon continue to be one of the best conferences around. Trust me, you’ll feel better. And you’ll get to stride authoritatively through the halls of PyCon with a cool 2-way radio. Awesome.

Installing Python 3 on Ubuntu (and Debian) Linux

I’m still seeing a certain amount of traffic hitting this site from queries searching for help installing Python 3 on Ubuntu and Debian Linux systems. Because of what I would call the conservatism of both distros, unless you are using the very latest releases, you won’t have the latest versions available. I do completely understand the reasons both the Ubuntu and Debian communities have for their update policies, but it does put users in a bind if they want to try Python 3 without updating their entire distribution.

Since I’m really behind the notion of people moving to Python 3 (not only did I write a book about it, Python 3 is a better language), let me recap your options for installing Python 3 on Ubuntu systems. The same approach will work for Debian boxes.

Use a package when you can…

As I mention in The Quick Python Book, the easiest and most reliable way to install Python is to use a precompiled package. If you’re using Karmic or later, that’s no problem. The packages are right there and ready to go. Go for the python3 and idle3 packages and you’ll get a complete Python 3.1 environment with no hassle at all.

You might be able to find precompiled Python 3 packages either on Launchpad.net or elsewhere, but I haven’t found any that are both the latest release of Python and that include IDLE. If you find packages that you trust that meet your requirements, by all means, go for it.

But if you can’t…

However, if you can’t get Python 3.1 packages, you will need to do things the hard way. Note that in Jaunty (8.10) there are Python 3.0 packages, but don’t bother – in the year since the initial release of Python 3.0, there have been several incremental improvements, particularly to file I/O, that mean 3.0 is now just history.

So “the hard way”. As far as I know, the only way to get a complete Python3+IDLE install on versions before Karmic is to compile from source. To do that, you need to follow the instructions in this post, except that you should get the latest version of Python 3.x, currently 3.1.1. The install is pretty resilient – that is, you’re likely to get something, even if you’re missing a few dependencies. If you follow the instructions, you should get everything you need and not have to upgrade your distro until you’re darned good and ready.