Beefing up the Python Shell to build apps faster and DRYer – Ben Plesser

One of the great mantras in Python is  avoid repetition.  And yet, when working on a Django app or with a new third party library, I often find myself stuck in the same pitiful cycle:

Start the Shell; encounter a bug or unexpected behavior; close the Shell; make some changes to my code; restart the Shell.  And so on….

This sucks. Not only does it end up wasting 3-4 seconds for each Shell restart on my Macbook Air (those seconds really add up over time), it also inevitably leads to soul crushing frustration.

As an alternative to incessantly restarting the Shell, you could use the builtin Python reload() function, which reimports a given module within the program.  Unfortunately, issues still crop up:

reload() must be passed a module as an argument, meaning you have to import the entire module at some point in your program. Typing reload(module_name) still takes too much time if done ad naseum Django models.py modules can’t be reloaded normally due to the AppCache singleton. People forget about the builtin in the heat of the moment, it just happens.

My solution (inspired by the builtin, multi-threaded Django server) was to add an auto-reloading thread to Shell_Plus.

Shell_Plus, an the extremely helpful script found in the Django Extensions library, is a Django Management Command which spins up an embedded Shell.  Within the embedded Shell, the script sets a number of objects in the global scope on startup via (i.e. your Django models and settings variables).

The major change here is that, before entering the mainloop of the IPython Shell, a Watchdog observer thread (another great library) is kicked off, which listens for file system events.  When a relevant event occurs, the thread automatically reloads the module into the global scope of the embedded Shell via a global dictionary.  It’s fast and completely transparent to the Shell user.

This rather large gist (https://gist.github.com/4499367) contains the code for the reloader thread, along with a heap of documentation.

At this point, I should mention that I decided to add some black magic to make the reloader as powerful as possible.

When a class definition is changed in a file and reloaded in the shell, all of the instances of the old version of that class inside the Shell’s global scope are dynamically assigned the reloaded class.

old_instance.__class__ = RefashionedKls

The implications of that last point are a little crazy. While inside a pdb debugger, you can add, delete, or modify class methods and immediately see the result inside the Shell session.  I’ve used it to great effect while debugging and generally experimenting with code but the danger is obvious so beware. Again, check out the gist to see more details.

You can find my Django Extensions fork at https://github.com/Bpless/django-extensions.

Even if you are not developing a Django app specifically, you can build off this concept of auto-reloading embedded Shells for any Python project.