Virtualenv's bin/activate is Doing It Wrong

I'm a Python programmer and frequently work with the excellent virtualenv tool by Ian Bicking.

Virtualenv is a great tool on the whole but there is one glaring problem: the activate script that virtualenv provides as a convenience to enable its functionality requires you to source it with your shell to invoke it. The activate script sets some environment variables in your current environment and defines for you a deactivateshell function which will (attempt to) help you to undo those changes later.

This pattern is abhorrently wrong and un-unix-y. activate should instead do what ssh-agent does, and launch a sub-shell or sub-command with a modified environment.

Problems

The approach of modifying the user's current environment suffers from a number of problems:

• It breaks if you don't use a supported shell.
• A separate activate script must be maintained for each supported shell syntax.
• What do you do if you use no shell at all? (I.E. run programs in a virtualenv from a GUI.)
• If the deactivate script fails to un-set an environment variable, it may contaminate other environments.
• If you want to edit deactivate or any other function sourced into your environment, you have to kill your shell and re-source the script to see the changes take effect.
• If you change the current directory from one to another virtual environment and forget to carefully deactivate and activate as you do so, you may end up using libraries from or making changes in the wrong one!

Virtualenv's activate suffers from a number of other warts as well:

• You can't simply run the script; you have to learn and employ your shell's "source this script" builtin. Many non-experts frequently stumble over this distinction. Doing away with the recommendation to source a shell script should make virtualenv easier to use.

  # This file must be used with "source bin/activate" *from bash*
  # you cannot run it directly

• In an attempt to preserve the user's old environment, it declares _OLD_VIRTUAL_PATH, _OLD_VIRTUAL_PYTHONHOME, and _OLD_VIRTUAL_PS1, and must define how to restore them upon deactivation. If you happen to want to modify activate to override more variables specific to your environment, you have to do the same.

• Its default means to display whether or not a virtual environment is currently active (modifying the user's PS1 variable) is fragile. On Debian and Ubuntu boxes it becomes confusing if one enters a subshell, or uses a tool like screen or tmux.

• It is not executable, and not meant to be used as an executable, yet it lives in a a directory named bin.

Doing It Right

Entering and exiting a virtual environment should be like using ssh to connect to another machine. When you're done, a simple exit should restore you to your original, unmodified environment.

An example of a program that does this the Right Way is ssh-agent. In order to communicate the port that it uses to other programs, it must set some variables into the environment. It provides an option to do what virtualenv does, but the better way is to simply ask ssh-agent to launch your command for you, with a modified environment. ssh-agent $SHELL will launch a sub-shell for you with its environment already modified appropriately for ssh-agent. Most Debian and Ubuntu machines even launch X11 this way; see /etc/X11/Xsession.d/90x11-common_ssh-agent.

Another advantage to the subshell approach is that it is far simpler than the hoops virtualenv jumps through to activate and deactivate an environment. There's no need to set _OLD_ variables since the former environment is restored automatically. There's no need for a deactivate function.

Finally, employing a prompt context variable instead of messing with PS1 would allow the user to define how that information is presented.

A better activate: "inve"

To differentiate, I'm calling this approach "inve" as in "inside this virtual environment, ..." I'll happily take name suggestions.

Launching a subcommand with a modified environment

How do we make an executable like ssh-agent that launches a subcommand with a modified environment? Easy. Call this my_launcher:

#!/bin/sh
export MY_VAR=xyz
exec "$@"

Calling "my_launcher firefox" will launch firefox with MY_VAR set to 'xyz' in its environment. The environment where "my_launcher" is called from will not be disturbed.

Simplifying activate

Let's now examine bin/activate to see what we can throw away if we assume that the system takes care of restoring the environment for us when we exit. We don't need the deactivate shell function at all. We don't need any _OLD_ variables. We don't mess with the prompt. What's left?

export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME

That's it. Three lines, down from 76. Down from 187 if you count all variants for other shells.

Wrap this with the launcher technique above, call it inve, and ./bin/inve $SHELL spawns a new subshell in the active virtualenv. What if you want a no-argument invocation to default to spawning an activated shell? This is the entire script:

#!/bin/sh
export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
exec "${@:-$SHELL}"

Now bin/inve does what bin/activateshould. By the way: this works for all shells. bash, zsh, csh, fish, ksh, and anything else, with one script.

More hacks

Re-enabling current environment modification

Some users source bin/activate from within their own shell scripts, which I don't find quite as offensive.

ssh-agent also supports this style of use. It too has to deal with the syntax differences between shells to do so. It's not hard to enable this; here's one proposal.

#!/bin/sh
# As above, do what's needed to activate
export VIRTUAL_ENV="/home/mike/var/virtualenvs/myvirtualenv"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
# If the first argument is -s or -c, do what ssh-agent does
if [ "$1" = "-s" ]; then cat <<- DONE
 export VIRTUAL_ENV="$VIRTUAL_ENV";
 export PATH="$PATH";
 unset PYTHON_HOME;
DONE
elif [ "$1" = "-c" ]; then cat <<- DONE
 setenv VIRTUAL_ENV "$VIRTUAL_ENV";
 setenv PATH "$PATH";
 unset PYTHON_HOME;
DONE
# Otherwise, launch a shell or subcommand
else
 exec "${@:-$SHELL}"
fi

Now inve supports the same -s and -c options that ssh-agent does. Where one might previously have written a script like this:

#!/bin/sh
source ./activate
... (commands) ...

One would now write instead:

#!/bin/sh
eval `./inve -s`
... (commands) ...

Or, for csh:

#!/bin/csh
eval `./inve -c`
... (commands) ...

Unfortunately, I don't know if this "eval the output of a command" technique works for all possible shells.

A system-level inve

I find it convenient to employ a "system-level" inve script that lives in my system $PATH, that I can run from anywhere within any virtual environment, and without specifying the full path to 'ENV/bin/inve'. This goes against the intention that "virtualenvs are self-sufficient once created" so I'm not advocating this technique be used instead of ENV/bin/inve.

#!/bin/sh
# inve
#
# usage: inve [COMMAND [ARGS]]
#
# For use with Ian Bicking's virtualenv tool. Attempts to find the root of
# a virtual environment. Then, executes COMMAND with ARGS in the context of
# the activated environment. If no COMMAND is given, activate defaults to a
# subshell.
# First, locate the root of the current virtualenv
while [ "$PWD" != "/" ]; do
 # Stop here if this the root of a virtualenv
 if [ \
 -x bin/python \
 -a -e lib/python*/site.py \
 -a -e include/python*/Python.h ]
 then
 break
 fi
 cd ..
done
if [ "$PWD" = "/" ]; then
 echo "Could not activate: no virtual environment found." >&2
 exit 1
fi
# Activate
export VIRTUAL_ENV="$PWD"
export PATH="$VIRTUAL_ENV/bin:$PATH"
unset PYTHON_HOME
exec "${@:-$SHELL}"

Until an inve-like script gets created in virtualenv bin/ directories, this system-level script will allow you to immediately use the subshell technique with all existing virtualenvs. If ever the inve script does land in virtualenv's bin/, this system level script could be simply a helper that searches for and invokes ENV/bin/inve:

# Locate the root of the current virtualenv
... (same as above) ...
# Activate
exec bin/inve "$@"

Don't mess with my prompt

But what about the prompt? Build a PS1 that does the right thing everywhere without needing to be modified to suit a particular purpose. I tend to have a function that collects all the context info this way, in my .bashrc:

function ps1_context {
 # For any of these bits of context that exist, display them and append
 # a space.
 virtualenv=`basename "$VIRTUAL_ENV"`
 for v in "$debian_chroot" "$virtualenv" "$PS1_CONTEXT"; do
 echo -n "${v:+$v }"
 done
}
export PS1="$(ps1_context)"'\u@\h:\w\$ '

This lets the user control their PS1 and it works everywhere, no matter how many subshells or screen sessions you're nested into. This is the only piece that has to be customized per-shell.

Conclusion

While using activate is intended only a convenience and is not necessary to work within a virtual environment, most of programmers I know treat it as a black box and never do without it. I suspect that, in part, the complexity of the script is what prevents more programmers from avoiding it.

Perhaps the worst part about a popular, useful tool like virtualenv using this antipattern is that many other programmers are adopting it as normative and using it for their own work. virtualenvwrapper and dustinlacewell/capn are two examples. Stop doing this, everyone!