Rate this page del.icio.us  Digg slashdot StumbleUpon

Writing simple python setup commands

by

Building software in most languages is a pain. Remember ant build.xml, maven2 pom files, and multi-level makefiles?

Python has a simple solution for building modules, applications, and extensions called distutils. Disutils comes as part of the Python distribution so there are no other packages required.

Pull down just about any python source code and you’re more than likely going to find a setup.py script that helps make building and installing a snap. Most engineers don’t add functionality when using distutils, instead opting to use the default commands.

In some cases, developers might provide secondary scripts to do other tasks for building and testing outside of the setup script, but I believe that can lead to unnecessary complication of common tasks.

For those who are not familiar with setup scripts, Figure 1 shows a simple example.

#!/usr/bin/env python
"""
Setup script.
"""

from distutils.core import setup


setup(name = "myapp",
    version = "1.0.0",
    description = "My simple application",
    long_description = "My simple application that doesn't do anything.",
    author = "Me",
    author_email = 'me@example.dom',
    url = "http://example.dom/myapp/",
    download_url = "http://example.dom/myapp/download/",
    platforms = ['any'],

    license = "GPLv3+",

    package_dir = {'myapp': 'src/myapp'},
    packages = ['myapp'],

    classifiers = [
        'License :: OSI Approved :: GNU General Public License (GPL)',
        'Development Status :: 5 - Production/Stable',
        'Topic :: Software Development :: Libraries :: Python Modules',
        'Programming Language :: Python'],
)
Figure 1.

This setup script lists the metadata, maps to a package directory, and provides the general commands expected from distutils setup. This is great, but as I’ve said, it may not do everything you need it to.

What if, for example, we want to be able to see all the TODO tags in the codebase on any platform? In other words, grep won’t cut it.

Let’s start by writing a function that searches for TODO tags in files and prints them back to the screen in a nice format like:

src/myapp/__init__.py (11): TODO: remove me

def report_todo():
    """
    Prints out TODO's in the code.
    """
    import os
    import re

    # The format of the string to print: file_path (line_no): %s line_str
    format_str = "%s (%i): %s"
    # regex to remove whitespace in front of TODO's
    remove_front_whitespace = re.compile("^[ ]*(.*)$")

    # Look at all non pyc files from current directory down
    for rootdir in ['src/', 'bin/']:
        # walk down each root directory
        for root, dirs, files in os.walk(rootdir):
            # for each single file in the files
            for afile in files:
                # if the file doesn't end with .pyc
                if not afile.endswith('.pyc'):
                    full_path = os.path.join(root, afile)
                    fobj = open(full_path, 'r')
                    line_no = 0
                    # look at each line for TODO's
                    for line in fobj.readlines():
                        if 'todo' in line.lower():
                            nice_line = remove_front_whitespace.match(
                                line).group(1)
                            # print the info if we have a TODO
                            print(format_str % (
                                full_path, line_no, nice_line))
                        line_no += 1
Figure 2.

The report_todo function is self-contained and quite simple, if a bit clunky by itself. Who wants to install and use a ‘report-todo’ command on machines just to look for TODO tags?

We want to turn this into a setup command so that we can ship it with our application ‘myapp’ to be used by developers via setup.py. We will probably add more setup commands in the future, so let’s create a class to subclass our commands from. We do this because most of our commands will probably be simple and won’t need to override the *_option methods (though they can if they need to).

import os

from distutils.core import setup, Command


class SetupBuildCommand(Command):
    """
    Master setup build command to subclass from.
    """

    user_options = []

    def initialize_options(self):
        """
        Setup the current dir.
        """
        self._dir = os.getcwd()

    def finalize_options(self):
        """
        Set final values for all the options that this command supports.
        """
        pass
Figure 3.

The SetupBuildCommand defines a few methods that are required for all setup commands. As I stated before, most of our commands will probably not deviate from these defaults, so defining them higher in the object hierarchy means simpler code in the commands themselves. If we do want to add arguments to to any of our commands, we can override initialize_options() and finalize_options() to handle the arguments properly.

Now that we have the SetupBuildCommand to subclass from we can merge our report_todo function into a SetupBuildCommand class. To do this, we create a class that subclasses SetupBuildCommand, and then add a description variable and a run method (which is what is executed when the command runs).

class TODOCommand(SetupBuildCommand):
    """
    Quick command to show code TODO's.
    """

    description = "prints out TODO's in the code"

    def run(self):
        """
        Prints out TODO's in the code.
        """
        import re

        # The format of the string to print: file_path (line_no): %s line_str
        format_str = "%s (%i): %s"
        # regex to remove whitespace in front of TODO's
        remove_front_whitespace = re.compile("^[ ]*(.*)$")

        # Look at all non pyc files in src/ and bin/
        for rootdir in ['src/', 'bin/']:
            # walk down each root directory
            for root, dirs, files in os.walk(rootdir):
                # for each single file in the files
                for afile in files:
                    # if the file doesn't end with .pyc
                    if not afile.endswith('.pyc'):
                        full_path = os.path.join(root, afile)
                        fobj = open(full_path, 'r')
                        line_no = 0
                        # look at each line for TODO's
                        for line in fobj.readlines():
                            if 'todo' in line.lower():
                                nice_line = remove_front_whitespace.match(
                                    line).group(1)
                                # print the info if we have a TODO
                                print(format_str % (
                                    full_path, line_no, nice_line))
                            line_no += 1
Figure 4.

The run method in this example is the same as report_todo but in a method format (with self) and renamed to run.

Now it’s time to bring all this together. If we add Figure 1 and Figure 2 to the setup script (Figure 1), then all that is left is to map the setup command with a command name. Do this via the cmdclass argument, and in the form:

cmdclass = {'name': CommandClass}

Figure 5 shows it all together in an abbreviated form.

#!/usr/bin/env python
"""
Setup script.
"""

import os

from distutils.core import setup, Command


class SetupBuildCommand(Command):
    """
    Master setup build command to subclass from.
    """
    # See Figure 3


class TODOCommand(SetupBuildCommand):
    """
    Quick command to show code TODO's.
    """
    # See Figure 4


setup(name = "myapp",
    version = "1.0.0",
    description = "My simple application",
    long_description = "My simple application that doesn't do anything.",
    author = "Me",
    author_email = 'me@example.dom',
    url = "http://example.dom/myapp/",
    download_url = "http://example.dom/myapp/download/",
    platforms = ['any'],

    license = "GPLv3+",

    package_dir = {'myapp': 'src/myapp'},
    packages = ['myapp'],

    classifiers = [
        'License :: OSI Approved :: GNU General Public License (GPL)',
        'Development Status :: 5 - Production/Stable',
        'Topic :: Software Development :: Libraries :: Python Modules',
        'Programming Language :: Python'],

    cmdclass = {'todo': TODOCommand},
)
Figure 5.

You’ll notice that `python setup.py –help-commands` now shows ‘Extra commands’ with our TODO command listed.

Give it a go and see what you have left to do in your code!

4 responses to “Writing simple python setup commands”

  1. dewente says:

    Hello,

    Nice perfomace, I even don’t know any scripts or languagues programmming.

    Question, which programming languagues is highly require for Red Hat administration position?

    Thanks,
    HB

  2. NexNova » Blog Archive » Links del giorno: April 17, 2009 says:

    […] Red Hat Magazine | Writing simple python setup commands […]

  3. stevemilner says:

    Hello HB,

    As far as I know there is no official programming or scripting language for administrator type positions. I personally believe that it’s a good idea to know at least one high level language and one machine bound language (IE Ruby and C or Python and C++, etc..) as it can give insight into both worlds. If the admin will be hosting web applications then the more dynamic languages known the better as dynamic languages have really taken strongly to the web space (Django, Rails, TurboGears, Pylons, CakePHP etc..)!

    Steve ‘Ashcrow’ Milner

  4. Drew Hates the Net » Blog Archive » Writing simple python setup commands says:

    […] is filed under technical,tips and tricks. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site. This entry was posted on […]