Red Hat Magazine » tips and tricks

Writing simple python setup commands

Steve 'Ashcrow' Milner — Thu, 09 Apr 2009 18:54:19 +0000

Building software in most languages is a pain. Remember ant build.xml, maven2 pom files, and multi-level makefiles?

Python has a simple solution for building modules, applications, and extensions called distutils. Disutils comes as part of the Python distribution so there are no other packages required.

Pull down just about any python source code and you’re more than likely going to find a setup.py script that helps make building and installing a snap. Most engineers don’t add functionality when using distutils, instead opting to use the default commands.

In some cases, developers might provide secondary scripts to do other tasks for building and testing outside of the setup script, but I believe that can lead to unnecessary complication of common tasks.

For those who are not familiar with setup scripts, Figure 1 shows a simple example.

#!/usr/bin/env python
"""
Setup script.
"""

from distutils.core import setup

setup(name = "myapp",
    version = "1.0.0",
    description = "My simple application",
    long_description = "My simple application that doesn't do anything.",
    author = "Me",
    author_email = 'me@example.dom',
    url = "http://example.dom/myapp/",
    download_url = "http://example.dom/myapp/download/",
    platforms = ['any'],

    license = "GPLv3+",

    package_dir = {'myapp': 'src/myapp'},
    packages = ['myapp'],

    classifiers = [
        'License :: OSI Approved :: GNU General Public License (GPL)',
        'Development Status :: 5 - Production/Stable',
        'Topic :: Software Development :: Libraries :: Python Modules',
        'Programming Language :: Python'],
)

Figure 1.

This setup script lists the metadata, maps to a package directory, and provides the general commands expected from distutils setup. This is great, but as I’ve said, it may not do everything you need it to.

What if, for example, we want to be able to see all the TODO tags in the codebase on any platform? In other words, grep won’t cut it.

Let’s start by writing a function that searches for TODO tags in files and prints them back to the screen in a nice format like:

src/myapp/__init__.py (11): TODO: remove me

def report_todo():
    """
    Prints out TODO's in the code.
    """
    import os
    import re

    # The format of the string to print: file_path (line_no): %s line_str
    format_str = "%s (%i): %s"
    # regex to remove whitespace in front of TODO's
    remove_front_whitespace = re.compile("^[ ]*(.*)$")

    # Look at all non pyc files from current directory down
    for rootdir in ['src/', 'bin/']:
        # walk down each root directory
        for root, dirs, files in os.walk(rootdir):
            # for each single file in the files
            for afile in files:
                # if the file doesn't end with .pyc
                if not afile.endswith('.pyc'):
                    full_path = os.path.join(root, afile)
                    fobj = open(full_path, 'r')
                    line_no = 0
                    # look at each line for TODO's
                    for line in fobj.readlines():
                        if 'todo' in line.lower():
                            nice_line = remove_front_whitespace.match(
                                line).group(1)
                            # print the info if we have a TODO
                            print(format_str % (
                                full_path, line_no, nice_line))
                        line_no += 1

Figure 2.

The report_todo function is self-contained and quite simple, if a bit clunky by itself. Who wants to install and use a ‘report-todo’ command on machines just to look for TODO tags?

We want to turn this into a setup command so that we can ship it with our application ‘myapp’ to be used by developers via setup.py. We will probably add more setup commands in the future, so let’s create a class to subclass our commands from. We do this because most of our commands will probably be simple and won’t need to override the *_option methods (though they can if they need to).

import os

from distutils.core import setup, Command

class SetupBuildCommand(Command):
    """
    Master setup build command to subclass from.
    """

    user_options = []

    def initialize_options(self):
        """
        Setup the current dir.
        """
        self._dir = os.getcwd()

    def finalize_options(self):
        """
        Set final values for all the options that this command supports.
        """
        pass

Figure 3.

The SetupBuildCommand defines a few methods that are required for all setup commands. As I stated before, most of our commands will probably not deviate from these defaults, so defining them higher in the object hierarchy means simpler code in the commands themselves. If we do want to add arguments to to any of our commands, we can override initialize_options() and finalize_options() to handle the arguments properly.

Now that we have the SetupBuildCommand to subclass from we can merge our report_todo function into a SetupBuildCommand class. To do this, we create a class that subclasses SetupBuildCommand, and then add a description variable and a run method (which is what is executed when the command runs).

class TODOCommand(SetupBuildCommand):
    """
    Quick command to show code TODO's.
    """

    description = "prints out TODO's in the code"

    def run(self):
        """
        Prints out TODO's in the code.
        """
        import re

        # The format of the string to print: file_path (line_no): %s line_str
        format_str = "%s (%i): %s"
        # regex to remove whitespace in front of TODO's
        remove_front_whitespace = re.compile("^[ ]*(.*)$")

        # Look at all non pyc files in src/ and bin/
        for rootdir in ['src/', 'bin/']:
            # walk down each root directory
            for root, dirs, files in os.walk(rootdir):
                # for each single file in the files
                for afile in files:
                    # if the file doesn't end with .pyc
                    if not afile.endswith('.pyc'):
                        full_path = os.path.join(root, afile)
                        fobj = open(full_path, 'r')
                        line_no = 0
                        # look at each line for TODO's
                        for line in fobj.readlines():
                            if 'todo' in line.lower():
                                nice_line = remove_front_whitespace.match(
                                    line).group(1)
                                # print the info if we have a TODO
                                print(format_str % (
                                    full_path, line_no, nice_line))
                            line_no += 1

Figure 4.

The run method in this example is the same as report_todo but in a method format (with self) and renamed to run.

Now it’s time to bring all this together. If we add Figure 1 and Figure 2 to the setup script (Figure 1), then all that is left is to map the setup command with a command name. Do this via the cmdclass argument, and in the form:

cmdclass = {'name': CommandClass}

Figure 5 shows it all together in an abbreviated form.

#!/usr/bin/env python
"""
Setup script.
"""

import os

from distutils.core import setup, Command

class SetupBuildCommand(Command):
    """
    Master setup build command to subclass from.
    """
    # See Figure 3

class TODOCommand(SetupBuildCommand):
    """
    Quick command to show code TODO's.
    """
    # See Figure 4

setup(name = "myapp",
    version = "1.0.0",
    description = "My simple application",
    long_description = "My simple application that doesn't do anything.",
    author = "Me",
    author_email = 'me@example.dom',
    url = "http://example.dom/myapp/",
    download_url = "http://example.dom/myapp/download/",
    platforms = ['any'],

    license = "GPLv3+",

    package_dir = {'myapp': 'src/myapp'},
    packages = ['myapp'],

    classifiers = [
        'License :: OSI Approved :: GNU General Public License (GPL)',
        'Development Status :: 5 - Production/Stable',
        'Topic :: Software Development :: Libraries :: Python Modules',
        'Programming Language :: Python'],

    cmdclass = {'todo': TODOCommand},
)

Figure 5.

You’ll notice that `python setup.py –help-commands` now shows ‘Extra commands’ with our TODO command listed.

Give it a go and see what you have left to do in your code!

Tips and tricks: How do I force users to change their passwords upon first login?

The editorial team — Mon, 22 Sep 2008 21:51:50 +0000

1.) Firstly, lock the account to prevent the user from using the login until the change has been made:

# usermod -L

2.) Change the password expiration date to 0 to ensure the user changes the password during the next login:

# chage -d 0

3.) To unlock the account after the change do the following:

# usermod -U

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Tips and tricks: How do I add raw device mapping in Red Hat Enterprise Linux 5?

The editorial team — Wed, 17 Sep 2008 21:58:09 +0000

Answer:

The raw devices interface has been deprecated in Red Hat® Enterprise Linux® 5. The rawdevices service and /etc/sysconfig/rawdevices file no longer exist and raw devices are now configured via udev rules. However the preferred method for performing raw I/O (ie. bypassing filesystem caching) is to open EXT3/EXT2 files with the O_DIRECT flag.

This is an excerpt from the raw command’s man page:

Although  Linux  includes  support  for rawio, it is now a deprecated interface. If your application performs device access using this
interface, Red Hat encourages you to modify your application to open the block device with the O_DIRECT flag. The rawio interface will
exist for the life of Red Hat Enterprise Linux 5, but is a candidate for removal from future releases.

Creating the raw devices:
Nevertheless, to create raw devices, add entries to /etc/udev/rules.d/60-raw.rules in the following formats:

For device names:

ACTION=="add", KERNEL=="", RUN+="raw /dev/raw/rawX %N"

For major / minor numbers:

ACTION=="add", ENV{MAJOR}="A", ENV{MINOR}="B", RUN+="raw /dev/raw/rawX %M %m"

Replace with the name of the device needed to bind (such as /dev/sda1). “A” and “B” are the major / minor numbers of the device needed for binding, an “X” is the raw device number that the system wants to use.

If there is a large, pre-existing /etc/sysconfig/rawdevices file, convert it with the following script:

#!/bin/sh
grep -v "^ *#" /etc/sysconfig/rawdevices | grep -v "^$" | while read dev major
minor ; do
        if [ -z "$minor" ]; then
                echo "ACTION==\"add\", KERNEL==\"${major##/dev/}\",
RUN+=\"/usr/bin/raw $dev %N\""
        else
                echo "ACTION==\"add\", ENV{MAJOR}==\"$major\",
ENV{MINOR}==\"$minor\", RUN+=\"/usr/bin/raw $dev %M %m\""
        fi
done

Creating persistent raw devices for single path LUNs:
If using unpartitioned LUNs, to create a single raw device for the whole LUN use this rule format:
```
ACTION=="add", KERNEL=="sd*[!0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601601bd2180072193a9242c3dc11", RUN+="/bin/raw /dev/raw/raw1 %N"
```
Set the RESULT value to the output of scsi_id -g -u -s /block/sdX (where sdX is the current path to the LUN).
This will create the raw device /dev/raw/raw1 that will be persistently bound to the LUN with WWID 3600601601bd2180072193a9242c3dc11.

If using partitioned LUNs, where raw devices are created for each of the partitions on the LUN, use this rule format:
```
ACTION=="add", KERNEL=="sd*[0-9]", PROGRAM=="/sbin/scsi_id -g -u -s %p", RESULT=="3600601601bd2180072193a9242c3dc11", RUN+="/bin/raw /dev/raw/raw%n %N"
```
Again, set RESULT to the output of scsi_id -g -u -s /block/sdX. This will create the raw device(s) /dev/raw/raw1, /dev/raw/raw2, etc. for each partition on the LUN and they will be persistently bound to the LUN with WWID 3600601601bd2180072193a9242c3dc11.
Setting ownership and permissions on the raw devices:
To set specific ownership and/or permissions for the raw devices, add entries to /etc/udev/rules.d/60-raw.rules in the following format:
```
ACTION=="add", KERNEL=="raw*", OWNER=="root", GROUP=="disk", MODE=="0660"
```

Testing and implementing the udev rules:

Before implementing them, use the udevtest command to verify the udev rules work as expected. To verify that the raw device is created for a specific disk or partition, eg /dev/sdb1:

[root@rhel5 rules.d]# udevtest /block/sdb/sdb1 | grep raw
main: run: '/bin/raw /dev/raw/raw1 /dev/.tmp-8-17'

To check ownership/permission settings for a particular raw device, eg /dev/raw/raw1:

[root@rhel5 rules.d]# udevtest /class/raw/raw1 | grep mode
udev_node_add: creating device node '/dev/raw/raw1', major = '162', minor = '1', mode = '0600', uid = '0', gid = '0'

Finally, to actually create the raw device(s), use the start_udev command:

[root@rhel5 rules.d]# start_udev
Starting udev:                                             [  OK  ]

Check that the raw device(s) have been created:

[root@rhel5 rules.d]# raw -qa
/dev/raw/raw1:  bound to major 8, minor 17
[root@rhel5 rules.d]# ls -l /dev/raw
total 0
crw-rw---- 1 root   disk 162,  1 Jan 29 02:47 raw1

Creating persistent raw devices for multipathed LUNs:
Unfortunately it is not possible to write udev rules for creating raw devices on multipath devices (/dev/dm-*) without manipulating existing udev rules. Modifying existing rules for this purpose could cause unforeseen problems and is not supported by Red Hat Global Support Services.
If absolutely necessary, an alternate method for creating raw devices on top of multipath devices could be to create the raw devices in /etc/rc.d/rc.local, so long as the raw device is not required before rc.local is executed. For example:
```
/bin/raw /dev/raw/raw1 /dev/mpath/mpath1p1
/bin/raw /dev/raw/raw2 /dev/mpath/mpath1p2
```

Note: Raw device support is not enabled on the s390 architecture.

https://bugzilla.redhat.com/show_bug.cgi?id=452534

Tips and tricks: What is devlabel, and how do I use it?

The editorial team — Mon, 15 Sep 2008 18:53:13 +0000

Devlabel is a script which manages symbolic links to storage devices on your system. This is accomplished by utilizing the inherent unique identifiers (UUID) that each device should have in order to maintain a correctly pointing symlink in the event that the device name changes (eg. /dev/sdc1 becomes /dev/sdd1).

By adding entries using devlabel, users can instead reference all devices by their symlink and no longer have to worry about what the true name of their device is. Devlabel works with both IDE and SCSI storage.

Devlabel first tries to find the uuid of the partition that you are attempting to add to devlabel by using the /usr/bin/partition_uuid program. Partition UUIDs are supported currently under ext2, ext3, xfs, jfs and ocfs. If no partition UUID can be found (or if it is not a partition you are adding) devlabel will then attempt to find either a SCSI uuid or IDE identifier.

To determine the unique identifier associated with a SCSI device, devlabel uses the program /usr/bin/scsi_unique_id. If this program cannot determine a unique identifier for your block device then the device cannot be used with devlabel.

Example usage of the devlabel command can be found below:

Add a device to be managed by devlabel
```
devlabel add -d  -s
```
Remove a devlabel symlink
```
devlabel remove -s
```
Report the status of all devlabel symlinks
```
devlabel status
```
Get the UUID of a particular device
```
devlabel printid -d 
```

Once a devlabel has been configured files such as /etc/fstab can be modified to reflect these changes and point to the symlink rather than the physical device itself.

The configuration file for devlabel is /etc/sysconfig/devlabel. This file tracks all the current device symlinks. It is not recommended to edit this file by hand.

The devlabel command is available from the devlabel package for Red Hat Enterprise Linux 3 and onwards.
For further information see the Red Hat Enterprise Linux System Administration Guide http://www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-devlabel.html

Tips and tricks: Registering Xen guests with RHN

The editorial team — Thu, 11 Sep 2008 22:05:17 +0000

Question: How do I register my Red Hat Enterprise Linux Xen Guest system with the Red Hat Network?

Answer:

Important: For Red Hat Enterprise Linux 4 fully-virtualized guests install the latest version of up2date.

This procedure will work for both fully-virtualized and para-virtualized guests.

The host system (dom0) must be registered with Red Hat Network. Follow standard procedures to register this system with Red Hat Network (RHN).
Under the System Properties page, subscribe the base system to the appropriate Virtualization Channel. Virtualization allows for up to 4 Xen guest registrations, while Virtualization Platform allows for an unlimited number of Xen guest registrations.
Make sure that the virtualization add on entitlement is checked “Add-On Entitlements: Virtualization *(0 open entitlements)*”. This is available via rhn.redhat.com –> Systems (Top Red Bar) –> Click on the host
system –> Click Edit These Properties –> Check the add on entitlements
box (Virtualization) –> Click update properties. If you have Advanced
Platform entitlement you can check the “Virtualization Platform” box
instead to be able to register an unlimited number of guests for this host.
Make sure that rhn-virtualization-common and rhn-virtualization-host are installed on the host system. If they are not installed, run:
```
yum install rhn-virtualization-common rhn-virtualization-host
```
Install the Xen guest.
Make sure that the Xen guest is running. Please note that xenguest is the name of the Xen guest.
```
xm create xenguest
```
From the command line on the host system, run:
```
rhn_check
```
From the command line on the host system, run:
```
rhn-profile-sync
```
From the Xen guest, run:
```
rhn_register
```

Now the Xen guest will show up as a registered, virtualized system.

Tips and tricks: How to get older kernel versions from Red Hat Network

The editorial team — Mon, 08 Sep 2008 22:39:46 +0000

Question: I recently used up2date to get the latest kernel for my Red Hat Enterprise Linux 3 system and now I have found out that I need an older version for my application that needs kernel 2.4.21-x. How do I download and install a different or an older kernel version then the one I got from Red Hat Network (RHN)?

Answer: Here’s how to get older kernel versions from Red Hat Network (RHN):

Log into RHN.
Select the “Channels” tab then select the “Red Hat Enterprise Linux” channel for your system in the channel name table. [Choose your appropriate channel, Red Hat Enterprise Linux AS, Red Hat Enterprise Linux ES or Red Hat Enterprise Linux WS, the correct version, and the correct architecture.
Select the “Errata” link between Details and Packages.
Search for the Kernel Update in the list:
For Red Hat Enterprise Linux version 3:
- RHBA-2003:308 – Updated kernel resolves 32-bit address space issue on AMD64
  2003-10-30, on page 1
- RHSA-2003:416 – Updated kernel resolves security vulnerability
- RHSA-2004:017 – Updated kernel packages available for Red Hat Enterprise Linux 3 Update 1
  2004-01-16, on page 2
- RHEA-2003:333 – Updated kernel-utils package includes enhanced irqbalance
  2004-01-16, on page 3
- RHSA-2004:066 – Updated kernel packages fix security vulnerability
  2004-02-20, on page 3
- RHSA-2004:183 – Updated kernel packages fix security vulnerabilities
  2004-04-22, on page 4
- RHSA-2004:188 – Updated kernel packages available for Red Hat Enterprise Linux 3 Update 2
  2004-05-11, on page 4
- RHBA-2004:187 – Updated kernel-utils package available
  2004-05-11, on page 6
- RHBA-2004:231 – Updated kernel-utils package adds dmidecode for ia64
  2004-05-30, on page 8
- RHSA-2004:255 – Updated kernel packages fix security vulnerabilities
  2004-06-17, on page 9
Select your appropriate advisory link:
RHBA-2003:308,
RHSA-2003:416,
RHSA-2004:017,
RHEA-2003:333,
RHSA-2004:066,
RHSA-2004:183,
RHBA-2004:187,
RHBA-2004:231, or
RHSA-2004:255.
After that, click on the “Packages” link between Details and Affected Systems.
You should see a list of all the old kernels. Select the kernel you need, download the .rpm file, and run md5sum filename.

You can then install kernel RPM packages with the following command:
rpm -ivh kernel-..rpm

Example:

# rpm -ivh kernel-2.4.21-4.0.1.EL.i686.rpm
# rpm -ivh kernel-smp-2.4.21-4.0.1.EL.i686.rpm
# rpm -ivh kernel-hugemem-2.4.21-4.0.1.EL.i686.rpm

Note: You must reboot your system after the kernel is installed in order to use it.

Tips and tricks: How do I re-enable the rhnsd service?

The editorial team — Wed, 03 Sep 2008 20:09:34 +0000

With many Red Hat Enterprise Linux 4 to Red Hat Enterprise Linux 5 upgrades, the rhnsd service was disabled. To re-enable this service, perform the following steps:

# chkconfig --add rhnsd
# chkconfig rhnsd on
# service rhnsd start

Tips and tricks: My system won’t boot from a burned CD

The editorial team — Wed, 27 Aug 2008 20:48:18 +0000

Question: After downloading and burning ISO files from Red Hat Network (RHN) why will the system not boot from the first burned CD?

Answer:

First, make sure that you have downloaded the correct files. For each distribution there are binary files and source files. To complete an installation the four binary files for a particular distribution are required.

Simply burning these files to CD as files will result in a single file being burnt to CD with a .iso extension. If this occurs your disks will not be bootable. The files available from Red Hat Network (RHN) are disk images and need to be burned to CD as an image.

Your burning software will extract the files from the .iso and burn them to CD. See your specific burning software documentation for more information on how to burn images to CD.

To check if you have burned each image correctly, simply examine the contents of the CD. Instead of a single .iso file the disk should contain multiple files and directories.

Tips and tricks: Alternative to mod_jk?

The editorial team — Mon, 25 Aug 2008 22:27:22 +0000

Question: Is there an alternative module to the mod_jk module that is provided in the standard channel for Red Hat® Enterprise Linux® 5?

Answer: The module mod_jk is currently included as part of the Application Stack channel and is not available in the base channel. However in Red Hat Enterprise Linux 5, the package httpd does provide a module by the name mod_proxy_ajp which helps connect between the two. The configuration file is located at /etc/httpd/conf.d/proxy_ajp.conf. Edit this configuration file according to the needs of the enviroment. At the very least, add i the webapps directory used by tomcat:

     ProxyPass /tomcat/ ajp://localhost:8009/

     ProxyPass /examples/ ajp://localhost:8009/jsp-examples/

Red Hat's customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Tips and tricks: Where is the kernel-source package for Red Hat Enterprise Linux 4?

The editorial team — Wed, 20 Aug 2008 23:00:12 +0000

Unlike Red Hat Enterprise Linux versions 2.1 and 3, there is no kernel-source package in the Red Hat Enterprise Linux 4 distribution. It was deemed redundant to provide a kernel-source package and a kernel .src.rpm package at the same time. Users that require access to the kernel sources can find them in the kernel.src.rpm file.

In Red Hat Enterprise Linux 4, The kernel-devel package includes the kernel headers files and you no longer require the kernel source package to build a third party kernel module. To install the kernel-devel package run the following command as root user in a terminal:

#up2date kernel-devel

A full source tree is not required in order to build modules against the current kernel you are using. You can simply point your Makefile to /lib/modules/`uname -r`/build. A more detailed explanation can also be found in the Release Notes.

If you require the kernel source package for reasons other than building a kernel module, you can obtain it in Red Hat Enterprise Linux 4 by typing the following as root user in a terminal:

# up2date redhat-rpm-config rpm-build

# up2date --get-source kernel

# rpm -ivh /var/spool/up2date/kernel*.src.rpm

# cd /usr/src/redhat/SPECS

# rpmbuild -bp --target=i686 kernel-2.6.spec

# cp -a /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9 /usr/src

# ln -s /usr/src/linux-2.6.9 /usr/src/linux

Note: This will build the source tree for a x86 based architecture. For different architectures, (i.e. x86_64) pass the appropriate target variable (i.e. rpmbuild -bp --target=x86_64 kernel-2.6.spec )

Once completed, a symlinked directory pointing to the latest Linux 2.6 kernel source should be available:

# ls -lt /usr/src
total 28
lrwxrwxrwx   1 root root   12 Mar  2 16:36 linux -> linux-2.6.9/
drwxr-xr-x  20 root root 4096 Mar  2 16:21 linux-2.6.9

Note:The steps are also provided in the Red Hat Enterprise Linux 4 Release Notes: http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/release-notes/as-x86/

This information has been provided by Red Hat, but is outside the scope of our posted Service Level Agreements (https://www.redhat.com/support/service/sla/) and support procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make your operating system unsupported by Red Hat Support Services. The intent of this article is to provide you with information to accomplish your system needs. Use the information in this article at your own risk.