Open source is answering the call at government agencies on all levels as they look for opportunities to carve out costs and improve security, transparency, public participation, and collaboration. Why? Open source is stable, trustworthy, and secure, and Red Hat solutions are being used across government agencies to create efficiencies, eliminate vendor lock-in, meet mission-critical IT demands, and improve service delivery.

On Monday March 30, Intel announced the availability of their much anticipated new line of processors, the Intel® Xeon® Processor 5500 series–nicknamed Nehalem.

Red Hat, a long-time partner of the market-leading chip maker , collaborated on the chip’s debut, testing and optimizing the recently released Red Hat® Enterprise Linux® 5.3 on the new processor.

Changes include a new processor architecture, platform architecture, memory subsystem, I/O subsystem, and options (including SSD and 10GbE).

So what’s the big deal? Why all the fuss? Here’s just a few of the improvements wrought by the combination of Intel’s processing power and Red Hat advancements in performance and efficiency.

Improved performance

According to Stream performance data, the new Intel Xeon 5500 series processor delivers a 2.25 times performance improvement, when compared to the performance of the preceding processor series (the Intel Xeon 5400). This allows the new processor to handle datacenter workloads at nearly twice the efficiency.

Intelligent performance

The processor can dynamically adapt throughput to the workload. Intel Hyper-Threading Technology lets system administrators increase workloads and add capabilities without slowing the system down—there’s plenty of reserve for usage peaks. Expanded physical server limits in Enterprise Linux 5.3 (255 CPUs and 1 TB main memory) improve system scalability dramatically.

Virtualization

Red Hat Enterprise Linux and Intel Virtualization Technology (Intel VT) deliver high consolidation ratios. These virtualization enhancements provide greater scalability and performance, and allow for the virtualization of a wide range of workloads, even those that are I/O intensive.

Automated energy efficiency

The combination of technologies supports low-latency changes between power states. This can help lower power consumption during off-peak hours . Integrated power gates and memory controllers deliver energy efficiency from the hardware side, while enhanced power management and CPU clock frequency scaling help conserve power from the Red Hat Enterprise Linux side.

Red Hat and Intel have a long history of working together to take open source technology to its full potential. Whether it’s combined open source contributions or corporate partnerships, the x86 platform and the open source software revolution have changed the face of computing. Integrated virtualization—and continued rapid improvements in processor technology—keeps the changes coming.

More information

• Red Hat and Intel: Industry Leaders Redefine Datacenter Price-Performance (Red Hat Press)
• What you need to know about Intel’s Nehalem CPU (ars technica)
• Internet: Meet your new processor (Intel.com)

Red Hat® Enterprise Linux® 4 was released on February 15th, 2005. This report takes a look at the state of security for the first four years from release. We look at key metrics, specific vulnerabilities, and the most common ways users were affected by security issues. We will show some best practices that could have been used to minimise the impact of the issues, and also take a look at how the included security innovations helped.

This report is an update to the three-year risk report published in Red Hat Magazine in February 2007.

1. Introduction

2. Vulnerabilities

2.1. Vulnerability Counts

2.2. Critical Flaws

2.3. Expanding “days of risk”

2.4. Riskiest packages

2.5. Advisory Workload

3. Threats

3.1. Exploits

3.1.1. Kernel exploits

3.1.2. Browser exploits

3.1.3. Other user-complicit exploits

3.1.4. PHP exploits

3.1.4. Servers and services exploits

3.2. Worms

4. Conclusion

5. Further Reading

6. About the Author

1. Introduction

We measure the overall risk of running Enterprise Linux 4 as a function of two factors; the vulnerabilities and the threats. Our first section covers the security vulnerabilities found in packages that are part of Enterprise Linux 4 and the advisories that address them. Our second section covers the threats by examining actual exploitation of those vulnerabilities through exploits and worms.

All the data used to generate this report, tables, and graphs, apply to Red Hat Enterprise Linux 4 AS from release day, 15 February 2005 to 14 February 2009 unless otherwise stated.

2. Vulnerabilities

At first sight it may appear that Red Hat have released a lot of updates for Enterprise Linux 4; in the last twelve months publishing a total of 107 security advisories to address 251 individual vulnerabilities. But in reality this is by far a worst-case metric, as it treats all vulnerabilities as equal, regardless of their severity and assumes a system that has installed every available package – which is not a default or even a likely installation.

With the release of Enterprise Linux 4, we started publishing severity levels with package errata to help users determine which advisories were the ones that mattered the most. Providing a prioritised risk assessment helps customers to understand and better schedule upgrades to their systems, being able to make a more informed decision on the risk that each issue places on their unique environment. Red Hat rates the impact of individual vulnerabilities on a four-point scale designed to be an at-a-glance guide to how worried Red Hat is about each security issue.

2.1. Vulnerability Counts

There are four variants of Red Hat Enterprise Linux 4; two targeted at server solutions with Enterprise Linux AS and ES, and two targeted at client solutions with Enterprise Linux WS and Red Hat Desktop. The package set available in Enterprise Linux WS and Red Hat Desktop is a subset of that available in Enterprise Linux AS.

During Enterprise Linux 4 installation, the user gets a choice of installing either the default selection of packages, or making a custom selection. Table 2 shows the vulnerability counts, normalised by CVE name, for some selected configurations.

A default install of Enterprise Linux 4 AS was only vulnerable to ten critical flaws in the whole four years. This is because most of the critical flaws have been in web browsers and their plug-ins: Firefox and Mozilla/SeaMonkey packages are not installed by default on distributions intended for server systems.

Client systems (Enterprise Linux WS and Red Hat Desktop) do include Firefox, Mozilla, and Helixplayer by default, leading to 126 critical vulnerabilities. A custom installation of AS, selecting every available package, would yield a system affected by the maximum possible number of critical vulnerabilities for the four years, 130.

For the purposes of this study we consider the worst-case scenario, a version of Red Hat Enterprise Linux 4 obtained on the day of release. During the first four years, six Update releases were made (Update 1 in June 2005, Update 2 in October 2005, Update 3 in March 2006, Update 4 in August 2006, Update 5 in May 2007, Update 6 in November 2007, Update 7 in July 2008). The Update releases are similar to a “service pack” and contain a roll-up of all security advisories. So, for example, a user who installed Enterprise Linux 4 in August 2008 would use Update 7 and be affected by only a subset of the issues. We’ve also counted vulnerabilities not advisories; it’s usual for a single security update of a package to fix a number of vulnerabilities at the same time, so the number of advisories and updates needed to be installed is far lower.

Tip

You can cut down the number of security issues you need to deal with by carefully choosing the right Enterprise Linux variant and package set when deploying a new system, and ensuring you install the latest
available Update release.

2.2. Critical Flaws

Vulnerabilities rated critical severity are the ones that can pose the most risk to an organisation. By definition, a critical vulnerability is one that could potentially be exploited remotely and automatically by a worm. However we also stretch the definition to include those flaws that affect web browsers or plug-ins where a user only needs to visit a malicious (or compromised) web site in order to be exploited. Since the vast majority of critical severity issues occurred due to web browsers or plugins, this is why there is such a difference between the number of critical issues that affects a default install of Enterprise Linux 4 AS and WS.

For the purposes of the severity classification we ignore what privileges the attacker would be able to gain: a remote root compromise via something like Samba would be of a much higher risk than a user-complicit Firefox flaw that results in running code as an unprivileged user, but both would be rated as critical on this scale.

To help qualify the risk we’ve split up the critical vulnerabilities into those that require some minimal user interaction to be exploitable (such as if a user visits malicious web page), and those that require no user interaction at all (and therefore could potentially be exploited by a worm).

For Enterprise Linux 4 AS with every package installed, Table 3 summarises all critical issues, and Table 4 breaks out the critical, non-browser flaws.

We’ve included in these tables the “days of risk” metric. This is commonly defined as the number of calendar days it takes for a vendor to produce updates that correct a flaw after the flaw is first known to the public.

Fixes for 87% of critical flaws were available from Red Hat Network the same day or next calendar day after public disclosure of the flaw. This fast response time is a deliberate goal of the Red Hat Security Response Team and forms an essential part of reducing customer risk from critical flaws.

2.3. Expanding “days of risk”

The “days of risk” metric has it’s limitations and so it isn’t particularly useful for comparing different software vendors against each other. The software that makes up the Enterprise Linux 4 distribution is open source, so we’re not the only vendor shipping each particular application. Unlike companies shipping proprietary software, Red Hat is not in sole control over the date each flaw is made public. This is actually a good thing and leads to much shorter response times between flaws being first reported to being made public. It also keeps us honest; Red Hat can’t play games to artificially reduce our #8220;days of risk” statistics by using tactics such as holding off public disclosure of important flaws for a long period, or until some regularly scheduled patch day.

A more useful metric to help assess risk would also take into account the amount of time that each issue was known to the vendor in advance. As part of our security measurement work since Enterprise Linux 4 we’ve been tracking how the Red Hat Security Response Team first found out about each vulnerability we fix. This information is interesting as it can also show us which relationships matter the most to us, and identify trends in vulnerability disclosure.

For each of the 1269 total vulnerabilities, across every package in Enterprise Linux in the 4 years, we determined if the flaw was something we knew about a day or more in advance of it being publicly disclosed, and how we found out ^[1] about the flaw. The results are summarised in Figure 2 and Figure 3.

Red Hat knew about 51% of the security vulnerabilities that we fixed at least a day in advance of them being publicly disclosed. For those issues, the average notice was 21 calendar days, although the median was much lower, with half the private issues having advance notice of 9 days or less. Figure 4 shows the distribution of notice periods in more detail.

2.4. Riskiest packages

In our work tracking and fixing vulnerabilities it sometimes seems like we produce a security advisory for the same packages every month. We therefore analysed Enterprise Linux 4 to find out which packages were
responsible for the most vulnerabilities, weighting them ^[2] to take into account their severity. The results are shown in Table 5, which lists the top 10, ranked across all four years.

These top 10 packages together totaled 79% of all the weighted vulnerabilities. The kernel, cups, php, krb5, and samba packages are part of the default installation of Enterprise Linux 4 AS.

Tip

You can reduce the number of vulnerabilities that will affect your systems by removing packages that you don’t need or don’t use, particularly those that have the worst security history. For example, if you don’t use thunderbird on a machine you could just remove the package.

2.5. Advisory Workload

In previous reports we’ve graphed the vulnerability workload, a measure of the number of vulnerabilities that security operations staff would need to worry about every day, weighted by severity. But the actual effort in maintaining an Enterprise Linux system is more related to the number of advisories we released, rather than the number of vulnerabilities: A single Firefox advisory may fix ten different issues of critical severity, but takes far less total effort to manage than ten separate advisories each fixing one critical Samba vulnerability.

Our Advisory Workload index gives a measure of the number of important advisories that users would need to worry about every day. The higher the number, the greater the workload, and the greater the general risk represented by the vulnerabilities addressed. This workload index is calculated in a similar way to the NIST workload index.

For a given month, Advisory Workload = weighted number of advisories ^[3] / days in the month. A workload of 1.0 would mean one important advisory a day.

Figure 5 shows the advisory workload index for a installation of Enterprise Linux 4 including every package. The initial peak during the first month looks surprising, but is easily explained, as the packages for Enterprise Linux 4 had a code freeze a few months prior to release. This led to a backlog of security issues that were fixed with updates on the date of release. The small peak in August 2005 aligns with the release of Update 1, and the other peaks align with Update releases or months during which there were several Firefox and SeaMonkey
updates.

Tip

Cut down on the number of alerts you receive. Register your systems with the Red Hat Network to get customised notifications for security updates for the packages your systems have installed. If you want to see all security updates for every Enterprise Linux version and package, subscribe to enterprise-watch-list mailing list as well.

3. Threats

The first part of this report analysed the total vulnerabilities found affecting Enterprise Linux 4. But to get a better evaluation of platform risk we also need to take into account the threat. This part therefore looks at
exploits and worms written to take advantage of the vulnerabilities.

Red Hat is continually developing technologies to help reduce the risk of security threats, and a number of these were consolidated into Red Hat Enterprise Linux 4. The most significant technologies were SELinux and
Exec-Shield. Exec-Shield is a project which includes support for the No eXecute (NX) memory permission, simulating NX via segment limits, Position Independent Executables (PIE), gcc, and glibc hardening. For more details, a table of the major security technology innovations in Enterprise 4 is available.

3.1. Exploits

An exploit is the way that an attacker makes use of a vulnerability. The Red Hat Security Response Team monitor numerous sources to track which vulnerabilities are being exploited. For this report we compiled a list of the publicly available exploits for the vulnerabilities that affected the first four years of Enterprise Linux 4.

We are interested in those exploits that have the potential to cause remote damage to the confidentiality or integrity of a system and we therefore don’t include exploits for vulnerabilities that are limited to a denial of service (affecting availability). We do, however, include exploits which are labeled “proof of concept”. A proof of concept exploit may only cause a crash or not quite work properly without modification, but in theory the vulnerability could be exploited properly leading to greater consequences. These proof of concept exploits often show techniques that a skilled attacker can turn into a full exploit.

We found exploits for 59 vulnerabilities for the first four years. 24 (40%) of these exploits are for buffer overflow vulnerabilities where in most cases the Exec-Shield technology should help prevent remote exploitation due to protections such as ASLR and enforcement of a non-executable stack.

3.1.1. Kernel exploits

The public exploits for the Linux kernel lead to one of two consequences: either a local unprivileged user can cause the machine to crash, or a local user can gain privileges.

We found exploits for nine vulnerabilities that had the potential to allow an unprivileged user to gain privileges on an unpatched Enterprise Linux 4 system. Of the nine, one required the target system to be using bluetooth drivers (CVE-2005-0750), another was exploitable only on systems with more than one CPU (CVE-2005-0001), one affected only x86_64 architectures (CVE-2007-4573), and one required a writable sgid directory (CVE-2008-4210).

The remainder (CVE-2006-3626, CVE-2006-2451, CVE-2005-0736, CVE-2004-1235, and CVE-2005-0531) could work on any default, unpatched system. Some of those exploits need unpublished source code adjustments in order to work against an Enterprise Linux 4 kernel.

3.1.2. Browser exploits

Around of quarter of the public exploits we found were for flaws in web browsers; and all but three targeted the Mozilla suite (Mozilla, Firefox, Thunderbird). These are detailed in Table 6. For each exploit, any resultant code execution would be limited to being run with the same rights as the user that is running the vulnerable browser. It is best practice to never use a web browser or email client as root. Some of these exploits are also blocked if JavaScript is disabled.

3.1.3. Other user-complicit exploits

The next class of exploits are those we term ‘user-complicit’, in that they need some involvement from the victim to be exploited. Some examples of user involvement would be opening a malicious file with a vulnerable application, or viewing an instant message from an unknown user. Table 7 lists the exploits we discovered that require some user involvement.

3.1.4. PHP exploits

During March 2007 the “Month of PHP bugs” uncovered a number of issues, some of which affected the PHP packages as distributed with Enterprise Linux 4. The PHP interpreter does not offer a reliable sand-boxed security layer (as found in, say, a JVM) in which untrusted scripts can be run, so any script run by the PHP interpreter must be trusted with the privileges of the interpreter itself. Therefore, in analysis of these issues, exploits which relied on an “untrusted local attacker” were not classified as security-sensitive since no trust boundary was crossed.

This leaves us with the exploits shown in Table 8. These exploits rely on the victim having PHP scripts installed that use the vulnerable PHP functions in a particular way or with untrusted data. In each case the default SELinux targeted policy for Apache would restrict what a successful exploit is able to do.

3.1.5. Servers and services exploits

Our final class of exploits are those that affect server applications and services, in Table 9. These are the most serious threats.

Tip

The way to reduce your risk from exploits is to make sure your systems have all applicable security updates installed. The Red Hat Network can help keep track of this.

3.2. Worms

Worms take advantage of vulnerabilities in order to compromise systems, then use the compromised system to seek out other systems to infect. By our definition, any vulnerability that could be exploited in this way would be classed as severity critical. In the first section of this report we listed every vulnerability that was rated as critical severity and showed that only a subset of those vulnerabilities could be actually used by worms. This is because we also class as critical some browser vulnerabilities where a victim has to take action (for example visiting a malicious web page) and therefore are not exploitable by a worm.

Worms affecting Linux platforms have been quite scarce in the last few years, and the anti-virus vendors who track malware recorded only two (although some variants of each exist) during the four year period of this study:

• Linux/MARE was discovered in November 2005 and was a worm that spread by exploiting a flaw in PHP-Nuke. PHP-Nuke is not shipped as part of Red Hat Enterprise Linux.
• Linux/Lupper was also discovered in November 2005 and was a worm designed to exploit CVE-2005-1921, a flaw in the PHP PEAR XML-RPC server package exploitable through a number of third party PHP applications. None of the affected third-party applications were shipped as part of Red Hat Enterprise Linux. Additionally, a PHP update in July 2005 fixed the underlying flaw in PHP. Even users that had not patched were also protected from this worm by the default SELinux configuration.

Without critical vulnerabilities to allow attackers to remotely exploit machines, we saw attackers instead try to focus on exploiting weak configurations. During the period of this study we tracked attempts by attackers to exploit machines with stolen passwords and brute-force password tools. The tools simply looked for internet-accessible SSH services they could connect to, then tried to log in with lots of combinations of common usernames and passwords. Restricting access to SSH remotely, moving the SSH daemon to a different port, and making sure all your users have strong passwords or use key authentication are all useful defenses against this particular attack.

4. Conclusion

The aim of this report was to get a measure of the security risk to users of Red Hat Enterprise Linux 4 during the first four years since release. We’ve shown that although on the surface it looks like Red Hat released a large number of security advisories, many of them do not apply to usual or default installations, and only a very small subset are a high risk. We’ve shown:

• A default installation of Enterprise Linux 4 AS was vulnerable to ten critical security issues over the first four years
• A customised installation of Enterprise Linux 4, selecting every package, would have been vulnerable to 114 critical browser security issues, and 16 in non-browser packages in the four years. 87% of those vulnerabilities had fixes to correct them available from the Red Hat Network within one calendar day of them being known to the public
• Red Hat knew about 51% of security issues affecting the first four years of Enterprise Linux 4 in advance. The average time between Red Hat knowing about an issue and it being made public was 21 days (median 9
  days)
• We found public exploits for 59 vulnerabilities that could have affected a customised full installation, although the majority relied on user interaction or non-default settings. Attempts to use many of the exploits would be caught by standard Enterprise Linux 4 security innovations
• The most likely successful exploits allowed a local unprivileged user to gain root privileges on an unpatched Enterprise Linux 4 machine
• Two worms targeting Linux systems were found during the four years, but both affected third party PHP applications not shipped in Red Hat Enterprise Linux 4. In addition, an update to PHP released over three months before one of the worms was released protected systems that had installed the third party applications

It would be foolish to draw conclusions about the future state of security in Red Hat Enterprise Linux 4 solely on the basis of this analysis of the past, however what we’ve tried to do is to enumerate the level of vulnerability and threat and hence overall platform risk. Red Hat treats vulnerabilities in our products and services seriously and the policies of the Red Hat Security Response Team are specifically designed to reduce the risk from security vulnerabilities:

• We place an emphasis on providing the fastest possible, highest quality, turnaround for critical vulnerabilities. We have a Security Response Team distributed globally which can draw on significant Engineering and Quality resources to get the things that matter the most fixed quickly
• We release updates for critical and important security issues as soon as possible rather than batching them into monthly or quarterly updates
• We provide transparency in the handling of vulnerabilities, our methods, and our metrics

All of the raw data used to generate the statistics in this report along with some tools to analyse them are available from the Red Hat Security Response Team. We also provide other tools and data that can help security measurement including CVE mappings for all our advisories and OVAL definitions.

5. Further Reading

• What’s new in security for Red Hat Enterprise Linux 4
• Vulnerability Types for Enterprise Linux 4
• Security Features in Red Hat Enterprise Linux and Fedora Core
• SELinux: A New Approach to Secure Systems
• Red Hat Enterprise Linux 4 Security Guide
• Statistics and data from the Security Response Team

6. About the author

Mark J Cox lives in Scotland and is Director of Red Hat Security Response. Over the last 14 years, Mark has developed software and worked on the security teams of some of the most popular open source projects including Apache, mod_ssl, and OpenSSL. Mark is a founding member of the Apache Software Foundation and the OpenSSL project, and a board member of the Mitre CVE project. In his spare time he finds geocaches with his family and occasionally plays music.

Red Hat Enterprise Linux 5.3 was released today, around 8 months since the release of 5.2 in May 2008. So let’s use this opportunity to take a quick look back over the vulnerabilities and security updates we’ve made in that time, specifically for Red Hat Enterprise Linux 5 Server.

The chart below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server as if you installed 5.2, up to and including the 5.3 release, broken down by severity. I’ve split it into two columns–one for the packages you’d get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will probably be somewhere between the two.

For a default install, from the release of 5.2 up to and including 5.3, we shipped 45 advisories to address 127 vulnerabilities. Seven advisories were rated critical, 21 were important, and the remaining 17 were moderate and low.

For all packages, from the release of 5.2 up to and including 5.3, we shipped 61 advisories to address 181 vulnerabilities. Seven advisories were rated critical, 28 were important, and the remaining 26 were moderate and low.

The 7 critical advisories were for just 3 different packages:

1. Five updates to Firefox (July, July, September, November, December) where a malicious web site could potentially run arbitrary code as the user running Firefox. Given the nature of the flaws, ExecShield protections in Red Hat Enterprise Linux 5 should make exploiting these memory flaws harder.

2. An update to Samba (May) where a remote attacker who can connect and send a print request to a Samba server could cause a heap overflow. The Red Hat Security Response Team believes it would be hard to remotely exploit this issue to execute arbitrary code due to the default enabled SELinux targeted policy and the default enabled SELinux memory protection tests. We are not aware of any public exploit for this issue.

3. An update to OpenSSH (August), provided to mitigate an intrusion into certain Red Hat computer systems. The attacker was able to sign a small number of tampered packages, but they were not distributed on the Red Hat Network. We classified this update as critical to ensure any tampered packages would be replaced with official packages.

Although not of critical severity, also of interest during this period were the spoofing attacks on DNS servers. We provided an update to BIND (July) adding source port randomization to help mitigate these attacks.

Updates to correct all of these critical vulnerabilities (as well as migitate the BIND issue) were available via Red Hat Network either the same day or one calendar day after the issues were public.

In fact, for Red Hat Enterprise Linux 5 since release and to date, every critical vulnerability has had an update to address it available from the Red Hat Network either the same day or the next calendar day after the issue was public.

To compare this with the last updates, we need to take into account that the time between each update is different. So looking at a default installation and calculating the number of advisories per month gives the following chart:

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and, in some cases, block exploits for certain flaw types completely. For 5.2 to 5.3 there were two flaws blocked that would otherwise have required updates:

1. A double-free flaw in unzip. The glibc pointer checking limited the exploitability of this issue to just a crash of unzip, a client application, which does not
have security implications. No security update was needed.

2. Two format string flaws in c++filt. The format string protection caused these issues to have no security implications. No security update was needed.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but it isn’t really useful for comparisons with other versions, distributions, or operating systems. For example, a default install of Red Hat Enterprise Linux 4 AS did not include Firefox, but 5 Server does. You can use our public security measurement data and tools, and run your own custom metrics for any given Red Hat product, package set, timescales, and severity range of interest.

See also: 5.1 to 5.2 risk report

Python has a good reputation for tasks like systems programming, network programming, and scripting, but Python for the web is becoming red hot. Part of this has to do with the very popular web framework Django, that was developed at a newspaper to help quickly create Content Management Sites. . Another reason is that Google App Engine–Google’s Cloud Computing offering for developers–only exposes a Python API.

If you are new to Python Web Development, then I’d recommend Django, as it is ideal for building CMS-type applications, social networking websites, and blogs. On the other hand, If you want a hacker’s framework, you might want to give Pylons a look.

Please note: By hacker, I am referring to the kind of hacker Eric Raymond refers to when he writes, “Becoming a hacker will take intelligence, practice, dedication, and hard work. Therefore, you have to learn to distrust attitude and respect competence of every kind. Hackers won’t let posers waste their time, but they worship competence — especially competence at hacking, but competence at anything is valued.”

Ok, so what problem does a hacker’s framework solve that a framework like Django doesn’t? According to some of the Pylons developers, their framework is geared to solve 80/20 problems. Most people—80% of people–want to build blogs, and CMS-type applications. And for that 80%, Django works just great. Of course, the other 20% is where Pylons comes in to play as a “hacker’s framework.”

Philosophically, Pylons is quite different. Pylons abstracts third-party libraries, such as WebOb, Mako, SQLAlchemy, Routes, and Beaker, to make a “hacker’s brew.” These libraries are loosely coupled–not in a internet marketing sense, but in a computer science sense. This means that it is quite easy to swap out the ORM, or templates, or URL routing, and create some alternate development stack. Note, that another hacker’s framework, Werkzeug, also follows a similar philosophy (See the references for more details).

What does a hacker’s framework buy you? Well, it allows you to change your web development paradigm. You no longer need to think in terms of what you can do with the choices defined by a framework. It allows the experienced developer to transcend this potential trap, and think about the actual problem at hand. For example, it may be more productive to start by using SQLAlchemy (the object-relational mapper) by itself.

Once the data model is working as expected, then it could turn into a command line tool, then potentially a WXPython application instead of a web application. Hacker’s frameworks let the developer decide what is best for them at any given situation. Additionally, by focusing on loosely coupling the “best of breed” components, it allows the user of a “hacker” framework, to use literally, the best component for the job. This extra power can come in handy with more complex problems.

Setup

In this article we dive into building an AJAX blog using Pylons. We cheat by using the bookmark tagging site, Del.icio.us as an admin interface that we don’t have to write. When a user creates content on Delicious, the Google AJAX feed API allows that content to be displayed locally on the web page. This is one of the most efficient type of blog that someone can create, as it reuses existing code, APIs, and services.

Building a Pylons AJAX blog

To get started you can download the whole example and run it, or you can follow the steps below. Note that in version control, each step is a separate Pylons project.

[ Step 1] http://code.google.com/p/pyatl-pylons/source/browse/#svn/trunk/1

1: Download http://www.pylonshq.com/download/0.9.7/go-pylons.py

ngift@noah][H:10471][J:0]# python gopylons.py --no-site-packages pylonsblog
New python executable in pylonsblog/bin/python
Installing setuptools..........................done.
Searching for Pylons
[snip]

Note: You will need to meet all requirements of the package.

2:

cd pylonsblog

3: Active environ

source bin/activate

4: Make project

mkdir -p src
paster create --template=pylons ajaxblog

5: Get Pylons running

cd ajaxblog

paster serve development.ini

6: Make a controller

paster controller blog

cd ajaxblog/controller to verify

go to page:

http://localhost:5000/blog

See Hello World, change it.

"Hello Red Hat Magazine"

[Step 2] http://code.google.com/p/pyatl-pylons/source/browse/#svn/trunk/step2

7: Change front page

Add your own content to public/index.html

8: Add a couple of templates, and hook up to blog controller

Create base.html
Create blog.html

Add this line to blog.html:

Hello from blog.html template: Red Hat Magazine

9: Change controller to render template

Edit controller/blog.py to this:

class BlogController(BaseController):

    def index(self):
        # Return a rendered template
        #   return render('/some/template.mako')
        # or, Return a response
        return render('/blog.html')

[Step 3] http://code.google.com/p/pyatl-pylons/source/browse/#svn/trunk/step3

10: Get Buzzword compliant: Adding AJAX, RSS, Mashup and Google

A. Sign up for AJAX RSS Developer Key:

http://code.google.com/apis/ajaxfeeds/

B. Add javascript Code to base.html
C. Call feed div in blog.html

11: See RSS Feed appear

If you go to localhost:5000/blog you will see my last RSS feeds

[Step 4] http://code.google.com/p/pyatl-pylons/source/browse/#svn/trunk/step4

Bonus Points: Try to add persistent comments using the SQLAlchemy ORM on your own!

Summary

In this article we explored some of the ideas behind Pylons, a hacker’s framework, and how it differs from a philosophy framework, like Django or Ruby on Rails. Note by philosophy framework, I am referring to how a developer must abide by the opinions of the developer of the framework, such as in the case of Django templates where a developer is handcuffed against running code in the template. For example, their idea of perfection might be different then yours. Some people refer to this as being an “opinionated” framework. Each one has its purpose, and place. If you just want to make a Content Management website MS website for your American Literature class project, then maybe a hacker’s framework isn’t suited for you or the project, as it is overkill. On the other hand, if you want maximum flexibility, power, (and, of course, “street cred”) you might give a hacker’s framework a try.

Finally, we got into making an actual Pylons AJAX blog that used Delicious to suck in feeds. This was accomplished by reusing code via the Google AJAX Feed API. Ok, enough talk, get to hacking….

Extra Credit: If you are interested diving into a more complex Pylons project template on your own take a look at this source code url. Jonathan Ellis, a Python hacker known for his work with SQLAlchemy, has donated a do-it-yourself tutorial on using FormAlchemy to create a simple blog in Pylons. I have included a link to his original article on the topic in the reference section.

References

Pylons Book: http://pylonsbook.com/alpha1/toc
Google AJAX Feed API: http://code.google.com/apis/ajaxfeeds/
SQLAlchemy: http://www.sqlalchemy.org/
Using SQLAlchemy: http://www.ibm.com/developerworks/aix/library/au-sqlalchemy/
How To Become a Hacker: http://www.catb.org/~esr/faqs/hacker-howto.html
Google App Engine: http://code.google.com/appengine/
WebOb: http://pythonpaste.org/webob/
Source Code For Example: http://pyatl-pylons.googlecode.com/svn/trunk/
Werkzeug (An alternate hacker’s framework): http://werkzeug.pocoo.org
Loose Coupling Computer Science Definition: http://en.wikipedia.org/wiki/Loose_coupling#Definition
Pylons FormAlchemy How To: http://spyced.blogspot.com/2008/10/formalchemy-10.html
Extra Credit Project: http://code.google.com/p/pyatl-pylons/source/browse/#svn/trunk/bonus_project_form_validation_jonathan_ellis

About the author

Noah Gift is the co-author of “Python For Unix and Linux” by O’Reilly, and “Google App Engine in Action” by Manning. He is an author, speaker, consultant, and community leader, writing for publications such as IBM Developerworks, Red Hat Magazine, O’Reilly, Manning and MacTech. He has a Master’s degree in CIS from Cal State Los Angeles, B.S. in Nutritional Science from Cal Poly San Luis Obispo, is an Apple and LPI certified sysadmin, and has worked at companies such as, Caltech, Disney Feature Animation, Sony Imageworks, Turner Studios, and–most recently–WetaDigital.

People often wonder how to get new capabilities—new packages, new features in existing packages, or even bug fixes—included in Red Hat Enterprise Linux. The process for doing so is straightforward, but may be foreign to those with a background in traditional software products.

To summarize, the process is:
1)Get the new code accepted upstream.
2)Get it included in Fedora.
3)Get it included in Red Hat Enterprise Linux.

Although this article focuses on the Linux kernel, the steps apply to all Red Hat Enterprise Linux components and packages.

The key element in the process is that Red Hat tracks upstream. This means that Red Hat works closely with the open source community. Any new features must first be accepted upstream before they’re added to Red Hat Enterprise Linux.

There are numerous benefits to this approach. The biggest is that it keeps the OS and its users closely aligned with Linux as it evolves. There are no dead-end branches, incompatible features, or Red Hat-specific changes that must be maintained. New features added to Linux and key packages are easily integrated. It also means that all new development can and must be accepted by the community before integration.

There are also a number of challenges to the open source model. Some of these are misconceptions, while others have at least a kernel of truth.

Open source: Multiple paths to new capabilities
With proprietary software, only the owner of the software can add new features and capabilities. This makes the question of new functions quite straightforward. There is one source for new features, and they either agree to or reject a request for a new feature. If they don’t agree to add the requested function, there’s no recourse.

With open source development, there are many ways to add a new capability:

• The original author of the package can add it.
• The Linux distributor (such as Red Hat) can add it.
• You can customize your installation by adding it yourself.
• You can contract with or persuade someone to add it for you.
• Perhaps most importantly, anyone can add it.

With open source, it isn’t a question of who can add or enhance a feature or capability, it is a question of how widely that feature will be adopted. This changes the dynamics of adding new features from one of focusing solely on writing the code that implements the feature to one of addressing creation, integration, acceptance, and adoption.

Of course you can make any changes you want to your own copy of the code. But if you want broad use and support of your new features or bugfixes, you need to go through an acceptance process where other groups are persuaded to distribute and support the new code.

There is no single point of control in open source—there is no one who can order a new feature to be accepted. Red Hat can’t demand that others accept new developments by Red Hat. By virtue of being a recognized leader, we have considerable influence, but we must work within the community structure. This community includes not just programmers, but interface designers, testers, documentation writers, project managers, support people, and marketers.

So what’s Red Hat’s involvement?
Red Hat is both a developer/contributor and an integrator. We consciously chose to do targeted development and to build on the work of thousands of other participants in the open source community. That is, we want to be a part of this community, rather than going on our own. In some cases we look to the people with a vested interest in a new feature—for example, a driver for a new piece of hardware—to develop the needed software and push it upstream (i.e. get it accepted).

We here at Red Hat are pleased to bring you a brand new set of videos aimed at showing off the latest and greatest enhancements in our technologies–featuring the very people who helped create them in the first place. The “SPOTLIGHT ON” series highlights the ways in which collaboration drives innovation by looking at projects that have been improved by community input. In our first installment, we track down Red Hat’s own Karsten Wade and Stephen Smoogen from the University of New Mexico to talk about Extra Packages for Enterprise Linux (EPEL), the Fedora-sourced repository of add-on packages for Red Hat Enterprise Linux. They discuss how EPEL is a tool for user-driven innovation that comes from and benefits enterprise customers with more stable code and lower business costs.

This past winter, Red Hat announced the release of a product called MRG–a computing platform that features high-speed messaging and allows high-throughput computing, realtime transactions, and workload management. Not sure what all that means? We weren’t either. So we contacted Bryan Che, the project manager for MRG, to see if we couldn’t get a few questions answered. He obliged, and so we bring you the MRG QandA. Still have questions of your own you want answered? Comment and let us know…

Red Hat has been working on the technologies behind MRG for quite some time–each of the components in MRG has had years of development. For example, Red Hat has been working on realtime technologies in the upstream kernel community for over seven years. Messaging has had a
similarly lengthy development history. Condor, the technology behind our grid scheduler, started development in the 1980’s!

We started work on these technologies because we saw the need for these capabilities, even if we didn’t know when or how we were going to bring
these technologies to market yet. For example, messaging is at the heart of enterprise computing. We had needs for messaging infrastructure at Red Hat–for building out our own capabilities around things like virtualization management. Many of Red Hat’s customers were asking us to provide an open source messaging offering. So, we started working on the AMQP specification and our messaging implementation, even though we didn’t know it was going to end up in something called “Red Hat Enterprise MRG”.

As we started working with customers and the community around the various technologies in MRG, it became apparent to us that the technologies had reached a point of maturity where we could support our most demanding customers with them. Also, we saw significant opportunities for building out fundamentally new capabilities by integrating messaging, realtime, and grid into one platform. And so, MRG was born.

We released MRG v1 at the Red Hat Summit on June 19, 2008. MRG v1 offers support for messaging and realtime, and grid is in Technology Preview. We’ll release a 1.1 update to MRG that will bring grid into full support as well.

JP Morgan Chase, like other investment banks, uses messaging for everything from executing stock trades to providing feeds of market data
to internal data distribution.

Realtime provides deterministic performance. The US Navy is deploying realtime in its DDG 1000 naval destroyers. Realtime is critical in this
environment, because the ships’ computers have to respond precisely without ever pausing, freezing, or getting out of sync with other
events. Otherwise, the results could be disastrous.

One of our large manufacturing customers has been working with Red Hat to build an on-demand grid in Amazon’s EC2 cloud environment for the times it needs access to a grid for calculations. Because this customer isn’t able to utilize fully a dedicated grid, having the option to deploy a grid in the cloud provides them significant cost savings and flexibility.

There isn’t an ideal customer–ultimately, we think that almost any customer will benefit from MRG. MRG provides a new platform and solution for many of the most pressing problems that enterprises face today. We have significant customer interest from many industries.

Having said that, many of our largest customers are MRG early adopters, such as investment banks like JP Morgan Chase, telco companies like
Alcatel Lucent, and multiple agencies in the US Government. We are also working across oil&gas, animation studios, Internet, shipping, stock exchanges, defense, travel, and so on.

MRG takes special advantage of and is highly optimized for Linux to deliver its performance. Additionally, at Red Hat, we have been driving
changes into Linux itself in order to benefit things like messaging performance. So, the fact that we are focusing on just one platform and optimizing both that platform and our implementation on that platform gives us tremendous gains (Note: everything we do is open source and contributed back to the community).

For example, we have written a new high performance journal for durable or persistent messaging that is highly tailored to Linux’s I/O model.
By using this journal, MRG Messaging can achieve throughputs up to about 500,000 durable messages/second/LUN. This rate is about 100 times
faster than other messaging solutions. For more details, you can read Carl Trieloff’s entry in the Red Hat Press blog.

Yes. For example, we support messaging clients across a wide variety of platforms and languages, from Linux to Solaris to Windows, and from C++ to Java/JMS to scription languages like Python. On the grid side, we’ll support scheduling to both Linux and Windows. And, of course, since we integrate with virtualization, this gives us a lot of flexibility in running on other operating systems.

We definitely see a lot of interest in cloud computing from customers. MRG integrates with cloud providers like Amazon EC2 so that you can dynamically provision and add capacity in the cloud from your grid scheduler. This means, for example, that you could have a scenario where you fully utilize your local data center but have additional work you want to compute.

MRG can automatically provision, say, 1000 extra servers for you at EC2, send your work over, get your results back, and tear down the servers when you’re done–all automatically. Some of our other customers are looking at provisioning most or all of their capacity in the cloud because they won’t utilize a data center fully and want to save on capital expenses.

In either case, one of the powerful features of MRG is that it can blend local capacity with cloud capacity. This means you don’t get locked into one cloud provider, and you can grow your infrastructure dynamically in the cloud or in your local data center.

One of the significant things about AMQP is that it is the first protocol standard for business messaging. All other standards, like JMS, aren’t comprehensive enough and don’t specify down to the wire level to provide true interoperability and an open ecosystem. So, I’m not concerned about competing standards–there aren’t really any right now. That’s why there is so much interest in AMQP.

I think that most big businesses will understand and appreciate what AMQP has to offer. Notably, many of the big businesses driving AMQP are not vendors but users. Eventually, if you want to work with these users, you’re going to have to adopt AMQP.

MRG is important to all of our offerings–it’s pretty strategic and central to many of the things that Red Hat is doing. MRG adds realtime capabilities to Red Hat Enterprise Linux and enables you to provide flexible scalability and performance for applications running on Red Hat Enterprise Linux. We’re working with Red Hat Network so that you can provision and manage MRG with our standard management tools. MRG Realtime and a realtime JVM from IBM or Sun can provide deterministic performance for JBoss Java applications. We’re working with the JBoss team to support MRG Messaging as a messaging transport for the JBoss ESB. And, many of our core products and technologies are using MRG technology. IPA and oVirt, for example, are both leveraging our messaging capabilities for distributing data.

More information

• Read more about MRG at the Red Hat Press blog.
• See the official MRG announcement.
• The official MRG pages on redhat.com.

Unlike Red Hat Enterprise Linux versions 2.1 and 3, there is no kernel-source package in the Red Hat Enterprise Linux 4 distribution. It was deemed redundant to provide a kernel-source package and a kernel .src.rpm package at the same time. Users that require access to the kernel sources can find them in the kernel.src.rpm file.

In Red Hat Enterprise Linux 4, The kernel-devel package includes the kernel headers files and you no longer require the kernel source package to build a third party kernel module. To install the kernel-devel package run the following command as root user in a terminal:

#up2date kernel-devel

A full source tree is not required in order to build modules against the current kernel you are using. You can simply point your Makefile to /lib/modules/`uname -r`/build. A more detailed explanation can also be found in the Release Notes.

If you require the kernel source package for reasons other than building a kernel module, you can obtain it in Red Hat Enterprise Linux 4 by typing the following as root user in a terminal:

# up2date redhat-rpm-config rpm-build

# up2date --get-source kernel

# rpm -ivh /var/spool/up2date/kernel*.src.rpm

# cd /usr/src/redhat/SPECS

# rpmbuild -bp --target=i686 kernel-2.6.spec

# cp -a /usr/src/redhat/BUILD/kernel-2.6.9/linux-2.6.9 /usr/src

# ln -s /usr/src/linux-2.6.9 /usr/src/linux

Note: This will build the source tree for a x86 based architecture. For different architectures, (i.e. x86_64) pass the appropriate target variable (i.e. rpmbuild -bp --target=x86_64 kernel-2.6.spec )

Once completed, a symlinked directory pointing to the latest Linux 2.6 kernel source should be available:

# ls -lt /usr/src
total 28
lrwxrwxrwx   1 root root   12 Mar  2 16:36 linux -> linux-2.6.9/
drwxr-xr-x  20 root root 4096 Mar  2 16:21 linux-2.6.9

Note:The steps are also provided in the Red Hat Enterprise Linux 4 Release Notes: http://www.redhat.com/docs/manuals/enterprise/RHEL-4-Manual/release-notes/as-x86/

This information has been provided by Red Hat, but is outside the scope of our posted Service Level Agreements (https://www.redhat.com/support/service/sla/) and support procedures. The information is provided as-is and any configuration settings or installed applications made from the information in this article could make your operating system unsupported by Red Hat Support Services. The intent of this article is to provide you with information to accomplish your system needs. Use the information in this article at your own risk.

Red Hat’s customer service and support teams receive technical support questions from users all over the world. Red Hat technicians add the questions and answers to Red Hat Knowledgebase on a daily basis. Access to Red Hat Knowledgebase is free. Red Hat Magazine offers a preview into the Red Hat Knowledgebase by highlighting some of the most recent entries. The information provided in this article is for your information only. The origin of this information may be internal or external to Red Hat. While Red Hat attempts to verify the validity of this information before it is posted, Red Hat makes no express or implied claims to its validity.

Here’s the final installment of Bill Nottingham’s series based on the talk he gave at this year’s Red Hat Summit. Find out about the latest and greatest Fedora™ developments… and the future of Red Hat® Enterprise Linux® from this experienced engineer. Missed the first part? Catch up in the archives.

Network handling

Another area that’s shown a lot of improvement since Enterprise Linux 5 is networking, especially for desktop and laptop computers. In Fedora 9, we’ve greatly enhanced NetworkManager, and as a result, have switched to NetworkManager by default for all installs. Some of the features we’ve added to NetworkManager include:

• MobileBroadband support – NetworkManager now supports configuring access via GSM and CDMA cards for even greater connectivity options.
• System configuration support – NetworkManager now reads my system configuration , as configured via anaconda or system-config-network. This allows support for things such as static IPs.
• Multiple device support – NetworkManager will automatically connect to both wireless and wired devices simultaneously. This means that if I disconnect the wired device, I’ll have seamless access through my wireless device, instead of having to wait for it to associate and get an IP address.
• Connection editing – NetworkManager also includes a connection editor. With this, I can easily configure my wireless network, my mobile broadband connection, or even 802.1x for my wired connection.

From the application side, we’re working on getting more and more apps tied into the NetworkManager infrastructure so they will automatically adapt to changing networks. For example, Firefox in Fedora 9 will now automatically go into offline mode if the network goes away, and go back online as soon as it returns.

In the future, we’re looking at extending NetworkManager to support Ipv6, as well as more device types (such as bridging and bonding devices).

Encrypted devices

One of the most requested features since the release of Enterprise Linux 5 is encrypted device support. We support encrypted devices via a technology called LUKS. LUKS, implemented on top of the existing device-mapper cryptography code, standardizes the partition header for the automatic detection of encrypted devices. It also allows for multiple passphrases to decrypt the device. For example, if I insert an encrypted USB stick, the encrypted device is detected via HAL, the GNOME file manager prompts me for the passphrase, and LUKS unlocks the device–which is then mounted and ready to use.

Fedora 9 goes even further: We support encrypting the entire system in the installer if the user desires. Anaconda will prompt for a password for physical devices, and then on boot, you’ll be prompted for that password before proceeding.

Audio handling

The addition of PulseAudio has greatly improved the audio subsystem in Fedora 9. PulseAudio is a networked sound server, used for mixing and playing audio streams on the system. Now, some of you might remember ESD, and wonder why we need another sound server. Simply put, the power and flexibility provided by PulseAudio is far beyond what ESD ever did.

For example, with PulseAudio I can easily adjust the volume of the audio streams individually. If I’m listening to music and I get a SIP call via ekiga, I can mute the music.

PulseAudio abstracts away your hardware devices as well. Say I decide to plug in a USB headset. I can then move any (or all) of my audio streams to that headset, while they’re playing. If I remove the headset, the audio streams are automatically moved back to the remaining device.

We’re working to use PulseAudio natively by all the applications shipped in Fedora and Enterprise Linux, and also use PulseAudio’s network support as a potential means for doing virtualized audio.

User switching and constrained users

Say I’m using my computer. My daughter comes up and says she wants to check her mail. In Fedora, we’ve made it easy for you to switch between users. When I log in, I see my user name on the panel. If I click there, my session is locked and a new login window will launch. From that window, I can select a different user. When this user logs out, I’ll automatically be switched back to my session.

If you notice, there’s a ‘Guest’ entry in the user list. The simple listing belies what’s available–this account is more than just an login named ‘guest’. Through cooperation between the Desktop and the SELinux teams, we’ve introduced a technology called ‘xguest.’

XGuest is a restricted kiosk-type user. I can log in as the guest user with no password. The guest user is specially confined via SELinux and only certain actions are allowed. For example, the only way to get out on the network is via the browser. Furthermore, on logout, any changes, customizations, or data saved by the guest user is thrown away so the next user will start with a clean slate (and guests can do little permanent damage). For example, I can change the background of the desktop. The next time I log in as the guest user, it will be reset to normal.

Virtual file systems

Another feature that was added in Fedora 9 is something called GVFS. GVFS is a userspace-based virtual filesystem. It replaces gnome-vfs in GNOME, adding many new features.

Let’s look at an example. One of the backends for GVFS is an archive mounter. I can enter and examine an archive–such as an ISO image or a tarball of source code– via the file manager by simply right-clicking and choosing ‘Open with Archive mounter.’ Once it’s open, I can navigate and open items with standard GNOME tools. Anything mounted with gvfs will also show up both in the GNOME ‘Places’ menu, and in the file chooser.

GVFS adds a new feature that makes it even more useful. Leveraging the power of FUSE userspace filesystems, anything mounted by GVFS is also exposed to your ’standard’ Linux tools under ~/.gvfs. I can navigate into it with a shell, and read/write files to these locations without actually porting your apps to GVFS.

GVFS has back-ends for archive mounting, Samba/CIFS shares, DAV shares, sftp network mounts, OBEX (for talking to your bluetooth phone), and more. In future releases, we plan on extending the backends available for GVFS, and porting more of the desktop stack to use it natively.

Virtualization

If you’ve been around any sort of technical presentation in the past two years, you’ve certainly heard about virtualization. That’s another area we’re working on improving in Fedora, although a lot of that improvement is done under the covers. For example, if I start virt-manager now, it looks much the same as it does under Red Hat Enterprise Linux 5.

However, the virtualization hypervisor has changed from Xen to KVM. KVM offers many benefits over Xen:

• KVM uses the standard kernel

  Xen runs as a hypervisor that requires a modified kernel underneath. Since it runs a (somewhat) non-standard kernel, it can lead to compatibility problems, especially for anything that has to call into, or use, the BIOS. For example, power management has had problems working under Xen, and console redirection and handling was often an interesting exercise., as anyone who’s used a serial console will tell you. By using the standard kernel, KVM allows full hardware compatibility–whatever works in the standard kernel, works in your virtualized host.

• KVM is in the upstream kernel

  KVM has been accepted into the upstream Linux kernel. This is a great step for virtualization, in that it will now always be available, and won’t need continual porting to newer kernels. As anyone who has followed Fedora knows, attempting to maintain the Xen kernel patchset against the upstream kernel is a large amount of work. Working in the upstream obviates the need for that effort, allowing work to be done on improving the virtualization experience rather than chasing the kernel of the day.

But wait, you say… what about my paravirtualized guests? We’ve got you covered. Fedora 9 introduces new technology called Xenner. Xenner emulates the Xen hypervisor interface as a thin layer on top of KVM.

This shows the power of our virtualization strategy–by abstracting the interfaces away via tools like libvirt and virt-manager, we can change out the virtualization hypervisor, yet still present the same interface to the user and administrator, and run the same guests.

And more…

There’s even more where this comes from–new versions of Firefox, new firewire stacks, and more. If you want to know what’s down the road for Red Hat Enterprise Linux, check out Fedora 9– that’s where the innovation happens.

About the author

Bill Nottingham is an engineer at Red Hat, where he’s worked on Red Hat Linux, Red Hat Enterprise Linux, and Fedora for the past ten years. (yipes!) He currently serves on the Fedora Project Board and the Fedora Engineering Steering Committee, and maintains a variety of packages in Fedora.