When Sally met Eddie: The Fedora package story
by Jesse Keating
There are lots of ways that software gets included in Fedora releases. Because Fedora is a community-powered Linux distribution, the most common (and likely the easiest) method is when a community member packages and “owns” the task of building it for Fedora. This story follows that process.
How Sally met Eddie
Sally is a Fedora user. She’s been using Fedora for a number of releases and really enjoys the project. However, she’s unhappy with the offered text editors and has been in search of something better. She was introduced to Eddie by a mutual friend in a community forum. You see, Eddie is a text editor, and many feel he’s a rather good one. Sally decides to give Eddie a chance.
Sally looks for Eddie in Fedora. She uses “Add/Remove Software” and searches for the term “Eddie”. She’s rather disappointed to discover that Eddie is not available yet in Fedora. She really wants to give him a chance so she visits Eddie’s home (let’s say he has a web site at eddietheeditor.org). She is saddened to find that there are no packaged version of Eddie for Fedora–just a source download and a Microsoft Windows binary. Not easily discouraged, Sally finds directions on how to download, compile, and install Eddie. It all seems pretty straightforward to Sally, so she does so.
Sharing Eddie with the community
Sally has a lot of fun with Eddie! He does everything she hoped he would do and more. The only sour note in their relationship is that every time Eddie learns a new trick or corrects some bad behavior, she has to download, compile, and install him all over again. Sally thinks this would be much easier if Eddie just came with Fedora.
Sally looks around at fedoraproject.org. There she finds directions for adding software to the Fedora package set. The best part? She discovers that anybody can suggest new software for Fedora!
She also finds instructions on how to make an RPM, which is the software packaging mechanism used by Fedora. She reviews the guidelines for package quality and correctness. At first, this seems like quite a lot of work, but she finds tools in Fedora that makes the process easier.
Sally prepares Eddie for submission. She learns that all Fedora packages must go through a peer review process to ensure the packaging guidelines are followed, the licensing is correct, the package builds correctly, and a number of other checks. To this end, Sally files a package review bug in Fedora’s bug tracking system, Bugzilla.
A fellow Fedora community member, Bob, is also a Fedora packager. He takes a look at the review request. Bob notices that Sally has not packaged anything for Fedora before. This means that she needs a Sponsor–somebody to mentor her in the ways of Fedora packaging. Bob is a Fedora Sponsor and is willing to take Sally on, so he processes the review. He notices a problem in the package submission and alerts Sally to the issues. Sally and Bob work together to bring Eddie up to spec for Fedora.
Sally returns to Fedoraproject.org and reviews what comes next. She reads about checking Eddie into Fedora’s source control system (CVS), and building Eddie for Rawhide. All Fedora packages are built from Fedora’s source control system. This ensures that the build can be reproduced when needed, and that all changes to upstream sources are tracked and managed centrally.
Once Eddie has been checked into CVS, Sally can request a build of Eddie. This request goes to Fedora’s Build System, Koji. Koji takes Sally’s request, checks Eddie out of CVS, and asks its builders to build Eddie for all the architectures that Fedora supports. At the end of each day, an automated process gathers up all the latest builds of Fedora packages and composes an installable package repository named Rawhide. Rawhide is the developmental tree of Fedora.
Each new Fedora release is driven through Rawhide. Sally decides that she would like to help in the development of the next Fedora release, by making sure Eddie works well. She upgrades one of her Fedora machines to Rawhide and installs Eddie. Now Eddie is available to anyone who is running Rawhide.
The Big Day, Eddie’s first Fedora release
Now Sally has Eddie in Rawhide. There is a Fedora release coming up, and she’d like more people to know about Eddie. How can she make it so that Eddie is seen by people during installation?
The package selection screen in the installer and on her installed system is driven by grouping information provided by comps. Comps is a simple XML file that organizes packages in groups, and describes groups with categories. She reads about how to update comps and sets to work. She finds an existing group called Editors within the Applications category and thinks that this would be the perfect place for Eddie.
She adds Eddie to the group, and sure enough in the next day’s rawhide release she can see Eddie in Add/Remove Software within the Editor group. She has marked Eddie as an optional editor. This means that when users view the Editor group, Eddie won’t be pre-selected. That’s OK, because Eddie isn’t for everybody.
In the days leading up to the Fedora release, Sally wants to make sure that Eddie is in tip-top shape. She’s received a few bug reports about how Eddie does things. The fact that people are using Eddie is really awesome in itself, but the users are also contributing to its improvement. She has communicated with Eddie’s upstream authors to report these issues and suggestions. Sally does a few more builds of Eddie for Rawhide. As each snapshot and release candidate comes out , Sally ensures that Eddie works as expected.
The big day comes–a Fedora release! Sally is very very happy to see Eddie in the release, and brags to all her friends that SHE is the one that made it happen. Thus ends the story of how Sally met Eddie, and shared Eddie with the Fedora community.
Fedora is a powerful project. Not only does it provide an awesome Linux distribution, it also provides a very large collection of packages for all your needs. In addition, the Fedora project is an open project, meaning that every day folks like Sally have the ability and power to join the project and bring new software to the collection. Without the vibrant Fedora community wrangling the packages, Fedora would not be as successful as it is today. In the next part, we’ll follow along as Sally releases updates of Eddie for the latest Fedora release.
About the author
Jesse Keating is a Senior Software Engineer for Red Hat, Inc. and is the lead Fedora Release Engineer. He has been involved in the Fedora community since its inception and has been a member of various Fedora committees.
February 28th, 2008 at 7:12 pm
Mea culpa. It appears that “Eddie” exists as an editor, and doesn’t appear to be open source. I made up the name while writing this article thinking nobody would be cheesy enough to name an editor that. Oops.
February 28th, 2008 at 9:07 pm
Excelent article, I can’t remembered about the comps xml package selection, today I updated them.
February 29th, 2008 at 2:12 am
Thanks for this Jesse, great article. I’ve been toying with the idea of packaging a couple of things for Fedora but I have no experience of building RPMs. This article has inspired me to go learn and find a sponsor who is willing to guide me. Hopefully in the future I can make a meaningful contribution to Fedora!
February 29th, 2008 at 8:24 am
I spend time in Fedora Forums and I feel this is where I can do the most good for Fedora. However, it is nice to know the process and ease of adding packages. Thank You
March 1st, 2008 at 1:19 am
“She upgrades one of her Fedora machines to Rawhide and installs Eddie. Now Eddie is available to anyone who is running Rawhide.”
Does this literally mean that one step in getting a package into Fedora is to install Rawhide locally? (By “Now Eddie is available…” you mean that the act of upgrading to RH causes Eddie to be available to RH users?)
March 1st, 2008 at 8:13 am
Nope, that was probably a poor construction on my part.
It is not necessary to run rawhide at all. It is helpful, but can be done with live images, or with virtualization. Or you can just build it and let others test the package and report problems.
Eddie was available to all rawhide users before Sally upgraded one of her machines. I just wanted to illustrate that once she upgraded to rawhide, she could install her build of Eddie and test it out.
In the next article about Sally and Eddie I’ll go over issuing an update of Eddie to an existing release, which is also a way of introducing new packages to existing releases.
March 1st, 2008 at 8:32 pm
Excellent article! I appreciate all the links you provided in the article. I plan to read them all and hope to be able to help maintain a package in the near future.
March 2nd, 2008 at 1:19 pm
Thanks for help me about how contribute to Fedora. I will read the links and see how i can add some useful Applications.
March 2nd, 2008 at 1:30 pm
“Mea culpa. It appears that “Eddie” exists as an editor, and doesn’t appear to be open source. I made up the name while writing this article thinking nobody would be cheesy enough to name an editor that. Oops.”
Why didn’t you choose Harry? Too cheesy?
March 2nd, 2008 at 6:15 pm
The rather convoluted and complex procedure required to add a software package to Fedora probably explains why so many interesting peices of software are not in it.
How much better it would be for all concerned if software could be given a distro independant auto install procedure that allows the software to be installed from the authors web site (or cd) without having to previously “add” it to the operating system bundle.
I dont care about so called “peer review” whatever the hell that is – if I want to run the software on MY PC I am the only person who needs to review it. I dont care what someone else thinks about the software, I only care what I think.
I wish Linux would break away from this stupid bazaar system of combining the applications with the operating system. It has no advantages whatsoever, but lots of disadvantages, at least from the perspective of an end user. (It has advantages for open source developers – it gives them many man years of fruitful work packaging and re-packaging the same bits of software over and over again).
This idiotic situation if simply the result of the absurd proliferation of distros. The bazaar system allows the creation of zillions of incompatible operating systems, for the purpose of providing software hackers with their Linux wet dreams, avoiding any requirement for standards of interoperability that would benefit end users.
The sooner the whole sorry mess goes away and some semblance of order and common sense returns the better.
March 3rd, 2008 at 9:44 am
stolennomenclature,
I won’t disagree with your goal to simplify things for the end user. I am just not convinced yet that packaging as it is done isn’t beneficial in more ways than not.
A main issue is that packaging is a way of keeping things tidy and of modest size (vs. bloated) in a way where things have been tested to work together.
Linux advances in simplicity, but we just might not be ready to take the next step yet whatever that is.
Notice that packaging is what allows a single vendor to provide some sort of guarantee over all the software on the system. Packaging also has its roots where one small group or person manages the software for many others (enterprise, etc). Things are more controlled.
Anyway, there is nothing stopping people from building one-click install packages that stay in their own little world (they do) and users from installing them to their hearts content (some do). You take that route when you want bleeding edge or if you are using a distro that is less well supported than fedora; however, naturally, it will not integrate as well with what you have as you might like.
Benefits of having this sort of integrated testing modern distro-based packaging provides is that you can upgrade your entire system quickly and easily and not have to deal with having all sorts of things break (when they break it’s a bug and usually gets fixed quickly).
Look at Microsoft, they do this same sort of things but only with their own products (and they likely comingle code much more). They have limited resources which is why you end up with fewer pieces of software when you go Microsoft only.
We do a similar form of integration where various packages coordinate, but this means that prior to the coordination people would step on each other’s toes to some extent.
Another alternative would be to just make packages that are very inflexible, but with so many people working on things, this would create too many incompatibility headaches without recourse (except to hack the code) and provide no smooth migration from the past.
Finally, LSB, freedesktop, and other standards help speed up the process and increase the likelihood that one-click installs will work fine (if perhaps bring unneeded bloat), but again, almost by definition bleeding edge means that it is going beyond what has been standardized.
For the time being, if you have software you want to distribute, you can do one-click sealed deals thing or tarball thing and let the community do the packaging for their favorite distros to provide the eventual tighter integration there.
Finally, feel free to state your detailed argument as to how your system beats the pants off everyone and find a few other enlightened souls to help you bring that system to life. Good luck.
March 3rd, 2008 at 10:02 am
>> Look at Microsoft, they do this same sort of things but only with their own products (and they likely comingle code much more). They have limited resources which is why you end up with fewer pieces of software when you go Microsoft only.
On Microsoft platforms, 3rd parties are second class citizens. They will never integrate as well as MS products can nor get the most efficient APIs to use to hook up with the core system or other MS apps.
Linux is open, allowing everyone to integrate tightly. The choice is that the tighter the integration, the more work it takes maintaining that software as the system evolves. At least you have that tradeoff option (vs Windows) if you are a 3rd party developer (in a sense, everyone on FOSS is a 3rd party developer). And if you go open source, the community will help out.
Microsoft’s public API will by necessity have to limit the amount of system integration. [And this assuming Microsoft was a kind-hearted nonprofit that didn't go further, and that they announced all changes as their internal devs made the changes.. but then they'd just about be FOSS which we know they aren't.] This is a major reason 3rd parties continue to fall to Microsoft on Windows (naturally, some vendors have or have had special deals with Microsoft).
March 3rd, 2008 at 4:23 pm
Great article! I loved the way you broke it down into a simple to understand way.
March 3rd, 2008 at 4:29 pm
Jose-X
If there was only one Linux distro (say Fedora for sake of argument), then any software author could create a Fedora RPM, publish it on their web site, and one could download and install from there, rather than the great big centralised Fedora repository. The only reason this does not happen is becuase there is not one Linux distro, but thousands.
True some authors publish a couple of binary packages for a subset of the major distros, but its a forgone conclusion that the distro im currently using is’nt one of them.
The proliferation of the Linux distros is just plain absurd. They are all just different enough to be incompatible, but similar enough that there is no justification whatever for there existence. Is having a different wallpaper or a new font really a good reason to create a whole new Linux distro and repackage the 10 thousand apps that go with it?
Some people say its about having choice. Mac OS-X, Windows, Linux – thats choice – they are different. Linux-1, Linux-2, Linux-3, …… Linux-10001 is not choice. They are all the same near enough. Its just a great example of what happens to software when the authors make the software to please themselves, rather than for the end users benefit.
The free software / open source communities, hackers and IT enthusiasts might like the current situation, but I doubt if people who really use the system do (how many of them are there i wonder).
Sally is not a typical end user. Typical end users dont compile programs or build packages. They use programs, they don’t write them or modify them.
As to me having a solution for the installation problems, I do. Scrap every version of Linux except one. End of install problems. QED.
March 9th, 2008 at 12:25 am
Dear stolennomenclature
and I guess that “one” would be a distro controlled by your friends at Microsoft … yes the one that lusts for the day when it can establish a taxation office at every turn of the global computing cycle.
No thanks.
March 10th, 2008 at 12:22 pm
stolennomenclature:
You are confusing two separate problems that aren’t as tightly related as you seem to think:
1. The variety of different Linux distributions creates extra work for third-party developers that wish to package their software for easy installation.
2. How to resolve dependency issues with third-party packages.
Taking the second problem first…even if there was only one Linux distribution in the world, the use of central repositories for distributing software can still be a good idea because tools like apt and yum can make sure all necessary dependencies are resolved without requiring the end-user to track down every prerequisite package needed to install some other package.
As for the first problem, I would humbly suggest that if you have THAT big of a problem with the fact that freedom of choice causes people to make different choices that you find to be absurdly picayune, Linux probably isn’t the platform for you. If the lack of point and click installers bothers you that much, you should probably stick with a proprietary computing platform.
March 25th, 2008 at 1:44 pm
We are so proud of you, Jesse!
You have always been super bright and this article will help a lot of people in the industry…. I loved the complaints from the Windows-hugger.
Keep building. You have a great support system.
Signed: Your grandfather.
January 17th, 2009 at 8:53 am
Where find next part ?
April 17th, 2009 at 4:02 pm
I have to agree, to some point, with stolennomenclature regarding the too many distros.
I wouldn’t like a scene with 1 distro on it… but how about 5 main distros and a few other specialty distros, based on one of the 5?
I have no problem with editions. Besides, if using Red Hat based distros, they could just publish kickstart files…
I loved the comments here and thanks for the great article! It’s super cool that your grandfather posted a comment!