Rate this page del.icio.us  Digg slashdot StumbleUpon

Automated failover and recovery of virtualized guests in Advanced Platform

by Rob Kenna

In our first article in this series, we introduced the Conga management interface. In this article we show how Red Hat Enterprise Linux 5.1′s Advanced Platform can be configured to automatically provide High Availability for Virtual Machine Guests. Using the Conga management interface, the reader is walked through the process of setting up a shared filesystem guest repository, enabling the failover settings, and demonstrating zero downtime failback.

Note: This article is based on the upcoming 5.1 version of Red Hat Enterprise Linux Advanced Platform.

The problem: guarding against failure of better utilized resources

You’ve spent the last several years trying to reduce your IT costs. You’ve consolidated labs and saved money on floor space, heating, cooling and electricity. You’ve automated system administration processes and have reduced costs on sysadmin staff. You’ve reduced hardware costs by replacing large numbers of small servers with a smaller number of blade servers. And finally, you’ve begun to replace physical servers with virtual machines.

Virtualization has become the promised land for you and your CIO. You’re running more and more software, including mission critical applications on virtual machines. Everything is going great. Your costs keep on dropping. And then, someone trips over a power cord in a lab and brings down one physical server and all the virtual machines that it hosted. Now you’ve got a problem, and it’s not a virtual one. Some of those virtual machines supported customer-facing applications, while others ran the company’s inventory control system. What you need is a failover capability for virtual hosts. You’ve long been aware of the failover capability of clustering for physical servers, but when you started to add virtual machines, no one thought about failover. It’s never been possible to even do that. Right? Well, with the RHEL Advanced Platform, now you can.

The solution – cluster failover

Let’s examine what it looks like to provide the failover protection to a virtual machine environment using Red Hat Enterprise Linux 5.0 Advanced Platform. In the diagram below, we have two Enterprise Linux guests on each of three machines. On the left, we see that a server physically fails. Ordinarily, guest instances A & B would stay down until an administrator is able to take action. But since we have Advanced Platform configured to manage these instances, the physical machine failure is automatically detected, and guests A & B are automatically restarted on the two other servers.

auto failover

Once the administrator is able to repair the problem with the machine on the left, we can then ask for those two guests to be moved, live, back to the now functioning system. This is an example of zero downtime failback. So not only did the virtual guests quickly and automatically get restarted, but there was no disruption to the application when re-balancing the configuration.

Details, please…

Let’s look at how this magic is constructed and then see the system in action. The key, fundamental change in the machine setup is to treat the three physical machines as shared resources. In this discussion, we’ll be using the new web-based management tool, Conga, to set up the cluster, shared storage, and failover configuration. We’ll also be using iSCSI SAN devices, which are a great alternative to Fibre Channel since they use standard Ethernet adapters, switches, and connections. Here’s the big picture:

virtual hosts

This solution makes use of GFS2, Advanced Platform’s cluster filesystem, to provide block level, high performance operation across a set of machines. The /guest_roots GFS2 filesystem is where the configuration and guest boot images are stored. This enables us to start a guest on any of the machines as well as perform live migration. Et-virt05, et-virt06, and et-virt07 are the physical machines hosting three virtual instances: guest1, guest2, and guest3.

Here are the five steps we’ll follow in creating and validating our example:

1.Form a cluster of the three physical machines.
2.Create the shared area for maintaining the virtual guests.
3.Move the virtual guests to a shared area, visible to all the nodes.
4.Have the cluster control the guests.
5.Try an example failover and recovery.

Step 1: Form a cluster

create cluster

Using Conga’s web-based management interface to form the cluster is easy. Make sure that your machines are on the same subnet, and then log in to Conga and navigate to the “Create a new cluster” page. Enter the machine names and root password, make sure “Enable shared storage” is selected, and click Submit. Conga will deal with installing packages and establishing the cluster. And don’t worry, the root password is encrypted over the network.

The submit step will restart the machines and within a few minutes, the cluster will be established. Next we will create the shared filesystem which holds the guest configurations and boot image files.

Step 2: Create the shared area for maintaining virtual guests

Conga makes it easy to create a shared filesystem across the machine set using the Cluster Logical Volume Manager (CLVM). CLVM provides volume management across a set of machines and enables concatenation, striping, mirroring, and expansion of storage underneath a filesystem. Start from the Storage tab and then select one of the machines; in our case et-virt05. Note that when initially viewing the storage page for a server, Conga will probe for a current, accurate view of disk and volume configurations. This will take a few seconds.

shared drives

A volume group is the logical storage area from which we draw space for the creation of our filesystem. Use of a volume group instead of a raw partition allows us to later extend the volume and the filesystem in case we need more space.

Once we’ve created the volume group, CLVM will provide a common name across the machine set which persists between reboots. For this case we have created a 100 GB volume group and will use a portion of that for the shared area. Specifically, we’ll create a volume group using two LUNs (storage partitions) from the array: /dev/sdd2 and /dev/sdg2. We’ll use the default extent size of 4 MB and name the volume group guest_vg. And since this is shared storage, we set the clustered state to “true.”

Note: It’s always a best practice to keep your servers time synchronized to a common source, like an ntp server. This is even more important for clusters and machines migrating guests.

create new volume

Having clicked “Create”, we then see all of the details of the new volume group.

volume group

Now click “New Logical Volume” to build the filesystem specification. There is a one-to-one correlation of logical volumes to filesystems. In one step we’ll create a new logical volume with a filesystem on top. Initially, the plan is to create three guests. Select GFS2 as the file system choice. Considering that each guest needs 6GB for the root volume and that we will need more guests later, we’ll carve out 60GB from the volume group. The filesystem is called guest_roots and we can use the same name for the logical volume name, filesystem name, and mount point. Let Conga create the fstab entry and also set the mount to occur at boot.

GFS2 uses a journal for filesystem crash recovery. Conga already knows that there are three machines involved, so it defaults to three journals, one for each server machine. Easy journal addition is one of the new features of GFS2, so it’s easy to add more journals later when expanding the cluster.

Now click “Create”.

unused space

You can see that we’ve got our filesystem. Note that we can later remove the filesystem from this page. Next, go and create a mount point with the same name (/guest_roots) and add an /etc/fstab entry on each machine.

logical volume

Step 3: Move the virtual guests to the shared area

At this point the guests can now be created and placed in the shared area. This article is not focused on the creation of virtual machine guests, but let’s review the settings needed for our configurations. Virt-manager is the easiest way to create new guests. In particular, specify that the disk image resides on the shared area, /guest_roots in our example. Also, you need to note that your system is in a networked environment, which will create the network bridge to your guest. Take a look at the configuration summary screen for virt-manager.

ready install

Once a guest has been constructed, its configuration file will be created in /etc/xen. In this case, it is /etc/xen/guest1. We need this file to also reside in the shared area. So, copy it over to /guests_root, right next to the disk image. Note that if you use virt-manager to modify the configuration, the /etc/xen configuration file needs to be re-copied to the shared area. Now /guests_roots will look like the listing shown.

guest root

Step 4: Have the cluster control the guests

We are now ready to place the control of the guests under cluster management. This includes starting a guest OS, restarting it if it crashes, and failover to another node in the event of machine failure. There are a few parts to this setup; disabling Xen, starting the guest, turning on live migration, and enabling the control of guests by the cluster management.

First disable the Xen script which starts the guest at boot time for the physical machine. On each of the servers issue the commands:

[root@et-virt06 ~]# chkconfig xendomains off
[root@et-virt06 ~]# service xendomains stop

This will immediately stop the service and prevent it from starting at the next boot time. Note that this will also stop currently running guests on the machine.

Next, enable live migration of a guest from one machine to another. This capability is off by default. On each of the machines, edit /etc/xen/xend-config.sxp. Uncomment the two relocation lines and set xend-relocation-server to “yes.”

(xend-relocation-server yes)
(xend-relocation-port 8002)

Now place the management of the virtual guest under the cluster. The cluster manager deals with services– whether they are a web service, database, or, as in this case, a virtual machine service. Since we are now in a cluster, we will specify how and where the guest OS is to be started. The first task is to configure a failover domain.

The failover domain simply states which machines can run the guest as well as which we prefer to have the guest run on. We will pick one machine out of the three to bias the running of the guest. We also will allow the guest to run on any other machine. You can get fancier with this configuration, but let’s keep it simple here. Look at the following failover domain.

We construct three in total; one for each physical machine. When later adding more guests, associate them to one of these three failover domains.

We don’t need fine-grained prioritization; only one machine is preferred and all others are treated equally. So we don’t check “Prioritized” or set the Priority fields.. Again, since we allow the guest to run on any machine, we don’t check “Restrict failover to the domain’s members.” This specification simply sets a bias to one machine. So we end up with three failover domain configurations: “bias-et-virt05”, “bias-et-virt06”, and “bias-et-virt07”.

add failover

failover list

Next construct the virtual service entries. Enter the name of the guest (guest1), the shared area (/guest_roots), that we want the service to automatically start, the failover domain (bias-etvirt05), and the recovery policy (restart the service if it fails).

properties for guest1

That’s it. Once you click “Update Virtual Machine Service,” the guest is under cluster management. If it’s not currently running, it will immediately be started. Notice that names are green, indicating that they are running and that the status lists which physical machine they are running on. They are shown running on the “bias” machine. Guest1 is on et-virt05 etc. Here’s what the service list now looks like:

services list

Step 5: Try an example failover and recovery

Now it’s time to validate our setup and see the system respond to various failure scenarios. First, let’s simply simulate a guest crash. This is easily performed by issuing the command “xm destroy” on one of the machines. Log in to one of the physical machines and try it. Here we used xm destroy to kill the guest.

[root@et-virt06 ~]# xm list
Name......................................ID.Mem(MiB).VCPUs.State...Time(s)
Domain-0...................................0.....1505.....8.r-----...6972.4
guest2.....................................5......499.....1.-b----.....21.0
[root@et-virt06 ~]# xm destroy guest2
[root@et-virt06 ~]# xm list
Name......................................ID.Mem(MiB).VCPUs.State...Time(s)
Domain-0...................................0.....1505.....8.r-----...6975.7
[root@et-virt06 ~]# xm list
Name......................................ID.Mem(MiB).VCPUs.State...Time(s)
Domain-0...................................0.....1505.....8.r-----...6988.7
guest2.....................................6......499.....1.-b----.....17.7

Note that about 10 seconds2 after destroying the guest, it restarts. The cluster manager has detected the failure and restarted the guest for you.

Let’s now simulate a physical machine failure. For this we’ll simply reboot one of the machines in the cluster. First note the report that Conga provides about the state and location of the guests. Also note that guest1 is running on et-virt05.

Now by simply logging into et-virt05 and issuing a reboot, we can watch the recovery kick in. With the absence of et-virt05, guest1 is restarted to run on et-virt07.

recovery

After we wait for et-virt05 to reboot, we can now use the same screen and choose the migrate task for guest1 to move it back to et-virt05. Remember, this failback happens while the guest is active with no further disruption in service.

Wrap Up

This article has shown the power of combining clustering and virtualization. We’ve demonstrated the construction of a robust, high-availability system, while taking advantage of the utilization efficiencies of processor virtualization. This was made very easy through the use of GFS2, the shared filesystem, for storing the boot images and configuration files for the guest OS’s. Further, the Conga web-based interface provided an easy tool for the configuration and management of the Advanced Platform.

What’s Next

In an upcoming article, we’ll cover:

  • Creating a virtualized cluster.
  • Managing shared partitions using CLVM across the Host OS’s.
  • Sharing GFS2 filesystems across a cluster of guests.

Acknowledgments

I’d like to tip my hat to the hard working teams at Red Hat and the open source community. The synergy of Xen virtualization, Linux, failover clustering and GFS make for a truly powerful operational environment. Thanks also to the reviewers of this article. In particular, Len DiMaggio was effectively a co-author as he sharpened the text and provided the “readers’ eye.” Be sure to check out Len’s earlier Red Hat Magazine article on using Conga!

About the Author

Rob Kenna is Senior Product Manager for Red Hat’s Storage and Clustering Products, including GFS (cluster file system), and RHCS (application failover). He brings a rich background as developer and manager for the creation of storage software.

References:

55 responses to “Automated failover and recovery of virtualized guests in Advanced Platform”

  1. Georgi says:

    I would like to see more of this kind of articles in RH Magazine.

  2. Rob Kenna says:

    Glad to hear it. We’ll be making additional articles around the use of Advanced Platform.

  3. Alexey Vasyukov says:

    Really great article! It shows the power of integrated technology in action rather then in theory.

    And the agenda of upcoming article looks even more promising. :-)

  4. Sagar says:

    Great article, looking forward to the upcoming articles, especially,
    Sharing GFS2 filesystems across a cluster of guests.

  5. Ahmed Kamal says:

    Excellent, easily one of the best articles I have ever read. Please, keep this quality and technical depth in redhat-mag. I am so excited about the next ones.
    I really think more articles should be published about the new and advanced features in RHEL5 that most people don’t know about (Cluster, GFS, SElinux policies, kerberos in AD environment, … )

  6. Roberto Polli says:

    Brilliant Rob!

    Just one thing. Is there any way to balance the load or to verify the available memory of the host machines to choose where the guest of the failing host should be located? I don’t think there is anything like the DRS, but any other workaround for that?

    I just can’t wait to read the next articles! You guys at RH and in the open source world are great!

    Thanks a lot!

  7. Davide P. says:

    I have implemented this on my new cluster (RedHat 5 Advanced Platform), all is ok with linux paravirtualized guest.

    While with Windows fullvirtualized guest in case of failure in dom0 the virtual service doesn’t restart on another host. Is this an expected beahviour ?

    Thanks!

  8. Rob Kenna says:

    re: comment 7: “service doesn’t restart…”

    This should restart. This may be an issue fixed in 5.1…

  9. Rodrique Heron says:

    I would like to see more discussion on the use fencing device when setting up a three node cluster. You mentioned you are using ISCSI, and it seems node “et-virt05″ is your ISCSI initiator. What happens when it fails ? How do you guard against that ?

  10. Rob Kenna says:

    re: Comment 9:

    The shared storage is on an EqualLogic iSCSI array, so each of the cluster nodes mount the LUN’s from the Host Domain. The failure of any single node will node cause issues for the array.

  11. Davide P. says:

    re: Comment 8

    My problems with windows server domU persists, I can’t relocate neither start the domU from luci interface.
    “xm create winguest1″ is successful but in this way the service is not part of the cluster.
    All is fine with para virt. linux domU.
    All nodes are updated to last rhn release.

    Thanks.

  12. Gary M. says:

    This article was a great help in setting up my 3 node cluster using 5.0 and Conga made the whole process a lot easier then my previous attempts.

    I did run into a few minor issues that I thought I would share in case anyone else following this article runs into them and what I did to get around them. Chances are some, if not all, of these issues will be fixed when 5.1 is released.

    * Conga does not seem to let you create a Physical Volume on a new drive / partition. I had to create the PV from the command line first and then was able to use Conga to create the Volume Group and Logical Volume.

    * I found that simply copying my VM container to the GFS2 shared volume caused GFS2 to get corrupted. I ran gfs2_fsck before the copy (no nodes had the fs mounted) and the fs was fine. I then mounted the fs and copied my VM image (16gigs) to the GFS2 volume. Then I unmounted the fs on all nodes again and ran gfs2_fsck and it found all sorts of errors. I decided to reformat using GFS1 and everything is working fine now. I think I found a bugzilla report that explains the GFS2 issue so it looks like this will be fixed in 5.1

    * Conga does not seem to let you delete Failover Domain entires. Adding and modifying works fine. I have not had the need to do this but would guess that manual editing of the cluster.conf file would be needed on all nodes as of right now.

    * Live migration did not work initially after setting the noted xend-relocation-* options from the article. I got it work by also setting the xend-relocation-address and xend-relocation-hosts-allow options.

    * Before fixing the live migration problem, every time I tried to migrate a VM from one node to another using Conga, the migration would of course fail and Conga would report there was and error before returning to the Services screen. Conga would report the VM that I tried to move was down even though xm list on the node showed the VM still running. Any attempt to restart the VM via Conga would continue to report errors. To resolve this I had to issue a clusvcadm -d VM from the command line which shutdown the VM. This seemed to get everything back in sync and I could restart the VM via Conga without errors.

    * Once I got everything working I started to test failover and recovery of a VM. I did the reboot test as described in the article and the VM restarted on one of the other nodes as expected. Once the rebooted node rejoined the cluster the cluster would automatically shutdown the VM and restart it on the rebooted node. I personally would like to manually do the failback using live migration or have the cluster do a auto failback using live migration instead of a shutdown and restart of the VM. I have not found a fix for this yet or maybe 5.1 will fix this.

    Thanks again for the great article.

  13. Justin Cirelli says:

    Great article. I cannot help but think that this is sooo not ready for primetime though. I run large Redhat shops, but most of it runs on ESX. In comparison the process you have to go through to get a guest running in a clustered fashion is insane. I am hoping redhat streamlines this process as VMware has, manually copying config files is not something I want to be doing when I have 300 vm’s.

  14. Agnieszka Kukalowicz says:

    Very useful article. But do I need GFS2 or I can use GFS1 for that?

  15. Rob Kenna says:

    In regards to comment #14, GFS works fine with this example as well. Given that GFS2 is not yet released for production support that would be the way to go.

    - Rob

  16. Pasi Kärkkäinen says:

    Why not use plain CLVM volumes for the domU/guest storage..? should be faster when there’s no filesystem (GFS) on top of the CLVM..

  17. Rob Kenna says:

    re: Comment 16…

    Yes, using CLVM volumes will certainly work as an option. You likely won’t see much of a performance advantage because the root files use direct IO. The main advantage using a file system is that you needn’t preallocate the space and it’s generally easier to manage a filesystem for backups etc… You can make a general shared file system space and split off from there.

  18. Agnieszka Kukalowicz says:

    What will happen when the application (for example MySQL, HTTP) running in the virtual machine crashes but the virtual machine is still working. This kind of virtualization/clustering where cluster manager is over virtuals doesn’t cover the above case. Am I wrong?

  19. Rob Kenna says:

    re: Comment 18. To cover this case, we need to construct a cluster out of the virtual machines and explicitly manage the services as are done in non-virt environments. This is actually a subject for my follow-up article. (thanks for the “kick”)

  20. Agnieszka Kukalowicz says:

    No problem:-) And I can’t wait it.

  21. Joe Lewinski says:

    Hello,

    With regard to comment #18, we have either Informix or Sybase databases in cooked filesystem space, and want to
    implement replication using the Operating System as our
    agent. Will Clustering support updating two copies of the
    database for us without the database engine knowing it?
    We would prefer to have the two copies of the database on
    two different disk arrays in different parts of the building, however on a common LAN.

    Thanks!

  22. Mark Nielsen says:

    This is a great article, one I’m constantly referring people to! We’re doing this at a gov’t client with minor changes. In response to the “not ready for primetime” comment, I can say with certainty it is ready. Thanks guys!

  23. Thomas von Steiger says:

    Great Idea with articles about the “Advanced Platform”!

    Is there a trick to have virtual servers with failover support from cluster suite and virtual servers without failover support from cluster suite on the same dom0 becouse of “chkconfig xendomains off” ?

    regards,
    Thomas

  24. Rodrique Heron says:

    I am reading your article again and I notice you are using disk images, is it possible to use LV for the guest root, I guest is the same a question #16. And why are you calling this a guest root, do you store the actual guest data some place else ? I mean, if this is a Web server with a /www/data, do you attach a LUN to the guest server just for that purpose ? If so, what happens to that LUN during a live migration ?

  25. Rob Kenna says:

    re: comment #24

    Yes, the guest root can either be files or LV’s (logical volumes). The main requirement is that they be equally visible to all nodes so that the failover and migration can occur. The advantage of a file solution is that you needn’t carve up you disks and instead simply provide enough space in a shared area. Also, the shared filesystem area can easily be extended. By the way, the roots are then used via direct IO which is quite efficient.

  26. Rodrique Heron says:

    re: comment #25

    The thing I like about LV’s are, you can snapshot them, this is great for backup or duplicating a VM. Is there a efficient way of doing this with disk images. In my non-clustered setup, cp command takes a while to copy a VM.

    I am looking at the Dell MD3000i to build my clustered setup, anyone using this Array ?

  27. Paolo Marini says:

    The article is great and very useful.

    I implemented a cluster using 3 physical machines hosting xen domain guest, using iSCSI storage, as described in the article, and the xen guest themself are configured as a second cluster mounting a GFS filesystem from the same iSCSI array. Everything works great, I am able to migrate the VMs from one physical machine to the other, but there is one important issue: if I reboot a VM guest part of the second cluster, e.g. from the luci interface, the second cluster looses the quorum and it is necessary to reboot each VM guest node to restart the cluster. It seems like the network connectivity is lost for enough time for the cluster to loose the heartbeat.

    I am currently using two gigabit ethernet with channel bonding and LACP on both the iSCSI device, the ethernet switch and the physical cluster.

    What can I do to correct this problem ?

  28. Rob Kenna says:

    re: comment #27

    Hmmm… Not sure I know what’s happening here. If your second cluster has 3+ VM’s in it, the loss of a single VM should still maintain quorum (2 out 3 still active).

  29. Rob Kenna says:

    re: comment #26

    I don’t know of a way to snap the file image. Also, watchout about snapping a clustered volume, that doesn’t work.

  30. Christian Nilsson says:

    Hi!
    Great article!
    I have a question:
    This is the scenario:
    A cluster with 2 nodes and 5 virtual services.
    2 of the virtual services are production systems and 3 are lab services, in the event of a failed cluster member i want to prioritize the production virtual servers meaning i want the failover service check to see if there are enough resources to start another service on the remaining server, if not then kill off the lab servers to make the production system allocate those resources. In this way a can effitiently utilize the resources in the cluster.
    Is this possible?

  31. Dan Levine says:

    I appreciate this article very much as well as the Conga one. I’ve been trying to figure out the best way to deliver fast disk access (~100MB/s over 1 GigE) to many of my user workstations. The solution I was toying with originally was to deploy GFS or GFS2 on all the workstations (~20) over iSCSI. The only problem I see with that option is the fencing policy requirement on the workstations. Users can reboot their workstations, but probably don’t want them automatically rebooted for them if the system thinks there’s a problem. :-(

    So in lieu of that, I figured perhaps I’d create a 3-tiered NFS solution using Virtual Server (for HA) and a Cluster to provide scalable NFS access to the shared GFS or GFS2 storage accessible to the cluster nodes to the workstations.

    Do I have the right idea here? If so, which one is the right idea? Just GFS or GFS2 seems so much simpler than the alternative, except for the fencing requirement.

    I’d love some actionable advice.

    -Dan

  32. Paolo Marini says:

    re: comment #27

    I solved the issue a few weeks ago but is important to notify to other having the same problem.

    The solution was to disable the IGMP snooping on the network switch (which is a 3COM baseline switch 2924-SFP Plus) and now everything is _VERY_ stable !

  33. Fortune 500 Manager says:

    Excellent article. I have one big problem with Xen on RedHat. No support for multiple VLANs. RedHat needs to do this to compete in large scale depolyments. We got it working by hacking at the system a bit, but we were told by support that they would not help us with any issues.

  34. Paolo Marini says:

    re: comment#33

    Can you share with us some ideas of your configuration ? I have a setup with two ethernet interfaces, bonded going to a switch supporting VLANs. What I would like to do is to isolate the cluster and iSCSI traffic on one VLAN and have all the user traffic on another, just for security reasons.

  35. John says:

    Hi Excellent article, keep up the good work red hat.

    I have two Compaq ML370 G2′s (both have 2 x 1.4Ghz CPU, 4 GB RAM and are connected to a Compaq Smart Array 500 with 14 x 18GB SCSI disks in a RAID 5 configuration). I have Xen (RHEL 5) running successfully on both machines and a shared storage area where I have four guests residing (2 guests on each physical machine) on a GFS2 shared disk. I now want to create a shared storage area that all four guests can read and write to. I have created the shared area with GFS2. However, when I assign the new disk to a guest using virt-manager the guest works fine but the moment I want to add the same shared area to a second or subsequent guest virt-manager tells me that the storage cannot be assigned because it is in use by another guest. I have trawled the internet for a solution but have not come up with anything. Is what I am trying to do possible and if so, how can I add the same shared FS to all four guests using GFS2?

  36. Tomash says:

    Hi, thanks for a great article. Why have you created one failover domain for each cluster node? What is the advantage from creating only one failvoer domain with three nodes, restrict to members?

    Thanks, TOmas

  37. Boris says:

    Hi, great article. I have one question. Is it possible to use SAS as a block level connection between servers and shared storage? By means of SAS expanders, all drives are available to all hosts, and GFS should take care of access controll. This should reduce the hardware costs, and SAS should be faster than GbE (and there is no TCP/IP stack overhead).

  38. graem says:

    Any news on when we going to see the article “Creating a virtualized cluster.”?

  39. Jack Jin says:

    Hi, I did almost exact test, but I am using RHEL5 update2, everything is fine, except when one cluster node reboot, the VM on it can failover to other node, but can’t automatically recover back after node bootup.
    In RHCS conga setting for failover domain, I did not check “Do not fail back services in this domain “, so it should fail back, but it can’t. I did lots of testing but can’t make it work.
    Do you have any idea?

    Thanks

  40. Francesco says:

    re: comment#39

    Hi,
    we have the same configuration with RHEL5 update2, and we have the same problem.
    Is there any solution available yet for this issue?

  41. Paras Pradhan says:

    hi.

    Great article Rob.

    Got a problem.

    1) created 3 nodes (all three under vmware fusion)

    2) Frist node runs luci and other nodes (node1 and node2) runs ricci and are clustred successfully

    3) node1 has got a xen VM

    4) When node1 is rebooted, node2 takes care of my VM. No problem at all

    5) Now when node1 comes back and try to migrate my VM back to node1 i get error as:

    Unable to retrieve batch 795347091 status from node1.writestbed.org:11111:
    vm01 is in the process of being migrated
    Starting cluster service “vm01″ on node “node2.writestbed.org” –
    You will be redirected in 5 seconds.

    What might be problem in migration.

    And BTW what is the difference in relocation and migrtion.

    Really need a help

    Thanks in advance

    Paras.

  42. Lon Hohberger says:

    The ‘unable to retrieve batch …’ sounds like a Conga issue; does the machine actually migrate?

    Relocation = Turns off the VM on node1, then turns it on on node2

    Migration = move the VM in question from one node to another node without stopping it. This can provide greater availability.

  43. Paras Pradhan says:

    No it doesn’t migrate.

    My xend-config has these entries.

    (xend-relocation-server yes)
    (xend-relocation-port 8002)
    (xend-relocation-address ”)
    (xend-relocation-hosts-allow ”)

    After doing migrate from node1 to node2 and if i do ‘xm list vm01′ on node2, will it show in node2 that the vm01 is running in node2?

    Thanks
    paras.

  44. Grigory says:

    Hi can anybody explain please how to configure MySQL resource.
    I’m getting this errors:
    RG service:mysql failed to stop; intervention required
    Service service:mysql failed to stop cleanly
    Service service:mysql is failed
    Stopping service service:mysql
    start on mysql “mysql” returned 1 (generic error)
    stop on mysql “mysql” returned 1 (generic error)
    Failed to start service:mysql; return value: 1
    Checking Existence Of File /var/run/cluster/mysql/mysql:mysql.pid [mysql:mysql] > Failed – File Doesn’t Exist
    Starting Service mysql:mysql > Failed – Timeout Error
    Stopping Service mysql:mysql > Failed

    My cluster.conf

    My my.cnf
    [mysqld]
    datadir=/data/mysql
    socket=/var/lib/mysql/mysql.sock
    user=mysql
    # Default to using old password format for compatibility with mysql 3.x
    # clients (those using the mysqlclient10 compatibility package).
    old_passwords=1
    enable-locking
    query_cache_wlock_invalidate
    query_cache_size= 0
    query_cache_type= 0
    delay_key_write = OFF
    flush

    [mysqld_safe]
    log-error=/var/log/mysqld.log
    pid-file=/var/run/mysqld/mysqld.pid

    It will be great if you can help me.

    Thanks allot
    Grigory

  45. Grigory says:

    My cluster.conf

  46. Grigory says:

    I’m not able to paste my cluster.conf

    name=”mysql”
    mysql config_file=”/etc/my.cnf” listen_address=”192.168.9.36″
    shutdown_wait=”0″/>

  47. Chris Edwards says:

    re: comment#39

    I am having the same problem, when I do a clustat its still showing that the vm service for that was running on the lost node is still up. No migration of the service.

  48. Paras Pradhan says:

    Got a question. Relocation and migration is working now. But, When I pull the power cable of one node, the VMs are not migrating/relocating to another node. I think it should do it automatically.. Am i correct?

    Paras.

  49. Paras Pradhan says:

    In the above setup, when testing for the physical failure of host et-virt05, the guest1 is restarted in et-virt07. This means the guest1 is relocated in et-virt07. Instead of relocation, is it possible to use migration ?

    Thanks
    Paras.

  50. Piergiorgio Andreani says:

    Very useful article!
    But I still have got an issue: I can’t create a new LV on a clustered Volume Group; it returns an “Error locking on node”. So I think one thing is not clear in this article: how do I have to prepare secondary nodes filesystem in order to “receive” propagating GFS filesystem? Here’s my tries:

    1 – Created a clustered VG on node1 and a clustered VG on node2 then tried to create a LV on node1: didn’t work

    2 – Created a clustered VG on node1 and no VGs on node2 then tried to create a LV on node1: didn’t work

    3 – Created a clustered VG on node1 and a no-clustered VG on node2 then tried to create a LV on node1: didn’t work

    4 – Created a no-clustered VG on node1 and a no-clustered VG on node2 then tried to create a LV on node1: worked but GFS was not propagated to node2

    So… where is the right Partition, VG and LV situation I must prepare on node1 and node2 ??? Thanks

  51. sequence says:

    Hi Piergiorgio!

    You forget to make partprobe on another nodes…
    (and vgscan-lvscan probably)

  52. Karaescomma says:

    Hi

    As newly registered user i just want to say hi to everyone else who uses this site :-D

  53. NextantastE says:

    For his part, Bush said he had come to admire Talabani
    President Bush during a news conference in Baghdad, Iraq, where the president was making a farewell visit Sunday.
    said Massachusetts Emergency Management Agency spokesman Peter Judge.

  54. Motoovavelp says:

    What is bumburbia?

  55. Thomas says:

    Hi,

    Maybe it’s time for a update of this article because there are many new feature in rhel5.3 and conga?

    Thomas