Posts Tagged ‘xenserver’

Netscaler 10.x, XenServer 6.2 and Cloudstack

November 1, 2013

Quick tech note. After my Cloudstack testbed upgrade my Netscaler VMs no longer booted. The issue was similar to the one described in CTX135226 but affected both Netscaler 9.3 and 10.1 VMs.

After wasting a few good hours it turns out that this is caused by the presence of a DVD drive in the VM. The mere presence of a DVD drive, even if no ISO is loaded, somehow messes up with the Netscaler boot process under XenServer 6.2. The workaround is trivial:

  1. 1. Spot the DVD drive (hdd) using the vbd-list command

  2. # xe vbd-list vm-name-label=NetScaler\ Virtual\ Appliance
    uuid ( RO) : 5e218b95-68e9-37b2-860f-8b97678e2c02
    vm-uuid ( RO): ab229b8f-2b18-4931-b80b-30ab8265b843
    vm-name-label ( RO): NetScaler Virtual Appliance
    vdi-uuid ( RO): 5bcf6d76-1d2f-411a-9dd7-469aad0f86ae
    empty ( RO): false
    device ( RO): hdd

    uuid ( RO) : 9d6ff62c-93fa-3af5-b4b5-c5c66a0556e4
    vm-uuid ( RO): ab229b8f-2b18-4931-b80b-30ab8265b843
    vm-name-label ( RO): NetScaler Virtual Appliance
    vdi-uuid ( RO): 7b1fa1ff-7388-4e5a-8c67-00e4d943a470
    empty ( RO): false
    device ( RO): hda

  3. Destroy it

  4. # xe vbd-destroy uuid=5e218b95-68e9-37b2-860f-8b97678e2c02

  5. 3. Power on the Netscaler VM again; this time it should work like a charm

This is good enough if you have access to the XenServer running the VM. Not as good of a workaround for a Cloudstack environment which requires you to jump a few extra hoops. Specifically, if you are using XenServer you can update the StartAnswer method to selectively invoke the createVbd function in if the guest OS type evaluate the guest OS type matches “Other install media” and the disk type is ISO skip VBD creation. Here is the relevant sample code snippet:

public StartAnswer execute(StartCommand cmd) {
String guestOsTypeName = getGuestOsType(vmSpec.getOs(), vmSpec.getBootloader() == BootloaderType.CD);
for (DiskTO disk : vmSpec.getDisks()) {
if (type == Volume.Type.ISO && gguestOsTypeName.toLowerCase().contains("other install media")) {
s_logger.debug("mperedim: skipping VBD creation");
} else {
createVbd(conn, disk, vmName, vm, vmSpec.getBootloader());

One can improve this workaround by mapping Netscaler VPXs to a dedicated OS template, so that DVD drives are still created for other VMs that get mapped to the “Other install media” XS template.

Xenserver: fake xen tools for Solaris 10 guests

March 31, 2013

Note: Also, you hopefully appreciate that this is completely unsupported.

Xenserver doesn’t enable Shutdown / Reboot buttons for VMs that don’t have the XenServer tools installed. This is an issue for my Solaris 10 guests since tools are not available for this platform. Which has been bugging me for some time.

So I went ahead and dug into the XenServer tools for linux. Turns out that the only thing they’re doing is updating a bunch of parameters on XenStore. However, Solaris 10 doesn’t have a device path for XenStore, putting us back into square one. Or not?

Not really. Turns out that Xen Tools installation is like orgasm memory. Sure, it’s a lot better if one has them, but if not one can modify the appropriate XenStore parameters from dom0 and fake it. XenServer couldn’t care less how the parameters were modified, as long as they are tweaked in the proper order the Suspend/Reboot/Shutdown buttons are enabled. So just get the dom-id of your Solaris VM:

[root@dom0 log]# xe vm-list name-label=i-2-323-VM params=dom-id
dom-id ( RO) : 154


[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/Installed 1
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MajorVersion 6
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MinorVersion 1
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MicroVersion 0
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/BuildVersion 59235
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/class "SunOS"
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/major "5"
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/minor "10"
[root@dom0 log]# xenstore-write /local/domain/154/data/updated 1

The above is enough to enable the shutdown/reboot/suspend buttons. Unfortunately in the process it also sets the “platform: viridian: true” parameter which doesn’t play nicely with Solaris VMs.

[root@dom0 log]# xe vm-list name-label=i-2-323-VM params=uuid
uuid ( RO)    : 5dc51848-bc9c-dd70-b670-2c7d263a7fe5
[root@dom0 log]# xe vm-param-remove param-name=platform param-key=viridian uuid=[...]

… and see the “Force shutdown”, “Force reboot” buttons disappearing.

So what works?

  1. Reboot: this does a clean reboot of the Solaris 10 domU
  2. Live migrate: not extensively tested, but a test VM does keep network connectivity after a single live migration.

Unfortunately shutdown only kind of works. Hitting the button does initiate a clean shutdown of the Solaris domU but the guest never seems to do an ACPI poweroff and gets stuck at “Press any key to reboot”. This is proving a slightly more touch nut to crack.

Update 2012/04/01: I’ve wasted a few too many hours on “shutdown” not working. Maybe I’ll revisit this in the future but calling it quits for now.

Cloudstack: OS type & xenserver templates

March 6, 2013

I’ve been using cloudstack for circa a month now for virtualising Solaris workloads. It has been mostly working like a charm, once I applied the appropriate workarounds (cf. my relevant findings, courtesy of my IoannisB citrix identity). However one thing has been bugging me for some time:

# xe vm-list name-label=i-2-271-VM params=name-description
name-description ( RW) : Template which allows VM installation from install media

My Solaris VMs are launched using the generic Xenserver template. This is not really to my liking for two reasons. Firstly, I have to apply the viridian:false modification to the default template. Secondly, there is no reason to appreciate whether a VM is a Solaris one or not using the Xenserver CLI.

The fix is to have Cloudstack using the “Solaris 10 (experimental)” template for my Solaris workloads.

  1. Download the cloudstack source code and uncompress to a folder of your choice.
  2. Apply a rather simple diff to the file:
    $ diff
    > _xenServer600GuestOsMap.put("Sun Solaris 10(64-bit)", "Solaris 10 (experimental)");
    > _xenServer602GuestOsMap.put("Sun Solaris 10(64-bit)", "Solaris 10 (experimental)");
  3. Build the JAR files, per the instructions in the Cloudstack installation guide page 16. No need to build DEB or RPM packages
  4. Replace /usr/share/java/cloud-plugin-hypervisor-xen.jar with cloud-plugin-hypervisor-xen-4.0.0-incubating-SNAPSHOT.jar that was built in the step above
  5. Restart the management server.

Slashdot geeks may want to add a Step-6: Profit. Launch again a Solaris 10 64-bit template and enjoy:

# xe vm-list name-label=i-2-272-VM params=name-description
name-description ( RW) : Clones of this template will automatically provision their storage when first booted and then reconfigure themselves with the optimal settings for Solaris 10 (experimental).

Solaris + xenserver + ovswitch

February 28, 2013

This has troubling me for quite some time, hopefully someone else can save a few hours by bumping in this post.

For some reason my Solaris 10 Virtual Machines on Xenserver failed when the Distributed Virtual Switch Controller was also running. I didn’t really troubleshoot the issue until recently since I could live without cross-server private networks. This no longer being the case I decided to look into it again.

Scroll forward a couple of hours and after losing quite some time on trying various tricks on the VM (disabling NIC checksum offload, lower MTUs etc) to no avail I concluded that it must be a hypervisor issue. Digging into the openvswitch tools revealed something interesting.

[root@xenserver ~]# ovs-vsctl list-ports xapi25

Specifically, for my Linux VMs only a vifX.Y interface was being added to the bridge, while for my Solaris ones both a tapX.Y and a vifX.Y. Clickety-click.

[root@xenserver ~]# ovs-vsctl del-port xapi25 tap47.0

Et voila! Network connectivity to the Solaris VM works like a charm. Now to make this change permanent:

[root@xenserver ~]# diff /etc/xensource/scripts/vif.orig /etc/xensource/scripts/vif
if [[ $dev != tap* ]]; then
> $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> else
> echo Skipping command $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> fi

I am not really certain of the ugly side-effects that this may have. But it does the trick for me.

Update 2013/03/10: A better workaround is to have the above behavior apply only to Solaris VMs. For example, assuming that these are based on the “Solaris 10 (experimental)” template, the following snippet skips the offending command only for the Solaris VMs:

if [[ $dev != tap* ]]; then
    $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
    xe vm-list dom-id=$DOMID params=name-description | grep 'Solaris 10' 2>&1 >/dev/null || \
        $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details