I am not really certain if your Greek subsidiary is a partner that just carries your trademark or a full subsidiary … but it just outright sucks. And you may want to look into it.

I could waste a number of keystrokes on the matter, but a picture is worth a thousand words:

2 working days (20 & 21 December) for a package to travel from the US all the way to Greece. Hopefully 4 … FOUR! working days (22, 23, 27, 28 December) for the package to travel another ~440km from Athens to Katerini (1 hour away from Thessaloniki, the 2nd largest city of Greece).

After struggling for something like 15′ trying in vain to figure out what idiotic mistake I had made I pulled out my laptop. After another 30′ or so, being unable to contact my HTPC through my laptop too, I found out that not even ARP is working. Afraid of a rootkit I started installing Wireshark on the HTPC. And after 5′ I was finding out in surprise that ARP broadcast requests were not even reaching the HTPC (?!?!).

Some googling later revealed that other people are facing the same problem: ARP simply fails with this DSL modem. And there is little info on whether this is a persistent problem. I can only tell that the problem was temporarily fixed by changing the encryption to WPA2 (vs. WPA+WAP2).

Who is to blame here? I will stand to my initial reaction. OTE, the largest ISP in Greece. True, they don’t build the firmware but they have selected and shipping and are getting paid for the hardware [*]. And if anyone still thinks that it’s not OTE to blame …

… I rest my case.

[*] One may argue that you get this specific CPE for free. Which is as free as a “free mobile phone with a two year contract”. Not free at all.

Selinux is crap.Sorry redhat fun boys but its true.Not even in redhat’s documentation doesnt have enough info.

via E.Balaskas

My own experience with SELinux today? A Virtual Machine with a forgotten root password. OK, that’s easy, boot in single user mode, type passwd(1), enter the new root password, reboot. I mean the process is documented in a shitload of pages (example) and has been working like that since … I don’t know 1996? Should be a piece of cake, right?

NOOOOOOOOOOOOOOOOO!

You see this is SELinux. There are procedures to follow, “passwd root” just won’t work in single user mode and will exit immediately without a prompt. A well-defined procedure that has been working for ages is now broken. Oh well …


# echo 0 >/selinux/enforce
# passwd root
Changing password for user root.
New password:


Oh-well I am fairly certain that there is one out of more than a billion parallel universes where SELinux just works. Just one though.

References: POLA

Setting RedHat Enterprise Linux 6 (hereby RHEL6) as a paravirtualized guest is well documented. However the virt-install command generates a 404 error when run on an Oracle VM server. I used tcpdump(8) to promptly discover that virt-install attempts to retrieve /images/xen/vmlinuz instead of the proper /isolinux/vmlinuz. Clickety-click:

# pwd
/opt/oracle/usr/lib/python2.4/site-packages/virtinst

# diff ParaVirtGuest.py ParaVirtGuest.py.orig
90c90
                 kernel = grabber.urlopen("%s/images/xen/vmlinuz"
92c92
                 initrd = grabber.urlopen("%s/images/xen/initrd.img"
104,105c104,105
<                 kernel = open("%s/isolinux/vmlinuz" %(nfsmntdir,), "r")
                 kernel = open("%s/images/xen/vmlinuz" %(nfsmntdir,), "r")
>                 initrd = open("%s/images/xen/initrd.img" %(nfsmntdir,), "r")

Then firing up virt-install again did the trick (remember to choose a suitable mirror):

# virt-install -n centos6 -r 2048 -f /OVS/publish_pool/centos6.disk.0 \
  --os-type=linux --vnc -p -l \
  http://ftp.ntua.gr/pub/linux/centos/6/os/x86_64/ -b br0 -d

Extra notes: [1] [2]. I only used ext2 for the /boot filesystem but YMMV.

#route print
Network Destination        Netmask          Gateway       Interface  Metric
         10.0.0.0        255.0.0.0      10.8.11.245     192.168.1.65     31

instead of the proper one:

Network Destination        Netmask          Gateway       Interface  Metric
         10.0.0.0        255.0.0.0         On-link       10.8.11.245     31

I’ve conveniently ignored the problem for some time, using a custom script to tear down and re-create the problematic routing entries, till today. Some well placed “echos” in /etc/vpnc/vpnc-script-win.js indicated that vpnc properly constructed the required route add commands, yet the routing table entries were still wrong. Clickety-click:

$ diff /etc/vpnc/vpnc-script-win.js /etc/vpnc/vpnc-script-win-BEDC.js
$ diff -U 1 /etc/vpnc/vpnc-script-win.js /etc/vpnc/vpnc-script-win-BEDC.js
--- /etc/vpnc/vpnc-script-win.js        2010-09-18 13:13:25.778339100 +0300
+++ /etc/vpnc/vpnc-script-win-BEDC.js   2011-02-18 21:35:53.279264500 +0200
@@ -80,2 +80,4 @@
         if (env("CISCO_SPLIT_INC")) {
+               echo("sleeping a little bit; don't ask why but this is needed");
+               run("sleep 5");
                for (var i = 0 ; i < parseInt(env("CISCO_SPLIT_INC")); i++) {

Seems that a timing issue of some sort causes these route add commands to run prematurely, before the TAP tunnel interface is properly configured, resulting in a problematic configuration. Holding them back for just 5 seconds consistently does the trick for me.

Update: if generally interested in configuring VPNC with Windows, check out Alessio Molteni’s detailed post.
Update 2: Corrected status of the official Cisco VPN client.

• suggests using a simple rule for extracting the IPv4 address from the MAC address within the VM
• manages just MAC addresses

This moves the burden of IPv4 configuration to the VM operating system, which has to dynamically calculate the IPv4 address details based on each interface MAC address. While Opennebula provides a relevant sample VM template and script to do this, it comes up a little bit short. Specifically, the script is linux specific, it will probably not work with other Unix O/S like Solaris or FreeBSD, let alone Windows. In addition, extra work is required in order to configure additional but required network parameters, like a default router or a DNS server.

This got me thinking that it should be possible to configure an ISC dhcpd server to do the “MAC to IPv4 address” work. This would make the VM configuration simpler (just configure DHCP for all network interfaces), the solution cross-platform and leverage any existing knowledge (existing subnets, routers, DNS servers, etc).

40,000 foot view

Here is how the solution works from a 40,000 ft. view:

1. A new, unknown Opennebula (ONE) VM performs a DHCP REQ
2. The VM gets assigned a temporary short-lived (i.e. 10 second) IPv4 address in the subnet it resides
3. The DHCP server determines this is a ONE VM based on the MAC prefix
4. The DHCP server determines this is an unknown ONE VM
5. The DHCP server creates on-the-fly (using omshell) a new static host entry for the unknown VM based on the MAC to IPv4 rule
6. The lease of the VM gets expired and a renewal is requested
7. A long term lease based on the reservation created in step 5 is assigned

Subnet configuration

Assuming that the subnet of interest is 192.168.254.0/24, its configuration in the DHCP server follows:

subnet 192.168.254.0 netmask 255.255.255.0 {
  option routers 192.168.254.1;
  option broadcast-address 192.168.254.255;
  pool {
    range 192.168.254.251 192.168.254.254;
    max-lease-time 10;
  }
}

Note the small pool of 4 IP addresses reserved within the subnet. This reflects the “short-lived leases” mentioned above. One should make sure that the relevant Opennebula network configuration leaves these out. For example:

NAME = "mynet"
TYPE = FIXED

BRIDGE = vbr1

LEASES = [IP=192.168.254.2]
LEASES = [IP=192.168.254.3]
...
LEASES = [IP=192.168.254.249]
LEASES = [IP=192.168.254.250]

One will notice that I’ve also left out the router IP address.

Enable OMAPI

One needs to enable OMAPI in order to update the DHCPD configuration during runtime. The following line does the trick:

# grep omapi /etc/dhcp/dhcpd.conf
omapi-port 7911;

In a production environment you should configure a key so that unauthorized users don’t wreak havoc in your server. There are a myriad of posts explaining how to do this, just google for them (random example)

Create an OMAPI script frontend

While OMAPI is useful, it is interactive. Towards this end we need a script that takes a MAC address, an IPv4 address and a hostname and generates a static lease.

# cat /var/tmp/omshell.sh
#!/bin/bash -i

. /root/.bashrc

if [ "$4" == "delayed" ]; then
  #fork to background
  bash $0 $1 $2 $3 "run" &
  exit 0
fi
if [ "$4" != "run" ]; then
  echo error
  exit 1
fi

# /bin/cat <<FOO
/usr/bin/omshell <<FOO
port 7911
connect
new host
set name = "$3"
set hardware-address = $1
set hardware-type = 1
set ip-address = $2
set known = 1
create
FOO

The above script also includes an extra “feature”: it allows specifying whether it will run normally or fork a copy of itself in the background to do the work.

Tying it all together: dhcp-eval

dhcp-eval(5) is the duct tape that ties it all together. It allows us to:

1. Define a class that applies only to DHCP clients with the magic MAC prefix
2. Calculate the appropriate IPv4 address
3. Create a DHCP reservation for the respective MAC:IPv4 pair (if one doesn’t exist already)

# more /etc/dhcp/dhcpd.conf
...
class "one-clients" {
  match if binary-to-ascii (16, 8, "-", substring (hardware, 1, 2)) = "0-3";
  if not (substring(host-decl-name,0,4) = "one-") {
    log (info, "hi one client; creating host entry");
    set onemacaddr = concat(binary-to-ascii (16, 8, "-", substring (hardware, 1, 1)), ":",
                            binary-to-ascii (16, 8, "-", substring (hardware, 2, 1)), ":",
                            binary-to-ascii (16, 8, "-", substring (hardware, 3, 1)), ":",
                            binary-to-ascii (16, 8, "-", substring (hardware, 4, 1)), ":",
                            binary-to-ascii (16, 8, "-", substring (hardware, 5, 1)), ":",
                            binary-to-ascii (16, 8, "-", substring (hardware, 6, 1)));
    set oneipaddr = concat(binary-to-ascii (10, 8, "-", substring (hardware, 3, 1)), ".",
                           binary-to-ascii (10, 8, "-", substring (hardware, 4, 1)), ".",
                           binary-to-ascii (10, 8, "-", substring (hardware, 5, 1)), ".",
                           binary-to-ascii (10, 8, "-", substring (hardware, 6, 1)));
    set onename = concat("one-",
                         binary-to-ascii (10, 8, "-", substring (hardware, 3, 1)),
                         binary-to-ascii (10, 8, "-", substring (hardware, 4, 1)),
                         binary-to-ascii (10, 8, "-", substring (hardware, 5, 1)),
                         binary-to-ascii (10, 8, "-", substring (hardware, 6, 1)));
    execute ("/var/tmp/omshell.sh", onemacaddr, oneipaddr, onename, "delayed");
  }
}
...

Why the short-lived lease?

Some readers will wonder why go through the short-lived 10-second lease. Why not go through the following steps:

1. DHCP req comes in
2. Client is recognized as a ONE VM
3. omshell is used to create static host entry
4. DHCP response is provided based on above entry

Which was what I initially shot for as well. Turns out that the ISC dhcpd doesn’t really allow that, for instance you can’t use the execute keyword unless a valid lease is reserved for the client. And once a valid lease has been reserved there is some kind of “lock” in place which prevents you from tweaking the lease database, unless the appropriate DHCP response has been sent.

Does it work?

Serving my conscription duty means (among other things) that I don’t have handy access to my OpenNebula setup.Hence I resorted to a simple Virtualbox testbed (one DHCP server, one DHCP client) leveraging a private network and dhclient(8) to verify the above. The test MAC address used was 00:03:C0:A8:FE:CF, corresponding to the 192.168.254.207 IP address.

dhcp-client# ifconfig eth0 | grep HWaddr | awk '{print $NF}'
00:03:C0:A8:FE:CF
dhcp-client# dhclient -v -d eth0
Internet Systems Consortium DHCP Client 4.2.0
Copyright 2004-2010 Internet Systems Consortium.
All rights reserved.
For info, please visit https://www.isc.org/software/dhcp/
Listening on LPF/eth0/00:03:c0:a8:fe:cf
Sending on   LPF/eth0/00:03:c0:a8:fe:cf
Sending on   Socket/fallback
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
DHCPOFFER from 192.168.254.1
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPNAK from 192.168.254.1
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPNAK from 192.168.254.1
DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 7
DHCPOFFER from 192.168.254.1
DHCPREQUEST on eth0 to 255.255.255.255 port 67
DHCPACK from 192.168.254.1
bound to 192.168.254.207 -- renewal in 228 seconds.

The respective log entries from the server side follow. One can clearly see that renewal requests for the “short-lived” lease are promptly rejected once the permanent entry has been added via omshell.


Feb 20 14:01:28 f14-server dhcpd: DHCPDISCOVER from 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:29 f14-server dhcpd: DHCPOFFER on 192.168.254.251 to 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:29 f14-server dhcpd: DHCPREQUEST for 192.168.254.251 (192.168.254.1) from 00:03:c0:a8:fe:cf via eth0: lease 192.168.254.251 unavailable.
Feb 20 14:01:29 f14-server dhcpd: DHCPNAK on 192.168.254.251 to 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:32 f14-server dhcpd: DHCPREQUEST for 192.168.254.251 (192.168.254.1) from 00:03:c0:a8:fe:cf via eth0: lease 192.168.254.251 unavailable.
Feb 20 14:01:32 f14-server dhcpd: DHCPNAK on 192.168.254.251 to 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:38 f14-server dhcpd: DHCPREQUEST for 192.168.254.251 (192.168.254.1) from 00:03:c0:a8:fe:cf via eth0: lease 192.168.254.251 unavailable.
Feb 20 14:01:38 f14-server dhcpd: DHCPNAK on 192.168.254.251 to 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:49 f14-server dhcpd: DHCPDISCOVER from 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:49 f14-server dhcpd: DHCPOFFER on 192.168.254.207 to 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:49 f14-server dhcpd: DHCPREQUEST for 192.168.254.207 (192.168.254.1) from 00:03:c0:a8:fe:cf via eth0
Feb 20 14:01:49 f14-server dhcpd: DHCPACK on 192.168.254.207 to 00:03:c0:a8:fe:cf via eth0


Potential issues

That said, the “short-lived” lease could affect the start-up of an actual system (if nothing else it will probably briefly result in the network being down for 3-5 seconds between getting a DHCPNACK for the short-lived lease and getting a DHCPACK for the final IPv4 address).

Other issues could include the short-lived leases being susceptible to a “DoS” attack, malicious clients/VMs getting hold of them and preventing new ones joining the cloud. Some ebtables magic (example) should do the trick here.

Which is more or less non-true. Conscription costs, and a lot of money for that matter. First of all there is a direct cost:
1. It costs approximately 300€ per conscript per month (link in Greek) or circa 100M€ per annum assuming 40,000 conscripts per year that serve for 9 months.
2. It costs up to the same amount per month to the conscript’s family, probably amounting to another 50M€.

Then there is a major and much larger opportunity cost. Most people in Greece and most pro-draft supporters around the world have a tendency to ignore this, which is kind of strange. It’s like claiming that the cost of a car accident, where 4 passengers of a Hyundai i30 get seriously injured for months when hitting a wall, amounts to circa 10,000€, that is the cost of the car plus any minor repairs to the wall. I mean who cares that 4 people got seriously injured?

So how much is the opportunity cost of 1 conscript that has to more or less cease working to serve the army? While there are surprisingly few articles on the matter it makes sense to assume it’s similar to the cost of someone that got seriously injured in a car accident and had to stop working. Mr. Dimitris Liakopoulos has done a good job of quantifying this cost in his Diploma Thesis. By making certain extra admissions, namely that:

1. The conscripts employment ratio is just 50%
2. Prices quoted are adjusted by +50% to take into account inflation between 1999 and 2010
3. A conscript has to cease working for 12 months instead of just 9 (cf. footnote 4, page 2 of The Dynamic Cost of the Draft)

It adds up to an additional sunk opportunity (lost productivity) cost of circa 14.150€ per conscript or circa 565M€ per year. And while this doesn’t directly impact the balance sheets, it does have a measurable impact on the GDP.

And this is not it. At least a couple of different sources claim that the wage of a civilian that has been a conscript vs. one that hasn’t is circa 5% in the long run (this is due to the importance of early professional training). Assuming a ratio of Greek men that have served of 3:1 vs. those that haven’t, this provides for a 5% wage impact for almost 2 million individual. Given the average wages this is almost 1.000€ per annum or 2 billion!

The above raise the conscription cost (direct + opportunity) to almost 2.5 billion euros, or almost 1% of the Greece GDP. Which is close to the 1.5% predicted by the strict mathematical model presented in “The Dynamic Cost of the Draft” paper (Page 12, Table 2, 50% subject to draft, 100% supplementary tax rate).

This doesn’t take into account extra hidden costs, such as employment costs (for military personnel dealing with conscription), retraining cost (not applicable in a professional army), procedural cost (access to what is perceived cheap labor leads to huge spending inefficiencies, at least in the Greek army). That said it should convince any reasonable person that conscription is not free and it may actually be worth spending half a billion dollars per annum to fully professionalize the Greek Army rather than wasting the time of its youth.

Η ώρα είναι περίπου 4 το απόγευμα και ένας ανθυπασπιστής (ή μήπως λοχίας; δε θυμάμαι) ζητάει άτομα να ξεφορτώσουν 3 μεγάλα φορτηγά, φορτωμένα με χαρτοκιβώτια γεμάτα ρούχα (άρβυλα, τζόκευ, τζάκετ, χιτώνια, κλπ) δεμένα πάνω σε ξύλινες παλέτες.

Για τις επόμενες 3-4 ώρες καμιά 10αριά άτομα (σε rotation όχι συνέχεια τα ίδια) λύνουν και ξεφορτώνουν μια-μια τις κούτες, τις κουβαλούν στην αποθήκη και επαναλαμβάνουν, υπό την καθοδήγηση του στελέχους. Κάποια στιγμή κάποιος αναφέρει τις δηλώσεις του Πάγκαλου. Το στέλεχος παίρνει ύφος χιλίων καρδιναλίων και λέει κάτι σαν “ας έρθει εδώ ο Πάγκαλος να δει πόσο γρήγορα ξεφορτώνουν τρία τεράστια φορτηγά στο πι και φι και ας μου ξανακάνει δηλώσεις για παραγωγικότητα”.

Με το μικρό μου μυαλουδάκι σκέφτομαι να του απαντήσω: “μπράβο. αντικατέστησες για μια δουλειά έναν άνθρωπο και ένα κλαρκ με 20-25 άτομα, ρισκάρεις τον τραυματισμό ενός ή περισσότερων από αυτούς και νομίζεις ότι είσαι και παραγωγικός”. Σκέφτομαι τον στρατιωτικό κανονισμό. Σκέφτομαι ότι εκεί που αρχίζει ο στρατός τελειώνει η λογική. Σκέφτομαι ότι απευθύνομαι στους ίδιους ανθρώπους που θεωρούν παραγωγικό και αναγκαίο το θαλαμοφυλίκι αντί για 4-5 motion detectors, 2-3 κάμερες και ένα κεντρικό σύστημα συναγερμού.

Σκέφτομαι “243 περίπου και σήμερα”. Και δε λέω τίποτα.

Fast forward a month or so and I was able to do 10km. At a still unimpressive pace (6+ min/km) but 10km nevertheless. In an attempt to keep myself motivated I set out a goal to run at the local half marathon at the first week of October. I ended up doing the distance, but it was during training and one and a half month earlier than I had hoped for. After that the poking from George, the availability of open seats, the will to step outside my comfort zone and the temptation of finishing the classic route were just irresistible. A full marathon sounded like too big of a bite to chew, 14 times more than what I could run 4 months ago, yet I registered and hoped for the best.

It ended up being a rocky start. I got off to a good pace, passing the half race mark in slightly less than 2 hours. The first couple of cramps hit me very soon after that. This made the uphill climb so painful that I might have quit if I knew what was waiting for me (ignorance is bliss ). I managed to reach the final 12km downhill part in a semi-decent shape thankful I had left the uphill behind. Then the cramps got even worse. But quitting never crossed my mind. I ran and when I couldn’t run I walked, until I could muster enough strength and determination to run again. It was slow and painful, as if having a root canal for two consecutive hours, the only difference that the affected nerves were all over the place in my legs rather than a single tooth (and no you don’t get any anesthesia).

The end result was totally worth it. Running down the final kilometer in Herodou Attikou under the cheers of the crowd, getting the Kallimarmaro into view and then crossing the finishing line is one of those priceless moments that one can hardly put into words. He can only savior it, feel proud about the achievement of having lived it … and even though the first time is always special train hard to experience it again

I ended up rejecting iSCSI mainly due to the additional requirements placed on the storage subsystem. A single ramdisk may be used by multiple nodes in the cluster, each client loads the ramdisk and then self-customizes the filesystem for host-specific parameters in the local RAM. Contrast this with iSCSI which requires a separate iSCSI LUN per client. The cost is not just about extra storage (which could be minimal in the presence of cloning and deduplication), there is an increased management cost (maintain 10 LUNs vs. a single ramdisk) as well as an increased CAPEX and OPEX due to the presence of an extra SAN. Specifically, you can’t really expect to have a highly available iSCSI solution with non-dedicated h/w, whereas a similar HA solution with ramdisks is trivial to setup and just needs two DHCP + TFTP servers (coupled with NIC bonding for extra redundancy).

The above said I thought I’d write some high level notes with regards to the pain of cloning an iSCSI LUN containing a Solaris installation. I can use them as a reference in the future or (if I’m lucky) someone will run into this blog post and suggest a more graceful approach.

1. Setup an iSCSI LUN: it doesn’t really matter how you’ll do it. For my setup I used the Solaris iSCSI target (greetz to @c0t0d0s0 for yet another excellent tutorial)
2. Install Solaris on the iSCSI LUN: Captain Jack provides a thorough step-by-step guide with screenshots with the relevant steps (I will admit wondering whether one can automate the process with Jumpstart and pre-install scripts but I never got there)
3. Boot the newly installed node for the first time, make any site-specific changes you need and then shut it down. Forget this LUN from now on, it will be your “golden image”
4. Clone the iSCSI LUN to a new one: This step is really dependent on your SAN. If you are using ZFS the steps are probably something as simple as the following:

# zfs snapshot rpool/iscsi/lun0@golden
# zfs clone rpool/iscsi/lun0@golden rpool/iscsi/lun1

5. Add the LUN to an existing or new iSCSI target and get its GUID

# iscsitadm create target -u 1 -b /dev/zvol/rdsk/rpool/iscsi/lun1 -t mytarget
# iscsitadm list target -v mytarget
Target: mytarget
    iSCSI Name: iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget
    Connections: 1
        Initiator:
            iSCSI Name: iqn.1986-03.com.sun:01:ba78c2f3ffff.49b911ad
            Alias: unknown
    ACL list:
    TPGT list:
    LUN information:
...
        LUN: 1
            GUID: 600144f04caf16fb00000c29324dee00
            VID: SUN
            PID: SOLARIS
            Type: disk
            Size: 4.0G
            Backing store: /dev/zvol/rdsk/rpool/iscsi/lun1
            Status: online
...

6. Configure a new system to boot from your newly created iSCSI LUN. Here is how a DHCP reservation for gPXE looks like:

host  {
  hardware ethernet ;
  fixed-address                   ;
  option routers                  ;
  option subnet-mask              ;
  option domain-name-servers      ;
  filename                      "";
  # iscsi root-path format        iscsi::[protocol]:[port]:[LUN]:
  option root-path
    "iscsi::::1:iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget;
}

Neat. You installed Solaris in a LUN and you cloned the LUN. One would expect that you can repeat this process as many times as necessary and by changing just the LUN id in gPXE boot as many Solaris systems as you want, right? WRONG!

Turns out that the Solaris installer “burns” the iSCSI boot device identifier in the root filesystem during installation. In fact it does a pretty good job of “burning” it all over the place to make your life miserable when it comes to cloning an iSCSI LUN and re-using it for another system. So you got to jump through some extra hoops, otherwise you will just get a nice kernel panic. The following steps assume that you are using UFS (don’t ask!) but they would probably work similarly with ZFS as well.

1. Mount the newly cloned iSCSI LUN from a Solaris system. This could be the iSCSI target itself if you are using Solaris for that task. Do notice the slight difference between the iSCSI target device and the device we are actually mounting.

# iscsiadm modify discovery -t enable
# iscsiadm list target -S
Target: iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget
        Alias: asmrootufs
        TPGT: 1
        ISID: 4000002a0000
        Connections: 1
        LUN: 0
             Vendor:  SUN
             Product: SOLARIS
             OS Device Name: /dev/rdsk/c2t600144F04CADE09C00000C29324DEE00d0s2
        LUN: 1
             Vendor:  SUN
             Product: SOLARIS
             OS Device Name: /dev/rdsk/c2t600144F04CAF16FB00000C29324DEE00d0s2
...
# ls -l /dev/rdsk/c2t600144F04CAF16FB00000C29324DEE00d0s2
lrwxrwxrwx  -> ../../devices/scsi_vhci/disk@g600144f04caf16fb00000c29324dee00:c,raw
# mount /devices/scsi_vhci/disk\@g600144f04caf16fb00000c29324dee00\:a /mnt/foo/

2. keep a note of the disk path above: “/devices/scsi_vhci/disk@g600144f04caf16fb00000c29324dee00:a”. You’re going to need it
3. Edit the files ./boot/solaris/bootenv.rc, etc/path_to_inst and etc/vfstab. In them you will find references to the iSCSI LUN0 device which was used as our golden image (cf. the iscsiadm command above). Change these to the “/devices” path corresponding to our iSCSI LUN 1.
4. Do a recursive grep (find /mnt/foo -type f | xargs grep) for any other occurences of the old iSCSI LUN. I think the above step covers everything but I played it from an old note and it may miss something.
5. Update the boot archive in the new LUN.

# bootadm list-archive -R /mnt/foo

6. Manually create the required symlink under /dev/dsk

# cd /mnt/foo/dev/dsk
# ln -s ../../devices/scsi_vhci/disk\@g600144f04caf16fb00000c29324dee00\:a c2t600144F04CAF16FB00000C29324DEE00d0s0

7. Unmount “/mnt/foo” and reboot your target node; now everything should work like a charm
8. …
9. Profit!