UPS Greece, you suck

December 27, 2011

Dear UPS,

I am not really certain if your Greek subsidiary is a partner that just carries your trademark or a full subsidiary … but it just outright sucks. And you may want to look into it.

I could waste a number of keystrokes on the matter, but a picture is worth a thousand words:

2 working days (20 & 21 December) for a package to travel from the US all the way to Greece. Hopefully 4 … FOUR! working days (22, 23, 27, 28 December) for the package to travel another ~440km from Athens to Katerini (1 hour away from Thessaloniki, the 2nd largest city of Greece).

On corporate responsibility and a shitty thomson TG585v8 DSL modem

December 27, 2011

Last night I ran into the strangest of problems. Having finally setup my HTPC in my living room I tried out XBMC and Constellation to conveniently control it from my iPad. Clicketty-click … and fail!

After struggling for something like 15′ trying in vain to figure out what idiotic mistake I had made I pulled out my laptop. After another 30′ or so, being unable to contact my HTPC through my laptop too, I found out that not even ARP is working. Afraid of a rootkit I started installing Wireshark on the HTPC. And after 5′ I was finding out in surprise that ARP broadcast requests were not even reaching the HTPC (?!?!).

Some googling later revealed that other people are facing the same problem: ARP simply fails with this DSL modem. And there is little info on whether this is a persistent problem. I can only tell that the problem was temporarily fixed by changing the encryption to WPA2 (vs. WPA+WAP2).

Who is to blame here? I will stand to my initial reaction. OTE, the largest ISP in Greece. True, they don’t build the firmware but they have selected and shipping and are getting paid for the hardware [*]. And if anyone still thinks that it’s not OTE to blame …

… I rest my case.

[*] One may argue that you get this specific CPE for free. Which is as free as a “free mobile phone with a two year contract”. Not free at all.

Selinux & POLA

July 21, 2011

Selinux is crap.Sorry redhat fun boys but its true.Not even in redhat’s documentation doesnt have enough info.

via E.Balaskas

My own experience with SELinux today? A Virtual Machine with a forgotten root password. OK, that’s easy, boot in single user mode, type passwd(1), enter the new root password, reboot. I mean the process is documented in a shitload of pages (example) and has been working like that since … I don’t know 1996? Should be a piece of cake, right?

NOOOOOOOOOOOOOOOOO!

You see this is SELinux. There are procedures to follow, “passwd root” just won’t work in single user mode and will exit immediately without a prompt. A well-defined procedure that has been working for ages is now broken. Oh well …


# echo 0 >/selinux/enforce
# passwd root
Changing password for user root.
New password:

Oh-well I am fairly certain that there is one out of more than a billion parallel universes where SELinux just works. Just one though.

References: POLA

Oracle VM server and RHEL-6 paravirtualized domU

July 14, 2011

This cost me something like 10′ of google search and 15′ troubleshooting. Writing it down so that it can cost the next person just 2′ of google search :-)

Setting RedHat Enterprise Linux 6 (hereby RHEL6) as a paravirtualized guest is well documented. However the virt-install command generates a 404 error when run on an Oracle VM server. I used tcpdump(8) to promptly discover that virt-install attempts to retrieve /images/xen/vmlinuz instead of the proper /isolinux/vmlinuz. Clickety-click:


# pwd
/opt/oracle/usr/lib/python2.4/site-packages/virtinst

# diff ParaVirtGuest.py ParaVirtGuest.py.orig
90c90
                 kernel = grabber.urlopen("%s/images/xen/vmlinuz"
92c92
                 initrd = grabber.urlopen("%s/images/xen/initrd.img"
104,105c104,105
<                 kernel = open("%s/isolinux/vmlinuz" %(nfsmntdir,), "r")
                 kernel = open("%s/images/xen/vmlinuz" %(nfsmntdir,), "r")
>                 initrd = open("%s/images/xen/initrd.img" %(nfsmntdir,), "r")

Then firing up virt-install again did the trick (remember to choose a suitable mirror):

# virt-install -n centos6 -r 2048 -f /OVS/publish_pool/centos6.disk.0 \
  --os-type=linux --vnc -p -l \
  http://ftp.ntua.gr/pub/linux/centos/6/os/x86_64/ -b br0 -d

Extra notes: [1] [2]. I only used ext2 for the /boot filesystem but YMMV.

vpnc & windows 7: sleep a little bit

February 18, 2011

For quite some time I’ve been using vpnc within Cygwin to connect to the aging Cisco VPN 3000 Series Concentrator at dayjob (thank you Cisco for not supporting 64-bit users as Ilias points out in the comments Cisco has finally added partial support for Windows 7 64-bit). However, I’ve been running into the erratic problem where my split tunnels were created eratically and didn’t work. Specifically, once a VPN connection got created route print indicated routes similar to the following:

#route print
Network Destination        Netmask          Gateway       Interface  Metric
         10.0.0.0        255.0.0.0      10.8.11.245     192.168.1.65     31

instead of the proper one:

Network Destination        Netmask          Gateway       Interface  Metric
         10.0.0.0        255.0.0.0         On-link       10.8.11.245     31

I’ve conveniently ignored the problem for some time, using a custom script to tear down and re-create the problematic routing entries, till today. Some well placed “echos” in /etc/vpnc/vpnc-script-win.js indicated that vpnc properly constructed the required route add commands, yet the routing table entries were still wrong. Clickety-click:

$ diff /etc/vpnc/vpnc-script-win.js /etc/vpnc/vpnc-script-win-BEDC.js
$ diff -U 1 /etc/vpnc/vpnc-script-win.js /etc/vpnc/vpnc-script-win-BEDC.js
--- /etc/vpnc/vpnc-script-win.js        2010-09-18 13:13:25.778339100 +0300
+++ /etc/vpnc/vpnc-script-win-BEDC.js   2011-02-18 21:35:53.279264500 +0200
@@ -80,2 +80,4 @@
         if (env("CISCO_SPLIT_INC")) {
+               echo("sleeping a little bit; don't ask why but this is needed");
+               run("sleep 5");
                for (var i = 0 ; i < parseInt(env("CISCO_SPLIT_INC")); i++) {

Seems that a timing issue of some sort causes these route add commands to run prematurely, before the TAP tunnel interface is properly configured, resulting in a problematic configuration. Holding them back for just 5 seconds consistently does the trick for me.

Update: if generally interested in configuring VPNC with Windows, check out Alessio Molteni’s detailed post.
Update 2: Corrected status of the official Cisco VPN client.

Opennebula: dhcpd contextualization magic

February 17, 2011

One of the most frequent questions on the Opennebula lists relates to network contextualization of Virtual Machines (VMs). Specifically, contrary to Eucalyptus or Nimbus, Opennebula does not directly manage a DHCP server. Instead Opennebula:

  • suggests using a simple rule for extracting the IPv4 address from the MAC address within the VM
  • manages just MAC addresses

This moves the burden of IPv4 configuration to the VM operating system, which has to dynamically calculate the IPv4 address details based on each interface MAC address. While Opennebula provides a relevant sample VM template and script to do this, it comes up a little bit short. Specifically, the script is linux specific, it will probably not work with other Unix O/S like Solaris or FreeBSD, let alone Windows. In addition, extra work is required in order to configure additional but required network parameters, like a default router or a DNS server.
Read the rest of this entry »

Estimated cost of conscription

December 12, 2010

A frequent pro-conscription argument that I keep hearing in Greece relates to its cost. Maintaining a 100% professional army is probably too expensive, practically a non-consideration given the austerity measures that the government is taking ever since the IMF saved the country.

Which is more or less non-true. Conscription costs, and a lot of money for that matter. First of all there is a direct cost:
1. It costs approximately 300€ per conscript per month (link in Greek) or circa 100M€ per annum assuming 40,000 conscripts per year that serve for 9 months.
2. It costs up to the same amount per month to the conscript’s family, probably amounting to another 50M€.

Then there is a major and much larger opportunity cost. Most people in Greece and most pro-draft supporters around the world have a tendency to ignore this, which is kind of strange. It’s like claiming that the cost of a car accident, where 4 passengers of a Hyundai i30 get seriously injured for months when hitting a wall, amounts to circa 10,000€, that is the cost of the car plus any minor repairs to the wall. I mean who cares that 4 people got seriously injured?

So how much is the opportunity cost of 1 conscript that has to more or less cease working to serve the army? While there are surprisingly few articles on the matter it makes sense to assume it’s similar to the cost of someone that got seriously injured in a car accident and had to stop working. Mr. Dimitris Liakopoulos has done a good job of quantifying this cost in his Diploma Thesis. By making certain extra admissions, namely that:

  1. The conscripts employment ratio is just 50%
  2. Prices quoted are adjusted by +50% to take into account inflation between 1999 and 2010
  3. A conscript has to cease working for 12 months instead of just 9 (cf. footnote 4, page 2 of The Dynamic Cost of the Draft)

It adds up to an additional sunk opportunity (lost productivity) cost of circa 14.150€ per conscript or circa 565M€ per year. And while this doesn’t directly impact the balance sheets, it does have a measurable impact on the GDP.

And this is not it. At least a couple of different sources claim that the wage of a civilian that has been a conscript vs. one that hasn’t is circa 5% in the long run (this is due to the importance of early professional training). Assuming a ratio of Greek men that have served of 3:1 vs. those that haven’t, this provides for a 5% wage impact for almost 2 million individual. Given the average wages this is almost 1.000€ per annum or 2 billion!

The above raise the conscription cost (direct + opportunity) to almost 2.5 billion euros, or almost 1% of the Greece GDP. Which is close to the 1.5% predicted by the strict mathematical model presented in “The Dynamic Cost of the Draft” paper (Page 12, Table 2, 50% subject to draft, 100% supplementary tax rate).

This doesn’t take into account extra hidden costs, such as employment costs (for military personnel dealing with conscription), retraining cost (not applicable in a professional army), procedural cost (access to what is perceived cheap labor leads to huge spending inefficiencies, at least in the Greek army). That said it should convince any reasonable person that conscription is not free and it may actually be worth spending half a billion dollars per annum to fully professionalize the Greek Army rather than wasting the time of its youth.

Η παραγωγικότητα του στρατού

December 3, 2010

Κόρινθος, 2 Νοεμβρίου 2010. Ημέρα προ-κατάταξης Δ’ ΕΣΣΟ 2010. Λίγο αφού ο Πάγκαλος έχει χαρακτηρίσει τις ένοπλες δυνάμεις αντιπαραγωγικές.

Η ώρα είναι περίπου 4 το απόγευμα και ένας ανθυπασπιστής (ή μήπως λοχίας; δε θυμάμαι) ζητάει άτομα να ξεφορτώσουν 3 μεγάλα φορτηγά, φορτωμένα με χαρτοκιβώτια γεμάτα ρούχα (άρβυλα, τζόκευ, τζάκετ, χιτώνια, κλπ) δεμένα πάνω σε ξύλινες παλέτες.

Για τις επόμενες 3-4 ώρες καμιά 10αριά άτομα (σε rotation όχι συνέχεια τα ίδια) λύνουν και ξεφορτώνουν μια-μια τις κούτες, τις κουβαλούν στην αποθήκη και επαναλαμβάνουν, υπό την καθοδήγηση του στελέχους. Κάποια στιγμή κάποιος αναφέρει τις δηλώσεις του Πάγκαλου. Το στέλεχος παίρνει ύφος χιλίων καρδιναλίων και λέει κάτι σαν “ας έρθει εδώ ο Πάγκαλος να δει πόσο γρήγορα ξεφορτώνουν τρία τεράστια φορτηγά στο πι και φι και ας μου ξανακάνει δηλώσεις για παραγωγικότητα”.

Με το μικρό μου μυαλουδάκι σκέφτομαι να του απαντήσω: “μπράβο. αντικατέστησες για μια δουλειά έναν άνθρωπο και ένα κλαρκ με 20-25 άτομα, ρισκάρεις τον τραυματισμό ενός ή περισσότερων από αυτούς και νομίζεις ότι είσαι και παραγωγικός”. Σκέφτομαι τον στρατιωτικό κανονισμό. Σκέφτομαι ότι εκεί που αρχίζει ο στρατός τελειώνει η λογική. Σκέφτομαι ότι απευθύνομαι στους ίδιους ανθρώπους που θεωρούν παραγωγικό και αναγκαίο το θαλαμοφυλίκι αντί για 4-5 motion detectors, 2-3 κάμερες και ένα κεντρικό σύστημα συναγερμού.

Σκέφτομαι “243 περίπου και σήμερα”. Και δε λέω τίποτα.

Times 14

November 1, 2010

Last April, after Orthodox Easter, I picked up running again. It seemed like a good way to maintain the weight loss after fasting for 49 days and fight the couch potato syndrome that seemed to had hit me since 2004 or so. A short visit to the most popular running ground in Patras was enough to find out that the almost 8 years since I ran regularly were not too kind with my stamina. I recall running just 3 km at what one can barely call a decent pace for such a short workout (circa 6 minute) then I gasping for breath and walking for the remaining kilometer to my car.

Fast forward a month or so and I was able to do 10km. At a still unimpressive pace (6+ min/km) but 10km nevertheless. In an attempt to keep myself motivated I set out a goal to run at the local half marathon at the first week of October. I ended up doing the distance, but it was during training and one and a half month earlier than I had hoped for. After that the poking from George, the availability of open seats, the will to step outside my comfort zone and the temptation of finishing the classic route were just irresistible. A full marathon sounded like too big of a bite to chew, 14 times more than what I could run 4 months ago, yet I registered and hoped for the best.

It ended up being a rocky start. I got off to a good pace, passing the half race mark in slightly less than 2 hours. The first couple of cramps hit me very soon after that. This made the uphill climb so painful that I might have quit if I knew what was waiting for me (ignorance is bliss :) ). I managed to reach the final 12km downhill part in a semi-decent shape thankful I had left the uphill behind. Then the cramps got even worse. But quitting never crossed my mind. I ran and when I couldn’t run I walked, until I could muster enough strength and determination to run again. It was slow and painful, as if having a root canal for two consecutive hours, the only difference that the affected nerves were all over the place in my legs rather than a single tooth (and no you don’t get any anesthesia).

The end result was totally worth it. Running down the final kilometer in Herodou Attikou under the cheers of the crowd, getting the Kallimarmaro into view and then crossing the finishing line is one of those priceless moments that one can hardly put into words. He can only savior it, feel proud about the achievement of having lived it … and even though the first time is always special train hard to experience it again :)

Solaris: cloning an iSCSI LUN

October 21, 2010

While I nailed down on a combination of ramdisk and golden Solaris container images for a diskless boot architectural prototype I had to implement for dayjob, I did toy around initially with iSCSI.

I ended up rejecting iSCSI mainly due to the additional requirements placed on the storage subsystem. A single ramdisk may be used by multiple nodes in the cluster, each client loads the ramdisk and then self-customizes the filesystem for host-specific parameters in the local RAM. Contrast this with iSCSI which requires a separate iSCSI LUN per client. The cost is not just about extra storage (which could be minimal in the presence of cloning and deduplication), there is an increased management cost (maintain 10 LUNs vs. a single ramdisk) as well as an increased CAPEX and OPEX due to the presence of an extra SAN. Specifically, you can’t really expect to have a highly available iSCSI solution with non-dedicated h/w, whereas a similar HA solution with ramdisks is trivial to setup and just needs two DHCP + TFTP servers (coupled with NIC bonding for extra redundancy).

The above said I thought I’d write some high level notes with regards to the pain of cloning an iSCSI LUN containing a Solaris installation. I can use them as a reference in the future or (if I’m lucky) someone will run into this blog post and suggest a more graceful approach.

  1. Setup an iSCSI LUN: it doesn’t really matter how you’ll do it. For my setup I used the Solaris iSCSI target (greetz to @c0t0d0s0 for yet another excellent tutorial)
  2. Install Solaris on the iSCSI LUN: Captain Jack provides a thorough step-by-step guide with screenshots with the relevant steps (I will admit wondering whether one can automate the process with Jumpstart and pre-install scripts but I never got there)
  3. Boot the newly installed node for the first time, make any site-specific changes you need and then shut it down. Forget this LUN from now on, it will be your “golden image”
  4. Clone the iSCSI LUN to a new one: This step is really dependent on your SAN. If you are using ZFS the steps are probably something as simple as the following:
  5. # zfs snapshot rpool/iscsi/lun0@golden
    # zfs clone rpool/iscsi/lun0@golden rpool/iscsi/lun1
    
  6. Add the LUN to an existing or new iSCSI target and get its GUID
  7. # iscsitadm create target -u 1 -b /dev/zvol/rdsk/rpool/iscsi/lun1 -t mytarget
    # iscsitadm list target -v mytarget
    Target: mytarget
        iSCSI Name: iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget
        Connections: 1
            Initiator:
                iSCSI Name: iqn.1986-03.com.sun:01:ba78c2f3ffff.49b911ad
                Alias: unknown
        ACL list:
        TPGT list:
        LUN information:
    ...
            LUN: 1
                GUID: 600144f04caf16fb00000c29324dee00
                VID: SUN
                PID: SOLARIS
                Type: disk
                Size: 4.0G
                Backing store: /dev/zvol/rdsk/rpool/iscsi/lun1
                Status: online
    ...
    
  8. Configure a new system to boot from your newly created iSCSI LUN. Here is how a DHCP reservation for gPXE looks like:
  9. host  {
      hardware ethernet ;
      fixed-address                   ;
      option routers                  ;
      option subnet-mask              ;
      option domain-name-servers      ;
      filename                      "";
      # iscsi root-path format        iscsi::[protocol]:[port]:[LUN]:
      option root-path
        "iscsi::::1:iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget;
    }
    

Neat. You installed Solaris in a LUN and you cloned the LUN. One would expect that you can repeat this process as many times as necessary and by changing just the LUN id in gPXE boot as many Solaris systems as you want, right? WRONG!

Turns out that the Solaris installer “burns” the iSCSI boot device identifier in the root filesystem during installation. In fact it does a pretty good job of “burning” it all over the place to make your life miserable when it comes to cloning an iSCSI LUN and re-using it for another system. So you got to jump through some extra hoops, otherwise you will just get a nice kernel panic. The following steps assume that you are using UFS (don’t ask!) but they would probably work similarly with ZFS as well.

  1. Mount the newly cloned iSCSI LUN from a Solaris system. This could be the iSCSI target itself if you are using Solaris for that task. Do notice the slight difference between the iSCSI target device and the device we are actually mounting.
  2. # iscsiadm modify discovery -t enable
    # iscsiadm list target -S
    Target: iqn.1986-03.com.sun:02:9c23130f-1d8e-6b20-8e95-a6ab8a227924.mytarget
            Alias: asmrootufs
            TPGT: 1
            ISID: 4000002a0000
            Connections: 1
            LUN: 0
                 Vendor:  SUN
                 Product: SOLARIS
                 OS Device Name: /dev/rdsk/c2t600144F04CADE09C00000C29324DEE00d0s2
            LUN: 1
                 Vendor:  SUN
                 Product: SOLARIS
                 OS Device Name: /dev/rdsk/c2t600144F04CAF16FB00000C29324DEE00d0s2
    ...
    # ls -l /dev/rdsk/c2t600144F04CAF16FB00000C29324DEE00d0s2
    lrwxrwxrwx  -> ../../devices/scsi_vhci/disk@g600144f04caf16fb00000c29324dee00:c,raw
    # mount /devices/scsi_vhci/disk\@g600144f04caf16fb00000c29324dee00\:a /mnt/foo/
    
  3. keep a note of the disk path above: “/devices/scsi_vhci/disk@g600144f04caf16fb00000c29324dee00:a”. You’re going to need it
  4. Edit the files ./boot/solaris/bootenv.rc, etc/path_to_inst and etc/vfstab. In them you will find references to the iSCSI LUN0 device which was used as our golden image (cf. the iscsiadm command above). Change these to the “/devices” path corresponding to our iSCSI LUN 1.
  5. Do a recursive grep (find /mnt/foo -type f | xargs grep) for any other occurences of the old iSCSI LUN. I think the above step covers everything but I played it from an old note and it may miss something.
  6. Update the boot archive in the new LUN.
  7. # bootadm list-archive -R /mnt/foo
    
  8. Manually create the required symlink under /dev/dsk
  9. # cd /mnt/foo/dev/dsk
    # ln -s ../../devices/scsi_vhci/disk\@g600144f04caf16fb00000c29324dee00\:a c2t600144F04CAF16FB00000C29324DEE00d0s0
    
  10. Unmount “/mnt/foo” and reboot your target node; now everything should work like a charm
  11. Profit!

Follow

Get every new post delivered to your Inbox.

Join 1,135 other followers