Archive for the ‘Uncategorized’ Category

Raspberry Pi: fun with docker

January 3, 2016

Today’s fun with docker

  1. Start with the Raspbian image
  2. docker pull resin/rpi-raspbian

  3. Install transmission and cleanup
  4. docker run -i -t resin/rpi-raspbian /bin/bash
    apt-get update
    apt-get install transmission-daemon vim-tiny
    mkdir /var/lib/transmission-daemon/incomplete
    rm -rf /var/lib/apt/lists/*
    apt-get clean

  5. Create a new image (note: use “docker ps -a” to identify your container ID)
  6. docker commit c072cafc6d18 mperedim/rpi-raspbian-transmission

  7. Fire up a new docker instance based on the image created. Replace the “media/bluedisk” paths below with the ones where you want your transmission downloads and incomplete files to be located
  8. docker run -d -p 9092:9091 --name downloads -v /media/bluedisk/transmission-daemon/downloads:/var/lib/transmission-daemon/downloads --name incomplete -v /media/bluedisk/transmission-daemon/incomplete:/var/lib/transmission-daemon/incomplete mperedim/rpi-raspbian-transmission /bin/bash -c "/usr/bin/transmission-daemon -a *.*.*.* -f --download-dir /var/lib/transmission-daemon/downloads --incomplete-dir /var/lib/transmission-daemon/incomplete --log-error"

  9. Hit port 9092 of your Raspberry Pi and start downloading stuff

TODO: autocreate from docker file, autostart on boot

my history of linux

November 6, 2013
  • Tried various linux distros between 1997-2003; being a mostly Windows guy back at the time (yeah I did start out as a Windows sysadmin) I ended up not investing any time trying to figure out why the mouse would not work, X didn’t start etc.
  • I ended up installing Gentoo stage 1 back in 2004. Having to manually configure pretty much everything in the system essentially forced me to learn a bunch of new stuff, allowing me to actually land a job that required Unix-fu a year later.
  • I ended up uninstalling Gentoo from my desktop system a year later when I decided that wasting 3 hours to fix my LVM setup was too much for what should be a simple emerge update.
  • In $dayjob I’ve been dutifully running, maintaining and otherwise working with a bunch of Linux systems since 2005, CentOS, Fedora, Ubuntu, RHEL 5.x and RHEL 6.x, OpenSUSE, CoreOS and maybe others I forget. Having been a mostly Solaris fanboy from 2005 onwards I have a love/hate relationship with it but I am willing to admit that it gets the job done most of the time.
  • I still think that Linux on the desktop is not worth my time and effort. If it’s worth yours then great.
  • Oh, I had an Android phone for a year or so and still love my Google gen-1 Nexus 7. Do Android devices count? 🙂

There, happy @ebalaskas?

Xenserver: fake xen tools for Solaris 10 guests

March 31, 2013

Note: Also, you hopefully appreciate that this is completely unsupported.

Xenserver doesn’t enable Shutdown / Reboot buttons for VMs that don’t have the XenServer tools installed. This is an issue for my Solaris 10 guests since tools are not available for this platform. Which has been bugging me for some time.

So I went ahead and dug into the XenServer tools for linux. Turns out that the only thing they’re doing is updating a bunch of parameters on XenStore. However, Solaris 10 doesn’t have a device path for XenStore, putting us back into square one. Or not?

Not really. Turns out that Xen Tools installation is like orgasm memory. Sure, it’s a lot better if one has them, but if not one can modify the appropriate XenStore parameters from dom0 and fake it. XenServer couldn’t care less how the parameters were modified, as long as they are tweaked in the proper order the Suspend/Reboot/Shutdown buttons are enabled. So just get the dom-id of your Solaris VM:

[root@dom0 log]# xe vm-list name-label=i-2-323-VM params=dom-id
dom-id ( RO) : 154


[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/Installed 1
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MajorVersion 6
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MinorVersion 1
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/MicroVersion 0
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/BuildVersion 59235
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/class "SunOS"
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/major "5"
[root@dom0 log]# xenstore-write /local/domain/154/attr/PVAddons/os/minor "10"
[root@dom0 log]# xenstore-write /local/domain/154/data/updated 1

The above is enough to enable the shutdown/reboot/suspend buttons. Unfortunately in the process it also sets the “platform: viridian: true” parameter which doesn’t play nicely with Solaris VMs.

[root@dom0 log]# xe vm-list name-label=i-2-323-VM params=uuid
uuid ( RO)    : 5dc51848-bc9c-dd70-b670-2c7d263a7fe5
[root@dom0 log]# xe vm-param-remove param-name=platform param-key=viridian uuid=[...]

… and see the “Force shutdown”, “Force reboot” buttons disappearing.

So what works?

  1. Reboot: this does a clean reboot of the Solaris 10 domU
  2. Live migrate: not extensively tested, but a test VM does keep network connectivity after a single live migration.

Unfortunately shutdown only kind of works. Hitting the button does initiate a clean shutdown of the Solaris domU but the guest never seems to do an ACPI poweroff and gets stuck at “Press any key to reboot”. This is proving a slightly more touch nut to crack.

Update 2012/04/01: I’ve wasted a few too many hours on “shutdown” not working. Maybe I’ll revisit this in the future but calling it quits for now.

Cloudstack: OS type & xenserver templates

March 6, 2013

I’ve been using cloudstack for circa a month now for virtualising Solaris workloads. It has been mostly working like a charm, once I applied the appropriate workarounds (cf. my relevant findings, courtesy of my IoannisB citrix identity). However one thing has been bugging me for some time:

# xe vm-list name-label=i-2-271-VM params=name-description
name-description ( RW) : Template which allows VM installation from install media

My Solaris VMs are launched using the generic Xenserver template. This is not really to my liking for two reasons. Firstly, I have to apply the viridian:false modification to the default template. Secondly, there is no reason to appreciate whether a VM is a Solaris one or not using the Xenserver CLI.

The fix is to have Cloudstack using the “Solaris 10 (experimental)” template for my Solaris workloads.

  1. Download the cloudstack source code and uncompress to a folder of your choice.
  2. Apply a rather simple diff to the file:
    $ diff
    > _xenServer600GuestOsMap.put("Sun Solaris 10(64-bit)", "Solaris 10 (experimental)");
    > _xenServer602GuestOsMap.put("Sun Solaris 10(64-bit)", "Solaris 10 (experimental)");
  3. Build the JAR files, per the instructions in the Cloudstack installation guide page 16. No need to build DEB or RPM packages
  4. Replace /usr/share/java/cloud-plugin-hypervisor-xen.jar with cloud-plugin-hypervisor-xen-4.0.0-incubating-SNAPSHOT.jar that was built in the step above
  5. Restart the management server.

Slashdot geeks may want to add a Step-6: Profit. Launch again a Solaris 10 64-bit template and enjoy:

# xe vm-list name-label=i-2-272-VM params=name-description
name-description ( RW) : Clones of this template will automatically provision their storage when first booted and then reconfigure themselves with the optimal settings for Solaris 10 (experimental).

Solaris + xenserver + ovswitch

February 28, 2013

This has troubling me for quite some time, hopefully someone else can save a few hours by bumping in this post.

For some reason my Solaris 10 Virtual Machines on Xenserver failed when the Distributed Virtual Switch Controller was also running. I didn’t really troubleshoot the issue until recently since I could live without cross-server private networks. This no longer being the case I decided to look into it again.

Scroll forward a couple of hours and after losing quite some time on trying various tricks on the VM (disabling NIC checksum offload, lower MTUs etc) to no avail I concluded that it must be a hypervisor issue. Digging into the openvswitch tools revealed something interesting.

[root@xenserver ~]# ovs-vsctl list-ports xapi25

Specifically, for my Linux VMs only a vifX.Y interface was being added to the bridge, while for my Solaris ones both a tapX.Y and a vifX.Y. Clickety-click.

[root@xenserver ~]# ovs-vsctl del-port xapi25 tap47.0

Et voila! Network connectivity to the Solaris VM works like a charm. Now to make this change permanent:

[root@xenserver ~]# diff /etc/xensource/scripts/vif.orig /etc/xensource/scripts/vif
if [[ $dev != tap* ]]; then
> $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> else
> echo Skipping command $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
> fi

I am not really certain of the ugly side-effects that this may have. But it does the trick for me.

Update 2013/03/10: A better workaround is to have the above behavior apply only to Solaris VMs. For example, assuming that these are based on the “Solaris 10 (experimental)” template, the following snippet skips the offending command only for the Solaris VMs:

if [[ $dev != tap* ]]; then
    $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details
    xe vm-list dom-id=$DOMID params=name-description | grep 'Solaris 10' 2>&1 >/dev/null || \
        $vsctl --timeout=30 -- --if-exists del-port $dev -- add-port $bridge $dev $vif_details

A Linux based firewall sandwich

March 7, 2012

While Linux has well documented server based load balancing features for both Layer-7 and Layer-4, there is little documentation on how one can make a firewall sandwich. Somehow it seems that this is a hard enough problem that even long time LVS contributor Roberto Nibali can be quoted to say on the topic “Buy a commercial load balancer and be done with it. Spend the spare time with your wife and kids or go to the pub with your buddies”. Problem is I don’t have a wife, and girlfriend is too far away so …

The 40,000ft. view

The following diagram (click to view in full size) captures the 40,000 feet view of the testbed in use. In short we have:

  1. a home-brew load generator simulating thousands of HTTP clients and servers (courtesy of my colleagues Alex and Leonidas)
  2. a couple of Solaris based firewalls
  3. a Linux load balancer running LVS in direct routing mode
  4. a Linux router

The firewalls setup

While the Solaris boxes run a proprietary dayjob product they can be treated as a regular firewall. That is assuming an HTTP request going through firewall-1, the HTTP response should also come back through firewall-1. In the event the response returns through firewall-2, firewall-2 will drop it altogether. Hence one should be able to reproduce the setup with an arbitrary linux firewall in place of the firewalls that:

  1. accepts new connections on the ingress interface (VLAN 162)
  2. reject packets on the egress interface (VLAN 165) that do not belong to an existing connection

Putting aside the above, the networking setup of the firewalls is pretty straightforward, configure the interfaces and add the routes to the HTTP client and server subnets.

firewall-1# more /etc/hostname.e1000g16*
firewall-1# cat /etc/inet/static_routes
# File generated by route(1M) - do not edit.

The load balancer

Linux Virtual Server was chosen for the load balancer setup. I installed Ubuntu 11.10 and then a couple of extra packages:

lvs# apt-get -y install vlan ipvsadm

Once done I configured the ingress interface to communicate with the HTTP clients:

lvs# tail -10 /etc/network/interfaces
iface eth1 inet manual
up ifconfig eth1 up

auto eth1.162
iface eth1.162 inet static
vlan_raw_device eth1

Then I proceeded to the load balancer setup. I want my firewalls to handle all web traffic (rather than a traffic to a specific VIP), so I had to use the firewall mark approach. I mark all HTTP related packets:

lvs# iptables -t mangle -N DIVERT
lvs# iptables -t mangle -A PREROUTING -i eth1.162 -p tcp --dport 80 -j DIVERT
lvs# iptables -t mangle -A DIVERT -j MARK --set-mark 1
lvs# iptables -t mangle -A DIVERT -j ACCEPT

then load balance them to my firewalls:

lvs# ipvsadm -A -f 1 -s sh
lvs# ipvsadm -a -f 1 -r -g -w 100
lvs# ipvsadm -a -f 1 -r -g -w 100

then make sure that my packets do get delivered to IPVS, even if their destination IP is not on a local interface (this caused a never-ending frustration till I found it):

lvs# ip rule add fwmark 1 lookup 100
lvs# ip route add local dev lo table 100

then add a route back to the HTTP clients via the appropriate interface (beats me why it’s needed, perhaps a Linux networking expert can explain why):

lvs# ip route add via

That’s it. Firing up my load generator I get a bunch of packets on eth1.162 which get load balanced to my two firewalls. The firewalls propagate the packet to the egress router, the egress router gets a response …

The egress router

And all it has to do is return the response via the originating firewall. It’s pretty much impossible to determine the originating load balancer through L4 criteria; sure enough if you use source-IP hashing on ingress you can do destination-IP hashing-load balancing on the egress. But what if you’re doing round robin distribution?

Towards this end the most simple way to solve this issue is via MAC persistence. If a request came through MAC-address “11:22:33:44:55:66” then return it through the same MAC. So all I need to do is write down the MAC addresses of the firewalls’ egress interfaces. Damn, I love iptables & iproute2:

egress-router# iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
egress-router# iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j ACCEPT
egress-router# iptables -t mangle -A PREROUTING -i eth2.165 -m mac --mac-source 00:0c:29:ef:a3:29 -j MARK --set-mark 1
egress-router# iptables -t mangle -A PREROUTING -i eth2.165 -m mac --mac-source 00:0c:29:c7:93:96 -j MARK --set-mark 2
egress-router# iptables -A POSTROUTING -t mangle -j CONNMARK --save-mark
egress-router# echo "101 firewall1" >> /etc/iproute2/rt_tables
egress-router# echo "102 firewall2" >> /etc/iproute2/rt_tables
egress-router# ip rule add fwmark 1 table firewall1
egress-router# ip rule add fwmark 2 table firewall2
egress-router# ip route add default via table firewall1
egress-router# ip route add via table firewall1
egress-router# ip route add default via table firewall2
egress-router# ip route add via table firewall2

What does the above snippet do:

  1. it uses a separate connection mark for traffic coming from each firewall; mark-1 for firewall-1, mark-2 for firewall-2
  2. it adds a couple of iproute2 routing tables, one for each firewall
  3. it adds a couple of iproute2 rules to look up the appropriate routing table, depending on the connection mark
  4. it adds all appropriate routes to the firewall specific routing tables

That’s it!

The above are enough to make the firewall sandwich work. Sure enough my load generator reports no errors, my firewall logs report traffic in both firewall with no errors, ipvsadm reports requests being load balanced:

lvs# ipvsadm -l --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
FWM 1 674 9579 0 609892 0
-> 367 3853 0 261303 0
-> 307 5726 0 348589 0

Everything looks great!

What’s missing

Actually a lot. This was just a prototype to make a point to a C-level exec. Notably missing are server healthchecking and loadbalancing failover. But these are problems well understood with a robust solution one can easily google for.

Solaris, VMWare & VGT mode

February 16, 2012

Today I had the strangest of problems. In a VMWare based testbed with a bunch of mixed systems (F5 Virtual appliances, a Linux host, 3 Solaris servers) I was facing severe connectivity issues with the Solaris hosts. Specifically, with all systems connected on VLAN 162 (L3 addressing: anything TCP related from the Solaris hosts failed. F5 and linux Virtual machines had no problem whatsoever.

I quickly fired up my trusted tcpdump tool to figure out what’s wrong. Then I issued a simple ICMP from a Solaris host to the load balancer to see what happens:

solaris-1# ping is alive

[root@loadbalancer-1:Active] config # tcpdump -q -n -i ingress-lb not arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ingress-lb, link-type EN10MB (Ethernet), capture size 108 bytes
19:43:05.048167 IP > ICMP echo request, id 12766, seq 0, length 64
19:43:05.048215 IP > ICMP echo reply, id 12766, seq 0, length 64

Nice. ICMP works. Everything looks nice in the packet capture. Now let’s try some TCP traffic for a change:

solaris-1:/root# telnet 22

[root@loadbalancer-1:Active] config # tcpdump -q -n -i ingress-lb not arp
19:44:06.816663 IP > tcp 0
19:44:07.949006 IP > tcp 0
19:44:10.215576 IP > tcp 0
19:44:14.730324 IP > tcp 0
19:44:23.739898 IP > tcp 0

Du-oh. The packet reaches the load balancer alright but the source IP is corrupted. Googling didn’t really help, other people have run into this or similar issues but no solution. Pinging my skilled colleague Leonidas didn’t help either, he was similarly baffled at what was happening as I was. And then it hit me.

solaris-1# echo "Clickety-click; disabling checksum offload" && echo "set ip:dohwcksum=0" >> /etc/system
Clickety-click; disabling checksum offload

solaris-1:/root# telnet 22
Connected to
Escape character is '^]'.

Uh! The joy! Too bad that 2′ after I figured this out Leonidas had signed off for the day and I can only brag about this in my blog 🙂

Apparmor (synonyms: selinux, crap)

February 8, 2012

Today’s fun was with apparmor. What was a simple MySQL statement to load a bunch of data from a file to a database:

mysql> LOAD DATA INFILE '/var/tmp/some_log_file'
-> INTO TABLE entries
ERROR 29 (HY000): File '/var/tmp/' not found (Errcode: 13)

… was constantly failing for no good reason. It took something like 30′ of pointless online searching until it hit me:

# tail -0f /var/log/syslog
Feb 8 19:11:44 hs21-a kernel: [15359.215686] type=1400 audit(1328721104.742:113): apparmor="DENIED" operation="open" parent=1 profile="/usr/sbin/mysqld" name="/var/tmp/" pid=15623 comm="mysqld" requested_mask="r" denied_mask="r" fsuid=105 ouid=0

Well I guess it’s just like SELinux. There is a parallel universe out there where apparmor just works. Just not this one.



On corporate responsibility and a shitty thomson TG585v8 DSL modem

December 27, 2011

Last night I ran into the strangest of problems. Having finally setup my HTPC in my living room I tried out XBMC and Constellation to conveniently control it from my iPad. Clicketty-click … and fail!

After struggling for something like 15′ trying in vain to figure out what idiotic mistake I had made I pulled out my laptop. After another 30′ or so, being unable to contact my HTPC through my laptop too, I found out that not even ARP is working. Afraid of a rootkit I started installing Wireshark on the HTPC. And after 5′ I was finding out in surprise that ARP broadcast requests were not even reaching the HTPC (?!?!).

Some googling later revealed that other people are facing the same problem: ARP simply fails with this DSL modem. And there is little info on whether this is a persistent problem. I can only tell that the problem was temporarily fixed by changing the encryption to WPA2 (vs. WPA+WAP2).

Who is to blame here? I will stand to my initial reaction. OTE, the largest ISP in Greece. True, they don’t build the firmware but they have selected and shipping and are getting paid for the hardware [*]. And if anyone still thinks that it’s not OTE to blame …

… I rest my case.

[*] One may argue that you get this specific CPE for free. Which is as free as a “free mobile phone with a two year contract”. Not free at all.

Oracle VM server and RHEL-6 paravirtualized domU

July 14, 2011

This cost me something like 10′ of google search and 15′ troubleshooting. Writing it down so that it can cost the next person just 2′ of google search 🙂

Setting RedHat Enterprise Linux 6 (hereby RHEL6) as a paravirtualized guest is well documented. However the virt-install command generates a 404 error when run on an Oracle VM server. I used tcpdump(8) to promptly discover that virt-install attempts to retrieve /images/xen/vmlinuz instead of the proper /isolinux/vmlinuz. Clickety-click:

# pwd

# diff
                 kernel = grabber.urlopen("%s/images/xen/vmlinuz"
                 initrd = grabber.urlopen("%s/images/xen/initrd.img"
<                 kernel = open("%s/isolinux/vmlinuz" %(nfsmntdir,), "r")
                 kernel = open("%s/images/xen/vmlinuz" %(nfsmntdir,), "r")
>                 initrd = open("%s/images/xen/initrd.img" %(nfsmntdir,), "r")

Then firing up virt-install again did the trick (remember to choose a suitable mirror):

# virt-install -n centos6 -r 2048 -f /OVS/publish_pool/centos6.disk.0 \
  --os-type=linux --vnc -p -l \ -b br0 -d

Extra notes: [1] [2]. I only used ext2 for the /boot filesystem but YMMV.