A bit about the stuff I've done

Friday 24 August 2012

Changing the OS on a remote server with no physical access

So this is a rather odd thing to want to do but actually with VPS providers having a limited range of pre-installed OSes available - which are not always the ones we are familar with, perhaps its not that odd after all.

So I have a number of VPS servers with CentOS installed.
Now I'm sure Centos is fine for a lot of people but personally I just can't get on with it.

I am a .NET developer, which means I need mono.
The version of mono which ships with CentOS 5 is just too far out of date for my needs now.
And CentOS 6 doesn't seem to have mono at all?!?

So after much battling with trying to compile applications from source, trying to add redhat repositories and install mono from there, trying a number of different solutions I finally gave up and decided to attempt to switch the OS to openSUSE, which I am familar with and I know has an up-to-date mono available with all the bells and whistles that I require.

So now the big question is - how to switch the OS on a remote (other side of world) server with no physical access?

Again I have tried a number of methods most of which ended in a dead-end but the solution I settled on seems to be quite functional, while not ideal.

IMPORTANT: before doing any of the things mentioned in this article please, please, (I cannot stress this enough) PLEASE! do a backup!

Fortunately my VPS provider has a handy "re-install" OS option in case things went spectacularly bad, but I hope never to have to use it.

Also I was working on a new VPS with nothing actually running on it yet, so if I had to delete the VPS and make a new one - it wouldn't be the end of the world.

I strongly recommend that you do NOT attempt any of the following on anything that even remotely resembles a live production server.

If you choose to ignore this warning - don't come running to me!

OK with that (somewhat verbose) disclaimer out of the way - let's get on with it!

N.B. For all the following I wil assume that you are familiar with YAST

So SUSE has this handy feature in YAST named "install to directory".

Sadly I couldn't find any way to run yast from within CentOS.

So you need to have an existing installation of SUSE somewhere, on some other server or computer.

So the first few steps are run from an existing installation of SUSE, somewhere.
Optionally you can skip to Step 11.
  1. Open yast and install the package "yast2-dirinstall"
  2. Close yast
  3. Open yast again (this is necassary to load the new module)
  4. Under software you should now have a new entry "Installation into directory" - open this.
  5. Set the target directory
  6. Turn OFF the option to run YAST and SUSEconfig.
  7. Create Image: No (actually I had this on, but it didn't seem to do anything for me?)
  8. Set the software.
  9.     At this point I recommend having a very minimal install to reduce the size of the files to be copied onto the server to be changed.
        With a minimal software selection the installed size is almost 900MB (SUSE 12.1)
        Also I think it is unlikely that you will be able to get a working X install (although I have not tried) with this method.
        You will however need at least the following packages:
    • bash
    • grep
    • sed
    • yast2
    • vim (or your favourite command line editor)
    • zypper
    • iputils
         that is a nice short list, but of course they pull in a thousand dependencies...
  10. Let that run for a while then quit YAST.
  11. Package the installed to directory into an archive and copy the archive to the server to be changed.
  12. You can optionally skip steps 1-10. By downloading the image I have created here: http://dev.dj-djl.com/TinySuse12.1.tgz

The following steps should be performed on the server to be changed.

  1. Unpack the archive somewhere sensible on your system.
    Do NOT unpack it to the root directory!
  2. Use your favorite editor to create a short script which will open the suse environment in a chroot session:
  3. user@centos# vim runsusechrooted.sh
    #!/bin/bash echo "mounting dev"
    sudo mount -o bind /dev tinysuse/dev
    echo "mounting proc"
    sudo mount -o bind /proc tinysuse/proc
    echo "mounting sys"
    sudo mount -o bind /sys tinysuse/sys

    echo "chrooting"
    sudo chroot tinysuse
    echo "chroot environment closed."
    echo "umounting dev"
    sudo umount tinysuse/dev
    echo "unmounting proc"
    sudo umount tinysuse/proc
    echo "unmounting sys"
    sudo umount tinysuse/sys

  4. set the script as executable
    chmod +x runsusechrooted.sh 
  5. run the script (your system may require you to run this as root):
    user@centos# ./runsusechrooted.sh
    mounting dev
    mounting proc
    mounting sys
    suse:/ #
  6. If you get this far then you now have a successful SUSE environment running inside the parent OS.
    Now the fun part begins!
    At this point in the process I, at first, attempted to get the VPS to boot from the suse system, with not much luck.
    I tried using yast within the suse system to overwrite the bootloader on the VPS.
    I tried editing the grub menu from the host centos system.
    Nothing that I tried would result in the VPS booting the suse system, it always booted CentOS. Unfortunately I can't even see the grub screen to see what errors may or may not be there. My suspicion is that the VPS provider is somehow skipping the bootloader and going straight into a predefined kernel.
    If anyone knows a way to rig it so that the VPS can boot from the new system it would save on all the faff that follows!
    The first thing is we need to get a working internet connection. so:
  7. Exit out of the chroot environment,copy the resolv.conf from the host, then go back into the chrooted environment:
    suse:/ # exit
    user@centos# cp /etc/resolv.conf tinysuse/etc/resolv.conf
    cp: overwrite `tinysuse/etc/resolv.conf'? y
    user@centos#  ./runsusechrooted.sh
    suse:/ # ping -c 1 www.google.com
    PING www.l.google.com ( 56(84) bytes of data.
    64 bytes from dfw06s06-in-f19.1e100.net ( icmp_seq=1 ttl=56 time=1.83 ms

    --- www.l.google.com ping statistics ---
    1 packets transmitted, 1 received, 0% packet loss, time 0ms
    rtt min/avg/max/mdev = 1.834/1.834/1.834/0.000 ms
  8. N.B. If you think your resolv.conf might change, you might want to try hard-linking it. I've not tested this so I have no idea how well it may work? 
  9.  Now we need to install any additional software that may be required:
  10. suse:/ # yast
  11. First open the Software->Software Repositories section and add at least the standard SUSE OSS repository by using the community repositories option. you may want to add other repos at this point too.
  12. Once you have done this you can install software using either yast or zipper.
So now we have a working environment, but what about services?
Thankfully I've devised a cunning hack to get services running in suse too.
  1. First make sure you don't have a conflicting service running in the host environment, I chose to completely uninstall  apache from the host to ensure there are no conflicts.
  2. user@centos# yum remove httpd
  3. Next configure the service in suse as normal (I used yast for the initial config, but will manually edit the config files later to suit my setup)
  4. finally we need a script to start the services.
    suse:/ # vim chrootboot.sh
    OLD=$( cat /runlevel )
     [ "a${OLD}a" == "aa" ] && OLD=0
    IGNORE="network halt reboot"
    echo Changing runlevel from $OLD to $NEW

            START=$(diff <(ls /etc/init.d/rc${OLD}.d -1 | grep "^S" | sed "s/^S[0-9]*//" |sort ) <(ls /etc/init.d/rc${NEW}.d -1 | grep "^S" | sed "s/^S[0-9]*//" | sort ) | grep "^>" | sed "s/^> //")

            STOP=$( diff <(ls /etc/init.d/rc${OLD}.d -1 | grep "^K" | sed "s/^K[0-9]*//" |sort ) <(ls /etc/init.d/rc${NEW}.d -1 | grep "^S" | sed "s/^S[0-9]*//" | sort ) | grep "^<" | sed "s/^< //")

    echo Stopping services in runlevel $OLD

            echo ${STOP} | grep -v "^ *$" | sed "s/ /\n/g" |  grep -v "$(echo $IGNORE | sed "s/ /\\\|/g")" |  sed -e "s#^.*\$#ls -1 /etc/rc.d/rc${OLD}.d/K*\0#" | bash | sort | sed -e "s#^/etc/rc.d/rc${OLD}.d/K[0-9]*\([^0-9].*\)\$#/sbin/service \1 stop#"  | bash

    echo Starting Services in runlevel $1

            echo ${START} | grep -v "^ *$" | sed "s/ /\n/g" |  grep -v "$(echo $IGNORE | sed "s/ /\\\|/g")" | sed -e "s#^.*\$#ls -1 /etc/rc.d/rc${NEW}.d/S*\0#" | bash | sort | sed -e "s#^/etc/rc.d/rc${NEW}.d/S[0-9]*\([^0-9].*\)\$#/sbin/service \1 start#"  | bash

    echo $1 > /runlevel

    N.B. You may want to disable some services to avoid them erroring when they try to start.
  5. Now if you exit from the chroot environment at this point you will notice that the unmounting fails - this is because the services remain running (this is good!)
  6. But - if you reboot the host, you will also notice that the services are not started again.
  7. So now we need a way to start the chrooted services on system boot, and close them cleanly when we reboot.
  8. For this we will create a new host service.
    user@centos# cd /etc/rc.d/init.d
    user@centos# vim susechroot
    #!/bin/bash . /etc/init.d/functions #echo -e "\n\n $(date +%Y-%m-%dT%H%M%S ) $0 $*" >> /susechroot.log case $1 in start)  mount  -o bind /dev /tinysuse/dev         mount -o bind /proc /tinysuse/proc         mount -o bind /sys /tinysuse/sys         mount none -t  devpts /tinysuse/dev/pts         chroot /tinysuse /chrootboot.sh 5         touch /var/lock/subsys/susechroot ;; stop)   chroot /tinysuse /chrootboot.sh 0         sleep 1         umount /tinysuse/dev         umount /tinysuse/proc         umount /tinysuse/sys         umount /tinysuse/dev/pts         rm -f /var/lock/subsys/susechroot ;; status) chroot /tinysuse service --status-all ;; esac
    cd ../rc3.d
    ln -s ../init.d/susechroot S99susechroot
    cd ../rc5.d
    ln -s ../init.d/susechroot S99susechroot
    cd ../rc6.d
    ln -s ../init.d/susechroot K0susechroot
    cd ../rc0.d
    ln -s ../init.d/susechroot K0susechroot
    /sbin/service susechroot start
eh Voila!

Now whenever we reboot the server it executes the scripts for runlevel 5 in the suse environment on startup.
So if we have, e.g. Apache installed in the suse environment and configured for runlevel 5. We can reboot the server and apache will be started with no manual intervention.

Now if anyone can work out how to use Suse's init system, rather than my custom script - please let me know!

Also just a note that it appears SUSE's and CentOS's init systems are actually quite different.
In SUSE the kill scripts live in the same runlevel directory as the Start scripts, but are only executed when there is no matching Start script in the target runlevel.

In CentOS the kill scripts live in all runlevel directories which do not have a start script.

This confusion delayed me in coming up with the above solution.

    Friday 17 August 2012

    Solving "Could not load file or assembly" problems on mono

    So once again I have been tearing my hair out for hours trying to solve this problem.
    Specifically in my case it was mod-mono-server2.exe that was producing this error. Now I *know* that that file exists - I checked, in fact that was the assembly I was executing.
    So it must be one of the dependencies - but which one?
    It turns out there IS a way to find out.
    By setting 2 environment variables before attempting to run the application you can get some debugging information which tells you exactly what files mono was trying to find when it errored.

    The 2 variables in question are:
    So if you run
    MONO_LOG_LEVEL=info MONO_LOG_MASK=asm mono myapp.exe
    You will get a stack load of debugging info on the console, and just before the exception you should see some lines which looks something similar to this:
    Mono-INFO: Assembly Loader probing location: '/usr/lib64/mono/gac/mod-mono-server2/'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mod-mono-server2.dll'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mono/gac/mod-mono-server2/'. Mono-INFO: Assembly Loader probing location: '/usr/lib64/mod-mono-server2.exe'. Handling exception type FileNotFoundException
    Bingo! It's looking for an assembly called "mod-mono-server2"
    But wait a minute! That's the assembly I'm running!

    And no - I have no idea why it can't find it when I've actually given it the path to look in!
    But since I can see where it is looking a quick symbolic link solves my issue:

    ln -s "/usr/local/lib/mono/2.0/mod-mono-server2.exe" /usr/lib64/
    This is probably a bad idea, there must be a good reason why it can't find it in the correct location.

    Now I'm getting a different error about non-blocking sockets.


    Thursday 16 August 2012

    Installing MONO on CENTOS

    I've spent literally hours tearing my hair out trying to get this sorted so I thought I would share what I have found here:

    Most of the search results on google are out of date and/or contain links which no longer work.

    I also tried compiling everything from source but ran into trouble here too.

    Eventually after much headscratching I found a solution :)

    Take a look at this page: http://fedoraproject.org/wiki/EPEL

    Ignore that this is a fedora project as it works just fine on centos.

    Part way down the page under the "How can I use these extra packages?" heading are a couple of links. Choose the one that is appropriate to your version of centos. e.g. I am using centos 6 so I used the "newest version of 'epel-release' for EL6" link.

    Download the rpm file and install it.
    You should probably verify the rpm before installing, but I couldn't work out how and I was loosing patience by this point!

    rpm -i epel-release-*.noarch.rpm
    Now you should be able to search for and install the mono packages (and others) using yum.

    yum install mono-core mono-data mono-devel mono-extras mono-web mono-winforms monodoc mono-web-devel
    N.B. there is no monodevelop included in this lot.
    And I can't find anywhere to get it either :(

    Friday 10 August 2012

    Fixing: Msg 15137, Level 16, State 1, Procedure sp_xp_cmdshell_proxy_account, Line 1 An error occurred during the execution of sp_xp_cmdshell_proxy_account. Possible reasons: the provided account was invalid or the '##xp_cmdshell_proxy_account##' credential could not be created. Error code: '0'.

    This one has been driving my crazy for ages. There are plenty of blog posts out there explaining how to fix this one by running SSMS as an administrator but in my case this did not help. The problem I was having was actually a really subtle one - getting the username right. In all other contexts I am able to use 'ip.add.re.ss\username' Not here so to fix the error in my case I had to use 'servername\username' Hopefully this post will save someone else a major headache!