Thursday, May 6, 2010

NASA and its Eucalyptus-based Nebula

Here I leave an article from The Register about NASA's Nebula infrastructure cloud. If you don't know yet how NASA's project relates to this blog, read the following paragraph which I quote from the article (Bold font added for emphasis).

"You can think of Nebula as a version of Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) that's used only within the federal government, a service that provides on-demand access to scalable processing and storage resources. It's based on Eucalyptus, the open source cloud platform that's bundled with the latest version of Ubuntu. Eucalyptus. Karmic Koala. You get the picture."

I cannot but wonder why NASA chose Eucalyptus to build Nebula...

Some changes...

Hello there,

I am back, posting about some changes that we have made in our cloud...

As we mentioned before, we only have one computer (homer) able to run KVM (due the required virtualization technology). Well... now we are testing bart as our cloud+cc+sc controller and homer as our node controller.

We are also taking the opportunity to test the fresh Ubuntu 10.04 server. So, bart is running a 32 bit version of it and homer is running a 64.
The ubuntu installation is for dummies (us?) and after little time eucalyptus 1.6.2 was running in both machines... without too much effort...

Then, we decided to go back to the issue of running our custom-made VMs. Now using the KVM hypervisor.

Happily, everything that failed before, trying with Xen, worked smoothly with KVM. We took a previous custom VM and it worked at the very first time, proving that the problem was related to Xen and its paravirtualization that needs a special kernel/ramdisk pair in order to run VMs using Eucalyptus.

At least now we don't have this doubt anymore...

Thanks for reading.

Lucio

Monday, April 26, 2010

Weather Report

We set forth a couple of months ago to try our hand at setting up a private cloud here at UWO, for educational use. Our goal was to determine the feasibility of using a cloud to support student coursework, exposing them to a wider array of environments and giving them unprecedented control and authority over them. We chose Eucalyptus as our cloud management software, and acquired a few spare machines to construct our prototype cloud. After a long series of struggles (chronicled throughout this blog), we have come to a few conclusions, and a few more questions.

First, let's address some of the key points of our original plan.

1. Registration of users (checking permission against a database of students)
Users in Eucalyptus are registered in the system by an administrator through the web interface. A user can request an account, but an administrator must authorize the request. There is no way to import a user list from a student database or list, but we do not see this as a hurdle to adoption, as user creation is straightforward and quick.

2. VMs should be accessible through SSH, Remote Desktop, HTTP, etc.
This presented a bit of an issue for us, as we were forced to put our cloud nodes on a separate network in order to provide IP addresses for our VMs. To address this, we set up a VPN server on our cloud controller (Homer), which has two NICs connecting it to both the separate cloud network and the outside world. By connecting to the VPN, access to the VMs through any means required is possible. Thus, this requirement is met.

3. There should be a timeout mechanism to suspend running VMs (as well, it should be possible to turn off the timeout mechanism for individual VMs)
Eucalyptus provides no such mechanism, and as such, this would require an administrator to manually check for old VMs that should be shut down.

4. Relatively simple creation of images tailored to course needs
As is painfully apparent from the last several blog entries, this turned out to be extremely non-trivial. Pre-packaged images run without issue, but provide very little in terms of useful applications and environments. Our options for custom image creation are at best limited and always painful. It may be the case that using KVM instead of Xen for virtualization may alleviate this problem, but since KVM was not an option for us, we have no way to test this possibility at present.

Only the first two points are adequately addressed, and the last two represent significant hurdles in the deployment and use of an educational cloud. These issues stunted our progress, and as such we were unable to truly explore the educational cloud concept. We had intended on looking at methods of managing a relationship between students, courses, professors, and VM images. If we assume that we can solve the issues with VM image creation, we believe that development of a management application or a set of management scripts to handle this relationship would be feasible. Unfortunately, we have been unable to reach a point where this can be evaluated, due mostly to our image creation issues.

So what is our conclusion? Can we deploy a private cloud for educational purposes using Eucalyptus? At the moment, the answer stands as "no", but optimistically it could be upgraded to "maybe" with some more work. Given our current set of hardware, which lacks hardware virtualization support, we are forced to use Xen's paravirtualization. It could be the case that this is the root of many of our issues. With proper hardware, we may be able to overcome our problems and continue to progress towards an educational cloud. As future work, we intend to build a small test cloud with the one machine we have that has hardware virtualization support, build our cloud using KVM, and evaluate this theory. If we are successful, we can proceed to evaluate and possibly develop management interfaces to allow easy management of an educational cloud.

It should be noted that all of this work applies only to the Eucalyptus cloud management software. Other such software exists, such as OpenQRM and OpenNebula. These alternatives should be evaluated against Eucalyptus, especially if Eucalyptus turns out to be inadequate for our purposes.

This represents the end of the opening chapter on our work on a private cloud for education. While we have encountered many road blocks, we have in fact made progress and learned a great deal in the process. We believe the concept is valuable and worth continued effort, and we hold out hope for the near future.

Issues running our custom-made VMs

Hello everybody,

Now it's time to talk about the issues that we have found while trying to use our own VMs on Eucalyptus rather than those pre-packed ones.

Basically the VMs worked, but not as smooth as the certified images. I'm just saying that, because we don't know yet why we had some problems, using our root filesystem with the pair kernel/ramdisk provided by them.

One issue is regarding the DHCP. When we tried to boot the VM with the kernel and ramdisk provided by the pre-packed images, the dhcp doesn't work. The system shows this msg: socket: Address family not supported by protocol - make sure CONFIG_PACKET (Packet socket) and CONFIG_FILTER (Socket Filtering) are enabled in your kernelconfiguration!

However, if we just select a manual IP in the linux configuration, then the boot works fine. The problem in this case is that Eucalyptus never realizes that the VM has an IP, and that is a problem in a cloud environment, assuming that everything should be automatic.

The same problem happened with the Debian image. We therefore suspect that something other than a normal system installation is required. We are trying to get some information from the Eucalyptus forums, but at this point we are stuck in this step. At least it worked, but not as well as we expected.

Another issue that we didn't figure out yet is related to the Xen hypervisor. When we try to boot our rootfs using the pair kernel/ramdisk mentioned before, we need to explicitly specify an extra configuration into the Xen setup file. Otherwise, after loading everything, the system stops and does not show the login prompt.

It took some time for us to realize what was happening. At the beginning we thought that the VMs were not working, since we didn't have their IPs (due to the previously mentioned problem) and we couldn't log into the VMs. Below is the line that we had to add to our Xen configuration file.

extra = 'xencons=tty console=tty1 console=hvc0'

Our assessment at this point is that using a pre-packed image works better and with less headache than trying to create our own image from scratch. However, it shouldn't be so hard to use a custom VM. Maybe it is something related to Xen...

For this reason we are now thinking in doing some extreme modifications in our academic cloud. Keep reading to see what happens.

Until next post.

Lucio

Friday, April 23, 2010

Creating new VMs - Running on Eucalyptus (part 2)

Hello there,

Now it is time to explain another method to install an OS into a VM from scratch.

We used the python-vmbuilder package, in a machine running Ubuntu 9.04. The process is very simple, as you can see below.

First, we installed the tool using the information provided in this link.

Second, we ran the following command to create the new VM:

# vmbuilder xen ubuntu --arch i386 --rootsize 600 --swapsize 256

where xen is the virtualization technology, ubuntu is the OS that we want to install, --arch indicates the desired architecture, and --rootsize and --swapsize specify the partitions sizes, in MB.

The process is straightforward. After running the command, we just had to wait a little bit and the system was created, like magic. There is absolutely nothing to do during the installation, except waiting for the end of it.

The final output is a directory with the Xen configuration file (that we used before uploading to Eucalyptus, just to install some packets) and both a root filesystem image and a swap file. The rootfs image is the one that we have to upload to Eucalyptus.

There is also some information about how to install Centos and then create the filesystem to run on Eucalpytus. You can check this information here.

Our next post will talk about the issues that we have found using with Eucalyptus the VMs we created and what we did to address those issues.

Thanks for reading.

Lucio

Creating new VMs - Running on Eucalyptus (part 1)

Hello there,

As the previous post mentioned, now it's time to talk about running Xen-based VMs that we created from scratch.

It was hard for us to figure out how to make it work. The truth is that we are still having problems. But at least something is working now.

Basically, since we used Xen instead of KVM (due to hardware limitations) several issues arose, since both people and linux distributions are running towards KVM. However, we didn't give up and at the end we succeeded in running our VMs - but not without effort.

Most of our problems were Xen configuration-related (running kernel, ramdisk and filesystem from local files rather than using a Xen bootloader) instead of Eucalyptus-related. However, it doesn't matter, since both softwares have to work together in order to have a good final product. Moreover, as Gastón mentioned before, the documentation is still unsatisfactory.

Well, let's describe our Debian installation first.

Installation of Debian5.0 (Lenny).

Basically we used the "virt-install" script as you can see below.

# virt-install --prompt
What is the name of your virtual machine? debian
How much RAM should be allocated (in megabytes)? 256
What would you like to use as the disk (file path)? debian.img
How large would you like the disk (debian.img) to be (in gigabytes)? 1
What is the install URL? http://mirror.csclub.uwaterloo.ca/debian/dists/stable/main/installer-i386/


Starting install...

So far so good... We used a default installation. (like Windows -> Next, Next, Finish).

After the OS installation, we logged into the machine through the Xen hypervisor and installed two packets: apache and ssh server (just to say that our VM has something different).

After that, we turned off the VM. Then, it was time to extract the root filesystem and upload it to eucalyptus.

Please note that this is not a trivial method and it might not work for different distros. The idea was to use a pair that we knew was working (we used those that were running with our eucalyptus pre-packed images).

# parted debian.img
GNU Parted 1.8.1
Using /vm/debian.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) U
Unit? [compact]? b
(parted) p

Model: (file)
Disk /vm/debian.img: 1073741823B
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number Start End Size Type File system Flags
1 32256B 954132479B 954100224B primary ext3 boot
2 954132480B 1069286399B 115153920B extended
5 954164736B 1069286399B 115121664B logical linux-swap

(parted) q

# nohup dd if=debian.img of=rootfs.img skip=32256 count=954100224

The file is now ready (rootfs.img) to be uploaded to Eucalyptus.

That's all for this post. We will explain more details later.

Until next post,

Lucio

Wednesday, April 21, 2010

Manipulating Existing Images

Hello there,

After all this time we were still running on our cloud the pre-packed images (they call as Eucalyptus-certified images) available in Eucalyptus' website. Until now, we were just worried about the system functionality itself, not about what was running on it.

Those pre-packed images work, but they are static copies and as we explained before, any modification to a image is simply discarded next time the system starts since Eucalyptus will start a new instance copying whatever is in walrus, hence the non-modified image.

So, we decided to work doing two things:

a) modify an existent pre-packed image, adjust it to our needs and then upload the new version to eucalyptus;

b) create a new Xen VM, install a fresh operating system on it and then upload the VM to eucalyptus.

This post is related to the first option listed above. We will post about the second one later, since we are having troubles making a new VM work on the cloud.

Well, let's explain how to customize an eucalyptus pre-packed image...

First, we put the three images (filesystem, ramdisk and kernel) in the same directory. As we didn't know the root's password for the image, we had to mount the filesystem in our local computer in order to modify the /etc/passwd file. To mount the filesystem we used the loop device. Assuming that we are inside the dir where the images are stored, fsimage.img is the name of our filesystem image and /any_dir is the location where we wanted to mount it.

# losetup /dev/loop5 fsimage.img
# mount /dev/loop5 /any_dir


Now, to remove the root password we edited the /any_dir/etc/passwd file as well as the /any_dir/etc/shadow file.

==> /any_dir/etc/passwd file before
root:x:0:0:root:/root:/bin/bash

==> /any_dir/etc/passwd file after
root::0:0:root:/root:/bin/bash

==> /any_dir/etc/shadow file before
root:$1$6lk5l5ux$LCsJdxM4AWzxDRKdcZa0Y1:14708:0:99999:7:::

==> /any_dir/etc/shadow file after
root::14708:0:99999:7:::

After saving the files in our local machine, we unmounted the image:

# umount /any_dir

Now, let's explain how to run the pre-packed image locally using Xen. To make it happen we created a basic config file for the machine in our /etc/xen directory. The content of the file can be seen below:

name = "digs_domain"
memory = 256
ramdisk = "/any_dir/initrd.img"
kernel = "/any_dir/vmlinuz.img"
root = "/dev/sda1 ro"
disk = [ "file:/any_dir/fsimage.img,sda1,w" ]
vif = [ "mac=00:16:36:48:17:dc,bridge=xenbr0,script=vif-bridge" ]
dhcp="on"


Then, we ran it!

# xm create digs_domain
# xm console digs_domain


Inside the VM we logged using the root passwordless account. Of course, we now have a new password simply after running the passwd command.
The modification that we made to the image was simple: we installed the apache server with its basic configuration.

The final step was to shutdown the VM and upload its new version to eucalyptus. We only uploaded our filesystem image, since the kernel and the ramdisk were not modified.

The new modified VM is now ready to be used, proving that system admins can have customized versions of VMs available in their academic cloud.

We will be back to post our adventures installing a system from scratch.

Keep in touch and thanks for reading.

Lucio