Here I leave an article from The Register about NASA's Nebula infrastructure cloud. If you don't know yet how NASA's project relates to this blog, read the following paragraph which I quote from the article (Bold font added for emphasis).
"You can think of Nebula as a version of Amazon's Elastic Compute Cloud (EC2) and Simple Storage Service (S3) that's used only within the federal government, a service that provides on-demand access to scalable processing and storage resources. It's based on Eucalyptus, the open source cloud platform that's bundled with the latest version of Ubuntu. Eucalyptus. Karmic Koala. You get the picture."
I cannot but wonder why NASA chose Eucalyptus to build Nebula...
Thursday, May 6, 2010
Some changes...
Hello there,
I am back, posting about some changes that we have made in our cloud...
As we mentioned before, we only have one computer (homer) able to run KVM (due the required virtualization technology). Well... now we are testing bart as our cloud+cc+sc controller and homer as our node controller.
We are also taking the opportunity to test the fresh Ubuntu 10.04 server. So, bart is running a 32 bit version of it and homer is running a 64.
The ubuntu installation is for dummies (us?) and after little time eucalyptus 1.6.2 was running in both machines... without too much effort...
Then, we decided to go back to the issue of running our custom-made VMs. Now using the KVM hypervisor.
Happily, everything that failed before, trying with Xen, worked smoothly with KVM. We took a previous custom VM and it worked at the very first time, proving that the problem was related to Xen and its paravirtualization that needs a special kernel/ramdisk pair in order to run VMs using Eucalyptus.
At least now we don't have this doubt anymore...
Thanks for reading.
Lucio
I am back, posting about some changes that we have made in our cloud...
As we mentioned before, we only have one computer (homer) able to run KVM (due the required virtualization technology). Well... now we are testing bart as our cloud+cc+sc controller and homer as our node controller.
We are also taking the opportunity to test the fresh Ubuntu 10.04 server. So, bart is running a 32 bit version of it and homer is running a 64.
The ubuntu installation is for dummies (us?) and after little time eucalyptus 1.6.2 was running in both machines... without too much effort...
Then, we decided to go back to the issue of running our custom-made VMs. Now using the KVM hypervisor.
Happily, everything that failed before, trying with Xen, worked smoothly with KVM. We took a previous custom VM and it worked at the very first time, proving that the problem was related to Xen and its paravirtualization that needs a special kernel/ramdisk pair in order to run VMs using Eucalyptus.
At least now we don't have this doubt anymore...
Thanks for reading.
Lucio
Monday, April 26, 2010
Weather Report
We set forth a couple of months ago to try our hand at setting up a private cloud here at UWO, for educational use. Our goal was to determine the feasibility of using a cloud to support student coursework, exposing them to a wider array of environments and giving them unprecedented control and authority over them. We chose Eucalyptus as our cloud management software, and acquired a few spare machines to construct our prototype cloud. After a long series of struggles (chronicled throughout this blog), we have come to a few conclusions, and a few more questions.
First, let's address some of the key points of our original plan.
1. Registration of users (checking permission against a database of students)
Users in Eucalyptus are registered in the system by an administrator through the web interface. A user can request an account, but an administrator must authorize the request. There is no way to import a user list from a student database or list, but we do not see this as a hurdle to adoption, as user creation is straightforward and quick.
2. VMs should be accessible through SSH, Remote Desktop, HTTP, etc.
This presented a bit of an issue for us, as we were forced to put our cloud nodes on a separate network in order to provide IP addresses for our VMs. To address this, we set up a VPN server on our cloud controller (Homer), which has two NICs connecting it to both the separate cloud network and the outside world. By connecting to the VPN, access to the VMs through any means required is possible. Thus, this requirement is met.
3. There should be a timeout mechanism to suspend running VMs (as well, it should be possible to turn off the timeout mechanism for individual VMs)
Eucalyptus provides no such mechanism, and as such, this would require an administrator to manually check for old VMs that should be shut down.
4. Relatively simple creation of images tailored to course needs
As is painfully apparent from the last several blog entries, this turned out to be extremely non-trivial. Pre-packaged images run without issue, but provide very little in terms of useful applications and environments. Our options for custom image creation are at best limited and always painful. It may be the case that using KVM instead of Xen for virtualization may alleviate this problem, but since KVM was not an option for us, we have no way to test this possibility at present.
Only the first two points are adequately addressed, and the last two represent significant hurdles in the deployment and use of an educational cloud. These issues stunted our progress, and as such we were unable to truly explore the educational cloud concept. We had intended on looking at methods of managing a relationship between students, courses, professors, and VM images. If we assume that we can solve the issues with VM image creation, we believe that development of a management application or a set of management scripts to handle this relationship would be feasible. Unfortunately, we have been unable to reach a point where this can be evaluated, due mostly to our image creation issues.
So what is our conclusion? Can we deploy a private cloud for educational purposes using Eucalyptus? At the moment, the answer stands as "no", but optimistically it could be upgraded to "maybe" with some more work. Given our current set of hardware, which lacks hardware virtualization support, we are forced to use Xen's paravirtualization. It could be the case that this is the root of many of our issues. With proper hardware, we may be able to overcome our problems and continue to progress towards an educational cloud. As future work, we intend to build a small test cloud with the one machine we have that has hardware virtualization support, build our cloud using KVM, and evaluate this theory. If we are successful, we can proceed to evaluate and possibly develop management interfaces to allow easy management of an educational cloud.
It should be noted that all of this work applies only to the Eucalyptus cloud management software. Other such software exists, such as OpenQRM and OpenNebula. These alternatives should be evaluated against Eucalyptus, especially if Eucalyptus turns out to be inadequate for our purposes.
This represents the end of the opening chapter on our work on a private cloud for education. While we have encountered many road blocks, we have in fact made progress and learned a great deal in the process. We believe the concept is valuable and worth continued effort, and we hold out hope for the near future.
First, let's address some of the key points of our original plan.
1. Registration of users (checking permission against a database of students)
Users in Eucalyptus are registered in the system by an administrator through the web interface. A user can request an account, but an administrator must authorize the request. There is no way to import a user list from a student database or list, but we do not see this as a hurdle to adoption, as user creation is straightforward and quick.
2. VMs should be accessible through SSH, Remote Desktop, HTTP, etc.
This presented a bit of an issue for us, as we were forced to put our cloud nodes on a separate network in order to provide IP addresses for our VMs. To address this, we set up a VPN server on our cloud controller (Homer), which has two NICs connecting it to both the separate cloud network and the outside world. By connecting to the VPN, access to the VMs through any means required is possible. Thus, this requirement is met.
3. There should be a timeout mechanism to suspend running VMs (as well, it should be possible to turn off the timeout mechanism for individual VMs)
Eucalyptus provides no such mechanism, and as such, this would require an administrator to manually check for old VMs that should be shut down.
4. Relatively simple creation of images tailored to course needs
As is painfully apparent from the last several blog entries, this turned out to be extremely non-trivial. Pre-packaged images run without issue, but provide very little in terms of useful applications and environments. Our options for custom image creation are at best limited and always painful. It may be the case that using KVM instead of Xen for virtualization may alleviate this problem, but since KVM was not an option for us, we have no way to test this possibility at present.
Only the first two points are adequately addressed, and the last two represent significant hurdles in the deployment and use of an educational cloud. These issues stunted our progress, and as such we were unable to truly explore the educational cloud concept. We had intended on looking at methods of managing a relationship between students, courses, professors, and VM images. If we assume that we can solve the issues with VM image creation, we believe that development of a management application or a set of management scripts to handle this relationship would be feasible. Unfortunately, we have been unable to reach a point where this can be evaluated, due mostly to our image creation issues.
So what is our conclusion? Can we deploy a private cloud for educational purposes using Eucalyptus? At the moment, the answer stands as "no", but optimistically it could be upgraded to "maybe" with some more work. Given our current set of hardware, which lacks hardware virtualization support, we are forced to use Xen's paravirtualization. It could be the case that this is the root of many of our issues. With proper hardware, we may be able to overcome our problems and continue to progress towards an educational cloud. As future work, we intend to build a small test cloud with the one machine we have that has hardware virtualization support, build our cloud using KVM, and evaluate this theory. If we are successful, we can proceed to evaluate and possibly develop management interfaces to allow easy management of an educational cloud.
It should be noted that all of this work applies only to the Eucalyptus cloud management software. Other such software exists, such as OpenQRM and OpenNebula. These alternatives should be evaluated against Eucalyptus, especially if Eucalyptus turns out to be inadequate for our purposes.
This represents the end of the opening chapter on our work on a private cloud for education. While we have encountered many road blocks, we have in fact made progress and learned a great deal in the process. We believe the concept is valuable and worth continued effort, and we hold out hope for the near future.
Issues running our custom-made VMs
Hello everybody,
Now it's time to talk about the issues that we have found while trying to use our own VMs on Eucalyptus rather than those pre-packed ones.
Basically the VMs worked, but not as smooth as the certified images. I'm just saying that, because we don't know yet why we had some problems, using our root filesystem with the pair kernel/ramdisk provided by them.
One issue is regarding the DHCP. When we tried to boot the VM with the kernel and ramdisk provided by the pre-packed images, the dhcp doesn't work. The system shows this msg: socket: Address family not supported by protocol - make sure CONFIG_PACKET (Packet socket) and CONFIG_FILTER (Socket Filtering) are enabled in your kernelconfiguration!
However, if we just select a manual IP in the linux configuration, then the boot works fine. The problem in this case is that Eucalyptus never realizes that the VM has an IP, and that is a problem in a cloud environment, assuming that everything should be automatic.
The same problem happened with the Debian image. We therefore suspect that something other than a normal system installation is required. We are trying to get some information from the Eucalyptus forums, but at this point we are stuck in this step. At least it worked, but not as well as we expected.
Another issue that we didn't figure out yet is related to the Xen hypervisor. When we try to boot our rootfs using the pair kernel/ramdisk mentioned before, we need to explicitly specify an extra configuration into the Xen setup file. Otherwise, after loading everything, the system stops and does not show the login prompt.
It took some time for us to realize what was happening. At the beginning we thought that the VMs were not working, since we didn't have their IPs (due to the previously mentioned problem) and we couldn't log into the VMs. Below is the line that we had to add to our Xen configuration file.
extra = 'xencons=tty console=tty1 console=hvc0'
Our assessment at this point is that using a pre-packed image works better and with less headache than trying to create our own image from scratch. However, it shouldn't be so hard to use a custom VM. Maybe it is something related to Xen...
For this reason we are now thinking in doing some extreme modifications in our academic cloud. Keep reading to see what happens.
Until next post.
Lucio
Now it's time to talk about the issues that we have found while trying to use our own VMs on Eucalyptus rather than those pre-packed ones.
Basically the VMs worked, but not as smooth as the certified images. I'm just saying that, because we don't know yet why we had some problems, using our root filesystem with the pair kernel/ramdisk provided by them.
One issue is regarding the DHCP. When we tried to boot the VM with the kernel and ramdisk provided by the pre-packed images, the dhcp doesn't work. The system shows this msg: socket: Address family not supported by protocol - make sure CONFIG_PACKET (Packet socket) and CONFIG_FILTER (Socket Filtering) are enabled in your kernelconfiguration!
However, if we just select a manual IP in the linux configuration, then the boot works fine. The problem in this case is that Eucalyptus never realizes that the VM has an IP, and that is a problem in a cloud environment, assuming that everything should be automatic.
The same problem happened with the Debian image. We therefore suspect that something other than a normal system installation is required. We are trying to get some information from the Eucalyptus forums, but at this point we are stuck in this step. At least it worked, but not as well as we expected.
Another issue that we didn't figure out yet is related to the Xen hypervisor. When we try to boot our rootfs using the pair kernel/ramdisk mentioned before, we need to explicitly specify an extra configuration into the Xen setup file. Otherwise, after loading everything, the system stops and does not show the login prompt.
It took some time for us to realize what was happening. At the beginning we thought that the VMs were not working, since we didn't have their IPs (due to the previously mentioned problem) and we couldn't log into the VMs. Below is the line that we had to add to our Xen configuration file.
extra = 'xencons=tty console=tty1 console=hvc0'
Our assessment at this point is that using a pre-packed image works better and with less headache than trying to create our own image from scratch. However, it shouldn't be so hard to use a custom VM. Maybe it is something related to Xen...
For this reason we are now thinking in doing some extreme modifications in our academic cloud. Keep reading to see what happens.
Until next post.
Lucio
Friday, April 23, 2010
Creating new VMs - Running on Eucalyptus (part 2)
Hello there,
Now it is time to explain another method to install an OS into a VM from scratch.
We used the python-vmbuilder package, in a machine running Ubuntu 9.04. The process is very simple, as you can see below.
First, we installed the tool using the information provided in this link.
Second, we ran the following command to create the new VM:
# vmbuilder xen ubuntu --arch i386 --rootsize 600 --swapsize 256
where xen is the virtualization technology, ubuntu is the OS that we want to install, --arch indicates the desired architecture, and --rootsize and --swapsize specify the partitions sizes, in MB.
The process is straightforward. After running the command, we just had to wait a little bit and the system was created, like magic. There is absolutely nothing to do during the installation, except waiting for the end of it.
The final output is a directory with the Xen configuration file (that we used before uploading to Eucalyptus, just to install some packets) and both a root filesystem image and a swap file. The rootfs image is the one that we have to upload to Eucalyptus.
There is also some information about how to install Centos and then create the filesystem to run on Eucalpytus. You can check this information here.
Our next post will talk about the issues that we have found using with Eucalyptus the VMs we created and what we did to address those issues.
Thanks for reading.
Lucio
Now it is time to explain another method to install an OS into a VM from scratch.
We used the python-vmbuilder package, in a machine running Ubuntu 9.04. The process is very simple, as you can see below.
First, we installed the tool using the information provided in this link.
Second, we ran the following command to create the new VM:
# vmbuilder xen ubuntu --arch i386 --rootsize 600 --swapsize 256
where xen is the virtualization technology, ubuntu is the OS that we want to install, --arch indicates the desired architecture, and --rootsize and --swapsize specify the partitions sizes, in MB.
The process is straightforward. After running the command, we just had to wait a little bit and the system was created, like magic. There is absolutely nothing to do during the installation, except waiting for the end of it.
The final output is a directory with the Xen configuration file (that we used before uploading to Eucalyptus, just to install some packets) and both a root filesystem image and a swap file. The rootfs image is the one that we have to upload to Eucalyptus.
There is also some information about how to install Centos and then create the filesystem to run on Eucalpytus. You can check this information here.
Our next post will talk about the issues that we have found using with Eucalyptus the VMs we created and what we did to address those issues.
Thanks for reading.
Lucio
Creating new VMs - Running on Eucalyptus (part 1)
Hello there,
As the previous post mentioned, now it's time to talk about running Xen-based VMs that we created from scratch.
It was hard for us to figure out how to make it work. The truth is that we are still having problems. But at least something is working now.
Basically, since we used Xen instead of KVM (due to hardware limitations) several issues arose, since both people and linux distributions are running towards KVM. However, we didn't give up and at the end we succeeded in running our VMs - but not without effort.
Most of our problems were Xen configuration-related (running kernel, ramdisk and filesystem from local files rather than using a Xen bootloader) instead of Eucalyptus-related. However, it doesn't matter, since both softwares have to work together in order to have a good final product. Moreover, as Gastón mentioned before, the documentation is still unsatisfactory.
Well, let's describe our Debian installation first.
Installation of Debian5.0 (Lenny).
Basically we used the "virt-install" script as you can see below.
# virt-install --prompt
What is the name of your virtual machine? debian
How much RAM should be allocated (in megabytes)? 256
What would you like to use as the disk (file path)? debian.img
How large would you like the disk (debian.img) to be (in gigabytes)? 1
What is the install URL? http://mirror.csclub.uwaterloo.ca/debian/dists/stable/main/installer-i386/
Starting install...
So far so good... We used a default installation. (like Windows -> Next, Next, Finish).
After the OS installation, we logged into the machine through the Xen hypervisor and installed two packets: apache and ssh server (just to say that our VM has something different).
After that, we turned off the VM. Then, it was time to extract the root filesystem and upload it to eucalyptus.
Please note that this is not a trivial method and it might not work for different distros. The idea was to use a pair that we knew was working (we used those that were running with our eucalyptus pre-packed images).
# parted debian.img
GNU Parted 1.8.1
Using /vm/debian.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) U
Unit? [compact]? b
(parted) p
Model: (file)
Disk /vm/debian.img: 1073741823B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32256B 954132479B 954100224B primary ext3 boot
2 954132480B 1069286399B 115153920B extended
5 954164736B 1069286399B 115121664B logical linux-swap
(parted) q
# nohup dd if=debian.img of=rootfs.img skip=32256 count=954100224
The file is now ready (rootfs.img) to be uploaded to Eucalyptus.
That's all for this post. We will explain more details later.
Until next post,
Lucio
As the previous post mentioned, now it's time to talk about running Xen-based VMs that we created from scratch.
It was hard for us to figure out how to make it work. The truth is that we are still having problems. But at least something is working now.
Basically, since we used Xen instead of KVM (due to hardware limitations) several issues arose, since both people and linux distributions are running towards KVM. However, we didn't give up and at the end we succeeded in running our VMs - but not without effort.
Most of our problems were Xen configuration-related (running kernel, ramdisk and filesystem from local files rather than using a Xen bootloader) instead of Eucalyptus-related. However, it doesn't matter, since both softwares have to work together in order to have a good final product. Moreover, as Gastón mentioned before, the documentation is still unsatisfactory.
Well, let's describe our Debian installation first.
Installation of Debian5.0 (Lenny).
Basically we used the "virt-install" script as you can see below.
# virt-install --prompt
What is the name of your virtual machine? debian
How much RAM should be allocated (in megabytes)? 256
What would you like to use as the disk (file path)? debian.img
How large would you like the disk (debian.img) to be (in gigabytes)? 1
What is the install URL? http://mirror.csclub.uwaterloo.ca/debian/dists/stable/main/installer-i386/
Starting install...
So far so good... We used a default installation. (like Windows -> Next, Next, Finish).
After the OS installation, we logged into the machine through the Xen hypervisor and installed two packets: apache and ssh server (just to say that our VM has something different).
After that, we turned off the VM. Then, it was time to extract the root filesystem and upload it to eucalyptus.
Please note that this is not a trivial method and it might not work for different distros. The idea was to use a pair
# parted debian.img
GNU Parted 1.8.1
Using /vm/debian.img
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) U
Unit? [compact]? b
(parted) p
Model: (file)
Disk /vm/debian.img: 1073741823B
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 32256B 954132479B 954100224B primary ext3 boot
2 954132480B 1069286399B 115153920B extended
5 954164736B 1069286399B 115121664B logical linux-swap
(parted) q
# nohup dd if=debian.img of=rootfs.img skip=32256 count=954100224
The file is now ready (rootfs.img) to be uploaded to Eucalyptus.
That's all for this post. We will explain more details later.
Until next post,
Lucio
Wednesday, April 21, 2010
Manipulating Existing Images
Hello there,
After all this time we were still running on our cloud the pre-packed images (they call as Eucalyptus-certified images) available in Eucalyptus' website. Until now, we were just worried about the system functionality itself, not about what was running on it.
Those pre-packed images work, but they are static copies and as we explained before, any modification to a image is simply discarded next time the system starts since Eucalyptus will start a new instance copying whatever is in walrus, hence the non-modified image.
So, we decided to work doing two things:
a) modify an existent pre-packed image, adjust it to our needs and then upload the new version to eucalyptus;
b) create a new Xen VM, install a fresh operating system on it and then upload the VM to eucalyptus.
This post is related to the first option listed above. We will post about the second one later, since we are having troubles making a new VM work on the cloud.
Well, let's explain how to customize an eucalyptus pre-packed image...
First, we put the three images (filesystem, ramdisk and kernel) in the same directory. As we didn't know the root's password for the image, we had to mount the filesystem in our local computer in order to modify the /etc/passwd file. To mount the filesystem we used the loop device. Assuming that we are inside the dir where the images are stored, fsimage.img is the name of our filesystem image and /any_dir is the location where we wanted to mount it.
# losetup /dev/loop5 fsimage.img
# mount /dev/loop5 /any_dir
Now, to remove the root password we edited the /any_dir/etc/passwd file as well as the /any_dir/etc/shadow file.
==> /any_dir/etc/passwd file before
root:x:0:0:root:/root:/bin/bash
==> /any_dir/etc/passwd file after
root::0:0:root:/root:/bin/bash
==> /any_dir/etc/shadow file before
root:$1$6lk5l5ux$LCsJdxM4AWzxDRKdcZa0Y1:14708:0:99999:7:::
==> /any_dir/etc/shadow file after
root::14708:0:99999:7:::
After saving the files in our local machine, we unmounted the image:
# umount /any_dir
Now, let's explain how to run the pre-packed image locally using Xen. To make it happen we created a basic config file for the machine in our /etc/xen directory. The content of the file can be seen below:
name = "digs_domain"
memory = 256
ramdisk = "/any_dir/initrd.img"
kernel = "/any_dir/vmlinuz.img"
root = "/dev/sda1 ro"
disk = [ "file:/any_dir/fsimage.img,sda1,w" ]
vif = [ "mac=00:16:36:48:17:dc,bridge=xenbr0,script=vif-bridge" ]
dhcp="on"
Then, we ran it!
# xm create digs_domain
# xm console digs_domain
Inside the VM we logged using the root passwordless account. Of course, we now have a new password simply after running the passwd command.
The modification that we made to the image was simple: we installed the apache server with its basic configuration.
The final step was to shutdown the VM and upload its new version to eucalyptus. We only uploaded our filesystem image, since the kernel and the ramdisk were not modified.
The new modified VM is now ready to be used, proving that system admins can have customized versions of VMs available in their academic cloud.
We will be back to post our adventures installing a system from scratch.
Keep in touch and thanks for reading.
Lucio
After all this time we were still running on our cloud the pre-packed images (they call as Eucalyptus-certified images) available in Eucalyptus' website. Until now, we were just worried about the system functionality itself, not about what was running on it.
Those pre-packed images work, but they are static copies and as we explained before, any modification to a image is simply discarded next time the system starts since Eucalyptus will start a new instance copying whatever is in walrus, hence the non-modified image.
So, we decided to work doing two things:
a) modify an existent pre-packed image, adjust it to our needs and then upload the new version to eucalyptus;
b) create a new Xen VM, install a fresh operating system on it and then upload the VM to eucalyptus.
This post is related to the first option listed above. We will post about the second one later, since we are having troubles making a new VM work on the cloud.
Well, let's explain how to customize an eucalyptus pre-packed image...
First, we put the three images (filesystem, ramdisk and kernel) in the same directory. As we didn't know the root's password for the image, we had to mount the filesystem in our local computer in order to modify the /etc/passwd file. To mount the filesystem we used the loop device. Assuming that we are inside the dir where the images are stored, fsimage.img is the name of our filesystem image and /any_dir is the location where we wanted to mount it.
# losetup /dev/loop5 fsimage.img
# mount /dev/loop5 /any_dir
Now, to remove the root password we edited the /any_dir/etc/passwd file as well as the /any_dir/etc/shadow file.
==> /any_dir/etc/passwd file before
root:x:0:0:root:/root:/bin/bash
==> /any_dir/etc/passwd file after
root::0:0:root:/root:/bin/bash
==> /any_dir/etc/shadow file before
root:$1$6lk5l5ux$LCsJdxM4AWzxDRKdcZa0Y1:14708:0:99999:7:::
==> /any_dir/etc/shadow file after
root::14708:0:99999:7:::
After saving the files in our local machine, we unmounted the image:
# umount /any_dir
Now, let's explain how to run the pre-packed image locally using Xen. To make it happen we created a basic config file for the machine in our /etc/xen directory. The content of the file can be seen below:
name = "digs_domain"
memory = 256
ramdisk = "/any_dir/initrd.img"
kernel = "/any_dir/vmlinuz.img"
root = "/dev/sda1 ro"
disk = [ "file:/any_dir/fsimage.img,sda1,w" ]
vif = [ "mac=00:16:36:48:17:dc,bridge=xenbr0,script=vif-bridge" ]
dhcp="on"
Then, we ran it!
# xm create digs_domain
# xm console digs_domain
Inside the VM we logged using the root passwordless account. Of course, we now have a new password simply after running the passwd command.
The modification that we made to the image was simple: we installed the apache server with its basic configuration.
The final step was to shutdown the VM and upload its new version to eucalyptus. We only uploaded our filesystem image, since the kernel and the ramdisk were not modified.
The new modified VM is now ready to be used, proving that system admins can have customized versions of VMs available in their academic cloud.
We will be back to post our adventures installing a system from scratch.
Keep in touch and thanks for reading.
Lucio
Monday, April 19, 2010
A Cloud for the Common Man
I decided the time had come to step down from the administrative role and take a look at the life of a normal user of our cloud. A user can apply to receive an account through the cloud web interface (Homer's IP:8443), after which an administrator can approve the account through the same interface.
I experienced no trouble downloading my user credentials and accessing the cloud. The view of the cloud from a regular user is a bit different, with all information about the available resources within the cloud hidden, as well as any information about the instances, volumes, etc. of other users. I was able to create my own instance without issue, create and attach a volume, create a snapshot of a volume, and perform any other task without incident.
This did, however, highlight a problem that we are all too aware of: our cloud runs in its own disconnected network, with Homer as the only point of access to the rest of the world. This means that once an instance has been started, there is no way to actually connect to it from outside of the private cloud network. Our interim solution has been to ssh into Homer and connect to the other machines and running instances from there. This works for us as administrators, but is a poor solution for general use. It also does not allow us to easily access other services running on an instance, for example, a web server.
To solve this problem, I decided to set up a VPN server on Homer to allow us to join the private cloud network from the outside, freely accessing any instance and viewing web pages served from them right from our browser. I installed OpenVPN server on Homer. Installation and configuration was straight forward and quick. Since our firewall on Homer was already disabled, the only system configuration I needed to perform was to enable NAT for the internal network Ethernet card (eth1).
Keys and certificates must be generated for each user of the VPN, using the easy-rsa tools with OpenVPN on the server. The user requires the OpenVPN client, the aforementioned certificate and key files, and a config file that contains some basic information (IP address of server, location of key/certificate files). A script to quickly build all of this for a new user could easily be written.
I was able to run the client and connect to Homer's VPN server without difficulty, and everything operates as expected. This seems to be a workable solution for general access to the cloud, and it doesn't seem unreasonable to require that students use VPN, should a cloud be deployed for academic use.
Adios amigos,
Mike
I experienced no trouble downloading my user credentials and accessing the cloud. The view of the cloud from a regular user is a bit different, with all information about the available resources within the cloud hidden, as well as any information about the instances, volumes, etc. of other users. I was able to create my own instance without issue, create and attach a volume, create a snapshot of a volume, and perform any other task without incident.
This did, however, highlight a problem that we are all too aware of: our cloud runs in its own disconnected network, with Homer as the only point of access to the rest of the world. This means that once an instance has been started, there is no way to actually connect to it from outside of the private cloud network. Our interim solution has been to ssh into Homer and connect to the other machines and running instances from there. This works for us as administrators, but is a poor solution for general use. It also does not allow us to easily access other services running on an instance, for example, a web server.
To solve this problem, I decided to set up a VPN server on Homer to allow us to join the private cloud network from the outside, freely accessing any instance and viewing web pages served from them right from our browser. I installed OpenVPN server on Homer. Installation and configuration was straight forward and quick. Since our firewall on Homer was already disabled, the only system configuration I needed to perform was to enable NAT for the internal network Ethernet card (eth1).
Keys and certificates must be generated for each user of the VPN, using the easy-rsa tools with OpenVPN on the server. The user requires the OpenVPN client, the aforementioned certificate and key files, and a config file that contains some basic information (IP address of server, location of key/certificate files). A script to quickly build all of this for a new user could easily be written.
I was able to run the client and connect to Homer's VPN server without difficulty, and everything operates as expected. This seems to be a workable solution for general access to the cloud, and it doesn't seem unreasonable to require that students use VPN, should a cloud be deployed for academic use.
Adios amigos,
Mike
Thursday, April 15, 2010
New layout + Volume Attachment
Hello everybody,
After a period of inactivity, due to the end of the academic term, we are back again working on our cloud... as we said before, we wanted to redefine our cloud's layout. We did it! Now, logically, we have something like this:
Every node is now attached to the cluster controller, which is running in Marge (not in Homer anymore). However, we had to register the new cluster and storage controller in Homer with these commands:
# euca_conf --register-cluster simpsons marge
# euca_conf --register-sc simpsons marge
After receiving the success from both commands we are now able to play around our cloud using the new configuration.
One particular issue of VM instances in Eucalyptus is that each time that we shutdown the instance, all working data is basically lost. To avoid this problem, a solution called the Storage Controller was used and now the user can create volumes in the front-end and dynamically attach them to VM instances running in the cloud.
The mechanism is pretty simple. First, in the front-end, we created the volume:
# euca-create-volume -s 1 -z simpsons
where -s specifies size in GB, and -z specifies the name of our cluster.
Next, we attached the new volume to a VM instance with this command:
# euca-attach-volume -i i-INSTNUMBER -d /dev/sdb vol-VOLNUMBER
where -i is the instance identifier and -d is the device name you are requesting to be assigned to your SC volume.
Now, after received the success response from euca2ools, we connected into our VM instance through ssh in order to check the creation of the new device. Happily, the new sdb device was there. But of course, it was like a new hard disk attached to a machine. The main difference is that we could do it with a running machine.
Basically, as it was the first time accessing the volume, we had to create a new partition (1) in our new virtual hd, as well as to format it:
# fdisk /dev/sdb
# mkfs -t ext3 /dev/sdb1
# fsck -f -y /dev/sdb1
After mounting the device we confirmed that it is now working. Also, we have made some tests detaching the volume from the front-end and attaching again and our data was still there. It really works!
That's all for this post fellas. Stay tuned.
Lucio
After a period of inactivity, due to the end of the academic term, we are back again working on our cloud... as we said before, we wanted to redefine our cloud's layout. We did it! Now, logically, we have something like this:
Every node is now attached to the cluster controller, which is running in Marge (not in Homer anymore). However, we had to register the new cluster and storage controller in Homer with these commands:
# euca_conf --register-cluster simpsons marge
# euca_conf --register-sc simpsons marge
After receiving the success from both commands we are now able to play around our cloud using the new configuration.
One particular issue of VM instances in Eucalyptus is that each time that we shutdown the instance, all working data is basically lost. To avoid this problem, a solution called the Storage Controller was used and now the user can create volumes in the front-end and dynamically attach them to VM instances running in the cloud.
The mechanism is pretty simple. First, in the front-end, we created the volume:
# euca-create-volume -s 1 -z simpsons
where -s specifies size in GB, and -z specifies the name of our cluster.
Next, we attached the new volume to a VM instance with this command:
# euca-attach-volume -i i-INSTNUMBER -d /dev/sdb vol-VOLNUMBER
where -i is the instance identifier and -d is the device name you are requesting to be assigned to your SC volume.
Now, after received the success response from euca2ools, we connected into our VM instance through ssh in order to check the creation of the new device. Happily, the new sdb device was there. But of course, it was like a new hard disk attached to a machine. The main difference is that we could do it with a running machine.
Basically, as it was the first time accessing the volume, we had to create a new partition (1) in our new virtual hd, as well as to format it:
# fdisk /dev/sdb
# mkfs -t ext3 /dev/sdb1
# fsck -f -y /dev/sdb1
After mounting the device we confirmed that it is now working. Also, we have made some tests detaching the volume from the front-end and attaching again and our data was still there. It really works!
That's all for this post fellas. Stay tuned.
Lucio
Wednesday, April 14, 2010
Documentation
Last week I registered in Eucalyptus' forum and immediately posted a question inquiring about the purpose of each controller in Eucalyptus. As I said then, it seems to me that there is a lack of clear and complete documentation about the system (mostly, regarding design and architecture), and as a consequence, many people are going through the motions of following tutorials and installation guides without really understanding the system at hand.
Somebody suggested reading the paper The Eucalyptus Open-source Cloud-computing System - which I had already read - to learn about the controllers. One problem with that paper is that it was written when Eucalyptus had no Storage Controller component besides Walrus. Therefore, there is no explanation about the purpose of the Storage Controller that has to exist now in every cluster running Eucalyptus. Additional note: that paper made back then an unfortunate use of the term storage controller to refer to Walrus, what contributes to the confusion.
Somebody also replied to my post providing a link to a discussion about the nature and purpose of Walrus and the Storage Controller. That helped to clarify the situation. However, I still think that Eucalyptus' documentation requires a lot of work if people are to start using the system.
Somebody suggested reading the paper The Eucalyptus Open-source Cloud-computing System - which I had already read - to learn about the controllers. One problem with that paper is that it was written when Eucalyptus had no Storage Controller component besides Walrus. Therefore, there is no explanation about the purpose of the Storage Controller that has to exist now in every cluster running Eucalyptus. Additional note: that paper made back then an unfortunate use of the term storage controller to refer to Walrus, what contributes to the confusion.
Somebody also replied to my post providing a link to a discussion about the nature and purpose of Walrus and the Storage Controller. That helped to clarify the situation. However, I still think that Eucalyptus' documentation requires a lot of work if people are to start using the system.
Friday, April 9, 2010
On Cloud APIs...
Here is an interesting post from Eucalyptus discussing their viewpoint on and approach towards Cloud APIs.
Monday, March 29, 2010
Xen - Migration
Hello everybody...
Here I am again posting about our cloud updates...
Today we decided to install a new physical machine as node in our cloud... the Homer's wife: Marge. The idea is to have in the future Marge as our storage and cluster controller making homer only our cloud and walrus controller.
After the installation of Marge (CentOS 5.4) I realized that our LAN had no internet. That was because when we installed the 2nd NIC, on Homer, more configurations were necessary (IP Masquerade). Since it is not a big deal, we made the required changes and now everything is running well. You can check here a good link of how to forward the internet to your lan if you have a server running Linux with 2 NICs as your lan's router. We needed the internet in order to run the CentOS updates on Marge.
Well, let's talk now about interesting things... we decided to test the live migration feature of Xen using our two currently installed physical nodes: Marge and Maggie. At this point of time we are not going to worry about Eucalyptus... The idea is just testing the Xen's live migration between our physical nodes.
So, we used a previously created CentOS image to run on Xen. If you want to know how to create a VM to run on Xen, check this.
At this point, the VM was working fine, but completely isolated in one physical node: Maggie. So now, we decided to create a NFS directory on Homer and store the image there (VM file). In that way, we could have access to the image from any physical node "without" the need to copy. To enable the NFS we just followed the instructions found on this page. It worked fine since the very first time.
Now, after rebooting the physical machines, all of them mount at startup time a directory shared by Homer. This directory contains the VM image.
Almost done! Now, we had to configure Xen in order to migrate the VMs...
To do that, you have to edit the file /etc/xen/xend-config.sxp and make the following changes:
(xend-relocation-server yes)
(xend-relocation-port 8002)
(xend-relocation-address '')
(xend-relocation-hosts-allow '')
Of course, we restarted the Xen service in all physical nodes:
... and then started the VM in Maggie with the following command:
Finally, we migrated the VM from Maggie to Marge:
That's all for today. Until next post.
Lucio
Of course, we restarted the Xen service in all physical nodes:
/etc/init.d/xend restart
... and then started the VM in Maggie with the following command:
xm create our-vm
Finally, we migrated the VM from Maggie to Marge:
xm migrate -l our-vm target-host
That's all for today. Until next post.
Lucio
Tuesday, March 23, 2010
2nd NIC + DHCP
Here we are again, posting about our cloud adventures...
As the previous post mentioned, we decided to install another NIC in homer (our server)... and have all our nodes in a private lan. We got an old NIC card (1998) which at least is 100 Mbps... well, it really doesn't matter right now since our network switch is also pretty old (but it is 10/100 Mbps as well).
The hardest job in order to make the installation work was how to open the computer!? yeah..., unfortunately it had some kind of security lock on it... but after contacting our system lab technician (or whatever) then it was finally opened (he had the power! I mean, the tool).
After inserting the NIC, our DHCP was now configured to work giving IPs only to our private lan. In this case, our new NIC called as eth1 (by default) is providing IPs to our cloud, while the other eth0 is connected to the external world. Assuming that nobody (except us) needs access to the nodes, there is absolutely no problem having this topology.
The last modification was in the eucalyptus.conf in our server homer, in order to inform it which interface is public (external) and which one is private (internal). In our case, it is something like this:
VNET_PUBINTERFACE="eth0"
VNET_PRIVINTERFACE="eth1"
After restarting all the services in both server and node everything was working "sweet".
Ah, of course... I still have to close the server... I'll do it soon... (or not)
Until next post,
Lucio
Monday, March 22, 2010
An Image of Perfection
The time had finally come to actually create a virtual machine instance in the cloud. First we required an image to instantiate, and for this we turned to a set of pre-packaged example images provided by Eucalyptus. The images here provide only 64 bit operating systems, but without too much trouble we found a 32 bit CentOS image. This was then bundled using Eucalyptus tools and registered with Walrus (cloud storage).
EDITED by Lucio: Make sure to register all files (image + kernel + ramdisk). That was our first issue... we were not using the ramdisk and the machine was not starting because of that... after lots of coffee and mate we figured out why the machine was not starting...
The next step was to create an instance, using the euca-run-instances command. It goes without saying that this failed. Our instances remained forever trapped in the pending state, never to be started. The node controller log file was of no real use, providing an error message that led us to no solutions despite extensive googling. Our first instinct was to attempt to start the image manually on our node. We created the necessary Xen configuration file and successfully started the virtual machine, proving that the image itself was not the issue. After much poking and prodding, we determined that we were only specifying the kernel and file system images, but also had a ramdisk image available that we were not making use of. We added this to Walrus and included it in our call to euca-run-instances, and met with success. Well, sort of.
euca-run-instances -k mykey --kernel eki-907A1380 --ramdisk eri-AE9113E8 emi-3F0A1253
Our virtual machine had no IP address, and thus no way for us to connect to it. While it was comforting to know that it was there, happily doing nothing, we needed to be able to actually talk to it. We experimented with the different network configuration modes of Eucalyptus, that is, SYSTEM, STATIC, and MANAGED, with little success. We then disconnected our switch from the network and installed a DHCP server on homer. The Eucalyptus network configuration mode was set to SYSTEM, which means that Eucalyptus simply assigns a random MAC address to each VM and lets DHCP handle the rest. With this setup, we were successfully able to start a VM with an IP address and connect to it via SSH.
The problem will be converting this into something usable. At the moment, our cloud is in complete isolation from the rest of the world. We have two options: 1) we reconnect our cloud to the network and secure ourselves a range of IP addresses that we have control over and can assign via our DHCP server, or 2) we keep the cloud on its own network and add a second NIC card to our cloud controller (homer) for external access. All access to the cloud would then be routed through this machine. While option #1 seems like the ideal solution, option #2 is far more feasible at the moment, and hence this will be our route for the time being.
Until next time,
Mike
The next step was to create an instance, using the euca-run-instances command. It goes without saying that this failed. Our instances remained forever trapped in the pending state, never to be started. The node controller log file was of no real use, providing an error message that led us to no solutions despite extensive googling. Our first instinct was to attempt to start the image manually on our node. We created the necessary Xen configuration file and successfully started the virtual machine, proving that the image itself was not the issue. After much poking and prodding, we determined that we were only specifying the kernel and file system images, but also had a ramdisk image available that we were not making use of. We added this to Walrus and included it in our call to euca-run-instances, and met with success. Well, sort of.
euca-run-instances -k mykey --kernel eki-907A1380 --ramdisk eri-AE9113E8 emi-3F0A1253
Our virtual machine had no IP address, and thus no way for us to connect to it. While it was comforting to know that it was there, happily doing nothing, we needed to be able to actually talk to it. We experimented with the different network configuration modes of Eucalyptus, that is, SYSTEM, STATIC, and MANAGED, with little success. We then disconnected our switch from the network and installed a DHCP server on homer. The Eucalyptus network configuration mode was set to SYSTEM, which means that Eucalyptus simply assigns a random MAC address to each VM and lets DHCP handle the rest. With this setup, we were successfully able to start a VM with an IP address and connect to it via SSH.
The problem will be converting this into something usable. At the moment, our cloud is in complete isolation from the rest of the world. We have two options: 1) we reconnect our cloud to the network and secure ourselves a range of IP addresses that we have control over and can assign via our DHCP server, or 2) we keep the cloud on its own network and add a second NIC card to our cloud controller (homer) for external access. All access to the cloud would then be routed through this machine. While option #1 seems like the ideal solution, option #2 is far more feasible at the moment, and hence this will be our route for the time being.
Until next time,
Mike
Configuring Eucalyptus
Well, the Eucalyptus provides a default web interface to the users. In order to configure it properly, some steps in the cloud controller were required, as they call as First step setup. You can check those steps here.
This web interface is quite simple . To access it, point your browser to https://ip_front_end:8443. At the first time, the user and password were both admin (does it make you remember anything?). Here is a screenshot of our login screen and here another screenshot after logged.
At this time we were excited to see something happening... I mean... something interesting, of course...
Our next step will be related to the Hypervisor (Xen) configuration. It seems that the time to have some fun (maybe I should say try to have...) is coming. Stay tuned.
Saturday, March 20, 2010
Installing CentOS 5.4 and Eucalyptus 1.6.2
We first visited CentOS web site to get the installation media. From there we navigated to a nearby mirror and downloaded the 6 CDs required to install CentOS 5.4. A hat tip to the CS Club of Waterloo.
We installed CentOS in two machines: homer and maggie. Both installations included the packages Server and Virtualization (the latter provides the Xen hypervisor).
We then installed Eucalyptus in both machines following this tutorial. Up to this point, no major difficulties. homer is running the CLC, the CC, the SC and Walrus. maggie is running a NC.
Next step: check that the controllers can talk to each other and instantiate a few VMs. =)
We installed CentOS in two machines: homer and maggie. Both installations included the packages Server and Virtualization (the latter provides the Xen hypervisor).
We then installed Eucalyptus in both machines following this tutorial. Up to this point, no major difficulties. homer is running the CLC, the CC, the SC and Walrus. maggie is running a NC.
Next step: check that the controllers can talk to each other and instantiate a few VMs. =)
Friday, March 19, 2010
Second attempt
We decided to go with Debian, which was supposed to support Xen. However, users reported stability problems. In addition, it seems Debian is dropping support for Xen until they release Debian 6.0.
We then found a list of suggested distributions to use with Xen (check it out here) and considered using CentOS.
This time around we decided to check whether Eucalyptus would work in CentOS before downloading the distribution and installing it. Lady Luck smiled to us in this opportunity, so we started downloading CentOS!
Private Cloud, get prepared 'cause here we go!
We then found a list of suggested distributions to use with Xen (check it out here) and considered using CentOS.
This time around we decided to check whether Eucalyptus would work in CentOS before downloading the distribution and installing it. Lady Luck smiled to us in this opportunity, so we started downloading CentOS!
Private Cloud, get prepared 'cause here we go!
First attempt
Ubuntu 9.10 Server Edition provides support for Eucalyptus, so we decided to go with it. We installed all the nodes and the configuration of everything was automatic and looked like magic (translation: we had no idea what was happening).
Later on, we tried to start a VM, but the process failed giving the following error:
[EUCAERROR ] libvirt: internal error no supported architecture for os type 'hvm' (code=1)
[EUCAFATAL ] hypervisor failed to start domain
We did some research with the help of Saint Google and we found out that Ubuntu was working by default with KVM as hypervisor. This wouldn't be a problem... except by the little fact that KVM requires hardware support for virtualization and our host nodes' CPUs were born far before hardware support for virtualization existed.
The first solution that came to our minds was to switch KVM for the Xen hypervisor. But we encountered another problem. Ubuntu stopped supporting Xen a while ago and reports from Xen users suggest that it can only run in an unstable state.
First road block.
Later on, we tried to start a VM, but the process failed giving the following error:
[EUCAERROR ] libvirt: internal error no supported architecture for os type 'hvm' (code=1)
[EUCAFATAL ] hypervisor failed to start domain
We did some research with the help of Saint Google and we found out that Ubuntu was working by default with KVM as hypervisor. This wouldn't be a problem... except by the little fact that KVM requires hardware support for virtualization and our host nodes' CPUs were born far before hardware support for virtualization existed.
The first solution that came to our minds was to switch KVM for the Xen hypervisor. But we encountered another problem. Ubuntu stopped supporting Xen a while ago and reports from Xen users suggest that it can only run in an unstable state.
First road block.
Hardware specifications
Our cloud is going to be a small one. Really. Small. We have only five computers, so we are going to use four for the nodes and one for the cluster head and cloud head. The latter machine will probably host Walrus as well, once I understand what Walrus is supposed to do. I think Mike already knows, but he is not telling me.
The one machine for the cluster head has the following specifications:
- Intel(R) Pentium(R) D CPU 3.0GHz
- 1.5 GB RAM
- 120 GB hard disk
- 10/100Base-T Ethernet
- DVD-ROM drive
The machines for the nodes have the following specifications:
- Intel(R) Pentium(R) 4 CPU 2.80GHz
- 1 GB RAM
- 80 GB hard disk
- 10/100/1000Base-T Ethernet
- CD-ROM drive !!!!
Can you believe it? CD-ROM drives! These are probably the last machines around to be built with a CD-ROM drive. =P
In addition, the network connection is limited to 10/100, so the NICs in the host nodes will be a little wasted.
The one machine for the cluster head has the following specifications:
- Intel(R) Pentium(R) D CPU 3.0GHz
- 1.5 GB RAM
- 120 GB hard disk
- 10/100Base-T Ethernet
- DVD-ROM drive
The machines for the nodes have the following specifications:
- Intel(R) Pentium(R) 4 CPU 2.80GHz
- 1 GB RAM
- 80 GB hard disk
- 10/100/1000Base-T Ethernet
- CD-ROM drive !!!!
Can you believe it? CD-ROM drives! These are probably the last machines around to be built with a CD-ROM drive. =P
In addition, the network connection is limited to 10/100, so the NICs in the host nodes will be a little wasted.
Our plans
The whole idea is to use Eucalyptus as the software to power our private cloud. Eucalyptus is composed of Node Controllers (NC), Cluster Controllers (CC) and a Cloud Controller (CLC). There is also a Storage Controller called Walrus, but I'm not sure yet how that one is supposed to work.
Node Controllers are to be installed in host nodes running a hypervisor. The nodes will host virtual machines.
A Cluster Controller is installed in the head of a cluster to work as an intermediary between the Cloud Controller and the Node Controllers and probably to do some additional management of the host nodes.
The Cloud Controller provides the front-end that registered users can use to create virtual machines.
Walrus? Somebody please enlighten me. =)
EDIT:
It seems there is an additional controller associated to Walrus: the Storage Controller (SC).
Node Controllers are to be installed in host nodes running a hypervisor. The nodes will host virtual machines.
A Cluster Controller is installed in the head of a cluster to work as an intermediary between the Cloud Controller and the Node Controllers and probably to do some additional management of the host nodes.
The Cloud Controller provides the front-end that registered users can use to create virtual machines.
Walrus? Somebody please enlighten me. =)
EDIT:
It seems there is an additional controller associated to Walrus: the Storage Controller (SC).
EDITED by Lucio:
Here are links to the tools that we are planning to use to build our cloud:
Thursday, March 18, 2010
Everybody has a cloud! We want OUR cloud!
So the title says it all. We are up for building our own private cloud. It's not that we want to sell hosting services... mmm... we could... nah, let's keep it academic for now. The idea is to experience the challenges of building and managing a private cloud (and what a challenge it will be given the hardware we have!). Hopefully, we'll learn as well what features are missing, what is a cloud useful for, and how can we contribute to the evolution of cloud computing.
The sole purpose of this blog is to document our adventure. We'll post some info on the infrastructure we have available (tiny one as of now), what software stacks we try and the like.
Stay tuned.
EDIT:
Just to make it clear what we are talking about, here's a definition of private cloud by IBM:
Private Clouds - activities and functions are provided "as a service," over a company's intranet. It is built by an organization for its own users, and everything is delivered within the organization's firewall (instead of the Internet). The private cloud owner does not share resources with any other companies, so multitenancy is not an issue. Also called an "internal cloud."
The sole purpose of this blog is to document our adventure. We'll post some info on the infrastructure we have available (tiny one as of now), what software stacks we try and the like.
Stay tuned.
EDIT:
Just to make it clear what we are talking about, here's a definition of private cloud by IBM:
Private Clouds - activities and functions are provided "as a service," over a company's intranet. It is built by an organization for its own users, and everything is delivered within the organization's firewall (instead of the Internet). The private cloud owner does not share resources with any other companies, so multitenancy is not an issue. Also called an "internal cloud."
Subscribe to:
Comments (Atom)
Posts
