Showing posts with label iso. Show all posts
Showing posts with label iso. Show all posts

Thursday, March 22, 2018

Thoughts on creating a large iso using dd

Let's see if I can avoid being long winded as usual:

So I wanted to create a .iso file that was going to be treated like a disk (more of that part in a future thread). Nothing better to do the deed but that old friend, dd:

raub@desktop:/export/scratch$ dd if=/dev/zero of=olddrive.iso bs=44G count=1
dd: memory exhausted by input buffer of size 47244640256 bytes (44 GiB)
raub@desktop:/export/scratch$

It so happens that dd is trying to create a 44G file by first allocating that in the RAM. And that will not work for me as I only have 32G total of memory.

The solution is rather obvious if you stop to think about it (which, I must admit, I did not): just don't use all the RAM! Instead, use smaller chunks and repeat that a lot. How about 1G chunks 44 times?

raub@desktop:/export/scratch$ dd if=/dev/zero of=olddrive.iso bs=1G count=44
44+0 records in
44+0 records out
47244640256 bytes (47 GB, 44 GiB) copied, 354.018 s, 133 MB/s
raub@desktop:/export/scratch$

Much better!

Saturday, January 30, 2016

Mounting user fileshare on boot2docker boot

For docker container development, and light using, I found boot2docker to be quite convenient. I have it to boot off its ISO and then mount a permanent drive for the containers (and to store config files).

/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
Whenever there is a new version, I shut the vm down, swap the ISOs (really just point the alias to the new file), and reboot. Nice and brainless.

I do not like to use the default account, docker, to do container development and running. Also, in a nice production environment you want to have other users running their containers. So, I created a user called ducker, which is a quick play on the default username. I also would prefer not having the user homedir in the drive where the containers are, which has been suggested before. You see, the way I see containers they are by design not important; blow them up if you feel like or wonder if they have been compromised. What matters is the data and the dockerfile required to rebuild the container. As a result, ducker has an account in the fileserver, which does its RAID and backup thingie as any good fileserver should. Now, if we want to have containers created and running from ducker's account when the server boots up, we need to have said account available.

So, the plan is that whenever the boot2docker server reboots, ducker will be there. And we then have a few issues to deal with. First, docker2file's ISO can't do automount. And second, usually when we shut down any user is lost because we are running of an ISO.

boot2docker's ISO is built so that we can provide some permanent stuff, which is why it mounts /dev/sda. But, that is not the only place we can mount things, nor the only way. In /opt/bootscript.sh we have the following interesting lines:

# Allow local bootsync.sh customisation
if [ -e /var/lib/boot2docker/bootsync.sh ]; then
    /bin/sh /var/lib/boot2docker/bootsync.sh
    echo "------------------- ran /var/lib/boot2docker/bootsync.sh"
fi

# Launch Docker
/etc/rc.d/docker

# Allow local HD customisation
if [ -e /var/lib/boot2docker/bootlocal.sh ]; then
    /bin/sh /var/lib/boot2docker/bootlocal.sh > /var/log/bootlocal.log 2>&1 &
    echo "------------------- ran /var/lib/boot2docker/bootlocal.sh"
fi

The files in /var/lib/boot2docker are in /dev/sda. Don't know which one to pick, but the second one does have a comforting Allow local HD customization message. So I will pick /var/lib/boot2docker/bootlocal.sh and add the following lines:

# Create local user, also creating the homedir
adduser -D -u 1003 ducker
# add user to docker group
adduser ducker docker
# Mount homedir
mount.nfs  fileserver.example.com:/export/home/ducker /home/ducker

After we create the file and reboot, we get

docker@boot2docker:~$ id ducker
uid=1003(ducker) gid=1003(ducker) groups=1003(ducker),100(docker)
docker@boot2docker:~$ df -h
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.6M    123.8M    772.8M  14% /
tmpfs                   498.1M         0    498.1M   0% /dev/shm
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda
cgroup                  498.1M         0    498.1M   0% /sys/fs/cgroup
df: /mnt/hgfs: Protocol error
fileserver.example.com:/home/ducker
                        295.3G    285.1G     10.0G  97% /home/ducker
/dev/sda                 19.6G     12.7G      5.9G  68% /mnt/sda/var/lib/docker/aufs
docker@boot2docker:~$

User ducker can login because there is a public ssh key already in place. However, we did not do anything for that user's password, but there are a few ways to take care of that such as using /var/lib/boot2docker/bootlocal.sh to copy the hash into /etc/passwd. And, you really want to take care of that otherwise you will not be able to login.

Monday, April 28, 2014

Booting a ESXi VM from a .iso in a NFS share

First time I created a VM in Vmware's ESXi, I placed the .iso containing the install image of the operating system in the host I run vsphere client from. And then told the VM where to get that. Some info on this procedure can be fount at http://community.spiceworks.com/topic/332810-create-vm-from-iso-without-uploading-to-datastore. At first glance, it seemed to be quick and easy; I would even add for a small deployment it is quite convenient. However, but it was just cumbersome (too many hoops to make that work) and slow (it expects the vsphere box to be in a fast and reliable connection to the ESXi one, which might not be the case). It really makes sense to have the images either in the server itself or mounted on the server (fileshare). I was not looking forward to having the images in the server itself because:

  • It would probably require me to download them somewhere else and then upload to the server. That does not sound like a dealbreaker to most since if I can ssh into my ESXi, which I do, I can scp the files into it. Well, I do not believe on having a single device/program/app/thingie that does it all; you see, I do like the Rule of Simplicity from the Unix Philosophy. And that dovetails to the next item:
  • I have a perfectly good fileserver (ok, a Synology NAS box) thank you verymuch. Many of my VMs already NFS mount shares from it, or just have their entire disk in an iSCSI LUN from said NAS.
  • I might want to use those images with my other vm host, which runs KVM.

What I want is maybe a NFS share that can be mounted somewhere where I can download the .iso files to and then have it available read-only to the vm hosts (ESXi, KVM/libvirt, Vbox). Let's see if we can make that happen, shall we?

Setup

We first need to begin with the fileshare itself. It is being exported read-only as a NFSv4 fileshare from the NAS; we won't go over how to do that in this discussion. Since I could not find showmount or even mount in ESXi, let's assume I know what I am doing and believe the share I want is fileserver.example.com:/public (I cheated and verified in a Linux box). We can add that to vmhost2 using the vsphere client:

Using the ISO

So now we did all this boring work, let's see if we can boot using a, well, boot .iso from the NFS share. So, On the vsphere client

  • Select the vmhost. In my case that is vmhost2.
    1. Select Configuration->Storage in the Hardware panel.
    2. Click Datastores and click Add Storage.

    3. Select Network File System and click Next.


    4. Enter

      • server name: fileserver.example.com
      • mount point folder name: /export/public (yep NFS3)
      • [x] Mount NFS read only
      • datastore name: public

    And that should result in a new datastore entry called public.
  • Select the vm client, which is called devcentos.
    1. select the vm in question, devcentos:
    2. Edit virtual machine properties->CD/DVD->Device Type->Datastore ISO File
    3. Hit Browse
    4. Datastores->public->ISOs->CentOS.iso
    5. Boot to BIOS. For some reason I have to manually select which device to boot. Even though most of the time the virtual hard drive is completely virgin, the bios does not failover to the ISO, Maybe that has been solved by now, but just learn this step... just in case.
    6. Turn CD player on on boot

References

https://communities.vmware.com/thread/456682