How to install Ubuntu Server on a bcached rootfs

Experimental
You can read a little bit more about bcache there Bcache

Summary

With this how-to we will install ubuntu server on a bcached rootfs. This server setup has two hardware RAIDs and one SSD:

• One SSD 120GB
• One HW RAID1 465GiB
• One HW RAID10 3,8TiB

For this guide we will setup disks like this:

  SSD (/dev/sdc1 here):
   * All LVM2

  RAIDs:
   * RAID1  (/dev/sda1 here): 1GiB for /boot, 
   * RAID1  (/dev/sda2 here): the rest is for LVM2
   * RAID10 (/dev/sdb1 here): full LVM2

Then on LVM we will create those logical volumes:

   SSD:
    * 12GiB for swap (8GiB RAM)
    * 64GiB for cache
    * Rest is reserved for other purposes.

   RAID1:
    * One big partition for root (will get containers and vms, so need some space)
   RAID10:
    * One big partition for /srv (Will store data for user and services)

Installation Guide

Live DVD setup

For setting the bcache root, it is easier for now to start with a live dvd desktop and debootstrap our server installation.

Download a live Ubuntu 15.04 Desktop AMD64 iso (Ubuntu Download Page), then boot your server with it.

Select 'Try Ubuntu', then when you access the desktop, open a terminal (search for gnome-terminal).

   Optional:
   I use a apt-cache-ng-proxy locally, so
    # sudo su
    # echo ‘Acquire::http::Proxy “http://IP.OF.THE.PROXY:3142”;’ > /etc/apt/apt.conf.d/01Proxy

Now we will install the necessary tools for our setup:

    # apt-get update && apt-get install -y mdadm lvm2 bcache-tools

   Note: lvm2 must already be in the iso, and we install mdadm if you want to use a linux software RAID stack.
         mdadm setup is not included in this guide.

Now you need to setup your partition layout:

Disk Layout Setup

Creation of partition for LVM Physical Volumes, with sgdisk:

   Create one big partition on SSD:
   Remove any remaining filesystem, raid or partition-table signatures:
   # wipefs -af /dev/sdc
   Create a new GPT partition table on /dev/sdc ( for some rare cases )
   # sgdisk -Z /dev/sdc
   Create partition:
   # sgdisk -n 1 /dev/sdc
   Reload partition table
   # partprobe
   Verify disk setup:
   # sgdisk -p /dev/sdc

   Create two partitions on RAID1: First: 1GiB(-n 1::1G), Second the remaining (-n 2)
   # wipefs -af /dev/sda
   For my particular (old) server, I need to make MBR boot, so I add -m to convert gpt to mbr here):
   # sgdisk -Z -m /dev/sda
   Now use fdisk to create 2 partition, 1 bootable of 1G, and the second for the remaining:
   # echo -e 'o\nn\np\n1\n\n+1G\na\nn\np\n\n\n\nw\n' | fdisk /dev/sda
   # partprobe
   # fdisk -l /dev/sda

   Create one big patition on RAID10
   # wipefs -af /dev/sdb
   # sgdisk -Z /dev/sdb
   # sgdisk -n 1 /dev/sdb
   # partprobe
   # sgdisk -p /dev/sdb

• If your disks were not empty, there may be some errors telling you that the kernel still uses old partitions, you will need to reboot, and redo the steps to install lvm2, bcache-tools, and mdadm if needed. Then pursue from here, else pvcreate or other instructions may fail, especially, if you don't reboot before installing bcache, you will have to purge/clean all disks from bcache and re-create them with reboots in between.

Formating /boot partition (here /dev/sda1):

   # mkfs.ext4 /dev/sda1

Creating Physical Volumes for LVM:

   # pvcreate /dev/sda2
   # pvcreate /dev/sdb1
   # pvcreate /dev/sdc1

This is what it looks like after those commands (lsblk):

   # lsblk

LVM Setup

Now we will create the Volume Groups:

   # vgcreate ssd /dev/sdc1
   # vgcreate RAID1 /dev/sda2
   # vgcreate RAID10 /dev/sdb1

Now we will create the Logical Volumes:
Create swap space:

   # lvcreate -n swap -L 12G ssd

Create a cache space:

   # lvcreate -n cache -L 64G ssd

Free space on the SSD for other uses:

   # lvcreate -n free -l 100%FREE ssd

Create root Logical Volume:

   # lvcreate -n root -l 100%FREE RAID1

Create /srv Logical Volume:

   # lvcreate -n srv -l 100%FREE RAID10

Bcache Setup

Here we will have one cache for serving two backing devices. We could have created two caches and use a 1:1 cache:backing setup to be sure to reserve a certain amount of cache per backing device (i.e: create 2 lvm cache partition 12G/52G for example). In this setup the cache will be shared between both devices.

Bcache Creation

Then create the bcache devices for both RAIDs:

   For avoiding left-overs:
   # wipefs -af /dev/mapper/ssd-cache
   # wipefs -af /dev/mapper/RAID1-root
   # wipefs -af /dev/mapper/RAID10-srv
   For the same reason, we will add --wipe-bcache:
   # make-bcache --writeback --wipe-bcache -B /dev/mapper/RAID1-root -B /dev/mapper/RAID10-srv \
                                           -C /dev/mapper/ssd-cache --wipe-bcache

Notes: make-bcache is the command that will create the bcache, it can take options.
       --writeback: for performance, but in production you may not want to use this mode for safety reasons. 
       -B refers to the backing devices, here we will use one cache to cache two disks.
       -C refers to the caching device, multiple devices not yet supported; use mdadm as a workaround if needed.
       --wipe-bcache, will overwrite any previous bcache superblocks, thus destroying previous bcache data.

This is what lsblk must show you at this step:

   # lsblk

Format baches

I format both ext4 (had a recent issue with bcache + btrfs):

   # mkfs.ext4 /dev/bcache0
   # mkfs.ext4 /dev/bcache1

Partitioning, and disk setup is now done, we head on to server install:

Debootstrap Server

Prepare Debootstrap

We create a mount point for those:

   # mkdir -p /media/target
   # mount /dev/bcache0 /media/target

We install our ubuntu server files with debootstrap:

   # apt-get install -y debootstrap

I prepare env for the debootstrap, I want to use my proxy for this:

   # export https_proxy=http://IP.OF.THE.PROXY:3142
   # export http_proxy=http://IP.OF.THE.PROXY:3142
   # export ftp_proxy=http://IP.OF.THE.PROXY:3142

Debootstrap

   # debootstrap --arch amd64 vivid /media/target http://archive.ubuntu.com/ubuntu/
   Note: you may use a different mirror (I use http://mirrors.ircam.fr/pub/ubuntu/archive)

Chroot Setup

Then, prepare chroot:

   # cd /media/target

Copy the proxy and DNS setup for this new install:

   # cp /etc/apt/apt.conf.d/01Proxy /media/target/etc/apt/apt.conf.d/
   # cp /etc/resolv.conf /media/target/etc/resolv.conf

Mount the system mounts:

   # mount -t proc proc proc
   # mount -o bind /dev/ dev
   # mount -o bind /sys sys

Setup Installation

Chroot to the new installation

Chroot into target install root:

   # chroot .

Prepare Mounts

Use lsblk to identify your srv bcache number (it can be either 0 or 1 on your particular setup):

   # mount /dev/bcacheX /srv
   # mount /dev/sda1 /boot

Install Kernel

Now proceed with final setup of the installation (kernel + grub setup + packages + users):

   # apt-get update 

We will need to install kernel, of course; here we will use the generic one, but you can choose an other. To list available ones use:
‘apt-cache search linux-image’:

   # apt-get install linux-image-generic

   When grub ask you where to be installed, in this setup:
    - Select your booting device, not the /boot partition (i.e.: Select /dev/sda)

For a standard Ubuntu server experience:

   # apt-get install -y server^ standard^

=== Update Grub2 ===

On Vivid, there is a warning message for now, so we will comment HIDDEN_TIMEOUT_OPTIONS:<<BR>> {{{
   # vi /etc/default/grub to comment HIDDEN_TIMEOUT_OPTIONS
   # update-grub2

Install mandatory packages

We need to install lvm2 and bcache-tools (which will add udev hooks), we need to add mdadm in case we used a linux software RAID option (Those are really important else system won’t boot).

   # apt-get install -y lvm2 bcache-tools mdadm 

Setup fstab

Now we will need to get UUIDs for disks to be sure to mount the good one on the good mount-point. Bcache number is not a guaranteed one (bcache0 can become bcache1 at next boot). For this, we will use blkid tool:

   # blkid

and setup /etc/fstab: i.e.:

   # blkid | grep “/bcache” | cut -d‘“‘ -f2 > /etc/fstab
   # cat /proc/mounts >> /etc/fstab

Now edit properly:

   # vi /etc/fstab

File /etc/fstab example:

   UUID=ebf07bc7-e45d-4bc6-95d5-ca120cc6c135 / ext4 rw,relatime,data=ordered 0 0
   UUID=e118a9ab-539b-406d-af91-78888b945fb7 /srv ext4 rw,relatime,data=ordered 0 0 
   #/dev/sda1 is the boot partition
   UUID=”ed353670-703f-4cb1-8a2a-fee649ab97bf” /boot ext4 rw,relatime,data=ordered 0 0

Setup Networking

Now edit /etc/hostname (i.e. bcache_server.example.com):

   # vi /etc/hostname

Now edit /etc/hosts, to add the ‘bcache_server bcache_server.example.com’ in the 127.0.0.1 line:

   # vi /etc/hosts

Configure network interface (as your specific needs):

   # vi /etc/network/interfaces

File /etc/network/interfaces example:

   # iface eth0 auto
   # iface eth0 inet dhcp

Install openssh-server:

   # apt-get install -y openssh-server

Create a user

Setting-up users: create a user and root passwd:

   # adduser kick
   # addgroup kick admin
   # passwd

Reboot to your new system

exit chroot and reboot:

   # exit
   # sudo reboot

On you new system if you issue a mount, you must see:

   # mount

Now you must have a working fresh new server on a bcached rootfs, time to test lxd?