HotplugRaid
Differences between revisions 1 and 39 (spanning 38 versions)
|
Size: 8843
Comment: A generally usefull setup, How-To and a basic test case.
|
← Revision 39 as of 2013-01-05 21:44:43 ⇥
Size: 6121
Comment: data loss bug
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 3: | Line 3: |
| #title Hotplugable raid or mirror partitions | #title Raid with hotplugable (possibly external) devices |
| Line 5: | Line 5: |
| ||<tablestyle="float:right; font-size: 0.9em; width:30%; background:#F1F1ED; background-repeat: no-repeat; background-position: 98% 0.5ex; margin: 0 0 1em 1em; padding: 0.5em;">'''Contents'''[[BR]][[TableOfContents]]|| | * Block devices connected over the network (drbd or nbd devices) are not covered here. * Alternative aproaches include the syncing of regular (possibly network mounted) filesystems (unison, ChironFS) or replicating file systems (GlusterFS, OpenAFS, coda, intermezzo, ...). ||<tablestyle="float:right; font-size: 0.9em; width:30%; background:#F1F1ED; background-repeat: no-repeat; background-position: 98% 0.5ex; margin: 0 0 1em 1em; padding: 0.5em;"><<TableOfContents>>|| |
| Line 10: | Line 13: |
| This is a generally usefull setup, a How-To and a basic test case. | A replicating raid (redundant array of independent disks) that holds the user data is a generally useful setup. A hotplugable raid with one ore more external drives is especially usefull for home or office laptop users. This page provides a How-To and a basic test case. |
| Line 13: | Line 16: |
For now we distiguish between having the root filesystem (/) on a hotplugable raid and having the home directories (/home) on a hotplugable raid. Having the root filesystem on a hotplugable raid will require modules like usb-storage be loaded in initramfs. |
Warning: Hotplugging differnt hardware with device mapper on top of the raid can cause data loss! https://bugs.launchpad.net/ubuntu/+source/linux/+bug/320638 |
| Line 23: | Line 24: |
| Consider you are doing critical work on a laptop computer. At home, or when on AC, you plug in one (or possibly several) of your USB harddisks containing mirror partions. Upon insertion lights of those drives will stay lit during idle times for a while. The mirror partition is getting synced in the background automaticaly. As long as the drive stay atatched any work you save will be written to the laptops internal disk as well as to the attached mirror partitions. | Consider you are doing critical work on a laptop computer. At home, or when on AC, you plug in one (or possibly several) of your USB hard disks containing mirror partitions. Upon insertion the lights of those drives will stay lit during idle times for a little while. This is the mirror partition being synced in the background automatically. As long as those drives stay attached any work you save will be written simultaneously to the laptops internal disk, as well as to the attached mirror partitions. |
| Line 25: | Line 26: |
| If the internal disk fails, crashes down the stairway together with your laptop when you are on the road, or it drives away in that fake taxi drivers trunk, well hopefully you have at least one up-to-date mirror partition on that drive in your backpack or on the one you keep at home. | If the internal disk fails, crashes down the stairway together with your laptop when you are on the road, or it drives away in that fake taxi drivers trunk, well hopefully you have at least one up-to-date mirror partition on that drive in your backpack, or on the one you kept at home. |
| Line 31: | Line 32: |
| Failsafe installations with filesystems residing on multi-disk raid arrays that contain hotplugable devices like removable USB and Firewire drives and (external / bay-mounted) [S/P]-ATA drives. | It's an installation with a filesystem residing on a multi-disk (md) raid array that contains hot-plugable devices like removable USB and Firewire drives or (possibly external / bay-mounted) (S)ATA harddrives. |
| Line 33: | Line 34: |
| As long as appropriate hotplug udev rules do not exist yet, you can mount a degraded md device with your backup mirror partition manually to access your data on another machine. | It is possible to have the entire root file system (/) on a hot-plugable raid or as in the example installation below just the home directories (/home). |
| Line 35: | Line 36: |
| The neccessary steps would be somethig like this: | Beginning with ubuntu 9.10 the arrays are properly assembled using udev rules and the --incremental option. Recovery of temporarily disconected (external) drives should now work out of the box. |
| Line 37: | Line 38: |
| {{{ mdadm --assemble --run /dev/md8 /dev/sdx5 # Assemble and run the device (in this case sdx2) as degraded raid (in this case md5 is an unused device name). cryptsetup luksOpen /dev/md5 my-rescued-mirror # Open the encrypted device using its luks header. mount /dev/mapper/my-rescued-mirror /mnt # Finally mount your data. }}} |
|
| Line 48: | Line 46: |
| 1. Attatch the external drives that you want to hold raid partitions. | See: http://testcases.qa.ubuntu.com/Install/AlternateRaidMirror and adopt the partitioning to your needs. |
| Line 50: | Line 48: |
| 1. Beginn the install process by booting from the alternate-cd. | * Having the root file system and swap on one encrypted lvm volume-group and /home on a second encrypted raid device, means you have to enter passphrases twice on each boot and resume. With fast devices like external SATA Drives you can include root swap and home into one volume group on one luks on raid mirror, and use a separate raid mirror only for /boot. |
| Line 52: | Line 50: |
| 1. Choose manual partioning. (The basic process is described more elaborately at https://help.ubuntu.com/community/Installation/SoftwareRAID) Since this setup and test case assumes that the installation is done on a laptop machine the system will also be set up using encrypted devices. (The encryption steps can be left if encryption is not desired.) First create a rather small (primary) partition that will hold the /boot files. (500M is more than needed.) Configure it to be a ext2 filesystem and be mounted as /boot. Then create a (logical) partion that will hold the rootfs and swap partion on the internal drive. Configure it to be used for dm-crypt with luks header. Enter the encryption setup to set the passphrase for it. Then configure the resulting _crypt device to be used as a physical volume for lvm. Enter the lvm setup to create a volume group (in this case vg_tsthost-local) and the logical volumes for the rootfs and swap (lv_root, lv_swap). Back in the partitioning tool configure lv_root to be mounted as / and lv_swap to be used as swap. Now setup equally sized (locical) partitions on the internal drive and each external drives as physical raid partitions that will hold the home directories. Choose to configure raid (multi-disks) from the menu and create a raid 1 (mirroring) device from the partitions you just created. (md0 will be assumed here.) The multi-disk raid partitions should start syncing in the background. Go on to create a partition on the md0 device and configure it to be used for dm-crypt with luks header. Enter the encryption setup to set the passphrase for it. Finally create an ext3 partion on the md0_crypt device and configure it to be mounted as /home. (Note that unlike it is the case with the first _crypt device there is no lvm used on top of the second _crypt device. If one wants it to hold multiple logical volumes it is of course possible to create a lvm volume group (vg_tsthost-raid) containing logical volumes (lv_home, lv_svn, ...) in between.) The partitioning of this example can therfore be summarized as: {{{ ext2 (/boot) first_crypt -> lvm [ext3 (/) and swap] [multi disk member partitions] -> md0 -> second_crypt -> ext3 (/home) }}} (During subsequent testing or when the disks already contain some lvm, raid or luks partitions you may encouter problems with the installer not being able to delete or use existing devices (with data) or inadvertently scraping previous setups. Please test this and file/report on appropriate bugs.) * Crypttab Bug The installer would not put the second md0_crypt device into the /etc/crypttab, so it won't get opened during the boot process. As a workaround do that manually while the installer is copying away. Change to another console and have a look: "cat /target/etc/crypttab". If md0_crypt is missing check "ls -l /dev/disks/by-uuid" to determine the uuid that points to the md0 device. Then enter "echo md0_crypt /dev/drives/by-uuid/<use_tab_completion_to_get_the_uuid_right> none luks >> /target/etc/crypttab" to add the necessary line to the crypttab. * Many mdadm boot script racing udev events bugs https://bugs.launchpad.net/ubuntu/+source/mdadm/+bugs The boot scripts try to assemble raid arrays before the partitions become available. Symptom: File system checks or opening dm-crypt devices fails because the raid device is not activated and can not be recognized. You are droped to a console. As long as the problem is not worked out (see also next issue) you may hack around the bug by introducing a "sleep 10" before do_open in the start section of /etc/cryptsetup-early. * When and what to do with degraded arrays during boot bug When you detatch a removable device with a raid member partiion on it when the machine is powered down the array won't come up during boot because it is not complete. Again you are droped to a console. A solution would be to rely on udev event rules also during boot: If all the md devices specified in /etc/mdadm/mdadm.conf or persistent superblocks have come up, do resume booting imediately. If after 30 sec there are still some raid partitions missing, run these arrays in degreaded mode and issue notifications. In the meantime you have to manualy start the array (mdadm --assemble --scan) and subsequently stop it again (mdadm --stop /dev/md0) on the console in order to have its status properly updated to degraded. (mdadm fails to stop RAID on shutdown Bug #111398) Exit from the console and reboot. https://wiki.ubuntu.com/BootDegradedRaid 1. Let the sync start when a raid member partition is attatched. "mdadm --add /dev/md0 $dev" will re-add the partion to the raid and start the syncing. But in order to get this automatically an udev rule is nessesary: {{{ ............................ }}} |
* multi-sync-bitmaps: If you want to have multiple external members and use them connected one at a time (rolling backup style) you may like to have a separate bitmap for each member! For this you need to stack md arrays of two members. The first array (i.e. md0) then consists of an internal member and one external member. The second array (i.e. md1) consists of md0 and the second external member. |
| Line 114: | Line 55: |
| Having the root filesystem and swap on one encrypted lvm volume-group and /home on a second encrypted raid device, means you have to enter passphrases twice on each boot and resume. | * Maybe enabling the mdadm options --write-mostly --write-behind for external drives. |
| Line 116: | Line 57: |
| With fast devices like external SATA Drives it may be bearable to include the root filesystem and swap partition into the raid mirror. You may even keep your /boot partition on a raid and have it mirrored to external drives. Maybe including a read only medium as a prefered mirror, showing discrepancies if the internal drive has been tempered with. Maybe even allowing you to boot from that external drive, possibly resyncing the internal /boot partition and master boot records (MBR). | * Maybe including a read only medium as a preferred mirror, showing discrepancies if the internal drive has been tempered with. Maybe even allowing you to boot from that external drive, possibly re-syncing the internal /boot partition and master boot records (MBR). |
| Line 118: | Line 59: |
| All this will require rootfs on HotplugRaid to work. | === Troubleshooting === 1. Preexisting Superblocks: If during subsequent install attempts or when the disks already contain some lvm, raid or luks partitions you encounter problems with the installer not being able to delete or use preexisting devices (with data) or inadvertently scraping previous setups. If you encounter this please file/report on appropriate bugs. (Workaround may be to delete existing superblocks: mdadm --zero-superblock)) 1. Booting with a degraded array (see ReliableRaid). When you detach a removable device with a raid member partition from the machine when it is powered down, the degraded array may not come up automatically during boot. Slight chance with 9.10: Do a "dpkg-reconfigure mdadm" after the install and set boot degraded to yes again. (Bug:462258) To start the array manually if you are droped to a console: {{{ mdadm --incremental --run /dev/sde5 # Assemble and run the device (in this case sde5) as a degraded raid. cryptsetup luksOpen /dev/mdX my-degraded-mirror # Open the encrypted md device using its luks header. mount /dev/mapper/my-degraded-mirror /mountpoint # Finally mount your data. }}} |
| Line 122: | Line 79: |
| == Root filesystem on HotplugRaid == | 1. Resyncing reconnected drives (readding raid member partitions to its array) when they are re-attached (see ReliableRaid). |
| Line 124: | Line 81: |
| 1. That early in the boot process we need to wait only for the raid device with the root filesystem to come up. And try to resume with an degraded array if this times out. | If a reconnected partitions that got marked faulty still needs to be re-added manually: "mdadm --add /dev/md0 $dev" will manually re-add the partition to the raid and start the syncing. A custom udev rule may be similar to this one: |
| Line 126: | Line 84: |
| 1. If the root filesystem resides on USB/Firewire/eSATA/PCMCIA/PC-Card/ExpressCard devices those drivers need to be present in initramfs and loaded prior to starting the timeout. 1. Present boot script racing bugs To workaround bugs here, one may need to edit /usr/share/initramfs-tools/init and put sleep 10 after line log_begin_msg "Mounting root file system...". After this update initramfs with "sudo update-initramfs -k all -u". |
{{{ ACTION=="add", BUS=="usb", SUBSYSTEM=="block", DEVPATH=="*[0-9]" SYSFS{serial}=="5B6B1B916C31", RUN+="mdadm --add /dev/md0 $DEVNAME" }}} * But the above rule actually identifies the USB device by its serial number and not the raid partition by superblock/uuid (which you may have moved to another device). |
| Line 134: | Line 91: |
= References = https://help.ubuntu.com/community/Installation/SoftwareRAID https://features.launchpad.net/distros/ubuntu/+spec/udev-mdadm https://blueprints.launchpad.net/ubuntu/+spec/udev-mdadm https://blueprints.edge.launchpad.net/ubuntu/+spec/boot-degraded-raid https://blueprints.launchpad.net/ubuntu/+spec/udev-lvm-mdadm-evms-gutsy |
|
| Line 135: | Line 105: |
| CategoryBootAndPartition | CategoryBootAndPartition CategoryUsb CategoryRecovery |
- Block devices connected over the network (drbd or nbd devices) are not covered here.
- Alternative aproaches include the syncing of regular (possibly network mounted) filesystems (unison, ChironFS) or replicating file systems (GlusterFS, OpenAFS, coda, intermezzo, ...).
HotplugRaid
A replicating raid (redundant array of independent disks) that holds the user data is a generally useful setup. A hotplugable raid with one ore more external drives is especially usefull for home or office laptop users. This page provides a How-To and a basic test case.
Warning: Hotplugging differnt hardware with device mapper on top of the raid can cause data loss! https://bugs.launchpad.net/ubuntu/+source/linux/+bug/320638
Home directories on HotplugRaid
Layman's Description:
Consider you are doing critical work on a laptop computer. At home, or when on AC, you plug in one (or possibly several) of your USB hard disks containing mirror partitions. Upon insertion the lights of those drives will stay lit during idle times for a little while. This is the mirror partition being synced in the background automatically. As long as those drives stay attached any work you save will be written simultaneously to the laptops internal disk, as well as to the attached mirror partitions.
If the internal disk fails, crashes down the stairway together with your laptop when you are on the road, or it drives away in that fake taxi drivers trunk, well hopefully you have at least one up-to-date mirror partition on that drive in your backpack, or on the one you kept at home.
Technical Description:
It's an installation with a filesystem residing on a multi-disk (md) raid array that contains hot-plugable devices like removable USB and Firewire drives or (possibly external / bay-mounted) (S)ATA harddrives.
It is possible to have the entire root file system (/) on a hot-plugable raid or as in the example installation below just the home directories (/home).
Beginning with ubuntu 9.10 the arrays are properly assembled using udev rules and the --incremental option. Recovery of temporarily disconected (external) drives should now work out of the box.
Installation Instructions
See: http://testcases.qa.ubuntu.com/Install/AlternateRaidMirror and adopt the partitioning to your needs.
- Having the root file system and swap on one encrypted lvm volume-group and /home on a second encrypted raid device, means you have to enter passphrases twice on each boot and resume. With fast devices like external SATA Drives you can include root swap and home into one volume group on one luks on raid mirror, and use a separate raid mirror only for /boot.
- multi-sync-bitmaps: If you want to have multiple external members and use them connected one at a time (rolling backup style) you may like to have a separate bitmap for each member! For this you need to stack md arrays of two members. The first array (i.e. md0) then consists of an internal member and one external member. The second array (i.e. md1) consists of md0 and the second external member.
Further Improvements
- Maybe enabling the mdadm options --write-mostly --write-behind for external drives.
- Maybe including a read only medium as a preferred mirror, showing discrepancies if the internal drive has been tempered with. Maybe even allowing you to boot from that external drive, possibly re-syncing the internal /boot partition and master boot records (MBR).
Troubleshooting
- Preexisting Superblocks: If during subsequent install attempts or when the disks already contain some lvm, raid or luks partitions you encounter problems with the installer not being able to delete or use preexisting devices (with data) or inadvertently scraping previous setups. If you encounter this please file/report on appropriate bugs. (Workaround may be to delete existing superblocks: mdadm --zero-superblock))
Booting with a degraded array (see ReliableRaid).
- When you detach a removable device with a raid member partition from the machine when it is powered down, the degraded array may not come up automatically during boot.
Slight chance with 9.10: Do a "dpkg-reconfigure mdadm" after the install and set boot degraded to yes again. (462258) To start the array manually if you are droped to a console:
mdadm --incremental --run /dev/sde5 # Assemble and run the device (in this case sde5) as a degraded raid. cryptsetup luksOpen /dev/mdX my-degraded-mirror # Open the encrypted md device using its luks header. mount /dev/mapper/my-degraded-mirror /mountpoint # Finally mount your data.
- When you detach a removable device with a raid member partition from the machine when it is powered down, the degraded array may not come up automatically during boot.
Resyncing reconnected drives (readding raid member partitions to its array) when they are re-attached (see ReliableRaid).
- If a reconnected partitions that got marked faulty still needs to be re-added manually: "mdadm --add /dev/md0 $dev" will manually re-add the partition to the raid and start the syncing. A custom udev rule may be similar to this one:
ACTION=="add", BUS=="usb", SUBSYSTEM=="block", DEVPATH=="*[0-9]" SYSFS{serial}=="5B6B1B916C31", RUN+="mdadm --add /dev/md0 $DEVNAME" - But the above rule actually identifies the USB device by its serial number and not the raid partition by superblock/uuid (which you may have moved to another device).
- If a reconnected partitions that got marked faulty still needs to be re-added manually: "mdadm --add /dev/md0 $dev" will manually re-add the partition to the raid and start the syncing. A custom udev rule may be similar to this one:
References
https://help.ubuntu.com/community/Installation/SoftwareRAID https://features.launchpad.net/distros/ubuntu/+spec/udev-mdadm https://blueprints.launchpad.net/ubuntu/+spec/udev-mdadm https://blueprints.edge.launchpad.net/ubuntu/+spec/boot-degraded-raid https://blueprints.launchpad.net/ubuntu/+spec/udev-lvm-mdadm-evms-gutsy
HotplugRaid (last edited 2013-01-05 21:44:43 by 77-22-90-94-dynip)