Last week, when we wanted to expand a volume, that a simple process:
- detach the volume from the instance
- create a snapshot of it
- create new volume from that snapshot
- attach that new volume to the instance
But when the instance started, it thrown out errors:
Starting udev: Unable to handle kernel paging request at virtual address..
I wondered there's some problems with the hardware, then tried to repeat several times, even I executed the fsck.ext3 to fix it if there's any hardware failure, but it didn't work although I could access that volume data by attaching it to another running instance..
So I decided to launch a new instance from a worked AMI to transfer the old instance data/configuration to it but unluckily it also wasn't started with the same the old errors.
Looked into the AWS forum, technicians recommended users to use pvgrub kernel instead because 2.6.16-xenU kernel is extremely old and does not receive any update... since 2011. That's odd because I still could do detach/attach volume a couple of months.. But it seems their suggestion is terminating the old non-working instance and launch a new one with pvgrub kernel.. That sounds like we will have to bake the AMI again, transfer data to it (although Puppet helps to provision for softwares and configurations..). But that will take time to do the same thing for bunch of current running instances with the old kernel module..
So finally I found a way to upgrade it to pvgrub kernel on existing non-boot volume with minimal impact.
Following is what I did (for someone has same problem):
1. detach the volume out of non-boot instance
2. attach the volume to a running instance as /dev/sdg for example.
3. login into that running instance and mount the new volume:
# mount /dev/sdg /mnt/sdg
4. install grub and kernel-xen for that device
# mkdir /mnt/sdg/sys/block
# yum -c /mnt/sdg/etc/yum.conf --installroot=/mnt/sdg -y install grub kernel-xen
5. check to see what installed vmlinuz and initrd version
# ls /mnt/sdg/boot
in my case they are vmlinuz-2.6.18-371.8.1.el5xen and initrd-2.6.18-371.8.1.el5xen.img
6. Recreate initial image
# mv /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img.org
# chroot /mnt/sdg mkinitrd /boot/initrd-2.6.18-371.8.1.el5xen.img 2.6.18-371.8.1.el5xen --preload=xenblk --preload=xennet --fstab=/etc/fstab
kernel /boot/vmlinuz-2.6.18-371.8.1.el5xen ro root=/dev/sda1
9. Unmount it
# umount /mnt/sdg
10. Detach that volume.
11. Create a snapshot from that volume.
12. Create an AMI from that snapshot with suitable kernel/Image ID from this
(in my case it's aki-f08f11c0)
13. Launch a new instance from that AMI
So we could recover existing installed softwares/configuration/data.