AWS : Starting udev: Unable to handle kernel paging request at virtual address..

I "baked" custom CentOS AMIs to use on the AWS 4 years ago with their module modules-2.6.16-ec2.tgz without any problems.

Last week, when we wanted to expand a volume, that a simple process:
- detach the volume from the instance
- create a snapshot of it
- create new volume from that snapshot
- attach that new volume to the instance

But when the instance started, it thrown out errors:
Starting udev: Unable to handle kernel paging request at virtual address..

I wondered there's some problems with the hardware, then tried to repeat several times, even I executed the fsck.ext3 to fix it if there's any hardware failure, but it didn't work although I could access that volume data by attaching it to another running instance..

So I decided to launch a new instance from a worked AMI to transfer the old instance data/configuration to it but unluckily it also wasn't started with the same the old errors.

Looked into the AWS forum, technicians recommended users to use pvgrub kernel instead because 2.6.16-xenU kernel is extremely old and does not receive any update... since 2011. That's odd because I still could do detach/attach volume a couple of months.. But it seems their suggestion is terminating the old non-working instance and launch a new one with pvgrub kernel.. That sounds like we will have to bake the AMI again, transfer data to it (although Puppet helps to provision for softwares and configurations..). But that will take time to do the same thing for bunch of current running instances with the old kernel module..

So finally I found a way to upgrade it to pvgrub kernel on existing non-boot volume with minimal impact.

Following is what I did (for someone has same problem):

1. detach the volume out of non-boot instance
2. attach the volume to a running instance as /dev/sdg for example.
3. login into that running instance and mount the new volume:
# mount /dev/sdg /mnt/sdg

4. install grub and kernel-xen for that device
# mkdir /mnt/sdg/sys/block
# yum -c /mnt/sdg/etc/yum.conf --installroot=/mnt/sdg -y install grub kernel-xen

5. check to see what installed vmlinuz and initrd version
# ls /mnt/sdg/boot

in my case they are vmlinuz-2.6.18-371.8.1.el5xen and initrd-2.6.18-371.8.1.el5xen.img

6. Recreate initial image
# mv /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img.org
# chroot /mnt/sdg mkinitrd /boot/initrd-2.6.18-371.8.1.el5xen.img 2.6.18-371.8.1.el5xen --preload=xenblk --preload=xennet --fstab=/etc/fstab

7. Install grub

# chroot /mnt/sdg grub-install /dev/sdg

8. Create the /mnt/sdg/boot/grub/menu.lst file with below content

default 0
timeout 5
title CentOS
root (hd0)
kernel /boot/vmlinuz-2.6.18-371.8.1.el5xen ro root=/dev/sda1
initrd /boot/initrd-2.6.18-371.8.1.el5xen.img

9. Unmount it
# umount /mnt/sdg

10. Detach that volume.

11. Create a snapshot from that volume.

12. Create an AMI from that snapshot with suitable kernel/Image ID from this
(in my case it's aki-f08f11c0)

13. Launch a new instance from that AMI

So we could recover existing installed softwares/configuration/data.

mongoose-paginate

When using mongoose-paginate 1.2.0 for pagination I got two following issues:

1. No paginate method on my Model class error

Regarding to the closed "No paginate method on my Model class" issue at #9, I found the problem that the mongoose module initializes new object:
module.exports = exports = new Mongoose; var mongoose = module.exports;so if we use :
var mongoose = require('mongoose'), paginate = require('mongoose-paginate');then the 1st mongoose variable is different from the mongoose variable with extend paginate method inside the mongoose-paginate module.

That why the error "No paginate method" was thrown out.

So I suggest we export the mongoose variable at the bottom of the mongoose-paginate.js:
module.exports = mongoose;

Then we use it as:
var mongoose = require('mongoose-paginate');

2. Mongoose also supports Population where we can get related documents within query - like join in RDBMS so I add it to mongoose-paginate. But I got an error:

"MissingSchemaError: Schema hasn't been registered for model "undefined"

I checked everything -  Schema, Model.. were correct and tested them well with the main Mongoose module (3.8.8) and finally it turned out that mongoose-paginate 1.2.0 is using Mongoose 3.5.1 so then upgraded to 3.8.8 and got it to work.