AWS : Starting udev: Unable to handle kernel paging request at virtual address..

I "baked" custom CentOS AMIs to use on the AWS 4 years ago with their module modules-2.6.16-ec2.tgz without any problems.

Last week, when we wanted to expand a volume, that a simple process:
- detach the volume from the instance
- create a snapshot of it
- create new volume from that snapshot
- attach that new volume to the instance

But when the instance started, it thrown out errors:
Starting udev: Unable to handle kernel paging request at virtual address..

I wondered there's some problems with the hardware, then tried to repeat several times, even I executed the fsck.ext3 to fix it if there's any hardware failure, but it didn't work although I could access that volume data by attaching it to another running instance..

So I decided to launch a new instance from a worked AMI to transfer the old instance data/configuration to it but unluckily it also wasn't started with the same the old errors.

Looked into the AWS forum, technicians recommended users to use pvgrub kernel instead because 2.6.16-xenU kernel is extremely old and does not receive any update... since 2011. That's odd because I still could do detach/attach volume a couple of months.. But it seems their suggestion is terminating the old non-working instance and launch a new one with pvgrub kernel.. That sounds like we will have to bake the AMI again, transfer data to it (although Puppet helps to provision for softwares and configurations..). But that will take time to do the same thing for bunch of current running instances with the old kernel module..

So finally I found a way to upgrade it to pvgrub kernel on existing non-boot volume with minimal impact.

Following is what I did (for someone has same problem):

1. detach the volume out of non-boot instance
2. attach the volume to a running instance as /dev/sdg for example.
3. login into that running instance and mount the new volume:
# mount /dev/sdg /mnt/sdg

4. install grub and kernel-xen for that device
# mkdir /mnt/sdg/sys/block
# yum -c /mnt/sdg/etc/yum.conf --installroot=/mnt/sdg -y install grub kernel-xen

5. check to see what installed vmlinuz and initrd version
# ls /mnt/sdg/boot

in my case they are vmlinuz-2.6.18-371.8.1.el5xen and initrd-2.6.18-371.8.1.el5xen.img

6. Recreate initial image
# mv /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img.org
# chroot /mnt/sdg mkinitrd /boot/initrd-2.6.18-371.8.1.el5xen.img 2.6.18-371.8.1.el5xen --preload=xenblk --preload=xennet --fstab=/etc/fstab

7. Install grub

# chroot /mnt/sdg grub-install /dev/sdg

8. Create the /mnt/sdg/boot/grub/menu.lst file with below content

default 0
timeout 5
title CentOS
root (hd0)
kernel /boot/vmlinuz-2.6.18-371.8.1.el5xen ro root=/dev/sda1
initrd /boot/initrd-2.6.18-371.8.1.el5xen.img

9. Unmount it
# umount /mnt/sdg

10. Detach that volume.

11. Create a snapshot from that volume.

12. Create an AMI from that snapshot with suitable kernel/Image ID from this
(in my case it's aki-f08f11c0)

13. Launch a new instance from that AMI

So we could recover existing installed softwares/configuration/data.

mongoose-paginate

When using mongoose-paginate 1.2.0 for pagination I got two following issues:

1. No paginate method on my Model class error

Regarding to the closed "No paginate method on my Model class" issue at #9, I found the problem that the mongoose module initializes new object:
module.exports = exports = new Mongoose; var mongoose = module.exports;so if we use :
var mongoose = require('mongoose'), paginate = require('mongoose-paginate');then the 1st mongoose variable is different from the mongoose variable with extend paginate method inside the mongoose-paginate module.

That why the error "No paginate method" was thrown out.

So I suggest we export the mongoose variable at the bottom of the mongoose-paginate.js:
module.exports = mongoose;

Then we use it as:
var mongoose = require('mongoose-paginate');

2. Mongoose also supports Population where we can get related documents within query - like join in RDBMS so I add it to mongoose-paginate. But I got an error:

"MissingSchemaError: Schema hasn't been registered for model "undefined"

I checked everything -  Schema, Model.. were correct and tested them well with the main Mongoose module (3.8.8) and finally it turned out that mongoose-paginate 1.2.0 is using Mongoose 3.5.1 so then upgraded to 3.8.8 and got it to work.

Using Python script to deploy Java J2EE apps

When deploy a J2EE app, we usually want to update configuration files such as web.xml, hibernate.cfg.xml, ehcache.xml.. We can use a bash script to do that but Python can do better since it provides XML parser.

To make the script for many environments, we will need a configuration file that contains specific environment's app information.
I prefer to make it in MS .INI file format. We will need some sections like environment, hibernate.cfg.xml, web.xml, ehcache.xml... such as:

 [environment]  

 tomcat-folder = /var/lib/tomcat6  
 tomcat-user = tomcat6  

 [hibernate.cfg.xml]  

 hibernate.connection.url = jdbc:postgresql://dbserver:5432/app  
 hibernate.connection.username = username
 hibernate.connection.password = password  

 [web.xml]  
 data1 = abc  
 Data2 = xyz  

To parse that file, we use ConfigParser:

 import sys, os, datetime, shutil, copy, ConfigParser
 from xml.etree import ElementTree as ET 
 app = sys.argv[1]  
 deployFolder = os.path.dirname(os.path.realpath(__file__)) + '/'  
 config = ConfigParser.ConfigParser()  
 config.optionxform = str #preserve case-sensitive  
 config_file = app + '.cfg'  
 config.read(deployFolder + config_file)  
 tomcatFolder = config.get('environment', 'tomcat-folder')  
 appFolder = tomcatFolder + '/webapps/' + app  

Then parse .xml configuration and update them

 #parse web.xml section into dictionary  
 webXMLdic = dict(config.items('web.xml'))  
 #parse the app web.xml  
 webXMLfile = appFolder + '/WEB-INF/web.xml'  
 webXML = ET.parse(webXMLfile)  
 for param in webXML.getroot().getiterator('context-param'):  
      name = param.find('param-name')  
      value = param.find('param-value')  
      if webXMLdic.has_key(name.text):  
           new_value = webXMLdic.get(name.text)  
           value.text = new_value  
      else:  
           print 'not found value for param ' + name.text + ' in ' + config_file  
 #update the app web.xml  
 with open(webXMLfile, 'w') as f:  
   f.write('<?xml version="1.0" encoding="UTF-8"?>\n' \  
           '<!DOCTYPE web-app \nPUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" \n' \  
     '"http://java.sun.com/dtd/web-app_2_3.dtd">\n')  
   ET.ElementTree(webXML.getroot()).write(f,'utf-8')  
 #parse hibernate.cfg.xml section into dictionary  
 hibernateXMLdic = dict(config.items('hibernate.cfg.xml'))  
 #parse the app hibernate.cfg.xml  
 hibernateXML = ET.parse(hibernateXMLfile)  
 hibernateXMLfile = appFolder + '/WEB-INF/classes/hibernate.cfg.xml'  
 #the XPath selector functionality was not implemented in ElementTree until version 1.3, which ships with Python 2.7, so I use iterator loop to work for lower Python versions  
 for property in hibernateXML.getroot().getiterator('property'):  
      name = property.get('name')  
      if hibernateXMLdic.has_key(name):  
           new_value = hibernateXMLdic.get(name)  
           property.text = new_value  
           print name, new_value  
      if (name == 'hbm2ddl.auto'): #remove update/create db if it's declared  
           hibernateXML.getroot().find('session-factory').remove(property)  
 #update the app hibernate.cfg.xml            
 with open(hibernateXMLfile, 'w') as f:  
   f.write('<!DOCTYPE hibernate-configuration PUBLIC \n' \  
      '"-//Hibernate/Hibernate Configuration DTD 3.0//EN" \n' \  
      '"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">\n')  
   ET.ElementTree(hibernateXML.getroot()).write(f)  

We will need other functions such as extract the war file, copy/backup..., but they are simple things to do with Python.

rsync for non-root user over SSH and preserve ownership/permissions/time..

If we need to archive files by rsync over ssh and preserve ownership/permissions/time..with a non-root account, we could do as follow steps:

1. Edit /etc/sudoers file (or exec visudo)
- Set NOPASSWD  for sync_user to execute the rsync command.
sync_user ALL= NOPASSWD:/usr/bin/rsync
- Skip tty requirement for sync_user (If we don't do this, we'll get the error: sudo: sorry, you must have a tty to run sudo)
Defaults:sync_user !requiretty

2. Then do rsync from local to remote with private key and rsync-path option:
rsync -avzH -e "ssh -i sync_user.pem" --rsync-path="sudo rsync" --delete  /local_dir/ sync_user@remote_host:/remote_dir

OpenConnect client for Cisco's AnyConnect SSL VPN

If we need to connect Cisco's AnyConnect SSL VPN from an AWS CentOS instance without using Cisco AnyConnect client, there's the OpenConnect as an alternative.

On CentOS 5.4, it's easy to install OpenConnect from enabled Red Hat Enterprise Linux / CentOS Linux Enable EPEL (Extra Packages for Enterprise Linux) repository.
If it is not, then install it:

rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

And then install openconnect: yum install openconnect

Then use it to connect VPN: openconnect -u auser cisco-vpn-server
But after enter the password, it throws an error: Failed to open tun device: No such device

Let's check the tun module:

[root@myhost ~]# modprobe tun
FATAL: Module tun not found.

[root@myhost ~]# cat /dev/net/tun
cat: /dev/net/tun: No such device

The module tun is /lib/modules/2.6.16-xenU/kernel/drivers/net/tun.ko. For some reason, if it's missed, so get it from http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz and unpack/copy entire folder lib/modules/2.6.16-xenU/kernel to /lib/modules/2.6.16-xenU

Then we need to reload modules:
[root@myhost ~]# depmod -ae 2.6.16-xenU

[root@myhost ~]# modprobe tun

Then connect VPN again, and it works: Connected tun0 as a.b.c.d, using SSL

Then just open another terminal and ssh to a server in the network.

Tynamo tapestry-security 0.5.1 and UNAUTHORIZED_URL bug

The Tynamo tapestry security 0.5.1 allows us to configure the URL path when unauthorized access happens. But it only works when we use annotation @RequiresRoles for each page, while it won't if we use Shiro createChain for global configuration, like

 configuration.add(factory.createChain("/admin/**").add(factory.roles(),User.Role.admin.name()).build());  

, the app displays:

HTTP ERROR 401
Problem accessing /admin/home. Reason:
Unauthorized
------------------------------------------------------------------------
/Powered by Jetty:///

instead of configured URL page.

The reason is the org.tynamo.security.shiro.authz.AuthorizationFilter.
getUnauthorizedUrl() overwrites the org.tynamo.security.shiro.AccessControlFilter.getUnauthorizedUrl() where the configured UNAUTHORIZED_URL passed in. The AuthorizationFilter.
getUnauthorizedUrl() returns value from class private variable unauthorizedUrl, so it's always null.

To fix it, just patch AuthorizationFilter by removing unauthorizedUrl, getUnauthorizedUrl(), setUnauthorizedUrl(.) as this.

Apache Shiro - Remember me and SerializationException: Unable to deserialze argument byte array

When implementing Tynamo tapestry security 0.5.1, I got exception when using Remember me

[WARN] mgt.DefaultSecurityManager Delegate RememberMeManager instance of type [$RememberMeManager_133e98f5ddc26344] threw an exception during getRememberedPrincipals().
org.apache.shiro.io.SerializationException: Unable to deserialze argument byte array.
at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:82)
at org.apache.shiro.mgt.AbstractRememberMeManager.deserialize(AbstractRememberMeManager.java:514)
at org.apache.shiro.mgt.AbstractRememberMeManager.convertBytesToPrincipals(AbstractRememberMeManager.java:431)

Caused by: java.io.StreamCorruptedException: invalid type code: EE
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1639)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1320)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at java.util.HashSet.readObject(HashSet.java:291)
...
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:77)

The Tynamo tapestry security 0.5.1 is based on Apache Shiro 1.2.0

Google around and found it was resolved for 1.2.0 release. That's interesting.

So I tried to debug the ClassResolvingObjectInputStream and ClassUtils and still didn't know the reason why the ClassUtils.forName(..) can't load byte array class. Finally google around for deserialize byte array exception and got JDK-6500212 : (cl) ClassLoader.loadClass() fails when byte array is passed resolved since 2007!

I'm running the app with 1.6.0.jdk on MacOS X so that might cause the problem.

So I patched the ClassUtils as below:

   public static Class forName(String fqcn) throws UnknownClassException {  
     .....  
     if (clazz == null) {//Luan - patch for byte array class (class name =[B - ref http://bugs.sun.com/view_bug.do?bug_id=6500212  
          try {  
                     clazz = Class.forName(fqcn,false,Thread.currentThread().getContextClassLoader());  
                } catch (ClassNotFoundException e) {  
                     if (log.isTraceEnabled()) {  
              log.trace("Unable to load class named [" + fqcn + "] from the current thread context.");  
            }  
                }  
     }  
     if (clazz == null) {  
       String msg = "Unable to load class named [" + fqcn + "] from the thread context, current, or " +  
           "system/application ClassLoaders. All heuristics have been exhausted. Class could not be found.";  
       throw new UnknownClassException(msg);  
     }  
     return clazz;  
   }  

and it worked!