Tuesday, February 11, 2014

Using Python script to deploy Java J2EE apps

When deploy a J2EE app, we usually want to update configuration files such as web.xml, hibernate.cfg.xml, ehcache.xml.. We can use a bash script to do that but Python can do better since it provides XML parser.

To make the script for many environments, we will need a configuration file that contains specific environment's app information.
I prefer to make it in MS .INI file format. We will need some sections like environment, hibernate.cfg.xml, web.xml, ehcache.xml... such as:

 [environment]  
 tomcat-folder = /var/lib/tomcat6  
 tomcat-user = tomcat6  
 [hibernate.cfg.xml]  
 hibernate.connection.url = jdbc:postgresql://dbserver:5432/app  
 hibernate.connection.username = username
 hibernate.connection.password = password  
 [web.xml]  
 data1 = abc  
 Data2 = xyz  
To parse that file, we use ConfigParser:
 import sys, os, datetime, shutil, copy, ConfigParser
 from xml.etree import ElementTree as ET 
 app = sys.argv[1]  
 deployFolder = os.path.dirname(os.path.realpath(__file__)) + '/'  
 config = ConfigParser.ConfigParser()  
 config.optionxform = str #preserve case-sensitive  
 config_file = app + '.cfg'  
 config.read(deployFolder + config_file)  
 tomcatFolder = config.get('environment', 'tomcat-folder')  
 appFolder = tomcatFolder + '/webapps/' + app  

Then parse .xml configuration and update them
 #parse web.xml section into dictionary  
 webXMLdic = dict(config.items('web.xml'))  
 #parse the app web.xml  
 webXMLfile = appFolder + '/WEB-INF/web.xml'  
 webXML = ET.parse(webXMLfile)  
 for param in webXML.getroot().getiterator('context-param'):  
      name = param.find('param-name')  
      value = param.find('param-value')  
      if webXMLdic.has_key(name.text):  
           new_value = webXMLdic.get(name.text)  
           value.text = new_value  
      else:  
           print 'not found value for param ' + name.text + ' in ' + config_file  
 #update the app web.xml  
 with open(webXMLfile, 'w') as f:  
   f.write('<?xml version="1.0" encoding="UTF-8"?>\n' \  
           '<!DOCTYPE web-app \nPUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" \n' \  
     '"http://java.sun.com/dtd/web-app_2_3.dtd">\n')  
   ET.ElementTree(webXML.getroot()).write(f,'utf-8')  
 #parse hibernate.cfg.xml section into dictionary  
 hibernateXMLdic = dict(config.items('hibernate.cfg.xml'))  
 #parse the app hibernate.cfg.xml  
 hibernateXML = ET.parse(hibernateXMLfile)  
 hibernateXMLfile = appFolder + '/WEB-INF/classes/hibernate.cfg.xml'  
 #the XPath selector functionality was not implemented in ElementTree until version 1.3, which ships with Python 2.7, so I use iterator loop to work for lower Python versions  
 for property in hibernateXML.getroot().getiterator('property'):  
      name = property.get('name')  
      if hibernateXMLdic.has_key(name):  
           new_value = hibernateXMLdic.get(name)  
           property.text = new_value  
           print name, new_value  
      if (name == 'hbm2ddl.auto'): #remove update/create db if it's declared  
           hibernateXML.getroot().find('session-factory').remove(property)  
 #update the app hibernate.cfg.xml            
 with open(hibernateXMLfile, 'w') as f:  
   f.write('<!DOCTYPE hibernate-configuration PUBLIC \n' \  
      '"-//Hibernate/Hibernate Configuration DTD 3.0//EN" \n' \  
      '"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">\n')  
   ET.ElementTree(hibernateXML.getroot()).write(f)  

We will need other functions such as extract the war file, copy/backup..., but they are simple things to do with Python.

Saturday, January 4, 2014

rsync for non-root user over SSH and preserve ownership/permissions/time..

If we need to archive files by rsync over ssh and preserve ownership/permissions/time..with a non-root account, we could do as follow steps:

1. Edit /etc/sudoers file (or exec visudo)
- Set NOPASSWD  for sync_user to execute the rsync command.
sync_user ALL= NOPASSWD:/usr/bin/rsync
- Skip tty requirement for sync_user (If we don't do this, we'll get the error: sudo: sorry, you must have a tty to run sudo)
Defaults:sync_user !requiretty

2. Then do rsync from local to remote with private key and rsync-path option:
rsync -avzH -e "ssh -i sync_user.pem" --rsync-path="sudo rsync" --delete  /local_dir/ sync_user@remote_host:/remote_dir

Tuesday, December 31, 2013

OpenConnect client for Cisco's AnyConnect SSL VPN

If we need to connect Cisco's AnyConnect SSL VPN from an AWS CentOS instance without using Cisco AnyConnect client, there's the OpenConnect as an alternative.

On CentOS 5.4, it's easy to install OpenConnect from enabled Red Hat Enterprise Linux / CentOS Linux Enable EPEL (Extra Packages for Enterprise Linux) repository.
If it is not, then install it:

rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

And then install openconnect: yum install openconnect
Then use it to connect VPN: openconnect -u auser cisco-vpn-server
But after enter the password, it throws an error: Failed to open tun device: No such device

Let's check the tun module:

[root@myhost ~]# modprobe tun
FATAL: Module tun not found.

[root@myhost ~]# cat /dev/net/tun
cat: /dev/net/tun: No such device

The module tun is /lib/modules/2.6.16-xenU/kernel/drivers/net/tun.ko. For some reason, if it's missed, so get it from http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz and unpack/copy entire folder lib/modules/2.6.16-xenU/kernel to /lib/modules/2.6.16-xenU

Then we need to reload modules:
[root@myhost ~]# depmod -ae 2.6.16-xenU
[root@myhost ~]# modprobe tun

Then connect VPN again, and it works: Connected tun0 as a.b.c.d, using SSL
Then just open another terminal and ssh to a server in the network.



Friday, December 13, 2013

Tynamo tapestry-security 0.5.1 and UNAUTHORIZED_URL bug

The Tynamo tapestry security 0.5.1 allows us to configure the URL path when unauthorized access happens. But it only works when we use annotation @RequiresRoles for each page, while it won't if we use Shiro createChain for global configuration, like
 configuration.add(factory.createChain("/admin/**").add(factory.roles(),User.Role.admin.name()).build());  
, the app displays:

HTTP ERROR 401
Problem accessing /admin/home. Reason:
Unauthorized
------------------------------------------------------------------------
/Powered by Jetty:///

instead of configured URL page.

The reason is the org.tynamo.security.shiro.authz.AuthorizationFilter.
getUnauthorizedUrl() overwrites the org.tynamo.security.shiro.AccessControlFilter.getUnauthorizedUrl() where the configured UNAUTHORIZED_URL passed in. The AuthorizationFilter.
getUnauthorizedUrl() returns value from class private variable unauthorizedUrl, so it's always null.

To fix it, just patch AuthorizationFilter by removing unauthorizedUrl, getUnauthorizedUrl(), setUnauthorizedUrl(.) as this.

Tuesday, December 10, 2013

Apache Shiro - Remember me and SerializationException: Unable to deserialze argument byte array

When implementing Tynamo tapestry security 0.5.1, I got exception when using Remember me

[WARN] mgt.DefaultSecurityManager Delegate RememberMeManager instance of type [$RememberMeManager_133e98f5ddc26344] threw an exception during getRememberedPrincipals().
org.apache.shiro.io.SerializationException: Unable to deserialze argument byte array.
at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:82)
at org.apache.shiro.mgt.AbstractRememberMeManager.deserialize(AbstractRememberMeManager.java:514)
at org.apache.shiro.mgt.AbstractRememberMeManager.convertBytesToPrincipals(AbstractRememberMeManager.java:431)

Caused by: java.io.StreamCorruptedException: invalid type code: EE
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1639)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1320)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at java.util.HashSet.readObject(HashSet.java:291)
...
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)
at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:77)

The Tynamo tapestry security 0.5.1 is based on Apache Shiro 1.2.0

Google around and found it was resolved for 1.2.0 release. That's interesting.

So I tried to debug the ClassResolvingObjectInputStream and ClassUtils and still didn't know the reason why the ClassUtils.forName(..) can't load byte array class. Finally google around for deserialize byte array exception and got JDK-6500212 : (cl) ClassLoader.loadClass() fails when byte array is passed resolved since 2007!

I'm running the app with 1.6.0.jdk on MacOS X so that might cause the problem.

So I patched the ClassUtils as below:

   public static Class forName(String fqcn) throws UnknownClassException {  
     .....  
     if (clazz == null) {//Luan - patch for byte array class (class name =[B - ref http://bugs.sun.com/view_bug.do?bug_id=6500212  
          try {  
                     clazz = Class.forName(fqcn,false,Thread.currentThread().getContextClassLoader());  
                } catch (ClassNotFoundException e) {  
                     if (log.isTraceEnabled()) {  
              log.trace("Unable to load class named [" + fqcn + "] from the current thread context.");  
            }  
                }  
     }  
     if (clazz == null) {  
       String msg = "Unable to load class named [" + fqcn + "] from the thread context, current, or " +  
           "system/application ClassLoaders. All heuristics have been exhausted. Class could not be found.";  
       throw new UnknownClassException(msg);  
     }  
     return clazz;  
   }  

and it worked!


Friday, November 22, 2013

Tapestry 5 https, Tomcat and load balancer plain HTTP proxy

We usually place Tapestry 5 apps running on Tomcat behind an Apache HTTPD with mod_jk / mod_proxy_ajp or mod_proxy_http.

Request protocols are plain HTTP or secure HTTPS from users to the Apache and same thing from Apache to Tomcat. Tapestry 5 auto-detects requests protocol and creates suitable URLs for further calling back actions for the application.

Below are 2 examples:
1. Browsers --http--> Apache --http--> Tomcat --Tapestry 5--http --> Apache --http--> Browsers ...
2. Browsers --https--> Apache --https--> Tomcat --Tapestry 5--https --> Apache --https--> Browsers ...

When we deploy an app required to access via https to load balancers connect to Tomcat in plain http proxy as:
3. Browsers --https--> Load balancers --http--> Tomcat--Tapestry 5 --http--> Load balancers --https --> Browsers ...

Then we get into the browser error “Blocked loading mixed active content” when loading an https page and mix some other http requests in background via AJAX within a page.

The reason is the Tomcat always got http requests from load balancers, so Tapestry 5 builds base URLs in http protocol.

Tapestry 5 allows us to override how these default URLs are created by the BaseURLSource service.

But that's not a good way to go with because we have to decide what protocol should be returned for each deploy environment in code...

Google around and found these posts:

Teaching Tapestry to use network path references
[T5]: BUG: Proxy Situation: Tapestry 5.3.3 Not Respecting isSecure for Form Action URL
Problem pushing application to production

The last one is very helpful when Kalle said about Tomcat connector configuration with proxyPort..

Study more about it and found a solution:

Tomcat http connector should be reconfigured as:
<Connector port="8080" proxyPort="443" secure="true" scheme="https"
protocol="HTTP/1.1" connectionTimeout="20000" URIEncoding="UTF-8" redirectPort="8443"/>

And http proxy in Apache/load balancer:
ProxyPreserveHost On
ProxyPass /T5app http://tomcat:8080/T5app
ProxyPassReverse /T5app http://tomcat:8080/T5app

So the Apache/load balancer handles https come from browsers, but makes http proxy to Tomcat and the Tomcat returns secure https/443 when Tapestry needs to build base URLs.

App web.xml
<context-param>
    <param-name>tapestry.production-mode</param-name>
    <param-value>true</param-value>
</context-param>
<context-param>
    <param-name>tapestry.secure-enabled</param-name>
    <param-value>false</param-value>
</context-param>










Saturday, November 16, 2013

Ehcache RMI Replicated Caching in cloud environments

The RMI Ehcache provides 2 ways for peer discovery: Automatic Peer Discovery and Manual Peer Discovery.

If the application is deployed in an environment where multicast is not allowed or not available like Amazon AWS or FireHost clouds, then we must use Manual Peer Discovery.

Follow the guide, configure Manual Peer Discovery as:

Configuration for server1
<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,
rmiUrls=//server2:40001/sampleCache11|//server2:40001/sampleCache12"/>

Configuration for server2

<cacheManagerPeerProviderFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"
properties="peerDiscovery=manual,
rmiUrls=//server1:40001/sampleCache11|//server1:40001/sampleCache12"/>

Configuring the CacheManagerPeerListener as:
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=server1, port=40001,
socketTimeoutMillis=2000"/>

<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=server2, port=40001,
socketTimeoutMillis=2000"/>

So we need to open the firewall for incoming TCP port 40001 on both peers.

But then each peer server can't send message to other with the connection timed out error:

RMIAsynchronousCacheReplicator Unable to send message to remote peer.  Message was: Connection refused to host: server2; nested exception is:
    java.net.ConnectException: Connection timed out
java.rmi.ConnectException: Connection refused to host: server2; nested exception is:
    java.net.ConnectException: Connection timed out
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
    at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
    at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.ConnectException: Connection timed out

Review the guide/google around... to find out the problem, but couldn't know why? Then tried to change the firewall to allow all ports and it worked!

So absolutely there was another port that was used by the Ehcache but blocked by the firewall.

Google around and found the answer here
"by default does use portmap and therefore random, non-root ports."

So we need to add remoteObjectPort to the cacheManagerPeerListenerFactory as:
<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=server1, port=40001, remoteObjectPort=40002,
socketTimeoutMillis=2000"/>

<cacheManagerPeerListenerFactory
class="net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"
properties="hostName=server2, port=40001, remoteObjectPort=40002,
socketTimeoutMillis=2000"/>

And close all ports on the firewall except TCP 40001, 40002 between those peers.

Problem is resolved.