tag:blogger.com,1999:blog-60877940198295056012024-03-13T16:05:37.504-07:00Heshan's Bloglog.info("Heshan Suriyaarachchi's Thread Dump");Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.comBlogger169125tag:blogger.com,1999:blog-6087794019829505601.post-24837362466465596302015-03-18T10:33:00.000-07:002015-03-18T10:33:28.998-07:00Fixing Error: Could not find a suitable provider in Puppet<div dir="ltr" style="text-align: left;" trbidi="on">
I'm quite new to Puppet and I had a Puppet script which configures a MySQL database working fine on a Puppet learning VM on VirtualBox. This issue happened when I installed a and setup Puppet on a server of my own. I kept seeing the following error and it was driving me crazy for some time.<br />
<br />
<pre class="brush:js">[hesxxxxxxx@xxxxxxpocx ~]$ sudo puppet apply --verbose --noop /etc/puppet/manifests/site.pp
Info: Loading facts
Info: Loading facts
Info: Loading facts
Warning: Config file /etc/puppet/hiera.yaml not found, using Hiera defaults
Notice: Compiled catalog for xxxxxxxxxxxxxxxxxxx in environment production in 0.99 seconds
Warning: The package type's allow_virtual parameter will be changing its default value from false to true in a future release. If you do not want to allow virtual packages, please explicitly set allow_virtual to false.
(at /usr/lib/ruby/site_ruby/1.8/puppet/type/package.rb:430:in `default')
Info: Applying configuration version '1426611197'
Notice: /Stage[main]/Mysql::Server::Install/Package[mysql-server]/ensure: current_value absent, should be present (noop)
Notice: /Stage[main]/Mysql::Server::Install/Exec[mysql_install_db]/returns: current_value notrun, should be 0 (noop)
Notice: Class[Mysql::Server::Install]: Would have triggered 'refresh' from 2 events
Notice: /Stage[main]/Mysql::Server::Config/File[mysql-config-file]/content: current_value {md5}8ace886bbe7e274448bc8bea16d3ead6, should be {md5}d0d209eb5ed544658b3f1a72274bc3ed (noop)
Notice: /Stage[main]/Mysql::Server::Config/File[/etc/my.cnf.d]/ensure: current_value absent, should be directory (noop)
Notice: Class[Mysql::Server::Config]: Would have triggered 'refresh' from 2 events
Notice: /Stage[main]/Mysql::Server::Service/Service[mysqld]/ensure: current_value stopped, should be running (noop)
Info: /Stage[main]/Mysql::Server::Service/Service[mysqld]: Unscheduling refresh on Service[mysqld]
Notice: /Stage[main]/Mysql::Server::Service/File[/var/log/mysqld.log]/ensure: current_value absent, should be present (noop)
Notice: Class[Mysql::Server::Service]: Would have triggered 'refresh' from 2 events
Error: Could not prefetch mysql_grant provider 'mysql': Command mysql is missing
Notice: /Stage[main]/Main/Node[default]/Mysql_grant[m_user@localhost/lvm.*]: Dependency Mysql_user[m_user@localhost] has failures: true
Warning: /Stage[main]/Main/Node[default]/Mysql_grant[m_user@localhost/lvm.*]: Skipping because of failed dependencies
Notice: Stage[main]: Would have triggered 'refresh' from 3 events
Error: Could not find a suitable provider for mysql_user
Error: Could not find a suitable provider for mysql_database
</pre>
<br />
The issue was I was running puppet with —noop mode. When my Puppet tries to configure the mysql setup it gives errors because it didn’t have a mysql setup to configure since I had —noop. Removing this did the trick.<br />
<br />
Although this is trivial, I thought of blogging this because someone might find this useful when facing the same issue as I did.<br />
<br />
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-34880938389002980912015-02-02T16:18:00.000-08:002015-02-02T16:19:12.878-08:00Resolving EACCES error when using Angular with Yeoman<div dir="ltr" style="text-align: left;" trbidi="on">
In one of my earlier posts, I discussed installing NodeJS, NPM and Yeoman. Although the setup was good to start off my initial work, it was not was giving me the following error when trying to install Angular generator. Following post describe how to resolve this error. <br />
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev/projects/myYoApp$yo
_-----_
| | .------------------------------------------.
|--(o)--| | Update available: 1.4.5 (current: 1.3.3) |
`---------´ | Run npm install -g yo to update. |
( _´U`_ ) '------------------------------------------'
/___A___\
| ~ |
__'.___.'__
´ ` |° ´ Y `
? 'Allo Heshan! What would you like to do? Install a generator
? Search NPM for generators: angular
? Here's what I found. Install one? angular
npm ERR! Darwin 14.0.0
npm ERR! argv "node" "/usr/local/bin/npm" "install" "-g" "generator-angular"
npm ERR! node v0.10.33
npm ERR! npm v2.1.11
npm ERR! path /Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md
npm ERR! code EACCES
npm ERR! errno 3
npm ERR! Error: EACCES, unlink '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md'
npm ERR! { [Error: EACCES, unlink '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md']
npm ERR! errno: 3,
npm ERR! code: 'EACCES',
npm ERR! path: '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.
npm ERR! error rolling back Error: EACCES, unlink '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md'
npm ERR! error rolling back { [Error: EACCES, unlink '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md']
npm ERR! error rolling back errno: 3,
npm ERR! error rolling back code: 'EACCES',
npm ERR! error rolling back path: '/Users/heshans/.node/lib/node_modules/generator-angular/CHANGELOG.md' }
npm ERR! Please include the following file with any support request:
npm ERR! /Users/heshans/Dev/projects/myYoApp/npm-debug.log
</pre>
<br />
It was due to a permission error in my setup. I tried giving my user the permission to access those files but it still didn't resolve the issue.
Then I decided to remove my existing NodeJS and NPM installations. Then I used the following <a href="https://gist.github.com/isaacs/579814#file-node-and-npm-in-30-seconds-sh">script</a> by <a href="https://gist.github.com/isaacs/579814">isaacs</a> with slight modifications. It worked like a charm. Then I was able to successfully install AngularJS generator to Yeoman.<br />
<br />
PS : Also make sure that you have updated your PATH variable in your ~/.bash_profile file.<br />
<b>export PATH=$HOME/local/bin:~/.node/bin:$PATH </b></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-15608457072765225582014-12-09T16:11:00.000-08:002014-12-09T16:11:40.229-08:00Install the Yeoman toolset<div dir="ltr" style="text-align: left;" trbidi="on">
1. Prerequisite: Node.js and NPM should be <a href="http://heshans.blogspot.com/2014/12/install-nodejs-and-npm-on-mac-osx.html">installed</a> in the system.<br />
<br />
2. Install Yeoman tools
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$npm install --global yo bower grunt-cli</pre>
<br />
3. Check installed versions.
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$yo --version && bower --version && grunt --version
1.3.3
1.3.12
grunt-cli v0.1.13</pre>
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-16555486583817129652014-12-09T15:41:00.000-08:002014-12-09T15:41:53.878-08:00 Install Node.js and NPM on Mac OSX <div dir="ltr" style="text-align: left;" trbidi="on">
I’m using Homebrew for the installation. If you don't have it installed, please install it. <br />
<br />
1. Install node
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$brew install node</pre>
<br />
2. Check installed versions.
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$node -v
v0.10.33
heshans@15mbp-08077.local:~/Dev$npm -v
2.1.8</pre>
<br />
3. Update Homebrew.
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$brew update
Already up-to-date.</pre>
<br />
4. Upgrade Node.
<br />
<pre class="brush:js">heshans@15mbp-08077.local:~/Dev$brew upgrade node</pre>
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-63962261623687406842014-10-15T12:30:00.000-07:002014-10-15T12:31:39.717-07:00How to find which method called the current method at runtime<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
I’m using the following helper class to find which method called the current method (at runtime.)
<pre class="brush:js">
import java.lang.reflect.Method;
import java.security.AccessController;
import java.security.PrivilegedAction;
public class TraceHelper {
private volatile static Method m;
private static Throwable t;
public TraceHelper() {
try {
m = Throwable.class.getDeclaredMethod("getStackTraceElement", int.class);
AccessController.doPrivileged(
new PrivilegedAction() {
public Object run() {
m.setAccessible(true);
return null;
}
}
);
t = new Throwable();
} catch (Exception e) {
e.printStackTrace();
}
}
public String getMethodName(final int depth, boolean useNew) {
return getMethod(depth, t != null && !useNew ? t : new Throwable());
}
public String getMethod(final int depth, Throwable t) {
try {
StackTraceElement element = (StackTraceElement) m.invoke(t, depth + 1);
return element.getClassName() + "$" + element.getMethodName();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}
}
</pre>
Then I use the following code inside my aspect to get the name of the method which called my current (executing) method.
<pre class="brush:js">
String previousMethodName = new TraceHelper().getMethodName(2, false);
</pre>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-37741013659635624082014-10-15T12:12:00.000-07:002014-10-15T12:12:14.121-07:00Include/exclude sources when using aspect-maven-plugin<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
AspectJ mavn plugin will add all .java and .aj files in the project source directories by default. By using <include/> and <exclude/> tags, you can add filtering on top of that. </div>
<br />
<br />
<pre class="brush:js"> <plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.5</version>
<configuration>
<complianceLevel>1.7</complianceLevel>
<source>1.7</source>
<target>1.7</target>
<sources>
<source>
<!--<basedir>src/main/java</basedir>-->
<!--<includes>-->
<!--<include>**/TransationAspect.java</include>-->
<!--</includes>-->
<excludes>
<exclude>**/DcXferHandler.java</exclude>
<!--<exclude>**/ChunkingSqlSerActor.java</exclude>-->
</excludes>
</source>
</sources>
</configuration>
<executions>
<execution>
<!--<phase>process-sources</phase>-->
<goals>
<goal>compile</goal>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
</plugin></pre>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-6799107985279170332014-10-14T14:46:00.000-07:002014-10-14T23:15:11.916-07:00Stackmap frame errors when building the aspectj project with Java 1.7<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
I had a project which used aspectj and it was building fine with Java 1.6. When I updated it to Java 1.7 I saw the following error. <br />
<br />
<blockquote class="tr_bq">
[INFO] Molva the Destroyer Aspects ....................... FAILURE [2.324s]<br />
[INFO] Molva The Destroyer Client ........................ SKIPPED<br />
[INFO] Molva The Destroyer Parent ........................ SKIPPED<br />
[INFO] Molva The Destroyer Distribution .................. SKIPPED<br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] BUILD FAILURE<br />
[INFO] ------------------------------------------------------------------------<br />
[INFO] Total time: 2.424s<br />
[INFO] Finished at: Tue Oct 14 11:16:19 PDT 2014<br />
[INFO] Final Memory: 12M/310M<br />
[INFO] ------------------------------------------------------------------------<br />
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.1:java (default) on project molva-the-destroyer-aspects: An exception occured while executing the Java class. Expecting a stackmap frame at branch target 30<br />
[ERROR] Exception Details:<br />
[ERROR] Location:<br />
[ERROR] com/concur/puma/molva/aspects/TestTarget.main([Ljava/lang/String;)V @12: invokestatic<br />
[ERROR] Reason:<br />
[ERROR] Expected stackmap frame at this location.<br />
[ERROR] Bytecode:<br />
[ERROR] 0000000: 2a4d b200 5e01 012c b800 644e b800 c62d<br />
[ERROR] 0000010: b600 ca2c 2db8 00bb 2db8 00bf 57b1 3a04<br />
[ERROR] 0000020: b800 c62d 1904 b600 ce19 04bf<br />
[ERROR] Exception Handler Table:<br />
[ERROR] bci [12, 30] => handler: 30<br />
[ERROR] -> [Help 1]<br />
[ERROR] <br />
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.<br />
[ERROR] Re-run Maven using the -X switch to enable full debug logging.<br />
[ERROR] <br />
[ERROR] For more information about the errors and possible solutions, please read the following articles:<br />
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException</blockquote>
<br />
My maven configuration looked like below. <br />
<pre class="brush:js"> <dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>3.8.1</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<version>1.6.5</version>
</dependency>
<dependency>
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
<version>1.2.17</version>
</dependency>
<dependency>
<groupId>org.perf4j</groupId>
<artifactId>perf4j</artifactId>
<version>0.9.16</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.2</version>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>exec-maven-plugin</artifactId>
<version>1.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>java</goal>
</goals>
</execution>
</executions>
<configuration>
<mainClass>com.concur.puma.molva.aspects.TestTarget</mainClass>
</configuration>
</plugin>
</plugins>
</build>
</pre>
<br />
<h3>
<u><b>Fix</b></u> </h3>
The default compliance level for aspectj-maven-plugin is 1.4 according to <a href="http://mojo.codehaus.org/aspectj-maven-plugin/compile-mojo.html#complianceLevel">http://mojo.codehaus.org/aspectj-maven-plugin/compile-mojo.html#complianceLevel</a>. Since I did not have that tag specified, the build was using the default value. Once I <a href="http://heshans.blogspot.com/2014/10/compile-aspectj-project-containing-java.html">inserted</a> the tag into the configuration, the build was successful.
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-45152874050665428952014-10-14T14:26:00.000-07:002014-10-14T14:47:33.662-07:00 Compile aspectj project containing Java 1.7 Source <div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" style="text-align: left;" trbidi="on">
<br /></div>
Following maven configuration let’s you compile a project with Java 1.7 source.
<br />
<pre class="brush:js"><dependencies>
<dependency>
<groupId>org.aspectj</groupId>
<artifactId>aspectjrt</artifactId>
<!--<version>1.6.5</version>-->
<version>1.8.2</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>aspectj-maven-plugin</artifactId>
<version>1.7</version>
<configuration>
<complianceLevel>1.7</complianceLevel>
<source>1.7</source>
<target>1.7</target>
</configuration>
<executions>
<execution>
<!--<phase>process-sources</phase>-->
<goals>
<goal>compile</goal>
<goal>test-compile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</pre>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Bellevue, WA, USA47.610377 -122.200678647.439057 -122.5234021 47.781697 -121.87795510000001tag:blogger.com,1999:blog-6087794019829505601.post-51803218288982817282014-09-23T15:20:00.000-07:002014-09-23T15:20:51.386-07:00Jmeter plugins for Apache Jmeter<div dir="ltr" style="text-align: left;" trbidi="on">
I came across <a href="http://jmeter-plugins.org/">Jmeter-plugins</a> project today and tried it out. It has a nice set of addons to support/complement existing functionality of Jmeter. </div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0tag:blogger.com,1999:blog-6087794019829505601.post-87704278253199112022014-03-27T14:03:00.000-07:002014-03-27T14:06:56.985-07:00Installing R on Mac OS X<div dir="ltr" style="text-align: left;" trbidi="on">
1) Install Homebrew in your system.<br />
<br />
2) Download and install <a href="https://xquartz.macosforge.org/landing/">XQuartz</a> (Homebrew does not package XQuartz and it is needed for step 3. Therefore install XQuartz.)<br />
<br />
3) Install Fortran and R.
<br />
<pre class="brush:js">heshans@15mbp-08077:/tmp$ brew update
heshans@15mbp-08077:/tmp$ brew tap homebrew/science
heshans@15mbp-08077:/tmp$ brew install gfortran r
</pre>
<br />
4) Verify the installation by running R.
<br />
<pre class="brush:js">heshans@15mbp-08077:/tmp$ R
R version 3.0.3 (2014-03-06) -- "Warm Puppy"
Copyright (C) 2014 The R Foundation for Statistical Computing
Platform: x86_64-apple-darwin13.1.0 (64-bit)
R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.
Natural language support but running in an English locale
R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.
Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.
</pre>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-13711681708027408422014-03-25T09:09:00.002-07:002014-03-25T09:09:45.405-07:00Custom Log Formatter <div dir="ltr" style="text-align: left;" trbidi="on">
Following is the source for a custom log formatter.<br />
<br />
<pre class="brush:js">import java.text.DateFormat;
import java.text.SimpleDateFormat;
import java.util.Date;
import java.util.logging.Formatter;
import java.util.logging.Handler;
import java.util.logging.LogRecord;
public class CustomFormatter extends Formatter {
private static final DateFormat df = new SimpleDateFormat("dd/MM/yyyy hh:mm:ss.SSS");
public String format(LogRecord record) {
StringBuilder builder = new StringBuilder(1000);
builder.append(df.format(new Date(record.getMillis()))).append(" - ");
builder.append("[").append(record.getSourceClassName()).append(".");
builder.append(record.getSourceMethodName()).append("] - ");
builder.append("").append(record.getLevel()).append(" - ");
builder.append(formatMessage(record));
builder.append("\n");
return builder.toString();
}
public String getHead(Handler h) {
return super.getHead(h);
}
public String getTail(Handler h) {
return super.getTail(h);
}
}
</pre>
<br />
Following is the way how the log messages will turn up in your log file.<br />
<br />
<pre class="brush:js">25/03/2014 08:46:17.770 - [com.company.util.memcache.EntityThreadTask.setDataToMemcache] - INFO - thread05 compositekey set time : 5 ms
25/03/2014 08:46:17.781 - [com.company.util.memcache.EntityThreadTask.setDataToMemcache] - INFO - thread03 compositekey set time : 11 ms
25/03/2014 08:46:17.783 - [com.company.util.memcache.EntityThreadTask.setDataToMemcache] - INFO - thread05 compositekey set time : 12 ms
25/03/2014 08:46:17.785 - [com.company.util.memcache.EntityThreadTask.setDataToMemcache] - INFO - thread03 compositekey set time : 4 ms
25/03/2014 08:46:17.787 - [com.company.util.memcache.EntityThreadTask.setDataToMemcache] - INFO - thread05 compositekey set time : 4 ms</pre>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0tag:blogger.com,1999:blog-6087794019829505601.post-69262077789584647002014-03-10T13:38:00.001-07:002014-03-10T13:39:54.777-07:00Fixing BSFException: unable to load language: java<div dir="ltr" style="text-align: left;" trbidi="on">
<div style="text-align: justify;">
I was using JMeter to execute BeanShell scripts that I have written and came across this exception. I had to waste sometime to find out what was the exact issue. Although I copied the BSF jar to the JMeter lib directory, it was not sufficient. When I added the bsh-bsf-2.0b4.jar, the script started running successfully. </div>
<div style="text-align: justify;">
</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
I thought someone else might find this tip useful, therefore blogging it. </div>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com2Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-12280252352842798902014-03-06T10:02:00.002-08:002014-03-06T10:02:40.383-08:00Fixing java.lang.ClassNotFoundException: org.apache.bsf.engines.java.JavaEngine<div dir="ltr" style="text-align: left;" trbidi="on">
When using JMeter’s Java request sampler, I started seeing the below error.
<br />
<pre class="brush:js">2014/03/06 09:46:39 ERROR - org.apache.bsf.BSFManager: Exception : java.lang.ClassNotFoundException: org.apache.bsf.engines.java.JavaEngine
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.bsf.BSFManager.loadScriptingEngine(BSFManager.java:693)
at org.apache.jmeter.util.BSFTestElement.processFileOrScript(BSFTestElement.java:98)
at org.apache.jmeter.visualizers.BSFListener.sampleOccurred(BSFListener.java:51)
at org.apache.jmeter.threads.ListenerNotifier.notifyListeners(ListenerNotifier.java:84)
at org.apache.jmeter.threads.JMeterThread.notifyListeners(JMeterThread.java:783)
at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:443)
at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257)
</pre>
<br />
Following steps will resolve this issue.<br />
<br />
1. Remove existing bsf jar from the jmeter/lib directory
<br />
<pre class="brush:js">heshans@15mbp-08077:~/Dev/tools$ rm apache-jmeter-2.11/lib/bsf-2.4.0.jar</pre>
<br />
2. Download and extract BSF http://wi.wu-wien.ac.at/rgf/rexx/bsf4rexx/current/BSF4Rexx_install.zip.<br />
<br />
3. Copy the following two jars to the jmeter/lib directory.
<br />
<pre class="brush:js">heshans@15mbp-08077:~/Dev/tools$ cp bsf4rexx/bsf-rexx-engine.jar apache-jmeter-2.11/lib/
heshans@15mbp-08077:~/Dev/tools$ cp bsf4rexx/bsf- apache-jmeter-2.11/lib/
</pre>
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07218687905915383016noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-8173746545239510862014-02-27T12:27:00.000-08:002014-02-27T12:27:08.760-08:00Fixing java.lang.NoClassDefFoundError: org/codehaus/classworlds/Launcher<div dir="ltr" style="text-align: left;" trbidi="on">
I came across this error after I installed a new version of maven.
<pre class="brush:js">$ mvn -version
java.lang.NoClassDefFoundError: org/codehaus/classworlds/Launcher
Caused by: java.lang.ClassNotFoundException: org.codehaus.classworlds.Launcher
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.codehaus.classworlds.Launcher. Program will exit.
Exception in thread "main"</pre>
The issue was with having two MAVEN_HOME locations (two versions) in the "path" variable. Once I removed a one, the issue resolved.
<pre class="brush:js">$ mvn -version
Apache Maven 3.1.1</pre>
<br /></div>Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-81448188238313537462014-02-24T11:43:00.000-08:002015-03-18T13:04:47.267-07:00Fixing mysql replication errors in a master/slave setup<div dir="ltr" style="text-align: left;" trbidi="on">
If you have a mysql master/slave replication setup and have run into replication errors, you can follow the below instructions to fix up the replication break and sync up the data.<br />
1) Stop mysql on the slave.
<br />
<pre class="brush:js">service mysql stop</pre>
2) Login to the master.<br />
<br />
3) Lock the master by running the following command
<br />
<pre class="brush:js">mysql> flush tables with read lock;</pre>
NOTE: No data can be written at this time to the master, so be quick in the following steps if your site is in production. It is important that you do not close mysql or this running ssh terminal. Login to a separate ssh terminal for the next part.<br />
<br />
4) Now, in a separate ssh screen, type the following command from the master mysql server.
<br />
<pre class="brush:js">rsync -varPe ssh /var/lib/mysql root@IP_ADDRESS:/var/lib/ —delete-after</pre>
5) After this is done, now you must get the binlog and position from the master before you unlock the tables. Run the following command on master
<br />
<pre class="brush:js">mysql> show master status\G;</pre>
Then you’ll get some information on master status. You need the file and the position because that’s what you’re going to use on the slave in a moment. See step 10 on how this information is used, but please do not skip to step 10.<br />
<br />
6) Unlock the master. It does not need to be locked now that you have the copy of the data and the log position.
<br />
<pre class="brush:js">mysql> unlock tables;</pre>
7) Login to the slave now via ssh. First remove the two files : /var/lib/mysql/master.info and /var/lib/mysql/relay-log.info<br />
<br />
8) Start mysql on the slave.
<br />
<pre class="brush:js">service mysqld start</pre>
9) Immediately login to mysql and stop slave
<br />
<pre class="brush:js">mysql> slave stop; </pre>
10) Now, you have to run the following query filling in the information from the show master status above (Step 5.)
<br />
<pre class="brush:js">mysql> CHANGE MASTER TO MASTER_HOST=MASTER_IP,
mysql> MASTER_USER=‘replicate’, MASTER_PASSWORD=‘replicate’,
mysql> MASTER_LOG_FILE=‘mysql-bin.000001’, MASTER_LOG_POS=1234512351;</pre>
11) Now start the slave.
<br />
<pre class="brush:js">mysql > slave start</pre>
12) Check slave status.
<br />
<pre class="brush:js">mysql> show slave status\G;</pre>
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com6Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-4555137688912824902014-02-20T14:51:00.000-08:002014-02-20T14:51:39.496-08:00XML to XML tranformation using XSLT<div dir="ltr" style="text-align: left;" trbidi="on">
In this example I am transforming an XML using XSLT by modifying certain child elements.
In coming XML message:
<pre class="brush:js">
<?xml version="1.0" encoding="utf-8"?><?xml-stylesheet type="text/xsl" href="/xsl/error.xsl"?>
<outtask_error>
<server>
<name>RSDC-C12</name>
</server>
<application>
<name>Jobs</name>
<version>2012_06_15</version>
<prod_status>qa</prod_status>
<datacenter_location>rtwa3</datacenter_location>
<request_id>n6dft440-8049-4dff-982a-ee05abc8433</request_id>
</application>
<error time="2014-02-06T20:01:05" timezone="Pacific Standard Time">
<error_id>aab10-825-4a52d-a261-7a845262f</error_id>
<error_default_source>ICSYC</error_default_source>
<request_id>b2263a40-8049-4dff-942a-ee05ef3f8433</request_id>
<error_source>ICSync</error_source>
<error_description>Connection Entity:abc no DB name is not configured</error_description>
<stack_trace>at Company.Core.Sync.GetConnectionInfo...
</stack_trace>
</error>
<server_variables>
<param name="SERVER_NAME">RSDC-C12</param>
<param name="SCRIPT_NAME">C:\Sync.exe</param>
<param name="QUERY_STRING">Sync.exe Day1</param>
</server_variables>
<Error_Context>
<param name="databasename">Company_MT</param>
<param name="error_defaultsource">SYS</param>
<param name="error_defaultapplicationname">Jobs</param>
<param name="requestid">b8943a40-1349-4dff-232a-ee05ef523452342f</param>
<param name="dbservername">fabc-sql02,1420</param>
</Error_Context>
</outtask_error>
</pre>
XSLT for the transformation:
<pre class="brush:js">
<?xml version="1.0" encoding="ISO-8859-1"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml"/>
<xsl:template match="@*|node()">
<xsl:copy>
<xsl:apply-templates select="@*|node()"/>
</xsl:copy>
</xsl:template>
<xsl:template match="outtask_error/server_variables">
<server_variables>
<xsl:for-each select="param">
<xsl:text disable-output-escaping="yes"><</xsl:text><xsl:value-of select="./@name"/><xsl:text disable-output-escaping="yes">></xsl:text>
<xsl:value-of select="."/>
<xsl:text disable-output-escaping="yes"><</xsl:text>/<xsl:value-of select="./@name"/><xsl:text disable-output-escaping="yes">></xsl:text>
</xsl:for-each>
</server_variables>
</xsl:template>
<xsl:template match="outtask_error/Error_Context">
<Error_Context>
<xsl:for-each select="param">
<xsl:text disable-output-escaping="yes"><</xsl:text><xsl:value-of select="./@name"/><xsl:text disable-output-escaping="yes">></xsl:text>
<xsl:value-of select="."/>
<xsl:text disable-output-escaping="yes"><</xsl:text>/<xsl:value-of select="./@name"/><xsl:text disable-output-escaping="yes">></xsl:text>
</xsl:for-each>
</Error_Context>
</xsl:template>
</xsl:stylesheet>
</pre>
Out going XML:
<pre class="brush:js">
<?xml version="1.0" encoding="UTF-8"?><?xml-stylesheet type="text/xsl" href="/xsl/error.xsl"?><outtask_error>
<server>
<name>RSDC-C12</name>
</server>
<application>
<name>Jobs</name>
<version>2012_06_15</version>
<prod_status>qa</prod_status>
<datacenter_location>rtwa3</datacenter_location>
<request_id>n6dft440-8049-4dff-982a-ee05abc8433</request_id>
</application>
<error time="2014-02-06T20:01:05" timezone="Pacific Standard Time">
<error_id>aab10-825-4a52d-a261-7a845262f</error_id>
<error_default_source>ICSYC</error_default_source>
<request_id>b2263a40-8049-4dff-942a-ee05ef3f8433</request_id>
<error_source>ICSync</error_source>
<error_description>Connection Entity:abc no DB name is not configured</error_description>
<stack_trace>at Company.Core.Sync.GetConnectionInfo...
</stack_trace>
</error>
<server_variables><SERVER_NAME>RSDC-C12</SERVER_NAME><SCRIPT_NAME>C:\Sync.exe</SCRIPT_NAME><QUERY_STRING>Sync.exe Day1</QUERY_STRING></server_variables>
<Error_Context><databasename>Company_MT</databasename><error_defaultsource>SYS</error_defaultsource><error_defaultapplicationname>Jobs</error_defaultapplicationname><requestid>b8943a40-1349-4dff-232a-ee05ef523452342f</requestid><dbservername>fabc-sql02,1420</dbservername></Error_Context>
</outtask_error>
</pre>
<br /></div>Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com2Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-88779001348378034512014-01-30T13:30:00.000-08:002014-01-30T13:52:52.858-08:00Aspect oriented programming with Java <div dir="ltr" style="text-align: left;" trbidi="on">
I started working with AspectJ and Perf4J recently. I did not have any prior AOP experience. I was looking for an example which did not use Spring AOP but I did not find a good sample which I could use to get my work done. Therefore, once I completed my project, thought of writing this sample up. There are a lot of tutorials out there which describes what AOP is. Therefore, I’m not going to introduce it here. You can do a Google search and find it out.<br />
<br />
What I wanted was to measure the method runtimes without doing any modifications to the existing source. AspectJ helped me to achieve that.<br />
<br />
Following are my sample source code.
Following is my test class which I'm using to measure method runtimes.
<br />
<pre class="brush:js">import org.apache.log4j.Logger;
public class TestTarget {
static Logger logger = Logger.getLogger(TestTarget.class);
public static void main(String[] args) throws Exception {
// Test case 01
testCountFast(1000);
// Test case 02
testCountSlow(1000);
}
public static void testCountSlow(int value) {
count(value,5);
}
public static void testCountFast(int value) {
count(value,0);
}
private static void count(int value, int delay) {
for (int i=0;i<value;i++) {
try {
Thread.sleep(delay);
} catch (Exception e) {
logger.error("Error occurred while sleeping", e);
}
}
}
}
</pre>
Following is my Aspect class which uses AspectJ and Perf4J.
<br />
<pre class="brush:js">import org.apache.log4j.Logger;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.perf4j.StopWatch;
import org.perf4j.log4j.Log4JStopWatch;
@Aspect
public class Perf4JAspect {
static Logger logger = Logger.getLogger(Perf4JAspect.class);
@Around("execution (* com.company.molva.aspectj.TestTarget.test*(..))")
public Object measureExecutionTime(ProceedingJoinPoint joinPoint) throws Throwable {
logger.info(MolvaLoggingUtil.getCurrentDateTime()
+ " MainAspect.measureExecutionTime() called on " + joinPoint.getSignature());
long startNanoTime = System.nanoTime();
StopWatch stopWatch = new Log4JStopWatch();
Object proceed = joinPoint.proceed();
stopWatch.stop(joinPoint.getSignature().toLongString());
long endNanoTime = System.nanoTime();
long executionNanoTime = endNanoTime - startNanoTime;
logger.info(MolvaLoggingUtil.getCurrentDateTime() + " "
+ joinPoint.getSignature()+" took "+ executionNanoTime/1000000.0 +"ms");
return proceed;
}
}
</pre>
My aop.xml configuration is as follows.
<br />
<pre class="brush:js"><aspectj>
<aspects>
<aspect name="com.company.molva.aspectj.MainAspect"/>
<aspect name="com.company.molva.aspectj.Perf4JAspect"/>
</aspects>
<weaver options="-verbose">
<include within="com.aspectj.*"/>
</weaver>
</aspectj>
</pre>
My log4j.xml looks as follows.
<br />
<pre class="brush:js">
<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">
<log4j:configuration debug="false" xmlns:log4j="http://jakarta.apache.org/log4j/">
<appender name="aspectjFileAppender" class="org.apache.log4j.FileAppender">
<param name="File" value="aspectj_stats.log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%m%n"/>
</layout>
</appender>
<appender name="CoalescingStatistics"
class="org.perf4j.log4j.AsyncCoalescingStatisticsAppender">
<param name="TimeSlice" value="10000"/>
<appender-ref ref="fileAppender"/>
</appender>
<appender name="fileAppender" class="org.apache.log4j.FileAppender">
<param name="File" value="perf_stats.log"/>
<layout class="org.apache.log4j.PatternLayout">
<param name="ConversionPattern" value="%m%n"/>
</layout>
</appender>
<logger name="org.perf4j.TimingLogger" additivity="false">
<level value="INFO"/>
<appender-ref ref="CoalescingStatistics"/>
</logger>
<root>
<level value="INFO"/>
<appender-ref ref="aspectjFileAppender"/>
</root>
</log4j:configuration>
</pre>
Once the sample completes running, I get the following logs at my log files.<br />
aspectj_stats.log:<br />
<pre class="brush:js">2014-01-30 13:19:25 MainAspect.measureExecutionTime() called on void com.company.molva.aspectj.TestTarget.testCountFast(int)
2014-01-30 13:19:25 void com.company.molva.aspectj.TestTarget.testCountFast(int) took 0.904ms
2014-01-30 13:19:25 MainAspect.measureExecutionTime() called on void com.company.molva.aspectj.TestTarget.testCountSlow(int)
2014-01-30 13:19:31 void com.company.molva.aspectj.TestTarget.testCountSlow(int) took 5608.996ms
</pre>
perf_stats.log:<br />
<pre class="brush:js">Performance Statistics 2014-01-30 13:19:20 - 2014-01-30 13:19:30
Tag Avg(ms) Min Max Std Dev Count
public static void com.company.molva.aspectj.TestTarget.testCountFast(int) 0.0 0 0 0.0 1
public static void com.company.molva.aspectj.TestTarget.testCountSlow(int) 5608.0 5608 5608 0.0 1
</pre>
As you can see, we can use AspectJ and Perf4J to generate some nice statistics.
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-92177810755375903442014-01-30T10:48:00.000-08:002014-01-30T13:31:47.275-08:00Java library to convert XML to JSON <div dir="ltr" style="text-align: left;" trbidi="on">
Recently I wanted to convert some XMLs to JSON messages. I came across this library called <a href="https://github.com/douglascrockford/JSON-java">JSON-Java</a> and found it very useful.
<br />
<pre class="brush:js">import org.json.JSONException;
import org.json.JSONObject;
import org.json.XML;
public class Main {
public static int PRETTY_PRINT_INDENT_FACTOR = 4;
public static String TEST_XML_STRING =
"<breakfast_menu>\n" +
"<food>\n" +
"<name>Belgian Waffles</name>\n" +
"<price>$5.95</price>\n" +
"<description>\n" +
"Two of our famous Belgian Waffles with plenty of real maple syrup\n" +
"</description>\n" +
"<calories>650</calories>\n" +
"</food>\n" +
"<food>\n" +
"<name>Strawberry Belgian Waffles</name>\n" +
"<price>$7.95</price>\n" +
"<description>\n" +
"Light Belgian waffles covered with strawberries and whipped cream\n" +
"</description>\n" +
"<calories>900</calories>\n" +
"</food>\n" +
"</breakfast_menu>";
public static void main(String[] args) {
try {
JSONObject xmlJSONObj = XML.toJSONObject(TEST_XML_STRING);
String jsonPrettyPrintString = xmlJSONObj.toString(PRETTY_PRINT_INDENT_FACTOR);
System.out.println(jsonPrettyPrintString);
} catch (JSONException je) {
System.out.println(je.toString());
}
}
}
</pre>
The XML:
<br />
<pre class="brush:js"><breakfast_menu>
<food>
<name>Belgian Waffles</name>
<price>$5.95</price>
<description>
Two of our famous Belgian Waffles with plenty of real maple syrup
</description>
<calories>650</calories>
</food>
<food>
<name>Strawberry Belgian Waffles</name>
<price>$7.95</price>
<description>
Light Belgian waffles covered with strawberries and whipped cream
</description>
<calories>900</calories>
</food>
</breakfast_menu>
</pre>
JSON:
<br />
<pre class="brush:js">{"breakfast_menu": {"food": [
{
"price": "$5.95",
"description": "Two of our famous Belgian Waffles with plenty of real maple syrup",
"name": "Belgian Waffles",
"calories": 650
},
{
"price": "$7.95",
"description": "Light Belgian waffles covered with strawberries and whipped cream",
"name": "Strawberry Belgian Waffles",
"calories": 900
}
]}}
</pre>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com3Seattle, WA, USA47.6062095 -122.332070847.2636815 -122.9775178 47.9487375 -121.68662379999999tag:blogger.com,1999:blog-6087794019829505601.post-58836898958114576542013-11-18T13:07:00.000-08:002013-11-18T13:41:15.973-08:00My first tandem sky dive experience<div dir="ltr" style="text-align: left;" trbidi="on">
<span class="userContent">Never thought jumping out of an air plane at 11000 feet would be this much fun. What a rush!</span><br />
<br />
Video quality is a bit low because I had to convert it to a lesser quality to upload it to youtube.<br />
<span class="userContent"><br /></span>
<iframe width="560" height="315" src="//www.youtube.com/embed/pS0G-Tjb2Sg" frameborder="0" allowfullscreen></iframe>
</div>Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Plant City, FL, USA28.0186323 -82.11286410000002427.7943648 -82.435587600000019 28.2428998 -81.790140600000029tag:blogger.com,1999:blog-6087794019829505601.post-89360902559299228232013-11-18T12:53:00.000-08:002013-11-18T12:59:11.101-08:00Techniques to backup and restore mysql databases - A comparison<div dir="ltr" style="text-align: left;" trbidi="on">
<div dir="ltr" id="docs-internal-guid-6a32e039-6cee-f5bc-5af1-f8546d48da8a" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">When I started working at Digital Mediat Solutions, one of the biggest problems that we had was the runtime of daily mysql backup scripts. They were taking 8 hours to complete. The first task I had was to fix the performance of the backups. The existing scripts were based on mysqldump. </span><span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span></div>
<div dir="ltr" id="docs-internal-guid-6a32e039-6cee-f5bc-5af1-f8546d48da8a" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" id="docs-internal-guid-6a32e039-6cee-f5bc-5af1-f8546d48da8a" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">I compiled a list of options to look into by referring the web. I'm sharing this, so that someone else might find it useful. </span></div>
<div dir="ltr" id="docs-internal-guid-6a32e039-6cee-f5bc-5af1-f8546d48da8a" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"><br /></span></div>
<div dir="ltr" id="docs-internal-guid-6a32e039-6cee-f5bc-5af1-f8546d48da8a" style="margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial;"><span style="font-size: 15px; line-height: 17px;">Out of the following options, the most feasible (economically, disk space wise and performance wise) option for us, was to use mysqlhotcopy. Now the backup script I have written based on mysqlhotcopy, completes in 1 hours time, which is a significant imporvement. </span></span></div>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<h2 dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 10pt;">
<span style="font-family: 'Trebuchet MS'; font-size: 17px; text-decoration: underline; vertical-align: baseline;">Options </span></h2>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">1) MySQL enterprise backup</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">pros</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">incremental backup</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">compressed backup</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Backing up the physical database files makes restore much faster than logical techniques such as the mysqldump command. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">InnoDB tables are copied using a hot backup mechanism. (Ideally, the InnoDB tables should represent a substantial majority of the data.) </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Tables from other storage engines are copied using a warm backup mechanism.</span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">cons</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">enterprise licensed</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Minimalistic license per 1 mysql server instance per 1 year is XXXXUSD.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">The cost will grow as the instances grow and the running time exceeds year by year. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">It even exceeds our current budget for underlying hardware. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Therefore, I don’t think it’s feasible for our budget and IMV it’s not needed for the current deployment. </span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">2) MySQL dumps with diffs</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">pros</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Saves space in the backed up device as we are backing up the diff. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">It won't result in network traffic as the only time the full dump is transferred, is when it’s run for the first time. </span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">cons</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">We are looking into stopping the overhead generated by the mysql dump command. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Since, we are using the same command over and over again, in this deployment, this wont satisfuy our need. Therefore, IMV this is not the option for us. </span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">3) Make incremental backups by enabling the binary logs.</span></div>
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">pros</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Used to set up replication </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Used for restore operations </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">The above two pros outweigh the con.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Can do pinpoint restorations</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Can tarball the existing logs and ship to the backup location. Just a matter of file zipping operation. </span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">cons</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Makes performance slightly slower.</span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">4) Suggestions to improve current backup scripts</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Improve the cron job (which gets triggered on low usage period) to perform the slaving task to avoid high CPU utilization. </span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">5) Should investigate and run some performance matrices on </span><a href="http://www.xaprb.com/blog/2007/09/30/introducing-mysql-parallel-dump/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">mysql-parallel-dump</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"> and mk-parallel-dump (</span><a href="http://www.maatkit.org/" style="text-decoration: none;"><span style="color: #1155cc; font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;">http://www.maatkit.org/</span></a><span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">) utilities. </span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">mk-parallel-dump is deprecated now. So, it not feasible. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">TODO: Talk to Matt regarding:</span></div>
</li>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Setting up a dev env, so that I could test these stuff. I don’t need any new fancy hardware for these. Just give me a PC and I’ll configure it with linux and try these out. We don’t have to buy new hardware. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Need to test these in a dev env. So, that we could roll these out to a production environment.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: circle; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Need to have a mock server (which have kind of the same traffic as the real servers), so that we could test and verify things properly</span></div>
</li>
</ul>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">6) Remove the current scripts and use a tool specifically written for this and see how it performs.</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://sourceforge.net/projects/automysqlbackup/" style="text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">http://sourceforge.net/projects/automysqlbackup/</span></a><span style="vertical-align: baseline;"></span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; text-decoration: underline; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">7) Update the current scripts with --quick option which avoids buffering of large tables and does row by row backups. Resulting in faster restores. This will be a minor tweak to the current scripts. </span></div>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">8) Use of mysqlhotcopy instead of mysquldump</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">It uses FLUSH TABLES, LOCK TABLES, and cp or scp to make a database backup. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">It is a fast way to make a backup of the database or single tables</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">but it can be run only on the same machine where the database directories are located. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">mysqlhotcopy works only for backing up MyISAM and ARCHIVE tables.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Since we are using MyISAM, this wont be a problem for us.</span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">9) Use SVN to backup databases.</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">This will help to have revisions. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">I don’t think we need such a mechanism for us.</span></div>
</li>
</ul>
<br />
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;"></span>
<br />
<div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="font-family: Arial; font-size: 15px; vertical-align: baseline;">10) If replication to a slave isn't an option, we could leverage the filesystem, depending on the OS we are using,</span></div>
<ul style="margin-bottom: 0pt; margin-top: 0pt;">
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://www.cyberciti.biz/tips/consistent-backup-linux-logical-volume-manager-snapshots.html" style="text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">Consistent backup with Linux Logical Volume Manager (LVM) snapshots</span></a><span style="vertical-align: baseline;">.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://www.oreillynet.com/databases/blog/2008/04/mysql_backups_using_zfs_snapsh_1.html" style="text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">MySQL backups using ZFS snapshots</span></a><span style="vertical-align: baseline;">.</span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<a href="http://blogs.digitar.com/jjww/2008/01/snapback-the-joys-of-backing-up-mysql-with-zfs/" style="text-decoration: none;"><span style="color: #1155cc; text-decoration: underline; vertical-align: baseline;">The joys of backing up MySQL with ZFS...</span></a><span style="vertical-align: baseline;"></span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><div dir="ltr" style="line-height: 1.15; margin-bottom: 0pt; margin-top: 0pt;">
<span style="vertical-align: baseline;">Some people have used ZFS snapshots on a quite large MySQL database (30GB+) as a backup method and it completes very quickly (never more than a few minutes) and says it doesn't block. </span></div>
</li>
<li dir="ltr" style="font-family: Arial; font-size: 15px; list-style-type: disc; vertical-align: baseline;"><span style="vertical-align: baseline;">They say that we can then mount the snapshot somewhere else and back it up to tape, etc.</span></li>
</ul>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Clearwater, FL, USA27.9658533 -82.800102627.8536853 -82.9614641 28.0780213 -82.6387411tag:blogger.com,1999:blog-6087794019829505601.post-25283877734474876452013-04-17T07:52:00.000-07:002013-04-17T07:55:35.592-07:00Apache Airavata 0.7 Released<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="text-align: justify;">The Apache Airavata PMC is pleased to announce the immediate availability of the Airavata 0.7 release.</span><br />
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
The release can be obtained from the Apache Airavata download page - <a href="http://airavata.apache.org/about/downloads.html" target="_blank">http://airavata.apache.org/<wbr></wbr>about/downloads.html</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Release notes are available at - <a href="https://svn.apache.org/repos/asf/airavata/tags/airavata-0.7/RELEASE_NOTES" target="_blank">https://svn.apache.org/repos/<wbr></wbr>asf/airavata/tags/airavata-0.<wbr></wbr>7/RELEASE_NOTES</a></div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Apache Airavata is a software framework providing API’s, sophisticated server-side tools, and graphical user interfaces to construct, execute, control and manage long running applications and workflows on distributed computing resources. Apache Airavata builds on general concepts of service oriented computing, distributed messaging, and workflow composition and orchestration.</div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
For general information on Apache Airavata, please visit the project website: <a href="http://airavata.apache.org/" target="_blank">http://airavata.apache.org/</a></div>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242tag:blogger.com,1999:blog-6087794019829505601.post-75156523082679941892013-04-05T09:56:00.000-07:002013-04-05T09:59:35.734-07:00Run EC2 Jobs with Airavata - Part III<div dir="ltr" style="text-align: left;" trbidi="on">
This is a followup to my earlier posts [1] [<a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html">2</a>]. Here we will execute the application mentioned in [<a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html">2</a>] programmetically using Airavata.<br />
<br />
<pre class="brush:js">import org.apache.airavata.commons.gfac.type.*;
import org.apache.airavata.gfac.GFacAPI;
import org.apache.airavata.gfac.GFacConfiguration;
import org.apache.airavata.gfac.GFacException;
import org.apache.airavata.gfac.context.security.AmazonSecurityContext;
import org.apache.airavata.gfac.context.ApplicationContext;
import org.apache.airavata.gfac.context.JobExecutionContext;
import org.apache.airavata.gfac.context.MessageContext;
import org.apache.airavata.schemas.gfac.*;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import java.io.File;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;
/**
* Your Amazon instance should be in a running state before running this test.
*/
public class EC2ProviderTest {
private JobExecutionContext jobExecutionContext;
private static final String hostName = "ec2-host";
private static final String hostAddress = "ec2-address";
private static final String sequence1 = "RR042383.21413#CTGGCACGGAGTTAGCCGATCCTTATTCATAAAGTACATGCAAACGGGTATCCATA" +
"CTCGACTTTATTCCTTTATAAAAGAAGTTTACAACCCATAGGGCAGTCATCCTTCACGCTACTTGGCTGGTTCAGGCCTGCGCCCATTGACCAATATTCCTCA" +
"CTGCTGCCTCCCGTAGGAGTTTGGACCGTGTCTCAGTTCCAATGTGGGGGACCTTCCTCTCAGAACCCCTATCCATCGAAGACTAGGTGGGCCGTTACCCCGC" +
"CTACTATCTAATGGAACGCATCCCCATCGTCTACCGGAATACCTTTAATCATGTGAACATGCGGACTCATGATGCCATCTTGTATTAATCTTCCTTTCAGAAG" +
"GCTGTCCAAGAGTAGACGGCAGGTTGGATACGTGTTACTCACCGTGCCGCCGGTCGCCATCAGTCTTAGCAAGCTAAGACCATGCTGCCCCTGACTTGCATGT" +
"GTTAAGCCTGTAGCTTAGCGTTC";
private static final String sequence2 = "RR042383.31934#CTGGCACGGAGTTAGCCGATCCTTATTCATAAAGTACATGCAAACGGGTATCCATA" +
"CCCGACTTTATTCCTTTATAAAAGAAGTTTACAACCCATAGGGCAGTCATCCTTCACGCTACTTGGCTGGTTCAGGCTCTCGCCCATTGACCAATATTCCTCA" +
"CTGCTGCCTCCCGTAGGAGTTTGGACCGTGTCTCAGTTCCAATGTGGGGGACCTTCCTCTCAGAACCCCTATCCATCGAAGACTAGGTGGGCCGTTACCCCGC" +
"CTACTATCTAATGGAACGCATCCCCATCGTCTACCGGAATACCTTTAATCATGTGAACATGCGGACTCATGATGCCATCTTGTATTAAATCTTCCTTTCAGAA" +
"GGCTATCCAAGAGTAGACGGCAGGTTGGATACGTGTTACTCACCGTGCG";
/* Following variables are needed to be set in-order to run the test. Since these are account specific information,
I'm not adding the values here. It's the responsibility of the person who's running the test to update
these variables accordingly.
*/
/* Username used to log into your ec2 instance eg.ec2-user */
private String userName = "";
/* Secret key used to connect to the image */
private String secretKey = "";
/* Access key used to connect to the image */
private String accessKey = "";
/* Instance id of the running instance of your image */
private String instanceId = "";
@Before
public void setUp() throws Exception {
URL resource = GramProviderTest.class.getClassLoader().getResource("gfac-config.xml");
assert resource != null;
System.out.println(resource.getFile());
GFacConfiguration gFacConfiguration = GFacConfiguration.create(new File(resource.getPath()), null, null);
/* EC2 Host */
HostDescription host = new HostDescription(Ec2HostType.type);
host.getType().setHostName(hostName);
host.getType().setHostAddress(hostAddress);
/* App */
ApplicationDescription ec2Desc = new ApplicationDescription(Ec2ApplicationDeploymentType.type);
Ec2ApplicationDeploymentType ec2App = (Ec2ApplicationDeploymentType)ec2Desc.getType();
String serviceName = "Gnome_distance_calculation_workflow";
ec2Desc.getType().addNewApplicationName().setStringValue(serviceName);
ec2App.setJobType(JobTypeType.EC_2);
ec2App.setExecutable("/home/ec2-user/run.sh");
ec2App.setExecutableType("sh");
/* Service */
ServiceDescription serv = new ServiceDescription();
serv.getType().setName("GenomeEC2");
List<inputparametertype> inputList = new ArrayList<inputparametertype>();
InputParameterType input1 = InputParameterType.Factory.newInstance();
input1.setParameterName("genome_input1");
input1.setParameterType(StringParameterType.Factory.newInstance());
inputList.add(input1);
InputParameterType input2 = InputParameterType.Factory.newInstance();
input2.setParameterName("genome_input2");
input2.setParameterType(StringParameterType.Factory.newInstance());
inputList.add(input2);
InputParameterType[] inputParamList = inputList.toArray(new InputParameterType[inputList.size()]);
List<outputparametertype> outputList = new ArrayList<outputparametertype>();
OutputParameterType output = OutputParameterType.Factory.newInstance();
output.setParameterName("genome_output");
output.setParameterType(StringParameterType.Factory.newInstance());
outputList.add(output);
OutputParameterType[] outputParamList = outputList
.toArray(new OutputParameterType[outputList.size()]);
serv.getType().setInputParametersArray(inputParamList);
serv.getType().setOutputParametersArray(outputParamList);
jobExecutionContext = new JobExecutionContext(gFacConfiguration,serv.getType().getName());
ApplicationContext applicationContext = new ApplicationContext();
jobExecutionContext.setApplicationContext(applicationContext);
applicationContext.setServiceDescription(serv);
applicationContext.setApplicationDeploymentDescription(ec2Desc);
applicationContext.setHostDescription(host);
AmazonSecurityContext amazonSecurityContext =
new AmazonSecurityContext(userName, accessKey, secretKey, instanceId);
jobExecutionContext.addSecurityContext(AmazonSecurityContext.AMAZON_SECURITY_CONTEXT, amazonSecurityContext);
MessageContext inMessage = new MessageContext();
ActualParameter genomeInput1 = new ActualParameter();
((StringParameterType)genomeInput1.getType()).setValue(sequence1);
inMessage.addParameter("genome_input1", genomeInput1);
ActualParameter genomeInput2 = new ActualParameter();
((StringParameterType)genomeInput2.getType()).setValue(sequence2);
inMessage.addParameter("genome_input2", genomeInput2);
MessageContext outMessage = new MessageContext();
ActualParameter echo_out = new ActualParameter();
outMessage.addParameter("distance", echo_out);
jobExecutionContext.setInMessageContext(inMessage);
jobExecutionContext.setOutMessageContext(outMessage);
}
@Test
public void testGramProvider() throws GFacException {
GFacAPI gFacAPI = new GFacAPI();
gFacAPI.submitJob(jobExecutionContext);
MessageContext outMessageContext = jobExecutionContext.getOutMessageContext();
Assert.assertEquals(MappingFactory.
toString((ActualParameter) outMessageContext.getParameter("genome_output")), "476");
}
}
</outputparametertype></outputparametertype></inputparametertype></inputparametertype></pre>
<br />
<b>References</b><br />
<inputparametertype> </inputparametertype><inputparametertype><inputparametertype><outputparametertype><outputparametertype>[<a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html">1</a>] - <a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html">http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html</a></outputparametertype></outputparametertype></inputparametertype></inputparametertype><br />
[<a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html">2</a>] - <a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html">http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html</a> </div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242tag:blogger.com,1999:blog-6087794019829505601.post-25383232320696169802013-04-05T09:38:00.003-07:002013-04-05T09:39:35.707-07:00Run EC2 Jobs with Airavata - Part II<div dir="ltr" style="text-align: left;" trbidi="on">
In this post we will look at how to compose a workflow out of an application that is installed in an Amazon Machine Image (AMI). In the earlier <a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html">post</a> we discussed how to do ec2 instance management using XBaya GUI. This is the followup to that post.<br />
<br />
For the Airavata EC2 integration testing, I created an AMI which has an
application which does gene sequence alignment using Smith-Waterman
algorithm. I will be using that application as a reference to this post. You can use an application of your preference that resides in your AMI.<br />
<br />
1. Unzip Airavata server distribution and start the server.<br />
<pre class="brush:js">unzip apache-airavata-server-0.7-bin.zip
cd apache-airavata-server-0.7/bin
./airavata-server.sh
</pre>
<br />
2. Unzip Airavata XBaya distribution and start XBaya.<br />
<pre class="brush:js">unzip apache-airavata-xbaya-gui-0.7-bin.zip
cd apache-airavata-xbaya-gui-0.7/bin
./xbaya-gui.sh
</pre>
<br />
Then you'll get the XBaya UI.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-_2FxX6BJ9v0/UV727pZ5WrI/AAAAAAAAAOQ/szNcsLrAT_g/s1600/Screenshot+from+2013-04-05+11:54:47.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-_2FxX6BJ9v0/UV727pZ5WrI/AAAAAAAAAOQ/szNcsLrAT_g/s400/Screenshot+from+2013-04-05+11:54:47.png" width="400" /></a></div>
<br />
3. Select "XBaya" Menu and click "Add Host" to register an EC2 Host. Once you add the details, click "ok".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-kubqhIYEzRY/UV727pp8QvI/AAAAAAAAAOU/GAi1PqZGE3I/s1600/Screenshot+from+2013-04-05+11:55:24.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/-kubqhIYEzRY/UV727pp8QvI/AAAAAAAAAOU/GAi1PqZGE3I/s400/Screenshot+from+2013-04-05+11:55:24.png" width="400" /></a></div>
<br />
4. You will then be prompted to enter "Airavata Registry" information. If you are using the default setup, you don't have to do any configuration. Just click "ok".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Nje_WkYvyP8/UV727pZP10I/AAAAAAAAAOY/bW6MS8BLR2k/s1600/Screenshot+from+2013-04-05+11:54:56.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-Nje_WkYvyP8/UV727pZP10I/AAAAAAAAAOY/bW6MS8BLR2k/s400/Screenshot+from+2013-04-05+11:54:56.png" width="400" /></a></div>
<br />
5. In order to use your application installed in the AMI, you must register it as an application in Airavata system. Select "XBaya" menu and click "Register Application". You will get the following dialog. Add the input parameters expected and the output parameters generated by your application.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-9JcykfereXY/UV728MXuLvI/AAAAAAAAAOo/6NXQArx2sl8/s1600/Screenshot+from+2013-04-05+11:57:00.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/-9JcykfereXY/UV728MXuLvI/AAAAAAAAAOo/6NXQArx2sl8/s400/Screenshot+from+2013-04-05+11:57:00.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
6. Then Click the "New deployment" button. You have to then select the EC2Host that you registered earlier as the Application Host. Configure the executable path to your application in your AMI and click "Add".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-XhQaBKC-LPc/UV728MNNfpI/AAAAAAAAAOk/vT1-QL381C0/s1600/Screenshot+from+2013-04-05+11:57:34.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-XhQaBKC-LPc/UV728MNNfpI/AAAAAAAAAOk/vT1-QL381C0/s400/Screenshot+from+2013-04-05+11:57:34.png" width="400" /></a></div>
<br />
7. Then click "Register". If the application registration was successful, you will be getting the following message.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-50UQKA-taq4/UV728QWTi-I/AAAAAAAAAOs/DN9t86JP5LM/s1600/Screenshot+from+2013-04-05+11:57:54.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-50UQKA-taq4/UV728QWTi-I/AAAAAAAAAOs/DN9t86JP5LM/s400/Screenshot+from+2013-04-05+11:57:54.png" width="400" /></a></div>
<br />
8. Now select "Registry" menu and click "Setup Airavata Registry". Click "ok".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-H4Jz1oMc2aA/UV728WyLCDI/AAAAAAAAAOw/MjmVILJWfSQ/s1600/Screenshot+from+2013-04-05+11:58:05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-H4Jz1oMc2aA/UV728WyLCDI/AAAAAAAAAOw/MjmVILJWfSQ/s400/Screenshot+from+2013-04-05+11:58:05.png" width="400" /></a></div>
<br />
9. Select "XBaya" menu and click "New workflow". Then configure it accordingly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/--EFgXbVo8UA/UV7287_eQbI/AAAAAAAAAO4/MgR6nscAFBc/s1600/Screenshot+from+2013-04-05+11:59:39.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/--EFgXbVo8UA/UV7287_eQbI/AAAAAAAAAO4/MgR6nscAFBc/s400/Screenshot+from+2013-04-05+11:59:39.png" width="400" /></a></div>
<br />
10. Select your registered application from the "Application Services" and drag drop it to the workflow window.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-28iOSNzfgSk/UV7285EJ3zI/AAAAAAAAAO8/eiiX1qjg0aQ/s1600/Screenshot+from+2013-04-05+11:59:55.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-28iOSNzfgSk/UV7285EJ3zI/AAAAAAAAAO8/eiiX1qjg0aQ/s400/Screenshot+from+2013-04-05+11:59:55.png" width="400" /></a></div>
<br />
11. Drag an "Instance" component from "Amazon Components" and drop it into workflow window. Then connect it to your application using Control ports.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-UVfdg0oApX0/UV729OCI4fI/AAAAAAAAAPI/ueCAL3RdrN8/s1600/Screenshot+from+2013-04-05+12:00:21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/-UVfdg0oApX0/UV729OCI4fI/AAAAAAAAAPI/ueCAL3RdrN8/s400/Screenshot+from+2013-04-05+12:00:21.png" width="400" /></a></div>
<br />
12. Click on top of the "Instance" components config label. Configure your instance accordingly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-DWnZA04iz54/UV729G_KtZI/AAAAAAAAAPM/pn9UzoIT8XM/s1600/Screenshot+from+2013-04-05+12:02:40.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-DWnZA04iz54/UV729G_KtZI/AAAAAAAAAPM/pn9UzoIT8XM/s400/Screenshot+from+2013-04-05+12:02:40.png" width="400" /></a></div>
<br />
13. Drag and drop two input components and one output component to the workflow from "System Components".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-IseTPzttUFU/UV729nSBDwI/AAAAAAAAAPQ/iiNJ5kKREJo/s1600/Screenshot+from+2013-04-05+12:03:21.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/-IseTPzttUFU/UV729nSBDwI/AAAAAAAAAPQ/iiNJ5kKREJo/s400/Screenshot+from+2013-04-05+12:03:21.png" width="400" /></a></div>
<br />
14. Connect the components together accordingly.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Y95KEx6amA8/UV7293OdVKI/AAAAAAAAAPc/ZvcRSWt86Ss/s1600/Screenshot+from+2013-04-05+12:03:36.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-Y95KEx6amA8/UV7293OdVKI/AAAAAAAAAPc/ZvcRSWt86Ss/s400/Screenshot+from+2013-04-05+12:03:36.png" width="400" /></a></div>
<br />
15. Now click the red colored "play" button to run your workflow. You will be prompted for the inputs values (in my case the gene sequences) and experiment id. Then click "Run" to execute your workflow.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-gQyDMIlaZfM/UV72-DGES-I/AAAAAAAAAPg/lNfMZPu1q5I/s1600/Screenshot+from+2013-04-05+12:06:51.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-gQyDMIlaZfM/UV72-DGES-I/AAAAAAAAAPg/lNfMZPu1q5I/s400/Screenshot+from+2013-04-05+12:06:51.png" width="400" /></a></div>
<br />
16. The execution result will be shown in the XBaya GUI.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://3.bp.blogspot.com/-OfK3noO2gmQ/UV72-DI3-LI/AAAAAAAAAPY/GMd0HoOsHJs/s1600/Screenshot+from+2013-04-05+12:07:13.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://3.bp.blogspot.com/-OfK3noO2gmQ/UV72-DI3-LI/AAAAAAAAAPY/GMd0HoOsHJs/s400/Screenshot+from+2013-04-05+12:07:13.png" width="400" /></a></div>
<br />
<b>References</b><br />
[<a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html">1</a>] - <a href="http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html">http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html</a></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242tag:blogger.com,1999:blog-6087794019829505601.post-33693709832755821572013-04-05T08:43:00.000-07:002013-04-05T09:09:57.742-07:00Run EC2 Jobs with Airavata - Part I<div dir="ltr" style="text-align: left;" trbidi="on">
This will be the first of many posts that I will be doing on Apache Airavata EC2 integration. First let's have a look at how you can use Airavata's "XBaya GUI" to manage amazon instances.<br />
<br />
Applies to : Airavata 0.7 and above<br />
<br />
1. Unzip Airavata server distribution and start the server.<br />
<pre class="brush:js">unzip apache-airavata-server-0.7-bin.zip
cd apache-airavata-server-0.7/bin
./airavata-server.sh
</pre>
2. Unzip Airavata XBaya distribution and start XBaya.<br />
<pre class="brush:js">unzip apache-airavata-xbaya-gui-0.7-bin.zip
cd apache-airavata-xbaya-gui-0.7/bin
./xbaya-gui.sh
</pre>
Then you'll get the XBaya UI.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-_1wdOzdtHiE/UV7uL6n9TII/AAAAAAAAANY/uPXx0tJPV0c/s1600/Screenshot+from+2013-04-05+11:08:59.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-_1wdOzdtHiE/UV7uL6n9TII/AAAAAAAAANY/uPXx0tJPV0c/s400/Screenshot+from+2013-04-05+11:08:59.png" width="400" /></a></div>
<br />
3. Then Select "Amazon" menu and click "Security Credentials". Specify your secret key and access key in the security credentials dialog box and click "ok".<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-7PbmKUthk30/UV7uL2zMr8I/AAAAAAAAANc/OLgzKTp9xCE/s1600/Screenshot+from+2013-04-05+11:10:26.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://4.bp.blogspot.com/-7PbmKUthk30/UV7uL2zMr8I/AAAAAAAAANc/OLgzKTp9xCE/s400/Screenshot+from+2013-04-05+11:10:26.png" width="400" /></a></div>
<br />
4. Then Select "Amazon" menu and click "EC2 Instance Management". It will give a glimpse of your running instances.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://2.bp.blogspot.com/-i70e56So-J8/UV7uMc2crUI/AAAAAAAAANg/x9AKhf3SNrA/s1600/Screenshot+from+2013-04-05+11:10:46.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://2.bp.blogspot.com/-i70e56So-J8/UV7uMc2crUI/AAAAAAAAANg/x9AKhf3SNrA/s400/Screenshot+from+2013-04-05+11:10:46.png" width="400" /></a></div>
<br />
5. Click the "launch" button to launch new instances and "terminate" button to terminate, running instances.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-vxVw0dCr7Ws/UV7uNcCbU9I/AAAAAAAAANk/R0TM16z4MV4/s1600/Screenshot+from+2013-04-05+11:12:05.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-vxVw0dCr7Ws/UV7uNcCbU9I/AAAAAAAAANk/R0TM16z4MV4/s400/Screenshot+from+2013-04-05+11:12:05.png" width="400" /></a></div>
<br />
6. When you launch a new instance, it will be showed in your "Amazon EC2 Management Console".<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-uSCUEdOQIYw/UV7uNg3fwtI/AAAAAAAAANs/HZ61oo3aNB0/s1600/Screenshot+from+2013-04-05+11:12:31.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="223" src="http://1.bp.blogspot.com/-uSCUEdOQIYw/UV7uNg3fwtI/AAAAAAAAANs/HZ61oo3aNB0/s400/Screenshot+from+2013-04-05+11:12:31.png" width="400" /></a></div>
<br />
<br /></div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242tag:blogger.com,1999:blog-6087794019829505601.post-38903780370333887862013-03-15T13:39:00.005-07:002013-03-15T14:25:42.956-07:00Airavata Deployment Studio (ADS)<div dir="ltr" style="text-align: left;" trbidi="on">
<span style="text-align: justify;"><br /></span>
<span style="text-align: justify;">This is an independent study that I have been doing for Apache Airavata [<a href="http://airavata.apache.org/">1</a>]. Airavata Deployment Studio or simply ADS, is a platform where an Airavata user can deploy his/her Airavata deployment on a Cloud computing resource on demand. Now let's dive into ADS and what's the actual problem that we are trying the solve here. </span><br />
<div style="text-align: justify;">
<br /></div>
<div style="text-align: center;">
<iframe allowfullscreen="" frameborder="0" height="356" marginheight="0" marginwidth="0" mozallowfullscreen="" scrolling="no" src="http://www.slideshare.net/slideshow/embed_code/17238354" style="border-width: 1px 1px 0; border: 1px solid #CCC; margin-bottom: 5px;" webkitallowfullscreen="" width="427"> </iframe> </div>
<div style="margin-bottom: 5px;">
<br /></div>
<h3 style="text-align: left;">
What is Airavata? </h3>
<div>
<br /></div>
<div style="text-align: justify;">
Airavata is a framework which enables a user to build Science Gateways. It is used to compose, manage, execute and monitor distributed applications and workflows on computational resources. These computational resources can range from local resources to computational grids and clouds. Therefore, various users with different backgrounds either contribute or use Airavata in their applications.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-fO1zo9uzB_I/UUOH9CxnzVI/AAAAAAAAAM4/5naUTshzCzs/s1600/user1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="284" src="http://4.bp.blogspot.com/-fO1zo9uzB_I/UUOH9CxnzVI/AAAAAAAAAM4/5naUTshzCzs/s640/user1.png" width="640" /></a></div>
<br /></div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: left;">
Who uses Airavata? </h3>
<div style="text-align: justify;">
From the Airavata standpoint, three main users can be identified.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-PTb91g49oNM/UUOIBk2KwfI/AAAAAAAAANM/WmoRdC8yl8o/s1600/user4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="385" src="http://1.bp.blogspot.com/-PTb91g49oNM/UUOIBk2KwfI/AAAAAAAAANM/WmoRdC8yl8o/s640/user4.png" width="640" /></a></div>
<br /></div>
<h4 style="text-align: left;">
1) End Users</h4>
<div>
<br /></div>
<div style="text-align: justify;">
End User is the one who will have a model code to do some scientific application. Sometimes this End User can be a Research Scientist. He/She writes scripts to wrap the applications up and by executing those scripts, they run the scientific workflows in Super Computers. This can be called a scientific experiment.<br />
<br /></div>
<h4 style="text-align: left;">
2) Gateway Developers</h4>
<div>
<br /></div>
<div style="text-align: justify;">
The Research Scientist is the one who comes up with requirement of bundling scientific applications together and composing as a workflow. The job of the Gateway Developer is to use Airavata and wrap the above mentioned model code and scripts together. Then, scientific workflows are created out these. In some cases, Scientist might be the Gateway Developer as well.<br />
<br /></div>
<h4 style="text-align: left;">
3) Core Developers</h4>
<div>
<br /></div>
<div style="text-align: justify;">
Core Developer is the one who develops and contributes to Airavata framework code-base. The Gateway Developers use the software developed by the Core Developers to create science gateways.</div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: left;">
Why ADS?</h3>
<div style="text-align: justify;">
According to the above description, Airavata is used by different people with different technical backgrounds. Some people will have in depth technical knowledge on their scientific domains; like chemistry, biology, astronomy, etc and may not have in depth knowledge on computer science aspects such as cluster configuration, configuring and trouble-shooting in VMs, etc. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
When it comes to ADS, it's targeted towards the first two types of users as they will be ones who will be running in to configuration issues with Airavata in their respective systems. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
Sometimes we come across instances where a user might run into issues while setting up Airavata on their Systems. These might be attributed to; </div>
<div style="text-align: justify;">
</div>
<ol>
<li>User not following the documented steps properly.</li>
<li>Issues in setting up the user environment. </li>
<li>User not being able to diagnose the issues at their end on their own.</li>
<li>Sometimes when we try to diagnose their issue remotely, we face difficulties trying to access user's VM remotely due to security policies defined in their System. </li>
<li>Different security policies at client's firewall.</li>
</ol>
<br />
<div style="text-align: justify;">
Due to the above mentioned issues, a first time user might go away with a bad impression due to a System/VM level issue that might not be directly related to Airavata. </div>
<div style="text-align: justify;">
<br /></div>
<div style="text-align: justify;">
What we are trying to do here is to give a first time user a good first impression as well as ease of configuring the Airavata eco system for production usage. </div>
<div style="text-align: justify;">
<br /></div>
<h3 style="text-align: left;">
How? </h3>
<div style="text-align: justify;">
Now you might be wondering how does ADS achieve this? ADS will use FutureGrid [<a href="https://portal.futuregrid.org/about">3</a>] as the underlying resource platform for this application. If you are interested in learning about what FutureGrid is, please refer [<a href="https://portal.futuregrid.org/about">3</a>] for more information. ADS will ultimately become a plugin to the FutureGrid's CloudMesh [<a href="http://cloudmesh.blogspot.com/">4</a>] environment.<br />
<br />
ADS will provide a user with a web interface which a user can use to configure his/her Airavata eco system. Once the configuration options are selected and user hits the submit button, a new VM with the selected configurations will be created. The user will be able to create his/her image with the following properties. </div>
<div style="text-align: justify;">
</div>
<ul>
<li>Infrastructure - eg: OpenStack, Eucalyptus, EC2, etc</li>
<li>Architecture - eg: 64-bit, 32-bit </li>
<li>Memory - eg: 2GB, 4GB, 8GB, etc</li>
<li>Operating System - eg: Ubuntu, CentOS, Fedora, etc</li>
<li>Java version - eg: Java 1.6, Java 1.7</li>
<li>Tomcat Version - eg: Tomcat6, Tomcat7</li>
<li>Airavata Version - eg: Airavata-0.6, Airavata-0.7</li>
</ul>
<div>
<br /></div>
<h3 style="text-align: left;">
<b>Advantages?</b></h3>
<div style="text-align: justify;">
</div>
<ol>
<li>One click install. </li>
<li>No need to interact with the shell to configure an Airavata environment.</li>
<li>Deploying on various Cloud platforms based on user preference.</li>
<li>Ease of use. </li>
<li>First time user will be able to quickly configure an insatnce of his own and run a sample workflow quickly. </li>
<li>On demand aspect.</li>
</ol>
<div>
<br /></div>
<h3 style="text-align: left;">
<b>Sneak Peak</b></h3>
<div style="text-align: justify;">
Following screenshots show how ADS will look like.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-Et1XZ0-R760/UUOGq_2PSOI/AAAAAAAAAMc/-u8DEBGCtHQ/s1600/Screenshot+from+2013-02-21+21:24:29.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="359" src="http://4.bp.blogspot.com/-Et1XZ0-R760/UUOGq_2PSOI/AAAAAAAAAMc/-u8DEBGCtHQ/s640/Screenshot+from+2013-02-21+21:24:29.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-ICuHzvBhnOA/UUOGrKTfKFI/AAAAAAAAAMk/rOFPKUszaOw/s1600/Screenshot+from+2013-02-22+17:32:22.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="http://1.bp.blogspot.com/-ICuHzvBhnOA/UUOGrKTfKFI/AAAAAAAAAMk/rOFPKUszaOw/s640/Screenshot+from+2013-02-22+17:32:22.png" width="640" /></a></div>
<br /></div>
<div style="text-align: justify;">
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://1.bp.blogspot.com/-KO60SXEFfD0/UUOGqzlwx3I/AAAAAAAAAM0/QTLAjmW_czQ/s1600/Screenshot+from+2013-02-21+21%253A27%253A16.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="http://1.bp.blogspot.com/-KO60SXEFfD0/UUOGqzlwx3I/AAAAAAAAAM0/QTLAjmW_czQ/s640/Screenshot+from+2013-02-21+21%253A27%253A16.png" width="640" /></a></div>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="http://4.bp.blogspot.com/-F2GSPpEX5l8/UUOGq2JvRcI/AAAAAAAAAMo/K8e-GP1WC18/s1600/Screenshot+from+2013-02-21+21%253A27%253A22.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="358" src="http://4.bp.blogspot.com/-F2GSPpEX5l8/UUOGq2JvRcI/AAAAAAAAAMo/K8e-GP1WC18/s640/Screenshot+from+2013-02-21+21%253A27%253A22.png" width="640" /></a></div>
<br /></div>
<h3 style="text-align: left;">
<b>References </b></h3>
<div style="text-align: justify;">
[<a href="http://airavata.apache.org/">1</a>] - <a href="http://airavata.apache.org/">http://airavata.apache.org</a></div>
<div style="text-align: justify;">
[<a href="http://airavata.apache.org/architecture/airavata-stakeholders.html">2</a>] - <a href="http://airavata.apache.org/architecture/airavata-stakeholders.html">http://airavata.apache.org/architecture/airavata-stakeholders.html</a></div>
<div style="text-align: justify;">
[<a href="https://portal.futuregrid.org/about">3</a>] - <a href="https://portal.futuregrid.org/about">https://portal.futuregrid.org/about</a><br />
[<a href="http://cloudmesh.blogspot.com/">4</a>] - <a href="http://cloudmesh.blogspot.com/">http://cloudmesh.blogspot.com</a></div>
<div style="text-align: justify;">
<br /></div>
</div>
Heshan Suriyaarachchihttp://www.blogger.com/profile/07333563024142418173noreply@blogger.com0Bloomington, IN, USA39.165325 -86.52638569999999239.066843500000004 -86.68774719999999 39.2638065 -86.3650242