Heshan's Blog: April 2013

Wednesday, April 17, 2013

Apache Airavata 0.7 Released

The Apache Airavata PMC is pleased to announce the immediate availability of the Airavata 0.7 release.

The release can be obtained from the Apache Airavata download page - http://airavata.apache.org/about/downloads.html

Release notes are available at - https://svn.apache.org/repos/asf/airavata/tags/airavata-0.7/RELEASE_NOTES

Apache Airavata is a software framework providing API’s, sophisticated server-side tools, and graphical user interfaces to construct, execute, control and manage long running applications and workflows on distributed computing resources. Apache Airavata builds on general concepts of service oriented computing, distributed messaging, and workflow composition and orchestration.

For general information on Apache Airavata, please visit the project website: http://airavata.apache.org/

Friday, April 5, 2013

Run EC2 Jobs with Airavata - Part III

This is a followup to my earlier posts [1] [2]. Here we will execute the application mentioned in [2] programmetically using Airavata.

import org.apache.airavata.commons.gfac.type.*;
import org.apache.airavata.gfac.GFacAPI;
import org.apache.airavata.gfac.GFacConfiguration;
import org.apache.airavata.gfac.GFacException;
import org.apache.airavata.gfac.context.security.AmazonSecurityContext;
import org.apache.airavata.gfac.context.ApplicationContext;
import org.apache.airavata.gfac.context.JobExecutionContext;
import org.apache.airavata.gfac.context.MessageContext;
import org.apache.airavata.schemas.gfac.*;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;

import java.io.File;
import java.net.URL;
import java.util.ArrayList;
import java.util.List;

/**
 * Your Amazon instance should be in a running state before running this test.
 */
public class EC2ProviderTest {
    private JobExecutionContext jobExecutionContext;

    private static final String hostName = "ec2-host";

    private static final String hostAddress = "ec2-address";

    private static final String sequence1 = "RR042383.21413#CTGGCACGGAGTTAGCCGATCCTTATTCATAAAGTACATGCAAACGGGTATCCATA" +
            "CTCGACTTTATTCCTTTATAAAAGAAGTTTACAACCCATAGGGCAGTCATCCTTCACGCTACTTGGCTGGTTCAGGCCTGCGCCCATTGACCAATATTCCTCA" +
            "CTGCTGCCTCCCGTAGGAGTTTGGACCGTGTCTCAGTTCCAATGTGGGGGACCTTCCTCTCAGAACCCCTATCCATCGAAGACTAGGTGGGCCGTTACCCCGC" +
            "CTACTATCTAATGGAACGCATCCCCATCGTCTACCGGAATACCTTTAATCATGTGAACATGCGGACTCATGATGCCATCTTGTATTAATCTTCCTTTCAGAAG" +
            "GCTGTCCAAGAGTAGACGGCAGGTTGGATACGTGTTACTCACCGTGCCGCCGGTCGCCATCAGTCTTAGCAAGCTAAGACCATGCTGCCCCTGACTTGCATGT" +
            "GTTAAGCCTGTAGCTTAGCGTTC";

    private static final String sequence2 = "RR042383.31934#CTGGCACGGAGTTAGCCGATCCTTATTCATAAAGTACATGCAAACGGGTATCCATA" +
            "CCCGACTTTATTCCTTTATAAAAGAAGTTTACAACCCATAGGGCAGTCATCCTTCACGCTACTTGGCTGGTTCAGGCTCTCGCCCATTGACCAATATTCCTCA" +
            "CTGCTGCCTCCCGTAGGAGTTTGGACCGTGTCTCAGTTCCAATGTGGGGGACCTTCCTCTCAGAACCCCTATCCATCGAAGACTAGGTGGGCCGTTACCCCGC" +
            "CTACTATCTAATGGAACGCATCCCCATCGTCTACCGGAATACCTTTAATCATGTGAACATGCGGACTCATGATGCCATCTTGTATTAAATCTTCCTTTCAGAA" +
            "GGCTATCCAAGAGTAGACGGCAGGTTGGATACGTGTTACTCACCGTGCG";

    /* Following variables are needed to be set in-order to run the test. Since these are account specific information,
       I'm not adding the values here. It's the responsibility of the person who's running the test to update
       these variables accordingly.
       */

    /* Username used to log into your ec2 instance eg.ec2-user */
    private String userName = "";

    /* Secret key used to connect to the image */
    private String secretKey = "";

    /* Access key used to connect to the image */
    private String accessKey = "";

    /* Instance id of the running instance of your image */
    private String instanceId = "";

    @Before
    public void setUp() throws Exception {
        URL resource = GramProviderTest.class.getClassLoader().getResource("gfac-config.xml");
        assert resource != null;
        System.out.println(resource.getFile());
        GFacConfiguration gFacConfiguration = GFacConfiguration.create(new File(resource.getPath()), null, null);

        /* EC2 Host */
        HostDescription host = new HostDescription(Ec2HostType.type);
        host.getType().setHostName(hostName);
        host.getType().setHostAddress(hostAddress);

        /* App */
        ApplicationDescription ec2Desc = new ApplicationDescription(Ec2ApplicationDeploymentType.type);
        Ec2ApplicationDeploymentType ec2App = (Ec2ApplicationDeploymentType)ec2Desc.getType();

        String serviceName = "Gnome_distance_calculation_workflow";
        ec2Desc.getType().addNewApplicationName().setStringValue(serviceName);
        ec2App.setJobType(JobTypeType.EC_2);
        ec2App.setExecutable("/home/ec2-user/run.sh");
        ec2App.setExecutableType("sh");

        /* Service */
        ServiceDescription serv = new ServiceDescription();
        serv.getType().setName("GenomeEC2");

        List inputList = new ArrayList();

        InputParameterType input1 = InputParameterType.Factory.newInstance();
        input1.setParameterName("genome_input1");
        input1.setParameterType(StringParameterType.Factory.newInstance());
        inputList.add(input1);

        InputParameterType input2 = InputParameterType.Factory.newInstance();
        input2.setParameterName("genome_input2");
        input2.setParameterType(StringParameterType.Factory.newInstance());
        inputList.add(input2);

        InputParameterType[] inputParamList = inputList.toArray(new InputParameterType[inputList.size()]);

        List outputList = new ArrayList();
        OutputParameterType output = OutputParameterType.Factory.newInstance();
        output.setParameterName("genome_output");
        output.setParameterType(StringParameterType.Factory.newInstance());
        outputList.add(output);

        OutputParameterType[] outputParamList = outputList
                .toArray(new OutputParameterType[outputList.size()]);

        serv.getType().setInputParametersArray(inputParamList);
        serv.getType().setOutputParametersArray(outputParamList);

        jobExecutionContext = new JobExecutionContext(gFacConfiguration,serv.getType().getName());
        ApplicationContext applicationContext = new ApplicationContext();
        jobExecutionContext.setApplicationContext(applicationContext);
        applicationContext.setServiceDescription(serv);
        applicationContext.setApplicationDeploymentDescription(ec2Desc);
        applicationContext.setHostDescription(host);

        AmazonSecurityContext amazonSecurityContext =
                new AmazonSecurityContext(userName, accessKey, secretKey, instanceId);
        jobExecutionContext.addSecurityContext(AmazonSecurityContext.AMAZON_SECURITY_CONTEXT, amazonSecurityContext);

        MessageContext inMessage = new MessageContext();
        ActualParameter genomeInput1 = new ActualParameter();
        ((StringParameterType)genomeInput1.getType()).setValue(sequence1);
        inMessage.addParameter("genome_input1", genomeInput1);

        ActualParameter genomeInput2 = new ActualParameter();
        ((StringParameterType)genomeInput2.getType()).setValue(sequence2);
        inMessage.addParameter("genome_input2", genomeInput2);

        MessageContext outMessage = new MessageContext();
        ActualParameter echo_out = new ActualParameter();
        outMessage.addParameter("distance", echo_out);

        jobExecutionContext.setInMessageContext(inMessage);
        jobExecutionContext.setOutMessageContext(outMessage);
    }

    @Test
    public void testGramProvider() throws GFacException {
        GFacAPI gFacAPI = new GFacAPI();
        gFacAPI.submitJob(jobExecutionContext);
        MessageContext outMessageContext = jobExecutionContext.getOutMessageContext();
        Assert.assertEquals(MappingFactory.
                toString((ActualParameter) outMessageContext.getParameter("genome_output")), "476");
    }
}

References
[1] - http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html
[2] - http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-ii.html

Run EC2 Jobs with Airavata - Part II

In this post we will look at how to compose a workflow out of an application that is installed in an Amazon Machine Image (AMI). In the earlier post we discussed how to do ec2 instance management using XBaya GUI. This is the followup to that post.

For the Airavata EC2 integration testing, I created an AMI which has an application which does gene sequence alignment using Smith-Waterman algorithm. I will be using that application as a reference to this post. You can use an application of your preference that resides in your AMI.

1. Unzip Airavata server distribution and start the server.

unzip apache-airavata-server-0.7-bin.zip
cd apache-airavata-server-0.7/bin
./airavata-server.sh

2. Unzip Airavata XBaya distribution and start XBaya.

unzip apache-airavata-xbaya-gui-0.7-bin.zip
cd apache-airavata-xbaya-gui-0.7/bin
./xbaya-gui.sh

Then you'll get the XBaya UI.

3. Select "XBaya" Menu and click "Add Host" to register an EC2 Host. Once you add the details, click "ok".

4. You will then be prompted to enter "Airavata Registry" information. If you are using the default setup, you don't have to do any configuration. Just click "ok".

5. In order to use your application installed in the AMI, you must register it as an application in Airavata system. Select "XBaya" menu and click "Register Application". You will get the following dialog. Add the input parameters expected and the output parameters generated by your application.

6. Then Click the "New deployment" button. You have to then select the EC2Host that you registered earlier as the Application Host. Configure the executable path to your application in your AMI and click "Add".

7. Then click "Register". If the application registration was successful, you will be getting the following message.

8. Now select "Registry" menu and click "Setup Airavata Registry". Click "ok".

9. Select "XBaya" menu and click "New workflow". Then configure it accordingly.

10. Select your registered application from the "Application Services" and drag drop it to the workflow window.

11. Drag an "Instance" component from "Amazon Components" and drop it into workflow window. Then connect it to your application using Control ports.

12. Click on top of the "Instance" components config label. Configure your instance accordingly.

13. Drag and drop two input components and one output component to the workflow from "System Components".

14. Connect the components together accordingly.

15. Now click the red colored "play" button to run your workflow. You will be prompted for the inputs values (in my case the gene sequences) and experiment id. Then click "Run" to execute your workflow.

16. The execution result will be shown in the XBaya GUI.

References
[1] - http://heshans.blogspot.com/2013/04/run-ec2-jobs-with-airavata-part-i.html

Run EC2 Jobs with Airavata - Part I

This will be the first of many posts that I will be doing on Apache Airavata EC2 integration. First let's have a look at how you can use Airavata's "XBaya GUI" to manage amazon instances.

Applies to : Airavata 0.7 and above

1. Unzip Airavata server distribution and start the server.

unzip apache-airavata-server-0.7-bin.zip
cd apache-airavata-server-0.7/bin
./airavata-server.sh

2. Unzip Airavata XBaya distribution and start XBaya.

unzip apache-airavata-xbaya-gui-0.7-bin.zip
cd apache-airavata-xbaya-gui-0.7/bin
./xbaya-gui.sh

Then you'll get the XBaya UI.

3. Then Select "Amazon" menu and click "Security Credentials". Specify your secret key and access key in the security credentials dialog box and click "ok".