Learning about Talend and AWS is always fun and the way it is interacted is also full of fun. In this post we will see how run the talend job by deploying them in lambda function and run the lambda function using Serverless framework.

Lets get started with the prerequisites which are very important to start the topic:

Prerequisites

Installation of Node JS :

Lets Download the Node JS binaries from (https://nodejs.org/en/) site and install.
Node JS installation will install both node JS runtime and npm (node package manager)
NPM is used to install packages.

For linux we have to use command as

<code

yum install-y gcc-c++make
curl -sL https://rpm.nodesource.com/setup_6.x |sudo-Ebash
yum install nodejs
node -v

For windows we have to install like : https://nodesource.com/blog/installing-nodejs-tutorial-windows/

Installation of Apache Maven:

As we are done with the Nodejs installation we will start with apache maven

Following are the ways how we can install apache maven :

In Windows :

  • Go to the link :https://maven.apache.org/download.cgi unzip it and add the bin path to the environment variables like :
    • PATH=C:\apache-maven-3.6.0-bin\apache-maven-3.6.0\bin
    • We can verify if it is installed by checking mvn -version [Which will give us the version of the maven which is installed.]

In Linux :

  • wget http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
  • Run the wget command from the dir you want to extract maven too.
  • run the following to extract the tar,
    tar xvf apache-maven-3.0.5-bin.tar.gz
  • move maven to /usr/local/apache-maven
    mv apache-maven-3.0.5  /usr/local/apache-maven
  • Next add the env variables to your ~/.bashrc fileexport M2_HOME=/usr/local/apache-maven
    export M2=$M2_HOME/bin
    export PATH=$M2:$PATH
  • Execute these commands
    source ~/.bashrc
  • Verify everything is working with the following commandmvn -version

Installation of Oracle JDK

Serverless configuration

We will need to install serverless and add our credentials in config which will help us to connect with the aws

  1. Run the below command to install Serverless globally

    serverless-stack-output is a plugin, aws-sdk is used to call Batch jobs and to install other dependencies using npm install

    npm install -g serverless
    npm install
  2. serverless config credentials

sudo serverless config credentials –provider aws –key <ACCESS_KEY> –secret <SECRET_KEY>

Serverless installation of the AWS Lambda function

This will create an AWS Lambda function for the talend job using the serverless.

  • First we will make empty directory named as : serverless-lambda-talendjobname
  • Now in Visual code we will browse to the folder as cd serverless-lambda-talendjobname
  • Now we will create a serverless java maven template as
     serverless create –template aws-java-maven
  • Now we have to install the talend related directories for that we have one zip attached as Supporting_Talend_Jobs_For Serverless
  • This zip will help us to install all the lib files and will also generate the pom.xml for us which we can use it for the serverless_Project.
  • Following will be the generic folder structure which we will follow :
  • Following are the details of all the folders and zips present in the above image
    • serverless_project : We can give the generic name to the serverless_project which will be parent directory and where we will place all the related folders and zips .
    • code : This will contain the serverless code which we will install and deploy
    • lib : This will contain all the libs which are present and which needs to do mvn install
    • Supporting_Talend_Job_For_Serverless : This is the attached zip which is used to install the libs of the talend projects
    • TalendProject : This is the talend project for which we have to create the lambda function
  •  Working Procedure of Supporting_Talend_Job_For_Serverless :
    • 1st we have to unzip the Supporting_Talend_Job_For_Serverless and also unzip the TalendProject
    • TalendProject have jars at 2 places :
      1. TalendProject\lib
      2. TalendProject\TalendProject\lib
      3. We will place this jars in the lib folder which is shown in above image
    • Then we will run the bat file present in the directory D:\serverless_project\Supporting_Talend_Job_For_Serverless_0.1\Supporting_Talend_Job_For_Serverless\Supporting_Talend_Job_For_Serverless_run.bat which will do all the installation of the jars and also will generate the pom.xml file named as pom_generated_by_talend.xml
  • We will now replace the pom.xml file created in the code directory from the template to the one which is generated by talend
  • We will change the Handler.java as per the code like below
    public ApiGatewayResponse handleRequest(Map<String, Object> input, Context context) {
            LOG.info(“received: {}”, input);
            testproject.newjob_0_1.newjob t2 = new testproject.newjob_0_1.newjob();
            String[] context2 ={};
            t2.runJob(context2);
            Response responseBody = new Response(“Success”,input);
            return ApiGatewayResponse.builder()
                .setStatusCode(200)
                .setObjectBody(responseBody)
                .setHeaders(Collections.singletonMap(“X-Powered-By”, “AWS Lambda & serverless”))
                .build();
    }
  • Now we will try to install the dependencies as :

    mvn clean install
  • When the build will be success we will deploy the sls as :

    sls deploy
  • We can now check the function by invoking it as :
    serverless invoke –function <functionname> -l

And we are done if we get any error then we can check the above steps .

Leave a Reply

Your email address will not be published. Required fields are marked *