Learning about Talend and AWS is always fun and the way it is interacted is also full of fun. In this post we will see how run the talend job by deploying them in lambda function and run the lambda function using Serverless framework.
Lets get started with the prerequisites which are very important to start the topic:
Prerequisites
- Node JS(https://nodejs.org/en/) (You can check appendix for more details)
- Apache Maven
- Serverless (https://serverless.com/framework/docs/providers/aws/guide/quick-start/)
- AWS Access Key and Secret Key
- Visual Studio Code
- Oracle JDK
Installation of Node JS
Lets Download the Node JS binaries from (https://nodejs.org/en/) site and install.
Node JS installation will install both node JS runtime and npm (node package manager)
NPM is used to install packages.
For linux we have to use command as
<code
yum install-y gcc-c++make
curl -sL https://rpm.nodesource.com/setup_6.x |sudo-Ebash
yum install nodejs
node -v
For windows we have to install like : https://nodesource.com/blog/installing-nodejs-tutorial-windows/
Installation of Apache Maven
As we are done with the Nodejs installation we will start with apache maven
Following are the ways how we can install apache maven :
In Windows :
- Go to the link :https://maven.apache.org/download.cgi unzip it and add the bin path to the environment variables like :
- PATH=C:\apache-maven-3.6.0-bin\apache-maven-3.6.0\bin
- We can verify if it is installed by checking mvn -version [Which will give us the version of the maven which is installed.]
In Linux :
- wget http://mirror.olnevhost.net/pub/apache/maven/maven-3/3.0.5/binaries/apache-maven-3.0.5-bin.tar.gz
- Run the wget command from the dir you want to extract maven too.
- run the following to extract the tar,
tar xvf apache-maven-3.0.5-bin.tar.gz - move maven to /usr/local/apache-maven
mv apache-maven-3.0.5 /usr/local/apache-maven - Next add the env variables to your ~/.bashrc fileexport M2_HOME=/usr/local/apache-maven
export M2=$M2_HOME/bin
export PATH=$M2:$PATH - Execute these commands
source ~/.bashrc - Verify everything is working with the following commandmvn -version
Installation of Oracle JDK
We can do the installation of Oracle JDK as follows :
In Windows
- Install the jdk 8 from the official website :
- https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
- We will need to set the environment variable of the jdk :
- JAVA_HOME=C:\Program Files\Java\jdk1.8.0_181
- PATh =%JAVA_HOME%\bin
- In Linux
- wget –no-check-certificate –no-cookies –header “Cookie: oraclelicense=accept-securebackup-cookie” http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.rpm
- sudo yum install -y jdk-8u141-linux-x64.rpm
- java -version
Serverless configuration
We will need to install serverless and add our credentials in config which will help us to connect with the aws
- Run the below command to install Serverless globallyserverless-stack-output is a plugin, aws-sdk is used to call Batch jobs and to install other dependencies using npm installnpm install -g serverlessnpm install
- serverless config credentials
sudo serverless config credentials –provider aws –key <ACCESS_KEY> –secret <SECRET_KEY>
Serverless installation of the AWS Lambda function
- This will create an AWS Lambda function for the talend job using the serverless.
- First we will make empty directory named as : serverless-lambda-talendjobname
- Now in Visual code we will browse to the folder as cd serverless-lambda-talendjobname
- Now we will create a serverless java maven template as
serverless create –template aws-java-maven - Now we have to install the talend related directories for that we have one zip attached as Supporting_Talend_Jobs_For Serverless
Supporting_Talend_Job_For_Serverless_0.1.zip - This zip will help us to install all the lib files and will also generate the pom.xml for us which we can use it for the serverless_Project.
- Following will be the generic folder structure which we will follow :
- serverless_project : We can give the generic name to the serverless_project which will be parent directory and where we will place all the related folders and zips .
- code : This will contain the serverless code which we will install and deploy
- lib : This will contain all the libs which are present and which needs to do mvn install
- Supporting_Talend_Job_For_Serverless : This is the attached zip which is used to install the libs of the talend projects
- TalendProject : This is the talend project for which we have to create the lambda function
- Working Procedure of Supporting_Talend_Job_For_Serverless :
- 1st we have to unzip the Supporting_Talend_Job_For_Serverless and also unzip the TalendProject
- TalendProject have jars at 2 places :
1. TalendProject\lib
2. TalendProject\TalendProject\lib
3. We will place this jars in the lib folder which is shown in above image - Then we will run the bat file present in the directory D:\serverless_project\Supporting_Talend_Job_For_Serverless_0.1\Supporting_Talend_Job_For_Serverless\Supporting_Talend_Job_For_Serverless_run.bat which will do all the installation of the jars and also will generate the pom.xml file named as pom_generated_by_talend.xml
- We will now replace the pom.xml file created in the code directory from the template to the one which is generated by talend
- We will change the Handler.java as per the code like below
public ApiGatewayResponse handleRequest(Map<String, Object> input, Context context) { LOG.info(“received: {}”, input);
testproject.newjob_0_1.newjob t2 = new testproject.newjob_0_1.newjob();
String[] context2 ={};
t2.runJob(context2);
Response responseBody = new Response(“Success”,input);
return ApiGatewayResponse.builder()
.setStatusCode(200)
.setObjectBody(responseBody)
.setHeaders(Collections.singletonMap(“X-Powered-By”, “AWS Lambda & serverless”))
.build();
}
Now we will try to install the dependencies as :
mvn clean install
When the build will be success we will deploy the sls as :
sls deploy
We can now check the function by invoking it as :
serverless invoke –function <functionname> -l
And we are done if we get any error then we can check the above steps .
Let us handle the heavy lifting and ensure your data is safe and secure throughout the process.
Benefits of Having Data Processing with ETL-Design in your Business
Serverless Lambda functions can offer several benefits for Talend jobs in a business setting. Some of these benefits include:
- Scalability: Lambda functions can be easily scaled up or down as needed, allowing for better performance and cost efficiency. This means that the function will automatically scale up when the workload increases and scale down when it decreases.
- Reduced cost: Lambda functions are only billed for the time they are running, which can result in significant cost savings compared to running a server 24/7. In addition, you don’t need to worry about maintaining servers, so you can save money on infrastructure costs.
- Faster time-to-market: By using serverless computing, you can quickly deploy your Talend jobs without the need for server configuration or management. This means you can deliver your solution faster and with less overhead.
- Improved reliability: Serverless computing is highly reliable because the cloud provider manages the infrastructure and automatically handles fault tolerance and failover. This means that you can focus on the business logic of your Talend jobs and not have to worry about infrastructure.
- Easy integration: Lambda functions can easily integrate with other AWS services and third-party tools, making it easier to create complex workflows and integrate with other applications.
Overall, using serverless Lambda functions for Talend jobs can provide a more efficient, cost-effective, and reliable solution for your business.
Complere can help
Complere is a consulting firm that can assist businesses in implementing serverless Lambda functions for Talend jobs. We provide expert guidance in architecture design, implementation, optimization, training, and support. Our team can help you design a serverless architecture, implement Lambda functions, and integrate them with other AWS services. We can also help you optimize your Lambda functions and other serverless services for cost and efficiency, as well as provide training and support for your team.
Using serverless Lambda functions for Talend jobs can provide several benefits for businesses, such as scalability, cost savings, faster time-to-market, improved reliability, and easy integration with other tools. Complere can help you leverage these benefits by providing expert guidance and support in implementing and optimizing your serverless solution. We can ensure that your solution is optimized for performance, cost, and reliability, and that your team is trained and supported in using it effectively.
Call the Complere team at 7042675588 today to learn more about our data services and how we can help you.
Complere can help
- 1
- 2
