Airflow is a platform to programmatically author, schedule and monitor workflows.
We use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies, we have heard about airflow but we never knew how to get it working for the talend jobs and it will be very easy as we will have the UI for scheduling and monitoring the working flow of the talend jobs.
Lets check our To Do to achieve the goal
- Launching the instance of the Ec2 : We will be launching the ubuntu server for the installation of the airflow and also for copying the talend jobs in the server
- Installing Airflow in Ec2 instance : We will follow the steps for the installation of the airflow and get the webserver of the airflow working
- Adding of the talend job and creating DAGs file
Launching an ec2 instance in aws.
We will launch ubuntu 16.04 instance for airflow