Workflow Management

Also available as:

PDF

Contents

Hortonworks Data Platform

Workflow Management

Copyright © 2012-2018 Hortonworks, Inc.

Except where otherwise noted, this document is licensed under Creative Commons Attribution ShareAlike 4.0 License

http://creativecommons.org/licenses/by-sa/4.0/legalcode

The Hortonworks Data Platform, powered by Apache Hadoop, is a massively scalable and 100% open source platform for storing, processing and analyzing large volumes of data. It is designed to deal with data from many sources and formats in a very quick, easy and cost-effective manner. The Hortonworks Data Platform consists of the essential set of Apache Hadoop projects including MapReduce, Hadoop Distributed File System (HDFS), HCatalog, Pig, Hive, HBase, ZooKeeper and Ambari. Hortonworks is the major contributor of code and patches to many of these projects. These projects have been integrated and tested as part of the Hortonworks Data Platform release process and installation and configuration tools have also been included.

Unlike other providers of platforms built using Apache Hadoop, Hortonworks contributes 100% of our code back to the Apache Software Foundation. The Hortonworks Data Platform is Apache-licensed and completely open source. We sell only expert technical support, training and partner-enablement services. All of our technology is, and will remain, free and open source.

Please visit the Hortonworks Data Platform page for more information on Hortonworks technology. For more information on Hortonworks services, please visit either the Support or Training page. You can contact us directly to discuss your specific needs.

Contents

1. Workflow Manager Basics

Workflow Manager Design Component
Workflow Manager Dashboard Component
Workflow Action Types

2. Content Roadmap for Workflow Manager

3. Quick Start

Configuring a Workflow Manager Ambari View

UI Elements of Workflow Manager

Understanding the Design Component

Using the Design Component

Using the Dashboard Component

Dashboard Job Details Tabs

Details Available for Workflow, Coordinator, and Bundle Jobs
Details Available Only for Workflow Jobs
Details Available Only for Coordinator Jobs
Details Available Only for Bundle Jobs

4. Designing Workflows Using the Design Component

Create a Workflow

Add Nodes to a Workflow

Add Action Nodes
Add Fork Control Nodes
Add Decision Control Nodes

Save a Workflow Draft

Validate and Save a Workflow

Import an Existing Workflow

Export a Workflow

Submitting and Executing a Workflow

Submit and Execute an Existing Workflow
Submit and Execute a New Workflow
Execute a Submitted Workflow

Modifying Workflows

Copy or Move Nodes on a Workflow Graph
Remove Nodes from a Workflow Graph
Resize and Move Graph Images

Reusing Node Configurations as Assets

Save an Ambari Database Asset to Use Within the WFM Instance
Save an Asset to Shared Storage to Use Across WFM Instances
Import an Asset from the Ambari Database Linked to a Single WFM Instance
Import a Shared Asset File from Shared Storage
Managing Assets

Creating Coordinators and Bundles

Create a Coordinator
Create a Bundle

View the XML Code

5. Monitoring Jobs Using the Dashboard

Verify the Status of a Job

View Job Details and Logs

Identify the Location of a Job XML File

Troubleshooting Job Errors

Basic Job Troubleshooting
Search the Log Output
Open an Existing Workflow to Edit

6. Sample ETL Use Case

Configure the Cluster

Create an HDFS Directory for Each New User
Create a Proxy User
Copy Files

Create and Submit the Workflow

Access the Workflow Manager View
Create the Sqoop Action to Extract Data
Create the Hive Action to Transform the Data
Create the Sqoop Action to Load the Data
Submit and Execute the Workflow Job

Monitor the Workflow Job

7. Workflow Parameters

Hive Action Parameters
Hive2 Action Parameters
Sqoop Action Parameters
Pig Action Parameters
Sub-Workflow Action Parameters
Java Action Parameters
Shell Action Parameters
DistCp Action Parameters
Map-Reduce (MR) Action Parameters
SSH Action Parameters
Spark Action Parameters
File System (FS) Action Parameters
Submit Dialog Parameters

8. Settings Menu Parameters

Global Configuration Parameters
Workflow Credentials Parameters
SLA for Workflow Parameters
Workflow Parameters
Workflow and Action Versions
How SLA Works

9. Job States

10. Workflow Manager Files

List of Tables

2.1. Workflow Manager Content roadmap
7.1. Hive Action, General Parameters
7.2. Hive Action, Transition Parameters
7.3. Hive Action, Advanced Properties Parameters
7.4. Hive Action, Configuration Parameters
7.5. Hive2 Action, General Parameters
7.6. Hive2 Action, Transition Parameters
7.7. Hive2 Action, Advanced Properties Parameters
7.8. Hive2 Action, Configuration Parameters
7.9. Sqoop Action, General Parameters
7.10. Sqoop Action, Transition Parameters
7.11. Sqoop Action, Advanced Properties Parameters
7.12. Sqoop Action, Configuration Parameters
7.13. Pig Action, General Parameters
7.14. Pig Action, Transition Parameters
7.15. Pig Action, Advanced Properties Parameters
7.16. Pig Action, Configuration Parameters
7.17. Sub-Workflow Action, General Parameters
7.18. Sub-Workflow Action, Configuration Parameters
7.19. Sub-Workflow Action, Transition Parameters
7.20. Java Action, General Parameters
7.21. Java Action, Transition Parameters
7.22. Java Action, Advanced Properties Parameters
7.23. Java Action, Configuration Parameters
7.24. Shell Action, General Parameters
7.25. Shell Action, Transition Parameters
7.26. Shell Action, Advanced Properties Parameters
7.27. Shell Action, Configuration Parameters
7.28. DistCp Action, General Parameters
7.29. DistCp Action, Transition Parameters
7.30. DistCp Action, Advanced Properties Parameters
7.31. DistCp Action, Configuration Parameters
7.32. MR Action, General Parameters
7.33. MR Action, Transition Parameters
7.34. MR Action, Advanced Properties Parameters
7.35. MR Action, Configuration Parameters
7.36. SSH Action, General Parameters
7.37. SSH Action, Transition Parameters
7.38. Spark Action, General Parameters
7.39. Spark Action, Transition Parameters
7.40. Spark Action, Advanced Properties Parameters
7.41. Spark Action, Configuration Parameters
7.42. FS Action, General Parameters
7.43. FS Action, Transition Parameters
7.44. FS Action, Advanced Properties Parameters
7.45. FS Action, Configuration Parameters
7.46. Submit Dialog, Configuration Parameters
8.1. Global Configuration Parameters
8.2. HCat Workflow Credentials Parameters
8.3. Hive2 Workflow Credentials Parameters
8.4. HBase Workflow Credentials Parameters
8.5. SLA for Workflows Parameters
8.6. Workflow Configuration Parameters