List of Pages in Category YARN (40 pages)
YARN (Yet Another Resource Negotiator)
A general architecture for running distributed applications. YARN specifies the following components:
- ResourceManager - A master daemon that authorizes submitted jobs to run, assigns an ApplicationMaster to them, and enforces resource limits.
- ApplicationMaster - A supervisory task that requests the resources needed for executor tasks. An ApplicationMaster runs on a different NodeManager for each application. The ApplicationMaster requests containers, which are sized by the resources a task requires to run.
- NodeManager - A worker daemon that launches and monitors the ApplicationMaster and task containers.
- JobHistory Server - Keeps track of completed applications.
The ApplicationMaster negotiates with the ResourceManager for cluster resources—described in terms of a number of containers, each with a certain memory limit—and then runs application-specific processes in those containers. The containers are overseen by NodeManagers running on cluster nodes, which ensure that the application does not use more resources than it has been allocated.
MapReduce v2 (MRv2) is implemented as a YARN application.
*
C
- Cluster Utilization Reports
- Configuring ADLS Connectivity for CDH
- Configuring Oozie
- Configuring Oozie to Enable MapReduce Jobs To Read/Write from Amazon S3
- Configuring Oozie to Enable MapReduce Jobs To Read/Write from Microsoft Azure (ADLS)
- Configuring Spark on YARN for Long-Running Applications
- Configuring the Fair Scheduler
- Configuring TLS/SSL for HDFS, YARN and MapReduce
- Configuring YARN Security
- ContainerExecutor Error Codes (YARN)
- Creating a Custom Cluster Utilization Report
)D
E
M
- Managing YARN
- Managing YARN (MRv2) and MapReduce (MRv1)
- MapReduce (MRv1) and YARN (MRv2) High Availability
- Migrating from MapReduce (MRv1) to MapReduce (MRv2)
- Migrating from MapReduce (MRv1) to MapReduce (MRv2)
- Monitoring YARN Applications
N
R
- REQUEST_POOL Query Option
- Resource Management for Impala
- ResourceManager Metrics
- Running Spark Applications on YARN
S
- Setting HADOOP_MAPRED_HOME for Apache Hive in CDH
- Setting Up Apache Sqoop 2 Using the Command Line
- Setting Up Apache Sqoop Using the Command Line
- Setting Up MapReduce v2 with YARN Using the Command Line
- Step 16: Configure Either MRv1 Security or YARN Security
- Step 2: Verify User Accounts and Groups in CDH 5 Due to Security
T
U
V
W
Y