Spark Executor on Task Node
Configure your EMR cluster to run Executors only on Spot TASK nodes and save money !
How to run Spark Executor on Task Nodes
While submitting a Spark application, we can add configuration such as 'spark.yarn.am.nodeLabelExpression' and 'spark.yarn.executor.nodeLabelExpression' to restrict the set of nodes AM and Executors will be scheduled on. However, as Task nodes are not labeled by default we cannot use this to Run Exceutors on Task Nodes. So we need to label task nodes and add node label 'Task' in YARN node labels.
- Create a TASK node label on YARN
- Add the task nodes with this label
- Update the scheduler configuration with TASK node label.
The main challenge here is, automating these changes and ensuring that it get implemented on cluster launch.
We will do this by following the below approach.
Install a script on all nodes of the cluster that shall add the TASK label after determining the NodeType .
We cannot do this using bootstrap action as this runs before the actual setup pf YARN environment. We will use the bootstrap action to download our labelling script from s3 to all nodes and execute the 'SecondStageBashScript' script after the provisioning completes.
BootstrapActionScript
#!/bin/bash -x
aws s3 cp s3://bucket/path/SecondStageBAScript.sh /home/hadoop/secondstage.sh && sudo bash /home/hadoop/secondstage.sh & exit 0#!/bin/bash
while true; do
NODEPROVISIONSTATE=` sed -n '/localInstance [{]/,/[}]/{
/nodeProvisionCheckinRecord [{]/,/[}]/ {
/status: / { p }
/[}]/a
}
/[}]/a
}' /emr/instance-controller/lib/info/job-flow-state.txt | awk ' { print $2 }'`
if [ "$NODEPROVISIONSTATE" == "SUCCESSFUL" ]; then
sleep 10;
echo "Running my post provision bootstrap"
NODETYPE=$(cat /mnt/var/lib/instance-controller/extraInstanceData.json | jq -r '.instanceRole' | awk '{print tolower($0)}')
if [ "$NODETYPE" == "master" ]
then
echo "In Master"
sudo yarn rmadmin -addToClusterNodeLabels "TASK(exclusive=false)"
CS_CONTENT="<property>\n<name>yarn.scheduler.capacity.root.accessible-node-labels.TASK.capacity</name>\n<value>100</value>\n</property>\n<property>\n<name>\nyarn.scheduler.capacity.root.default.accessible-node-labels.TASK.capacity</name>\n<value>100</value>\n</property>\n</configuration>"
sudo sed -i '$d' /etc/hadoop/conf/capacity-scheduler.xml
sudo echo -en $CS_CONTENT >> /etc/hadoop/conf/capacity-scheduler.xml
sudo stop hadoop-yarn-resourcemanager
sudo start hadoop-yarn-resourcemanager
sudo yarn rmadmin -refreshQueues
elif [ "$NODETYPE" == "task" ]
then
echo "In Task"
YARN_CONTENT="<property>\n<name>yarn.node-labels.am.default-node-label-expression</name>\n<value>CORE</value>\n</property>\n<property>\n<name>\nyarn.nodemanager.node-labels.provider</name>\n<value>config</value>\n</property>\n<property>\n<name>yarn.nodemanager.node-labels.provider.configured-node-partition</name>\n<value>TASK</value>\n</property>\n</configuration>"
sudo sed -i '$d' /etc/hadoop/conf/yarn-site.xml
sudo echo -en $YARN_CONTENT >> /etc/hadoop/conf/yarn-site.xml
sudo stop hadoop-yarn-nodemanager
sudo start hadoop-yarn-nodemanager
fi
exit;
fi
sleep 10;
doneUpload 'Second Stage' Bootstrap Action Script to S3
s3://<yourBucketName>/<path>/SecondStageBAScript.shUpdate the path of Second Stage Script.sh in the BootstrapActionScript.sh at Line 2 and upload it to s3.
s3://<yourBucketName>/<path>/BootstrapActionScript.shPass the s3 location of "BootstrapActionScript.sh" for the 'Script Location' parameter in Bootstrap Action configuration of you EMR cluster.
Once the cluster is lauched, you can check the yarn node lables using the following commands:
yarn cluster --list-node-labels
yarn node -list -all -showDetails
# Sample spark-submit commads for testing
spark-submit --conf spark.yarn.executor.nodeLabelExpression=TASK --class org.apache.spark.examples.SparkPi /usr/lib/spark/examples/jars/spark-examples.jar 500