Spark1.5.2 on yarn fair scheduler 配置
2015-11-19 15:20
671 查看
1.fair scheduler 配置
<?xml version="1.0"?> <allocations> <pool name="production"> <schedulingMode>FAIR</schedulingMode> <weight>1</weight> <minShare>2</minShare> </pool> <pool name="test"> <schedulingMode>FIFO</schedulingMode> <weight>2</weight> <minShare>3</minShare> </pool> <pool name="spark"> <schedulingMode>FAIR</schedulingMode> <weight>2</weight> <minShare>3</minShare> </pool> <pool name="ztai"> <schedulingMode>FAIR</schedulingMode> <weight>1</weight> <minShare>3</minShare> </pool> </allocations>
2.代码提交
mport org.apache.spark.SparkConf
import org.apache.spark.SparkContext
/**
* Created by root on 11/18/15.
*/
object HiveUtility {
def createTable(sc: SparkContext, db: String, table: String, system_user_name: String) = {
val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)
println("user.name " + System.getProperty("user.name"))
println("HADOOP_USER_NAME " + System.getProperty("HADOOP_USER_NAME"))
sqlContext.sql("use " + db)
sqlContext.sql("CREATE TABLE IF NOT EXISTS " + db + "." + table + "(key INT, value STRING)")
sqlContext.sql("LOAD DATA INPATH 'hdfs:///user/spark/examples/src/main/resources/kv1.txt' INTO TABLE " + db + "." + table)
sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").collect().foreach(println)
sqlContext.sql("FROM " + db + "." + table + " SELECT key, value").count()
sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 20").collect().foreach(println)
sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 5").collect().foreach(println)
sqlContext.sql("FROM " + db + "." + table + " SELECT key, value limit 15").collect().foreach(println)
}
def main(args: Array[String]) {
if (args.length < 3) {
print("useage:HiveUtility.jar db table system_user_name")
}
System.setProperty("user.name", args(2))
System.setProperty("HADOOP_USER_NAME", args(2))
val conf = new SparkConf()
// conf.set("spark.scheduler.allocation.file", "hdfs:///user/spark/fairscheduler.xml")
conf.set("spark.scheduler.mode", "FAIR") // set scheduler mode
val sc = new SparkContext(conf)
sc.setLocalProperty("spark.scheduler.pool", "spark") // set fair scheduler
createTable(sc, args(0), args(1), args(2))
}
}提交命令：
mainclass=com.suning.spark.hive.HiveUtility mainjar=HiveUtility-v1.jar hadoop fs -rm /home/spark/software/spark/examples/src/main/resources/kv1.txt hadoop fs -put /home/spark/software/spark/examples/src/main/resources/kv1.txt examples/src/main/resources export HADOOP_USER_NAME=bi djars=/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar,/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar,/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar,/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar spark-submit --master yarn-cluster \ --queue spark \ --files /home/spark/software/spark/conf/hive-site.xml,/home/spark/software/spark/conf/fairscheduler.xml \ --jars $djars \ --class $mainclass $mainjar bi_dm hive3 bi
3.结果验证
日志：
5/11/19 15:16:51 INFO Client: Will allocate AM container, with 4505 MB memory including 409 MB overhead 15/11/19 15:16:51 INFO Client: Setting up container launch context for our AM 15/11/19 15:16:51 INFO Client: Setting up the launch environment for our AM container 15/11/19 15:16:51 INFO Client: Preparing resources for our AM container 15/11/19 15:16:51 INFO Client: Uploading resource file:/home/spark/software/spark-1.5.2-bin-2.4.0/lib/spark-assembly-1.5.2-hadoop2.4.0.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/spark-assembly-1.5.2-hadoop2.4.0.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/workspace/tersort/HiveUtility-v1.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/HiveUtility-v1.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-api-jdo-3.2.6.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-core-3.2.10.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-core-3.2.10.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/lib/datanucleus-rdbms-3.2.9.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/datanucleus-rdbms-3.2.9.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/ext/mysql-connector-java-5.1.27-bin.jar -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/mysql-connector-java-5.1.27-bin.jar 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/hive-site.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/hive-site.xml 15/11/19 15:16:54 INFO Client: Uploading resource file:/home/spark/software/spark/conf/fairscheduler.xml -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/fairscheduler.xml 15/11/19 15:16:54 INFO Client: Uploading resource file:/data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0/__spark_conf__6425041649655761245.zip -> hdfs://SuningHadoop2/user/bi/.sparkStaging/application_1447783148941_6897/__spark_conf__6425041649655761245.zip 15/11/19 15:16:54 INFO SecurityManager: Changing view acls to: spark,bi 15/11/19 15:16:54 INFO SecurityManager: Changing modify acls to: spark,bi 15/11/19 15:16:54 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(spark, bi); users with modify permissions: Set(spark, bi) 15/11/19 15:16:54 INFO Client: Submitting application 6897 to ResourceManager 15/11/19 15:16:54 INFO YarnClientImpl: Submitted application application_1447783148941_6897 15/11/19 15:16:55 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED) 15/11/19 15:16:55 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: N/A ApplicationMaster RPC port: -1 queue: root.spark start time: 1447917414919 final status: UNDEFINED tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/ user: bi 15/11/19 15:16:56 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED) 15/11/19 15:16:57 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED) 15/11/19 15:16:58 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED) 15/11/19 15:16:59 INFO Client: Application report for application_1447783148941_6897 (state: ACCEPTED) 15/11/19 15:17:00 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:00 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.27.1.143 ApplicationMaster RPC port: 0 queue: root.spark start time: 1447917414919 final status: UNDEFINED tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/ user: bi 15/11/19 15:17:01 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:02 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:03 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:04 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:05 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:06 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:07 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:09 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:10 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:11 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:12 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:13 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:14 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:15 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:16 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:17 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:18 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:19 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:20 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:21 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:22 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:23 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:24 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:25 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:26 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:27 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:28 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:29 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:30 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:31 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:32 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:33 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:34 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:35 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:36 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:37 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:38 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:39 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:40 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:41 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:42 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:43 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:44 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:45 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:46 INFO Client: Application report for application_1447783148941_6897 (state: RUNNING) 15/11/19 15:17:47 INFO Client: Application report for application_1447783148941_6897 (state: FINISHED) 15/11/19 15:17:47 INFO Client: client token: N/A diagnostics: N/A ApplicationMaster host: 10.27.1.143 ApplicationMaster RPC port: 0 queue: root.spark start time: 1447917414919 final status: SUCCEEDED tracking URL: http://namenode1-sit.cnsuning.com:8088/proxy/application_1447783148941_6897/ user: bi 15/11/19 15:17:47 INFO ShutdownHookManager: Shutdown hook called 15/11/19 15:17:47 INFO ShutdownHookManager: Deleting directory /data/spark/sparkLocalDir/spark-e634db07-a87d-4c36-b52b-b062a4e55ef0
参考：http://spark.apache.org/docs/latest/job-scheduling.html
内容来自用户分享和网络整理，不保证内容的准确性，如有侵权内容，可联系管理员处理 
相关文章推荐
- ovs-vsctl: unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory)
- stitching_detail算法介绍（转载）
- linker command failed with exit code 1
- LeetCode（220） Contains Duplicate III
- LeetCode（220） Contains Duplicate III
- Xcode6中如何去掉默认的Main.storyboard
- LeetCode（219） Contains Duplicate II
- LeetCode（219） Contains Duplicate II
- Archive failed
- Fibonacci Again模板
- mybatis 批量增加 Parameter '__frch_item_0' not found. Available parameters are [list]
- [1021]：Fibonacci Again
- AIDL中使用自定义类型时无法导包的问题
- FAL[server, ARC3]: FAL archive failed Error 16401 creating standby archive log file at host
- 测试用例 自动生成工具PICT与AllPairs
- 配置SharePoint 2013失败--Max Degree of Parallelism
- DELL T420塔式服务器RAID配置及OS安装
- HDU ACM 1789 Doing Homework again
- POJ 1273 Drainage Ditches
- ASP.NTE 5 Target framework dnx451 and dnxcore50(转)原文：http://www.cnblogs.com/xishuai/p/aspnet5-target-framework-dnx451-and-dnxcore50.html