러닝 스파크(Learning Spark) 책의 2장을 따라하는데, 윈도우에서 spark-shell을 실행하면 다음과 같은 이상한 에러가 발생한다.
C:\devtool\spark-1.6.0-bin-hadoop2.6>bin\spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.0
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51)
...
16/02/17 13:09:58 WARN : Your hostname, user resolves to a loopback/non-reachable address: fe80:0:0:0:35a2:f2cc:2a09:416
e%eth8, but we couldn't find any external IP address!
java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: 액세스가 거부되었습니다
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.ClientWrapper.<init>(ClientWrapper.scala:194)
...(너무 길어 생략)
Caused by: java.io.IOException: 액세스가 거부되었습니다
at java.io.WinNTFileSystem.createFileExclusively(Native Method)
at java.io.File.createTempFile(File.java:2024)
at org.apache.hadoop.hive.ql.session.SessionState.createTempFile(SessionState.java:818)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:513)
... 62 more
<console>:16: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:16: error: not found: value sqlContext
import sqlContext.sql
^
scala>
이 에러가 발생하지 않도록 하려면 다음의 두 가지를 해 주면 된다.
- winutils.exe 파일을 c:\hadoop\bin에 복사한다. c:\haoop을 HADOOP_HOME 환경 변수로 설정한다. (winutils.exe 파일은 https://github.com/steveloughran/winutils/tree/master/hadoop-2.6.0/bin 에서 다운로드 받을 수 있다)
- 관리자 권한으로 명령 프롬프트를 열고 "C:\Hadoop\bin\winutils.exe chmod 777 /tmp/hive" 명령어를 실행한다.
이제 관리자 권한으로 실행한 명령 프롬프트에서 bin\spark-shell을 실행한다. 다음과 같이 정상으로 실행되는 것을 확인할 수 있다.
C:\devtool\spark-1.6.0-bin-hadoop2.6>bin\spark-shell
log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties
To adjust logging level use sc.setLogLevel("INFO")
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 1.6.0
/_/
Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_51)
Type in expressions to have them evaluated.
Type :help for more information.
Spark context available as sc.
16/02/17 13:20:05 WARN General: Plugin (Bundle) "org.datanucleus" is already registered. Ensure you dont have multiple J
...생략
16/02/17 13:20:12 WARN : Your hostname, user resolves to a loopback/non-reachable address: fe80:0:0:0:35a2:f2cc:2a09:416
e%eth8, but we couldn't find any external IP address!
...생략
16/02/17 13:20:18 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
SQL context available as sqlContext.
scala> val lines = sc.textFile("README.md")
lines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[1] at textFile at <console>:27
scala>