Skip to main content

Hadoop Exceptions – JobTracker Couldn’t Start

Recently, I’m learning hadoop, but encountered many problems, here is the jobtracker problem.

System environment: ubuntu 13.04, hadoop 1.1.2

Exception description:

************************************************************/
2013-05-19 13:38:50,190 INFO org.apache.hadoop.mapred.JobTracker: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting JobTracker
STARTUP_MSG: host = xuqian-desktop/127.0.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.1.2
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.1 -r 1440782; compiled by 'hortonfo' on Thu Jan 31 02:03:24 UTC 2013
************************************************************/
2013-05-19 13:38:50,298 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
2013-05-19 13:38:50,310 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source MetricsSystem,sub=Stats registered.
2013-05-19 13:38:50,311 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
2013-05-19 13:38:50,311 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: JobTracker metrics system started
2013-05-19 13:38:50,416 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source QueueMetrics,q=default registered.
2013-05-19 13:38:50,641 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source ugi registered.
2013-05-19 13:38:50,641 WARN org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Source name ugi already exists!
2013-05-19 13:38:50,642 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-05-19 13:38:50,644 INFO org.apache.hadoop.mapred.JobTracker: Scheduler configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT, limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
2013-05-19 13:38:50,645 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Starting expired delegation token remover thread, tokenRemoverScanInterval=60 min(s)
2013-05-19 13:38:50,645 INFO org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: Updating the current master key for generating delegation tokens
2013-05-19 13:38:50,645 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude) list
2013-05-19 13:38:50,655 INFO org.apache.hadoop.mapred.JobTracker: Starting jobtracker with owner as xuqian
2013-05-19 13:38:50,675 INFO org.apache.hadoop.ipc.Server: Starting SocketReader
2013-05-19 13:38:50,678 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcDetailedActivityForPort9001 registered.
2013-05-19 13:38:50,678 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source RpcActivityForPort9001 registered.
2013-05-19 13:38:55,738 INFO org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2013-05-19 13:38:55,787 INFO org.apache.hadoop.http.HttpServer: Added global filtersafety (class=org.apache.hadoop.http.HttpServer$QuotingInputFilter)
2013-05-19 13:38:55,790 INFO org.apache.hadoop.http.HttpServer: Port returned by webServer.getConnectors()[0].getLocalPort() before open() is -1. Opening the listener on 50030
2013-05-19 13:38:55,790 INFO org.apache.hadoop.http.HttpServer: listener.getLocalPort() returned 50030 webServer.getConnectors()[0].getLocalPort() returned 50030
2013-05-19 13:38:55,790 INFO org.apache.hadoop.http.HttpServer: Jetty bound to port 50030
2013-05-19 13:38:55,791 INFO org.mortbay.log: jetty-6.1.26
2013-05-19 13:38:56,039 INFO org.mortbay.log: Started SelectChannelConnector@0.0.0.0:50030
2013-05-19 13:38:56,044 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source jvm registered.
2013-05-19 13:38:56,045 INFO org.apache.hadoop.metrics2.impl.MetricsSourceAdapter: MBean for source JobTrackerMetrics registered.
2013-05-19 13:38:56,054 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 9001
2013-05-19 13:38:56,054 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030
2013-05-19 13:38:56,064 INFO org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2013-05-19 13:38:56,064 INFO org.apache.hadoop.ipc.Server: IPC Server listener on 9001: starting
2013-05-19 13:38:56,065 INFO org.apache.hadoop.ipc.Server: IPC Server handler 0 on 9001: starting
2013-05-19 13:38:56,065 INFO org.apache.hadoop.ipc.Server: IPC Server handler 1 on 9001: starting
2013-05-19 13:38:56,065 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001: starting
2013-05-19 13:38:56,065 INFO org.apache.hadoop.ipc.Server: IPC Server handler 2 on 9001: starting
2013-05-19 13:38:56,066 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001: starting
2013-05-19 13:38:56,069 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5 on 9001: starting
2013-05-19 13:38:56,069 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9001: starting
2013-05-19 13:38:56,069 INFO org.apache.hadoop.ipc.Server: IPC Server handler 6 on 9001: starting
2013-05-19 13:38:56,069 INFO org.apache.hadoop.ipc.Server: IPC Server handler 9 on 9001: starting
2013-05-19 13:38:56,070 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to true. Requested by : xuqian
2013-05-19 13:38:56,073 INFO org.apache.hadoop.ipc.Server: IPC Server handler 8 on 9001: starting
2013-05-19 13:38:56,252 INFO org.apache.hadoop.mapred.JobTracker: Setting safe mode to false. Requested by : xuqian
2013-05-19 13:38:56,326 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory
2013-05-19 13:38:56,430 INFO org.apache.hadoop.mapred.JobHistory: Creating DONE folder at file:/usr/local/hadoop/logs/history/done
2013-05-19 13:38:56,432 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library
2013-05-19 13:38:56,434 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:xuqian cause:ENOENT: No such file or directory
2013-05-19 13:38:56,435 FATAL org.apache.hadoop.mapred.JobTracker: ENOENT: No such file or directory
at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:699)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:654)
at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)
at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:344)
at org.apache.hadoop.fs.FilterFileSystem.mkdirs(FilterFileSystem.java:189)
at org.apache.hadoop.mapred.JobHistory.initDone(JobHistory.java:564)
at org.apache.hadoop.mapred.JobHistory.initDone(JobHistory.java:540)
at org.apache.hadoop.mapred.JobTracker$4.run(JobTracker.java:2023)
at org.apache.hadoop.mapred.JobTracker$4.run(JobTracker.java:2021)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1149)
at org.apache.hadoop.mapred.JobTracker.initialize(JobTracker.java:2020)
at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2328)
at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4792)
2013-05-19 13:38:56,446 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down JobTracker at xuqian-desktop/127.0.1.1

 

Solution: The history folder was created after I execute ‘chown’ on hadoop installation folder, so just run:

sudo chown -R hadoop_user path/to/your/hadoop_home

For me is

chown -R xuqian /usr/local/hadoopt
  • V字龍

    Hi, I recently encounter the similar problem too(Ubuntu 12.04, Hadoop PPA)
    I eventually found out it is the jobtracker’s job history directory(default at file://${hadoop.log.dir}/history) not writable by jobtracker causes this error.
    The solution is change the permission of that directory