How-to: resolve "java.lang.NoClassDefFoundError: org/htrace/Trace" when hbase Export

Caused by: java.lang.NoClassDefFoundError: org/htrace/Trace
at org.apache.hadoop.hbase.client.ResultBoundedCompletionService.submit(ResultBoundedCompletionService.java:142)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.addCallsForCurrentReplica(ScannerCallableWithReplicas.java:276)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:165)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:57)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
... 17 more

Debugging:

The related jar is named htrace-core-3.0.4.jar which is located in ${HBASE_HOME}. But hadoop nodemanager does not find this. This is because hadoop clients classpath does not include this jar. I think this should be a hbase bug. That is during hbase Export, hbase should add this jar into runtime classpath.

Solution:

One workaround is adding this jar into hadoop lib in which all jars will be loaded during jop runtime. Like: ${HADOOP_HOME}/share/hadoop/mapreduce/lib/. Do this in each nodemanger as you are not sure which node mapreduce job will run.