背景

在hive sql中执行添加临时udf的操作

add jar hdfs:///user/hive/lib/tools-1.0.jar;
create temporary function decode as 'com.test.etl.Decoder';

在定时任务重偶尔会报错，报错比较随机，报错信息如下：

INFO  : Query ID = hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6INFO  : Total jobs = 1INFO  : Launching Job 1 out of 1INFO  : Starting task [Stage-1:MAPRED] in serial modeINFO  : Number of reduce tasks not specified. Estimated from input data size: 197INFO  : In order to change the average load for a reducer (in bytes):INFO  :   set hive.exec.reducers.bytes.per.reducer=INFO  : In order to limit the maximum number of reducers:INFO  :   set hive.exec.reducers.max=INFO  : In order to set a constant number of reducers:INFO  :   set mapreduce.job.reduces=INFO  : Cleaning up the staging area /user/yarn/hive/.staging/job_1629262398375_145140ERROR : Job Submission failed with exception 'java.io.FileNotFoundException(File file:/disk1/hive/download/tools-1.0.jar does not exist)'java.io.FileNotFoundException: File file:/disk1/hive/download/tools-1.0.jar does not exist    at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)    at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:378)    at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:329)    at org.apache.hadoop.mapreduce.JobResourceUploader.copyRemoteFiles(JobResourceUploader.java:680)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadLibJars(JobResourceUploader.java:313)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResourcesInternal(JobResourceUploader.java:205)    at org.apache.hadoop.mapreduce.JobResourceUploader.uploadResources(JobResourceUploader.java:133)    at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:99)    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:194)    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1570)    at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1567)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1567)    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:576)    at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:571)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:571)    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:562)    at org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:441)    at org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:146)    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2250)    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1893)    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1613)    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1332)    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1327)    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:256)    at org.apache.hive.service.cli.operation.SQLOperation.access$600(SQLOperation.java:92)    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:345)    at java.security.AccessController.doPrivileged(Native Method)    at javax.security.auth.Subject.doAs(Subject.java:422)    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1731)    at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:357)    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)    at java.util.concurrent.FutureTask.run(FutureTask.java:266)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)    at java.lang.Thread.run(Thread.java:748)ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not existINFO  : Completed executing command(queryId=hive_20211026010225_545899e7-7afa-4b5c-b7db-fd71565a89c6); Time taken: 0.133 secondsError: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask. File file:/disk1/hive/download/tools-1.0.jar does not exist (state=08S01,code=1)Closing: 0: jdbc:hive2://master:10000Process exited with an error: 1 (Exit value: 1)

定位

报错是偶发的，没有规律，查看报错的目录，为hive下载目录，配置为：

hive.downloaded.resources.dir=/disk1/hive/download

检查发现集群中很多节点该目录不存在，怀疑是服务器环境初始化有问题，有的节点没有正确初始化。
进一步检查发现同一个节点这个目录，有时存在，有时不存在，排除上边的怀疑。
hive中这个配置的默认值为：

hive.downloaded.resources.dir=${system:java.io.tmpdir}/${hive.session.id}_resources

可见这个目录与session相关，每次session关闭时会做删除，如果配置的路径没有sessionid，则不同session之间会相互影响

DOWNLOADED_RESOURCES_DIR("hive.downloaded.resources.dir",    "${system:java.io.tmpdir}" + File.separator + "${hive.session.id}_resources",    "Temporary local directory for added resources in the remote file system."),

修改配置为：

hive.downloaded.resources.dir=/disk1/hive/download/${hive.session.id}_resources

问题解决。