CDH-5.14.2使用阿里的jindoFS文件系统访问OSS数据时报错
  GPYyDLfgzzIb 2023年11月02日 64 0


CDH-5.14.2使用阿里的jindoFS文件系统访问OSS数据时报错_apache

报错结果:

CDH-5.14.2使用阿里的jindoFS文件系统访问OSS数据时报错_hadoop_02

Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.aliyun.jindodata.oss.JindoOssFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2349)
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2790)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
        at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:343)
        ... 34 more
Caused by: java.lang.ClassNotFoundException: Class com.aliyun.jindodata.oss.JindoOssFileSystem not found
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2347)
        ... 42 more

报错原因:没有相关权限


解决方案:

#需要进入root用户
[root@bgtest001 lib]# cp jindo*  /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/
[root@bgtest001 jars]# ll jindo-core-4.6.5.jar
-rw-r--r-- 1 root root 15353705 May  5 17:37 jindo-core-4.6.5.jar
#注意查看是否有对应的权限,如果没有进行赋权
[root@bgtest001 jars]# chmod 777 jindo-core-4.6.5.jar

hive>
    > insert into ods_db.ods_seller_config_detail_oss_test partition(ds = '2023-05-05')
    > select  id ,group_id ,seller_id ,seller_config_id ,service_module ,field_code ,related_field_code ,related_field_value ,field_value ,data_type ,is_seller_read,is_seller_write ,status ,created_at ,updated_at ,deleted_at ,is_deleted from ods_db.ods_seller_config_detail ;
Query ID = root_20230505173939_2158fe6c-1deb-4212-a1da-c08c584dba3f
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1682071687253_0541, Tracking URL = http://bgtest002:34490/proxy/application_1682071687253_0541/
Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/hadoop job  -kill job_1682071687253_0541
Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 0
2023-05-05 17:39:38,707 Stage-1 map = 0%,  reduce = 0%
2023-05-05 17:40:13,909 Stage-1 map = 1%,  reduce = 0%, Cumulative CPU 24.81 sec
2023-05-05 17:40:15,030 Stage-1 map = 2%,  reduce = 0%, Cumulative CPU 26.4 sec
2023-05-05 17:40:16,236 Stage-1 map = 14%,  reduce = 0%, Cumulative CPU 29.46 sec
2023-05-05 17:40:17,789 Stage-1 map = 16%,  reduce = 0%, Cumulative CPU 34.36 sec
2023-05-05 17:40:21,227 Stage-1 map = 17%,  reduce = 0%, Cumulative CPU 37.33 sec
2023-05-05 17:40:22,335 Stage-1 map = 18%,  reduce = 0%, Cumulative CPU 43.05 sec
2023-05-05 17:40:23,428 Stage-1 map = 20%,  reduce = 0%, Cumulative CPU 45.83 sec
2023-05-05 17:40:25,555 Stage-1 map = 22%,  reduce = 0%, Cumulative CPU 58.67 sec
2023-05-05 17:40:26,638 Stage-1 map = 24%,  reduce = 0%, Cumulative CPU 61.03 sec
2023-05-05 17:40:27,733 Stage-1 map = 26%,  reduce = 0%, Cumulative CPU 63.09 sec
2023-05-05 17:40:29,048 Stage-1 map = 29%,  reduce = 0%, Cumulative CPU 67.53 sec
2023-05-05 17:40:30,106 Stage-1 map = 32%,  reduce = 0%, Cumulative CPU 70.49 sec
2023-05-05 17:40:31,331 Stage-1 map = 34%,  reduce = 0%, Cumulative CPU 72.81 sec
2023-05-05 17:40:32,411 Stage-1 map = 38%,  reduce = 0%, Cumulative CPU 78.54 sec
2023-05-05 17:40:33,478 Stage-1 map = 41%,  reduce = 0%, Cumulative CPU 81.03 sec
2023-05-05 17:40:34,550 Stage-1 map = 43%,  reduce = 0%, Cumulative CPU 83.55 sec
2023-05-05 17:40:35,614 Stage-1 map = 47%,  reduce = 0%, Cumulative CPU 88.67 sec
2023-05-05 17:40:37,869 Stage-1 map = 52%,  reduce = 0%, Cumulative CPU 94.52 sec
2023-05-05 17:40:39,021 Stage-1 map = 54%,  reduce = 0%, Cumulative CPU 96.55 sec
2023-05-05 17:40:40,092 Stage-1 map = 56%,  reduce = 0%, Cumulative CPU 98.95 sec
2023-05-05 17:40:41,167 Stage-1 map = 60%,  reduce = 0%, Cumulative CPU 103.48 sec
2023-05-05 17:40:42,263 Stage-1 map = 62%,  reduce = 0%, Cumulative CPU 106.39 sec
2023-05-05 17:40:43,382 Stage-1 map = 64%,  reduce = 0%, Cumulative CPU 109.03 sec
2023-05-05 17:40:44,439 Stage-1 map = 68%,  reduce = 0%, Cumulative CPU 113.75 sec
2023-05-05 17:40:45,577 Stage-1 map = 70%,  reduce = 0%, Cumulative CPU 115.95 sec
2023-05-05 17:40:46,644 Stage-1 map = 73%,  reduce = 0%, Cumulative CPU 118.68 sec
2023-05-05 17:40:47,709 Stage-1 map = 76%,  reduce = 0%, Cumulative CPU 123.21 sec
2023-05-05 17:40:49,855 Stage-1 map = 79%,  reduce = 0%, Cumulative CPU 126.37 sec
2023-05-05 17:40:50,959 Stage-1 map = 81%,  reduce = 0%, Cumulative CPU 127.94 sec
2023-05-05 17:40:52,033 Stage-1 map = 83%,  reduce = 0%, Cumulative CPU 129.88 sec
2023-05-05 17:40:53,122 Stage-1 map = 88%,  reduce = 0%, Cumulative CPU 135.91 sec
2023-05-05 17:40:54,249 Stage-1 map = 89%,  reduce = 0%, Cumulative CPU 136.47 sec
2023-05-05 17:40:56,408 Stage-1 map = 93%,  reduce = 0%, Cumulative CPU 142.61 sec
2023-05-05 17:40:57,485 Stage-1 map = 95%,  reduce = 0%, Cumulative CPU 144.23 sec
2023-05-05 17:40:58,550 Stage-1 map = 97%,  reduce = 0%, Cumulative CPU 146.21 sec
2023-05-05 17:40:59,738 Stage-1 map = 99%,  reduce = 0%, Cumulative CPU 147.69 sec
2023-05-05 17:41:01,075 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 149.38 sec
MapReduce Total cumulative CPU time: 2 minutes 29 seconds 380 msec
Ended Job = job_1682071687253_0541
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: oss://qm-bigdata-hdfs/user/hive/qm/bigdata/ods_db.db/ods_customer_account_flow_oss_di/ds=2023-05-05/.hive-staging_hive_2023-05-05_17-39-26_053_1171860901158736223-1/-ext-10000
Loading data to table ods_db.ods_seller_config_detail_oss_test partition (ds=2023-05-05)
Partition ods_db.ods_seller_config_detail_oss_test{ds=2023-05-05} stats: [numFiles=8, numRows=10666897, totalSize=1848787374, rawDataSize=1838120477]
[ods_db@ods_seller_config_detail_oss_test@ds=2023-05-05]
Fri May 05 17:41:04 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
MapReduce Jobs Launched:
Stage-Stage-1: Map: 8   Cumulative CPU: 149.38 sec   HDFS Read: 1849656964 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 2 minutes 29 seconds 380 msec
OK
Time taken: 99.503 seconds

CDH-5.14.2使用阿里的jindoFS文件系统访问OSS数据时报错_apache_03





【版权声明】本文内容来自摩杜云社区用户原创、第三方投稿、转载,内容版权归原作者所有。本网站的目的在于传递更多信息,不拥有版权,亦不承担相应法律责任。如果您发现本社区中有涉嫌抄袭的内容,欢迎发送邮件进行举报,并提供相关证据,一经查实,本社区将立刻删除涉嫌侵权内容,举报邮箱: cloudbbs@moduyun.com

上一篇: Hive 面试篇《一》 下一篇: Go操作Kafka
  1. 分享:
最后一次编辑于 2023年11月08日 0

暂无评论

推荐阅读
  KRe60ogUm4le   2024年04月26日   30   0   0 java算法
  KRe60ogUm4le   2024年05月03日   56   0   0 javascala
GPYyDLfgzzIb
最新推荐 更多

2024-05-31