报错结果:
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.aliyun.jindodata.oss.JindoOssFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2349)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2790)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2853)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2835)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:343)
... 34 more
Caused by: java.lang.ClassNotFoundException: Class com.aliyun.jindodata.oss.JindoOssFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2255)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2347)
... 42 more
报错原因:没有相关权限
解决方案:
#需要进入root用户
[root@bgtest001 lib]# cp jindo* /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/jars/
[root@bgtest001 jars]# ll jindo-core-4.6.5.jar
-rw-r--r-- 1 root root 15353705 May 5 17:37 jindo-core-4.6.5.jar
#注意查看是否有对应的权限,如果没有进行赋权
[root@bgtest001 jars]# chmod 777 jindo-core-4.6.5.jar
hive>
> insert into ods_db.ods_seller_config_detail_oss_test partition(ds = '2023-05-05')
> select id ,group_id ,seller_id ,seller_config_id ,service_module ,field_code ,related_field_code ,related_field_value ,field_value ,data_type ,is_seller_read,is_seller_write ,status ,created_at ,updated_at ,deleted_at ,is_deleted from ods_db.ods_seller_config_detail ;
Query ID = root_20230505173939_2158fe6c-1deb-4212-a1da-c08c584dba3f
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1682071687253_0541, Tracking URL = http://bgtest002:34490/proxy/application_1682071687253_0541/
Kill Command = /opt/cloudera/parcels/CDH-5.14.2-1.cdh5.14.2.p0.3/lib/hadoop/bin/hadoop job -kill job_1682071687253_0541
Hadoop job information for Stage-1: number of mappers: 8; number of reducers: 0
2023-05-05 17:39:38,707 Stage-1 map = 0%, reduce = 0%
2023-05-05 17:40:13,909 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 24.81 sec
2023-05-05 17:40:15,030 Stage-1 map = 2%, reduce = 0%, Cumulative CPU 26.4 sec
2023-05-05 17:40:16,236 Stage-1 map = 14%, reduce = 0%, Cumulative CPU 29.46 sec
2023-05-05 17:40:17,789 Stage-1 map = 16%, reduce = 0%, Cumulative CPU 34.36 sec
2023-05-05 17:40:21,227 Stage-1 map = 17%, reduce = 0%, Cumulative CPU 37.33 sec
2023-05-05 17:40:22,335 Stage-1 map = 18%, reduce = 0%, Cumulative CPU 43.05 sec
2023-05-05 17:40:23,428 Stage-1 map = 20%, reduce = 0%, Cumulative CPU 45.83 sec
2023-05-05 17:40:25,555 Stage-1 map = 22%, reduce = 0%, Cumulative CPU 58.67 sec
2023-05-05 17:40:26,638 Stage-1 map = 24%, reduce = 0%, Cumulative CPU 61.03 sec
2023-05-05 17:40:27,733 Stage-1 map = 26%, reduce = 0%, Cumulative CPU 63.09 sec
2023-05-05 17:40:29,048 Stage-1 map = 29%, reduce = 0%, Cumulative CPU 67.53 sec
2023-05-05 17:40:30,106 Stage-1 map = 32%, reduce = 0%, Cumulative CPU 70.49 sec
2023-05-05 17:40:31,331 Stage-1 map = 34%, reduce = 0%, Cumulative CPU 72.81 sec
2023-05-05 17:40:32,411 Stage-1 map = 38%, reduce = 0%, Cumulative CPU 78.54 sec
2023-05-05 17:40:33,478 Stage-1 map = 41%, reduce = 0%, Cumulative CPU 81.03 sec
2023-05-05 17:40:34,550 Stage-1 map = 43%, reduce = 0%, Cumulative CPU 83.55 sec
2023-05-05 17:40:35,614 Stage-1 map = 47%, reduce = 0%, Cumulative CPU 88.67 sec
2023-05-05 17:40:37,869 Stage-1 map = 52%, reduce = 0%, Cumulative CPU 94.52 sec
2023-05-05 17:40:39,021 Stage-1 map = 54%, reduce = 0%, Cumulative CPU 96.55 sec
2023-05-05 17:40:40,092 Stage-1 map = 56%, reduce = 0%, Cumulative CPU 98.95 sec
2023-05-05 17:40:41,167 Stage-1 map = 60%, reduce = 0%, Cumulative CPU 103.48 sec
2023-05-05 17:40:42,263 Stage-1 map = 62%, reduce = 0%, Cumulative CPU 106.39 sec
2023-05-05 17:40:43,382 Stage-1 map = 64%, reduce = 0%, Cumulative CPU 109.03 sec
2023-05-05 17:40:44,439 Stage-1 map = 68%, reduce = 0%, Cumulative CPU 113.75 sec
2023-05-05 17:40:45,577 Stage-1 map = 70%, reduce = 0%, Cumulative CPU 115.95 sec
2023-05-05 17:40:46,644 Stage-1 map = 73%, reduce = 0%, Cumulative CPU 118.68 sec
2023-05-05 17:40:47,709 Stage-1 map = 76%, reduce = 0%, Cumulative CPU 123.21 sec
2023-05-05 17:40:49,855 Stage-1 map = 79%, reduce = 0%, Cumulative CPU 126.37 sec
2023-05-05 17:40:50,959 Stage-1 map = 81%, reduce = 0%, Cumulative CPU 127.94 sec
2023-05-05 17:40:52,033 Stage-1 map = 83%, reduce = 0%, Cumulative CPU 129.88 sec
2023-05-05 17:40:53,122 Stage-1 map = 88%, reduce = 0%, Cumulative CPU 135.91 sec
2023-05-05 17:40:54,249 Stage-1 map = 89%, reduce = 0%, Cumulative CPU 136.47 sec
2023-05-05 17:40:56,408 Stage-1 map = 93%, reduce = 0%, Cumulative CPU 142.61 sec
2023-05-05 17:40:57,485 Stage-1 map = 95%, reduce = 0%, Cumulative CPU 144.23 sec
2023-05-05 17:40:58,550 Stage-1 map = 97%, reduce = 0%, Cumulative CPU 146.21 sec
2023-05-05 17:40:59,738 Stage-1 map = 99%, reduce = 0%, Cumulative CPU 147.69 sec
2023-05-05 17:41:01,075 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 149.38 sec
MapReduce Total cumulative CPU time: 2 minutes 29 seconds 380 msec
Ended Job = job_1682071687253_0541
Stage-4 is selected by condition resolver.
Stage-3 is filtered out by condition resolver.
Stage-5 is filtered out by condition resolver.
Moving data to: oss://qm-bigdata-hdfs/user/hive/qm/bigdata/ods_db.db/ods_customer_account_flow_oss_di/ds=2023-05-05/.hive-staging_hive_2023-05-05_17-39-26_053_1171860901158736223-1/-ext-10000
Loading data to table ods_db.ods_seller_config_detail_oss_test partition (ds=2023-05-05)
Partition ods_db.ods_seller_config_detail_oss_test{ds=2023-05-05} stats: [numFiles=8, numRows=10666897, totalSize=1848787374, rawDataSize=1838120477]
[ods_db@ods_seller_config_detail_oss_test@ds=2023-05-05]
Fri May 05 17:41:04 CST 2023 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
MapReduce Jobs Launched:
Stage-Stage-1: Map: 8 Cumulative CPU: 149.38 sec HDFS Read: 1849656964 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 2 minutes 29 seconds 380 msec
OK
Time taken: 99.503 seconds