当前位置 博文首页 > 大英小二黑new的博客:Caused by: java.lang.NoSuchMethodError:

    大英小二黑new的博客:Caused by: java.lang.NoSuchMethodError:

    作者:[db:作者] 时间:2021-09-13 09:52

    (1)在pyspark连接,mongodb时连接代码如下:版本,spark2.3.0,scala2.11.8

    # 创建会话
    spark = SparkSession \
        .builder \
        .appName("pyspark-mongodb data source example") \
        .config("spark.mongodb.input.uri",
                "mongodb://192.168.5.247:27017/test.company_info") \
        .config("spark.mongodb.output.uri",
                "mongodb://192.168.5.247:27017/test.company_info") \
        .master('local[*]') \
        .getOrCreate()
    
    # df = spark.read.format("com.mongodb.spark.sql").option("spark.mongodb.input.uri",
                                                                
    #                              "mongodb://192.168.5.247:27017/test.company_info").load()
    
    df = spark.read.format("com.mongodb.spark.sql").load()

    (2)在连接时,报错:如下:

    Caused by: java.lang.NoSuchMethodError: org.apache.spark.sql.catalyst.analysis.TypeCoercion$.findTightestCommonTypeOfTwo()Lscala/Function2;
            at com.mongodb.spark.sql.MongoInferSchema$.com$mongodb$spark$sql$MongoInferSchema$$compatibleType(MongoInferSchema.scala:135)
            at com.mongodb.spark.sql.MongoInferSchema$$anonfun$3.apply(MongoInferSchema.scala:78)
            at com.mongodb.spark.sql.MongoInferSchema$$anonfun$3.apply(MongoInferSchema.scala:78)
            at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
            at scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
            at scala.collection.Iterator$class.foreach(Iterator.scala:893)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
            at scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
            at scala.collection.AbstractIterator.foldLeft(Iterator.scala:1336)
            at scala.collection.TraversableOnce$class.aggregate(TraversableOnce.scala:214)
            at scala.collection.AbstractIterator.aggregate(Iterator.scala:1336)
            at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1139)
            at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$23.apply(RDD.scala:1139)
            at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1140)
            at org.apache.spark.rdd.RDD$$anonfun$treeAggregate$1$$anonfun$24.apply(RDD.scala:1140)
            at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
            at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
            at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
            at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
            at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
            at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
            at org.apache.spark.scheduler.Task.run(Task.scala:109)
            at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            ... 1 more
    

    (3)问题分析:

    出现Caused by: java.lang.NoSuchMethodError这样的错误,一般都是jar包的版本冲突造成。

    1. jar包的版本不对
    2. 有两个版本的jar包,jar包版本冲突

    (4)问题解决

    1. 下载对应的包https://download.csdn.net/download/qq_21735341/10657593?(mongo-java-driver-3.8.0.jar,mongo-spark-connector_2.11-2.3.0.jar)
    2. 在spark的jar包下替换掉之前的对应版本,注意:一定要将之前的版本删掉,一定要删掉,不然jar包版本冲突,会报一样的错误
    cs