to get output from the Scala project
But after getting the output and running it, I encounter this error
What is the reason?
scala compile and run
with target/scala-2.11/scala spark-kafka-streaming_2.11-0.1.jar
error:
java.lang.ClassNotFoundException: org.apache.spark.sql.streaming.StreamingQueryException
???
Вероятно, у вас конфликт версий скалы. Проверьте, что все зависимости имеют один и тот же постфикс версии скалы (например, _2.11)
Мои версии такие же. Это мои библиотеки
name := "spark-kafka-streaming" version := "0.1" scalaVersion := "2.11.12" val sparkVersion = "2.4.7" val kafkaVersion = "2.4.1" libraryDependencies += "org.apache.spark" %% "spark-streaming" % sparkVersion libraryDependencies += "org.apache.spark" %% "spark-streaming-kafka-0-8" % sparkVersion libraryDependencies += "org.apache.kafka" % "kafka-clients" % kafkaVersion val spark = "org.apache.spark" % "spark-core_2.11" % sparkVersion val kafka = "org.apache.spark" % "spark-sql-kafka-0-10_2.11" % kafkaVersion val sql = "org.apache.spark" % "spark-sql_2.11" % sparkVersion libraryDependencies += spark libraryDependencies += sql libraryDependencies += kafka libraryDependencies += "com.codahale.metrics" % "metrics-core" % "3.0.2" libraryDependencies += "com.sun.mail" % "javax.mail" % "1.6.2" libraryDependencies += "org.postgresql" % "postgresql" % "42.2.19" libraryDependencies += "io.projectreactor.netty" % "reactor-netty" % "1.0.6"
Обсуждают сегодня