java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

Je obtenir NoClassDefFoundErrorerror tout en utilisant Spark streaming API. Voici mon Streaming code.

Je sais que c'est un problème avec certains mising les pots et les dépendances, mais je ne pourrais pas comprendre exactement ce que c'est.

Je suis à l'aide de kafka 0.9.0, étincelle 1.6.1 - Sont présentes avec des dépendances amende ou dois-je le changer? J'ai joint pom.xml ci-dessous.

Voici le streaming API que j'utilise.

JavaPairInputDStream directKafkaStream = KafkaUtils.createDirectStream(cdc, String.class,
byte[].de la classe, StringDecoder.class, DefaultDecoder.class, kafkaParams, topicSet);

voici mon code de la pièce. Je reçois une erreur à tout(itr.next())

directKafkaStream.foreachRDD(rdd -> {

    rdd.foreachPartition(itr -> {


        try {

            while (itr.hasNext()) {

java.lang.NoClassDefFoundError: org/apache/kafka/common/message/KafkaLZ4BlockOutputStream

Voici mon POM.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>com.abcd.rep.xyz</groupId>
        <artifactId>xyz</artifactId>
        <version>1.0</version>
        <relativePath>../pom.xml</relativePath>
    </parent>
        <artifactId>SparkPOC</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>
        <name>SparkPOCde</name>
        <url>http://maven.apache.org</url>
<properties>

<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>


<spark-version>1.6.1</spark-version>

<kafka-version>0.9.0.0</kafka-version>

</properties>


<dependencies>

<!-- http://mvnrepository.com/artifact/org.springframework/spring-core -->



<!-- http://mvnrepository.com/artifact/org.springframework/spring-jdbc -->
<dependency>

<groupId>log4j</groupId>

<artifactId>log4j</artifactId>

<version>1.2.17</version>

</dependency>

<!-- http://mvnrepository.com/artifact/org.apache.spark/spark-streaming_2.10 -->

<dependency>
<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming_2.10</artifactId>

<version>${spark-version}</version>

</dependency>

<!-- https://mvnrepository.com/artifact/org.apache.spark/spark-streaming-kafka_2.10 -->



<dependency>

<groupId>org.apache.spark</groupId>

<artifactId>spark-streaming-kafka_2.10</artifactId>

<version>1.6.2</version>

 <exclusions>
        <exclusion>
            <groupId>io.netty</groupId>
            <artifactId>netty</artifactId>
        </exclusion>

        <exclusion>
            <groupId>io.jboss.netty</groupId>
            <artifactId>netty</artifactId>
        </exclusion>
    </exclusions> 

</dependency>

   <dependency>
            <groupId>com.abcd.rep.xyz</groupId>
            <artifactId>xyzCommon</artifactId>
            <version>1.0</version>
            <type>jar</type>
        </dependency>


<!-- http://mvnrepository.com/artifact/ojdbc/ojdbc -->


<!-- <dependency> <groupId>ojdbc</groupId> <artifactId>ojdbc</artifactId> <version>14</version> </dependency>-->


<!-- https://mvnrepository.com/artifact/org.mongodb/mongo-java-driver -->




<!-- http://mvnrepository.com/artifact/org.springframework.data/spring-data-mongodb -->




<!-- https://mvnrepository.com/artifact/com.googlecode.json-simple/json-simple -->





</dependencies>


<build>

<finalName>appname</finalName>


<resources>


<resource>

<directory>src/main/resources</directory>


<excludes>

<exclude>eventRules.json</exclude>

<exclude>log4j.xml</exclude>

<exclude>resources.properties</exclude>

</excludes>

</resource>

</resources>


<plugins>


<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-compiler-plugin</artifactId>

<version>3.5.1</version>


<configuration>

<source>1.8</source>

<target>1.8</target>

</configuration>

</plugin>


<plugin>

<groupId>org.apache.maven.plugins</groupId>

<artifactId>maven-shade-plugin</artifactId>

<version>2.4.3</version>


<executions>


<execution>

<phase>package</phase>


<goals>

<goal>shade</goal>

</goals>


<configuration>


<filters>


<filter>

<artifact>*:*</artifact>


-<excludes>

<exclude>META-INF/*.SF</exclude>

<exclude>META-INF/*.DSA</exclude>

<exclude>META-INF/*.RSA</exclude>

</excludes>

</filter>

</filters>


<transformers>


-<transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">

<resource>reference.conf</resource>

</transformer>


-<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">

<mainClass>com.abcd.rep.xyz.SparkPOCde.EventConsumerServiceImpl</mainClass>

</transformer>

</transformers>

</configuration>

</execution>

</executions>

</plugin>

</plugins>

</build>

</project>