NiFi and Spark Streaming

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi and Spark Streaming

Savage, Edward (ISE)
Hi,

I have been reading your blog: https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark.

This has been a useful resource for learning a new technology. So thanks for putting it together.
Firstly I believe there is a typo. The code when creating the receiver passes in an object called "ClientConfig". However from the code above I believe it is meant to be simply "config".

Secondly when trying to run your example on my own machine I am getting issues when submitting the job to Spark. It moans about not finding the "NiFiReceiver" class. How do I add this external dependency to spark?

Thanks in advance for any help you can offer,
Edd


EDWARD SAVAGE | Software Engineer, Cyber | NORTHROP GRUMMAN

Northrop Grumman
Cyber Division
95, Promenade
Cheltenham,
GL50 1HH

Email: [hidden email]<mailto:[hidden email]>

Reply | Threaded
Open this post in threaded view
|

RE: NiFi and Spark Streaming

Mark Payne
Ed,

Thanks for your interest!

You're definitely right - ClientConfig should be "config". I have updated the code on the blog post. Thanks for pointing that out.

To include the Receiver in your application, you'll need the following dependency in your application:

  <dependency>
    <groupId>org.apache.nifi</groupId>
    <artifactId>nifi-spark-receiver</artifactId>
    <version>0.1.0-incubating</version>
  </dependency>

Then, you should create an "uber jar" that you can deploy (in Maven it's usually done with something like the following snippet):

        <build>
                <plugins>
                        <plugin>
                                <groupId>org.apache.maven.plugins</groupId>
                                <artifactId>maven-shade-plugin</artifactId>
                                <executions>
                                        <execution>
                                                <phase>package</phase>
                                                <goals>
                                                        <goal>shade</goal>
                                                </goals>
                                        </execution>
                                </executions>
                        </plugin>
                </plugins>
        </build>
You should then be able to use spark-deploy or whatever mechanism you usually use to deploy your applications to Spark.

Thanks
-Mark



----------------------------------------

> From: [hidden email]
> To: [hidden email]
> Subject: NiFi and Spark Streaming
> Date: Tue, 26 May 2015 15:04:36 +0000
>
> Hi,
>
> I have been reading your blog: https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark.
>
> This has been a useful resource for learning a new technology. So thanks for putting it together.
> Firstly I believe there is a typo. The code when creating the receiver passes in an object called "ClientConfig". However from the code above I believe it is meant to be simply "config".
>
> Secondly when trying to run your example on my own machine I am getting issues when submitting the job to Spark. It moans about not finding the "NiFiReceiver" class. How do I add this external dependency to spark?
>
> Thanks in advance for any help you can offer,
> Edd
>
>
> EDWARD SAVAGE | Software Engineer, Cyber | NORTHROP GRUMMAN
>
> Northrop Grumman
> Cyber Division
> 95, Promenade
> Cheltenham,
> GL50 1HH
>
> Email: [hidden email]<mailto:[hidden email]>
>