NiFi Spark Processor

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi Spark Processor

shankhamajumdar
Hi,

I am trying to use custom Spark Nifi processor. I have downloaded the processor from https://github.com/qiansl127/nifi-spark-bundle and then executed the maven build. After the maven build I have copied nifi-spark-nar-0.3.0.nar inside NiFi lib folder and restarted the NiFi server which is not working.

Please advice.

Regards,
Shankha
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Spark Processor

Joey Frazee
Shanka,

It's hard to tell without more details, which you can probably find in the logs, but the pom.xml in that project has not been updated for any of the recent (< 1y) releases of NiFi (0.6, 0.7, 1.0), so you're probably going to have to change the version properties and, if using NiFi 1.0, change ProcessorLog to ComponentLog in the actual processor implementation.

If you want to check the logs for relevant messages you can find these in ./logs/nifi-app.log and ./logs/nifi-bootstrap.log.

I'll add that, from a stylistic point of view, you might be better off looking into one the Spark job servers (spark-jobserver, livy) and using NiFi's HTTP processors to submit jobs that way. It should be more manageable and then you won't need this custom processor.

-joey

> On Nov 16, 2016, at 2:16 AM, shankhamajumdar <[hidden email]> wrote:
>
> Hi,
>
> I am trying to use custom Spark Nifi processor. I have downloaded the
> processor from https://github.com/qiansl127/nifi-spark-bundle and then
> executed the maven build. After the maven build I have copied
> nifi-spark-nar-0.3.0.nar inside NiFi lib folder and restarted the NiFi
> server which is not working.
>
> Please advice.
>
> Regards,
> Shankha
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Spark Processor

shankhamajumdar
Hi joey,

I am trying the POC in different way now.

1. GetFile processor which is getting the files from the local drive.

2. Connecting the processor to the NiFi Output Port.

3. Written a Spark streaming Job which is connecting to the NiFi Output Port to stream the data.

The Spark Job is able to connect the NiFi Output Port successfully. But not able to get any data in spark though in the NiFi the data has been queued from GetFile processor. I have checked the NiFi Data Provenance of the NiFi Output Port but no data is showing there as well where as GetFile to Output Port connector is showing the data has been queued.

Regards,
Shankha
Reply | Threaded
Open this post in threaded view
|

Re: NiFi Spark Processor

Oleg Zhurakousky
Shankha

I know it may not always be all that obvious, but have you started the OutputPort? Also, have you defined ‘nifi.remote.input.socket.port’ property in wifi.properties?
Also, unless you’ve already discovered it there is an excellent blog (https://blogs.apache.org/nifi/entry/stream_processing_nifi_and_spark) from Mark on this which covers every little detail, so hope that will help.

Cheers
Oleg

On Nov 18, 2016, at 12:29 AM, shankhamajumdar <[hidden email]<mailto:[hidden email]>> wrote:

Hi joey,

I am trying the POC in different way now.

1. GetFile processor which is getting the files from the local drive.

2. Connecting the processor to the NiFi Output Port.

3. Written a Spark streaming Job which is connecting to the NiFi Output Port
to stream the data.

The Spark Job is able to connect the NiFi Output Port successfully. But not
able to get any data in spark though in the NiFi the data has been queued
from GetFile processor. I have checked the NiFi Data Provenance of the NiFi
Output Port but no data is showing there as well where as GetFile to Output
Port connector is showing the data has been queued.

Regards,
Shankha



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Spark-Processor-tp13899p13913.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com<http://nabble.com>.