wifi questions

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

wifi questions

Clay Teahouse
Hello All,

1) Why would one need both nifi and Kafka in an environment, considering
that NiFi can handle back pressure, set up and deal with queues? I have an
environment where I would be collecting data via NiFi and I would need to
write the data to hdfs after some post processing. Can't I just process the
records in NiFi, change the format and write the data to HDFS via a NiFi
HDFS processor, PutHDFS?

2) Similarly, if I need to do some stream processing, can't I just pull the
data from NiFi processor via NiFiSpout, do the processing via some bolts,
and write the data to HDFS either via HDFS bolt or A NiFi HDFS processor?

Do I even need storm in the picture?

3) Is NiFi suited for microbatch processing? Would it be better to pull the
data from NiFi via spark streaming and do microbatching there? Which
approach is most performant and reliable?

thanks

Clay
Reply | Threaded
Open this post in threaded view
|

Re: wifi questions

Andy LoPresto-2
Hi Clay,

A common use case for NiFi and Kafka in conjunction is when you want the capabilities of a message broker system like Kafka with very low latency and multiple publishers/consumers, but you also have the need for some of the features NiFi provides like backpressure, as you mentioned. This is frequently found in industrial control systems or hardware/IoT integration (sometimes interfacing with MQTT). 

In the scenario you call out in 1), yes, NiFi can be a complete solution for record transformation and writing to HDFS. 

I am not a Storm expert, but you correctly identify NiFi as a good “deliverer” of data to stream processing applications. 

I also won’t address microbatching, but I’m confident some other community members will have good input on the topic. 

I’ve included a couple resources you may find helpful, and I would suggest you might also get good results sending this email to the [hidden email] mailing list, as this list tends to focus more on the internals of NiFi, extensibility, and feature development. The users list has many contributors who deploy NiFi in real world scenarios and integrate with other systems that may not monitor this list. 

Good luck. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On May 2, 2018, at 7:45 PM, Clay Teahouse <[hidden email]> wrote:

Hello All,

1) Why would one need both nifi and Kafka in an environment, considering
that NiFi can handle back pressure, set up and deal with queues? I have an
environment where I would be collecting data via NiFi and I would need to
write the data to hdfs after some post processing. Can't I just process the
records in NiFi, change the format and write the data to HDFS via a NiFi
HDFS processor, PutHDFS?

2) Similarly, if I need to do some stream processing, can't I just pull the
data from NiFi processor via NiFiSpout, do the processing via some bolts,
and write the data to HDFS either via HDFS bolt or A NiFi HDFS processor?

Do I even need storm in the picture?

3) Is NiFi suited for microbatch processing? Would it be better to pull the
data from NiFi via spark streaming and do microbatching there? Which
approach is most performant and reliable?

thanks

Clay


signature.asc (849 bytes) Download Attachment