Distribute FlowFiles among Nodes in a Cluster

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Distribute FlowFiles among Nodes in a Cluster

dale.chang13
I was wondering if there was a way for the NCM to distribute FlowFiles to different nodes.

Currently I see that all of the nodes in my cluster run the same dataflow. I know I can restrict certain processors to the primary node, but it seems like the NCM does not distribute the FlowFiles to different nodes once the primary node have finished their operations.

Additionally, I have checked nifi.properties and I do not have any nifi.state properties configured
Reply | Threaded
Open this post in threaded view
|

Re: Distribute FlowFiles among Nodes in a Cluster

Joe Witt
Dale,

The common pattern here is to use NiFi's site-to-site function to
distribute the load to the rest of the cluster.  So you'd setup a
remote input port on your flow and then connect the output of your
primary only processor to a remote process group of that same cluster
using that new port.

Thanks
Joe

On Mon, May 16, 2016 at 9:13 AM, dale.chang13 <[hidden email]> wrote:

> I was wondering if there was a way for the NCM to distribute FlowFiles to
> different nodes.
>
> Currently I see that all of the nodes in my cluster run the same dataflow. I
> know I can restrict certain processors to the primary node, but it seems
> like the NCM does not distribute the FlowFiles to different nodes once the
> primary node have finished their operations.
>
> Additionally, I have checked nifi.properties and I do not have any
> nifi.state properties configured
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Distribute-FlowFiles-among-Nodes-in-a-Cluster-tp10293.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Distribute FlowFiles among Nodes in a Cluster

Bryan Bende
This post has a diagram of what Joe described:

https://community.hortonworks.com/articles/16120/how-do-i-distribute-data-across-a-nifi-cluster.html

-Bryan

On Mon, May 16, 2016 at 10:20 AM, Joe Witt <[hidden email]> wrote:

> Dale,
>
> The common pattern here is to use NiFi's site-to-site function to
> distribute the load to the rest of the cluster.  So you'd setup a
> remote input port on your flow and then connect the output of your
> primary only processor to a remote process group of that same cluster
> using that new port.
>
> Thanks
> Joe
>
> On Mon, May 16, 2016 at 9:13 AM, dale.chang13 <[hidden email]>
> wrote:
> > I was wondering if there was a way for the NCM to distribute FlowFiles to
> > different nodes.
> >
> > Currently I see that all of the nodes in my cluster run the same
> dataflow. I
> > know I can restrict certain processors to the primary node, but it seems
> > like the NCM does not distribute the FlowFiles to different nodes once
> the
> > primary node have finished their operations.
> >
> > Additionally, I have checked nifi.properties and I do not have any
> > nifi.state properties configured
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Distribute-FlowFiles-among-Nodes-in-a-Cluster-tp10293.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>