Reg. Nifi Clustering Load Balancing

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Reg. Nifi Clustering Load Balancing

Nishant Gupta
Hello Sir/Madam,

I need to know how we can implement load balancing and single point access
to all data producer

I have suppose 100 machines that produce data.(flowfiles)
That data is sent over HTTP from 100 machines to nifi and that we can
access using listenHTTP processor in Nifi.(Currently I am able to do that
for single node Nifi)
Is there any way to provide 100 machines with a single URL (for all 4-5
nodes in Nifi Cluster)

Can you please let me know how to achieve it.??

Thanks and Regards,
Nishant Gupta
Reply | Threaded
Open this post in threaded view
|

Re: Reg. Nifi Clustering Load Balancing

mayank rathi
Does this help?



On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta <[hidden email]> wrote:
Hello Sir/Madam,

I need to know how we can implement load balancing and single point access
to all data producer

I have suppose 100 machines that produce data.(flowfiles)
That data is sent over HTTP from 100 machines to nifi and that we can
access using listenHTTP processor in Nifi.(Currently I am able to do that
for single node Nifi)
Is there any way to provide 100 machines with a single URL (for all 4-5
nodes in Nifi Cluster)

Can you please let me know how to achieve it.??

Thanks and Regards,
Nishant Gupta



--
NOTICE: This email message is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.
Reply | Threaded
Open this post in threaded view
|

Re: Reg. Nifi Clustering Load Balancing

Bryan Bende
Hello,

You can run a standard HTTP load-balancer in front of ListenHTTP and have
your producers use the URL of the load-balancer.

Nginx or apache httpd can be used.

Thanks,

Bryan


On Tue, Aug 29, 2017 at 11:40 AM, mayank rathi <[hidden email]>
wrote:

> Does this help?
>
> [image: Inline image 1]
>
> On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta <
> [hidden email]> wrote:
>
>> Hello Sir/Madam,
>>
>> I need to know how we can implement load balancing and single point access
>> to all data producer
>>
>> I have suppose 100 machines that produce data.(flowfiles)
>> That data is sent over HTTP from 100 machines to nifi and that we can
>> access using listenHTTP processor in Nifi.(Currently I am able to do that
>> for single node Nifi)
>> Is there any way to provide 100 machines with a single URL (for all 4-5
>> nodes in Nifi Cluster)
>>
>> Can you please let me know how to achieve it.??
>>
>> Thanks and Regards,
>> Nishant Gupta
>>
>
>
>
> --
> NOTICE: This email message is for the sole use of the intended
> recipient(s) and may contain confidential and privileged information. Any
> unauthorized review, use, disclosure or distribution is prohibited. If you
> are not the intended recipient, please contact the sender by reply email
> and destroy all copies of the original message.
>
Reply | Threaded
Open this post in threaded view
|

Re: Reg. Nifi Clustering Load Balancing

Kevin Doran
Hi Nishant,

A HTTP load-balancer as Bryan suggests is a good option for server-side load balancing so that your client only needs to keep track of a single URL. In addition to the options Bryan mentioned, which are great, most cloud providers offer managed load balancing services, which many be convenient if you happen to be using a commercial cloud for your NiFi cluster.

Another option is client-side load balancing, in which clients of the NiFi cluster discover all available nodes in the cluster and cycle through them. There are many ways of implementing client-side load balancing, most of which involve periodically refreshing a list of server nodes/endpoints. If you want to explore this path, the NiFi API [1] makes it easy to discover all nodes known to the cluster and their status using this endpoint:

GET /controller/cluster

This can be called on any node in the cluster. It will return a JSON object that includes an array of nodes, each of which has an address and port you could use to build a URL as well as some other status information that might be useful for implementing dynamic load-balancing logic in the client. This could be consumed directly by the client or by an agent of a service discovery framework that would maintain a directory/registry of endpoints. Examples of frameworks that can be used as a service discovery registry include Apache Zookeeper, Consul, Netlfix Eureka, and etcd.

The easiest way to get started for most is definitely the server-side load balancing approach using an HTTP load balancer. There are pros and cons to both server-side and client-side load balancing that one has to evaluate based on a particular use case. So I agree with Bryan's advice, and am only pointing out another option that may be preferable to some on this list that have a similar need.

[1] https://nifi.apache.org/docs/nifi-docs/rest-api/ 

Cheers,
Kevin

On 8/29/17, 12:15, "Bryan Bende" <[hidden email]> wrote:

    Hello,
   
    You can run a standard HTTP load-balancer in front of ListenHTTP and have
    your producers use the URL of the load-balancer.
   
    Nginx or apache httpd can be used.
   
    Thanks,
   
    Bryan
   
   
    On Tue, Aug 29, 2017 at 11:40 AM, mayank rathi <[hidden email]>
    wrote:
   
    > Does this help?
    >
    > [image: Inline image 1]
    >
    > On Tue, Aug 29, 2017 at 11:07 AM, Nishant Gupta <
    > [hidden email]> wrote:
    >
    >> Hello Sir/Madam,
    >>
    >> I need to know how we can implement load balancing and single point access
    >> to all data producer
    >>
    >> I have suppose 100 machines that produce data.(flowfiles)
    >> That data is sent over HTTP from 100 machines to nifi and that we can
    >> access using listenHTTP processor in Nifi.(Currently I am able to do that
    >> for single node Nifi)
    >> Is there any way to provide 100 machines with a single URL (for all 4-5
    >> nodes in Nifi Cluster)
    >>
    >> Can you please let me know how to achieve it.??
    >>
    >> Thanks and Regards,
    >> Nishant Gupta
    >>
    >
    >
    >
    > --
    > NOTICE: This email message is for the sole use of the intended
    > recipient(s) and may contain confidential and privileged information. Any
    > unauthorized review, use, disclosure or distribution is prohibited. If you
    > are not the intended recipient, please contact the sender by reply email
    > and destroy all copies of the original message.
    >