Load Balancing

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Load Balancing

Mark Bean
I am trying to understand how the load balancing works in NiFi 1.8.0.

I have a 2-node Cluster. I set an UpdateAttribute to set the value of a
property, "balancer", to either 0 or 1. I am using stateful EL for this:
${getStateValue('balancer'):plus(1):mod(2)}.

The connection for the output of the UpdateAttribute processor is load
balanced.
Load Balance Strategy: Partition by attribute
Attribute name: balancer

The queue in the connection contains 5 objects, but when I perform a "List
queue", I only see 3 flowfiles. All of the flowfiles are on the same Node,
and as expected have the same "balancer" attribute value.

Presumably, the other 2 flowfiles were load-balanced to the other Node.
However, they should still be visible in List queue, correct?

Perhaps related, the load balance icon on the connection indicates
"Actively balancing...". There are only two 10 byte files, but the
balancing never seems to complete.
Reply | Threaded
Open this post in threaded view
|

Re: Load Balancing

Mark Bean
I found the problem: iptables was blocking the load balancing port. Once the port was opened, the balance completed and all files were visible via List queue.

In this scenario, should the nifi.cluster.load.balance.comms.timeout have caused the balancing operation to terminate (unsuccessful)?

Another question: the usage of nifi.cluster.load.balance.host (and .port) values is not clear to me. If Node A set's this value for Node B's FQDN, would this allow Node A to "spoof" Node B and accept load balanced items intended for Node B?


On 2018/10/31 17:45:29, Mark Bean <[hidden email]> wrote:

> I am trying to understand how the load balancing works in NiFi 1.8.0.
>
> I have a 2-node Cluster. I set an UpdateAttribute to set the value of a
> property, "balancer", to either 0 or 1. I am using stateful EL for this:
> ${getStateValue('balancer'):plus(1):mod(2)}.
>
> The connection for the output of the UpdateAttribute processor is load
> balanced.
> Load Balance Strategy: Partition by attribute
> Attribute name: balancer
>
> The queue in the connection contains 5 objects, but when I perform a "List
> queue", I only see 3 flowfiles. All of the flowfiles are on the same Node,
> and as expected have the same "balancer" attribute value.
>
> Presumably, the other 2 flowfiles were load-balanced to the other Node.
> However, they should still be visible in List queue, correct?
>
> Perhaps related, the load balance icon on the connection indicates
> "Actively balancing...". There are only two 10 byte files, but the
> balancing never seems to complete.
>
Reply | Threaded
Open this post in threaded view
|

Re: Load Balancing

Koji Kawamura-2
Hi Mark,

> In this scenario, should the nifi.cluster.load.balance.comms.timeout have caused the balancing operation to terminate (unsuccessful)?

I agree with that. Wasn't there any WARN log messages written?
Currently NiFi UI doesn't have capability to show load-balancing
related error on the canvas, other than currently active or not.

> Another question: the usage of nifi.cluster.load.balance.host (and .port) values is not clear to me. If Node A set's this value for Node B's FQDN, would this allow Node A to "spoof" Node B and accept load balanced items intended for Node B?

The two properties are used how a NiFi node opens its socket to
receive load-balanced data from other node. It can be useful when a
node have different NICs and you want to use specific one for
load-balancing. If you specify a hostname or ip-address that is not
for the node, then you'll get an exception when the node tries opening
a socket since it can't bind to the specified address.

Thanks,
Koji
On Thu, Nov 1, 2018 at 3:04 AM [hidden email]
<[hidden email]> wrote:

>
> I found the problem: iptables was blocking the load balancing port. Once the port was opened, the balance completed and all files were visible via List queue.
>
> In this scenario, should the nifi.cluster.load.balance.comms.timeout have caused the balancing operation to terminate (unsuccessful)?
>
> Another question: the usage of nifi.cluster.load.balance.host (and .port) values is not clear to me. If Node A set's this value for Node B's FQDN, would this allow Node A to "spoof" Node B and accept load balanced items intended for Node B?
>
>
> On 2018/10/31 17:45:29, Mark Bean <[hidden email]> wrote:
> > I am trying to understand how the load balancing works in NiFi 1.8.0.
> >
> > I have a 2-node Cluster. I set an UpdateAttribute to set the value of a
> > property, "balancer", to either 0 or 1. I am using stateful EL for this:
> > ${getStateValue('balancer'):plus(1):mod(2)}.
> >
> > The connection for the output of the UpdateAttribute processor is load
> > balanced.
> > Load Balance Strategy: Partition by attribute
> > Attribute name: balancer
> >
> > The queue in the connection contains 5 objects, but when I perform a "List
> > queue", I only see 3 flowfiles. All of the flowfiles are on the same Node,
> > and as expected have the same "balancer" attribute value.
> >
> > Presumably, the other 2 flowfiles were load-balanced to the other Node.
> > However, they should still be visible in List queue, correct?
> >
> > Perhaps related, the load balance icon on the connection indicates
> > "Actively balancing...". There are only two 10 byte files, but the
> > balancing never seems to complete.
> >