NIFI single node in cluster mode

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NIFI single node in cluster mode

Milan Das
Hello Nifi team,

Is it possible to run a single NIFI node in cluster mode ? I have this requirement because we will add other nodes soon down line.

I tried that by setting  “nifi.cluster.is.node” but and zookeeper setting. But seems it waits ever for election.

 

Appreciate your thoughts.

 

Thanks,

Milan Das

Reply | Threaded
Open this post in threaded view
|

Re: NIFI single node in cluster mode

Jon Logan
It waits for election for a specific period of time, which if I recall is
fairly high (I think 5 minutes?). If you lower this it'll still wait for an
election but will complete faster (we do 30 seconds, but you could do
lower). There's a property controlling this.

On Fri, Oct 12, 2018 at 5:41 PM Milan Das <[hidden email]> wrote:

> Hello Nifi team,
>
> Is it possible to run a single NIFI node in cluster mode ? I have this
> requirement because we will add other nodes soon down line.
>
> I tried that by setting  “nifi.cluster.is.node” but and zookeeper setting.
> But seems it waits ever for election.
>
>
>
> Appreciate your thoughts.
>
>
>
> Thanks,
>
> Milan Das
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NIFI single node in cluster mode

Bryan Bende
There is also another property for the # of candidates to wait for when
voting, if it sees the # of candidates first it will short circuit the time
period. So setting the candidates to 1 for a single node cluster should
start immediately.

On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <[hidden email]> wrote:

> It waits for election for a specific period of time, which if I recall is
> fairly high (I think 5 minutes?). If you lower this it'll still wait for an
> election but will complete faster (we do 30 seconds, but you could do
> lower). There's a property controlling this.
>
> On Fri, Oct 12, 2018 at 5:41 PM Milan Das <[hidden email]> wrote:
>
> > Hello Nifi team,
> >
> > Is it possible to run a single NIFI node in cluster mode ? I have this
> > requirement because we will add other nodes soon down line.
> >
> > I tried that by setting  “nifi.cluster.is.node” but and zookeeper
> setting.
> > But seems it waits ever for election.
> >
> >
> >
> > Appreciate your thoughts.
> >
> >
> >
> > Thanks,
> >
> > Milan Das
> >
> >
>
--
Sent from Gmail Mobile
Reply | Threaded
Open this post in threaded view
|

Re: NIFI single node in cluster mode

Milan Das
Problem is I am using Kerbrized zookeeper and it is failing to create nifi basepath. Even if TGT is getting created Authentication is failing.


2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT refresh thread started.
2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT valid starting at:        Sat Oct 13 13:33:53 UTC 2018
2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT expires:                  Sun Oct 14 13:33:53 UTC 2018
2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
2018-10-13 13:33:53,577 INFO [main-SendThread(ip-172-30-1-132.ec2.internal:2181)] o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL mechanism.
2018-10-13 13:33:53,606 INFO [main-EventThread] o.a.c.f.state.ConnectionStateManager State change: CONNECTED
2018-10-13 13:33:53,616 ERROR [main-SendThread(ip-172-30-1-132.ec2.internal:2181)] o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using login context 'Client'.
2018-10-13 13:33:53,723 WARN [main] o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected Leader for role 'Cluster Coordinator' due to org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode = AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has been elected
2018-10-13 13:33:53,724 INFO [Curator-Framework-0] o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
2018-10-13 13:33:53,726 INFO [main] o.apache.nifi.controller.FlowController It appears that no Cluster Coordinator has been Elected yet. Registering for Cluster Coordinator Role.


Thanks,
Milan Das


On 10/12/18, 6:26 PM, "Bryan Bende" <[hidden email]> wrote:

    There is also another property for the # of candidates to wait for when
    voting, if it sees the # of candidates first it will short circuit the time
    period. So setting the candidates to 1 for a single node cluster should
    start immediately.
   
    On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <[hidden email]> wrote:
   
    > It waits for election for a specific period of time, which if I recall is
    > fairly high (I think 5 minutes?). If you lower this it'll still wait for an
    > election but will complete faster (we do 30 seconds, but you could do
    > lower). There's a property controlling this.
    >
    > On Fri, Oct 12, 2018 at 5:41 PM Milan Das <[hidden email]> wrote:
    >
    > > Hello Nifi team,
    > >
    > > Is it possible to run a single NIFI node in cluster mode ? I have this
    > > requirement because we will add other nodes soon down line.
    > >
    > > I tried that by setting  “nifi.cluster.is.node” but and zookeeper
    > setting.
    > > But seems it waits ever for election.
    > >
    > >
    > >
    > > Appreciate your thoughts.
    > >
    > >
    > >
    > > Thanks,
    > >
    > > Milan Das
    > >
    > >
    >
    --
    Sent from Gmail Mobile
   


Reply | Threaded
Open this post in threaded view
|

Re: NIFI single node in cluster mode

Jeff
Milan,

If you haven't already done so, please take a look at the NiFi Admin
Guide's sections "Securing Zookeeper" [1] and "Kerberizing NiFi’s ZooKeeper
Client" [2], which should help you configure NiFi to use a kerberized
ZooKeeper.

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#securing_zookeeper
[2]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client

On Sat, Oct 13, 2018 at 9:38 AM Milan Das <[hidden email]> wrote:

> Problem is I am using Kerbrized zookeeper and it is failing to create nifi
> basepath. Even if TGT is getting created Authentication is failing.
>
>
> 2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh thread started.
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> valid starting at:        Sat Oct 13 13:33:53 UTC 2018
> 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
> expires:                  Sun Oct 14 13:33:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT
> refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
> 2018-10-13 13:33:53,577 INFO
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL
> mechanism.
> 2018-10-13 13:33:53,606 INFO [main-EventThread]
> o.a.c.f.state.ConnectionStateManager State change: CONNECTED
> 2018-10-13 13:33:53,616 ERROR
> [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
> o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using
> login context 'Client'.
> 2018-10-13 13:33:53,723 WARN [main]
> o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected
> Leader for role 'Cluster Coordinator' due to
> org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
> AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has
> been elected
> 2018-10-13 13:33:53,724 INFO [Curator-Framework-0]
> o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
> 2018-10-13 13:33:53,726 INFO [main]
> o.apache.nifi.controller.FlowController It appears that no Cluster
> Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
>
>
> Thanks,
> Milan Das
>
>
> On 10/12/18, 6:26 PM, "Bryan Bende" <[hidden email]> wrote:
>
>     There is also another property for the # of candidates to wait for when
>     voting, if it sees the # of candidates first it will short circuit the
> time
>     period. So setting the candidates to 1 for a single node cluster should
>     start immediately.
>
>     On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <[hidden email]> wrote:
>
>     > It waits for election for a specific period of time, which if I
> recall is
>     > fairly high (I think 5 minutes?). If you lower this it'll still wait
> for an
>     > election but will complete faster (we do 30 seconds, but you could do
>     > lower). There's a property controlling this.
>     >
>     > On Fri, Oct 12, 2018 at 5:41 PM Milan Das <[hidden email]> wrote:
>     >
>     > > Hello Nifi team,
>     > >
>     > > Is it possible to run a single NIFI node in cluster mode ? I have
> this
>     > > requirement because we will add other nodes soon down line.
>     > >
>     > > I tried that by setting  “nifi.cluster.is.node” but and zookeeper
>     > setting.
>     > > But seems it waits ever for election.
>     > >
>     > >
>     > >
>     > > Appreciate your thoughts.
>     > >
>     > >
>     > >
>     > > Thanks,
>     > >
>     > > Milan Das
>     > >
>     > >
>     >
>     --
>     Sent from Gmail Mobile
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: NIFI single node in cluster mode

Milan Das
Thanks all for the advise.
Found the problem. I was adding two Djava parameters in java.arg.N. I added then two different  lines it worked.

Now I see the problem when I list Queue on Success Queye: My flow is simple GenerateFlowFile (success) --> Funnel.
Yes I added all policies at root level to user nifiadmin1. This works when I set the cluster to false.

NIFI version : 1.6.0



Error:

2018-10-14 15:03:21,620 INFO [NiFi Web Server-38] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for [hidden email]
2018-10-14 15:03:21,621 INFO [NiFi Web Server-38] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[[hidden email]], groups[] does not have permission to access the requested resource. Unable to view the data for Processor with ID 7312084e-0166-1000-0000-00006ef08dd3. Returning Forbidden response.
2018-10-14 15:03:21,623 INFO [NiFi Web Server-40] o.a.n.w.a.c.AccessDeniedExceptionMapper identity[[hidden email]], groups[] does not have permission to access the requested resource. Node ip-172-30-1-235.ec2.internal:8443 is unable to fulfill this request due to: Unable to view the data for Processor with ID 7312084e-0166-1000-0000-00006ef08dd3. Contact the system administrator. Returning Forbidden response.
2018-10-14 15:03:21,633 INFO [NiFi Web Server-138] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<[hidden email]><CN=ip-172-30-1-235.ec2.internal, O=Interset, ST=California, C=US>) POST https://ip-172-30-1-235.ec2.internal:8443/nifi-api/flowfile-queues/73121f31-0166-1000-0000-000024972726/listing-requests (source ip: 172.30.1.235)
2018-10-14 15:03:21,633 INFO [NiFi Web Server-138] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for nifiadmin1@

Regards,
Milan Das


Milan Das
Sr. System Architect
email: [hidden email]
<https://www.linkedin.com/in/milandas/>
www.interset.com <http://www.interset.com/>
 


On 10/13/18, 2:39 PM, "Jeff" <[hidden email]> wrote:

    Milan,
   
    If you haven't already done so, please take a look at the NiFi Admin
    Guide's sections "Securing Zookeeper" [1] and "Kerberizing NiFi’s ZooKeeper
    Client" [2], which should help you configure NiFi to use a kerberized
    ZooKeeper.
   
    [1]
    https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#securing_zookeeper
    [2]
    https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#zk_kerberos_client
   
    On Sat, Oct 13, 2018 at 9:38 AM Milan Das <[hidden email]> wrote:
   
    > Problem is I am using Kerbrized zookeeper and it is failing to create nifi
    > basepath. Even if TGT is getting created Authentication is failing.
    >
    >
    > 2018-10-13 13:33:53,573 INFO [Thread-12] org.apache.zookeeper.Login TGT
    > refresh thread started.
    > 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
    > valid starting at:        Sat Oct 13 13:33:53 UTC 2018
    > 2018-10-13 13:33:53,576 INFO [Thread-12] org.apache.zookeeper.Login TGT
    > expires:                  Sun Oct 14 13:33:53 UTC 2018
    > 2018-10-13 13:33:53,577 INFO [Thread-12] org.apache.zookeeper.Login TGT
    > refresh sleeping until: Sun Oct 14 09:38:53 UTC 2018
    > 2018-10-13 13:33:53,577 INFO
    > [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
    > o.a.zookeeper.client.ZooKeeperSaslClient Client will use GSSAPI as SASL
    > mechanism.
    > 2018-10-13 13:33:53,606 INFO [main-EventThread]
    > o.a.c.f.state.ConnectionStateManager State change: CONNECTED
    > 2018-10-13 13:33:53,616 ERROR
    > [main-SendThread(ip-172-30-1-132.ec2.internal:2181)]
    > o.a.zookeeper.client.ZooKeeperSaslClient SASL authentication failed using
    > login context 'Client'.
    > 2018-10-13 13:33:53,723 WARN [main]
    > o.a.n.c.l.e.CuratorLeaderElectionManager Unable to determine the Elected
    > Leader for role 'Cluster Coordinator' due to
    > org.apache.zookeeper.KeeperException$AuthFailedException: KeeperErrorCode =
    > AuthFailed for /nifi/leaders/Cluster Coordinator; assuming no leader has
    > been elected
    > 2018-10-13 13:33:53,724 INFO [Curator-Framework-0]
    > o.a.c.f.imps.CuratorFrameworkImpl backgroundOperationsLoop exiting
    > 2018-10-13 13:33:53,726 INFO [main]
    > o.apache.nifi.controller.FlowController It appears that no Cluster
    > Coordinator has been Elected yet. Registering for Cluster Coordinator Role.
    >
    >
    > Thanks,
    > Milan Das
    >
    >
    > On 10/12/18, 6:26 PM, "Bryan Bende" <[hidden email]> wrote:
    >
    >     There is also another property for the # of candidates to wait for when
    >     voting, if it sees the # of candidates first it will short circuit the
    > time
    >     period. So setting the candidates to 1 for a single node cluster should
    >     start immediately.
    >
    >     On Fri, Oct 12, 2018 at 5:59 PM Jon Logan <[hidden email]> wrote:
    >
    >     > It waits for election for a specific period of time, which if I
    > recall is
    >     > fairly high (I think 5 minutes?). If you lower this it'll still wait
    > for an
    >     > election but will complete faster (we do 30 seconds, but you could do
    >     > lower). There's a property controlling this.
    >     >
    >     > On Fri, Oct 12, 2018 at 5:41 PM Milan Das <[hidden email]> wrote:
    >     >
    >     > > Hello Nifi team,
    >     > >
    >     > > Is it possible to run a single NIFI node in cluster mode ? I have
    > this
    >     > > requirement because we will add other nodes soon down line.
    >     > >
    >     > > I tried that by setting  “nifi.cluster.is.node” but and zookeeper
    >     > setting.
    >     > > But seems it waits ever for election.
    >     > >
    >     > >
    >     > >
    >     > > Appreciate your thoughts.
    >     > >
    >     > >
    >     > >
    >     > > Thanks,
    >     > >
    >     > > Milan Das
    >     > >
    >     > >
    >     >
    >     --
    >     Sent from Gmail Mobile
    >
    >
    >
    >