Cluster Setup

classic Classic list List threaded Threaded
5 messages Options
plj
Reply | Threaded
Open this post in threaded view
|

Cluster Setup

plj
Howdy,

  I'm trying to set up a cluster for the 1st time.  I 1st tried to setup a NCM and a node on the same machine.  In nifi.properties I set:
nifi.web.http.port=8081
nifi.cluster.is.node=true
nifi.cluster.node.address=
nifi.cluster.node.protocol.port=8083
nifi.cluster.is.manager=true
nifi.cluster.manager.address=
nifi.cluster.manager.protocol.port=8082

I got the error:
2015-12-18 11:14:56,827 ERROR [NiFi logging handler] org.apache.nifi.StdErr Failed to start web server: ...
nested exception is java.lang.IllegalStateException: Application may be configured as a cluster manager or a node, but not both.

What did I do wrong?  The admin guild says :
"it is also perfectly fine to install the NCM and one of the nodes on the same server, as the NCM is very lightweight".

So I decided I could figure that out later and set nifi.cluster.is.node=false
I started my NCM and then started a node on another machine.  I got the following error.

2015-12-18 11:06:45,112 INFO [Handle Controller Startup Failure Message from [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost, apiPort=8081, socketAddress=localhost, socketPort=8083]] o.a.n.c.manager.impl.WebClusterManager Node Event: [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost, apiPort=8081, socketAddress=localhost, socketPort=8083] -- 'Node could not join cluster because it failed to start up properly. Setting node to Disconnected. Node reported the following error: Failed to connect node to cluster because local flow is different than cluster flow.'

How do I get the cluster flow and the local flow to be the same?

thank you

Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup

Bryan Bende
Hello,

You can run the NCM on the same machine as a node, but they still have to
be two separate processes. You would create two copies of the directory
where you extracted nifi and set one to be the manager and one to be the
node, you'll also have to give them different nifi.web.http.port values.

For the second error, you could try stopping the instance that is the node
and remove the flow.xml.gz that is under conf, and then start it again to
have it pull the flow from the NCM.

-Bryan

On Fri, Dec 18, 2015 at 10:50 AM, plj <[hidden email]> wrote:

> Howdy,
>
>   I'm trying to set up a cluster for the 1st time.  I 1st tried to setup a
> NCM and a node on the same machine.  In nifi.properties I set:
> nifi.web.http.port=8081
> nifi.cluster.is.node=true
> nifi.cluster.node.address=
> nifi.cluster.node.protocol.port=8083
> nifi.cluster.is.manager=true
> nifi.cluster.manager.address=
> nifi.cluster.manager.protocol.port=8082
>
> I got the error:
> 2015-12-18 11:14:56,827 ERROR [NiFi logging handler] org.apache.nifi.StdErr
> Failed to start web server: ...
> nested exception is java.lang.IllegalStateException: Application may be
> configured as a cluster manager or a node, but not both.
>
> What did I do wrong?  The admin guild says :
> "it is also perfectly fine to install the NCM and one of the nodes on the
> same server, as the NCM is very lightweight".
>
> So I decided I could figure that out later and set
> nifi.cluster.is.node=false
> I started my NCM and then started a node on another machine.  I got the
> following error.
>
> 2015-12-18 11:06:45,112 INFO [Handle Controller Startup Failure Message
> from
> [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
> apiPort=8081, socketAddress=localhost, socketPort=8083]]
> o.a.n.c.manager.impl.WebClusterManager Node Event:
> [id=24b57ab2-1853-4814-8c31-4e467d3364e3, apiAddress=localhost,
> apiPort=8081, socketAddress=localhost, socketPort=8083] -- 'Node could not
> join cluster because it failed to start up properly. Setting node to
> Disconnected. Node reported the following error: Failed to connect node to
> cluster because local flow is different than cluster flow.'
>
> How do I get the cluster flow and the local flow to be the same?
>
> thank you
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
plj
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup

plj
Thank you for your help.  I deleted the flow.xml.gz .  So now I have the NCM on machine 'b' and a node on machine 'c'.  I start up a and then start up b.  In b's logs I see logs of heartbeats:
2015-12-18 14:41:05,807 INFO [Process NCM Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 790c06c7-6857-4a90-9d4f-76ec99159369 (type=HEARTBEAT, length=3250 bytes) in 2 millis

On machine c is also see heartbeats:
2015-12-18 14:39:55,741 INFO [Clustering Tasks Thread-3] org.apache.nifi.cluster.heartbeat Heartbeat created at 2015-12-18 14:39:55,664 and sent at 2015-12-18 14:39:55,741; send took 6 millis

Everything seems fine.

Then I try to view NiFi on the  NCM which I set the web to:
nifi.web.http.port=8089

http://machine-b.mitre.org:8089/nifi/

I see:
An unexpected error has occurred

No nodes were able to process this request.


I don't see anything in the logs to suggest what went wrong.

Thoughts?


Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup

Matthew Clarke
I believe you are having issues related to hostname resolution on your
nodes and NCM. By leaving the varies host properties in your NiFi
properties file blank, they are resolving to localhost. Each stem thinks
localhost is itself. Try filling in the properties either with FQDNs that
each node can successfully resolve or populate these host properties with
IPs.  So the rest are multiple host properties to populate. One under
site-to-site properties, one under web properties, and a couple under
cluster properties.
On Dec 18, 2015 3:00 PM, "plj" <[hidden email]> wrote:

> Thank you for your help.  I deleted the flow.xml.gz .  So now I have the
> NCM
> on machine 'b' and a node on machine 'c'.  I start up a and then start up
> b.
> In b's logs I see logs of heartbeats:
> 2015-12-18 14:41:05,807 INFO [Process NCM Request-10]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> 790c06c7-6857-4a90-9d4f-76ec99159369 (type=HEARTBEAT, length=3250 bytes) in
> 2 millis
>
> On machine c is also see heartbeats:
> 2015-12-18 14:39:55,741 INFO [Clustering Tasks Thread-3]
> org.apache.nifi.cluster.heartbeat Heartbeat created at 2015-12-18
> 14:39:55,664 and sent at 2015-12-18 14:39:55,741; send took 6 millis
>
> Everything seems fine.
>
> Then I try to view NiFi on the  NCM which I set the web to:
> nifi.web.http.port=8089
>
> http://machine-b.mitre.org:8089/nifi/
>
> I see:
> An unexpected error has occurred
>
> No nodes were able to process this request.
>
>
> I don't see anything in the logs to suggest what went wrong.
>
> Thoughts?
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Cluster-Setup-tp5853p5860.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
plj
Reply | Threaded
Open this post in threaded view
|

Re: Cluster Setup

plj
Thank you very much.  That solved the problem.  I changed from the default blank value to the name of the machine on the NCM but I still got errors.  When I filled in the value on the node machine it started working.