Failed to connect node to cluster because local flow is different than cluster flow.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Failed to connect node to cluster because local flow is different than cluster flow.

Oscar de la Pena
How does NiFi handle syncing of data flow setup across slave nodes in a cluster?

I changed my data flow while one of my node was disconnected. Now, I can't reconnect it back
to the cluster and the log file has this error message:

2015-03-27 14:53:08,289 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
java.lang.Exception: Unable to load flow due to: java.io.IOException: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster because
 local flow is different than cluster flow.

Thanks!
Owie
Reply | Threaded
Open this post in threaded view
|

Re: Failed to connect node to cluster because local flow is different than cluster flow.

Corey Flowers
Good morning Owie,

       When you are running in a cluster, the NCM syncs the flow.xml.gz
between all of your nodes. The gold copy is kept on the NCM within the
flow.tar in the conf directory. Actually when you look at the graph (GUI),
you are looking at the graph of the NCM not a node within the cluster. It
does this to protect the integrity of the changes. Meaning, if you change a
node that is disconnected and someone else, changes the graph still running
in the cluster, how would the NCM know which changes took priority or to
over write?

On another note, if a node is down within your cluster, the NCM will not
let you change the graph either, because, again, it doesn't want to change
the underlying flow.xml.gz and cause the node that has fallen out to be out
of sync. If you want to change the cluster graph, you will need to
disconnect the down node from the cluster or work to get it back into the
cluster so that you can work in the graph again.

If you want the node to reconnect, you will need to delete the flow.xml.gz
from the node and restart. It will then call up to the NCM and get a copy
of the flow.xml.gz again, from the master gold copy kept within the
flow.tar. Unfortunately you will have to redo your changes. Normally the
only time operationally we would disconnect a node to make changes is to
troubleshoot an error on the node in question. Other than that, all changes
would be done within the Cluster graph.

BTW, we call it a graph because of the graph paper background. :-)


If you have any other questions, just let me know.

Thanks,
Corey Flowers


On Fri, Mar 27, 2015 at 3:08 AM, Oscar dela Pena <[hidden email]>
wrote:

> How does NiFi handle syncing of data flow setup across slave nodes in a
> cluster?
>
> I changed my data flow while one of my node was disconnected. Now, I can't
> reconnect it back
> to the cluster and the log file has this error message:
>
> 2015-03-27 14:53:08,289 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster because
>  local flow is different than cluster flow.
>
> Thanks!
> Owie
>



--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: Failed to connect node to cluster because local flow is different than cluster flow.

Oscar de la Pena
Hi Cory, thank you for the detailed explanation.

I remember all nodes were on CONNECTED status when I added a  new processor. It seemed like one node got disconnected while syncing the changes I made in the NCM. The nodes list indicated DISCONNECTED status for that one node. When I tried reconnecting it via the UI, that's when the error occurred and I couldn't got it synced even after several reconnect tries.

I will try the fix you provided. Thanks again.


----- Original Message -----
From: Corey Flowers <[hidden email]>
To: [hidden email]
Sent: Fri, 27 Mar 2015 20:55:08 +0800 (PHT)
Subject: Re: Failed to connect node to cluster because local flow is different than cluster flow.

Good morning Owie,

 When you are running in a cluster, the NCM syncs the flow.xml.gz
between all of your nodes. The gold copy is kept on the NCM within the
flow.tar in the conf directory. Actually when you look at the graph (GUI),
you are looking at the graph of the NCM not a node within the cluster. It
does this to protect the integrity of the changes. Meaning, if you change a
node that is disconnected and someone else, changes the graph still running
in the cluster, how would the NCM know which changes took priority or to
over write?

On another note, if a node is down within your cluster, the NCM will not
let you change the graph either, because, again, it doesn't want to change
the underlying flow.xml.gz and cause the node that has fallen out to be out
of sync. If you want to change the cluster graph, you will need to
disconnect the down node from the cluster or work to get it back into the
cluster so that you can work in the graph again.

If you want the node to reconnect, you will need to delete the flow.xml.gz
from the node and restart. It will then call up to the NCM and get a copy
of the flow.xml.gz again, from the master gold copy kept within the
flow.tar. Unfortunately you will have to redo your changes. Normally the
only time operationally we would disconnect a node to make changes is to
troubleshoot an error on the node in question. Other than that, all changes
would be done within the Cluster graph.

BTW, we call it a graph because of the graph paper background. :-)


If you have any other questions, just let me know.

Thanks,
Corey Flowers


On Fri, Mar 27, 2015 at 3:08 AM, Oscar dela Pena <[hidden email]>
wrote:

> How does NiFi handle syncing of data flow setup across slave nodes in a
> cluster?
>
> I changed my data flow while one of my node was disconnected. Now, I can't
> reconnect it back
> to the cluster and the log file has this error message:
>
> 2015-03-27 14:53:08,289 WARN [main] org.apache.nifi.web.server.JettyServer
> Failed to start web server... shutting down.
> java.lang.Exception: Unable to load flow due to: java.io.IOException:
> org.apache.nifi.cluster.ConnectionException: Failed to connect node to
> cluster because
> local flow is different than cluster flow.
>
> Thanks!
> Owie
>



--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --