Remote process group networking

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Remote process group networking

Rick Braddy
I have a question about network paths required for proper operation of remote process groups.

By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).

The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?

The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).

If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.

Thanks
Rick
Reply | Threaded
Open this post in threaded view
|

RE: Remote process group networking

Rick Braddy
Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?

Rick

-----Original Message-----
From: Rick Braddy [mailto:[hidden email]]
Sent: Saturday, October 03, 2015 4:59 PM
To: [hidden email]
Subject: Remote process group networking

I have a question about network paths required for proper operation of remote process groups.

By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).

The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?

The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).

If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.

Thanks
Rick
Reply | Threaded
Open this post in threaded view
|

RE: Remote process group networking

Rick Braddy
In reply to this post by Rick Braddy
Still no definitive answers...

My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall.  Even with those iptables rules, it seems something is missing.  I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.

Rick

-----Original Message-----
From: Rick Braddy
Sent: Monday, October 05, 2015 10:18 AM
To: [hidden email]
Subject: RE: Remote process group networking

Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?

Rick

-----Original Message-----
From: Rick Braddy [mailto:[hidden email]]
Sent: Saturday, October 03, 2015 4:59 PM
To: [hidden email]
Subject: Remote process group networking

I have a question about network paths required for proper operation of remote process groups.

By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).

The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?

The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).

If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.

Thanks
Rick
Reply | Threaded
Open this post in threaded view
|

RE: Remote process group networking

Rick Braddy
Just to close this topic off...

First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.

On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall).  No other ports are required across firewall boundaries.

nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).

localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).

Best,
Rick

-----Original Message-----
From: Rick Braddy [mailto:[hidden email]]
Sent: Monday, October 05, 2015 4:45 PM
To: [hidden email]
Subject: RE: Remote process group networking

Still no definitive answers...

My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall.  Even with those iptables rules, it seems something is missing.  I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.

Rick

-----Original Message-----
From: Rick Braddy
Sent: Monday, October 05, 2015 10:18 AM
To: [hidden email]
Subject: RE: Remote process group networking

Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?

Rick

-----Original Message-----
From: Rick Braddy [mailto:[hidden email]]
Sent: Saturday, October 03, 2015 4:59 PM
To: [hidden email]
Subject: Remote process group networking

I have a question about network paths required for proper operation of remote process groups.

By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).

The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?

The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).

If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.

Thanks
Rick
Reply | Threaded
Open this post in threaded view
|

Re: Remote process group networking

Joe Witt
...wonder if we should turn this into a FAQ/explanation.

Thanks for writing this up and following through with resolution Ricky.

On Fri, Oct 16, 2015 at 2:36 PM, Rick Braddy <[hidden email]> wrote:

> Just to close this topic off...
>
> First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.
>
> On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall).  No other ports are required across firewall boundaries.
>
> nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).
>
> localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).
>
> Best,
> Rick
>
> -----Original Message-----
> From: Rick Braddy [mailto:[hidden email]]
> Sent: Monday, October 05, 2015 4:45 PM
> To: [hidden email]
> Subject: RE: Remote process group networking
>
> Still no definitive answers...
>
> My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall.  Even with those iptables rules, it seems something is missing.  I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
>
> Rick
>
> -----Original Message-----
> From: Rick Braddy
> Sent: Monday, October 05, 2015 10:18 AM
> To: [hidden email]
> Subject: RE: Remote process group networking
>
> Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
>
> Rick
>
> -----Original Message-----
> From: Rick Braddy [mailto:[hidden email]]
> Sent: Saturday, October 03, 2015 4:59 PM
> To: [hidden email]
> Subject: Remote process group networking
>
> I have a question about network paths required for proper operation of remote process groups.
>
> By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
>
> The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
>
> The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
>
> If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
>
> Thanks
> Rick
Reply | Threaded
Open this post in threaded view
|

Re: Remote process group networking

Joe Witt
"Rick"  - sorry for the extra Y.

On Tue, Nov 3, 2015 at 9:48 AM, Joe Witt <[hidden email]> wrote:

> ...wonder if we should turn this into a FAQ/explanation.
>
> Thanks for writing this up and following through with resolution Ricky.
>
> On Fri, Oct 16, 2015 at 2:36 PM, Rick Braddy <[hidden email]> wrote:
>> Just to close this topic off...
>>
>> First, I found an error in my remote target node flow that was preventing proper connection from the source node and hampering troubleshooting - had a connector inside a process group, but no connector at top level of graph, which is required for Remote Process Group access.
>>
>> On firewall configuration, indeed only TCP traffic on the UI port 8080 plus the site-to-site port (e.g., 8081) need to be open on the target node for unidirectional site-to-site operation (not required to be open on the source node's firewall).  No other ports are required across firewall boundaries.
>>
>> nifi.remote.input.socket.host must be set to the external (Internet) NAT firewall address is the other key configuration item, because when site-to-site connection is established, the source node must connect to the firewall (not directly to the remote target node's local IP, which is the default if this value is not configured).
>>
>> localhost must also be enabled for local operation, as the "service nifi status" (and probably other stuff) makes calls via localhost (in case you're using iptables, as I was for testing).
>>
>> Best,
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy [mailto:[hidden email]]
>> Sent: Monday, October 05, 2015 4:45 PM
>> To: [hidden email]
>> Subject: RE: Remote process group networking
>>
>> Still no definitive answers...
>>
>> My testing shows that both the NiFi UI port (e.g., 8080) and the data port (e.g., 8081) must be open in both directions through a firewall.  Even with those iptables rules, it seems something is missing.  I will figure it out eventually, and let everyone know what's required to use Nifi across firewall boundaries.
>>
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy
>> Sent: Monday, October 05, 2015 10:18 AM
>> To: [hidden email]
>> Subject: RE: Remote process group networking
>>
>> Let me ask this in a simpler way... for Nifi Remote Process Group communications across firewall boundaries, which ports must be open through firewalls between a source node running the local graph processes and the Remote Process Group node?
>>
>> Rick
>>
>> -----Original Message-----
>> From: Rick Braddy [mailto:[hidden email]]
>> Sent: Saturday, October 03, 2015 4:59 PM
>> To: [hidden email]
>> Subject: Remote process group networking
>>
>> I have a question about network paths required for proper operation of remote process groups.
>>
>> By default, the initial connection from source node to remote process group target node is on port 8080.  Then, there's a second port (e.g., I set it to 8081 and a setting for whether it's SSL secured or not).
>>
>> The question is, are the TCP connection one way, from source node where graph is running to the remote process group's node only, or are bidirectional TCP connections required?
>>
>> The reason I ask is encountering problems trying to connect from data center that has open outbound firewall, but allows no incoming connections.  On the target node, there is no indication in nifi-app.log of the source node even attempting connect (not sure if debug logging is required).
>>
>> If there's some other information on remote process group network topology setup and/or troubleshooting, would be great to read up on it.
>>
>> Thanks
>> Rick