NiFi remote connections

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi remote connections

Clay Teahouse
Hi All,

Are the connections between the primary node and RPG persistent, and if no,
is there a way to make them persistent?

Similarly, are the http connection from PostHTTP and InvokeHTTP to the
destination persistent, meaning keep-alive is set to true?

thanks a lot

Clay
Reply | Threaded
Open this post in threaded view
|

Re: NiFi remote connections

Koji Kawamura-2
Hi Clay,

RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
no distinction between a primary node and the remote cluster, or nodes
other than the primary node and the remote cluster.
E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
Each node must be able to communicate with every remote node. Node a1
will communicate with all of node b1, b2 and b3. So do node a2 and a3.
Those connections are persistent.
S2S RAW uses socket based connection. S2S HTTP uses
PoolingHttpClientConnectionManager internally to reuse connection.

PostHTTP uses PoolingHttpClientConnectionManager, too.
InvokeHTTP uses different HTTP client library, that is okhttp. I
didn't check it, but I assume it supports keep-alive.

Do you have any specific concern about keep-alive?
The keep-alive technology can be used to improve performance. However,
we should not depend on that for any load-balancing rule.
If you are looking for a solution to distribute FlowFiles based on
some rules, NIFI-5516 will be useful (under development).
https://issues.apache.org/jira/browse/NIFI-5516

Hope this helps.

Thanks,
Koji
On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <[hidden email]> wrote:

>
> Hi All,
>
> Are the connections between the primary node and RPG persistent, and if no,
> is there a way to make them persistent?
>
> Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> destination persistent, meaning keep-alive is set to true?
>
> thanks a lot
>
> Clay
Reply | Threaded
Open this post in threaded view
|

Re: NiFi remote connections

Clay Teahouse
Thanks for the reply, Koji.

In case of RPG, are there circumstances where the connections are not
persistent?


On Tue, Sep 25, 2018 at 12:14 AM Koji Kawamura <[hidden email]>
wrote:

> Hi Clay,
>
> RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
> no distinction between a primary node and the remote cluster, or nodes
> other than the primary node and the remote cluster.
> E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
> Each node must be able to communicate with every remote node. Node a1
> will communicate with all of node b1, b2 and b3. So do node a2 and a3.
> Those connections are persistent.
> S2S RAW uses socket based connection. S2S HTTP uses
> PoolingHttpClientConnectionManager internally to reuse connection.
>
> PostHTTP uses PoolingHttpClientConnectionManager, too.
> InvokeHTTP uses different HTTP client library, that is okhttp. I
> didn't check it, but I assume it supports keep-alive.
>
> Do you have any specific concern about keep-alive?
> The keep-alive technology can be used to improve performance. However,
> we should not depend on that for any load-balancing rule.
> If you are looking for a solution to distribute FlowFiles based on
> some rules, NIFI-5516 will be useful (under development).
> https://issues.apache.org/jira/browse/NIFI-5516
>
> Hope this helps.
>
> Thanks,
> Koji
> On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <[hidden email]>
> wrote:
> >
> > Hi All,
> >
> > Are the connections between the primary node and RPG persistent, and if
> no,
> > is there a way to make them persistent?
> >
> > Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> > destination persistent, meaning keep-alive is set to true?
> >
> > thanks a lot
> >
> > Clay
>
Reply | Threaded
Open this post in threaded view
|

Re: NiFi remote connections

Koji Kawamura-2
Hi Clay,

Excuse me for the confusing response.
I looked at the source code again and do some testing to see how RPG
(HTTP S2S) manages connections.
It uses both sync and async HTTP client simultaneously and it
establishes multiple connection and those are not persistent.
I just remember that in case of RPG, it was difficult to share a
single connection with multiple HTTP requests especially with async
HTTP client while getting along with the existing S2S mechanism.
Currently HTTP S2S protocol consists of a series of requests and uses
multiple connections.

Thanks,
Koji

On Wed, Sep 26, 2018 at 1:16 PM Clay Teahouse <[hidden email]> wrote:

>
> Thanks for the reply, Koji.
>
> In case of RPG, are there circumstances where the connections are not
> persistent?
>
>
> On Tue, Sep 25, 2018 at 12:14 AM Koji Kawamura <[hidden email]>
> wrote:
>
> > Hi Clay,
> >
> > RPG (Site-to-Site) is a Peer-to-Peer communication protocol. There's
> > no distinction between a primary node and the remote cluster, or nodes
> > other than the primary node and the remote cluster.
> > E.g. With Cluster A (node a1, a2 and a3) and Cluster B (node b1, b2 and b3)
> > Each node must be able to communicate with every remote node. Node a1
> > will communicate with all of node b1, b2 and b3. So do node a2 and a3.
> > Those connections are persistent.
> > S2S RAW uses socket based connection. S2S HTTP uses
> > PoolingHttpClientConnectionManager internally to reuse connection.
> >
> > PostHTTP uses PoolingHttpClientConnectionManager, too.
> > InvokeHTTP uses different HTTP client library, that is okhttp. I
> > didn't check it, but I assume it supports keep-alive.
> >
> > Do you have any specific concern about keep-alive?
> > The keep-alive technology can be used to improve performance. However,
> > we should not depend on that for any load-balancing rule.
> > If you are looking for a solution to distribute FlowFiles based on
> > some rules, NIFI-5516 will be useful (under development).
> > https://issues.apache.org/jira/browse/NIFI-5516
> >
> > Hope this helps.
> >
> > Thanks,
> > Koji
> > On Tue, Sep 25, 2018 at 12:07 PM Clay Teahouse <[hidden email]>
> > wrote:
> > >
> > > Hi All,
> > >
> > > Are the connections between the primary node and RPG persistent, and if
> > no,
> > > is there a way to make them persistent?
> > >
> > > Similarly, are the http connection from PostHTTP and InvokeHTTP to the
> > > destination persistent, meaning keep-alive is set to true?
> > >
> > > thanks a lot
> > >
> > > Clay
> >