Slow flowfile transfer from process group port to output port.

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow flowfile transfer from process group port to output port.

ashwin.konale@gmail.com
Hi I have the following flow which I am trying to increase the throughput.
I am runnign 3 node nifi cluster(v1.6)

CDC -> processGroupOutputPort -> externalOutputPort
RPG -> downstream

Flowfiles are always queued up between processGroupOutputPort to
externalOutputPort. I dont see any configuration to increase the number of
threads for this operation as well. I have set backpressure to be 10,000
flowfiles. Since the CDC produces at much faster rate than this, I want to
increase the throughput of the flow. Also I see in the UI
externalOutputPort has number of threads running as 3 all the time no
matter how much I increase number of threads to consume in RPG.

Any pointers here will be much helpful.

Ashwin
Reply | Threaded
Open this post in threaded view
|

Re: Slow flowfile transfer from process group port to output port.

Mark Payne
Ashwin,

You'll want to Right-Click on the RPG and then choose to configure Remote Ports. From there, you can
configure how many threads should be used to pull data from each port. So you've updated the Output Port
to use up to 3 threads per node to provide the data, but each node is still only using 1 thread to pull the data.
By configuring the Remote Ports, you can configure the RPG to use 3 threads to pull the data as well.

Thanks
-Mark


> On Sep 25, 2018, at 11:29 AM, ashwin konale <[hidden email]> wrote:
>
> Hi I have the following flow which I am trying to increase the throughput.
> I am runnign 3 node nifi cluster(v1.6)
>
> CDC -> processGroupOutputPort -> externalOutputPort
> RPG -> downstream
>
> Flowfiles are always queued up between processGroupOutputPort to
> externalOutputPort. I dont see any configuration to increase the number of
> threads for this operation as well. I have set backpressure to be 10,000
> flowfiles. Since the CDC produces at much faster rate than this, I want to
> increase the throughput of the flow. Also I see in the UI
> externalOutputPort has number of threads running as 3 all the time no
> matter how much I increase number of threads to consume in RPG.
>
> Any pointers here will be much helpful.
>
> Ashwin

Reply | Threaded
Open this post in threaded view
|

Re: Slow flowfile transfer from process group port to output port.

ashwin.konale@gmail.com


On 2018/09/25 15:34:03, Mark Payne <[hidden email]> wrote:

> Ashwin,
>
> You'll want to Right-Click on the RPG and then choose to configure Remote Ports. From there, you can
> configure how many threads should be used to pull data from each port. So you've updated the Output Port
> to use up to 3 threads per node to provide the data, but each node is still only using 1 thread to pull the data.
> By configuring the Remote Ports, you can configure the RPG to use 3 threads to pull the data as well.
>
> Thanks
> -Mark
>
>
> > On Sep 25, 2018, at 11:29 AM, ashwin konale <[hidden email]> wrote:
> >
> > Hi I have the following flow which I am trying to increase the throughput.
> > I am runnign 3 node nifi cluster(v1.6)
> >
> > CDC -> processGroupOutputPort -> externalOutputPort
> > RPG -> downstream
> >
> > Flowfiles are always queued up between processGroupOutputPort to
> > externalOutputPort. I dont see any configuration to increase the number of
> > threads for this operation as well. I have set backpressure to be 10,000
> > flowfiles. Since the CDC produces at much faster rate than this, I want to
> > increase the throughput of the flow. Also I see in the UI
> > externalOutputPort has number of threads running as 3 all the time no
> > matter how much I increase number of threads to consume in RPG.
> >
> > Any pointers here will be much helpful.
> >
> > Ashwin
>
> Hello Mark, Thanks a lot for the reply.
I have already configured number of threads to in remote ports in RPG. But no matter how many threads I set it to, the number of threads running in output port is showing as 3. And there is no setting to update the number of threads for the output port. I quickly went through LocalPort.java . (https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/connectable/LocalPort.java) Looks like it transfers in the batches of 100. Is there any way to increase the throughput.?
Reply | Threaded
Open this post in threaded view
|

Re: Slow flowfile transfer from process group port to output port.

ashwin.konale@gmail.com
In reply to this post by Mark Payne


On 2018/09/25 15:34:03, Mark Payne <[hidden email]> wrote:

> Ashwin,
>
> You'll want to Right-Click on the RPG and then choose to configure Remote Ports. From there, you can
> configure how many threads should be used to pull data from each port. So you've updated the Output Port
> to use up to 3 threads per node to provide the data, but each node is still only using 1 thread to pull the data.
> By configuring the Remote Ports, you can configure the RPG to use 3 threads to pull the data as well.
>
> Thanks
> -Mark
>
>
> > On Sep 25, 2018, at 11:29 AM, ashwin konale <[hidden email]> wrote:
> >
> > Hi I have the following flow which I am trying to increase the throughput.
> > I am runnign 3 node nifi cluster(v1.6)
> >
> > CDC -> processGroupOutputPort -> externalOutputPort
> > RPG -> downstream
> >
> > Flowfiles are always queued up between processGroupOutputPort to
> > externalOutputPort. I dont see any configuration to increase the number of
> > threads for this operation as well. I have set backpressure to be 10,000
> > flowfiles. Since the CDC produces at much faster rate than this, I want to
> > increase the throughput of the flow. Also I see in the UI
> > externalOutputPort has number of threads running as 3 all the time no
> > matter how much I increase number of threads to consume in RPG.
> >
> > Any pointers here will be much helpful.
> >
> > Ashwin
>
> Hi Mark
Thanks a lot for the reply. I have already configured RPG to use more threads on remote ports. But since nifi does not provide any option to increase number of threads on externalOutputPort, it by default runs on 3 threads. (I am not sure how it chose 3 threads.) So flowfiles are always blocked between processGroupOutputPort to externalOutputPort. I quickly went through sourcecode of LocalPort.java. Looks like it transfers in batches of 100. Can you provide some guidance here on how to improve the throughput ?
Thanks
- Ashwin
Reply | Threaded
Open this post in threaded view
|

Re: Slow flowfile transfer from process group port to output port.

Ashwin Konale
In reply to this post by ashwin.konale@gmail.com
Hello Mike,
Thanks a lot for looking at it. But I have already configured threads for remote port in RPG. But congestion is in the transfer between process group to external port. And there are no configuration option to increase number of threads in externalOutputPort. Number of running threads for output port is always showing as 3 in ui. So I am not sure how to increase the throughput now. I quickly went through implementation of LocalPort.java. Looks like flowflies are transferred at batches of 100. Can you give me any other suggestion to increase throughput ?

-Ashwin

On 2018/09/25 15:29:32, ashwin konale <[hidden email]> wrote:

> Hi I have the following flow which I am trying to increase the throughput.
> I am runnign 3 node nifi cluster(v1.6)
>
> CDC -> processGroupOutputPort -> externalOutputPort
> RPG -> downstream
>
> Flowfiles are always queued up between processGroupOutputPort to
> externalOutputPort. I dont see any configuration to increase the number of
> threads for this operation as well. I have set backpressure to be 10,000
> flowfiles. Since the CDC produces at much faster rate than this, I want to
> increase the throughput of the flow. Also I see in the UI
> externalOutputPort has number of threads running as 3 all the time no
> matter how much I increase number of threads to consume in RPG.
>
> Any pointers here will be much helpful.
>
> Ashwin
>