Funnels and available thread count

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Funnels and available thread count

Mark Bean
It seems that funnels should not require a thread; they do not actually
process data. However, looking into the code for both 1.8.0 [1] and master
[2], it seems that a thread is allocated to each funnel when gathering the
status of a Process Group. Does this count against threads available to be
allocated to processors?

[1]
https://github.com/apache/nifi/blob/98aabf2c50f857efc72fd6f2bfdd9965b97fa195/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L3326
[2]
https://github.com/apache/nifi/blob/e4be06c7c8a4801e375344954e442bb15aa3f636/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1936

Thanks,
Mark
Reply | Threaded
Open this post in threaded view
|

Re: Funnels and available thread count

Joe Witt
Mark

I believe it does pull from the same thread pool but not sure if it is
'counted'.

Alternatively it could be implemented such that whenever data is written to
a queue we check if the target of that queue is a funnel and if so then
immediately do the right 'next' thing until the next target isn't a
funnel.  Could do the same with local ports..

Thanks

On Fri, Feb 8, 2019 at 10:23 AM Mark Bean <[hidden email]> wrote:

> It seems that funnels should not require a thread; they do not actually
> process data. However, looking into the code for both 1.8.0 [1] and master
> [2], it seems that a thread is allocated to each funnel when gathering the
> status of a Process Group. Does this count against threads available to be
> allocated to processors?
>
> [1]
>
> https://github.com/apache/nifi/blob/98aabf2c50f857efc72fd6f2bfdd9965b97fa195/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L3326
> [2]
>
> https://github.com/apache/nifi/blob/e4be06c7c8a4801e375344954e442bb15aa3f636/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1936
>
> Thanks,
> Mark
>
Reply | Threaded
Open this post in threaded view
|

Re: Funnels and available thread count

Mark Bean
Digging a little deeper I see that a Funnel gets its active thread count
from LifecycleState which is initialized with an activeThreadCount of 0. I
do not believe Funnels ever update this. So, the short answer is that they
do not count towards the active (and therefore available) thread count.

The question arose due to backpressure building on a connection leading
into a funnel. I was curious how this could ever happen. (The output
connection of the funnel has a backpressure setting of 0 object/0 bytes.)


On Fri, Feb 8, 2019 at 10:40 AM Joe Witt <[hidden email]> wrote:

> Mark
>
> I believe it does pull from the same thread pool but not sure if it is
> 'counted'.
>
> Alternatively it could be implemented such that whenever data is written to
> a queue we check if the target of that queue is a funnel and if so then
> immediately do the right 'next' thing until the next target isn't a
> funnel.  Could do the same with local ports..
>
> Thanks
>
> On Fri, Feb 8, 2019 at 10:23 AM Mark Bean <[hidden email]> wrote:
>
> > It seems that funnels should not require a thread; they do not actually
> > process data. However, looking into the code for both 1.8.0 [1] and
> master
> > [2], it seems that a thread is allocated to each funnel when gathering
> the
> > status of a Process Group. Does this count against threads available to
> be
> > allocated to processors?
> >
> > [1]
> >
> >
> https://github.com/apache/nifi/blob/98aabf2c50f857efc72fd6f2bfdd9965b97fa195/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L3326
> > [2]
> >
> >
> https://github.com/apache/nifi/blob/e4be06c7c8a4801e375344954e442bb15aa3f636/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-core/src/main/java/org/apache/nifi/controller/FlowController.java#L1936
> >
> > Thanks,
> > Mark
> >
>