Threads

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Threads

Edgardo Vega
The admin guide says the Maximum Forked Processes section says  "NiFi may
be configured to generate a significant number of threads." How is this
done? I read the rest of the documentation and didn't see a way to define
the number of threads to use. Also any recommendation on number of threads
per code?

I have been seeing the number of threads in the gui pegged at 10 that is
why I went looking for how to change that number.

Thanks in advance,

Edgardo
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Matt Gilman
The existing documentation is a little short but it is mentioned in the
Other Management Features section [1]. In the UI can you click the wrench
and screwdriver icon in the upper right. There you can set the name of your
dataflow, it's description, and the size of the timer/event driven thread
pools.

Matt

[1]
https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features

On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]> wrote:

> The admin guide says the Maximum Forked Processes section says  "NiFi may
> be configured to generate a significant number of threads." How is this
> done? I read the rest of the documentation and didn't see a way to define
> the number of threads to use. Also any recommendation on number of threads
> per code?
>
> I have been seeing the number of threads in the gui pegged at 10 that is
> why I went looking for how to change that number.
>
> Thanks in advance,
>
> Edgardo
>
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Edgardo Vega
Thanks Matt. Is this configurable in a file or through a java argument?
That way we can set that via puppet.

Cheers,

Edgardo

On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]> wrote:

> The existing documentation is a little short but it is mentioned in the
> Other Management Features section [1]. In the UI can you click the wrench
> and screwdriver icon in the upper right. There you can set the name of your
> dataflow, it's description, and the size of the timer/event driven thread
> pools.
>
> Matt
>
> [1]
>
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>
> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]>
> wrote:
>
> > The admin guide says the Maximum Forked Processes section says  "NiFi may
> > be configured to generate a significant number of threads." How is this
> > done? I read the rest of the documentation and didn't see a way to define
> > the number of threads to use. Also any recommendation on number of
> threads
> > per code?
> >
> > I have been seeing the number of threads in the gui pegged at 10 that is
> > why I went looking for how to change that number.
> >
> > Thanks in advance,
> >
> > Edgardo
> >
>



--
Cheers,

Edgardo
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Matt Gilman
It's actually saved as part of your flow. So assuming your flow is part of
your puppet configuration, you should be good. In a standalone instance it
would be in <NIFI_HOME>/conf/flow.xml.gz.

Matt

On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <[hidden email]> wrote:

> Thanks Matt. Is this configurable in a file or through a java argument?
> That way we can set that via puppet.
>
> Cheers,
>
> Edgardo
>
> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]>
> wrote:
>
> > The existing documentation is a little short but it is mentioned in the
> > Other Management Features section [1]. In the UI can you click the wrench
> > and screwdriver icon in the upper right. There you can set the name of
> your
> > dataflow, it's description, and the size of the timer/event driven thread
> > pools.
> >
> > Matt
> >
> > [1]
> >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> >
> > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]>
> > wrote:
> >
> > > The admin guide says the Maximum Forked Processes section says  "NiFi
> may
> > > be configured to generate a significant number of threads." How is this
> > > done? I read the rest of the documentation and didn't see a way to
> define
> > > the number of threads to use. Also any recommendation on number of
> > threads
> > > per code?
> > >
> > > I have been seeing the number of threads in the gui pegged at 10 that
> is
> > > why I went looking for how to change that number.
> > >
> > > Thanks in advance,
> > >
> > > Edgardo
> > >
> >
>
>
>
> --
> Cheers,
>
> Edgardo
>
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Edgardo Vega
That's an interesting place to store that information wouldn't it make more
sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
machines with different specs you would have to manually tweak the flow on
each machine.

I would gladly make a ticket if it seems to make more sense for these
properties to be moved.  What's the consensus?

On Wednesday, May 6, 2015, Matt Gilman <[hidden email]> wrote:

> It's actually saved as part of your flow. So assuming your flow is part of
> your puppet configuration, you should be good. In a standalone instance it
> would be in <NIFI_HOME>/conf/flow.xml.gz.
>
> Matt
>
> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <[hidden email]
> <javascript:;>> wrote:
>
> > Thanks Matt. Is this configurable in a file or through a java argument?
> > That way we can set that via puppet.
> >
> > Cheers,
> >
> > Edgardo
> >
> > On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]
> <javascript:;>>
> > wrote:
> >
> > > The existing documentation is a little short but it is mentioned in the
> > > Other Management Features section [1]. In the UI can you click the
> wrench
> > > and screwdriver icon in the upper right. There you can set the name of
> > your
> > > dataflow, it's description, and the size of the timer/event driven
> thread
> > > pools.
> > >
> > > Matt
> > >
> > > [1]
> > >
> > >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> > >
> > > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]
> <javascript:;>>
> > > wrote:
> > >
> > > > The admin guide says the Maximum Forked Processes section says  "NiFi
> > may
> > > > be configured to generate a significant number of threads." How is
> this
> > > > done? I read the rest of the documentation and didn't see a way to
> > define
> > > > the number of threads to use. Also any recommendation on number of
> > > threads
> > > > per code?
> > > >
> > > > I have been seeing the number of threads in the gui pegged at 10 that
> > is
> > > > why I went looking for how to change that number.
> > > >
> > > > Thanks in advance,
> > > >
> > > > Edgardo
> > > >
> > >
> >
> >
> >
> > --
> > Cheers,
> >
> > Edgardo
> >
>


--
Cheers,

Edgardo

Sent from Gmail Mobile
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Corey Flowers
Hey guys,

       The flow.xml.gz should not be updated by puppet. Puppet is great for
config files, controller services, reporting tasks, really anything
"variable/value" or "tag/value" like that but not the flow.xml.gz. It is
constantly changed and dynamic in its nature, which doesn't lend well to
the overall concept of puppet manifest control. Just a heads up.

I will let you guys decide about the idea to make these values config
variables.

Later!


On Wednesday, May 6, 2015, Edgardo Vega <[hidden email]> wrote:

> That's an interesting place to store that information wouldn't it make more
> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
> machines with different specs you would have to manually tweak the flow on
> each machine.
>
> I would gladly make a ticket if it seems to make more sense for these
> properties to be moved.  What's the consensus?
>
> On Wednesday, May 6, 2015, Matt Gilman <[hidden email]
> <javascript:;>> wrote:
>
> > It's actually saved as part of your flow. So assuming your flow is part
> of
> > your puppet configuration, you should be good. In a standalone instance
> it
> > would be in <NIFI_HOME>/conf/flow.xml.gz.
> >
> > Matt
> >
> > On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <[hidden email]
> <javascript:;>
> > <javascript:;>> wrote:
> >
> > > Thanks Matt. Is this configurable in a file or through a java argument?
> > > That way we can set that via puppet.
> > >
> > > Cheers,
> > >
> > > Edgardo
> > >
> > > On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]
> <javascript:;>
> > <javascript:;>>
> > > wrote:
> > >
> > > > The existing documentation is a little short but it is mentioned in
> the
> > > > Other Management Features section [1]. In the UI can you click the
> > wrench
> > > > and screwdriver icon in the upper right. There you can set the name
> of
> > > your
> > > > dataflow, it's description, and the size of the timer/event driven
> > thread
> > > > pools.
> > > >
> > > > Matt
> > > >
> > > > [1]
> > > >
> > > >
> > >
> >
> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
> > > >
> > > > On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]
> <javascript:;>
> > <javascript:;>>
> > > > wrote:
> > > >
> > > > > The admin guide says the Maximum Forked Processes section says
> "NiFi
> > > may
> > > > > be configured to generate a significant number of threads." How is
> > this
> > > > > done? I read the rest of the documentation and didn't see a way to
> > > define
> > > > > the number of threads to use. Also any recommendation on number of
> > > > threads
> > > > > per code?
> > > > >
> > > > > I have been seeing the number of threads in the gui pegged at 10
> that
> > > is
> > > > > why I went looking for how to change that number.
> > > > >
> > > > > Thanks in advance,
> > > > >
> > > > > Edgardo
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers,
> > >
> > > Edgardo
> > >
> >
>
>
> --
> Cheers,
>
> Edgardo
>
> Sent from Gmail Mobile
>


--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Matt Gilman
Ah, yes Corey you're totally right. Sorry about that. Also wanted to point out that starting with this upcoming release (0.1.0-incubating) the controller services and reporting task will configurable in the UI and become part of the flow configuration.

Sent from my iPhone

> On May 6, 2015, at 6:23 PM, Corey Flowers <[hidden email]> wrote:
>
> Hey guys,
>
>       The flow.xml.gz should not be updated by puppet. Puppet is great for
> config files, controller services, reporting tasks, really anything
> "variable/value" or "tag/value" like that but not the flow.xml.gz. It is
> constantly changed and dynamic in its nature, which doesn't lend well to
> the overall concept of puppet manifest control. Just a heads up.
>
> I will let you guys decide about the idea to make these values config
> variables.
>
> Later!
>
>
>> On Wednesday, May 6, 2015, Edgardo Vega <[hidden email]> wrote:
>>
>> That's an interesting place to store that information wouldn't it make more
>> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
>> machines with different specs you would have to manually tweak the flow on
>> each machine.
>>
>> I would gladly make a ticket if it seems to make more sense for these
>> properties to be moved.  What's the consensus?
>>
>> On Wednesday, May 6, 2015, Matt Gilman <[hidden email]
>> <javascript:;>> wrote:
>>
>>> It's actually saved as part of your flow. So assuming your flow is part
>> of
>>> your puppet configuration, you should be good. In a standalone instance
>> it
>>> would be in <NIFI_HOME>/conf/flow.xml.gz.
>>>
>>> Matt
>>>
>>> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <[hidden email]
>> <javascript:;>
>>> <javascript:;>> wrote:
>>>
>>>> Thanks Matt. Is this configurable in a file or through a java argument?
>>>> That way we can set that via puppet.
>>>>
>>>> Cheers,
>>>>
>>>> Edgardo
>>>>
>>>> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]
>> <javascript:;>
>>> <javascript:;>>
>>>> wrote:
>>>>
>>>>> The existing documentation is a little short but it is mentioned in
>> the
>>>>> Other Management Features section [1]. In the UI can you click the
>>> wrench
>>>>> and screwdriver icon in the upper right. There you can set the name
>> of
>>>> your
>>>>> dataflow, it's description, and the size of the timer/event driven
>>> thread
>>>>> pools.
>>>>>
>>>>> Matt
>>>>>
>>>>> [1]
>> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>>>>>
>>>>> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]
>> <javascript:;>
>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> The admin guide says the Maximum Forked Processes section says
>> "NiFi
>>>> may
>>>>>> be configured to generate a significant number of threads." How is
>>> this
>>>>>> done? I read the rest of the documentation and didn't see a way to
>>>> define
>>>>>> the number of threads to use. Also any recommendation on number of
>>>>> threads
>>>>>> per code?
>>>>>>
>>>>>> I have been seeing the number of threads in the gui pegged at 10
>> that
>>>> is
>>>>>> why I went looking for how to change that number.
>>>>>>
>>>>>> Thanks in advance,
>>>>>>
>>>>>> Edgardo
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers,
>>>>
>>>> Edgardo
>>
>>
>> --
>> Cheers,
>>
>> Edgardo
>>
>> Sent from Gmail Mobile
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> [hidden email]
>
> -- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: Threads

Joe Witt
Edgardo

It's not really a clear winner in either place.  Your intuition here
is totally fair and indeed it used to be a nifi properties config
item.  Then we had stronger cases for it being a flow specific
configuration item.  I think the reality is that you need a bit of
both.  The total number of available threads can impact the way
someone designing a flow thinks.  This is seen in how they allocate a
number of 'concurrent tasks' to a given processor.  This means it is
in a sense more of a 'flow configuration' item.  But you're absolutely
right that the other way of looking at this is that the total number
of threads will often be a function of the capabilities of a given
system.

So, I'd propose us supporting both if this turns out to be worth
tackling.  The current method works well.  But we could add back a
configuration property item that is more tied to the 'system' than the
flow and that property would be 'max threads' or something.  Then at
runtime we'd set the actual 'max threads' to the lesser of the flow
configured value or the nifi properties value.

Thanks
Joe

On Wed, May 6, 2015 at 7:07 PM, Matt Gilman <[hidden email]> wrote:

> Ah, yes Corey you're totally right. Sorry about that. Also wanted to point out that starting with this upcoming release (0.1.0-incubating) the controller services and reporting task will configurable in the UI and become part of the flow configuration.
>
> Sent from my iPhone
>
>> On May 6, 2015, at 6:23 PM, Corey Flowers <[hidden email]> wrote:
>>
>> Hey guys,
>>
>>       The flow.xml.gz should not be updated by puppet. Puppet is great for
>> config files, controller services, reporting tasks, really anything
>> "variable/value" or "tag/value" like that but not the flow.xml.gz. It is
>> constantly changed and dynamic in its nature, which doesn't lend well to
>> the overall concept of puppet manifest control. Just a heads up.
>>
>> I will let you guys decide about the idea to make these values config
>> variables.
>>
>> Later!
>>
>>
>>> On Wednesday, May 6, 2015, Edgardo Vega <[hidden email]> wrote:
>>>
>>> That's an interesting place to store that information wouldn't it make more
>>> sense to be in nifi.properties? If your sharing the flow.xml.gz on multiple
>>> machines with different specs you would have to manually tweak the flow on
>>> each machine.
>>>
>>> I would gladly make a ticket if it seems to make more sense for these
>>> properties to be moved.  What's the consensus?
>>>
>>> On Wednesday, May 6, 2015, Matt Gilman <[hidden email]
>>> <javascript:;>> wrote:
>>>
>>>> It's actually saved as part of your flow. So assuming your flow is part
>>> of
>>>> your puppet configuration, you should be good. In a standalone instance
>>> it
>>>> would be in <NIFI_HOME>/conf/flow.xml.gz.
>>>>
>>>> Matt
>>>>
>>>> On Wed, May 6, 2015 at 3:47 PM, Edgardo Vega <[hidden email]
>>> <javascript:;>
>>>> <javascript:;>> wrote:
>>>>
>>>>> Thanks Matt. Is this configurable in a file or through a java argument?
>>>>> That way we can set that via puppet.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Edgardo
>>>>>
>>>>> On Wed, May 6, 2015 at 3:31 PM, Matt Gilman <[hidden email]
>>> <javascript:;>
>>>> <javascript:;>>
>>>>> wrote:
>>>>>
>>>>>> The existing documentation is a little short but it is mentioned in
>>> the
>>>>>> Other Management Features section [1]. In the UI can you click the
>>>> wrench
>>>>>> and screwdriver icon in the upper right. There you can set the name
>>> of
>>>>> your
>>>>>> dataflow, it's description, and the size of the timer/event driven
>>>> thread
>>>>>> pools.
>>>>>>
>>>>>> Matt
>>>>>>
>>>>>> [1]
>>> https://nifi.incubator.apache.org/docs/nifi-docs/user-guide.html#other_management_features
>>>>>>
>>>>>> On Wed, May 6, 2015 at 3:22 PM, Edgardo Vega <[hidden email]
>>> <javascript:;>
>>>> <javascript:;>>
>>>>>> wrote:
>>>>>>
>>>>>>> The admin guide says the Maximum Forked Processes section says
>>> "NiFi
>>>>> may
>>>>>>> be configured to generate a significant number of threads." How is
>>>> this
>>>>>>> done? I read the rest of the documentation and didn't see a way to
>>>>> define
>>>>>>> the number of threads to use. Also any recommendation on number of
>>>>>> threads
>>>>>>> per code?
>>>>>>>
>>>>>>> I have been seeing the number of threads in the gui pegged at 10
>>> that
>>>>> is
>>>>>>> why I went looking for how to change that number.
>>>>>>>
>>>>>>> Thanks in advance,
>>>>>>>
>>>>>>> Edgardo
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers,
>>>>>
>>>>> Edgardo
>>>
>>>
>>> --
>>> Cheers,
>>>
>>> Edgardo
>>>
>>> Sent from Gmail Mobile
>>
>>
>> --
>> Corey Flowers
>> Vice President, Onyx Point, Inc
>> (410) 541-6699
>> [hidden email]
>>
>> -- This account not approved for unencrypted proprietary information --