MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

Andy Christianson-3
All,

Currently in the middle of MINIFICPP-357.

v3 includes core/repository configuration in the yml file, while traditionally in minificpp this has been set with the .properties files in config/.

E.g.

Core Properties:

flow controller graceful shutdown period: 10 sec

flow service write delay interval: 500 ms

administrative yield duration: 30 sec

bored yield duration: 10 millis

max concurrent threads: 1

variable registry properties: ''

FlowFile Repository:

partitions: 256

checkpoint interval: 2 mins

always sync: false

Swap:

threshold: 20000

in period: 5 sec

in threads: 1

out period: 5 sec

out threads: 4

Content Repository:

content claim max appendable size: 10 MB

content claim max flow files: 100

always sync: false

How should we handle this in minificpp with v3 support? Do we want to keep the convention of using the .properties file, or move all that into the yml?

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.
Reply | Threaded
Open this post in threaded view
|

Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

Aldrin Piri
For the time being, I kind of view the C++ variant's configuration to be a
subset of that which is available for Java.  Many of these items currently
have no backing, so don't know that we need to provide explicit support for
them at this juncture but ensure that their presence does not preclude
proper operation of a C++ instance.

Looking at things from a general handling standpoint, I see these as
considerations of flow vs instance configuration.  My general inclination
is to provide such separation at some point and is at the heart of
MINIFI-66 [1].  I think what is important regardless of how this carries
out, and in view of the first release of Registry, is to think how we can
make use of what facilities Registry provides both now in terms of flows as
well as future resource types.  Some combination of the command and control
work (both server and the initial C2 API in C++) with Registry should allow
us to manage both sets of configuration that map to and enable the most
common workflows users would expect.  I suspect the C++ approach is more
aligned with typical usage, properties generally bootstrapped in
.properties file(s), with some C2 API type interactions that may allow
adjusting these items in a separate context outside of the flow itself.

[1] https://issues.apache.org/jira/browse/MINIFI-66

On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <[hidden email]>
wrote:

> All,
>
> Currently in the middle of MINIFICPP-357.
>
> v3 includes core/repository configuration in the yml file, while
> traditionally in minificpp this has been set with the .properties files in
> config/.
>
> E.g.
>
> Core Properties:
>
> flow controller graceful shutdown period: 10 sec
>
> flow service write delay interval: 500 ms
>
> administrative yield duration: 30 sec
>
> bored yield duration: 10 millis
>
> max concurrent threads: 1
>
> variable registry properties: ''
>
> FlowFile Repository:
>
> partitions: 256
>
> checkpoint interval: 2 mins
>
> always sync: false
>
> Swap:
>
> threshold: 20000
>
> in period: 5 sec
>
> in threads: 1
>
> out period: 5 sec
>
> out threads: 4
>
> Content Repository:
>
> content claim max appendable size: 10 MB
>
> content claim max flow files: 100
>
> always sync: false
>
> How should we handle this in minificpp with v3 support? Do we want to keep
> the convention of using the .properties file, or move all that into the yml?
>
> Regards,
>
> Andy I.C.
>
> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
> email.
Reply | Threaded
Open this post in threaded view
|

Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

Joe Witt
Hello

My preference is that we follow similar logic to how NiFi has
'instance configuration' details in nifi.properties and 'flow
configuration' details in flow.xml.gz.

What we do in minificpp will hopefully be aligned with what we do in
minifi-java and hopefully minifi-java just becomes an assembly of
nifi.

Thanks
Joe

On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri <[hidden email]> wrote:

> For the time being, I kind of view the C++ variant's configuration to be a
> subset of that which is available for Java.  Many of these items currently
> have no backing, so don't know that we need to provide explicit support for
> them at this juncture but ensure that their presence does not preclude
> proper operation of a C++ instance.
>
> Looking at things from a general handling standpoint, I see these as
> considerations of flow vs instance configuration.  My general inclination
> is to provide such separation at some point and is at the heart of
> MINIFI-66 [1].  I think what is important regardless of how this carries
> out, and in view of the first release of Registry, is to think how we can
> make use of what facilities Registry provides both now in terms of flows as
> well as future resource types.  Some combination of the command and control
> work (both server and the initial C2 API in C++) with Registry should allow
> us to manage both sets of configuration that map to and enable the most
> common workflows users would expect.  I suspect the C++ approach is more
> aligned with typical usage, properties generally bootstrapped in
> .properties file(s), with some C2 API type interactions that may allow
> adjusting these items in a separate context outside of the flow itself.
>
> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <[hidden email]>
> wrote:
>
>> All,
>>
>> Currently in the middle of MINIFICPP-357.
>>
>> v3 includes core/repository configuration in the yml file, while
>> traditionally in minificpp this has been set with the .properties files in
>> config/.
>>
>> E.g.
>>
>> Core Properties:
>>
>> flow controller graceful shutdown period: 10 sec
>>
>> flow service write delay interval: 500 ms
>>
>> administrative yield duration: 30 sec
>>
>> bored yield duration: 10 millis
>>
>> max concurrent threads: 1
>>
>> variable registry properties: ''
>>
>> FlowFile Repository:
>>
>> partitions: 256
>>
>> checkpoint interval: 2 mins
>>
>> always sync: false
>>
>> Swap:
>>
>> threshold: 20000
>>
>> in period: 5 sec
>>
>> in threads: 1
>>
>> out period: 5 sec
>>
>> out threads: 4
>>
>> Content Repository:
>>
>> content claim max appendable size: 10 MB
>>
>> content claim max flow files: 100
>>
>> always sync: false
>>
>> How should we handle this in minificpp with v3 support? Do we want to keep
>> the convention of using the .properties file, or move all that into the yml?
>>
>> Regards,
>>
>> Andy I.C.
>>
>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>> email.
Reply | Threaded
Open this post in threaded view
|

Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

Kevin Doran
Hi Andy,

My preference would also be to keep the .yml scoped to flows, and keep instance config in a .properties file.

When we get further along the C2 roadmap, both of these should be updatable via C2 interactions.

Thanks,
Kevin

On 1/4/18, 16:04, "Joe Witt" <[hidden email]> wrote:

    Hello
   
    My preference is that we follow similar logic to how NiFi has
    'instance configuration' details in nifi.properties and 'flow
    configuration' details in flow.xml.gz.
   
    What we do in minificpp will hopefully be aligned with what we do in
    minifi-java and hopefully minifi-java just becomes an assembly of
    nifi.
   
    Thanks
    Joe
   
    On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri <[hidden email]> wrote:
    > For the time being, I kind of view the C++ variant's configuration to be a
    > subset of that which is available for Java.  Many of these items currently
    > have no backing, so don't know that we need to provide explicit support for
    > them at this juncture but ensure that their presence does not preclude
    > proper operation of a C++ instance.
    >
    > Looking at things from a general handling standpoint, I see these as
    > considerations of flow vs instance configuration.  My general inclination
    > is to provide such separation at some point and is at the heart of
    > MINIFI-66 [1].  I think what is important regardless of how this carries
    > out, and in view of the first release of Registry, is to think how we can
    > make use of what facilities Registry provides both now in terms of flows as
    > well as future resource types.  Some combination of the command and control
    > work (both server and the initial C2 API in C++) with Registry should allow
    > us to manage both sets of configuration that map to and enable the most
    > common workflows users would expect.  I suspect the C++ approach is more
    > aligned with typical usage, properties generally bootstrapped in
    > .properties file(s), with some C2 API type interactions that may allow
    > adjusting these items in a separate context outside of the flow itself.
    >
    > [1] https://issues.apache.org/jira/browse/MINIFI-66
    >
    > On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <[hidden email]>
    > wrote:
    >
    >> All,
    >>
    >> Currently in the middle of MINIFICPP-357.
    >>
    >> v3 includes core/repository configuration in the yml file, while
    >> traditionally in minificpp this has been set with the .properties files in
    >> config/.
    >>
    >> E.g.
    >>
    >> Core Properties:
    >>
    >> flow controller graceful shutdown period: 10 sec
    >>
    >> flow service write delay interval: 500 ms
    >>
    >> administrative yield duration: 30 sec
    >>
    >> bored yield duration: 10 millis
    >>
    >> max concurrent threads: 1
    >>
    >> variable registry properties: ''
    >>
    >> FlowFile Repository:
    >>
    >> partitions: 256
    >>
    >> checkpoint interval: 2 mins
    >>
    >> always sync: false
    >>
    >> Swap:
    >>
    >> threshold: 20000
    >>
    >> in period: 5 sec
    >>
    >> in threads: 1
    >>
    >> out period: 5 sec
    >>
    >> out threads: 4
    >>
    >> Content Repository:
    >>
    >> content claim max appendable size: 10 MB
    >>
    >> content claim max flow files: 100
    >>
    >> always sync: false
    >>
    >> How should we handle this in minificpp with v3 support? Do we want to keep
    >> the convention of using the .properties file, or move all that into the yml?
    >>
    >> Regards,
    >>
    >> Andy I.C.
    >>
    >> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
    >> email.
   


Reply | Threaded
Open this post in threaded view
|

Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties

Andy Christianson-3
Thanks for the feedback. It looks like we have consensus on keeping instance configuration in the properties files and keeping the yml focused on flow.

I'll run some tests to make sure the instance config properties, if present in the yml, do not interfere with the instance startup/operations.

Regards,

Andy I.C.

Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted email.

> -------- Original Message --------
> Subject: Re: MINIFICPP-357 minifi-cpp Design decision: Flow Configuration v3 yml vs properties
> Local Time: January 4, 2018 4:22 PM
> UTC Time: January 4, 2018 9:22 PM
> From: [hidden email]
> To: [hidden email]
>
> Hi Andy,
>
> My preference would also be to keep the .yml scoped to flows, and keep instance config in a .properties file.
>
> When we get further along the C2 roadmap, both of these should be updatable via C2 interactions.
>
> Thanks,
> Kevin
>
> On 1/4/18, 16:04, "Joe Witt" [hidden email] wrote:
>
> Hello
>
> My preference is that we follow similar logic to how NiFi has
> 'instance configuration' details in nifi.properties and 'flow
> configuration' details in flow.xml.gz.
>
> What we do in minificpp will hopefully be aligned with what we do in
> minifi-java and hopefully minifi-java just becomes an assembly of
> nifi.
>
> Thanks
> Joe
>
> On Thu, Jan 4, 2018 at 3:59 PM, Aldrin Piri [hidden email] wrote:
>
>> For the time being, I kind of view the C++ variant's configuration to be a
>
>> subset of that which is available for Java.  Many of these items currently
>
>> have no backing, so don't know that we need to provide explicit support for
>
>> them at this juncture but ensure that their presence does not preclude
>
>> proper operation of a C++ instance.
>
>>
>
>> Looking at things from a general handling standpoint, I see these as
>
>> considerations of flow vs instance configuration.  My general inclination
>
>> is to provide such separation at some point and is at the heart of
>
>> MINIFI-66 [1].  I think what is important regardless of how this carries
>
>> out, and in view of the first release of Registry, is to think how we can
>
>> make use of what facilities Registry provides both now in terms of flows as
>
>> well as future resource types.  Some combination of the command and control
>
>> work (both server and the initial C2 API in C++) with Registry should allow
>
>> us to manage both sets of configuration that map to and enable the most
>
>> common workflows users would expect.  I suspect the C++ approach is more
>
>> aligned with typical usage, properties generally bootstrapped in
>
>> .properties file(s), with some C2 API type interactions that may allow
>
>> adjusting these items in a separate context outside of the flow itself.
>
>>
>
>> [1] https://issues.apache.org/jira/browse/MINIFI-66
>
>>
>
>> On Thu, Jan 4, 2018 at 1:27 PM, Andy Christianson <[hidden email]>
>
>> wrote:
>
>>
>
>>> All,
>
>>>
>
>>> Currently in the middle of MINIFICPP-357.
>
>>>
>
>>> v3 includes core/repository configuration in the yml file, while
>
>>> traditionally in minificpp this has been set with the .properties files in
>
>>> config/.
>
>>>
>
>>> E.g.
>
>>>
>
>>> Core Properties:
>
>>>
>
>>> flow controller graceful shutdown period: 10 sec
>
>>>
>
>>> flow service write delay interval: 500 ms
>
>>>
>
>>> administrative yield duration: 30 sec
>
>>>
>
>>> bored yield duration: 10 millis
>
>>>
>
>>> max concurrent threads: 1
>
>>>
>
>>> variable registry properties: ''
>
>>>
>
>>> FlowFile Repository:
>
>>>
>
>>> partitions: 256
>
>>>
>
>>> checkpoint interval: 2 mins
>
>>>
>
>>> always sync: false
>
>>>
>
>>> Swap:
>
>>>
>
>>> threshold: 20000
>
>>>
>
>>> in period: 5 sec
>
>>>
>
>>> in threads: 1
>
>>>
>
>>> out period: 5 sec
>
>>>
>
>>> out threads: 4
>
>>>
>
>>> Content Repository:
>
>>>
>
>>> content claim max appendable size: 10 MB
>
>>>
>
>>> content claim max flow files: 100
>
>>>
>
>>> always sync: false
>
>>>
>
>>> How should we handle this in minificpp with v3 support? Do we want to keep
>
>>> the convention of using the .properties file, or move all that into the yml?
>
>>>
>
>>> Regards,
>
>>>
>
>>> Andy I.C.
>
>>>
>
>>> Sent from [ProtonMail](https://protonmail.com), Swiss-based encrypted
>
>>> email.