Configuration Service

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuration Service

Simon Ball
One of the problems with complex flows is repetition of common configuration. Many people also want to be able to configure things like connection strings into an environment specific location outside of the Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context service help in part with this, but the problem could be generalised.

What do people think of the idea of a configuration provider controller service, providing essentially the same concept as environment variables, which holds essentially a key value store which processors can refer to? A simple way to refer to contents could be an extension on expression language providing a config(‘key’) function.

What are your thoughts on this? Worth knocking up a quick prototype implementation (I’m happy to do so if there is community interest)?

Simon
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Simon Ball
Other thoughts:

This will have implications for broadcasting OnPropertyChanged events, and potentially locking processors around the changes in properties held by the service. I’d love to hear if anyone can think of any other significant land mines, or has had any thoughts on anything similar.

Simon

On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:[hidden email]>> wrote:

One of the problems with complex flows is repetition of common configuration. Many people also want to be able to configure things like connection strings into an environment specific location outside of the Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context service help in part with this, but the problem could be generalised.

What do people think of the idea of a configuration provider controller service, providing essentially the same concept as environment variables, which holds essentially a key value store which processors can refer to? A simple way to refer to contents could be an extension on expression language providing a config(‘key’) function.

What are your thoughts on this? Worth knocking up a quick prototype implementation (I’m happy to do so if there is community interest)?

Simon

Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Matt Gilman
Simon,

One idea that's been thrown around is adding a 'variable registry' [1]
where you could define variables at a group level that could be referenced
by the encapsulated components. Additionally, this would help with the
portability of templates when needing to define different values for
different environments.

Matt

[1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry

On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball <[hidden email]> wrote:

> Other thoughts:
>
> This will have implications for broadcasting OnPropertyChanged events, and
> potentially locking processors around the changes in properties held by the
> service. I’d love to hear if anyone can think of any other significant land
> mines, or has had any thoughts on anything similar.
>
> Simon
>
> On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:
> [hidden email]>> wrote:
>
> One of the problems with complex flows is repetition of common
> configuration. Many people also want to be able to configure things like
> connection strings into an environment specific location outside of the
> Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context
> service help in part with this, but the problem could be generalised.
>
> What do people think of the idea of a configuration provider controller
> service, providing essentially the same concept as environment variables,
> which holds essentially a key value store which processors can refer to? A
> simple way to refer to contents could be an extension on expression
> language providing a config(‘key’) function.
>
> What are your thoughts on this? Worth knocking up a quick prototype
> implementation (I’m happy to do so if there is community interest)?
>
> Simon
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

mring33621
Why can't we just have a built-in persistent k-v store controller service
that any processor can use? Could even modify the expression language to
interact with it. Maybe something like ${props:<key>} where 'props' refers
to the built-in store.

-Matt R

On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman <[hidden email]>
wrote:

> Simon,
>
> One idea that's been thrown around is adding a 'variable registry' [1]
> where you could define variables at a group level that could be referenced
> by the encapsulated components. Additionally, this would help with the
> portability of templates when needing to define different values for
> different environments.
>
> Matt
>
> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>
> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball <[hidden email]> wrote:
>
> > Other thoughts:
> >
> > This will have implications for broadcasting OnPropertyChanged events,
> and
> > potentially locking processors around the changes in properties held by
> the
> > service. I’d love to hear if anyone can think of any other significant
> land
> > mines, or has had any thoughts on anything similar.
> >
> > Simon
> >
> > On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:
> > [hidden email]>> wrote:
> >
> > One of the problems with complex flows is repetition of common
> > configuration. Many people also want to be able to configure things like
> > connection strings into an environment specific location outside of the
> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context
> > service help in part with this, but the problem could be generalised.
> >
> > What do people think of the idea of a configuration provider controller
> > service, providing essentially the same concept as environment variables,
> > which holds essentially a key value store which processors can refer to?
> A
> > simple way to refer to contents could be an extension on expression
> > language providing a config(‘key’) function.
> >
> > What are your thoughts on this? Worth knocking up a quick prototype
> > implementation (I’m happy to do so if there is community interest)?
> >
> > Simon
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Joe Witt
Matt,

That is basically what we're talking about providing but it would be
transparent through the expression language capability.  This approach
will make templates far more portable than they are today.

Thanks
Joe

On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio
<[hidden email]> wrote:

> Why can't we just have a built-in persistent k-v store controller service
> that any processor can use? Could even modify the expression language to
> interact with it. Maybe something like ${props:<key>} where 'props' refers
> to the built-in store.
>
> -Matt R
>
> On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman <[hidden email]>
> wrote:
>
>> Simon,
>>
>> One idea that's been thrown around is adding a 'variable registry' [1]
>> where you could define variables at a group level that could be referenced
>> by the encapsulated components. Additionally, this would help with the
>> portability of templates when needing to define different values for
>> different environments.
>>
>> Matt
>>
>> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>
>> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball <[hidden email]> wrote:
>>
>> > Other thoughts:
>> >
>> > This will have implications for broadcasting OnPropertyChanged events,
>> and
>> > potentially locking processors around the changes in properties held by
>> the
>> > service. I’d love to hear if anyone can think of any other significant
>> land
>> > mines, or has had any thoughts on anything similar.
>> >
>> > Simon
>> >
>> > On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:
>> > [hidden email]>> wrote:
>> >
>> > One of the problems with complex flows is repetition of common
>> > configuration. Many people also want to be able to configure things like
>> > connection strings into an environment specific location outside of the
>> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc Context
>> > service help in part with this, but the problem could be generalised.
>> >
>> > What do people think of the idea of a configuration provider controller
>> > service, providing essentially the same concept as environment variables,
>> > which holds essentially a key value store which processors can refer to?
>> A
>> > simple way to refer to contents could be an extension on expression
>> > language providing a config(‘key’) function.
>> >
>> > What are your thoughts on this? Worth knocking up a quick prototype
>> > implementation (I’m happy to do so if there is community interest)?
>> >
>> > Simon
>> >
>> >
>>
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Bryan Bende
FWIW you can externalize connection strings and other similar properties
into nifi.properties and have your processors references these environment
variables through expression language. This way you can move the
flow.xml.gz between two environments without changing anything after you
setup the nifi.properties for each environment. The variable registry would
just make this a much better user experience.

On Wed, Jan 27, 2016 at 12:07 PM, Joe Witt <[hidden email]> wrote:

> Matt,
>
> That is basically what we're talking about providing but it would be
> transparent through the expression language capability.  This approach
> will make templates far more portable than they are today.
>
> Thanks
> Joe
>
> On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio
> <[hidden email]> wrote:
> > Why can't we just have a built-in persistent k-v store controller service
> > that any processor can use? Could even modify the expression language to
> > interact with it. Maybe something like ${props:<key>} where 'props'
> refers
> > to the built-in store.
> >
> > -Matt R
> >
> > On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman <[hidden email]>
> > wrote:
> >
> >> Simon,
> >>
> >> One idea that's been thrown around is adding a 'variable registry' [1]
> >> where you could define variables at a group level that could be
> referenced
> >> by the encapsulated components. Additionally, this would help with the
> >> portability of templates when needing to define different values for
> >> different environments.
> >>
> >> Matt
> >>
> >> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
> >>
> >> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball <[hidden email]>
> wrote:
> >>
> >> > Other thoughts:
> >> >
> >> > This will have implications for broadcasting OnPropertyChanged events,
> >> and
> >> > potentially locking processors around the changes in properties held
> by
> >> the
> >> > service. I’d love to hear if anyone can think of any other significant
> >> land
> >> > mines, or has had any thoughts on anything similar.
> >> >
> >> > Simon
> >> >
> >> > On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:
> >> > [hidden email]>> wrote:
> >> >
> >> > One of the problems with complex flows is repetition of common
> >> > configuration. Many people also want to be able to configure things
> like
> >> > connection strings into an environment specific location outside of
> the
> >> > Flow, and parameterise the flow. Things like the Kerberos|SSL|etc
> Context
> >> > service help in part with this, but the problem could be generalised.
> >> >
> >> > What do people think of the idea of a configuration provider
> controller
> >> > service, providing essentially the same concept as environment
> variables,
> >> > which holds essentially a key value store which processors can refer
> to?
> >> A
> >> > simple way to refer to contents could be an extension on expression
> >> > language providing a config(‘key’) function.
> >> >
> >> > What are your thoughts on this? Worth knocking up a quick prototype
> >> > implementation (I’m happy to do so if there is community interest)?
> >> >
> >> > Simon
> >> >
> >> >
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Simon Ball
That certainly covers a part of it, but I take it since that is read at start time, you couldn't have a property change on a flow in progress. If we had something like the variable repository, we could fire property change events on updates, and change centralised properties at runtime. For example, failover of non load balanced endpoints, or just adjustments of other global variables that might feed a flow.

Simon


> On 27 Jan 2016, at 17:16, Bryan Bende <[hidden email]> wrote:
>
> FWIW you can externalize connection strings and other similar properties
> into nifi.properties and have your processors references these environment
> variables through expression language. This way you can move the
> flow.xml.gz between two environments without changing anything after you
> setup the nifi.properties for each environment. The variable registry would
> just make this a much better user experience.
>
>> On Wed, Jan 27, 2016 at 12:07 PM, Joe Witt <[hidden email]> wrote:
>>
>> Matt,
>>
>> That is basically what we're talking about providing but it would be
>> transparent through the expression language capability.  This approach
>> will make templates far more portable than they are today.
>>
>> Thanks
>> Joe
>>
>> On Wed, Jan 27, 2016 at 11:41 AM, Angry Duck Studio
>> <[hidden email]> wrote:
>>> Why can't we just have a built-in persistent k-v store controller service
>>> that any processor can use? Could even modify the expression language to
>>> interact with it. Maybe something like ${props:<key>} where 'props'
>> refers
>>> to the built-in store.
>>>
>>> -Matt R
>>>
>>> On Wed, Jan 27, 2016 at 7:16 AM, Matt Gilman <[hidden email]>
>>> wrote:
>>>
>>>> Simon,
>>>>
>>>> One idea that's been thrown around is adding a 'variable registry' [1]
>>>> where you could define variables at a group level that could be
>> referenced
>>>> by the encapsulated components. Additionally, this would help with the
>>>> portability of templates when needing to define different values for
>>>> different environments.
>>>>
>>>> Matt
>>>>
>>>> [1] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry
>>>>
>>>> On Wed, Jan 27, 2016 at 4:30 AM, Simon Ball <[hidden email]>
>> wrote:
>>>>
>>>>> Other thoughts:
>>>>>
>>>>> This will have implications for broadcasting OnPropertyChanged events,
>>>> and
>>>>> potentially locking processors around the changes in properties held
>> by
>>>> the
>>>>> service. I’d love to hear if anyone can think of any other significant
>>>> land
>>>>> mines, or has had any thoughts on anything similar.
>>>>>
>>>>> Simon
>>>>>
>>>>> On 27 Jan 2016, at 09:26, Simon Ball <[hidden email]<mailto:
>>>>> [hidden email]>> wrote:
>>>>>
>>>>> One of the problems with complex flows is repetition of common
>>>>> configuration. Many people also want to be able to configure things
>> like
>>>>> connection strings into an environment specific location outside of
>> the
>>>>> Flow, and parameterise the flow. Things like the Kerberos|SSL|etc
>> Context
>>>>> service help in part with this, but the problem could be generalised.
>>>>>
>>>>> What do people think of the idea of a configuration provider
>> controller
>>>>> service, providing essentially the same concept as environment
>> variables,
>>>>> which holds essentially a key value store which processors can refer
>> to?
>>>> A
>>>>> simple way to refer to contents could be an extension on expression
>>>>> language providing a config(‘key’) function.
>>>>>
>>>>> What are your thoughts on this? Worth knocking up a quick prototype
>>>>> implementation (I’m happy to do so if there is community interest)?
>>>>>
>>>>> Simon
>>
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Abhay Pasad
In reply to this post by Bryan Bende
One of the major issues with an external variable registry file which we
refer to in the nifi.properties file is that it needs NiFi instance to be
bounced. While this may serve your purpose in the lower environments in a
purpose, at an enterprise level cluster this might not be feasible in higher
environments like Production. What this also means is that due to lack of
common global variables/parameters, you may need to spin up a new controller
service every time. I think a Configuration service is a must and should be
worked upon. :)



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: Configuration Service

Bryan Bende
This original thread was from quite a long time ago, and there have
been many improvements since then, one of which is a variable registry
that is exposed directly through the UI [1] and does not require
bouncing the NiFi instance.

[1] https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#Variables_Window

On Thu, Mar 14, 2019 at 6:51 AM Abhay Pasad <[hidden email]> wrote:

>
> One of the major issues with an external variable registry file which we
> refer to in the nifi.properties file is that it needs NiFi instance to be
> bounced. While this may serve your purpose in the lower environments in a
> purpose, at an enterprise level cluster this might not be feasible in higher
> environments like Production. What this also means is that due to lack of
> common global variables/parameters, you may need to spin up a new controller
> service every time. I think a Configuration service is a must and should be
> worked upon. :)
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/