Making NiFi upgrade smoother

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Making NiFi upgrade smoother

Andre
dev,

What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?

In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_

as they contain the links to all other files (logback.xml excluded).

Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).

This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!

I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts

Cheers
Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Yolanda Davis
Hi Andre,

Just wanted to add my thoughts on this.  One of the things I have been
thinking and working through in creating utilities to automate upgrades
(see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
ability to migrate to new configurations/settings, especially in the
instance that  bootstrap.conf or a nifi.properties have been changed. Even
master bootstrap and nifi.properties may be subject to upgrade. I think a
large pain point is that users have needed to address those changes
manually, so automating the migration might help in making an upgrade
better.

Also when thinking about smoother upgrades, I think there are some
additional considerations that may need to be accounted for, such as
whether or not certain repos should stay in place or be relocated (e.g. if
a repository lives under the conf folder in the older installation, and not
external, should that be moved to the newer one?). Or whether an existing
flow template or state information (for zookeeper) needs to be migrated
into the new installation as well. And how to easily manage a need to
rollback if upgrade did not work for whatever reason (e.g. a flow is
impacted by upgraded libraries). My hope is that upgrade tool will have a
comprehensive approach for these things and also lead us down the road
towards supporting rolling upgrades in a cluster.

-yolanda





On Tue, Apr 25, 2017 at 11:06 AM, Andre <[hidden email]> wrote:

> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers
>



--
--
[hidden email]
@YolandaMDavis
Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Uwe@Moosheimer.com
👍

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 25.04.2017 um 18:16 schrieb Yolanda Davis <[hidden email]>:
>
> Hi Andre,
>
> Just wanted to add my thoughts on this.  One of the things I have been
> thinking and working through in creating utilities to automate upgrades
> (see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
> ability to migrate to new configurations/settings, especially in the
> instance that  bootstrap.conf or a nifi.properties have been changed. Even
> master bootstrap and nifi.properties may be subject to upgrade. I think a
> large pain point is that users have needed to address those changes
> manually, so automating the migration might help in making an upgrade
> better.
>
> Also when thinking about smoother upgrades, I think there are some
> additional considerations that may need to be accounted for, such as
> whether or not certain repos should stay in place or be relocated (e.g. if
> a repository lives under the conf folder in the older installation, and not
> external, should that be moved to the newer one?). Or whether an existing
> flow template or state information (for zookeeper) needs to be migrated
> into the new installation as well. And how to easily manage a need to
> rollback if upgrade did not work for whatever reason (e.g. a flow is
> impacted by upgraded libraries). My hope is that upgrade tool will have a
> comprehensive approach for these things and also lead us down the road
> towards supporting rolling upgrades in a cluster.
>
> -yolanda
>
>
>
>
>
>> On Tue, Apr 25, 2017 at 11:06 AM, Andre <[hidden email]> wrote:
>>
>> dev,
>>
>> What do you think about the idea of allowing an user to configure overrides
>> for the "master" configuration files in order to smooth upgrades?
>>
>> In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_
>>
>> as they contain the links to all other files (logback.xml excluded).
>>
>> Upon starting bootstrap and nifi would load first the standard files as it
>> does today, but once the load is done, the process would proceed to load
>> the local overrides (reaching its final configuration).
>>
>> This way that whenever a user decides to upgrade, it would simply copy or
>> link the override file to the NiFi folder and voilà!
>>
>> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
>> to configure local properties and so does a number of other JAVA based
>> applications I have dealt with) so I am keen to hear your thoughts
>>
>> Cheers
>>
>
>
>
> --
> --
> [hidden email]
> @YolandaMDavis

Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Adam Lamar
Great idea Yolanda!