Making NiFi upgrade smoother

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Making NiFi upgrade smoother

Andre F de Miranda
dev,

What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?

In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_

as they contain the links to all other files (logback.xml excluded).

Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).

This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!

I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts

Cheers
Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

James Wing
I think this is a great idea.  It would also provide a standard integration
point for the tls-toolkit command and similar utilities.


On Tue, Apr 25, 2017 at 8:03 AM, Andre F de Miranda <[hidden email]> wrote:

> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers
>
Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Uwe@Moosheimer.com
In reply to this post by Andre F de Miranda
This is a great idea!

Mit freundlichen Grüßen / best regards
Kay-Uwe Moosheimer

> Am 25.04.2017 um 17:03 schrieb Andre F de Miranda <[hidden email]>:
>
> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers

Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Yolanda Davis
In reply to this post by Andre F de Miranda
Hi Andre,

Realized I responded on the wrong thread...

Just wanted to add my thoughts on this.  One of the things I have been
thinking and working through in creating utilities to automate upgrades
(see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
ability to migrate to new configurations/settings, especially in the
instance that  bootstrap.conf or a nifi.properties have been changed. Even
master bootstrap and nifi.properties may be subject to upgrade. I think a
large pain point is that users have needed to address those changes
manually, so automating the migration might help in making an upgrade
better.

Also when thinking about smoother upgrades, I think there are some
additional considerations that may need to be accounted for, such as
whether or not certain repos should stay in place or be relocated (e.g. if
a repository lives under the conf folder in the older installation, and not
external, should that be moved to the newer one?). Or whether an existing
flow template or state information (for zookeeper) needs to be migrated
into the new installation as well. And how to easily manage a need to
rollback if upgrade did not work for whatever reason (e.g. a flow is
impacted by upgraded libraries). My hope is that upgrade tool will have a
comprehensive approach for these things and also lead us down the road
towards supporting rolling upgrades in a cluster in the future.

-yolanda

On Tue, Apr 25, 2017 at 11:03 AM, Andre F de Miranda <[hidden email]> wrote:

> dev,
>
> What do you think about the idea of allowing an user to configure overrides
> for the "master" configuration files in order to smooth upgrades?
>
> In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_
>
> as they contain the links to all other files (logback.xml excluded).
>
> Upon starting bootstrap and nifi would load first the standard files as it
> does today, but once the load is done, the process would proceed to load
> the local overrides (reaching its final configuration).
>
> This way that whenever a user decides to upgrade, it would simply copy or
> link the override file to the NiFi folder and voilà!
>
> I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
> to configure local properties and so does a number of other JAVA based
> applications I have dealt with) so I am keen to hear your thoughts
>
> Cheers
>



--
--
[hidden email]
@YolandaMDavis
Reply | Threaded
Open this post in threaded view
|

Re: Making NiFi upgrade smoother

Andy LoPresto-2
Andre,

Aldrin and I discussed something similar in the comments of NIFI-2974 last year but didn’t follow up on it. I think you’ve gotten sufficient support here to file a Jira and scope it out. I know as someone who is constantly rebuilding and redeploying the application, I have shortcuts to copy over pre-configured files, but making the process smoother and more robust for all users would be a huge gain. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Apr 25, 2017, at 9:27 AM, Yolanda Davis <[hidden email]> wrote:

Hi Andre,

Realized I responded on the wrong thread...

Just wanted to add my thoughts on this.  One of the things I have been
thinking and working through in creating utilities to automate upgrades
(see https://issues.apache.org/jira/browse/NIFI-3663) is creating the
ability to migrate to new configurations/settings, especially in the
instance that  bootstrap.conf or a nifi.properties have been changed. Even
master bootstrap and nifi.properties may be subject to upgrade. I think a
large pain point is that users have needed to address those changes
manually, so automating the migration might help in making an upgrade
better.

Also when thinking about smoother upgrades, I think there are some
additional considerations that may need to be accounted for, such as
whether or not certain repos should stay in place or be relocated (e.g. if
a repository lives under the conf folder in the older installation, and not
external, should that be moved to the newer one?). Or whether an existing
flow template or state information (for zookeeper) needs to be migrated
into the new installation as well. And how to easily manage a need to
rollback if upgrade did not work for whatever reason (e.g. a flow is
impacted by upgraded libraries). My hope is that upgrade tool will have a
comprehensive approach for these things and also lead us down the road
towards supporting rolling upgrades in a cluster in the future.

-yolanda

On Tue, Apr 25, 2017 at 11:03 AM, Andre F de Miranda <[hidden email]> wrote:

dev,

What do you think about the idea of allowing an user to configure overrides
for the "master" configuration files in order to smooth upgrades?

In my opinion candidates would be_ bootstrap.conf_ and  _nifi.properties_

as they contain the links to all other files (logback.xml excluded).

Upon starting bootstrap and nifi would load first the standard files as it
does today, but once the load is done, the process would proceed to load
the local overrides (reaching its final configuration).

This way that whenever a user decides to upgrade, it would simply copy or
link the override file to the NiFi folder and voilà!

I realise this is not particularly a new idea, (YARN uses the "*-site.xml"
to configure local properties and so does a number of other JAVA based
applications I have dealt with) so I am keen to hear your thoughts

Cheers




--
--
[hidden email]
@YolandaMDavis


signature.asc (859 bytes) Download Attachment