Upgrading NiFi Registry

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrading NiFi Registry

Mark Bean
How does one upgrade the NiFi Registry?

After unpacking the .tar.gz file, how does one get all the flows registered
in a previous version of NiFi Registry into the newly installed version?
And, how does one ensure all the policies transfer as well?

Thanks,
Mark
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Bryan Bende
Mark,

The database directory and flow storage directory are where all the
data are. By default these are created in the root of NiFi Registry,
so depending how you want to set it up you could move those
directories to the new install, or you could set them up to be
external locations so you don't have to move them every time, or you
could upgrade the lib directory of your current install and leave
everything in place.

The policies are the same as NiFi... stored in users.xml and
authorizations.xml the conf directory, depending how you configured
everything. So just copying those two files over to the new install.

-Bryan


On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]> wrote:
> How does one upgrade the NiFi Registry?
>
> After unpacking the .tar.gz file, how does one get all the flows registered
> in a previous version of NiFi Registry into the newly installed version?
> And, how does one ensure all the policies transfer as well?
>
> Thanks,
> Mark
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Mark Bean
Thanks Bryan. There is actually another step not explicitly mentioned. At
least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties
file as well. The 0.2.0 version has new properties/values not in the 0.1.0.
And, I had to set the following for the database (using values from 0.1.0).
This was only required on the first startup; these properties can be empty
values on subsequent startups.
nifi.registry.db.directory=
nifi.registry.db.url.append=

In the future, if using a database location that is external to the
installation directory, is nifi.registry.db.url the only property that
needs to be modified?


On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]> wrote:

> Mark,
>
> The database directory and flow storage directory are where all the
> data are. By default these are created in the root of NiFi Registry,
> so depending how you want to set it up you could move those
> directories to the new install, or you could set them up to be
> external locations so you don't have to move them every time, or you
> could upgrade the lib directory of your current install and leave
> everything in place.
>
> The policies are the same as NiFi... stored in users.xml and
> authorizations.xml the conf directory, depending how you configured
> everything. So just copying those two files over to the new install.
>
> -Bryan
>
>
> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]> wrote:
> > How does one upgrade the NiFi Registry?
> >
> > After unpacking the .tar.gz file, how does one get all the flows
> registered
> > in a previous version of NiFi Registry into the newly installed version?
> > And, how does one ensure all the policies transfer as well?
> >
> > Thanks,
> > Mark
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Kevin Doran-2
Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2 migration guide to include these steps as part of updating the site with news of the new release.

Thanks,
Kevin

________________________________
From: Mark Bean <[hidden email]>
Sent: Wednesday, June 20, 2018 8:53:39 AM
To: [hidden email]
Subject: Re: Upgrading NiFi Registry

Thanks Bryan. There is actually another step not explicitly mentioned. At
least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties
file as well. The 0.2.0 version has new properties/values not in the 0.1.0.
And, I had to set the following for the database (using values from 0.1.0).
This was only required on the first startup; these properties can be empty
values on subsequent startups.
nifi.registry.db.directory=
nifi.registry.db.url.append=

In the future, if using a database location that is external to the
installation directory, is nifi.registry.db.url the only property that
needs to be modified?


On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]> wrote:

> Mark,
>
> The database directory and flow storage directory are where all the
> data are. By default these are created in the root of NiFi Registry,
> so depending how you want to set it up you could move those
> directories to the new install, or you could set them up to be
> external locations so you don't have to move them every time, or you
> could upgrade the lib directory of your current install and leave
> everything in place.
>
> The policies are the same as NiFi... stored in users.xml and
> authorizations.xml the conf directory, depending how you configured
> everything. So just copying those two files over to the new install.
>
> -Bryan
>
>
> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]> wrote:
> > How does one upgrade the NiFi Registry?
> >
> > After unpacking the .tar.gz file, how does one get all the flows
> registered
> > in a previous version of NiFi Registry into the newly installed version?
> > And, how does one ensure all the policies transfer as well?
> >
> > Thanks,
> > Mark
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Bryan Bende
Ah good point Mark... yes the old db properties are only need this
first time so that it can auto-migrate the old DB to the new one,
after that you don't need the old properties anymore.

Thanks Kevin!

On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <[hidden email]> wrote:

> Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2 migration guide to include these steps as part of updating the site with news of the new release.
>
> Thanks,
> Kevin
>
> ________________________________
> From: Mark Bean <[hidden email]>
> Sent: Wednesday, June 20, 2018 8:53:39 AM
> To: [hidden email]
> Subject: Re: Upgrading NiFi Registry
>
> Thanks Bryan. There is actually another step not explicitly mentioned. At
> least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties
> file as well. The 0.2.0 version has new properties/values not in the 0.1.0.
> And, I had to set the following for the database (using values from 0.1.0).
> This was only required on the first startup; these properties can be empty
> values on subsequent startups.
> nifi.registry.db.directory=
> nifi.registry.db.url.append=
>
> In the future, if using a database location that is external to the
> installation directory, is nifi.registry.db.url the only property that
> needs to be modified?
>
>
> On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]> wrote:
>
>> Mark,
>>
>> The database directory and flow storage directory are where all the
>> data are. By default these are created in the root of NiFi Registry,
>> so depending how you want to set it up you could move those
>> directories to the new install, or you could set them up to be
>> external locations so you don't have to move them every time, or you
>> could upgrade the lib directory of your current install and leave
>> everything in place.
>>
>> The policies are the same as NiFi... stored in users.xml and
>> authorizations.xml the conf directory, depending how you configured
>> everything. So just copying those two files over to the new install.
>>
>> -Bryan
>>
>>
>> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]> wrote:
>> > How does one upgrade the NiFi Registry?
>> >
>> > After unpacking the .tar.gz file, how does one get all the flows
>> registered
>> > in a previous version of NiFi Registry into the newly installed version?
>> > And, how does one ensure all the policies transfer as well?
>> >
>> > Thanks,
>> > Mark
>>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Mike Thomsen
Kevin/Bryan,

We have a bunch of users that are using Docker and might not have set
things up to point to a docker volume for the H2 database location. Do
either of you know if H2 will allow you to copy the files over to the
Docker volume while everything's running?

Thanks,

Mike

On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]> wrote:

> Ah good point Mark... yes the old db properties are only need this
> first time so that it can auto-migrate the old DB to the new one,
> after that you don't need the old properties anymore.
>
> Thanks Kevin!
>
> On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <[hidden email]> wrote:
> > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2 migration
> guide to include these steps as part of updating the site with news of the
> new release.
> >
> > Thanks,
> > Kevin
> >
> > ________________________________
> > From: Mark Bean <[hidden email]>
> > Sent: Wednesday, June 20, 2018 8:53:39 AM
> > To: [hidden email]
> > Subject: Re: Upgrading NiFi Registry
> >
> > Thanks Bryan. There is actually another step not explicitly mentioned. At
> > least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties
> > file as well. The 0.2.0 version has new properties/values not in the
> 0.1.0.
> > And, I had to set the following for the database (using values from
> 0.1.0).
> > This was only required on the first startup; these properties can be
> empty
> > values on subsequent startups.
> > nifi.registry.db.directory=
> > nifi.registry.db.url.append=
> >
> > In the future, if using a database location that is external to the
> > installation directory, is nifi.registry.db.url the only property that
> > needs to be modified?
> >
> >
> > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]> wrote:
> >
> >> Mark,
> >>
> >> The database directory and flow storage directory are where all the
> >> data are. By default these are created in the root of NiFi Registry,
> >> so depending how you want to set it up you could move those
> >> directories to the new install, or you could set them up to be
> >> external locations so you don't have to move them every time, or you
> >> could upgrade the lib directory of your current install and leave
> >> everything in place.
> >>
> >> The policies are the same as NiFi... stored in users.xml and
> >> authorizations.xml the conf directory, depending how you configured
> >> everything. So just copying those two files over to the new install.
> >>
> >> -Bryan
> >>
> >>
> >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]>
> wrote:
> >> > How does one upgrade the NiFi Registry?
> >> >
> >> > After unpacking the .tar.gz file, how does one get all the flows
> >> registered
> >> > in a previous version of NiFi Registry into the newly installed
> version?
> >> > And, how does one ensure all the policies transfer as well?
> >> >
> >> > Thanks,
> >> > Mark
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Kevin Doran-2
Mike,

I don't know the extent of H2's capabilities, but for NiFi Registry changing the location of those files will require restarting the server to pick up the new database location from nifi-registry.properties, correct? So if I follow correctly what the objective is, my recommendation would be the following steps:

1. Stop NiFi Registry
2. Move/copy the H2 database files to the Docker volume location (should be safe to do when Registry is stopped)
3. Change nifi-registry.properties to point the database config props to the new location
4. Start NiFi Registry

If the goal is actually just to copy a snapshot of the H2 database to the docker volume and not re-point NiFi Registry, which you want to keep running, I suppose you could do that. It should be fine to copy it if nothing is being written, but of course no guarantees and YMMV, so at the least I'd try to verify that it is a validate db file after copying it, using something like [1].

[1] https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
 
Hope this helps,
Kevin

On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]> wrote:

    Kevin/Bryan,
   
    We have a bunch of users that are using Docker and might not have set
    things up to point to a docker volume for the H2 database location. Do
    either of you know if H2 will allow you to copy the files over to the
    Docker volume while everything's running?
   
    Thanks,
   
    Mike
   
    On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]> wrote:
   
    > Ah good point Mark... yes the old db properties are only need this
    > first time so that it can auto-migrate the old DB to the new one,
    > after that you don't need the old properties anymore.
    >
    > Thanks Kevin!
    >
    > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <[hidden email]> wrote:
    > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2 migration
    > guide to include these steps as part of updating the site with news of the
    > new release.
    > >
    > > Thanks,
    > > Kevin
    > >
    > > ________________________________
    > > From: Mark Bean <[hidden email]>
    > > Sent: Wednesday, June 20, 2018 8:53:39 AM
    > > To: [hidden email]
    > > Subject: Re: Upgrading NiFi Registry
    > >
    > > Thanks Bryan. There is actually another step not explicitly mentioned. At
    > > least for 0.1.0 -> 0.2.0, I needed to modify the nifi-registry.properties
    > > file as well. The 0.2.0 version has new properties/values not in the
    > 0.1.0.
    > > And, I had to set the following for the database (using values from
    > 0.1.0).
    > > This was only required on the first startup; these properties can be
    > empty
    > > values on subsequent startups.
    > > nifi.registry.db.directory=
    > > nifi.registry.db.url.append=
    > >
    > > In the future, if using a database location that is external to the
    > > installation directory, is nifi.registry.db.url the only property that
    > > needs to be modified?
    > >
    > >
    > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]> wrote:
    > >
    > >> Mark,
    > >>
    > >> The database directory and flow storage directory are where all the
    > >> data are. By default these are created in the root of NiFi Registry,
    > >> so depending how you want to set it up you could move those
    > >> directories to the new install, or you could set them up to be
    > >> external locations so you don't have to move them every time, or you
    > >> could upgrade the lib directory of your current install and leave
    > >> everything in place.
    > >>
    > >> The policies are the same as NiFi... stored in users.xml and
    > >> authorizations.xml the conf directory, depending how you configured
    > >> everything. So just copying those two files over to the new install.
    > >>
    > >> -Bryan
    > >>
    > >>
    > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <[hidden email]>
    > wrote:
    > >> > How does one upgrade the NiFi Registry?
    > >> >
    > >> > After unpacking the .tar.gz file, how does one get all the flows
    > >> registered
    > >> > in a previous version of NiFi Registry into the newly installed
    > version?
    > >> > And, how does one ensure all the policies transfer as well?
    > >> >
    > >> > Thanks,
    > >> > Mark
    > >>
    >
   


Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Mike Thomsen
Kevin,

The problem with our situation is that it's all inside Docker so I don't
think I can just stop the registry and proceed. That's why I was asking if
it sounded feasible to just say "hands off for 5 minutes while we copy the
data."

Also, what is the maturity level of the GitFlow option at this point? That
was particularly attractive as a storage solution because it should make it
much easier to export everything from our dev team to our end users.

Thanks,

Mike

On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]> wrote:

> Mike,
>
> I don't know the extent of H2's capabilities, but for NiFi Registry
> changing the location of those files will require restarting the server to
> pick up the new database location from nifi-registry.properties, correct?
> So if I follow correctly what the objective is, my recommendation would be
> the following steps:
>
> 1. Stop NiFi Registry
> 2. Move/copy the H2 database files to the Docker volume location (should
> be safe to do when Registry is stopped)
> 3. Change nifi-registry.properties to point the database config props to
> the new location
> 4. Start NiFi Registry
>
> If the goal is actually just to copy a snapshot of the H2 database to the
> docker volume and not re-point NiFi Registry, which you want to keep
> running, I suppose you could do that. It should be fine to copy it if
> nothing is being written, but of course no guarantees and YMMV, so at the
> least I'd try to verify that it is a validate db file after copying it,
> using something like [1].
>
> [1]
> https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
>
> Hope this helps,
> Kevin
>
> On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]> wrote:
>
>     Kevin/Bryan,
>
>     We have a bunch of users that are using Docker and might not have set
>     things up to point to a docker volume for the H2 database location. Do
>     either of you know if H2 will allow you to copy the files over to the
>     Docker volume while everything's running?
>
>     Thanks,
>
>     Mike
>
>     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]> wrote:
>
>     > Ah good point Mark... yes the old db properties are only need this
>     > first time so that it can auto-migrate the old DB to the new one,
>     > after that you don't need the old properties anymore.
>     >
>     > Thanks Kevin!
>     >
>     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <[hidden email]>
> wrote:
>     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2
> migration
>     > guide to include these steps as part of updating the site with news
> of the
>     > new release.
>     > >
>     > > Thanks,
>     > > Kevin
>     > >
>     > > ________________________________
>     > > From: Mark Bean <[hidden email]>
>     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
>     > > To: [hidden email]
>     > > Subject: Re: Upgrading NiFi Registry
>     > >
>     > > Thanks Bryan. There is actually another step not explicitly
> mentioned. At
>     > > least for 0.1.0 -> 0.2.0, I needed to modify the
> nifi-registry.properties
>     > > file as well. The 0.2.0 version has new properties/values not in
> the
>     > 0.1.0.
>     > > And, I had to set the following for the database (using values from
>     > 0.1.0).
>     > > This was only required on the first startup; these properties can
> be
>     > empty
>     > > values on subsequent startups.
>     > > nifi.registry.db.directory=
>     > > nifi.registry.db.url.append=
>     > >
>     > > In the future, if using a database location that is external to the
>     > > installation directory, is nifi.registry.db.url the only property
> that
>     > > needs to be modified?
>     > >
>     > >
>     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]>
> wrote:
>     > >
>     > >> Mark,
>     > >>
>     > >> The database directory and flow storage directory are where all
> the
>     > >> data are. By default these are created in the root of NiFi
> Registry,
>     > >> so depending how you want to set it up you could move those
>     > >> directories to the new install, or you could set them up to be
>     > >> external locations so you don't have to move them every time, or
> you
>     > >> could upgrade the lib directory of your current install and leave
>     > >> everything in place.
>     > >>
>     > >> The policies are the same as NiFi... stored in users.xml and
>     > >> authorizations.xml the conf directory, depending how you
> configured
>     > >> everything. So just copying those two files over to the new
> install.
>     > >>
>     > >> -Bryan
>     > >>
>     > >>
>     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
> [hidden email]>
>     > wrote:
>     > >> > How does one upgrade the NiFi Registry?
>     > >> >
>     > >> > After unpacking the .tar.gz file, how does one get all the flows
>     > >> registered
>     > >> > in a previous version of NiFi Registry into the newly installed
>     > version?
>     > >> > And, how does one ensure all the policies transfer as well?
>     > >> >
>     > >> > Thanks,
>     > >> > Mark
>     > >>
>     >
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Kevin Doran-2
Oh right. My mistake. Not sure if there is a way to stop the nifi registry process without stopping the container. Other's more knowledgeable of Docker might know... In that case I'd say it's worth a shot as that DB file should be valid so long as nothing is being written to it.

The adoption of the git persistence provider stuff is probably still small given how new it is, but anecdotally, I've been using it for about a week now, syncing to a github repo as a remote, and I've not run into any issues with it nor have I heard of anyone else having a problem with it. It seems pretty solid to me. One of the nice things about the git option is that the format saved to git in the snapshots is compatible with a tool like NiFi CLI, so (while some custom scripting still required), you should be in a pretty good position to do disaster recovery to recreate a NiFi Registry instance with your bucket/flow/flow-snapshot data just from the contents of your git repository should you need to for whatever reason (again, would need to script the right NiFi CLI commands to crawl the git directory tree, but it's pretty straightforward as it's just a matter of parsing json and yml files). One really nice feature is that deletes (a flow or bucket for instance) happen as a new commit so you can even recover objects that were deleted from NiFi Registry by accident, for instance.

If your team/users are using NiFi Registry and have feedback, positive or otherwise, let us know as it would be great to factor that in to the existing backlog for what to target in future releases.

Best,
Kevin

If your team runs into any trouble with this, do let us know as I'm not sure the deployment footprint

On 6/28/18, 13:31, "Mike Thomsen" <[hidden email]> wrote:

    Kevin,
   
    The problem with our situation is that it's all inside Docker so I don't
    think I can just stop the registry and proceed. That's why I was asking if
    it sounded feasible to just say "hands off for 5 minutes while we copy the
    data."
   
    Also, what is the maturity level of the GitFlow option at this point? That
    was particularly attractive as a storage solution because it should make it
    much easier to export everything from our dev team to our end users.
   
    Thanks,
   
    Mike
   
    On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]> wrote:
   
    > Mike,
    >
    > I don't know the extent of H2's capabilities, but for NiFi Registry
    > changing the location of those files will require restarting the server to
    > pick up the new database location from nifi-registry.properties, correct?
    > So if I follow correctly what the objective is, my recommendation would be
    > the following steps:
    >
    > 1. Stop NiFi Registry
    > 2. Move/copy the H2 database files to the Docker volume location (should
    > be safe to do when Registry is stopped)
    > 3. Change nifi-registry.properties to point the database config props to
    > the new location
    > 4. Start NiFi Registry
    >
    > If the goal is actually just to copy a snapshot of the H2 database to the
    > docker volume and not re-point NiFi Registry, which you want to keep
    > running, I suppose you could do that. It should be fine to copy it if
    > nothing is being written, but of course no guarantees and YMMV, so at the
    > least I'd try to verify that it is a validate db file after copying it,
    > using something like [1].
    >
    > [1]
    > https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
    >
    > Hope this helps,
    > Kevin
    >
    > On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]> wrote:
    >
    >     Kevin/Bryan,
    >
    >     We have a bunch of users that are using Docker and might not have set
    >     things up to point to a docker volume for the H2 database location. Do
    >     either of you know if H2 will allow you to copy the files over to the
    >     Docker volume while everything's running?
    >
    >     Thanks,
    >
    >     Mike
    >
    >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]> wrote:
    >
    >     > Ah good point Mark... yes the old db properties are only need this
    >     > first time so that it can auto-migrate the old DB to the new one,
    >     > after that you don't need the old properties anymore.
    >     >
    >     > Thanks Kevin!
    >     >
    >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <[hidden email]>
    > wrote:
    >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2
    > migration
    >     > guide to include these steps as part of updating the site with news
    > of the
    >     > new release.
    >     > >
    >     > > Thanks,
    >     > > Kevin
    >     > >
    >     > > ________________________________
    >     > > From: Mark Bean <[hidden email]>
    >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
    >     > > To: [hidden email]
    >     > > Subject: Re: Upgrading NiFi Registry
    >     > >
    >     > > Thanks Bryan. There is actually another step not explicitly
    > mentioned. At
    >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
    > nifi-registry.properties
    >     > > file as well. The 0.2.0 version has new properties/values not in
    > the
    >     > 0.1.0.
    >     > > And, I had to set the following for the database (using values from
    >     > 0.1.0).
    >     > > This was only required on the first startup; these properties can
    > be
    >     > empty
    >     > > values on subsequent startups.
    >     > > nifi.registry.db.directory=
    >     > > nifi.registry.db.url.append=
    >     > >
    >     > > In the future, if using a database location that is external to the
    >     > > installation directory, is nifi.registry.db.url the only property
    > that
    >     > > needs to be modified?
    >     > >
    >     > >
    >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <[hidden email]>
    > wrote:
    >     > >
    >     > >> Mark,
    >     > >>
    >     > >> The database directory and flow storage directory are where all
    > the
    >     > >> data are. By default these are created in the root of NiFi
    > Registry,
    >     > >> so depending how you want to set it up you could move those
    >     > >> directories to the new install, or you could set them up to be
    >     > >> external locations so you don't have to move them every time, or
    > you
    >     > >> could upgrade the lib directory of your current install and leave
    >     > >> everything in place.
    >     > >>
    >     > >> The policies are the same as NiFi... stored in users.xml and
    >     > >> authorizations.xml the conf directory, depending how you
    > configured
    >     > >> everything. So just copying those two files over to the new
    > install.
    >     > >>
    >     > >> -Bryan
    >     > >>
    >     > >>
    >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
    > [hidden email]>
    >     > wrote:
    >     > >> > How does one upgrade the NiFi Registry?
    >     > >> >
    >     > >> > After unpacking the .tar.gz file, how does one get all the flows
    >     > >> registered
    >     > >> > in a previous version of NiFi Registry into the newly installed
    >     > version?
    >     > >> > And, how does one ensure all the policies transfer as well?
    >     > >> >
    >     > >> > Thanks,
    >     > >> > Mark
    >     > >>
    >     >
    >
    >
    >
    >
   


Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Aldrin Piri
Hi Mike,

Registry's docker image is a little off from some of the other NiFi project
configurations but the following approach should be fine and is fairly
conservative in how it is carried out.

   1. Exec into the container - *docker exec -it <registry container
   name/id> /bin/bash*
   2. Inside container: stop Registry, and have your interactive session
   terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
   3. On your host, copy the h2 file to your host - *docker cp <registry
   container name/id>
**/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
   <path on host>*
   4. Copy the database file in question to your volume for your new
   instance
   5. Start the new container to verify import/migration
   6. Cleanup the old container and files as appropriate

1 & 2 are questionable in terms of their necessity but with the potential
for the container to get sigkilled, lend to a safer approach.  The Java
process doesn't seem to handle the sigterm from my quick testing of docker
stop.

You have the ability to restart where you left off on the original
container via *docker start **<registry container name/id>*.

I did a quick run through of these steps with vanilla versions of 0.1.0 and
0.2.0 and it seemed to work as anticipated.

Let us know how it goes.

On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <[hidden email]> wrote:

> Oh right. My mistake. Not sure if there is a way to stop the nifi registry
> process without stopping the container. Other's more knowledgeable of
> Docker might know... In that case I'd say it's worth a shot as that DB file
> should be valid so long as nothing is being written to it.
>
> The adoption of the git persistence provider stuff is probably still small
> given how new it is, but anecdotally, I've been using it for about a week
> now, syncing to a github repo as a remote, and I've not run into any issues
> with it nor have I heard of anyone else having a problem with it. It seems
> pretty solid to me. One of the nice things about the git option is that the
> format saved to git in the snapshots is compatible with a tool like NiFi
> CLI, so (while some custom scripting still required), you should be in a
> pretty good position to do disaster recovery to recreate a NiFi Registry
> instance with your bucket/flow/flow-snapshot data just from the contents of
> your git repository should you need to for whatever reason (again, would
> need to script the right NiFi CLI commands to crawl the git directory tree,
> but it's pretty straightforward as it's just a matter of parsing json and
> yml files). One really nice feature is that deletes (a flow or bucket for
> instance) happen as a new commit so you can even recover objects that were
> deleted from NiFi Registry by accident, for instance.
>
> If your team/users are using NiFi Registry and have feedback, positive or
> otherwise, let us know as it would be great to factor that in to the
> existing backlog for what to target in future releases.
>
> Best,
> Kevin
>
> If your team runs into any trouble with this, do let us know as I'm not
> sure the deployment footprint
>
> On 6/28/18, 13:31, "Mike Thomsen" <[hidden email]> wrote:
>
>     Kevin,
>
>     The problem with our situation is that it's all inside Docker so I
> don't
>     think I can just stop the registry and proceed. That's why I was
> asking if
>     it sounded feasible to just say "hands off for 5 minutes while we copy
> the
>     data."
>
>     Also, what is the maturity level of the GitFlow option at this point?
> That
>     was particularly attractive as a storage solution because it should
> make it
>     much easier to export everything from our dev team to our end users.
>
>     Thanks,
>
>     Mike
>
>     On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]> wrote:
>
>     > Mike,
>     >
>     > I don't know the extent of H2's capabilities, but for NiFi Registry
>     > changing the location of those files will require restarting the
> server to
>     > pick up the new database location from nifi-registry.properties,
> correct?
>     > So if I follow correctly what the objective is, my recommendation
> would be
>     > the following steps:
>     >
>     > 1. Stop NiFi Registry
>     > 2. Move/copy the H2 database files to the Docker volume location
> (should
>     > be safe to do when Registry is stopped)
>     > 3. Change nifi-registry.properties to point the database config
> props to
>     > the new location
>     > 4. Start NiFi Registry
>     >
>     > If the goal is actually just to copy a snapshot of the H2 database
> to the
>     > docker volume and not re-point NiFi Registry, which you want to keep
>     > running, I suppose you could do that. It should be fine to copy it if
>     > nothing is being written, but of course no guarantees and YMMV, so
> at the
>     > least I'd try to verify that it is a validate db file after copying
> it,
>     > using something like [1].
>     >
>     > [1]
>     >
> https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
>     >
>     > Hope this helps,
>     > Kevin
>     >
>     > On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]> wrote:
>     >
>     >     Kevin/Bryan,
>     >
>     >     We have a bunch of users that are using Docker and might not
> have set
>     >     things up to point to a docker volume for the H2 database
> location. Do
>     >     either of you know if H2 will allow you to copy the files over
> to the
>     >     Docker volume while everything's running?
>     >
>     >     Thanks,
>     >
>     >     Mike
>     >
>     >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]>
> wrote:
>     >
>     >     > Ah good point Mark... yes the old db properties are only need
> this
>     >     > first time so that it can auto-migrate the old DB to the new
> one,
>     >     > after that you don't need the old properties anymore.
>     >     >
>     >     > Thanks Kevin!
>     >     >
>     >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
> [hidden email]>
>     > wrote:
>     >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2
>     > migration
>     >     > guide to include these steps as part of updating the site with
> news
>     > of the
>     >     > new release.
>     >     > >
>     >     > > Thanks,
>     >     > > Kevin
>     >     > >
>     >     > > ________________________________
>     >     > > From: Mark Bean <[hidden email]>
>     >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
>     >     > > To: [hidden email]
>     >     > > Subject: Re: Upgrading NiFi Registry
>     >     > >
>     >     > > Thanks Bryan. There is actually another step not explicitly
>     > mentioned. At
>     >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
>     > nifi-registry.properties
>     >     > > file as well. The 0.2.0 version has new properties/values
> not in
>     > the
>     >     > 0.1.0.
>     >     > > And, I had to set the following for the database (using
> values from
>     >     > 0.1.0).
>     >     > > This was only required on the first startup; these
> properties can
>     > be
>     >     > empty
>     >     > > values on subsequent startups.
>     >     > > nifi.registry.db.directory=
>     >     > > nifi.registry.db.url.append=
>     >     > >
>     >     > > In the future, if using a database location that is external
> to the
>     >     > > installation directory, is nifi.registry.db.url the only
> property
>     > that
>     >     > > needs to be modified?
>     >     > >
>     >     > >
>     >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
> [hidden email]>
>     > wrote:
>     >     > >
>     >     > >> Mark,
>     >     > >>
>     >     > >> The database directory and flow storage directory are where
> all
>     > the
>     >     > >> data are. By default these are created in the root of NiFi
>     > Registry,
>     >     > >> so depending how you want to set it up you could move those
>     >     > >> directories to the new install, or you could set them up to
> be
>     >     > >> external locations so you don't have to move them every
> time, or
>     > you
>     >     > >> could upgrade the lib directory of your current install and
> leave
>     >     > >> everything in place.
>     >     > >>
>     >     > >> The policies are the same as NiFi... stored in users.xml and
>     >     > >> authorizations.xml the conf directory, depending how you
>     > configured
>     >     > >> everything. So just copying those two files over to the new
>     > install.
>     >     > >>
>     >     > >> -Bryan
>     >     > >>
>     >     > >>
>     >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
>     > [hidden email]>
>     >     > wrote:
>     >     > >> > How does one upgrade the NiFi Registry?
>     >     > >> >
>     >     > >> > After unpacking the .tar.gz file, how does one get all
> the flows
>     >     > >> registered
>     >     > >> > in a previous version of NiFi Registry into the newly
> installed
>     >     > version?
>     >     > >> > And, how does one ensure all the policies transfer as
> well?
>     >     > >> >
>     >     > >> > Thanks,
>     >     > >> > Mark
>     >     > >>
>     >     >
>     >
>     >
>     >
>     >
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Kevin Doran-2
Thanks Aldrin! That sequence makes sense to me and should result in a clean copy of the H2 Database. Mike, you're probably aware of this, but you will also want to grab the files in the HOME/flow_storage directory assuming those were left I nthe default settings and are not on your Docker volume.

Aldrin, is there anything you see regarding the Registry Docker image that could be improved, for which we could open a JIRA? Just curious to get your insight while it is fresh in your mind.

Thanks!
Kevin

On 6/28/18, 14:24, "Aldrin Piri" <[hidden email]> wrote:

    Hi Mike,
   
    Registry's docker image is a little off from some of the other NiFi project
    configurations but the following approach should be fine and is fairly
    conservative in how it is carried out.
   
       1. Exec into the container - *docker exec -it <registry container
       name/id> /bin/bash*
       2. Inside container: stop Registry, and have your interactive session
       terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
       3. On your host, copy the h2 file to your host - *docker cp <registry
       container name/id>
    **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
       <path on host>*
       4. Copy the database file in question to your volume for your new
       instance
       5. Start the new container to verify import/migration
       6. Cleanup the old container and files as appropriate
   
    1 & 2 are questionable in terms of their necessity but with the potential
    for the container to get sigkilled, lend to a safer approach.  The Java
    process doesn't seem to handle the sigterm from my quick testing of docker
    stop.
   
    You have the ability to restart where you left off on the original
    container via *docker start **<registry container name/id>*.
   
    I did a quick run through of these steps with vanilla versions of 0.1.0 and
    0.2.0 and it seemed to work as anticipated.
   
    Let us know how it goes.
   
    On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <[hidden email]> wrote:
   
    > Oh right. My mistake. Not sure if there is a way to stop the nifi registry
    > process without stopping the container. Other's more knowledgeable of
    > Docker might know... In that case I'd say it's worth a shot as that DB file
    > should be valid so long as nothing is being written to it.
    >
    > The adoption of the git persistence provider stuff is probably still small
    > given how new it is, but anecdotally, I've been using it for about a week
    > now, syncing to a github repo as a remote, and I've not run into any issues
    > with it nor have I heard of anyone else having a problem with it. It seems
    > pretty solid to me. One of the nice things about the git option is that the
    > format saved to git in the snapshots is compatible with a tool like NiFi
    > CLI, so (while some custom scripting still required), you should be in a
    > pretty good position to do disaster recovery to recreate a NiFi Registry
    > instance with your bucket/flow/flow-snapshot data just from the contents of
    > your git repository should you need to for whatever reason (again, would
    > need to script the right NiFi CLI commands to crawl the git directory tree,
    > but it's pretty straightforward as it's just a matter of parsing json and
    > yml files). One really nice feature is that deletes (a flow or bucket for
    > instance) happen as a new commit so you can even recover objects that were
    > deleted from NiFi Registry by accident, for instance.
    >
    > If your team/users are using NiFi Registry and have feedback, positive or
    > otherwise, let us know as it would be great to factor that in to the
    > existing backlog for what to target in future releases.
    >
    > Best,
    > Kevin
    >
    > If your team runs into any trouble with this, do let us know as I'm not
    > sure the deployment footprint
    >
    > On 6/28/18, 13:31, "Mike Thomsen" <[hidden email]> wrote:
    >
    >     Kevin,
    >
    >     The problem with our situation is that it's all inside Docker so I
    > don't
    >     think I can just stop the registry and proceed. That's why I was
    > asking if
    >     it sounded feasible to just say "hands off for 5 minutes while we copy
    > the
    >     data."
    >
    >     Also, what is the maturity level of the GitFlow option at this point?
    > That
    >     was particularly attractive as a storage solution because it should
    > make it
    >     much easier to export everything from our dev team to our end users.
    >
    >     Thanks,
    >
    >     Mike
    >
    >     On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]> wrote:
    >
    >     > Mike,
    >     >
    >     > I don't know the extent of H2's capabilities, but for NiFi Registry
    >     > changing the location of those files will require restarting the
    > server to
    >     > pick up the new database location from nifi-registry.properties,
    > correct?
    >     > So if I follow correctly what the objective is, my recommendation
    > would be
    >     > the following steps:
    >     >
    >     > 1. Stop NiFi Registry
    >     > 2. Move/copy the H2 database files to the Docker volume location
    > (should
    >     > be safe to do when Registry is stopped)
    >     > 3. Change nifi-registry.properties to point the database config
    > props to
    >     > the new location
    >     > 4. Start NiFi Registry
    >     >
    >     > If the goal is actually just to copy a snapshot of the H2 database
    > to the
    >     > docker volume and not re-point NiFi Registry, which you want to keep
    >     > running, I suppose you could do that. It should be fine to copy it if
    >     > nothing is being written, but of course no guarantees and YMMV, so
    > at the
    >     > least I'd try to verify that it is a validate db file after copying
    > it,
    >     > using something like [1].
    >     >
    >     > [1]
    >     >
    > https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
    >     >
    >     > Hope this helps,
    >     > Kevin
    >     >
    >     > On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]> wrote:
    >     >
    >     >     Kevin/Bryan,
    >     >
    >     >     We have a bunch of users that are using Docker and might not
    > have set
    >     >     things up to point to a docker volume for the H2 database
    > location. Do
    >     >     either of you know if H2 will allow you to copy the files over
    > to the
    >     >     Docker volume while everything's running?
    >     >
    >     >     Thanks,
    >     >
    >     >     Mike
    >     >
    >     >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <[hidden email]>
    > wrote:
    >     >
    >     >     > Ah good point Mark... yes the old db properties are only need
    > this
    >     >     > first time so that it can auto-migrate the old DB to the new
    > one,
    >     >     > after that you don't need the old properties anymore.
    >     >     >
    >     >     > Thanks Kevin!
    >     >     >
    >     >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
    > [hidden email]>
    >     > wrote:
    >     >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1 -> 0.2
    >     > migration
    >     >     > guide to include these steps as part of updating the site with
    > news
    >     > of the
    >     >     > new release.
    >     >     > >
    >     >     > > Thanks,
    >     >     > > Kevin
    >     >     > >
    >     >     > > ________________________________
    >     >     > > From: Mark Bean <[hidden email]>
    >     >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
    >     >     > > To: [hidden email]
    >     >     > > Subject: Re: Upgrading NiFi Registry
    >     >     > >
    >     >     > > Thanks Bryan. There is actually another step not explicitly
    >     > mentioned. At
    >     >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
    >     > nifi-registry.properties
    >     >     > > file as well. The 0.2.0 version has new properties/values
    > not in
    >     > the
    >     >     > 0.1.0.
    >     >     > > And, I had to set the following for the database (using
    > values from
    >     >     > 0.1.0).
    >     >     > > This was only required on the first startup; these
    > properties can
    >     > be
    >     >     > empty
    >     >     > > values on subsequent startups.
    >     >     > > nifi.registry.db.directory=
    >     >     > > nifi.registry.db.url.append=
    >     >     > >
    >     >     > > In the future, if using a database location that is external
    > to the
    >     >     > > installation directory, is nifi.registry.db.url the only
    > property
    >     > that
    >     >     > > needs to be modified?
    >     >     > >
    >     >     > >
    >     >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
    > [hidden email]>
    >     > wrote:
    >     >     > >
    >     >     > >> Mark,
    >     >     > >>
    >     >     > >> The database directory and flow storage directory are where
    > all
    >     > the
    >     >     > >> data are. By default these are created in the root of NiFi
    >     > Registry,
    >     >     > >> so depending how you want to set it up you could move those
    >     >     > >> directories to the new install, or you could set them up to
    > be
    >     >     > >> external locations so you don't have to move them every
    > time, or
    >     > you
    >     >     > >> could upgrade the lib directory of your current install and
    > leave
    >     >     > >> everything in place.
    >     >     > >>
    >     >     > >> The policies are the same as NiFi... stored in users.xml and
    >     >     > >> authorizations.xml the conf directory, depending how you
    >     > configured
    >     >     > >> everything. So just copying those two files over to the new
    >     > install.
    >     >     > >>
    >     >     > >> -Bryan
    >     >     > >>
    >     >     > >>
    >     >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
    >     > [hidden email]>
    >     >     > wrote:
    >     >     > >> > How does one upgrade the NiFi Registry?
    >     >     > >> >
    >     >     > >> > After unpacking the .tar.gz file, how does one get all
    > the flows
    >     >     > >> registered
    >     >     > >> > in a previous version of NiFi Registry into the newly
    > installed
    >     >     > version?
    >     >     > >> > And, how does one ensure all the policies transfer as
    > well?
    >     >     > >> >
    >     >     > >> > Thanks,
    >     >     > >> > Mark
    >     >     > >>
    >     >     >
    >     >
    >     >
    >     >
    >     >
    >
    >
    >
    >
   


Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Aldrin Piri
I don't know that it necessarily needs to be changed but the convention we
have in other images where we make use of a wrapper init script gives a
little more flexibility for investigating/manipulating instance internals.
That convention would have made things a bit more straightforward for this
scenario.  In terms of the signals handling, I was a bit surprised they
weren't being picked up by Java but would need to investigate a bit more to
see if it was just my hastiness or if there is something else at play.

On Thu, Jun 28, 2018 at 2:33 PM Kevin Doran <[hidden email]> wrote:

> Thanks Aldrin! That sequence makes sense to me and should result in a
> clean copy of the H2 Database. Mike, you're probably aware of this, but you
> will also want to grab the files in the HOME/flow_storage directory
> assuming those were left I nthe default settings and are not on your Docker
> volume.
>
> Aldrin, is there anything you see regarding the Registry Docker image that
> could be improved, for which we could open a JIRA? Just curious to get your
> insight while it is fresh in your mind.
>
> Thanks!
> Kevin
>
> On 6/28/18, 14:24, "Aldrin Piri" <[hidden email]> wrote:
>
>     Hi Mike,
>
>     Registry's docker image is a little off from some of the other NiFi
> project
>     configurations but the following approach should be fine and is fairly
>     conservative in how it is carried out.
>
>        1. Exec into the container - *docker exec -it <registry container
>        name/id> /bin/bash*
>        2. Inside container: stop Registry, and have your interactive
> session
>        terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
>        3. On your host, copy the h2 file to your host - *docker cp
> <registry
>        container name/id>
>     **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
>        <path on host>*
>        4. Copy the database file in question to your volume for your new
>        instance
>        5. Start the new container to verify import/migration
>        6. Cleanup the old container and files as appropriate
>
>     1 & 2 are questionable in terms of their necessity but with the
> potential
>     for the container to get sigkilled, lend to a safer approach.  The Java
>     process doesn't seem to handle the sigterm from my quick testing of
> docker
>     stop.
>
>     You have the ability to restart where you left off on the original
>     container via *docker start **<registry container name/id>*.
>
>     I did a quick run through of these steps with vanilla versions of
> 0.1.0 and
>     0.2.0 and it seemed to work as anticipated.
>
>     Let us know how it goes.
>
>     On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <[hidden email]> wrote:
>
>     > Oh right. My mistake. Not sure if there is a way to stop the nifi
> registry
>     > process without stopping the container. Other's more knowledgeable of
>     > Docker might know... In that case I'd say it's worth a shot as that
> DB file
>     > should be valid so long as nothing is being written to it.
>     >
>     > The adoption of the git persistence provider stuff is probably still
> small
>     > given how new it is, but anecdotally, I've been using it for about a
> week
>     > now, syncing to a github repo as a remote, and I've not run into any
> issues
>     > with it nor have I heard of anyone else having a problem with it. It
> seems
>     > pretty solid to me. One of the nice things about the git option is
> that the
>     > format saved to git in the snapshots is compatible with a tool like
> NiFi
>     > CLI, so (while some custom scripting still required), you should be
> in a
>     > pretty good position to do disaster recovery to recreate a NiFi
> Registry
>     > instance with your bucket/flow/flow-snapshot data just from the
> contents of
>     > your git repository should you need to for whatever reason (again,
> would
>     > need to script the right NiFi CLI commands to crawl the git
> directory tree,
>     > but it's pretty straightforward as it's just a matter of parsing
> json and
>     > yml files). One really nice feature is that deletes (a flow or
> bucket for
>     > instance) happen as a new commit so you can even recover objects
> that were
>     > deleted from NiFi Registry by accident, for instance.
>     >
>     > If your team/users are using NiFi Registry and have feedback,
> positive or
>     > otherwise, let us know as it would be great to factor that in to the
>     > existing backlog for what to target in future releases.
>     >
>     > Best,
>     > Kevin
>     >
>     > If your team runs into any trouble with this, do let us know as I'm
> not
>     > sure the deployment footprint
>     >
>     > On 6/28/18, 13:31, "Mike Thomsen" <[hidden email]> wrote:
>     >
>     >     Kevin,
>     >
>     >     The problem with our situation is that it's all inside Docker so
> I
>     > don't
>     >     think I can just stop the registry and proceed. That's why I was
>     > asking if
>     >     it sounded feasible to just say "hands off for 5 minutes while
> we copy
>     > the
>     >     data."
>     >
>     >     Also, what is the maturity level of the GitFlow option at this
> point?
>     > That
>     >     was particularly attractive as a storage solution because it
> should
>     > make it
>     >     much easier to export everything from our dev team to our end
> users.
>     >
>     >     Thanks,
>     >
>     >     Mike
>     >
>     >     On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]>
> wrote:
>     >
>     >     > Mike,
>     >     >
>     >     > I don't know the extent of H2's capabilities, but for NiFi
> Registry
>     >     > changing the location of those files will require restarting
> the
>     > server to
>     >     > pick up the new database location from
> nifi-registry.properties,
>     > correct?
>     >     > So if I follow correctly what the objective is, my
> recommendation
>     > would be
>     >     > the following steps:
>     >     >
>     >     > 1. Stop NiFi Registry
>     >     > 2. Move/copy the H2 database files to the Docker volume
> location
>     > (should
>     >     > be safe to do when Registry is stopped)
>     >     > 3. Change nifi-registry.properties to point the database config
>     > props to
>     >     > the new location
>     >     > 4. Start NiFi Registry
>     >     >
>     >     > If the goal is actually just to copy a snapshot of the H2
> database
>     > to the
>     >     > docker volume and not re-point NiFi Registry, which you want
> to keep
>     >     > running, I suppose you could do that. It should be fine to
> copy it if
>     >     > nothing is being written, but of course no guarantees and
> YMMV, so
>     > at the
>     >     > least I'd try to verify that it is a validate db file after
> copying
>     > it,
>     >     > using something like [1].
>     >     >
>     >     > [1]
>     >     >
>     >
> https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
>     >     >
>     >     > Hope this helps,
>     >     > Kevin
>     >     >
>     >     > On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]>
> wrote:
>     >     >
>     >     >     Kevin/Bryan,
>     >     >
>     >     >     We have a bunch of users that are using Docker and might
> not
>     > have set
>     >     >     things up to point to a docker volume for the H2 database
>     > location. Do
>     >     >     either of you know if H2 will allow you to copy the files
> over
>     > to the
>     >     >     Docker volume while everything's running?
>     >     >
>     >     >     Thanks,
>     >     >
>     >     >     Mike
>     >     >
>     >     >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <
> [hidden email]>
>     > wrote:
>     >     >
>     >     >     > Ah good point Mark... yes the old db properties are only
> need
>     > this
>     >     >     > first time so that it can auto-migrate the old DB to the
> new
>     > one,
>     >     >     > after that you don't need the old properties anymore.
>     >     >     >
>     >     >     > Thanks Kevin!
>     >     >     >
>     >     >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
>     > [hidden email]>
>     >     > wrote:
>     >     >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1
> -> 0.2
>     >     > migration
>     >     >     > guide to include these steps as part of updating the
> site with
>     > news
>     >     > of the
>     >     >     > new release.
>     >     >     > >
>     >     >     > > Thanks,
>     >     >     > > Kevin
>     >     >     > >
>     >     >     > > ________________________________
>     >     >     > > From: Mark Bean <[hidden email]>
>     >     >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
>     >     >     > > To: [hidden email]
>     >     >     > > Subject: Re: Upgrading NiFi Registry
>     >     >     > >
>     >     >     > > Thanks Bryan. There is actually another step not
> explicitly
>     >     > mentioned. At
>     >     >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
>     >     > nifi-registry.properties
>     >     >     > > file as well. The 0.2.0 version has new
> properties/values
>     > not in
>     >     > the
>     >     >     > 0.1.0.
>     >     >     > > And, I had to set the following for the database (using
>     > values from
>     >     >     > 0.1.0).
>     >     >     > > This was only required on the first startup; these
>     > properties can
>     >     > be
>     >     >     > empty
>     >     >     > > values on subsequent startups.
>     >     >     > > nifi.registry.db.directory=
>     >     >     > > nifi.registry.db.url.append=
>     >     >     > >
>     >     >     > > In the future, if using a database location that is
> external
>     > to the
>     >     >     > > installation directory, is nifi.registry.db.url the
> only
>     > property
>     >     > that
>     >     >     > > needs to be modified?
>     >     >     > >
>     >     >     > >
>     >     >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
>     > [hidden email]>
>     >     > wrote:
>     >     >     > >
>     >     >     > >> Mark,
>     >     >     > >>
>     >     >     > >> The database directory and flow storage directory are
> where
>     > all
>     >     > the
>     >     >     > >> data are. By default these are created in the root of
> NiFi
>     >     > Registry,
>     >     >     > >> so depending how you want to set it up you could move
> those
>     >     >     > >> directories to the new install, or you could set them
> up to
>     > be
>     >     >     > >> external locations so you don't have to move them
> every
>     > time, or
>     >     > you
>     >     >     > >> could upgrade the lib directory of your current
> install and
>     > leave
>     >     >     > >> everything in place.
>     >     >     > >>
>     >     >     > >> The policies are the same as NiFi... stored in
> users.xml and
>     >     >     > >> authorizations.xml the conf directory, depending how
> you
>     >     > configured
>     >     >     > >> everything. So just copying those two files over to
> the new
>     >     > install.
>     >     >     > >>
>     >     >     > >> -Bryan
>     >     >     > >>
>     >     >     > >>
>     >     >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
>     >     > [hidden email]>
>     >     >     > wrote:
>     >     >     > >> > How does one upgrade the NiFi Registry?
>     >     >     > >> >
>     >     >     > >> > After unpacking the .tar.gz file, how does one get
> all
>     > the flows
>     >     >     > >> registered
>     >     >     > >> > in a previous version of NiFi Registry into the
> newly
>     > installed
>     >     >     > version?
>     >     >     > >> > And, how does one ensure all the policies transfer
> as
>     > well?
>     >     >     > >> >
>     >     >     > >> > Thanks,
>     >     >     > >> > Mark
>     >     >     > >>
>     >     >     >
>     >     >
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>     >
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Upgrading NiFi Registry

Kevin Doran-2
Thanks, I'll take a look at the wrapper init script.

On 6/28/18, 15:11, "Aldrin Piri" <[hidden email]> wrote:

    I don't know that it necessarily needs to be changed but the convention we
    have in other images where we make use of a wrapper init script gives a
    little more flexibility for investigating/manipulating instance internals.
    That convention would have made things a bit more straightforward for this
    scenario.  In terms of the signals handling, I was a bit surprised they
    weren't being picked up by Java but would need to investigate a bit more to
    see if it was just my hastiness or if there is something else at play.
   
    On Thu, Jun 28, 2018 at 2:33 PM Kevin Doran <[hidden email]> wrote:
   
    > Thanks Aldrin! That sequence makes sense to me and should result in a
    > clean copy of the H2 Database. Mike, you're probably aware of this, but you
    > will also want to grab the files in the HOME/flow_storage directory
    > assuming those were left I nthe default settings and are not on your Docker
    > volume.
    >
    > Aldrin, is there anything you see regarding the Registry Docker image that
    > could be improved, for which we could open a JIRA? Just curious to get your
    > insight while it is fresh in your mind.
    >
    > Thanks!
    > Kevin
    >
    > On 6/28/18, 14:24, "Aldrin Piri" <[hidden email]> wrote:
    >
    >     Hi Mike,
    >
    >     Registry's docker image is a little off from some of the other NiFi
    > project
    >     configurations but the following approach should be fine and is fairly
    >     conservative in how it is carried out.
    >
    >        1. Exec into the container - *docker exec -it <registry container
    >        name/id> /bin/bash*
    >        2. Inside container: stop Registry, and have your interactive
    > session
    >        terminated as PID 1 was terminated - *./bin/nifi-registry.sh stop*
    >        3. On your host, copy the h2 file to your host - *docker cp
    > <registry
    >        container name/id>
    >     **/opt/nifi-registry/nifi-registry-0.1.0/database/nifi-registry.mv.db
    >        <path on host>*
    >        4. Copy the database file in question to your volume for your new
    >        instance
    >        5. Start the new container to verify import/migration
    >        6. Cleanup the old container and files as appropriate
    >
    >     1 & 2 are questionable in terms of their necessity but with the
    > potential
    >     for the container to get sigkilled, lend to a safer approach.  The Java
    >     process doesn't seem to handle the sigterm from my quick testing of
    > docker
    >     stop.
    >
    >     You have the ability to restart where you left off on the original
    >     container via *docker start **<registry container name/id>*.
    >
    >     I did a quick run through of these steps with vanilla versions of
    > 0.1.0 and
    >     0.2.0 and it seemed to work as anticipated.
    >
    >     Let us know how it goes.
    >
    >     On Thu, Jun 28, 2018 at 1:52 PM Kevin Doran <[hidden email]> wrote:
    >
    >     > Oh right. My mistake. Not sure if there is a way to stop the nifi
    > registry
    >     > process without stopping the container. Other's more knowledgeable of
    >     > Docker might know... In that case I'd say it's worth a shot as that
    > DB file
    >     > should be valid so long as nothing is being written to it.
    >     >
    >     > The adoption of the git persistence provider stuff is probably still
    > small
    >     > given how new it is, but anecdotally, I've been using it for about a
    > week
    >     > now, syncing to a github repo as a remote, and I've not run into any
    > issues
    >     > with it nor have I heard of anyone else having a problem with it. It
    > seems
    >     > pretty solid to me. One of the nice things about the git option is
    > that the
    >     > format saved to git in the snapshots is compatible with a tool like
    > NiFi
    >     > CLI, so (while some custom scripting still required), you should be
    > in a
    >     > pretty good position to do disaster recovery to recreate a NiFi
    > Registry
    >     > instance with your bucket/flow/flow-snapshot data just from the
    > contents of
    >     > your git repository should you need to for whatever reason (again,
    > would
    >     > need to script the right NiFi CLI commands to crawl the git
    > directory tree,
    >     > but it's pretty straightforward as it's just a matter of parsing
    > json and
    >     > yml files). One really nice feature is that deletes (a flow or
    > bucket for
    >     > instance) happen as a new commit so you can even recover objects
    > that were
    >     > deleted from NiFi Registry by accident, for instance.
    >     >
    >     > If your team/users are using NiFi Registry and have feedback,
    > positive or
    >     > otherwise, let us know as it would be great to factor that in to the
    >     > existing backlog for what to target in future releases.
    >     >
    >     > Best,
    >     > Kevin
    >     >
    >     > If your team runs into any trouble with this, do let us know as I'm
    > not
    >     > sure the deployment footprint
    >     >
    >     > On 6/28/18, 13:31, "Mike Thomsen" <[hidden email]> wrote:
    >     >
    >     >     Kevin,
    >     >
    >     >     The problem with our situation is that it's all inside Docker so
    > I
    >     > don't
    >     >     think I can just stop the registry and proceed. That's why I was
    >     > asking if
    >     >     it sounded feasible to just say "hands off for 5 minutes while
    > we copy
    >     > the
    >     >     data."
    >     >
    >     >     Also, what is the maturity level of the GitFlow option at this
    > point?
    >     > That
    >     >     was particularly attractive as a storage solution because it
    > should
    >     > make it
    >     >     much easier to export everything from our dev team to our end
    > users.
    >     >
    >     >     Thanks,
    >     >
    >     >     Mike
    >     >
    >     >     On Thu, Jun 28, 2018 at 1:28 PM Kevin Doran <[hidden email]>
    > wrote:
    >     >
    >     >     > Mike,
    >     >     >
    >     >     > I don't know the extent of H2's capabilities, but for NiFi
    > Registry
    >     >     > changing the location of those files will require restarting
    > the
    >     > server to
    >     >     > pick up the new database location from
    > nifi-registry.properties,
    >     > correct?
    >     >     > So if I follow correctly what the objective is, my
    > recommendation
    >     > would be
    >     >     > the following steps:
    >     >     >
    >     >     > 1. Stop NiFi Registry
    >     >     > 2. Move/copy the H2 database files to the Docker volume
    > location
    >     > (should
    >     >     > be safe to do when Registry is stopped)
    >     >     > 3. Change nifi-registry.properties to point the database config
    >     > props to
    >     >     > the new location
    >     >     > 4. Start NiFi Registry
    >     >     >
    >     >     > If the goal is actually just to copy a snapshot of the H2
    > database
    >     > to the
    >     >     > docker volume and not re-point NiFi Registry, which you want
    > to keep
    >     >     > running, I suppose you could do that. It should be fine to
    > copy it if
    >     >     > nothing is being written, but of course no guarantees and
    > YMMV, so
    >     > at the
    >     >     > least I'd try to verify that it is a validate db file after
    > copying
    >     > it,
    >     >     > using something like [1].
    >     >     >
    >     >     > [1]
    >     >     >
    >     >
    > https://stackoverflow.com/questions/32727588/how-to-check-h2-database-health-and-corruption
    >     >     >
    >     >     > Hope this helps,
    >     >     > Kevin
    >     >     >
    >     >     > On 6/28/18, 12:34, "Mike Thomsen" <[hidden email]>
    > wrote:
    >     >     >
    >     >     >     Kevin/Bryan,
    >     >     >
    >     >     >     We have a bunch of users that are using Docker and might
    > not
    >     > have set
    >     >     >     things up to point to a docker volume for the H2 database
    >     > location. Do
    >     >     >     either of you know if H2 will allow you to copy the files
    > over
    >     > to the
    >     >     >     Docker volume while everything's running?
    >     >     >
    >     >     >     Thanks,
    >     >     >
    >     >     >     Mike
    >     >     >
    >     >     >     On Wed, Jun 20, 2018 at 11:58 AM Bryan Bende <
    > [hidden email]>
    >     > wrote:
    >     >     >
    >     >     >     > Ah good point Mark... yes the old db properties are only
    > need
    >     > this
    >     >     >     > first time so that it can auto-migrate the old DB to the
    > new
    >     > one,
    >     >     >     > after that you don't need the old properties anymore.
    >     >     >     >
    >     >     >     > Thanks Kevin!
    >     >     >     >
    >     >     >     > On Wed, Jun 20, 2018 at 11:57 AM, Kevin Doran <
    >     > [hidden email]>
    >     >     > wrote:
    >     >     >     > > Thanks Mark and Bryan. I will add a NiFi Registry 0.1
    > -> 0.2
    >     >     > migration
    >     >     >     > guide to include these steps as part of updating the
    > site with
    >     > news
    >     >     > of the
    >     >     >     > new release.
    >     >     >     > >
    >     >     >     > > Thanks,
    >     >     >     > > Kevin
    >     >     >     > >
    >     >     >     > > ________________________________
    >     >     >     > > From: Mark Bean <[hidden email]>
    >     >     >     > > Sent: Wednesday, June 20, 2018 8:53:39 AM
    >     >     >     > > To: [hidden email]
    >     >     >     > > Subject: Re: Upgrading NiFi Registry
    >     >     >     > >
    >     >     >     > > Thanks Bryan. There is actually another step not
    > explicitly
    >     >     > mentioned. At
    >     >     >     > > least for 0.1.0 -> 0.2.0, I needed to modify the
    >     >     > nifi-registry.properties
    >     >     >     > > file as well. The 0.2.0 version has new
    > properties/values
    >     > not in
    >     >     > the
    >     >     >     > 0.1.0.
    >     >     >     > > And, I had to set the following for the database (using
    >     > values from
    >     >     >     > 0.1.0).
    >     >     >     > > This was only required on the first startup; these
    >     > properties can
    >     >     > be
    >     >     >     > empty
    >     >     >     > > values on subsequent startups.
    >     >     >     > > nifi.registry.db.directory=
    >     >     >     > > nifi.registry.db.url.append=
    >     >     >     > >
    >     >     >     > > In the future, if using a database location that is
    > external
    >     > to the
    >     >     >     > > installation directory, is nifi.registry.db.url the
    > only
    >     > property
    >     >     > that
    >     >     >     > > needs to be modified?
    >     >     >     > >
    >     >     >     > >
    >     >     >     > > On Wed, Jun 20, 2018 at 11:18 AM Bryan Bende <
    >     > [hidden email]>
    >     >     > wrote:
    >     >     >     > >
    >     >     >     > >> Mark,
    >     >     >     > >>
    >     >     >     > >> The database directory and flow storage directory are
    > where
    >     > all
    >     >     > the
    >     >     >     > >> data are. By default these are created in the root of
    > NiFi
    >     >     > Registry,
    >     >     >     > >> so depending how you want to set it up you could move
    > those
    >     >     >     > >> directories to the new install, or you could set them
    > up to
    >     > be
    >     >     >     > >> external locations so you don't have to move them
    > every
    >     > time, or
    >     >     > you
    >     >     >     > >> could upgrade the lib directory of your current
    > install and
    >     > leave
    >     >     >     > >> everything in place.
    >     >     >     > >>
    >     >     >     > >> The policies are the same as NiFi... stored in
    > users.xml and
    >     >     >     > >> authorizations.xml the conf directory, depending how
    > you
    >     >     > configured
    >     >     >     > >> everything. So just copying those two files over to
    > the new
    >     >     > install.
    >     >     >     > >>
    >     >     >     > >> -Bryan
    >     >     >     > >>
    >     >     >     > >>
    >     >     >     > >> On Wed, Jun 20, 2018 at 11:13 AM, Mark Bean <
    >     >     > [hidden email]>
    >     >     >     > wrote:
    >     >     >     > >> > How does one upgrade the NiFi Registry?
    >     >     >     > >> >
    >     >     >     > >> > After unpacking the .tar.gz file, how does one get
    > all
    >     > the flows
    >     >     >     > >> registered
    >     >     >     > >> > in a previous version of NiFi Registry into the
    > newly
    >     > installed
    >     >     >     > version?
    >     >     >     > >> > And, how does one ensure all the policies transfer
    > as
    >     > well?
    >     >     >     > >> >
    >     >     >     > >> > Thanks,
    >     >     >     > >> > Mark
    >     >     >     > >>
    >     >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >     >
    >     >
    >     >
    >     >
    >     >
    >
    >
    >
    >