Variable / Nifi-EL for RPG Endpoint?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Variable / Nifi-EL for RPG Endpoint?

Jon Logan
We are running into a slight issue when we wish to migrate between NiFi
cluster instances...the RPG endpoint in our flow seems to disallow setting
of a variable, which makes our flow have to be specific to the specific
environment it's being deployed in. It seems odd that it won't let us use
an expression there and set the value to a variable -- is this an
oversight? Is there some design decision here that we're missing?

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Bryan Bende
Jon,

The RPG URL is treated similar to sensitive properties or parent
controller services, meaning that after deploying your flow you can
issue an update to the RPG to set the URL to given environment's value
and that change will not be considered a change as far as version
control and will be retained when upgrading the process group to newer
versions.

Thanks,

Bryan

On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
> We are running into a slight issue when we wish to migrate between NiFi
> cluster instances...the RPG endpoint in our flow seems to disallow setting
> of a variable, which makes our flow have to be specific to the specific
> environment it's being deployed in. It seems odd that it won't let us use
> an expression there and set the value to a variable -- is this an
> oversight? Is there some design decision here that we're missing?
>
> Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Jon Logan
Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
Docker Registry, but our plan was to bundle a flow.gz into a Docker image
directly. I'm not sure if maybe we're approaching this problem in a weird
way? How do most people handle multiple instances in different environments
that are disconnected from one another? We considered running a Docker
Registry container as an option, or specifying the flow.gz directly as an
option...both seem to have some issues and inconveniences.

Thanks!
Jon

On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende <[hidden email]> wrote:

> Jon,
>
> The RPG URL is treated similar to sensitive properties or parent
> controller services, meaning that after deploying your flow you can
> issue an update to the RPG to set the URL to given environment's value
> and that change will not be considered a change as far as version
> control and will be retained when upgrading the process group to newer
> versions.
>
> Thanks,
>
> Bryan
>
> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
> > We are running into a slight issue when we wish to migrate between NiFi
> > cluster instances...the RPG endpoint in our flow seems to disallow
> setting
> > of a variable, which makes our flow have to be specific to the specific
> > environment it's being deployed in. It seems odd that it won't let us use
> > an expression there and set the value to a variable -- is this an
> > oversight? Is there some design decision here that we're missing?
> >
> > Thanks!
>
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Andy LoPresto-2
I think Bryan was referring to Apache NiFi Registry [1] which is the complementary standalone application which hosts version controlled flow snippets. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Sep 6, 2018, at 1:09 PM, Jon Logan <[hidden email]> wrote:

Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
Docker Registry, but our plan was to bundle a flow.gz into a Docker image
directly. I'm not sure if maybe we're approaching this problem in a weird
way? How do most people handle multiple instances in different environments
that are disconnected from one another? We considered running a Docker
Registry container as an option, or specifying the flow.gz directly as an
option...both seem to have some issues and inconveniences.

Thanks!
Jon

On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende <[hidden email]> wrote:

Jon,

The RPG URL is treated similar to sensitive properties or parent
controller services, meaning that after deploying your flow you can
issue an update to the RPG to set the URL to given environment's value
and that change will not be considered a change as far as version
control and will be retained when upgrading the process group to newer
versions.

Thanks,

Bryan

On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
We are running into a slight issue when we wish to migrate between NiFi
cluster instances...the RPG endpoint in our flow seems to disallow
setting
of a variable, which makes our flow have to be specific to the specific
environment it's being deployed in. It seems odd that it won't let us use
an expression there and set the value to a variable -- is this an
oversight? Is there some design decision here that we're missing?

Thanks!



signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Bryan Bende
Sorry Jon, I made the assumption that you were talking about saving a
flow to NiFi Registry from say you dev environment and then importing
into another environment and figuring out how to deal with the RPG
URL.

The process I described is how RPG URLs are handled with versioned
flows in NiFi Registry, but if you are taking a different approach
based around templates, or some other mechanism, then you are correct
that the target URL doesn't currently support EL.

Traditionally EL is primarily for property descriptors which are the
fields on the properties tab of processors, controller services, and
reporting tasks. So RPG URLs kind of fall outside of that, not to say
it can't ever be supported, but I haven't thought through all the
ramifications of what that mean internally in the framework.


On Thu, Sep 6, 2018 at 4:11 PM, Andy LoPresto <[hidden email]> wrote:

> I think Bryan was referring to Apache NiFi Registry [1] which is the
> complementary standalone application which hosts version controlled flow
> snippets.
>
> [1] https://nifi.apache.org/registry.html
>
> Andy LoPresto
> [hidden email]
> [hidden email]
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 6, 2018, at 1:09 PM, Jon Logan <[hidden email]> wrote:
>
> Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
> Docker Registry, but our plan was to bundle a flow.gz into a Docker image
> directly. I'm not sure if maybe we're approaching this problem in a weird
> way? How do most people handle multiple instances in different environments
> that are disconnected from one another? We considered running a Docker
> Registry container as an option, or specifying the flow.gz directly as an
> option...both seem to have some issues and inconveniences.
>
> Thanks!
> Jon
>
> On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende <[hidden email]> wrote:
>
> Jon,
>
> The RPG URL is treated similar to sensitive properties or parent
> controller services, meaning that after deploying your flow you can
> issue an update to the RPG to set the URL to given environment's value
> and that change will not be considered a change as far as version
> control and will be retained when upgrading the process group to newer
> versions.
>
> Thanks,
>
> Bryan
>
> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
>
> We are running into a slight issue when we wish to migrate between NiFi
> cluster instances...the RPG endpoint in our flow seems to disallow
>
> setting
>
> of a variable, which makes our flow have to be specific to the specific
> environment it's being deployed in. It seems odd that it won't let us use
> an expression there and set the value to a variable -- is this an
> oversight? Is there some design decision here that we're missing?
>
> Thanks!
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Jon Logan
In reply to this post by Andy LoPresto-2
Yeah, I meant to say NiFi Registry but substituted in Docker instead! Part
of the reason we've approached NiFi registry with caution is the lack of
connectivity between installations. Would the recommended approach be
shipping a registry containing the flows desired, and then shipping the
NiFi containers without a baked-in flow? Unfortunately that would require a
manual step step out of the box as the RPG and other potential secrets
would need to be specified when pulling from the registry into the NiFi
cluster (though it could probably be done via the API)?

On Thu, Sep 6, 2018 at 4:11 PM, Andy LoPresto <[hidden email]> wrote:

> I think Bryan was referring to Apache NiFi Registry [1] which is the
> complementary standalone application which hosts version controlled flow
> snippets.
>
> [1] https://nifi.apache.org/registry.html
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 6, 2018, at 1:09 PM, Jon Logan <[hidden email]> wrote:
>
> Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
> Docker Registry, but our plan was to bundle a flow.gz into a Docker image
> directly. I'm not sure if maybe we're approaching this problem in a weird
> way? How do most people handle multiple instances in different environments
> that are disconnected from one another? We considered running a Docker
> Registry container as an option, or specifying the flow.gz directly as an
> option...both seem to have some issues and inconveniences.
>
> Thanks!
> Jon
>
> On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende <[hidden email]> wrote:
>
> Jon,
>
> The RPG URL is treated similar to sensitive properties or parent
> controller services, meaning that after deploying your flow you can
> issue an update to the RPG to set the URL to given environment's value
> and that change will not be considered a change as far as version
> control and will be retained when upgrading the process group to newer
> versions.
>
> Thanks,
>
> Bryan
>
> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
>
> We are running into a slight issue when we wish to migrate between NiFi
> cluster instances...the RPG endpoint in our flow seems to disallow
>
> setting
>
> of a variable, which makes our flow have to be specific to the specific
> environment it's being deployed in. It seems odd that it won't let us use
> an expression there and set the value to a variable -- is this an
> oversight? Is there some design decision here that we're missing?
>
> Thanks!
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Variable / Nifi-EL for RPG Endpoint?

Bryan Bende
That makes sense, not having connectivity definitely introduces some
additional work.

The approach you could take is to have a NiFi Registry instance per
environment [1] and then use the NiFi CLI [2] to export and import
between them.

You are right that the RPG and secrets would have to be applied
through some scripts that used the CLI or the REST API and maybe some
conventions you come up with for how you take some config file and
locate the components and which properties need which values, but
anything is possible via the API.

[1] https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/
[2] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli


On Thu, Sep 6, 2018 at 4:33 PM, Jon Logan <[hidden email]> wrote:

> Yeah, I meant to say NiFi Registry but substituted in Docker instead! Part
> of the reason we've approached NiFi registry with caution is the lack of
> connectivity between installations. Would the recommended approach be
> shipping a registry containing the flows desired, and then shipping the
> NiFi containers without a baked-in flow? Unfortunately that would require a
> manual step step out of the box as the RPG and other potential secrets
> would need to be specified when pulling from the registry into the NiFi
> cluster (though it could probably be done via the API)?
>
> On Thu, Sep 6, 2018 at 4:11 PM, Andy LoPresto <[hidden email]> wrote:
>
>> I think Bryan was referring to Apache NiFi Registry [1] which is the
>> complementary standalone application which hosts version controlled flow
>> snippets.
>>
>> [1] https://nifi.apache.org/registry.html
>>
>> Andy LoPresto
>> [hidden email]
>> *[hidden email] <[hidden email]>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Sep 6, 2018, at 1:09 PM, Jon Logan <[hidden email]> wrote:
>>
>> Thanks Bryan -- Is that in terms of Docker Registry? We don't currently use
>> Docker Registry, but our plan was to bundle a flow.gz into a Docker image
>> directly. I'm not sure if maybe we're approaching this problem in a weird
>> way? How do most people handle multiple instances in different environments
>> that are disconnected from one another? We considered running a Docker
>> Registry container as an option, or specifying the flow.gz directly as an
>> option...both seem to have some issues and inconveniences.
>>
>> Thanks!
>> Jon
>>
>> On Thu, Sep 6, 2018 at 4:01 PM, Bryan Bende <[hidden email]> wrote:
>>
>> Jon,
>>
>> The RPG URL is treated similar to sensitive properties or parent
>> controller services, meaning that after deploying your flow you can
>> issue an update to the RPG to set the URL to given environment's value
>> and that change will not be considered a change as far as version
>> control and will be retained when upgrading the process group to newer
>> versions.
>>
>> Thanks,
>>
>> Bryan
>>
>> On Thu, Sep 6, 2018 at 3:48 PM, Jon Logan <[hidden email]> wrote:
>>
>> We are running into a slight issue when we wish to migrate between NiFi
>> cluster instances...the RPG endpoint in our flow seems to disallow
>>
>> setting
>>
>> of a variable, which makes our flow have to be specific to the specific
>> environment it's being deployed in. It seems odd that it won't let us use
>> an expression there and set the value to a variable -- is this an
>> oversight? Is there some design decision here that we're missing?
>>
>> Thanks!
>>
>>
>>
>>