Authorization problems of NiFi secured cluster

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Authorization problems of NiFi secured cluster

Takanobu Asanuma
Hello experts,

When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.

For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.

This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.

Do you have any thoughts?

Thanks,
Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Koji Kawamura-2
Hello Takanobu,

If the issue doesn't happen with standalone mode, I assume it happens
because the security policy does not allow NiFi node to "view the
data".

When a user sends a request to a node within a cluster, the node
proxies the request to other nodes within the same cluster.
I'd recommend to check if conf/authorizers.xml has Node Identity
properties, looks like this:

<authorizer>
  ...
  <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
</authorizer>

IIRC, if you define the Node Identity before starting the secured
cluster at the first time, NiFi automatically creates necessary
policies for each node to proxy user request (I maybe wrong on
this..). If you already have the cluster started, then you can add
NiFi node as a user then add it to the "view the data" policy manually
(probably at the root PG's policy would be the most appropriate
place).

I confirmed that the issue can be reproduced by removing NiFi node
user from "view the data" policy.

Please try above and let us know if it addresses your issue.

Thanks,
Koji

On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma
<[hidden email]> wrote:

> Hello experts,
>
> When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.
>
> For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.
>
> This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.
>
> Do you have any thoughts?
>
> Thanks,
> Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

RE: Authorization problems of NiFi secured cluster

Takanobu Asanuma
Hi Koji,

Thank you for your quick and valuable answer! That's exactly what I need. After adding "Node Identity" of authorizers.xml to the "view the data" policy, the authorized user can list the queue.

>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..).

Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.

Thanks again!
Takanobu

-----Original Message-----
From: Koji Kawamura [mailto:[hidden email]]
Sent: Tuesday, June 27, 2017 2:32 PM
To: dev <[hidden email]>
Subject: Re: Authorization problems of NiFi secured cluster

Hello Takanobu,

If the issue doesn't happen with standalone mode, I assume it happens because the security policy does not allow NiFi node to "view the data".

When a user sends a request to a node within a cluster, the node proxies the request to other nodes within the same cluster.
I'd recommend to check if conf/authorizers.xml has Node Identity properties, looks like this:

<authorizer>
  ...
  <property name="Node Identity 1">CN=localhost, OU=NIFI</property> </authorizer>

IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..). If you already have the cluster started, then you can add NiFi node as a user then add it to the "view the data" policy manually (probably at the root PG's policy would be the most appropriate place).

I confirmed that the issue can be reproduced by removing NiFi node user from "view the data" policy.

Please try above and let us know if it addresses your issue.

Thanks,
Koji

On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <[hidden email]> wrote:

> Hello experts,
>
> When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.
>
> For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.
>
> This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.
>
> Do you have any thoughts?
>
> Thanks,
> Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Koji Kawamura-2
Hi Takanobu,

Glad to hear that you have it fixed.

> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.

Thanks for sharing, ideally NiFi cluster should work without adding
the policy manually.
I will try to setup a brand-new secured NiFi cluster to see what
initial policy setting will look like.
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster-node-identities

Thanks,
Koji

On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
<[hidden email]> wrote:

> Hi Koji,
>
> Thank you for your quick and valuable answer! That's exactly what I need. After adding "Node Identity" of authorizers.xml to the "view the data" policy, the authorized user can list the queue.
>
>>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..).
>
> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.
>
> Thanks again!
> Takanobu
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 2:32 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> Hello Takanobu,
>
> If the issue doesn't happen with standalone mode, I assume it happens because the security policy does not allow NiFi node to "view the data".
>
> When a user sends a request to a node within a cluster, the node proxies the request to other nodes within the same cluster.
> I'd recommend to check if conf/authorizers.xml has Node Identity properties, looks like this:
>
> <authorizer>
>   ...
>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property> </authorizer>
>
> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..). If you already have the cluster started, then you can add NiFi node as a user then add it to the "view the data" policy manually (probably at the root PG's policy would be the most appropriate place).
>
> I confirmed that the issue can be reproduced by removing NiFi node user from "view the data" policy.
>
> Please try above and let us know if it addresses your issue.
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <[hidden email]> wrote:
>> Hello experts,
>>
>> When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.
>>
>> For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.
>>
>> This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.
>>
>> Do you have any thoughts?
>>
>> Thanks,
>> Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Koji Kawamura-2
I just created a brand-new secured cluster now. NiFi automatically
created a policy "view the data" (and others) with the user defined as
"Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
It seems working as expected.

Koji

On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura <[hidden email]> wrote:

> Hi Takanobu,
>
> Glad to hear that you have it fixed.
>
>> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.
>
> Thanks for sharing, ideally NiFi cluster should work without adding
> the policy manually.
> I will try to setup a brand-new secured NiFi cluster to see what
> initial policy setting will look like.
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#cluster-node-identities
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> <[hidden email]> wrote:
>> Hi Koji,
>>
>> Thank you for your quick and valuable answer! That's exactly what I need. After adding "Node Identity" of authorizers.xml to the "view the data" policy, the authorized user can list the queue.
>>
>>>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..).
>>
>> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.
>>
>> Thanks again!
>> Takanobu
>>
>> -----Original Message-----
>> From: Koji Kawamura [mailto:[hidden email]]
>> Sent: Tuesday, June 27, 2017 2:32 PM
>> To: dev <[hidden email]>
>> Subject: Re: Authorization problems of NiFi secured cluster
>>
>> Hello Takanobu,
>>
>> If the issue doesn't happen with standalone mode, I assume it happens because the security policy does not allow NiFi node to "view the data".
>>
>> When a user sends a request to a node within a cluster, the node proxies the request to other nodes within the same cluster.
>> I'd recommend to check if conf/authorizers.xml has Node Identity properties, looks like this:
>>
>> <authorizer>
>>   ...
>>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property> </authorizer>
>>
>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..). If you already have the cluster started, then you can add NiFi node as a user then add it to the "view the data" policy manually (probably at the root PG's policy would be the most appropriate place).
>>
>> I confirmed that the issue can be reproduced by removing NiFi node user from "view the data" policy.
>>
>> Please try above and let us know if it addresses your issue.
>>
>> Thanks,
>> Koji
>>
>> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <[hidden email]> wrote:
>>> Hello experts,
>>>
>>> When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.
>>>
>>> For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.
>>>
>>> This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.
>>>
>>> Do you have any thoughts?
>>>
>>> Thanks,
>>> Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

RE: Authorization problems of NiFi secured cluster

Takanobu Asanuma
Hi Koji,

Thank you very much for the confirmation. Hmm... I will continue to investigate why my cluster does not work correctly.

Thanks again,
Takanobu

-----Original Message-----
From: Koji Kawamura [mailto:[hidden email]]
Sent: Tuesday, June 27, 2017 5:59 PM
To: dev <[hidden email]>
Subject: Re: Authorization problems of NiFi secured cluster

I just created a brand-new secured cluster now. NiFi automatically created a policy "view the data" (and others) with the user defined as "Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
It seems working as expected.

Koji

On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura <[hidden email]> wrote:

> Hi Takanobu,
>
> Glad to hear that you have it fixed.
>
>> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.
>
> Thanks for sharing, ideally NiFi cluster should work without adding
> the policy manually.
> I will try to setup a brand-new secured NiFi cluster to see what
> initial policy setting will look like.
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> cluster-node-identities
>
> Thanks,
> Koji
>
> On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> <[hidden email]> wrote:
>> Hi Koji,
>>
>> Thank you for your quick and valuable answer! That's exactly what I need. After adding "Node Identity" of authorizers.xml to the "view the data" policy, the authorized user can list the queue.
>>
>>>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..).
>>
>> Although I defined the Node Identity before stating the cluster at the first time, it seemed NiFi did not automatically create the policies and I needed to add the Node Identity to the policy explicitly.
>>
>> Thanks again!
>> Takanobu
>>
>> -----Original Message-----
>> From: Koji Kawamura [mailto:[hidden email]]
>> Sent: Tuesday, June 27, 2017 2:32 PM
>> To: dev <[hidden email]>
>> Subject: Re: Authorization problems of NiFi secured cluster
>>
>> Hello Takanobu,
>>
>> If the issue doesn't happen with standalone mode, I assume it happens because the security policy does not allow NiFi node to "view the data".
>>
>> When a user sends a request to a node within a cluster, the node proxies the request to other nodes within the same cluster.
>> I'd recommend to check if conf/authorizers.xml has Node Identity properties, looks like this:
>>
>> <authorizer>
>>   ...
>>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
>> </authorizer>
>>
>> IIRC, if you define the Node Identity before starting the secured cluster at the first time, NiFi automatically creates necessary policies for each node to proxy user request (I maybe wrong on this..). If you already have the cluster started, then you can add NiFi node as a user then add it to the "view the data" policy manually (probably at the root PG's policy would be the most appropriate place).
>>
>> I confirmed that the issue can be reproduced by removing NiFi node user from "view the data" policy.
>>
>> Please try above and let us know if it addresses your issue.
>>
>> Thanks,
>> Koji
>>
>> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <[hidden email]> wrote:
>>> Hello experts,
>>>
>>> When I created a NiFi cluster with security, any users can't list any queues due to "insufficient permissions" though the users have the permissions.
>>>
>>> For example, there is a dataflow which contains processor-A and processor-B, and processor-A is connecting to processor-B. In this case, even if user1 has the policies which are view/modify the component/data of processor-A and processor-B, he can't list the queue of the processors.
>>>
>>> This problem only occurs when the secured NiFi instance is clustering mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone mode, the problem doesn't happen. I have faced this problem with the latest release version, 1.3.0.
>>>
>>> Do you have any thoughts?
>>>
>>> Thanks,
>>> Takanobu Asanuma
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Matt Gilman
Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are
only granted for new instances when there is an existing flow.xml.gz in
your <NIFI_HOME>/conf directory. When there is no flow and the NiFi
instance is joining a cluster the policies cannot be granted at start up
because the components technically do not exist yet. However, your Initial
Admin is given the required permissions to grant those dataflow-specific
policies once the nodes have all joined the cluster. There is a short
snippet in the Admin guide describing this behavior [1] (if you scroll down
a little bit looking for the little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma <[hidden email]>
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically created
> a policy "view the data" (and others) with the user defined as "Initial
> Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura <[hidden email]>
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> > <[hidden email]> wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the
> data" policy, the authorized user can list the queue.
> >>
> >>>> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -----Original Message-----
> >> From: Koji Kawamura [mailto:[hidden email]]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev <[hidden email]>
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> <authorizer>
> >>   ...
> >>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
> >> </authorizer>
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..). If you
> already have the cluster started, then you can add NiFi node as a user then
> add it to the "view the data" policy manually (probably at the root PG's
> policy would be the most appropriate place).
> >>
> >> I confirmed that the issue can be reproduced by removing NiFi node user
> from "view the data" policy.
> >>
> >> Please try above and let us know if it addresses your issue.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> [hidden email]> wrote:
> >>> Hello experts,
> >>>
> >>> When I created a NiFi cluster with security, any users can't list any
> queues due to "insufficient permissions" though the users have the
> permissions.
> >>>
> >>> For example, there is a dataflow which contains processor-A and
> processor-B, and processor-A is connecting to processor-B. In this case,
> even if user1 has the policies which are view/modify the component/data of
> processor-A and processor-B, he can't list the queue of the processors.
> >>>
> >>> This problem only occurs when the secured NiFi instance is clustering
> mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone
> mode, the problem doesn't happen. I have faced this problem with the latest
> release version, 1.3.0.
> >>>
> >>> Do you have any thoughts?
> >>>
> >>> Thanks,
> >>> Takanobu Asanuma
>
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Koji Kawamura-2
Thanks Matt for clarification. My cluster had an existing flow.xml I
happened copied from another NiFi instance.

On Jun 27, 2017 9:14 PM, "Matt Gilman" <[hidden email]> wrote:

Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are
only granted for new instances when there is an existing flow.xml.gz in
your <NIFI_HOME>/conf directory. When there is no flow and the NiFi
instance is joining a cluster the policies cannot be granted at start up
because the components technically do not exist yet. However, your Initial
Admin is given the required permissions to grant those dataflow-specific
policies once the nodes have all joined the cluster. There is a short
snippet in the Admin guide describing this behavior [1] (if you scroll down
a little bit looking for the little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-
guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma <[hidden email]>
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically created
> a policy "view the data" (and others) with the user defined as "Initial
> Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura <[hidden email]>
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> > <[hidden email]> wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the
> data" policy, the authorized user can list the queue.
> >>
> >>>> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at the
> first time, it seemed NiFi did not automatically create the policies and I
> needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -----Original Message-----
> >> From: Koji Kawamura [mailto:[hidden email]]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev <[hidden email]>
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> <authorizer>
> >>   ...
> >>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
> >> </authorizer>
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary policies
> for each node to proxy user request (I maybe wrong on this..). If you
> already have the cluster started, then you can add NiFi node as a user
then

> add it to the "view the data" policy manually (probably at the root PG's
> policy would be the most appropriate place).
> >>
> >> I confirmed that the issue can be reproduced by removing NiFi node user
> from "view the data" policy.
> >>
> >> Please try above and let us know if it addresses your issue.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> [hidden email]> wrote:
> >>> Hello experts,
> >>>
> >>> When I created a NiFi cluster with security, any users can't list any
> queues due to "insufficient permissions" though the users have the
> permissions.
> >>>
> >>> For example, there is a dataflow which contains processor-A and
> processor-B, and processor-A is connecting to processor-B. In this case,
> even if user1 has the policies which are view/modify the component/data of
> processor-A and processor-B, he can't list the queue of the processors.
> >>>
> >>> This problem only occurs when the secured NiFi instance is clustering
> mode (nifi.cluster.is.node=true). If secured NiFi instance is standalone
> mode, the problem doesn't happen. I have faced this problem with the
latest
> release version, 1.3.0.
> >>>
> >>> Do you have any thoughts?
> >>>
> >>> Thanks,
> >>> Takanobu Asanuma
>
Reply | Threaded
Open this post in threaded view
|

RE: Authorization problems of NiFi secured cluster

Takanobu Asanuma
Hi Matt and Koji,

Thanks for the information. So if there is not any flow.xml.gz in conf directory when a secured nifi cluster is starting, we need to add "Node Identity" (and "Initial Admin Identity") to the policies (each component or PG) explicitly, right? That's my case. After adding flow.xml.gz and then starting the secured cluster, I confirmed that the policies are set automatically.

-----Original Message-----
From: Koji Kawamura [mailto:[hidden email]]
Sent: Tuesday, June 27, 2017 10:06 PM
To: dev <[hidden email]>
Subject: Re: Authorization problems of NiFi secured cluster

Thanks Matt for clarification. My cluster had an existing flow.xml I happened copied from another NiFi instance.

On Jun 27, 2017 9:14 PM, "Matt Gilman" <[hidden email]> wrote:

Takanobu,

The dataflow-specific policies (any policies on the root Process Group) are only granted for new instances when there is an existing flow.xml.gz in your <NIFI_HOME>/conf directory. When there is no flow and the NiFi instance is joining a cluster the policies cannot be granted at start up because the components technically do not exist yet. However, your Initial Admin is given the required permissions to grant those dataflow-specific policies once the nodes have all joined the cluster. There is a short snippet in the Admin guide describing this behavior [1] (if you scroll down a little bit looking for the little info (i) icon on the left).

Hope that clears it up.

Matt

[1]
https://nifi.apache.org/docs/nifi-docs/html/administration-
guide.html#authorizer-configuration

On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma <[hidden email]>
wrote:

> Hi Koji,
>
> Thank you very much for the confirmation. Hmm... I will continue to
> investigate why my cluster does not work correctly.
>
> Thanks again,
> Takanobu
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 5:59 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> I just created a brand-new secured cluster now. NiFi automatically
> created a policy "view the data" (and others) with the user defined as
> "Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
> It seems working as expected.
>
> Koji
>
> On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura
> <[hidden email]>
> wrote:
> > Hi Takanobu,
> >
> > Glad to hear that you have it fixed.
> >
> >> Although I defined the Node Identity before stating the cluster at
> >> the
> first time, it seemed NiFi did not automatically create the policies
> and I needed to add the Node Identity to the policy explicitly.
> >
> > Thanks for sharing, ideally NiFi cluster should work without adding
> > the policy manually.
> > I will try to setup a brand-new secured NiFi cluster to see what
> > initial policy setting will look like.
> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> > l#
> > cluster-node-identities
> >
> > Thanks,
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> > <[hidden email]> wrote:
> >> Hi Koji,
> >>
> >> Thank you for your quick and valuable answer! That's exactly what I
> need. After adding "Node Identity" of authorizers.xml to the "view the
> data" policy, the authorized user can list the queue.
> >>
> >>>> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary
> policies for each node to proxy user request (I maybe wrong on this..).
> >>
> >> Although I defined the Node Identity before stating the cluster at
> >> the
> first time, it seemed NiFi did not automatically create the policies
> and I needed to add the Node Identity to the policy explicitly.
> >>
> >> Thanks again!
> >> Takanobu
> >>
> >> -----Original Message-----
> >> From: Koji Kawamura [mailto:[hidden email]]
> >> Sent: Tuesday, June 27, 2017 2:32 PM
> >> To: dev <[hidden email]>
> >> Subject: Re: Authorization problems of NiFi secured cluster
> >>
> >> Hello Takanobu,
> >>
> >> If the issue doesn't happen with standalone mode, I assume it
> >> happens
> because the security policy does not allow NiFi node to "view the data".
> >>
> >> When a user sends a request to a node within a cluster, the node
> proxies the request to other nodes within the same cluster.
> >> I'd recommend to check if conf/authorizers.xml has Node Identity
> properties, looks like this:
> >>
> >> <authorizer>
> >>   ...
> >>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
> >> </authorizer>
> >>
> >> IIRC, if you define the Node Identity before starting the secured
> cluster at the first time, NiFi automatically creates necessary
> policies for each node to proxy user request (I maybe wrong on
> this..). If you already have the cluster started, then you can add
> NiFi node as a user
then

> add it to the "view the data" policy manually (probably at the root
> PG's policy would be the most appropriate place).
> >>
> >> I confirmed that the issue can be reproduced by removing NiFi node
> >> user
> from "view the data" policy.
> >>
> >> Please try above and let us know if it addresses your issue.
> >>
> >> Thanks,
> >> Koji
> >>
> >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> [hidden email]> wrote:
> >>> Hello experts,
> >>>
> >>> When I created a NiFi cluster with security, any users can't list
> >>> any
> queues due to "insufficient permissions" though the users have the
> permissions.
> >>>
> >>> For example, there is a dataflow which contains processor-A and
> processor-B, and processor-A is connecting to processor-B. In this
> case, even if user1 has the policies which are view/modify the
> component/data of processor-A and processor-B, he can't list the queue of the processors.
> >>>
> >>> This problem only occurs when the secured NiFi instance is
> >>> clustering
> mode (nifi.cluster.is.node=true). If secured NiFi instance is
> standalone mode, the problem doesn't happen. I have faced this problem
> with the
latest
> release version, 1.3.0.
> >>>
> >>> Do you have any thoughts?
> >>>
> >>> Thanks,
> >>> Takanobu Asanuma
>
Reply | Threaded
Open this post in threaded view
|

Re: Authorization problems of NiFi secured cluster

Mark Bean
This is correct. And, there is an alternative to adding the cluster Nodes
as "Node Identity" in the authorizers.xml file. Instead, you can use a
legacy authorized-users.xml file specifying the file location/name in the
"Legacy Authorized Users File" property. In this file, each cluster Node
should have the ROLE_PROXY role. This must be done when the cluster/node is
started for the first time and the authorizations.xml and users.xml files
are generated. If those files already exist, the Legacy Authorized Users
File will be ignored.


On Tue, Jun 27, 2017 at 11:59 PM, Takanobu Asanuma <[hidden email]>
wrote:

> Hi Matt and Koji,
>
> Thanks for the information. So if there is not any flow.xml.gz in conf
> directory when a secured nifi cluster is starting, we need to add "Node
> Identity" (and "Initial Admin Identity") to the policies (each component or
> PG) explicitly, right? That's my case. After adding flow.xml.gz and then
> starting the secured cluster, I confirmed that the policies are set
> automatically.
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 10:06 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> Thanks Matt for clarification. My cluster had an existing flow.xml I
> happened copied from another NiFi instance.
>
> On Jun 27, 2017 9:14 PM, "Matt Gilman" <[hidden email]> wrote:
>
> Takanobu,
>
> The dataflow-specific policies (any policies on the root Process Group)
> are only granted for new instances when there is an existing flow.xml.gz in
> your <NIFI_HOME>/conf directory. When there is no flow and the NiFi
> instance is joining a cluster the policies cannot be granted at start up
> because the components technically do not exist yet. However, your Initial
> Admin is given the required permissions to grant those dataflow-specific
> policies once the nodes have all joined the cluster. There is a short
> snippet in the Admin guide describing this behavior [1] (if you scroll down
> a little bit looking for the little info (i) icon on the left).
>
> Hope that clears it up.
>
> Matt
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#authorizer-configuration
>
> On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma <[hidden email]>
> wrote:
>
> > Hi Koji,
> >
> > Thank you very much for the confirmation. Hmm... I will continue to
> > investigate why my cluster does not work correctly.
> >
> > Thanks again,
> > Takanobu
> >
> > -----Original Message-----
> > From: Koji Kawamura [mailto:[hidden email]]
> > Sent: Tuesday, June 27, 2017 5:59 PM
> > To: dev <[hidden email]>
> > Subject: Re: Authorization problems of NiFi secured cluster
> >
> > I just created a brand-new secured cluster now. NiFi automatically
> > created a policy "view the data" (and others) with the user defined as
> > "Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
> > It seems working as expected.
> >
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura
> > <[hidden email]>
> > wrote:
> > > Hi Takanobu,
> > >
> > > Glad to hear that you have it fixed.
> > >
> > >> Although I defined the Node Identity before stating the cluster at
> > >> the
> > first time, it seemed NiFi did not automatically create the policies
> > and I needed to add the Node Identity to the policy explicitly.
> > >
> > > Thanks for sharing, ideally NiFi cluster should work without adding
> > > the policy manually.
> > > I will try to setup a brand-new secured NiFi cluster to see what
> > > initial policy setting will look like.
> > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.htm
> > > l#
> > > cluster-node-identities
> > >
> > > Thanks,
> > > Koji
> > >
> > > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> > > <[hidden email]> wrote:
> > >> Hi Koji,
> > >>
> > >> Thank you for your quick and valuable answer! That's exactly what I
> > need. After adding "Node Identity" of authorizers.xml to the "view the
> > data" policy, the authorized user can list the queue.
> > >>
> > >>>> IIRC, if you define the Node Identity before starting the secured
> > cluster at the first time, NiFi automatically creates necessary
> > policies for each node to proxy user request (I maybe wrong on this..).
> > >>
> > >> Although I defined the Node Identity before stating the cluster at
> > >> the
> > first time, it seemed NiFi did not automatically create the policies
> > and I needed to add the Node Identity to the policy explicitly.
> > >>
> > >> Thanks again!
> > >> Takanobu
> > >>
> > >> -----Original Message-----
> > >> From: Koji Kawamura [mailto:[hidden email]]
> > >> Sent: Tuesday, June 27, 2017 2:32 PM
> > >> To: dev <[hidden email]>
> > >> Subject: Re: Authorization problems of NiFi secured cluster
> > >>
> > >> Hello Takanobu,
> > >>
> > >> If the issue doesn't happen with standalone mode, I assume it
> > >> happens
> > because the security policy does not allow NiFi node to "view the data".
> > >>
> > >> When a user sends a request to a node within a cluster, the node
> > proxies the request to other nodes within the same cluster.
> > >> I'd recommend to check if conf/authorizers.xml has Node Identity
> > properties, looks like this:
> > >>
> > >> <authorizer>
> > >>   ...
> > >>   <property name="Node Identity 1">CN=localhost, OU=NIFI</property>
> > >> </authorizer>
> > >>
> > >> IIRC, if you define the Node Identity before starting the secured
> > cluster at the first time, NiFi automatically creates necessary
> > policies for each node to proxy user request (I maybe wrong on
> > this..). If you already have the cluster started, then you can add
> > NiFi node as a user
> then
> > add it to the "view the data" policy manually (probably at the root
> > PG's policy would be the most appropriate place).
> > >>
> > >> I confirmed that the issue can be reproduced by removing NiFi node
> > >> user
> > from "view the data" policy.
> > >>
> > >> Please try above and let us know if it addresses your issue.
> > >>
> > >> Thanks,
> > >> Koji
> > >>
> > >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> > [hidden email]> wrote:
> > >>> Hello experts,
> > >>>
> > >>> When I created a NiFi cluster with security, any users can't list
> > >>> any
> > queues due to "insufficient permissions" though the users have the
> > permissions.
> > >>>
> > >>> For example, there is a dataflow which contains processor-A and
> > processor-B, and processor-A is connecting to processor-B. In this
> > case, even if user1 has the policies which are view/modify the
> > component/data of processor-A and processor-B, he can't list the queue
> of the processors.
> > >>>
> > >>> This problem only occurs when the secured NiFi instance is
> > >>> clustering
> > mode (nifi.cluster.is.node=true). If secured NiFi instance is
> > standalone mode, the problem doesn't happen. I have faced this problem
> > with the
> latest
> > release version, 1.3.0.
> > >>>
> > >>> Do you have any thoughts?
> > >>>
> > >>> Thanks,
> > >>> Takanobu Asanuma
> >
>
Reply | Threaded
Open this post in threaded view
|

RE: Authorization problems of NiFi secured cluster

Takanobu Asanuma
Hi Mark,
Thanks for your reply and the new suggestion. I will try it, too!

-----Original Message-----
From: Mark Bean [mailto:[hidden email]]
Sent: Wednesday, June 28, 2017 9:13 PM
To: [hidden email]
Subject: Re: Authorization problems of NiFi secured cluster

This is correct. And, there is an alternative to adding the cluster Nodes as "Node Identity" in the authorizers.xml file. Instead, you can use a legacy authorized-users.xml file specifying the file location/name in the "Legacy Authorized Users File" property. In this file, each cluster Node should have the ROLE_PROXY role. This must be done when the cluster/node is started for the first time and the authorizations.xml and users.xml files are generated. If those files already exist, the Legacy Authorized Users File will be ignored.


On Tue, Jun 27, 2017 at 11:59 PM, Takanobu Asanuma <[hidden email]>
wrote:

> Hi Matt and Koji,
>
> Thanks for the information. So if there is not any flow.xml.gz in conf
> directory when a secured nifi cluster is starting, we need to add
> "Node Identity" (and "Initial Admin Identity") to the policies (each
> component or
> PG) explicitly, right? That's my case. After adding flow.xml.gz and
> then starting the secured cluster, I confirmed that the policies are
> set automatically.
>
> -----Original Message-----
> From: Koji Kawamura [mailto:[hidden email]]
> Sent: Tuesday, June 27, 2017 10:06 PM
> To: dev <[hidden email]>
> Subject: Re: Authorization problems of NiFi secured cluster
>
> Thanks Matt for clarification. My cluster had an existing flow.xml I
> happened copied from another NiFi instance.
>
> On Jun 27, 2017 9:14 PM, "Matt Gilman" <[hidden email]> wrote:
>
> Takanobu,
>
> The dataflow-specific policies (any policies on the root Process
> Group) are only granted for new instances when there is an existing
> flow.xml.gz in your <NIFI_HOME>/conf directory. When there is no flow
> and the NiFi instance is joining a cluster the policies cannot be
> granted at start up because the components technically do not exist
> yet. However, your Initial Admin is given the required permissions to
> grant those dataflow-specific policies once the nodes have all joined
> the cluster. There is a short snippet in the Admin guide describing
> this behavior [1] (if you scroll down a little bit looking for the little info (i) icon on the left).
>
> Hope that clears it up.
>
> Matt
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/html/administration-
> guide.html#authorizer-configuration
>
> On Tue, Jun 27, 2017 at 6:03 AM, Takanobu Asanuma
> <[hidden email]>
> wrote:
>
> > Hi Koji,
> >
> > Thank you very much for the confirmation. Hmm... I will continue to
> > investigate why my cluster does not work correctly.
> >
> > Thanks again,
> > Takanobu
> >
> > -----Original Message-----
> > From: Koji Kawamura [mailto:[hidden email]]
> > Sent: Tuesday, June 27, 2017 5:59 PM
> > To: dev <[hidden email]>
> > Subject: Re: Authorization problems of NiFi secured cluster
> >
> > I just created a brand-new secured cluster now. NiFi automatically
> > created a policy "view the data" (and others) with the user defined
> > as "Initial Admin Identity" and "Node Identity" in conf/authorizers.xml.
> > It seems working as expected.
> >
> > Koji
> >
> > On Tue, Jun 27, 2017 at 5:26 PM, Koji Kawamura
> > <[hidden email]>
> > wrote:
> > > Hi Takanobu,
> > >
> > > Glad to hear that you have it fixed.
> > >
> > >> Although I defined the Node Identity before stating the cluster
> > >> at the
> > first time, it seemed NiFi did not automatically create the policies
> > and I needed to add the Node Identity to the policy explicitly.
> > >
> > > Thanks for sharing, ideally NiFi cluster should work without
> > > adding the policy manually.
> > > I will try to setup a brand-new secured NiFi cluster to see what
> > > initial policy setting will look like.
> > > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.h
> > > tm
> > > l#
> > > cluster-node-identities
> > >
> > > Thanks,
> > > Koji
> > >
> > > On Tue, Jun 27, 2017 at 5:08 PM, Takanobu Asanuma
> > > <[hidden email]> wrote:
> > >> Hi Koji,
> > >>
> > >> Thank you for your quick and valuable answer! That's exactly what
> > >> I
> > need. After adding "Node Identity" of authorizers.xml to the "view
> > the data" policy, the authorized user can list the queue.
> > >>
> > >>>> IIRC, if you define the Node Identity before starting the
> > >>>> secured
> > cluster at the first time, NiFi automatically creates necessary
> > policies for each node to proxy user request (I maybe wrong on this..).
> > >>
> > >> Although I defined the Node Identity before stating the cluster
> > >> at the
> > first time, it seemed NiFi did not automatically create the policies
> > and I needed to add the Node Identity to the policy explicitly.
> > >>
> > >> Thanks again!
> > >> Takanobu
> > >>
> > >> -----Original Message-----
> > >> From: Koji Kawamura [mailto:[hidden email]]
> > >> Sent: Tuesday, June 27, 2017 2:32 PM
> > >> To: dev <[hidden email]>
> > >> Subject: Re: Authorization problems of NiFi secured cluster
> > >>
> > >> Hello Takanobu,
> > >>
> > >> If the issue doesn't happen with standalone mode, I assume it
> > >> happens
> > because the security policy does not allow NiFi node to "view the data".
> > >>
> > >> When a user sends a request to a node within a cluster, the node
> > proxies the request to other nodes within the same cluster.
> > >> I'd recommend to check if conf/authorizers.xml has Node Identity
> > properties, looks like this:
> > >>
> > >> <authorizer>
> > >>   ...
> > >>   <property name="Node Identity 1">CN=localhost,
> > >> OU=NIFI</property> </authorizer>
> > >>
> > >> IIRC, if you define the Node Identity before starting the secured
> > cluster at the first time, NiFi automatically creates necessary
> > policies for each node to proxy user request (I maybe wrong on
> > this..). If you already have the cluster started, then you can add
> > NiFi node as a user
> then
> > add it to the "view the data" policy manually (probably at the root
> > PG's policy would be the most appropriate place).
> > >>
> > >> I confirmed that the issue can be reproduced by removing NiFi
> > >> node user
> > from "view the data" policy.
> > >>
> > >> Please try above and let us know if it addresses your issue.
> > >>
> > >> Thanks,
> > >> Koji
> > >>
> > >> On Tue, Jun 27, 2017 at 1:12 PM, Takanobu Asanuma <
> > [hidden email]> wrote:
> > >>> Hello experts,
> > >>>
> > >>> When I created a NiFi cluster with security, any users can't
> > >>> list any
> > queues due to "insufficient permissions" though the users have the
> > permissions.
> > >>>
> > >>> For example, there is a dataflow which contains processor-A and
> > processor-B, and processor-A is connecting to processor-B. In this
> > case, even if user1 has the policies which are view/modify the
> > component/data of processor-A and processor-B, he can't list the
> > queue
> of the processors.
> > >>>
> > >>> This problem only occurs when the secured NiFi instance is
> > >>> clustering
> > mode (nifi.cluster.is.node=true). If secured NiFi instance is
> > standalone mode, the problem doesn't happen. I have faced this
> > problem with the
> latest
> > release version, 1.3.0.
> > >>>
> > >>> Do you have any thoughts?
> > >>>
> > >>> Thanks,
> > >>> Takanobu Asanuma
> >
>