API to get all Policies

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

API to get all Policies

Lars Francke
Hi,

I was tasked with writing a tool to generate a kind of "audit report". For
that I need to get all policies that people have across various systems.
NiFi is one of them.

I see that we have a REST API for Policies but that doesn't expose a method
to expose _all_ policies. I'd like to add a REST endpoint that allows
retrieving all policies.
Before I open a Jira I wanted to get feedback whether this addition would
be acceptable.

Implementation notes
This is how I see the current flow of requests from the
AccessPolicyResource to the actual AccessPolicyProider:

AccessPolicyResource -> NiFiServiceFacade (StandardNiFiServiceFacade) ->
AccessPolicyDAO (StandardPolicyBasedAuthorizerDAO) -> AccessPolicyProvider

Fortunately the AccessPolicyProvider already has a method to retrieve all
policies. Should there be custom implementations by third-parties they
already support the necessary methods and I believe the classes that need
to be changed are all NiFi internal:

* AccessPolicyResource
* NiFiServiceFacade
* StandardNiFiServiceFacade
* AccessPolicyDAO
* StandardPolicyBasedAuthorizerDAO
* And probably a bunch of others especially test classes

If I don't hear any objections I will open a Jira issue and would try and
provide a patch.

Cheers,
Lars
Reply | Threaded
Open this post in threaded view
|

Re: API to get all Policies

Andy LoPresto-2
Lars,

What access controls do you anticipate putting on this API endpoint and what potential issues do you see? I could see this being abused if not secured very carefully, and it doesn’t seem like a common use case (notwithstanding your current requirement). Is this something that can be done by using the NiFi CLI to iterate/recurse through the various PGs and retrieve these policies?

Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 9, 2018, at 3:31 AM, Lars Francke <[hidden email]> wrote:
>
> Hi,
>
> I was tasked with writing a tool to generate a kind of "audit report". For
> that I need to get all policies that people have across various systems.
> NiFi is one of them.
>
> I see that we have a REST API for Policies but that doesn't expose a method
> to expose _all_ policies. I'd like to add a REST endpoint that allows
> retrieving all policies.
> Before I open a Jira I wanted to get feedback whether this addition would
> be acceptable.
>
> Implementation notes
> This is how I see the current flow of requests from the
> AccessPolicyResource to the actual AccessPolicyProider:
>
> AccessPolicyResource -> NiFiServiceFacade (StandardNiFiServiceFacade) ->
> AccessPolicyDAO (StandardPolicyBasedAuthorizerDAO) -> AccessPolicyProvider
>
> Fortunately the AccessPolicyProvider already has a method to retrieve all
> policies. Should there be custom implementations by third-parties they
> already support the necessary methods and I believe the classes that need
> to be changed are all NiFi internal:
>
> * AccessPolicyResource
> * NiFiServiceFacade
> * StandardNiFiServiceFacade
> * AccessPolicyDAO
> * StandardPolicyBasedAuthorizerDAO
> * And probably a bunch of others especially test classes
>
> If I don't hear any objections I will open a Jira issue and would try and
> provide a patch.
>
> Cheers,
> Lars

Reply | Threaded
Open this post in threaded view
|

Re: API to get all Policies

Lars Francke
Andy,

that's a good question. I have to admit that I thought about it and then
saw that there is already an Authorizable for this scenario so I assumed
that part was already taken care of. So whoever has the permission to view
"access all policies" would also be able to use the API? Were you thinking
of something different?

Cheers,
Lars



On Fri, Nov 9, 2018 at 12:35 AM Andy LoPresto <[hidden email]> wrote:

> Lars,
>
> What access controls do you anticipate putting on this API endpoint and
> what potential issues do you see? I could see this being abused if not
> secured very carefully, and it doesn’t seem like a common use case
> (notwithstanding your current requirement). Is this something that can be
> done by using the NiFi CLI to iterate/recurse through the various PGs and
> retrieve these policies?
>
> Andy LoPresto
> [hidden email]
> [hidden email]
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Nov 9, 2018, at 3:31 AM, Lars Francke <[hidden email]> wrote:
> >
> > Hi,
> >
> > I was tasked with writing a tool to generate a kind of "audit report".
> For
> > that I need to get all policies that people have across various systems.
> > NiFi is one of them.
> >
> > I see that we have a REST API for Policies but that doesn't expose a
> method
> > to expose _all_ policies. I'd like to add a REST endpoint that allows
> > retrieving all policies.
> > Before I open a Jira I wanted to get feedback whether this addition would
> > be acceptable.
> >
> > Implementation notes
> > This is how I see the current flow of requests from the
> > AccessPolicyResource to the actual AccessPolicyProider:
> >
> > AccessPolicyResource -> NiFiServiceFacade (StandardNiFiServiceFacade) ->
> > AccessPolicyDAO (StandardPolicyBasedAuthorizerDAO) ->
> AccessPolicyProvider
> >
> > Fortunately the AccessPolicyProvider already has a method to retrieve all
> > policies. Should there be custom implementations by third-parties they
> > already support the necessary methods and I believe the classes that need
> > to be changed are all NiFi internal:
> >
> > * AccessPolicyResource
> > * NiFiServiceFacade
> > * StandardNiFiServiceFacade
> > * AccessPolicyDAO
> > * StandardPolicyBasedAuthorizerDAO
> > * And probably a bunch of others especially test classes
> >
> > If I don't hear any objections I will open a Jira issue and would try and
> > provide a patch.
> >
> > Cheers,
> > Lars
>
>
Reply | Threaded
Open this post in threaded view
|

Re: API to get all Policies

Lars Francke
I've just tried implementing my new resource and it seems to work fine and
as I expect it to. Also in regards to authorization. Users cannot see
anything that they are not allowed to do anyway.

Regarding your other comments: I agree that it's probably not a super
common use case.

Either way I'd love to use a API that I can access remotely as I need to
connect to other systems as well (e.g. Kafka, HBase etc.) so I don't want
to colocate my service on one of the NiFi machines.
But yes I could probably get a list of all resources somehow using the API
and then send one request per resource. But that seems overly complicated.

So if you don't object I'd create a Jira.

Cheers,
Lars


On Fri, Nov 9, 2018 at 10:01 AM Lars Francke <[hidden email]> wrote:

> Andy,
>
> that's a good question. I have to admit that I thought about it and then
> saw that there is already an Authorizable for this scenario so I assumed
> that part was already taken care of. So whoever has the permission to view
> "access all policies" would also be able to use the API? Were you thinking
> of something different?
>
> Cheers,
> Lars
>
>
>
> On Fri, Nov 9, 2018 at 12:35 AM Andy LoPresto <[hidden email]>
> wrote:
>
>> Lars,
>>
>> What access controls do you anticipate putting on this API endpoint and
>> what potential issues do you see? I could see this being abused if not
>> secured very carefully, and it doesn’t seem like a common use case
>> (notwithstanding your current requirement). Is this something that can be
>> done by using the NiFi CLI to iterate/recurse through the various PGs and
>> retrieve these policies?
>>
>> Andy LoPresto
>> [hidden email]
>> [hidden email]
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> > On Nov 9, 2018, at 3:31 AM, Lars Francke <[hidden email]>
>> wrote:
>> >
>> > Hi,
>> >
>> > I was tasked with writing a tool to generate a kind of "audit report".
>> For
>> > that I need to get all policies that people have across various systems.
>> > NiFi is one of them.
>> >
>> > I see that we have a REST API for Policies but that doesn't expose a
>> method
>> > to expose _all_ policies. I'd like to add a REST endpoint that allows
>> > retrieving all policies.
>> > Before I open a Jira I wanted to get feedback whether this addition
>> would
>> > be acceptable.
>> >
>> > Implementation notes
>> > This is how I see the current flow of requests from the
>> > AccessPolicyResource to the actual AccessPolicyProider:
>> >
>> > AccessPolicyResource -> NiFiServiceFacade (StandardNiFiServiceFacade) ->
>> > AccessPolicyDAO (StandardPolicyBasedAuthorizerDAO) ->
>> AccessPolicyProvider
>> >
>> > Fortunately the AccessPolicyProvider already has a method to retrieve
>> all
>> > policies. Should there be custom implementations by third-parties they
>> > already support the necessary methods and I believe the classes that
>> need
>> > to be changed are all NiFi internal:
>> >
>> > * AccessPolicyResource
>> > * NiFiServiceFacade
>> > * StandardNiFiServiceFacade
>> > * AccessPolicyDAO
>> > * StandardPolicyBasedAuthorizerDAO
>> > * And probably a bunch of others especially test classes
>> >
>> > If I don't hear any objections I will open a Jira issue and would try
>> and
>> > provide a patch.
>> >
>> > Cheers,
>> > Lars
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: API to get all Policies

Kevin Doran
Hi Lars,

I think as long as the following are true (it sounds like they are from what you have looked at):

1. the proposed endpoint does not require adding any additional Authorizable or policy to protect, and
2. the proposed endpoint does not expose any information that the authenticated client/user would not already have access to view, and is merely acting as a convenience method to return a list of things they could fetch individually

then in that case this is probably fine. No objection from me.

Any time we are adding a collection endpoint, my main concern is if pagination of the results also needs to be added (i.e., if for typical usage of NiFi, the response size of the JSON result would be larger than is reasonable to transmit in a single HTTP round trip, or if creating the response would be unreasonable load on the server). In typical usage of NiFi, I don't think the number of policies is that large (perhaps others can chime in if they feel differently?), so it would come down to what is the size of a policy element when returned in a list. If it is very large, you may also want to introduce a summary view/perspective of the policy that reduces the amount of information to the minimal that is required for a list view... I think that may already exist for NiFi in the AccessPolicySummary object, but it's been a while since I've looked at the code so I may be forgetting the details or confusing it with the NiFi Registry implementation, which does have a get all policies endpoint.

Lastly, take care that the Swagger annotations that are used to drive the Rest API documentation are accurate. If you have any questions on that let me know. Happy to help review a PR if you submit one.

Regards,
Kevin

On 11/9/18, 06:23, "Lars Francke" <[hidden email]> wrote:

    I've just tried implementing my new resource and it seems to work fine and
    as I expect it to. Also in regards to authorization. Users cannot see
    anything that they are not allowed to do anyway.
   
    Regarding your other comments: I agree that it's probably not a super
    common use case.
   
    Either way I'd love to use a API that I can access remotely as I need to
    connect to other systems as well (e.g. Kafka, HBase etc.) so I don't want
    to colocate my service on one of the NiFi machines.
    But yes I could probably get a list of all resources somehow using the API
    and then send one request per resource. But that seems overly complicated.
   
    So if you don't object I'd create a Jira.
   
    Cheers,
    Lars
   
   
    On Fri, Nov 9, 2018 at 10:01 AM Lars Francke <[hidden email]> wrote:
   
    > Andy,
    >
    > that's a good question. I have to admit that I thought about it and then
    > saw that there is already an Authorizable for this scenario so I assumed
    > that part was already taken care of. So whoever has the permission to view
    > "access all policies" would also be able to use the API? Were you thinking
    > of something different?
    >
    > Cheers,
    > Lars
    >
    >
    >
    > On Fri, Nov 9, 2018 at 12:35 AM Andy LoPresto <[hidden email]>
    > wrote:
    >
    >> Lars,
    >>
    >> What access controls do you anticipate putting on this API endpoint and
    >> what potential issues do you see? I could see this being abused if not
    >> secured very carefully, and it doesn’t seem like a common use case
    >> (notwithstanding your current requirement). Is this something that can be
    >> done by using the NiFi CLI to iterate/recurse through the various PGs and
    >> retrieve these policies?
    >>
    >> Andy LoPresto
    >> [hidden email]
    >> [hidden email]
    >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
    >>
    >> > On Nov 9, 2018, at 3:31 AM, Lars Francke <[hidden email]>
    >> wrote:
    >> >
    >> > Hi,
    >> >
    >> > I was tasked with writing a tool to generate a kind of "audit report".
    >> For
    >> > that I need to get all policies that people have across various systems.
    >> > NiFi is one of them.
    >> >
    >> > I see that we have a REST API for Policies but that doesn't expose a
    >> method
    >> > to expose _all_ policies. I'd like to add a REST endpoint that allows
    >> > retrieving all policies.
    >> > Before I open a Jira I wanted to get feedback whether this addition
    >> would
    >> > be acceptable.
    >> >
    >> > Implementation notes
    >> > This is how I see the current flow of requests from the
    >> > AccessPolicyResource to the actual AccessPolicyProider:
    >> >
    >> > AccessPolicyResource -> NiFiServiceFacade (StandardNiFiServiceFacade) ->
    >> > AccessPolicyDAO (StandardPolicyBasedAuthorizerDAO) ->
    >> AccessPolicyProvider
    >> >
    >> > Fortunately the AccessPolicyProvider already has a method to retrieve
    >> all
    >> > policies. Should there be custom implementations by third-parties they
    >> > already support the necessary methods and I believe the classes that
    >> need
    >> > to be changed are all NiFi internal:
    >> >
    >> > * AccessPolicyResource
    >> > * NiFiServiceFacade
    >> > * StandardNiFiServiceFacade
    >> > * AccessPolicyDAO
    >> > * StandardPolicyBasedAuthorizerDAO
    >> > * And probably a bunch of others especially test classes
    >> >
    >> > If I don't hear any objections I will open a Jira issue and would try
    >> and
    >> > provide a patch.
    >> >
    >> > Cheers,
    >> > Lars
    >>
    >>