Duplicated processors when using nifi processors dependency

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Duplicated processors when using nifi processors dependency

Robert
For my custom processor I am extending AbstractS3Processor. For that I use nifi-aws-processors as a dependency.
With upgrade to 1.2.0, Nifi now loads also all aws processors that are packed in nar as a depencency. Is there any way to avoid duplicated processors that get loaded in that way?
Reply | Threaded
Open this post in threaded view
|

Re: Duplicated processors when using nifi processors dependency

Bryan Bende
Hi Robert,

It sounds like your NAR is including nifi-aws-processors, and so is
the regular AWS NAR. Previously it wasn't obvious this was happening,
and when NiFi started up it was finding those processors in one of the
NARs and skipping the other one. With component versioning it is now
more specific as to where the processors are coming from.

The preferred solution here would be to create a utility project
like"nifi-aws-utils" underneath nifi-nar-bundles/nifi-extension-utils:

https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-extension-utils

This project would then contain the abstract processor, then your
custom processors and the regular AWS processors could both depend on
and include this JAR.

If you wanted to submit a PR/patch that created this utility project,
we'd be happy to help get it reviewed/merged, although I realize you
would have to be dependent on the latest code in master until a
release occurred with the change.

Hope that helps.

Thanks,

Bryan



On Thu, May 18, 2017 at 2:58 AM, Robert <[hidden email]> wrote:

> For my custom processor I am extending AbstractS3Processor. For that I use
> nifi-aws-processors as a dependency.
> With upgrade to 1.2.0, Nifi now loads also all aws processors that are
> packed in nar as a depencency. Is there any way to avoid duplicated
> processors that get loaded in that way?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Duplicated processors when using nifi processors dependency

Robert
I tired to do move abstract aws processors from nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created nifi-aws-utils.

Problem that I face now is with AWSCredentialsProviderControllerService that imo should stay under nifi-aws-bundle, but it is required by AbstractAWSCredentialsProviderProcessor that should be moved to nifi-aws-utils as it is used by abstract aws processors.

What is the preferred way to solve this?
Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and both nifi-aws-processors and nifi-aws-utils can depend on it?

Reply | Threaded
Open this post in threaded view
|

Re: Duplicated processors when using nifi processors dependency

Bryan Bende
Currently the AWS NAR contains the processors, controller service API, and controller service implementation, all in one NAR.

Typically the controller service API should be in its own NAR, so there should be something like:

nifi-aws-bunde
        nifi-aws-service-api
        nifi-aws-service-api-nar
        nifi-aws-processors
        nifi-aws-nar

The nifi-aws-service-api module would have the AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR dependency on nifi-aws-service-api-nar.

Setting it up this way, you nifi-aws-utils project could have a provided dependency on the nifi-aws-service-api, and then your custom NAR could have a NAR dependency on nifi-aws-service-api-nar.

There is some more information here about setting up controller services:

https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices <https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices>

https://github.com/bbende/nifi-dependency-example <https://github.com/bbende/nifi-dependency-example>


> On May 20, 2017, at 10:06 AM, Robert <[hidden email]> wrote:
>
> I tired to do move abstract aws processors from
> nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created
> nifi-aws-utils.
>
> Problem that I face now is with AWSCredentialsProviderControllerService that
> imo should stay under nifi-aws-bundle, but it is required by
> AbstractAWSCredentialsProviderProcessor that should be moved to
> nifi-aws-utils as it is used by abstract aws processors.
>
> What is the preferred way to solve this?
> Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and
> both nifi-aws-processors and nifi-aws-utils can depend on it?
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902p15937.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Duplicated processors when using nifi processors dependency

James Wing
I created a JIRA for the nifi-aws-bundle structure,
https://issues.apache.org/jira/browse/NIFI-3950.

Is there a best practice for the abstract classes like Robert describes?
Are they a better fit for the service API bundle or the processors bundle?
Or does that become irrelevant once the NAR dependencies are aligned and
they can be references as Java libraries?

Thanks,

James

On Sun, May 21, 2017 at 10:28 AM, Bryan Bende <[hidden email]> wrote:

> Currently the AWS NAR contains the processors, controller service API, and
> controller service implementation, all in one NAR.
>
> Typically the controller service API should be in its own NAR, so there
> should be something like:
>
> nifi-aws-bunde
>         nifi-aws-service-api
>         nifi-aws-service-api-nar
>         nifi-aws-processors
>         nifi-aws-nar
>
> The nifi-aws-service-api module would have the
> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR
> dependency on nifi-aws-service-api-nar.
>
> Setting it up this way, you nifi-aws-utils project could have a provided
> dependency on the nifi-aws-service-api, and then your custom NAR could have
> a NAR dependency on nifi-aws-service-api-nar.
>
> There is some more information here about setting up controller services:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+
> Projects+for+Extensions#MavenProjectsforExtensions-
> LinkingProcessorsandControllerServices <https://cwiki.apache.org/
> confluence/display/NIFI/Maven+Projects+for+Extensions#
> MavenProjectsforExtensions-LinkingProcessorsandControllerServices>
>
> https://github.com/bbende/nifi-dependency-example <
> https://github.com/bbende/nifi-dependency-example>
>
>
> > On May 20, 2017, at 10:06 AM, Robert <[hidden email]> wrote:
> >
> > I tired to do move abstract aws processors from
> > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created
> > nifi-aws-utils.
> >
> > Problem that I face now is with AWSCredentialsProviderControllerService
> that
> > imo should stay under nifi-aws-bundle, but it is required by
> > AbstractAWSCredentialsProviderProcessor that should be moved to
> > nifi-aws-utils as it is used by abstract aws processors.
> >
> > What is the preferred way to solve this?
> > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and
> > both nifi-aws-processors and nifi-aws-utils can depend on it?
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-
> dependency-tp15902p15937.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplicated processors when using nifi processors dependency

Bryan Bende
Thanks James.

For the abstract AWS processors, I would go with one of the following...

1) Another module under nifi-aws-bundle like:

nifi-aws-bunde
    nifi-aws-service-api
    nifi-aws-service-api-nar
    nifi-aws-abstract-processors (or some similar name)
    nifi-aws-processors
    nifi-aws-nar

2) A new module under nifi-nar-bundles/nifi-extension-utils like nifi-aws-utils

Either way, the desired result is to have a JAR that someone can
depend on that only brings in the abstract processors.

We have been putting generic abstract processors into
nifi-nar-bundles/nifi-extension-utils /nifi-processor-utils, but since
this case depends on the AWS SDK, then we would be introducing that
dependency to all the NARs that depend on nifi-processor-utils, which
is why I was suggesting a new module.

Let me know if this doesn't make sense.


On Mon, May 22, 2017 at 1:24 AM, James Wing <[hidden email]> wrote:

> I created a JIRA for the nifi-aws-bundle structure,
> https://issues.apache.org/jira/browse/NIFI-3950.
>
> Is there a best practice for the abstract classes like Robert describes?
> Are they a better fit for the service API bundle or the processors bundle?
> Or does that become irrelevant once the NAR dependencies are aligned and
> they can be references as Java libraries?
>
> Thanks,
>
> James
>
> On Sun, May 21, 2017 at 10:28 AM, Bryan Bende <[hidden email]> wrote:
>
>> Currently the AWS NAR contains the processors, controller service API, and
>> controller service implementation, all in one NAR.
>>
>> Typically the controller service API should be in its own NAR, so there
>> should be something like:
>>
>> nifi-aws-bunde
>>         nifi-aws-service-api
>>         nifi-aws-service-api-nar
>>         nifi-aws-processors
>>         nifi-aws-nar
>>
>> The nifi-aws-service-api module would have the
>> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR
>> dependency on nifi-aws-service-api-nar.
>>
>> Setting it up this way, you nifi-aws-utils project could have a provided
>> dependency on the nifi-aws-service-api, and then your custom NAR could have
>> a NAR dependency on nifi-aws-service-api-nar.
>>
>> There is some more information here about setting up controller services:
>>
>> https://cwiki.apache.org/confluence/display/NIFI/Maven+
>> Projects+for+Extensions#MavenProjectsforExtensions-
>> LinkingProcessorsandControllerServices <https://cwiki.apache.org/
>> confluence/display/NIFI/Maven+Projects+for+Extensions#
>> MavenProjectsforExtensions-LinkingProcessorsandControllerServices>
>>
>> https://github.com/bbende/nifi-dependency-example <
>> https://github.com/bbende/nifi-dependency-example>
>>
>>
>> > On May 20, 2017, at 10:06 AM, Robert <[hidden email]> wrote:
>> >
>> > I tired to do move abstract aws processors from
>> > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created
>> > nifi-aws-utils.
>> >
>> > Problem that I face now is with AWSCredentialsProviderControllerService
>> that
>> > imo should stay under nifi-aws-bundle, but it is required by
>> > AbstractAWSCredentialsProviderProcessor that should be moved to
>> > nifi-aws-utils as it is used by abstract aws processors.
>> >
>> > What is the preferred way to solve this?
>> > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and
>> > both nifi-aws-processors and nifi-aws-utils can depend on it?
>> >
>> >
>> >
>> >
>> >
>> > --
>> > View this message in context: http://apache-nifi-developer-
>> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-
>> dependency-tp15902p15937.html
>> > Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
>>