nifi/commons consolidation suggestion

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

nifi/commons consolidation suggestion

Michael Moser
I'm not sure if this is the most appropriate forum or I should have
just written a Jira ticket, but here goes.

I believe we should consolidate the number of artifacts we have in the
nifi/commons module.  We create three jars that contain just 1 class
each and there are three more jars with 3 or fewer classes in them.
This makes it annoying (especially for beginners) to find the location
of classes that you need and slightly bloats our footprint for number
of artifacts that nifi create.  I believe we can improve this.

I analyzed all of the nar-bundles to find where each common library
was used.  Several are used by many framework, services, and
processors bundles already, so consolidating these common jars is a
no-brainer.  Other jars that are used more sparingly contain just 1 or
2 classes, so it really will have minimal impact to consolidate them
even if the classes aren't needed by a nar.

So, I propose we consolidate these artifacts into the nifi-utils
artifact.  The number in (parentheses) is the number of classes in
them.

core-flowfile-attributes (2)
flowfile-packager (9)
naive-search-ring-buffer (1)
nifi-file-utils (1)
nifi-logging-utils (1)
nifi-properties (2)
nifi-security-utils (5)
nifi-socket-utils (24)
nifi-stream-utils (17)
processor-utilities (3) (this would also resolve why the name doesn't
start with "nifi")

nifi-utils would go from 24 classes to 89 classes.

nifi-web-utils (3), remote-communications-utils (13), and search-utils
(5) I did not include because their use is limited to just one or two
places.

Thanks,
-- Mike
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Joe Witt
Mike,

I think this is a great point and a great analysis.

+1 and unless anyone specifically objects I'll go ahead and do this
tonight.  If i run into any curveballs I'll throw it on this thread.

Thanks
Joe

On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]> wrote:

>
> I'm not sure if this is the most appropriate forum or I should have
> just written a Jira ticket, but here goes.
>
> I believe we should consolidate the number of artifacts we have in the
> nifi/commons module.  We create three jars that contain just 1 class
> each and there are three more jars with 3 or fewer classes in them.
> This makes it annoying (especially for beginners) to find the location
> of classes that you need and slightly bloats our footprint for number
> of artifacts that nifi create.  I believe we can improve this.
>
> I analyzed all of the nar-bundles to find where each common library
> was used.  Several are used by many framework, services, and
> processors bundles already, so consolidating these common jars is a
> no-brainer.  Other jars that are used more sparingly contain just 1 or
> 2 classes, so it really will have minimal impact to consolidate them
> even if the classes aren't needed by a nar.
>
> So, I propose we consolidate these artifacts into the nifi-utils
> artifact.  The number in (parentheses) is the number of classes in
> them.
>
> core-flowfile-attributes (2)
> flowfile-packager (9)
> naive-search-ring-buffer (1)
> nifi-file-utils (1)
> nifi-logging-utils (1)
> nifi-properties (2)
> nifi-security-utils (5)
> nifi-socket-utils (24)
> nifi-stream-utils (17)
> processor-utilities (3) (this would also resolve why the name doesn't
> start with "nifi")
>
> nifi-utils would go from 24 classes to 89 classes.
>
> nifi-web-utils (3), remote-communications-utils (13), and search-utils
> (5) I did not include because their use is limited to just one or two
> places.
>
> Thanks,
> -- Mike
>
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Joey Echeverria
Do we know how many transitive dependencies this ends up mixing together?

I bring it up because that's often a reason for splitting a small
number of classes into their own module. For example, if I care about
socket-based data flow maybe I don't need the dependancies utilities
related to file-based data flow. I'll try to take a look at the actual
modules, but I thought I would throw that out there for others to
think on.

One thing I've seen work well is creating a dependency aggregator
module for users that don't care about the extra dependencies.

-Joey

On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:

> Mike,
>
> I think this is a great point and a great analysis.
>
> +1 and unless anyone specifically objects I'll go ahead and do this
> tonight.  If i run into any curveballs I'll throw it on this thread.
>
> Thanks
> Joe
>
> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]> wrote:
>>
>> I'm not sure if this is the most appropriate forum or I should have
>> just written a Jira ticket, but here goes.
>>
>> I believe we should consolidate the number of artifacts we have in the
>> nifi/commons module.  We create three jars that contain just 1 class
>> each and there are three more jars with 3 or fewer classes in them.
>> This makes it annoying (especially for beginners) to find the location
>> of classes that you need and slightly bloats our footprint for number
>> of artifacts that nifi create.  I believe we can improve this.
>>
>> I analyzed all of the nar-bundles to find where each common library
>> was used.  Several are used by many framework, services, and
>> processors bundles already, so consolidating these common jars is a
>> no-brainer.  Other jars that are used more sparingly contain just 1 or
>> 2 classes, so it really will have minimal impact to consolidate them
>> even if the classes aren't needed by a nar.
>>
>> So, I propose we consolidate these artifacts into the nifi-utils
>> artifact.  The number in (parentheses) is the number of classes in
>> them.
>>
>> core-flowfile-attributes (2)
>> flowfile-packager (9)
>> naive-search-ring-buffer (1)
>> nifi-file-utils (1)
>> nifi-logging-utils (1)
>> nifi-properties (2)
>> nifi-security-utils (5)
>> nifi-socket-utils (24)
>> nifi-stream-utils (17)
>> processor-utilities (3) (this would also resolve why the name doesn't
>> start with "nifi")
>>
>> nifi-utils would go from 24 classes to 89 classes.
>>
>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
>> (5) I did not include because their use is limited to just one or two
>> places.
>>
>> Thanks,
>> -- Mike
>>



--
Joey Echeverria
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Michael Moser
Joey makes a good point.  Without nifi-socket-utils, the transitive
dependencies will be commons-codec, commons-compress and
commons-lang3.  slf4j-api is also in there but that's marked provided
by all of NiFi.

When you add nifi-socket-utils to the equation, that adds commons-io
and commons-net.  Interestingly, nifi-socket-utils also depends on
three of the other nifi-*-utils.  So if you need nifi-socket-utils,
you also get nifi-properties, nifi-logging-utils, and nifi-utils
anyway.

It's probably worth leaving nifi-socket-utils separate for now.  It
had the biggest footprint of all of the utils to begin with.

flowfile-packager is the only one that pulls in commons-compress.
nifi-file-utils is the only one that pulls in commons-codec (though
that dependency could be removed with a clever refactor of
computeMd5Digest(File file) using just the JDK).

-- Mike


On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]> wrote:

> Do we know how many transitive dependencies this ends up mixing together?
>
> I bring it up because that's often a reason for splitting a small
> number of classes into their own module. For example, if I care about
> socket-based data flow maybe I don't need the dependancies utilities
> related to file-based data flow. I'll try to take a look at the actual
> modules, but I thought I would throw that out there for others to
> think on.
>
> One thing I've seen work well is creating a dependency aggregator
> module for users that don't care about the extra dependencies.
>
> -Joey
>
> On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:
>> Mike,
>>
>> I think this is a great point and a great analysis.
>>
>> +1 and unless anyone specifically objects I'll go ahead and do this
>> tonight.  If i run into any curveballs I'll throw it on this thread.
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]> wrote:
>>>
>>> I'm not sure if this is the most appropriate forum or I should have
>>> just written a Jira ticket, but here goes.
>>>
>>> I believe we should consolidate the number of artifacts we have in the
>>> nifi/commons module.  We create three jars that contain just 1 class
>>> each and there are three more jars with 3 or fewer classes in them.
>>> This makes it annoying (especially for beginners) to find the location
>>> of classes that you need and slightly bloats our footprint for number
>>> of artifacts that nifi create.  I believe we can improve this.
>>>
>>> I analyzed all of the nar-bundles to find where each common library
>>> was used.  Several are used by many framework, services, and
>>> processors bundles already, so consolidating these common jars is a
>>> no-brainer.  Other jars that are used more sparingly contain just 1 or
>>> 2 classes, so it really will have minimal impact to consolidate them
>>> even if the classes aren't needed by a nar.
>>>
>>> So, I propose we consolidate these artifacts into the nifi-utils
>>> artifact.  The number in (parentheses) is the number of classes in
>>> them.
>>>
>>> core-flowfile-attributes (2)
>>> flowfile-packager (9)
>>> naive-search-ring-buffer (1)
>>> nifi-file-utils (1)
>>> nifi-logging-utils (1)
>>> nifi-properties (2)
>>> nifi-security-utils (5)
>>> nifi-socket-utils (24)
>>> nifi-stream-utils (17)
>>> processor-utilities (3) (this would also resolve why the name doesn't
>>> start with "nifi")
>>>
>>> nifi-utils would go from 24 classes to 89 classes.
>>>
>>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
>>> (5) I did not include because their use is limited to just one or two
>>> places.
>>>
>>> Thanks,
>>> -- Mike
>>>
>
>
>
> --
> Joey Echeverria
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Joe Witt
i'm sure mark payne was laughing at my response on this thread pretty
hard.  I've given him grief before for the many split utilities jars we
have and he'd each time quickly remind me that it was to avoid pulling
needless deps into spaces we dont want them.  So fell into the trap again
today...

Thanks
Joe

On Thu, Dec 18, 2014 at 5:47 PM, Michael Moser <[hidden email]> wrote:

>
> Joey makes a good point.  Without nifi-socket-utils, the transitive
> dependencies will be commons-codec, commons-compress and
> commons-lang3.  slf4j-api is also in there but that's marked provided
> by all of NiFi.
>
> When you add nifi-socket-utils to the equation, that adds commons-io
> and commons-net.  Interestingly, nifi-socket-utils also depends on
> three of the other nifi-*-utils.  So if you need nifi-socket-utils,
> you also get nifi-properties, nifi-logging-utils, and nifi-utils
> anyway.
>
> It's probably worth leaving nifi-socket-utils separate for now.  It
> had the biggest footprint of all of the utils to begin with.
>
> flowfile-packager is the only one that pulls in commons-compress.
> nifi-file-utils is the only one that pulls in commons-codec (though
> that dependency could be removed with a clever refactor of
> computeMd5Digest(File file) using just the JDK).
>
> -- Mike
>
>
> On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]>
> wrote:
> > Do we know how many transitive dependencies this ends up mixing together?
> >
> > I bring it up because that's often a reason for splitting a small
> > number of classes into their own module. For example, if I care about
> > socket-based data flow maybe I don't need the dependancies utilities
> > related to file-based data flow. I'll try to take a look at the actual
> > modules, but I thought I would throw that out there for others to
> > think on.
> >
> > One thing I've seen work well is creating a dependency aggregator
> > module for users that don't care about the extra dependencies.
> >
> > -Joey
> >
> > On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:
> >> Mike,
> >>
> >> I think this is a great point and a great analysis.
> >>
> >> +1 and unless anyone specifically objects I'll go ahead and do this
> >> tonight.  If i run into any curveballs I'll throw it on this thread.
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]>
> wrote:
> >>>
> >>> I'm not sure if this is the most appropriate forum or I should have
> >>> just written a Jira ticket, but here goes.
> >>>
> >>> I believe we should consolidate the number of artifacts we have in the
> >>> nifi/commons module.  We create three jars that contain just 1 class
> >>> each and there are three more jars with 3 or fewer classes in them.
> >>> This makes it annoying (especially for beginners) to find the location
> >>> of classes that you need and slightly bloats our footprint for number
> >>> of artifacts that nifi create.  I believe we can improve this.
> >>>
> >>> I analyzed all of the nar-bundles to find where each common library
> >>> was used.  Several are used by many framework, services, and
> >>> processors bundles already, so consolidating these common jars is a
> >>> no-brainer.  Other jars that are used more sparingly contain just 1 or
> >>> 2 classes, so it really will have minimal impact to consolidate them
> >>> even if the classes aren't needed by a nar.
> >>>
> >>> So, I propose we consolidate these artifacts into the nifi-utils
> >>> artifact.  The number in (parentheses) is the number of classes in
> >>> them.
> >>>
> >>> core-flowfile-attributes (2)
> >>> flowfile-packager (9)
> >>> naive-search-ring-buffer (1)
> >>> nifi-file-utils (1)
> >>> nifi-logging-utils (1)
> >>> nifi-properties (2)
> >>> nifi-security-utils (5)
> >>> nifi-socket-utils (24)
> >>> nifi-stream-utils (17)
> >>> processor-utilities (3) (this would also resolve why the name doesn't
> >>> start with "nifi")
> >>>
> >>> nifi-utils would go from 24 classes to 89 classes.
> >>>
> >>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
> >>> (5) I did not include because their use is limited to just one or two
> >>> places.
> >>>
> >>> Thanks,
> >>> -- Mike
> >>>
> >
> >
> >
> > --
> > Joey Echeverria
>
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Michael Moser
You didn't fall too far, Joe.  core-flowfile-attributes,
naive-search-ring-buffer, nifi-properties, nifi-stream-utils, and
processor-utilities have no dependencies.  And nifi-logging-utils only
depends on the provided slf4j-api.  Just sayin'  ;)

-- Mike

On Thu, Dec 18, 2014 at 8:06 PM, Joe Witt <[hidden email]> wrote:

> i'm sure mark payne was laughing at my response on this thread pretty
> hard.  I've given him grief before for the many split utilities jars we
> have and he'd each time quickly remind me that it was to avoid pulling
> needless deps into spaces we dont want them.  So fell into the trap again
> today...
>
> Thanks
> Joe
>
> On Thu, Dec 18, 2014 at 5:47 PM, Michael Moser <[hidden email]> wrote:
>>
>> Joey makes a good point.  Without nifi-socket-utils, the transitive
>> dependencies will be commons-codec, commons-compress and
>> commons-lang3.  slf4j-api is also in there but that's marked provided
>> by all of NiFi.
>>
>> When you add nifi-socket-utils to the equation, that adds commons-io
>> and commons-net.  Interestingly, nifi-socket-utils also depends on
>> three of the other nifi-*-utils.  So if you need nifi-socket-utils,
>> you also get nifi-properties, nifi-logging-utils, and nifi-utils
>> anyway.
>>
>> It's probably worth leaving nifi-socket-utils separate for now.  It
>> had the biggest footprint of all of the utils to begin with.
>>
>> flowfile-packager is the only one that pulls in commons-compress.
>> nifi-file-utils is the only one that pulls in commons-codec (though
>> that dependency could be removed with a clever refactor of
>> computeMd5Digest(File file) using just the JDK).
>>
>> -- Mike
>>
>>
>> On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]>
>> wrote:
>> > Do we know how many transitive dependencies this ends up mixing together?
>> >
>> > I bring it up because that's often a reason for splitting a small
>> > number of classes into their own module. For example, if I care about
>> > socket-based data flow maybe I don't need the dependancies utilities
>> > related to file-based data flow. I'll try to take a look at the actual
>> > modules, but I thought I would throw that out there for others to
>> > think on.
>> >
>> > One thing I've seen work well is creating a dependency aggregator
>> > module for users that don't care about the extra dependencies.
>> >
>> > -Joey
>> >
>> > On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:
>> >> Mike,
>> >>
>> >> I think this is a great point and a great analysis.
>> >>
>> >> +1 and unless anyone specifically objects I'll go ahead and do this
>> >> tonight.  If i run into any curveballs I'll throw it on this thread.
>> >>
>> >> Thanks
>> >> Joe
>> >>
>> >> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]>
>> wrote:
>> >>>
>> >>> I'm not sure if this is the most appropriate forum or I should have
>> >>> just written a Jira ticket, but here goes.
>> >>>
>> >>> I believe we should consolidate the number of artifacts we have in the
>> >>> nifi/commons module.  We create three jars that contain just 1 class
>> >>> each and there are three more jars with 3 or fewer classes in them.
>> >>> This makes it annoying (especially for beginners) to find the location
>> >>> of classes that you need and slightly bloats our footprint for number
>> >>> of artifacts that nifi create.  I believe we can improve this.
>> >>>
>> >>> I analyzed all of the nar-bundles to find where each common library
>> >>> was used.  Several are used by many framework, services, and
>> >>> processors bundles already, so consolidating these common jars is a
>> >>> no-brainer.  Other jars that are used more sparingly contain just 1 or
>> >>> 2 classes, so it really will have minimal impact to consolidate them
>> >>> even if the classes aren't needed by a nar.
>> >>>
>> >>> So, I propose we consolidate these artifacts into the nifi-utils
>> >>> artifact.  The number in (parentheses) is the number of classes in
>> >>> them.
>> >>>
>> >>> core-flowfile-attributes (2)
>> >>> flowfile-packager (9)
>> >>> naive-search-ring-buffer (1)
>> >>> nifi-file-utils (1)
>> >>> nifi-logging-utils (1)
>> >>> nifi-properties (2)
>> >>> nifi-security-utils (5)
>> >>> nifi-socket-utils (24)
>> >>> nifi-stream-utils (17)
>> >>> processor-utilities (3) (this would also resolve why the name doesn't
>> >>> start with "nifi")
>> >>>
>> >>> nifi-utils would go from 24 classes to 89 classes.
>> >>>
>> >>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
>> >>> (5) I did not include because their use is limited to just one or two
>> >>> places.
>> >>>
>> >>> Thanks,
>> >>> -- Mike
>> >>>
>> >
>> >
>> >
>> > --
>> > Joey Echeverria
>>
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Joey Echeverria
Those sound like really good candidates for consolidation. It might
also be worth looking at the dependency graph to find a lot of
co-occurence. If every module that depends on A also depends on B,
there's less of an argument to keep them separate.

-Joey

On Thu, Dec 18, 2014 at 7:37 PM, Michael Moser <[hidden email]> wrote:

> You didn't fall too far, Joe.  core-flowfile-attributes,
> naive-search-ring-buffer, nifi-properties, nifi-stream-utils, and
> processor-utilities have no dependencies.  And nifi-logging-utils only
> depends on the provided slf4j-api.  Just sayin'  ;)
>
> -- Mike
>
> On Thu, Dec 18, 2014 at 8:06 PM, Joe Witt <[hidden email]> wrote:
>> i'm sure mark payne was laughing at my response on this thread pretty
>> hard.  I've given him grief before for the many split utilities jars we
>> have and he'd each time quickly remind me that it was to avoid pulling
>> needless deps into spaces we dont want them.  So fell into the trap again
>> today...
>>
>> Thanks
>> Joe
>>
>> On Thu, Dec 18, 2014 at 5:47 PM, Michael Moser <[hidden email]> wrote:
>>>
>>> Joey makes a good point.  Without nifi-socket-utils, the transitive
>>> dependencies will be commons-codec, commons-compress and
>>> commons-lang3.  slf4j-api is also in there but that's marked provided
>>> by all of NiFi.
>>>
>>> When you add nifi-socket-utils to the equation, that adds commons-io
>>> and commons-net.  Interestingly, nifi-socket-utils also depends on
>>> three of the other nifi-*-utils.  So if you need nifi-socket-utils,
>>> you also get nifi-properties, nifi-logging-utils, and nifi-utils
>>> anyway.
>>>
>>> It's probably worth leaving nifi-socket-utils separate for now.  It
>>> had the biggest footprint of all of the utils to begin with.
>>>
>>> flowfile-packager is the only one that pulls in commons-compress.
>>> nifi-file-utils is the only one that pulls in commons-codec (though
>>> that dependency could be removed with a clever refactor of
>>> computeMd5Digest(File file) using just the JDK).
>>>
>>> -- Mike
>>>
>>>
>>> On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]>
>>> wrote:
>>> > Do we know how many transitive dependencies this ends up mixing together?
>>> >
>>> > I bring it up because that's often a reason for splitting a small
>>> > number of classes into their own module. For example, if I care about
>>> > socket-based data flow maybe I don't need the dependancies utilities
>>> > related to file-based data flow. I'll try to take a look at the actual
>>> > modules, but I thought I would throw that out there for others to
>>> > think on.
>>> >
>>> > One thing I've seen work well is creating a dependency aggregator
>>> > module for users that don't care about the extra dependencies.
>>> >
>>> > -Joey
>>> >
>>> > On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:
>>> >> Mike,
>>> >>
>>> >> I think this is a great point and a great analysis.
>>> >>
>>> >> +1 and unless anyone specifically objects I'll go ahead and do this
>>> >> tonight.  If i run into any curveballs I'll throw it on this thread.
>>> >>
>>> >> Thanks
>>> >> Joe
>>> >>
>>> >> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]>
>>> wrote:
>>> >>>
>>> >>> I'm not sure if this is the most appropriate forum or I should have
>>> >>> just written a Jira ticket, but here goes.
>>> >>>
>>> >>> I believe we should consolidate the number of artifacts we have in the
>>> >>> nifi/commons module.  We create three jars that contain just 1 class
>>> >>> each and there are three more jars with 3 or fewer classes in them.
>>> >>> This makes it annoying (especially for beginners) to find the location
>>> >>> of classes that you need and slightly bloats our footprint for number
>>> >>> of artifacts that nifi create.  I believe we can improve this.
>>> >>>
>>> >>> I analyzed all of the nar-bundles to find where each common library
>>> >>> was used.  Several are used by many framework, services, and
>>> >>> processors bundles already, so consolidating these common jars is a
>>> >>> no-brainer.  Other jars that are used more sparingly contain just 1 or
>>> >>> 2 classes, so it really will have minimal impact to consolidate them
>>> >>> even if the classes aren't needed by a nar.
>>> >>>
>>> >>> So, I propose we consolidate these artifacts into the nifi-utils
>>> >>> artifact.  The number in (parentheses) is the number of classes in
>>> >>> them.
>>> >>>
>>> >>> core-flowfile-attributes (2)
>>> >>> flowfile-packager (9)
>>> >>> naive-search-ring-buffer (1)
>>> >>> nifi-file-utils (1)
>>> >>> nifi-logging-utils (1)
>>> >>> nifi-properties (2)
>>> >>> nifi-security-utils (5)
>>> >>> nifi-socket-utils (24)
>>> >>> nifi-stream-utils (17)
>>> >>> processor-utilities (3) (this would also resolve why the name doesn't
>>> >>> start with "nifi")
>>> >>>
>>> >>> nifi-utils would go from 24 classes to 89 classes.
>>> >>>
>>> >>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
>>> >>> (5) I did not include because their use is limited to just one or two
>>> >>> places.
>>> >>>
>>> >>> Thanks,
>>> >>> -- Mike
>>> >>>
>>> >
>>> >
>>> >
>>> > --
>>> > Joey Echeverria
>>>



--
Joey Echeverria
Reply | Threaded
Open this post in threaded view
|

RE: nifi/commons consolidation suggestion

Mark Payne
These probably can be cleaned up some now, as we now have a much more unified build, with a high level dependency management.
Careful, though: nifi-properties cannot be merged with anything else! It is on the root of the lib/ directory, so it cannot have any transitive dependencies at all.
-Mark

> Date: Fri, 19 Dec 2014 09:17:34 -0600
> Subject: Re: nifi/commons consolidation suggestion
> From: [hidden email]
> To: [hidden email]
>
> Those sound like really good candidates for consolidation. It might
> also be worth looking at the dependency graph to find a lot of
> co-occurence. If every module that depends on A also depends on B,
> there's less of an argument to keep them separate.
>
> -Joey
>
> On Thu, Dec 18, 2014 at 7:37 PM, Michael Moser <[hidden email]> wrote:
> > You didn't fall too far, Joe.  core-flowfile-attributes,
> > naive-search-ring-buffer, nifi-properties, nifi-stream-utils, and
> > processor-utilities have no dependencies.  And nifi-logging-utils only
> > depends on the provided slf4j-api.  Just sayin'  ;)
> >
> > -- Mike
> >
> > On Thu, Dec 18, 2014 at 8:06 PM, Joe Witt <[hidden email]> wrote:
> >> i'm sure mark payne was laughing at my response on this thread pretty
> >> hard.  I've given him grief before for the many split utilities jars we
> >> have and he'd each time quickly remind me that it was to avoid pulling
> >> needless deps into spaces we dont want them.  So fell into the trap again
> >> today...
> >>
> >> Thanks
> >> Joe
> >>
> >> On Thu, Dec 18, 2014 at 5:47 PM, Michael Moser <[hidden email]> wrote:
> >>>
> >>> Joey makes a good point.  Without nifi-socket-utils, the transitive
> >>> dependencies will be commons-codec, commons-compress and
> >>> commons-lang3.  slf4j-api is also in there but that's marked provided
> >>> by all of NiFi.
> >>>
> >>> When you add nifi-socket-utils to the equation, that adds commons-io
> >>> and commons-net.  Interestingly, nifi-socket-utils also depends on
> >>> three of the other nifi-*-utils.  So if you need nifi-socket-utils,
> >>> you also get nifi-properties, nifi-logging-utils, and nifi-utils
> >>> anyway.
> >>>
> >>> It's probably worth leaving nifi-socket-utils separate for now.  It
> >>> had the biggest footprint of all of the utils to begin with.
> >>>
> >>> flowfile-packager is the only one that pulls in commons-compress.
> >>> nifi-file-utils is the only one that pulls in commons-codec (though
> >>> that dependency could be removed with a clever refactor of
> >>> computeMd5Digest(File file) using just the JDK).
> >>>
> >>> -- Mike
> >>>
> >>>
> >>> On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]>
> >>> wrote:
> >>> > Do we know how many transitive dependencies this ends up mixing together?
> >>> >
> >>> > I bring it up because that's often a reason for splitting a small
> >>> > number of classes into their own module. For example, if I care about
> >>> > socket-based data flow maybe I don't need the dependancies utilities
> >>> > related to file-based data flow. I'll try to take a look at the actual
> >>> > modules, but I thought I would throw that out there for others to
> >>> > think on.
> >>> >
> >>> > One thing I've seen work well is creating a dependency aggregator
> >>> > module for users that don't care about the extra dependencies.
> >>> >
> >>> > -Joey
> >>> >
> >>> > On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]> wrote:
> >>> >> Mike,
> >>> >>
> >>> >> I think this is a great point and a great analysis.
> >>> >>
> >>> >> +1 and unless anyone specifically objects I'll go ahead and do this
> >>> >> tonight.  If i run into any curveballs I'll throw it on this thread.
> >>> >>
> >>> >> Thanks
> >>> >> Joe
> >>> >>
> >>> >> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <[hidden email]>
> >>> wrote:
> >>> >>>
> >>> >>> I'm not sure if this is the most appropriate forum or I should have
> >>> >>> just written a Jira ticket, but here goes.
> >>> >>>
> >>> >>> I believe we should consolidate the number of artifacts we have in the
> >>> >>> nifi/commons module.  We create three jars that contain just 1 class
> >>> >>> each and there are three more jars with 3 or fewer classes in them.
> >>> >>> This makes it annoying (especially for beginners) to find the location
> >>> >>> of classes that you need and slightly bloats our footprint for number
> >>> >>> of artifacts that nifi create.  I believe we can improve this.
> >>> >>>
> >>> >>> I analyzed all of the nar-bundles to find where each common library
> >>> >>> was used.  Several are used by many framework, services, and
> >>> >>> processors bundles already, so consolidating these common jars is a
> >>> >>> no-brainer.  Other jars that are used more sparingly contain just 1 or
> >>> >>> 2 classes, so it really will have minimal impact to consolidate them
> >>> >>> even if the classes aren't needed by a nar.
> >>> >>>
> >>> >>> So, I propose we consolidate these artifacts into the nifi-utils
> >>> >>> artifact.  The number in (parentheses) is the number of classes in
> >>> >>> them.
> >>> >>>
> >>> >>> core-flowfile-attributes (2)
> >>> >>> flowfile-packager (9)
> >>> >>> naive-search-ring-buffer (1)
> >>> >>> nifi-file-utils (1)
> >>> >>> nifi-logging-utils (1)
> >>> >>> nifi-properties (2)
> >>> >>> nifi-security-utils (5)
> >>> >>> nifi-socket-utils (24)
> >>> >>> nifi-stream-utils (17)
> >>> >>> processor-utilities (3) (this would also resolve why the name doesn't
> >>> >>> start with "nifi")
> >>> >>>
> >>> >>> nifi-utils would go from 24 classes to 89 classes.
> >>> >>>
> >>> >>> nifi-web-utils (3), remote-communications-utils (13), and search-utils
> >>> >>> (5) I did not include because their use is limited to just one or two
> >>> >>> places.
> >>> >>>
> >>> >>> Thanks,
> >>> >>> -- Mike
> >>> >>>
> >>> >
> >>> >
> >>> >
> >>> > --
> >>> > Joey Echeverria
> >>>
>
>
>
> --
> Joey Echeverria
     
Reply | Threaded
Open this post in threaded view
|

Re: nifi/commons consolidation suggestion

Joe Witt
https://issues.apache.org/jira/browse/NIFI-189

On Fri, Dec 19, 2014 at 11:05 AM, Mark Payne <[hidden email]> wrote:

> These probably can be cleaned up some now, as we now have a much more
> unified build, with a high level dependency management.
> Careful, though: nifi-properties cannot be merged with anything else! It
> is on the root of the lib/ directory, so it cannot have any transitive
> dependencies at all.
> -Mark
> > Date: Fri, 19 Dec 2014 09:17:34 -0600
> > Subject: Re: nifi/commons consolidation suggestion
> > From: [hidden email]
> > To: [hidden email]
> >
> > Those sound like really good candidates for consolidation. It might
> > also be worth looking at the dependency graph to find a lot of
> > co-occurence. If every module that depends on A also depends on B,
> > there's less of an argument to keep them separate.
> >
> > -Joey
> >
> > On Thu, Dec 18, 2014 at 7:37 PM, Michael Moser <[hidden email]>
> wrote:
> > > You didn't fall too far, Joe.  core-flowfile-attributes,
> > > naive-search-ring-buffer, nifi-properties, nifi-stream-utils, and
> > > processor-utilities have no dependencies.  And nifi-logging-utils only
> > > depends on the provided slf4j-api.  Just sayin'  ;)
> > >
> > > -- Mike
> > >
> > > On Thu, Dec 18, 2014 at 8:06 PM, Joe Witt <[hidden email]> wrote:
> > >> i'm sure mark payne was laughing at my response on this thread pretty
> > >> hard.  I've given him grief before for the many split utilities jars
> we
> > >> have and he'd each time quickly remind me that it was to avoid pulling
> > >> needless deps into spaces we dont want them.  So fell into the trap
> again
> > >> today...
> > >>
> > >> Thanks
> > >> Joe
> > >>
> > >> On Thu, Dec 18, 2014 at 5:47 PM, Michael Moser <[hidden email]>
> wrote:
> > >>>
> > >>> Joey makes a good point.  Without nifi-socket-utils, the transitive
> > >>> dependencies will be commons-codec, commons-compress and
> > >>> commons-lang3.  slf4j-api is also in there but that's marked provided
> > >>> by all of NiFi.
> > >>>
> > >>> When you add nifi-socket-utils to the equation, that adds commons-io
> > >>> and commons-net.  Interestingly, nifi-socket-utils also depends on
> > >>> three of the other nifi-*-utils.  So if you need nifi-socket-utils,
> > >>> you also get nifi-properties, nifi-logging-utils, and nifi-utils
> > >>> anyway.
> > >>>
> > >>> It's probably worth leaving nifi-socket-utils separate for now.  It
> > >>> had the biggest footprint of all of the utils to begin with.
> > >>>
> > >>> flowfile-packager is the only one that pulls in commons-compress.
> > >>> nifi-file-utils is the only one that pulls in commons-codec (though
> > >>> that dependency could be removed with a clever refactor of
> > >>> computeMd5Digest(File file) using just the JDK).
> > >>>
> > >>> -- Mike
> > >>>
> > >>>
> > >>> On Thu, Dec 18, 2014 at 2:38 PM, Joey Echeverria <[hidden email]>
> > >>> wrote:
> > >>> > Do we know how many transitive dependencies this ends up mixing
> together?
> > >>> >
> > >>> > I bring it up because that's often a reason for splitting a small
> > >>> > number of classes into their own module. For example, if I care
> about
> > >>> > socket-based data flow maybe I don't need the dependancies
> utilities
> > >>> > related to file-based data flow. I'll try to take a look at the
> actual
> > >>> > modules, but I thought I would throw that out there for others to
> > >>> > think on.
> > >>> >
> > >>> > One thing I've seen work well is creating a dependency aggregator
> > >>> > module for users that don't care about the extra dependencies.
> > >>> >
> > >>> > -Joey
> > >>> >
> > >>> > On Thu, Dec 18, 2014 at 12:13 PM, Joe Witt <[hidden email]>
> wrote:
> > >>> >> Mike,
> > >>> >>
> > >>> >> I think this is a great point and a great analysis.
> > >>> >>
> > >>> >> +1 and unless anyone specifically objects I'll go ahead and do
> this
> > >>> >> tonight.  If i run into any curveballs I'll throw it on this
> thread.
> > >>> >>
> > >>> >> Thanks
> > >>> >> Joe
> > >>> >>
> > >>> >> On Thu, Dec 18, 2014 at 11:04 AM, Michael Moser <
> [hidden email]>
> > >>> wrote:
> > >>> >>>
> > >>> >>> I'm not sure if this is the most appropriate forum or I should
> have
> > >>> >>> just written a Jira ticket, but here goes.
> > >>> >>>
> > >>> >>> I believe we should consolidate the number of artifacts we have
> in the
> > >>> >>> nifi/commons module.  We create three jars that contain just 1
> class
> > >>> >>> each and there are three more jars with 3 or fewer classes in
> them.
> > >>> >>> This makes it annoying (especially for beginners) to find the
> location
> > >>> >>> of classes that you need and slightly bloats our footprint for
> number
> > >>> >>> of artifacts that nifi create.  I believe we can improve this.
> > >>> >>>
> > >>> >>> I analyzed all of the nar-bundles to find where each common
> library
> > >>> >>> was used.  Several are used by many framework, services, and
> > >>> >>> processors bundles already, so consolidating these common jars
> is a
> > >>> >>> no-brainer.  Other jars that are used more sparingly contain
> just 1 or
> > >>> >>> 2 classes, so it really will have minimal impact to consolidate
> them
> > >>> >>> even if the classes aren't needed by a nar.
> > >>> >>>
> > >>> >>> So, I propose we consolidate these artifacts into the nifi-utils
> > >>> >>> artifact.  The number in (parentheses) is the number of classes
> in
> > >>> >>> them.
> > >>> >>>
> > >>> >>> core-flowfile-attributes (2)
> > >>> >>> flowfile-packager (9)
> > >>> >>> naive-search-ring-buffer (1)
> > >>> >>> nifi-file-utils (1)
> > >>> >>> nifi-logging-utils (1)
> > >>> >>> nifi-properties (2)
> > >>> >>> nifi-security-utils (5)
> > >>> >>> nifi-socket-utils (24)
> > >>> >>> nifi-stream-utils (17)
> > >>> >>> processor-utilities (3) (this would also resolve why the name
> doesn't
> > >>> >>> start with "nifi")
> > >>> >>>
> > >>> >>> nifi-utils would go from 24 classes to 89 classes.
> > >>> >>>
> > >>> >>> nifi-web-utils (3), remote-communications-utils (13), and
> search-utils
> > >>> >>> (5) I did not include because their use is limited to just one
> or two
> > >>> >>> places.
> > >>> >>>
> > >>> >>> Thanks,
> > >>> >>> -- Mike
> > >>> >>>
> > >>> >
> > >>> >
> > >>> >
> > >>> > --
> > >>> > Joey Echeverria
> > >>>
> >
> >
> >
> > --
> > Joey Echeverria
>
>