Planning for the next release

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Planning for the next release

Joe Witt
All,

With the 0.1.0 release hopefully soon available it is time to turn
towards the next release or so and get a sense of what we should focus
on.

This should include both 0.1.x but also 0.2.0.

Obviously first and foremost we need to work the existing PRs and
patches that exist.

Beyond that we have slated the following so far:

0.1.1
https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel

0.2.0
https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel

But what i'd like to throw out for general discussion is what are some
of the bigger thematic things we should focus on?  Things which will
help further with community growth and utility of the product for
those folks using it now?

For example:
I'd like to see us start digging into the cluster robustness issues
(HA cluster manager w/ legit leader election, etc..).  But there are
other things as well that may be more important sooner.

Please share your thoughts as this is a great time to effect those releases.

Thanks
Joe
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Sean Busbey
how easy / hard is it to dynamically add and remove nodes from cluster mode
right now? Being able to scale elastically would make gaining ground in
cloud deployments way easier.

On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:

> All,
>
> With the 0.1.0 release hopefully soon available it is time to turn
> towards the next release or so and get a sense of what we should focus
> on.
>
> This should include both 0.1.x but also 0.2.0.
>
> Obviously first and foremost we need to work the existing PRs and
> patches that exist.
>
> Beyond that we have slated the following so far:
>
> 0.1.1
>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> 0.2.0
>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> But what i'd like to throw out for general discussion is what are some
> of the bigger thematic things we should focus on?  Things which will
> help further with community growth and utility of the product for
> those folks using it now?
>
> For example:
> I'd like to see us start digging into the cluster robustness issues
> (HA cluster manager w/ legit leader election, etc..).  But there are
> other things as well that may be more important sooner.
>
> Please share your thoughts as this is a great time to effect those
> releases.
>
> Thanks
> Joe
>



--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Matthew Gaulin
I think it would be awesome to have a java driver for sending and receiving
data to/from a nifi flow directly, similar to the nifi-spark-receiver.
Then it would make it much easier to directly link with any java
application or something like storm.  Currently our team is using kafka to
bridge that gap, but it would be great if that could be streamlined by
going directly to and from a nifi flow.

Thanks!
Matt

On Thu, May 14, 2015 at 10:58 PM Sean Busbey <[hidden email]> wrote:

> how easy / hard is it to dynamically add and remove nodes from cluster mode
> right now? Being able to scale elastically would make gaining ground in
> cloud deployments way easier.
>
> On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:
>
> > All,
> >
> > With the 0.1.0 release hopefully soon available it is time to turn
> > towards the next release or so and get a sense of what we should focus
> > on.
> >
> > This should include both 0.1.x but also 0.2.0.
> >
> > Obviously first and foremost we need to work the existing PRs and
> > patches that exist.
> >
> > Beyond that we have slated the following so far:
> >
> > 0.1.1
> >
> >
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >
> > 0.2.0
> >
> >
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >
> > But what i'd like to throw out for general discussion is what are some
> > of the bigger thematic things we should focus on?  Things which will
> > help further with community growth and utility of the product for
> > those folks using it now?
> >
> > For example:
> > I'd like to see us start digging into the cluster robustness issues
> > (HA cluster manager w/ legit leader election, etc..).  But there are
> > other things as well that may be more important sooner.
> >
> > Please share your thoughts as this is a great time to effect those
> > releases.
> >
> > Thanks
> > Joe
> >
>
>
>
> --
> Sean
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Joe Witt
Matt G:

Understood.  It turns out that the spark receiver uses our client
library which does exactly what you suggest.  We should probably put
together a blog/writeup/howto for clients that want to push or pull
from nifi using this direct mechanism.  We've always focused on
supporting whatever protocols a given client and producer wants to use
so we've never really emphasized this.  But both models make sense.

Thanks
Joe

On Fri, May 15, 2015 at 9:34 AM, Matthew Gaulin <[hidden email]> wrote:

> I think it would be awesome to have a java driver for sending and receiving
> data to/from a nifi flow directly, similar to the nifi-spark-receiver.
> Then it would make it much easier to directly link with any java
> application or something like storm.  Currently our team is using kafka to
> bridge that gap, but it would be great if that could be streamlined by
> going directly to and from a nifi flow.
>
> Thanks!
> Matt
>
> On Thu, May 14, 2015 at 10:58 PM Sean Busbey <[hidden email]> wrote:
>
>> how easy / hard is it to dynamically add and remove nodes from cluster mode
>> right now? Being able to scale elastically would make gaining ground in
>> cloud deployments way easier.
>>
>> On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:
>>
>> > All,
>> >
>> > With the 0.1.0 release hopefully soon available it is time to turn
>> > towards the next release or so and get a sense of what we should focus
>> > on.
>> >
>> > This should include both 0.1.x but also 0.2.0.
>> >
>> > Obviously first and foremost we need to work the existing PRs and
>> > patches that exist.
>> >
>> > Beyond that we have slated the following so far:
>> >
>> > 0.1.1
>> >
>> >
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>> >
>> > 0.2.0
>> >
>> >
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>> >
>> > But what i'd like to throw out for general discussion is what are some
>> > of the bigger thematic things we should focus on?  Things which will
>> > help further with community growth and utility of the product for
>> > those folks using it now?
>> >
>> > For example:
>> > I'd like to see us start digging into the cluster robustness issues
>> > (HA cluster manager w/ legit leader election, etc..).  But there are
>> > other things as well that may be more important sooner.
>> >
>> > Please share your thoughts as this is a great time to effect those
>> > releases.
>> >
>> > Thanks
>> > Joe
>> >
>>
>>
>>
>> --
>> Sean
>>
Reply | Threaded
Open this post in threaded view
|

RE: Planning for the next release

Mark Payne
In reply to this post by Matthew Gaulin
Matt,

When we created the Spark Receiver, we did exactly that: we factored out a SiteToSite Client. The Spark Receiver is a very simple wrapper around that library.

You can pull the library in with the following dependency:

<dependency>
        <groupId>org.apache.nifi</groupId>
        <artifactId>nifi-site-to-site-client</artifactId>
        <version>0.0.2-incubating</version>
</dependency>

The main starting point, where you'll find most of the JavaDocs on how to use it are also available at:

https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;a=blob_plain;f=nifi/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java;hb=refs/heads/develop

If you have any issues making use of that, let us know and we'll help out!

Thanks
-Mark


----------------------------------------

> From: [hidden email]
> Date: Fri, 15 May 2015 13:34:17 +0000
> Subject: Re: Planning for the next release
> To: [hidden email]
>
> I think it would be awesome to have a java driver for sending and receiving
> data to/from a nifi flow directly, similar to the nifi-spark-receiver.
> Then it would make it much easier to directly link with any java
> application or something like storm. Currently our team is using kafka to
> bridge that gap, but it would be great if that could be streamlined by
> going directly to and from a nifi flow.
>
> Thanks!
> Matt
>
> On Thu, May 14, 2015 at 10:58 PM Sean Busbey <[hidden email]> wrote:
>
>> how easy / hard is it to dynamically add and remove nodes from cluster mode
>> right now? Being able to scale elastically would make gaining ground in
>> cloud deployments way easier.
>>
>> On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:
>>
>>> All,
>>>
>>> With the 0.1.0 release hopefully soon available it is time to turn
>>> towards the next release or so and get a sense of what we should focus
>>> on.
>>>
>>> This should include both 0.1.x but also 0.2.0.
>>>
>>> Obviously first and foremost we need to work the existing PRs and
>>> patches that exist.
>>>
>>> Beyond that we have slated the following so far:
>>>
>>> 0.1.1
>>>
>>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>> 0.2.0
>>>
>>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>> But what i'd like to throw out for general discussion is what are some
>>> of the bigger thematic things we should focus on? Things which will
>>> help further with community growth and utility of the product for
>>> those folks using it now?
>>>
>>> For example:
>>> I'd like to see us start digging into the cluster robustness issues
>>> (HA cluster manager w/ legit leader election, etc..). But there are
>>> other things as well that may be more important sooner.
>>>
>>> Please share your thoughts as this is a great time to effect those
>>> releases.
>>>
>>> Thanks
>>> Joe
>>>
>>
>>
>>
>> --
>> Sean
>>
     
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Corey Flowers
Every one I have talked to, have mentioned things like:

Centralized dash board to monitor all their NIFI instances (some of the
enterprises are projected to be in the hundreds of instances).
A way to remotely change processor state or properties.

Corey


On Fri, May 15, 2015 at 9:40 AM, Mark Payne <[hidden email]> wrote:

> Matt,
>
> When we created the Spark Receiver, we did exactly that: we factored out a
> SiteToSite Client. The Spark Receiver is a very simple wrapper around that
> library.
>
> You can pull the library in with the following dependency:
>
> <dependency>
>         <groupId>org.apache.nifi</groupId>
>         <artifactId>nifi-site-to-site-client</artifactId>
>         <version>0.0.2-incubating</version>
> </dependency>
>
> The main starting point, where you'll find most of the JavaDocs on how to
> use it are also available at:
>
>
> https://git-wip-us.apache.org/repos/asf?p=incubator-nifi.git;a=blob_plain;f=nifi/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/client/SiteToSiteClient.java;hb=refs/heads/develop
>
> If you have any issues making use of that, let us know and we'll help out!
>
> Thanks
> -Mark
>
>
> ----------------------------------------
> > From: [hidden email]
> > Date: Fri, 15 May 2015 13:34:17 +0000
> > Subject: Re: Planning for the next release
> > To: [hidden email]
> >
> > I think it would be awesome to have a java driver for sending and
> receiving
> > data to/from a nifi flow directly, similar to the nifi-spark-receiver.
> > Then it would make it much easier to directly link with any java
> > application or something like storm. Currently our team is using kafka to
> > bridge that gap, but it would be great if that could be streamlined by
> > going directly to and from a nifi flow.
> >
> > Thanks!
> > Matt
> >
> > On Thu, May 14, 2015 at 10:58 PM Sean Busbey <[hidden email]>
> wrote:
> >
> >> how easy / hard is it to dynamically add and remove nodes from cluster
> mode
> >> right now? Being able to scale elastically would make gaining ground in
> >> cloud deployments way easier.
> >>
> >> On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:
> >>
> >>> All,
> >>>
> >>> With the 0.1.0 release hopefully soon available it is time to turn
> >>> towards the next release or so and get a sense of what we should focus
> >>> on.
> >>>
> >>> This should include both 0.1.x but also 0.2.0.
> >>>
> >>> Obviously first and foremost we need to work the existing PRs and
> >>> patches that exist.
> >>>
> >>> Beyond that we have slated the following so far:
> >>>
> >>> 0.1.1
> >>>
> >>>
> >>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >>>
> >>> 0.2.0
> >>>
> >>>
> >>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >>>
> >>> But what i'd like to throw out for general discussion is what are some
> >>> of the bigger thematic things we should focus on? Things which will
> >>> help further with community growth and utility of the product for
> >>> those folks using it now?
> >>>
> >>> For example:
> >>> I'd like to see us start digging into the cluster robustness issues
> >>> (HA cluster manager w/ legit leader election, etc..). But there are
> >>> other things as well that may be more important sooner.
> >>>
> >>> Please share your thoughts as this is a great time to effect those
> >>> releases.
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>
> >>
> >>
> >> --
> >> Sean
> >>
>
>



--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Joe Witt
In reply to this post by Sean Busbey
Sean i definitely think this is the right track to head down.  My own
sense here is that we need to solve some of the fundamental clustering
items like effective leader election first.

On Thu, May 14, 2015 at 10:58 PM, Sean Busbey <[hidden email]> wrote:

> how easy / hard is it to dynamically add and remove nodes from cluster mode
> right now? Being able to scale elastically would make gaining ground in
> cloud deployments way easier.
>
> On Thu, May 14, 2015 at 9:29 PM, Joe Witt <[hidden email]> wrote:
>
>> All,
>>
>> With the 0.1.0 release hopefully soon available it is time to turn
>> towards the next release or so and get a sense of what we should focus
>> on.
>>
>> This should include both 0.1.x but also 0.2.0.
>>
>> Obviously first and foremost we need to work the existing PRs and
>> patches that exist.
>>
>> Beyond that we have slated the following so far:
>>
>> 0.1.1
>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>
>> 0.2.0
>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>
>> But what i'd like to throw out for general discussion is what are some
>> of the bigger thematic things we should focus on?  Things which will
>> help further with community growth and utility of the product for
>> those folks using it now?
>>
>> For example:
>> I'd like to see us start digging into the cluster robustness issues
>> (HA cluster manager w/ legit leader election, etc..).  But there are
>> other things as well that may be more important sooner.
>>
>> Please share your thoughts as this is a great time to effect those
>> releases.
>>
>> Thanks
>> Joe
>>
>
>
>
> --
> Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Sergio Fernández
In reply to this post by Joe Witt
Have you thought about graduating as TLP? FMPOV the project has
demonstrated enough maturity level to, at least, discuss it.
 On May 15, 2015 4:29 AM, "Joe Witt" <[hidden email]> wrote:

> All,
>
> With the 0.1.0 release hopefully soon available it is time to turn
> towards the next release or so and get a sense of what we should focus
> on.
>
> This should include both 0.1.x but also 0.2.0.
>
> Obviously first and foremost we need to work the existing PRs and
> patches that exist.
>
> Beyond that we have slated the following so far:
>
> 0.1.1
>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> 0.2.0
>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> But what i'd like to throw out for general discussion is what are some
> of the bigger thematic things we should focus on?  Things which will
> help further with community growth and utility of the product for
> those folks using it now?
>
> For example:
> I'd like to see us start digging into the cluster robustness issues
> (HA cluster manager w/ legit leader election, etc..).  But there are
> other things as well that may be more important sooner.
>
> Please share your thoughts as this is a great time to effect those
> releases.
>
> Thanks
> Joe
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Joe Witt
Sergio

Am really happy you think so.  Was just waiting to get through the release
before having that discussion.

Do others feel that we are at the point to start that as well?

Thanks
Joe
On May 15, 2015 12:21 PM, "Sergio Fernández" <[hidden email]> wrote:

> Have you thought about graduating as TLP? FMPOV the project has
> demonstrated enough maturity level to, at least, discuss it.
>  On May 15, 2015 4:29 AM, "Joe Witt" <[hidden email]> wrote:
>
> > All,
> >
> > With the 0.1.0 release hopefully soon available it is time to turn
> > towards the next release or so and get a sense of what we should focus
> > on.
> >
> > This should include both 0.1.x but also 0.2.0.
> >
> > Obviously first and foremost we need to work the existing PRs and
> > patches that exist.
> >
> > Beyond that we have slated the following so far:
> >
> > 0.1.1
> >
> >
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >
> > 0.2.0
> >
> >
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >
> > But what i'd like to throw out for general discussion is what are some
> > of the bigger thematic things we should focus on?  Things which will
> > help further with community growth and utility of the product for
> > those folks using it now?
> >
> > For example:
> > I'd like to see us start digging into the cluster robustness issues
> > (HA cluster manager w/ legit leader election, etc..).  But there are
> > other things as well that may be more important sooner.
> >
> > Please share your thoughts as this is a great time to effect those
> > releases.
> >
> > Thanks
> > Joe
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Ryan Blue
I think this community is ready to graduate.

rb

On 05/15/2015 09:26 AM, Joe Witt wrote:

> Sergio
>
> Am really happy you think so.  Was just waiting to get through the release
> before having that discussion.
>
> Do others feel that we are at the point to start that as well?
>
> Thanks
> Joe
> On May 15, 2015 12:21 PM, "Sergio Fernández" <[hidden email]> wrote:
>
>> Have you thought about graduating as TLP? FMPOV the project has
>> demonstrated enough maturity level to, at least, discuss it.
>>   On May 15, 2015 4:29 AM, "Joe Witt" <[hidden email]> wrote:
>>
>>> All,
>>>
>>> With the 0.1.0 release hopefully soon available it is time to turn
>>> towards the next release or so and get a sense of what we should focus
>>> on.
>>>
>>> This should include both 0.1.x but also 0.2.0.
>>>
>>> Obviously first and foremost we need to work the existing PRs and
>>> patches that exist.
>>>
>>> Beyond that we have slated the following so far:
>>>
>>> 0.1.1
>>>
>>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>> 0.2.0
>>>
>>>
>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>> But what i'd like to throw out for general discussion is what are some
>>> of the bigger thematic things we should focus on?  Things which will
>>> help further with community growth and utility of the product for
>>> those folks using it now?
>>>
>>> For example:
>>> I'd like to see us start digging into the cluster robustness issues
>>> (HA cluster manager w/ legit leader election, etc..).  But there are
>>> other things as well that may be more important sooner.
>>>
>>> Please share your thoughts as this is a great time to effect those
>>> releases.
>>>
>>> Thanks
>>> Joe
>>>
>>
>


--
Ryan Blue
Software Engineer
Cloudera, Inc.
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Corey Flowers
+1

On Fri, May 15, 2015 at 12:31 PM, Ryan Blue <[hidden email]> wrote:

> I think this community is ready to graduate.
>
> rb
>
>
> On 05/15/2015 09:26 AM, Joe Witt wrote:
>
>> Sergio
>>
>> Am really happy you think so.  Was just waiting to get through the release
>> before having that discussion.
>>
>> Do others feel that we are at the point to start that as well?
>>
>> Thanks
>> Joe
>> On May 15, 2015 12:21 PM, "Sergio Fernández" <[hidden email]> wrote:
>>
>>  Have you thought about graduating as TLP? FMPOV the project has
>>> demonstrated enough maturity level to, at least, discuss it.
>>>   On May 15, 2015 4:29 AM, "Joe Witt" <[hidden email]> wrote:
>>>
>>>  All,
>>>>
>>>> With the 0.1.0 release hopefully soon available it is time to turn
>>>> towards the next release or so and get a sense of what we should focus
>>>> on.
>>>>
>>>> This should include both 0.1.x but also 0.2.0.
>>>>
>>>> Obviously first and foremost we need to work the existing PRs and
>>>> patches that exist.
>>>>
>>>> Beyond that we have slated the following so far:
>>>>
>>>> 0.1.1
>>>>
>>>>
>>>>
>>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>>>
>>>> 0.2.0
>>>>
>>>>
>>>>
>>> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>>>
>>>>
>>>> But what i'd like to throw out for general discussion is what are some
>>>> of the bigger thematic things we should focus on?  Things which will
>>>> help further with community growth and utility of the product for
>>>> those folks using it now?
>>>>
>>>> For example:
>>>> I'd like to see us start digging into the cluster robustness issues
>>>> (HA cluster manager w/ legit leader election, etc..).  But there are
>>>> other things as well that may be more important sooner.
>>>>
>>>> Please share your thoughts as this is a great time to effect those
>>>> releases.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>>
>>>
>>
>
> --
> Ryan Blue
> Software Engineer
> Cloudera, Inc.
>



--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Brandon DeVries
If my vote on it counts for anything,  definitely +1 as well.

Brandon

On Fri, May 15, 2015, 12:32 PM Corey Flowers <[hidden email]> wrote:

> +1
>
> On Fri, May 15, 2015 at 12:31 PM, Ryan Blue <[hidden email]> wrote:
>
> > I think this community is ready to graduate.
> >
> > rb
> >
> >
> > On 05/15/2015 09:26 AM, Joe Witt wrote:
> >
> >> Sergio
> >>
> >> Am really happy you think so.  Was just waiting to get through the
> release
> >> before having that discussion.
> >>
> >> Do others feel that we are at the point to start that as well?
> >>
> >> Thanks
> >> Joe
> >> On May 15, 2015 12:21 PM, "Sergio Fernández" <[hidden email]> wrote:
> >>
> >>  Have you thought about graduating as TLP? FMPOV the project has
> >>> demonstrated enough maturity level to, at least, discuss it.
> >>>   On May 15, 2015 4:29 AM, "Joe Witt" <[hidden email]> wrote:
> >>>
> >>>  All,
> >>>>
> >>>> With the 0.1.0 release hopefully soon available it is time to turn
> >>>> towards the next release or so and get a sense of what we should focus
> >>>> on.
> >>>>
> >>>> This should include both 0.1.x but also 0.2.0.
> >>>>
> >>>> Obviously first and foremost we need to work the existing PRs and
> >>>> patches that exist.
> >>>>
> >>>> Beyond that we have slated the following so far:
> >>>>
> >>>> 0.1.1
> >>>>
> >>>>
> >>>>
> >>>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >>>
> >>>>
> >>>> 0.2.0
> >>>>
> >>>>
> >>>>
> >>>
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
> >>>
> >>>>
> >>>> But what i'd like to throw out for general discussion is what are some
> >>>> of the bigger thematic things we should focus on?  Things which will
> >>>> help further with community growth and utility of the product for
> >>>> those folks using it now?
> >>>>
> >>>> For example:
> >>>> I'd like to see us start digging into the cluster robustness issues
> >>>> (HA cluster manager w/ legit leader election, etc..).  But there are
> >>>> other things as well that may be more important sooner.
> >>>>
> >>>> Please share your thoughts as this is a great time to effect those
> >>>> releases.
> >>>>
> >>>> Thanks
> >>>> Joe
> >>>>
> >>>>
> >>>
> >>
> >
> > --
> > Ryan Blue
> > Software Engineer
> > Cloudera, Inc.
> >
>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> [hidden email]
>
> -- This account not approved for unencrypted proprietary information --
>
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Ryan Blue
In reply to this post by Joe Witt
I would like to see:

* Leader election

* Ability to balance channels across a cluster

* A discussion or design around better fault-tolerance: if a worker goes
down, how would we process its data elsewhere? (a short-term solution
could be adding Kafka-based channels)

* A discussion or design for balancing work across a cluster: pulling
data from HDFS has to be done in a single node, but if processors
supported some notion of pending work and could balance it across the
cluster, that would be helpful. For pulling from HDFS, that would be
listing the paths to process, then pulling them in parallel and marking
the path/task finished. This should be fault-tolerant so even if a node
goes down, another node does the work (otherwise we could just use a
simple partitioning scheme).

Obviously, these get complicated quick. But, I think some of these
features would really help adoption.

rb

On 05/14/2015 07:29 PM, Joe Witt wrote:

> All,
>
> With the 0.1.0 release hopefully soon available it is time to turn
> towards the next release or so and get a sense of what we should focus
> on.
>
> This should include both 0.1.x but also 0.2.0.
>
> Obviously first and foremost we need to work the existing PRs and
> patches that exist.
>
> Beyond that we have slated the following so far:
>
> 0.1.1
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12332286/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> 0.2.0
> https://issues.apache.org/jira/browse/NIFI/fixforversion/12329653/?selectedTab=com.atlassian.jira.jira-projects-plugin:version-summary-panel
>
> But what i'd like to throw out for general discussion is what are some
> of the bigger thematic things we should focus on?  Things which will
> help further with community growth and utility of the product for
> those folks using it now?
>
> For example:
> I'd like to see us start digging into the cluster robustness issues
> (HA cluster manager w/ legit leader election, etc..).  But there are
> other things as well that may be more important sooner.
>
> Please share your thoughts as this is a great time to effect those releases.
>
> Thanks
> Joe
>


--
Ryan Blue
Software Engineer
Cloudera, Inc.
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Sean Busbey
In reply to this post by Joe Witt
On Fri, May 15, 2015 at 11:26 AM, Joe Witt <[hidden email]> wrote:

> Sergio
>
> Am really happy you think so.  Was just waiting to get through the release
> before having that discussion.
>
> Do others feel that we are at the point to start that as well?
>
>
Can we get a current accounting for how the community has grown thus far in
incubation?

I think we have the release cadence bit mostly down.


--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Joe Witt
Sean,

Mailing Lists Subscribers:
  Our first status report:
  http://wiki.apache.org/incubator/December2014

  Our most recent status report:
  http://wiki.apache.org/incubator/May2015

  Current counts on mailing lists (excluding the archive addresses)
  - Dev: 122
  - Users: 30
  - Commits: 30

Mailing List Activity
  Users is a few weeks old and usage is limited thus far.  Dev list
has had a couple to a few hundred exchanges each month.

Github (as a measure of code and documentation contribution):
  - Shows 27 individuals have contributed code or documentation.
  - 13 of them are repeat contributors.
  - 18 have contributed in the past month.
  - 9 PRs in the past month
  - 55 PRs in the past 5 months.  37 were merged.  The difference of
those numbers were the result of reviews/discussion.

JIRA
  - More than 190 of the JIRA tickets are from people outside the PPMC
which is nearly 1/3 of all tickets.
  - Several of these include contributions as attached patches.
Represents people adding new features.  Fixing coding defects.  Adding
or correcting documentation.

PPMC Growth:
  - We've added two new members since we joined the Incubator in November, 2014.
  - There are community members providing the type of contribution to
work toward PPMC status.  The pipeline is strong and highly
encouraging.

Committer Independence:
  - The PPMC membership includes people from a variety of companies
with independent influence.  The PPMC membership is employed by at
least 11 different organizations.  This includes Hortonworks,
Streamsets, Redlink, Basis Tech, Booze Allen Hamilton, Salesforce,
Onyara, Requitest, Twitter, iJET, US Gov.  If you look at the
contributor base beyond the PPMC the independence is even more
encouraging.  Given nifi's origins that independence has been
established outside the defense space so quickly is fantastic.

In short: The community is growing and we continue to incrementally
improve and do the things which help community growth occur.

Thanks
Joe


On Fri, May 15, 2015 at 4:26 PM, Sean Busbey <[hidden email]> wrote:

> On Fri, May 15, 2015 at 11:26 AM, Joe Witt <[hidden email]> wrote:
>
>> Sergio
>>
>> Am really happy you think so.  Was just waiting to get through the release
>> before having that discussion.
>>
>> Do others feel that we are at the point to start that as well?
>>
>>
> Can we get a current accounting for how the community has grown thus far in
> incubation?
>
> I think we have the release cadence bit mostly down.
>
>
> --
> Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Sean Busbey
> Given nifi's origins that independence has been
> established outside the defense space so quickly is fantastic.

This is my main concern. If the contributor pipeline is showing strong
representation outside of the defense industry and the MD/DC/VA area, then
I'd say +1.

On Fri, May 15, 2015 at 10:30 PM, Joe Witt <[hidden email]> wrote:

> Sean,
>
> Mailing Lists Subscribers:
>   Our first status report:
>   http://wiki.apache.org/incubator/December2014
>
>   Our most recent status report:
>   http://wiki.apache.org/incubator/May2015
>
>   Current counts on mailing lists (excluding the archive addresses)
>   - Dev: 122
>   - Users: 30
>   - Commits: 30
>
> Mailing List Activity
>   Users is a few weeks old and usage is limited thus far.  Dev list
> has had a couple to a few hundred exchanges each month.
>
> Github (as a measure of code and documentation contribution):
>   - Shows 27 individuals have contributed code or documentation.
>   - 13 of them are repeat contributors.
>   - 18 have contributed in the past month.
>   - 9 PRs in the past month
>   - 55 PRs in the past 5 months.  37 were merged.  The difference of
> those numbers were the result of reviews/discussion.
>
> JIRA
>   - More than 190 of the JIRA tickets are from people outside the PPMC
> which is nearly 1/3 of all tickets.
>   - Several of these include contributions as attached patches.
> Represents people adding new features.  Fixing coding defects.  Adding
> or correcting documentation.
>
> PPMC Growth:
>   - We've added two new members since we joined the Incubator in November,
> 2014.
>   - There are community members providing the type of contribution to
> work toward PPMC status.  The pipeline is strong and highly
> encouraging.
>
> Committer Independence:
>   - The PPMC membership includes people from a variety of companies
> with independent influence.  The PPMC membership is employed by at
> least 11 different organizations.  This includes Hortonworks,
> Streamsets, Redlink, Basis Tech, Booze Allen Hamilton, Salesforce,
> Onyara, Requitest, Twitter, iJET, US Gov.  If you look at the
> contributor base beyond the PPMC the independence is even more
> encouraging.  Given nifi's origins that independence has been
> established outside the defense space so quickly is fantastic.
>
> In short: The community is growing and we continue to incrementally
> improve and do the things which help community growth occur.
>
> Thanks
> Joe
>
>
> On Fri, May 15, 2015 at 4:26 PM, Sean Busbey <[hidden email]> wrote:
> > On Fri, May 15, 2015 at 11:26 AM, Joe Witt <[hidden email]> wrote:
> >
> >> Sergio
> >>
> >> Am really happy you think so.  Was just waiting to get through the
> release
> >> before having that discussion.
> >>
> >> Do others feel that we are at the point to start that as well?
> >>
> >>
> > Can we get a current accounting for how the community has grown thus far
> in
> > incubation?
> >
> > I think we have the release cadence bit mostly down.
> >
> >
> > --
> > Sean
>



--
Sean
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Mike Drob-2
In reply to this post by Joe Witt
On Fri, May 15, 2015 at 10:30 PM, Joe Witt <[hidden email]> wrote:

>
> Committer Independence:
>   - The PPMC membership includes people from a variety of companies
> with independent influence.  The PPMC membership is employed by at
> least 11 different organizations.  This includes Hortonworks,
> Streamsets, Redlink, Basis Tech, Booze Allen Hamilton, Salesforce,
> Onyara, Requitest, Twitter, iJET, US Gov.  If you look at the
> contributor base beyond the PPMC the independence is even more
> encouraging.  Given nifi's origins that independence has been
> established outside the defense space so quickly is fantastic.
>
> In short: The community is growing and we continue to incrementally
> improve and do the things which help community growth occur.
>
> What do these numbers look like when weighted by which companies are
contributing code? It's fine to say that you have representation from X
companies, but if all the work is really done by a single organization then
I don't think we can count it as committer independence. I've seen in the
past that mentors tend to lose interest after graduation as they move on to
newer podlings that need their attention when they are not also
contributors.

Maybe this deserves its own thread?


> Thanks
> Joe
>
>
> On Fri, May 15, 2015 at 4:26 PM, Sean Busbey <[hidden email]> wrote:
> > On Fri, May 15, 2015 at 11:26 AM, Joe Witt <[hidden email]> wrote:
> >
> >> Sergio
> >>
> >> Am really happy you think so.  Was just waiting to get through the
> release
> >> before having that discussion.
> >>
> >> Do others feel that we are at the point to start that as well?
> >>
> >>
> > Can we get a current accounting for how the community has grown thus far
> in
> > incubation?
> >
> > I think we have the release cadence bit mostly down.
> >
> >
> > --
> > Sean
>

Mike
Reply | Threaded
Open this post in threaded view
|

Re: Planning for the next release

Joe Witt
> What do these numbers look like when weighted by which companies are
contributing code?

A new thread will start soon.  If you remove mentors and only consider ppmc
we have recent commits from onyara, us gov, requitest, and ijet.  And that
is assuming the employer is relevant to that individuals contribution.

Thanks
Joe
On May 18, 2015 3:14 PM, "Mike Drob" <[hidden email]> wrote:

> On Fri, May 15, 2015 at 10:30 PM, Joe Witt <[hidden email]> wrote:
>
> >
> > Committer Independence:
> >   - The PPMC membership includes people from a variety of companies
> > with independent influence.  The PPMC membership is employed by at
> > least 11 different organizations.  This includes Hortonworks,
> > Streamsets, Redlink, Basis Tech, Booze Allen Hamilton, Salesforce,
> > Onyara, Requitest, Twitter, iJET, US Gov.  If you look at the
> > contributor base beyond the PPMC the independence is even more
> > encouraging.  Given nifi's origins that independence has been
> > established outside the defense space so quickly is fantastic.
> >
> > In short: The community is growing and we continue to incrementally
> > improve and do the things which help community growth occur.
> >
> > What do these numbers look like when weighted by which companies are
> contributing code? It's fine to say that you have representation from X
> companies, but if all the work is really done by a single organization then
> I don't think we can count it as committer independence. I've seen in the
> past that mentors tend to lose interest after graduation as they move on to
> newer podlings that need their attention when they are not also
> contributors.
>
> Maybe this deserves its own thread?
>
>
> > Thanks
> > Joe
> >
> >
> > On Fri, May 15, 2015 at 4:26 PM, Sean Busbey <[hidden email]>
> wrote:
> > > On Fri, May 15, 2015 at 11:26 AM, Joe Witt <[hidden email]> wrote:
> > >
> > >> Sergio
> > >>
> > >> Am really happy you think so.  Was just waiting to get through the
> > release
> > >> before having that discussion.
> > >>
> > >> Do others feel that we are at the point to start that as well?
> > >>
> > >>
> > > Can we get a current accounting for how the community has grown thus
> far
> > in
> > > incubation?
> > >
> > > I think we have the release cadence bit mostly down.
> > >
> > >
> > > --
> > > Sean
> >
>
> Mike
>