Lowering the barrier of entry

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Lowering the barrier of entry

Andy LoPresto-2
Hi folks,

Based on some recent (and long-term) experiences, I wanted to discuss with the community what we could do to lower the barrier of entry to using & contributing to NiFi. I hope to get some good feedback from both long-time and newer members, and determine some immediate concrete steps we can take.

Problems identified:
* NiFi has a number of custom profiles, so a simple “mvn clean install” in project root doesn’t get a new developer up and running immediately
* The API is very well defined, but for new contributors, it can be a challenge to know where to put functionality, and building a custom processor + NAR and deploying isn’t a one-step process
* Project size (and build size/time) is large. This can restrict the minimum hardware necessary, elongate the development cycle, etc.
* Some new users do not receive mailing list replies

Possible solutions:
* On a clean git clone, “mvn clean install” should build a working instance. Maybe we provide a quickstart.sh script to handle the default maven build, change to the target directory, and start NiFi?
* Individual contributors have written excellent blogs, and documentation exists, but making it more prominent or more easily accessed could help?
* Extension registry will solve all the world’s problems (related to bundling and build time)
* Not sure about this one — I don’t know if it’s because they’re not subscribed, their mail client is blocking them, etc.

I’ve said my bit, now I am eager to hear from other community members on their experiences, steps that helped them, and suggestions for the future to continue to make the NiFi community welcoming to new users. Thanks.


Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Bryan Bende
I think we have a lot more documentation than most projects, but I
think an issue is that content is scattered in many different
locations, and some of the docs are huge reference guides where it can
be hard to find all the pieces of what you are trying to do.

The first thing a new contributor wants to do is get the code and run
a build, and we do have a quick-start guide linked to on the site, but
I think there is a lot of extra information in there that is not
really relevant to someone just wanting get the code and build. We
could have separate guides per OS like "Build NiFi on Linux", "Build
NiFi on Windows", etc, where each guide was 4-5 steps like:

- Clone repo
- checkout master
- run maven
- cd to assembly
- ./bin/nifi.sh

The next thing they want to do is contribute a change, and we have a
great contributor guide, but again I think there could be a very short
tutorial for the most common steps:

- fork repo
- clone fork
- create branch
- make changes
- push branch
- submit pr

and then say something like "for a more detailed description of the
contribution process, please reference the Contributor Guide".

If we then make these getting started guides more prominent right in
the middle of the NiFi homepage, then maybe they will be easier to
find for new community members.

We can keep extending this idea to other common tasks beyond just
building and contributing.


On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]> wrote:

>
> Hi folks,
>
> Based on some recent (and long-term) experiences, I wanted to discuss with the community what we could do to lower the barrier of entry to using & contributing to NiFi. I hope to get some good feedback from both long-time and newer members, and determine some immediate concrete steps we can take.
>
> Problems identified:
> * NiFi has a number of custom profiles, so a simple “mvn clean install” in project root doesn’t get a new developer up and running immediately
> * The API is very well defined, but for new contributors, it can be a challenge to know where to put functionality, and building a custom processor + NAR and deploying isn’t a one-step process
> * Project size (and build size/time) is large. This can restrict the minimum hardware necessary, elongate the development cycle, etc.
> * Some new users do not receive mailing list replies
>
> Possible solutions:
> * On a clean git clone, “mvn clean install” should build a working instance. Maybe we provide a quickstart.sh script to handle the default maven build, change to the target directory, and start NiFi?
> * Individual contributors have written excellent blogs, and documentation exists, but making it more prominent or more easily accessed could help?
> * Extension registry will solve all the world’s problems (related to bundling and build time)
> * Not sure about this one — I don’t know if it’s because they’re not subscribed, their mail client is blocking them, etc.
>
> I’ve said my bit, now I am eager to hear from other community members on their experiences, steps that helped them, and suggestions for the future to continue to make the NiFi community welcoming to new users. Thanks.
>
>
> Andy LoPresto
> [hidden email]
> [hidden email]
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Mike Thomsen
One of the changes we should make is to create a separate guide for product
vendors on how to build and maintain a bundle. We're at that point where
vendors will have to do it on their own as extension providers, so it would
be very helpful for them to have a simple and straight forward document
showing them what should be there, best practices for maintainability and
where to announce it.

On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:

> I think we have a lot more documentation than most projects, but I
> think an issue is that content is scattered in many different
> locations, and some of the docs are huge reference guides where it can
> be hard to find all the pieces of what you are trying to do.
>
> The first thing a new contributor wants to do is get the code and run
> a build, and we do have a quick-start guide linked to on the site, but
> I think there is a lot of extra information in there that is not
> really relevant to someone just wanting get the code and build. We
> could have separate guides per OS like "Build NiFi on Linux", "Build
> NiFi on Windows", etc, where each guide was 4-5 steps like:
>
> - Clone repo
> - checkout master
> - run maven
> - cd to assembly
> - ./bin/nifi.sh
>
> The next thing they want to do is contribute a change, and we have a
> great contributor guide, but again I think there could be a very short
> tutorial for the most common steps:
>
> - fork repo
> - clone fork
> - create branch
> - make changes
> - push branch
> - submit pr
>
> and then say something like "for a more detailed description of the
> contribution process, please reference the Contributor Guide".
>
> If we then make these getting started guides more prominent right in
> the middle of the NiFi homepage, then maybe they will be easier to
> find for new community members.
>
> We can keep extending this idea to other common tasks beyond just
> building and contributing.
>
>
> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> wrote:
> >
> > Hi folks,
> >
> > Based on some recent (and long-term) experiences, I wanted to discuss
> with the community what we could do to lower the barrier of entry to using
> & contributing to NiFi. I hope to get some good feedback from both
> long-time and newer members, and determine some immediate concrete steps we
> can take.
> >
> > Problems identified:
> > * NiFi has a number of custom profiles, so a simple “mvn clean install”
> in project root doesn’t get a new developer up and running immediately
> > * The API is very well defined, but for new contributors, it can be a
> challenge to know where to put functionality, and building a custom
> processor + NAR and deploying isn’t a one-step process
> > * Project size (and build size/time) is large. This can restrict the
> minimum hardware necessary, elongate the development cycle, etc.
> > * Some new users do not receive mailing list replies
> >
> > Possible solutions:
> > * On a clean git clone, “mvn clean install” should build a working
> instance. Maybe we provide a quickstart.sh script to handle the default
> maven build, change to the target directory, and start NiFi?
> > * Individual contributors have written excellent blogs, and
> documentation exists, but making it more prominent or more easily accessed
> could help?
> > * Extension registry will solve all the world’s problems (related to
> bundling and build time)
> > * Not sure about this one — I don’t know if it’s because they’re not
> subscribed, their mail client is blocking them, etc.
> >
> > I’ve said my bit, now I am eager to hear from other community members on
> their experiences, steps that helped them, and suggestions for the future
> to continue to make the NiFi community welcoming to new users. Thanks.
> >
> >
> > Andy LoPresto
> > [hidden email]
> > [hidden email]
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Andrew Grande-2
I consistently see my users struggling when they move up the nifi food
chain and start looking at custom processors. The good content about
prototyping processsors via scripting processors and finalizing with a full
NAR bundle is everywhere but where it should be.

A few simple changes could help (not *more* docs). They are great, much
better than in many other projecta, but people are already drowning in
those.

How about:

+ ISP has a pre-populated processor sceleton. A simple no-op to fill in is
miles better than a blank text area (which invokes a blank stare).

+ As much as we may loook down on this, but... A simple guide to a full NAR
build as a series of copy/paste commands.

There's more, but this should fit the context for now.

Andrew

On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]> wrote:

> One of the changes we should make is to create a separate guide for product
> vendors on how to build and maintain a bundle. We're at that point where
> vendors will have to do it on their own as extension providers, so it would
> be very helpful for them to have a simple and straight forward document
> showing them what should be there, best practices for maintainability and
> where to announce it.
>
> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
>
> > I think we have a lot more documentation than most projects, but I
> > think an issue is that content is scattered in many different
> > locations, and some of the docs are huge reference guides where it can
> > be hard to find all the pieces of what you are trying to do.
> >
> > The first thing a new contributor wants to do is get the code and run
> > a build, and we do have a quick-start guide linked to on the site, but
> > I think there is a lot of extra information in there that is not
> > really relevant to someone just wanting get the code and build. We
> > could have separate guides per OS like "Build NiFi on Linux", "Build
> > NiFi on Windows", etc, where each guide was 4-5 steps like:
> >
> > - Clone repo
> > - checkout master
> > - run maven
> > - cd to assembly
> > - ./bin/nifi.sh
> >
> > The next thing they want to do is contribute a change, and we have a
> > great contributor guide, but again I think there could be a very short
> > tutorial for the most common steps:
> >
> > - fork repo
> > - clone fork
> > - create branch
> > - make changes
> > - push branch
> > - submit pr
> >
> > and then say something like "for a more detailed description of the
> > contribution process, please reference the Contributor Guide".
> >
> > If we then make these getting started guides more prominent right in
> > the middle of the NiFi homepage, then maybe they will be easier to
> > find for new community members.
> >
> > We can keep extending this idea to other common tasks beyond just
> > building and contributing.
> >
> >
> > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > wrote:
> > >
> > > Hi folks,
> > >
> > > Based on some recent (and long-term) experiences, I wanted to discuss
> > with the community what we could do to lower the barrier of entry to
> using
> > & contributing to NiFi. I hope to get some good feedback from both
> > long-time and newer members, and determine some immediate concrete steps
> we
> > can take.
> > >
> > > Problems identified:
> > > * NiFi has a number of custom profiles, so a simple “mvn clean install”
> > in project root doesn’t get a new developer up and running immediately
> > > * The API is very well defined, but for new contributors, it can be a
> > challenge to know where to put functionality, and building a custom
> > processor + NAR and deploying isn’t a one-step process
> > > * Project size (and build size/time) is large. This can restrict the
> > minimum hardware necessary, elongate the development cycle, etc.
> > > * Some new users do not receive mailing list replies
> > >
> > > Possible solutions:
> > > * On a clean git clone, “mvn clean install” should build a working
> > instance. Maybe we provide a quickstart.sh script to handle the default
> > maven build, change to the target directory, and start NiFi?
> > > * Individual contributors have written excellent blogs, and
> > documentation exists, but making it more prominent or more easily
> accessed
> > could help?
> > > * Extension registry will solve all the world’s problems (related to
> > bundling and build time)
> > > * Not sure about this one — I don’t know if it’s because they’re not
> > subscribed, their mail client is blocking them, etc.
> > >
> > > I’ve said my bit, now I am eager to hear from other community members
> on
> > their experiences, steps that helped them, and suggestions for the future
> > to continue to make the NiFi community welcoming to new users. Thanks.
> > >
> > >
> > > Andy LoPresto
> > > [hidden email]
> > > [hidden email]
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Mike Thomsen
I should be able to find some time to set up some skeletons based on code
that I've written. Would storing them in the docs folder work, so we can do
PRs?

On Fri, Jan 25, 2019 at 11:50 AM Andrew Grande <[hidden email]> wrote:

> I consistently see my users struggling when they move up the nifi food
> chain and start looking at custom processors. The good content about
> prototyping processsors via scripting processors and finalizing with a full
> NAR bundle is everywhere but where it should be.
>
> A few simple changes could help (not *more* docs). They are great, much
> better than in many other projecta, but people are already drowning in
> those.
>
> How about:
>
> + ISP has a pre-populated processor sceleton. A simple no-op to fill in is
> miles better than a blank text area (which invokes a blank stare).
>
> + As much as we may loook down on this, but... A simple guide to a full NAR
> build as a series of copy/paste commands.
>
> There's more, but this should fit the context for now.
>
> Andrew
>
> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]> wrote:
>
> > One of the changes we should make is to create a separate guide for
> product
> > vendors on how to build and maintain a bundle. We're at that point where
> > vendors will have to do it on their own as extension providers, so it
> would
> > be very helpful for them to have a simple and straight forward document
> > showing them what should be there, best practices for maintainability and
> > where to announce it.
> >
> > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
> >
> > > I think we have a lot more documentation than most projects, but I
> > > think an issue is that content is scattered in many different
> > > locations, and some of the docs are huge reference guides where it can
> > > be hard to find all the pieces of what you are trying to do.
> > >
> > > The first thing a new contributor wants to do is get the code and run
> > > a build, and we do have a quick-start guide linked to on the site, but
> > > I think there is a lot of extra information in there that is not
> > > really relevant to someone just wanting get the code and build. We
> > > could have separate guides per OS like "Build NiFi on Linux", "Build
> > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > >
> > > - Clone repo
> > > - checkout master
> > > - run maven
> > > - cd to assembly
> > > - ./bin/nifi.sh
> > >
> > > The next thing they want to do is contribute a change, and we have a
> > > great contributor guide, but again I think there could be a very short
> > > tutorial for the most common steps:
> > >
> > > - fork repo
> > > - clone fork
> > > - create branch
> > > - make changes
> > > - push branch
> > > - submit pr
> > >
> > > and then say something like "for a more detailed description of the
> > > contribution process, please reference the Contributor Guide".
> > >
> > > If we then make these getting started guides more prominent right in
> > > the middle of the NiFi homepage, then maybe they will be easier to
> > > find for new community members.
> > >
> > > We can keep extending this idea to other common tasks beyond just
> > > building and contributing.
> > >
> > >
> > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > > wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > Based on some recent (and long-term) experiences, I wanted to discuss
> > > with the community what we could do to lower the barrier of entry to
> > using
> > > & contributing to NiFi. I hope to get some good feedback from both
> > > long-time and newer members, and determine some immediate concrete
> steps
> > we
> > > can take.
> > > >
> > > > Problems identified:
> > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> install”
> > > in project root doesn’t get a new developer up and running immediately
> > > > * The API is very well defined, but for new contributors, it can be a
> > > challenge to know where to put functionality, and building a custom
> > > processor + NAR and deploying isn’t a one-step process
> > > > * Project size (and build size/time) is large. This can restrict the
> > > minimum hardware necessary, elongate the development cycle, etc.
> > > > * Some new users do not receive mailing list replies
> > > >
> > > > Possible solutions:
> > > > * On a clean git clone, “mvn clean install” should build a working
> > > instance. Maybe we provide a quickstart.sh script to handle the default
> > > maven build, change to the target directory, and start NiFi?
> > > > * Individual contributors have written excellent blogs, and
> > > documentation exists, but making it more prominent or more easily
> > accessed
> > > could help?
> > > > * Extension registry will solve all the world’s problems (related to
> > > bundling and build time)
> > > > * Not sure about this one — I don’t know if it’s because they’re not
> > > subscribed, their mail client is blocking them, etc.
> > > >
> > > > I’ve said my bit, now I am eager to hear from other community members
> > on
> > > their experiences, steps that helped them, and suggestions for the
> future
> > > to continue to make the NiFi community welcoming to new users. Thanks.
> > > >
> > > >
> > > > Andy LoPresto
> > > > [hidden email]
> > > > [hidden email]
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Otto Fowler
In reply to this post by Andrew Grande-2
I think this ties into my other discuss thread on refreshing the archetypes


On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email]) wrote:

I consistently see my users struggling when they move up the nifi food
chain and start looking at custom processors. The good content about
prototyping processsors via scripting processors and finalizing with a full
NAR bundle is everywhere but where it should be.

A few simple changes could help (not *more* docs). They are great, much
better than in many other projecta, but people are already drowning in
those.

How about:

+ ISP has a pre-populated processor sceleton. A simple no-op to fill in is
miles better than a blank text area (which invokes a blank stare).

+ As much as we may loook down on this, but... A simple guide to a full NAR
build as a series of copy/paste commands.

There's more, but this should fit the context for now.

Andrew

On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]> wrote:

> One of the changes we should make is to create a separate guide for
product
> vendors on how to build and maintain a bundle. We're at that point where
> vendors will have to do it on their own as extension providers, so it
would

> be very helpful for them to have a simple and straight forward document
> showing them what should be there, best practices for maintainability and
> where to announce it.
>
> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
>
> > I think we have a lot more documentation than most projects, but I
> > think an issue is that content is scattered in many different
> > locations, and some of the docs are huge reference guides where it can
> > be hard to find all the pieces of what you are trying to do.
> >
> > The first thing a new contributor wants to do is get the code and run
> > a build, and we do have a quick-start guide linked to on the site, but
> > I think there is a lot of extra information in there that is not
> > really relevant to someone just wanting get the code and build. We
> > could have separate guides per OS like "Build NiFi on Linux", "Build
> > NiFi on Windows", etc, where each guide was 4-5 steps like:
> >
> > - Clone repo
> > - checkout master
> > - run maven
> > - cd to assembly
> > - ./bin/nifi.sh
> >
> > The next thing they want to do is contribute a change, and we have a
> > great contributor guide, but again I think there could be a very short
> > tutorial for the most common steps:
> >
> > - fork repo
> > - clone fork
> > - create branch
> > - make changes
> > - push branch
> > - submit pr
> >
> > and then say something like "for a more detailed description of the
> > contribution process, please reference the Contributor Guide".
> >
> > If we then make these getting started guides more prominent right in
> > the middle of the NiFi homepage, then maybe they will be easier to
> > find for new community members.
> >
> > We can keep extending this idea to other common tasks beyond just
> > building and contributing.
> >
> >
> > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > wrote:
> > >
> > > Hi folks,
> > >
> > > Based on some recent (and long-term) experiences, I wanted to discuss
> > with the community what we could do to lower the barrier of entry to
> using
> > & contributing to NiFi. I hope to get some good feedback from both
> > long-time and newer members, and determine some immediate concrete
steps
> we
> > can take.
> > >
> > > Problems identified:
> > > * NiFi has a number of custom profiles, so a simple “mvn clean
install”

> > in project root doesn’t get a new developer up and running immediately
> > > * The API is very well defined, but for new contributors, it can be a
> > challenge to know where to put functionality, and building a custom
> > processor + NAR and deploying isn’t a one-step process
> > > * Project size (and build size/time) is large. This can restrict the
> > minimum hardware necessary, elongate the development cycle, etc.
> > > * Some new users do not receive mailing list replies
> > >
> > > Possible solutions:
> > > * On a clean git clone, “mvn clean install” should build a working
> > instance. Maybe we provide a quickstart.sh script to handle the default
> > maven build, change to the target directory, and start NiFi?
> > > * Individual contributors have written excellent blogs, and
> > documentation exists, but making it more prominent or more easily
> accessed
> > could help?
> > > * Extension registry will solve all the world’s problems (related to
> > bundling and build time)
> > > * Not sure about this one — I don’t know if it’s because they’re not
> > subscribed, their mail client is blocking them, etc.
> > >
> > > I’ve said my bit, now I am eager to hear from other community members
> on
> > their experiences, steps that helped them, and suggestions for the
future

> > to continue to make the NiFi community welcoming to new users. Thanks.
> > >
> > >
> > > Andy LoPresto
> > > [hidden email]
> > > [hidden email]
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Bryan Bende
Andrew,

I'm not disagreeing with your points, but I'm curious how you see
those two ideas being different from the processor archetype and the
wiki page with the archetype commands?

Is it just that people don't know about it?

-Bryan

[1] https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions

On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]> wrote:

>
> I think this ties into my other discuss thread on refreshing the archetypes
>
>
> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email]) wrote:
>
> I consistently see my users struggling when they move up the nifi food
> chain and start looking at custom processors. The good content about
> prototyping processsors via scripting processors and finalizing with a full
> NAR bundle is everywhere but where it should be.
>
> A few simple changes could help (not *more* docs). They are great, much
> better than in many other projecta, but people are already drowning in
> those.
>
> How about:
>
> + ISP has a pre-populated processor sceleton. A simple no-op to fill in is
> miles better than a blank text area (which invokes a blank stare).
>
> + As much as we may loook down on this, but... A simple guide to a full NAR
> build as a series of copy/paste commands.
>
> There's more, but this should fit the context for now.
>
> Andrew
>
> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]> wrote:
>
> > One of the changes we should make is to create a separate guide for
> product
> > vendors on how to build and maintain a bundle. We're at that point where
> > vendors will have to do it on their own as extension providers, so it
> would
> > be very helpful for them to have a simple and straight forward document
> > showing them what should be there, best practices for maintainability and
> > where to announce it.
> >
> > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
> >
> > > I think we have a lot more documentation than most projects, but I
> > > think an issue is that content is scattered in many different
> > > locations, and some of the docs are huge reference guides where it can
> > > be hard to find all the pieces of what you are trying to do.
> > >
> > > The first thing a new contributor wants to do is get the code and run
> > > a build, and we do have a quick-start guide linked to on the site, but
> > > I think there is a lot of extra information in there that is not
> > > really relevant to someone just wanting get the code and build. We
> > > could have separate guides per OS like "Build NiFi on Linux", "Build
> > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > >
> > > - Clone repo
> > > - checkout master
> > > - run maven
> > > - cd to assembly
> > > - ./bin/nifi.sh
> > >
> > > The next thing they want to do is contribute a change, and we have a
> > > great contributor guide, but again I think there could be a very short
> > > tutorial for the most common steps:
> > >
> > > - fork repo
> > > - clone fork
> > > - create branch
> > > - make changes
> > > - push branch
> > > - submit pr
> > >
> > > and then say something like "for a more detailed description of the
> > > contribution process, please reference the Contributor Guide".
> > >
> > > If we then make these getting started guides more prominent right in
> > > the middle of the NiFi homepage, then maybe they will be easier to
> > > find for new community members.
> > >
> > > We can keep extending this idea to other common tasks beyond just
> > > building and contributing.
> > >
> > >
> > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > > wrote:
> > > >
> > > > Hi folks,
> > > >
> > > > Based on some recent (and long-term) experiences, I wanted to discuss
> > > with the community what we could do to lower the barrier of entry to
> > using
> > > & contributing to NiFi. I hope to get some good feedback from both
> > > long-time and newer members, and determine some immediate concrete
> steps
> > we
> > > can take.
> > > >
> > > > Problems identified:
> > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> install”
> > > in project root doesn’t get a new developer up and running immediately
> > > > * The API is very well defined, but for new contributors, it can be a
> > > challenge to know where to put functionality, and building a custom
> > > processor + NAR and deploying isn’t a one-step process
> > > > * Project size (and build size/time) is large. This can restrict the
> > > minimum hardware necessary, elongate the development cycle, etc.
> > > > * Some new users do not receive mailing list replies
> > > >
> > > > Possible solutions:
> > > > * On a clean git clone, “mvn clean install” should build a working
> > > instance. Maybe we provide a quickstart.sh script to handle the default
> > > maven build, change to the target directory, and start NiFi?
> > > > * Individual contributors have written excellent blogs, and
> > > documentation exists, but making it more prominent or more easily
> > accessed
> > > could help?
> > > > * Extension registry will solve all the world’s problems (related to
> > > bundling and build time)
> > > > * Not sure about this one — I don’t know if it’s because they’re not
> > > subscribed, their mail client is blocking them, etc.
> > > >
> > > > I’ve said my bit, now I am eager to hear from other community members
> > on
> > > their experiences, steps that helped them, and suggestions for the
> future
> > > to continue to make the NiFi community welcoming to new users. Thanks.
> > > >
> > > >
> > > > Andy LoPresto
> > > > [hidden email]
> > > > [hidden email]
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Andrew Grande-2
We assume they create new projects from archetypes every day. They don't.

We also assume they know how to deploy new NARs. Most don't. Especially if
we want them to follow best practices and create an additional NAR bundles
directory entry im the config (vs dumping into nifi lib).

I can attest that I feel a bit lost myself every time I need to come back
to this and refresh my brain synapses. If we could make these not require
any of that and make simple thinga dead simple....

Andrew

On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:

> Andrew,
>
> I'm not disagreeing with your points, but I'm curious how you see
> those two ideas being different from the processor archetype and the
> wiki page with the archetype commands?
>
> Is it just that people don't know about it?
>
> -Bryan
>
> [1]
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> wrote:
> >
> > I think this ties into my other discuss thread on refreshing the
> archetypes
> >
> >
> > On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> wrote:
> >
> > I consistently see my users struggling when they move up the nifi food
> > chain and start looking at custom processors. The good content about
> > prototyping processsors via scripting processors and finalizing with a
> full
> > NAR bundle is everywhere but where it should be.
> >
> > A few simple changes could help (not *more* docs). They are great, much
> > better than in many other projecta, but people are already drowning in
> > those.
> >
> > How about:
> >
> > + ISP has a pre-populated processor sceleton. A simple no-op to fill in
> is
> > miles better than a blank text area (which invokes a blank stare).
> >
> > + As much as we may loook down on this, but... A simple guide to a full
> NAR
> > build as a series of copy/paste commands.
> >
> > There's more, but this should fit the context for now.
> >
> > Andrew
> >
> > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> wrote:
> >
> > > One of the changes we should make is to create a separate guide for
> > product
> > > vendors on how to build and maintain a bundle. We're at that point
> where
> > > vendors will have to do it on their own as extension providers, so it
> > would
> > > be very helpful for them to have a simple and straight forward document
> > > showing them what should be there, best practices for maintainability
> and
> > > where to announce it.
> > >
> > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
> > >
> > > > I think we have a lot more documentation than most projects, but I
> > > > think an issue is that content is scattered in many different
> > > > locations, and some of the docs are huge reference guides where it
> can
> > > > be hard to find all the pieces of what you are trying to do.
> > > >
> > > > The first thing a new contributor wants to do is get the code and run
> > > > a build, and we do have a quick-start guide linked to on the site,
> but
> > > > I think there is a lot of extra information in there that is not
> > > > really relevant to someone just wanting get the code and build. We
> > > > could have separate guides per OS like "Build NiFi on Linux", "Build
> > > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > >
> > > > - Clone repo
> > > > - checkout master
> > > > - run maven
> > > > - cd to assembly
> > > > - ./bin/nifi.sh
> > > >
> > > > The next thing they want to do is contribute a change, and we have a
> > > > great contributor guide, but again I think there could be a very
> short
> > > > tutorial for the most common steps:
> > > >
> > > > - fork repo
> > > > - clone fork
> > > > - create branch
> > > > - make changes
> > > > - push branch
> > > > - submit pr
> > > >
> > > > and then say something like "for a more detailed description of the
> > > > contribution process, please reference the Contributor Guide".
> > > >
> > > > If we then make these getting started guides more prominent right in
> > > > the middle of the NiFi homepage, then maybe they will be easier to
> > > > find for new community members.
> > > >
> > > > We can keep extending this idea to other common tasks beyond just
> > > > building and contributing.
> > > >
> > > >
> > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > > > wrote:
> > > > >
> > > > > Hi folks,
> > > > >
> > > > > Based on some recent (and long-term) experiences, I wanted to
> discuss
> > > > with the community what we could do to lower the barrier of entry to
> > > using
> > > > & contributing to NiFi. I hope to get some good feedback from both
> > > > long-time and newer members, and determine some immediate concrete
> > steps
> > > we
> > > > can take.
> > > > >
> > > > > Problems identified:
> > > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> > install”
> > > > in project root doesn’t get a new developer up and running
> immediately
> > > > > * The API is very well defined, but for new contributors, it can
> be a
> > > > challenge to know where to put functionality, and building a custom
> > > > processor + NAR and deploying isn’t a one-step process
> > > > > * Project size (and build size/time) is large. This can restrict
> the
> > > > minimum hardware necessary, elongate the development cycle, etc.
> > > > > * Some new users do not receive mailing list replies
> > > > >
> > > > > Possible solutions:
> > > > > * On a clean git clone, “mvn clean install” should build a working
> > > > instance. Maybe we provide a quickstart.sh script to handle the
> default
> > > > maven build, change to the target directory, and start NiFi?
> > > > > * Individual contributors have written excellent blogs, and
> > > > documentation exists, but making it more prominent or more easily
> > > accessed
> > > > could help?
> > > > > * Extension registry will solve all the world’s problems (related
> to
> > > > bundling and build time)
> > > > > * Not sure about this one — I don’t know if it’s because they’re
> not
> > > > subscribed, their mail client is blocking them, etc.
> > > > >
> > > > > I’ve said my bit, now I am eager to hear from other community
> members
> > > on
> > > > their experiences, steps that helped them, and suggestions for the
> > future
> > > > to continue to make the NiFi community welcoming to new users.
> Thanks.
> > > > >
> > > > >
> > > > > Andy LoPresto
> > > > > [hidden email]
> > > > > [hidden email]
> > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > >
> > > >
> > >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Bryan Bende
That makes sense about the best practice for deploying to an
additional lib directory.

So for the project structure you are saying it would be easier to have
a repo somewhere with essentially the same thing that is in the
archetype, but they just clone it and rename it themselves (what the
archetype does for you)?

Something that I think would be awesome is if we could provide a
web-based project initializer that would essentially run the archetype
behind the scenes and then let you download the archive of the code,
just like the spring-boot starter [1]. Not sure if their initializr is
something that can be re-used and customized [2].

The problem is we would need to host that somewhere.

[1] https://start.spring.io/
[2] https://github.com/spring-io/initializr

On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:

>
> We assume they create new projects from archetypes every day. They don't.
>
> We also assume they know how to deploy new NARs. Most don't. Especially if
> we want them to follow best practices and create an additional NAR bundles
> directory entry im the config (vs dumping into nifi lib).
>
> I can attest that I feel a bit lost myself every time I need to come back
> to this and refresh my brain synapses. If we could make these not require
> any of that and make simple thinga dead simple....
>
> Andrew
>
> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
>
> > Andrew,
> >
> > I'm not disagreeing with your points, but I'm curious how you see
> > those two ideas being different from the processor archetype and the
> > wiki page with the archetype commands?
> >
> > Is it just that people don't know about it?
> >
> > -Bryan
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> >
> > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> > wrote:
> > >
> > > I think this ties into my other discuss thread on refreshing the
> > archetypes
> > >
> > >
> > > On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> > wrote:
> > >
> > > I consistently see my users struggling when they move up the nifi food
> > > chain and start looking at custom processors. The good content about
> > > prototyping processsors via scripting processors and finalizing with a
> > full
> > > NAR bundle is everywhere but where it should be.
> > >
> > > A few simple changes could help (not *more* docs). They are great, much
> > > better than in many other projecta, but people are already drowning in
> > > those.
> > >
> > > How about:
> > >
> > > + ISP has a pre-populated processor sceleton. A simple no-op to fill in
> > is
> > > miles better than a blank text area (which invokes a blank stare).
> > >
> > > + As much as we may loook down on this, but... A simple guide to a full
> > NAR
> > > build as a series of copy/paste commands.
> > >
> > > There's more, but this should fit the context for now.
> > >
> > > Andrew
> > >
> > > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> > wrote:
> > >
> > > > One of the changes we should make is to create a separate guide for
> > > product
> > > > vendors on how to build and maintain a bundle. We're at that point
> > where
> > > > vendors will have to do it on their own as extension providers, so it
> > > would
> > > > be very helpful for them to have a simple and straight forward document
> > > > showing them what should be there, best practices for maintainability
> > and
> > > > where to announce it.
> > > >
> > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]> wrote:
> > > >
> > > > > I think we have a lot more documentation than most projects, but I
> > > > > think an issue is that content is scattered in many different
> > > > > locations, and some of the docs are huge reference guides where it
> > can
> > > > > be hard to find all the pieces of what you are trying to do.
> > > > >
> > > > > The first thing a new contributor wants to do is get the code and run
> > > > > a build, and we do have a quick-start guide linked to on the site,
> > but
> > > > > I think there is a lot of extra information in there that is not
> > > > > really relevant to someone just wanting get the code and build. We
> > > > > could have separate guides per OS like "Build NiFi on Linux", "Build
> > > > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > > >
> > > > > - Clone repo
> > > > > - checkout master
> > > > > - run maven
> > > > > - cd to assembly
> > > > > - ./bin/nifi.sh
> > > > >
> > > > > The next thing they want to do is contribute a change, and we have a
> > > > > great contributor guide, but again I think there could be a very
> > short
> > > > > tutorial for the most common steps:
> > > > >
> > > > > - fork repo
> > > > > - clone fork
> > > > > - create branch
> > > > > - make changes
> > > > > - push branch
> > > > > - submit pr
> > > > >
> > > > > and then say something like "for a more detailed description of the
> > > > > contribution process, please reference the Contributor Guide".
> > > > >
> > > > > If we then make these getting started guides more prominent right in
> > > > > the middle of the NiFi homepage, then maybe they will be easier to
> > > > > find for new community members.
> > > > >
> > > > > We can keep extending this idea to other common tasks beyond just
> > > > > building and contributing.
> > > > >
> > > > >
> > > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <[hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > Hi folks,
> > > > > >
> > > > > > Based on some recent (and long-term) experiences, I wanted to
> > discuss
> > > > > with the community what we could do to lower the barrier of entry to
> > > > using
> > > > > & contributing to NiFi. I hope to get some good feedback from both
> > > > > long-time and newer members, and determine some immediate concrete
> > > steps
> > > > we
> > > > > can take.
> > > > > >
> > > > > > Problems identified:
> > > > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> > > install”
> > > > > in project root doesn’t get a new developer up and running
> > immediately
> > > > > > * The API is very well defined, but for new contributors, it can
> > be a
> > > > > challenge to know where to put functionality, and building a custom
> > > > > processor + NAR and deploying isn’t a one-step process
> > > > > > * Project size (and build size/time) is large. This can restrict
> > the
> > > > > minimum hardware necessary, elongate the development cycle, etc.
> > > > > > * Some new users do not receive mailing list replies
> > > > > >
> > > > > > Possible solutions:
> > > > > > * On a clean git clone, “mvn clean install” should build a working
> > > > > instance. Maybe we provide a quickstart.sh script to handle the
> > default
> > > > > maven build, change to the target directory, and start NiFi?
> > > > > > * Individual contributors have written excellent blogs, and
> > > > > documentation exists, but making it more prominent or more easily
> > > > accessed
> > > > > could help?
> > > > > > * Extension registry will solve all the world’s problems (related
> > to
> > > > > bundling and build time)
> > > > > > * Not sure about this one — I don’t know if it’s because they’re
> > not
> > > > > subscribed, their mail client is blocking them, etc.
> > > > > >
> > > > > > I’ve said my bit, now I am eager to hear from other community
> > members
> > > > on
> > > > > their experiences, steps that helped them, and suggestions for the
> > > future
> > > > > to continue to make the NiFi community welcoming to new users.
> > Thanks.
> > > > > >
> > > > > >
> > > > > > Andy LoPresto
> > > > > > [hidden email]
> > > > > > [hidden email]
> > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > > > >
> > > > >
> > > >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Andrew Grande-2
I am not against the archetype. But we need to spell out every step of the
way. I'd like to see a user thinking about their custom logic ASAP rather
than fighting the tools to get started. Those steps should be brain-dead,
just reflexes, if you know what I mean. Hell, let them create a custom
processor project or prototype in a script by accident even! :)

On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:

> That makes sense about the best practice for deploying to an
> additional lib directory.
>
> So for the project structure you are saying it would be easier to have
> a repo somewhere with essentially the same thing that is in the
> archetype, but they just clone it and rename it themselves (what the
> archetype does for you)?
>
> Something that I think would be awesome is if we could provide a
> web-based project initializer that would essentially run the archetype
> behind the scenes and then let you download the archive of the code,
> just like the spring-boot starter [1]. Not sure if their initializr is
> something that can be re-used and customized [2].
>
> The problem is we would need to host that somewhere.
>
> [1] https://start.spring.io/
> [2] https://github.com/spring-io/initializr
>
> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:
> >
> > We assume they create new projects from archetypes every day. They don't.
> >
> > We also assume they know how to deploy new NARs. Most don't. Especially
> if
> > we want them to follow best practices and create an additional NAR
> bundles
> > directory entry im the config (vs dumping into nifi lib).
> >
> > I can attest that I feel a bit lost myself every time I need to come back
> > to this and refresh my brain synapses. If we could make these not require
> > any of that and make simple thinga dead simple....
> >
> > Andrew
> >
> > On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> >
> > > Andrew,
> > >
> > > I'm not disagreeing with your points, but I'm curious how you see
> > > those two ideas being different from the processor archetype and the
> > > wiki page with the archetype commands?
> > >
> > > Is it just that people don't know about it?
> > >
> > > -Bryan
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > >
> > > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> > > wrote:
> > > >
> > > > I think this ties into my other discuss thread on refreshing the
> > > archetypes
> > > >
> > > >
> > > > On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> > > wrote:
> > > >
> > > > I consistently see my users struggling when they move up the nifi
> food
> > > > chain and start looking at custom processors. The good content about
> > > > prototyping processsors via scripting processors and finalizing with
> a
> > > full
> > > > NAR bundle is everywhere but where it should be.
> > > >
> > > > A few simple changes could help (not *more* docs). They are great,
> much
> > > > better than in many other projecta, but people are already drowning
> in
> > > > those.
> > > >
> > > > How about:
> > > >
> > > > + ISP has a pre-populated processor sceleton. A simple no-op to fill
> in
> > > is
> > > > miles better than a blank text area (which invokes a blank stare).
> > > >
> > > > + As much as we may loook down on this, but... A simple guide to a
> full
> > > NAR
> > > > build as a series of copy/paste commands.
> > > >
> > > > There's more, but this should fit the context for now.
> > > >
> > > > Andrew
> > > >
> > > > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> > > wrote:
> > > >
> > > > > One of the changes we should make is to create a separate guide for
> > > > product
> > > > > vendors on how to build and maintain a bundle. We're at that point
> > > where
> > > > > vendors will have to do it on their own as extension providers, so
> it
> > > > would
> > > > > be very helpful for them to have a simple and straight forward
> document
> > > > > showing them what should be there, best practices for
> maintainability
> > > and
> > > > > where to announce it.
> > > > >
> > > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> wrote:
> > > > >
> > > > > > I think we have a lot more documentation than most projects, but
> I
> > > > > > think an issue is that content is scattered in many different
> > > > > > locations, and some of the docs are huge reference guides where
> it
> > > can
> > > > > > be hard to find all the pieces of what you are trying to do.
> > > > > >
> > > > > > The first thing a new contributor wants to do is get the code
> and run
> > > > > > a build, and we do have a quick-start guide linked to on the
> site,
> > > but
> > > > > > I think there is a lot of extra information in there that is not
> > > > > > really relevant to someone just wanting get the code and build.
> We
> > > > > > could have separate guides per OS like "Build NiFi on Linux",
> "Build
> > > > > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > > > >
> > > > > > - Clone repo
> > > > > > - checkout master
> > > > > > - run maven
> > > > > > - cd to assembly
> > > > > > - ./bin/nifi.sh
> > > > > >
> > > > > > The next thing they want to do is contribute a change, and we
> have a
> > > > > > great contributor guide, but again I think there could be a very
> > > short
> > > > > > tutorial for the most common steps:
> > > > > >
> > > > > > - fork repo
> > > > > > - clone fork
> > > > > > - create branch
> > > > > > - make changes
> > > > > > - push branch
> > > > > > - submit pr
> > > > > >
> > > > > > and then say something like "for a more detailed description of
> the
> > > > > > contribution process, please reference the Contributor Guide".
> > > > > >
> > > > > > If we then make these getting started guides more prominent
> right in
> > > > > > the middle of the NiFi homepage, then maybe they will be easier
> to
> > > > > > find for new community members.
> > > > > >
> > > > > > We can keep extending this idea to other common tasks beyond just
> > > > > > building and contributing.
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi folks,
> > > > > > >
> > > > > > > Based on some recent (and long-term) experiences, I wanted to
> > > discuss
> > > > > > with the community what we could do to lower the barrier of
> entry to
> > > > > using
> > > > > > & contributing to NiFi. I hope to get some good feedback from
> both
> > > > > > long-time and newer members, and determine some immediate
> concrete
> > > > steps
> > > > > we
> > > > > > can take.
> > > > > > >
> > > > > > > Problems identified:
> > > > > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> > > > install”
> > > > > > in project root doesn’t get a new developer up and running
> > > immediately
> > > > > > > * The API is very well defined, but for new contributors, it
> can
> > > be a
> > > > > > challenge to know where to put functionality, and building a
> custom
> > > > > > processor + NAR and deploying isn’t a one-step process
> > > > > > > * Project size (and build size/time) is large. This can
> restrict
> > > the
> > > > > > minimum hardware necessary, elongate the development cycle, etc.
> > > > > > > * Some new users do not receive mailing list replies
> > > > > > >
> > > > > > > Possible solutions:
> > > > > > > * On a clean git clone, “mvn clean install” should build a
> working
> > > > > > instance. Maybe we provide a quickstart.sh script to handle the
> > > default
> > > > > > maven build, change to the target directory, and start NiFi?
> > > > > > > * Individual contributors have written excellent blogs, and
> > > > > > documentation exists, but making it more prominent or more easily
> > > > > accessed
> > > > > > could help?
> > > > > > > * Extension registry will solve all the world’s problems
> (related
> > > to
> > > > > > bundling and build time)
> > > > > > > * Not sure about this one — I don’t know if it’s because
> they’re
> > > not
> > > > > > subscribed, their mail client is blocking them, etc.
> > > > > > >
> > > > > > > I’ve said my bit, now I am eager to hear from other community
> > > members
> > > > > on
> > > > > > their experiences, steps that helped them, and suggestions for
> the
> > > > future
> > > > > > to continue to make the NiFi community welcoming to new users.
> > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > > Andy LoPresto
> > > > > > > [hidden email]
> > > > > > > [hidden email]
> > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> EF69
> > > > > > >
> > > > > >
> > > > >
> > >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

James Srinivasan
As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
realise I could and possibly should submit PRs :)

1) I'm used to Java and Maven, so used the archetype. It worked fine,
it would have been nice it if set up unit tests for me.
2) The User and Developer documentation is great and comprehensive.
Finding the developer docs is a little painful (handful of items at
the end of a scrolling list of 200+ processors)
3) The Developer docs could possibly do with a little more clarity on
processor lifetime i.e. what is called when ^h^h^h - skimming back
over the docs, it looks pretty clear now
4) Some example code for common operations e.g. getting/setting
attributes or reading/writing/modifying flowfile content would be
great.
5) When using existing processors for inspiration, best practices
weren't always clear e.g. some generated properties inside
getSupportedPropertyDescriptors(), others generated a private static
list on init and returned that. Such differences are inevitable in a
large project, but it would be nice to have something blessed to start
from.
6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
7) I couldn't find (m?)any docs about the Groovy scripting API, but
the great blog posts by Matt Burgess and others were invaluable
8) In case this all sounds too negative, NiFi is fab!

On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:

>
> I am not against the archetype. But we need to spell out every step of the
> way. I'd like to see a user thinking about their custom logic ASAP rather
> than fighting the tools to get started. Those steps should be brain-dead,
> just reflexes, if you know what I mean. Hell, let them create a custom
> processor project or prototype in a script by accident even! :)
>
> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
>
> > That makes sense about the best practice for deploying to an
> > additional lib directory.
> >
> > So for the project structure you are saying it would be easier to have
> > a repo somewhere with essentially the same thing that is in the
> > archetype, but they just clone it and rename it themselves (what the
> > archetype does for you)?
> >
> > Something that I think would be awesome is if we could provide a
> > web-based project initializer that would essentially run the archetype
> > behind the scenes and then let you download the archive of the code,
> > just like the spring-boot starter [1]. Not sure if their initializr is
> > something that can be re-used and customized [2].
> >
> > The problem is we would need to host that somewhere.
> >
> > [1] https://start.spring.io/
> > [2] https://github.com/spring-io/initializr
> >
> > On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:
> > >
> > > We assume they create new projects from archetypes every day. They don't.
> > >
> > > We also assume they know how to deploy new NARs. Most don't. Especially
> > if
> > > we want them to follow best practices and create an additional NAR
> > bundles
> > > directory entry im the config (vs dumping into nifi lib).
> > >
> > > I can attest that I feel a bit lost myself every time I need to come back
> > > to this and refresh my brain synapses. If we could make these not require
> > > any of that and make simple thinga dead simple....
> > >
> > > Andrew
> > >
> > > On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> > >
> > > > Andrew,
> > > >
> > > > I'm not disagreeing with your points, but I'm curious how you see
> > > > those two ideas being different from the processor archetype and the
> > > > wiki page with the archetype commands?
> > > >
> > > > Is it just that people don't know about it?
> > > >
> > > > -Bryan
> > > >
> > > > [1]
> > > >
> > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > >
> > > > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> > > > wrote:
> > > > >
> > > > > I think this ties into my other discuss thread on refreshing the
> > > > archetypes
> > > > >
> > > > >
> > > > > On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> > > > wrote:
> > > > >
> > > > > I consistently see my users struggling when they move up the nifi
> > food
> > > > > chain and start looking at custom processors. The good content about
> > > > > prototyping processsors via scripting processors and finalizing with
> > a
> > > > full
> > > > > NAR bundle is everywhere but where it should be.
> > > > >
> > > > > A few simple changes could help (not *more* docs). They are great,
> > much
> > > > > better than in many other projecta, but people are already drowning
> > in
> > > > > those.
> > > > >
> > > > > How about:
> > > > >
> > > > > + ISP has a pre-populated processor sceleton. A simple no-op to fill
> > in
> > > > is
> > > > > miles better than a blank text area (which invokes a blank stare).
> > > > >
> > > > > + As much as we may loook down on this, but... A simple guide to a
> > full
> > > > NAR
> > > > > build as a series of copy/paste commands.
> > > > >
> > > > > There's more, but this should fit the context for now.
> > > > >
> > > > > Andrew
> > > > >
> > > > > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > One of the changes we should make is to create a separate guide for
> > > > > product
> > > > > > vendors on how to build and maintain a bundle. We're at that point
> > > > where
> > > > > > vendors will have to do it on their own as extension providers, so
> > it
> > > > > would
> > > > > > be very helpful for them to have a simple and straight forward
> > document
> > > > > > showing them what should be there, best practices for
> > maintainability
> > > > and
> > > > > > where to announce it.
> > > > > >
> > > > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > I think we have a lot more documentation than most projects, but
> > I
> > > > > > > think an issue is that content is scattered in many different
> > > > > > > locations, and some of the docs are huge reference guides where
> > it
> > > > can
> > > > > > > be hard to find all the pieces of what you are trying to do.
> > > > > > >
> > > > > > > The first thing a new contributor wants to do is get the code
> > and run
> > > > > > > a build, and we do have a quick-start guide linked to on the
> > site,
> > > > but
> > > > > > > I think there is a lot of extra information in there that is not
> > > > > > > really relevant to someone just wanting get the code and build.
> > We
> > > > > > > could have separate guides per OS like "Build NiFi on Linux",
> > "Build
> > > > > > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > > > > >
> > > > > > > - Clone repo
> > > > > > > - checkout master
> > > > > > > - run maven
> > > > > > > - cd to assembly
> > > > > > > - ./bin/nifi.sh
> > > > > > >
> > > > > > > The next thing they want to do is contribute a change, and we
> > have a
> > > > > > > great contributor guide, but again I think there could be a very
> > > > short
> > > > > > > tutorial for the most common steps:
> > > > > > >
> > > > > > > - fork repo
> > > > > > > - clone fork
> > > > > > > - create branch
> > > > > > > - make changes
> > > > > > > - push branch
> > > > > > > - submit pr
> > > > > > >
> > > > > > > and then say something like "for a more detailed description of
> > the
> > > > > > > contribution process, please reference the Contributor Guide".
> > > > > > >
> > > > > > > If we then make these getting started guides more prominent
> > right in
> > > > > > > the middle of the NiFi homepage, then maybe they will be easier
> > to
> > > > > > > find for new community members.
> > > > > > >
> > > > > > > We can keep extending this idea to other common tasks beyond just
> > > > > > > building and contributing.
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > [hidden email]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi folks,
> > > > > > > >
> > > > > > > > Based on some recent (and long-term) experiences, I wanted to
> > > > discuss
> > > > > > > with the community what we could do to lower the barrier of
> > entry to
> > > > > > using
> > > > > > > & contributing to NiFi. I hope to get some good feedback from
> > both
> > > > > > > long-time and newer members, and determine some immediate
> > concrete
> > > > > steps
> > > > > > we
> > > > > > > can take.
> > > > > > > >
> > > > > > > > Problems identified:
> > > > > > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> > > > > install”
> > > > > > > in project root doesn’t get a new developer up and running
> > > > immediately
> > > > > > > > * The API is very well defined, but for new contributors, it
> > can
> > > > be a
> > > > > > > challenge to know where to put functionality, and building a
> > custom
> > > > > > > processor + NAR and deploying isn’t a one-step process
> > > > > > > > * Project size (and build size/time) is large. This can
> > restrict
> > > > the
> > > > > > > minimum hardware necessary, elongate the development cycle, etc.
> > > > > > > > * Some new users do not receive mailing list replies
> > > > > > > >
> > > > > > > > Possible solutions:
> > > > > > > > * On a clean git clone, “mvn clean install” should build a
> > working
> > > > > > > instance. Maybe we provide a quickstart.sh script to handle the
> > > > default
> > > > > > > maven build, change to the target directory, and start NiFi?
> > > > > > > > * Individual contributors have written excellent blogs, and
> > > > > > > documentation exists, but making it more prominent or more easily
> > > > > > accessed
> > > > > > > could help?
> > > > > > > > * Extension registry will solve all the world’s problems
> > (related
> > > > to
> > > > > > > bundling and build time)
> > > > > > > > * Not sure about this one — I don’t know if it’s because
> > they’re
> > > > not
> > > > > > > subscribed, their mail client is blocking them, etc.
> > > > > > > >
> > > > > > > > I’ve said my bit, now I am eager to hear from other community
> > > > members
> > > > > > on
> > > > > > > their experiences, steps that helped them, and suggestions for
> > the
> > > > > future
> > > > > > > to continue to make the NiFi community welcoming to new users.
> > > > Thanks.
> > > > > > > >
> > > > > > > >
> > > > > > > > Andy LoPresto
> > > > > > > > [hidden email]
> > > > > > > > [hidden email]
> > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > EF69
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

James Srinivasan
9) Oh, and the wiki is a little hard to navigate and the contents rather patchy

On Fri, 25 Jan 2019 at 21:57, James Srinivasan
<[hidden email]> wrote:

>
> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> realise I could and possibly should submit PRs :)
>
> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> it would have been nice it if set up unit tests for me.
> 2) The User and Developer documentation is great and comprehensive.
> Finding the developer docs is a little painful (handful of items at
> the end of a scrolling list of 200+ processors)
> 3) The Developer docs could possibly do with a little more clarity on
> processor lifetime i.e. what is called when ^h^h^h - skimming back
> over the docs, it looks pretty clear now
> 4) Some example code for common operations e.g. getting/setting
> attributes or reading/writing/modifying flowfile content would be
> great.
> 5) When using existing processors for inspiration, best practices
> weren't always clear e.g. some generated properties inside
> getSupportedPropertyDescriptors(), others generated a private static
> list on init and returned that. Such differences are inevitable in a
> large project, but it would be nice to have something blessed to start
> from.
> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> the great blog posts by Matt Burgess and others were invaluable
> 8) In case this all sounds too negative, NiFi is fab!
>
> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
> >
> > I am not against the archetype. But we need to spell out every step of the
> > way. I'd like to see a user thinking about their custom logic ASAP rather
> > than fighting the tools to get started. Those steps should be brain-dead,
> > just reflexes, if you know what I mean. Hell, let them create a custom
> > processor project or prototype in a script by accident even! :)
> >
> > On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
> >
> > > That makes sense about the best practice for deploying to an
> > > additional lib directory.
> > >
> > > So for the project structure you are saying it would be easier to have
> > > a repo somewhere with essentially the same thing that is in the
> > > archetype, but they just clone it and rename it themselves (what the
> > > archetype does for you)?
> > >
> > > Something that I think would be awesome is if we could provide a
> > > web-based project initializer that would essentially run the archetype
> > > behind the scenes and then let you download the archive of the code,
> > > just like the spring-boot starter [1]. Not sure if their initializr is
> > > something that can be re-used and customized [2].
> > >
> > > The problem is we would need to host that somewhere.
> > >
> > > [1] https://start.spring.io/
> > > [2] https://github.com/spring-io/initializr
> > >
> > > On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:
> > > >
> > > > We assume they create new projects from archetypes every day. They don't.
> > > >
> > > > We also assume they know how to deploy new NARs. Most don't. Especially
> > > if
> > > > we want them to follow best practices and create an additional NAR
> > > bundles
> > > > directory entry im the config (vs dumping into nifi lib).
> > > >
> > > > I can attest that I feel a bit lost myself every time I need to come back
> > > > to this and refresh my brain synapses. If we could make these not require
> > > > any of that and make simple thinga dead simple....
> > > >
> > > > Andrew
> > > >
> > > > On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> > > >
> > > > > Andrew,
> > > > >
> > > > > I'm not disagreeing with your points, but I'm curious how you see
> > > > > those two ideas being different from the processor archetype and the
> > > > > wiki page with the archetype commands?
> > > > >
> > > > > Is it just that people don't know about it?
> > > > >
> > > > > -Bryan
> > > > >
> > > > > [1]
> > > > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > > >
> > > > > On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > I think this ties into my other discuss thread on refreshing the
> > > > > archetypes
> > > > > >
> > > > > >
> > > > > > On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> > > > > wrote:
> > > > > >
> > > > > > I consistently see my users struggling when they move up the nifi
> > > food
> > > > > > chain and start looking at custom processors. The good content about
> > > > > > prototyping processsors via scripting processors and finalizing with
> > > a
> > > > > full
> > > > > > NAR bundle is everywhere but where it should be.
> > > > > >
> > > > > > A few simple changes could help (not *more* docs). They are great,
> > > much
> > > > > > better than in many other projecta, but people are already drowning
> > > in
> > > > > > those.
> > > > > >
> > > > > > How about:
> > > > > >
> > > > > > + ISP has a pre-populated processor sceleton. A simple no-op to fill
> > > in
> > > > > is
> > > > > > miles better than a blank text area (which invokes a blank stare).
> > > > > >
> > > > > > + As much as we may loook down on this, but... A simple guide to a
> > > full
> > > > > NAR
> > > > > > build as a series of copy/paste commands.
> > > > > >
> > > > > > There's more, but this should fit the context for now.
> > > > > >
> > > > > > Andrew
> > > > > >
> > > > > > On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > One of the changes we should make is to create a separate guide for
> > > > > > product
> > > > > > > vendors on how to build and maintain a bundle. We're at that point
> > > > > where
> > > > > > > vendors will have to do it on their own as extension providers, so
> > > it
> > > > > > would
> > > > > > > be very helpful for them to have a simple and straight forward
> > > document
> > > > > > > showing them what should be there, best practices for
> > > maintainability
> > > > > and
> > > > > > > where to announce it.
> > > > > > >
> > > > > > > On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > I think we have a lot more documentation than most projects, but
> > > I
> > > > > > > > think an issue is that content is scattered in many different
> > > > > > > > locations, and some of the docs are huge reference guides where
> > > it
> > > > > can
> > > > > > > > be hard to find all the pieces of what you are trying to do.
> > > > > > > >
> > > > > > > > The first thing a new contributor wants to do is get the code
> > > and run
> > > > > > > > a build, and we do have a quick-start guide linked to on the
> > > site,
> > > > > but
> > > > > > > > I think there is a lot of extra information in there that is not
> > > > > > > > really relevant to someone just wanting get the code and build.
> > > We
> > > > > > > > could have separate guides per OS like "Build NiFi on Linux",
> > > "Build
> > > > > > > > NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > > > > > >
> > > > > > > > - Clone repo
> > > > > > > > - checkout master
> > > > > > > > - run maven
> > > > > > > > - cd to assembly
> > > > > > > > - ./bin/nifi.sh
> > > > > > > >
> > > > > > > > The next thing they want to do is contribute a change, and we
> > > have a
> > > > > > > > great contributor guide, but again I think there could be a very
> > > > > short
> > > > > > > > tutorial for the most common steps:
> > > > > > > >
> > > > > > > > - fork repo
> > > > > > > > - clone fork
> > > > > > > > - create branch
> > > > > > > > - make changes
> > > > > > > > - push branch
> > > > > > > > - submit pr
> > > > > > > >
> > > > > > > > and then say something like "for a more detailed description of
> > > the
> > > > > > > > contribution process, please reference the Contributor Guide".
> > > > > > > >
> > > > > > > > If we then make these getting started guides more prominent
> > > right in
> > > > > > > > the middle of the NiFi homepage, then maybe they will be easier
> > > to
> > > > > > > > find for new community members.
> > > > > > > >
> > > > > > > > We can keep extending this idea to other common tasks beyond just
> > > > > > > > building and contributing.
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > Hi folks,
> > > > > > > > >
> > > > > > > > > Based on some recent (and long-term) experiences, I wanted to
> > > > > discuss
> > > > > > > > with the community what we could do to lower the barrier of
> > > entry to
> > > > > > > using
> > > > > > > > & contributing to NiFi. I hope to get some good feedback from
> > > both
> > > > > > > > long-time and newer members, and determine some immediate
> > > concrete
> > > > > > steps
> > > > > > > we
> > > > > > > > can take.
> > > > > > > > >
> > > > > > > > > Problems identified:
> > > > > > > > > * NiFi has a number of custom profiles, so a simple “mvn clean
> > > > > > install”
> > > > > > > > in project root doesn’t get a new developer up and running
> > > > > immediately
> > > > > > > > > * The API is very well defined, but for new contributors, it
> > > can
> > > > > be a
> > > > > > > > challenge to know where to put functionality, and building a
> > > custom
> > > > > > > > processor + NAR and deploying isn’t a one-step process
> > > > > > > > > * Project size (and build size/time) is large. This can
> > > restrict
> > > > > the
> > > > > > > > minimum hardware necessary, elongate the development cycle, etc.
> > > > > > > > > * Some new users do not receive mailing list replies
> > > > > > > > >
> > > > > > > > > Possible solutions:
> > > > > > > > > * On a clean git clone, “mvn clean install” should build a
> > > working
> > > > > > > > instance. Maybe we provide a quickstart.sh script to handle the
> > > > > default
> > > > > > > > maven build, change to the target directory, and start NiFi?
> > > > > > > > > * Individual contributors have written excellent blogs, and
> > > > > > > > documentation exists, but making it more prominent or more easily
> > > > > > > accessed
> > > > > > > > could help?
> > > > > > > > > * Extension registry will solve all the world’s problems
> > > (related
> > > > > to
> > > > > > > > bundling and build time)
> > > > > > > > > * Not sure about this one — I don’t know if it’s because
> > > they’re
> > > > > not
> > > > > > > > subscribed, their mail client is blocking them, etc.
> > > > > > > > >
> > > > > > > > > I’ve said my bit, now I am eager to hear from other community
> > > > > members
> > > > > > > on
> > > > > > > > their experiences, steps that helped them, and suggestions for
> > > the
> > > > > > future
> > > > > > > > to continue to make the NiFi community welcoming to new users.
> > > > > Thanks.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Andy LoPresto
> > > > > > > > > [hidden email]
> > > > > > > > > [hidden email]
> > > > > > > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > > EF69
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > >
> > >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Andy LoPresto-2
James,

I’m wondering if a page outlining a toy processor (something like `CountText` or `ReverseContent`) and doing a line-by-line annotation from a developer’s perspective would be helpful. It could be a few sections:

1. How to get to this point
        * running the maven archetype
        * choosing the directory to install to
        * putting the class name in the manifest file
        * etc.
2. The code
        * here’s the class
        * here’s what extending AbstractProcessor gets you, etc. A lot of this is currently in the Developer Guide, but in textbook mode
        * here’s how to modify some code (i.e. write one line of Java that switches it from straight content pass-through to reversing the text)
        * here’s how to make a unit test (introduce the TestRunner framework, etc.)
3. Running, building, installing
        * Run your unit test from the IDE/maven
        * Build the NAR module
        * Install the NAR in NiFi lib/ or custom/
        * Restart NiFi
                * See the NAR loaded in the log
                * Deploy the component on the canvas

I imagine this being written more conversationally/blog-like than most of our current reference documentation to be used as a split-screen walkthrough. Each section could certainly link to the existing detailed documentation for various topics, like the processor lifecycle, etc.

Does this sounds like something that would have helped you?

Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 25, 2019, at 1:59 PM, James Srinivasan <[hidden email]> wrote:
>
> 9) Oh, and the wiki is a little hard to navigate and the contents rather patchy
>
> On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> <[hidden email]> wrote:
>>
>> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
>> realise I could and possibly should submit PRs :)
>>
>> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
>> it would have been nice it if set up unit tests for me.
>> 2) The User and Developer documentation is great and comprehensive.
>> Finding the developer docs is a little painful (handful of items at
>> the end of a scrolling list of 200+ processors)
>> 3) The Developer docs could possibly do with a little more clarity on
>> processor lifetime i.e. what is called when ^h^h^h - skimming back
>> over the docs, it looks pretty clear now
>> 4) Some example code for common operations e.g. getting/setting
>> attributes or reading/writing/modifying flowfile content would be
>> great.
>> 5) When using existing processors for inspiration, best practices
>> weren't always clear e.g. some generated properties inside
>> getSupportedPropertyDescriptors(), others generated a private static
>> list on init and returned that. Such differences are inevitable in a
>> large project, but it would be nice to have something blessed to start
>> from.
>> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
>> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
>> the great blog posts by Matt Burgess and others were invaluable
>> 8) In case this all sounds too negative, NiFi is fab!
>>
>> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
>>>
>>> I am not against the archetype. But we need to spell out every step of the
>>> way. I'd like to see a user thinking about their custom logic ASAP rather
>>> than fighting the tools to get started. Those steps should be brain-dead,
>>> just reflexes, if you know what I mean. Hell, let them create a custom
>>> processor project or prototype in a script by accident even! :)
>>>
>>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
>>>
>>>> That makes sense about the best practice for deploying to an
>>>> additional lib directory.
>>>>
>>>> So for the project structure you are saying it would be easier to have
>>>> a repo somewhere with essentially the same thing that is in the
>>>> archetype, but they just clone it and rename it themselves (what the
>>>> archetype does for you)?
>>>>
>>>> Something that I think would be awesome is if we could provide a
>>>> web-based project initializer that would essentially run the archetype
>>>> behind the scenes and then let you download the archive of the code,
>>>> just like the spring-boot starter [1]. Not sure if their initializr is
>>>> something that can be re-used and customized [2].
>>>>
>>>> The problem is we would need to host that somewhere.
>>>>
>>>> [1] https://start.spring.io/
>>>> [2] https://github.com/spring-io/initializr
>>>>
>>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:
>>>>>
>>>>> We assume they create new projects from archetypes every day. They don't.
>>>>>
>>>>> We also assume they know how to deploy new NARs. Most don't. Especially
>>>> if
>>>>> we want them to follow best practices and create an additional NAR
>>>> bundles
>>>>> directory entry im the config (vs dumping into nifi lib).
>>>>>
>>>>> I can attest that I feel a bit lost myself every time I need to come back
>>>>> to this and refresh my brain synapses. If we could make these not require
>>>>> any of that and make simple thinga dead simple....
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> I'm not disagreeing with your points, but I'm curious how you see
>>>>>> those two ideas being different from the processor archetype and the
>>>>>> wiki page with the archetype commands?
>>>>>>
>>>>>> Is it just that people don't know about it?
>>>>>>
>>>>>> -Bryan
>>>>>>
>>>>>> [1]
>>>>>>
>>>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>>>>>>
>>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>> I think this ties into my other discuss thread on refreshing the
>>>>>> archetypes
>>>>>>>
>>>>>>>
>>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
>>>>>> wrote:
>>>>>>>
>>>>>>> I consistently see my users struggling when they move up the nifi
>>>> food
>>>>>>> chain and start looking at custom processors. The good content about
>>>>>>> prototyping processsors via scripting processors and finalizing with
>>>> a
>>>>>> full
>>>>>>> NAR bundle is everywhere but where it should be.
>>>>>>>
>>>>>>> A few simple changes could help (not *more* docs). They are great,
>>>> much
>>>>>>> better than in many other projecta, but people are already drowning
>>>> in
>>>>>>> those.
>>>>>>>
>>>>>>> How about:
>>>>>>>
>>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to fill
>>>> in
>>>>>> is
>>>>>>> miles better than a blank text area (which invokes a blank stare).
>>>>>>>
>>>>>>> + As much as we may loook down on this, but... A simple guide to a
>>>> full
>>>>>> NAR
>>>>>>> build as a series of copy/paste commands.
>>>>>>>
>>>>>>> There's more, but this should fit the context for now.
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>>> One of the changes we should make is to create a separate guide for
>>>>>>> product
>>>>>>>> vendors on how to build and maintain a bundle. We're at that point
>>>>>> where
>>>>>>>> vendors will have to do it on their own as extension providers, so
>>>> it
>>>>>>> would
>>>>>>>> be very helpful for them to have a simple and straight forward
>>>> document
>>>>>>>> showing them what should be there, best practices for
>>>> maintainability
>>>>>> and
>>>>>>>> where to announce it.
>>>>>>>>
>>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
>>>> wrote:
>>>>>>>>
>>>>>>>>> I think we have a lot more documentation than most projects, but
>>>> I
>>>>>>>>> think an issue is that content is scattered in many different
>>>>>>>>> locations, and some of the docs are huge reference guides where
>>>> it
>>>>>> can
>>>>>>>>> be hard to find all the pieces of what you are trying to do.
>>>>>>>>>
>>>>>>>>> The first thing a new contributor wants to do is get the code
>>>> and run
>>>>>>>>> a build, and we do have a quick-start guide linked to on the
>>>> site,
>>>>>> but
>>>>>>>>> I think there is a lot of extra information in there that is not
>>>>>>>>> really relevant to someone just wanting get the code and build.
>>>> We
>>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
>>>> "Build
>>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
>>>>>>>>>
>>>>>>>>> - Clone repo
>>>>>>>>> - checkout master
>>>>>>>>> - run maven
>>>>>>>>> - cd to assembly
>>>>>>>>> - ./bin/nifi.sh
>>>>>>>>>
>>>>>>>>> The next thing they want to do is contribute a change, and we
>>>> have a
>>>>>>>>> great contributor guide, but again I think there could be a very
>>>>>> short
>>>>>>>>> tutorial for the most common steps:
>>>>>>>>>
>>>>>>>>> - fork repo
>>>>>>>>> - clone fork
>>>>>>>>> - create branch
>>>>>>>>> - make changes
>>>>>>>>> - push branch
>>>>>>>>> - submit pr
>>>>>>>>>
>>>>>>>>> and then say something like "for a more detailed description of
>>>> the
>>>>>>>>> contribution process, please reference the Contributor Guide".
>>>>>>>>>
>>>>>>>>> If we then make these getting started guides more prominent
>>>> right in
>>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
>>>> to
>>>>>>>>> find for new community members.
>>>>>>>>>
>>>>>>>>> We can keep extending this idea to other common tasks beyond just
>>>>>>>>> building and contributing.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
>>>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
>>>>>> discuss
>>>>>>>>> with the community what we could do to lower the barrier of
>>>> entry to
>>>>>>>> using
>>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
>>>> both
>>>>>>>>> long-time and newer members, and determine some immediate
>>>> concrete
>>>>>>> steps
>>>>>>>> we
>>>>>>>>> can take.
>>>>>>>>>>
>>>>>>>>>> Problems identified:
>>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
>>>>>>> install”
>>>>>>>>> in project root doesn’t get a new developer up and running
>>>>>> immediately
>>>>>>>>>> * The API is very well defined, but for new contributors, it
>>>> can
>>>>>> be a
>>>>>>>>> challenge to know where to put functionality, and building a
>>>> custom
>>>>>>>>> processor + NAR and deploying isn’t a one-step process
>>>>>>>>>> * Project size (and build size/time) is large. This can
>>>> restrict
>>>>>> the
>>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
>>>>>>>>>> * Some new users do not receive mailing list replies
>>>>>>>>>>
>>>>>>>>>> Possible solutions:
>>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
>>>> working
>>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
>>>>>> default
>>>>>>>>> maven build, change to the target directory, and start NiFi?
>>>>>>>>>> * Individual contributors have written excellent blogs, and
>>>>>>>>> documentation exists, but making it more prominent or more easily
>>>>>>>> accessed
>>>>>>>>> could help?
>>>>>>>>>> * Extension registry will solve all the world’s problems
>>>> (related
>>>>>> to
>>>>>>>>> bundling and build time)
>>>>>>>>>> * Not sure about this one — I don’t know if it’s because
>>>> they’re
>>>>>> not
>>>>>>>>> subscribed, their mail client is blocking them, etc.
>>>>>>>>>>
>>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
>>>>>> members
>>>>>>>> on
>>>>>>>>> their experiences, steps that helped them, and suggestions for
>>>> the
>>>>>>> future
>>>>>>>>> to continue to make the NiFi community welcoming to new users.
>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Andy LoPresto
>>>>>>>>>> [hidden email]
>>>>>>>>>> [hidden email]
>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
>>>> EF69
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>

Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

James Srinivasan
That would have been great - the only things I would add are
reading/writing attributes & reading the processor's properties. Maybe
something about testing in NiFi using GenerateFlowFile too?

I ended up referring to the bundled processors a lot, but it was
sometimes hard to see the wood for the trees.

On Fri, 25 Jan 2019 at 23:04, Andy LoPresto <[hidden email]> wrote:

>
> James,
>
> I’m wondering if a page outlining a toy processor (something like `CountText` or `ReverseContent`) and doing a line-by-line annotation from a developer’s perspective would be helpful. It could be a few sections:
>
> 1. How to get to this point
>         * running the maven archetype
>         * choosing the directory to install to
>         * putting the class name in the manifest file
>         * etc.
> 2. The code
>         * here’s the class
>         * here’s what extending AbstractProcessor gets you, etc. A lot of this is currently in the Developer Guide, but in textbook mode
>         * here’s how to modify some code (i.e. write one line of Java that switches it from straight content pass-through to reversing the text)
>         * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> 3. Running, building, installing
>         * Run your unit test from the IDE/maven
>         * Build the NAR module
>         * Install the NAR in NiFi lib/ or custom/
>         * Restart NiFi
>                 * See the NAR loaded in the log
>                 * Deploy the component on the canvas
>
> I imagine this being written more conversationally/blog-like than most of our current reference documentation to be used as a split-screen walkthrough. Each section could certainly link to the existing detailed documentation for various topics, like the processor lifecycle, etc.
>
> Does this sounds like something that would have helped you?
>
> Andy LoPresto
> [hidden email]
> [hidden email]
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> > On Jan 25, 2019, at 1:59 PM, James Srinivasan <[hidden email]> wrote:
> >
> > 9) Oh, and the wiki is a little hard to navigate and the contents rather patchy
> >
> > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > <[hidden email]> wrote:
> >>
> >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> >> realise I could and possibly should submit PRs :)
> >>
> >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> >> it would have been nice it if set up unit tests for me.
> >> 2) The User and Developer documentation is great and comprehensive.
> >> Finding the developer docs is a little painful (handful of items at
> >> the end of a scrolling list of 200+ processors)
> >> 3) The Developer docs could possibly do with a little more clarity on
> >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> >> over the docs, it looks pretty clear now
> >> 4) Some example code for common operations e.g. getting/setting
> >> attributes or reading/writing/modifying flowfile content would be
> >> great.
> >> 5) When using existing processors for inspiration, best practices
> >> weren't always clear e.g. some generated properties inside
> >> getSupportedPropertyDescriptors(), others generated a private static
> >> list on init and returned that. Such differences are inevitable in a
> >> large project, but it would be nice to have something blessed to start
> >> from.
> >> 6) (Minor niggle - layout of the docs doesn't work great on a phone screen)
> >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> >> the great blog posts by Matt Burgess and others were invaluable
> >> 8) In case this all sounds too negative, NiFi is fab!
> >>
> >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
> >>>
> >>> I am not against the archetype. But we need to spell out every step of the
> >>> way. I'd like to see a user thinking about their custom logic ASAP rather
> >>> than fighting the tools to get started. Those steps should be brain-dead,
> >>> just reflexes, if you know what I mean. Hell, let them create a custom
> >>> processor project or prototype in a script by accident even! :)
> >>>
> >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
> >>>
> >>>> That makes sense about the best practice for deploying to an
> >>>> additional lib directory.
> >>>>
> >>>> So for the project structure you are saying it would be easier to have
> >>>> a repo somewhere with essentially the same thing that is in the
> >>>> archetype, but they just clone it and rename it themselves (what the
> >>>> archetype does for you)?
> >>>>
> >>>> Something that I think would be awesome is if we could provide a
> >>>> web-based project initializer that would essentially run the archetype
> >>>> behind the scenes and then let you download the archive of the code,
> >>>> just like the spring-boot starter [1]. Not sure if their initializr is
> >>>> something that can be re-used and customized [2].
> >>>>
> >>>> The problem is we would need to host that somewhere.
> >>>>
> >>>> [1] https://start.spring.io/
> >>>> [2] https://github.com/spring-io/initializr
> >>>>
> >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]> wrote:
> >>>>>
> >>>>> We assume they create new projects from archetypes every day. They don't.
> >>>>>
> >>>>> We also assume they know how to deploy new NARs. Most don't. Especially
> >>>> if
> >>>>> we want them to follow best practices and create an additional NAR
> >>>> bundles
> >>>>> directory entry im the config (vs dumping into nifi lib).
> >>>>>
> >>>>> I can attest that I feel a bit lost myself every time I need to come back
> >>>>> to this and refresh my brain synapses. If we could make these not require
> >>>>> any of that and make simple thinga dead simple....
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> >>>>>
> >>>>>> Andrew,
> >>>>>>
> >>>>>> I'm not disagreeing with your points, but I'm curious how you see
> >>>>>> those two ideas being different from the processor archetype and the
> >>>>>> wiki page with the archetype commands?
> >>>>>>
> >>>>>> Is it just that people don't know about it?
> >>>>>>
> >>>>>> -Bryan
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> >>>>>>
> >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I think this ties into my other discuss thread on refreshing the
> >>>>>> archetypes
> >>>>>>>
> >>>>>>>
> >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I consistently see my users struggling when they move up the nifi
> >>>> food
> >>>>>>> chain and start looking at custom processors. The good content about
> >>>>>>> prototyping processsors via scripting processors and finalizing with
> >>>> a
> >>>>>> full
> >>>>>>> NAR bundle is everywhere but where it should be.
> >>>>>>>
> >>>>>>> A few simple changes could help (not *more* docs). They are great,
> >>>> much
> >>>>>>> better than in many other projecta, but people are already drowning
> >>>> in
> >>>>>>> those.
> >>>>>>>
> >>>>>>> How about:
> >>>>>>>
> >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to fill
> >>>> in
> >>>>>> is
> >>>>>>> miles better than a blank text area (which invokes a blank stare).
> >>>>>>>
> >>>>>>> + As much as we may loook down on this, but... A simple guide to a
> >>>> full
> >>>>>> NAR
> >>>>>>> build as a series of copy/paste commands.
> >>>>>>>
> >>>>>>> There's more, but this should fit the context for now.
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> One of the changes we should make is to create a separate guide for
> >>>>>>> product
> >>>>>>>> vendors on how to build and maintain a bundle. We're at that point
> >>>>>> where
> >>>>>>>> vendors will have to do it on their own as extension providers, so
> >>>> it
> >>>>>>> would
> >>>>>>>> be very helpful for them to have a simple and straight forward
> >>>> document
> >>>>>>>> showing them what should be there, best practices for
> >>>> maintainability
> >>>>>> and
> >>>>>>>> where to announce it.
> >>>>>>>>
> >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> I think we have a lot more documentation than most projects, but
> >>>> I
> >>>>>>>>> think an issue is that content is scattered in many different
> >>>>>>>>> locations, and some of the docs are huge reference guides where
> >>>> it
> >>>>>> can
> >>>>>>>>> be hard to find all the pieces of what you are trying to do.
> >>>>>>>>>
> >>>>>>>>> The first thing a new contributor wants to do is get the code
> >>>> and run
> >>>>>>>>> a build, and we do have a quick-start guide linked to on the
> >>>> site,
> >>>>>> but
> >>>>>>>>> I think there is a lot of extra information in there that is not
> >>>>>>>>> really relevant to someone just wanting get the code and build.
> >>>> We
> >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
> >>>> "Build
> >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> >>>>>>>>>
> >>>>>>>>> - Clone repo
> >>>>>>>>> - checkout master
> >>>>>>>>> - run maven
> >>>>>>>>> - cd to assembly
> >>>>>>>>> - ./bin/nifi.sh
> >>>>>>>>>
> >>>>>>>>> The next thing they want to do is contribute a change, and we
> >>>> have a
> >>>>>>>>> great contributor guide, but again I think there could be a very
> >>>>>> short
> >>>>>>>>> tutorial for the most common steps:
> >>>>>>>>>
> >>>>>>>>> - fork repo
> >>>>>>>>> - clone fork
> >>>>>>>>> - create branch
> >>>>>>>>> - make changes
> >>>>>>>>> - push branch
> >>>>>>>>> - submit pr
> >>>>>>>>>
> >>>>>>>>> and then say something like "for a more detailed description of
> >>>> the
> >>>>>>>>> contribution process, please reference the Contributor Guide".
> >>>>>>>>>
> >>>>>>>>> If we then make these getting started guides more prominent
> >>>> right in
> >>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
> >>>> to
> >>>>>>>>> find for new community members.
> >>>>>>>>>
> >>>>>>>>> We can keep extending this idea to other common tasks beyond just
> >>>>>>>>> building and contributing.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> >>>> [hidden email]>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi folks,
> >>>>>>>>>>
> >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
> >>>>>> discuss
> >>>>>>>>> with the community what we could do to lower the barrier of
> >>>> entry to
> >>>>>>>> using
> >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
> >>>> both
> >>>>>>>>> long-time and newer members, and determine some immediate
> >>>> concrete
> >>>>>>> steps
> >>>>>>>> we
> >>>>>>>>> can take.
> >>>>>>>>>>
> >>>>>>>>>> Problems identified:
> >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
> >>>>>>> install”
> >>>>>>>>> in project root doesn’t get a new developer up and running
> >>>>>> immediately
> >>>>>>>>>> * The API is very well defined, but for new contributors, it
> >>>> can
> >>>>>> be a
> >>>>>>>>> challenge to know where to put functionality, and building a
> >>>> custom
> >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> >>>>>>>>>> * Project size (and build size/time) is large. This can
> >>>> restrict
> >>>>>> the
> >>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
> >>>>>>>>>> * Some new users do not receive mailing list replies
> >>>>>>>>>>
> >>>>>>>>>> Possible solutions:
> >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> >>>> working
> >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
> >>>>>> default
> >>>>>>>>> maven build, change to the target directory, and start NiFi?
> >>>>>>>>>> * Individual contributors have written excellent blogs, and
> >>>>>>>>> documentation exists, but making it more prominent or more easily
> >>>>>>>> accessed
> >>>>>>>>> could help?
> >>>>>>>>>> * Extension registry will solve all the world’s problems
> >>>> (related
> >>>>>> to
> >>>>>>>>> bundling and build time)
> >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> >>>> they’re
> >>>>>> not
> >>>>>>>>> subscribed, their mail client is blocking them, etc.
> >>>>>>>>>>
> >>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
> >>>>>> members
> >>>>>>>> on
> >>>>>>>>> their experiences, steps that helped them, and suggestions for
> >>>> the
> >>>>>>> future
> >>>>>>>>> to continue to make the NiFi community welcoming to new users.
> >>>>>> Thanks.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Andy LoPresto
> >>>>>>>>>> [hidden email]
> >>>>>>>>>> [hidden email]
> >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> >>>> EF69
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Otto Fowler
In reply to this post by Andy LoPresto-2
Why wouldn’t we make the archetypes do this?


On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email]) wrote:

James,

I’m wondering if a page outlining a toy processor (something like
`CountText` or `ReverseContent`) and doing a line-by-line annotation from a
developer’s perspective would be helpful. It could be a few sections:

1. How to get to this point
* running the maven archetype
* choosing the directory to install to
* putting the class name in the manifest file
* etc.
2. The code
* here’s the class
* here’s what extending AbstractProcessor gets you, etc. A lot of this is
currently in the Developer Guide, but in textbook mode
* here’s how to modify some code (i.e. write one line of Java that switches
it from straight content pass-through to reversing the text)
* here’s how to make a unit test (introduce the TestRunner framework, etc.)
3. Running, building, installing
* Run your unit test from the IDE/maven
* Build the NAR module
* Install the NAR in NiFi lib/ or custom/
* Restart NiFi
* See the NAR loaded in the log
* Deploy the component on the canvas

I imagine this being written more conversationally/blog-like than most of
our current reference documentation to be used as a split-screen
walkthrough. Each section could certainly link to the existing detailed
documentation for various topics, like the processor lifecycle, etc.

Does this sounds like something that would have helped you?

Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69

> On Jan 25, 2019, at 1:59 PM, James Srinivasan <[hidden email]>
wrote:
>
> 9) Oh, and the wiki is a little hard to navigate and the contents rather
patchy

>
> On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> <[hidden email]> wrote:
>>
>> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
>> realise I could and possibly should submit PRs :)
>>
>> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
>> it would have been nice it if set up unit tests for me.
>> 2) The User and Developer documentation is great and comprehensive.
>> Finding the developer docs is a little painful (handful of items at
>> the end of a scrolling list of 200+ processors)
>> 3) The Developer docs could possibly do with a little more clarity on
>> processor lifetime i.e. what is called when ^h^h^h - skimming back
>> over the docs, it looks pretty clear now
>> 4) Some example code for common operations e.g. getting/setting
>> attributes or reading/writing/modifying flowfile content would be
>> great.
>> 5) When using existing processors for inspiration, best practices
>> weren't always clear e.g. some generated properties inside
>> getSupportedPropertyDescriptors(), others generated a private static
>> list on init and returned that. Such differences are inevitable in a
>> large project, but it would be nice to have something blessed to start
>> from.
>> 6) (Minor niggle - layout of the docs doesn't work great on a phone
screen)
>> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
>> the great blog posts by Matt Burgess and others were invaluable
>> 8) In case this all sounds too negative, NiFi is fab!
>>
>> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
>>>
>>> I am not against the archetype. But we need to spell out every step of
the
>>> way. I'd like to see a user thinking about their custom logic ASAP
rather
>>> than fighting the tools to get started. Those steps should be
brain-dead,

>>> just reflexes, if you know what I mean. Hell, let them create a custom
>>> processor project or prototype in a script by accident even! :)
>>>
>>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
>>>
>>>> That makes sense about the best practice for deploying to an
>>>> additional lib directory.
>>>>
>>>> So for the project structure you are saying it would be easier to have
>>>> a repo somewhere with essentially the same thing that is in the
>>>> archetype, but they just clone it and rename it themselves (what the
>>>> archetype does for you)?
>>>>
>>>> Something that I think would be awesome is if we could provide a
>>>> web-based project initializer that would essentially run the archetype
>>>> behind the scenes and then let you download the archive of the code,
>>>> just like the spring-boot starter [1]. Not sure if their initializr is
>>>> something that can be re-used and customized [2].
>>>>
>>>> The problem is we would need to host that somewhere.
>>>>
>>>> [1] https://start.spring.io/
>>>> [2] https://github.com/spring-io/initializr
>>>>
>>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]>
wrote:
>>>>>
>>>>> We assume they create new projects from archetypes every day. They
don't.
>>>>>
>>>>> We also assume they know how to deploy new NARs. Most don't.
Especially
>>>> if
>>>>> we want them to follow best practices and create an additional NAR
>>>> bundles
>>>>> directory entry im the config (vs dumping into nifi lib).
>>>>>
>>>>> I can attest that I feel a bit lost myself every time I need to come
back
>>>>> to this and refresh my brain synapses. If we could make these not
require

>>>>> any of that and make simple thinga dead simple....
>>>>>
>>>>> Andrew
>>>>>
>>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
>>>>>
>>>>>> Andrew,
>>>>>>
>>>>>> I'm not disagreeing with your points, but I'm curious how you see
>>>>>> those two ideas being different from the processor archetype and the
>>>>>> wiki page with the archetype commands?
>>>>>>
>>>>>> Is it just that people don't know about it?
>>>>>>
>>>>>> -Bryan
>>>>>>
>>>>>> [1]
>>>>>>
>>>>
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>>>>>>
>>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <[hidden email]>

>>>>>> wrote:
>>>>>>>
>>>>>>> I think this ties into my other discuss thread on refreshing the
>>>>>> archetypes
>>>>>>>
>>>>>>>
>>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email])
>>>>>> wrote:
>>>>>>>
>>>>>>> I consistently see my users struggling when they move up the nifi
>>>> food
>>>>>>> chain and start looking at custom processors. The good content
about
>>>>>>> prototyping processsors via scripting processors and finalizing
with

>>>> a
>>>>>> full
>>>>>>> NAR bundle is everywhere but where it should be.
>>>>>>>
>>>>>>> A few simple changes could help (not *more* docs). They are great,
>>>> much
>>>>>>> better than in many other projecta, but people are already drowning
>>>> in
>>>>>>> those.
>>>>>>>
>>>>>>> How about:
>>>>>>>
>>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to
fill

>>>> in
>>>>>> is
>>>>>>> miles better than a blank text area (which invokes a blank stare).
>>>>>>>
>>>>>>> + As much as we may loook down on this, but... A simple guide to a
>>>> full
>>>>>> NAR
>>>>>>> build as a series of copy/paste commands.
>>>>>>>
>>>>>>> There's more, but this should fit the context for now.
>>>>>>>
>>>>>>> Andrew
>>>>>>>
>>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]>
>>>>>> wrote:
>>>>>>>
>>>>>>>> One of the changes we should make is to create a separate guide
for

>>>>>>> product
>>>>>>>> vendors on how to build and maintain a bundle. We're at that point
>>>>>> where
>>>>>>>> vendors will have to do it on their own as extension providers, so
>>>> it
>>>>>>> would
>>>>>>>> be very helpful for them to have a simple and straight forward
>>>> document
>>>>>>>> showing them what should be there, best practices for
>>>> maintainability
>>>>>> and
>>>>>>>> where to announce it.
>>>>>>>>
>>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
>>>> wrote:
>>>>>>>>
>>>>>>>>> I think we have a lot more documentation than most projects, but
>>>> I
>>>>>>>>> think an issue is that content is scattered in many different
>>>>>>>>> locations, and some of the docs are huge reference guides where
>>>> it
>>>>>> can
>>>>>>>>> be hard to find all the pieces of what you are trying to do.
>>>>>>>>>
>>>>>>>>> The first thing a new contributor wants to do is get the code
>>>> and run
>>>>>>>>> a build, and we do have a quick-start guide linked to on the
>>>> site,
>>>>>> but
>>>>>>>>> I think there is a lot of extra information in there that is not
>>>>>>>>> really relevant to someone just wanting get the code and build.
>>>> We
>>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
>>>> "Build
>>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
>>>>>>>>>
>>>>>>>>> - Clone repo
>>>>>>>>> - checkout master
>>>>>>>>> - run maven
>>>>>>>>> - cd to assembly
>>>>>>>>> - ./bin/nifi.sh
>>>>>>>>>
>>>>>>>>> The next thing they want to do is contribute a change, and we
>>>> have a
>>>>>>>>> great contributor guide, but again I think there could be a very
>>>>>> short
>>>>>>>>> tutorial for the most common steps:
>>>>>>>>>
>>>>>>>>> - fork repo
>>>>>>>>> - clone fork
>>>>>>>>> - create branch
>>>>>>>>> - make changes
>>>>>>>>> - push branch
>>>>>>>>> - submit pr
>>>>>>>>>
>>>>>>>>> and then say something like "for a more detailed description of
>>>> the
>>>>>>>>> contribution process, please reference the Contributor Guide".
>>>>>>>>>
>>>>>>>>> If we then make these getting started guides more prominent
>>>> right in
>>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
>>>> to
>>>>>>>>> find for new community members.
>>>>>>>>>
>>>>>>>>> We can keep extending this idea to other common tasks beyond just
>>>>>>>>> building and contributing.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
>>>> [hidden email]>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi folks,
>>>>>>>>>>
>>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
>>>>>> discuss
>>>>>>>>> with the community what we could do to lower the barrier of
>>>> entry to
>>>>>>>> using
>>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
>>>> both
>>>>>>>>> long-time and newer members, and determine some immediate
>>>> concrete
>>>>>>> steps
>>>>>>>> we
>>>>>>>>> can take.
>>>>>>>>>>
>>>>>>>>>> Problems identified:
>>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
>>>>>>> install”
>>>>>>>>> in project root doesn’t get a new developer up and running
>>>>>> immediately
>>>>>>>>>> * The API is very well defined, but for new contributors, it
>>>> can
>>>>>> be a
>>>>>>>>> challenge to know where to put functionality, and building a
>>>> custom
>>>>>>>>> processor + NAR and deploying isn’t a one-step process
>>>>>>>>>> * Project size (and build size/time) is large. This can
>>>> restrict
>>>>>> the
>>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
>>>>>>>>>> * Some new users do not receive mailing list replies
>>>>>>>>>>
>>>>>>>>>> Possible solutions:
>>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
>>>> working
>>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
>>>>>> default
>>>>>>>>> maven build, change to the target directory, and start NiFi?
>>>>>>>>>> * Individual contributors have written excellent blogs, and
>>>>>>>>> documentation exists, but making it more prominent or more easily
>>>>>>>> accessed
>>>>>>>>> could help?
>>>>>>>>>> * Extension registry will solve all the world’s problems
>>>> (related
>>>>>> to
>>>>>>>>> bundling and build time)
>>>>>>>>>> * Not sure about this one — I don’t know if it’s because
>>>> they’re
>>>>>> not
>>>>>>>>> subscribed, their mail client is blocking them, etc.
>>>>>>>>>>
>>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
>>>>>> members
>>>>>>>> on
>>>>>>>>> their experiences, steps that helped them, and suggestions for
>>>> the
>>>>>>> future
>>>>>>>>> to continue to make the NiFi community welcoming to new users.
>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Andy LoPresto
>>>>>>>>>> [hidden email]
>>>>>>>>>> [hidden email]
>>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
>>>> EF69
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Joe Witt
...we can.  but the discussion is about how to both lower the bar and offer
more routes to the bar.

On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <[hidden email] wrote:

> Why wouldn’t we make the archetypes do this?
>
>
> On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email])
> wrote:
>
> James,
>
> I’m wondering if a page outlining a toy processor (something like
> `CountText` or `ReverseContent`) and doing a line-by-line annotation from a
> developer’s perspective would be helpful. It could be a few sections:
>
> 1. How to get to this point
> * running the maven archetype
> * choosing the directory to install to
> * putting the class name in the manifest file
> * etc.
> 2. The code
> * here’s the class
> * here’s what extending AbstractProcessor gets you, etc. A lot of this is
> currently in the Developer Guide, but in textbook mode
> * here’s how to modify some code (i.e. write one line of Java that switches
> it from straight content pass-through to reversing the text)
> * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> 3. Running, building, installing
> * Run your unit test from the IDE/maven
> * Build the NAR module
> * Install the NAR in NiFi lib/ or custom/
> * Restart NiFi
> * See the NAR loaded in the log
> * Deploy the component on the canvas
>
> I imagine this being written more conversationally/blog-like than most of
> our current reference documentation to be used as a split-screen
> walkthrough. Each section could certainly link to the existing detailed
> documentation for various topics, like the processor lifecycle, etc.
>
> Does this sounds like something that would have helped you?
>
> Andy LoPresto
> [hidden email]
> [hidden email]
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
>
> > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> [hidden email]>
> wrote:
> >
> > 9) Oh, and the wiki is a little hard to navigate and the contents rather
> patchy
> >
> > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > <[hidden email]> wrote:
> >>
> >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> >> realise I could and possibly should submit PRs :)
> >>
> >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> >> it would have been nice it if set up unit tests for me.
> >> 2) The User and Developer documentation is great and comprehensive.
> >> Finding the developer docs is a little painful (handful of items at
> >> the end of a scrolling list of 200+ processors)
> >> 3) The Developer docs could possibly do with a little more clarity on
> >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> >> over the docs, it looks pretty clear now
> >> 4) Some example code for common operations e.g. getting/setting
> >> attributes or reading/writing/modifying flowfile content would be
> >> great.
> >> 5) When using existing processors for inspiration, best practices
> >> weren't always clear e.g. some generated properties inside
> >> getSupportedPropertyDescriptors(), others generated a private static
> >> list on init and returned that. Such differences are inevitable in a
> >> large project, but it would be nice to have something blessed to start
> >> from.
> >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> screen)
> >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> >> the great blog posts by Matt Burgess and others were invaluable
> >> 8) In case this all sounds too negative, NiFi is fab!
> >>
> >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
> >>>
> >>> I am not against the archetype. But we need to spell out every step of
> the
> >>> way. I'd like to see a user thinking about their custom logic ASAP
> rather
> >>> than fighting the tools to get started. Those steps should be
> brain-dead,
> >>> just reflexes, if you know what I mean. Hell, let them create a custom
> >>> processor project or prototype in a script by accident even! :)
> >>>
> >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
> >>>
> >>>> That makes sense about the best practice for deploying to an
> >>>> additional lib directory.
> >>>>
> >>>> So for the project structure you are saying it would be easier to have
> >>>> a repo somewhere with essentially the same thing that is in the
> >>>> archetype, but they just clone it and rename it themselves (what the
> >>>> archetype does for you)?
> >>>>
> >>>> Something that I think would be awesome is if we could provide a
> >>>> web-based project initializer that would essentially run the archetype
> >>>> behind the scenes and then let you download the archive of the code,
> >>>> just like the spring-boot starter [1]. Not sure if their initializr is
> >>>> something that can be re-used and customized [2].
> >>>>
> >>>> The problem is we would need to host that somewhere.
> >>>>
> >>>> [1] https://start.spring.io/
> >>>> [2] https://github.com/spring-io/initializr
> >>>>
> >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]>
> wrote:
> >>>>>
> >>>>> We assume they create new projects from archetypes every day. They
> don't.
> >>>>>
> >>>>> We also assume they know how to deploy new NARs. Most don't.
> Especially
> >>>> if
> >>>>> we want them to follow best practices and create an additional NAR
> >>>> bundles
> >>>>> directory entry im the config (vs dumping into nifi lib).
> >>>>>
> >>>>> I can attest that I feel a bit lost myself every time I need to come
> back
> >>>>> to this and refresh my brain synapses. If we could make these not
> require
> >>>>> any of that and make simple thinga dead simple....
> >>>>>
> >>>>> Andrew
> >>>>>
> >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> >>>>>
> >>>>>> Andrew,
> >>>>>>
> >>>>>> I'm not disagreeing with your points, but I'm curious how you see
> >>>>>> those two ideas being different from the processor archetype and the
> >>>>>> wiki page with the archetype commands?
> >>>>>>
> >>>>>> Is it just that people don't know about it?
> >>>>>>
> >>>>>> -Bryan
> >>>>>>
> >>>>>> [1]
> >>>>>>
> >>>>
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> >>>>>>
> >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> [hidden email]>
>
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I think this ties into my other discuss thread on refreshing the
> >>>>>> archetypes
> >>>>>>>
> >>>>>>>
> >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email]
> )
> >>>>>> wrote:
> >>>>>>>
> >>>>>>> I consistently see my users struggling when they move up the nifi
> >>>> food
> >>>>>>> chain and start looking at custom processors. The good content
> about
> >>>>>>> prototyping processsors via scripting processors and finalizing
> with
> >>>> a
> >>>>>> full
> >>>>>>> NAR bundle is everywhere but where it should be.
> >>>>>>>
> >>>>>>> A few simple changes could help (not *more* docs). They are great,
> >>>> much
> >>>>>>> better than in many other projecta, but people are already drowning
> >>>> in
> >>>>>>> those.
> >>>>>>>
> >>>>>>> How about:
> >>>>>>>
> >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to
> fill
> >>>> in
> >>>>>> is
> >>>>>>> miles better than a blank text area (which invokes a blank stare).
> >>>>>>>
> >>>>>>> + As much as we may loook down on this, but... A simple guide to a
> >>>> full
> >>>>>> NAR
> >>>>>>> build as a series of copy/paste commands.
> >>>>>>>
> >>>>>>> There's more, but this should fit the context for now.
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]
> >
> >>>>>> wrote:
> >>>>>>>
> >>>>>>>> One of the changes we should make is to create a separate guide
> for
> >>>>>>> product
> >>>>>>>> vendors on how to build and maintain a bundle. We're at that point
> >>>>>> where
> >>>>>>>> vendors will have to do it on their own as extension providers, so
> >>>> it
> >>>>>>> would
> >>>>>>>> be very helpful for them to have a simple and straight forward
> >>>> document
> >>>>>>>> showing them what should be there, best practices for
> >>>> maintainability
> >>>>>> and
> >>>>>>>> where to announce it.
> >>>>>>>>
> >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> >>>> wrote:
> >>>>>>>>
> >>>>>>>>> I think we have a lot more documentation than most projects, but
> >>>> I
> >>>>>>>>> think an issue is that content is scattered in many different
> >>>>>>>>> locations, and some of the docs are huge reference guides where
> >>>> it
> >>>>>> can
> >>>>>>>>> be hard to find all the pieces of what you are trying to do.
> >>>>>>>>>
> >>>>>>>>> The first thing a new contributor wants to do is get the code
> >>>> and run
> >>>>>>>>> a build, and we do have a quick-start guide linked to on the
> >>>> site,
> >>>>>> but
> >>>>>>>>> I think there is a lot of extra information in there that is not
> >>>>>>>>> really relevant to someone just wanting get the code and build.
> >>>> We
> >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
> >>>> "Build
> >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> >>>>>>>>>
> >>>>>>>>> - Clone repo
> >>>>>>>>> - checkout master
> >>>>>>>>> - run maven
> >>>>>>>>> - cd to assembly
> >>>>>>>>> - ./bin/nifi.sh
> >>>>>>>>>
> >>>>>>>>> The next thing they want to do is contribute a change, and we
> >>>> have a
> >>>>>>>>> great contributor guide, but again I think there could be a very
> >>>>>> short
> >>>>>>>>> tutorial for the most common steps:
> >>>>>>>>>
> >>>>>>>>> - fork repo
> >>>>>>>>> - clone fork
> >>>>>>>>> - create branch
> >>>>>>>>> - make changes
> >>>>>>>>> - push branch
> >>>>>>>>> - submit pr
> >>>>>>>>>
> >>>>>>>>> and then say something like "for a more detailed description of
> >>>> the
> >>>>>>>>> contribution process, please reference the Contributor Guide".
> >>>>>>>>>
> >>>>>>>>> If we then make these getting started guides more prominent
> >>>> right in
> >>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
> >>>> to
> >>>>>>>>> find for new community members.
> >>>>>>>>>
> >>>>>>>>> We can keep extending this idea to other common tasks beyond just
> >>>>>>>>> building and contributing.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> >>>> [hidden email]>
> >>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi folks,
> >>>>>>>>>>
> >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
> >>>>>> discuss
> >>>>>>>>> with the community what we could do to lower the barrier of
> >>>> entry to
> >>>>>>>> using
> >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
> >>>> both
> >>>>>>>>> long-time and newer members, and determine some immediate
> >>>> concrete
> >>>>>>> steps
> >>>>>>>> we
> >>>>>>>>> can take.
> >>>>>>>>>>
> >>>>>>>>>> Problems identified:
> >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
> >>>>>>> install”
> >>>>>>>>> in project root doesn’t get a new developer up and running
> >>>>>> immediately
> >>>>>>>>>> * The API is very well defined, but for new contributors, it
> >>>> can
> >>>>>> be a
> >>>>>>>>> challenge to know where to put functionality, and building a
> >>>> custom
> >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> >>>>>>>>>> * Project size (and build size/time) is large. This can
> >>>> restrict
> >>>>>> the
> >>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
> >>>>>>>>>> * Some new users do not receive mailing list replies
> >>>>>>>>>>
> >>>>>>>>>> Possible solutions:
> >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> >>>> working
> >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
> >>>>>> default
> >>>>>>>>> maven build, change to the target directory, and start NiFi?
> >>>>>>>>>> * Individual contributors have written excellent blogs, and
> >>>>>>>>> documentation exists, but making it more prominent or more easily
> >>>>>>>> accessed
> >>>>>>>>> could help?
> >>>>>>>>>> * Extension registry will solve all the world’s problems
> >>>> (related
> >>>>>> to
> >>>>>>>>> bundling and build time)
> >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> >>>> they’re
> >>>>>> not
> >>>>>>>>> subscribed, their mail client is blocking them, etc.
> >>>>>>>>>>
> >>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
> >>>>>> members
> >>>>>>>> on
> >>>>>>>>> their experiences, steps that helped them, and suggestions for
> >>>> the
> >>>>>>> future
> >>>>>>>>> to continue to make the NiFi community welcoming to new users.
> >>>>>> Thanks.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> Andy LoPresto
> >>>>>>>>>> [hidden email]
> >>>>>>>>>> [hidden email]
> >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> >>>> EF69
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>
> >>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Bryan Bende
What does everyone think about bumping the "Developer" section of the
docs ahead of "Processors" so that it's easier to find?

Here is what it would look like -
https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f

I also added links to the "Contributor Guide" and the "Maven Projects"
page since I think it would be helpful to make the Developer section
be the one-stop place people look for development help, although I can
see an argument for not mixing wiki content with the docs content.

One issue would be if we want the docs to be fully usable in an
off-line environment, then linking to the wiki won't work, so we could
remove those links, or convert those pages to be part of the docs now
that they are more stable.

For reference, we already have some links in the docs to the wiki:

https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution

On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <[hidden email]> wrote:

>
> ...we can.  but the discussion is about how to both lower the bar and offer
> more routes to the bar.
>
> On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <[hidden email] wrote:
>
> > Why wouldn’t we make the archetypes do this?
> >
> >
> > On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email])
> > wrote:
> >
> > James,
> >
> > I’m wondering if a page outlining a toy processor (something like
> > `CountText` or `ReverseContent`) and doing a line-by-line annotation from a
> > developer’s perspective would be helpful. It could be a few sections:
> >
> > 1. How to get to this point
> > * running the maven archetype
> > * choosing the directory to install to
> > * putting the class name in the manifest file
> > * etc.
> > 2. The code
> > * here’s the class
> > * here’s what extending AbstractProcessor gets you, etc. A lot of this is
> > currently in the Developer Guide, but in textbook mode
> > * here’s how to modify some code (i.e. write one line of Java that switches
> > it from straight content pass-through to reversing the text)
> > * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> > 3. Running, building, installing
> > * Run your unit test from the IDE/maven
> > * Build the NAR module
> > * Install the NAR in NiFi lib/ or custom/
> > * Restart NiFi
> > * See the NAR loaded in the log
> > * Deploy the component on the canvas
> >
> > I imagine this being written more conversationally/blog-like than most of
> > our current reference documentation to be used as a split-screen
> > walkthrough. Each section could certainly link to the existing detailed
> > documentation for various topics, like the processor lifecycle, etc.
> >
> > Does this sounds like something that would have helped you?
> >
> > Andy LoPresto
> > [hidden email]
> > [hidden email]
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> >
> > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > [hidden email]>
> > wrote:
> > >
> > > 9) Oh, and the wiki is a little hard to navigate and the contents rather
> > patchy
> > >
> > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > <[hidden email]> wrote:
> > >>
> > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > >> realise I could and possibly should submit PRs :)
> > >>
> > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> > >> it would have been nice it if set up unit tests for me.
> > >> 2) The User and Developer documentation is great and comprehensive.
> > >> Finding the developer docs is a little painful (handful of items at
> > >> the end of a scrolling list of 200+ processors)
> > >> 3) The Developer docs could possibly do with a little more clarity on
> > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > >> over the docs, it looks pretty clear now
> > >> 4) Some example code for common operations e.g. getting/setting
> > >> attributes or reading/writing/modifying flowfile content would be
> > >> great.
> > >> 5) When using existing processors for inspiration, best practices
> > >> weren't always clear e.g. some generated properties inside
> > >> getSupportedPropertyDescriptors(), others generated a private static
> > >> list on init and returned that. Such differences are inevitable in a
> > >> large project, but it would be nice to have something blessed to start
> > >> from.
> > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > screen)
> > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > >> the great blog posts by Matt Burgess and others were invaluable
> > >> 8) In case this all sounds too negative, NiFi is fab!
> > >>
> > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
> > >>>
> > >>> I am not against the archetype. But we need to spell out every step of
> > the
> > >>> way. I'd like to see a user thinking about their custom logic ASAP
> > rather
> > >>> than fighting the tools to get started. Those steps should be
> > brain-dead,
> > >>> just reflexes, if you know what I mean. Hell, let them create a custom
> > >>> processor project or prototype in a script by accident even! :)
> > >>>
> > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
> > >>>
> > >>>> That makes sense about the best practice for deploying to an
> > >>>> additional lib directory.
> > >>>>
> > >>>> So for the project structure you are saying it would be easier to have
> > >>>> a repo somewhere with essentially the same thing that is in the
> > >>>> archetype, but they just clone it and rename it themselves (what the
> > >>>> archetype does for you)?
> > >>>>
> > >>>> Something that I think would be awesome is if we could provide a
> > >>>> web-based project initializer that would essentially run the archetype
> > >>>> behind the scenes and then let you download the archive of the code,
> > >>>> just like the spring-boot starter [1]. Not sure if their initializr is
> > >>>> something that can be re-used and customized [2].
> > >>>>
> > >>>> The problem is we would need to host that somewhere.
> > >>>>
> > >>>> [1] https://start.spring.io/
> > >>>> [2] https://github.com/spring-io/initializr
> > >>>>
> > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]>
> > wrote:
> > >>>>>
> > >>>>> We assume they create new projects from archetypes every day. They
> > don't.
> > >>>>>
> > >>>>> We also assume they know how to deploy new NARs. Most don't.
> > Especially
> > >>>> if
> > >>>>> we want them to follow best practices and create an additional NAR
> > >>>> bundles
> > >>>>> directory entry im the config (vs dumping into nifi lib).
> > >>>>>
> > >>>>> I can attest that I feel a bit lost myself every time I need to come
> > back
> > >>>>> to this and refresh my brain synapses. If we could make these not
> > require
> > >>>>> any of that and make simple thinga dead simple....
> > >>>>>
> > >>>>> Andrew
> > >>>>>
> > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> > >>>>>
> > >>>>>> Andrew,
> > >>>>>>
> > >>>>>> I'm not disagreeing with your points, but I'm curious how you see
> > >>>>>> those two ideas being different from the processor archetype and the
> > >>>>>> wiki page with the archetype commands?
> > >>>>>>
> > >>>>>> Is it just that people don't know about it?
> > >>>>>>
> > >>>>>> -Bryan
> > >>>>>>
> > >>>>>> [1]
> > >>>>>>
> > >>>>
> >
> > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > >>>>>>
> > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> > [hidden email]>
> >
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> I think this ties into my other discuss thread on refreshing the
> > >>>>>> archetypes
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email]
> > )
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>> I consistently see my users struggling when they move up the nifi
> > >>>> food
> > >>>>>>> chain and start looking at custom processors. The good content
> > about
> > >>>>>>> prototyping processsors via scripting processors and finalizing
> > with
> > >>>> a
> > >>>>>> full
> > >>>>>>> NAR bundle is everywhere but where it should be.
> > >>>>>>>
> > >>>>>>> A few simple changes could help (not *more* docs). They are great,
> > >>>> much
> > >>>>>>> better than in many other projecta, but people are already drowning
> > >>>> in
> > >>>>>>> those.
> > >>>>>>>
> > >>>>>>> How about:
> > >>>>>>>
> > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to
> > fill
> > >>>> in
> > >>>>>> is
> > >>>>>>> miles better than a blank text area (which invokes a blank stare).
> > >>>>>>>
> > >>>>>>> + As much as we may loook down on this, but... A simple guide to a
> > >>>> full
> > >>>>>> NAR
> > >>>>>>> build as a series of copy/paste commands.
> > >>>>>>>
> > >>>>>>> There's more, but this should fit the context for now.
> > >>>>>>>
> > >>>>>>> Andrew
> > >>>>>>>
> > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]
> > >
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> One of the changes we should make is to create a separate guide
> > for
> > >>>>>>> product
> > >>>>>>>> vendors on how to build and maintain a bundle. We're at that point
> > >>>>>> where
> > >>>>>>>> vendors will have to do it on their own as extension providers, so
> > >>>> it
> > >>>>>>> would
> > >>>>>>>> be very helpful for them to have a simple and straight forward
> > >>>> document
> > >>>>>>>> showing them what should be there, best practices for
> > >>>> maintainability
> > >>>>>> and
> > >>>>>>>> where to announce it.
> > >>>>>>>>
> > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> > >>>> wrote:
> > >>>>>>>>
> > >>>>>>>>> I think we have a lot more documentation than most projects, but
> > >>>> I
> > >>>>>>>>> think an issue is that content is scattered in many different
> > >>>>>>>>> locations, and some of the docs are huge reference guides where
> > >>>> it
> > >>>>>> can
> > >>>>>>>>> be hard to find all the pieces of what you are trying to do.
> > >>>>>>>>>
> > >>>>>>>>> The first thing a new contributor wants to do is get the code
> > >>>> and run
> > >>>>>>>>> a build, and we do have a quick-start guide linked to on the
> > >>>> site,
> > >>>>>> but
> > >>>>>>>>> I think there is a lot of extra information in there that is not
> > >>>>>>>>> really relevant to someone just wanting get the code and build.
> > >>>> We
> > >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
> > >>>> "Build
> > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> > >>>>>>>>>
> > >>>>>>>>> - Clone repo
> > >>>>>>>>> - checkout master
> > >>>>>>>>> - run maven
> > >>>>>>>>> - cd to assembly
> > >>>>>>>>> - ./bin/nifi.sh
> > >>>>>>>>>
> > >>>>>>>>> The next thing they want to do is contribute a change, and we
> > >>>> have a
> > >>>>>>>>> great contributor guide, but again I think there could be a very
> > >>>>>> short
> > >>>>>>>>> tutorial for the most common steps:
> > >>>>>>>>>
> > >>>>>>>>> - fork repo
> > >>>>>>>>> - clone fork
> > >>>>>>>>> - create branch
> > >>>>>>>>> - make changes
> > >>>>>>>>> - push branch
> > >>>>>>>>> - submit pr
> > >>>>>>>>>
> > >>>>>>>>> and then say something like "for a more detailed description of
> > >>>> the
> > >>>>>>>>> contribution process, please reference the Contributor Guide".
> > >>>>>>>>>
> > >>>>>>>>> If we then make these getting started guides more prominent
> > >>>> right in
> > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
> > >>>> to
> > >>>>>>>>> find for new community members.
> > >>>>>>>>>
> > >>>>>>>>> We can keep extending this idea to other common tasks beyond just
> > >>>>>>>>> building and contributing.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > >>>> [hidden email]>
> > >>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hi folks,
> > >>>>>>>>>>
> > >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
> > >>>>>> discuss
> > >>>>>>>>> with the community what we could do to lower the barrier of
> > >>>> entry to
> > >>>>>>>> using
> > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
> > >>>> both
> > >>>>>>>>> long-time and newer members, and determine some immediate
> > >>>> concrete
> > >>>>>>> steps
> > >>>>>>>> we
> > >>>>>>>>> can take.
> > >>>>>>>>>>
> > >>>>>>>>>> Problems identified:
> > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
> > >>>>>>> install”
> > >>>>>>>>> in project root doesn’t get a new developer up and running
> > >>>>>> immediately
> > >>>>>>>>>> * The API is very well defined, but for new contributors, it
> > >>>> can
> > >>>>>> be a
> > >>>>>>>>> challenge to know where to put functionality, and building a
> > >>>> custom
> > >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> > >>>>>>>>>> * Project size (and build size/time) is large. This can
> > >>>> restrict
> > >>>>>> the
> > >>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
> > >>>>>>>>>> * Some new users do not receive mailing list replies
> > >>>>>>>>>>
> > >>>>>>>>>> Possible solutions:
> > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> > >>>> working
> > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
> > >>>>>> default
> > >>>>>>>>> maven build, change to the target directory, and start NiFi?
> > >>>>>>>>>> * Individual contributors have written excellent blogs, and
> > >>>>>>>>> documentation exists, but making it more prominent or more easily
> > >>>>>>>> accessed
> > >>>>>>>>> could help?
> > >>>>>>>>>> * Extension registry will solve all the world’s problems
> > >>>> (related
> > >>>>>> to
> > >>>>>>>>> bundling and build time)
> > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> > >>>> they’re
> > >>>>>> not
> > >>>>>>>>> subscribed, their mail client is blocking them, etc.
> > >>>>>>>>>>
> > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
> > >>>>>> members
> > >>>>>>>> on
> > >>>>>>>>> their experiences, steps that helped them, and suggestions for
> > >>>> the
> > >>>>>>> future
> > >>>>>>>>> to continue to make the NiFi community welcoming to new users.
> > >>>>>> Thanks.
> > >>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>> Andy LoPresto
> > >>>>>>>>>> [hidden email]
> > >>>>>>>>>> [hidden email]
> > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > >>>> EF69
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>
> >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Andrew Grande-2
+1 it's so obvious it should've always been there :)

As for offline access. Well, every time I showed an offline help section to
every user thry were startled :) tells about awareness. The first hunch is
always to google insread of going to a locally hosted doc, so it won't be
an issue, IMO.

Andrew

On Mon, Jan 28, 2019, 8:13 AM Bryan Bende <[hidden email]> wrote:

> What does everyone think about bumping the "Developer" section of the
> docs ahead of "Processors" so that it's easier to find?
>
> Here is what it would look like -
> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>
> I also added links to the "Contributor Guide" and the "Maven Projects"
> page since I think it would be helpful to make the Developer section
> be the one-stop place people look for development help, although I can
> see an argument for not mixing wiki content with the docs content.
>
> One issue would be if we want the docs to be fully usable in an
> off-line environment, then linking to the wiki won't work, so we could
> remove those links, or convert those pages to be part of the docs now
> that they are more stable.
>
> For reference, we already have some links in the docs to the wiki:
>
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
>
> On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <[hidden email]> wrote:
> >
> > ...we can.  but the discussion is about how to both lower the bar and
> offer
> > more routes to the bar.
> >
> > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <[hidden email]
> wrote:
> >
> > > Why wouldn’t we make the archetypes do this?
> > >
> > >
> > > On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email])
> > > wrote:
> > >
> > > James,
> > >
> > > I’m wondering if a page outlining a toy processor (something like
> > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> from a
> > > developer’s perspective would be helpful. It could be a few sections:
> > >
> > > 1. How to get to this point
> > > * running the maven archetype
> > > * choosing the directory to install to
> > > * putting the class name in the manifest file
> > > * etc.
> > > 2. The code
> > > * here’s the class
> > > * here’s what extending AbstractProcessor gets you, etc. A lot of this
> is
> > > currently in the Developer Guide, but in textbook mode
> > > * here’s how to modify some code (i.e. write one line of Java that
> switches
> > > it from straight content pass-through to reversing the text)
> > > * here’s how to make a unit test (introduce the TestRunner framework,
> etc.)
> > > 3. Running, building, installing
> > > * Run your unit test from the IDE/maven
> > > * Build the NAR module
> > > * Install the NAR in NiFi lib/ or custom/
> > > * Restart NiFi
> > > * See the NAR loaded in the log
> > > * Deploy the component on the canvas
> > >
> > > I imagine this being written more conversationally/blog-like than most
> of
> > > our current reference documentation to be used as a split-screen
> > > walkthrough. Each section could certainly link to the existing detailed
> > > documentation for various topics, like the processor lifecycle, etc.
> > >
> > > Does this sounds like something that would have helped you?
> > >
> > > Andy LoPresto
> > > [hidden email]
> > > [hidden email]
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > [hidden email]>
> > > wrote:
> > > >
> > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> rather
> > > patchy
> > > >
> > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > <[hidden email]> wrote:
> > > >>
> > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > >> realise I could and possibly should submit PRs :)
> > > >>
> > > >> 1) I'm used to Java and Maven, so used the archetype. It worked
> fine,
> > > >> it would have been nice it if set up unit tests for me.
> > > >> 2) The User and Developer documentation is great and comprehensive.
> > > >> Finding the developer docs is a little painful (handful of items at
> > > >> the end of a scrolling list of 200+ processors)
> > > >> 3) The Developer docs could possibly do with a little more clarity
> on
> > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > >> over the docs, it looks pretty clear now
> > > >> 4) Some example code for common operations e.g. getting/setting
> > > >> attributes or reading/writing/modifying flowfile content would be
> > > >> great.
> > > >> 5) When using existing processors for inspiration, best practices
> > > >> weren't always clear e.g. some generated properties inside
> > > >> getSupportedPropertyDescriptors(), others generated a private static
> > > >> list on init and returned that. Such differences are inevitable in a
> > > >> large project, but it would be nice to have something blessed to
> start
> > > >> from.
> > > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > > screen)
> > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > > >> the great blog posts by Matt Burgess and others were invaluable
> > > >> 8) In case this all sounds too negative, NiFi is fab!
> > > >>
> > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]>
> wrote:
> > > >>>
> > > >>> I am not against the archetype. But we need to spell out every
> step of
> > > the
> > > >>> way. I'd like to see a user thinking about their custom logic ASAP
> > > rather
> > > >>> than fighting the tools to get started. Those steps should be
> > > brain-dead,
> > > >>> just reflexes, if you know what I mean. Hell, let them create a
> custom
> > > >>> processor project or prototype in a script by accident even! :)
> > > >>>
> > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]>
> wrote:
> > > >>>
> > > >>>> That makes sense about the best practice for deploying to an
> > > >>>> additional lib directory.
> > > >>>>
> > > >>>> So for the project structure you are saying it would be easier to
> have
> > > >>>> a repo somewhere with essentially the same thing that is in the
> > > >>>> archetype, but they just clone it and rename it themselves (what
> the
> > > >>>> archetype does for you)?
> > > >>>>
> > > >>>> Something that I think would be awesome is if we could provide a
> > > >>>> web-based project initializer that would essentially run the
> archetype
> > > >>>> behind the scenes and then let you download the archive of the
> code,
> > > >>>> just like the spring-boot starter [1]. Not sure if their
> initializr is
> > > >>>> something that can be re-used and customized [2].
> > > >>>>
> > > >>>> The problem is we would need to host that somewhere.
> > > >>>>
> > > >>>> [1] https://start.spring.io/
> > > >>>> [2] https://github.com/spring-io/initializr
> > > >>>>
> > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <
> [hidden email]>
> > > wrote:
> > > >>>>>
> > > >>>>> We assume they create new projects from archetypes every day.
> They
> > > don't.
> > > >>>>>
> > > >>>>> We also assume they know how to deploy new NARs. Most don't.
> > > Especially
> > > >>>> if
> > > >>>>> we want them to follow best practices and create an additional
> NAR
> > > >>>> bundles
> > > >>>>> directory entry im the config (vs dumping into nifi lib).
> > > >>>>>
> > > >>>>> I can attest that I feel a bit lost myself every time I need to
> come
> > > back
> > > >>>>> to this and refresh my brain synapses. If we could make these not
> > > require
> > > >>>>> any of that and make simple thinga dead simple....
> > > >>>>>
> > > >>>>> Andrew
> > > >>>>>
> > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]>
> wrote:
> > > >>>>>
> > > >>>>>> Andrew,
> > > >>>>>>
> > > >>>>>> I'm not disagreeing with your points, but I'm curious how you
> see
> > > >>>>>> those two ideas being different from the processor archetype
> and the
> > > >>>>>> wiki page with the archetype commands?
> > > >>>>>>
> > > >>>>>> Is it just that people don't know about it?
> > > >>>>>>
> > > >>>>>> -Bryan
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>
> > >
> > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > >>>>>>
> > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> > > [hidden email]>
> > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I think this ties into my other discuss thread on refreshing
> the
> > > >>>>>> archetypes
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande (
> [hidden email]
> > > )
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I consistently see my users struggling when they move up the
> nifi
> > > >>>> food
> > > >>>>>>> chain and start looking at custom processors. The good content
> > > about
> > > >>>>>>> prototyping processsors via scripting processors and finalizing
> > > with
> > > >>>> a
> > > >>>>>> full
> > > >>>>>>> NAR bundle is everywhere but where it should be.
> > > >>>>>>>
> > > >>>>>>> A few simple changes could help (not *more* docs). They are
> great,
> > > >>>> much
> > > >>>>>>> better than in many other projecta, but people are already
> drowning
> > > >>>> in
> > > >>>>>>> those.
> > > >>>>>>>
> > > >>>>>>> How about:
> > > >>>>>>>
> > > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to
> > > fill
> > > >>>> in
> > > >>>>>> is
> > > >>>>>>> miles better than a blank text area (which invokes a blank
> stare).
> > > >>>>>>>
> > > >>>>>>> + As much as we may loook down on this, but... A simple guide
> to a
> > > >>>> full
> > > >>>>>> NAR
> > > >>>>>>> build as a series of copy/paste commands.
> > > >>>>>>>
> > > >>>>>>> There's more, but this should fit the context for now.
> > > >>>>>>>
> > > >>>>>>> Andrew
> > > >>>>>>>
> > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <
> [hidden email]
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> One of the changes we should make is to create a separate
> guide
> > > for
> > > >>>>>>> product
> > > >>>>>>>> vendors on how to build and maintain a bundle. We're at that
> point
> > > >>>>>> where
> > > >>>>>>>> vendors will have to do it on their own as extension
> providers, so
> > > >>>> it
> > > >>>>>>> would
> > > >>>>>>>> be very helpful for them to have a simple and straight forward
> > > >>>> document
> > > >>>>>>>> showing them what should be there, best practices for
> > > >>>> maintainability
> > > >>>>>> and
> > > >>>>>>>> where to announce it.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]
> >
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I think we have a lot more documentation than most projects,
> but
> > > >>>> I
> > > >>>>>>>>> think an issue is that content is scattered in many different
> > > >>>>>>>>> locations, and some of the docs are huge reference guides
> where
> > > >>>> it
> > > >>>>>> can
> > > >>>>>>>>> be hard to find all the pieces of what you are trying to do.
> > > >>>>>>>>>
> > > >>>>>>>>> The first thing a new contributor wants to do is get the code
> > > >>>> and run
> > > >>>>>>>>> a build, and we do have a quick-start guide linked to on the
> > > >>>> site,
> > > >>>>>> but
> > > >>>>>>>>> I think there is a lot of extra information in there that is
> not
> > > >>>>>>>>> really relevant to someone just wanting get the code and
> build.
> > > >>>> We
> > > >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
> > > >>>> "Build
> > > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > >>>>>>>>>
> > > >>>>>>>>> - Clone repo
> > > >>>>>>>>> - checkout master
> > > >>>>>>>>> - run maven
> > > >>>>>>>>> - cd to assembly
> > > >>>>>>>>> - ./bin/nifi.sh
> > > >>>>>>>>>
> > > >>>>>>>>> The next thing they want to do is contribute a change, and we
> > > >>>> have a
> > > >>>>>>>>> great contributor guide, but again I think there could be a
> very
> > > >>>>>> short
> > > >>>>>>>>> tutorial for the most common steps:
> > > >>>>>>>>>
> > > >>>>>>>>> - fork repo
> > > >>>>>>>>> - clone fork
> > > >>>>>>>>> - create branch
> > > >>>>>>>>> - make changes
> > > >>>>>>>>> - push branch
> > > >>>>>>>>> - submit pr
> > > >>>>>>>>>
> > > >>>>>>>>> and then say something like "for a more detailed description
> of
> > > >>>> the
> > > >>>>>>>>> contribution process, please reference the Contributor
> Guide".
> > > >>>>>>>>>
> > > >>>>>>>>> If we then make these getting started guides more prominent
> > > >>>> right in
> > > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be
> easier
> > > >>>> to
> > > >>>>>>>>> find for new community members.
> > > >>>>>>>>>
> > > >>>>>>>>> We can keep extending this idea to other common tasks beyond
> just
> > > >>>>>>>>> building and contributing.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > > >>>> [hidden email]>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi folks,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted
> to
> > > >>>>>> discuss
> > > >>>>>>>>> with the community what we could do to lower the barrier of
> > > >>>> entry to
> > > >>>>>>>> using
> > > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
> > > >>>> both
> > > >>>>>>>>> long-time and newer members, and determine some immediate
> > > >>>> concrete
> > > >>>>>>> steps
> > > >>>>>>>> we
> > > >>>>>>>>> can take.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Problems identified:
> > > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn
> clean
> > > >>>>>>> install”
> > > >>>>>>>>> in project root doesn’t get a new developer up and running
> > > >>>>>> immediately
> > > >>>>>>>>>> * The API is very well defined, but for new contributors, it
> > > >>>> can
> > > >>>>>> be a
> > > >>>>>>>>> challenge to know where to put functionality, and building a
> > > >>>> custom
> > > >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> > > >>>>>>>>>> * Project size (and build size/time) is large. This can
> > > >>>> restrict
> > > >>>>>> the
> > > >>>>>>>>> minimum hardware necessary, elongate the development cycle,
> etc.
> > > >>>>>>>>>> * Some new users do not receive mailing list replies
> > > >>>>>>>>>>
> > > >>>>>>>>>> Possible solutions:
> > > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> > > >>>> working
> > > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle
> the
> > > >>>>>> default
> > > >>>>>>>>> maven build, change to the target directory, and start NiFi?
> > > >>>>>>>>>> * Individual contributors have written excellent blogs, and
> > > >>>>>>>>> documentation exists, but making it more prominent or more
> easily
> > > >>>>>>>> accessed
> > > >>>>>>>>> could help?
> > > >>>>>>>>>> * Extension registry will solve all the world’s problems
> > > >>>> (related
> > > >>>>>> to
> > > >>>>>>>>> bundling and build time)
> > > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> > > >>>> they’re
> > > >>>>>> not
> > > >>>>>>>>> subscribed, their mail client is blocking them, etc.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other
> community
> > > >>>>>> members
> > > >>>>>>>> on
> > > >>>>>>>>> their experiences, steps that helped them, and suggestions
> for
> > > >>>> the
> > > >>>>>>> future
> > > >>>>>>>>> to continue to make the NiFi community welcoming to new
> users.
> > > >>>>>> Thanks.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Andy LoPresto
> > > >>>>>>>>>> [hidden email]
> > > >>>>>>>>>> [hidden email]
> > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B
> 2F7D
> > > >>>> EF69
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > >
>
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

James Srinivasan
In reply to this post by Bryan Bende
How about separating out User/Developer/Admin into separate docs?

On Mon, 28 Jan 2019 at 16:13, Bryan Bende <[hidden email]> wrote:

>
> What does everyone think about bumping the "Developer" section of the
> docs ahead of "Processors" so that it's easier to find?
>
> Here is what it would look like -
> https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
>
> I also added links to the "Contributor Guide" and the "Maven Projects"
> page since I think it would be helpful to make the Developer section
> be the one-stop place people look for development help, although I can
> see an argument for not mixing wiki content with the docs content.
>
> One issue would be if we want the docs to be fully usable in an
> off-line environment, then linking to the wiki won't work, so we could
> remove those links, or convert those pages to be part of the docs now
> that they are more stable.
>
> For reference, we already have some links in the docs to the wiki:
>
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
>
> On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <[hidden email]> wrote:
> >
> > ...we can.  but the discussion is about how to both lower the bar and offer
> > more routes to the bar.
> >
> > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <[hidden email] wrote:
> >
> > > Why wouldn’t we make the archetypes do this?
> > >
> > >
> > > On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email])
> > > wrote:
> > >
> > > James,
> > >
> > > I’m wondering if a page outlining a toy processor (something like
> > > `CountText` or `ReverseContent`) and doing a line-by-line annotation from a
> > > developer’s perspective would be helpful. It could be a few sections:
> > >
> > > 1. How to get to this point
> > > * running the maven archetype
> > > * choosing the directory to install to
> > > * putting the class name in the manifest file
> > > * etc.
> > > 2. The code
> > > * here’s the class
> > > * here’s what extending AbstractProcessor gets you, etc. A lot of this is
> > > currently in the Developer Guide, but in textbook mode
> > > * here’s how to modify some code (i.e. write one line of Java that switches
> > > it from straight content pass-through to reversing the text)
> > > * here’s how to make a unit test (introduce the TestRunner framework, etc.)
> > > 3. Running, building, installing
> > > * Run your unit test from the IDE/maven
> > > * Build the NAR module
> > > * Install the NAR in NiFi lib/ or custom/
> > > * Restart NiFi
> > > * See the NAR loaded in the log
> > > * Deploy the component on the canvas
> > >
> > > I imagine this being written more conversationally/blog-like than most of
> > > our current reference documentation to be used as a split-screen
> > > walkthrough. Each section could certainly link to the existing detailed
> > > documentation for various topics, like the processor lifecycle, etc.
> > >
> > > Does this sounds like something that would have helped you?
> > >
> > > Andy LoPresto
> > > [hidden email]
> > > [hidden email]
> > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > >
> > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > [hidden email]>
> > > wrote:
> > > >
> > > > 9) Oh, and the wiki is a little hard to navigate and the contents rather
> > > patchy
> > > >
> > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > <[hidden email]> wrote:
> > > >>
> > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > >> realise I could and possibly should submit PRs :)
> > > >>
> > > >> 1) I'm used to Java and Maven, so used the archetype. It worked fine,
> > > >> it would have been nice it if set up unit tests for me.
> > > >> 2) The User and Developer documentation is great and comprehensive.
> > > >> Finding the developer docs is a little painful (handful of items at
> > > >> the end of a scrolling list of 200+ processors)
> > > >> 3) The Developer docs could possibly do with a little more clarity on
> > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > >> over the docs, it looks pretty clear now
> > > >> 4) Some example code for common operations e.g. getting/setting
> > > >> attributes or reading/writing/modifying flowfile content would be
> > > >> great.
> > > >> 5) When using existing processors for inspiration, best practices
> > > >> weren't always clear e.g. some generated properties inside
> > > >> getSupportedPropertyDescriptors(), others generated a private static
> > > >> list on init and returned that. Such differences are inevitable in a
> > > >> large project, but it would be nice to have something blessed to start
> > > >> from.
> > > >> 6) (Minor niggle - layout of the docs doesn't work great on a phone
> > > screen)
> > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API, but
> > > >> the great blog posts by Matt Burgess and others were invaluable
> > > >> 8) In case this all sounds too negative, NiFi is fab!
> > > >>
> > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]> wrote:
> > > >>>
> > > >>> I am not against the archetype. But we need to spell out every step of
> > > the
> > > >>> way. I'd like to see a user thinking about their custom logic ASAP
> > > rather
> > > >>> than fighting the tools to get started. Those steps should be
> > > brain-dead,
> > > >>> just reflexes, if you know what I mean. Hell, let them create a custom
> > > >>> processor project or prototype in a script by accident even! :)
> > > >>>
> > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]> wrote:
> > > >>>
> > > >>>> That makes sense about the best practice for deploying to an
> > > >>>> additional lib directory.
> > > >>>>
> > > >>>> So for the project structure you are saying it would be easier to have
> > > >>>> a repo somewhere with essentially the same thing that is in the
> > > >>>> archetype, but they just clone it and rename it themselves (what the
> > > >>>> archetype does for you)?
> > > >>>>
> > > >>>> Something that I think would be awesome is if we could provide a
> > > >>>> web-based project initializer that would essentially run the archetype
> > > >>>> behind the scenes and then let you download the archive of the code,
> > > >>>> just like the spring-boot starter [1]. Not sure if their initializr is
> > > >>>> something that can be re-used and customized [2].
> > > >>>>
> > > >>>> The problem is we would need to host that somewhere.
> > > >>>>
> > > >>>> [1] https://start.spring.io/
> > > >>>> [2] https://github.com/spring-io/initializr
> > > >>>>
> > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <[hidden email]>
> > > wrote:
> > > >>>>>
> > > >>>>> We assume they create new projects from archetypes every day. They
> > > don't.
> > > >>>>>
> > > >>>>> We also assume they know how to deploy new NARs. Most don't.
> > > Especially
> > > >>>> if
> > > >>>>> we want them to follow best practices and create an additional NAR
> > > >>>> bundles
> > > >>>>> directory entry im the config (vs dumping into nifi lib).
> > > >>>>>
> > > >>>>> I can attest that I feel a bit lost myself every time I need to come
> > > back
> > > >>>>> to this and refresh my brain synapses. If we could make these not
> > > require
> > > >>>>> any of that and make simple thinga dead simple....
> > > >>>>>
> > > >>>>> Andrew
> > > >>>>>
> > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]> wrote:
> > > >>>>>
> > > >>>>>> Andrew,
> > > >>>>>>
> > > >>>>>> I'm not disagreeing with your points, but I'm curious how you see
> > > >>>>>> those two ideas being different from the processor archetype and the
> > > >>>>>> wiki page with the archetype commands?
> > > >>>>>>
> > > >>>>>> Is it just that people don't know about it?
> > > >>>>>>
> > > >>>>>> -Bryan
> > > >>>>>>
> > > >>>>>> [1]
> > > >>>>>>
> > > >>>>
> > >
> > > https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > >>>>>>
> > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> > > [hidden email]>
> > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I think this ties into my other discuss thread on refreshing the
> > > >>>>>> archetypes
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande ([hidden email]
> > > )
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I consistently see my users struggling when they move up the nifi
> > > >>>> food
> > > >>>>>>> chain and start looking at custom processors. The good content
> > > about
> > > >>>>>>> prototyping processsors via scripting processors and finalizing
> > > with
> > > >>>> a
> > > >>>>>> full
> > > >>>>>>> NAR bundle is everywhere but where it should be.
> > > >>>>>>>
> > > >>>>>>> A few simple changes could help (not *more* docs). They are great,
> > > >>>> much
> > > >>>>>>> better than in many other projecta, but people are already drowning
> > > >>>> in
> > > >>>>>>> those.
> > > >>>>>>>
> > > >>>>>>> How about:
> > > >>>>>>>
> > > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op to
> > > fill
> > > >>>> in
> > > >>>>>> is
> > > >>>>>>> miles better than a blank text area (which invokes a blank stare).
> > > >>>>>>>
> > > >>>>>>> + As much as we may loook down on this, but... A simple guide to a
> > > >>>> full
> > > >>>>>> NAR
> > > >>>>>>> build as a series of copy/paste commands.
> > > >>>>>>>
> > > >>>>>>> There's more, but this should fit the context for now.
> > > >>>>>>>
> > > >>>>>>> Andrew
> > > >>>>>>>
> > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <[hidden email]
> > > >
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>> One of the changes we should make is to create a separate guide
> > > for
> > > >>>>>>> product
> > > >>>>>>>> vendors on how to build and maintain a bundle. We're at that point
> > > >>>>>> where
> > > >>>>>>>> vendors will have to do it on their own as extension providers, so
> > > >>>> it
> > > >>>>>>> would
> > > >>>>>>>> be very helpful for them to have a simple and straight forward
> > > >>>> document
> > > >>>>>>>> showing them what should be there, best practices for
> > > >>>> maintainability
> > > >>>>>> and
> > > >>>>>>>> where to announce it.
> > > >>>>>>>>
> > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <[hidden email]>
> > > >>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>>> I think we have a lot more documentation than most projects, but
> > > >>>> I
> > > >>>>>>>>> think an issue is that content is scattered in many different
> > > >>>>>>>>> locations, and some of the docs are huge reference guides where
> > > >>>> it
> > > >>>>>> can
> > > >>>>>>>>> be hard to find all the pieces of what you are trying to do.
> > > >>>>>>>>>
> > > >>>>>>>>> The first thing a new contributor wants to do is get the code
> > > >>>> and run
> > > >>>>>>>>> a build, and we do have a quick-start guide linked to on the
> > > >>>> site,
> > > >>>>>> but
> > > >>>>>>>>> I think there is a lot of extra information in there that is not
> > > >>>>>>>>> really relevant to someone just wanting get the code and build.
> > > >>>> We
> > > >>>>>>>>> could have separate guides per OS like "Build NiFi on Linux",
> > > >>>> "Build
> > > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > >>>>>>>>>
> > > >>>>>>>>> - Clone repo
> > > >>>>>>>>> - checkout master
> > > >>>>>>>>> - run maven
> > > >>>>>>>>> - cd to assembly
> > > >>>>>>>>> - ./bin/nifi.sh
> > > >>>>>>>>>
> > > >>>>>>>>> The next thing they want to do is contribute a change, and we
> > > >>>> have a
> > > >>>>>>>>> great contributor guide, but again I think there could be a very
> > > >>>>>> short
> > > >>>>>>>>> tutorial for the most common steps:
> > > >>>>>>>>>
> > > >>>>>>>>> - fork repo
> > > >>>>>>>>> - clone fork
> > > >>>>>>>>> - create branch
> > > >>>>>>>>> - make changes
> > > >>>>>>>>> - push branch
> > > >>>>>>>>> - submit pr
> > > >>>>>>>>>
> > > >>>>>>>>> and then say something like "for a more detailed description of
> > > >>>> the
> > > >>>>>>>>> contribution process, please reference the Contributor Guide".
> > > >>>>>>>>>
> > > >>>>>>>>> If we then make these getting started guides more prominent
> > > >>>> right in
> > > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be easier
> > > >>>> to
> > > >>>>>>>>> find for new community members.
> > > >>>>>>>>>
> > > >>>>>>>>> We can keep extending this idea to other common tasks beyond just
> > > >>>>>>>>> building and contributing.
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > > >>>> [hidden email]>
> > > >>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi folks,
> > > >>>>>>>>>>
> > > >>>>>>>>>> Based on some recent (and long-term) experiences, I wanted to
> > > >>>>>> discuss
> > > >>>>>>>>> with the community what we could do to lower the barrier of
> > > >>>> entry to
> > > >>>>>>>> using
> > > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback from
> > > >>>> both
> > > >>>>>>>>> long-time and newer members, and determine some immediate
> > > >>>> concrete
> > > >>>>>>> steps
> > > >>>>>>>> we
> > > >>>>>>>>> can take.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Problems identified:
> > > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn clean
> > > >>>>>>> install”
> > > >>>>>>>>> in project root doesn’t get a new developer up and running
> > > >>>>>> immediately
> > > >>>>>>>>>> * The API is very well defined, but for new contributors, it
> > > >>>> can
> > > >>>>>> be a
> > > >>>>>>>>> challenge to know where to put functionality, and building a
> > > >>>> custom
> > > >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> > > >>>>>>>>>> * Project size (and build size/time) is large. This can
> > > >>>> restrict
> > > >>>>>> the
> > > >>>>>>>>> minimum hardware necessary, elongate the development cycle, etc.
> > > >>>>>>>>>> * Some new users do not receive mailing list replies
> > > >>>>>>>>>>
> > > >>>>>>>>>> Possible solutions:
> > > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> > > >>>> working
> > > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to handle the
> > > >>>>>> default
> > > >>>>>>>>> maven build, change to the target directory, and start NiFi?
> > > >>>>>>>>>> * Individual contributors have written excellent blogs, and
> > > >>>>>>>>> documentation exists, but making it more prominent or more easily
> > > >>>>>>>> accessed
> > > >>>>>>>>> could help?
> > > >>>>>>>>>> * Extension registry will solve all the world’s problems
> > > >>>> (related
> > > >>>>>> to
> > > >>>>>>>>> bundling and build time)
> > > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> > > >>>> they’re
> > > >>>>>> not
> > > >>>>>>>>> subscribed, their mail client is blocking them, etc.
> > > >>>>>>>>>>
> > > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other community
> > > >>>>>> members
> > > >>>>>>>> on
> > > >>>>>>>>> their experiences, steps that helped them, and suggestions for
> > > >>>> the
> > > >>>>>>> future
> > > >>>>>>>>> to continue to make the NiFi community welcoming to new users.
> > > >>>>>> Thanks.
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Andy LoPresto
> > > >>>>>>>>>> [hidden email]
> > > >>>>>>>>>> [hidden email]
> > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D
> > > >>>> EF69
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>
> > > >>>>
> > >
Reply | Threaded
Open this post in threaded view
|

Re: Lowering the barrier of entry

Bryan Bende
Currently it’s broken into General and Developer, so were you thinking of
splitting General into User and Admin?

On Mon, Jan 28, 2019 at 11:34 AM James Srinivasan <
[hidden email]> wrote:

> How about separating out User/Developer/Admin into separate docs?
>
> On Mon, 28 Jan 2019 at 16:13, Bryan Bende <[hidden email]> wrote:
> >
> > What does everyone think about bumping the "Developer" section of the
> > docs ahead of "Processors" so that it's easier to find?
> >
> > Here is what it would look like -
> > https://gist.github.com/bbende/279c983f5c80d4fad1431ae81862060f
> >
> > I also added links to the "Contributor Guide" and the "Maven Projects"
> > page since I think it would be helpful to make the Developer section
> > be the one-stop place people look for development help, although I can
> > see an argument for not mixing wiki content with the docs content.
> >
> > One issue would be if we want the docs to be fully usable in an
> > off-line environment, then linking to the wiki won't work, so we could
> > remove those links, or convert those pages to be part of the docs now
> > that they are more stable.
> >
> > For reference, we already have some links in the docs to the wiki:
> >
> >
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#supplying-a-contribution
> >
> > On Sat, Jan 26, 2019 at 10:49 AM Joe Witt <[hidden email]> wrote:
> > >
> > > ...we can.  but the discussion is about how to both lower the bar and
> offer
> > > more routes to the bar.
> > >
> > > On Sat, Jan 26, 2019, 10:45 AM Otto Fowler <[hidden email]
> wrote:
> > >
> > > > Why wouldn’t we make the archetypes do this?
> > > >
> > > >
> > > > On January 25, 2019 at 18:04:25, Andy LoPresto ([hidden email]
> )
> > > > wrote:
> > > >
> > > > James,
> > > >
> > > > I’m wondering if a page outlining a toy processor (something like
> > > > `CountText` or `ReverseContent`) and doing a line-by-line annotation
> from a
> > > > developer’s perspective would be helpful. It could be a few sections:
> > > >
> > > > 1. How to get to this point
> > > > * running the maven archetype
> > > > * choosing the directory to install to
> > > > * putting the class name in the manifest file
> > > > * etc.
> > > > 2. The code
> > > > * here’s the class
> > > > * here’s what extending AbstractProcessor gets you, etc. A lot of
> this is
> > > > currently in the Developer Guide, but in textbook mode
> > > > * here’s how to modify some code (i.e. write one line of Java that
> switches
> > > > it from straight content pass-through to reversing the text)
> > > > * here’s how to make a unit test (introduce the TestRunner
> framework, etc.)
> > > > 3. Running, building, installing
> > > > * Run your unit test from the IDE/maven
> > > > * Build the NAR module
> > > > * Install the NAR in NiFi lib/ or custom/
> > > > * Restart NiFi
> > > > * See the NAR loaded in the log
> > > > * Deploy the component on the canvas
> > > >
> > > > I imagine this being written more conversationally/blog-like than
> most of
> > > > our current reference documentation to be used as a split-screen
> > > > walkthrough. Each section could certainly link to the existing
> detailed
> > > > documentation for various topics, like the processor lifecycle, etc.
> > > >
> > > > Does this sounds like something that would have helped you?
> > > >
> > > > Andy LoPresto
> > > > [hidden email]
> > > > [hidden email]
> > > > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> > > >
> > > > > On Jan 25, 2019, at 1:59 PM, James Srinivasan <
> > > > [hidden email]>
> > > > wrote:
> > > > >
> > > > > 9) Oh, and the wiki is a little hard to navigate and the contents
> rather
> > > > patchy
> > > > >
> > > > > On Fri, 25 Jan 2019 at 21:57, James Srinivasan
> > > > > <[hidden email]> wrote:
> > > > >>
> > > > >> As someone relatively new to NiFi dev, here's my £0.02. (Yes, I
> > > > >> realise I could and possibly should submit PRs :)
> > > > >>
> > > > >> 1) I'm used to Java and Maven, so used the archetype. It worked
> fine,
> > > > >> it would have been nice it if set up unit tests for me.
> > > > >> 2) The User and Developer documentation is great and
> comprehensive.
> > > > >> Finding the developer docs is a little painful (handful of items
> at
> > > > >> the end of a scrolling list of 200+ processors)
> > > > >> 3) The Developer docs could possibly do with a little more
> clarity on
> > > > >> processor lifetime i.e. what is called when ^h^h^h - skimming back
> > > > >> over the docs, it looks pretty clear now
> > > > >> 4) Some example code for common operations e.g. getting/setting
> > > > >> attributes or reading/writing/modifying flowfile content would be
> > > > >> great.
> > > > >> 5) When using existing processors for inspiration, best practices
> > > > >> weren't always clear e.g. some generated properties inside
> > > > >> getSupportedPropertyDescriptors(), others generated a private
> static
> > > > >> list on init and returned that. Such differences are inevitable
> in a
> > > > >> large project, but it would be nice to have something blessed to
> start
> > > > >> from.
> > > > >> 6) (Minor niggle - layout of the docs doesn't work great on a
> phone
> > > > screen)
> > > > >> 7) I couldn't find (m?)any docs about the Groovy scripting API,
> but
> > > > >> the great blog posts by Matt Burgess and others were invaluable
> > > > >> 8) In case this all sounds too negative, NiFi is fab!
> > > > >>
> > > > >> On Fri, 25 Jan 2019 at 18:47, Andrew Grande <[hidden email]>
> wrote:
> > > > >>>
> > > > >>> I am not against the archetype. But we need to spell out every
> step of
> > > > the
> > > > >>> way. I'd like to see a user thinking about their custom logic
> ASAP
> > > > rather
> > > > >>> than fighting the tools to get started. Those steps should be
> > > > brain-dead,
> > > > >>> just reflexes, if you know what I mean. Hell, let them create a
> custom
> > > > >>> processor project or prototype in a script by accident even! :)
> > > > >>>
> > > > >>> On Fri, Jan 25, 2019, 10:43 AM Bryan Bende <[hidden email]>
> wrote:
> > > > >>>
> > > > >>>> That makes sense about the best practice for deploying to an
> > > > >>>> additional lib directory.
> > > > >>>>
> > > > >>>> So for the project structure you are saying it would be easier
> to have
> > > > >>>> a repo somewhere with essentially the same thing that is in the
> > > > >>>> archetype, but they just clone it and rename it themselves
> (what the
> > > > >>>> archetype does for you)?
> > > > >>>>
> > > > >>>> Something that I think would be awesome is if we could provide a
> > > > >>>> web-based project initializer that would essentially run the
> archetype
> > > > >>>> behind the scenes and then let you download the archive of the
> code,
> > > > >>>> just like the spring-boot starter [1]. Not sure if their
> initializr is
> > > > >>>> something that can be re-used and customized [2].
> > > > >>>>
> > > > >>>> The problem is we would need to host that somewhere.
> > > > >>>>
> > > > >>>> [1] https://start.spring.io/
> > > > >>>> [2] https://github.com/spring-io/initializr
> > > > >>>>
> > > > >>>> On Fri, Jan 25, 2019 at 12:56 PM Andrew Grande <
> [hidden email]>
> > > > wrote:
> > > > >>>>>
> > > > >>>>> We assume they create new projects from archetypes every day.
> They
> > > > don't.
> > > > >>>>>
> > > > >>>>> We also assume they know how to deploy new NARs. Most don't.
> > > > Especially
> > > > >>>> if
> > > > >>>>> we want them to follow best practices and create an additional
> NAR
> > > > >>>> bundles
> > > > >>>>> directory entry im the config (vs dumping into nifi lib).
> > > > >>>>>
> > > > >>>>> I can attest that I feel a bit lost myself every time I need
> to come
> > > > back
> > > > >>>>> to this and refresh my brain synapses. If we could make these
> not
> > > > require
> > > > >>>>> any of that and make simple thinga dead simple....
> > > > >>>>>
> > > > >>>>> Andrew
> > > > >>>>>
> > > > >>>>> On Fri, Jan 25, 2019, 9:47 AM Bryan Bende <[hidden email]>
> wrote:
> > > > >>>>>
> > > > >>>>>> Andrew,
> > > > >>>>>>
> > > > >>>>>> I'm not disagreeing with your points, but I'm curious how you
> see
> > > > >>>>>> those two ideas being different from the processor archetype
> and the
> > > > >>>>>> wiki page with the archetype commands?
> > > > >>>>>>
> > > > >>>>>> Is it just that people don't know about it?
> > > > >>>>>>
> > > > >>>>>> -Bryan
> > > > >>>>>>
> > > > >>>>>> [1]
> > > > >>>>>>
> > > > >>>>
> > > >
> > > >
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
> > > > >>>>>>
> > > > >>>>>> On Fri, Jan 25, 2019 at 12:23 PM Otto Fowler <
> > > > [hidden email]>
> > > >
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> I think this ties into my other discuss thread on refreshing
> the
> > > > >>>>>> archetypes
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> On January 25, 2019 at 11:50:10, Andrew Grande (
> [hidden email]
> > > > )
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> I consistently see my users struggling when they move up the
> nifi
> > > > >>>> food
> > > > >>>>>>> chain and start looking at custom processors. The good
> content
> > > > about
> > > > >>>>>>> prototyping processsors via scripting processors and
> finalizing
> > > > with
> > > > >>>> a
> > > > >>>>>> full
> > > > >>>>>>> NAR bundle is everywhere but where it should be.
> > > > >>>>>>>
> > > > >>>>>>> A few simple changes could help (not *more* docs). They are
> great,
> > > > >>>> much
> > > > >>>>>>> better than in many other projecta, but people are already
> drowning
> > > > >>>> in
> > > > >>>>>>> those.
> > > > >>>>>>>
> > > > >>>>>>> How about:
> > > > >>>>>>>
> > > > >>>>>>> + ISP has a pre-populated processor sceleton. A simple no-op
> to
> > > > fill
> > > > >>>> in
> > > > >>>>>> is
> > > > >>>>>>> miles better than a blank text area (which invokes a blank
> stare).
> > > > >>>>>>>
> > > > >>>>>>> + As much as we may loook down on this, but... A simple
> guide to a
> > > > >>>> full
> > > > >>>>>> NAR
> > > > >>>>>>> build as a series of copy/paste commands.
> > > > >>>>>>>
> > > > >>>>>>> There's more, but this should fit the context for now.
> > > > >>>>>>>
> > > > >>>>>>> Andrew
> > > > >>>>>>>
> > > > >>>>>>> On Fri, Jan 25, 2019, 8:13 AM Mike Thomsen <
> [hidden email]
> > > > >
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> One of the changes we should make is to create a separate
> guide
> > > > for
> > > > >>>>>>> product
> > > > >>>>>>>> vendors on how to build and maintain a bundle. We're at
> that point
> > > > >>>>>> where
> > > > >>>>>>>> vendors will have to do it on their own as extension
> providers, so
> > > > >>>> it
> > > > >>>>>>> would
> > > > >>>>>>>> be very helpful for them to have a simple and straight
> forward
> > > > >>>> document
> > > > >>>>>>>> showing them what should be there, best practices for
> > > > >>>> maintainability
> > > > >>>>>> and
> > > > >>>>>>>> where to announce it.
> > > > >>>>>>>>
> > > > >>>>>>>> On Fri, Jan 25, 2019 at 9:59 AM Bryan Bende <
> [hidden email]>
> > > > >>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>>> I think we have a lot more documentation than most
> projects, but
> > > > >>>> I
> > > > >>>>>>>>> think an issue is that content is scattered in many
> different
> > > > >>>>>>>>> locations, and some of the docs are huge reference guides
> where
> > > > >>>> it
> > > > >>>>>> can
> > > > >>>>>>>>> be hard to find all the pieces of what you are trying to
> do.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The first thing a new contributor wants to do is get the
> code
> > > > >>>> and run
> > > > >>>>>>>>> a build, and we do have a quick-start guide linked to on
> the
> > > > >>>> site,
> > > > >>>>>> but
> > > > >>>>>>>>> I think there is a lot of extra information in there that
> is not
> > > > >>>>>>>>> really relevant to someone just wanting get the code and
> build.
> > > > >>>> We
> > > > >>>>>>>>> could have separate guides per OS like "Build NiFi on
> Linux",
> > > > >>>> "Build
> > > > >>>>>>>>> NiFi on Windows", etc, where each guide was 4-5 steps like:
> > > > >>>>>>>>>
> > > > >>>>>>>>> - Clone repo
> > > > >>>>>>>>> - checkout master
> > > > >>>>>>>>> - run maven
> > > > >>>>>>>>> - cd to assembly
> > > > >>>>>>>>> - ./bin/nifi.sh
> > > > >>>>>>>>>
> > > > >>>>>>>>> The next thing they want to do is contribute a change, and
> we
> > > > >>>> have a
> > > > >>>>>>>>> great contributor guide, but again I think there could be
> a very
> > > > >>>>>> short
> > > > >>>>>>>>> tutorial for the most common steps:
> > > > >>>>>>>>>
> > > > >>>>>>>>> - fork repo
> > > > >>>>>>>>> - clone fork
> > > > >>>>>>>>> - create branch
> > > > >>>>>>>>> - make changes
> > > > >>>>>>>>> - push branch
> > > > >>>>>>>>> - submit pr
> > > > >>>>>>>>>
> > > > >>>>>>>>> and then say something like "for a more detailed
> description of
> > > > >>>> the
> > > > >>>>>>>>> contribution process, please reference the Contributor
> Guide".
> > > > >>>>>>>>>
> > > > >>>>>>>>> If we then make these getting started guides more prominent
> > > > >>>> right in
> > > > >>>>>>>>> the middle of the NiFi homepage, then maybe they will be
> easier
> > > > >>>> to
> > > > >>>>>>>>> find for new community members.
> > > > >>>>>>>>>
> > > > >>>>>>>>> We can keep extending this idea to other common tasks
> beyond just
> > > > >>>>>>>>> building and contributing.
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Thu, Jan 24, 2019 at 8:03 PM Andy LoPresto <
> > > > >>>> [hidden email]>
> > > > >>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Hi folks,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Based on some recent (and long-term) experiences, I
> wanted to
> > > > >>>>>> discuss
> > > > >>>>>>>>> with the community what we could do to lower the barrier of
> > > > >>>> entry to
> > > > >>>>>>>> using
> > > > >>>>>>>>> & contributing to NiFi. I hope to get some good feedback
> from
> > > > >>>> both
> > > > >>>>>>>>> long-time and newer members, and determine some immediate
> > > > >>>> concrete
> > > > >>>>>>> steps
> > > > >>>>>>>> we
> > > > >>>>>>>>> can take.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Problems identified:
> > > > >>>>>>>>>> * NiFi has a number of custom profiles, so a simple “mvn
> clean
> > > > >>>>>>> install”
> > > > >>>>>>>>> in project root doesn’t get a new developer up and running
> > > > >>>>>> immediately
> > > > >>>>>>>>>> * The API is very well defined, but for new contributors,
> it
> > > > >>>> can
> > > > >>>>>> be a
> > > > >>>>>>>>> challenge to know where to put functionality, and building
> a
> > > > >>>> custom
> > > > >>>>>>>>> processor + NAR and deploying isn’t a one-step process
> > > > >>>>>>>>>> * Project size (and build size/time) is large. This can
> > > > >>>> restrict
> > > > >>>>>> the
> > > > >>>>>>>>> minimum hardware necessary, elongate the development
> cycle, etc.
> > > > >>>>>>>>>> * Some new users do not receive mailing list replies
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Possible solutions:
> > > > >>>>>>>>>> * On a clean git clone, “mvn clean install” should build a
> > > > >>>> working
> > > > >>>>>>>>> instance. Maybe we provide a quickstart.sh script to
> handle the
> > > > >>>>>> default
> > > > >>>>>>>>> maven build, change to the target directory, and start
> NiFi?
> > > > >>>>>>>>>> * Individual contributors have written excellent blogs,
> and
> > > > >>>>>>>>> documentation exists, but making it more prominent or more
> easily
> > > > >>>>>>>> accessed
> > > > >>>>>>>>> could help?
> > > > >>>>>>>>>> * Extension registry will solve all the world’s problems
> > > > >>>> (related
> > > > >>>>>> to
> > > > >>>>>>>>> bundling and build time)
> > > > >>>>>>>>>> * Not sure about this one — I don’t know if it’s because
> > > > >>>> they’re
> > > > >>>>>> not
> > > > >>>>>>>>> subscribed, their mail client is blocking them, etc.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> I’ve said my bit, now I am eager to hear from other
> community
> > > > >>>>>> members
> > > > >>>>>>>> on
> > > > >>>>>>>>> their experiences, steps that helped them, and suggestions
> for
> > > > >>>> the
> > > > >>>>>>> future
> > > > >>>>>>>>> to continue to make the NiFi community welcoming to new
> users.
> > > > >>>>>> Thanks.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Andy LoPresto
> > > > >>>>>>>>>> [hidden email]
> > > > >>>>>>>>>> [hidden email]
> > > > >>>>>>>>>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B
> 2F7D
> > > > >>>> EF69
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > >
>
--
Sent from Gmail Mobile
12