Compiling custom processor

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Compiling custom processor

John McGinn
Ok, I am far from a maven expert, and am struggling on this problem.

I created a new project using the maven generate process, and compiled that sample processor and everything was fine, and I could see it in my local NiFi instance. I then copied over the Wait processor on top of the MyProcessor.java class, changed the package name, and the class name, and attempted to compile. Got errors due to dependency issues. This is where I get confused.

In the processor directory, I modify the pom.xml with a dependency line for nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and nifi-standard-processors. This seemed to make things work, and I get a 33 meg NAR file, which contains lots of JARs. I load that up, and I get a problem with CalculateStats, or similar, because of RecordReaderFactory. I go back in and include nifi-record-serialization-services, and it compiles, and the JAR is included in the NAR file now, and I still get class not found.

I've also tried to change the dependencies to a status of provided, and therefore get a minimal 25k or so NAR file with no JARs included. I get the same issue with the class not found.

I had also attempted to copy Wait.java to AnotherWait.java within the actually NiFi source code, and it compiles, and shows up in the JARs and NARs as I'd expect, but I cannot get to it within my NiFi instance.

Can someone let me know what I'm doing wrong.

Thanks,
John McGinn
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

Bryan Bende
In general, if your processor uses a controller service then the
processors pom file needs a provided dependency on the API of the CS,
and your NAR pom needs a NAR dependency on the NAR where the CS API
is.

Example is shown here in the section linking processors and controller
services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions

Also, you do not want to include NiFI's standard processors in your
own NAR, this will result in two copies of every standard processor.
You will want to copy whatever code you need into your own NAR.

In you last statement, if you made a new processor in
nifi-standard-processors and it didn't show up, it is most likely
because you didn't update META-INF/services file to include the new
processor.
On Tue, Oct 16, 2018 at 9:36 AM John McGinn
<[hidden email]> wrote:

>
> Ok, I am far from a maven expert, and am struggling on this problem.
>
> I created a new project using the maven generate process, and compiled that sample processor and everything was fine, and I could see it in my local NiFi instance. I then copied over the Wait processor on top of the MyProcessor.java class, changed the package name, and the class name, and attempted to compile. Got errors due to dependency issues. This is where I get confused.
>
> In the processor directory, I modify the pom.xml with a dependency line for nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and nifi-standard-processors. This seemed to make things work, and I get a 33 meg NAR file, which contains lots of JARs. I load that up, and I get a problem with CalculateStats, or similar, because of RecordReaderFactory. I go back in and include nifi-record-serialization-services, and it compiles, and the JAR is included in the NAR file now, and I still get class not found.
>
> I've also tried to change the dependencies to a status of provided, and therefore get a minimal 25k or so NAR file with no JARs included. I get the same issue with the class not found.
>
> I had also attempted to copy Wait.java to AnotherWait.java within the actually NiFi source code, and it compiles, and shows up in the JARs and NARs as I'd expect, but I cannot get to it within my NiFi instance.
>
> Can someone let me know what I'm doing wrong.
>
> Thanks,
> John McGinn
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

Mike Thomsen
If you want to look at a large Maven project that builds several custom
NARs from one unified build system, take a look at my data generation
bundle here:

https://github.com/MikeThomsen/nifi-datageneration-bundle

On Tue, Oct 16, 2018 at 9:53 AM Bryan Bende <[hidden email]> wrote:

> In general, if your processor uses a controller service then the
> processors pom file needs a provided dependency on the API of the CS,
> and your NAR pom needs a NAR dependency on the NAR where the CS API
> is.
>
> Example is shown here in the section linking processors and controller
> services -
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
> Also, you do not want to include NiFI's standard processors in your
> own NAR, this will result in two copies of every standard processor.
> You will want to copy whatever code you need into your own NAR.
>
> In you last statement, if you made a new processor in
> nifi-standard-processors and it didn't show up, it is most likely
> because you didn't update META-INF/services file to include the new
> processor.
> On Tue, Oct 16, 2018 at 9:36 AM John McGinn
> <[hidden email]> wrote:
> >
> > Ok, I am far from a maven expert, and am struggling on this problem.
> >
> > I created a new project using the maven generate process, and compiled
> that sample processor and everything was fine, and I could see it in my
> local NiFi instance. I then copied over the Wait processor on top of the
> MyProcessor.java class, changed the package name, and the class name, and
> attempted to compile. Got errors due to dependency issues. This is where I
> get confused.
> >
> > In the processor directory, I modify the pom.xml with a dependency line
> for nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and
> nifi-standard-processors. This seemed to make things work, and I get a 33
> meg NAR file, which contains lots of JARs. I load that up, and I get a
> problem with CalculateStats, or similar, because of RecordReaderFactory. I
> go back in and include nifi-record-serialization-services, and it compiles,
> and the JAR is included in the NAR file now, and I still get class not
> found.
> >
> > I've also tried to change the dependencies to a status of provided, and
> therefore get a minimal 25k or so NAR file with no JARs included. I get the
> same issue with the class not found.
> >
> > I had also attempted to copy Wait.java to AnotherWait.java within the
> actually NiFi source code, and it compiles, and shows up in the JARs and
> NARs as I'd expect, but I cannot get to it within my NiFi instance.
> >
> > Can someone let me know what I'm doing wrong.
> >
> > Thanks,
> > John McGinn
>
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

John McGinn-2
In reply to this post by John McGinn
Thanks Bryan,

I restarted with a fresh maven generate, and only added the API to the processor pom, and then the API-NAR to the NAR pom. I then figured out a subsequent dependency (lang3), and got that in, and now my processor shows up in local NiFi instance.

Regarding the copying of the standard processor, when I copied Wait over, I renamed it from Wait to MyProcessorWait to be unique, and not copied. I'm just attempting changes to Wait without modifying the actual one. Also, thanks on the META-INF/services file information.

Now I can debug and test.

Sincerely,
John McGinn

--------------------------------------------
On Tue, 10/16/18, Bryan Bende <[hidden email]> wrote:

 Subject: Re: Compiling custom processor
 To: [hidden email]
 Date: Tuesday, October 16, 2018, 9:52 AM
 
 In general, if your processor
 uses a controller service then the
 processors pom file needs a provided dependency
 on the API of the CS,
 and your NAR pom needs
 a NAR dependency on the NAR where the CS API
 is.
 
 Example is
 shown here in the section linking processors and
 controller
 services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 
 Also, you do not want to
 include NiFI's standard processors in your
 own NAR, this will result in two copies of
 every standard processor.
 You will want to
 copy whatever code you need into your own NAR.
 
 In you last statement, if you
 made a new processor in
 nifi-standard-processors and it didn't show
 up, it is most likely
 because you didn't
 update META-INF/services file to include the new
 processor.
 On Tue, Oct 16, 2018 at
 9:36 AM John McGinn
 <[hidden email]>
 wrote:
 >
 > Ok, I am
 far from a maven expert, and am struggling on this
 problem.
 >
 > I created
 a new project using the maven generate process, and compiled
 that sample processor and everything was fine, and I could
 see it in my local NiFi instance. I then copied over the
 Wait processor on top of the MyProcessor.java class, changed
 the package name, and the class name, and attempted to
 compile. Got errors due to dependency issues. This is where
 I get confused.
 >
 > In
 the processor directory, I modify the pom.xml with a
 dependency line for nifi-api, nifi-utils,
 nifi-distribute-cache-client-service-api and
 nifi-standard-processors. This seemed to make things work,
 and I get a 33 meg NAR file, which contains lots of JARs. I
 load that up, and I get a problem with CalculateStats, or
 similar, because of RecordReaderFactory. I go back in and
 include nifi-record-serialization-services, and it compiles,
 and the JAR is included in the NAR file now, and I still get
 class not found.
 >
 >
 I've also tried to change the dependencies to a status
 of provided, and therefore get a minimal 25k or so NAR file
 with no JARs included. I get the same issue with the class
 not found.
 >
 > I had
 also attempted to copy Wait.java to AnotherWait.java within
 the actually NiFi source code, and it compiles, and shows up
 in the JARs and NARs as I'd expect, but I cannot get to
 it within my NiFi instance.
 >
 > Can someone let me know what I'm doing
 wrong.
 >
 > Thanks,
 > John McGinn
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

John McGinn-2
In reply to this post by John McGinn
And I'm back.

So, I had my processor running, no errors, so things were good.

I then add 2 if statements with logger messages, run the same mvn command, copied the new NAR on top of the old one, same versionId and all, and now I get a class not found error, that didn't occur before I stopped, redeployed and started. And I can't delete the event in the queue even though source and destination are stopped.

Is there problems when redeploying a custom processor NAR on top of itself? Even if the NiFi instance is stopped? Serialization issues?

Frustratedly yours,
John McGinn

--------------------------------------------
On Tue, 10/16/18, John McGinn <[hidden email]> wrote:

 Subject: Re: Compiling custom processor
 To: [hidden email]
 Date: Tuesday, October 16, 2018, 1:21 PM
 
 Thanks Bryan,
 
 I restarted with a fresh maven
 generate, and only added the API to the processor pom, and
 then the API-NAR to the NAR pom. I then figured out a
 subsequent dependency (lang3), and got that in, and now my
 processor shows up in local NiFi instance.
 
 Regarding the copying of the standard
 processor, when I copied Wait over, I renamed it from Wait
 to MyProcessorWait to be unique, and not copied. I'm just
 attempting changes to Wait without modifying the actual one.
 Also, thanks on the META-INF/services file information.
 
 Now I can debug and test.
 
 Sincerely,
 John McGinn
 
 --------------------------------------------
 On Tue, 10/16/18, Bryan Bende <[hidden email]>
 wrote:
 
  Subject: Re: Compiling custom
 processor
  To: [hidden email]
  Date: Tuesday, October 16, 2018, 9:52
 AM
 
  In general, if your processor
  uses a controller service then the
  processors pom file needs a provided
 dependency
  on the API of the CS,
  and your NAR pom needs
  a NAR dependency on the NAR where the
 CS API
  is.
 
  Example is
  shown here in the section linking
 processors and
  controller
  services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 
  Also, you do not want to
  include NiFI's standard processors in
 your
  own NAR, this will result in two
 copies of
  every standard processor.
  You will want to
  copy whatever code you need into your
 own NAR.
 
  In you last statement, if you
  made a new processor in
  nifi-standard-processors and it didn't
 show
  up, it is most likely
  because you didn't
  update META-INF/services file to
 include the new
  processor.
  On Tue, Oct 16, 2018 at
  9:36 AM John McGinn
  <[hidden email]>
  wrote:
  >
  > Ok, I am
  far from a maven expert, and am
 struggling on this
  problem.
  >
  > I created
  a new project using the maven generate
 process, and compiled
  that sample processor and everything
 was fine, and I could
  see it in my local NiFi instance. I
 then copied over the
  Wait processor on top of the
 MyProcessor.java class, changed
  the package name, and the class name,
 and attempted to
  compile. Got errors due to dependency
 issues. This is where
  I get confused.
  >
  > In
  the processor directory, I modify the
 pom.xml with a
  dependency line for nifi-api,
 nifi-utils,
 
 nifi-distribute-cache-client-service-api and
  nifi-standard-processors. This seemed
 to make things work,
  and I get a 33 meg NAR file, which
 contains lots of JARs. I
  load that up, and I get a problem with
 CalculateStats, or
  similar, because of
 RecordReaderFactory. I go back in and
  include
 nifi-record-serialization-services, and it compiles,
  and the JAR is included in the NAR
 file now, and I still get
  class not found.
  >
  >
  I've also tried to change the
 dependencies to a status
  of provided, and therefore get a
 minimal 25k or so NAR file
  with no JARs included. I get the same
 issue with the class
  not found.
  >
  > I had
  also attempted to copy Wait.java to
 AnotherWait.java within
  the actually NiFi source code, and it
 compiles, and shows up
  in the JARs and NARs as I'd expect,
 but I cannot get to
  it within my NiFi instance.
  >
  > Can someone let me know what I'm
 doing
  wrong.
  >
  > Thanks,
  > John McGinn
 
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

Bryan Bende
There shouldn't be any issue doing that, generally you just copy the
new NAR into the lib directory and restart.

What were the specifics of the class not found error?

On Tue, Oct 16, 2018 at 3:24 PM John McGinn <[hidden email]> wrote:

>
> And I'm back.
>
> So, I had my processor running, no errors, so things were good.
>
> I then add 2 if statements with logger messages, run the same mvn command, copied the new NAR on top of the old one, same versionId and all, and now I get a class not found error, that didn't occur before I stopped, redeployed and started. And I can't delete the event in the queue even though source and destination are stopped.
>
> Is there problems when redeploying a custom processor NAR on top of itself? Even if the NiFi instance is stopped? Serialization issues?
>
> Frustratedly yours,
> John McGinn
>
> --------------------------------------------
> On Tue, 10/16/18, John McGinn <[hidden email]> wrote:
>
>  Subject: Re: Compiling custom processor
>  To: [hidden email]
>  Date: Tuesday, October 16, 2018, 1:21 PM
>
>  Thanks Bryan,
>
>  I restarted with a fresh maven
>  generate, and only added the API to the processor pom, and
>  then the API-NAR to the NAR pom. I then figured out a
>  subsequent dependency (lang3), and got that in, and now my
>  processor shows up in local NiFi instance.
>
>  Regarding the copying of the standard
>  processor, when I copied Wait over, I renamed it from Wait
>  to MyProcessorWait to be unique, and not copied. I'm just
>  attempting changes to Wait without modifying the actual one.
>  Also, thanks on the META-INF/services file information.
>
>  Now I can debug and test.
>
>  Sincerely,
>  John McGinn
>
>  --------------------------------------------
>  On Tue, 10/16/18, Bryan Bende <[hidden email]>
>  wrote:
>
>   Subject: Re: Compiling custom
>  processor
>   To: [hidden email]
>   Date: Tuesday, October 16, 2018, 9:52
>  AM
>
>   In general, if your processor
>   uses a controller service then the
>   processors pom file needs a provided
>  dependency
>   on the API of the CS,
>   and your NAR pom needs
>   a NAR dependency on the NAR where the
>  CS API
>   is.
>
>   Example is
>   shown here in the section linking
>  processors and
>   controller
>   services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
>   Also, you do not want to
>   include NiFI's standard processors in
>  your
>   own NAR, this will result in two
>  copies of
>   every standard processor.
>   You will want to
>   copy whatever code you need into your
>  own NAR.
>
>   In you last statement, if you
>   made a new processor in
>   nifi-standard-processors and it didn't
>  show
>   up, it is most likely
>   because you didn't
>   update META-INF/services file to
>  include the new
>   processor.
>   On Tue, Oct 16, 2018 at
>   9:36 AM John McGinn
>   <[hidden email]>
>   wrote:
>   >
>   > Ok, I am
>   far from a maven expert, and am
>  struggling on this
>   problem.
>   >
>   > I created
>   a new project using the maven generate
>  process, and compiled
>   that sample processor and everything
>  was fine, and I could
>   see it in my local NiFi instance. I
>  then copied over the
>   Wait processor on top of the
>  MyProcessor.java class, changed
>   the package name, and the class name,
>  and attempted to
>   compile. Got errors due to dependency
>  issues. This is where
>   I get confused.
>   >
>   > In
>   the processor directory, I modify the
>  pom.xml with a
>   dependency line for nifi-api,
>  nifi-utils,
>
>  nifi-distribute-cache-client-service-api and
>   nifi-standard-processors. This seemed
>  to make things work,
>   and I get a 33 meg NAR file, which
>  contains lots of JARs. I
>   load that up, and I get a problem with
>  CalculateStats, or
>   similar, because of
>  RecordReaderFactory. I go back in and
>   include
>  nifi-record-serialization-services, and it compiles,
>   and the JAR is included in the NAR
>  file now, and I still get
>   class not found.
>   >
>   >
>   I've also tried to change the
>  dependencies to a status
>   of provided, and therefore get a
>  minimal 25k or so NAR file
>   with no JARs included. I get the same
>  issue with the class
>   not found.
>   >
>   > I had
>   also attempted to copy Wait.java to
>  AnotherWait.java within
>   the actually NiFi source code, and it
>  compiles, and shows up
>   in the JARs and NARs as I'd expect,
>  but I cannot get to
>   it within my NiFi instance.
>   >
>   > Can someone let me know what I'm
>  doing
>   wrong.
>   >
>   > Thanks,
>   > John McGinn
>
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

John McGinn-2
In reply to this post by John McGinn
I have a directory /home/user/nifi-nars, which is listed in the nifi.properties file located under the nifi-assembly directory from the nifi-master code. That's where I redropped my NAR file at.

The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils". I added a provided dependency of version 3.4 for commons-lang3 in my processors pom.xml. That worked initially, but with the redropping of the NAR file, the error occurs. The stacktrace points to the session.get() lambda function trying to use StringUtils.isBlank(). (Again, this is a copy and renamed class of the Wait Processor. Trying to turn it into a Yield, as it were, hold unless it's free to move forward.)
--------------------------------------------
On Tue, 10/16/18, Bryan Bende <[hidden email]> wrote:

 Subject: Re: Compiling custom processor
 To: [hidden email]
 Date: Tuesday, October 16, 2018, 3:34 PM
 
 There shouldn't be any issue
 doing that, generally you just copy the
 new
 NAR into the lib directory and restart.
 
 What were the specifics of the class not found
 error?
 
 On Tue, Oct 16, 2018 at
 3:24 PM John McGinn <[hidden email]>
 wrote:
 >
 > And I'm
 back.
 >
 > So, I had my
 processor running, no errors, so things were good.
 >
 > I then add 2 if
 statements with logger messages, run the same mvn command,
 copied the new NAR on top of the old one, same versionId and
 all, and now I get a class not found error, that didn't
 occur before I stopped, redeployed and started. And I
 can't delete the event in the queue even though source
 and destination are stopped.
 >
 > Is there problems when redeploying a
 custom processor NAR on top of itself? Even if the NiFi
 instance is stopped? Serialization issues?
 >
 > Frustratedly
 yours,
 > John McGinn
 >
 >
 --------------------------------------------
 > On Tue, 10/16/18, John McGinn <[hidden email]>
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: [hidden email]
 >  Date: Tuesday, October 16, 2018, 1:21
 PM
 >
 >  Thanks
 Bryan,
 >
 >  I
 restarted with a fresh maven
 > 
 generate, and only added the API to the processor pom,
 and
 >  then the API-NAR to the NAR pom.
 I then figured out a
 >  subsequent
 dependency (lang3), and got that in, and now my
 >  processor shows up in local NiFi
 instance.
 >
 > 
 Regarding the copying of the standard
 > 
 processor, when I copied Wait over, I renamed it from
 Wait
 >  to MyProcessorWait to be unique,
 and not copied. I'm just
 > 
 attempting changes to Wait without modifying the actual
 one.
 >  Also, thanks on the
 META-INF/services file information.
 >
 >  Now I can debug and test.
 >
 >  Sincerely,
 >  John McGinn
 >
 > 
 --------------------------------------------
 >  On Tue, 10/16/18, Bryan Bende <[hidden email]>
 >  wrote:
 >
 >   Subject: Re: Compiling custom
 >  processor
 >   To: [hidden email]
 >   Date: Tuesday, October 16, 2018,
 9:52
 >  AM
 >
 >   In general, if your processor
 >   uses a controller service then the
 >   processors pom file needs a provided
 >  dependency
 >   on
 the API of the CS,
 >   and your NAR pom
 needs
 >   a NAR dependency on the NAR
 where the
 >  CS API
 >   is.
 >
 >   Example is
 > 
 shown here in the section linking
 > 
 processors and
 >   controller
 >   services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 >
 >   Also, you do not
 want to
 >   include NiFI's standard
 processors in
 >  your
 >   own NAR, this will result in two
 >  copies of
 >   every
 standard processor.
 >   You will want
 to
 >   copy whatever code you need into
 your
 >  own NAR.
 >
 >   In you last statement, if you
 >   made a new processor in
 >   nifi-standard-processors and it
 didn't
 >  show
 >   up, it is most likely
 >   because you didn't
 >   update META-INF/services file to
 >  include the new
 > 
 processor.
 >   On Tue, Oct 16, 2018
 at
 >   9:36 AM John McGinn
 >   <[hidden email]>
 >   wrote:
 >   >
 >   > Ok, I am
 > 
 far from a maven expert, and am
 > 
 struggling on this
 >   problem.
 >   >
 >   > I
 created
 >   a new project using the
 maven generate
 >  process, and
 compiled
 >   that sample processor and
 everything
 >  was fine, and I could
 >   see it in my local NiFi instance. I
 >  then copied over the
 >   Wait processor on top of the
 >  MyProcessor.java class, changed
 >   the package name, and the class
 name,
 >  and attempted to
 >   compile. Got errors due to
 dependency
 >  issues. This is where
 >   I get confused.
 > 
 >
 >   > In
 > 
 the processor directory, I modify the
 > 
 pom.xml with a
 >   dependency line for
 nifi-api,
 >  nifi-utils,
 >
 > 
 nifi-distribute-cache-client-service-api and
 >   nifi-standard-processors. This
 seemed
 >  to make things work,
 >   and I get a 33 meg NAR file, which
 >  contains lots of JARs. I
 >   load that up, and I get a problem
 with
 >  CalculateStats, or
 >   similar, because of
 >  RecordReaderFactory. I go back in and
 >   include
 > 
 nifi-record-serialization-services, and it compiles,
 >   and the JAR is included in the NAR
 >  file now, and I still get
 >   class not found.
 > 
  >
 >   >
 > 
 I've also tried to change the
 > 
 dependencies to a status
 >   of
 provided, and therefore get a
 >  minimal
 25k or so NAR file
 >   with no JARs
 included. I get the same
 >  issue with
 the class
 >   not found.
 >   >
 >   > I
 had
 >   also attempted to copy Wait.java
 to
 >  AnotherWait.java within
 >   the actually NiFi source code, and
 it
 >  compiles, and shows up
 >   in the JARs and NARs as I'd
 expect,
 >  but I cannot get to
 >   it within my NiFi instance.
 >   >
 >   > Can
 someone let me know what I'm
 > 
 doing
 >   wrong.
 > 
 >
 >   > Thanks,
 >   > John McGinn
 >
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

Bryan Bende
Any reason why the commons-lang3 dependency is provided and not normal
compile scope?

Generally the only cases where a dependency should be provided are
when a parent NAR provides it (the API scenario earlier), or when it
is a jar directly in lib which means it is on the classpath of
everything (like nifi-api or slf4j).
On Tue, Oct 16, 2018 at 3:47 PM John McGinn <[hidden email]> wrote:

>
> I have a directory /home/user/nifi-nars, which is listed in the nifi.properties file located under the nifi-assembly directory from the nifi-master code. That's where I redropped my NAR file at.
>
> The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils". I added a provided dependency of version 3.4 for commons-lang3 in my processors pom.xml. That worked initially, but with the redropping of the NAR file, the error occurs. The stacktrace points to the session.get() lambda function trying to use StringUtils.isBlank(). (Again, this is a copy and renamed class of the Wait Processor. Trying to turn it into a Yield, as it were, hold unless it's free to move forward.)
> --------------------------------------------
> On Tue, 10/16/18, Bryan Bende <[hidden email]> wrote:
>
>  Subject: Re: Compiling custom processor
>  To: [hidden email]
>  Date: Tuesday, October 16, 2018, 3:34 PM
>
>  There shouldn't be any issue
>  doing that, generally you just copy the
>  new
>  NAR into the lib directory and restart.
>
>  What were the specifics of the class not found
>  error?
>
>  On Tue, Oct 16, 2018 at
>  3:24 PM John McGinn <[hidden email]>
>  wrote:
>  >
>  > And I'm
>  back.
>  >
>  > So, I had my
>  processor running, no errors, so things were good.
>  >
>  > I then add 2 if
>  statements with logger messages, run the same mvn command,
>  copied the new NAR on top of the old one, same versionId and
>  all, and now I get a class not found error, that didn't
>  occur before I stopped, redeployed and started. And I
>  can't delete the event in the queue even though source
>  and destination are stopped.
>  >
>  > Is there problems when redeploying a
>  custom processor NAR on top of itself? Even if the NiFi
>  instance is stopped? Serialization issues?
>  >
>  > Frustratedly
>  yours,
>  > John McGinn
>  >
>  >
>  --------------------------------------------
>  > On Tue, 10/16/18, John McGinn <[hidden email]>
>  wrote:
>  >
>  >  Subject:
>  Re: Compiling custom processor
>  >  To: [hidden email]
>  >  Date: Tuesday, October 16, 2018, 1:21
>  PM
>  >
>  >  Thanks
>  Bryan,
>  >
>  >  I
>  restarted with a fresh maven
>  >
>  generate, and only added the API to the processor pom,
>  and
>  >  then the API-NAR to the NAR pom.
>  I then figured out a
>  >  subsequent
>  dependency (lang3), and got that in, and now my
>  >  processor shows up in local NiFi
>  instance.
>  >
>  >
>  Regarding the copying of the standard
>  >
>  processor, when I copied Wait over, I renamed it from
>  Wait
>  >  to MyProcessorWait to be unique,
>  and not copied. I'm just
>  >
>  attempting changes to Wait without modifying the actual
>  one.
>  >  Also, thanks on the
>  META-INF/services file information.
>  >
>  >  Now I can debug and test.
>  >
>  >  Sincerely,
>  >  John McGinn
>  >
>  >
>  --------------------------------------------
>  >  On Tue, 10/16/18, Bryan Bende <[hidden email]>
>  >  wrote:
>  >
>  >   Subject: Re: Compiling custom
>  >  processor
>  >   To: [hidden email]
>  >   Date: Tuesday, October 16, 2018,
>  9:52
>  >  AM
>  >
>  >   In general, if your processor
>  >   uses a controller service then the
>  >   processors pom file needs a provided
>  >  dependency
>  >   on
>  the API of the CS,
>  >   and your NAR pom
>  needs
>  >   a NAR dependency on the NAR
>  where the
>  >  CS API
>  >   is.
>  >
>  >   Example is
>  >
>  shown here in the section linking
>  >
>  processors and
>  >   controller
>  >   services - https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>  >
>  >   Also, you do not
>  want to
>  >   include NiFI's standard
>  processors in
>  >  your
>  >   own NAR, this will result in two
>  >  copies of
>  >   every
>  standard processor.
>  >   You will want
>  to
>  >   copy whatever code you need into
>  your
>  >  own NAR.
>  >
>  >   In you last statement, if you
>  >   made a new processor in
>  >   nifi-standard-processors and it
>  didn't
>  >  show
>  >   up, it is most likely
>  >   because you didn't
>  >   update META-INF/services file to
>  >  include the new
>  >
>  processor.
>  >   On Tue, Oct 16, 2018
>  at
>  >   9:36 AM John McGinn
>  >   <[hidden email]>
>  >   wrote:
>  >   >
>  >   > Ok, I am
>  >
>  far from a maven expert, and am
>  >
>  struggling on this
>  >   problem.
>  >   >
>  >   > I
>  created
>  >   a new project using the
>  maven generate
>  >  process, and
>  compiled
>  >   that sample processor and
>  everything
>  >  was fine, and I could
>  >   see it in my local NiFi instance. I
>  >  then copied over the
>  >   Wait processor on top of the
>  >  MyProcessor.java class, changed
>  >   the package name, and the class
>  name,
>  >  and attempted to
>  >   compile. Got errors due to
>  dependency
>  >  issues. This is where
>  >   I get confused.
>  >
>  >
>  >   > In
>  >
>  the processor directory, I modify the
>  >
>  pom.xml with a
>  >   dependency line for
>  nifi-api,
>  >  nifi-utils,
>  >
>  >
>  nifi-distribute-cache-client-service-api and
>  >   nifi-standard-processors. This
>  seemed
>  >  to make things work,
>  >   and I get a 33 meg NAR file, which
>  >  contains lots of JARs. I
>  >   load that up, and I get a problem
>  with
>  >  CalculateStats, or
>  >   similar, because of
>  >  RecordReaderFactory. I go back in and
>  >   include
>  >
>  nifi-record-serialization-services, and it compiles,
>  >   and the JAR is included in the NAR
>  >  file now, and I still get
>  >   class not found.
>  >
>   >
>  >   >
>  >
>  I've also tried to change the
>  >
>  dependencies to a status
>  >   of
>  provided, and therefore get a
>  >  minimal
>  25k or so NAR file
>  >   with no JARs
>  included. I get the same
>  >  issue with
>  the class
>  >   not found.
>  >   >
>  >   > I
>  had
>  >   also attempted to copy Wait.java
>  to
>  >  AnotherWait.java within
>  >   the actually NiFi source code, and
>  it
>  >  compiles, and shows up
>  >   in the JARs and NARs as I'd
>  expect,
>  >  but I cannot get to
>  >   it within my NiFi instance.
>  >   >
>  >   > Can
>  someone let me know what I'm
>  >
>  doing
>  >   wrong.
>  >
>  >
>  >   > Thanks,
>  >   > John McGinn
>  >
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

John McGinn-2
In reply to this post by John McGinn
Thanks Bryan.

That was a mistake on my part. I thought that I could use part of the NiFi standard processors in my custom processor. I suppose this also means it's not possible to extend a standard processor class, say Wait, and override a method, for instance, onTrigger(), to do different work. Instead, as you state, I should copy the portions I would need to my custom processor. Is that accurate?

In the end, as I was looking through how to modify Wait, I think it is bigger than the need that I have, so I restarted with a new generated processor project, and just added the cache service, and appropriate relationships, and simple logic to control flow through my processor. I'll work on getting a github account and uploading my simple processor.

Thanks for the assistance
John McGinn

--------------------------------------------
On Tue, 10/16/18, Bryan Bende <[hidden email]> wrote:

 Subject: Re: Compiling custom processor
 To: [hidden email]
 Date: Tuesday, October 16, 2018, 4:45 PM
 
 You shouldn't really try to
 depend on code inside other NARs like
 standard processors, you'll want to copy
 WaitNotifyProtocol into your
 own NAR.
 On
 Tue, Oct 16, 2018 at 4:40 PM John McGinn <[hidden email]>
 wrote:
 >
 > The
 commons-lang3-3.7.jar file is located in the lib/bootstrap
 directory, therefore, I figured it could be considered
 provided. I've since change it to normal scope, and it
 is included in my NAR and that error goes away. So, this
 > then leans on the next error I get, which
 is no class found for the WaitNotifyProtocol in
 nifi.processors.standard. Is there an easy way to figure out
 what NiFi class/object I'm using, and what NAR should be
 listed as provided? I tried to add
 nifi-standard-processors-nar, but that doesn't exist for
 maven, nor does it exist in the lib directory. I looked
 inside the nifi-standard-nar NAR file, and the
 nifi-standard-processors is there, and inside that is
 WaitNotifyProtocol, but yet, my maven won't build if I
 have nifi-standard-nar listed, either with type of NAR, or
 scope of provided.
 >
 --------------------------------------------
 > On Tue, 10/16/18, Bryan Bende <[hidden email]>
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: [hidden email],
 [hidden email]
 >  Date: Tuesday, October 16, 2018, 4:01
 PM
 >
 >  Any reason
 why the commons-lang3
 >  dependency is
 provided and not normal
 >  compile
 scope?
 >
 >  Generally
 the only cases where a dependency
 > 
 should be provided are
 >  when a parent
 NAR
 >  provides it (the API scenario
 earlier), or when it
 >  is a jar
 directly in lib which means it is on
 > 
 the classpath of
 >  everything (like
 nifi-api
 >  or slf4j).
 >  On Tue, Oct 16, 2018 at 3:47 PM John
 McGinn
 >  <[hidden email]>
 >  wrote:
 >  >
 >  > I have a
 > 
 directory /home/user/nifi-nars, which is listed in the
 >  nifi.properties file located under the
 nifi-assembly
 >  directory from the
 nifi-master code. That's where I
 > 
 redropped my NAR file at.
 >  >
 >  > The error was "due to
 uncaught
 >  Exception:
 java.lang.NoClassDefFoundError:
 > 
 org/apache/commons/lang3/StringUtils". I added a
 >  provided dependency of version 3.4 for
 commons-lang3 in my
 >  processors
 pom.xml. That worked initially, but with the
 >  redropping of the NAR file, the error
 occurs. The stacktrace
 >  points to the
 session.get() lambda function trying to use
 >  StringUtils.isBlank(). (Again, this is a
 copy and renamed
 >  class of the Wait
 Processor. Trying to turn it into a Yield,
 >  as it were, hold unless it's free to
 move forward.)
 >  >
 > 
 --------------------------------------------
 >  > On Tue, 10/16/18, Bryan Bende
 <[hidden email]>
 >  wrote:
 >  >
 >  >  Subject:
 > 
 Re: Compiling custom processor
 >  > 
 To: [hidden email]
 >  >  Date: Tuesday, October 16, 2018,
 3:34
 >  PM
 > 
 >
 >  >  There
 >  shouldn't be any issue
 >  >  doing
 > 
 that, generally you just copy the
 > 
 >
 >  new
 > 
 >  NAR into the lib directory and
 > 
 restart.
 >  >
 > 
 >  What
 >  were the specifics of the
 class not found
 >  >  error?
 >  >
 >  >  On
 Tue, Oct 16, 2018 at
 >  >  3:24 PM
 John McGinn <[hidden email]>
 >  >  wrote:
 > 
 >  >
 >  >  > And
 I'm
 >  >
 > 
 back.
 >  >  >
 >  >
 >  > So, I
 had my
 >  >  processor running,
 >  no errors, so things were good.
 >  >
 >  >
 >  >  > I then add 2 if
 >  >  statements with logger messages,
 run the
 >  same mvn command,
 >  >  copied the new NAR
 >  on top of the old one, same versionId
 and
 >  >  all, and now I get a class
 not found
 >  error, that didn't
 >  >  occur before I
 >  stopped, redeployed and started. And
 I
 >  >  can't delete the event
 in the queue
 >  even though source
 >  >  and destination
 >  are stopped.
 > 
 >  >
 >  >  > Is there
 problems when redeploying
 >  a
 >  >  custom processor NAR on top of
 >  itself? Even if the NiFi
 >  >  instance is
 >  stopped? Serialization issues?
 >  >
 >  >
 >  >  > Frustratedly
 >  >  yours,
 > 
 >  > John
 >  McGinn
 >  >  >
 > 
 >
 >  >
 > 
 >
 > 
 --------------------------------------------
 >  >  > On Tue, 10/16/18, John
 McGinn
 >  <[hidden email]>
 >  >  wrote:
 > 
 >  >
 >  >  >  Subject:
 >  >
 >  Re:
 Compiling custom processor
 >  > 
 >
 >  To: [hidden email]
 >  >  >  Date: Tuesday, October 16,
 2018,
 >  1:21
 > 
 >  PM
 >  >
 > 
 >
 >  >  >  Thanks
 >  >  Bryan,
 > 
 >  >
 >  >  >  I
 >  >
 >  restarted
 with a fresh maven
 >  >  >
 >  >  generate, and only added the API
 to the
 >  processor pom,
 >  >  and
 >  > 
 >  then the API-NAR to the NAR
 > 
 pom.
 >  >  I then figured out a
 >  >  >  subsequent
 >  >
 >  dependency
 (lang3), and got that in, and now my
 > 
 >  >  processor shows up in local
 >  NiFi
 >  > 
 instance.
 >  >
 > 
 >
 >  >  >
 >  >
 >  Regarding
 the copying of the standard
 >  >
 >  >
 >  > 
 processor, when I copied Wait
 >  over, I
 renamed it from
 >  >  Wait
 >  >  >  to MyProcessorWait to
 be
 >  unique,
 > 
 >  and not copied. I'm
 > 
 just
 >  >  >
 >  >
 >  attempting
 changes to Wait without modifying the actual
 >  >  one.
 > 
 >  >  Also,
 >  thanks on the
 >  >  META-INF/services file
 >  information.
 > 
 >  >
 >  >  >  Now I can
 debug and test.
 >  >  >
 >  >  >
 > 
 Sincerely,
 >  >  >  John
 McGinn
 >  >  >
 >  >  >
 > 
 >
 > 
 --------------------------------------------
 >  >  >  On Tue, 10/16/18, Bryan
 Bende
 >  <[hidden email]>
 >  >  >  wrote:
 >  >
 >  >
 >  >  >   Subject: Re: Compiling
 >  custom
 >  > 
 >  processor
 >  >  >   To:
 [hidden email]
 >  >  >   Date: Tuesday, October
 16,
 >  2018,
 > 
 >  9:52
 >  >
 >  >  AM
 >  > 
 >
 >  >
 > 
 >   In general, if your processor
 > 
 >
 >  >   uses a controller
 service then the
 >  >  > 
 processors pom file needs a
 > 
 provided
 >  >  >  dependency
 >  >  >   on
 > 
 >  the
 >  API of the CS,
 >  >  >   and your NAR
 >  pom
 >  > 
 needs
 >  >
 > 
 >   a NAR dependency on the NAR
 > 
 >
 >  where the
 > 
 >  >  CS API
 >  >  > 
 is.
 >  >
 > 
 >
 >  >  >   Example is
 >  >  >
 > 
 >  shown here
 >  in the section
 linking
 >  >  >
 >  >  processors and
 >  >
 >  > 
 controller
 >  >  >   services
 -
 >  https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 >  >  >
 > 
 >  >
 >  Also, you do not
 >  >  want to
 > 
 >  >   include NiFI's standard
 >  >  processors in
 >  >
 >  > 
 your
 >  >  >   own NAR, this
 >  will result in two
 >  >  >  copies
 >  of
 >  >  > 
 every
 >  >  standard processor.
 >  >  >   You will want
 >  >  to
 >  > 
 >   copy
 >  whatever code you need
 into
 >  >  your
 >  >  >  own NAR.
 >  >
 >  >
 >  >  >   In you last statement,
 >  if you
 >  > 
 >   made a new processor
 >  in
 >  >  > 
 nifi-standard-processors
 >  and it
 >  >  didn't
 > 
 >  >  show
 >  >
 >  >   up, it is most likely
 >  >  >
 > 
 because you didn't
 >  >  > 
 update
 >  META-INF/services file to
 >  >  >
 > 
 include the new
 >  >  >
 >  >  processor.
 > 
 >  >
 >   On Tue, Oct 16, 2018
 >  >  at
 >  > 
 >   9:36 AM John McGinn
 >  > 
 >   <[hidden email]>
 >  >  >   wrote:
 >  >
 >  > 
 >
 >  >  >   > Ok, I am
 >  >  >
 > 
 >  far from a
 >  maven expert, and
 am
 >  >  >
 > 
 >  struggling on this
 >  > 
 >   problem.
 >  >
 >  >   >
 > 
 >  >   > I
 >  > 
 created
 >  >  >   a
 >  new project using the
 >  >  maven
 > 
 generate
 >  >  >  process,
 and
 >  >  compiled
 >  >  >
 >  that
 sample processor and
 >  >
 >  everything
 >  > 
 >  was fine, and I
 >  could
 >  >  >   see it in my local
 NiFi
 >  instance. I
 >  >  >  then copied over
 >  the
 >  >  > 
  Wait processor on top of
 >  the
 >  >  >  MyProcessor.java class,
 >  changed
 >  > 
 >   the package name, and
 >  the
 class
 >  >  name,
 >  >  >  and attempted to
 >  >  >   compile. Got errors due
 to
 >  >  dependency
 >  >  >
 > 
 issues. This is where
 >  >  > 
 I get
 >  confused.
 >  >  >
 > 
 >
 >  >
 > 
 >  >   > In
 >  >  >
 >  >  the
 > 
 processor directory, I modify the
 > 
 >
 >  >
 > 
 >  pom.xml with a
 >  >  > 
 dependency line for
 >  > 
 nifi-api,
 >  >  >
 >  nifi-utils,
 > 
 >  >
 >  >  >
 >  >
 > 
 nifi-distribute-cache-client-service-api and
 >  >  > 
 nifi-standard-processors.
 >  This
 >  >  seemed
 > 
 >
 >  >  to make things work,
 >  >  >
 >  and I
 get a 33 meg NAR file, which
 >  >
 >  >  contains lots of JARs. I
 >  >
 >  >   load
 that up, and I get a problem
 >  > 
 with
 >  >  >
 >  CalculateStats, or
 >  >  >   similar,
 >  because of
 >  > 
 >
 >  RecordReaderFactory. I go back
 in and
 >  >
 > 
 >   include
 >  >  >
 >  > 
 nifi-record-serialization-services, and
 >  it compiles,
 > 
 >  >   and the JAR is
 > 
 included in the NAR
 >  >  > 
 file now,
 >  and I still get
 >  >  >   class not
 >  found.
 >  > 
 >
 >  >
 > 
 >
 >  >  >   >
 >  >  >
 > 
 >  I've
 >  also tried to change
 the
 >  >  >
 >  >  dependencies to a status
 >  >  >   of
 > 
 >
 >  provided, and therefore get a
 >  >  >
 > 
 minimal
 >  >  25k or so NAR file
 >  >  >   with no JARs
 >  >  included. I get the same
 >  >  >  issue with
 >  >
 >  the class
 >  >  >   not found.
 >  >  >   >
 >  >
 >  >   >
 I
 >  >  had
 > 
 >  >   also attempted to copy
 > 
 Wait.java
 >  >  to
 >  >
 >  > 
 AnotherWait.java within
 >  > 
 >
 >   the actually NiFi source code,
 and
 >  >
 >  it
 >  >  >  compiles, and shows up
 >  >  >   in the JARs and NARs
 as
 >  I'd
 > 
 >  expect,
 >  >  >  but I
 cannot get to
 >  >  >   it
 within my NiFi instance.
 >  > 
 >   >
 >  >
 >  >   > Can
 > 
 >  someone let me know
 >  what
 I'm
 >  >  >
 >  >  doing
 > 
 >  >
 >  wrong.
 >  >  >
 > 
 >
 >  >
 > 
 >  >   > Thanks,
 >  > 
 >   > John McGinn
 >  > 
 >
Reply | Threaded
Open this post in threaded view
|

Re: Compiling custom processor

Bryan Bende
That is accurate. Generally the intention isn't really to extend
processors across NARs. It is better to create a shared module with a
base class that both processors can extend from, or utility code they
can both call, but in the case that shared module doesn't exist then
the main option is to copy over the code you need.

On Wed, Oct 17, 2018 at 2:52 PM John McGinn <[hidden email]> wrote:

>
> Thanks Bryan.
>
> That was a mistake on my part. I thought that I could use part of the NiFi standard processors in my custom processor. I suppose this also means it's not possible to extend a standard processor class, say Wait, and override a method, for instance, onTrigger(), to do different work. Instead, as you state, I should copy the portions I would need to my custom processor. Is that accurate?
>
> In the end, as I was looking through how to modify Wait, I think it is bigger than the need that I have, so I restarted with a new generated processor project, and just added the cache service, and appropriate relationships, and simple logic to control flow through my processor. I'll work on getting a github account and uploading my simple processor.
>
> Thanks for the assistance
> John McGinn
>
> --------------------------------------------
> On Tue, 10/16/18, Bryan Bende <[hidden email]> wrote:
>
>  Subject: Re: Compiling custom processor
>  To: [hidden email]
>  Date: Tuesday, October 16, 2018, 4:45 PM
>
>  You shouldn't really try to
>  depend on code inside other NARs like
>  standard processors, you'll want to copy
>  WaitNotifyProtocol into your
>  own NAR.
>  On
>  Tue, Oct 16, 2018 at 4:40 PM John McGinn <[hidden email]>
>  wrote:
>  >
>  > The
>  commons-lang3-3.7.jar file is located in the lib/bootstrap
>  directory, therefore, I figured it could be considered
>  provided. I've since change it to normal scope, and it
>  is included in my NAR and that error goes away. So, this
>  > then leans on the next error I get, which
>  is no class found for the WaitNotifyProtocol in
>  nifi.processors.standard. Is there an easy way to figure out
>  what NiFi class/object I'm using, and what NAR should be
>  listed as provided? I tried to add
>  nifi-standard-processors-nar, but that doesn't exist for
>  maven, nor does it exist in the lib directory. I looked
>  inside the nifi-standard-nar NAR file, and the
>  nifi-standard-processors is there, and inside that is
>  WaitNotifyProtocol, but yet, my maven won't build if I
>  have nifi-standard-nar listed, either with type of NAR, or
>  scope of provided.
>  >
>  --------------------------------------------
>  > On Tue, 10/16/18, Bryan Bende <[hidden email]>
>  wrote:
>  >
>  >  Subject:
>  Re: Compiling custom processor
>  >  To: [hidden email],
>  [hidden email]
>  >  Date: Tuesday, October 16, 2018, 4:01
>  PM
>  >
>  >  Any reason
>  why the commons-lang3
>  >  dependency is
>  provided and not normal
>  >  compile
>  scope?
>  >
>  >  Generally
>  the only cases where a dependency
>  >
>  should be provided are
>  >  when a parent
>  NAR
>  >  provides it (the API scenario
>  earlier), or when it
>  >  is a jar
>  directly in lib which means it is on
>  >
>  the classpath of
>  >  everything (like
>  nifi-api
>  >  or slf4j).
>  >  On Tue, Oct 16, 2018 at 3:47 PM John
>  McGinn
>  >  <[hidden email]>
>  >  wrote:
>  >  >
>  >  > I have a
>  >
>  directory /home/user/nifi-nars, which is listed in the
>  >  nifi.properties file located under the
>  nifi-assembly
>  >  directory from the
>  nifi-master code. That's where I
>  >
>  redropped my NAR file at.
>  >  >
>  >  > The error was "due to
>  uncaught
>  >  Exception:
>  java.lang.NoClassDefFoundError:
>  >
>  org/apache/commons/lang3/StringUtils". I added a
>  >  provided dependency of version 3.4 for
>  commons-lang3 in my
>  >  processors
>  pom.xml. That worked initially, but with the
>  >  redropping of the NAR file, the error
>  occurs. The stacktrace
>  >  points to the
>  session.get() lambda function trying to use
>  >  StringUtils.isBlank(). (Again, this is a
>  copy and renamed
>  >  class of the Wait
>  Processor. Trying to turn it into a Yield,
>  >  as it were, hold unless it's free to
>  move forward.)
>  >  >
>  >
>  --------------------------------------------
>  >  > On Tue, 10/16/18, Bryan Bende
>  <[hidden email]>
>  >  wrote:
>  >  >
>  >  >  Subject:
>  >
>  Re: Compiling custom processor
>  >  >
>  To: [hidden email]
>  >  >  Date: Tuesday, October 16, 2018,
>  3:34
>  >  PM
>  >
>  >
>  >  >  There
>  >  shouldn't be any issue
>  >  >  doing
>  >
>  that, generally you just copy the
>  >
>  >
>  >  new
>  >
>  >  NAR into the lib directory and
>  >
>  restart.
>  >  >
>  >
>  >  What
>  >  were the specifics of the
>  class not found
>  >  >  error?
>  >  >
>  >  >  On
>  Tue, Oct 16, 2018 at
>  >  >  3:24 PM
>  John McGinn <[hidden email]>
>  >  >  wrote:
>  >
>  >  >
>  >  >  > And
>  I'm
>  >  >
>  >
>  back.
>  >  >  >
>  >  >
>  >  > So, I
>  had my
>  >  >  processor running,
>  >  no errors, so things were good.
>  >  >
>  >  >
>  >  >  > I then add 2 if
>  >  >  statements with logger messages,
>  run the
>  >  same mvn command,
>  >  >  copied the new NAR
>  >  on top of the old one, same versionId
>  and
>  >  >  all, and now I get a class
>  not found
>  >  error, that didn't
>  >  >  occur before I
>  >  stopped, redeployed and started. And
>  I
>  >  >  can't delete the event
>  in the queue
>  >  even though source
>  >  >  and destination
>  >  are stopped.
>  >
>  >  >
>  >  >  > Is there
>  problems when redeploying
>  >  a
>  >  >  custom processor NAR on top of
>  >  itself? Even if the NiFi
>  >  >  instance is
>  >  stopped? Serialization issues?
>  >  >
>  >  >
>  >  >  > Frustratedly
>  >  >  yours,
>  >
>  >  > John
>  >  McGinn
>  >  >  >
>  >
>  >
>  >  >
>  >
>  >
>  >
>  --------------------------------------------
>  >  >  > On Tue, 10/16/18, John
>  McGinn
>  >  <[hidden email]>
>  >  >  wrote:
>  >
>  >  >
>  >  >  >  Subject:
>  >  >
>  >  Re:
>  Compiling custom processor
>  >  >
>  >
>  >  To: [hidden email]
>  >  >  >  Date: Tuesday, October 16,
>  2018,
>  >  1:21
>  >
>  >  PM
>  >  >
>  >
>  >
>  >  >  >  Thanks
>  >  >  Bryan,
>  >
>  >  >
>  >  >  >  I
>  >  >
>  >  restarted
>  with a fresh maven
>  >  >  >
>  >  >  generate, and only added the API
>  to the
>  >  processor pom,
>  >  >  and
>  >  >
>  >  then the API-NAR to the NAR
>  >
>  pom.
>  >  >  I then figured out a
>  >  >  >  subsequent
>  >  >
>  >  dependency
>  (lang3), and got that in, and now my
>  >
>  >  >  processor shows up in local
>  >  NiFi
>  >  >
>  instance.
>  >  >
>  >
>  >
>  >  >  >
>  >  >
>  >  Regarding
>  the copying of the standard
>  >  >
>  >  >
>  >  >
>  processor, when I copied Wait
>  >  over, I
>  renamed it from
>  >  >  Wait
>  >  >  >  to MyProcessorWait to
>  be
>  >  unique,
>  >
>  >  and not copied. I'm
>  >
>  just
>  >  >  >
>  >  >
>  >  attempting
>  changes to Wait without modifying the actual
>  >  >  one.
>  >
>  >  >  Also,
>  >  thanks on the
>  >  >  META-INF/services file
>  >  information.
>  >
>  >  >
>  >  >  >  Now I can
>  debug and test.
>  >  >  >
>  >  >  >
>  >
>  Sincerely,
>  >  >  >  John
>  McGinn
>  >  >  >
>  >  >  >
>  >
>  >
>  >
>  --------------------------------------------
>  >  >  >  On Tue, 10/16/18, Bryan
>  Bende
>  >  <[hidden email]>
>  >  >  >  wrote:
>  >  >
>  >  >
>  >  >  >   Subject: Re: Compiling
>  >  custom
>  >  >
>  >  processor
>  >  >  >   To:
>  [hidden email]
>  >  >  >   Date: Tuesday, October
>  16,
>  >  2018,
>  >
>  >  9:52
>  >  >
>  >  >  AM
>  >  >
>  >
>  >  >
>  >
>  >   In general, if your processor
>  >
>  >
>  >  >   uses a controller
>  service then the
>  >  >  >
>  processors pom file needs a
>  >
>  provided
>  >  >  >  dependency
>  >  >  >   on
>  >
>  >  the
>  >  API of the CS,
>  >  >  >   and your NAR
>  >  pom
>  >  >
>  needs
>  >  >
>  >
>  >   a NAR dependency on the NAR
>  >
>  >
>  >  where the
>  >
>  >  >  CS API
>  >  >  >
>  is.
>  >  >
>  >
>  >
>  >  >  >   Example is
>  >  >  >
>  >
>  >  shown here
>  >  in the section
>  linking
>  >  >  >
>  >  >  processors and
>  >  >
>  >  >
>  controller
>  >  >  >   services
>  -
>  >  https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>  >  >  >
>  >
>  >  >
>  >  Also, you do not
>  >  >  want to
>  >
>  >  >   include NiFI's standard
>  >  >  processors in
>  >  >
>  >  >
>  your
>  >  >  >   own NAR, this
>  >  will result in two
>  >  >  >  copies
>  >  of
>  >  >  >
>  every
>  >  >  standard processor.
>  >  >  >   You will want
>  >  >  to
>  >  >
>  >   copy
>  >  whatever code you need
>  into
>  >  >  your
>  >  >  >  own NAR.
>  >  >
>  >  >
>  >  >  >   In you last statement,
>  >  if you
>  >  >
>  >   made a new processor
>  >  in
>  >  >  >
>  nifi-standard-processors
>  >  and it
>  >  >  didn't
>  >
>  >  >  show
>  >  >
>  >  >   up, it is most likely
>  >  >  >
>  >
>  because you didn't
>  >  >  >
>  update
>  >  META-INF/services file to
>  >  >  >
>  >
>  include the new
>  >  >  >
>  >  >  processor.
>  >
>  >  >
>  >   On Tue, Oct 16, 2018
>  >  >  at
>  >  >
>  >   9:36 AM John McGinn
>  >  >
>  >   <[hidden email]>
>  >  >  >   wrote:
>  >  >
>  >  >
>  >
>  >  >  >   > Ok, I am
>  >  >  >
>  >
>  >  far from a
>  >  maven expert, and
>  am
>  >  >  >
>  >
>  >  struggling on this
>  >  >
>  >   problem.
>  >  >
>  >  >   >
>  >
>  >  >   > I
>  >  >
>  created
>  >  >  >   a
>  >  new project using the
>  >  >  maven
>  >
>  generate
>  >  >  >  process,
>  and
>  >  >  compiled
>  >  >  >
>  >  that
>  sample processor and
>  >  >
>  >  everything
>  >  >
>  >  was fine, and I
>  >  could
>  >  >  >   see it in my local
>  NiFi
>  >  instance. I
>  >  >  >  then copied over
>  >  the
>  >  >  >
>   Wait processor on top of
>  >  the
>  >  >  >  MyProcessor.java class,
>  >  changed
>  >  >
>  >   the package name, and
>  >  the
>  class
>  >  >  name,
>  >  >  >  and attempted to
>  >  >  >   compile. Got errors due
>  to
>  >  >  dependency
>  >  >  >
>  >
>  issues. This is where
>  >  >  >
>  I get
>  >  confused.
>  >  >  >
>  >
>  >
>  >  >
>  >
>  >  >   > In
>  >  >  >
>  >  >  the
>  >
>  processor directory, I modify the
>  >
>  >
>  >  >
>  >
>  >  pom.xml with a
>  >  >  >
>  dependency line for
>  >  >
>  nifi-api,
>  >  >  >
>  >  nifi-utils,
>  >
>  >  >
>  >  >  >
>  >  >
>  >
>  nifi-distribute-cache-client-service-api and
>  >  >  >
>  nifi-standard-processors.
>  >  This
>  >  >  seemed
>  >
>  >
>  >  >  to make things work,
>  >  >  >
>  >  and I
>  get a 33 meg NAR file, which
>  >  >
>  >  >  contains lots of JARs. I
>  >  >
>  >  >   load
>  that up, and I get a problem
>  >  >
>  with
>  >  >  >
>  >  CalculateStats, or
>  >  >  >   similar,
>  >  because of
>  >  >
>  >
>  >  RecordReaderFactory. I go back
>  in and
>  >  >
>  >
>  >   include
>  >  >  >
>  >  >
>  nifi-record-serialization-services, and
>  >  it compiles,
>  >
>  >  >   and the JAR is
>  >
>  included in the NAR
>  >  >  >
>  file now,
>  >  and I still get
>  >  >  >   class not
>  >  found.
>  >  >
>  >
>  >  >
>  >
>  >
>  >  >  >   >
>  >  >  >
>  >
>  >  I've
>  >  also tried to change
>  the
>  >  >  >
>  >  >  dependencies to a status
>  >  >  >   of
>  >
>  >
>  >  provided, and therefore get a
>  >  >  >
>  >
>  minimal
>  >  >  25k or so NAR file
>  >  >  >   with no JARs
>  >  >  included. I get the same
>  >  >  >  issue with
>  >  >
>  >  the class
>  >  >  >   not found.
>  >  >  >   >
>  >  >
>  >  >   >
>  I
>  >  >  had
>  >
>  >  >   also attempted to copy
>  >
>  Wait.java
>  >  >  to
>  >  >
>  >  >
>  AnotherWait.java within
>  >  >
>  >
>  >   the actually NiFi source code,
>  and
>  >  >
>  >  it
>  >  >  >  compiles, and shows up
>  >  >  >   in the JARs and NARs
>  as
>  >  I'd
>  >
>  >  expect,
>  >  >  >  but I
>  cannot get to
>  >  >  >   it
>  within my NiFi instance.
>  >  >
>  >   >
>  >  >
>  >  >   > Can
>  >
>  >  someone let me know
>  >  what
>  I'm
>  >  >  >
>  >  >  doing
>  >
>  >  >
>  >  wrong.
>  >  >  >
>  >
>  >
>  >  >
>  >
>  >  >   > Thanks,
>  >  >
>  >   > John McGinn
>  >  >
>  >