Will you accept contributions in Scala?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Will you accept contributions in Scala?

Weiss, Adam
Devs,

I have some interns starting with my team and we use Scala internally for our work.
If I wanted to have them work to contribute some new processors, would they have to be Java to be included with the base distribution or could they use Scala as long as it fit within your current build and test process?

Thanks,
-Adam
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Andy LoPresto-2
Hi Adam,

It’s great that your team wants to contribute to NiFi. I think there should be some discussion from other community members on this, but I’ll give my view here. At this time, I believe we should only accept core contributions in Java. It’s not because I have a predisposition against Scala or any other language (frankly, I would prefer to write my processors in Groovy), but because Apache prioritizes community over code, and in order to make the code accessible to the community at large, Java is the standard for this project. 

That said, I think there are some important ways you can contribute in other languages:

* Testing — while we try to ensure the “production” code is pure Java, the test frameworks around NiFi can use other languages. We currently have a number of tests in Groovy/Spock. 
* Custom repositories — you are welcome to publish processors written in Scala on GitHub/Maven and anyone in the community who is interested can add that functionality into their installations very easily. 

Hopefully this helps guide your instruction to your interns. If others in the community feel strongly about this, I hope they weigh in. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Feb 9, 2018, at 6:00 AM, Weiss, Adam <[hidden email]> wrote:

Devs,

I have some interns starting with my team and we use Scala internally for our work.
If I wanted to have them work to contribute some new processors, would they have to be Java to be included with the base distribution or could they use Scala as long as it fit within your current build and test process?

Thanks,
-Adam


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Mike Thomsen
In reply to this post by Weiss, Adam
I'm just a community contributor, so take that FWIW, but a compromise might
be to publish the Scala code as separate maven modules to maven central and
then submit a thoroughly tested processor written in Java. As long as you
have enough unit and integration tests to give strong coverage, I wouldn't
imagine anyone here would have issues reviewing it. If the tests fail
because of code issues in the external dependencies, the obvious answer is
to just hold the PR until the tests pass.

On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <[hidden email]>
wrote:

> Devs,
>
> I have some interns starting with my team and we use Scala internally for
> our work.
> If I wanted to have them work to contribute some new processors, would
> they have to be Java to be included with the base distribution or could
> they use Scala as long as it fit within your current build and test process?
>
> Thanks,
> -Adam
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Joe Witt
i personally would be ok with it for an extension/processor provided it
integrates well with the build.

i would agree with andys view for core framework stuff but for extensions i
think we can do it like mikethomsen suggested.

others?

thanks
joe

On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]> wrote:

> I'm just a community contributor, so take that FWIW, but a compromise might
> be to publish the Scala code as separate maven modules to maven central and
> then submit a thoroughly tested processor written in Java. As long as you
> have enough unit and integration tests to give strong coverage, I wouldn't
> imagine anyone here would have issues reviewing it. If the tests fail
> because of code issues in the external dependencies, the obvious answer is
> to just hold the PR until the tests pass.
>
> On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <[hidden email]>
> wrote:
>
> > Devs,
> >
> > I have some interns starting with my team and we use Scala internally for
> > our work.
> > If I wanted to have them work to contribute some new processors, would
> > they have to be Java to be included with the base distribution or could
> > they use Scala as long as it fit within your current build and test
> process?
> >
> > Thanks,
> > -Adam
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Jeff
+1 to Joe's response.  If you can develop a component in Groovy or Scala
(or Clojure!) more quickly/comfortably, or if allowing components written
in other languages would encourage people to contribute more, I'm all for
it.

On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]> wrote:

> i personally would be ok with it for an extension/processor provided it
> integrates well with the build.
>
> i would agree with andys view for core framework stuff but for extensions i
> think we can do it like mikethomsen suggested.
>
> others?
>
> thanks
> joe
>
> On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]> wrote:
>
> > I'm just a community contributor, so take that FWIW, but a compromise
> might
> > be to publish the Scala code as separate maven modules to maven central
> and
> > then submit a thoroughly tested processor written in Java. As long as you
> > have enough unit and integration tests to give strong coverage, I
> wouldn't
> > imagine anyone here would have issues reviewing it. If the tests fail
> > because of code issues in the external dependencies, the obvious answer
> is
> > to just hold the PR until the tests pass.
> >
> > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <[hidden email]>
> > wrote:
> >
> > > Devs,
> > >
> > > I have some interns starting with my team and we use Scala internally
> for
> > > our work.
> > > If I wanted to have them work to contribute some new processors, would
> > > they have to be Java to be included with the base distribution or could
> > > they use Scala as long as it fit within your current build and test
> > process?
> > >
> > > Thanks,
> > > -Adam
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Bryan Bende
I agree more with Andy about sticking with Java. The more varying languages
used, the more challenging it is to maintain. Once the code is part of the
Apache NiFi git repo, it is now the responsibility of the committers and
PMC members to maintain it.

I’d even say I am somewhat against the groovy/Spock test code that Andy
mentioned. I have frequently spent hours trying to fix a Spock test that
broke from something I was working on. Every committer is familiar with
JUnit, but only a couple know Spock. Just using this as an example that
every committer knows Java, but only a couple probably know Scala, Clojure,
etc.

On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]> wrote:

> +1 to Joe's response.  If you can develop a component in Groovy or Scala
> (or Clojure!) more quickly/comfortably, or if allowing components written
> in other languages would encourage people to contribute more, I'm all for
> it.
>
> On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]> wrote:
>
> > i personally would be ok with it for an extension/processor provided it
> > integrates well with the build.
> >
> > i would agree with andys view for core framework stuff but for
> extensions i
> > think we can do it like mikethomsen suggested.
> >
> > others?
> >
> > thanks
> > joe
> >
> > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]> wrote:
> >
> > > I'm just a community contributor, so take that FWIW, but a compromise
> > might
> > > be to publish the Scala code as separate maven modules to maven central
> > and
> > > then submit a thoroughly tested processor written in Java. As long as
> you
> > > have enough unit and integration tests to give strong coverage, I
> > wouldn't
> > > imagine anyone here would have issues reviewing it. If the tests fail
> > > because of code issues in the external dependencies, the obvious answer
> > is
> > > to just hold the PR until the tests pass.
> > >
> > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> [hidden email]>
> > > wrote:
> > >
> > > > Devs,
> > > >
> > > > I have some interns starting with my team and we use Scala internally
> > for
> > > > our work.
> > > > If I wanted to have them work to contribute some new processors,
> would
> > > > they have to be Java to be included with the base distribution or
> could
> > > > they use Scala as long as it fit within your current build and test
> > > process?
> > > >
> > > > Thanks,
> > > > -Adam
> > > >
> > >
> >
>
--
Sent from Gmail Mobile
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Joey Frazee
This probably necessitates a vote, yeah?

Frankly, I’m usually happier writing Scala, and I’ve not encountered any problems using processors written in Scala, but I think it’ll be important to tread lightly.

There’s a few things that pop into my head:

- Maintainability and reviewability. A very very good Java developer need not, by definition, either know how to write or identify good Scala or spot problems and bugs.
- Every Scala processor would either end up with a 5MB scala-lang.jar packaged into the .nar or we’d have to start including it in the core somewhere, if it’s not. It’s possible it might have already gotten pulled up from other dependencies.
- Style. There’s a tremendous amount of variation in Scala style because of its type system, implicits, macros, and functional nature. There are very good people out there that can write good Scala that isn’t readable by the 99%.
- Binary compatibility. Scala tends to be a little more brazen about breaking binary compatibility in major releases and those happen a bit more often than with Java. That’s not a problem for any potential source code in the project, but it could present some dependency issues someday.
- Testing. There’s N > 1 test frameworks and testing styles within those, so there’s a lot of options for introducing more variability into the tests.
- NiFi uses a lot of statics in setting up properties and relationships and the like, and idiomatic Scala approaches that stuff a bit differently, so it’ll be necessary to impose some style guidelines so there isn’t too much variation.

That said, there are some things that won’t be problematic:

- As mentioned, processors written in Scala do just work.
- The scala-maven-plugin works just fine allowing mixed Java-Scala projects (btw, it’d probably be not super great to do mixed Java-Scala and mixed Maven-SBT though).
- A lot of the above concerns could be addressed by having clear style guidelines.

Another thing: most of the projects I see deliver separate jars for Scala components are delivering idiomatic APIs wrapping Java (or vice versa). I think publishing   a separate set of jars/nars for stuff written in Scala would be odd since here it’d mostly be processors with new functionality and not functionality for using Scala. I could imagine a lib of implicits, traits, classes that could make the Scala development more enjoyable. That probably would make sense to deliver that way.

-joey

On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]>, wrote:

> I agree more with Andy about sticking with Java. The more varying languages
> used, the more challenging it is to maintain. Once the code is part of the
> Apache NiFi git repo, it is now the responsibility of the committers and
> PMC members to maintain it.
>
> I’d even say I am somewhat against the groovy/Spock test code that Andy
> mentioned. I have frequently spent hours trying to fix a Spock test that
> broke from something I was working on. Every committer is familiar with
> JUnit, but only a couple know Spock. Just using this as an example that
> every committer knows Java, but only a couple probably know Scala, Clojure,
> etc.
>
> On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]> wrote:
>
> > +1 to Joe's response. If you can develop a component in Groovy or Scala
> > (or Clojure!) more quickly/comfortably, or if allowing components written
> > in other languages would encourage people to contribute more, I'm all for
> > it.
> >
> > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]> wrote:
> >
> > > i personally would be ok with it for an extension/processor provided it
> > > integrates well with the build.
> > >
> > > i would agree with andys view for core framework stuff but for
> > extensions i
> > > think we can do it like mikethomsen suggested.
> > >
> > > others?
> > >
> > > thanks
> > > joe
> > >
> > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]> wrote:
> > >
> > > > I'm just a community contributor, so take that FWIW, but a compromise
> > > might
> > > > be to publish the Scala code as separate maven modules to maven central
> > > and
> > > > then submit a thoroughly tested processor written in Java. As long as
> > you
> > > > have enough unit and integration tests to give strong coverage, I
> > > wouldn't
> > > > imagine anyone here would have issues reviewing it. If the tests fail
> > > > because of code issues in the external dependencies, the obvious answer
> > > is
> > > > to just hold the PR until the tests pass.
> > > >
> > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> > [hidden email]
> > > > wrote:
> > > >
> > > > > Devs,
> > > > >
> > > > > I have some interns starting with my team and we use Scala internally
> > > for
> > > > > our work.
> > > > > If I wanted to have them work to contribute some new processors,
> > would
> > > > > they have to be Java to be included with the base distribution or
> > could
> > > > > they use Scala as long as it fit within your current build and test
> > > > process?
> > > > >
> > > > > Thanks,
> > > > > -Adam
> > > > >
> > > >
> > >
> >
> --
> Sent from Gmail Mobile
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Andrew Grande-2
Wasn't there a warning trigger about the NiFi distro size from Apache
recently? IMO, before talking alternative languages, solve the modularity
and NAR distribution problem. I think the implementation of a module won't
matter much then, the point being not everything has to go in the core,
base distribution, but can still be easily sourced from a known repo, for
example.

I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)

Andrew

On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]> wrote:

> This probably necessitates a vote, yeah?
>
> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
> problems using processors written in Scala, but I think it’ll be important
> to tread lightly.
>
> There’s a few things that pop into my head:
>
> - Maintainability and reviewability. A very very good Java developer need
> not, by definition, either know how to write or identify good Scala or spot
> problems and bugs.
> - Every Scala processor would either end up with a 5MB scala-lang.jar
> packaged into the .nar or we’d have to start including it in the core
> somewhere, if it’s not. It’s possible it might have already gotten pulled
> up from other dependencies.
> - Style. There’s a tremendous amount of variation in Scala style because
> of its type system, implicits, macros, and functional nature. There are
> very good people out there that can write good Scala that isn’t readable by
> the 99%.
> - Binary compatibility. Scala tends to be a little more brazen about
> breaking binary compatibility in major releases and those happen a bit more
> often than with Java. That’s not a problem for any potential source code in
> the project, but it could present some dependency issues someday.
> - Testing. There’s N > 1 test frameworks and testing styles within those,
> so there’s a lot of options for introducing more variability into the tests.
> - NiFi uses a lot of statics in setting up properties and relationships
> and the like, and idiomatic Scala approaches that stuff a bit differently,
> so it’ll be necessary to impose some style guidelines so there isn’t too
> much variation.
>
> That said, there are some things that won’t be problematic:
>
> - As mentioned, processors written in Scala do just work.
> - The scala-maven-plugin works just fine allowing mixed Java-Scala
> projects (btw, it’d probably be not super great to do mixed Java-Scala and
> mixed Maven-SBT though).
> - A lot of the above concerns could be addressed by having clear style
> guidelines.
>
> Another thing: most of the projects I see deliver separate jars for Scala
> components are delivering idiomatic APIs wrapping Java (or vice versa). I
> think publishing   a separate set of jars/nars for stuff written in Scala
> would be odd since here it’d mostly be processors with new functionality
> and not functionality for using Scala. I could imagine a lib of implicits,
> traits, classes that could make the Scala development more enjoyable. That
> probably would make sense to deliver that way.
>
> -joey
>
> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]>, wrote:
> > I agree more with Andy about sticking with Java. The more varying
> languages
> > used, the more challenging it is to maintain. Once the code is part of
> the
> > Apache NiFi git repo, it is now the responsibility of the committers and
> > PMC members to maintain it.
> >
> > I’d even say I am somewhat against the groovy/Spock test code that Andy
> > mentioned. I have frequently spent hours trying to fix a Spock test that
> > broke from something I was working on. Every committer is familiar with
> > JUnit, but only a couple know Spock. Just using this as an example that
> > every committer knows Java, but only a couple probably know Scala,
> Clojure,
> > etc.
> >
> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]> wrote:
> >
> > > +1 to Joe's response. If you can develop a component in Groovy or Scala
> > > (or Clojure!) more quickly/comfortably, or if allowing components
> written
> > > in other languages would encourage people to contribute more, I'm all
> for
> > > it.
> > >
> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]> wrote:
> > >
> > > > i personally would be ok with it for an extension/processor provided
> it
> > > > integrates well with the build.
> > > >
> > > > i would agree with andys view for core framework stuff but for
> > > extensions i
> > > > think we can do it like mikethomsen suggested.
> > > >
> > > > others?
> > > >
> > > > thanks
> > > > joe
> > > >
> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]>
> wrote:
> > > >
> > > > > I'm just a community contributor, so take that FWIW, but a
> compromise
> > > > might
> > > > > be to publish the Scala code as separate maven modules to maven
> central
> > > > and
> > > > > then submit a thoroughly tested processor written in Java. As long
> as
> > > you
> > > > > have enough unit and integration tests to give strong coverage, I
> > > > wouldn't
> > > > > imagine anyone here would have issues reviewing it. If the tests
> fail
> > > > > because of code issues in the external dependencies, the obvious
> answer
> > > > is
> > > > > to just hold the PR until the tests pass.
> > > > >
> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> > > [hidden email]
> > > > > wrote:
> > > > >
> > > > > > Devs,
> > > > > >
> > > > > > I have some interns starting with my team and we use Scala
> internally
> > > > for
> > > > > > our work.
> > > > > > If I wanted to have them work to contribute some new processors,
> > > would
> > > > > > they have to be Java to be included with the base distribution or
> > > could
> > > > > > they use Scala as long as it fit within your current build and
> test
> > > > > process?
> > > > > >
> > > > > > Thanks,
> > > > > > -Adam
> > > > > >
> > > > >
> > > >
> > >
> > --
> > Sent from Gmail Mobile
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Matt Burgess-2
I'm fine with a vote, but I'll be voting to keep Java as the single
language for the (non-test) code. I share the same concerns as many of
the other folks as far as accepting other languages, it's mainly the
"slippery slope" argument that I don't want to turn into a
JVM-language flame war.  If Scala, why not Groovy? Certainly the
syntax is closer to Java, and the community has accepted it as a valid
language for writing unit tests, although we stopped short for
allowing it for the deployable NiFi codebase, for the same reasons
IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument
(albeit more tenuous) goes for Clojure and just about every other JVM
language (although I don't expect a call for LuaJ processors lol).

Whether we decide to support various languages ad-hoc or not, I would
strenuously object to multiple/hybrid build systems for the deployed
artifacts. If I could switch NiFi completely to Gradle I would, but I
realize there are good reasons for not doing so (yet?) in the Apache
NiFi community, and I would never want any hybrid Maven/Gradle build
for the deployable code, likewise for SBT, Leiningen, etc. With a
custom Mojo for Maven NAR builds, and the complexity for hybrid builds
in general, I think this would create a maintenance nightmare.

The language thing is a tough decision though, it's not awesome that
specifying a single language can be a barrier to a more diverse
community, certainly Scala-based bundles would be more than welcome in
the overall NiFi ecosystem, I just think the cons outweigh the pros
for the baseline code. I've written Groovy processors/NARs using
Gradle as the build system, and I'm good with keeping them in my own
repo, especially when the Extension Registry becomes a thing. I can
see the Extension Registry perhaps making this a moot point, but
clearly we need to have this discussion in the meantime.

Regards,
Matt


On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]> wrote:

> Wasn't there a warning trigger about the NiFi distro size from Apache
> recently? IMO, before talking alternative languages, solve the modularity
> and NAR distribution problem. I think the implementation of a module won't
> matter much then, the point being not everything has to go in the core,
> base distribution, but can still be easily sourced from a known repo, for
> example.
>
> I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)
>
> Andrew
>
> On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]> wrote:
>
>> This probably necessitates a vote, yeah?
>>
>> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
>> problems using processors written in Scala, but I think it’ll be important
>> to tread lightly.
>>
>> There’s a few things that pop into my head:
>>
>> - Maintainability and reviewability. A very very good Java developer need
>> not, by definition, either know how to write or identify good Scala or spot
>> problems and bugs.
>> - Every Scala processor would either end up with a 5MB scala-lang.jar
>> packaged into the .nar or we’d have to start including it in the core
>> somewhere, if it’s not. It’s possible it might have already gotten pulled
>> up from other dependencies.
>> - Style. There’s a tremendous amount of variation in Scala style because
>> of its type system, implicits, macros, and functional nature. There are
>> very good people out there that can write good Scala that isn’t readable by
>> the 99%.
>> - Binary compatibility. Scala tends to be a little more brazen about
>> breaking binary compatibility in major releases and those happen a bit more
>> often than with Java. That’s not a problem for any potential source code in
>> the project, but it could present some dependency issues someday.
>> - Testing. There’s N > 1 test frameworks and testing styles within those,
>> so there’s a lot of options for introducing more variability into the tests.
>> - NiFi uses a lot of statics in setting up properties and relationships
>> and the like, and idiomatic Scala approaches that stuff a bit differently,
>> so it’ll be necessary to impose some style guidelines so there isn’t too
>> much variation.
>>
>> That said, there are some things that won’t be problematic:
>>
>> - As mentioned, processors written in Scala do just work.
>> - The scala-maven-plugin works just fine allowing mixed Java-Scala
>> projects (btw, it’d probably be not super great to do mixed Java-Scala and
>> mixed Maven-SBT though).
>> - A lot of the above concerns could be addressed by having clear style
>> guidelines.
>>
>> Another thing: most of the projects I see deliver separate jars for Scala
>> components are delivering idiomatic APIs wrapping Java (or vice versa). I
>> think publishing   a separate set of jars/nars for stuff written in Scala
>> would be odd since here it’d mostly be processors with new functionality
>> and not functionality for using Scala. I could imagine a lib of implicits,
>> traits, classes that could make the Scala development more enjoyable. That
>> probably would make sense to deliver that way.
>>
>> -joey
>>
>> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]>, wrote:
>> > I agree more with Andy about sticking with Java. The more varying
>> languages
>> > used, the more challenging it is to maintain. Once the code is part of
>> the
>> > Apache NiFi git repo, it is now the responsibility of the committers and
>> > PMC members to maintain it.
>> >
>> > I’d even say I am somewhat against the groovy/Spock test code that Andy
>> > mentioned. I have frequently spent hours trying to fix a Spock test that
>> > broke from something I was working on. Every committer is familiar with
>> > JUnit, but only a couple know Spock. Just using this as an example that
>> > every committer knows Java, but only a couple probably know Scala,
>> Clojure,
>> > etc.
>> >
>> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]> wrote:
>> >
>> > > +1 to Joe's response. If you can develop a component in Groovy or Scala
>> > > (or Clojure!) more quickly/comfortably, or if allowing components
>> written
>> > > in other languages would encourage people to contribute more, I'm all
>> for
>> > > it.
>> > >
>> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]> wrote:
>> > >
>> > > > i personally would be ok with it for an extension/processor provided
>> it
>> > > > integrates well with the build.
>> > > >
>> > > > i would agree with andys view for core framework stuff but for
>> > > extensions i
>> > > > think we can do it like mikethomsen suggested.
>> > > >
>> > > > others?
>> > > >
>> > > > thanks
>> > > > joe
>> > > >
>> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]>
>> wrote:
>> > > >
>> > > > > I'm just a community contributor, so take that FWIW, but a
>> compromise
>> > > > might
>> > > > > be to publish the Scala code as separate maven modules to maven
>> central
>> > > > and
>> > > > > then submit a thoroughly tested processor written in Java. As long
>> as
>> > > you
>> > > > > have enough unit and integration tests to give strong coverage, I
>> > > > wouldn't
>> > > > > imagine anyone here would have issues reviewing it. If the tests
>> fail
>> > > > > because of code issues in the external dependencies, the obvious
>> answer
>> > > > is
>> > > > > to just hold the PR until the tests pass.
>> > > > >
>> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
>> > > [hidden email]
>> > > > > wrote:
>> > > > >
>> > > > > > Devs,
>> > > > > >
>> > > > > > I have some interns starting with my team and we use Scala
>> internally
>> > > > for
>> > > > > > our work.
>> > > > > > If I wanted to have them work to contribute some new processors,
>> > > would
>> > > > > > they have to be Java to be included with the base distribution or
>> > > could
>> > > > > > they use Scala as long as it fit within your current build and
>> test
>> > > > > process?
>> > > > > >
>> > > > > > Thanks,
>> > > > > > -Adam
>> > > > > >
>> > > > >
>> > > >
>> > >
>> > --
>> > Sent from Gmail Mobile
>>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

trkurc
Administrator
It is like Matt read my mind.

On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <[hidden email]> wrote:

> I'm fine with a vote, but I'll be voting to keep Java as the single
> language for the (non-test) code. I share the same concerns as many of
> the other folks as far as accepting other languages, it's mainly the
> "slippery slope" argument that I don't want to turn into a
> JVM-language flame war.  If Scala, why not Groovy? Certainly the
> syntax is closer to Java, and the community has accepted it as a valid
> language for writing unit tests, although we stopped short for
> allowing it for the deployable NiFi codebase, for the same reasons
> IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument
> (albeit more tenuous) goes for Clojure and just about every other JVM
> language (although I don't expect a call for LuaJ processors lol).
>
> Whether we decide to support various languages ad-hoc or not, I would
> strenuously object to multiple/hybrid build systems for the deployed
> artifacts. If I could switch NiFi completely to Gradle I would, but I
> realize there are good reasons for not doing so (yet?) in the Apache
> NiFi community, and I would never want any hybrid Maven/Gradle build
> for the deployable code, likewise for SBT, Leiningen, etc. With a
> custom Mojo for Maven NAR builds, and the complexity for hybrid builds
> in general, I think this would create a maintenance nightmare.
>
> The language thing is a tough decision though, it's not awesome that
> specifying a single language can be a barrier to a more diverse
> community, certainly Scala-based bundles would be more than welcome in
> the overall NiFi ecosystem, I just think the cons outweigh the pros
> for the baseline code. I've written Groovy processors/NARs using
> Gradle as the build system, and I'm good with keeping them in my own
> repo, especially when the Extension Registry becomes a thing. I can
> see the Extension Registry perhaps making this a moot point, but
> clearly we need to have this discussion in the meantime.
>
> Regards,
> Matt
>
>
> On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]> wrote:
> > Wasn't there a warning trigger about the NiFi distro size from Apache
> > recently? IMO, before talking alternative languages, solve the modularity
> > and NAR distribution problem. I think the implementation of a module
> won't
> > matter much then, the point being not everything has to go in the core,
> > base distribution, but can still be easily sourced from a known repo, for
> > example.
> >
> > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)
> >
> > Andrew
> >
> > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]>
> wrote:
> >
> >> This probably necessitates a vote, yeah?
> >>
> >> Frankly, I’m usually happier writing Scala, and I’ve not encountered any
> >> problems using processors written in Scala, but I think it’ll be
> important
> >> to tread lightly.
> >>
> >> There’s a few things that pop into my head:
> >>
> >> - Maintainability and reviewability. A very very good Java developer
> need
> >> not, by definition, either know how to write or identify good Scala or
> spot
> >> problems and bugs.
> >> - Every Scala processor would either end up with a 5MB scala-lang.jar
> >> packaged into the .nar or we’d have to start including it in the core
> >> somewhere, if it’s not. It’s possible it might have already gotten
> pulled
> >> up from other dependencies.
> >> - Style. There’s a tremendous amount of variation in Scala style because
> >> of its type system, implicits, macros, and functional nature. There are
> >> very good people out there that can write good Scala that isn’t
> readable by
> >> the 99%.
> >> - Binary compatibility. Scala tends to be a little more brazen about
> >> breaking binary compatibility in major releases and those happen a bit
> more
> >> often than with Java. That’s not a problem for any potential source
> code in
> >> the project, but it could present some dependency issues someday.
> >> - Testing. There’s N > 1 test frameworks and testing styles within
> those,
> >> so there’s a lot of options for introducing more variability into the
> tests.
> >> - NiFi uses a lot of statics in setting up properties and relationships
> >> and the like, and idiomatic Scala approaches that stuff a bit
> differently,
> >> so it’ll be necessary to impose some style guidelines so there isn’t too
> >> much variation.
> >>
> >> That said, there are some things that won’t be problematic:
> >>
> >> - As mentioned, processors written in Scala do just work.
> >> - The scala-maven-plugin works just fine allowing mixed Java-Scala
> >> projects (btw, it’d probably be not super great to do mixed Java-Scala
> and
> >> mixed Maven-SBT though).
> >> - A lot of the above concerns could be addressed by having clear style
> >> guidelines.
> >>
> >> Another thing: most of the projects I see deliver separate jars for
> Scala
> >> components are delivering idiomatic APIs wrapping Java (or vice versa).
> I
> >> think publishing   a separate set of jars/nars for stuff written in
> Scala
> >> would be odd since here it’d mostly be processors with new functionality
> >> and not functionality for using Scala. I could imagine a lib of
> implicits,
> >> traits, classes that could make the Scala development more enjoyable.
> That
> >> probably would make sense to deliver that way.
> >>
> >> -joey
> >>
> >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]>, wrote:
> >> > I agree more with Andy about sticking with Java. The more varying
> >> languages
> >> > used, the more challenging it is to maintain. Once the code is part of
> >> the
> >> > Apache NiFi git repo, it is now the responsibility of the committers
> and
> >> > PMC members to maintain it.
> >> >
> >> > I’d even say I am somewhat against the groovy/Spock test code that
> Andy
> >> > mentioned. I have frequently spent hours trying to fix a Spock test
> that
> >> > broke from something I was working on. Every committer is familiar
> with
> >> > JUnit, but only a couple know Spock. Just using this as an example
> that
> >> > every committer knows Java, but only a couple probably know Scala,
> >> Clojure,
> >> > etc.
> >> >
> >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]> wrote:
> >> >
> >> > > +1 to Joe's response. If you can develop a component in Groovy or
> Scala
> >> > > (or Clojure!) more quickly/comfortably, or if allowing components
> >> written
> >> > > in other languages would encourage people to contribute more, I'm
> all
> >> for
> >> > > it.
> >> > >
> >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]>
> wrote:
> >> > >
> >> > > > i personally would be ok with it for an extension/processor
> provided
> >> it
> >> > > > integrates well with the build.
> >> > > >
> >> > > > i would agree with andys view for core framework stuff but for
> >> > > extensions i
> >> > > > think we can do it like mikethomsen suggested.
> >> > > >
> >> > > > others?
> >> > > >
> >> > > > thanks
> >> > > > joe
> >> > > >
> >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]>
> >> wrote:
> >> > > >
> >> > > > > I'm just a community contributor, so take that FWIW, but a
> >> compromise
> >> > > > might
> >> > > > > be to publish the Scala code as separate maven modules to maven
> >> central
> >> > > > and
> >> > > > > then submit a thoroughly tested processor written in Java. As
> long
> >> as
> >> > > you
> >> > > > > have enough unit and integration tests to give strong coverage,
> I
> >> > > > wouldn't
> >> > > > > imagine anyone here would have issues reviewing it. If the tests
> >> fail
> >> > > > > because of code issues in the external dependencies, the obvious
> >> answer
> >> > > > is
> >> > > > > to just hold the PR until the tests pass.
> >> > > > >
> >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <
> >> > > [hidden email]
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Devs,
> >> > > > > >
> >> > > > > > I have some interns starting with my team and we use Scala
> >> internally
> >> > > > for
> >> > > > > > our work.
> >> > > > > > If I wanted to have them work to contribute some new
> processors,
> >> > > would
> >> > > > > > they have to be Java to be included with the base
> distribution or
> >> > > could
> >> > > > > > they use Scala as long as it fit within your current build and
> >> test
> >> > > > > process?
> >> > > > > >
> >> > > > > > Thanks,
> >> > > > > > -Adam
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> > --
> >> > Sent from Gmail Mobile
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Milan Das
I think we should not add blindly any language but should be open to add couple of language like Scala.
In Bigdata world Scala/Python/Java are widely accepted.

Thanks,
Milan Das
Interset

On 2/13/18, 10:20 AM, "Weiss, Adam" <[hidden email]> wrote:

    I think it makes the most sense to me for us to publish a separate repo with a module and nar build for now and post when it's available in the users group.
   
    Thanks for the discussion everyone, hopefully we can start making some helpful contributions soon.
   
    -Adam
   
   
    On 2018/02/10 23:43:31, Tony Kurc <[hidden email]<mailto:[hidden email]>> wrote:
    > It is like Matt read my mind.>
    >
    > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <[hidden email]<mailto:[hidden email]>> wrote:>
    >
    > > I'm fine with a vote, but I'll be voting to keep Java as the single>
    > > language for the (non-test) code. I share the same concerns as many of>
    > > the other folks as far as accepting other languages, it's mainly the>
    > > "slippery slope" argument that I don't want to turn into a>
    > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
    > > syntax is closer to Java, and the community has accepted it as a valid>
    > > language for writing unit tests, although we stopped short for>
    > > allowing it for the deployable NiFi codebase, for the same reasons>
    > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
    > > (albeit more tenuous) goes for Clojure and just about every other JVM>
    > > language (although I don't expect a call for LuaJ processors lol).>
    > >>
    > > Whether we decide to support various languages ad-hoc or not, I would>
    > > strenuously object to multiple/hybrid build systems for the deployed>
    > > artifacts. If I could switch NiFi completely to Gradle I would, but I>
    > > realize there are good reasons for not doing so (yet?) in the Apache>
    > > NiFi community, and I would never want any hybrid Maven/Gradle build>
    > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
    > > custom Mojo for Maven NAR builds, and the complexity for hybrid builds>
    > > in general, I think this would create a maintenance nightmare.>
    > >>
    > > The language thing is a tough decision though, it's not awesome that>
    > > specifying a single language can be a barrier to a more diverse>
    > > community, certainly Scala-based bundles would be more than welcome in>
    > > the overall NiFi ecosystem, I just think the cons outweigh the pros>
    > > for the baseline code. I've written Groovy processors/NARs using>
    > > Gradle as the build system, and I'm good with keeping them in my own>
    > > repo, especially when the Extension Registry becomes a thing. I can>
    > > see the Extension Registry perhaps making this a moot point, but>
    > > clearly we need to have this discussion in the meantime.>
    > >>
    > > Regards,>
    > > Matt>
    > >>
    > >>
    > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]<mailto:[hidden email]>> wrote:>
    > > > Wasn't there a warning trigger about the NiFi distro size from Apache>
    > > > recently? IMO, before talking alternative languages, solve the modularity>
    > > > and NAR distribution problem. I think the implementation of a module>
    > > won't>
    > > > matter much then, the point being not everything has to go in the core,>
    > > > base distribution, but can still be easily sourced from a known repo, for>
    > > > example.>
    > > >>
    > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size soon :)>
    > > >>
    > > > Andrew>
    > > >>
    > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]<mailto:[hidden email]>>>
    > > wrote:>
    > > >>
    > > >> This probably necessitates a vote, yeah?>
    > > >>>
    > > >> Frankly, I’m usually happier writing Scala, and I’ve not encountered any>
    > > >> problems using processors written in Scala, but I think it’ll be>
    > > important>
    > > >> to tread lightly.>
    > > >>>
    > > >> There’s a few things that pop into my head:>
    > > >>>
    > > >> - Maintainability and reviewability. A very very good Java developer>
    > > need>
    > > >> not, by definition, either know how to write or identify good Scala or>
    > > spot>
    > > >> problems and bugs.>
    > > >> - Every Scala processor would either end up with a 5MB scala-lang.jar>
    > > >> packaged into the .nar or we’d have to start including it in the core>
    > > >> somewhere, if it’s not. It’s possible it might have already gotten>
    > > pulled>
    > > >> up from other dependencies.>
    > > >> - Style. There’s a tremendous amount of variation in Scala style because>
    > > >> of its type system, implicits, macros, and functional nature. There are>
    > > >> very good people out there that can write good Scala that isn’t>
    > > readable by>
    > > >> the 99%.>
    > > >> - Binary compatibility. Scala tends to be a little more brazen about>
    > > >> breaking binary compatibility in major releases and those happen a bit>
    > > more>
    > > >> often than with Java. That’s not a problem for any potential source>
    > > code in>
    > > >> the project, but it could present some dependency issues someday.>
    > > >> - Testing. There’s N > 1 test frameworks and testing styles within>
    > > those,>
    > > >> so there’s a lot of options for introducing more variability into the>
    > > tests.>
    > > >> - NiFi uses a lot of statics in setting up properties and relationships>
    > > >> and the like, and idiomatic Scala approaches that stuff a bit>
    > > differently,>
    > > >> so it’ll be necessary to impose some style guidelines so there isn’t too>
    > > >> much variation.>
    > > >>>
    > > >> That said, there are some things that won’t be problematic:>
    > > >>>
    > > >> - As mentioned, processors written in Scala do just work.>
    > > >> - The scala-maven-plugin works just fine allowing mixed Java-Scala>
    > > >> projects (btw, it’d probably be not super great to do mixed Java-Scala>
    > > and>
    > > >> mixed Maven-SBT though).>
    > > >> - A lot of the above concerns could be addressed by having clear style>
    > > >> guidelines.>
    > > >>>
    > > >> Another thing: most of the projects I see deliver separate jars for>
    > > Scala>
    > > >> components are delivering idiomatic APIs wrapping Java (or vice versa).>
    > > I>
    > > >> think publishing   a separate set of jars/nars for stuff written in>
    > > Scala>
    > > >> would be odd since here it’d mostly be processors with new functionality>
    > > >> and not functionality for using Scala. I could imagine a lib of>
    > > implicits,>
    > > >> traits, classes that could make the Scala development more enjoyable.>
    > > That>
    > > >> probably would make sense to deliver that way.>
    > > >>>
    > > >> -joey>
    > > >>>
    > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]<mailto:[hidden email]>>, wrote:>
    > > >> > I agree more with Andy about sticking with Java. The more varying>
    > > >> languages>
    > > >> > used, the more challenging it is to maintain. Once the code is part of>
    > > >> the>
    > > >> > Apache NiFi git repo, it is now the responsibility of the committers>
    > > and>
    > > >> > PMC members to maintain it.>
    > > >> >>
    > > >> > I’d even say I am somewhat against the groovy/Spock test code that>
    > > Andy>
    > > >> > mentioned. I have frequently spent hours trying to fix a Spock test>
    > > that>
    > > >> > broke from something I was working on. Every committer is familiar>
    > > with>
    > > >> > JUnit, but only a couple know Spock. Just using this as an example>
    > > that>
    > > >> > every committer knows Java, but only a couple probably know Scala,>
    > > >> Clojure,>
    > > >> > etc.>
    > > >> >>
    > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]<mailto:[hidden email]>> wrote:>
    > > >> >>
    > > >> > > +1 to Joe's response. If you can develop a component in Groovy or>
    > > Scala>
    > > >> > > (or Clojure!) more quickly/comfortably, or if allowing components>
    > > >> written>
    > > >> > > in other languages would encourage people to contribute more, I'm>
    > > all>
    > > >> for>
    > > >> > > it.>
    > > >> > >>
    > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]<mailto:[hidden email]>>>
    > > wrote:>
    > > >> > >>
    > > >> > > > i personally would be ok with it for an extension/processor>
    > > provided>
    > > >> it>
    > > >> > > > integrates well with the build.>
    > > >> > > >>
    > > >> > > > i would agree with andys view for core framework stuff but for>
    > > >> > > extensions i>
    > > >> > > > think we can do it like mikethomsen suggested.>
    > > >> > > >>
    > > >> > > > others?>
    > > >> > > >>
    > > >> > > > thanks>
    > > >> > > > joe>
    > > >> > > >>
    > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]<mailto:[hidden email]>>>
    > > >> wrote:>
    > > >> > > >>
    > > >> > > > > I'm just a community contributor, so take that FWIW, but a>
    > > >> compromise>
    > > >> > > > might>
    > > >> > > > > be to publish the Scala code as separate maven modules to maven>
    > > >> central>
    > > >> > > > and>
    > > >> > > > > then submit a thoroughly tested processor written in Java. As>
    > > long>
    > > >> as>
    > > >> > > you>
    > > >> > > > > have enough unit and integration tests to give strong coverage,>
    > > I>
    > > >> > > > wouldn't>
    > > >> > > > > imagine anyone here would have issues reviewing it. If the tests>
    > > >> fail>
    > > >> > > > > because of code issues in the external dependencies, the obvious>
    > > >> answer>
    > > >> > > > is>
    > > >> > > > > to just hold the PR until the tests pass.>
    > > >> > > > >>
    > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <>
    > > >> > > [hidden email]<mailto:[hidden email]>>
    > > >> > > > > wrote:>
    > > >> > > > >>
    > > >> > > > > > Devs,>
    > > >> > > > > >>
    > > >> > > > > > I have some interns starting with my team and we use Scala>
    > > >> internally>
    > > >> > > > for>
    > > >> > > > > > our work.>
    > > >> > > > > > If I wanted to have them work to contribute some new>
    > > processors,>
    > > >> > > would>
    > > >> > > > > > they have to be Java to be included with the base>
    > > distribution or>
    > > >> > > could>
    > > >> > > > > > they use Scala as long as it fit within your current build and>
    > > >> test>
    > > >> > > > > process?>
    > > >> > > > > >>
    > > >> > > > > > Thanks,>
    > > >> > > > > > -Adam>
    > > >> > > > > >>
    > > >> > > > >>
    > > >> > > >>
    > > >> > >>
    > > >> > -->
    > > >> > Sent from Gmail Mobile>
    > > >>>
    > >>
    >
   
    Adam Weiss | Senior Manager, Digital Solutions R&D
    PerkinElmer | Innovating for a Healthier World
    [hidden email]<mailto:[hidden email]>
    Mobile: +1 716.429.7324
    77 Goodell St. Suite 470, Buffalo, NY, 14203
    www.perkinelmer.com
   


Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Mike Thomsen
Milan,

I don't think you can do that without creating a lot of fuel for a flame
war. I have personally never met a Scala developer who was incapable of
writing decent Java. The same is true of Groovy. I think anyone who finds
the requirement to use Java over their language of choice to be a deal
breaker on contributing is probably someone unlikely to be more help than
trouble anyway.

Mike

On Tue, Feb 13, 2018 at 11:35 AM, Milan Das <[hidden email]> wrote:

> I think we should not add blindly any language but should be open to add
> couple of language like Scala.
> In Bigdata world Scala/Python/Java are widely accepted.
>
> Thanks,
> Milan Das
> Interset
>
> On 2/13/18, 10:20 AM, "Weiss, Adam" <[hidden email]> wrote:
>
>     I think it makes the most sense to me for us to publish a separate
> repo with a module and nar build for now and post when it's available in
> the users group.
>
>     Thanks for the discussion everyone, hopefully we can start making some
> helpful contributions soon.
>
>     -Adam
>
>
>     On 2018/02/10 23:43:31, Tony Kurc <[hidden email]<mailto:t...@
> gmail.com>> wrote:
>     > It is like Matt read my mind.>
>     >
>     > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <[hidden email]
> <mailto:[hidden email]>> wrote:>
>     >
>     > > I'm fine with a vote, but I'll be voting to keep Java as the
> single>
>     > > language for the (non-test) code. I share the same concerns as
> many of>
>     > > the other folks as far as accepting other languages, it's mainly
> the>
>     > > "slippery slope" argument that I don't want to turn into a>
>     > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
>     > > syntax is closer to Java, and the community has accepted it as a
> valid>
>     > > language for writing unit tests, although we stopped short for>
>     > > allowing it for the deployable NiFi codebase, for the same reasons>
>     > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
>     > > (albeit more tenuous) goes for Clojure and just about every other
> JVM>
>     > > language (although I don't expect a call for LuaJ processors lol).>
>     > >>
>     > > Whether we decide to support various languages ad-hoc or not, I
> would>
>     > > strenuously object to multiple/hybrid build systems for the
> deployed>
>     > > artifacts. If I could switch NiFi completely to Gradle I would,
> but I>
>     > > realize there are good reasons for not doing so (yet?) in the
> Apache>
>     > > NiFi community, and I would never want any hybrid Maven/Gradle
> build>
>     > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>     > > custom Mojo for Maven NAR builds, and the complexity for hybrid
> builds>
>     > > in general, I think this would create a maintenance nightmare.>
>     > >>
>     > > The language thing is a tough decision though, it's not awesome
> that>
>     > > specifying a single language can be a barrier to a more diverse>
>     > > community, certainly Scala-based bundles would be more than
> welcome in>
>     > > the overall NiFi ecosystem, I just think the cons outweigh the
> pros>
>     > > for the baseline code. I've written Groovy processors/NARs using>
>     > > Gradle as the build system, and I'm good with keeping them in my
> own>
>     > > repo, especially when the Extension Registry becomes a thing. I
> can>
>     > > see the Extension Registry perhaps making this a moot point, but>
>     > > clearly we need to have this discussion in the meantime.>
>     > >>
>     > > Regards,>
>     > > Matt>
>     > >>
>     > >>
>     > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]
> <mailto:[hidden email]>> wrote:>
>     > > > Wasn't there a warning trigger about the NiFi distro size from
> Apache>
>     > > > recently? IMO, before talking alternative languages, solve the
> modularity>
>     > > > and NAR distribution problem. I think the implementation of a
> module>
>     > > won't>
>     > > > matter much then, the point being not everything has to go in
> the core,>
>     > > > base distribution, but can still be easily sourced from a known
> repo, for>
>     > > > example.>
>     > > >>
>     > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
> soon :)>
>     > > >>
>     > > > Andrew>
>     > > >>
>     > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]
> <mailto:[hidden email]>>>
>     > > wrote:>
>     > > >>
>     > > >> This probably necessitates a vote, yeah?>
>     > > >>>
>     > > >> Frankly, I’m usually happier writing Scala, and I’ve not
> encountered any>
>     > > >> problems using processors written in Scala, but I think it’ll
> be>
>     > > important>
>     > > >> to tread lightly.>
>     > > >>>
>     > > >> There’s a few things that pop into my head:>
>     > > >>>
>     > > >> - Maintainability and reviewability. A very very good Java
> developer>
>     > > need>
>     > > >> not, by definition, either know how to write or identify good
> Scala or>
>     > > spot>
>     > > >> problems and bugs.>
>     > > >> - Every Scala processor would either end up with a 5MB
> scala-lang.jar>
>     > > >> packaged into the .nar or we’d have to start including it in
> the core>
>     > > >> somewhere, if it’s not. It’s possible it might have already
> gotten>
>     > > pulled>
>     > > >> up from other dependencies.>
>     > > >> - Style. There’s a tremendous amount of variation in Scala
> style because>
>     > > >> of its type system, implicits, macros, and functional nature.
> There are>
>     > > >> very good people out there that can write good Scala that isn’t>
>     > > readable by>
>     > > >> the 99%.>
>     > > >> - Binary compatibility. Scala tends to be a little more brazen
> about>
>     > > >> breaking binary compatibility in major releases and those
> happen a bit>
>     > > more>
>     > > >> often than with Java. That’s not a problem for any potential
> source>
>     > > code in>
>     > > >> the project, but it could present some dependency issues
> someday.>
>     > > >> - Testing. There’s N > 1 test frameworks and testing styles
> within>
>     > > those,>
>     > > >> so there’s a lot of options for introducing more variability
> into the>
>     > > tests.>
>     > > >> - NiFi uses a lot of statics in setting up properties and
> relationships>
>     > > >> and the like, and idiomatic Scala approaches that stuff a bit>
>     > > differently,>
>     > > >> so it’ll be necessary to impose some style guidelines so there
> isn’t too>
>     > > >> much variation.>
>     > > >>>
>     > > >> That said, there are some things that won’t be problematic:>
>     > > >>>
>     > > >> - As mentioned, processors written in Scala do just work.>
>     > > >> - The scala-maven-plugin works just fine allowing mixed
> Java-Scala>
>     > > >> projects (btw, it’d probably be not super great to do mixed
> Java-Scala>
>     > > and>
>     > > >> mixed Maven-SBT though).>
>     > > >> - A lot of the above concerns could be addressed by having
> clear style>
>     > > >> guidelines.>
>     > > >>>
>     > > >> Another thing: most of the projects I see deliver separate jars
> for>
>     > > Scala>
>     > > >> components are delivering idiomatic APIs wrapping Java (or vice
> versa).>
>     > > I>
>     > > >> think publishing   a separate set of jars/nars for stuff
> written in>
>     > > Scala>
>     > > >> would be odd since here it’d mostly be processors with new
> functionality>
>     > > >> and not functionality for using Scala. I could imagine a lib of>
>     > > implicits,>
>     > > >> traits, classes that could make the Scala development more
> enjoyable.>
>     > > That>
>     > > >> probably would make sense to deliver that way.>
>     > > >>>
>     > > >> -joey>
>     > > >>>
>     > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]
> <mailto:[hidden email]>>, wrote:>
>     > > >> > I agree more with Andy about sticking with Java. The more
> varying>
>     > > >> languages>
>     > > >> > used, the more challenging it is to maintain. Once the code
> is part of>
>     > > >> the>
>     > > >> > Apache NiFi git repo, it is now the responsibility of the
> committers>
>     > > and>
>     > > >> > PMC members to maintain it.>
>     > > >> >>
>     > > >> > I’d even say I am somewhat against the groovy/Spock test code
> that>
>     > > Andy>
>     > > >> > mentioned. I have frequently spent hours trying to fix a
> Spock test>
>     > > that>
>     > > >> > broke from something I was working on. Every committer is
> familiar>
>     > > with>
>     > > >> > JUnit, but only a couple know Spock. Just using this as an
> example>
>     > > that>
>     > > >> > every committer knows Java, but only a couple probably know
> Scala,>
>     > > >> Clojure,>
>     > > >> > etc.>
>     > > >> >>
>     > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]
> <mailto:[hidden email]>> wrote:>
>     > > >> >>
>     > > >> > > +1 to Joe's response. If you can develop a component in
> Groovy or>
>     > > Scala>
>     > > >> > > (or Clojure!) more quickly/comfortably, or if allowing
> components>
>     > > >> written>
>     > > >> > > in other languages would encourage people to contribute
> more, I'm>
>     > > all>
>     > > >> for>
>     > > >> > > it.>
>     > > >> > >>
>     > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]
> <mailto:[hidden email]>>>
>     > > wrote:>
>     > > >> > >>
>     > > >> > > > i personally would be ok with it for an
> extension/processor>
>     > > provided>
>     > > >> it>
>     > > >> > > > integrates well with the build.>
>     > > >> > > >>
>     > > >> > > > i would agree with andys view for core framework stuff
> but for>
>     > > >> > > extensions i>
>     > > >> > > > think we can do it like mikethomsen suggested.>
>     > > >> > > >>
>     > > >> > > > others?>
>     > > >> > > >>
>     > > >> > > > thanks>
>     > > >> > > > joe>
>     > > >> > > >>
>     > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]
> <mailto:[hidden email]>>>
>     > > >> wrote:>
>     > > >> > > >>
>     > > >> > > > > I'm just a community contributor, so take that FWIW,
> but a>
>     > > >> compromise>
>     > > >> > > > might>
>     > > >> > > > > be to publish the Scala code as separate maven modules
> to maven>
>     > > >> central>
>     > > >> > > > and>
>     > > >> > > > > then submit a thoroughly tested processor written in
> Java. As>
>     > > long>
>     > > >> as>
>     > > >> > > you>
>     > > >> > > > > have enough unit and integration tests to give strong
> coverage,>
>     > > I>
>     > > >> > > > wouldn't>
>     > > >> > > > > imagine anyone here would have issues reviewing it. If
> the tests>
>     > > >> fail>
>     > > >> > > > > because of code issues in the external dependencies,
> the obvious>
>     > > >> answer>
>     > > >> > > > is>
>     > > >> > > > > to just hold the PR until the tests pass.>
>     > > >> > > > >>
>     > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <>
>     > > >> > > [hidden email]<mailto:Adam.Weiss@perkinelmer.
> com>>
>     > > >> > > > > wrote:>
>     > > >> > > > >>
>     > > >> > > > > > Devs,>
>     > > >> > > > > >>
>     > > >> > > > > > I have some interns starting with my team and we use
> Scala>
>     > > >> internally>
>     > > >> > > > for>
>     > > >> > > > > > our work.>
>     > > >> > > > > > If I wanted to have them work to contribute some new>
>     > > processors,>
>     > > >> > > would>
>     > > >> > > > > > they have to be Java to be included with the base>
>     > > distribution or>
>     > > >> > > could>
>     > > >> > > > > > they use Scala as long as it fit within your current
> build and>
>     > > >> test>
>     > > >> > > > > process?>
>     > > >> > > > > >>
>     > > >> > > > > > Thanks,>
>     > > >> > > > > > -Adam>
>     > > >> > > > > >>
>     > > >> > > > >>
>     > > >> > > >>
>     > > >> > >>
>     > > >> > -->
>     > > >> > Sent from Gmail Mobile>
>     > > >>>
>     > >>
>     >
>
>     Adam Weiss | Senior Manager, Digital Solutions R&D
>     PerkinElmer | Innovating for a Healthier World
>     [hidden email]<mailto:[hidden email]>
>     Mobile: +1 716.429.7324
>     77 Goodell St. Suite 470, Buffalo, NY, 14203
>     www.perkinelmer.com
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Russell Bateman-2
I completely second Mike's sentiment here as well as cautionary
statements by other contributors over this thread's history (Matt
Burgess' post in particular). Besides fueling flame wars and religious
inquisition, you cannot open the door to Scala without opening it to
other (JVM languages at least) and there's no umbrella that will shield
you from the chaos doing so will unleash rendering NiFi difficult to
develop to (and maybe even to deploy). As it is presently, all
JVM-language developers are able to consume NiFi easily for development
and to deal with their own deployment issues.

Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:

> Milan,
>
> I don't think you can do that without creating a lot of fuel for a flame
> war. I have personally never met a Scala developer who was incapable of
> writing decent Java. The same is true of Groovy. I think anyone who finds
> the requirement to use Java over their language of choice to be a deal
> breaker on contributing is probably someone unlikely to be more help than
> trouble anyway.
>
> Mike
>
> On Tue, Feb 13, 2018 at 11:35 AM, Milan Das <[hidden email]> wrote:
>
>> I think we should not add blindly any language but should be open to add
>> couple of language like Scala.
>> In Bigdata world Scala/Python/Java are widely accepted.
>>
>> Thanks,
>> Milan Das
>> Interset
>>
>> On 2/13/18, 10:20 AM, "Weiss, Adam" <[hidden email]> wrote:
>>
>>      I think it makes the most sense to me for us to publish a separate
>> repo with a module and nar build for now and post when it's available in
>> the users group.
>>
>>      Thanks for the discussion everyone, hopefully we can start making some
>> helpful contributions soon.
>>
>>      -Adam
>>
>>
>>      On 2018/02/10 23:43:31, Tony Kurc <[hidden email]<mailto:t...@
>> gmail.com>> wrote:
>>      > It is like Matt read my mind.>
>>      >
>>      > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>>      >
>>      > > I'm fine with a vote, but I'll be voting to keep Java as the
>> single>
>>      > > language for the (non-test) code. I share the same concerns as
>> many of>
>>      > > the other folks as far as accepting other languages, it's mainly
>> the>
>>      > > "slippery slope" argument that I don't want to turn into a>
>>      > > JVM-language flame war.  If Scala, why not Groovy? Certainly the>
>>      > > syntax is closer to Java, and the community has accepted it as a
>> valid>
>>      > > language for writing unit tests, although we stopped short for>
>>      > > allowing it for the deployable NiFi codebase, for the same reasons>
>>      > > IIRC.  If Scala and/or Groovy, why not Kotlin?  The same argument>
>>      > > (albeit more tenuous) goes for Clojure and just about every other
>> JVM>
>>      > > language (although I don't expect a call for LuaJ processors lol).>
>>      > >>
>>      > > Whether we decide to support various languages ad-hoc or not, I
>> would>
>>      > > strenuously object to multiple/hybrid build systems for the
>> deployed>
>>      > > artifacts. If I could switch NiFi completely to Gradle I would,
>> but I>
>>      > > realize there are good reasons for not doing so (yet?) in the
>> Apache>
>>      > > NiFi community, and I would never want any hybrid Maven/Gradle
>> build>
>>      > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>>      > > custom Mojo for Maven NAR builds, and the complexity for hybrid
>> builds>
>>      > > in general, I think this would create a maintenance nightmare.>
>>      > >>
>>      > > The language thing is a tough decision though, it's not awesome
>> that>
>>      > > specifying a single language can be a barrier to a more diverse>
>>      > > community, certainly Scala-based bundles would be more than
>> welcome in>
>>      > > the overall NiFi ecosystem, I just think the cons outweigh the
>> pros>
>>      > > for the baseline code. I've written Groovy processors/NARs using>
>>      > > Gradle as the build system, and I'm good with keeping them in my
>> own>
>>      > > repo, especially when the Extension Registry becomes a thing. I
>> can>
>>      > > see the Extension Registry perhaps making this a moot point, but>
>>      > > clearly we need to have this discussion in the meantime.>
>>      > >>
>>      > > Regards,>
>>      > > Matt>
>>      > >>
>>      > >>
>>      > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>>      > > > Wasn't there a warning trigger about the NiFi distro size from
>> Apache>
>>      > > > recently? IMO, before talking alternative languages, solve the
>> modularity>
>>      > > > and NAR distribution problem. I think the implementation of a
>> module>
>>      > > won't>
>>      > > > matter much then, the point being not everything has to go in
>> the core,>
>>      > > > base distribution, but can still be easily sourced from a known
>> repo, for>
>>      > > > example.>
>>      > > >>
>>      > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
>> soon :)>
>>      > > >>
>>      > > > Andrew>
>>      > > >>
>>      > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]
>> <mailto:[hidden email]>>>
>>      > > wrote:>
>>      > > >>
>>      > > >> This probably necessitates a vote, yeah?>
>>      > > >>>
>>      > > >> Frankly, I’m usually happier writing Scala, and I’ve not
>> encountered any>
>>      > > >> problems using processors written in Scala, but I think it’ll
>> be>
>>      > > important>
>>      > > >> to tread lightly.>
>>      > > >>>
>>      > > >> There’s a few things that pop into my head:>
>>      > > >>>
>>      > > >> - Maintainability and reviewability. A very very good Java
>> developer>
>>      > > need>
>>      > > >> not, by definition, either know how to write or identify good
>> Scala or>
>>      > > spot>
>>      > > >> problems and bugs.>
>>      > > >> - Every Scala processor would either end up with a 5MB
>> scala-lang.jar>
>>      > > >> packaged into the .nar or we’d have to start including it in
>> the core>
>>      > > >> somewhere, if it’s not. It’s possible it might have already
>> gotten>
>>      > > pulled>
>>      > > >> up from other dependencies.>
>>      > > >> - Style. There’s a tremendous amount of variation in Scala
>> style because>
>>      > > >> of its type system, implicits, macros, and functional nature.
>> There are>
>>      > > >> very good people out there that can write good Scala that isn’t>
>>      > > readable by>
>>      > > >> the 99%.>
>>      > > >> - Binary compatibility. Scala tends to be a little more brazen
>> about>
>>      > > >> breaking binary compatibility in major releases and those
>> happen a bit>
>>      > > more>
>>      > > >> often than with Java. That’s not a problem for any potential
>> source>
>>      > > code in>
>>      > > >> the project, but it could present some dependency issues
>> someday.>
>>      > > >> - Testing. There’s N > 1 test frameworks and testing styles
>> within>
>>      > > those,>
>>      > > >> so there’s a lot of options for introducing more variability
>> into the>
>>      > > tests.>
>>      > > >> - NiFi uses a lot of statics in setting up properties and
>> relationships>
>>      > > >> and the like, and idiomatic Scala approaches that stuff a bit>
>>      > > differently,>
>>      > > >> so it’ll be necessary to impose some style guidelines so there
>> isn’t too>
>>      > > >> much variation.>
>>      > > >>>
>>      > > >> That said, there are some things that won’t be problematic:>
>>      > > >>>
>>      > > >> - As mentioned, processors written in Scala do just work.>
>>      > > >> - The scala-maven-plugin works just fine allowing mixed
>> Java-Scala>
>>      > > >> projects (btw, it’d probably be not super great to do mixed
>> Java-Scala>
>>      > > and>
>>      > > >> mixed Maven-SBT though).>
>>      > > >> - A lot of the above concerns could be addressed by having
>> clear style>
>>      > > >> guidelines.>
>>      > > >>>
>>      > > >> Another thing: most of the projects I see deliver separate jars
>> for>
>>      > > Scala>
>>      > > >> components are delivering idiomatic APIs wrapping Java (or vice
>> versa).>
>>      > > I>
>>      > > >> think publishing   a separate set of jars/nars for stuff
>> written in>
>>      > > Scala>
>>      > > >> would be odd since here it’d mostly be processors with new
>> functionality>
>>      > > >> and not functionality for using Scala. I could imagine a lib of>
>>      > > implicits,>
>>      > > >> traits, classes that could make the Scala development more
>> enjoyable.>
>>      > > That>
>>      > > >> probably would make sense to deliver that way.>
>>      > > >>>
>>      > > >> -joey>
>>      > > >>>
>>      > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]
>> <mailto:[hidden email]>>, wrote:>
>>      > > >> > I agree more with Andy about sticking with Java. The more
>> varying>
>>      > > >> languages>
>>      > > >> > used, the more challenging it is to maintain. Once the code
>> is part of>
>>      > > >> the>
>>      > > >> > Apache NiFi git repo, it is now the responsibility of the
>> committers>
>>      > > and>
>>      > > >> > PMC members to maintain it.>
>>      > > >> >>
>>      > > >> > I’d even say I am somewhat against the groovy/Spock test code
>> that>
>>      > > Andy>
>>      > > >> > mentioned. I have frequently spent hours trying to fix a
>> Spock test>
>>      > > that>
>>      > > >> > broke from something I was working on. Every committer is
>> familiar>
>>      > > with>
>>      > > >> > JUnit, but only a couple know Spock. Just using this as an
>> example>
>>      > > that>
>>      > > >> > every committer knows Java, but only a couple probably know
>> Scala,>
>>      > > >> Clojure,>
>>      > > >> > etc.>
>>      > > >> >>
>>      > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>>      > > >> >>
>>      > > >> > > +1 to Joe's response. If you can develop a component in
>> Groovy or>
>>      > > Scala>
>>      > > >> > > (or Clojure!) more quickly/comfortably, or if allowing
>> components>
>>      > > >> written>
>>      > > >> > > in other languages would encourage people to contribute
>> more, I'm>
>>      > > all>
>>      > > >> for>
>>      > > >> > > it.>
>>      > > >> > >>
>>      > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]
>> <mailto:[hidden email]>>>
>>      > > wrote:>
>>      > > >> > >>
>>      > > >> > > > i personally would be ok with it for an
>> extension/processor>
>>      > > provided>
>>      > > >> it>
>>      > > >> > > > integrates well with the build.>
>>      > > >> > > >>
>>      > > >> > > > i would agree with andys view for core framework stuff
>> but for>
>>      > > >> > > extensions i>
>>      > > >> > > > think we can do it like mikethomsen suggested.>
>>      > > >> > > >>
>>      > > >> > > > others?>
>>      > > >> > > >>
>>      > > >> > > > thanks>
>>      > > >> > > > joe>
>>      > > >> > > >>
>>      > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]
>> <mailto:[hidden email]>>>
>>      > > >> wrote:>
>>      > > >> > > >>
>>      > > >> > > > > I'm just a community contributor, so take that FWIW,
>> but a>
>>      > > >> compromise>
>>      > > >> > > > might>
>>      > > >> > > > > be to publish the Scala code as separate maven modules
>> to maven>
>>      > > >> central>
>>      > > >> > > > and>
>>      > > >> > > > > then submit a thoroughly tested processor written in
>> Java. As>
>>      > > long>
>>      > > >> as>
>>      > > >> > > you>
>>      > > >> > > > > have enough unit and integration tests to give strong
>> coverage,>
>>      > > I>
>>      > > >> > > > wouldn't>
>>      > > >> > > > > imagine anyone here would have issues reviewing it. If
>> the tests>
>>      > > >> fail>
>>      > > >> > > > > because of code issues in the external dependencies,
>> the obvious>
>>      > > >> answer>
>>      > > >> > > > is>
>>      > > >> > > > > to just hold the PR until the tests pass.>
>>      > > >> > > > >>
>>      > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <>
>>      > > >> > > [hidden email]<mailto:Adam.Weiss@perkinelmer.
>> com>>
>>      > > >> > > > > wrote:>
>>      > > >> > > > >>
>>      > > >> > > > > > Devs,>
>>      > > >> > > > > >>
>>      > > >> > > > > > I have some interns starting with my team and we use
>> Scala>
>>      > > >> internally>
>>      > > >> > > > for>
>>      > > >> > > > > > our work.>
>>      > > >> > > > > > If I wanted to have them work to contribute some new>
>>      > > processors,>
>>      > > >> > > would>
>>      > > >> > > > > > they have to be Java to be included with the base>
>>      > > distribution or>
>>      > > >> > > could>
>>      > > >> > > > > > they use Scala as long as it fit within your current
>> build and>
>>      > > >> test>
>>      > > >> > > > > process?>
>>      > > >> > > > > >>
>>      > > >> > > > > > Thanks,>
>>      > > >> > > > > > -Adam>
>>      > > >> > > > > >>
>>      > > >> > > > >>
>>      > > >> > > >>
>>      > > >> > >>
>>      > > >> > -->
>>      > > >> > Sent from Gmail Mobile>
>>      > > >>>
>>      > >>
>>      >
>>
>>      Adam Weiss | Senior Manager, Digital Solutions R&D
>>      PerkinElmer | Innovating for a Healthier World
>>      [hidden email]<mailto:[hidden email]>
>>      Mobile: +1 716.429.7324
>>      77 Goodell St. Suite 470, Buffalo, NY, 14203
>>      www.perkinelmer.com
>>
>>
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: Will you accept contributions in Scala?

Otto Fowler
Sorry for the late response, but the point about the Extension Repo making
this a moot point is an important one.
The idea that you need to get your processor into the main codebase to have
it be known and useful is something
that is easier to overcome than a language war.


On February 13, 2018 at 12:15:23, Russell Bateman ([hidden email])
wrote:

I completely second Mike's sentiment here as well as cautionary
statements by other contributors over this thread's history (Matt
Burgess' post in particular). Besides fueling flame wars and religious
inquisition, you cannot open the door to Scala without opening it to
other (JVM languages at least) and there's no umbrella that will shield
you from the chaos doing so will unleash rendering NiFi difficult to
develop to (and maybe even to deploy). As it is presently, all
JVM-language developers are able to consume NiFi easily for development
and to deal with their own deployment issues.

Russ

On 02/13/2018 10:06 AM, Mike Thomsen wrote:

> Milan,
>
> I don't think you can do that without creating a lot of fuel for a flame
> war. I have personally never met a Scala developer who was incapable of
> writing decent Java. The same is true of Groovy. I think anyone who finds
> the requirement to use Java over their language of choice to be a deal
> breaker on contributing is probably someone unlikely to be more help than
> trouble anyway.
>
> Mike
>
> On Tue, Feb 13, 2018 at 11:35 AM, Milan Das <[hidden email]> wrote:
>
>> I think we should not add blindly any language but should be open to add
>> couple of language like Scala.
>> In Bigdata world Scala/Python/Java are widely accepted.
>>
>> Thanks,
>> Milan Das
>> Interset
>>
>> On 2/13/18, 10:20 AM, "Weiss, Adam" <[hidden email]> wrote:
>>
>> I think it makes the most sense to me for us to publish a separate
>> repo with a module and nar build for now and post when it's available in
>> the users group.
>>
>> Thanks for the discussion everyone, hopefully we can start making some
>> helpful contributions soon.
>>
>> -Adam
>>
>>
>> On 2018/02/10 23:43:31, Tony Kurc <[hidden email]<mailto:t...@
>> gmail.com>> wrote:
>> > It is like Matt read my mind.>
>> >
>> > On Sat, Feb 10, 2018 at 6:26 PM, Matt Burgess <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>> >
>> > > I'm fine with a vote, but I'll be voting to keep Java as the
>> single>
>> > > language for the (non-test) code. I share the same concerns as
>> many of>
>> > > the other folks as far as accepting other languages, it's mainly
>> the>
>> > > "slippery slope" argument that I don't want to turn into a>
>> > > JVM-language flame war. If Scala, why not Groovy? Certainly the>
>> > > syntax is closer to Java, and the community has accepted it as a
>> valid>
>> > > language for writing unit tests, although we stopped short for>
>> > > allowing it for the deployable NiFi codebase, for the same reasons>
>> > > IIRC. If Scala and/or Groovy, why not Kotlin? The same argument>
>> > > (albeit more tenuous) goes for Clojure and just about every other
>> JVM>
>> > > language (although I don't expect a call for LuaJ processors lol).>
>> > >>
>> > > Whether we decide to support various languages ad-hoc or not, I
>> would>
>> > > strenuously object to multiple/hybrid build systems for the
>> deployed>
>> > > artifacts. If I could switch NiFi completely to Gradle I would,
>> but I>
>> > > realize there are good reasons for not doing so (yet?) in the
>> Apache>
>> > > NiFi community, and I would never want any hybrid Maven/Gradle
>> build>
>> > > for the deployable code, likewise for SBT, Leiningen, etc. With a>
>> > > custom Mojo for Maven NAR builds, and the complexity for hybrid
>> builds>
>> > > in general, I think this would create a maintenance nightmare.>
>> > >>
>> > > The language thing is a tough decision though, it's not awesome
>> that>
>> > > specifying a single language can be a barrier to a more diverse>
>> > > community, certainly Scala-based bundles would be more than
>> welcome in>
>> > > the overall NiFi ecosystem, I just think the cons outweigh the
>> pros>
>> > > for the baseline code. I've written Groovy processors/NARs using>
>> > > Gradle as the build system, and I'm good with keeping them in my
>> own>
>> > > repo, especially when the Extension Registry becomes a thing. I
>> can>
>> > > see the Extension Registry perhaps making this a moot point, but>
>> > > clearly we need to have this discussion in the meantime.>
>> > >>
>> > > Regards,>
>> > > Matt>
>> > >>
>> > >>
>> > > On Sat, Feb 10, 2018 at 5:23 PM, Andrew Grande <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>> > > > Wasn't there a warning trigger about the NiFi distro size from
>> Apache>
>> > > > recently? IMO, before talking alternative languages, solve the
>> modularity>
>> > > > and NAR distribution problem. I think the implementation of a
>> module>
>> > > won't>
>> > > > matter much then, the point being not everything has to go in
>> the core,>
>> > > > base distribution, but can still be easily sourced from a known
>> repo, for>
>> > > > example.>
>> > > >>
>> > > > I have a feeling NiFi 1.6+ can be approaching 2GB distro size
>> soon :)>
>> > > >>
>> > > > Andrew>
>> > > >>
>> > > > On Sat, Feb 10, 2018, 5:12 PM Joey Frazee <[hidden email]
>> <mailto:[hidden email]>>>
>> > > wrote:>
>> > > >>
>> > > >> This probably necessitates a vote, yeah?>
>> > > >>>
>> > > >> Frankly, I’m usually happier writing Scala, and I’ve not
>> encountered any>
>> > > >> problems using processors written in Scala, but I think it’ll
>> be>
>> > > important>
>> > > >> to tread lightly.>
>> > > >>>
>> > > >> There’s a few things that pop into my head:>
>> > > >>>
>> > > >> - Maintainability and reviewability. A very very good Java
>> developer>
>> > > need>
>> > > >> not, by definition, either know how to write or identify good
>> Scala or>
>> > > spot>
>> > > >> problems and bugs.>
>> > > >> - Every Scala processor would either end up with a 5MB
>> scala-lang.jar>
>> > > >> packaged into the .nar or we’d have to start including it in
>> the core>
>> > > >> somewhere, if it’s not. It’s possible it might have already
>> gotten>
>> > > pulled>
>> > > >> up from other dependencies.>
>> > > >> - Style. There’s a tremendous amount of variation in Scala
>> style because>
>> > > >> of its type system, implicits, macros, and functional nature.
>> There are>
>> > > >> very good people out there that can write good Scala that isn’t>
>> > > readable by>
>> > > >> the 99%.>
>> > > >> - Binary compatibility. Scala tends to be a little more brazen
>> about>
>> > > >> breaking binary compatibility in major releases and those
>> happen a bit>
>> > > more>
>> > > >> often than with Java. That’s not a problem for any potential
>> source>
>> > > code in>
>> > > >> the project, but it could present some dependency issues
>> someday.>
>> > > >> - Testing. There’s N > 1 test frameworks and testing styles
>> within>
>> > > those,>
>> > > >> so there’s a lot of options for introducing more variability
>> into the>
>> > > tests.>
>> > > >> - NiFi uses a lot of statics in setting up properties and
>> relationships>
>> > > >> and the like, and idiomatic Scala approaches that stuff a bit>
>> > > differently,>
>> > > >> so it’ll be necessary to impose some style guidelines so there
>> isn’t too>
>> > > >> much variation.>
>> > > >>>
>> > > >> That said, there are some things that won’t be problematic:>
>> > > >>>
>> > > >> - As mentioned, processors written in Scala do just work.>
>> > > >> - The scala-maven-plugin works just fine allowing mixed
>> Java-Scala>
>> > > >> projects (btw, it’d probably be not super great to do mixed
>> Java-Scala>
>> > > and>
>> > > >> mixed Maven-SBT though).>
>> > > >> - A lot of the above concerns could be addressed by having
>> clear style>
>> > > >> guidelines.>
>> > > >>>
>> > > >> Another thing: most of the projects I see deliver separate jars
>> for>
>> > > Scala>
>> > > >> components are delivering idiomatic APIs wrapping Java (or vice
>> versa).>
>> > > I>
>> > > >> think publishing a separate set of jars/nars for stuff
>> written in>
>> > > Scala>
>> > > >> would be odd since here it’d mostly be processors with new
>> functionality>
>> > > >> and not functionality for using Scala. I could imagine a lib of>
>> > > implicits,>
>> > > >> traits, classes that could make the Scala development more
>> enjoyable.>
>> > > That>
>> > > >> probably would make sense to deliver that way.>
>> > > >>>
>> > > >> -joey>
>> > > >>>
>> > > >> On Feb 10, 2018, 10:33 AM -0600, Bryan Bende <[hidden email]
>> <mailto:[hidden email]>>, wrote:>
>> > > >> > I agree more with Andy about sticking with Java. The more
>> varying>
>> > > >> languages>
>> > > >> > used, the more challenging it is to maintain. Once the code
>> is part of>
>> > > >> the>
>> > > >> > Apache NiFi git repo, it is now the responsibility of the
>> committers>
>> > > and>
>> > > >> > PMC members to maintain it.>
>> > > >> >>
>> > > >> > I’d even say I am somewhat against the groovy/Spock test code
>> that>
>> > > Andy>
>> > > >> > mentioned. I have frequently spent hours trying to fix a
>> Spock test>
>> > > that>
>> > > >> > broke from something I was working on. Every committer is
>> familiar>
>> > > with>
>> > > >> > JUnit, but only a couple know Spock. Just using this as an
>> example>
>> > > that>
>> > > >> > every committer knows Java, but only a couple probably know
>> Scala,>
>> > > >> Clojure,>
>> > > >> > etc.>
>> > > >> >>
>> > > >> > On Sat, Feb 10, 2018 at 10:25 AM Jeff <[hidden email]
>> <mailto:[hidden email]>> wrote:>
>> > > >> >>
>> > > >> > > +1 to Joe's response. If you can develop a component in
>> Groovy or>
>> > > Scala>
>> > > >> > > (or Clojure!) more quickly/comfortably, or if allowing
>> components>
>> > > >> written>
>> > > >> > > in other languages would encourage people to contribute
>> more, I'm>
>> > > all>
>> > > >> for>
>> > > >> > > it.>
>> > > >> > >>
>> > > >> > > On Sat, Feb 10, 2018 at 7:42 AM Joe Witt <[hidden email]
>> <mailto:[hidden email]>>>
>> > > wrote:>
>> > > >> > >>
>> > > >> > > > i personally would be ok with it for an
>> extension/processor>
>> > > provided>
>> > > >> it>
>> > > >> > > > integrates well with the build.>
>> > > >> > > >>
>> > > >> > > > i would agree with andys view for core framework stuff
>> but for>
>> > > >> > > extensions i>
>> > > >> > > > think we can do it like mikethomsen suggested.>
>> > > >> > > >>
>> > > >> > > > others?>
>> > > >> > > >>
>> > > >> > > > thanks>
>> > > >> > > > joe>
>> > > >> > > >>
>> > > >> > > > On Feb 10, 2018 7:30 AM, "Mike Thomsen" <[hidden email]
>> <mailto:[hidden email]>>>
>> > > >> wrote:>
>> > > >> > > >>
>> > > >> > > > > I'm just a community contributor, so take that FWIW,
>> but a>
>> > > >> compromise>
>> > > >> > > > might>
>> > > >> > > > > be to publish the Scala code as separate maven modules
>> to maven>
>> > > >> central>
>> > > >> > > > and>
>> > > >> > > > > then submit a thoroughly tested processor written in
>> Java. As>
>> > > long>
>> > > >> as>
>> > > >> > > you>
>> > > >> > > > > have enough unit and integration tests to give strong
>> coverage,>
>> > > I>
>> > > >> > > > wouldn't>
>> > > >> > > > > imagine anyone here would have issues reviewing it. If
>> the tests>
>> > > >> fail>
>> > > >> > > > > because of code issues in the external dependencies,
>> the obvious>
>> > > >> answer>
>> > > >> > > > is>
>> > > >> > > > > to just hold the PR until the tests pass.>
>> > > >> > > > >>
>> > > >> > > > > On Fri, Feb 9, 2018 at 9:00 AM, Weiss, Adam <>
>> > > >> > > [hidden email]<mailto:Adam.Weiss@perkinelmer.
>> com>>
>> > > >> > > > > wrote:>
>> > > >> > > > >>
>> > > >> > > > > > Devs,>
>> > > >> > > > > >>
>> > > >> > > > > > I have some interns starting with my team and we use
>> Scala>
>> > > >> internally>
>> > > >> > > > for>
>> > > >> > > > > > our work.>
>> > > >> > > > > > If I wanted to have them work to contribute some new>
>> > > processors,>
>> > > >> > > would>
>> > > >> > > > > > they have to be Java to be included with the base>
>> > > distribution or>
>> > > >> > > could>
>> > > >> > > > > > they use Scala as long as it fit within your current
>> build and>
>> > > >> test>
>> > > >> > > > > process?>
>> > > >> > > > > >>
>> > > >> > > > > > Thanks,>
>> > > >> > > > > > -Adam>
>> > > >> > > > > >>
>> > > >> > > > >>
>> > > >> > > >>
>> > > >> > >>
>> > > >> > -->
>> > > >> > Sent from Gmail Mobile>
>> > > >>>
>> > >>
>> >
>>
>> Adam Weiss | Senior Manager, Digital Solutions R&D
>> PerkinElmer | Innovating for a Healthier World
>> [hidden email]<mailto:[hidden email]>
>> Mobile: +1 716.429.7324
>> 77 Goodell St. Suite 470, Buffalo, NY, 14203
>> www.perkinelmer.com
>>
>>
>>
>>