native (JNI) code

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

native (JNI) code

trkurc
Administrator
I've gotten some work done on NIFI-455. I'm building a shared library in C,
and calling functions in that library using JNI. I've been working in a
separate repo (https://github.com/trkurc/offheap) for now as I'm struggling
a bit with where native code should live in the source tree and how it
should be built and distributed. I'm hoping to get some suggestions.. I've
seen quite a bit of variety in the way other java projects include native
code such as (mixed into maven-friendly source tree, maven build with ant
run plugin with cmake and make) [1] (similar, but just with ant-run with
cmake) [2] (separate part of source tree with pom that invokes make via
executable plugin) [3].

Building and putting in the source tree is one issue. The next couple
integration steps are getting the shared library somewhere sensible and
setting up the environment to load, (and doing sensible things if the
shared library can't be found).

If people have thoughts, or idioms that other projects use that we may be
able to borrow, I'd like to know!

Tony

[1]
https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
[2]
https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
[3]
https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native

*parens included to improve groupin
Reply | Threaded
Open this post in threaded view
|

Re: native (JNI) code

Mark Payne
Tony,


Off the top of my head I’d vote just put it in nifi-commons. I wouldn’t worry about the fact that it’s C rather than Java - it’s all handled by Maven. But I’m not sure what exactly is involved, so if it doesn’t make sense then by all means do whatever you think is best in that respect.


As far as integrating it into the application: I wouldn’t integrate it at all. I would just make it a library that a NAR can bring in. NAR’s are allowed to have a “native” directory where native libraries live, so I’d just leave it up to the NAR to bring it in.


-Mark






From: Tony Kurc
Sent: ‎Wednesday‎, ‎March‎ ‎25‎, ‎2015 ‎11‎:‎53‎ ‎PM
To: [hidden email]





I've gotten some work done on NIFI-455. I'm building a shared library in C,
and calling functions in that library using JNI. I've been working in a
separate repo (https://github.com/trkurc/offheap) for now as I'm struggling
a bit with where native code should live in the source tree and how it
should be built and distributed. I'm hoping to get some suggestions.. I've
seen quite a bit of variety in the way other java projects include native
code such as (mixed into maven-friendly source tree, maven build with ant
run plugin with cmake and make) [1] (similar, but just with ant-run with
cmake) [2] (separate part of source tree with pom that invokes make via
executable plugin) [3].

Building and putting in the source tree is one issue. The next couple
integration steps are getting the shared library somewhere sensible and
setting up the environment to load, (and doing sensible things if the
shared library can't be found).

If people have thoughts, or idioms that other projects use that we may be
able to borrow, I'd like to know!

Tony

[1]
https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
[2]
https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
[3]
https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native

*parens included to improve groupin
Reply | Threaded
Open this post in threaded view
|

Re: native (JNI) code

Benson Margulies
Time for fun and confusion. The nar-maven-plugin (the other one) does
a working job in many cases of just folding the C++ build into the
Maven build.


On Thu, Mar 26, 2015 at 8:50 AM, Mark Payne <[hidden email]> wrote:

> Tony,
>
>
> Off the top of my head I’d vote just put it in nifi-commons. I wouldn’t worry about the fact that it’s C rather than Java - it’s all handled by Maven. But I’m not sure what exactly is involved, so if it doesn’t make sense then by all means do whatever you think is best in that respect.
>
>
> As far as integrating it into the application: I wouldn’t integrate it at all. I would just make it a library that a NAR can bring in. NAR’s are allowed to have a “native” directory where native libraries live, so I’d just leave it up to the NAR to bring it in.
>
>
> -Mark
>
>
>
>
>
>
> From: Tony Kurc
> Sent: ‎Wednesday‎, ‎March‎ ‎25‎, ‎2015 ‎11‎:‎53‎ ‎PM
> To: [hidden email]
>
>
>
>
>
> I've gotten some work done on NIFI-455. I'm building a shared library in C,
> and calling functions in that library using JNI. I've been working in a
> separate repo (https://github.com/trkurc/offheap) for now as I'm struggling
> a bit with where native code should live in the source tree and how it
> should be built and distributed. I'm hoping to get some suggestions.. I've
> seen quite a bit of variety in the way other java projects include native
> code such as (mixed into maven-friendly source tree, maven build with ant
> run plugin with cmake and make) [1] (similar, but just with ant-run with
> cmake) [2] (separate part of source tree with pom that invokes make via
> executable plugin) [3].
>
> Building and putting in the source tree is one issue. The next couple
> integration steps are getting the shared library somewhere sensible and
> setting up the environment to load, (and doing sensible things if the
> shared library can't be found).
>
> If people have thoughts, or idioms that other projects use that we may be
> able to borrow, I'd like to know!
>
> Tony
>
> [1]
> https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
> [2]
> https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
> [3]
> https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native
>
> *parens included to improve groupin
Reply | Threaded
Open this post in threaded view
|

Re: native (JNI) code

Adam Taft
In reply to this post by trkurc
​Tony,

Why native code for this?  What does this do that NIO Direct Buffers can't
do?

If not NIO, why not use an existing open source?  Netty has a direct buffer
wrapper.  And there's the DirectMemory Apache Incubator [1] (that I know
nothing about).

If still not NIO, why not use memcached, redis or the equivalent?

​Just curious your thinking.

Adam

[1] http://directmemory.apache.org/


On Wed, Mar 25, 2015 at 11:52 PM, Tony Kurc <[hidden email]> wrote:

> I've gotten some work done on NIFI-455. I'm building a shared library in C,
> and calling functions in that library using JNI. I've been working in a
> separate repo (https://github.com/trkurc/offheap) for now as I'm
> struggling
> a bit with where native code should live in the source tree and how it
> should be built and distributed. I'm hoping to get some suggestions.. I've
> seen quite a bit of variety in the way other java projects include native
> code such as (mixed into maven-friendly source tree, maven build with ant
> run plugin with cmake and make) [1] (similar, but just with ant-run with
> cmake) [2] (separate part of source tree with pom that invokes make via
> executable plugin) [3].
>
> Building and putting in the source tree is one issue. The next couple
> integration steps are getting the shared library somewhere sensible and
> setting up the environment to load, (and doing sensible things if the
> shared library can't be found).
>
> If people have thoughts, or idioms that other projects use that we may be
> able to borrow, I'd like to know!
>
> Tony
>
> [1]
>
> https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
> [2]
>
> https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
> [3]
>
> https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native
>
> *parens included to improve groupin
>
Reply | Threaded
Open this post in threaded view
|

Re: native (JNI) code

Mike Drob-2
Adam, I've copied your questions to the JIRA issue so that we can keep the
conversation in one place. If folks think it's better directly on the
mailing list, then we can switch back.

On Thu, Mar 26, 2015 at 8:44 AM, Adam Taft <[hidden email]> wrote:

> ​Tony,
>
> Why native code for this?  What does this do that NIO Direct Buffers can't
> do?
>
> If not NIO, why not use an existing open source?  Netty has a direct buffer
> wrapper.  And there's the DirectMemory Apache Incubator [1] (that I know
> nothing about).
>
> If still not NIO, why not use memcached, redis or the equivalent?
>
> ​Just curious your thinking.
>
> Adam
>
> [1] http://directmemory.apache.org/
>
>
> On Wed, Mar 25, 2015 at 11:52 PM, Tony Kurc <[hidden email]> wrote:
>
> > I've gotten some work done on NIFI-455. I'm building a shared library in
> C,
> > and calling functions in that library using JNI. I've been working in a
> > separate repo (https://github.com/trkurc/offheap) for now as I'm
> > struggling
> > a bit with where native code should live in the source tree and how it
> > should be built and distributed. I'm hoping to get some suggestions..
> I've
> > seen quite a bit of variety in the way other java projects include native
> > code such as (mixed into maven-friendly source tree, maven build with ant
> > run plugin with cmake and make) [1] (similar, but just with ant-run with
> > cmake) [2] (separate part of source tree with pom that invokes make via
> > executable plugin) [3].
> >
> > Building and putting in the source tree is one issue. The next couple
> > integration steps are getting the shared library somewhere sensible and
> > setting up the environment to load, (and doing sensible things if the
> > shared library can't be found).
> >
> > If people have thoughts, or idioms that other projects use that we may be
> > able to borrow, I'd like to know!
> >
> > Tony
> >
> > [1]
> >
> >
> https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
> > [2]
> >
> >
> https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
> > [3]
> >
> >
> https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native
> >
> > *parens included to improve groupin
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: native (JNI) code

trkurc
Administrator
In reply to this post by Benson Margulies
I think I'm going to give the nar-maven-plugin a try to compile the C code
and then try to build a nar with the nifi-nar-maven-plugin. I hope I can
find some other nar plugins to complete my maven plugin bingo board.

As for reasons for offheap storage, joe alluded to some of the reasons in
the ticket.

Some reasons:
1) manage dynamically allocated memory without gc pressure (why in C)
2) ability to tune data structures for best load and lookup times (the lack
of threadsafety and collision detection on insertion are examples)
3) I'm attempting to write code using C's standard library to avoid
dependency and build challenges (in conjunction with 2, which helps me
rationalize not use an existing library )
4) Have a library which is "optimized" for nifi operations ("tried and
true" factor)
5) Writing in C makes me happy.

Frankly, offheap aside, my question was more about native code. I think
having a robust way of including libs that aren't jvm friendly is probably
a good "feature" to have.


On Thu, Mar 26, 2015 at 8:55 AM, Benson Margulies <[hidden email]>
wrote:

> Time for fun and confusion. The nar-maven-plugin (the other one) does
> a working job in many cases of just folding the C++ build into the
> Maven build.
>
>
> On Thu, Mar 26, 2015 at 8:50 AM, Mark Payne <[hidden email]> wrote:
> > Tony,
> >
> >
> > Off the top of my head I’d vote just put it in nifi-commons. I wouldn’t
> worry about the fact that it’s C rather than Java - it’s all handled by
> Maven. But I’m not sure what exactly is involved, so if it doesn’t make
> sense then by all means do whatever you think is best in that respect.
> >
> >
> > As far as integrating it into the application: I wouldn’t integrate it
> at all. I would just make it a library that a NAR can bring in. NAR’s are
> allowed to have a “native” directory where native libraries live, so I’d
> just leave it up to the NAR to bring it in.
> >
> >
> > -Mark
> >
> >
> >
> >
> >
> >
> > From: Tony Kurc
> > Sent: ‎Wednesday‎, ‎March‎ ‎25‎, ‎2015 ‎11‎:‎53‎ ‎PM
> > To: [hidden email]
> >
> >
> >
> >
> >
> > I've gotten some work done on NIFI-455. I'm building a shared library in
> C,
> > and calling functions in that library using JNI. I've been working in a
> > separate repo (https://github.com/trkurc/offheap) for now as I'm
> struggling
> > a bit with where native code should live in the source tree and how it
> > should be built and distributed. I'm hoping to get some suggestions..
> I've
> > seen quite a bit of variety in the way other java projects include native
> > code such as (mixed into maven-friendly source tree, maven build with ant
> > run plugin with cmake and make) [1] (similar, but just with ant-run with
> > cmake) [2] (separate part of source tree with pom that invokes make via
> > executable plugin) [3].
> >
> > Building and putting in the source tree is one issue. The next couple
> > integration steps are getting the shared library somewhere sensible and
> > setting up the environment to load, (and doing sensible things if the
> > shared library can't be found).
> >
> > If people have thoughts, or idioms that other projects use that we may be
> > able to borrow, I'd like to know!
> >
> > Tony
> >
> > [1]
> >
> https://github.com/apache/hadoop/tree/3c9181722b05a9192f5440ea8f3f77231f84eac6/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
> > [2]
> >
> https://github.com/apache/hbase/tree/543c64d2e9d2fc608ef65cafd485a910d4b8d480/hbase-server/src/main/native/src
> > [3]
> >
> https://github.com/apache/accumulo/tree/be4aade67b0f0711e25de9ac01552471b48a75af/server/native
> >
> > *parens included to improve groupin
>