New Database Connection Pooling Controller Service

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

New Database Connection Pooling Controller Service

Toivo Adams
Often DataFlows contain many processors which deal with database - select, update or delete different data in different tables.
Yet database is same and connection pooling helps to speed up connecting to database (open connection is fairly expensive). Also configuration must be done only in one place.
Database Connection Pooling Controller Service helps to solve this in consistent way.

Or is this solved already or underway?

Thanks
Toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
Toivo,


I don’t think this has been addressed, but I do agree that it is needed.


We do have a ticket https://issues.apache.org/jira/browse/NIFI-293 : Add a JDBC Processor for executing arbitrary SQL queries. This ticket I think is related.


Feel free to create a ticket to address this. I would also call out in the ticket description or a comment on the ticket that this is related to NIFI-293, just to provide a convenient link so that if someone is interested in addressing one they can look at both.


Thanks

-Mark






From: Toivo Adams
Sent: ‎Thursday‎, ‎February‎ ‎5‎, ‎2015 ‎9‎:‎59‎ ‎AM
To: [hidden email]





Often DataFlows contain many processors which deal with database - select,
update or delete different data in different tables.
Yet database is same and connection pooling helps to speed up connecting to
database (open connection is fairly expensive). Also configuration must be
done only in one place.
Database Connection Pooling Controller Service helps to solve this in
consistent way.

Or is this solved already or underway?

Thanks
Toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
https://issues.apache.org/jira/browse/NIFI-322
New Database Connection Pooling Controller Service
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
I'd like to start implementing this.

1. Is the /nifi-standard-services right place?

2. There are several Database Connection Pool implementations available.
Is Apache Commons DBCP suitable?
http://commons.apache.org/proper/commons-dbcp/

3. Service should be database independent.  So JDBC driver configuration and loading should be dynamic. Any recommendations and considerations how do to it?

What should be taken into account when implementing service?


Toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Toivo,


Answers below, inline.


Unfortunately, I don’t really have a lot of recommendations here, but I didn’t want to leave you hanging without responding to the e-mail. Anyone else have any recommendations?


Thanks

-Mark






From: [hidden email]
Sent: ‎Saturday‎, ‎February‎ ‎21‎, ‎2015 ‎9‎:‎19‎ ‎AM
To: [hidden email]





I'd like to start implementing this.

1. Is the /nifi-standard-services right place?

>> I think this is a good place.

2. There are several Database Connection Pool implementations available.
Is Apache Commons DBCP suitable?
http://commons.apache.org/proper/commons-dbcp/


>> This should be fine, assuming that the licensing is okay.


3. Service should be database independent.  So JDBC driver configuration and
loading should be dynamic. Any recommendations and considerations how do to
it?

>> I haven’t given this service enough thought to really provide any answer to this personally, but any specific questions I’d be happy to help with. I think I would have to refresh myself a bit on JDBC connectivity to provide much insight here.

What should be taken into account when implementing service?
>> As above, I haven’t given this service enough thought to really provide any answer here.


Toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p796.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Chad Zobrisky
Tovio,

I've used a Database service with nifi at work and just wanted to give some
input with how it was implemented.  I'm not sure if it was done with best
practices in mind and I'm sure Mark and Joe can add some input on that part.

All configuration was done in the controllerservices.xml in the conf
directory.  This includes the jdbc driver
("oracle.jdbc.driver.OracleDriver"), database url, username, password, and
a name for the connection.  Database pooling was then done, I think through
c3po, using the Class.from("oracle.jdbc.driver.OracleDriver") approach.
This connection was then selected from SQL processors to use from a
dropdown by getting the controller service created and setting the
connections available as acceptable options.

I was also looking at approaching this ticket and feel the implementation
will affect how the ExecuteSQL processor(s) down the road will be
implemented.  I've thought about and use the service fairly often so if you
want to discuss it further I'm all for it.

Chad

On Sun, Feb 22, 2015 at 10:45 AM, Mark Payne <[hidden email]> wrote:

> Toivo,
>
>
> Answers below, inline.
>
>
> Unfortunately, I don’t really have a lot of recommendations here, but I
> didn’t want to leave you hanging without responding to the e-mail. Anyone
> else have any recommendations?
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
> From: [hidden email]
> Sent: ‎Saturday‎, ‎February‎ ‎21‎, ‎2015 ‎9‎:‎19‎ ‎AM
> To: [hidden email]
>
>
>
>
>
> I'd like to start implementing this.
>
> 1. Is the /nifi-standard-services right place?
>
> >> I think this is a good place.
>
> 2. There are several Database Connection Pool implementations available.
> Is Apache Commons DBCP suitable?
> http://commons.apache.org/proper/commons-dbcp/
>
>
> >> This should be fine, assuming that the licensing is okay.
>
>
> 3. Service should be database independent.  So JDBC driver configuration
> and
> loading should be dynamic. Any recommendations and considerations how do to
> it?
>
> >> I haven’t given this service enough thought to really provide any
> answer to this personally, but any specific questions I’d be happy to help
> with. I think I would have to refresh myself a bit on JDBC connectivity to
> provide much insight here.
>
> What should be taken into account when implementing service?
> >> As above, I haven’t given this service enough thought to really provide
> any answer here.
>
>
> Toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p796.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Chad,
Thank you for hints.

So you have similar existing Controller Service.
Can this be contributed to NiFi?
No need to reinvent the wheel again.

Toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Chad Zobrisky
Toivo,

It was developed quickly and in a closed environment so the process of
trying to have it approved to open source would be more of a hassle than
just re-writing it.  The ideas and concepts used are of course open and
therefore it would be easy and valid to take the service itself and
reproduce it.

-Chad

On Tue, Feb 24, 2015 at 9:32 AM, Toivo Adams <[hidden email]> wrote:

> Chad,
> Thank you for hints.
>
> So you have similar existing Controller Service.
> Can this be contributed to NiFi?
> No need to reinvent the wheel again.
>
> Toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p824.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Chad,

Right, I understand.
So I move on.

This is my first NiFi controller service, so don't expect much.
:)

But I am sure together we can make it usable.

Where do you keep JDBC driver jar file?
Is it inside nar or in the NiFi classpath?

Mark, Joe,
JDBC and pooling implementations are capable of loading JDBC drivers for different databases by driver class name, for example (oracle.jdbc.driver.OracleDriver), but jar which contain driver class must be in classpath.
Don't know how NiFi ClassLoaders will handle this.

Toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Toivo,


When you do this, you can use:


Class.forName(“…”, true, Thread.currentThread().getContextClassLoader())


That will ensure that you are using the ClassLoader for you NAR.


Thanks

-Mark






From: [hidden email]
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎10‎:‎13‎ ‎AM
To: [hidden email]





Chad,

Right, I understand.
So I move on.

This is my first NiFi controller service, so don't expect much.
:)

But I am sure together we can make it usable.

Where do you keep JDBC driver jar file?
Is it inside nar or in the NiFi classpath?

Mark, Joe,
JDBC and pooling implementations are capable of loading JDBC drivers for
different databases by driver class name, for example
(oracle.jdbc.driver.OracleDriver), but jar which contain driver class must
be in classpath.
Don't know how NiFi ClassLoaders will handle this.

Toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p828.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
I created nifi-dbcp-service-api under nifi-standard-services.
nifi-dbcp-service-api  contain service interface only.

So current structure is:

  + nifi-standard-services
          + nifi-dbcp-service-api

next I must create nifi-dbcp-service-bundle

So final structure will be :

  + nifi-standard-services
          + nifi-dbcp-service-api
          + nifi-dbcp-service-bundle
                      + nifi-dbcp-service-nar
                      + nifi-dbcp-service

Is this correct?

toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Toivo,


Yes, exactly. You will also want to update the standard-services-api-nar to pull in your nifi-dbcp-service-api.


Thanks

-Mark






From: [hidden email]
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎11‎:‎57‎ ‎AM
To: [hidden email]





I created nifi-dbcp-service-api under nifi-standard-services.
nifi-dbcp-service-api  contain service interface only.

So current structure is:

  + nifi-standard-services
          + nifi-dbcp-service-api

next I must create nifi-dbcp-service-bundle

So final structure will be :

  + nifi-standard-services
          + nifi-dbcp-service-api
          + nifi-dbcp-service-bundle
                      + nifi-dbcp-service-nar
                      + nifi-dbcp-service

Is this correct?

toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p838.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Chad Zobrisky
Toivo

In regards to the driver jar, I'm not sure if it is okay to place a jar in
the lib directory, but it does work if you place, for example, ojbdc.jar in
that directory and load it via the Class.from() method.  I'm not sure if
the community wants to use that directory this way though, if it would be
seen as polluting or not.

-Chad

On Tue, Feb 24, 2015 at 11:59 AM, Mark Payne <[hidden email]> wrote:

> Toivo,
>
>
> Yes, exactly. You will also want to update the standard-services-api-nar
> to pull in your nifi-dbcp-service-api.
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
> From: [hidden email]
> Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎11‎:‎57‎ ‎AM
> To: [hidden email]
>
>
>
>
>
> I created nifi-dbcp-service-api under nifi-standard-services.
> nifi-dbcp-service-api  contain service interface only.
>
> So current structure is:
>
>   + nifi-standard-services
>           + nifi-dbcp-service-api
>
> next I must create nifi-dbcp-service-bundle
>
> So final structure will be :
>
>   + nifi-standard-services
>           + nifi-dbcp-service-api
>           + nifi-dbcp-service-bundle
>                       + nifi-dbcp-service-nar
>                       + nifi-dbcp-service
>
> Is this correct?
>
> toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p838.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Chad,


Exactly - that willwork but is “polluting” the lib directory. We put the absolute minimum amount we can in there. It is, essentially, only those things that are required to start the application. Everything else goes in NAR’s. You can get the appropriate ClassLoader from your code by calling Thread.currentThread().contextClassLoader(). Then, you can use that ClassLoader to load the class: Class.forName( “my.class”, true, Thread.currentThread().contextClassLoader() );


Thanks

-Mark






From: Chad Zobrisky
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎12‎:‎17‎ ‎PM
To: [hidden email]





Toivo

In regards to the driver jar, I'm not sure if it is okay to place a jar in
the lib directory, but it does work if you place, for example, ojbdc.jar in
that directory and load it via the Class.from() method.  I'm not sure if
the community wants to use that directory this way though, if it would be
seen as polluting or not.

-Chad

On Tue, Feb 24, 2015 at 11:59 AM, Mark Payne <[hidden email]> wrote:

> Toivo,
>
>
> Yes, exactly. You will also want to update the standard-services-api-nar
> to pull in your nifi-dbcp-service-api.
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
> From: [hidden email]
> Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎11‎:‎57‎ ‎AM
> To: [hidden email]
>
>
>
>
>
> I created nifi-dbcp-service-api under nifi-standard-services.
> nifi-dbcp-service-api  contain service interface only.
>
> So current structure is:
>
>   + nifi-standard-services
>           + nifi-dbcp-service-api
>
> next I must create nifi-dbcp-service-bundle
>
> So final structure will be :
>
>   + nifi-standard-services
>           + nifi-dbcp-service-api
>           + nifi-dbcp-service-bundle
>                       + nifi-dbcp-service-nar
>                       + nifi-dbcp-service
>
> Is this correct?
>
> toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p838.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
In reply to this post by Chad Zobrisky
I'd like not to put JDBC driver jar files to nar because every database vendor has it's own JDBC driver jar file.
This way service will work with any database which have JDBC compliant driver.

Don't know is NiFi lib right place for JDBC drivers, but at least this is better than in nar file.
But when you can add your custom nar files to NiFi lib directory, then why not JDBC jar's?


Thanks
toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Chad Zobrisky
Mark - I agree.  The current one I'm using it is built into the nar as a
dependency, but this can be prohibitive.

The biggest advantage of putting it in the lib is what I believe Toivo is
getting at.  If someone wanted to connect to Postgres, all they would have
to do is drop the postgres jdbc driver into the lib verse building or
rebuilding the Database service package.

Having the flexibility of dropping a jar in the lib directory is nice, but
it does pollute the base of nifi.  Maybe including some drivers in the nar,
but still allowing a user the option to drop a driver in the lib directory
would be a good choice?  This makes it easy for the end user but also
doesn't bloat the Database service with unused drivers.

Just some thoughts.
Chad

On Tue, Feb 24, 2015 at 12:34 PM, Toivo Adams <[hidden email]> wrote:

> I'd like not to put JDBC driver jar files to nar because every database
> vendor has it's own JDBC driver jar file.
> This way service will work with any database which have JDBC compliant
> driver.
>
> Don't know is NiFi lib right place for JDBC drivers, but at least this is
> better than in nar file.
> But when you can add your custom nar files to NiFi lib directory, then why
> not JDBC jar's?
>
>
> Thanks
> toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p842.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Toivo,


It’s okay to add NARs to the lib/ directory because the NAR provides classloader isolation. I.e., all of the jar files that you put in your NAR are accessible only by your NAR (or anyone that marks your NAR as its parent). In this way, if you add a bunch of jar files to your NAR, you don’t provide conflicts with someone else’s classpath.


All NARs inherit from the root classloader though, which loads the lib/ directory. So if you put the JDBC drivers into the lib/ directory, you are forcing all NARs to use your jar file. This can cause all sorts of weird and unexpected behavior if you modify a NAR’s classpath.


Thanks

-Mark






From: [hidden email]
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎12‎:‎43‎ ‎PM
To: [hidden email]





I'd like not to put JDBC driver jar files to nar because every database
vendor has it's own JDBC driver jar file.
This way service will work with any database which have JDBC compliant
driver.

Don't know is NiFi lib right place for JDBC drivers, but at least this is
better than in nar file.
But when you can add your custom nar files to NiFi lib directory, then why
not JDBC jar's?


Thanks
toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p842.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Chad,


Agreed, putting it in the lib/ directory does provide more flexibility. However, it can cause crazy results due to altering the classpath. You could get really crazy and allow the Controller Service to take a path to the JDBC driver and then create your own classloader within there, but I wouldn’t recommend it, because the configuration is really weird then. The operator has to know about JDBC jars - and remember, the operator should NOT be assumed to be a developer. Often, operators are not developers, so I think expecting them to understand how to configure that is a bit odd.


I think the ideal approach is to provide a NAR registry that people could provide their NARs to. We could then have a mysql connection pool, a postgres pool, etc., etc., as different NARs. This allows admins to choose what they want in their NiFi instance and avoid the bloat. However, building that registry is a very large undertaking, so it’s not really a viable solution for short term.


Depending on how large the nar would be, i’d recommend just putting the drivers we want to support in the nar for now.


Another possible solution would be to go ahead and create separate nars for each vendor, but not include them all in the build. Rather, we would release it but not include it in the assembly. Instructions could then be provided that point admins to the locations to pull the NARs.


Anyone else have thoughts on how to set this up?


Thanks

-Mark









From: Chad Zobrisky
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎12‎:‎52‎ ‎PM
To: [hidden email]





Mark - I agree.  The current one I'm using it is built into the nar as a
dependency, but this can be prohibitive.

The biggest advantage of putting it in the lib is what I believe Toivo is
getting at.  If someone wanted to connect to Postgres, all they would have
to do is drop the postgres jdbc driver into the lib verse building or
rebuilding the Database service package.

Having the flexibility of dropping a jar in the lib directory is nice, but
it does pollute the base of nifi.  Maybe including some drivers in the nar,
but still allowing a user the option to drop a driver in the lib directory
would be a good choice?  This makes it easy for the end user but also
doesn't bloat the Database service with unused drivers.

Just some thoughts.
Chad

On Tue, Feb 24, 2015 at 12:34 PM, Toivo Adams <[hidden email]> wrote:

> I'd like not to put JDBC driver jar files to nar because every database
> vendor has it's own JDBC driver jar file.
> This way service will work with any database which have JDBC compliant
> driver.
>
> Don't know is NiFi lib right place for JDBC drivers, but at least this is
> better than in nar file.
> But when you can add your custom nar files to NiFi lib directory, then why
> not JDBC jar's?
>
>
> Thanks
> toivo
>
>
>
>
> --
> View this message in context:
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p842.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive
> at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Chad Zobrisky
Mark,

After reading your response and now better understanding NiFi's class
loader, I completely agree that database driver jars should not be put in
the lib directory.  It seems the easiest way for now would be to include it
in the nar.

And the end user I mention would be an administrator over a non-developer
operator.  Someone has to be the one to install and configure nifi, and
that person should be knowledgeable enough of database connections and jdbc
drivers.

-Chad

On Tue, Feb 24, 2015 at 1:29 PM, Mark Payne <[hidden email]> wrote:

> Chad,
>
>
> Agreed, putting it in the lib/ directory does provide more flexibility.
> However, it can cause crazy results due to altering the classpath. You
> could get really crazy and allow the Controller Service to take a path to
> the JDBC driver and then create your own classloader within there, but I
> wouldn’t recommend it, because the configuration is really weird then. The
> operator has to know about JDBC jars - and remember, the operator should
> NOT be assumed to be a developer. Often, operators are not developers, so I
> think expecting them to understand how to configure that is a bit odd.
>
>
> I think the ideal approach is to provide a NAR registry that people could
> provide their NARs to. We could then have a mysql connection pool, a
> postgres pool, etc., etc., as different NARs. This allows admins to choose
> what they want in their NiFi instance and avoid the bloat. However,
> building that registry is a very large undertaking, so it’s not really a
> viable solution for short term.
>
>
> Depending on how large the nar would be, i’d recommend just putting the
> drivers we want to support in the nar for now.
>
>
> Another possible solution would be to go ahead and create separate nars
> for each vendor, but not include them all in the build. Rather, we would
> release it but not include it in the assembly. Instructions could then be
> provided that point admins to the locations to pull the NARs.
>
>
> Anyone else have thoughts on how to set this up?
>
>
> Thanks
>
> -Mark
>
>
>
>
>
>
>
>
>
> From: Chad Zobrisky
> Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎12‎:‎52‎ ‎PM
> To: [hidden email]
>
>
>
>
>
> Mark - I agree.  The current one I'm using it is built into the nar as a
> dependency, but this can be prohibitive.
>
> The biggest advantage of putting it in the lib is what I believe Toivo is
> getting at.  If someone wanted to connect to Postgres, all they would have
> to do is drop the postgres jdbc driver into the lib verse building or
> rebuilding the Database service package.
>
> Having the flexibility of dropping a jar in the lib directory is nice, but
> it does pollute the base of nifi.  Maybe including some drivers in the nar,
> but still allowing a user the option to drop a driver in the lib directory
> would be a good choice?  This makes it easy for the end user but also
> doesn't bloat the Database service with unused drivers.
>
> Just some thoughts.
> Chad
>
> On Tue, Feb 24, 2015 at 12:34 PM, Toivo Adams <[hidden email]>
> wrote:
>
> > I'd like not to put JDBC driver jar files to nar because every database
> > vendor has it's own JDBC driver jar file.
> > This way service will work with any database which have JDBC compliant
> > driver.
> >
> > Don't know is NiFi lib right place for JDBC drivers, but at least this is
> > better than in nar file.
> > But when you can add your custom nar files to NiFi lib directory, then
> why
> > not JDBC jar's?
> >
> >
> > Thanks
> > toivo
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p842.html
> > Sent from the Apache NiFi (incubating) Developer List mailing list
> archive
> > at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
In reply to this post by Mark Payne
Mark,

Thanks, now I understand.

But at the same time would be very useful to load JDBC drivers which are not included in nar file.
For example to use new JDBC version (bug fixes) or other vendor database.
Or some JDBC drivers may not be open source at all (Sybase JDBC, Microsoft JDBC?)
Does Oracle JDBC driver license allows to be included in NiFi?
Also including many JDBC driver jar's in nar which are not mostly used increase nar size.

Can NiFi have separate some sort of ext lib directory for such jar files?

Because Java JDK includes Java DB (Derby version) with JDBC driver, testing can be done using this.
Or Apache Derby can be also included in nar.
But others? Should Postgres, MySql, etc, also included?

Thanks
toivo
12