New Database Connection Pooling Controller Service

classic Classic list List threaded Threaded
28 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
Toivo,


You have a good point - not all drivers could be included in Apache land. One possible solution then is to have a property to configure which database you’re using and allow that to be a set of AllowableValue’s so that the user can choose any of the pre-provided choices, but also allow a “Custom” value. If they chose Custom, they would be required to provide the path to the JDBC jar. Then you would probably want to create your own instance of URLClassLoader and load the specified jar. This allows for the ease of configuring if you want to use one of the pre-loaded drivers (if any?) and allows plugability or other drivers.


-Mark






From: [hidden email]
Sent: ‎Tuesday‎, ‎February‎ ‎24‎, ‎2015 ‎1‎:‎47‎ ‎PM
To: [hidden email]





Mark,

Thanks, now I understand.

But at the same time would be very useful to load JDBC drivers which are not
included in nar file.
For example to use new JDBC version (bug fixes) or other vendor database.
Or some JDBC drivers may not be open source at all (Sybase JDBC, Microsoft
JDBC?)
Does Oracle JDBC driver license allows to be included in NiFi?
Also including many JDBC driver jar's in nar which are not mostly used
increase nar size.

Can NiFi have separate some sort of ext lib directory for such jar files?

Because Java JDK includes Java DB (Derby version) with JDBC driver, testing
can be done using this.
Or Apache Derby can be also included in nar.
But others? Should Postgres, MySql, etc, also included?

Thanks
toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p847.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Mark,

Sounds like a solution!


Thanks
toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Hi,
How I can change properties dynamically?
For example service has PropertyDescriptors:
DATABASE_SYSTEM   which contains all known databases as AllowableValue’s.
DRIVERNAME database specific driver name, initially empty
SERVERPORT database server specific port, initially empty

When user will choose DATABASE_SYSTEM, service will catch this using onPropertyModified
Now I want fill DRIVERNAME and SERVERPORT values for user, so user don’t need to enter them. But can change when needed.

Thanks
Toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Michael Moser
Toivo,
Mark,

I'm late to the conversation, but you might consider taking a look at
how the nifi-hadoop-nar depends on the nifi-hadoop-libraries-nar for
inheriting the appropriate Hadoop jars.  Create a jdbc-drivers-nar
containing one or two Apache compatible JDBC driver jars.  Then
someone who wants to add their JDBC driver of choice can simply
rebuild that nar after adding their JDBC driver dependency.

Then your nar inheritance structure would look something like this:

+ nifi-standard-services-api-nar
        + nifi-dbcp-service-api-nar
                + nifi-dbcp-drivers-nar
                        + nifi-dbcp-service-nar

It's a bit more complicated, but we've proven that this works well
with Hadoop in NiFi.

-- Mike


On Wed, Feb 25, 2015 at 11:33 AM, Toivo Adams <[hidden email]> wrote:

> Hi,
> How I can change properties dynamically?
> For example service has PropertyDescriptors:
> DATABASE_SYSTEM   which contains all known databases as AllowableValue’s.
> DRIVERNAME              database specific driver name, initially empty
> SERVERPORT              database server specific port, initially empty
>
> When user will choose DATABASE_SYSTEM, service will catch this using
> onPropertyModified
> Now I want fill DRIVERNAME and SERVERPORT values for user, so user don’t
> need to enter them. But can change when needed.
>
> Thanks
> Toivo
>
>
>
>
> --
> View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p854.html
> Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Hi,

Please help.

I still don't know how to set programatically some property value after user will change some other value.

My current idea is to show to user only one property:
DATABASE_SYSTEM  which contains all known databases as AllowableValue’s.

When user will choose DATABASE_SYSTEM, service will catch this using onPropertyModified.
And now service will create other property's and show them to user.
Don't know is this possible at all?

Usual practice is create unmodifiable list of property's.
Can this list be altered later?


Tried to test using TestRunner but onPropertyModified() is not called.

Service DBCPServiceApacheDBCP14 contains:

    @Override
    public void onPropertyModified(final PropertyDescriptor descriptor, final String oldValue, final String newValue) {
        super.onPropertyModified(descriptor, oldValue, newValue);

        if (descriptor.equals(DATABASE_SYSTEM)) {
       
        DatabaseSystemDescriptor databaseSystemDescriptor = DatabaseSystems.getDescriptor(newValue);

        // create other PropertyDescriptor's
        }        
    }

and test:

    @Test
    public void testGood() throws InitializationException {
        final TestRunner runner = TestRunners.newTestRunner(TestProcessor.class);
        final DBCPServiceApacheDBCP14 service = new DBCPServiceApacheDBCP14();
        runner.addControllerService("test-good1", service);
       
        // Should setProperty call also generate DBCPServiceApacheDBCP14.onPropertyModified() method call?
        // It does not currently.
       
        runner.setProperty(service, DBCPServiceApacheDBCP14.DATABASE_SYSTEM, "JavaDB");
        runner.enableControllerService(service);

        runner.assertValid(service);
        DBCPService dbcpService = (DBCPService) runner.getProcessContext().getControllerServiceLookup().getControllerService("test-good1");
        Assert.assertNotNull(dbcpService);
        Connection connection = dbcpService.getConnection();
        Assert.assertNotNull(connection);
    }


Thanks
toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
I added some code under

https://issues.apache.org/jira/browse/NIFI-322

toivo
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Mark Payne
In reply to this post by Toivo Adams
Toivo,


At this time, NiFi does not allow the developer to set property values programmatically. There are a few reasons for this, but the biggest reason is that if a processor can change its properties dynamically then one node in a cluster may change the value while another node does not. As a result, the nodes would be out of sync and would not be able to rejoin the cluster if they are disconnected for any reason.


The list of property descriptors, though, does not have to be static. It can change. However, it’s a little bit frowned upon only because it means that after the user drags the processor onto the graph, they can configure it and choose the appropriate value for that property. Then they have to apply those changes in order for the onPropertyModified method to get called and the properties to get refreshed. So the user would configure the processor, set the value, then click OK and then have to go back and configure the processor again for the new properties to show up.


That being said, we have recently discussed briefly the idea of allowing a property to depend on another. In this case, we would hide the properties unless they are applicable. I think this is really what you are wanting. But it’s not something that we support at this time.


Thanks

-Mark






From: [hidden email]
Sent: ‎Saturday‎, ‎February‎ ‎28‎, ‎2015 ‎11‎:‎01‎ ‎AM
To: [hidden email]





Hi,

Please help.

I still don't know how to set programatically some property value after user
will change some other value.

My current idea is to show to user only one property:
DATABASE_SYSTEM  which contains all known databases as AllowableValue’s.

When user will choose DATABASE_SYSTEM, service will catch this using
onPropertyModified.
And now service will create other property's and show them to user.
Don't know is this possible at all?

Usual practice is create unmodifiable list of property's.
Can this list be altered later?


Tried to test using TestRunner but onPropertyModified() is not called.

Service DBCPServiceApacheDBCP14 contains:

    @Override
    public void onPropertyModified(final PropertyDescriptor descriptor,
final String oldValue, final String newValue) {
        super.onPropertyModified(descriptor, oldValue, newValue);

        if (descriptor.equals(DATABASE_SYSTEM)) {
         
         DatabaseSystemDescriptor databaseSystemDescriptor =
DatabaseSystems.getDescriptor(newValue);

         // create other PropertyDescriptor's
        }        
    }

and test:

    @Test
    public void testGood() throws InitializationException {
        final TestRunner runner =
TestRunners.newTestRunner(TestProcessor.class);
        final DBCPServiceApacheDBCP14 service = new
DBCPServiceApacheDBCP14();
        runner.addControllerService("test-good1", service);
       
        // Should setProperty call also generate
DBCPServiceApacheDBCP14.onPropertyModified() method call?
        // It does not currently.
       
        runner.setProperty(service, DBCPServiceApacheDBCP14.DATABASE_SYSTEM,
"JavaDB");
        runner.enableControllerService(service);

        runner.assertValid(service);
        DBCPService dbcpService = (DBCPService)
runner.getProcessContext().getControllerServiceLookup().getControllerService("test-good1");
        Assert.assertNotNull(dbcpService);
        Connection connection = dbcpService.getConnection();
        Assert.assertNotNull(connection);
    }


Thanks
toivo




--
View this message in context: http://apache-nifi-incubating-developer-list.39713.n7.nabble.com/New-Database-Connection-Pooling-Controller-Service-tp582p857.html
Sent from the Apache NiFi (incubating) Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: New Database Connection Pooling Controller Service

Toivo Adams
Initial implementation.

https://issues.apache.org/jira/browse/NIFI-322

NIFI-322_01mar2015.patch



Please review.

Thanks
Toivo
12