Secure Cluster Mode Issues

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Secure Cluster Mode Issues

Ricky Saltzer
Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
key store while in clustered mode. If I set the node in clustered mode, and
also provide a valid keystore, I receive a KeyStoreException [1]. If I set
the configuration to not use clustered mode, NiFi will start up fine with
the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


        at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

        at
org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

        at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

        at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

        at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

        ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available

        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

        at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]

        at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

        ... 73 common frames omitted
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on all nodes of the cluster? It appears from the log message that the keystore is not found during startup. To further debug, you can add the following line in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
key store while in clustered mode. If I set the node in clustered mode, and
also provide a valid keystore, I receive a KeyStoreException [1]. If I set
the configuration to not use clustered mode, NiFi will start up fine with
the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


       at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

       at
org.apache.nifi.io.socket.SSLContextFactory.<init>(SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

       at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

       at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFactoryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

       ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available

       at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

       at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]

       at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

       ... 73 common frames omitted


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out of
clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt

On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]> wrote:

> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on all
> nodes of the cluster? It appears from the log message that the keystore is
> not found during startup. To further debug, you can add the following line
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
> key store while in clustered mode. If I set the node in clustered mode, and
> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
> the configuration to not use clustered mode, NiFi will start up fine with
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>        at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>        at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>        at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>        at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>        at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>        ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available
>
>        at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>        at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>
>        at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>        ... 73 common frames omitted
>
>
>


--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Mark Payne
Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out of
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0fedabc88bb/gistfile1.txt>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email] <mailto:[hidden email]>> wrote:
>
>> Hi Ricky,
>>
>> Sorry to hear you are having this issue. Is the keystore available on all
>> nodes of the cluster? It appears from the log message that the keystore is
>> not found during startup. To further debug, you can add the following line
>> in bootstrap.conf to provide additional logging:
>>
>> java.arg.15=-Djavax.net.debug=ssl,handshake
>>
>> Andy LoPresto
>> [hidden email] <mailto:[hidden email]>
>> *[hidden email] <mailto:[hidden email]> <[hidden email] <mailto:[hidden email]>>*
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>>
>> Hey all -
>>
>> I'm using NiFi 1.0 and I'm having an issue using secure mode with a local
>> key store while in clustered mode. If I set the node in clustered mode, and
>> also provide a valid keystore, I receive a KeyStoreException [1]. If I set
>> the configuration to not use clustered mode, NiFi will start up fine with
>> the provided key store. Am I supposed to be storing this key store in
>> Zookeeper somewhere?
>>
>>
>> [1]
>>
>>
>> Caused by: java.security.KeyStoreException:  not found
>>
>>
>>       at java.security.KeyStore.getInstance(KeyStore.java:839)
>> ~[na:1.8.0_11]
>>
>>       at
>> org.apache.nifi.io.socket.SSLContextFactory.<init>(
>> SSLContextFactory.java:61)
>> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>>
>>       at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>>       at
>> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
>> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
>> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>>
>>       at
>> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
>> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
>> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>>
>>       ... 69 common frames omitted
>>
>> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not available
>>
>>       at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
>> ~[na:1.8.0_11]
>>
>>       at java.security.Security.getImpl(Security.java:695) ~[na:1.8.0_11]
>>
>>       at java.security.KeyStore.getInstance(KeyStore.java:836)
>> ~[na:1.8.0_11]
>>
>>       ... 73 common frames omitted
>>
>>
>>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
> >
> > Hey Andy -
> >
> > Thanks for the response. I'm currently just trying to get one node in
> > clustered mode before adding a second. The keystore is stored locally and
> > I've confirmed it's readable, as it was able to start once I took it out
> of
> > clustered mode. I added that line to the bootstrap.conf, but I don't
> > believe any additional logging was produced in regards to troubleshooting
> > this problem. Just in case, I've attached the entire log [1].
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
> >
> > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
> <mailto:[hidden email]>> wrote:
> >
> >> Hi Ricky,
> >>
> >> Sorry to hear you are having this issue. Is the keystore available on
> all
> >> nodes of the cluster? It appears from the log message that the keystore
> is
> >> not found during startup. To further debug, you can add the following
> line
> >> in bootstrap.conf to provide additional logging:
> >>
> >> java.arg.15=-Djavax.net.debug=ssl,handshake
> >>
> >> Andy LoPresto
> >> [hidden email] <mailto:[hidden email]>
> >> *[hidden email] <mailto:[hidden email]> <
> [hidden email] <mailto:[hidden email]>>*
> >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >>
> >> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
> >>
> >> Hey all -
> >>
> >> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
> local
> >> key store while in clustered mode. If I set the node in clustered mode,
> and
> >> also provide a valid keystore, I receive a KeyStoreException [1]. If I
> set
> >> the configuration to not use clustered mode, NiFi will start up fine
> with
> >> the provided key store. Am I supposed to be storing this key store in
> >> Zookeeper somewhere?
> >>
> >>
> >> [1]
> >>
> >>
> >> Caused by: java.security.KeyStoreException:  not found
> >>
> >>
> >>       at java.security.KeyStore.getInstance(KeyStore.java:839)
> >> ~[na:1.8.0_11]
> >>
> >>       at
> >> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> >> SSLContextFactory.java:61)
> >> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >>
> >>       at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >>       at
> >> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> >> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> >> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >>
> >>       at
> >> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> >> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> >> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
> >>
> >>       ... 69 common frames omitted
> >>
> >> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
> available
> >>
> >>       at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> >> ~[na:1.8.0_11]
> >>
> >>       at java.security.Security.getImpl(Security.java:695)
> ~[na:1.8.0_11]
> >>
> >>       at java.security.KeyStore.getInstance(KeyStore.java:836)
> >> ~[na:1.8.0_11]
> >>
> >>       ... 73 common frames omitted
> >>
> >>
> >>
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com <http://www.cloudera.com/>
>



--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Hi Ricky,

Sorry, should have noted that the debug output goes to nifi-bootstrap.log, so thanks Mark for jumping in to help there. 

If you look at the top of that log, you’ll note that there is no keystore file provided and the truststore loaded is the default JRE cacerts truststore. Can you please provide your nifi.properties file in a Gist, *taking care to redact any sensitive values* like keystore/truststore passwords, although I think from looking at your log output, you are taking advantage of the encrypted configuration feature, so even viewing the encrypted values should be ok. Could you also please provide the directory listing where the keystore and truststore are located including the permissions and ownership information?

There may be a bug in the logic between cluster and standalone mode, but I haven’t encountered this behavior before. If you can start NiFi in standalone mode, could you please provide the output of the following command run from the terminal? It will simulate an HTTPS connection to the server and verify the key and certificate presented by NiFi. 

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the CA which signed your NiFi server certificate (if self-signed, provide that certificate)

$ openssl s_client -connect <host:port> -debug -state -cert <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile <path_to_your_CA_cert.pem>

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out
of
clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/
ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>

On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
<mailto:[hidden email]>> wrote:

Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on
all
nodes of the cluster? It appears from the log message that the keystore
is
not found during startup. To further debug, you can add the following
line
in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <mailto:[hidden email]>
*[hidden email] <mailto:[hidden email]> <
[hidden email] <mailto:[hidden email]>>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a
local
key store while in clustered mode. If I set the node in clustered mode,
and
also provide a valid keystore, I receive a KeyStoreException [1]. If I
set
the configuration to not use clustered mode, NiFi will start up fine
with
the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


     at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

     at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

     at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

     at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

     at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

     ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
available

     at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

     at java.security.Security.getImpl(Security.java:695)
~[na:1.8.0_11]

     at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

     ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>




--
Ricky Saltzer
http://www.cloudera.com


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]> wrote:

> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to nifi-bootstrap.log,
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist, **taking
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking advantage
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions and
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but I
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to the
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the CA
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?) so
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/rickysaltzer/
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
>
> <mailto:[hidden email]>> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> [hidden email] <mailto:[hidden email]>
> *[hidden email] <mailto:[hidden email]> <
>
> [hidden email] <mailto:[hidden email]>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>      at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>      at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>      at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>      at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>      at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>      ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
>
> available
>
>
>      at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>      at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
>      at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>      ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>


--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andre
Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> > Hi Ricky,
> >
> > Sorry, should have noted that the debug output goes to
> nifi-bootstrap.log,
> > so thanks Mark for jumping in to help there.
> >
> > If you look at the top of that log, you’ll note that there is no keystore
> > file provided and the truststore loaded is the default JRE cacerts
> > truststore. Can you please provide your nifi.properties file in a Gist,
> **taking
> > care to redact any sensitive values** like keystore/truststore passwords,
> > although I think from looking at your log output, you are taking
> advantage
> > of the encrypted configuration feature, so even viewing the encrypted
> > values should be ok. Could you also please provide the directory listing
> > where the keystore and truststore are located including the permissions
> and
> > ownership information?
> >
> > There may be a bug in the logic between cluster and standalone mode, but
> I
> > haven’t encountered this behavior before. If you can start NiFi in
> > standalone mode, could you please provide the output of the following
> > command run from the terminal? It will simulate an HTTPS connection to
> the
> > server and verify the key and certificate presented by NiFi.
> >
> > * host — the NiFi hostname
> > * port — the port NiFi is running on
> > * path_to_your_cert.pem — the public key certificate identifying the
> > client/user (i.e. what you load into your browser to authenticate)
> > * path_to_your_key.key — the private key identifying the client/user
> > * path_to_your_CA_cert.pem — the public key certificate identifying the
> CA
> > which signed your NiFi server certificate (if self-signed, provide that
> > certificate)
> >
> > $ openssl s_client -connect <host:port> -debug -state -cert
> > <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> > <path_to_your_CA_cert.pem>
> >
> > Andy LoPresto
> > [hidden email]
> > *[hidden email] <[hidden email]>*
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:
> >
> > Hey guys -
> >
> > I went ahead and uploaded the boostrap log. I took a look at it and it
> > seems to be the same error [1]
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> > b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> > 9e70c57e49/gistfile1.txt
> >
> > On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:
> >
> > Hey Ricky,
> >
> > When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
> so
> > it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> > Can you give that a look?
> >
> > Thanks
> > -Mark
> >
> > On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
> >
> > Hey Andy -
> >
> > Thanks for the response. I'm currently just trying to get one node in
> > clustered mode before adding a second. The keystore is stored locally and
> > I've confirmed it's readable, as it was able to start once I took it out
> >
> > of
> >
> > clustered mode. I added that line to the bootstrap.conf, but I don't
> > believe any additional logging was produced in regards to troubleshooting
> > this problem. Just in case, I've attached the entire log [1].
> >
> > [1]:
> > https://gist.githubusercontent.com/rickysaltzer/
> >
> > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> > fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
> rickysaltzer/
> > ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> > fedabc88bb/gistfile1.txt>
> >
> >
> > On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
> >
> > <mailto:[hidden email]>> wrote:
> >
> >
> > Hi Ricky,
> >
> > Sorry to hear you are having this issue. Is the keystore available on
> >
> > all
> >
> > nodes of the cluster? It appears from the log message that the keystore
> >
> > is
> >
> > not found during startup. To further debug, you can add the following
> >
> > line
> >
> > in bootstrap.conf to provide additional logging:
> >
> > java.arg.15=-Djavax.net.debug=ssl,handshake
> >
> > Andy LoPresto
> > [hidden email] <mailto:[hidden email]>
> > *[hidden email] <mailto:[hidden email]> <
> >
> > [hidden email] <mailto:[hidden email]>>*
> >
> > PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
> >
> > On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
> >
> > Hey all -
> >
> > I'm using NiFi 1.0 and I'm having an issue using secure mode with a
> >
> > local
> >
> > key store while in clustered mode. If I set the node in clustered mode,
> >
> > and
> >
> > also provide a valid keystore, I receive a KeyStoreException [1]. If I
> >
> > set
> >
> > the configuration to not use clustered mode, NiFi will start up fine
> >
> > with
> >
> > the provided key store. Am I supposed to be storing this key store in
> > Zookeeper somewhere?
> >
> >
> > [1]
> >
> >
> > Caused by: java.security.KeyStoreException:  not found
> >
> >
> >      at java.security.KeyStore.getInstance(KeyStore.java:839)
> > ~[na:1.8.0_11]
> >
> >      at
> > org.apache.nifi.io.socket.SSLContextFactory.<init>(
> > SSLContextFactory.java:61)
> > ~[nifi-socket-utils-1.0.0.jar:1.0.0]
> >
> >      at
> > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >
> >      at
> > org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> > ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> > ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
> >
> >      at
> > org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> > doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> > ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
> >
> >      ... 69 common frames omitted
> >
> > Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
> >
> > available
> >
> >
> >      at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> > ~[na:1.8.0_11]
> >
> >      at java.security.Security.getImpl(Security.java:695)
> >
> > ~[na:1.8.0_11]
> >
> >
> >      at java.security.KeyStore.getInstance(KeyStore.java:836)
> > ~[na:1.8.0_11]
> >
> >      ... 73 common frames omitted
> >
> >
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com <http://www.cloudera.com/>
> >
> >
> >
> >
> >
> > --
> > Ricky Saltzer
> > http://www.cloudera.com
> >
> >
> >
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Hi Ricky,

Sorry for the delay in my response. I see a couple things which could be causing an issue:

1. You do not specify a hostname for the NiFi node (nifi.web.https.host=) which you should do. I have not set up a one-node cluster before, but my suspicion is that this field needs to be populated, and the value needs to match the CN for the issued server certificate. This value does not need to be unique (i.e. it could be nifi.cloudera.com or localhost and you could run multiple instances on the same machine, identified by the same certificate and different ports), but I think it has to be populated. 
1.a. While you have nifi.security.needClientAuth=false set in both cluster and standalone mode, I am not sure if this allows for the truststore to be missing completely. I would have to explore this further. There is a current Jira for clarifying cluster, UI/API, and S2S TLS configurations [1]. You can try using the default JRE truststore, located at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”. 
2. The certificate you have put into the keystore appears to be a certificate identifying you (Ricky the person) rather than a server entity. You should create a certificate identifying the server itself, so the CN is the FQDN as mentioned above. Then import into (or generate directly inside) the keystore. You can also use the provided NiFi TLS Toolkit to automate this entire process [2].  
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzer
compare to an example connection to google.com:443:

Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=*.google.com
   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

3. It appears the certificate you are using is expired. If you look at the output of the OpenSSL command you ran, you can see that the result code was 10, not 0 as would be in the case of a successful connection (I understand that it looks successful because it negotiates a key and encrypts the channel). 
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: 10 (certificate has expired)
---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output, and it appears it expired on October 25, 2016. You can verify this by copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END CERTIFICATE-----“ (including these markers) into a text file and naming it “ricky.pem” and then running the following command:

hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 485683207 (0x1cf2f007)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer
        Validity
            Not Before: Jul 27 17:15:46 2016 GMT
            Not After : Oct 25 17:15:46 2016 GMT
        Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team, Cloudera, CN=Ricky Saltzer
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
                    46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
                    1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
                    2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
                    c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
                    ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
                    fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
                    ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
                    d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
                    b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
                    09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
                    18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
                    52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
                    0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
                    bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
                    16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
                    79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
                    2b:c5
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                1C:DD:E1:16:5F:41:CC:7C:69:27:E3:B1:6B:78:EF:FA:16:DA:1F:6F
    Signature Algorithm: sha256WithRSAEncryption
         da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
         30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
         0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
         1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
         05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
         25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
         c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
         94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
         5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
         4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
         76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
         5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
         f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
         ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
         72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $

I would recommend using the TLS Toolkit to generate a valid CA, server certificate, client certificate, and keystore & truststore (seriously, one command does all of this) and then re-trying. From this stable baseline, I think I’ll be able to better help iron out the cluster/standalone issues.  



Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:

Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry, should have noted that the debug output goes to
nifi-bootstrap.log,
so thanks Mark for jumping in to help there.

If you look at the top of that log, you’ll note that there is no keystore
file provided and the truststore loaded is the default JRE cacerts
truststore. Can you please provide your nifi.properties file in a Gist,
**taking
care to redact any sensitive values** like keystore/truststore passwords,
although I think from looking at your log output, you are taking
advantage
of the encrypted configuration feature, so even viewing the encrypted
values should be ok. Could you also please provide the directory listing
where the keystore and truststore are located including the permissions
and
ownership information?

There may be a bug in the logic between cluster and standalone mode, but
I
haven’t encountered this behavior before. If you can start NiFi in
standalone mode, could you please provide the output of the following
command run from the terminal? It will simulate an HTTPS connection to
the
server and verify the key and certificate presented by NiFi.

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the
client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the
CA
which signed your NiFi server certificate (if self-signed, provide that
certificate)

$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/
b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
so
it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out

of

clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
rickysaltzer/
ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>


On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]

<[hidden email]>> wrote:


Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on

all

nodes of the cluster? It appears from the log message that the keystore

is

not found during startup. To further debug, you can add the following

line

in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <[hidden email]>
*[hidden email] <[hidden email]> <

[hidden email] <[hidden email]>>*

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a

local

key store while in clustered mode. If I set the node in clustered mode,

and

also provide a valid keystore, I receive a KeyStoreException [1]. If I

set

the configuration to not use clustered mode, NiFi will start up fine

with

the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


    at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

    at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

    at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

    at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

    at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

    ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not

available


    at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

    at java.security.Security.getImpl(Security.java:695)

~[na:1.8.0_11]


    at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

    ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com



signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]> wrote:

> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
>  0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
>    i:/C=US/O=Google Inc/CN=Google Internet Authority G2
>  1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>    i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>  2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>    i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
>     Data:
>         Version: 3 (0x2)
>         Serial Number: 485683207 (0x1cf2f007)
>     Signature Algorithm: sha256WithRSAEncryption
>         Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>         Validity
>             Not Before: Jul 27 17:15:46 2016 GMT
>             *Not After : Oct 25 17:15:46 2016 GMT*
>         Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>         Subject Public Key Info:
>             Public Key Algorithm: rsaEncryption
>                 Public-Key: (2048 bit)
>                 Modulus:
>                     00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>                     46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>                     1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>                     2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>                     c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>                     ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>                     fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>                     ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>                     d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>                     b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>                     09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>                     18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>                     52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>                     0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>                     bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>                     16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>                     79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>                     2b:c5
>                 Exponent: 65537 (0x10001)
>         X509v3 extensions:
>             X509v3 Subject Key Identifier:
>                 1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>     Signature Algorithm: sha256WithRSAEncryption
>          da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>          30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>          0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>          1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>          05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>          25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>          c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>          94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>          5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>          4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>          76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>          5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>          f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>          ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>          72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
>
> <mailto:[hidden email] <[hidden email]>>> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> [hidden email] <mailto:[hidden email] <[hidden email]>>
> *[hidden email] <mailto:[hidden email]
> <[hidden email]>> <
>
> [hidden email] <mailto:[hidden email]
> <[hidden email]>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>     at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>     at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>     at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>     at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>     at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>     ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
>
> available
>
>
>     at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>     at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
>     at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>     ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>


--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Hi Ricky,

The ERR_CONNECTION_CLOSED is likely because you are not sending a client certificate on the HTTP request. By default, a secured cluster requires client certificate authentication unless LDAP or Kerberos are configured as identity providers [1]. The TLS Toolkit provides a quick way to generate a valid client certificate which you can load into your browser in order to access the site. 

First, verify the cluster is running and accepting incoming connections (we’re going to cheat here just to be quick about it; disclaimer that this is not the RIGHT way to do this):

In the directory where you ran the toolkit, you noted there was a “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded public certificate of the NiFi CA cert that was generated by the toolkit, and the key file is the PEM-encoded private key. Because this is the same certificate that signed the NiFi server key loaded in the keystore, it is also loaded into the truststore. That means the server will accept an incoming connection with any certificate signed by the CA cert. Coincidentally, the CA cert is self-signed, so…

$ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem -key nifi-key.key -CAfile nifi-cert.pem

That command will attempt to negotiate a TLS connection to your server by presenting the CA cert and key as the client. Again, not semantically correct, but  technically will work. You’ll get a long output, but it should end in a section like this:

---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-SHA384
    Session-ID: 583DCD...9E828C
    Session-ID-ctx:
    Master-Key: 5477C0...A51E85
    Key-Arg   : None
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1480445265
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

The important part is the last line — you want the Verify return code to be 0 for success. Once you have verified this, run the TLS toolkit again to generate a valid client certificate:

$ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi' -B thisIsABadPassword

This will generate a PKCS12 keystore (*.p12) containing your public certificate AND the private key, as well as an additional file (.password) containing the password you provided for -B. 

hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi' -B password
2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using embedded one.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Running standalone certificate generation with output directory ../nifi-toolkit-1.1.0
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key ../nifi-toolkit-1.1.0/nifi-key.key
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: No hostnames specified, not generating any host certificates or configuration.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Generating new client certificate ../nifi-toolkit-1.1.0/CN=Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone: tls-toolkit standalone completed successfully
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 7s @ 10:50:22 $ ll
total 136
drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
-rw-------   1 alopresto  staff   3.4K Nov 29 10:50 CN=Ricky_Saltzer_OU=Apache_NiFi.p12
-rw-------   1 alopresto  staff     8B Nov 29 10:50 CN=Ricky_Saltzer_OU=Apache_NiFi.password
...
-rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
-rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0 (master) alopresto
🔓 4s @ 10:50:27 $

You can double-click on the *.p12 file to open it in your default handler — on OS X for example, this is Keychain Access. You can also manually load it into Firefox to keep it isolated from your system certificates. 

Now when you visit the UI through the browser, you should receive a prompt for which certificate to present, and select this entry from the presented list. 

If you get a NiFi error message in the browser that you do not have sufficient access, remember to update conf/authorizers.xml with an Initial Admin Identity, which should match EXACTLY the DN of this certificate — “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed authentication message in logs/nifi-user.log to be sure. 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:

Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]> wrote:

Hi Ricky,

Sorry for the delay in my response. I see a couple things which could be
causing an issue:

1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
which you should do. I have not set up a one-node cluster before, but my
suspicion is that this field needs to be populated, and the value needs to
match the CN for the issued server certificate. This value does not need to
be unique (i.e. it could be nifi.cloudera.com or localhost and you could
run multiple instances on the same machine, identified by the same
certificate and different ports), but I think it has to be populated.
1.a. While you have nifi.security.needClientAuth=false set in both
cluster and standalone mode, I am not sure if this allows for the
truststore to be missing completely. I would have to explore this further.
There is a current Jira for clarifying cluster, UI/API, and S2S TLS
configurations [1]. You can try using the default JRE truststore, located
at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
2. The certificate you have put into the keystore appears to be a
certificate identifying you (Ricky the person) rather than a server entity.
You should create a certificate identifying the server itself, so the CN is
the FQDN as mentioned above. Then import into (or generate directly inside)
the keystore. You can also use the provided NiFi TLS Toolkit to automate
this entire process [2].
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky Saltzercompare
to an example connection to google.com:443:

Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
<http://google.com>*
  i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
  i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
  i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

3. It appears the certificate you are using is expired. If you look at the
output of the OpenSSL command you ran, you can see that the result code was
*10*, not *0* as would be in the case of a successful connection (I
understand that it looks successful because it negotiates a key and
encrypts the channel).
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: *10* *(certificate has expired)*
---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output,
and it appears it expired on October 25, 2016. You can verify this by
copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
CERTIFICATE-----“ (including these markers) into a text file and naming it
“ricky.pem” and then running the following command:

hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
   Data:
       Version: 3 (0x2)
       Serial Number: 485683207 (0x1cf2f007)
   Signature Algorithm: sha256WithRSAEncryption
       Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
       Validity
           Not Before: Jul 27 17:15:46 2016 GMT
           *Not After : Oct 25 17:15:46 2016 GMT*
       Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
       Subject Public Key Info:
           Public Key Algorithm: rsaEncryption
               Public-Key: (2048 bit)
               Modulus:
                   00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
                   46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
                   1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
                   2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
                   c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
                   ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
                   fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
                   ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
                   d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
                   b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
                   09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
                   18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
                   52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
                   0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
                   bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
                   16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
                   79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
                   2b:c5
               Exponent: 65537 (0x10001)
       X509v3 extensions:
           X509v3 Subject Key Identifier:
               1C:DD:E1:16:5F:41:CC:7C:69:27:
E3:B1:6B:78:EF:FA:16:DA:1F:6F
   Signature Algorithm: sha256WithRSAEncryption
        da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
        30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
        0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
        1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
        05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
        25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
        c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
        94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
        5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
        4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
        76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
        5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
        f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
        ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
        72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $

I would recommend using the TLS Toolkit to generate a valid CA, server
certificate, client certificate, and keystore & truststore (seriously, one
command does all of this) and then re-trying. From this stable baseline, I
think I’ll be able to better help iron out the cluster/standalone issues.

[1] https://issues.apache.org/jira/browse/NIFI-1990
[2] https://nifi.apache.org/docs/nifi-docs/html/
administration-guide.html#tls-generation-toolkit


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:

Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry, should have noted that the debug output goes to

nifi-bootstrap.log,

so thanks Mark for jumping in to help there.

If you look at the top of that log, you’ll note that there is no keystore
file provided and the truststore loaded is the default JRE cacerts
truststore. Can you please provide your nifi.properties file in a Gist,

**taking

care to redact any sensitive values** like keystore/truststore passwords,
although I think from looking at your log output, you are taking

advantage

of the encrypted configuration feature, so even viewing the encrypted
values should be ok. Could you also please provide the directory listing
where the keystore and truststore are located including the permissions

and

ownership information?

There may be a bug in the logic between cluster and standalone mode, but

I

haven’t encountered this behavior before. If you can start NiFi in
standalone mode, could you please provide the output of the following
command run from the terminal? It will simulate an HTTPS connection to

the

server and verify the key and certificate presented by NiFi.

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the
client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the

CA

which signed your NiFi server certificate (if self-signed, provide that
certificate)

$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/
b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?)

so

it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out

of

clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/

rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>


On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]

<[hidden email] <[hidden email]>>> wrote:


Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on

all

nodes of the cluster? It appears from the log message that the keystore

is

not found during startup. To further debug, you can add the following

line

in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <[hidden email] <[hidden email]>>
*[hidden email] <[hidden email]
<[hidden email]>> <

[hidden email] <[hidden email]
<[hidden email]>>>*

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a

local

key store while in clustered mode. If I set the node in clustered mode,

and

also provide a valid keystore, I receive a KeyStoreException [1]. If I

set

the configuration to not use clustered mode, NiFi will start up fine

with

the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


   at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

   at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

   at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

   at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

   at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

   ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not

available


   at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

   at java.security.Security.getImpl(Security.java:695)

~[na:1.8.0_11]


   at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

   ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





-- 
Ricky Saltzer
http://www.cloudera.com


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hey Andy -

Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of
importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.

On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[hidden email]> wrote:

> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but  technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
>     Protocol  : TLSv1.2
>     Cipher    : ECDHE-RSA-AES256-SHA384
>     Session-ID: 583DCD...9E828C
>     Session-ID-ctx:
>     Master-Key: 5477C0...A51E85
>     Key-Arg   : None
>     PSK identity: None
>     PSK identity hint: None
>     SRP username: None
>     Start Time: 1480445265
>     Timeout   : 300 (sec)
>     Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
> tls-toolkit standalone completed successfully
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 7s @ 10:50:22 $ ll
> total 136
> drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
> drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
> -rw-------   1 alopresto  staff   3.4K Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
> -rw-------   1 alopresto  staff     8B Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.password
> ...
> -rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
> -rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 4s @ 10:50:27 $
>
> You can double-click on the *.p12 file to open it in your default handler
> — on OS X for example, this is Keychain Access. You can also manually load
> it into Firefox to keep it isolated from your system certificates.
>
> Now when you visit the UI through the browser, you should receive a prompt
> for which certificate to present, and select this entry from the presented
> list.
>
> If you get a NiFi error message in the browser that you do not have
> sufficient access, remember to update conf/authorizers.xml with an Initial
> Admin Identity, which should match EXACTLY the DN of this certificate —
> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
> authentication message in logs/nifi-user.log to be sure.
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
>   i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>   i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>   i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
>
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
>    Data:
>        Version: 3 (0x2)
>        Serial Number: 485683207 (0x1cf2f007)
>    Signature Algorithm: sha256WithRSAEncryption
>        Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>        Validity
>            Not Before: Jul 27 17:15:46 2016 GMT
>            *Not After : Oct 25 17:15:46 2016 GMT*
>
>        Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>        Subject Public Key Info:
>            Public Key Algorithm: rsaEncryption
>                Public-Key: (2048 bit)
>                Modulus:
>                    00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>                    46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>                    1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>                    2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>                    c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>                    ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>                    fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>                    ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>                    d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>                    b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>                    09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>                    18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>                    52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>                    0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>                    bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>                    16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>                    79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>                    2b:c5
>                Exponent: 65537 (0x10001)
>        X509v3 extensions:
>            X509v3 Subject Key Identifier:
>                1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>    Signature Algorithm: sha256WithRSAEncryption
>         da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>         30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>         0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>         1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>         05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>         25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>         c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>         94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>         5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>         4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>         76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>         5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>         f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>         ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>         72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
>
> <mailto:[hidden email] <[hidden email]> <[hidden email]>>>
> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> [hidden email] <mailto:[hidden email] <[hidden email]> <
> [hidden email]>>
> *[hidden email] <mailto:[hidden email]
> <[hidden email]>
> <[hidden email]>> <
>
> [hidden email] <mailto:[hidden email]
> <[hidden email]>
>
> <[hidden email]>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>    at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>    at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>    at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>    at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>    at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>    ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
>
> available
>
>
>    at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>    at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
>    at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>    ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>


--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Ricky,

When using HTTPS in non-cluster mode, NiFi still requires user authentication — this can be either client certificate (perhaps you already had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI over HTTPS without presenting some authentication, something is seriously broken. The warning in the browser is because the CA certificate that signed the server certificate (the one being presented to the browser by the application) is not trusted in the browser’s pre-installed trust chain. If, for example, that CA cert had been imported to the browser ahead of time, or if it was signed by a publicly known entity like DigiCert, Verisign, Comodo, etc., you would not receive a warning. 

For small teams, client certificates can be manageable, but if you want to allow multiple users to connect with minimal identity management, I recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory, Apache Directory Studio, etc.) and administering users there. Then the users will just enter a username and password into a login field on their first connection to NiFi and be authenticated. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of
importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.

On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[hidden email]> wrote:

Hi Ricky,

The ERR_CONNECTION_CLOSED is likely because you are not sending a client
certificate on the HTTP request. By default, a secured cluster requires
client certificate authentication unless LDAP or Kerberos are configured as
identity providers [1]. The TLS Toolkit provides a quick way to generate a
valid client certificate which you can load into your browser in order to
access the site.

First, verify the cluster is running and accepting incoming connections
(we’re going to cheat here just to be quick about it; disclaimer that this
is not the RIGHT way to do this):

In the directory where you ran the toolkit, you noted there was a
“nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
public certificate of the NiFi CA cert that was generated by the toolkit,
and the key file is the PEM-encoded private key. Because this is the same
certificate that signed the NiFi server key loaded in the keystore, it is
also loaded into the truststore. That means the server will accept an
incoming connection with any certificate signed by the CA cert.
Coincidentally, the CA cert is self-signed, so…

$ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
-key nifi-key.key -CAfile nifi-cert.pem

That command will attempt to negotiate a TLS connection to your server by
presenting the CA cert and key as the client. Again, not semantically
correct, but  technically will work. You’ll get a long output, but it
should end in a section like this:

---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
   Protocol  : TLSv1.2
   Cipher    : ECDHE-RSA-AES256-SHA384
   Session-ID: 583DCD...9E828C
   Session-ID-ctx:
   Master-Key: 5477C0...A51E85
   Key-Arg   : None
   PSK identity: None
   PSK identity hint: None
   SRP username: None
   Start Time: 1480445265
   Timeout   : 300 (sec)
   Verify return code: 0 (ok)
---

The important part is the last line — you want the *Verify return code* to
be 0 for success. Once you have verified this, run the TLS toolkit again to
generate a valid client certificate:

$ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
-B thisIsABadPassword

This will generate a PKCS12 keystore (*.p12) containing your public
certificate AND the private key, as well as an additional file (.password)
containing the password you provided for -B.

hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
OU=Apache NiFi' -B password
2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
embedded one.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Running standalone certificate generation with output directory
../nifi-toolkit-1.1.0
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
../nifi-toolkit-1.1.0/nifi-key.key
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
No hostnames specified, not generating any host certificates or
configuration.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Generating new client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.TlsToolkitStandalone:
tls-toolkit standalone completed successfully
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 7s @ 10:50:22 $ ll
total 136
drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
-rw-------   1 alopresto  staff   3.4K Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.p12
-rw-------   1 alopresto  staff     8B Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.password
...
-rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
-rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 4s @ 10:50:27 $

You can double-click on the *.p12 file to open it in your default handler
— on OS X for example, this is Keychain Access. You can also manually load
it into Firefox to keep it isolated from your system certificates.

Now when you visit the UI through the browser, you should receive a prompt
for which certificate to present, and select this entry from the presented
list.

If you get a NiFi error message in the browser that you do not have
sufficient access, remember to update conf/authorizers.xml with an Initial
Admin Identity, which should match EXACTLY the DN of this certificate —
“CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
authentication message in logs/nifi-user.log to be sure.

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:

Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry for the delay in my response. I see a couple things which could be
causing an issue:

1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
which you should do. I have not set up a one-node cluster before, but my
suspicion is that this field needs to be populated, and the value needs to
match the CN for the issued server certificate. This value does not need to
be unique (i.e. it could be nifi.cloudera.com or localhost and you could
run multiple instances on the same machine, identified by the same
certificate and different ports), but I think it has to be populated.
1.a. While you have nifi.security.needClientAuth=false set in both
cluster and standalone mode, I am not sure if this allows for the
truststore to be missing completely. I would have to explore this further.
There is a current Jira for clarifying cluster, UI/API, and S2S TLS
configurations [1]. You can try using the default JRE truststore, located
at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
2. The certificate you have put into the keystore appears to be a
certificate identifying you (Ricky the person) rather than a server entity.
You should create a certificate identifying the server itself, so the CN is
the FQDN as mentioned above. Then import into (or generate directly inside)
the keystore. You can also use the provided NiFi TLS Toolkit to automate
this entire process [2].
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzercompare
to an example connection to google.com:443:

Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
<http://google.com>*
 i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
 i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
 i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

3. It appears the certificate you are using is expired. If you look at the
output of the OpenSSL command you ran, you can see that the result code was
*10*, not *0* as would be in the case of a successful connection (I
understand that it looks successful because it negotiates a key and
encrypts the channel).
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: *10* *(certificate has expired)*

---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output,
and it appears it expired on October 25, 2016. You can verify this by
copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
CERTIFICATE-----“ (including these markers) into a text file and naming it
“ricky.pem” and then running the following command:

hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
  Data:
      Version: 3 (0x2)
      Serial Number: 485683207 (0x1cf2f007)
  Signature Algorithm: sha256WithRSAEncryption
      Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
      Validity
          Not Before: Jul 27 17:15:46 2016 GMT
          *Not After : Oct 25 17:15:46 2016 GMT*

      Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
      Subject Public Key Info:
          Public Key Algorithm: rsaEncryption
              Public-Key: (2048 bit)
              Modulus:
                  00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
                  46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
                  1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
                  2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
                  c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
                  ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
                  fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
                  ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
                  d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
                  b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
                  09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
                  18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
                  52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
                  0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
                  bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
                  16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
                  79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
                  2b:c5
              Exponent: 65537 (0x10001)
      X509v3 extensions:
          X509v3 Subject Key Identifier:
              1C:DD:E1:16:5F:41:CC:7C:69:27:
E3:B1:6B:78:EF:FA:16:DA:1F:6F
  Signature Algorithm: sha256WithRSAEncryption
       da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
       30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
       0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
       1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
       05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
       25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
       c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
       94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
       5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
       4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
       76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
       5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
       f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
       ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
       72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $

I would recommend using the TLS Toolkit to generate a valid CA, server
certificate, client certificate, and keystore & truststore (seriously, one
command does all of this) and then re-trying. From this stable baseline, I
think I’ll be able to better help iron out the cluster/standalone issues.

[1] https://issues.apache.org/jira/browse/NIFI-1990
[2] https://nifi.apache.org/docs/nifi-docs/html/
administration-guide.html#tls-generation-toolkit


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:

Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry, should have noted that the debug output goes to

nifi-bootstrap.log,

so thanks Mark for jumping in to help there.

If you look at the top of that log, you’ll note that there is no keystore
file provided and the truststore loaded is the default JRE cacerts
truststore. Can you please provide your nifi.properties file in a Gist,

**taking

care to redact any sensitive values** like keystore/truststore passwords,
although I think from looking at your log output, you are taking

advantage

of the encrypted configuration feature, so even viewing the encrypted
values should be ok. Could you also please provide the directory listing
where the keystore and truststore are located including the permissions

and

ownership information?

There may be a bug in the logic between cluster and standalone mode, but

I

haven’t encountered this behavior before. If you can start NiFi in
standalone mode, could you please provide the output of the following
command run from the terminal? It will simulate an HTTPS connection to

the

server and verify the key and certificate presented by NiFi.

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the
client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the

CA

which signed your NiFi server certificate (if self-signed, provide that
certificate)

$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/
b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?)

so

it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out

of

clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/

rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>


On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]

<[hidden email] <[hidden email]> <[hidden email]>>>
wrote:


Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on

all

nodes of the cluster? It appears from the log message that the keystore

is

not found during startup. To further debug, you can add the following

line

in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <[hidden email] <[hidden email]> <
[hidden email]>>
*[hidden email] <[hidden email]
<[hidden email]>
<[hidden email]>> <

[hidden email] <[hidden email]
<[hidden email]>

<[hidden email]>>>*

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a

local

key store while in clustered mode. If I set the node in clustered mode,

and

also provide a valid keystore, I receive a KeyStoreException [1]. If I

set

the configuration to not use clustered mode, NiFi will start up fine

with

the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


  at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

  at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

  at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

  at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

  at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

  ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not

available


  at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

  at java.security.Security.getImpl(Security.java:695)

~[na:1.8.0_11]


  at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

  ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





-- 
Ricky Saltzer
http://www.cloudera.com


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Hey Andy -

I think I may have figured out the problem. Although the keystorePasswd and
keyPasswd are the same, after completely removing the value for
nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
without manually importing the certificate.

On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <[hidden email]> wrote:

> Ricky,
>
> When using HTTPS in non-cluster mode, NiFi still requires user
> authentication — this can be either client certificate (perhaps you already
> had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
> over HTTPS without presenting some authentication, something is seriously
> broken. The warning in the browser is because the CA certificate that
> signed the server certificate (the one being presented *to* the browser
> by the application) is not trusted in the browser’s pre-installed trust
> chain. If, for example, that CA cert had been imported to the browser ahead
> of time, or if it was signed by a publicly known entity like DigiCert,
> Verisign, Comodo, etc., you would not receive a warning.
>
> For small teams, client certificates can be manageable, but if you want to
> allow multiple users to connect with minimal identity management, I
> recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
> Apache Directory Studio, etc.) and administering users there. Then the
> users will just enter a username and password into a login field on their
> first connection to NiFi and be authenticated.
>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the reply, I used the openssl command you provided and indeed
> the return code was *OK*. Before proceeding with the recommendation of
>
> importing the key into my OSX keychain, I would like to understand why this
> required. When using HTTPS mode in non-clustered mode, it does not require
> clients to have a special key or cert imported to their machine. Instead,
> the client is given a warning in the browser, and it's up to them to
> proceed. This UI will serve as an endpoint to several users, and I would
> really like to avoid the cumbersomeness of having members of multiple teams
> follow instructions for importing keys just so they can access a web UI.
>
> On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> The ERR_CONNECTION_CLOSED is likely because you are not sending a client
> certificate on the HTTP request. By default, a secured cluster requires
> client certificate authentication unless LDAP or Kerberos are configured as
> identity providers [1]. The TLS Toolkit provides a quick way to generate a
> valid client certificate which you can load into your browser in order to
> access the site.
>
> First, verify the cluster is running and accepting incoming connections
> (we’re going to cheat here just to be quick about it; disclaimer that this
> is not the RIGHT way to do this):
>
> In the directory where you ran the toolkit, you noted there was a
> “nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
> public certificate of the NiFi CA cert that was generated by the toolkit,
> and the key file is the PEM-encoded private key. Because this is the same
> certificate that signed the NiFi server key loaded in the keystore, it is
> also loaded into the truststore. That means the server will accept an
> incoming connection with any certificate signed by the CA cert.
> Coincidentally, the CA cert is self-signed, so…
>
> $ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
> -key nifi-key.key -CAfile nifi-cert.pem
>
> That command will attempt to negotiate a TLS connection to your server by
> presenting the CA cert and key as the client. Again, not semantically
> correct, but  technically will work. You’ll get a long output, but it
> should end in a section like this:
>
> ---
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> No ALPN negotiated
> SSL-Session:
>    Protocol  : TLSv1.2
>    Cipher    : ECDHE-RSA-AES256-SHA384
>    Session-ID: 583DCD...9E828C
>    Session-ID-ctx:
>    Master-Key: 5477C0...A51E85
>    Key-Arg   : None
>    PSK identity: None
>    PSK identity hint: None
>    SRP username: None
>    Start Time: 1480445265
>    Timeout   : 300 (sec)
>    Verify return code: 0 (ok)
> ---
>
> The important part is the last line — you want the *Verify return code* to
>
> be 0 for success. Once you have verified this, run the TLS toolkit again to
> generate a valid client certificate:
>
> $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
> -B thisIsABadPassword
>
> This will generate a PKCS12 keystore (*.p12) containing your public
> certificate AND the private key, as well as an additional file (.password)
> containing the password you provided for -B.
>
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
> OU=Apache NiFi' -B password
> 2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
> embedded one.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Running standalone certificate generation with output directory
> ../nifi-toolkit-1.1.0
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
> ../nifi-toolkit-1.1.0/nifi-key.key
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> No hostnames specified, not generating any host certificates or
> configuration.
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Generating new client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
> Ricky_Saltzer_OU=Apache_NiFi.p12
> 2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
> TlsToolkitStandalone:
> tls-toolkit standalone completed successfully
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 7s @ 10:50:22 $ ll
> total 136
> drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
> drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
> -rw-------   1 alopresto  staff   3.4K Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.p12
> -rw-------   1 alopresto  staff     8B Nov 29 10:50
> CN=Ricky_Saltzer_OU=Apache_NiFi.password
> ...
> -rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
> -rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
> hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
> 1.1.0-bin/nifi-toolkit-1.1.0
> (master) alopresto
> 🔓 4s @ 10:50:27 $
>
> You can double-click on the *.p12 file to open it in your default handler
> — on OS X for example, this is Keychain Access. You can also manually load
> it into Firefox to keep it isolated from your system certificates.
>
> Now when you visit the UI through the browser, you should receive a prompt
> for which certificate to present, and select this entry from the presented
> list.
>
> If you get a NiFi error message in the browser that you do not have
> sufficient access, remember to update conf/authorizers.xml with an Initial
> Admin Identity, which should match EXACTLY the DN of this certificate —
> “CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
> authentication message in logs/nifi-user.log to be sure.
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hi Andy -
>
> Thanks again for the help, sorry I've been a little preoccupied with some
> other tasks. I took your advice and used the tls-toolkit and things are
> looking a lot better. I generated a keystore/truststore for both of my
> nodes. It also generated a pem and key file. I was able to get NiFi to
> start in clustered mode (horray)! The only issue I'm seeing now is that I
> can't actually connect to the web UI using HTTPS, as it immediately throws
> a "ERR_CONNECTION_CLOSED" message in my browser.
>
> Judging by the logs, NiFi is seemingly running. A leader election took
> place and the node is heart beating to itself. Am I missing a step where
> I'm supposed to be doing something with the pem/key files?
>
> Thanks again,
> Ricky
>
> On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> Sorry for the delay in my response. I see a couple things which could be
> causing an issue:
>
> 1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
> which you should do. I have not set up a one-node cluster before, but my
> suspicion is that this field needs to be populated, and the value needs to
> match the CN for the issued server certificate. This value does not need to
> be unique (i.e. it could be nifi.cloudera.com or localhost and you could
> run multiple instances on the same machine, identified by the same
> certificate and different ports), but I think it has to be populated.
> 1.a. While you have nifi.security.needClientAuth=false set in both
> cluster and standalone mode, I am not sure if this allows for the
> truststore to be missing completely. I would have to explore this further.
> There is a current Jira for clarifying cluster, UI/API, and S2S TLS
> configurations [1]. You can try using the default JRE truststore, located
> at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
> 2. The certificate you have put into the keystore appears to be a
> certificate identifying you (Ricky the person) rather than a server entity.
> You should create a certificate identifying the server itself, so the CN is
> the FQDN as mentioned above. Then import into (or generate directly inside)
> the keystore. You can also use the provided NiFi TLS Toolkit to automate
> this entire process [2].
> Certificate chain
> 0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzer
> i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
> Saltzercompare
> to an example connection to google.com:443:
>
> Certificate chain
> 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
> <http://google.com>*
>  i:/C=US/O=Google Inc/CN=Google Internet Authority G2
> 1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
>  i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
> 2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
>  i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority
>
> 3. It appears the certificate you are using is expired. If you look at the
> output of the OpenSSL command you ran, you can see that the result code was
> *10*, not *0* as would be in the case of a successful connection (I
> understand that it looks successful because it negotiates a key and
> encrypts the channel).
> New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
> Server public key is 2048 bit
> Secure Renegotiation IS supported
> Compression: NONE
> Expansion: NONE
> SSL-Session:
> Protocol : TLSv1.2
> Cipher : ECDHE-RSA-AES256-SHA384
> Session-ID: 581F5A5...41153B
> Session-ID-ctx:
> Master-Key: CEEED2...944C30
> Key-Arg : None
> PSK identity: None
> PSK identity hint: None
> SRP username: None
> Start Time: 1478449748
> Timeout : 300 (sec)
> Verify return code: *10* *(certificate has expired)*
>
> ---
> HTTP/1.1 400 No URI
> Content-Length: 0
> Connection: close
> Server: Jetty(9.3.9.v20160517)
> I rebuilt your certificate from the Base64-encoded version in the output,
> and it appears it expired on October 25, 2016. You can verify this by
> copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
> CERTIFICATE-----“ (including these markers) into a text file and naming it
> “ricky.pem” and then running the following command:
>
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:30 $ more ricky.pem
> -----BEGIN CERTIFICATE-----
> MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
> ...
> 45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
> -----END CERTIFICATE-----
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
> Certificate:
>   Data:
>       Version: 3 (0x2)
>       Serial Number: 485683207 (0x1cf2f007)
>   Signature Algorithm: sha256WithRSAEncryption
>       Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>       Validity
>           Not Before: Jul 27 17:15:46 2016 GMT
>           *Not After : Oct 25 17:15:46 2016 GMT*
>
>       Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
> Cloudera, CN=Ricky Saltzer
>       Subject Public Key Info:
>           Public Key Algorithm: rsaEncryption
>               Public-Key: (2048 bit)
>               Modulus:
>                   00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
>                   46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
>                   1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
>                   2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
>                   c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
>                   ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
>                   fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
>                   ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
>                   d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
>                   b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
>                   09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
>                   18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
>                   52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
>                   0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
>                   bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
>                   16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
>                   79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
>                   2b:c5
>               Exponent: 65537 (0x10001)
>       X509v3 extensions:
>           X509v3 Subject Key Identifier:
>               1C:DD:E1:16:5F:41:CC:7C:69:27:
> E3:B1:6B:78:EF:FA:16:DA:1F:6F
>   Signature Algorithm: sha256WithRSAEncryption
>        da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
>        30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
>        0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
>        1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
>        05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
>        25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
>        c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
>        94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
>        5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
>        4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
>        76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
>        5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
>        f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
>        ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
>        72:a0:9f:35
> hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
> 🔓 11s @ 17:09:47 $
>
> I would recommend using the TLS Toolkit to generate a valid CA, server
> certificate, client certificate, and keystore & truststore (seriously, one
> command does all of this) and then re-trying. From this stable baseline, I
> think I’ll be able to better help iron out the cluster/standalone issues.
>
> [1] https://issues.apache.org/jira/browse/NIFI-1990
> [2] https://nifi.apache.org/docs/nifi-docs/html/
> administration-guide.html#tls-generation-toolkit
>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:
>
> Rick,
>
> Can you confirm the certificate has a chain of trust with the default JDK
> trusted certs? (i.e. trusted by the JVM)
>
> Cheers
>
>
> On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks again for the help.
>
> The error message seems indicative that it doesn't seem to properly read
> the keystore file. One thing to note, if I point the nifi properties to a
> bogus keystore location, then it actually throws a FileNotFound exception.
> This is really odd behavior, because as I mentioned I'm able to start it in
> standalone mode using the correct keystore location, just as I try to do in
> clustered mode.
>
> I've attached both the clustered [1] nifi.properties, which doesn't work,
> and the standalone [2] which does work. . I restored it to a more basic
> configuration without the encrypted configuration, but with SSL still
> enabled. I also added a diff [3] of both the standalone and clustered
> properties file. Note that I I only have NiFi configured to use the
> keystore and not a truststore. I've redacted a few of the values in the
> property files, but be assured that the keystore is most definitely valid
> and is readable / locatable, as starting in standalone works just fine.
>
> I ran the SSL command [4] you gave me, minus the three PEM file arguments
> as I don't have any of those on hand. I hope that is fine. The output still
> looks good.
>
> [1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
> [2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
> [3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
> [4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a
>
>
>
>
> On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
> wrote:
>
> Hi Ricky,
>
> Sorry, should have noted that the debug output goes to
>
> nifi-bootstrap.log,
>
> so thanks Mark for jumping in to help there.
>
> If you look at the top of that log, you’ll note that there is no keystore
> file provided and the truststore loaded is the default JRE cacerts
> truststore. Can you please provide your nifi.properties file in a Gist,
>
> **taking
>
> care to redact any sensitive values** like keystore/truststore passwords,
> although I think from looking at your log output, you are taking
>
> advantage
>
> of the encrypted configuration feature, so even viewing the encrypted
> values should be ok. Could you also please provide the directory listing
> where the keystore and truststore are located including the permissions
>
> and
>
> ownership information?
>
> There may be a bug in the logic between cluster and standalone mode, but
>
> I
>
> haven’t encountered this behavior before. If you can start NiFi in
> standalone mode, could you please provide the output of the following
> command run from the terminal? It will simulate an HTTPS connection to
>
> the
>
> server and verify the key and certificate presented by NiFi.
>
> * host — the NiFi hostname
> * port — the port NiFi is running on
> * path_to_your_cert.pem — the public key certificate identifying the
> client/user (i.e. what you load into your browser to authenticate)
> * path_to_your_key.key — the private key identifying the client/user
> * path_to_your_CA_cert.pem — the public key certificate identifying the
>
> CA
>
> which signed your NiFi server certificate (if self-signed, provide that
> certificate)
>
> $ openssl s_client -connect <host:port> -debug -state -cert
> <path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
> <path_to_your_CA_cert.pem>
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey guys -
>
> I went ahead and uploaded the boostrap log. I took a look at it and it
> seems to be the same error [1]
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
> b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
> 9e70c57e49/gistfile1.txt
>
> On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:
>
> Hey Ricky,
>
> When you enable debug logging for SSL, it writes to StdErr (or StdOut?)
>
> so
>
> it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
> Can you give that a look?
>
> Thanks
> -Mark
>
> On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey Andy -
>
> Thanks for the response. I'm currently just trying to get one node in
> clustered mode before adding a second. The keystore is stored locally and
> I've confirmed it's readable, as it was able to start once I took it out
>
> of
>
> clustered mode. I added that line to the bootstrap.conf, but I don't
> believe any additional logging was produced in regards to troubleshooting
> this problem. Just in case, I've attached the entire log [1].
>
> [1]:
> https://gist.githubusercontent.com/rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/
>
> rickysaltzer/
>
> ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
> fedabc88bb/gistfile1.txt>
>
>
> On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]
>
> <mailto:[hidden email] <[hidden email]> <[hidden email]>
> <[hidden email]>>>
> wrote:
>
>
> Hi Ricky,
>
> Sorry to hear you are having this issue. Is the keystore available on
>
> all
>
> nodes of the cluster? It appears from the log message that the keystore
>
> is
>
> not found during startup. To further debug, you can add the following
>
> line
>
> in bootstrap.conf to provide additional logging:
>
> java.arg.15=-Djavax.net.debug=ssl,handshake
>
> Andy LoPresto
> [hidden email] <mailto:[hidden email] <[hidden email]> <
> [hidden email]> <
> [hidden email]>>
> *[hidden email] <mailto:[hidden email]
> <[hidden email]>
> <[hidden email]>
> <[hidden email]>> <
>
> [hidden email] <mailto:[hidden email]
> <[hidden email]>
>
> <[hidden email]>
>
> <[hidden email]>>>*
>
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:
>
> Hey all -
>
> I'm using NiFi 1.0 and I'm having an issue using secure mode with a
>
> local
>
> key store while in clustered mode. If I set the node in clustered mode,
>
> and
>
> also provide a valid keystore, I receive a KeyStoreException [1]. If I
>
> set
>
> the configuration to not use clustered mode, NiFi will start up fine
>
> with
>
> the provided key store. Am I supposed to be storing this key store in
> Zookeeper somewhere?
>
>
> [1]
>
>
> Caused by: java.security.KeyStoreException:  not found
>
>
>   at java.security.KeyStore.getInstance(KeyStore.java:839)
> ~[na:1.8.0_11]
>
>   at
> org.apache.nifi.io.socket.SSLContextFactory.<init>(
> SSLContextFactory.java:61)
> ~[nifi-socket-utils-1.0.0.jar:1.0.0]
>
>   at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>   at
> org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
> ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
> ~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]
>
>   at
> org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
> doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
> ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
>
>   ... 69 common frames omitted
>
> Caused by: java.security.NoSuchAlgorithmException:  KeyStore not
>
> available
>
>
>   at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
> ~[na:1.8.0_11]
>
>   at java.security.Security.getImpl(Security.java:695)
>
> ~[na:1.8.0_11]
>
>
>   at java.security.KeyStore.getInstance(KeyStore.java:836)
> ~[na:1.8.0_11]
>
>   ... 73 common frames omitted
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com <http://www.cloudera.com/>
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>
>
>
> --
> Ricky Saltzer
> http://www.cloudera.com
>
>
>


--
Ricky Saltzer
http://www.cloudera.com
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Andy LoPresto-2
Ricky,

Removing the redundant key password property shouldn’t have an impact (although you may be running a legacy version before NIFI-2222 [1] and NIFI-2466 [2] were fixed). Can you look at the top right of your NiFi UI and see what user is accessing the system? It should look like the screenshot I have attached. This, and the contents of logs/nifi-user.log, will indicate the authenticated user. That should help you figure out how the authentication is occurring (client certificate, LDAP, or Kerberos). If you still cannot determine it, you can update conf/logback.xml and change the logging level for the following loggers from INFO to DEBUG:

    <logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>


I only ask for this information because your results do not make sense and I fear that they will not be reproducible for the rest of your team when you try to deploy the system and let them access NiFi and I would hope we can provide the best experience from the beginning. 



Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 30, 2016, at 11:12 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

I think I may have figured out the problem. Although the keystorePasswd and
keyPasswd are the same, after completely removing the value for
nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
without manually importing the certificate.

On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <[hidden email]> wrote:

Ricky,

When using HTTPS in non-cluster mode, NiFi still requires user
authentication — this can be either client certificate (perhaps you already
had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
over HTTPS without presenting some authentication, something is seriously
broken. The warning in the browser is because the CA certificate that
signed the server certificate (the one being presented *to* the browser
by the application) is not trusted in the browser’s pre-installed trust
chain. If, for example, that CA cert had been imported to the browser ahead
of time, or if it was signed by a publicly known entity like DigiCert,
Verisign, Comodo, etc., you would not receive a warning.

For small teams, client certificates can be manageable, but if you want to
allow multiple users to connect with minimal identity management, I
recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
Apache Directory Studio, etc.) and administering users there. Then the
users will just enter a username and password into a login field on their
first connection to NiFi and be authenticated.


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of

importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.

On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

The ERR_CONNECTION_CLOSED is likely because you are not sending a client
certificate on the HTTP request. By default, a secured cluster requires
client certificate authentication unless LDAP or Kerberos are configured as
identity providers [1]. The TLS Toolkit provides a quick way to generate a
valid client certificate which you can load into your browser in order to
access the site.

First, verify the cluster is running and accepting incoming connections
(we’re going to cheat here just to be quick about it; disclaimer that this
is not the RIGHT way to do this):

In the directory where you ran the toolkit, you noted there was a
“nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
public certificate of the NiFi CA cert that was generated by the toolkit,
and the key file is the PEM-encoded private key. Because this is the same
certificate that signed the NiFi server key loaded in the keystore, it is
also loaded into the truststore. That means the server will accept an
incoming connection with any certificate signed by the CA cert.
Coincidentally, the CA cert is self-signed, so…

$ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
-key nifi-key.key -CAfile nifi-cert.pem

That command will attempt to negotiate a TLS connection to your server by
presenting the CA cert and key as the client. Again, not semantically
correct, but  technically will work. You’ll get a long output, but it
should end in a section like this:

---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
  Protocol  : TLSv1.2
  Cipher    : ECDHE-RSA-AES256-SHA384
  Session-ID: 583DCD...9E828C
  Session-ID-ctx:
  Master-Key: 5477C0...A51E85
  Key-Arg   : None
  PSK identity: None
  PSK identity hint: None
  SRP username: None
  Start Time: 1480445265
  Timeout   : 300 (sec)
  Verify return code: 0 (ok)
---

The important part is the last line — you want the *Verify return code* to

be 0 for success. Once you have verified this, run the TLS toolkit again to
generate a valid client certificate:

$ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
-B thisIsABadPassword

This will generate a PKCS12 keystore (*.p12) containing your public
certificate AND the private key, as well as an additional file (.password)
containing the password you provided for -B.

hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
OU=Apache NiFi' -B password
2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
embedded one.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Running standalone certificate generation with output directory
../nifi-toolkit-1.1.0
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
../nifi-toolkit-1.1.0/nifi-key.key
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
No hostnames specified, not generating any host certificates or
configuration.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Generating new client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
tls-toolkit standalone completed successfully
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 7s @ 10:50:22 $ ll
total 136
drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
-rw-------   1 alopresto  staff   3.4K Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.p12
-rw-------   1 alopresto  staff     8B Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.password
...
-rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
-rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 4s @ 10:50:27 $

You can double-click on the *.p12 file to open it in your default handler
— on OS X for example, this is Keychain Access. You can also manually load
it into Firefox to keep it isolated from your system certificates.

Now when you visit the UI through the browser, you should receive a prompt
for which certificate to present, and select this entry from the presented
list.

If you get a NiFi error message in the browser that you do not have
sufficient access, remember to update conf/authorizers.xml with an Initial
Admin Identity, which should match EXACTLY the DN of this certificate —
“CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
authentication message in logs/nifi-user.log to be sure.

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:

Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry for the delay in my response. I see a couple things which could be
causing an issue:

1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
which you should do. I have not set up a one-node cluster before, but my
suspicion is that this field needs to be populated, and the value needs to
match the CN for the issued server certificate. This value does not need to
be unique (i.e. it could be nifi.cloudera.com or localhost and you could
run multiple instances on the same machine, identified by the same
certificate and different ports), but I think it has to be populated.
1.a. While you have nifi.security.needClientAuth=false set in both
cluster and standalone mode, I am not sure if this allows for the
truststore to be missing completely. I would have to explore this further.
There is a current Jira for clarifying cluster, UI/API, and S2S TLS
configurations [1]. You can try using the default JRE truststore, located
at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
2. The certificate you have put into the keystore appears to be a
certificate identifying you (Ricky the person) rather than a server entity.
You should create a certificate identifying the server itself, so the CN is
the FQDN as mentioned above. Then import into (or generate directly inside)
the keystore. You can also use the provided NiFi TLS Toolkit to automate
this entire process [2].
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzercompare
to an example connection to google.com:443:

Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
<http://google.com>*
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

3. It appears the certificate you are using is expired. If you look at the
output of the OpenSSL command you ran, you can see that the result code was
*10*, not *0* as would be in the case of a successful connection (I
understand that it looks successful because it negotiates a key and
encrypts the channel).
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: *10* *(certificate has expired)*

---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output,
and it appears it expired on October 25, 2016. You can verify this by
copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
CERTIFICATE-----“ (including these markers) into a text file and naming it
“ricky.pem” and then running the following command:

hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
 Data:
     Version: 3 (0x2)
     Serial Number: 485683207 (0x1cf2f007)
 Signature Algorithm: sha256WithRSAEncryption
     Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
     Validity
         Not Before: Jul 27 17:15:46 2016 GMT
         *Not After : Oct 25 17:15:46 2016 GMT*

     Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
     Subject Public Key Info:
         Public Key Algorithm: rsaEncryption
             Public-Key: (2048 bit)
             Modulus:
                 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
                 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
                 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
                 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
                 c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
                 ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
                 fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
                 ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
                 d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
                 b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
                 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
                 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
                 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
                 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
                 bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
                 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
                 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
                 2b:c5
             Exponent: 65537 (0x10001)
     X509v3 extensions:
         X509v3 Subject Key Identifier:
             1C:DD:E1:16:5F:41:CC:7C:69:27:
E3:B1:6B:78:EF:FA:16:DA:1F:6F
 Signature Algorithm: sha256WithRSAEncryption
      da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
      30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
      0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
      1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
      05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
      25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
      c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
      94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
      5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
      4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
      76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
      5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
      f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
      ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
      72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $

I would recommend using the TLS Toolkit to generate a valid CA, server
certificate, client certificate, and keystore & truststore (seriously, one
command does all of this) and then re-trying. From this stable baseline, I
think I’ll be able to better help iron out the cluster/standalone issues.

[1] https://issues.apache.org/jira/browse/NIFI-1990
[2] https://nifi.apache.org/docs/nifi-docs/html/
administration-guide.html#tls-generation-toolkit


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:

Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry, should have noted that the debug output goes to

nifi-bootstrap.log,

so thanks Mark for jumping in to help there.

If you look at the top of that log, you’ll note that there is no keystore
file provided and the truststore loaded is the default JRE cacerts
truststore. Can you please provide your nifi.properties file in a Gist,

**taking

care to redact any sensitive values** like keystore/truststore passwords,
although I think from looking at your log output, you are taking

advantage

of the encrypted configuration feature, so even viewing the encrypted
values should be ok. Could you also please provide the directory listing
where the keystore and truststore are located including the permissions

and

ownership information?

There may be a bug in the logic between cluster and standalone mode, but

I

haven’t encountered this behavior before. If you can start NiFi in
standalone mode, could you please provide the output of the following
command run from the terminal? It will simulate an HTTPS connection to

the

server and verify the key and certificate presented by NiFi.

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the
client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the

CA

which signed your NiFi server certificate (if self-signed, provide that
certificate)

$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/
b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?)

so

it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out

of

clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/

rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>


On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]

<[hidden email] <[hidden email]> <[hidden email]>
<[hidden email]>>>
wrote:


Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on

all

nodes of the cluster? It appears from the log message that the keystore

is

not found during startup. To further debug, you can add the following

line

in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <[hidden email] <[hidden email]> <
[hidden email]> <
[hidden email]>>
*[hidden email] <[hidden email]
<[hidden email]>
<[hidden email]>
<[hidden email]>> <

[hidden email] <[hidden email]
<[hidden email]>

<[hidden email]>

<[hidden email]>>>*

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a

local

key store while in clustered mode. If I set the node in clustered mode,

and

also provide a valid keystore, I receive a KeyStoreException [1]. If I

set

the configuration to not use clustered mode, NiFi will start up fine

with

the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


 at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

 at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

 at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

 at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

 at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

 ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not

available


 at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

 at java.security.Security.getImpl(Security.java:695)

~[na:1.8.0_11]


 at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

 ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





-- 
Ricky Saltzer
http://www.cloudera.com


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Secure Cluster Mode Issues

Ricky Saltzer
Andy -

I'm using version 1.1.0 (official binary). I'm using kerberos authentication, and able to log in using my internal Kerberos principal. To be clear, the only difference between blanking out the keyPasswd and keystorePasswd was that I was allowed to access the UI without manually importing the certificate, but instead agreeing to proceed even though I know the certificate was untrusted. 




Ricky

On Wed, Nov 30, 2016 at 7:43 PM, Andy LoPresto <[hidden email]> wrote:
Ricky,

Removing the redundant key password property shouldn’t have an impact (although you may be running a legacy version before NIFI-2222 [1] and NIFI-2466 [2] were fixed). Can you look at the top right of your NiFi UI and see what user is accessing the system? It should look like the screenshot I have attached. This, and the contents of logs/nifi-user.log, will indicate the authenticated user. That should help you figure out how the authentication is occurring (client certificate, LDAP, or Kerberos). If you still cannot determine it, you can update conf/logback.xml and change the logging level for the following loggers from INFO to DEBUG:

    <logger name="org.apache.nifi.web.security" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.web.api.config" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>
    <logger name="org.apache.nifi.authorization" level="INFO" additivity="false">
        <appender-ref ref="USER_FILE"/>
    </logger>


I only ask for this information because your results do not make sense and I fear that they will not be reproducible for the rest of your team when you try to deploy the system and let them access NiFi and I would hope we can provide the best experience from the beginning. 



Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 30, 2016, at 11:12 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

I think I may have figured out the problem. Although the keystorePasswd and
keyPasswd are the same, after completely removing the value for
nifi.security.keyPasswd,and restarting NiFi...I'm able to access the web UI
without manually importing the certificate.

On Tue, Nov 29, 2016 at 2:19 PM, Andy LoPresto <[hidden email]> wrote:

Ricky,

When using HTTPS in non-cluster mode, NiFi still requires user
authentication — this can be either client certificate (perhaps you already
had one loaded?), LDAP, or Kerberos. If you are able to access the NiFi UI
over HTTPS without presenting some authentication, something is seriously
broken. The warning in the browser is because the CA certificate that
signed the server certificate (the one being presented *to* the browser
by the application) is not trusted in the browser’s pre-installed trust
chain. If, for example, that CA cert had been imported to the browser ahead
of time, or if it was signed by a publicly known entity like DigiCert,
Verisign, Comodo, etc., you would not receive a warning.

For small teams, client certificates can be manageable, but if you want to
allow multiple users to connect with minimal identity management, I
recommend setting up an LDAP server (OpenLDAP, Microsoft ActiveDirectory,
Apache Directory Studio, etc.) and administering users there. Then the
users will just enter a username and password into a login field on their
first connection to NiFi and be authenticated.


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 11:07 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the reply, I used the openssl command you provided and indeed
the return code was *OK*. Before proceeding with the recommendation of

importing the key into my OSX keychain, I would like to understand why this
required. When using HTTPS mode in non-clustered mode, it does not require
clients to have a special key or cert imported to their machine. Instead,
the client is given a warning in the browser, and it's up to them to
proceed. This UI will serve as an endpoint to several users, and I would
really like to avoid the cumbersomeness of having members of multiple teams
follow instructions for importing keys just so they can access a web UI.

On Tue, Nov 29, 2016 at 1:55 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

The ERR_CONNECTION_CLOSED is likely because you are not sending a client
certificate on the HTTP request. By default, a secured cluster requires
client certificate authentication unless LDAP or Kerberos are configured as
identity providers [1]. The TLS Toolkit provides a quick way to generate a
valid client certificate which you can load into your browser in order to
access the site.

First, verify the cluster is running and accepting incoming connections
(we’re going to cheat here just to be quick about it; disclaimer that this
is not the RIGHT way to do this):

In the directory where you ran the toolkit, you noted there was a
“nifi-cert.pem” and “nifi-cert.key” file. The pem file is the PEM-encoded
public certificate of the NiFi CA cert that was generated by the toolkit,
and the key file is the PEM-encoded private key. Because this is the same
certificate that signed the NiFi server key loaded in the keystore, it is
also loaded into the truststore. That means the server will accept an
incoming connection with any certificate signed by the CA cert.
Coincidentally, the CA cert is self-signed, so…

$ openssl s_client -connect <host:port> -debug -state -cert nifi-cert.pem
-key nifi-key.key -CAfile nifi-cert.pem

That command will attempt to negotiate a TLS connection to your server by
presenting the CA cert and key as the client. Again, not semantically
correct, but  technically will work. You’ll get a long output, but it
should end in a section like this:

---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
  Protocol  : TLSv1.2
  Cipher    : ECDHE-RSA-AES256-SHA384
  Session-ID: 583DCD...9E828C
  Session-ID-ctx:
  Master-Key: 5477C0...A51E85
  Key-Arg   : None
  PSK identity: None
  PSK identity hint: None
  SRP username: None
  Start Time: 1480445265
  Timeout   : 300 (sec)
  Verify return code: 0 (ok)
---

The important part is the last line — you want the *Verify return code* to

be 0 for success. Once you have verified this, run the TLS toolkit again to
generate a valid client certificate:

$ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer, OU=Apache NiFi'
-B thisIsABadPassword

This will generate a PKCS12 keystore (*.p12) containing your public
certificate AND the private key, as well as an additional file (.password)
containing the password you provided for -B.

hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 146s @ 10:50:14 $ ./bin/tls-toolkit.sh standalone -C 'CN=Ricky Saltzer,
OU=Apache NiFi' -B password
2016/11/29 10:50:20 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandaloneCommandLine: No nifiPropertiesFile specified, using
embedded one.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Running standalone certificate generation with output directory
../nifi-toolkit-1.1.0
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Using existing CA certificate ../nifi-toolkit-1.1.0/nifi-cert.pem and key
../nifi-toolkit-1.1.0/nifi-key.key
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
No hostnames specified, not generating any host certificates or
configuration.
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Generating new client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
Successfully generated client certificate ../nifi-toolkit-1.1.0/CN=
Ricky_Saltzer_OU=Apache_NiFi.p12
2016/11/29 10:50:21 INFO [main] org.apache.nifi.toolkit.tls.standalone.
TlsToolkitStandalone:
tls-toolkit standalone completed successfully
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 7s @ 10:50:22 $ ll
total 136
drwxr-xr-x  20 alopresto  staff   680B Nov 29 10:50 ./
drwxr-xr-x   4 alopresto  staff   136B Nov 28 17:16 ../
-rw-------   1 alopresto  staff   3.4K Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.p12
-rw-------   1 alopresto  staff     8B Nov 29 10:50
CN=Ricky_Saltzer_OU=Apache_NiFi.password
...
-rw-------   1 alopresto  staff   1.2K Nov 28 16:31 nifi-cert.pem
-rw-------   1 alopresto  staff   1.6K Nov 28 16:31 nifi-key.key
hw12203:...lkit/nifi-toolkit-assembly/target/nifi-toolkit-
1.1.0-bin/nifi-toolkit-1.1.0
(master) alopresto
🔓 4s @ 10:50:27 $

You can double-click on the *.p12 file to open it in your default handler
— on OS X for example, this is Keychain Access. You can also manually load
it into Firefox to keep it isolated from your system certificates.

Now when you visit the UI through the browser, you should receive a prompt
for which certificate to present, and select this entry from the presented
list.

If you get a NiFi error message in the browser that you do not have
sufficient access, remember to update conf/authorizers.xml with an Initial
Admin Identity, which should match EXACTLY the DN of this certificate —
“CN=Ricky Saltzer, OU=Apache NiFi”. You can copy it from the failed
authentication message in logs/nifi-user.log to be sure.

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 29, 2016, at 10:27 AM, Ricky Saltzer <[hidden email]> wrote:

Hi Andy -

Thanks again for the help, sorry I've been a little preoccupied with some
other tasks. I took your advice and used the tls-toolkit and things are
looking a lot better. I generated a keystore/truststore for both of my
nodes. It also generated a pem and key file. I was able to get NiFi to
start in clustered mode (horray)! The only issue I'm seeing now is that I
can't actually connect to the web UI using HTTPS, as it immediately throws
a "ERR_CONNECTION_CLOSED" message in my browser.

Judging by the logs, NiFi is seemingly running. A leader election took
place and the node is heart beating to itself. Am I missing a step where
I'm supposed to be doing something with the pem/key files?

Thanks again,
Ricky

On Tue, Nov 8, 2016 at 8:22 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry for the delay in my response. I see a couple things which could be
causing an issue:

1. You do not specify a hostname for the NiFi node (nifi.web.https.host=)
which you should do. I have not set up a one-node cluster before, but my
suspicion is that this field needs to be populated, and the value needs to
match the CN for the issued server certificate. This value does not need to
be unique (i.e. it could be nifi.cloudera.com or localhost and you could
run multiple instances on the same machine, identified by the same
certificate and different ports), but I think it has to be populated.
1.a. While you have nifi.security.needClientAuth=false set in both
cluster and standalone mode, I am not sure if this allows for the
truststore to be missing completely. I would have to explore this further.
There is a current Jira for clarifying cluster, UI/API, and S2S TLS
configurations [1]. You can try using the default JRE truststore, located
at “$JAVA_HOME/jre/lib/security/cacerts” with password “changeit”.
2. The certificate you have put into the keystore appears to be a
certificate identifying you (Ricky the person) rather than a server entity.
You should create a certificate identifying the server itself, so the CN is
the FQDN as mentioned above. Then import into (or generate directly inside)
the keystore. You can also use the provided NiFi TLS Toolkit to automate
this entire process [2].
Certificate chain
0 s:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzer
i:/C=CA/ST=CA/L=Palo Alto/O=Cloudera/OU=EDH Team, Cloudera/CN=Ricky
Saltzercompare
to an example connection to google.com:443:

Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/*CN=*.google.com
<http://google.com>*
i:/C=US/O=Google Inc/CN=Google Internet Authority G2
1 s:/C=US/O=Google Inc/CN=Google Internet Authority G2
i:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
2 s:/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA
i:/C=US/O=Equifax/OU=Equifax Secure Certificate Authority

3. It appears the certificate you are using is expired. If you look at the
output of the OpenSSL command you ran, you can see that the result code was
*10*, not *0* as would be in the case of a successful connection (I
understand that it looks successful because it negotiates a key and
encrypts the channel).
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-SHA384
Session-ID: 581F5A5...41153B
Session-ID-ctx:
Master-Key: CEEED2...944C30
Key-Arg : None
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1478449748
Timeout : 300 (sec)
Verify return code: *10* *(certificate has expired)*

---
HTTP/1.1 400 No URI
Content-Length: 0
Connection: close
Server: Jetty(9.3.9.v20160517)
I rebuilt your certificate from the Base64-encoded version in the output,
and it appears it expired on October 25, 2016. You can verify this by
copying the section between "-----BEGIN CERTIFICATE-----“ and "-----END
CERTIFICATE-----“ (including these markers) into a text file and naming it
“ricky.pem” and then running the following command:

hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:30 $ more ricky.pem
-----BEGIN CERTIFICATE-----
MIIDizCCAnOgAwIBAgIEHPLwBzANBgkqhkiG9w0BAQsFADB2MQswCQYDVQQGEwJD
...
45GrxLz7n/ZXZk8q9aXcrSGaj+HHCyrO0/9NYtMNP2V5wpYcBRiHO9f3sHKgnzU=
-----END CERTIFICATE-----
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 5s @ 17:09:35 $ openssl x509 -in ricky.pem -text -noout
Certificate:
 Data:
     Version: 3 (0x2)
     Serial Number: 485683207 (0x1cf2f007)
 Signature Algorithm: sha256WithRSAEncryption
     Issuer: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
     Validity
         Not Before: Jul 27 17:15:46 2016 GMT
         *Not After : Oct 25 17:15:46 2016 GMT*

     Subject: C=CA, ST=CA, L=Palo Alto, O=Cloudera, OU=EDH Team,
Cloudera, CN=Ricky Saltzer
     Subject Public Key Info:
         Public Key Algorithm: rsaEncryption
             Public-Key: (2048 bit)
             Modulus:
                 00:f0:4c:fe:7e:af:69:89:f0:4b:a0:e0:df:e8:8c:
                 46:6e:d7:17:4d:17:eb:ee:89:25:ab:5b:35:6a:b2:
                 1e:07:84:95:5f:39:1e:c3:d9:5c:d4:54:09:0c:1a:
                 2a:60:df:9f:ce:7d:87:00:d7:31:23:37:62:8c:02:
                 c9:98:a3:d4:f5:3e:71:4e:1d:fb:bb:5b:86:35:af:
                 ce:6d:9d:ee:fb:65:d1:01:22:e3:f8:df:c0:81:d7:
                 fe:fd:f3:c8:48:c2:bd:23:e2:44:1e:8d:48:5f:81:
                 ed:10:62:77:71:3e:15:82:5a:c2:49:70:bb:ef:95:
                 d9:ff:f2:37:6c:06:63:fc:3c:98:a3:c9:49:3a:1d:
                 b8:37:e3:0b:50:0f:23:5f:26:32:19:c8:59:4e:5a:
                 09:5a:87:7f:b6:40:b4:37:18:20:74:fb:d2:f8:e9:
                 18:e8:d7:23:fa:83:be:2a:c3:33:8d:17:a8:b3:7c:
                 52:03:03:7c:24:80:c5:9c:a5:0d:21:3d:57:7c:d4:
                 0f:78:00:fd:be:69:dd:d8:fe:07:a0:09:77:81:b7:
                 bf:fd:0f:bd:8c:9a:35:42:4f:89:8e:31:fd:00:22:
                 16:bb:76:0b:51:b0:c4:81:64:69:f7:c4:5d:37:1e:
                 79:b7:08:d2:5a:6a:a7:43:98:d3:81:a9:08:00:57:
                 2b:c5
             Exponent: 65537 (0x10001)
     X509v3 extensions:
         X509v3 Subject Key Identifier:
             1C:DD:E1:16:5F:41:CC:7C:69:27:
E3:B1:6B:78:EF:FA:16:DA:1F:6F
 Signature Algorithm: sha256WithRSAEncryption
      da:5a:31:4b:3b:a6:bf:84:8d:6a:56:2a:82:88:2c:e0:95:49:
      30:f2:3a:31:ed:b5:bc:5b:92:e6:ad:b3:5b:9f:7a:71:bf:62:
      0f:38:27:4c:71:02:a1:2f:7f:fa:c0:46:da:78:64:36:35:34:
      1d:91:b3:f8:e2:23:22:1a:e8:e1:14:2b:15:2c:aa:36:71:39:
      05:d7:c0:2a:e0:cd:4d:f9:72:07:03:b9:f2:c8:6d:59:fb:c9:
      25:49:ae:3f:a2:c2:fe:d0:54:1e:45:13:8d:95:05:86:8b:a4:
      c4:7c:df:e1:7a:35:cc:48:e3:cd:20:87:c5:50:a7:43:d2:66:
      94:81:89:b1:a2:66:20:44:ee:cb:ea:08:c9:7c:aa:61:c7:24:
      5f:71:ca:6e:c0:a8:15:1f:4a:d3:79:e3:19:fd:7f:86:96:1a:
      4d:db:9d:38:6f:b3:16:fd:39:68:cf:1c:57:86:26:93:6d:0e:
      76:6b:aa:b8:9a:e7:c4:05:c6:c5:ae:5a:e1:a1:2a:ad:aa:e3:
      5b:9a:f1:70:a5:f0:54:76:b3:0e:00:e3:91:ab:c4:bc:fb:9f:
      f6:57:66:4f:2a:f5:a5:dc:ad:21:9a:8f:e1:c7:0b:2a:ce:d3:
      ff:4d:62:d3:0d:3f:65:79:c2:96:1c:05:18:87:3b:d7:f7:b0:
      72:a0:9f:35
hw12203:/Users/alopresto/Workspace/scratch (master) alopresto
🔓 11s @ 17:09:47 $

I would recommend using the TLS Toolkit to generate a valid CA, server
certificate, client certificate, and keystore & truststore (seriously, one
command does all of this) and then re-trying. From this stable baseline, I
think I’ll be able to better help iron out the cluster/standalone issues.

[1] https://issues.apache.org/jira/browse/NIFI-1990
[2] https://nifi.apache.org/docs/nifi-docs/html/
administration-guide.html#tls-generation-toolkit


Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 8, 2016, at 1:31 AM, Andre <[hidden email]> wrote:

Rick,

Can you confirm the certificate has a chain of trust with the default JDK
trusted certs? (i.e. trusted by the JVM)

Cheers


On Mon, Nov 7, 2016 at 3:38 AM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks again for the help.

The error message seems indicative that it doesn't seem to properly read
the keystore file. One thing to note, if I point the nifi properties to a
bogus keystore location, then it actually throws a FileNotFound exception.
This is really odd behavior, because as I mentioned I'm able to start it in
standalone mode using the correct keystore location, just as I try to do in
clustered mode.

I've attached both the clustered [1] nifi.properties, which doesn't work,
and the standalone [2] which does work. . I restored it to a more basic
configuration without the encrypted configuration, but with SSL still
enabled. I also added a diff [3] of both the standalone and clustered
properties file. Note that I I only have NiFi configured to use the
keystore and not a truststore. I've redacted a few of the values in the
property files, but be assured that the keystore is most definitely valid
and is readable / locatable, as starting in standalone works just fine.

I ran the SSL command [4] you gave me, minus the three PEM file arguments
as I don't have any of those on hand. I hope that is fine. The output still
looks good.

[1] https://gist.github.com/rickysaltzer/712aa6586592fe6628db2d57cec7a562
[2] https://gist.github.com/rickysaltzer/fe11c8233e4434eacedd7fd0a083d950
[3] https://gist.github.com/rickysaltzer/d715c7451eb554a54f14ec6e64da8558
[4] https://gist.github.com/rickysaltzer/5d7cdeff8868bfc1f47010189735411a




On Fri, Nov 4, 2016 at 7:48 PM, Andy LoPresto <[hidden email]>
wrote:

Hi Ricky,

Sorry, should have noted that the debug output goes to

nifi-bootstrap.log,

so thanks Mark for jumping in to help there.

If you look at the top of that log, you’ll note that there is no keystore
file provided and the truststore loaded is the default JRE cacerts
truststore. Can you please provide your nifi.properties file in a Gist,

**taking

care to redact any sensitive values** like keystore/truststore passwords,
although I think from looking at your log output, you are taking

advantage

of the encrypted configuration feature, so even viewing the encrypted
values should be ok. Could you also please provide the directory listing
where the keystore and truststore are located including the permissions

and

ownership information?

There may be a bug in the logic between cluster and standalone mode, but

I

haven’t encountered this behavior before. If you can start NiFi in
standalone mode, could you please provide the output of the following
command run from the terminal? It will simulate an HTTPS connection to

the

server and verify the key and certificate presented by NiFi.

* host — the NiFi hostname
* port — the port NiFi is running on
* path_to_your_cert.pem — the public key certificate identifying the
client/user (i.e. what you load into your browser to authenticate)
* path_to_your_key.key — the private key identifying the client/user
* path_to_your_CA_cert.pem — the public key certificate identifying the

CA

which signed your NiFi server certificate (if self-signed, provide that
certificate)

$ openssl s_client -connect <host:port> -debug -state -cert
<path_to_your_cert.pem> -key <path_to_your_key.pem> -CAfile
<path_to_your_CA_cert.pem>

Andy LoPresto
[hidden email]
*[hidden email] <[hidden email]>*
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 4, 2016, at 11:21 AM, Ricky Saltzer <[hidden email]> wrote:

Hey guys -

I went ahead and uploaded the boostrap log. I took a look at it and it
seems to be the same error [1]

[1]:
https://gist.githubusercontent.com/rickysaltzer/
b156594f92066873f80928eddf84e7bb/raw/4d0e018038b332f4fdf14644910dfa
9e70c57e49/gistfile1.txt

On Fri, Nov 4, 2016 at 2:14 PM, Mark Payne <[hidden email]> wrote:

Hey Ricky,

When you enable debug logging for SSL, it writes to StdErr (or StdOut?)

so

it will end up in your logs/nifi-bootstrap.log instead of nifi-app.log.
Can you give that a look?

Thanks
-Mark

On Nov 4, 2016, at 2:07 PM, Ricky Saltzer <[hidden email]> wrote:

Hey Andy -

Thanks for the response. I'm currently just trying to get one node in
clustered mode before adding a second. The keystore is stored locally and
I've confirmed it's readable, as it was able to start once I took it out

of

clustered mode. I added that line to the bootstrap.conf, but I don't
believe any additional logging was produced in regards to troubleshooting
this problem. Just in case, I've attached the entire log [1].

[1]:
https://gist.githubusercontent.com/rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt <https://gist.githubusercontent.com/

rickysaltzer/

ed454d87d2207d5acab401a473d4be57/raw/425c42da762fc5cc997153d48b09f0
fedabc88bb/gistfile1.txt>


On Wed, Nov 2, 2016 at 7:08 PM, Andy LoPresto <[hidden email]

<[hidden email] <[hidden email]> <[hidden email]>
<[hidden email]>>>
wrote:


Hi Ricky,

Sorry to hear you are having this issue. Is the keystore available on

all

nodes of the cluster? It appears from the log message that the keystore

is

not found during startup. To further debug, you can add the following

line

in bootstrap.conf to provide additional logging:

java.arg.15=-Djavax.net.debug=ssl,handshake

Andy LoPresto
[hidden email] <[hidden email] <[hidden email]> <
[hidden email]> <
[hidden email]>>
*[hidden email] <[hidden email]
<[hidden email]>
<[hidden email]>
<[hidden email]>> <

[hidden email] <[hidden email]
<[hidden email]>

<[hidden email]>

<[hidden email]>>>*

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Nov 2, 2016, at 2:25 PM, Ricky Saltzer <[hidden email]> wrote:

Hey all -

I'm using NiFi 1.0 and I'm having an issue using secure mode with a

local

key store while in clustered mode. If I set the node in clustered mode,

and

also provide a valid keystore, I receive a KeyStoreException [1]. If I

set

the configuration to not use clustered mode, NiFi will start up fine

with

the provided key store. Am I supposed to be storing this key store in
Zookeeper somewhere?


[1]


Caused by: java.security.KeyStoreException:  not found


 at java.security.KeyStore.getInstance(KeyStore.java:839)
~[na:1.8.0_11]

 at
org.apache.nifi.io.socket.SSLContextFactory.<init>(
SSLContextFactory.java:61)
~[nifi-socket-utils-1.0.0.jar:1.0.0]

 at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:45)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

 at
org.apache.nifi.cluster.protocol.spring.ServerSocketConfigurationFacto
ryBean.getObject(ServerSocketConfigurationFactoryBean.java:30)
~[nifi-framework-cluster-protocol-1.0.0.jar:1.0.0]

 at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.
doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]

 ... 69 common frames omitted

Caused by: java.security.NoSuchAlgorithmException:  KeyStore not

available


 at sun.security.jca.GetInstance.getInstance(GetInstance.java:159)
~[na:1.8.0_11]

 at java.security.Security.getImpl(Security.java:695)

~[na:1.8.0_11]


 at java.security.KeyStore.getInstance(KeyStore.java:836)
~[na:1.8.0_11]

 ... 73 common frames omitted





--
Ricky Saltzer
http://www.cloudera.com <http://www.cloudera.com/>





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





--
Ricky Saltzer
http://www.cloudera.com





-- 
Ricky Saltzer
http://www.cloudera.com




--