NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

bmichaud
Fedora, Java 1.8, NiFi 1.1.1

I was able to start up NiFi as a single stand-alone server, but when I tried to start it as a cluster, configured the same way I had configured it in 1.0.0:
nifi.properties:
nifi.state.management.embedded.zookeeper.start=true
nifi.cluster.is.node=true
nifi.cluster.node.address=server1
nifi.zookeeper.connect.string=server1:2181,server2:2181,server3:2181

zookeeper.properties:
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888

state-management.xml:
    <cluster-provider>
        <id>zk-provider</id>
        <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
        <property name="Connect String">server1:2181,server2:2181,server3:2181</property>
        <property name="Root Node">/nifi</property>
        <property name="Session Timeout">10 seconds</property>
        <property name="Access Control">Open</property>
    </cluster-provider>


Errors:
=====
2017-01-20 17:54:25,503 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 3 milliseconds
2017-01-20 17:54:25,522 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@24df4805 checkpointed with 0 Records and 0 Swap Files in 18 milliseconds (Stop-the-world time = 2 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1
2017-01-20 17:54:25,578 ERROR [main] o.a.z.server.quorum.QuorumPeerConfig  does not have the form host:port or host:port:port  or host:port:port:type
2017-01-20 17:54:26,041 WARN [main] org.eclipse.jetty.webapp.WebAppContext Failed startup of context o.e.j.w.WebAppContext@1dc37d4f{/nifi-api,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-api-1.1.1.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.1.war}
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
        at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85) ~[na:na]
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.start(Server.java:411) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.doStart(Server.java:378) [jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) [jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675) [nifi-jetty-1.1.1.jar:1.1.1]
        at org.apache.nifi.NiFi.<init>(NiFi.java:156) [nifi-runtime-1.1.1.jar:1.1.1]
        at org.apache.nifi.NiFi.main(NiFi.java:262) [nifi-runtime-1.1.1.jar:1.1.1]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53) ~[na:na]
        ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        ... 34 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:188) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:53) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:176) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.FlowController.<init>(FlowController.java:562) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:403) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:61) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        ... 41 common frames omitted
2017-01-20 17:54:26,633 INFO [main] /nifi-content-viewer No Spring WebApplicationInitializer types detected on classpath
2017-01-20 17:54:26,668 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@5a349644{/nifi-content-viewer,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-content-viewer-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.1.1.war}
2017-01-20 17:54:26,669 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.s.h.ContextHandler@1a819769{/nifi-docs,null,AVAILABLE}
2017-01-20 17:54:26,724 INFO [main] /nifi-docs No Spring WebApplicationInitializer types detected on classpath
2017-01-20 17:54:26,726 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@6479b4cb{/nifi-docs,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-docs-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.1.1.war}
2017-01-20 17:54:26,755 INFO [main] / No Spring WebApplicationInitializer types detected on classpath
2017-01-20 17:54:26,778 INFO [main] o.e.jetty.server.handler.ContextHandler Started o.e.j.w.WebAppContext@582b6362{/,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-error-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.1.1.war}
2017-01-20 17:54:26,818 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@64f55630{HTTP/1.1,[http/1.1]}{apsrt3401.uhc.com:9080}
2017-01-20 17:54:26,821 INFO [main] o.e.jetty.util.ssl.SslContextFactory x509=X509@2ab0ca04(coda-nifi-ssl-cert,h=[apsrt3387.uhc.com, apsrt3389.uhc.com, apsrt3388.uhc.com, apsrt3391.uhc.com, apsrt3390.uhc.com, apsrt3402.uhc.com, apsrt3395.uhc.com, apsrt3394.uhc.com, apsrt3393.uhc.com, apsrt3396.uhc.com, apsrt3398.uhc.com, apsrt3397.uhc.com, apsrt3399.uhc.com, apsrt3409.uhc.com, apsrt3408.uhc.com, apsrt3403.uhc.com, apsrt3400.uhc.com, apsrt3401.uhc.com, apsrt3410.uhc.com],w=[]) for SslContextFactory@53c68ea5(file:///app_2/runtime/nifi-1.1.1/conf/keystore.jks,file:///app_2/runtime/nifi-1.1.1/conf/truststore.jks)
2017-01-20 17:54:26,830 INFO [main] o.eclipse.jetty.server.AbstractConnector Started ServerConnector@2cf29dce{SSL,[ssl, http/1.1]}{apsrt3401.uhc.com:9443}
2017-01-20 17:54:26,831 INFO [main] org.eclipse.jetty.server.Server Started @77025ms
2017-01-20 17:54:26,834 WARN [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
        at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85) ~[na:na]
        at org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) ~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520) ~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.start(Server.java:411) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.server.Server.doStart(Server.java:378) ~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
        at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675) ~[nifi-jetty-1.1.1.jar:1.1.1]
        at org.apache.nifi.NiFi.<init>(NiFi.java:156) [nifi-runtime-1.1.1.jar:1.1.1]
        at org.apache.nifi.NiFi.main(NiFi.java:262) [nifi-runtime-1.1.1.jar:1.1.1]
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53) ~[na:na]
        ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060) ~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        at org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        ... 34 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
        at org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:188) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
        at org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:53) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:176) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.FlowController.<init>(FlowController.java:562) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:403) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:61) ~[nifi-framework-core-1.1.1.jar:1.1.1]
        at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
        ... 41 common frames omitted
2017-01-20 17:54:26,835 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server...
2017-01-20 17:54:26,845 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@64f55630{HTTP/1.1,[http/1.1]}{apsrt3401.uhc.com:9080}
2017-01-20 17:54:26,849 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@2cf29dce{SSL,[ssl, http/1.1]}{apsrt3401.uhc.com:9443}
2017-01-20 17:54:26,855 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@582b6362{/,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.1.1.war}
2017-01-20 17:54:26,858 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@6479b4cb{/nifi-docs,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.1.1.war}
2017-01-20 17:54:26,858 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.s.h.ContextHandler@1a819769{/nifi-docs,null,UNAVAILABLE}
2017-01-20 17:54:26,861 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@5a349644{/nifi-content-viewer,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.1.1.war}
2017-01-20 17:54:26,862 INFO [Thread-1] o.a.n.w.c.ApplicationStartupContextListener Initiating shutdown of flow service...
2017-01-20 17:54:26,877 INFO [Thread-1] o.e.jetty.server.handler.ContextHandler Stopped o.e.j.w.WebAppContext@1dc37d4f{/nifi-api,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-api-1.1.1.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.1.war}
2017-01-20 17:54:26,880 WARN [Thread-1] org.apache.nifi.web.server.JettyServer Failed to stop web server
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

Andy LoPresto-2
Hi Ben,

Is your deployment secured with certificates or running in plaintext? Pierre Villiard has written step-by-step instructions for a secured cluster on 1.1.0 [1] and default cluster on 1.0.0 [2]. I don’t believe anything changed in the configuration of ZK between 1.0.0 and 1.1.1. Which of these nodes (none, 1, 2, all?) are running embedded ZooKeeper nodes? If it’s more than one, you’ll need to set the ZK myid value (see [2] for instructions). 

I’ve included the relevant sections of my last cluster configs. Note that while the nifi.cluster.node.address in each nifi.properties must be the name of the server running it, the nifi.zookeeper.connect.string should be the embedded ZK host. You could also try changing your nifi.zookeeper.connect.string to just point to one instance of ZK and see if that works (no, this is not normal, but AIOOBE usually means something’s not getting parsed correctly). 

Hope this helps. Please let us know what results you achieve or if you have further questions. 

hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 26s @ 16:49:13 $ more node1/conf/zookeeper.properties
clientPort=2181
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30

#
# Specifies the servers that are part of this zookeeper ensemble. For
# every NiFi instance running an embedded zookeeper, there needs to be
# a server entry below. For instance:
#
# server.1=nifi-node1-hostname:2888:3888
# server.2=nifi-node2-hostname:2888:3888
# server.3=nifi-node3-hostname:2888:3888
#
# The index of the server corresponds to the myid file that gets created
# in the dataDir of each node running an embedded zookeeper. See the
# administration guide for more details.
#

server.1=ncm.nifi.apache.org:2888:3888
hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 36s @ 16:49:51 $ more node2/conf/zookeeper.properties
clientPort=2182
initLimit=10
autopurge.purgeInterval=24
syncLimit=5
tickTime=2000
dataDir=./state/zookeeper
autopurge.snapRetainCount=30

...comments...

server.1=ncm.nifi.apache.org:2888:3888
hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 55s @ 16:50:47 $ more node1/state/zookeeper/myid
1
hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 38s @ 16:51:26 $ more node2/state/zookeeper/myid
2
hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 38s @ 16:51:26 $ more node1/conf/nifi.properties
...
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=ncm.nifi.apache.org
nifi.cluster.node.protocol.port=10000
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.firewall.file=

# How long a request should be allowed to hold a 'lock' on a component. #
nifi.cluster.request.replication.claim.timeout=15 secs

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=ncm.nifi.apache.org:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

hw12203:/Users/alopresto/Workspace/scratch/NIFI-2186 (master) alopresto
🔓 38s @ 16:51:28 $ more node2/conf/nifi.properties
# cluster common properties (all nodes must have same values) #
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true

# cluster node properties (only configure for cluster nodes) #
nifi.cluster.is.node=true
nifi.cluster.node.address=node.nifi.apache.org
nifi.cluster.node.protocol.port=10001
nifi.cluster.node.protocol.threads=10
nifi.cluster.node.event.history.size=25
nifi.cluster.node.connection.timeout=5 sec
nifi.cluster.node.read.timeout=5 sec
nifi.cluster.firewall.file=

# How long a request should be allowed to hold a 'lock' on a component. #
nifi.cluster.request.replication.claim.timeout=15 secs

# zookeeper properties, used for cluster management #
nifi.zookeeper.connect.string=ncm.nifi.apache.org:2181
nifi.zookeeper.connect.timeout=3 secs
nifi.zookeeper.session.timeout=3 secs
nifi.zookeeper.root.node=/nifi

 
Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 20, 2017, at 2:39 PM, bmichaud <[hidden email]> wrote:

Fedora, Java 1.8, NiFi 1.1.1

I was able to start up NiFi as a single stand-alone server, but when I tried
to start it as a cluster, configured the same way I had configured it in
1.0.0:
nifi.properties:
nifi.state.management.embedded.zookeeper.start=true
nifi.cluster.is.node=true
nifi.cluster.node.address=server1
nifi.zookeeper.connect.string=server1:2181,server2:2181,server3:2181

zookeeper.properties:
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888

state-management.xml:
   <cluster-provider>
       <id>zk-provider</id>

<class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class>
       <property name="Connect
String">server1:2181,server2:2181,server3:2181</property>
       <property name="Root Node">/nifi</property>
       <property name="Session Timeout">10 seconds</property>
       <property name="Access Control">Open</property>
   </cluster-provider>


Errors:
=====
2017-01-20 17:54:25,503 INFO [main] org.wali.MinimalLockingWriteAheadLog
Successfully recovered 0 records in 3 milliseconds
2017-01-20 17:54:25,522 INFO [main] org.wali.MinimalLockingWriteAheadLog
org.wali.MinimalLockingWriteAheadLog@24df4805 checkpointed with 0 Records
and 0 Swap Files in 18 milliseconds (Stop-the-world time = 2 milliseconds,
Clear Edit Logs time = 2 millis), max Transaction ID -1
2017-01-20 17:54:25,578 ERROR [main] o.a.z.server.quorum.QuorumPeerConfig
does not have the form host:port or host:port:port  or host:port:port:type
2017-01-20 17:54:26,041 WARN [main] org.eclipse.jetty.webapp.WebAppContext
Failed startup of context
o.e.j.w.WebAppContext@1dc37d4f{/nifi-api,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-api-1.1.1.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.1.war}
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
       at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85)
~[na:na]
       at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at org.eclipse.jetty.server.Server.start(Server.java:411)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at org.eclipse.jetty.server.Server.doStart(Server.java:378)
[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675)
[nifi-jetty-1.1.1.jar:1.1.1]
       at org.apache.nifi.NiFi.<init>(NiFi.java:156)
[nifi-runtime-1.1.1.jar:1.1.1]
       at org.apache.nifi.NiFi.main(NiFi.java:262)
[nifi-runtime-1.1.1.jar:1.1.1]
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowService': FactoryBean threw exception on object
creation; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'flowController': FactoryBean threw exception on object creation;
nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53)
~[na:na]
       ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowController': FactoryBean threw exception on
object creation; nested exception is
java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       ... 34 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:188)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
       at
org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:53)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:176)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.FlowController.<init>(FlowController.java:562)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:403)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:61)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       ... 41 common frames omitted
2017-01-20 17:54:26,633 INFO [main] /nifi-content-viewer No Spring
WebApplicationInitializer types detected on classpath
2017-01-20 17:54:26,668 INFO [main] o.e.jetty.server.handler.ContextHandler
Started
o.e.j.w.WebAppContext@5a349644{/nifi-content-viewer,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-content-viewer-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.1.1.war}
2017-01-20 17:54:26,669 INFO [main] o.e.jetty.server.handler.ContextHandler
Started o.e.j.s.h.ContextHandler@1a819769{/nifi-docs,null,AVAILABLE}
2017-01-20 17:54:26,724 INFO [main] /nifi-docs No Spring
WebApplicationInitializer types detected on classpath
2017-01-20 17:54:26,726 INFO [main] o.e.jetty.server.handler.ContextHandler
Started
o.e.j.w.WebAppContext@6479b4cb{/nifi-docs,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-docs-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.1.1.war}
2017-01-20 17:54:26,755 INFO [main] / No Spring WebApplicationInitializer
types detected on classpath
2017-01-20 17:54:26,778 INFO [main] o.e.jetty.server.handler.ContextHandler
Started
o.e.j.w.WebAppContext@582b6362{/,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-error-1.1.1.war/webapp/,AVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.1.1.war}
2017-01-20 17:54:26,818 INFO [main] o.eclipse.jetty.server.AbstractConnector
Started
ServerConnector@64f55630{HTTP/1.1,[http/1.1]}{apsrt3401.uhc.com:9080}
2017-01-20 17:54:26,821 INFO [main] o.e.jetty.util.ssl.SslContextFactory
x509=X509@2ab0ca04(coda-nifi-ssl-cert,h=[apsrt3387.uhc.com,
apsrt3389.uhc.com, apsrt3388.uhc.com, apsrt3391.uhc.com, apsrt3390.uhc.com,
apsrt3402.uhc.com, apsrt3395.uhc.com, apsrt3394.uhc.com, apsrt3393.uhc.com,
apsrt3396.uhc.com, apsrt3398.uhc.com, apsrt3397.uhc.com, apsrt3399.uhc.com,
apsrt3409.uhc.com, apsrt3408.uhc.com, apsrt3403.uhc.com, apsrt3400.uhc.com,
apsrt3401.uhc.com, apsrt3410.uhc.com],w=[]) for
SslContextFactory@53c68ea5(file:///app_2/runtime/nifi-1.1.1/conf/keystore.jks,file:///app_2/runtime/nifi-1.1.1/conf/truststore.jks)
2017-01-20 17:54:26,830 INFO [main] o.eclipse.jetty.server.AbstractConnector
Started ServerConnector@2cf29dce{SSL,[ssl,
http/1.1]}{apsrt3401.uhc.com:9443}
2017-01-20 17:54:26,831 INFO [main] org.eclipse.jetty.server.Server Started
@77025ms
2017-01-20 17:54:26,834 WARN [main] org.apache.nifi.web.server.JettyServer
Failed to start web server... shutting down.
org.apache.nifi.web.NiFiCoreException: Unable to start Flow Controller.
       at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:85)
~[na:na]
       at
org.eclipse.jetty.server.handler.ContextHandler.callContextInitialized(ContextHandler.java:837)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.callContextInitialized(ServletContextHandler.java:533)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.ContextHandler.startContext(ContextHandler.java:810)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:345)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1404)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1366)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
~[jetty-servlet-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:520)
~[jetty-webapp-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.gzip.GzipHandler.doStart(GzipHandler.java:231)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at org.eclipse.jetty.server.Server.start(Server.java:411)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:106)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at org.eclipse.jetty.server.Server.doStart(Server.java:378)
~[jetty-server-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
~[jetty-util-9.3.9.v20160517.jar:9.3.9.v20160517]
       at
org.apache.nifi.web.server.JettyServer.start(JettyServer.java:675)
~[nifi-jetty-1.1.1.jar:1.1.1]
       at org.apache.nifi.NiFi.<init>(NiFi.java:156)
[nifi-runtime-1.1.1.jar:1.1.1]
       at org.apache.nifi.NiFi.main(NiFi.java:262)
[nifi-runtime-1.1.1.jar:1.1.1]
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowService': FactoryBean threw exception on object
creation; nested exception is
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'flowController': FactoryBean threw exception on object creation;
nested exception is java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.apache.nifi.web.contextlistener.ApplicationStartupContextListener.contextInitialized(ApplicationStartupContextListener.java:53)
~[na:na]
       ... 28 common frames omitted
Caused by: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'flowController': FactoryBean threw exception on
object creation; nested exception is
java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.getObjectFromFactoryBean(FactoryBeanRegistrySupport.java:103)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getObjectForBeanInstance(AbstractBeanFactory.java:1585)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:317)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1060)
~[spring-context-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       at
org.apache.nifi.spring.StandardFlowServiceFactoryBean.getObject(StandardFlowServiceFactoryBean.java:48)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       ... 34 common frames omitted
Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
       at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:188)
~[zookeeper-3.4.6.jar:3.4.6-1569965]
       at
org.apache.nifi.controller.state.server.ZooKeeperStateServer.<init>(ZooKeeperStateServer.java:53)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.state.server.ZooKeeperStateServer.create(ZooKeeperStateServer.java:176)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.FlowController.<init>(FlowController.java:562)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.controller.FlowController.createClusteredInstance(FlowController.java:403)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:61)
~[nifi-framework-core-1.1.1.jar:1.1.1]
       at
org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168)
~[spring-beans-4.2.4.RELEASE.jar:4.2.4.RELEASE]
       ... 41 common frames omitted
2017-01-20 17:54:26,835 INFO [Thread-1] org.apache.nifi.NiFi Initiating
shutdown of Jetty web server...
2017-01-20 17:54:26,845 INFO [Thread-1]
o.eclipse.jetty.server.AbstractConnector Stopped
ServerConnector@64f55630{HTTP/1.1,[http/1.1]}{apsrt3401.uhc.com:9080}
2017-01-20 17:54:26,849 INFO [Thread-1]
o.eclipse.jetty.server.AbstractConnector Stopped
ServerConnector@2cf29dce{SSL,[ssl, http/1.1]}{apsrt3401.uhc.com:9443}
2017-01-20 17:54:26,855 INFO [Thread-1]
o.e.jetty.server.handler.ContextHandler Stopped
o.e.j.w.WebAppContext@582b6362{/,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-error-1.1.1.war}
2017-01-20 17:54:26,858 INFO [Thread-1]
o.e.jetty.server.handler.ContextHandler Stopped
o.e.j.w.WebAppContext@6479b4cb{/nifi-docs,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-docs-1.1.1.war}
2017-01-20 17:54:26,858 INFO [Thread-1]
o.e.jetty.server.handler.ContextHandler Stopped
o.e.j.s.h.ContextHandler@1a819769{/nifi-docs,null,UNAVAILABLE}
2017-01-20 17:54:26,861 INFO [Thread-1]
o.e.jetty.server.handler.ContextHandler Stopped
o.e.j.w.WebAppContext@5a349644{/nifi-content-viewer,null,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-content-viewer-1.1.1.war}
2017-01-20 17:54:26,862 INFO [Thread-1]
o.a.n.w.c.ApplicationStartupContextListener Initiating shutdown of flow
service...
2017-01-20 17:54:26,877 INFO [Thread-1]
o.e.jetty.server.handler.ContextHandler Stopped
o.e.j.w.WebAppContext@1dc37d4f{/nifi-api,file:///app_2/runtime/nifi-1.1.1/work/jetty/nifi-web-api-1.1.1.war/webapp/,UNAVAILABLE}{./work/nar/framework/nifi-framework-nar-1.1.1.nar-unpacked/META-INF/bundled-dependencies/nifi-web-api-1.1.1.war}
2017-01-20 17:54:26,880 WARN [Thread-1]
org.apache.nifi.web.server.JettyServer Failed to stop web server




--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-1-1-1-can-t-start-as-a-cluster-OverlappingFileLockException-tp14486.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

bmichaud
Andy LoPresto-2 wrote
Is your deployment secured with certificates or running in plaintext? Pierre Villiard has written step-by-step instructions for a secured cluster on 1.1.0 [1] and default cluster on 1.0.0 [2]. I don’t believe anything changed in the configuration of ZK between 1.0.0 and 1.1.1.
certificates. Where are these instructions? in the admin part of the manual, because I followed those instructions back when we went to 1.0.0.

Andy LoPresto-2 wrote
Which of these nodes (none, 1, 2, all?) are running embedded ZooKeeper nodes? If it’s more than one, you’ll need to set the ZK myid value (see [2] for instructions).
All nodes run Zookeeper. I had already set the myid values.

Andy LoPresto-2 wrote
I’ve included the relevant sections of my last cluster configs.
thanks

Andy LoPresto-2 wrote
Note that while the nifi.cluster.node.address in each nifi.properties must be the name of the server running it, the nifi.zookeeper.connect.string should be the embedded ZK host.
I have the connect string pointing to all three, but I tried this change and got the same result as before. Then I checked the zookeeper.properties files and found that they had not been configured by my configuration script. Now the configurations have all three zookeeper servers as I had before. The servers are all clustered, but I am getting an SSL error (see below)

Do you recommend I still make this change? Do I also need to change the zookeeper.properties to only include the local node address?

Andy LoPresto-2 wrote
You could also try changing your nifi.zookeeper.connect.string to just point to one instance of ZK and see if that works (no, this is not normal, but AIOOBE usually means something’s not getting parsed correctly).
So all three nodes' nifi.properties file points to the same Zookeeper, even though all three servers have a zookeeper running? Does this instruction contradict the one just above?

Andy LoPresto-2 wrote
Hope this helps. Please let us know what results you achieve or if you have further questions.
Thanks it did, because it forced me to check all the files and see that my zookeeper.properties was not configured in my script to include node addresses.

Now I am getting another error. The certificate I have has over twenty servers in it, and it is apparently not trusted. Its CN is named for a server that is not even in this cluster I am upgrading now. Should I create a new issue?

Error
Untrusted proxy CN=server15, O=My Company Inc., L=Main, ST=State, C=US
Untrusted proxy CN=server15, O=My Company Inc., L=Main, ST=State, C=US


Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

Andy LoPresto-2
The instructions by Pierre that I mentioned were linked at the bottom of my email as [1] (for 1.1.0) and [2] (for 1.0.0). 

Changing the ZK connect strings was just to isolate the issue during debugging. You should not have to make that change now. 

The error you listed is incomplete and I need more information (preferably a full stacktrace) to debug it. I don’t know if it’s a true SSL/TLS error or if it’s an authorization error because you didn’t add the CN of each server as an authorized user in the users.xml and authorizations.xml. Please provide the complete output of logs/nifi-app.log and logs/nifi-user.log as well as conf/users.xml and conf/authorizations.xml. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 23, 2017, at 4:14 AM, bmichaud <[hidden email]> wrote:

Andy LoPresto-2 wrote
Is your deployment secured with certificates or running in plaintext?
Pierre Villiard has written step-by-step instructions for a secured
cluster on 1.1.0 [1] and default cluster on 1.0.0 [2]. I don’t believe
anything changed in the configuration of ZK between 1.0.0 and 1.1.1.

certificates. Where are these instructions? in the admin part of the manual,
because I followed those instructions back when we went to 1.0.0.


Andy LoPresto-2 wrote
Which of these nodes (none, 1, 2, all?) are running embedded ZooKeeper
nodes? If it’s more than one, you’ll need to set the ZK myid value (see
[2] for instructions).

All nodes run Zookeeper. I had already set the myid values.


Andy LoPresto-2 wrote
I’ve included the relevant sections of my last cluster configs.

thanks


Andy LoPresto-2 wrote
Note that while the nifi.cluster.node.address in each nifi.properties must
be the name of the server running it, the nifi.zookeeper.connect.string
should be the embedded ZK host.

I have the connect string pointing to all three, but I tried this change and
got the same result as before. Then I checked the zookeeper.properties files
and found that they had not been configured by my configuration script. Now
the configurations have all three zookeeper servers as I had before. The
servers are all clustered, but I am getting an SSL error (see below)

Do you recommend I still make this change? Do I also need to change the
zookeeper.properties to only include the local node address?


Andy LoPresto-2 wrote
You could also try changing your nifi.zookeeper.connect.string to just
point to one instance of ZK and see if that works (no, this is not normal,
but AIOOBE usually means something’s not getting parsed correctly).

So all three nodes' nifi.properties file points to the same Zookeeper, even
though all three servers have a zookeeper running? Does this instruction
contradict the one just above?


Andy LoPresto-2 wrote
Hope this helps. Please let us know what results you achieve or if you
have further questions.

Thanks it did, because it forced me to check all the files and see that my
zookeeper.properties was not configured in my script to include node
addresses.

Now I am getting another error. The certificate I have has over twenty
servers in it, and it is apparently not trusted. Its CN is named for a
server that is not even in this cluster I am upgrading now. Should I create
a new issue?

Error
Untrusted proxy CN=server15, O=My Company Inc., L=Main, ST=State, C=US
Untrusted proxy CN=server15, O=My Company Inc., L=Main, ST=State, C=US






--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-1-1-1-can-t-start-as-a-cluster-OverlappingFileLockException-tp14486p14494.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

bmichaud
Andy LoPresto-2 wrote
The instructions by Pierre that I mentioned were linked at the bottom of my email as [1] (for 1.1.0) and [2] (for 1.0.0).
Thanks those are helpful.

The logs contain internal server names and IP addresses. The nifi-app.log does not seem to have any more information, only an unrelated error repeated over and over.

Here is the nifi-user.log error (obfuscated-IP 1.2.3.4 is my PC connecting to the server GUI, 5.6.7.8 is the node IP, server name "common.name.server.company.com" is the common name of the certificate in the keystore file, and server name "this.server.company.com" is the current server node.):

====
GUI:
====
Access Denied
home
Untrusted proxy CN=common.name.server.company.com, O=UnitedHealth Group Inc., L=Plymouth, ST=Minnesota, C=US

==============
nifi-user.log:
==============
2017-01-24 08:35:47,051 INFO [NiFi Web Server-81306] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (EMAILADDRESS=ben_michaud@company.com, CN=bmichau1, CN=Users, DC=ms, DC=ds, DC=uhc, DC=com) GET https://this.server.company.com:9443/nifi-api/flow/current-user (source ip: 1.2.3.4)
2017-01-24 08:35:47,051 INFO [NiFi Web Server-81306] o.a.n.w.s.NiFiAuthenticationFilter Authentication success for EMAILADDRESS=ben_michaud@company.com, CN=bmichau1, CN=Users, DC=ms, DC=ds, DC=uhc, DC=com
2017-01-24 08:35:47,127 INFO [NiFi Web Server-81360] o.a.n.w.s.NiFiAuthenticationFilter Attempting request for (<EMAILADDRESS=ben_michaud@company.com, CN=bmichau1, CN=Users, DC=ms, DC=ds, DC=uhc, DC=com><CN=common.name.server.company.com, O=UnitedHealth Group Inc., L=Plymouth, ST=Minnesota, C=US>) GET https://this.server.company.com:9443/nifi-api/flow/current-user (source ip: 5.6.7.8)
2017-01-24 08:35:47,129 INFO [NiFi Web Server-81360] o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted proxy CN=common.name.server.company.com, O=UnitedHealth Group Inc., L=Plymouth, ST=Minnesota, C=US
codaops@this.server:/app_2/runtime/nifi/logs
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

Andy LoPresto-2
Ben,

Please ensure that each of the node CNs is added to users.xml as a user and that the corresponding user entry has the /proxy resource available to it in authorizations.xml. See a full example here [1]. I subbed the UUIDs for simple string identifiers to make it easier to match in the example below. 


users.xml

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<tenants>
    <groups/>
    <users>
        <user identifier=“abc" identity="[hidden email], CN=bmichau1, CN=Users, DC=ms, DC=ds, DC=uhc, DC=com"/>
        <user identifier=“def" identity="CN=common.name.server.company.com, O=UnitedHealth Group Inc., L=Plymouth, ST=Minnesota, C=US"/> ... repeat for all nodes in cluster ...
    </users>
</tenants>
authorizations.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizations>
    <policies>
        <policy identifier=“1" resource="/flow" action="R">
            <user identifier=“abc"/>
        </policy>
        ...
        <policy identifier=“9" resource="/proxy" action="R">
            <user identifier=“def"/>
        </policy>
        <policy identifier=“10" resource="/proxy" action="W">
            <user identifier=“def"/>
        </policy>
    </policies>
</authorizations>


[1] http://bryanbende.com/development/2016/08/17/apache-nifi-1-0-0-authorization-and-multi-tenancy
Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 24, 2017, at 8:15 AM, bmichaud <[hidden email]> wrote:

Andy LoPresto-2 wrote
The instructions by Pierre that I mentioned were linked at the bottom of
my email as [1] (for 1.1.0) and [2] (for 1.0.0).

Thanks those are helpful.

The logs contain internal server names and IP addresses. The nifi-app.log
does not seem to have any more information, only an unrelated error repeated
over and over.

Here is the nifi-user.log error (obfuscated-IP 1.2.3.4 is my PC connecting
to the server GUI, 5.6.7.8 is the node IP, server name
"common.name.server.company.com" is the common name of the certificate in
the keystore file, and server name "this.server.company.com" is the current
server node.):

====
GUI:
====
Access Denied
home
Untrusted proxy CN=common.name.server.company.com, O=UnitedHealth Group
Inc., L=Plymouth, ST=Minnesota, C=US

==============
nifi-user.log:
==============
2017-01-24 08:35:47,051 INFO [NiFi Web Server-81306]
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
([hidden email], CN=bmichau1, CN=Users, DC=ms, DC=ds,
DC=uhc, DC=com) GET
https://this.server.company.com:9443/nifi-api/flow/current-user (source ip:
1.2.3.4)
2017-01-24 08:35:47,051 INFO [NiFi Web Server-81306]
o.a.n.w.s.NiFiAuthenticationFilter Authentication success for
[hidden email], CN=bmichau1, CN=Users, DC=ms, DC=ds,
DC=uhc, DC=com
2017-01-24 08:35:47,127 INFO [NiFi Web Server-81360]
o.a.n.w.s.NiFiAuthenticationFilter Attempting request for
(<[hidden email], CN=bmichau1, CN=Users, DC=ms, DC=ds,
DC=uhc, DC=com><CN=common.name.server.company.com, O=UnitedHealth Group
Inc., L=Plymouth, ST=Minnesota, C=US>) GET
https://this.server.company.com:9443/nifi-api/flow/current-user (source ip:
5.6.7.8)
2017-01-24 08:35:47,129 INFO [NiFi Web Server-81360]
o.a.n.w.s.NiFiAuthenticationFilter Rejecting access to web api: Untrusted
proxy CN=common.name.server.company.com, O=UnitedHealth Group Inc.,
L=Plymouth, ST=Minnesota, C=US
[hidden email]:/app_2/runtime/nifi/logs



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-1-1-1-can-t-start-as-a-cluster-OverlappingFileLockException-tp14486p14508.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

bmichaud
It appears that, for the time being the authorization issue is solved, but I did nothing to solve it. Thanks for the help.
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 proxy authorization error

bmichaud
This post was updated on .
In reply to this post by Andy LoPresto-2
I spoke too soon. I just tried on another cluster, and I am having the same "Untrusted proxy" error for the common name server.

The way I was setting up initial security was with the legacy authorized-users.xml file. additionally, my certificate keystore.jks has been and must be generated  by a company internal tool. I have done so, and generated a truststore file using the java keytool, adding trusted entries for various public keys we use for our company. This has worked fine in 1.0.0 and prior versions.

The instructions from Brian Bende suggests that I use the tls toolkit to generate keystore and truststore, but I just need to append the common server and the three servers in my cluster. I know that all of these servers's certificates are included in my keystore file already, so I believe that I just need to add truststore entries.

Is the best way to do that to use a plain text PEM file and edit it with a test editor?
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 proxy authorization error

Andy LoPresto-2
Yes, you can manually construct the keystore and truststore files if you know how. The toolkit is provided as a convenience for users who do not have the experience or a dedicated security team to generate those files. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 25, 2017, at 6:42 AM, bmichaud <[hidden email]> wrote:

The way I was setting up initial security was with the legacy
authorized-users.xml file. additionally, my certificate keystore.jks has
been and must be generated  by a company internal tool. I have done so, and
generated a truststore file using the java keytool, adding trusted entries
for various public keys we use for our company. This has worked fine in
1.0.0 and prior versions.

The instructions from Brian Bende suggests that I use the tls toolkit to
generate keystore and truststore, but I just need to append the common
server and the three servers in my cluster. I know that all of these
servers's certificates are included in my keystore file already, so I
believe that I just need to add truststore entries.

Is the best way to do that to use a plain text PEM file and edit it with a
test editor?



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-1-1-1-can-t-start-as-a-cluster-OverlappingFileLockException-tp14486p14518.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

bmichaud
This post was updated on .
In reply to this post by Andy LoPresto-2
Using the legacy security file and this in the authorizers.xml:
        <property name="Node Identity 1">CN=node1.company.com, O=Company Inc., L=Plymouth, ST=Minnesota, C=US</property>
        <property name="Node Identity 2">CN=node2.company.com, O=Company Inc., L=Plymouth, ST=Minnesota, C=US</property>
        <property name="Node Identity 3">CN=node3.company.com, O=Company Inc., L=Plymouth, ST=Minnesota, C=US</property>

I was able to generate those servers in the users.xml and each was given read and write access to the /proxy in the authorizations.xml.

Then, I still got the same error. From what you said, I needed to generate the same information for the common name server from my trust store. I did so like this

        <property name="Node Identity C">CN=common.company.com, O=Company Inc., L=Plymouth, ST=Minnesota, C=US</property>

This allowed me to log in to the UI and administrate. What impact would there by to having this extra node in the list of user authorizations which is not actually a node in the cluster? Is it a security hole?

I think I may have to create a separate certificate for each cluster to avoid this.
Reply | Threaded
Open this post in threaded view
|

Re: NiFi 1.1.1 can't start as a cluster OverlappingFileLockException

Andy LoPresto-2
The legacy authorized-users.xml is read from once during initial NiFi startup, translated to users.xml and authorizations.xml, and then ignored for the future. If you continue making changes to it, it will have no effect. 

If you are upgrading from a 0.x NiFi instance, you can convert your previously configured users and roles to the multi-tenant authorization model. In the authorizers.xml file, specify the location of your existing authorized-users.xml file in the “Legacy Authorized Users File” property.

After you have edited and saved the authorizers.xml file, restart NiFi. Users and roles from the authorized-users.xml file are converted and added as identities and policies in the users.xml and authorizations.xml files. Once the application starts, users who previously had a legacy Administrator role can access the UI and begin managing users, groups, and policies.


The authorizers.xml establishes the Authorizer implementation (FileAuthorizer, Ranger, etc.) which will perform policy decisions. The authorizations.xml file (confusing, I understand) is a file store of assigned permissions to the user/entity which the FileAuthorizer reads from. 

Can you provide a full copy of your authorizations.xml and users.xml files? You can make them GitHub secret gists and just reply directly to me if you want. 

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Jan 26, 2017, at 10:41 AM, bmichaud <[hidden email]> wrote:

Using the legacy security file and this in the authorizers.xml:
       <property name="Node Identity 1">CN=node1.company.com, O=Company
Inc., L=Plymouth, ST=Minnesota, C=US</property>
       <property name="Node Identity 2">CN=node2.company.com, O=Company
Inc., L=Plymouth, ST=Minnesota, C=US</property>
       <property name="Node Identity 3">CN=node3.company.com, O=Company
Inc., L=Plymouth, ST=Minnesota, C=US</property>

I was able to generate those servers in the users.xml and each was given
read and write access to the /proxy in the authorizations.xml.

However, I still get the same error. From what you said, I need to generate
the same information for the common name server from my trust store.
Correct?

If so, and if I use the same method to properly populate the users.xml and
authorizations.xml using the authorizers.xml property entries, then how do I
specify this other server? The same way? What name should I give it? Does it
matter? Can I do this?:

       <property name="Common Name Server">CN=common.company.com, O=Company
Inc., L=Plymouth, ST=Minnesota, C=US</property>





--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-1-1-1-can-t-start-as-a-cluster-OverlappingFileLockException-tp14486p14533.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment