Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

ashwin.konale@gmail.com
Hi,

I am running 3 node nifi cluster (embedded zookeeper) with following flow.
It pulls the data from mysql binlogs using CaptureChangeMysql and sending
it to HDFS.

CDC[Primary node only] -> outputPort -> RPG

inputPort -> MergeContent -> PutHdfs

I am using RPG to distribute load among other nodes in cluster. CDC reads
large amount of data (~10 Mbps). But as soon as I start CDC processor, nifi
ui hangs, and I get "javax.ws.rs.ProcessingException:
java.net.SocketTimeoutException: Read timed out" , I have gone through
various threads and tweaked certain properties like,

nifi.cluster.node.connection.timeout = 30 seconds
nifi.cluster.node.read.timeout = 30 sec
nifi.cluster.node.protocol.threads = 20
nifi.web.jetty.threads = 400

Nothing unusual in the logs except data flow seems to be continuing. I am
not able to figure out what is going wrong. Can someone please help me with
it. Let me know what more information I need to provide in order to debug
this.

Stacktrace:

2018-09-25 12:00:49,888 ERROR [LearnerHandler-/10.10.10.10:31955]
o.a.z.server.quorum.LearnerHandler Unexpected exception causing shutdown
while sock still open

java.net.SocketTimeoutException: Read timed out

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)

at java.net.SocketInputStream.read(SocketInputStream.java:171)

at java.net.SocketInputStream.read(SocketInputStream.java:141)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)

at java.io.BufferedInputStream.read(BufferedInputStream.java:265)

at java.io.DataInputStream.readInt(DataInputStream.java:387)

at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)

at
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)

at
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)

at
org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:546)

2018-09-25 12:00:51,901 INFO [Process Cluster Protocol Request-7]
o.a.n.c.p.impl.SocketProtocolListener Finished processing request
e181bdbd-00c7-4f3b-b69a-34ff7c1c9cee (type=HEARTBEAT, length=3565 bytes)
from nifiapp-1002:8080 in 1 millis


Any help would be much appreciated.
Ashwin
Reply | Threaded
Open this post in threaded view
|

Re: Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

Mark Payne
Hi Ashwin,

The embedded ZooKeeper is provided as a convenience so that you can easily test running things in a cluster
on a laptop, etc. However, it struggles when your nodes are handling any kind of significant data rate. It is always
recommended that an external ZooKeeper be used for any sort of production use.

Thanks
-Mark

> On Sep 25, 2018, at 6:03 AM, ashwin konale <[hidden email]> wrote:
>
> Hi,
>
> I am running 3 node nifi cluster (embedded zookeeper) with following flow.
> It pulls the data from mysql binlogs using CaptureChangeMysql and sending
> it to HDFS.
>
> CDC[Primary node only] -> outputPort -> RPG
>
> inputPort -> MergeContent -> PutHdfs
>
> I am using RPG to distribute load among other nodes in cluster. CDC reads
> large amount of data (~10 Mbps). But as soon as I start CDC processor, nifi
> ui hangs, and I get "javax.ws.rs.ProcessingException:
> java.net.SocketTimeoutException: Read timed out" , I have gone through
> various threads and tweaked certain properties like,
>
> nifi.cluster.node.connection.timeout = 30 seconds
> nifi.cluster.node.read.timeout = 30 sec
> nifi.cluster.node.protocol.threads = 20
> nifi.web.jetty.threads = 400
>
> Nothing unusual in the logs except data flow seems to be continuing. I am
> not able to figure out what is going wrong. Can someone please help me with
> it. Let me know what more information I need to provide in order to debug
> this.
>
> Stacktrace:
>
> 2018-09-25 12:00:49,888 ERROR [LearnerHandler-/10.10.10.10:31955]
> o.a.z.server.quorum.LearnerHandler Unexpected exception causing shutdown
> while sock still open
>
> java.net.SocketTimeoutException: Read timed out
>
> at java.net.SocketInputStream.socketRead0(Native Method)
>
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
>
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
>
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
>
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
>
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
>
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
>
> at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
>
> at
> org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
>
> at
> org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
>
> at
> org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:546)
>
> 2018-09-25 12:00:51,901 INFO [Process Cluster Protocol Request-7]
> o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> e181bdbd-00c7-4f3b-b69a-34ff7c1c9cee (type=HEARTBEAT, length=3565 bytes)
> from nifiapp-1002:8080 in 1 millis
>
>
> Any help would be much appreciated.
> Ashwin

Reply | Threaded
Open this post in threaded view
|

Re: Unable to operate nifi ui due to "java.net.SocketTimeoutException: Read timed out"

ashwin.konale@gmail.com


On 2018/09/25 13:13:12, Mark Payne <[hidden email]> wrote:

> Hi Ashwin,
>
> The embedded ZooKeeper is provided as a convenience so that you can easily test running things in a cluster
> on a laptop, etc. However, it struggles when your nodes are handling any kind of significant data rate. It is always
> recommended that an external ZooKeeper be used for any sort of production use.
>
> Thanks
> -Mark
>
> > On Sep 25, 2018, at 6:03 AM, ashwin konale <[hidden email]> wrote:
> >
> > Hi,
> >
> > I am running 3 node nifi cluster (embedded zookeeper) with following flow.
> > It pulls the data from mysql binlogs using CaptureChangeMysql and sending
> > it to HDFS.
> >
> > CDC[Primary node only] -> outputPort -> RPG
> >
> > inputPort -> MergeContent -> PutHdfs
> >
> > I am using RPG to distribute load among other nodes in cluster. CDC reads
> > large amount of data (~10 Mbps). But as soon as I start CDC processor, nifi
> > ui hangs, and I get "javax.ws.rs.ProcessingException:
> > java.net.SocketTimeoutException: Read timed out" , I have gone through
> > various threads and tweaked certain properties like,
> >
> > nifi.cluster.node.connection.timeout = 30 seconds
> > nifi.cluster.node.read.timeout = 30 sec
> > nifi.cluster.node.protocol.threads = 20
> > nifi.web.jetty.threads = 400
> >
> > Nothing unusual in the logs except data flow seems to be continuing. I am
> > not able to figure out what is going wrong. Can someone please help me with
> > it. Let me know what more information I need to provide in order to debug
> > this.
> >
> > Stacktrace:
> >
> > 2018-09-25 12:00:49,888 ERROR [LearnerHandler-/10.10.10.10:31955]
> > o.a.z.server.quorum.LearnerHandler Unexpected exception causing shutdown
> > while sock still open
> >
> > java.net.SocketTimeoutException: Read timed out
> >
> > at java.net.SocketInputStream.socketRead0(Native Method)
> >
> > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> >
> > at java.net.SocketInputStream.read(SocketInputStream.java:171)
> >
> > at java.net.SocketInputStream.read(SocketInputStream.java:141)
> >
> > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> >
> > at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> >
> > at java.io.DataInputStream.readInt(DataInputStream.java:387)
> >
> > at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
> >
> > at
> > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83)
> >
> > at
> > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:103)
> >
> > at
> > org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:546)
> >
> > 2018-09-25 12:00:51,901 INFO [Process Cluster Protocol Request-7]
> > o.a.n.c.p.impl.SocketProtocolListener Finished processing request
> > e181bdbd-00c7-4f3b-b69a-34ff7c1c9cee (type=HEARTBEAT, length=3565 bytes)
> > from nifiapp-1002:8080 in 1 millis
> >
> >
> > Any help would be much appreciated.
> > Ashwin
>
> Thanks Mark. Will try that