NiFi starts very slowly

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

NiFi starts very slowly

luby
Hi, All,

We created about 1600+ processor groups based on a NiFi template, but
after creating those groups, we found
tht NiFi service start very slowly, it takes about 40min ~ 1 hour to
start.

We are using NiFi 1.3.

Does anyone know how to speed it up?

Thanks

Boying


 
本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。

 
This email message may contain confidential and/or privileged information.
If you are not the intended recipient, please do not read, save, forward,
disclose or copy the contents of this email or open any file attached to
this email. We will be grateful if you could advise the sender immediately
by replying this email, and delete this email and any attachment or links
to this email completely and immediately from your computer system.



Reply | Threaded
Open this post in threaded view
|

Re: NiFi starts very slowly

Mohammed Nadeem
Hi Luby,

If you are using a cluster based setup then Nifi is unstable due to nodes
getting disconnected by the cluster cordinator. Cluster coordinator
disconnects nodes from the cluster when the nodes take a little longer to
respond ( beyond timeout) . The cluster coordinator timeout based on the
settings in the nifi.properties file.

We need to adjust the following properties

*nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
*nifi.cluster.node.read.timeout *=  5 sec ( by default)

By Increasing the timeout ,from 5 sec to higher value would probably solve a
problem


Thanks,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

答复: Re: NiFi starts very slowly

luby
Thanks for your reply.

We are not using a cluster, our NiFi 1.3 runs in the single node mode.



发件人:
"Mohammed Nadeem" <[hidden email]>
收件人:
[hidden email]
日期:
2017/12/28 20:11
主题:
Re: NiFi starts very slowly



Hi Luby,

If you are using a cluster based setup then Nifi is unstable due to nodes
getting disconnected by the cluster cordinator. Cluster coordinator
disconnects nodes from the cluster when the nodes take a little longer to
respond ( beyond timeout) . The cluster coordinator timeout based on the
settings in the nifi.properties file.

We need to adjust the following properties

*nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
*nifi.cluster.node.read.timeout *=  5 sec ( by default)

By Increasing the timeout ,from 5 sec to higher value would probably solve
a
problem


Thanks,
Nadeem



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/





 
本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。

 
This email message may contain confidential and/or privileged information.
If you are not the intended recipient, please do not read, save, forward,
disclose or copy the contents of this email or open any file attached to
this email. We will be grateful if you could advise the sender immediately
by replying this email, and delete this email and any attachment or links
to this email completely and immediately from your computer system.



Reply | Threaded
Open this post in threaded view
|

Re: 答复: Re: NiFi starts very slowly

Joe Witt
hello

the total number of processors is probably a factor.  how many are there?
can you please list how many processors of each type there are?  Any custom
processors?

creating 1600 disjoint flows is probably going against the grain of nifi.
cant the flows be designed to operate as a directed graph amd leverage
attributes?

keep in mind to start each processor we call validate.  this could be the
cause of the slowness.

help us collect the above info and we can give pointers.  if you could
share your flow.xml.gz that would be great.

thanks
joe

On Dec 28, 2017 8:13 PM, <[hidden email]> wrote:

> Thanks for your reply.
>
> We are not using a cluster, our NiFi 1.3 runs in the single node mode.
>
>
>
> 发件人:
> "Mohammed Nadeem" <[hidden email]>
> 收件人:
> [hidden email]
> 日期:
> 2017/12/28 20:11
> 主题:
> Re: NiFi starts very slowly
>
>
>
> Hi Luby,
>
> If you are using a cluster based setup then Nifi is unstable due to nodes
> getting disconnected by the cluster cordinator. Cluster coordinator
> disconnects nodes from the cluster when the nodes take a little longer to
> respond ( beyond timeout) . The cluster coordinator timeout based on the
> settings in the nifi.properties file.
>
> We need to adjust the following properties
>
> *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> *nifi.cluster.node.read.timeout *=  5 sec ( by default)
>
> By Increasing the timeout ,from 5 sec to higher value would probably solve
> a
> problem
>
>
> Thanks,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
> 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the intended recipient, please do not read, save, forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender immediately
> by replying this email, and delete this email and any attachment or links
> to this email completely and immediately from your computer system.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

答复: Re: 答复: Re: NiFi starts very slowly

luby
Thanks for your help.

Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
work.

There are about 10 processors in each group, so there will be 16000
processors in total.

I understand that the processor should be validate before it starts, but
could this validation be defered until the processor (or the group it
belongs to) is to start for the first time
instead of doing this when NiFi service starts up?



发件人:
Joe Witt <[hidden email]>
收件人:
[hidden email]
日期:
2017/12/29 09:20
主题:
Re: 答复: Re: NiFi starts very slowly



hello

the total number of processors is probably a factor.  how many are there?
can you please list how many processors of each type there are?  Any
custom
processors?

creating 1600 disjoint flows is probably going against the grain of nifi.
cant the flows be designed to operate as a directed graph amd leverage
attributes?

keep in mind to start each processor we call validate.  this could be the
cause of the slowness.

help us collect the above info and we can give pointers.  if you could
share your flow.xml.gz that would be great.

thanks
joe

On Dec 28, 2017 8:13 PM, <[hidden email]> wrote:

> Thanks for your reply.
>
> We are not using a cluster, our NiFi 1.3 runs in the single node mode.
>
>
>
> 发件人:
> "Mohammed Nadeem" <[hidden email]>
> 收件人:
> [hidden email]
> 日期:
> 2017/12/28 20:11
> 主题:
> Re: NiFi starts very slowly
>
>
>
> Hi Luby,
>
> If you are using a cluster based setup then Nifi is unstable due to
nodes
> getting disconnected by the cluster cordinator. Cluster coordinator
> disconnects nodes from the cluster when the nodes take a little longer
to
> respond ( beyond timeout) . The cluster coordinator timeout based on the
> settings in the nifi.properties file.
>
> We need to adjust the following properties
>
> *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> *nifi.cluster.node.read.timeout *=  5 sec ( by default)
>
> By Increasing the timeout ,from 5 sec to higher value would probably
solve

> a
> problem
>
>
> Thanks,
> Nadeem
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对

> 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发

> 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
>
>
> This email message may contain confidential and/or privileged
information.
> If you are not the intended recipient, please do not read, save,
forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender
immediately
> by replying this email, and delete this email and any attachment or
links
> to this email completely and immediately from your computer system.
>
>
>
>





 
本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。

 
This email message may contain confidential and/or privileged information.
If you are not the intended recipient, please do not read, save, forward,
disclose or copy the contents of this email or open any file attached to
this email. We will be grateful if you could advise the sender immediately
by replying this email, and delete this email and any attachment or links
to this email completely and immediately from your computer system.



Reply | Threaded
Open this post in threaded view
|

Re: Re: 答复: Re: NiFi starts very slowly

Joe Witt
Ok...validation may not be the issue.

During the startup phase at 15 sec intervals please do a series of thread
dumps.  Please send those dump logs with timing info.

Having 1600 groups that aren’t taking advantage of nifis ability to support
directed graphs and expression language and flow file attributes will
provide the least performant and highest overhead processing.  Is it
required to be this way in your current usage?

Thanks

On Thu, Dec 28, 2017 at 8:54 PM <[hidden email]> wrote:

> Thanks for your help.
>
> Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
> work.
>
> There are about 10 processors in each group, so there will be 16000
> processors in total.
>
> I understand that the processor should be validate before it starts, but
> could this validation be defered until the processor (or the group it
> belongs to) is to start for the first time
> instead of doing this when NiFi service starts up?
>
>
>
> 发件人:
> Joe Witt <[hidden email]>
> 收件人:
> [hidden email]
> 日期:
> 2017/12/29 09:20
> 主题:
> Re: 答复: Re: NiFi starts very slowly
>
>
>
> hello
>
> the total number of processors is probably a factor.  how many are there?
> can you please list how many processors of each type there are?  Any
> custom
> processors?
>
> creating 1600 disjoint flows is probably going against the grain of nifi.
> cant the flows be designed to operate as a directed graph amd leverage
> attributes?
>
> keep in mind to start each processor we call validate.  this could be the
> cause of the slowness.
>
> help us collect the above info and we can give pointers.  if you could
> share your flow.xml.gz that would be great.
>
> thanks
> joe
>
> On Dec 28, 2017 8:13 PM, <[hidden email]> wrote:
>
> > Thanks for your reply.
> >
> > We are not using a cluster, our NiFi 1.3 runs in the single node mode.
> >
> >
> >
> > 发件人:
> > "Mohammed Nadeem" <[hidden email]>
> > 收件人:
> > [hidden email]
> > 日期:
> > 2017/12/28 20:11
> > 主题:
> > Re: NiFi starts very slowly
> >
> >
> >
> > Hi Luby,
> >
> > If you are using a cluster based setup then Nifi is unstable due to
> nodes
> > getting disconnected by the cluster cordinator. Cluster coordinator
> > disconnects nodes from the cluster when the nodes take a little longer
> to
> > respond ( beyond timeout) . The cluster coordinator timeout based on the
> > settings in the nifi.properties file.
> >
> > We need to adjust the following properties
> >
> > *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> > *nifi.cluster.node.read.timeout *=  5 sec ( by default)
> >
> > By Increasing the timeout ,from 5 sec to higher value would probably
> solve
> > a
> > problem
> >
> >
> > Thanks,
> > Nadeem
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对
> 外
> > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发
> 件
> > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save,
> forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or
> links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
> 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the intended recipient, please do not read, save, forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender immediately
> by replying this email, and delete this email and any attachment or links
> to this email completely and immediately from your computer system.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: 答复: Re: 答复: Re: NiFi starts very slowly

Mike Thomsen
In reply to this post by luby
How many of these 1600 processor groups represent distinct functionality
that cannot be genericized?

Thanks,

Mike

On Thu, Dec 28, 2017 at 8:54 PM, <[hidden email]> wrote:

> Thanks for your help.
>
> Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
> work.
>
> There are about 10 processors in each group, so there will be 16000
> processors in total.
>
> I understand that the processor should be validate before it starts, but
> could this validation be defered until the processor (or the group it
> belongs to) is to start for the first time
> instead of doing this when NiFi service starts up?
>
>
>
> 发件人:
> Joe Witt <[hidden email]>
> 收件人:
> [hidden email]
> 日期:
> 2017/12/29 09:20
> 主题:
> Re: 答复: Re: NiFi starts very slowly
>
>
>
> hello
>
> the total number of processors is probably a factor.  how many are there?
> can you please list how many processors of each type there are?  Any
> custom
> processors?
>
> creating 1600 disjoint flows is probably going against the grain of nifi.
> cant the flows be designed to operate as a directed graph amd leverage
> attributes?
>
> keep in mind to start each processor we call validate.  this could be the
> cause of the slowness.
>
> help us collect the above info and we can give pointers.  if you could
> share your flow.xml.gz that would be great.
>
> thanks
> joe
>
> On Dec 28, 2017 8:13 PM, <[hidden email]> wrote:
>
> > Thanks for your reply.
> >
> > We are not using a cluster, our NiFi 1.3 runs in the single node mode.
> >
> >
> >
> > 发件人:
> > "Mohammed Nadeem" <[hidden email]>
> > 收件人:
> > [hidden email]
> > 日期:
> > 2017/12/28 20:11
> > 主题:
> > Re: NiFi starts very slowly
> >
> >
> >
> > Hi Luby,
> >
> > If you are using a cluster based setup then Nifi is unstable due to
> nodes
> > getting disconnected by the cluster cordinator. Cluster coordinator
> > disconnects nodes from the cluster when the nodes take a little longer
> to
> > respond ( beyond timeout) . The cluster coordinator timeout based on the
> > settings in the nifi.properties file.
> >
> > We need to adjust the following properties
> >
> > *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> > *nifi.cluster.node.read.timeout *=  5 sec ( by default)
> >
> > By Increasing the timeout ,from 5 sec to higher value would probably
> solve
> > a
> > problem
> >
> >
> > Thanks,
> > Nadeem
> >
> >
> >
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对
> 外
> > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发
> 件
> > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save,
> forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or
> links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>
>
>
>
>
>
> 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
> 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
> 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
>
>
> This email message may contain confidential and/or privileged information.
> If you are not the intended recipient, please do not read, save, forward,
> disclose or copy the contents of this email or open any file attached to
> this email. We will be grateful if you could advise the sender immediately
> by replying this email, and delete this email and any attachment or links
> to this email completely and immediately from your computer system.
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: 答复: Re: 答复: Re: NiFi starts very slowly

Michael Moser
I was thinking that with 16000 processors, you would need a large amount of
heap space allocated to the JVM.  A small heap would slow NiFi's ability to
do its work, because the JVM garbage collector would be consuming more CPU
time than what is healthy.  A 64 GB heap size, allocated in the NiFi
boostrap.conf, can work.  Hopefully the system you are running this NiFi
has adequate CPU and RAM resources.

-- Mike


On Fri, Dec 29, 2017 at 9:28 AM, Mike Thomsen <[hidden email]>
wrote:

> How many of these 1600 processor groups represent distinct functionality
> that cannot be genericized?
>
> Thanks,
>
> Mike
>
> On Thu, Dec 28, 2017 at 8:54 PM, <[hidden email]> wrote:
>
> > Thanks for your help.
> >
> > Actually we are using Kylo (https://kylo.io/) which uses NiFi to do the
> > work.
> >
> > There are about 10 processors in each group, so there will be 16000
> > processors in total.
> >
> > I understand that the processor should be validate before it starts, but
> > could this validation be defered until the processor (or the group it
> > belongs to) is to start for the first time
> > instead of doing this when NiFi service starts up?
> >
> >
> >
> > 发件人:
> > Joe Witt <[hidden email]>
> > 收件人:
> > [hidden email]
> > 日期:
> > 2017/12/29 09:20
> > 主题:
> > Re: 答复: Re: NiFi starts very slowly
> >
> >
> >
> > hello
> >
> > the total number of processors is probably a factor.  how many are there?
> > can you please list how many processors of each type there are?  Any
> > custom
> > processors?
> >
> > creating 1600 disjoint flows is probably going against the grain of nifi.
> > cant the flows be designed to operate as a directed graph amd leverage
> > attributes?
> >
> > keep in mind to start each processor we call validate.  this could be the
> > cause of the slowness.
> >
> > help us collect the above info and we can give pointers.  if you could
> > share your flow.xml.gz that would be great.
> >
> > thanks
> > joe
> >
> > On Dec 28, 2017 8:13 PM, <[hidden email]> wrote:
> >
> > > Thanks for your reply.
> > >
> > > We are not using a cluster, our NiFi 1.3 runs in the single node mode.
> > >
> > >
> > >
> > > 发件人:
> > > "Mohammed Nadeem" <[hidden email]>
> > > 收件人:
> > > [hidden email]
> > > 日期:
> > > 2017/12/28 20:11
> > > 主题:
> > > Re: NiFi starts very slowly
> > >
> > >
> > >
> > > Hi Luby,
> > >
> > > If you are using a cluster based setup then Nifi is unstable due to
> > nodes
> > > getting disconnected by the cluster cordinator. Cluster coordinator
> > > disconnects nodes from the cluster when the nodes take a little longer
> > to
> > > respond ( beyond timeout) . The cluster coordinator timeout based on
> the
> > > settings in the nifi.properties file.
> > >
> > > We need to adjust the following properties
> > >
> > > *nifi.cluster.node.connection.timeout * = 5 sec  ( by default )
> > > *nifi.cluster.node.read.timeout *=  5 sec ( by default)
> > >
> > > By Increasing the timeout ,from 5 sec to higher value would probably
> > solve
> > > a
> > > problem
> > >
> > >
> > > Thanks,
> > > Nadeem
> > >
> > >
> > >
> > > --
> > > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
> > >
> > >
> > >
> > >
> > >
> > >
> > > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对
> > 外
> > > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发
> > 件
> > > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
> > >
> > >
> > > This email message may contain confidential and/or privileged
> > information.
> > > If you are not the intended recipient, please do not read, save,
> > forward,
> > > disclose or copy the contents of this email or open any file attached
> to
> > > this email. We will be grateful if you could advise the sender
> > immediately
> > > by replying this email, and delete this email and any attachment or
> > links
> > > to this email completely and immediately from your computer system.
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> >
> > 本邮件内容包含保密信息。如阁下并非拟发送的收件人,请您不要阅读、保存、对外
> > 披露或复制本邮件的任何内容,或者打开本邮件的任何附件。请即回复邮件告知发件
> > 人,并立刻将该邮件及其附件从您的电脑系统中全部删除,不胜感激。
> >
> >
> > This email message may contain confidential and/or privileged
> information.
> > If you are not the intended recipient, please do not read, save, forward,
> > disclose or copy the contents of this email or open any file attached to
> > this email. We will be grateful if you could advise the sender
> immediately
> > by replying this email, and delete this email and any attachment or links
> > to this email completely and immediately from your computer system.
> >
> >
> >
> >
>