Status history graphs over 24hrs (v1.3.0)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Status history graphs over 24hrs (v1.3.0)

Phil H
Hi there,

Is there a way to plot graphs longer than 24 hours? We are interested in
seeing things that happen over the course of a week or so

Thanks,
Phil
Reply | Threaded
Open this post in threaded view
|

Re: Status history graphs over 24hrs (v1.3.0)

Pierre Villard
Hi Phil,

That's not possible at the moment. Usually users would use the reporting
tasks to export data to external dashboarding tools in case they want
options to see statistics on a longer period of time.

Thanks,
Pierre

2018-08-03 1:59 GMT+02:00 Phil H <[hidden email]>:

> Hi there,
>
> Is there a way to plot graphs longer than 24 hours? We are interested in
> seeing things that happen over the course of a week or so
>
> Thanks,
> Phil
>
Reply | Threaded
Open this post in threaded view
|

Re: Status history graphs over 24hrs (v1.3.0)

Mark Payne
In reply to this post by Phil H
Hey Phil,

There are two properties that control this:

nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min

The snapshot.frequency tells NiFi how often to take a 'snapshot' of the stats on your canvas and add to the 'Status History'.
The buffer.size says how many of these to hold in memory.

So the default is to take a snapshot every minute and keep 1,440 minutes' worth (which is 24 hours). If you want to expand this
to a week, you can do so by making the buffer size larger or by making the snapshot frequency less often. Keep in mind, though,
that there are a lot of stats being collected so if you have a lot of processors, the amount of heap/memory that gets used up can
grow pretty quickly. I'd recommend changing the snapshot frequency to "5 mins" and the buffer size to 2016 to make 7 days' worth
of stats.

You'll also want to keep in mind that these stats are all stored in memory so if you restart NiFi, you will lose the stats that you've gathered.

Of note, I am currently working on NIFI-5466 [1], which should significantly reduce the amount of memory needed to store the stats, especially
for processors that are stopped.

Hope this helps!

Thanks
-Mark

[1] https://issues.apache.org/jira/browse/NIFI-5466

On Aug 2, 2018, at 7:59 PM, Phil H <[hidden email]<mailto:[hidden email]>> wrote:

Hi there,

Is there a way to plot graphs longer than 24 hours? We are interested in
seeing things that happen over the course of a week or so

Thanks,
Phil

Reply | Threaded
Open this post in threaded view
|

Re: Status history graphs over 24hrs (v1.3.0)

Joe Witt
are there any good local/embedded style time series type DBs we can
use for a non transient option?

On Fri, Aug 3, 2018 at 9:54 AM, Mark Payne <[hidden email]> wrote:

> Hey Phil,
>
> There are two properties that control this:
>
> nifi.components.status.repository.buffer.size=1440
> nifi.components.status.snapshot.frequency=1 min
>
> The snapshot.frequency tells NiFi how often to take a 'snapshot' of the stats on your canvas and add to the 'Status History'.
> The buffer.size says how many of these to hold in memory.
>
> So the default is to take a snapshot every minute and keep 1,440 minutes' worth (which is 24 hours). If you want to expand this
> to a week, you can do so by making the buffer size larger or by making the snapshot frequency less often. Keep in mind, though,
> that there are a lot of stats being collected so if you have a lot of processors, the amount of heap/memory that gets used up can
> grow pretty quickly. I'd recommend changing the snapshot frequency to "5 mins" and the buffer size to 2016 to make 7 days' worth
> of stats.
>
> You'll also want to keep in mind that these stats are all stored in memory so if you restart NiFi, you will lose the stats that you've gathered.
>
> Of note, I am currently working on NIFI-5466 [1], which should significantly reduce the amount of memory needed to store the stats, especially
> for processors that are stopped.
>
> Hope this helps!
>
> Thanks
> -Mark
>
> [1] https://issues.apache.org/jira/browse/NIFI-5466
>
> On Aug 2, 2018, at 7:59 PM, Phil H <[hidden email]<mailto:[hidden email]>> wrote:
>
> Hi there,
>
> Is there a way to plot graphs longer than 24 hours? We are interested in
> seeing things that happen over the course of a week or so
>
> Thanks,
> Phil
>
Reply | Threaded
Open this post in threaded view
|

Re: Status history graphs over 24hrs (v1.3.0)

Pierre Villard
I stand corrected. I completely forgot about these properties!


2018-08-03 16:03 GMT+02:00 Joe Witt <[hidden email]>:

> are there any good local/embedded style time series type DBs we can
> use for a non transient option?
>
> On Fri, Aug 3, 2018 at 9:54 AM, Mark Payne <[hidden email]> wrote:
> > Hey Phil,
> >
> > There are two properties that control this:
> >
> > nifi.components.status.repository.buffer.size=1440
> > nifi.components.status.snapshot.frequency=1 min
> >
> > The snapshot.frequency tells NiFi how often to take a 'snapshot' of the
> stats on your canvas and add to the 'Status History'.
> > The buffer.size says how many of these to hold in memory.
> >
> > So the default is to take a snapshot every minute and keep 1,440
> minutes' worth (which is 24 hours). If you want to expand this
> > to a week, you can do so by making the buffer size larger or by making
> the snapshot frequency less often. Keep in mind, though,
> > that there are a lot of stats being collected so if you have a lot of
> processors, the amount of heap/memory that gets used up can
> > grow pretty quickly. I'd recommend changing the snapshot frequency to "5
> mins" and the buffer size to 2016 to make 7 days' worth
> > of stats.
> >
> > You'll also want to keep in mind that these stats are all stored in
> memory so if you restart NiFi, you will lose the stats that you've gathered.
> >
> > Of note, I am currently working on NIFI-5466 [1], which should
> significantly reduce the amount of memory needed to store the stats,
> especially
> > for processors that are stopped.
> >
> > Hope this helps!
> >
> > Thanks
> > -Mark
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5466
> >
> > On Aug 2, 2018, at 7:59 PM, Phil H <[hidden email]<mailto:gi
> [hidden email]>> wrote:
> >
> > Hi there,
> >
> > Is there a way to plot graphs longer than 24 hours? We are interested in
> > seeing things that happen over the course of a week or so
> >
> > Thanks,
> > Phil
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Status history graphs over 24hrs (v1.3.0)

Matt Burgess-2
In reply to this post by Joe Witt
Netflix uses Atlas [1], the standalone JAR is only 8 KB [2] but it's
written in Scala, not sure what the overall footprint of the Scala
libraries and other dependencies are...

Regards,
Matt

[1] https://github.com/netflix/atlas/wiki
[2] https://mvnrepository.com/artifact/com.netflix.atlas/atlas-standalone/1.4.4

On Fri, Aug 3, 2018 at 10:03 AM Joe Witt <[hidden email]> wrote:

>
> are there any good local/embedded style time series type DBs we can
> use for a non transient option?
>
> On Fri, Aug 3, 2018 at 9:54 AM, Mark Payne <[hidden email]> wrote:
> > Hey Phil,
> >
> > There are two properties that control this:
> >
> > nifi.components.status.repository.buffer.size=1440
> > nifi.components.status.snapshot.frequency=1 min
> >
> > The snapshot.frequency tells NiFi how often to take a 'snapshot' of the stats on your canvas and add to the 'Status History'.
> > The buffer.size says how many of these to hold in memory.
> >
> > So the default is to take a snapshot every minute and keep 1,440 minutes' worth (which is 24 hours). If you want to expand this
> > to a week, you can do so by making the buffer size larger or by making the snapshot frequency less often. Keep in mind, though,
> > that there are a lot of stats being collected so if you have a lot of processors, the amount of heap/memory that gets used up can
> > grow pretty quickly. I'd recommend changing the snapshot frequency to "5 mins" and the buffer size to 2016 to make 7 days' worth
> > of stats.
> >
> > You'll also want to keep in mind that these stats are all stored in memory so if you restart NiFi, you will lose the stats that you've gathered.
> >
> > Of note, I am currently working on NIFI-5466 [1], which should significantly reduce the amount of memory needed to store the stats, especially
> > for processors that are stopped.
> >
> > Hope this helps!
> >
> > Thanks
> > -Mark
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-5466
> >
> > On Aug 2, 2018, at 7:59 PM, Phil H <[hidden email]<mailto:[hidden email]>> wrote:
> >
> > Hi there,
> >
> > Is there a way to plot graphs longer than 24 hours? We are interested in
> > seeing things that happen over the course of a week or so
> >
> > Thanks,
> > Phil
> >