Status of MetricsReportingTask?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Status of MetricsReportingTask?

Jon Logan
Hi All,

I was looking at utilizing the MetricsReportingTask service, but I was
wondering what the status of it is -- it seems to be lacking some features
that I thought it'd have (ex. reporting counters), and I'm not sure there's
an ability to extend the metrics being produced. Is this something that is
still being worked on, or is a legacy component? We are going to have to
write our own Service implementation, as the only supported one seems to be
Graphite, and wanted to make sure we're not going down a legacy path.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Status of MetricsReportingTask?

Koji Kawamura-2
Hi Jon,

About reporting counter values, there is an existing JIRA for the
improvement idea to expose counters to Reporting task context. That
requires NiFi framework level improvements. I'd suggest taking a look
on it, and resuming discussion there if needed.
https://issues.apache.org/jira/browse/NIFI-3293

Although MetricsReportingTask currently only supports Graphite, the
component is well designed for generic reporting use-cases. I don't
think it is a legacy component. The underlying dropwizard metrics
project seems active, too.
https://github.com/dropwizard/metrics

I'm interested in what service implementation you're going to write.
Is it going to use one of already available reporters?
https://metrics.dropwizard.io/3.1.0/manual/third-party/

Thanks,
Koji
On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <[hidden email]> wrote:

>
> Hi All,
>
> I was looking at utilizing the MetricsReportingTask service, but I was
> wondering what the status of it is -- it seems to be lacking some features
> that I thought it'd have (ex. reporting counters), and I'm not sure there's
> an ability to extend the metrics being produced. Is this something that is
> still being worked on, or is a legacy component? We are going to have to
> write our own Service implementation, as the only supported one seems to be
> Graphite, and wanted to make sure we're not going down a legacy path.
>
> Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Status of MetricsReportingTask?

Jon Logan
We were planning on having a reporter to pipe the metrics off to AWS
CloudWatch. The one issue being ran into -- and we've ran into it for other
components -- is the classloading. What is the recommended way of adding
alternate implementations of services (in this case MetricReporterService)?
We run into issues that I think stem from classloading isolation between
NARs, such that our custom implementations of services are not being
detected from the other NAR's context. We end up having to duplicate the
service into our custom NAR, but that hardly seems like a good idea.

Thanks!
Jon

On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <[hidden email]>
wrote:

> Hi Jon,
>
> About reporting counter values, there is an existing JIRA for the
> improvement idea to expose counters to Reporting task context. That
> requires NiFi framework level improvements. I'd suggest taking a look
> on it, and resuming discussion there if needed.
> https://issues.apache.org/jira/browse/NIFI-3293
>
> Although MetricsReportingTask currently only supports Graphite, the
> component is well designed for generic reporting use-cases. I don't
> think it is a legacy component. The underlying dropwizard metrics
> project seems active, too.
> https://github.com/dropwizard/metrics
>
> I'm interested in what service implementation you're going to write.
> Is it going to use one of already available reporters?
> https://metrics.dropwizard.io/3.1.0/manual/third-party/
>
> Thanks,
> Koji
> On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <[hidden email]> wrote:
> >
> > Hi All,
> >
> > I was looking at utilizing the MetricsReportingTask service, but I was
> > wondering what the status of it is -- it seems to be lacking some
> features
> > that I thought it'd have (ex. reporting counters), and I'm not sure
> there's
> > an ability to extend the metrics being produced. Is this something that
> is
> > still being worked on, or is a legacy component? We are going to have to
> > write our own Service implementation, as the only supported one seems to
> be
> > Graphite, and wanted to make sure we're not going down a legacy path.
> >
> > Thanks!
>
Reply | Threaded
Open this post in threaded view
|

Re: Status of MetricsReportingTask?

Bryan Bende
You should be able to make your own NAR with two modules like:

nifi-cloudwatch-metrics - contains your service implementation with a
provided dependency on nifi-metrics-reporter-service-api
nifi-cloudwatch-metrics-nar - packages your implementation and has a
NAR dependency on nifi-metrics-reporter-service-api-nar

Similar to this example:
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices

On Fri, Nov 2, 2018 at 3:24 PM Jon Logan <[hidden email]> wrote:

>
> We were planning on having a reporter to pipe the metrics off to AWS
> CloudWatch. The one issue being ran into -- and we've ran into it for other
> components -- is the classloading. What is the recommended way of adding
> alternate implementations of services (in this case MetricReporterService)?
> We run into issues that I think stem from classloading isolation between
> NARs, such that our custom implementations of services are not being
> detected from the other NAR's context. We end up having to duplicate the
> service into our custom NAR, but that hardly seems like a good idea.
>
> Thanks!
> Jon
>
> On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <[hidden email]>
> wrote:
>
> > Hi Jon,
> >
> > About reporting counter values, there is an existing JIRA for the
> > improvement idea to expose counters to Reporting task context. That
> > requires NiFi framework level improvements. I'd suggest taking a look
> > on it, and resuming discussion there if needed.
> > https://issues.apache.org/jira/browse/NIFI-3293
> >
> > Although MetricsReportingTask currently only supports Graphite, the
> > component is well designed for generic reporting use-cases. I don't
> > think it is a legacy component. The underlying dropwizard metrics
> > project seems active, too.
> > https://github.com/dropwizard/metrics
> >
> > I'm interested in what service implementation you're going to write.
> > Is it going to use one of already available reporters?
> > https://metrics.dropwizard.io/3.1.0/manual/third-party/
> >
> > Thanks,
> > Koji
> > On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <[hidden email]> wrote:
> > >
> > > Hi All,
> > >
> > > I was looking at utilizing the MetricsReportingTask service, but I was
> > > wondering what the status of it is -- it seems to be lacking some
> > features
> > > that I thought it'd have (ex. reporting counters), and I'm not sure
> > there's
> > > an ability to extend the metrics being produced. Is this something that
> > is
> > > still being worked on, or is a legacy component? We are going to have to
> > > write our own Service implementation, as the only supported one seems to
> > be
> > > Graphite, and wanted to make sure we're not going down a legacy path.
> > >
> > > Thanks!
> >
Reply | Threaded
Open this post in threaded view
|

Re: Status of MetricsReportingTask?

Omer Hadari
Hi Jon. Just wanted to say that I have developed this ReportingTask since
we had a similar issue to yours, only we had to report metrics to graphite.
We are currently using it in production.
I tried to implement as generic design as possible, so reporting to other
endpoints (e.g. cloudwatch) should be easy enough.
I performed a quick google search about dropwizard-metrics cloudwatch
reporter implementation. Found many small projects, such as this one:
https://github.com/azagniotov/codahale-aggregated-metrics-cloudwatch-reporter.
It might fit your needs or you can at least take a look at it.

When I designed this I only had system and process group metrics in mind. I
was trying to be generic with the endpoints, so I used dropwizard-metrics
but I admit I haven't thought about allowing custom tasks to report custom
metrics. It shouldn't be too hard to implement even now, so let me know if
you need this option and I will try getting to submitting a contribution.
Feel free to let me know if you have any questions regarding this reporting
task.

On Fri, Nov 2, 2018 at 9:35 PM Bryan Bende <[hidden email]> wrote:

> You should be able to make your own NAR with two modules like:
>
> nifi-cloudwatch-metrics - contains your service implementation with a
> provided dependency on nifi-metrics-reporter-service-api
> nifi-cloudwatch-metrics-nar - packages your implementation and has a
> NAR dependency on nifi-metrics-reporter-service-api-nar
>
> Similar to this example:
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>
> On Fri, Nov 2, 2018 at 3:24 PM Jon Logan <[hidden email]> wrote:
> >
> > We were planning on having a reporter to pipe the metrics off to AWS
> > CloudWatch. The one issue being ran into -- and we've ran into it for
> other
> > components -- is the classloading. What is the recommended way of adding
> > alternate implementations of services (in this case
> MetricReporterService)?
> > We run into issues that I think stem from classloading isolation between
> > NARs, such that our custom implementations of services are not being
> > detected from the other NAR's context. We end up having to duplicate the
> > service into our custom NAR, but that hardly seems like a good idea.
> >
> > Thanks!
> > Jon
> >
> > On Thu, Nov 1, 2018 at 10:27 PM Koji Kawamura <[hidden email]>
> > wrote:
> >
> > > Hi Jon,
> > >
> > > About reporting counter values, there is an existing JIRA for the
> > > improvement idea to expose counters to Reporting task context. That
> > > requires NiFi framework level improvements. I'd suggest taking a look
> > > on it, and resuming discussion there if needed.
> > > https://issues.apache.org/jira/browse/NIFI-3293
> > >
> > > Although MetricsReportingTask currently only supports Graphite, the
> > > component is well designed for generic reporting use-cases. I don't
> > > think it is a legacy component. The underlying dropwizard metrics
> > > project seems active, too.
> > > https://github.com/dropwizard/metrics
> > >
> > > I'm interested in what service implementation you're going to write.
> > > Is it going to use one of already available reporters?
> > > https://metrics.dropwizard.io/3.1.0/manual/third-party/
> > >
> > > Thanks,
> > > Koji
> > > On Thu, Nov 1, 2018 at 3:17 AM Jon Logan <[hidden email]> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I was looking at utilizing the MetricsReportingTask service, but I
> was
> > > > wondering what the status of it is -- it seems to be lacking some
> > > features
> > > > that I thought it'd have (ex. reporting counters), and I'm not sure
> > > there's
> > > > an ability to extend the metrics being produced. Is this something
> that
> > > is
> > > > still being worked on, or is a legacy component? We are going to
> have to
> > > > write our own Service implementation, as the only supported one
> seems to
> > > be
> > > > Graphite, and wanted to make sure we're not going down a legacy path.
> > > >
> > > > Thanks!
> > >
>