Monitoring API? JMX?

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Monitoring API? JMX?

Otis Gospodnetic
Hi,

Nifi looks mightily impressive!
I see Nifi has built-in monitoring and metrics.... but is there an API that
would allow something like SPM to add support for Nifi monitoring?  Or JMX?

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
Otis,

Glad you are liking it.

We do provide support for both push and pull based metrics
reporting/gathering.

For NiFi being able to push out metrics:
We do have an API for what we call "Reporting Tasks".  For example we
have a Ganglia reporter today.

For a client being able to pull metrics from NiFI:
The REST API is the way to go.

Do either of these sound like a path that can work for you?  Let us
know if you'd like more details.

Thanks
Joe

On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
<[hidden email]> wrote:

> Hi,
>
> Nifi looks mightily impressive!
> I see Nifi has built-in monitoring and metrics.... but is there an API that
> would allow something like SPM to add support for Nifi monitoring?  Or JMX?
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Ashish
That's cool Joe !

The Ganglia Reporting Task uses Metrics package, so it should be
possible to make it generic to support other reporting methods
provided by Metrics, correct?



On Wed, Feb 18, 2015 at 10:45 AM, Joe Witt <[hidden email]> wrote:

> Otis,
>
> Glad you are liking it.
>
> We do provide support for both push and pull based metrics
> reporting/gathering.
>
> For NiFi being able to push out metrics:
> We do have an API for what we call "Reporting Tasks".  For example we
> have a Ganglia reporter today.
>
> For a client being able to pull metrics from NiFI:
> The REST API is the way to go.
>
> Do either of these sound like a path that can work for you?  Let us
> know if you'd like more details.
>
> Thanks
> Joe
>
> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> <[hidden email]> wrote:
>> Hi,
>>
>> Nifi looks mightily impressive!
>> I see Nifi has built-in monitoring and metrics.... but is there an API that
>> would allow something like SPM to add support for Nifi monitoring?  Or JMX?
>>
>> Thanks,
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Otis Gospodnetic
In reply to this post by Joe Witt
Hi Joe,

On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:

> Otis,
>
> Glad you are liking it.
>
> We do provide support for both push and pull based metrics
> reporting/gathering.
>
> For NiFi being able to push out metrics:
> We do have an API for what we call "Reporting Tasks".  For example we
> have a Ganglia reporter today.
>

So we are talking about the Yammer/Coda Hale's Metrics being used here?
If so, are the metrics also available via JMX, too?

For a client being able to pull metrics from NiFI:
> The REST API is the way to go.
>

This sounds good.  Are this API and the metrics it exposes documented
anywhere?
I looked in the guides, but couldn't find anything.


> Do either of these sound like a path that can work for you?  Let us
> know if you'd like more details.
>

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/




>
> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> <[hidden email]> wrote:
> > Hi,
> >
> > Nifi looks mightily impressive!
> > I see Nifi has built-in monitoring and metrics.... but is there an API
> that
> > would allow something like SPM to add support for Nifi monitoring?  Or
> JMX?
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
In reply to this post by Ashish
" it should be possible to make it generic to support other reporting
methods provided by Metrics, correct?"

  We think so.  We're using a slightly older version of it because it
appears that upon close inspection newer versions might have issues
with ASF compatible licensing.  But in any event if the thing you want
to report metrics to has a client library/API/interface then the
ReportingTask is likely a good way to go about feeding it (if you're
good with a push model).  We've also done some interesting things
using our Reporting Task API with Graphite in the past.

I think our general philosophy here, as is the case with much of the
app, is that we should provide a solid dataflow user experience
built-in to NiFI and then we should also expose this information
outward to apps purpose built to address those common cross cutting
enterprise items that go beyond our sphere.

Are there other metrics/monitoring repository apps you think we should
take a look at?

Thanks
Joe



On Wed, Feb 18, 2015 at 12:38 AM, Ashish <[hidden email]> wrote:

> That's cool Joe !
>
> The Ganglia Reporting Task uses Metrics package, so it should be
> possible to make it generic to support other reporting methods
> provided by Metrics, correct?
>
>
>
> On Wed, Feb 18, 2015 at 10:45 AM, Joe Witt <[hidden email]> wrote:
>> Otis,
>>
>> Glad you are liking it.
>>
>> We do provide support for both push and pull based metrics
>> reporting/gathering.
>>
>> For NiFi being able to push out metrics:
>> We do have an API for what we call "Reporting Tasks".  For example we
>> have a Ganglia reporter today.
>>
>> For a client being able to pull metrics from NiFI:
>> The REST API is the way to go.
>>
>> Do either of these sound like a path that can work for you?  Let us
>> know if you'd like more details.
>>
>> Thanks
>> Joe
>>
>> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>> <[hidden email]> wrote:
>>> Hi,
>>>
>>> Nifi looks mightily impressive!
>>> I see Nifi has built-in monitoring and metrics.... but is there an API that
>>> would allow something like SPM to add support for Nifi monitoring?  Or JMX?
>>>
>>> Thanks,
>>> Otis
>>> --
>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
In reply to this post by Otis Gospodnetic
Otis,

We're quite light on documentation thus far for the Reporting Task
API.  But you can see a bit about it here:

  http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
  http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task

The javadocs are published to Maven as well so that will do nicely to
give a sense of what we expose from the Internals of NiFi to an
implementation of a reporting task.

But feel free to ask detailed questions and as many as you want.  That
will give us good information on the types of things we need to cover
and in how much detail.

Thanks
Joe

On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
<[hidden email]> wrote:

> Hi Joe,
>
> On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
>
>> Otis,
>>
>> Glad you are liking it.
>>
>> We do provide support for both push and pull based metrics
>> reporting/gathering.
>>
>> For NiFi being able to push out metrics:
>> We do have an API for what we call "Reporting Tasks".  For example we
>> have a Ganglia reporter today.
>>
>
> So we are talking about the Yammer/Coda Hale's Metrics being used here?
> If so, are the metrics also available via JMX, too?
>
> For a client being able to pull metrics from NiFI:
>> The REST API is the way to go.
>>
>
> This sounds good.  Are this API and the metrics it exposes documented
> anywhere?
> I looked in the guides, but couldn't find anything.
>
>
>> Do either of these sound like a path that can work for you?  Let us
>> know if you'd like more details.
>>
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
>
>>
>> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>> <[hidden email]> wrote:
>> > Hi,
>> >
>> > Nifi looks mightily impressive!
>> > I see Nifi has built-in monitoring and metrics.... but is there an API
>> that
>> > would allow something like SPM to add support for Nifi monitoring?  Or
>> JMX?
>> >
>> > Thanks,
>> > Otis
>> > --
>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > Solr & Elasticsearch Support * http://sematext.com/
>>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Ashish
Joe,

Latest version is ASL 2.0, pls check https://github.com/dropwizard/metrics

I am still in initial stages of hacking Nifi, shall send a PR with
upgraded version, if it works for you?

Web UI would be a good place to see these API's in action, I suppose.

On Wed, Feb 18, 2015 at 11:20 AM, Joe Witt <[hidden email]> wrote:

> Otis,
>
> We're quite light on documentation thus far for the Reporting Task
> API.  But you can see a bit about it here:
>
>   http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>   http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>
> The javadocs are published to Maven as well so that will do nicely to
> give a sense of what we expose from the Internals of NiFi to an
> implementation of a reporting task.
>
> But feel free to ask detailed questions and as many as you want.  That
> will give us good information on the types of things we need to cover
> and in how much detail.
>
> Thanks
> Joe
>
> On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> <[hidden email]> wrote:
>> Hi Joe,
>>
>> On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
>>
>>> Otis,
>>>
>>> Glad you are liking it.
>>>
>>> We do provide support for both push and pull based metrics
>>> reporting/gathering.
>>>
>>> For NiFi being able to push out metrics:
>>> We do have an API for what we call "Reporting Tasks".  For example we
>>> have a Ganglia reporter today.
>>>
>>
>> So we are talking about the Yammer/Coda Hale's Metrics being used here?
>> If so, are the metrics also available via JMX, too?
>>
>> For a client being able to pull metrics from NiFI:
>>> The REST API is the way to go.
>>>
>>
>> This sounds good.  Are this API and the metrics it exposes documented
>> anywhere?
>> I looked in the guides, but couldn't find anything.
>>
>>
>>> Do either of these sound like a path that can work for you?  Let us
>>> know if you'd like more details.
>>>
>>
>> Thanks,
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>>
>>
>>>
>>> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>>> <[hidden email]> wrote:
>>> > Hi,
>>> >
>>> > Nifi looks mightily impressive!
>>> > I see Nifi has built-in monitoring and metrics.... but is there an API
>>> that
>>> > would allow something like SPM to add support for Nifi monitoring?  Or
>>> JMX?
>>> >
>>> > Thanks,
>>> > Otis
>>> > --
>>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> > Solr & Elasticsearch Support * http://sematext.com/
>>>



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Otis Gospodnetic
In reply to this post by Joe Witt
Hi Joe,

Kafka devs describe their MBeans here - https://kafka.apache.org/08/ops.html
- section 6.5, which is good for people who want to understand what each
metric means.
It sounds like the reporting task stuff is all about the push.  I think
pull may be better for us, so if you can document the end point (I assume
we are talking about an HTTP/REST API?  That returns various metrics in
JSON?), that would be very very helpful (and not just to us).

Thanks,
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]> wrote:

> Otis,
>
> We're quite light on documentation thus far for the Reporting Task
> API.  But you can see a bit about it here:
>
>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>
> The javadocs are published to Maven as well so that will do nicely to
> give a sense of what we expose from the Internals of NiFi to an
> implementation of a reporting task.
>
> But feel free to ask detailed questions and as many as you want.  That
> will give us good information on the types of things we need to cover
> and in how much detail.
>
> Thanks
> Joe
>
> On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> <[hidden email]> wrote:
> > Hi Joe,
> >
> > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
> >
> >> Otis,
> >>
> >> Glad you are liking it.
> >>
> >> We do provide support for both push and pull based metrics
> >> reporting/gathering.
> >>
> >> For NiFi being able to push out metrics:
> >> We do have an API for what we call "Reporting Tasks".  For example we
> >> have a Ganglia reporter today.
> >>
> >
> > So we are talking about the Yammer/Coda Hale's Metrics being used here?
> > If so, are the metrics also available via JMX, too?
> >
> > For a client being able to pull metrics from NiFI:
> >> The REST API is the way to go.
> >>
> >
> > This sounds good.  Are this API and the metrics it exposes documented
> > anywhere?
> > I looked in the guides, but couldn't find anything.
> >
> >
> >> Do either of these sound like a path that can work for you?  Let us
> >> know if you'd like more details.
> >>
> >
> > Thanks,
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> >
> >
> >>
> >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> >> <[hidden email]> wrote:
> >> > Hi,
> >> >
> >> > Nifi looks mightily impressive!
> >> > I see Nifi has built-in monitoring and metrics.... but is there an API
> >> that
> >> > would allow something like SPM to add support for Nifi monitoring?  Or
> >> JMX?
> >> >
> >> > Thanks,
> >> > Otis
> >> > --
> >> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> > Solr & Elasticsearch Support * http://sematext.com/
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Matt Gilman
Otis,

We've disabled the automatic generation of the REST API documentation do to
license concerns of the generated output. If you've built NiFi from source,
you could enable it using the maven profile 'docs'. We have an open ticket
to address this [1]. If the application is built with the REST API
documentation it can be accessed in the Help section within the NiFi
instance your running. After clicking the help link, the Developer section
is located on the bottom of the side navigation.

I realize this isn't great. The next best thing I could suggest is opening
your browser's Developer Tools to watch the requests in the Summary page.
This is where we show metrics for the data flow and various system
diagnostics.

[1] https://issues.apache.org/jira/browse/NIFI-292

On Wed, Feb 18, 2015 at 1:25 AM, Otis Gospodnetic <
[hidden email]> wrote:

> Hi Joe,
>
> Kafka devs describe their MBeans here -
> https://kafka.apache.org/08/ops.html
> - section 6.5, which is good for people who want to understand what each
> metric means.
> It sounds like the reporting task stuff is all about the push.  I think
> pull may be better for us, so if you can document the end point (I assume
> we are talking about an HTTP/REST API?  That returns various metrics in
> JSON?), that would be very very helpful (and not just to us).
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]> wrote:
>
> > Otis,
> >
> > We're quite light on documentation thus far for the Reporting Task
> > API.  But you can see a bit about it here:
> >
> >
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
> >
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
> >
> > The javadocs are published to Maven as well so that will do nicely to
> > give a sense of what we expose from the Internals of NiFi to an
> > implementation of a reporting task.
> >
> > But feel free to ask detailed questions and as many as you want.  That
> > will give us good information on the types of things we need to cover
> > and in how much detail.
> >
> > Thanks
> > Joe
> >
> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> > <[hidden email]> wrote:
> > > Hi Joe,
> > >
> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
> > >
> > >> Otis,
> > >>
> > >> Glad you are liking it.
> > >>
> > >> We do provide support for both push and pull based metrics
> > >> reporting/gathering.
> > >>
> > >> For NiFi being able to push out metrics:
> > >> We do have an API for what we call "Reporting Tasks".  For example we
> > >> have a Ganglia reporter today.
> > >>
> > >
> > > So we are talking about the Yammer/Coda Hale's Metrics being used here?
> > > If so, are the metrics also available via JMX, too?
> > >
> > > For a client being able to pull metrics from NiFI:
> > >> The REST API is the way to go.
> > >>
> > >
> > > This sounds good.  Are this API and the metrics it exposes documented
> > > anywhere?
> > > I looked in the guides, but couldn't find anything.
> > >
> > >
> > >> Do either of these sound like a path that can work for you?  Let us
> > >> know if you'd like more details.
> > >>
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > >
> > >>
> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> > >> <[hidden email]> wrote:
> > >> > Hi,
> > >> >
> > >> > Nifi looks mightily impressive!
> > >> > I see Nifi has built-in monitoring and metrics.... but is there an
> API
> > >> that
> > >> > would allow something like SPM to add support for Nifi monitoring?
> Or
> > >> JMX?
> > >> >
> > >> > Thanks,
> > >> > Otis
> > >> > --
> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > >> > Solr & Elasticsearch Support * http://sematext.com/
> > >>
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Andrew Purtell-2
In reply to this post by Otis Gospodnetic
Would it be possible to instantiate a Metrics JmxReporter for each process?
Sounds like enabling metrics export via JMX could be an interesting JIRA at
any rate.

I did a quick search of the User, Developer, and Admin guides and there are
no occurrences of the string "JMX"

On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
[hidden email]> wrote:

> Hi Joe,
>
> Kafka devs describe their MBeans here -
> https://kafka.apache.org/08/ops.html
> - section 6.5, which is good for people who want to understand what each
> metric means.
> It sounds like the reporting task stuff is all about the push.  I think
> pull may be better for us, so if you can document the end point (I assume
> we are talking about an HTTP/REST API?  That returns various metrics in
> JSON?), that would be very very helpful (and not just to us).
>
> Thanks,
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]> wrote:
>
> > Otis,
> >
> > We're quite light on documentation thus far for the Reporting Task
> > API.  But you can see a bit about it here:
> >
> >
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
> >
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
> >
> > The javadocs are published to Maven as well so that will do nicely to
> > give a sense of what we expose from the Internals of NiFi to an
> > implementation of a reporting task.
> >
> > But feel free to ask detailed questions and as many as you want.  That
> > will give us good information on the types of things we need to cover
> > and in how much detail.
> >
> > Thanks
> > Joe
> >
> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> > <[hidden email]> wrote:
> > > Hi Joe,
> > >
> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
> > >
> > >> Otis,
> > >>
> > >> Glad you are liking it.
> > >>
> > >> We do provide support for both push and pull based metrics
> > >> reporting/gathering.
> > >>
> > >> For NiFi being able to push out metrics:
> > >> We do have an API for what we call "Reporting Tasks".  For example we
> > >> have a Ganglia reporter today.
> > >>
> > >
> > > So we are talking about the Yammer/Coda Hale's Metrics being used here?
> > > If so, are the metrics also available via JMX, too?
> > >
> > > For a client being able to pull metrics from NiFI:
> > >> The REST API is the way to go.
> > >>
> > >
> > > This sounds good.  Are this API and the metrics it exposes documented
> > > anywhere?
> > > I looked in the guides, but couldn't find anything.
> > >
> > >
> > >> Do either of these sound like a path that can work for you?  Let us
> > >> know if you'd like more details.
> > >>
> > >
> > > Thanks,
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > >
> > >
> > >>
> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> > >> <[hidden email]> wrote:
> > >> > Hi,
> > >> >
> > >> > Nifi looks mightily impressive!
> > >> > I see Nifi has built-in monitoring and metrics.... but is there an
> API
> > >> that
> > >> > would allow something like SPM to add support for Nifi monitoring?
> Or
> > >> JMX?
> > >> >
> > >> > Thanks,
> > >> > Otis
> > >> > --
> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> > >> > Solr & Elasticsearch Support * http://sematext.com/
> > >>
> >
>



--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
Andrew,

That should certainly be doable.  It just never seemed clear that JMX
was the way to go.  I think the theory had always been that if we
expose these sorts of things through the REST API over which we have
clearly defined authentication/authorization schemes we could then
support a variety of clients (people, systems).

I'll have to read up more on the options for security around JMX and
the sort of client interactions that opens up.  That said, it would
also be good to have a sense of tradeoffs of opening up further
avenues of client interaction beyond the REST API.  Once we get our
REST API docs legit again (had to pull due to licensing concerns) then
that should help a lot too.

Thanks
Joe


On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]> wrote:

> Would it be possible to instantiate a Metrics JmxReporter for each process?
> Sounds like enabling metrics export via JMX could be an interesting JIRA at
> any rate.
>
> I did a quick search of the User, Developer, and Admin guides and there are
> no occurrences of the string "JMX"
>
> On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
> [hidden email]> wrote:
>
>> Hi Joe,
>>
>> Kafka devs describe their MBeans here -
>> https://kafka.apache.org/08/ops.html
>> - section 6.5, which is good for people who want to understand what each
>> metric means.
>> It sounds like the reporting task stuff is all about the push.  I think
>> pull may be better for us, so if you can document the end point (I assume
>> we are talking about an HTTP/REST API?  That returns various metrics in
>> JSON?), that would be very very helpful (and not just to us).
>>
>> Thanks,
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]> wrote:
>>
>> > Otis,
>> >
>> > We're quite light on documentation thus far for the Reporting Task
>> > API.  But you can see a bit about it here:
>> >
>> >
>> >
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>> >
>> >
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>> >
>> > The javadocs are published to Maven as well so that will do nicely to
>> > give a sense of what we expose from the Internals of NiFi to an
>> > implementation of a reporting task.
>> >
>> > But feel free to ask detailed questions and as many as you want.  That
>> > will give us good information on the types of things we need to cover
>> > and in how much detail.
>> >
>> > Thanks
>> > Joe
>> >
>> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
>> > <[hidden email]> wrote:
>> > > Hi Joe,
>> > >
>> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]> wrote:
>> > >
>> > >> Otis,
>> > >>
>> > >> Glad you are liking it.
>> > >>
>> > >> We do provide support for both push and pull based metrics
>> > >> reporting/gathering.
>> > >>
>> > >> For NiFi being able to push out metrics:
>> > >> We do have an API for what we call "Reporting Tasks".  For example we
>> > >> have a Ganglia reporter today.
>> > >>
>> > >
>> > > So we are talking about the Yammer/Coda Hale's Metrics being used here?
>> > > If so, are the metrics also available via JMX, too?
>> > >
>> > > For a client being able to pull metrics from NiFI:
>> > >> The REST API is the way to go.
>> > >>
>> > >
>> > > This sounds good.  Are this API and the metrics it exposes documented
>> > > anywhere?
>> > > I looked in the guides, but couldn't find anything.
>> > >
>> > >
>> > >> Do either of these sound like a path that can work for you?  Let us
>> > >> know if you'd like more details.
>> > >>
>> > >
>> > > Thanks,
>> > > Otis
>> > > --
>> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >
>> > >
>> > >
>> > >
>> > >>
>> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>> > >> <[hidden email]> wrote:
>> > >> > Hi,
>> > >> >
>> > >> > Nifi looks mightily impressive!
>> > >> > I see Nifi has built-in monitoring and metrics.... but is there an
>> API
>> > >> that
>> > >> > would allow something like SPM to add support for Nifi monitoring?
>> Or
>> > >> JMX?
>> > >> >
>> > >> > Thanks,
>> > >> > Otis
>> > >> > --
>> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> Management
>> > >> > Solr & Elasticsearch Support * http://sematext.com/
>> > >>
>> >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Andrew Purtell-2
> That should certainly be doable.  It just never seemed clear that JMX
> was the way to go.  I think the theory had always been that if we
> expose these sorts of things through the REST API over which we have
> clearly defined authentication/authorization schemes we could then
> support a variety of clients (people, systems).

That's definitely true.

What we do over on HBase is wire up our metrics to JMX and provide a REST
endpoint in every process that presents all mbean data as JSON. We don't
set up JMX remote access, but let users edit configs for launch scripts if
RMI JMX access is desired. This has some minor advantages in my opinion:
- We don't need to do anything special to expose JVM runtime data like
detailed information on GC pools and times.
- If it is desired for some reason, devs/admins then do have the option to
set up the usual properties in the environment for RMI to JMX using the
standard options for auth and such.

In my experience some shops do want to work with JMX, usually when other
systems that export operational data via JMX are already under management.



On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:

> Andrew,
>
> That should certainly be doable.  It just never seemed clear that JMX
> was the way to go.  I think the theory had always been that if we
> expose these sorts of things through the REST API over which we have
> clearly defined authentication/authorization schemes we could then
> support a variety of clients (people, systems).
>
> I'll have to read up more on the options for security around JMX and
> the sort of client interactions that opens up.  That said, it would
> also be good to have a sense of tradeoffs of opening up further
> avenues of client interaction beyond the REST API.  Once we get our
> REST API docs legit again (had to pull due to licensing concerns) then
> that should help a lot too.
>
> Thanks
> Joe
>
>
> On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]>
> wrote:
> > Would it be possible to instantiate a Metrics JmxReporter for each
> process?
> > Sounds like enabling metrics export via JMX could be an interesting JIRA
> at
> > any rate.
> >
> > I did a quick search of the User, Developer, and Admin guides and there
> are
> > no occurrences of the string "JMX"
> >
> > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
> > [hidden email]> wrote:
> >
> >> Hi Joe,
> >>
> >> Kafka devs describe their MBeans here -
> >> https://kafka.apache.org/08/ops.html
> >> - section 6.5, which is good for people who want to understand what each
> >> metric means.
> >> It sounds like the reporting task stuff is all about the push.  I think
> >> pull may be better for us, so if you can document the end point (I
> assume
> >> we are talking about an HTTP/REST API?  That returns various metrics in
> >> JSON?), that would be very very helpful (and not just to us).
> >>
> >> Thanks,
> >> Otis
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> >>
> >>
> >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]> wrote:
> >>
> >> > Otis,
> >> >
> >> > We're quite light on documentation thus far for the Reporting Task
> >> > API.  But you can see a bit about it here:
> >> >
> >> >
> >> >
> >>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
> >> >
> >> >
> >>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
> >> >
> >> > The javadocs are published to Maven as well so that will do nicely to
> >> > give a sense of what we expose from the Internals of NiFi to an
> >> > implementation of a reporting task.
> >> >
> >> > But feel free to ask detailed questions and as many as you want.  That
> >> > will give us good information on the types of things we need to cover
> >> > and in how much detail.
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> >> > <[hidden email]> wrote:
> >> > > Hi Joe,
> >> > >
> >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]>
> wrote:
> >> > >
> >> > >> Otis,
> >> > >>
> >> > >> Glad you are liking it.
> >> > >>
> >> > >> We do provide support for both push and pull based metrics
> >> > >> reporting/gathering.
> >> > >>
> >> > >> For NiFi being able to push out metrics:
> >> > >> We do have an API for what we call "Reporting Tasks".  For example
> we
> >> > >> have a Ganglia reporter today.
> >> > >>
> >> > >
> >> > > So we are talking about the Yammer/Coda Hale's Metrics being used
> here?
> >> > > If so, are the metrics also available via JMX, too?
> >> > >
> >> > > For a client being able to pull metrics from NiFI:
> >> > >> The REST API is the way to go.
> >> > >>
> >> > >
> >> > > This sounds good.  Are this API and the metrics it exposes
> documented
> >> > > anywhere?
> >> > > I looked in the guides, but couldn't find anything.
> >> > >
> >> > >
> >> > >> Do either of these sound like a path that can work for you?  Let us
> >> > >> know if you'd like more details.
> >> > >>
> >> > >
> >> > > Thanks,
> >> > > Otis
> >> > > --
> >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >
> >> > >
> >> > >
> >> > >
> >> > >>
> >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> >> > >> <[hidden email]> wrote:
> >> > >> > Hi,
> >> > >> >
> >> > >> > Nifi looks mightily impressive!
> >> > >> > I see Nifi has built-in monitoring and metrics.... but is there
> an
> >> API
> >> > >> that
> >> > >> > would allow something like SPM to add support for Nifi
> monitoring?
> >> Or
> >> > >> JMX?
> >> > >> >
> >> > >> > Thanks,
> >> > >> > Otis
> >> > >> > --
> >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> Management
> >> > >> > Solr & Elasticsearch Support * http://sematext.com/
> >> > >>
> >> >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>



--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Otis Gospodnetic
+1 to everything that Andrew said.
I think there is actually no licensing issue with Metric library - it's
ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE

re JMX and auth stuff, we've documented it for SPM users here:
https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]> wrote:

> > That should certainly be doable.  It just never seemed clear that JMX
> > was the way to go.  I think the theory had always been that if we
> > expose these sorts of things through the REST API over which we have
> > clearly defined authentication/authorization schemes we could then
> > support a variety of clients (people, systems).
>
> That's definitely true.
>
> What we do over on HBase is wire up our metrics to JMX and provide a REST
> endpoint in every process that presents all mbean data as JSON. We don't
> set up JMX remote access, but let users edit configs for launch scripts if
> RMI JMX access is desired. This has some minor advantages in my opinion:
> - We don't need to do anything special to expose JVM runtime data like
> detailed information on GC pools and times.
> - If it is desired for some reason, devs/admins then do have the option to
> set up the usual properties in the environment for RMI to JMX using the
> standard options for auth and such.
>
> In my experience some shops do want to work with JMX, usually when other
> systems that export operational data via JMX are already under management.
>
>
>
> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
>
> > Andrew,
> >
> > That should certainly be doable.  It just never seemed clear that JMX
> > was the way to go.  I think the theory had always been that if we
> > expose these sorts of things through the REST API over which we have
> > clearly defined authentication/authorization schemes we could then
> > support a variety of clients (people, systems).
> >
> > I'll have to read up more on the options for security around JMX and
> > the sort of client interactions that opens up.  That said, it would
> > also be good to have a sense of tradeoffs of opening up further
> > avenues of client interaction beyond the REST API.  Once we get our
> > REST API docs legit again (had to pull due to licensing concerns) then
> > that should help a lot too.
> >
> > Thanks
> > Joe
> >
> >
> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]>
> > wrote:
> > > Would it be possible to instantiate a Metrics JmxReporter for each
> > process?
> > > Sounds like enabling metrics export via JMX could be an interesting
> JIRA
> > at
> > > any rate.
> > >
> > > I did a quick search of the User, Developer, and Admin guides and there
> > are
> > > no occurrences of the string "JMX"
> > >
> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
> > > [hidden email]> wrote:
> > >
> > >> Hi Joe,
> > >>
> > >> Kafka devs describe their MBeans here -
> > >> https://kafka.apache.org/08/ops.html
> > >> - section 6.5, which is good for people who want to understand what
> each
> > >> metric means.
> > >> It sounds like the reporting task stuff is all about the push.  I
> think
> > >> pull may be better for us, so if you can document the end point (I
> > assume
> > >> we are talking about an HTTP/REST API?  That returns various metrics
> in
> > >> JSON?), that would be very very helpful (and not just to us).
> > >>
> > >> Thanks,
> > >> Otis
> > >> --
> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > >> Solr & Elasticsearch Support * http://sematext.com/
> > >>
> > >>
> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
> wrote:
> > >>
> > >> > Otis,
> > >> >
> > >> > We're quite light on documentation thus far for the Reporting Task
> > >> > API.  But you can see a bit about it here:
> > >> >
> > >> >
> > >> >
> > >>
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
> > >> >
> > >> >
> > >>
> >
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
> > >> >
> > >> > The javadocs are published to Maven as well so that will do nicely
> to
> > >> > give a sense of what we expose from the Internals of NiFi to an
> > >> > implementation of a reporting task.
> > >> >
> > >> > But feel free to ask detailed questions and as many as you want.
> That
> > >> > will give us good information on the types of things we need to
> cover
> > >> > and in how much detail.
> > >> >
> > >> > Thanks
> > >> > Joe
> > >> >
> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> > >> > <[hidden email]> wrote:
> > >> > > Hi Joe,
> > >> > >
> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]>
> > wrote:
> > >> > >
> > >> > >> Otis,
> > >> > >>
> > >> > >> Glad you are liking it.
> > >> > >>
> > >> > >> We do provide support for both push and pull based metrics
> > >> > >> reporting/gathering.
> > >> > >>
> > >> > >> For NiFi being able to push out metrics:
> > >> > >> We do have an API for what we call "Reporting Tasks".  For
> example
> > we
> > >> > >> have a Ganglia reporter today.
> > >> > >>
> > >> > >
> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being used
> > here?
> > >> > > If so, are the metrics also available via JMX, too?
> > >> > >
> > >> > > For a client being able to pull metrics from NiFI:
> > >> > >> The REST API is the way to go.
> > >> > >>
> > >> > >
> > >> > > This sounds good.  Are this API and the metrics it exposes
> > documented
> > >> > > anywhere?
> > >> > > I looked in the guides, but couldn't find anything.
> > >> > >
> > >> > >
> > >> > >> Do either of these sound like a path that can work for you?  Let
> us
> > >> > >> know if you'd like more details.
> > >> > >>
> > >> > >
> > >> > > Thanks,
> > >> > > Otis
> > >> > > --
> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > Management
> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >
> > >> > >
> > >> > >
> > >> > >
> > >> > >>
> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> > >> > >> <[hidden email]> wrote:
> > >> > >> > Hi,
> > >> > >> >
> > >> > >> > Nifi looks mightily impressive!
> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is there
> > an
> > >> API
> > >> > >> that
> > >> > >> > would allow something like SPM to add support for Nifi
> > monitoring?
> > >> Or
> > >> > >> JMX?
> > >> > >> >
> > >> > >> > Thanks,
> > >> > >> > Otis
> > >> > >> > --
> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> > >> Management
> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
> > >> > >>
> > >> >
> > >>
> > >
> > >
> > >
> > > --
> > > Best regards,
> > >
> > >    - Andy
> > >
> > > Problems worthy of attack prove their worth by hitting back. - Piet
> Hein
> > > (via Tom White)
> >
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
Otis,

We'll take a fresh look at JMX but if you want to help contribute in
this area that would be awesome as well.

Regarding the Metric library and its licensing fitness:
----------------------------

It's important to keep in mind that transitive dependencies and their
licenses are just as important as direct dependencies.  If you start
here:

https://github.com/dropwizard/metrics   [yep good this one is ASLv2]
  from that we want to use

https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep
good still ASLv2 from parent but let's look at deps]
  specifically this one: info.ganglia.gmetric4j:gmetric4j

https://github.com/ganglia/gmetric4j [great it is BSD licensed and
thats ok too as it is Category-A but lets check its deps]
  specifically this one: org.acplt.remotetea:remotetea-oncrpc

http://sourceforge.net/projects/remotetea/  [And this is where things
are problematic.  This one is LGPLv2 which is Category-X - not allowed
for either source or binary dependency]
  Upon further review the source does indeed depend on the remotetea
library so we can't simply ignore that dependency.  Which means we can
use the gmetric4j library and thus can't use metrics-ganglia library.

Now this is just from the ganglia tree.  Other parts of the metrics
library might well be fine.  We're getting around this problem today
by using the older metrics library and its older ganglia support which
doesn't have this dependency licensing issue.


Thanks
Joe

On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic
<[hidden email]> wrote:

> +1 to everything that Andrew said.
> I think there is actually no licensing issue with Metric library - it's
> ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE
>
> re JMX and auth stuff, we've documented it for SPM users here:
> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]> wrote:
>
>> > That should certainly be doable.  It just never seemed clear that JMX
>> > was the way to go.  I think the theory had always been that if we
>> > expose these sorts of things through the REST API over which we have
>> > clearly defined authentication/authorization schemes we could then
>> > support a variety of clients (people, systems).
>>
>> That's definitely true.
>>
>> What we do over on HBase is wire up our metrics to JMX and provide a REST
>> endpoint in every process that presents all mbean data as JSON. We don't
>> set up JMX remote access, but let users edit configs for launch scripts if
>> RMI JMX access is desired. This has some minor advantages in my opinion:
>> - We don't need to do anything special to expose JVM runtime data like
>> detailed information on GC pools and times.
>> - If it is desired for some reason, devs/admins then do have the option to
>> set up the usual properties in the environment for RMI to JMX using the
>> standard options for auth and such.
>>
>> In my experience some shops do want to work with JMX, usually when other
>> systems that export operational data via JMX are already under management.
>>
>>
>>
>> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
>>
>> > Andrew,
>> >
>> > That should certainly be doable.  It just never seemed clear that JMX
>> > was the way to go.  I think the theory had always been that if we
>> > expose these sorts of things through the REST API over which we have
>> > clearly defined authentication/authorization schemes we could then
>> > support a variety of clients (people, systems).
>> >
>> > I'll have to read up more on the options for security around JMX and
>> > the sort of client interactions that opens up.  That said, it would
>> > also be good to have a sense of tradeoffs of opening up further
>> > avenues of client interaction beyond the REST API.  Once we get our
>> > REST API docs legit again (had to pull due to licensing concerns) then
>> > that should help a lot too.
>> >
>> > Thanks
>> > Joe
>> >
>> >
>> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]>
>> > wrote:
>> > > Would it be possible to instantiate a Metrics JmxReporter for each
>> > process?
>> > > Sounds like enabling metrics export via JMX could be an interesting
>> JIRA
>> > at
>> > > any rate.
>> > >
>> > > I did a quick search of the User, Developer, and Admin guides and there
>> > are
>> > > no occurrences of the string "JMX"
>> > >
>> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
>> > > [hidden email]> wrote:
>> > >
>> > >> Hi Joe,
>> > >>
>> > >> Kafka devs describe their MBeans here -
>> > >> https://kafka.apache.org/08/ops.html
>> > >> - section 6.5, which is good for people who want to understand what
>> each
>> > >> metric means.
>> > >> It sounds like the reporting task stuff is all about the push.  I
>> think
>> > >> pull may be better for us, so if you can document the end point (I
>> > assume
>> > >> we are talking about an HTTP/REST API?  That returns various metrics
>> in
>> > >> JSON?), that would be very very helpful (and not just to us).
>> > >>
>> > >> Thanks,
>> > >> Otis
>> > >> --
>> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > >> Solr & Elasticsearch Support * http://sematext.com/
>> > >>
>> > >>
>> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
>> wrote:
>> > >>
>> > >> > Otis,
>> > >> >
>> > >> > We're quite light on documentation thus far for the Reporting Task
>> > >> > API.  But you can see a bit about it here:
>> > >> >
>> > >> >
>> > >> >
>> > >>
>> >
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>> > >> >
>> > >> >
>> > >>
>> >
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>> > >> >
>> > >> > The javadocs are published to Maven as well so that will do nicely
>> to
>> > >> > give a sense of what we expose from the Internals of NiFi to an
>> > >> > implementation of a reporting task.
>> > >> >
>> > >> > But feel free to ask detailed questions and as many as you want.
>> That
>> > >> > will give us good information on the types of things we need to
>> cover
>> > >> > and in how much detail.
>> > >> >
>> > >> > Thanks
>> > >> > Joe
>> > >> >
>> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
>> > >> > <[hidden email]> wrote:
>> > >> > > Hi Joe,
>> > >> > >
>> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]>
>> > wrote:
>> > >> > >
>> > >> > >> Otis,
>> > >> > >>
>> > >> > >> Glad you are liking it.
>> > >> > >>
>> > >> > >> We do provide support for both push and pull based metrics
>> > >> > >> reporting/gathering.
>> > >> > >>
>> > >> > >> For NiFi being able to push out metrics:
>> > >> > >> We do have an API for what we call "Reporting Tasks".  For
>> example
>> > we
>> > >> > >> have a Ganglia reporter today.
>> > >> > >>
>> > >> > >
>> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being used
>> > here?
>> > >> > > If so, are the metrics also available via JMX, too?
>> > >> > >
>> > >> > > For a client being able to pull metrics from NiFI:
>> > >> > >> The REST API is the way to go.
>> > >> > >>
>> > >> > >
>> > >> > > This sounds good.  Are this API and the metrics it exposes
>> > documented
>> > >> > > anywhere?
>> > >> > > I looked in the guides, but couldn't find anything.
>> > >> > >
>> > >> > >
>> > >> > >> Do either of these sound like a path that can work for you?  Let
>> us
>> > >> > >> know if you'd like more details.
>> > >> > >>
>> > >> > >
>> > >> > > Thanks,
>> > >> > > Otis
>> > >> > > --
>> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > Management
>> > >> > > Solr & Elasticsearch Support * http://sematext.com/
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >
>> > >> > >>
>> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>> > >> > >> <[hidden email]> wrote:
>> > >> > >> > Hi,
>> > >> > >> >
>> > >> > >> > Nifi looks mightily impressive!
>> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is there
>> > an
>> > >> API
>> > >> > >> that
>> > >> > >> > would allow something like SPM to add support for Nifi
>> > monitoring?
>> > >> Or
>> > >> > >> JMX?
>> > >> > >> >
>> > >> > >> > Thanks,
>> > >> > >> > Otis
>> > >> > >> > --
>> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> > >> Management
>> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
>> > >> > >>
>> > >> >
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Best regards,
>> > >
>> > >    - Andy
>> > >
>> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> Hein
>> > > (via Tom White)
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
>>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Andrew Purtell-2
Would it suffice to not support Ganglia reporting? Then you'd not be
bringing in metrics-ganglia* and its transitive dependencies?

* - And can actively exclude it in POMs if necessary



On Thu, Feb 19, 2015 at 1:39 PM, Joe Witt <[hidden email]> wrote:

> Otis,
>
> We'll take a fresh look at JMX but if you want to help contribute in
> this area that would be awesome as well.
>
> Regarding the Metric library and its licensing fitness:
> ----------------------------
>
> It's important to keep in mind that transitive dependencies and their
> licenses are just as important as direct dependencies.  If you start
> here:
>
> https://github.com/dropwizard/metrics   [yep good this one is ASLv2]
>   from that we want to use
>
> https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep
> good still ASLv2 from parent but let's look at deps]
>   specifically this one: info.ganglia.gmetric4j:gmetric4j
>
> https://github.com/ganglia/gmetric4j [great it is BSD licensed and
> thats ok too as it is Category-A but lets check its deps]
>   specifically this one: org.acplt.remotetea:remotetea-oncrpc
>
> http://sourceforge.net/projects/remotetea/  [And this is where things
> are problematic.  This one is LGPLv2 which is Category-X - not allowed
> for either source or binary dependency]
>   Upon further review the source does indeed depend on the remotetea
> library so we can't simply ignore that dependency.  Which means we can
> use the gmetric4j library and thus can't use metrics-ganglia library.
>
> Now this is just from the ganglia tree.  Other parts of the metrics
> library might well be fine.  We're getting around this problem today
> by using the older metrics library and its older ganglia support which
> doesn't have this dependency licensing issue.
>
>
> Thanks
> Joe
>
> On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic
> <[hidden email]> wrote:
> > +1 to everything that Andrew said.
> > I think there is actually no licensing issue with Metric library - it's
> > ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE
> >
> > re JMX and auth stuff, we've documented it for SPM users here:
> >
> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]>
> wrote:
> >
> >> > That should certainly be doable.  It just never seemed clear that JMX
> >> > was the way to go.  I think the theory had always been that if we
> >> > expose these sorts of things through the REST API over which we have
> >> > clearly defined authentication/authorization schemes we could then
> >> > support a variety of clients (people, systems).
> >>
> >> That's definitely true.
> >>
> >> What we do over on HBase is wire up our metrics to JMX and provide a
> REST
> >> endpoint in every process that presents all mbean data as JSON. We don't
> >> set up JMX remote access, but let users edit configs for launch scripts
> if
> >> RMI JMX access is desired. This has some minor advantages in my opinion:
> >> - We don't need to do anything special to expose JVM runtime data like
> >> detailed information on GC pools and times.
> >> - If it is desired for some reason, devs/admins then do have the option
> to
> >> set up the usual properties in the environment for RMI to JMX using the
> >> standard options for auth and such.
> >>
> >> In my experience some shops do want to work with JMX, usually when other
> >> systems that export operational data via JMX are already under
> management.
> >>
> >>
> >>
> >> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
> >>
> >> > Andrew,
> >> >
> >> > That should certainly be doable.  It just never seemed clear that JMX
> >> > was the way to go.  I think the theory had always been that if we
> >> > expose these sorts of things through the REST API over which we have
> >> > clearly defined authentication/authorization schemes we could then
> >> > support a variety of clients (people, systems).
> >> >
> >> > I'll have to read up more on the options for security around JMX and
> >> > the sort of client interactions that opens up.  That said, it would
> >> > also be good to have a sense of tradeoffs of opening up further
> >> > avenues of client interaction beyond the REST API.  Once we get our
> >> > REST API docs legit again (had to pull due to licensing concerns) then
> >> > that should help a lot too.
> >> >
> >> > Thanks
> >> > Joe
> >> >
> >> >
> >> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]
> >
> >> > wrote:
> >> > > Would it be possible to instantiate a Metrics JmxReporter for each
> >> > process?
> >> > > Sounds like enabling metrics export via JMX could be an interesting
> >> JIRA
> >> > at
> >> > > any rate.
> >> > >
> >> > > I did a quick search of the User, Developer, and Admin guides and
> there
> >> > are
> >> > > no occurrences of the string "JMX"
> >> > >
> >> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
> >> > > [hidden email]> wrote:
> >> > >
> >> > >> Hi Joe,
> >> > >>
> >> > >> Kafka devs describe their MBeans here -
> >> > >> https://kafka.apache.org/08/ops.html
> >> > >> - section 6.5, which is good for people who want to understand what
> >> each
> >> > >> metric means.
> >> > >> It sounds like the reporting task stuff is all about the push.  I
> >> think
> >> > >> pull may be better for us, so if you can document the end point (I
> >> > assume
> >> > >> we are talking about an HTTP/REST API?  That returns various
> metrics
> >> in
> >> > >> JSON?), that would be very very helpful (and not just to us).
> >> > >>
> >> > >> Thanks,
> >> > >> Otis
> >> > >> --
> >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >> > >> Solr & Elasticsearch Support * http://sematext.com/
> >> > >>
> >> > >>
> >> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
> >> wrote:
> >> > >>
> >> > >> > Otis,
> >> > >> >
> >> > >> > We're quite light on documentation thus far for the Reporting
> Task
> >> > >> > API.  But you can see a bit about it here:
> >> > >> >
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
> >> > >> >
> >> > >> >
> >> > >>
> >> >
> >>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
> >> > >> >
> >> > >> > The javadocs are published to Maven as well so that will do
> nicely
> >> to
> >> > >> > give a sense of what we expose from the Internals of NiFi to an
> >> > >> > implementation of a reporting task.
> >> > >> >
> >> > >> > But feel free to ask detailed questions and as many as you want.
> >> That
> >> > >> > will give us good information on the types of things we need to
> >> cover
> >> > >> > and in how much detail.
> >> > >> >
> >> > >> > Thanks
> >> > >> > Joe
> >> > >> >
> >> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
> >> > >> > <[hidden email]> wrote:
> >> > >> > > Hi Joe,
> >> > >> > >
> >> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]
> >
> >> > wrote:
> >> > >> > >
> >> > >> > >> Otis,
> >> > >> > >>
> >> > >> > >> Glad you are liking it.
> >> > >> > >>
> >> > >> > >> We do provide support for both push and pull based metrics
> >> > >> > >> reporting/gathering.
> >> > >> > >>
> >> > >> > >> For NiFi being able to push out metrics:
> >> > >> > >> We do have an API for what we call "Reporting Tasks".  For
> >> example
> >> > we
> >> > >> > >> have a Ganglia reporter today.
> >> > >> > >>
> >> > >> > >
> >> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being
> used
> >> > here?
> >> > >> > > If so, are the metrics also available via JMX, too?
> >> > >> > >
> >> > >> > > For a client being able to pull metrics from NiFI:
> >> > >> > >> The REST API is the way to go.
> >> > >> > >>
> >> > >> > >
> >> > >> > > This sounds good.  Are this API and the metrics it exposes
> >> > documented
> >> > >> > > anywhere?
> >> > >> > > I looked in the guides, but couldn't find anything.
> >> > >> > >
> >> > >> > >
> >> > >> > >> Do either of these sound like a path that can work for you?
> Let
> >> us
> >> > >> > >> know if you'd like more details.
> >> > >> > >>
> >> > >> > >
> >> > >> > > Thanks,
> >> > >> > > Otis
> >> > >> > > --
> >> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > Management
> >> > >> > > Solr & Elasticsearch Support * http://sematext.com/
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >
> >> > >> > >>
> >> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
> >> > >> > >> <[hidden email]> wrote:
> >> > >> > >> > Hi,
> >> > >> > >> >
> >> > >> > >> > Nifi looks mightily impressive!
> >> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is
> there
> >> > an
> >> > >> API
> >> > >> > >> that
> >> > >> > >> > would allow something like SPM to add support for Nifi
> >> > monitoring?
> >> > >> Or
> >> > >> > >> JMX?
> >> > >> > >> >
> >> > >> > >> > Thanks,
> >> > >> > >> > Otis
> >> > >> > >> > --
> >> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
> >> > >> Management
> >> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
> >> > >> > >>
> >> > >> >
> >> > >>
> >> > >
> >> > >
> >> > >
> >> > > --
> >> > > Best regards,
> >> > >
> >> > >    - Andy
> >> > >
> >> > > Problems worthy of attack prove their worth by hitting back. - Piet
> >> Hein
> >> > > (via Tom White)
> >> >
> >>
> >>
> >>
> >> --
> >> Best regards,
> >>
> >>    - Andy
> >>
> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> >> (via Tom White)
> >>
>



--
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
yeah absolutely - i was just addressing the fact that there is a
licensing issue (in general) with that library.

We can still totally do fun things with JMX, ganglia, etc..  Just
takes a little more work when the best libs aren't' readily useable.

Thanks
Joe

On Thu, Feb 19, 2015 at 4:42 PM, Andrew Purtell <[hidden email]> wrote:

> Would it suffice to not support Ganglia reporting? Then you'd not be
> bringing in metrics-ganglia* and its transitive dependencies?
>
> * - And can actively exclude it in POMs if necessary
>
>
>
> On Thu, Feb 19, 2015 at 1:39 PM, Joe Witt <[hidden email]> wrote:
>
>> Otis,
>>
>> We'll take a fresh look at JMX but if you want to help contribute in
>> this area that would be awesome as well.
>>
>> Regarding the Metric library and its licensing fitness:
>> ----------------------------
>>
>> It's important to keep in mind that transitive dependencies and their
>> licenses are just as important as direct dependencies.  If you start
>> here:
>>
>> https://github.com/dropwizard/metrics   [yep good this one is ASLv2]
>>   from that we want to use
>>
>> https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep
>> good still ASLv2 from parent but let's look at deps]
>>   specifically this one: info.ganglia.gmetric4j:gmetric4j
>>
>> https://github.com/ganglia/gmetric4j [great it is BSD licensed and
>> thats ok too as it is Category-A but lets check its deps]
>>   specifically this one: org.acplt.remotetea:remotetea-oncrpc
>>
>> http://sourceforge.net/projects/remotetea/  [And this is where things
>> are problematic.  This one is LGPLv2 which is Category-X - not allowed
>> for either source or binary dependency]
>>   Upon further review the source does indeed depend on the remotetea
>> library so we can't simply ignore that dependency.  Which means we can
>> use the gmetric4j library and thus can't use metrics-ganglia library.
>>
>> Now this is just from the ganglia tree.  Other parts of the metrics
>> library might well be fine.  We're getting around this problem today
>> by using the older metrics library and its older ganglia support which
>> doesn't have this dependency licensing issue.
>>
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic
>> <[hidden email]> wrote:
>> > +1 to everything that Andrew said.
>> > I think there is actually no licensing issue with Metric library - it's
>> > ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE
>> >
>> > re JMX and auth stuff, we've documented it for SPM users here:
>> >
>> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
>> >
>> > Otis
>> > --
>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> > Solr & Elasticsearch Support * http://sematext.com/
>> >
>> >
>> > On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]>
>> wrote:
>> >
>> >> > That should certainly be doable.  It just never seemed clear that JMX
>> >> > was the way to go.  I think the theory had always been that if we
>> >> > expose these sorts of things through the REST API over which we have
>> >> > clearly defined authentication/authorization schemes we could then
>> >> > support a variety of clients (people, systems).
>> >>
>> >> That's definitely true.
>> >>
>> >> What we do over on HBase is wire up our metrics to JMX and provide a
>> REST
>> >> endpoint in every process that presents all mbean data as JSON. We don't
>> >> set up JMX remote access, but let users edit configs for launch scripts
>> if
>> >> RMI JMX access is desired. This has some minor advantages in my opinion:
>> >> - We don't need to do anything special to expose JVM runtime data like
>> >> detailed information on GC pools and times.
>> >> - If it is desired for some reason, devs/admins then do have the option
>> to
>> >> set up the usual properties in the environment for RMI to JMX using the
>> >> standard options for auth and such.
>> >>
>> >> In my experience some shops do want to work with JMX, usually when other
>> >> systems that export operational data via JMX are already under
>> management.
>> >>
>> >>
>> >>
>> >> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
>> >>
>> >> > Andrew,
>> >> >
>> >> > That should certainly be doable.  It just never seemed clear that JMX
>> >> > was the way to go.  I think the theory had always been that if we
>> >> > expose these sorts of things through the REST API over which we have
>> >> > clearly defined authentication/authorization schemes we could then
>> >> > support a variety of clients (people, systems).
>> >> >
>> >> > I'll have to read up more on the options for security around JMX and
>> >> > the sort of client interactions that opens up.  That said, it would
>> >> > also be good to have a sense of tradeoffs of opening up further
>> >> > avenues of client interaction beyond the REST API.  Once we get our
>> >> > REST API docs legit again (had to pull due to licensing concerns) then
>> >> > that should help a lot too.
>> >> >
>> >> > Thanks
>> >> > Joe
>> >> >
>> >> >
>> >> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]
>> >
>> >> > wrote:
>> >> > > Would it be possible to instantiate a Metrics JmxReporter for each
>> >> > process?
>> >> > > Sounds like enabling metrics export via JMX could be an interesting
>> >> JIRA
>> >> > at
>> >> > > any rate.
>> >> > >
>> >> > > I did a quick search of the User, Developer, and Admin guides and
>> there
>> >> > are
>> >> > > no occurrences of the string "JMX"
>> >> > >
>> >> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
>> >> > > [hidden email]> wrote:
>> >> > >
>> >> > >> Hi Joe,
>> >> > >>
>> >> > >> Kafka devs describe their MBeans here -
>> >> > >> https://kafka.apache.org/08/ops.html
>> >> > >> - section 6.5, which is good for people who want to understand what
>> >> each
>> >> > >> metric means.
>> >> > >> It sounds like the reporting task stuff is all about the push.  I
>> >> think
>> >> > >> pull may be better for us, so if you can document the end point (I
>> >> > assume
>> >> > >> we are talking about an HTTP/REST API?  That returns various
>> metrics
>> >> in
>> >> > >> JSON?), that would be very very helpful (and not just to us).
>> >> > >>
>> >> > >> Thanks,
>> >> > >> Otis
>> >> > >> --
>> >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log
>> Management
>> >> > >> Solr & Elasticsearch Support * http://sematext.com/
>> >> > >>
>> >> > >>
>> >> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
>> >> wrote:
>> >> > >>
>> >> > >> > Otis,
>> >> > >> >
>> >> > >> > We're quite light on documentation thus far for the Reporting
>> Task
>> >> > >> > API.  But you can see a bit about it here:
>> >> > >> >
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>> >> > >> >
>> >> > >> >
>> >> > >>
>> >> >
>> >>
>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>> >> > >> >
>> >> > >> > The javadocs are published to Maven as well so that will do
>> nicely
>> >> to
>> >> > >> > give a sense of what we expose from the Internals of NiFi to an
>> >> > >> > implementation of a reporting task.
>> >> > >> >
>> >> > >> > But feel free to ask detailed questions and as many as you want.
>> >> That
>> >> > >> > will give us good information on the types of things we need to
>> >> cover
>> >> > >> > and in how much detail.
>> >> > >> >
>> >> > >> > Thanks
>> >> > >> > Joe
>> >> > >> >
>> >> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
>> >> > >> > <[hidden email]> wrote:
>> >> > >> > > Hi Joe,
>> >> > >> > >
>> >> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]
>> >
>> >> > wrote:
>> >> > >> > >
>> >> > >> > >> Otis,
>> >> > >> > >>
>> >> > >> > >> Glad you are liking it.
>> >> > >> > >>
>> >> > >> > >> We do provide support for both push and pull based metrics
>> >> > >> > >> reporting/gathering.
>> >> > >> > >>
>> >> > >> > >> For NiFi being able to push out metrics:
>> >> > >> > >> We do have an API for what we call "Reporting Tasks".  For
>> >> example
>> >> > we
>> >> > >> > >> have a Ganglia reporter today.
>> >> > >> > >>
>> >> > >> > >
>> >> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being
>> used
>> >> > here?
>> >> > >> > > If so, are the metrics also available via JMX, too?
>> >> > >> > >
>> >> > >> > > For a client being able to pull metrics from NiFI:
>> >> > >> > >> The REST API is the way to go.
>> >> > >> > >>
>> >> > >> > >
>> >> > >> > > This sounds good.  Are this API and the metrics it exposes
>> >> > documented
>> >> > >> > > anywhere?
>> >> > >> > > I looked in the guides, but couldn't find anything.
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >> Do either of these sound like a path that can work for you?
>> Let
>> >> us
>> >> > >> > >> know if you'd like more details.
>> >> > >> > >>
>> >> > >> > >
>> >> > >> > > Thanks,
>> >> > >> > > Otis
>> >> > >> > > --
>> >> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> >> > Management
>> >> > >> > > Solr & Elasticsearch Support * http://sematext.com/
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >
>> >> > >> > >>
>> >> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>> >> > >> > >> <[hidden email]> wrote:
>> >> > >> > >> > Hi,
>> >> > >> > >> >
>> >> > >> > >> > Nifi looks mightily impressive!
>> >> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is
>> there
>> >> > an
>> >> > >> API
>> >> > >> > >> that
>> >> > >> > >> > would allow something like SPM to add support for Nifi
>> >> > monitoring?
>> >> > >> Or
>> >> > >> > >> JMX?
>> >> > >> > >> >
>> >> > >> > >> > Thanks,
>> >> > >> > >> > Otis
>> >> > >> > >> > --
>> >> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
>> >> > >> Management
>> >> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
>> >> > >> > >>
>> >> > >> >
>> >> > >>
>> >> > >
>> >> > >
>> >> > >
>> >> > > --
>> >> > > Best regards,
>> >> > >
>> >> > >    - Andy
>> >> > >
>> >> > > Problems worthy of attack prove their worth by hitting back. - Piet
>> >> Hein
>> >> > > (via Tom White)
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best regards,
>> >>
>> >>    - Andy
>> >>
>> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> >> (via Tom White)
>> >>
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Ashish
Joe,

other way round is to support it, but let the End User download and
drop the binaries needed (not shipping them by default).

I am keen on working to get the JMX support and based on my reading of
the code so far, it would be a Reporting Task (shall keep it generic
around Metrics) and add initially enable it only for JMX and later as
we have more clarity, we can add more functions.

Shall I create a JIRA for this?

On Fri, Feb 20, 2015 at 3:15 AM, Joe Witt <[hidden email]> wrote:

> yeah absolutely - i was just addressing the fact that there is a
> licensing issue (in general) with that library.
>
> We can still totally do fun things with JMX, ganglia, etc..  Just
> takes a little more work when the best libs aren't' readily useable.
>
> Thanks
> Joe
>
> On Thu, Feb 19, 2015 at 4:42 PM, Andrew Purtell <[hidden email]> wrote:
>> Would it suffice to not support Ganglia reporting? Then you'd not be
>> bringing in metrics-ganglia* and its transitive dependencies?
>>
>> * - And can actively exclude it in POMs if necessary
>>
>>
>>
>> On Thu, Feb 19, 2015 at 1:39 PM, Joe Witt <[hidden email]> wrote:
>>
>>> Otis,
>>>
>>> We'll take a fresh look at JMX but if you want to help contribute in
>>> this area that would be awesome as well.
>>>
>>> Regarding the Metric library and its licensing fitness:
>>> ----------------------------
>>>
>>> It's important to keep in mind that transitive dependencies and their
>>> licenses are just as important as direct dependencies.  If you start
>>> here:
>>>
>>> https://github.com/dropwizard/metrics   [yep good this one is ASLv2]
>>>   from that we want to use
>>>
>>> https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep
>>> good still ASLv2 from parent but let's look at deps]
>>>   specifically this one: info.ganglia.gmetric4j:gmetric4j
>>>
>>> https://github.com/ganglia/gmetric4j [great it is BSD licensed and
>>> thats ok too as it is Category-A but lets check its deps]
>>>   specifically this one: org.acplt.remotetea:remotetea-oncrpc
>>>
>>> http://sourceforge.net/projects/remotetea/  [And this is where things
>>> are problematic.  This one is LGPLv2 which is Category-X - not allowed
>>> for either source or binary dependency]
>>>   Upon further review the source does indeed depend on the remotetea
>>> library so we can't simply ignore that dependency.  Which means we can
>>> use the gmetric4j library and thus can't use metrics-ganglia library.
>>>
>>> Now this is just from the ganglia tree.  Other parts of the metrics
>>> library might well be fine.  We're getting around this problem today
>>> by using the older metrics library and its older ganglia support which
>>> doesn't have this dependency licensing issue.
>>>
>>>
>>> Thanks
>>> Joe
>>>
>>> On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic
>>> <[hidden email]> wrote:
>>> > +1 to everything that Andrew said.
>>> > I think there is actually no licensing issue with Metric library - it's
>>> > ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE
>>> >
>>> > re JMX and auth stuff, we've documented it for SPM users here:
>>> >
>>> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
>>> >
>>> > Otis
>>> > --
>>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> > Solr & Elasticsearch Support * http://sematext.com/
>>> >
>>> >
>>> > On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]>
>>> wrote:
>>> >
>>> >> > That should certainly be doable.  It just never seemed clear that JMX
>>> >> > was the way to go.  I think the theory had always been that if we
>>> >> > expose these sorts of things through the REST API over which we have
>>> >> > clearly defined authentication/authorization schemes we could then
>>> >> > support a variety of clients (people, systems).
>>> >>
>>> >> That's definitely true.
>>> >>
>>> >> What we do over on HBase is wire up our metrics to JMX and provide a
>>> REST
>>> >> endpoint in every process that presents all mbean data as JSON. We don't
>>> >> set up JMX remote access, but let users edit configs for launch scripts
>>> if
>>> >> RMI JMX access is desired. This has some minor advantages in my opinion:
>>> >> - We don't need to do anything special to expose JVM runtime data like
>>> >> detailed information on GC pools and times.
>>> >> - If it is desired for some reason, devs/admins then do have the option
>>> to
>>> >> set up the usual properties in the environment for RMI to JMX using the
>>> >> standard options for auth and such.
>>> >>
>>> >> In my experience some shops do want to work with JMX, usually when other
>>> >> systems that export operational data via JMX are already under
>>> management.
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
>>> >>
>>> >> > Andrew,
>>> >> >
>>> >> > That should certainly be doable.  It just never seemed clear that JMX
>>> >> > was the way to go.  I think the theory had always been that if we
>>> >> > expose these sorts of things through the REST API over which we have
>>> >> > clearly defined authentication/authorization schemes we could then
>>> >> > support a variety of clients (people, systems).
>>> >> >
>>> >> > I'll have to read up more on the options for security around JMX and
>>> >> > the sort of client interactions that opens up.  That said, it would
>>> >> > also be good to have a sense of tradeoffs of opening up further
>>> >> > avenues of client interaction beyond the REST API.  Once we get our
>>> >> > REST API docs legit again (had to pull due to licensing concerns) then
>>> >> > that should help a lot too.
>>> >> >
>>> >> > Thanks
>>> >> > Joe
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]
>>> >
>>> >> > wrote:
>>> >> > > Would it be possible to instantiate a Metrics JmxReporter for each
>>> >> > process?
>>> >> > > Sounds like enabling metrics export via JMX could be an interesting
>>> >> JIRA
>>> >> > at
>>> >> > > any rate.
>>> >> > >
>>> >> > > I did a quick search of the User, Developer, and Admin guides and
>>> there
>>> >> > are
>>> >> > > no occurrences of the string "JMX"
>>> >> > >
>>> >> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
>>> >> > > [hidden email]> wrote:
>>> >> > >
>>> >> > >> Hi Joe,
>>> >> > >>
>>> >> > >> Kafka devs describe their MBeans here -
>>> >> > >> https://kafka.apache.org/08/ops.html
>>> >> > >> - section 6.5, which is good for people who want to understand what
>>> >> each
>>> >> > >> metric means.
>>> >> > >> It sounds like the reporting task stuff is all about the push.  I
>>> >> think
>>> >> > >> pull may be better for us, so if you can document the end point (I
>>> >> > assume
>>> >> > >> we are talking about an HTTP/REST API?  That returns various
>>> metrics
>>> >> in
>>> >> > >> JSON?), that would be very very helpful (and not just to us).
>>> >> > >>
>>> >> > >> Thanks,
>>> >> > >> Otis
>>> >> > >> --
>>> >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log
>>> Management
>>> >> > >> Solr & Elasticsearch Support * http://sematext.com/
>>> >> > >>
>>> >> > >>
>>> >> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
>>> >> wrote:
>>> >> > >>
>>> >> > >> > Otis,
>>> >> > >> >
>>> >> > >> > We're quite light on documentation thus far for the Reporting
>>> Task
>>> >> > >> > API.  But you can see a bit about it here:
>>> >> > >> >
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>>> >> > >> >
>>> >> > >> >
>>> >> > >>
>>> >> >
>>> >>
>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>>> >> > >> >
>>> >> > >> > The javadocs are published to Maven as well so that will do
>>> nicely
>>> >> to
>>> >> > >> > give a sense of what we expose from the Internals of NiFi to an
>>> >> > >> > implementation of a reporting task.
>>> >> > >> >
>>> >> > >> > But feel free to ask detailed questions and as many as you want.
>>> >> That
>>> >> > >> > will give us good information on the types of things we need to
>>> >> cover
>>> >> > >> > and in how much detail.
>>> >> > >> >
>>> >> > >> > Thanks
>>> >> > >> > Joe
>>> >> > >> >
>>> >> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
>>> >> > >> > <[hidden email]> wrote:
>>> >> > >> > > Hi Joe,
>>> >> > >> > >
>>> >> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]
>>> >
>>> >> > wrote:
>>> >> > >> > >
>>> >> > >> > >> Otis,
>>> >> > >> > >>
>>> >> > >> > >> Glad you are liking it.
>>> >> > >> > >>
>>> >> > >> > >> We do provide support for both push and pull based metrics
>>> >> > >> > >> reporting/gathering.
>>> >> > >> > >>
>>> >> > >> > >> For NiFi being able to push out metrics:
>>> >> > >> > >> We do have an API for what we call "Reporting Tasks".  For
>>> >> example
>>> >> > we
>>> >> > >> > >> have a Ganglia reporter today.
>>> >> > >> > >>
>>> >> > >> > >
>>> >> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being
>>> used
>>> >> > here?
>>> >> > >> > > If so, are the metrics also available via JMX, too?
>>> >> > >> > >
>>> >> > >> > > For a client being able to pull metrics from NiFI:
>>> >> > >> > >> The REST API is the way to go.
>>> >> > >> > >>
>>> >> > >> > >
>>> >> > >> > > This sounds good.  Are this API and the metrics it exposes
>>> >> > documented
>>> >> > >> > > anywhere?
>>> >> > >> > > I looked in the guides, but couldn't find anything.
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > >> Do either of these sound like a path that can work for you?
>>> Let
>>> >> us
>>> >> > >> > >> know if you'd like more details.
>>> >> > >> > >>
>>> >> > >> > >
>>> >> > >> > > Thanks,
>>> >> > >> > > Otis
>>> >> > >> > > --
>>> >> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>>> >> > Management
>>> >> > >> > > Solr & Elasticsearch Support * http://sematext.com/
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > >
>>> >> > >> > >>
>>> >> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>>> >> > >> > >> <[hidden email]> wrote:
>>> >> > >> > >> > Hi,
>>> >> > >> > >> >
>>> >> > >> > >> > Nifi looks mightily impressive!
>>> >> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is
>>> there
>>> >> > an
>>> >> > >> API
>>> >> > >> > >> that
>>> >> > >> > >> > would allow something like SPM to add support for Nifi
>>> >> > monitoring?
>>> >> > >> Or
>>> >> > >> > >> JMX?
>>> >> > >> > >> >
>>> >> > >> > >> > Thanks,
>>> >> > >> > >> > Otis
>>> >> > >> > >> > --
>>> >> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
>>> >> > >> Management
>>> >> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
>>> >> > >> > >>
>>> >> > >> >
>>> >> > >>
>>> >> > >
>>> >> > >
>>> >> > >
>>> >> > > --
>>> >> > > Best regards,
>>> >> > >
>>> >> > >    - Andy
>>> >> > >
>>> >> > > Problems worthy of attack prove their worth by hitting back. - Piet
>>> >> Hein
>>> >> > > (via Tom White)
>>> >> >
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Best regards,
>>> >>
>>> >>    - Andy
>>> >>
>>> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> >> (via Tom White)
>>> >>
>>>
>>
>>
>>
>> --
>> Best regards,
>>
>>    - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)



--
thanks
ashish

Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring API? JMX?

Joe Witt
Ashish,

Yes that sounds good and thanks for your interest in contributing.
Please feel free to ask questions early and often.

Thanks and look forward to working with you

Joe

On Thu, Feb 19, 2015 at 10:00 PM, Ashish <[hidden email]> wrote:

> Joe,
>
> other way round is to support it, but let the End User download and
> drop the binaries needed (not shipping them by default).
>
> I am keen on working to get the JMX support and based on my reading of
> the code so far, it would be a Reporting Task (shall keep it generic
> around Metrics) and add initially enable it only for JMX and later as
> we have more clarity, we can add more functions.
>
> Shall I create a JIRA for this?
>
> On Fri, Feb 20, 2015 at 3:15 AM, Joe Witt <[hidden email]> wrote:
>> yeah absolutely - i was just addressing the fact that there is a
>> licensing issue (in general) with that library.
>>
>> We can still totally do fun things with JMX, ganglia, etc..  Just
>> takes a little more work when the best libs aren't' readily useable.
>>
>> Thanks
>> Joe
>>
>> On Thu, Feb 19, 2015 at 4:42 PM, Andrew Purtell <[hidden email]> wrote:
>>> Would it suffice to not support Ganglia reporting? Then you'd not be
>>> bringing in metrics-ganglia* and its transitive dependencies?
>>>
>>> * - And can actively exclude it in POMs if necessary
>>>
>>>
>>>
>>> On Thu, Feb 19, 2015 at 1:39 PM, Joe Witt <[hidden email]> wrote:
>>>
>>>> Otis,
>>>>
>>>> We'll take a fresh look at JMX but if you want to help contribute in
>>>> this area that would be awesome as well.
>>>>
>>>> Regarding the Metric library and its licensing fitness:
>>>> ----------------------------
>>>>
>>>> It's important to keep in mind that transitive dependencies and their
>>>> licenses are just as important as direct dependencies.  If you start
>>>> here:
>>>>
>>>> https://github.com/dropwizard/metrics   [yep good this one is ASLv2]
>>>>   from that we want to use
>>>>
>>>> https://github.com/dropwizard/metrics/tree/master/metrics-ganglia [yep
>>>> good still ASLv2 from parent but let's look at deps]
>>>>   specifically this one: info.ganglia.gmetric4j:gmetric4j
>>>>
>>>> https://github.com/ganglia/gmetric4j [great it is BSD licensed and
>>>> thats ok too as it is Category-A but lets check its deps]
>>>>   specifically this one: org.acplt.remotetea:remotetea-oncrpc
>>>>
>>>> http://sourceforge.net/projects/remotetea/  [And this is where things
>>>> are problematic.  This one is LGPLv2 which is Category-X - not allowed
>>>> for either source or binary dependency]
>>>>   Upon further review the source does indeed depend on the remotetea
>>>> library so we can't simply ignore that dependency.  Which means we can
>>>> use the gmetric4j library and thus can't use metrics-ganglia library.
>>>>
>>>> Now this is just from the ganglia tree.  Other parts of the metrics
>>>> library might well be fine.  We're getting around this problem today
>>>> by using the older metrics library and its older ganglia support which
>>>> doesn't have this dependency licensing issue.
>>>>
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Thu, Feb 19, 2015 at 3:56 PM, Otis Gospodnetic
>>>> <[hidden email]> wrote:
>>>> > +1 to everything that Andrew said.
>>>> > I think there is actually no licensing issue with Metric library - it's
>>>> > ASLv2 - https://github.com/dropwizard/metrics/blob/master/LICENSE
>>>> >
>>>> > re JMX and auth stuff, we've documented it for SPM users here:
>>>> >
>>>> https://sematext.atlassian.net/wiki/display/PUBSPM/SPM+Monitor+-+Standalone
>>>> >
>>>> > Otis
>>>> > --
>>>> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>>> > Solr & Elasticsearch Support * http://sematext.com/
>>>> >
>>>> >
>>>> > On Thu, Feb 19, 2015 at 1:12 PM, Andrew Purtell <[hidden email]>
>>>> wrote:
>>>> >
>>>> >> > That should certainly be doable.  It just never seemed clear that JMX
>>>> >> > was the way to go.  I think the theory had always been that if we
>>>> >> > expose these sorts of things through the REST API over which we have
>>>> >> > clearly defined authentication/authorization schemes we could then
>>>> >> > support a variety of clients (people, systems).
>>>> >>
>>>> >> That's definitely true.
>>>> >>
>>>> >> What we do over on HBase is wire up our metrics to JMX and provide a
>>>> REST
>>>> >> endpoint in every process that presents all mbean data as JSON. We don't
>>>> >> set up JMX remote access, but let users edit configs for launch scripts
>>>> if
>>>> >> RMI JMX access is desired. This has some minor advantages in my opinion:
>>>> >> - We don't need to do anything special to expose JVM runtime data like
>>>> >> detailed information on GC pools and times.
>>>> >> - If it is desired for some reason, devs/admins then do have the option
>>>> to
>>>> >> set up the usual properties in the environment for RMI to JMX using the
>>>> >> standard options for auth and such.
>>>> >>
>>>> >> In my experience some shops do want to work with JMX, usually when other
>>>> >> systems that export operational data via JMX are already under
>>>> management.
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Feb 19, 2015 at 9:52 AM, Joe Witt <[hidden email]> wrote:
>>>> >>
>>>> >> > Andrew,
>>>> >> >
>>>> >> > That should certainly be doable.  It just never seemed clear that JMX
>>>> >> > was the way to go.  I think the theory had always been that if we
>>>> >> > expose these sorts of things through the REST API over which we have
>>>> >> > clearly defined authentication/authorization schemes we could then
>>>> >> > support a variety of clients (people, systems).
>>>> >> >
>>>> >> > I'll have to read up more on the options for security around JMX and
>>>> >> > the sort of client interactions that opens up.  That said, it would
>>>> >> > also be good to have a sense of tradeoffs of opening up further
>>>> >> > avenues of client interaction beyond the REST API.  Once we get our
>>>> >> > REST API docs legit again (had to pull due to licensing concerns) then
>>>> >> > that should help a lot too.
>>>> >> >
>>>> >> > Thanks
>>>> >> > Joe
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 19, 2015 at 12:42 PM, Andrew Purtell <[hidden email]
>>>> >
>>>> >> > wrote:
>>>> >> > > Would it be possible to instantiate a Metrics JmxReporter for each
>>>> >> > process?
>>>> >> > > Sounds like enabling metrics export via JMX could be an interesting
>>>> >> JIRA
>>>> >> > at
>>>> >> > > any rate.
>>>> >> > >
>>>> >> > > I did a quick search of the User, Developer, and Admin guides and
>>>> there
>>>> >> > are
>>>> >> > > no occurrences of the string "JMX"
>>>> >> > >
>>>> >> > > On Tue, Feb 17, 2015 at 10:25 PM, Otis Gospodnetic <
>>>> >> > > [hidden email]> wrote:
>>>> >> > >
>>>> >> > >> Hi Joe,
>>>> >> > >>
>>>> >> > >> Kafka devs describe their MBeans here -
>>>> >> > >> https://kafka.apache.org/08/ops.html
>>>> >> > >> - section 6.5, which is good for people who want to understand what
>>>> >> each
>>>> >> > >> metric means.
>>>> >> > >> It sounds like the reporting task stuff is all about the push.  I
>>>> >> think
>>>> >> > >> pull may be better for us, so if you can document the end point (I
>>>> >> > assume
>>>> >> > >> we are talking about an HTTP/REST API?  That returns various
>>>> metrics
>>>> >> in
>>>> >> > >> JSON?), that would be very very helpful (and not just to us).
>>>> >> > >>
>>>> >> > >> Thanks,
>>>> >> > >> Otis
>>>> >> > >> --
>>>> >> > >> Monitoring * Alerting * Anomaly Detection * Centralized Log
>>>> Management
>>>> >> > >> Solr & Elasticsearch Support * http://sematext.com/
>>>> >> > >>
>>>> >> > >>
>>>> >> > >> On Wed, Feb 18, 2015 at 12:50 AM, Joe Witt <[hidden email]>
>>>> >> wrote:
>>>> >> > >>
>>>> >> > >> > Otis,
>>>> >> > >> >
>>>> >> > >> > We're quite light on documentation thus far for the Reporting
>>>> Task
>>>> >> > >> > API.  But you can see a bit about it here:
>>>> >> > >> >
>>>> >> > >> >
>>>> >> > >> >
>>>> >> > >>
>>>> >> >
>>>> >>
>>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#reporting-tasks
>>>> >> > >> >
>>>> >> > >> >
>>>> >> > >>
>>>> >> >
>>>> >>
>>>> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#developing-a-reporting-task
>>>> >> > >> >
>>>> >> > >> > The javadocs are published to Maven as well so that will do
>>>> nicely
>>>> >> to
>>>> >> > >> > give a sense of what we expose from the Internals of NiFi to an
>>>> >> > >> > implementation of a reporting task.
>>>> >> > >> >
>>>> >> > >> > But feel free to ask detailed questions and as many as you want.
>>>> >> That
>>>> >> > >> > will give us good information on the types of things we need to
>>>> >> cover
>>>> >> > >> > and in how much detail.
>>>> >> > >> >
>>>> >> > >> > Thanks
>>>> >> > >> > Joe
>>>> >> > >> >
>>>> >> > >> > On Wed, Feb 18, 2015 at 12:46 AM, Otis Gospodnetic
>>>> >> > >> > <[hidden email]> wrote:
>>>> >> > >> > > Hi Joe,
>>>> >> > >> > >
>>>> >> > >> > > On Wed, Feb 18, 2015 at 12:15 AM, Joe Witt <[hidden email]
>>>> >
>>>> >> > wrote:
>>>> >> > >> > >
>>>> >> > >> > >> Otis,
>>>> >> > >> > >>
>>>> >> > >> > >> Glad you are liking it.
>>>> >> > >> > >>
>>>> >> > >> > >> We do provide support for both push and pull based metrics
>>>> >> > >> > >> reporting/gathering.
>>>> >> > >> > >>
>>>> >> > >> > >> For NiFi being able to push out metrics:
>>>> >> > >> > >> We do have an API for what we call "Reporting Tasks".  For
>>>> >> example
>>>> >> > we
>>>> >> > >> > >> have a Ganglia reporter today.
>>>> >> > >> > >>
>>>> >> > >> > >
>>>> >> > >> > > So we are talking about the Yammer/Coda Hale's Metrics being
>>>> used
>>>> >> > here?
>>>> >> > >> > > If so, are the metrics also available via JMX, too?
>>>> >> > >> > >
>>>> >> > >> > > For a client being able to pull metrics from NiFI:
>>>> >> > >> > >> The REST API is the way to go.
>>>> >> > >> > >>
>>>> >> > >> > >
>>>> >> > >> > > This sounds good.  Are this API and the metrics it exposes
>>>> >> > documented
>>>> >> > >> > > anywhere?
>>>> >> > >> > > I looked in the guides, but couldn't find anything.
>>>> >> > >> > >
>>>> >> > >> > >
>>>> >> > >> > >> Do either of these sound like a path that can work for you?
>>>> Let
>>>> >> us
>>>> >> > >> > >> know if you'd like more details.
>>>> >> > >> > >>
>>>> >> > >> > >
>>>> >> > >> > > Thanks,
>>>> >> > >> > > Otis
>>>> >> > >> > > --
>>>> >> > >> > > Monitoring * Alerting * Anomaly Detection * Centralized Log
>>>> >> > Management
>>>> >> > >> > > Solr & Elasticsearch Support * http://sematext.com/
>>>> >> > >> > >
>>>> >> > >> > >
>>>> >> > >> > >
>>>> >> > >> > >
>>>> >> > >> > >>
>>>> >> > >> > >> On Wed, Feb 18, 2015 at 12:01 AM, Otis Gospodnetic
>>>> >> > >> > >> <[hidden email]> wrote:
>>>> >> > >> > >> > Hi,
>>>> >> > >> > >> >
>>>> >> > >> > >> > Nifi looks mightily impressive!
>>>> >> > >> > >> > I see Nifi has built-in monitoring and metrics.... but is
>>>> there
>>>> >> > an
>>>> >> > >> API
>>>> >> > >> > >> that
>>>> >> > >> > >> > would allow something like SPM to add support for Nifi
>>>> >> > monitoring?
>>>> >> > >> Or
>>>> >> > >> > >> JMX?
>>>> >> > >> > >> >
>>>> >> > >> > >> > Thanks,
>>>> >> > >> > >> > Otis
>>>> >> > >> > >> > --
>>>> >> > >> > >> > Monitoring * Alerting * Anomaly Detection * Centralized Log
>>>> >> > >> Management
>>>> >> > >> > >> > Solr & Elasticsearch Support * http://sematext.com/
>>>> >> > >> > >>
>>>> >> > >> >
>>>> >> > >>
>>>> >> > >
>>>> >> > >
>>>> >> > >
>>>> >> > > --
>>>> >> > > Best regards,
>>>> >> > >
>>>> >> > >    - Andy
>>>> >> > >
>>>> >> > > Problems worthy of attack prove their worth by hitting back. - Piet
>>>> >> Hein
>>>> >> > > (via Tom White)
>>>> >> >
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Best regards,
>>>> >>
>>>> >>    - Andy
>>>> >>
>>>> >> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>>> >> (via Tom White)
>>>> >>
>>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>>    - Andy
>>>
>>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>>> (via Tom White)
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal