Creating a custom processor for JSON documents

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Creating a custom processor for JSON documents

István
Hi,

I was wondering what would be the best way of doing this. The basic
scenario is that we have a Kafka cluster that has lots of messages (JSON
documents). I would like to go though all of those messages and parse each
individual message to extract fields from it, like if you are thinking
about the HashMap representation of a JSON document I need a value of a
certain key. I need some filtering capabilities as well, only process
messages that has a certain key and the value of that key is configurable
in the UI.

I have seen that there are standard processors here:

https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard

Which one should I use as a kind of example to do this? Am I on the right
track that this is achievable with Nifi and I am not trying to do something
silly?

It seems that TransformXml.java would be a good start, using it as an
example to implement something like this.

Thank you in advance,
Istvan

--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

RE: Creating a custom processor for JSON documents

Mark Payne
Istvan,
I think this is a great use case for NiFi! The TransformXml processor could be applicable, but I would actually guide you more toward the EvaluateXPath. Specifically, I would break the problem into two processors:
EvaluateJsonPath and RouteOnAttribute.
EvaluateJsonPath would allow you to specify one or more "JsonPath" expressions and put the results into user-defined attributes.Then, you can route and filter on those attributes via the RouteOnAttribute processor.
I have actually considered doing something like this a few times, but I know there are a few different "JsonPath" types of languages/libs out there and wasn't sure which library made most sense to use.  I know there is one at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I would want to look a bit closer to make sure that all of its dependencies are also Apache-compatible, but that may be a good starting point.
Then, the user would configure this processor by adding their own properties. For instance, a property named "orderId" could have value "/order/orderId". That would create an attribute on the FlowFile named "orderId" whose value would be the result of evaluating the "/order/orderId" JsonPath. Then RouteOnAttribute could route on that attribute.
Does this make sense at all?
I would be happy to go into more detail or explain further with better examples if it doesn't make sense.
Thanks-Mark

> Date: Wed, 4 Feb 2015 14:08:22 -0800
> Subject: Creating a custom processor for JSON documents
> From: [hidden email]
> To: [hidden email]
>
> Hi,
>
> I was wondering what would be the best way of doing this. The basic
> scenario is that we have a Kafka cluster that has lots of messages (JSON
> documents). I would like to go though all of those messages and parse each
> individual message to extract fields from it, like if you are thinking
> about the HashMap representation of a JSON document I need a value of a
> certain key. I need some filtering capabilities as well, only process
> messages that has a certain key and the value of that key is configurable
> in the UI.
>
> I have seen that there are standard processors here:
>
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
>
> Which one should I use as a kind of example to do this? Am I on the right
> track that this is achievable with Nifi and I am not trying to do something
> silly?
>
> It seems that TransformXml.java would be a good start, using it as an
> example to implement something like this.
>
> Thank you in advance,
> Istvan
>
> --
> the sun shines for all
     
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

István
Mark,

This is exactly the feedback I was looking for. Thank you very much! I am
trying to play with it for a while and get back if I have any questions.

Regards,
Istvan


On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:

> Istvan,
> I think this is a great use case for NiFi! The TransformXml processor
> could be applicable, but I would actually guide you more toward the
> EvaluateXPath. Specifically, I would break the problem into two processors:
> EvaluateJsonPath and RouteOnAttribute.
> EvaluateJsonPath would allow you to specify one or more "JsonPath"
> expressions and put the results into user-defined attributes.Then, you can
> route and filter on those attributes via the RouteOnAttribute processor.
> I have actually considered doing something like this a few times, but I
> know there are a few different "JsonPath" types of languages/libs out there
> and wasn't sure which library made most sense to use.  I know there is one
> at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> would want to look a bit closer to make sure that all of its dependencies
> are also Apache-compatible, but that may be a good starting point.
> Then, the user would configure this processor by adding their own
> properties. For instance, a property named "orderId" could have value
> "/order/orderId". That would create an attribute on the FlowFile named
> "orderId" whose value would be the result of evaluating the
> "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> attribute.
> Does this make sense at all?
> I would be happy to go into more detail or explain further with better
> examples if it doesn't make sense.
> Thanks-Mark
>
> > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > Subject: Creating a custom processor for JSON documents
> > From: [hidden email]
> > To: [hidden email]
> >
> > Hi,
> >
> > I was wondering what would be the best way of doing this. The basic
> > scenario is that we have a Kafka cluster that has lots of messages (JSON
> > documents). I would like to go though all of those messages and parse
> each
> > individual message to extract fields from it, like if you are thinking
> > about the HashMap representation of a JSON document I need a value of a
> > certain key. I need some filtering capabilities as well, only process
> > messages that has a certain key and the value of that key is configurable
> > in the UI.
> >
> > I have seen that there are standard processors here:
> >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> >
> > Which one should I use as a kind of example to do this? Am I on the right
> > track that this is achievable with Nifi and I am not trying to do
> something
> > silly?
> >
> > It seems that TransformXml.java would be a good start, using it as an
> > example to implement something like this.
> >
> > Thank you in advance,
> > Istvan
> >
> > --
> > the sun shines for all
>
>



--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

Bryan Bende
Not sure if this is helpful, but I recently attempted writing a processor
to rename and/or exclude fields from JSON documents. My thought being that
one system is producing documents that need to be slightly modified before
sending to another system.

https://github.com/bbende/nifi-example-bundles/blob/master/nifi-example-utils-bundle/nifi-example-utils-processors/src/main/java/org/apache/nifi/processors/example/utils/RenameJSONFields.java

I'll put a big disclaimer on this and say that it hasn't really been tested
other than the unit tests, but it might be helpful in getting you started
writing your own EvaluateJsonPath processor.

On Wed, Feb 4, 2015 at 5:25 PM, István <[hidden email]> wrote:

> Mark,
>
> This is exactly the feedback I was looking for. Thank you very much! I am
> trying to play with it for a while and get back if I have any questions.
>
> Regards,
> Istvan
>
>
> On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
>
> > Istvan,
> > I think this is a great use case for NiFi! The TransformXml processor
> > could be applicable, but I would actually guide you more toward the
> > EvaluateXPath. Specifically, I would break the problem into two
> processors:
> > EvaluateJsonPath and RouteOnAttribute.
> > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > expressions and put the results into user-defined attributes.Then, you
> can
> > route and filter on those attributes via the RouteOnAttribute processor.
> > I have actually considered doing something like this a few times, but I
> > know there are a few different "JsonPath" types of languages/libs out
> there
> > and wasn't sure which library made most sense to use.  I know there is
> one
> > at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> > would want to look a bit closer to make sure that all of its dependencies
> > are also Apache-compatible, but that may be a good starting point.
> > Then, the user would configure this processor by adding their own
> > properties. For instance, a property named "orderId" could have value
> > "/order/orderId". That would create an attribute on the FlowFile named
> > "orderId" whose value would be the result of evaluating the
> > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > attribute.
> > Does this make sense at all?
> > I would be happy to go into more detail or explain further with better
> > examples if it doesn't make sense.
> > Thanks-Mark
> >
> > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > Subject: Creating a custom processor for JSON documents
> > > From: [hidden email]
> > > To: [hidden email]
> > >
> > > Hi,
> > >
> > > I was wondering what would be the best way of doing this. The basic
> > > scenario is that we have a Kafka cluster that has lots of messages
> (JSON
> > > documents). I would like to go though all of those messages and parse
> > each
> > > individual message to extract fields from it, like if you are thinking
> > > about the HashMap representation of a JSON document I need a value of a
> > > certain key. I need some filtering capabilities as well, only process
> > > messages that has a certain key and the value of that key is
> configurable
> > > in the UI.
> > >
> > > I have seen that there are standard processors here:
> > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > >
> > > Which one should I use as a kind of example to do this? Am I on the
> right
> > > track that this is achievable with Nifi and I am not trying to do
> > something
> > > silly?
> > >
> > > It seems that TransformXml.java would be a good start, using it as an
> > > example to implement something like this.
> > >
> > > Thank you in advance,
> > > Istvan
> > >
> > > --
> > > the sun shines for all
> >
> >
>
>
>
> --
> the sun shines for all
>
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

István
Thank you Bryan,

Much appreciated!

I.

On Wed, Feb 4, 2015 at 2:51 PM, Bryan Bende <[hidden email]> wrote:

> Not sure if this is helpful, but I recently attempted writing a processor
> to rename and/or exclude fields from JSON documents. My thought being that
> one system is producing documents that need to be slightly modified before
> sending to another system.
>
>
> https://github.com/bbende/nifi-example-bundles/blob/master/nifi-example-utils-bundle/nifi-example-utils-processors/src/main/java/org/apache/nifi/processors/example/utils/RenameJSONFields.java
>
> I'll put a big disclaimer on this and say that it hasn't really been tested
> other than the unit tests, but it might be helpful in getting you started
> writing your own EvaluateJsonPath processor.
>
> On Wed, Feb 4, 2015 at 5:25 PM, István <[hidden email]> wrote:
>
> > Mark,
> >
> > This is exactly the feedback I was looking for. Thank you very much! I am
> > trying to play with it for a while and get back if I have any questions.
> >
> > Regards,
> > Istvan
> >
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> > processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> > can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> > there
> > > and wasn't sure which library made most sense to use.  I know there is
> > one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> > (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> > configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> > right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
> >
>



--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

István
In reply to this post by Mark Payne
Hi Mark,

I read through the EvaluateXPath.java yesterday. I have few (I guess
newbie) questions. I hope you don't mind.

I have created the new EvaluateJSON.java file and started to write the
parser. I am wondering what is the best way of recompiling the standard
processors and try out the newly created feature. I did a mvn -T C2.0
-DskipTests=true install but the new file is not showing up in the menu
when I drop the processor to the worksheet. I was wondering what makes Nifi
to accept a new processor. Do I need to add anything else somewhere to let
the system know about a new file like that?

Thank you in advance.

Best regards,
Istvan

On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:

> Istvan,
> I think this is a great use case for NiFi! The TransformXml processor
> could be applicable, but I would actually guide you more toward the
> EvaluateXPath. Specifically, I would break the problem into two processors:
> EvaluateJsonPath and RouteOnAttribute.
> EvaluateJsonPath would allow you to specify one or more "JsonPath"
> expressions and put the results into user-defined attributes.Then, you can
> route and filter on those attributes via the RouteOnAttribute processor.
> I have actually considered doing something like this a few times, but I
> know there are a few different "JsonPath" types of languages/libs out there
> and wasn't sure which library made most sense to use.  I know there is one
> at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> would want to look a bit closer to make sure that all of its dependencies
> are also Apache-compatible, but that may be a good starting point.
> Then, the user would configure this processor by adding their own
> properties. For instance, a property named "orderId" could have value
> "/order/orderId". That would create an attribute on the FlowFile named
> "orderId" whose value would be the result of evaluating the
> "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> attribute.
> Does this make sense at all?
> I would be happy to go into more detail or explain further with better
> examples if it doesn't make sense.
> Thanks-Mark
>
> > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > Subject: Creating a custom processor for JSON documents
> > From: [hidden email]
> > To: [hidden email]
> >
> > Hi,
> >
> > I was wondering what would be the best way of doing this. The basic
> > scenario is that we have a Kafka cluster that has lots of messages (JSON
> > documents). I would like to go though all of those messages and parse
> each
> > individual message to extract fields from it, like if you are thinking
> > about the HashMap representation of a JSON document I need a value of a
> > certain key. I need some filtering capabilities as well, only process
> > messages that has a certain key and the value of that key is configurable
> > in the UI.
> >
> > I have seen that there are standard processors here:
> >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> >
> > Which one should I use as a kind of example to do this? Am I on the right
> > track that this is achievable with Nifi and I am not trying to do
> something
> > silly?
> >
> > It seems that TransformXml.java would be a good start, using it as an
> > example to implement something like this.
> >
> > Thank you in advance,
> > Istvan
> >
> > --
> > the sun shines for all
>
>



--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

Aldrin Piri
Istvan,

Have you included the processor class in the
org.apache.nifi.processor.Processor
file in the services directory?  The accompanying reference in the
developer guide can be found at
http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api
.



On Wed, Feb 11, 2015 at 1:31 PM, István <[hidden email]> wrote:

> Hi Mark,
>
> I read through the EvaluateXPath.java yesterday. I have few (I guess
> newbie) questions. I hope you don't mind.
>
> I have created the new EvaluateJSON.java file and started to write the
> parser. I am wondering what is the best way of recompiling the standard
> processors and try out the newly created feature. I did a mvn -T C2.0
> -DskipTests=true install but the new file is not showing up in the menu
> when I drop the processor to the worksheet. I was wondering what makes Nifi
> to accept a new processor. Do I need to add anything else somewhere to let
> the system know about a new file like that?
>
> Thank you in advance.
>
> Best regards,
> Istvan
>
> On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
>
> > Istvan,
> > I think this is a great use case for NiFi! The TransformXml processor
> > could be applicable, but I would actually guide you more toward the
> > EvaluateXPath. Specifically, I would break the problem into two
> processors:
> > EvaluateJsonPath and RouteOnAttribute.
> > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > expressions and put the results into user-defined attributes.Then, you
> can
> > route and filter on those attributes via the RouteOnAttribute processor.
> > I have actually considered doing something like this a few times, but I
> > know there are a few different "JsonPath" types of languages/libs out
> there
> > and wasn't sure which library made most sense to use.  I know there is
> one
> > at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> > would want to look a bit closer to make sure that all of its dependencies
> > are also Apache-compatible, but that may be a good starting point.
> > Then, the user would configure this processor by adding their own
> > properties. For instance, a property named "orderId" could have value
> > "/order/orderId". That would create an attribute on the FlowFile named
> > "orderId" whose value would be the result of evaluating the
> > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > attribute.
> > Does this make sense at all?
> > I would be happy to go into more detail or explain further with better
> > examples if it doesn't make sense.
> > Thanks-Mark
> >
> > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > Subject: Creating a custom processor for JSON documents
> > > From: [hidden email]
> > > To: [hidden email]
> > >
> > > Hi,
> > >
> > > I was wondering what would be the best way of doing this. The basic
> > > scenario is that we have a Kafka cluster that has lots of messages
> (JSON
> > > documents). I would like to go though all of those messages and parse
> > each
> > > individual message to extract fields from it, like if you are thinking
> > > about the HashMap representation of a JSON document I need a value of a
> > > certain key. I need some filtering capabilities as well, only process
> > > messages that has a certain key and the value of that key is
> configurable
> > > in the UI.
> > >
> > > I have seen that there are standard processors here:
> > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > >
> > > Which one should I use as a kind of example to do this? Am I on the
> right
> > > track that this is achievable with Nifi and I am not trying to do
> > something
> > > silly?
> > >
> > > It seems that TransformXml.java would be a good start, using it as an
> > > example to implement something like this.
> > >
> > > Thank you in advance,
> > > Istvan
> > >
> > > --
> > > the sun shines for all
> >
> >
>
>
>
> --
> the sun shines for all
>
Reply | Threaded
Open this post in threaded view
|

RE: Creating a custom processor for JSON documents

Mark Payne
In reply to this post by István
Istvan,

NiFi Processors are loaded using Java's Service Loader Mechanism. There is a file in the standard-processors module named src/main/resources/META-INF/services/org.apache.nifi.processor.Processor.

The new class needs to be added to that file.

For more information, you can reference the developer guide: see second paragraph http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api

Does this give you what you need?

Thanks
-Mark

> Date: Wed, 11 Feb 2015 10:31:47 -0800
> Subject: Re: Creating a custom processor for JSON documents
> From: [hidden email]
> To: [hidden email]; [hidden email]
>
> Hi Mark,
>
> I read through the EvaluateXPath.java yesterday. I have few (I guess
> newbie) questions. I hope you don't mind.
>
> I have created the new EvaluateJSON.java file and started to write the
> parser. I am wondering what is the best way of recompiling the standard
> processors and try out the newly created feature. I did a mvn -T C2.0
> -DskipTests=true install but the new file is not showing up in the menu
> when I drop the processor to the worksheet. I was wondering what makes Nifi
> to accept a new processor. Do I need to add anything else somewhere to let
> the system know about a new file like that?
>
> Thank you in advance.
>
> Best regards,
> Istvan
>
> On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
>
> > Istvan,
> > I think this is a great use case for NiFi! The TransformXml processor
> > could be applicable, but I would actually guide you more toward the
> > EvaluateXPath. Specifically, I would break the problem into two processors:
> > EvaluateJsonPath and RouteOnAttribute.
> > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > expressions and put the results into user-defined attributes.Then, you can
> > route and filter on those attributes via the RouteOnAttribute processor.
> > I have actually considered doing something like this a few times, but I
> > know there are a few different "JsonPath" types of languages/libs out there
> > and wasn't sure which library made most sense to use.  I know there is one
> > at https://github.com/jayway/JsonPath that has an Apache 2.0 License. I
> > would want to look a bit closer to make sure that all of its dependencies
> > are also Apache-compatible, but that may be a good starting point.
> > Then, the user would configure this processor by adding their own
> > properties. For instance, a property named "orderId" could have value
> > "/order/orderId". That would create an attribute on the FlowFile named
> > "orderId" whose value would be the result of evaluating the
> > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > attribute.
> > Does this make sense at all?
> > I would be happy to go into more detail or explain further with better
> > examples if it doesn't make sense.
> > Thanks-Mark
> >
> > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > Subject: Creating a custom processor for JSON documents
> > > From: [hidden email]
> > > To: [hidden email]
> > >
> > > Hi,
> > >
> > > I was wondering what would be the best way of doing this. The basic
> > > scenario is that we have a Kafka cluster that has lots of messages (JSON
> > > documents). I would like to go though all of those messages and parse
> > each
> > > individual message to extract fields from it, like if you are thinking
> > > about the HashMap representation of a JSON document I need a value of a
> > > certain key. I need some filtering capabilities as well, only process
> > > messages that has a certain key and the value of that key is configurable
> > > in the UI.
> > >
> > > I have seen that there are standard processors here:
> > >
> > >
> > https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > >
> > > Which one should I use as a kind of example to do this? Am I on the right
> > > track that this is achievable with Nifi and I am not trying to do
> > something
> > > silly?
> > >
> > > It seems that TransformXml.java would be a good start, using it as an
> > > example to implement something like this.
> > >
> > > Thank you in advance,
> > > Istvan
> > >
> > > --
> > > the sun shines for all
> >
> >
>
>
>
> --
> the sun shines for all
     
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

István
Yes, thank you Mark, this is exactly it.

I.

On Wed, Feb 11, 2015 at 10:51 AM, Mark Payne <[hidden email]> wrote:

> Istvan,
>
> NiFi Processors are loaded using Java's Service Loader Mechanism. There is
> a file in the standard-processors module named
> src/main/resources/META-INF/services/org.apache.nifi.processor.Processor.
>
> The new class needs to be added to that file.
>
> For more information, you can reference the developer guide: see second
> paragraph
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api
>
> Does this give you what you need?
>
> Thanks
> -Mark
>
> > Date: Wed, 11 Feb 2015 10:31:47 -0800
> > Subject: Re: Creating a custom processor for JSON documents
> > From: [hidden email]
> > To: [hidden email]; [hidden email]
> >
> > Hi Mark,
> >
> > I read through the EvaluateXPath.java yesterday. I have few (I guess
> > newbie) questions. I hope you don't mind.
> >
> > I have created the new EvaluateJSON.java file and started to write the
> > parser. I am wondering what is the best way of recompiling the standard
> > processors and try out the newly created feature. I did a mvn -T C2.0
> > -DskipTests=true install but the new file is not showing up in the menu
> > when I drop the processor to the worksheet. I was wondering what makes
> Nifi
> > to accept a new processor. Do I need to add anything else somewhere to
> let
> > the system know about a new file like that?
> >
> > Thank you in advance.
> >
> > Best regards,
> > Istvan
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> there
> > > and wasn't sure which library made most sense to use.  I know there is
> one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
>
>



--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

István
In reply to this post by Aldrin Piri
Aldrin,

I think this might be it. I forgot to do that.

Thanks!

On Wed, Feb 11, 2015 at 10:51 AM, Aldrin Piri <[hidden email]> wrote:

> Istvan,
>
> Have you included the processor class in the
> org.apache.nifi.processor.Processor
> file in the services directory?  The accompanying reference in the
> developer guide can be found at
>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api
> .
>
>
>
> On Wed, Feb 11, 2015 at 1:31 PM, István <[hidden email]> wrote:
>
> > Hi Mark,
> >
> > I read through the EvaluateXPath.java yesterday. I have few (I guess
> > newbie) questions. I hope you don't mind.
> >
> > I have created the new EvaluateJSON.java file and started to write the
> > parser. I am wondering what is the best way of recompiling the standard
> > processors and try out the newly created feature. I did a mvn -T C2.0
> > -DskipTests=true install but the new file is not showing up in the menu
> > when I drop the processor to the worksheet. I was wondering what makes
> Nifi
> > to accept a new processor. Do I need to add anything else somewhere to
> let
> > the system know about a new file like that?
> >
> > Thank you in advance.
> >
> > Best regards,
> > Istvan
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> > processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> > can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> > there
> > > and wasn't sure which library made most sense to use.  I know there is
> > one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> > (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> > configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> > right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
> >
>



--
the sun shines for all
Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

Phillip G
Just for future use bryanbende.com who wrote the arche type has a post on developing processors, and there is another on Developing A Custom Apache Nifi Processor (JSON) - NiFi.rocks
|   |
|   |  |   |   |   |   |   |
| bryanbende.comThis post will cover the initial steps to get started developing a custom processor for Apache NiFi. The focus will be on setting up a new processor bundle and and... |
|  |
| View on bryanbende.com | Preview by Yahoo |
|  |
|   |

  
|   |
|   |   |   |   |   |
| Developing A Custom Apache Nifi Processor (JSON) - NiFi.rocksHow to develop a custom Apache Nifi processor. The example in this post will utilize json, and json path |
|  |
| View on www.nifi.rocks | Preview by Yahoo |
|  |
|   |

  

     On Wednesday, February 11, 2015 2:26 PM, István <[hidden email]> wrote:
   

 Aldrin,

I think this might be it. I forgot to do that.

Thanks!

On Wed, Feb 11, 2015 at 10:51 AM, Aldrin Piri <[hidden email]> wrote:

> Istvan,
>
> Have you included the processor class in the
> org.apache.nifi.processor.Processor
> file in the services directory?  The accompanying reference in the
> developer guide can be found at
>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api
> .
>
>
>
> On Wed, Feb 11, 2015 at 1:31 PM, István <[hidden email]> wrote:
>
> > Hi Mark,
> >
> > I read through the EvaluateXPath.java yesterday. I have few (I guess
> > newbie) questions. I hope you don't mind.
> >
> > I have created the new EvaluateJSON.java file and started to write the
> > parser. I am wondering what is the best way of recompiling the standard
> > processors and try out the newly created feature. I did a mvn -T C2.0
> > -DskipTests=true install but the new file is not showing up in the menu
> > when I drop the processor to the worksheet. I was wondering what makes
> Nifi
> > to accept a new processor. Do I need to add anything else somewhere to
> let
> > the system know about a new file like that?
> >
> > Thank you in advance.
> >
> > Best regards,
> > Istvan
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> > processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> > can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> > there
> > > and wasn't sure which library made most sense to use.  I know there is
> > one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> > (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> > configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> > right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
> >
>



--
the sun shines for all

Reply | Threaded
Open this post in threaded view
|

Re: Creating a custom processor for JSON documents

Mark Payne
In reply to this post by István
In general, I would recommend looking to the NiFi Developer Guide (http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html ) and the standard processors as your examples. These processors have all been used in production for quite a while.






From: Phillip G
Sent: ‎Wednesday‎, ‎February‎ ‎11‎, ‎2015 ‎7‎:‎06‎ ‎PM
To: [hidden email]
Cc: Mark Payne





Just for future use bryanbende.com who wrote the arche type has a post on developing processors, and there is another on Developing A Custom Apache Nifi Processor (JSON) - NiFi.rocks
|   |
|   |  |   |   |   |   |   |
| bryanbende.comThis post will cover the initial steps to get started developing a custom processor for Apache NiFi. The focus will be on setting up a new processor bundle and and... |
|  |
| View on bryanbende.com | Preview by Yahoo |
|  |
|   |

 
|   |
|   |   |   |   |   |
| Developing A Custom Apache Nifi Processor (JSON) - NiFi.rocksHow to develop a custom Apache Nifi processor. The example in this post will utilize json, and json path |
|  |
| View on www.nifi.rocks | Preview by Yahoo |
|  |
|   |

   

     On Wednesday, February 11, 2015 2:26 PM, István <[hidden email]> wrote:
   

 Aldrin,

I think this might be it. I forgot to do that.

Thanks!

On Wed, Feb 11, 2015 at 10:51 AM, Aldrin Piri <[hidden email]> wrote:

> Istvan,
>
> Have you included the processor class in the
> org.apache.nifi.processor.Processor
> file in the services directory?  The accompanying reference in the
> developer guide can be found at
>
> http://nifi.incubator.apache.org/docs/nifi-docs/developer-guide.html#processor_api
> .
>
>
>
> On Wed, Feb 11, 2015 at 1:31 PM, István <[hidden email]> wrote:
>
> > Hi Mark,
> >
> > I read through the EvaluateXPath.java yesterday. I have few (I guess
> > newbie) questions. I hope you don't mind.
> >
> > I have created the new EvaluateJSON.java file and started to write the
> > parser. I am wondering what is the best way of recompiling the standard
> > processors and try out the newly created feature. I did a mvn -T C2.0
> > -DskipTests=true install but the new file is not showing up in the menu
> > when I drop the processor to the worksheet. I was wondering what makes
> Nifi
> > to accept a new processor. Do I need to add anything else somewhere to
> let
> > the system know about a new file like that?
> >
> > Thank you in advance.
> >
> > Best regards,
> > Istvan
> >
> > On Wed, Feb 4, 2015 at 2:22 PM, Mark Payne <[hidden email]> wrote:
> >
> > > Istvan,
> > > I think this is a great use case for NiFi! The TransformXml processor
> > > could be applicable, but I would actually guide you more toward the
> > > EvaluateXPath. Specifically, I would break the problem into two
> > processors:
> > > EvaluateJsonPath and RouteOnAttribute.
> > > EvaluateJsonPath would allow you to specify one or more "JsonPath"
> > > expressions and put the results into user-defined attributes.Then, you
> > can
> > > route and filter on those attributes via the RouteOnAttribute
> processor.
> > > I have actually considered doing something like this a few times, but I
> > > know there are a few different "JsonPath" types of languages/libs out
> > there
> > > and wasn't sure which library made most sense to use.  I know there is
> > one
> > > at https://github.com/jayway/JsonPath that has an Apache 2.0 License.
> I
> > > would want to look a bit closer to make sure that all of its
> dependencies
> > > are also Apache-compatible, but that may be a good starting point.
> > > Then, the user would configure this processor by adding their own
> > > properties. For instance, a property named "orderId" could have value
> > > "/order/orderId". That would create an attribute on the FlowFile named
> > > "orderId" whose value would be the result of evaluating the
> > > "/order/orderId" JsonPath. Then RouteOnAttribute could route on that
> > > attribute.
> > > Does this make sense at all?
> > > I would be happy to go into more detail or explain further with better
> > > examples if it doesn't make sense.
> > > Thanks-Mark
> > >
> > > > Date: Wed, 4 Feb 2015 14:08:22 -0800
> > > > Subject: Creating a custom processor for JSON documents
> > > > From: [hidden email]
> > > > To: [hidden email]
> > > >
> > > > Hi,
> > > >
> > > > I was wondering what would be the best way of doing this. The basic
> > > > scenario is that we have a Kafka cluster that has lots of messages
> > (JSON
> > > > documents). I would like to go though all of those messages and parse
> > > each
> > > > individual message to extract fields from it, like if you are
> thinking
> > > > about the HashMap representation of a JSON document I need a value
> of a
> > > > certain key. I need some filtering capabilities as well, only process
> > > > messages that has a certain key and the value of that key is
> > configurable
> > > > in the UI.
> > > >
> > > > I have seen that there are standard processors here:
> > > >
> > > >
> > >
> >
> https://github.com/apache/incubator-nifi/tree/develop/nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard
> > > >
> > > > Which one should I use as a kind of example to do this? Am I on the
> > right
> > > > track that this is achievable with Nifi and I am not trying to do
> > > something
> > > > silly?
> > > >
> > > > It seems that TransformXml.java would be a good start, using it as an
> > > > example to implement something like this.
> > > >
> > > > Thank you in advance,
> > > > Istvan
> > > >
> > > > --
> > > > the sun shines for all
> > >
> > >
> >
> >
> >
> > --
> > the sun shines for all
> >
>



--
the sun shines for all