RecordReader and RecordWriter: development work on ruleengine

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

RecordReader and RecordWriter: development work on ruleengine

Uwe Geercken
Hello,
 
I would like to pick up development on my processor that uses a ruleengine. It is working since several month already but I want to convert the processor to use the new RecordReader/RecordWriter framework. This way I can use any data format, which is really, really cool and will benefit it's purpose.
 
What the processor does is to get a record and run it through the ruleengine to evaluate results based on business rules. The business rules are written in an external web application. So the benefit is, that the business logic is not embedded in the code (process). You can change the logic without touvhing the code/process. The ruleengine uses Java reflection to construct objects and run methods based on the rules and against the data. Using reflection I can instantiate any Java object, which makes the ruleengine universally usable. Btw: I have used the ruleengine with Pentaho ETL, Nifi, Kafka and Hadoop - and from Java code of course.
 
The ruleengine also knows "actions". Actions are fired based on the the result of rules (group of rules). So this is a way to fire an event based on logic. Again - all outside of the code. You can manipulate the records of the flowfile this way, e.g.
 
So I am also interested in the methods that RecordReader provides to get to the data. I am confident that I can make a really easy implementation that works for any data because the framework abstracts everything away that is about data format.
 
Now looking through code of the existing processors helps and is educational but to get started a documentation would be very helpful. Maybe you can point me to a sample or documentation which shows the usage of the api for RecordReader/RecordWriter and how I get a hold on the data/content of the flowfile.
 
Thanks for your time. Nifi is getting better and better. I am totally convinced that the ruleengine will benefit many people and help externalize logic, so that a proper division of responsibilities is possible: between those that write code/create flows and those that manage a complete set of logic, which is in fact the knowledge of a company. Central logic and clean code will benefit both quality and agility (time to implement changes).
 
Greetings,
 
Uwe
Reply | Threaded
Open this post in threaded view
|

Re: RecordReader and RecordWriter: development work on ruleengine

Bryan Bende
Uwe,

I don't think there is specific documentation on how to write code
using the record readers and writers, but the best example to look at
would be ConvertRecord....

ConvertRecord actually extends from AbstractRecordProcessor:

https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/AbstractRecordProcessor.java

AbstractRecordProcessor does the following:

- Create a stream callback for a flow file
- Create a reader from the input stream of the flow file
- Create a writer for the output stream of the flow file
- Read each record from the reader and pass it to the writer

For each record it delegates to a "process" method to let sub-classes
take action on the record, but ConvertRecord doesn't need to do
anything because the act of taking a record from a reader for format 1
and passing it to writer for format 2, is conversion in itself.

Hope that helps.

-Bryan


On Thu, Oct 5, 2017 at 2:14 PM, Uwe Geercken <[hidden email]> wrote:

> Hello,
>
> I would like to pick up development on my processor that uses a ruleengine. It is working since several month already but I want to convert the processor to use the new RecordReader/RecordWriter framework. This way I can use any data format, which is really, really cool and will benefit it's purpose.
>
> What the processor does is to get a record and run it through the ruleengine to evaluate results based on business rules. The business rules are written in an external web application. So the benefit is, that the business logic is not embedded in the code (process). You can change the logic without touvhing the code/process. The ruleengine uses Java reflection to construct objects and run methods based on the rules and against the data. Using reflection I can instantiate any Java object, which makes the ruleengine universally usable. Btw: I have used the ruleengine with Pentaho ETL, Nifi, Kafka and Hadoop - and from Java code of course.
>
> The ruleengine also knows "actions". Actions are fired based on the the result of rules (group of rules). So this is a way to fire an event based on logic. Again - all outside of the code. You can manipulate the records of the flowfile this way, e.g.
>
> So I am also interested in the methods that RecordReader provides to get to the data. I am confident that I can make a really easy implementation that works for any data because the framework abstracts everything away that is about data format.
>
> Now looking through code of the existing processors helps and is educational but to get started a documentation would be very helpful. Maybe you can point me to a sample or documentation which shows the usage of the api for RecordReader/RecordWriter and how I get a hold on the data/content of the flowfile.
>
> Thanks for your time. Nifi is getting better and better. I am totally convinced that the ruleengine will benefit many people and help externalize logic, so that a proper division of responsibilities is possible: between those that write code/create flows and those that manage a complete set of logic, which is in fact the knowledge of a company. Central logic and clean code will benefit both quality and agility (time to implement changes).
>
> Greetings,
>
> Uwe