Lineage in custom nodes that aren't directly connected

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view

Lineage in custom nodes that aren't directly connected

Hi developers,

I have the following setup:
A sender node reads data from a flow file and puts that via messages into a Kafka topic. It appends the UUID of the flow-file to each message created.
The messages are then processed by someone else and a follow-up message is created and put back to Kafka (another topic).
A collector node reads data from the response topic in Kafka and creates a flow file per UUID found in the response. I use {{session.getProvenanceReporter().fetch(...)}} to report that there is a newly created flow file. Also, I set the property "parent.uuid" using {{session.putAttribute( flowFile, "parent.uuid", sourceUuid )}}.

Once data is processed and viewing the provenance graph, I observe the following:
- The sender stops reporting after the message was sent to Kafka
- The collector starts reporting from the Kafka receiving point onwards with a fetch event
- The parent.uuid information of the collector is correctly displayed within the collector stream and can be used to manually search within the Nifi UI for the original flow file. This works.

What I want to achieve is that the flow file sent to Kafka isn't stopping in the provenance graph -- it shall contain the graph and therewith the created final flow files of the collector as well. How can I achieve this?