Apache NiFI - Atlas Bridge Builders Needed

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache NiFI - Atlas Bridge Builders Needed

Lawrance, Greg J JTST:EX
We are working with Apache NiFi and Apache Atlas (within the Hortonworks
HDP framework) and we are looking for some paid help to develop something
that meets our needs.  We posted the opportunity as an issue on GitHub
associated with the repo where code would be delivered

https://github.com/bcgov/nifi-atlas/issues/1 .


 Any suggestions of where
we might advertise / share this information would be greatly appreciated.

Thanks in advance.

Greg

Reply | Threaded
Open this post in threaded view
|

Re: Apache NiFI - Atlas Bridge Builders Needed

Koji Kawamura-2
Hello Lawrance,

I've been working on NIFI-3709, adding a Apache NiFi reporting task to
report lineage to Apache Atlas.
The original PullRequest did not use Provenance events, but I'm now
adding it so that it can cover more use-cases.

It's still under development and not ready to be reviewed fully, but
you can look at the code to see if it covers your needs, I hope it
does, if not, please let me know.
https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task

The core part of analyzing NiFi Provenance events is implemented in
this package, and extensible.
An analyzer is picked by following matching rules:
1. The component type that emitted the provenance event
2. The transit URI of provenance event
3. The event type of the provenance event

For general protocols such as hdfs or hbase can be covered by #2.
(http, jdbc and such can be done similarly in the future)
If specific implementation is needed, it can be done as #1.

https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/provenance/analyzer

The work also has a mechanism to resolve Atlas cluster name from
hostname/ip address.
User can define mapping of those at the ReportingTask's configuration.
https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nifi-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atlas/resolver

Thanks,
Koji


On Wed, Sep 6, 2017 at 6:00 AM, Lawrance, Greg J JTST:EX
<[hidden email]> wrote:

> We are working with Apache NiFi and Apache Atlas (within the Hortonworks
> HDP framework) and we are looking for some paid help to develop something
> that meets our needs.  We posted the opportunity as an issue on GitHub
> associated with the repo where code would be delivered
>
> https://github.com/bcgov/nifi-atlas/issues/1 .
>
>
>  Any suggestions of where
> we might advertise / share this information would be greatly appreciated.
>
> Thanks in advance.
>
> Greg
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache NiFI - Atlas Bridge Builders Needed

Lawrance, Greg J JTST:EX
Koji,

Thank you very much for this! It looks very promising - and may go a long
way towards resolving our problem. We will give it a more thorough review
/ trial over the next few days.

Greg


On 2017-09-05, 5:30 PM, "Koji Kawamura" <[hidden email]> wrote:

>Hello Lawrance,
>
>I've been working on NIFI-3709, adding a Apache NiFi reporting task to
>report lineage to Apache Atlas.
>The original PullRequest did not use Provenance events, but I'm now
>adding it so that it can cover more use-cases.
>
>It's still under development and not ready to be reviewed fully, but
>you can look at the code to see if it covers your needs, I hope it
>does, if not, please let me know.
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task
>
>The core part of analyzing NiFi Provenance events is implemented in
>this package, and extensible.
>An analyzer is picked by following matching rules:
>1. The component type that emitted the provenance event
>2. The transit URI of provenance event
>3. The event type of the provenance event
>
>For general protocols such as hdfs or hbase can be covered by #2.
>(http, jdbc and such can be done similarly in the future)
>If specific implementation is needed, it can be done as #1.
>
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atl
>as/provenance/analyzer
>
>The work also has a mechanism to resolve Atlas cluster name from
>hostname/ip address.
>User can define mapping of those at the ReportingTask's configuration.
>https://github.com/ijokarumawak/nifi/tree/nifi-3709-2/nifi-nar-bundles/nif
>i-atlas-bundle/nifi-atlas-reporting-task/src/main/java/org/apache/nifi/atl
>as/resolver
>
>Thanks,
>Koji
>
>
>On Wed, Sep 6, 2017 at 6:00 AM, Lawrance, Greg J JTST:EX
><[hidden email]> wrote:
>> We are working with Apache NiFi and Apache Atlas (within the Hortonworks
>> HDP framework) and we are looking for some paid help to develop
>>something
>> that meets our needs.  We posted the opportunity as an issue on GitHub
>> associated with the repo where code would be delivered
>>
>> https://github.com/bcgov/nifi-atlas/issues/1 .
>>
>>
>>  Any suggestions of where
>> we might advertise / share this information would be greatly
>>appreciated.
>>
>> Thanks in advance.
>>
>> Greg
>>