Lookup services for cleanup and other tasks

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Lookup services for cleanup and other tasks

Mike Thomsen
A lot of the data we get is very dirty and in need of cleanup to such a
point that UpdateRecord cannot handle it. So we use lookup services and
LookupRecord to do things like normalize certain fields that are often
wildly off from what we need. I wrote this processor over the weekend, and
am wondering if I should add it to a PR:

https://github.com/MikeThomsen/nifi-record-enhancement-bundle

It's similar to LookupRecord, but it's designed for one purpose: let a user
apply a bunch of cleanup tasks in a single hop through the NiFi graph.

Any interest?

Thanks,

Mike
Reply | Threaded
Open this post in threaded view
|

Re: Lookup services for cleanup and other tasks

Andy LoPresto-2
Mike,

This looks useful to me as an offering, but also very complex and almost abstracting an arbitrary DSL out. One of the patterns of NiFi has been the Unix tool model of each component doing a single activity and separating responsibilities between “black box” processors. This seems a bit complex to add into core NiFi, but I think it would be great to offer from your repository for users who could take advantage of it. Perhaps with the coming Extension Registry, a component marketplace will emerge and it will gain traction there.

Thanks.


Andy LoPresto
[hidden email]
[hidden email]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jan 21, 2019, at 9:45 AM, Mike Thomsen <[hidden email]> wrote:
>
> A lot of the data we get is very dirty and in need of cleanup to such a
> point that UpdateRecord cannot handle it. So we use lookup services and
> LookupRecord to do things like normalize certain fields that are often
> wildly off from what we need. I wrote this processor over the weekend, and
> am wondering if I should add it to a PR:
>
> https://github.com/MikeThomsen/nifi-record-enhancement-bundle
>
> It's similar to LookupRecord, but it's designed for one purpose: let a user
> apply a bunch of cleanup tasks in a single hop through the NiFi graph.
>
> Any interest?
>
> Thanks,
>
> Mike