Help with NiFi LookupRecord & MergeRecord Processors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with NiFi LookupRecord & MergeRecord Processors

matthewmichala@northwesternmutual.com
NiFi Developers,

I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just wondering if  anyone can help us with the NiFi LookupRecord, or MergeRecord Processors?

We keep getting this error when trying to use any Lookup Service (NiFi or 3rd party) to append additional attributes to our initial DB2 Query:

     Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup Service

Has anyone had any success merging data from 2 different sql queries based on a common Unique ID from both queries?   We've tried LookupRecord which uses the Lookup Service, and also tried MergeRecord, and MergeContent - nothing seems to work??

Bottom line is - we have an initial main query (ID & Status) - then want to append an additional attribute (Name) from a 2nd query (which just contains ID & Name) - onto each record of the main query - after joining on ID.

Whenever we try using LookupRecord, and pass the ID as a parameter via ? marks, we keep getting the above error.  We've gotten nowhere with MergeRecord too.  We have learned how to MergeContent, but that is not really what we want (is just a workaround).
Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668


Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668



This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.


Reply | Threaded
Open this post in threaded view
|

RE: Help with NiFi LookupRecord & MergeRecord Processors

matthewmichala@northwesternmutual.com
FYI, I did get the following responses from the Stack Overflow Nifi Tag Site - so until the DB Lookup Service is available, does anyone have any other workaround or custom built solutions:

Stack Overflow Reply 1:

When you use LookupRecord, what LookupService are you using? There currently is not a released implementation of a LookupService that works against a relatonial DB, although there is active work to implement one

Stack Overflow Reply 2:

MergeRecord understands records and can merge them together as proper JSON. In both cases though, merging is not joining, merging is writing content or records one after another to create larger flow files.



-----Original Message-----
From: [hidden email] <[hidden email]>
Sent: Monday, March 11, 2019 5:51 PM
To: [hidden email]
Subject: [EXTERNAL] Help with NiFi LookupRecord & MergeRecord Processors

NiFi Developers,

I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just wondering if  anyone can help us with the NiFi LookupRecord, or MergeRecord Processors?

We keep getting this error when trying to use any Lookup Service (NiFi or 3rd party) to append additional attributes to our initial DB2 Query:

     Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup Service

Has anyone had any success merging data from 2 different sql queries based on a common Unique ID from both queries?   We've tried LookupRecord which uses the Lookup Service, and also tried MergeRecord, and MergeContent - nothing seems to work??

Bottom line is - we have an initial main query (ID & Status) - then want to append an additional attribute (Name) from a 2nd query (which just contains ID & Name) - onto each record of the main query - after joining on ID.

Whenever we try using LookupRecord, and pass the ID as a parameter via ? marks, we keep getting the above error.  We've gotten nowhere with MergeRecord too.  We have learned how to MergeContent, but that is not really what we want (is just a workaround).
Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668


Thanks & Regards,

Matt Michala
Senior Engineer
Digital Workplace & Corp. Solutions
Northwestern Mutual (T07-150)
414-661-4668



This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.




This e-mail and any attachments may contain confidential information of Northwestern Mutual. If you are not the intended recipient of this message, be aware that any disclosure, copying, distribution or use of this e-mail and any attachments is prohibited. If you have received this e-mail in error, please notify Northwestern Mutual immediately by returning it to the sender and delete all copies from your system. Please be advised that communications with {SECURE MESSAGE} in the subject line have been sent using a secure messaging system. Communications that do not have this tag may not be secure and could be observed by a third party. Thank you for your cooperation.


Reply | Threaded
Open this post in threaded view
|

Re: Help with NiFi LookupRecord & MergeRecord Processors

Mike Thomsen
If you're not a Java developer and/or don't have access to any on your
team, your best work around would be to look at massaging your lookup data
into HBase, Mongo or ElasticSearch and using those. If you are a Java
developer  or have one on your team I can point you to some sample projects
you can hack up to quickly spin up a custom component that is tailored to
your needs.

On Mon, Mar 11, 2019 at 6:57 PM [hidden email] <
[hidden email]> wrote:

> FYI, I did get the following responses from the Stack Overflow Nifi Tag
> Site - so until the DB Lookup Service is available, does anyone have any
> other workaround or custom built solutions:
>
> Stack Overflow Reply 1:
>
> When you use LookupRecord, what LookupService are you using? There
> currently is not a released implementation of a LookupService that works
> against a relatonial DB, although there is active work to implement one
>
> Stack Overflow Reply 2:
>
> MergeRecord understands records and can merge them together as proper
> JSON. In both cases though, merging is not joining, merging is writing
> content or records one after another to create larger flow files.
>
>
>
> -----Original Message-----
> From: [hidden email] <
> [hidden email]>
> Sent: Monday, March 11, 2019 5:51 PM
> To: [hidden email]
> Subject: [EXTERNAL] Help with NiFi LookupRecord & MergeRecord Processors
>
> NiFi Developers,
>
> I'm a senior engineer at Northwestern Mutual in Milwaukee, and was just
> wondering if  anyone can help us with the NiFi LookupRecord, or MergeRecord
> Processors?
>
> We keep getting this error when trying to use any Lookup Service (NiFi or
> 3rd party) to append additional attributes to our initial DB2 Query:
>
>      Failed to lookup coordinates {LEGAL_ENTITY_ID_NUM=13366} in Lookup
> Service
>
> Has anyone had any success merging data from 2 different sql queries based
> on a common Unique ID from both queries?   We've tried LookupRecord which
> uses the Lookup Service, and also tried MergeRecord, and MergeContent -
> nothing seems to work??
>
> Bottom line is - we have an initial main query (ID & Status) - then want
> to append an additional attribute (Name) from a 2nd query (which just
> contains ID & Name) - onto each record of the main query - after joining on
> ID.
>
> Whenever we try using LookupRecord, and pass the ID as a parameter via ?
> marks, we keep getting the above error.  We've gotten nowhere with
> MergeRecord too.  We have learned how to MergeContent, but that is not
> really what we want (is just a workaround).
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
> Thanks & Regards,
>
> Matt Michala
> Senior Engineer
> Digital Workplace & Corp. Solutions
> Northwestern Mutual (T07-150)
> 414-661-4668
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>
>
> This e-mail and any attachments may contain confidential information of
> Northwestern Mutual. If you are not the intended recipient of this message,
> be aware that any disclosure, copying, distribution or use of this e-mail
> and any attachments is prohibited. If you have received this e-mail in
> error, please notify Northwestern Mutual immediately by returning it to the
> sender and delete all copies from your system. Please be advised that
> communications with {SECURE MESSAGE} in the subject line have been sent
> using a secure messaging system. Communications that do not have this tag
> may not be secure and could be observed by a third party. Thank you for
> your cooperation.
>
>
>