Bug/Issue with ReplaceTextWithMapping

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug/Issue with ReplaceTextWithMapping

Ameer Mawia
Guys,

It seems that NIFI  ReplaceTextWithMapping   Processors has a BUG with
Refreshing its Mapped file. We are using its functionality in PROD and
getting odd behaviour.

Our USAGE Scenario:

   - We use NIFI primarily as a TRANSFORMATION Tool.
   - Our flow involves:


   1. Getting a raw csv file.
      2. Split the file on per line basis:
         1. So from one source flow file - we may 10000 flows
         generated/splitted out.
      3. For each of the splitted flow file(flowfiles for individual lines)
      we perform transformation on the attributes.
      4. We merge these flowfiles back and write the Output file.


As part of the transformation in Step#4, we do some mapping for one of the
field in the csv. For this we use ReplaceTextWithMapping  Processor. Also
to note we update ourmapping file just before starting our flow(ie. Step #1)

Our Issue:


   - We have noted for SAME key we get two DIFFERENT values in two
   different flowfile.
   - We noted: that one the value mapped existed in older Mapping file.
   - So in essence: ReplaceTextWithMapping Processor didn't refresh its
   cash uptil certain time. And thus return old value for few mapping file and
   then - once in the meanwhile it has refreshed it cache - returned new
   mapped value.
   - So this cause the issue.

Question:

   - Is this a known issue with  ReplaceTextWithMapping Processor?
   - If not how can I create an issue for this?
   - How can I confirm this behaviour?

Thanks,
Ameer Mawia




--
http://ca.linkedin.com/in/ameermawia
Toronto, ON
Reply | Threaded
Open this post in threaded view
|

Re: Bug/Issue with ReplaceTextWithMapping

Ameer Mawia
Correcting Typo.

On Fri, Jul 19, 2019 at 2:03 PM Ameer Mawia <[hidden email]> wrote:

> Guys,
>
> It seems that NIFI  ReplaceTextWithMapping   Processors has a BUG with
> Refreshing its Mapped file. We are using its functionality in PROD and
> getting odd behaviour.
>
> Our USAGE Scenario:
>
>    - We use NIFI primarily as a TRANSFORMATION Tool.
>    - Our flow involves:
>
>
>    1. Getting a raw csv file.
>       2. Split the file on per line basis:
>          1. So from one source flowfile - we may have 10000 flowfile
>          generated/splitted out.
>       3. For each of the splitted flow file(flowfiles for individual
>       lines) we perform transformation on the attributes.
>       4. We merge these flowfiles back and write the Output file.
>
>
> As part of the transformation in Step#3, we do some mapping for one of the
> field in the csv. For this we use ReplaceTextWithMapping  Processor. Also
> to note we update our mapping file just before starting our flow(ie. Step
> #1)
>
> Our Issue:
>
>
>    - We have noted for SAME key we get two DIFFERENT values in two
>    different flowfiles.
>    - We noted that one of the value mapped, existed in an older Mapping
>    file.
>    - So in essence: ReplaceTextWithMapping Processor didn't refresh its
>    cash uptill certain time. And thus return the old value for few mapping
>    file and then - once in the meanwhile it has refreshed it cache - returned
>    new updated value.
>    - And this cause the issue?
>
> Question:
>
>    - Is this a known issue with  ReplaceTextWithMapping Processor?
>    - If not how can I create an issue for this?
>    - How can I confirm this behaviour?
>
> Thanks,
> Ameer Mawia
>
>
>
>
> --
> http://ca.linkedin.com/in/ameermawia
> Toronto, ON
>
>

--
http://ca.linkedin.com/in/ameermawia
Toronto, ON