When should MergeContent stop and proceed to next processor?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

When should MergeContent stop and proceed to next processor?

srini
This post was updated on .
Hi,
I have a subflow like this. From SplitJson to MergeContent it is in loop. I expect it loops based on the number of splits of that record. How it know that splits for that record is over, and it needs to be proceed to next processor that is ExtractText?

I have 3 records. In my case it is merging 25 (8 + 5 + 12 = 25). It is merging all records data into one record. It shouldn't merge all, instead after each record it should proceed to next processor.
1st record: Merge 8 items
2nd record:  Merge 5 items.
3rd record: Merge 12 items.



What changes do you recommend to my flow?

Reply | Threaded
Open this post in threaded view
|

Re: When should MergeContent stop and proceed to next processor?

leelaim
Srini,
You should set your merge strategy to "defragment"  on the MergeContent
processor.  This will merge based on the parent record that was split at
splitJson.
Thanks,
Lee


On Wed, Mar 15, 2017 at 3:55 PM, srini <[hidden email]> wrote:

> Hi,
> I have a subflow like this. From SplitJson to MergeContent it is in loop. I
> expect it loops based on the number of splits of that record. How it know
> that splits for that record is over, and it needs to be proceed to next
> processor that is ExtractText?
>
> I have 3 records. In my case it is merging 25 (8 + 5 + 12 = 25). It is
> merging all records data into one record. It shouldn't merge all, instead
> after each record it should proceed to next processor.
> 1st record: Merge 8 items
> 2nd record:  Merge 5 items.
> 3rd record: Merge 12 items.
>
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/
> n15148/Screenshot_2017-03-15_17.png>
>
> What changes do you recommend to my flow?
> Here is my MergeContent screenshot.
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/
> n15148/Screenshot_2017-03-15_17.png>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/When-should-MergeContent-stop-and-
> proceed-to-next-processor-tp15148.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: When should MergeContent stop and proceed to next processor?

Oleg Zhurakousky
In reply to this post by srini
Hi

Is there any chance you can share your processor’s configuration? I am curious as to what are you using as “Correlation Attribute Name” in the MergeContent processor.
Basically this attribute allows to distinguish groups of flow files so, since you have SplitJson as an upstream processor feeding MergeContent you can use “fragment.identifier” as correlation attribute.
Anyway, please share what you can.

Cheers
Oleg

> On Mar 15, 2017, at 5:55 PM, srini <[hidden email]> wrote:
>
> Hi,
> I have a subflow like this. From SplitJson to MergeContent it is in loop. I
> expect it loops based on the number of splits of that record. How it know
> that splits for that record is over, and it needs to be proceed to next
> processor that is ExtractText?
>
> I have 3 records. In my case it is merging 25 (8 + 5 + 12 = 25). It is
> merging all records data into one record. It shouldn't merge all, instead
> after each record it should proceed to next processor.
> 1st record: Merge 8 items
> 2nd record:  Merge 5 items.
> 3rd record: Merge 12 items.
>
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n15148/Screenshot_2017-03-15_17.png>
>
> What changes do you recommend to my flow?
> Here is my MergeContent screenshot.
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n15148/Screenshot_2017-03-15_17.png>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/When-should-MergeContent-stop-and-proceed-to-next-processor-tp15148.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: When should MergeContent stop and proceed to next processor?

srini
Hi Oleg,

Here is MergetContent screenshot. My flowfiles don't give any clue about what record it belongs to. I have an attribute called recordId which distinguishes each record. But I shouldn't add recordId in the flowfiles to be merged.


thanks
Srini
Reply | Threaded
Open this post in threaded view
|

Re: When should MergeContent stop and proceed to next processor?

Oleg Zhurakousky
Ok, can you please set the “Correlation Identifier” to “fragment.identifier”?
That is what I was trying to explain in the previous email.

Cheers
Oleg

> On Mar 16, 2017, at 11:06 AM, srini <[hidden email]> wrote:
>
> Hi Oleg,
>
> Here is MergetContent screenshot. My flowfiles don't give any clue about
> what record it belongs to. I have an attribute called recordId which
> distinguishes each record. But I shouldn't add recordId in the flowfiles to
> be merged.
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/n15164/Screenshot_2017-03-16_10.png>
>
> thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/When-should-MergeContent-stop-and-proceed-to-next-processor-tp15148p15164.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
>