Help with "MergeContent"

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Help with "MergeContent"

srini
This post was updated on .
Hello,

Why this merged only 13 out of 170, why it can't merge all? http://screencast.com/t/qDwcGDZIK


I see this WARNings on PutFile. Why? Why it can't merge all the same file?:

2016-12-08 13:28:01,118 WARN [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.PutFile PutFile[id=d58c6d67-0158-1000-a03a-0df7d62c6d22] Penalizing StandardFlowFileRecord[uuid=0f46f6c9-7cc1-4339-be0d-9d6e5a9557b5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1481221670979-72, container=default, section=72], offset=535929, length=231],offset=0,name=15235853714738,size=231] and routing to failure as configured because file with the same name already exists

I see statements like this in the logs:
Merged 23 FlowFiles into StandardFlowFileRecord
Merged 19 FlowFiles into StandardFlowFileRecord
Merged 25 FlowFiles into StandardFlowFileRecord
......

Why it can't merge all before send it them to PutFile?


Thanks
Srini
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

Matt Burgess-2
Srini,

Can you share your configuration for MergeContent? It appears that
some merging was done (170 into 13, not merging 13 together), but
depending on your RouteOnAttribute processor, you may have removed
some elements from a fragmented set without updating the
fragment.count attribute, so MergeContent wouldn't know when a
fragment has been completely merged successfully.

Regards,
Matt

On Thu, Dec 8, 2016 at 11:33 AM, srini <[hidden email]> wrote:

> Hello,
>
> Why this merged only 13 out of 170? http://screencast.com/t/qDwcGDZIK
>
> Thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Help-with-MergeContent-tp14169.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

srini
This post was updated on .
Hi Matt,
I updated the the origianl describtion little bit.

I see statements like this in the logs:
Merged 23 FlowFiles into StandardFlowFileRecord
Merged 19 FlowFiles into StandardFlowFileRecord
Merged 25 FlowFiles into StandardFlowFileRecord
......

Why it can't merge all before send it them to PutFile?

And here is the MergeContent screenshot.
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

Matthew Clarke
What is the desired outcome? Are you trying to have a single merged file
that included all 170 input files?

The merger content processor is looking at the incoming connection's queue
Everytime it runs. If you run schedule is 0, that means as fast as
possible. When it sees file it immediately bins those files. If at that
time the bin requirements have been met, the FlowFiles in that bin are
merged. You min bin requirement is only 1. So my guess here is that the 170
incoming files did not all land on the incoming queue at the same time, so
13 separate binning actions occurred.

If you want only a single output file, adjust the min bin requirements.

Thanks,
Matt

On Dec 8, 2016 2:24 PM, "srini" <[hidden email]> wrote:

Hi Matt,
I updated the the origianl describtion little bit.

And here is the MergeContent screenshot.
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/
n14171/Screenshot_2016-12-08_13.png>



--
View this message in context: http://apache-nifi-developer-
list.39713.n7.nabble.com/Help-with-MergeContent-tp14169p14171.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

srini
Hi Matt,

I see some Minimum properties. But if I set 100, but the number of flowfiles is 93, then it never complete.

And PutFile doesn't have append option. That is a big problem for me.

My desired outcome is to have a single merged file that included all 170 input files.

thanks
Srini
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

Matthew Clarke
You can set a min number of files, but also set a max bin age.  This will
allow a bin to continue to grow until it reaches the min, but also force it
to merge if the max bin age occurs before that min is reached. This solves
your 93 file issue. You just need to decide how long your are willing to
wait to insure you have gotten every file you wanted merged into a single
file.

Matt

On Dec 8, 2016 8:03 PM, "srini" <[hidden email]> wrote:

> Hi Matt,
>
> I see some Minimum properties. But if I set 100, but the number of
> flowfiles
> is 93, then it never complete.
>
> And PutFile doesn't have append option. That is a big problem for me.
>
> My desired outcome is to have a single merged file that included all 170
> input files.
>
> thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Help-with-MergeContent-tp14169p14174.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Help with "MergeContent"

srini
Thanks a lot, Matt. I am good now.