I see this WARNings on PutFile. Why? Why it can't merge all the same file?:
2016-12-08 13:28:01,118 WARN [Timer-Driven Process Thread-3] o.a.nifi.processors.standard.PutFile PutFile[id=d58c6d67-0158-1000-a03a-0df7d62c6d22] Penalizing StandardFlowFileRecord[uuid=0f46f6c9-7cc1-4339-be0d-9d6e5a9557b5,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1481221670979-72, container=default, section=72], offset=535929, length=231],offset=0,name=15235853714738,size=231] and routing to failure as configured because file with the same name already exists
I see statements like this in the logs:
Merged 23 FlowFiles into StandardFlowFileRecord
Merged 19 FlowFiles into StandardFlowFileRecord
Merged 25 FlowFiles into StandardFlowFileRecord
Why it can't merge all before send it them to PutFile?
Can you share your configuration for MergeContent? It appears that
some merging was done (170 into 13, not merging 13 together), but
depending on your RouteOnAttribute processor, you may have removed
some elements from a fragmented set without updating the
fragment.count attribute, so MergeContent wouldn't know when a
fragment has been completely merged successfully.
What is the desired outcome? Are you trying to have a single merged file
that included all 170 input files?
The merger content processor is looking at the incoming connection's queue
Everytime it runs. If you run schedule is 0, that means as fast as
possible. When it sees file it immediately bins those files. If at that
time the bin requirements have been met, the FlowFiles in that bin are
merged. You min bin requirement is only 1. So my guess here is that the 170
incoming files did not all land on the incoming queue at the same time, so
13 separate binning actions occurred.
If you want only a single output file, adjust the min bin requirements.
View this message in context: http://apache-nifi-developer- list.39713.n7.nabble.com/Help-with-MergeContent-tp14169p14171.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
You can set a min number of files, but also set a max bin age. This will
allow a bin to continue to grow until it reaches the min, but also force it
to merge if the max bin age occurs before that min is reached. This solves
your 93 file issue. You just need to decide how long your are willing to
wait to insure you have gotten every file you wanted merged into a single
> Hi Matt,
> I see some Minimum properties. But if I set 100, but the number of
> is 93, then it never complete.
> And PutFile doesn't have append option. That is a big problem for me.
> My desired outcome is to have a single merged file that included all 170
> input files.
> View this message in context: http://apache-nifi-developer- > list.39713.n7.nabble.com/Help-with-MergeContent-tp14169p14174.html
> Sent from the Apache NiFi Developer List mailing list archive at