All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual intervention will be required.

[srini]

Hi,
We have single nifi instance, and this is our production environment.
Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and Xmx, and restarted nifi.
Then I see this error [1]. Then I deleted the folder ../nifi-1.1.0/flowfile_repository and restarted the nifi.
Then everything is fine, and it has been working close 20 hours without any issue.

Now all of sudden, I am seeing the same error. What should I do?

[1] All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual intervention will be required.
error.txt attached.

thanks
Srinierror.txt

[James Wing]

Srini,

The error message "FlowFile Repository failed to update" matches a known
issue where NiFi has trouble persisting attributes larger than 64 KB (
https://issues.apache.org/jira/browse/NIFI-3389).  Is it possible that your
EvaluateJsonPath is creating attributes exceeding 64 KB?  From your error
file:

o.a.n.p.standard.EvaluateJsonPath
EvaluateJsonPath[id=b71b17e5-1046-115a-5d5c-e4f368141f13] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException:
FlowFile Repository failed to update:
org.apache.nifi.processor.exception.ProcessException: FlowFile Repository
failed to update


Thanks,

James

On Thu, Mar 16, 2017 at 11:40 AM, srini <[hidden email]> wrote:

> Hi,
> We have single nifi instance, and this is our production environment.
> Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and
> Xmx, and restarted nifi.
> Then I see this error [1]. Then I deleted the folder
> ../nifi-1.1.0/flowfile_repository and restarted the nifi.
> Then everything is fine, and it has been working close 20 hours without any
> issue.
>
> Now all of sudden, I am seeing the same error. What should I do?
>
> [1] All Partitions have been blacklisted due to failures when attempting to
> update. If the Write-Ahead Log is able to perform a checkpoint, this issue
> may resolve itself. Otherwise, manual intervention will be required.
>
> thanks
> Srini error.txt
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/
> n15167/error.txt>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

[srini]

Hi James,

Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I do to avoid this?
thanks
Srini

[James Wing]

Would it be possible to configure EvaluateJsonPath to place the selected
JSON fragment in the flowfile content instead of an attribute?  Or to break
the selection across multiple attributes rather than one big one?

Thanks,

James

On Thu, Mar 16, 2017 at 11:57 AM, srini <[hidden email]> wrote:

> Hi James,
>
> Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I
> do to avoid this?
> thanks
> Srini
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/All-Partitions-have-been-
> blacklisted-due-to-failures-when-attempting-to-update-If-
> the-Write-Ahead-Lo-tp15167p15169.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

[srini]

Hello everybody,
The size of the attribute nifi accepts is a show stopper for me. As my flow is big, the flowfile is changes few times. I can't add the attribute (size is close to 120KB) to each flowfile. If I put the attribute in the flowfile, I can't maintain the same flowfile till the end. As some processes replaces the flowfile automatically.

This is a show stopper for me. I have to move all my flow logic to a custom processor. That is a huge task.

thanks
Srini

[Andy LoPresto-2]

Srini,

Rather than storing the large amounts of data in the attribute of the original flowfile, can you put them into the content of a new flowfile (correlated with an identifier attribute) and then route the original flowfile on and reference the associated flowfile content when necessary?

Andy LoPresto

[hidden email]

[hidden email]

PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Mar 21, 2017, at 8:41 AM, srini <[hidden email]> wrote:

Hello everybody,
The size of the attribute nifi accepts is a show stopper for me. As my flow
is big, the flowfile is changes few times. I can't add the attribute (size
is close to 120KB) to each flowfile. If I put the attribute in the flowfile,
I can't maintain the same flowfile till the end. As some processes replaces
the flowfile automatically.

This is a show stopper for me. I have to move all my flow logic to a custom
processor. That is a huge task.

thanks
Srini





--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Write-Ahead-Lo-tp15167p15207.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.