This post was updated on .
Hi,
We have single nifi instance, and this is our production environment. Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and Xmx, and restarted nifi. Then I see this error [1]. Then I deleted the folder ../nifi-1.1.0/flowfile_repository and restarted the nifi. Then everything is fine, and it has been working close 20 hours without any issue. Now all of sudden, I am seeing the same error. What should I do? [1] All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual intervention will be required. error.txt attached. thanks Srinierror.txt |
Srini,
The error message "FlowFile Repository failed to update" matches a known issue where NiFi has trouble persisting attributes larger than 64 KB ( https://issues.apache.org/jira/browse/NIFI-3389). Is it possible that your EvaluateJsonPath is creating attributes exceeding 64 KB? From your error file: o.a.n.p.standard.EvaluateJsonPath EvaluateJsonPath[id=b71b17e5-1046-115a-5d5c-e4f368141f13] Failed to process session due to org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update: org.apache.nifi.processor.exception.ProcessException: FlowFile Repository failed to update Thanks, James On Thu, Mar 16, 2017 at 11:40 AM, srini <[hidden email]> wrote: > Hi, > We have single nifi instance, and this is our production environment. > Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and > Xmx, and restarted nifi. > Then I see this error [1]. Then I deleted the folder > ../nifi-1.1.0/flowfile_repository and restarted the nifi. > Then everything is fine, and it has been working close 20 hours without any > issue. > > Now all of sudden, I am seeing the same error. What should I do? > > [1] All Partitions have been blacklisted due to failures when attempting to > update. If the Write-Ahead Log is able to perform a checkpoint, this issue > may resolve itself. Otherwise, manual intervention will be required. > > thanks > Srini error.txt > <http://apache-nifi-developer-list.39713.n7.nabble.com/file/ > n15167/error.txt> > > > > -- > View this message in context: http://apache-nifi-developer- > list.39713.n7.nabble.com/All-Partitions-have-been- > blacklisted-due-to-failures-when-attempting-to-update-If- > the-Write-Ahead-Lo-tp15167.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. > |
Hi James,
Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I do to avoid this? thanks Srini |
Would it be possible to configure EvaluateJsonPath to place the selected
JSON fragment in the flowfile content instead of an attribute? Or to break the selection across multiple attributes rather than one big one? Thanks, James On Thu, Mar 16, 2017 at 11:57 AM, srini <[hidden email]> wrote: > Hi James, > > Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I > do to avoid this? > thanks > Srini > > > > -- > View this message in context: http://apache-nifi-developer- > list.39713.n7.nabble.com/All-Partitions-have-been- > blacklisted-due-to-failures-when-attempting-to-update-If- > the-Write-Ahead-Lo-tp15167p15169.html > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. > |
Hello everybody,
The size of the attribute nifi accepts is a show stopper for me. As my flow is big, the flowfile is changes few times. I can't add the attribute (size is close to 120KB) to each flowfile. If I put the attribute in the flowfile, I can't maintain the same flowfile till the end. As some processes replaces the flowfile automatically. This is a show stopper for me. I have to move all my flow logic to a custom processor. That is a huge task. thanks Srini |
Srini,
Rather than storing the large amounts of data in the attribute of the original flowfile, can you put them into the content of a new flowfile (correlated with an identifier attribute) and then route the original flowfile on and reference the associated flowfile content when necessary? Andy LoPresto PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
|
Free forum by Nabble | Edit this page |