QueryRecord error

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

QueryRecord error

Joe Gresock
I'm changing the subject name so as not to muddy the [VOTE] thread.

I finally found the error in the logs.  This was also not happening prior
to the upgrade.  I thinkI'll debug the server to determine the root cause:

2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81]
o.a.nifi.processors.standard.QueryRecord
QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query
StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1506960462126-919,
container=default, section=919], offset=55116,
length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]
due to org.apache.nifi.processor.exception.ProcessException: Failed to read
next record in stream for
StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1506960462126-919,
container=default, section=919], offset=55116,
length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]:
{}
org.apache.nifi.processor.exception.ProcessException: Failed to read next
record in stream for
StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1506960462126-919,
container=default, section=919], offset=55116,
length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]
        at
org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:65)
        at Baz$1$1.moveNext(Unknown Source)
        at
org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:664)
        at
org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
        at
org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
        at
org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89)
        at
org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:196)
        at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
        at
org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
        at
org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:513)
        at
org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:132)
        at
org.apache.nifi.processors.standard.QueryRecord.queryWithCache(QueryRecord.java:445)
        at
org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:276)
        at
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
        at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
        at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)


On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <[hidden email]> wrote:

> Hey Joe,
>
> I've seen this error crop up a time or two before. The issue is a bug in
> how we are handling a failure
> condition. Haven't yet figured out the best way to tackle it, but the
> thing to note is that this error occurs
> when there is some other sort of failure -- typically, a failure to obtain
> the schema for the Record Reader
> or Record Writer. Can you check your logs for any other errors that may be
> occurring?
>
> Thanks
> -Mark
>
>
> > On Oct 2, 2017, at 11:32 AM, Joe Gresock <[hidden email]> wrote:
> >
> > I have to change my non-binding +1 vote to a 0, unless there is a
> > workaround for the bug [1] I just submitted, regarding the QueryRecord
> > processor being broken.
> >
> >
> > [1] https://issues.apache.org/jira/browse/NIFI-4452
> >
> >
> > On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <[hidden email]>
> wrote:
> >
> >> +1, non-binding
> >>
> >> - Verified hashes
> >> - Full build with contrib-check passes
> >> - Was able to run nifi server and send flow files to remote input port
> >> over s2s protocol. Web UI and API working as expected.
> >>
> >> Platform:
> >> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
> >> 2017-04-03T15:39:06-04:00)
> >> Java version: 1.8.0_131, vendor: Oracle Corporation
> >> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
> >>
> >> Looks good! Nice work everyone!
> >>
> >> Kevin
> >>
> >> On 9/28/17, 15:54, "Jeff" <[hidden email]> wrote:
> >>
> >>    Hello,
> >>
> >>    I am pleased to be calling this vote for the source release of Apache
> >> NiFi
> >>    nifi-1.4.0.
> >>
> >>    The source zip, including signatures, digests, etc. can be found at:
> >>    https://repository.apache.org/content/repositories/
> orgapachenifi-1111
> >>
> >>    The Git tag is nifi-1.4.0-RC2
> >>    The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> >>    https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> >> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> >>
> >>    Checksums of nifi-1.4.0-source-release.zip:
> >>    MD5: 41e4083e602883a3e180032f32913414
> >>    SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
> >>
> >>    Release artifacts are signed with the following key:
> >>    https://people.apache.org/keys/committer/jstorck.asc
> >>
> >>    KEYS file available here:
> >>    https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>
> >>    199 issues were closed/resolved for this release:
> >>    https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >> projectId=12316020&version=12340589
> >>
> >>    Release note highlights can be found here:
> >>    https://cwiki.apache.org/confluence/display/NIFI/
> >> Release+Notes#ReleaseNotes-Version-1.4.0
> >>
> >>    The vote will be open for 72 hours.
> >>    Please download the release candidate and evaluate the necessary
> items
> >>    including checking hashes, signatures, build
> >>    from source, and test.  The please vote:
> >>
> >>    [ ] +1 Release this package as nifi-1.4.0
> >>    [ ] +0 no opinion
> >>    [ ] -1 Do not release this package because...
> >>
> >>
> >>
> >>
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
>
>


--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: QueryRecord error

Mark Payne
Joe,

Thanks. I was going to suggest we spawn off a new thread if this continued with more back-and-forth.

I did notice that in the JIRA you indicated that
you are using a CSVReader service to read the data. In the message, it indicates that the FlowFIle's
filename is "billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro" -- which would imply
to me that it is Avro (perhaps converted from CSV to avro upstream?) Is it possible that you just selected
the wrong reader?

Thanks
-Mark

> On Oct 2, 2017, at 12:24 PM, Joe Gresock <[hidden email]> wrote:
>
> I'm changing the subject name so as not to muddy the [VOTE] thread.
>
> I finally found the error in the logs.  This was also not happening prior
> to the upgrade.  I thinkI'll debug the server to determine the root cause:
>
> 2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81]
> o.a.nifi.processors.standard.QueryRecord
> QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query
> StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> container=default, section=919], offset=55116,
> length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]
> due to org.apache.nifi.processor.exception.ProcessException: Failed to read
> next record in stream for
> StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> container=default, section=919], offset=55116,
> length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]:
> {}
> org.apache.nifi.processor.exception.ProcessException: Failed to read next
> record in stream for
> StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> container=default, section=919], offset=55116,
> length=616757],offset=0,name=billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro,size=616757]
>        at
> org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:65)
>        at Baz$1$1.moveNext(Unknown Source)
>        at
> org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:664)
>        at
> org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
>        at
> org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33)
>        at
> org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89)
>        at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:196)
>        at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
>        at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
>        at
> org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(AvaticaConnection.java:513)
>        at
> org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(AvaticaPreparedStatement.java:132)
>        at
> org.apache.nifi.processors.standard.QueryRecord.queryWithCache(QueryRecord.java:445)
>        at
> org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:276)
>        at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
>        at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
>        at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>        at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>        at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>        at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>        at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>        at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>        at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>        at java.lang.Thread.run(Thread.java:748)
>
>
> On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <[hidden email]> wrote:
>
>> Hey Joe,
>>
>> I've seen this error crop up a time or two before. The issue is a bug in
>> how we are handling a failure
>> condition. Haven't yet figured out the best way to tackle it, but the
>> thing to note is that this error occurs
>> when there is some other sort of failure -- typically, a failure to obtain
>> the schema for the Record Reader
>> or Record Writer. Can you check your logs for any other errors that may be
>> occurring?
>>
>> Thanks
>> -Mark
>>
>>
>>> On Oct 2, 2017, at 11:32 AM, Joe Gresock <[hidden email]> wrote:
>>>
>>> I have to change my non-binding +1 vote to a 0, unless there is a
>>> workaround for the bug [1] I just submitted, regarding the QueryRecord
>>> processor being broken.
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-4452
>>>
>>>
>>> On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <[hidden email]>
>> wrote:
>>>
>>>> +1, non-binding
>>>>
>>>> - Verified hashes
>>>> - Full build with contrib-check passes
>>>> - Was able to run nifi server and send flow files to remote input port
>>>> over s2s protocol. Web UI and API working as expected.
>>>>
>>>> Platform:
>>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
>>>> 2017-04-03T15:39:06-04:00)
>>>> Java version: 1.8.0_131, vendor: Oracle Corporation
>>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
>>>>
>>>> Looks good! Nice work everyone!
>>>>
>>>> Kevin
>>>>
>>>> On 9/28/17, 15:54, "Jeff" <[hidden email]> wrote:
>>>>
>>>>   Hello,
>>>>
>>>>   I am pleased to be calling this vote for the source release of Apache
>>>> NiFi
>>>>   nifi-1.4.0.
>>>>
>>>>   The source zip, including signatures, digests, etc. can be found at:
>>>>   https://repository.apache.org/content/repositories/
>> orgapachenifi-1111
>>>>
>>>>   The Git tag is nifi-1.4.0-RC2
>>>>   The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>>>>   https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>>>> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>>>>
>>>>   Checksums of nifi-1.4.0-source-release.zip:
>>>>   MD5: 41e4083e602883a3e180032f32913414
>>>>   SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
>>>>
>>>>   Release artifacts are signed with the following key:
>>>>   https://people.apache.org/keys/committer/jstorck.asc
>>>>
>>>>   KEYS file available here:
>>>>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>>>>
>>>>   199 issues were closed/resolved for this release:
>>>>   https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>>>> projectId=12316020&version=12340589
>>>>
>>>>   Release note highlights can be found here:
>>>>   https://cwiki.apache.org/confluence/display/NIFI/
>>>> Release+Notes#ReleaseNotes-Version-1.4.0
>>>>
>>>>   The vote will be open for 72 hours.
>>>>   Please download the release candidate and evaluate the necessary
>> items
>>>>   including checking hashes, signatures, build
>>>>   from source, and test.  The please vote:
>>>>
>>>>   [ ] +1 Release this package as nifi-1.4.0
>>>>   [ ] +0 no opinion
>>>>   [ ] -1 Do not release this package because...
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> I know what it is to be in need, and I know what it is to have plenty.  I
>>> have learned the secret of being content in any and every situation,
>>> whether well fed or hungry, whether living in plenty or in want.  I can
>> do
>>> all this through him who gives me strength.    *-Philippians 4:12-13*
>>
>>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*

Reply | Threaded
Open this post in threaded view
|

Re: QueryRecord error

Joe Gresock
Good question -- no, this must be an artifact of running InferAvroSchema
before the QueryRecord processor.  InferAvroSchema infers the schema and
places it in a flow file attribute, which the CSVReader uses in its
configuration.

I just debugged nifi and found that it thinks one of my fields is null.
There is indeed a record with an empty string in the csv for this field,
but the same thing happens every day when this flow runs, so I suspect
something must have changed in the implementation of either CSVReader or
InferAvroSchema that made the parsing stricter.  I'm going to try to figure
out how to allow blank values in the CSV.

On Mon, Oct 2, 2017 at 4:29 PM, Mark Payne <[hidden email]> wrote:

> Joe,
>
> Thanks. I was going to suggest we spawn off a new thread if this continued
> with more back-and-forth.
>
> I did notice that in the JIRA you indicated that
> you are using a CSVReader service to read the data. In the message, it
> indicates that the FlowFIle's
> filename is "billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro"
> -- which would imply
> to me that it is Avro (perhaps converted from CSV to avro upstream?) Is it
> possible that you just selected
> the wrong reader?
>
> Thanks
> -Mark
>
> > On Oct 2, 2017, at 12:24 PM, Joe Gresock <[hidden email]> wrote:
> >
> > I'm changing the subject name so as not to muddy the [VOTE] thread.
> >
> > I finally found the error in the logs.  This was also not happening prior
> > to the upgrade.  I thinkI'll debug the server to determine the root
> cause:
> >
> > 2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81]
> > o.a.nifi.processors.standard.QueryRecord
> > QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query
> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=
> StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> > container=default, section=919], offset=55116,
> > length=616757],offset=0,name=billing-detailed-line-items-
> with-resources-and-tags-2017-10.csv.avro,size=616757]
> > due to org.apache.nifi.processor.exception.ProcessException: Failed to
> read
> > next record in stream for
> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=
> StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> > container=default, section=919], offset=55116,
> > length=616757],offset=0,name=billing-detailed-line-items-
> with-resources-and-tags-2017-10.csv.avro,size=616757]:
> > {}
> > org.apache.nifi.processor.exception.ProcessException: Failed to read
> next
> > record in stream for
> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f74911,claim=
> StandardContentClaim
> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
> > container=default, section=919], offset=55116,
> > length=616757],offset=0,name=billing-detailed-line-items-
> with-resources-and-tags-2017-10.csv.avro,size=616757]
> >        at
> > org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(
> FlowFileEnumerator.java:65)
> >        at Baz$1$1.moveNext(Unknown Source)
> >        at
> > org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<
> init>(Linq4j.java:664)
> >        at
> > org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
> >        at
> > org.apache.calcite.linq4j.AbstractEnumerable.iterator(
> AbstractEnumerable.java:33)
> >        at
> > org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89)
> >        at
> > org.apache.calcite.avatica.AvaticaResultSet.execute(
> AvaticaResultSet.java:196)
> >        at
> > org.apache.calcite.jdbc.CalciteResultSet.execute(
> CalciteResultSet.java:67)
> >        at
> > org.apache.calcite.jdbc.CalciteResultSet.execute(
> CalciteResultSet.java:44)
> >        at
> > org.apache.calcite.avatica.AvaticaConnection.executeQueryInternal(
> AvaticaConnection.java:513)
> >        at
> > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQuery(
> AvaticaPreparedStatement.java:132)
> >        at
> > org.apache.nifi.processors.standard.QueryRecord.
> queryWithCache(QueryRecord.java:445)
> >        at
> > org.apache.nifi.processors.standard.QueryRecord.
> onTrigger(QueryRecord.java:276)
> >        at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:27)
> >        at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1119)
> >        at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:147)
> >        at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> ContinuallyRunProcessorTask.java:47)
> >        at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> TimerDrivenSchedulingAgent.java:128)
> >        at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >        at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> >        at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >        at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >        at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >        at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >        at java.lang.Thread.run(Thread.java:748)
> >
> >
> > On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <[hidden email]> wrote:
> >
> >> Hey Joe,
> >>
> >> I've seen this error crop up a time or two before. The issue is a bug in
> >> how we are handling a failure
> >> condition. Haven't yet figured out the best way to tackle it, but the
> >> thing to note is that this error occurs
> >> when there is some other sort of failure -- typically, a failure to
> obtain
> >> the schema for the Record Reader
> >> or Record Writer. Can you check your logs for any other errors that may
> be
> >> occurring?
> >>
> >> Thanks
> >> -Mark
> >>
> >>
> >>> On Oct 2, 2017, at 11:32 AM, Joe Gresock <[hidden email]> wrote:
> >>>
> >>> I have to change my non-binding +1 vote to a 0, unless there is a
> >>> workaround for the bug [1] I just submitted, regarding the QueryRecord
> >>> processor being broken.
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/NIFI-4452
> >>>
> >>>
> >>> On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <[hidden email]>
> >> wrote:
> >>>
> >>>> +1, non-binding
> >>>>
> >>>> - Verified hashes
> >>>> - Full build with contrib-check passes
> >>>> - Was able to run nifi server and send flow files to remote input port
> >>>> over s2s protocol. Web UI and API working as expected.
> >>>>
> >>>> Platform:
> >>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
> >>>> 2017-04-03T15:39:06-04:00)
> >>>> Java version: 1.8.0_131, vendor: Oracle Corporation
> >>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family: "mac"
> >>>>
> >>>> Looks good! Nice work everyone!
> >>>>
> >>>> Kevin
> >>>>
> >>>> On 9/28/17, 15:54, "Jeff" <[hidden email]> wrote:
> >>>>
> >>>>   Hello,
> >>>>
> >>>>   I am pleased to be calling this vote for the source release of
> Apache
> >>>> NiFi
> >>>>   nifi-1.4.0.
> >>>>
> >>>>   The source zip, including signatures, digests, etc. can be found at:
> >>>>   https://repository.apache.org/content/repositories/
> >> orgapachenifi-1111
> >>>>
> >>>>   The Git tag is nifi-1.4.0-RC2
> >>>>   The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> >>>>   https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
> >>>> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
> >>>>
> >>>>   Checksums of nifi-1.4.0-source-release.zip:
> >>>>   MD5: 41e4083e602883a3e180032f32913414
> >>>>   SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
> >>>>
> >>>>   Release artifacts are signed with the following key:
> >>>>   https://people.apache.org/keys/committer/jstorck.asc
> >>>>
> >>>>   KEYS file available here:
> >>>>   https://dist.apache.org/repos/dist/release/nifi/KEYS
> >>>>
> >>>>   199 issues were closed/resolved for this release:
> >>>>   https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> >>>> projectId=12316020&version=12340589
> >>>>
> >>>>   Release note highlights can be found here:
> >>>>   https://cwiki.apache.org/confluence/display/NIFI/
> >>>> Release+Notes#ReleaseNotes-Version-1.4.0
> >>>>
> >>>>   The vote will be open for 72 hours.
> >>>>   Please download the release candidate and evaluate the necessary
> >> items
> >>>>   including checking hashes, signatures, build
> >>>>   from source, and test.  The please vote:
> >>>>
> >>>>   [ ] +1 Release this package as nifi-1.4.0
> >>>>   [ ] +0 no opinion
> >>>>   [ ] -1 Do not release this package because...
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> I know what it is to be in need, and I know what it is to have
> plenty.  I
> >>> have learned the secret of being content in any and every situation,
> >>> whether well fed or hungry, whether living in plenty or in want.  I can
> >> do
> >>> all this through him who gives me strength.    *-Philippians 4:12-13*
> >>
> >>
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
>
>


--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: QueryRecord error

Joe Gresock
I was able to configure the reader to infer the schema from the first line
of the CSV, and the writer to use the same schema, and now the processor
works.  I think these are new properties in 1.4.0, and they certainly help
simplify the flow!  I was able to remove the InferAvroSchema processor as a
result.

Joe

On Mon, Oct 2, 2017 at 4:44 PM, Joe Gresock <[hidden email]> wrote:

> Good question -- no, this must be an artifact of running InferAvroSchema
> before the QueryRecord processor.  InferAvroSchema infers the schema and
> places it in a flow file attribute, which the CSVReader uses in its
> configuration.
>
> I just debugged nifi and found that it thinks one of my fields is null.
> There is indeed a record with an empty string in the csv for this field,
> but the same thing happens every day when this flow runs, so I suspect
> something must have changed in the implementation of either CSVReader or
> InferAvroSchema that made the parsing stricter.  I'm going to try to figure
> out how to allow blank values in the CSV.
>
> On Mon, Oct 2, 2017 at 4:29 PM, Mark Payne <[hidden email]> wrote:
>
>> Joe,
>>
>> Thanks. I was going to suggest we spawn off a new thread if this
>> continued with more back-and-forth.
>>
>> I did notice that in the JIRA you indicated that
>> you are using a CSVReader service to read the data. In the message, it
>> indicates that the FlowFIle's
>> filename is "billing-detailed-line-items-with-resources-and-tags-2017-10.csv.avro"
>> -- which would imply
>> to me that it is Avro (perhaps converted from CSV to avro upstream?) Is
>> it possible that you just selected
>> the wrong reader?
>>
>> Thanks
>> -Mark
>>
>> > On Oct 2, 2017, at 12:24 PM, Joe Gresock <[hidden email]> wrote:
>> >
>> > I'm changing the subject name so as not to muddy the [VOTE] thread.
>> >
>> > I finally found the error in the logs.  This was also not happening
>> prior
>> > to the upgrade.  I thinkI'll debug the server to determine the root
>> cause:
>> >
>> > 2017-10-02 16:09:47,004 ERROR [Timer-Driven Process Thread-81]
>> > o.a.nifi.processors.standard.QueryRecord
>> > QueryRecord[id=21570e18-015c-1000-0000-000058832883] Unable to query
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]
>> > due to org.apache.nifi.processor.exception.ProcessException: Failed to
>> read
>> > next record in stream for
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]:
>> > {}
>> > org.apache.nifi.processor.exception.ProcessException: Failed to read
>> next
>> > record in stream for
>> > StandardFlowFileRecord[uuid=2da6d0f9-3856-420e-99bd-9589f7f7
>> 4911,claim=StandardContentClaim
>> > [resourceClaim=StandardResourceClaim[id=1506960462126-919,
>> > container=default, section=919], offset=55116,
>> > length=616757],offset=0,name=billing-detailed-line-items-wit
>> h-resources-and-tags-2017-10.csv.avro,size=616757]
>> >        at
>> > org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(Flow
>> FileEnumerator.java:65)
>> >        at Baz$1$1.moveNext(Unknown Source)
>> >        at
>> > org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(
>> Linq4j.java:664)
>> >        at
>> > org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:98)
>> >        at
>> > org.apache.calcite.linq4j.AbstractEnumerable.iterator(Abstra
>> ctEnumerable.java:33)
>> >        at
>> > org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:89)
>> >        at
>> > org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaR
>> esultSet.java:196)
>> >        at
>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>> ltSet.java:67)
>> >        at
>> > org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResu
>> ltSet.java:44)
>> >        at
>> > org.apache.calcite.avatica.AvaticaConnection.executeQueryInt
>> ernal(AvaticaConnection.java:513)
>> >        at
>> > org.apache.calcite.avatica.AvaticaPreparedStatement.executeQ
>> uery(AvaticaPreparedStatement.java:132)
>> >        at
>> > org.apache.nifi.processors.standard.QueryRecord.queryWithCac
>> he(QueryRecord.java:445)
>> >        at
>> > org.apache.nifi.processors.standard.QueryRecord.onTrigger(
>> QueryRecord.java:276)
>> >        at
>> > org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:27)
>> >        at
>> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1119)
>> >        at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:147)
>> >        at
>> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> >        at
>> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:128)
>> >        at
>> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> >        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> >        at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >        at
>> > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >        at
>> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> >        at
>> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> >        at java.lang.Thread.run(Thread.java:748)
>> >
>> >
>> > On Mon, Oct 2, 2017 at 4:03 PM, Mark Payne <[hidden email]>
>> wrote:
>> >
>> >> Hey Joe,
>> >>
>> >> I've seen this error crop up a time or two before. The issue is a bug
>> in
>> >> how we are handling a failure
>> >> condition. Haven't yet figured out the best way to tackle it, but the
>> >> thing to note is that this error occurs
>> >> when there is some other sort of failure -- typically, a failure to
>> obtain
>> >> the schema for the Record Reader
>> >> or Record Writer. Can you check your logs for any other errors that
>> may be
>> >> occurring?
>> >>
>> >> Thanks
>> >> -Mark
>> >>
>> >>
>> >>> On Oct 2, 2017, at 11:32 AM, Joe Gresock <[hidden email]> wrote:
>> >>>
>> >>> I have to change my non-binding +1 vote to a 0, unless there is a
>> >>> workaround for the bug [1] I just submitted, regarding the QueryRecord
>> >>> processor being broken.
>> >>>
>> >>>
>> >>> [1] https://issues.apache.org/jira/browse/NIFI-4452
>> >>>
>> >>>
>> >>> On Mon, Oct 2, 2017 at 3:23 AM, Kevin Doran <[hidden email]>
>> >> wrote:
>> >>>
>> >>>> +1, non-binding
>> >>>>
>> >>>> - Verified hashes
>> >>>> - Full build with contrib-check passes
>> >>>> - Was able to run nifi server and send flow files to remote input
>> port
>> >>>> over s2s protocol. Web UI and API working as expected.
>> >>>>
>> >>>> Platform:
>> >>>> Apache Maven 3.5.0 (ff8f5e7444045639af65f6095c62210b5713f426;
>> >>>> 2017-04-03T15:39:06-04:00)
>> >>>> Java version: 1.8.0_131, vendor: Oracle Corporation
>> >>>> OS name: "mac os x", version: "10.12.6", arch: "x86_64", family:
>> "mac"
>> >>>>
>> >>>> Looks good! Nice work everyone!
>> >>>>
>> >>>> Kevin
>> >>>>
>> >>>> On 9/28/17, 15:54, "Jeff" <[hidden email]> wrote:
>> >>>>
>> >>>>   Hello,
>> >>>>
>> >>>>   I am pleased to be calling this vote for the source release of
>> Apache
>> >>>> NiFi
>> >>>>   nifi-1.4.0.
>> >>>>
>> >>>>   The source zip, including signatures, digests, etc. can be found
>> at:
>> >>>>   https://repository.apache.org/content/repositories/
>> >> orgapachenifi-1111
>> >>>>
>> >>>>   The Git tag is nifi-1.4.0-RC2
>> >>>>   The Git commit ID is e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>> >>>>   https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=
>> >>>> e6508ba7d3da5bba54abd6233a7a8f9dd4c32151
>> >>>>
>> >>>>   Checksums of nifi-1.4.0-source-release.zip:
>> >>>>   MD5: 41e4083e602883a3e180032f32913414
>> >>>>   SHA1: 26770625138126f45bed4989adb0a6b65a767aa2
>> >>>>
>> >>>>   Release artifacts are signed with the following key:
>> >>>>   https://people.apache.org/keys/committer/jstorck.asc
>> >>>>
>> >>>>   KEYS file available here:
>> >>>>   https://dist.apache.org/repos/dist/release/nifi/KEYS
>> >>>>
>> >>>>   199 issues were closed/resolved for this release:
>> >>>>   https://issues.apache.org/jira/secure/ReleaseNote.jspa?
>> >>>> projectId=12316020&version=12340589
>> >>>>
>> >>>>   Release note highlights can be found here:
>> >>>>   https://cwiki.apache.org/confluence/display/NIFI/
>> >>>> Release+Notes#ReleaseNotes-Version-1.4.0
>> >>>>
>> >>>>   The vote will be open for 72 hours.
>> >>>>   Please download the release candidate and evaluate the necessary
>> >> items
>> >>>>   including checking hashes, signatures, build
>> >>>>   from source, and test.  The please vote:
>> >>>>
>> >>>>   [ ] +1 Release this package as nifi-1.4.0
>> >>>>   [ ] +0 no opinion
>> >>>>   [ ] -1 Do not release this package because...
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>> --
>> >>> I know what it is to be in need, and I know what it is to have
>> plenty.  I
>> >>> have learned the secret of being content in any and every situation,
>> >>> whether well fed or hungry, whether living in plenty or in want.  I
>> can
>> >> do
>> >>> all this through him who gives me strength.    *-Philippians 4:12-13*
>> >>
>> >>
>> >
>> >
>> > --
>> > I know what it is to be in need, and I know what it is to have plenty.
>> I
>> > have learned the secret of being content in any and every situation,
>> > whether well fed or hungry, whether living in plenty or in want.  I can
>> do
>> > all this through him who gives me strength.    *-Philippians 4:12-13*
>>
>>
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can
> do all this through him who gives me strength.    *-Philippians 4:12-13*
>



--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*