multi-row updates makes ArrayIndexOutOfBoundsException.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

multi-row updates makes ArrayIndexOutOfBoundsException.

gywndi
Hi~

I'm testing CDC from MySQL to another system(ex, Redis or Logging).
And I set simple CDC flow with CaptureChangeMySQL.
The thing only I config is just use "LogAttribute" and write down every
event from MySQL.

Single rows was no matter.
But.. I changed updated multi-rows with "update xx set aa = md5(rand()) ",
then I've got in big problem with this error output.

I wanted to skip this events.. but nothing I can do.
Could tell me what I misunderstand using NiFi?

I think this is the final solution if it works well. (I can do a log of
things.)
Thanks.
Chan.

==========================================
2017-11-04 03:21:50,565 ERROR [Timer-Driven Process Thread-8]
o.a.n.c.m.processors.CaptureChangeMySQL
CaptureChangeMySQL[id=82d07299-015f-1000-5dba-3ead930d9c90]
CaptureChangeMySQL[id=82d07299-015f-1000-5dba-3ead930d9c90] failed to
process session due to java.lang.ArrayIndexOutOfBoundsException: 1: {}
java.lang.ArrayIndexOutOfBoundsException: 1
        at
org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeRow(UpdateRowsWriter.java:87)
        at
org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.lambda$writeEvent$0(UpdateRowsWriter.java:57)
        at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570)
        at
org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeEvent(UpdateRowsWriter.java:51)
        at
org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:887)
        at
org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
        at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
        at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
        at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
        at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: multi-row updates makes ArrayIndexOutOfBoundsException.

Matt Burgess-2
Chan,

This definitely looks like a bug, you should not see an
ArrayIndexOutOfBoundsException, it should be handled more gracefully.
It looks like the processor is not handling the (possibly missing)
information in the binlog event. I will investigate and will likely
write up a Jira case to track any needed fix.

Regards,
Matt

On Mon, Nov 6, 2017 at 4:45 AM, gywndi <[hidden email]> wrote:

> Hi~
>
> I'm testing CDC from MySQL to another system(ex, Redis or Logging).
> And I set simple CDC flow with CaptureChangeMySQL.
> The thing only I config is just use "LogAttribute" and write down every
> event from MySQL.
>
> Single rows was no matter.
> But.. I changed updated multi-rows with "update xx set aa = md5(rand()) ",
> then I've got in big problem with this error output.
>
> I wanted to skip this events.. but nothing I can do.
> Could tell me what I misunderstand using NiFi?
>
> I think this is the final solution if it works well. (I can do a log of
> things.)
> Thanks.
> Chan.
>
> ==========================================
> 2017-11-04 03:21:50,565 ERROR [Timer-Driven Process Thread-8]
> o.a.n.c.m.processors.CaptureChangeMySQL
> CaptureChangeMySQL[id=82d07299-015f-1000-5dba-3ead930d9c90]
> CaptureChangeMySQL[id=82d07299-015f-1000-5dba-3ead930d9c90] failed to
> process session due to java.lang.ArrayIndexOutOfBoundsException: 1: {}
> java.lang.ArrayIndexOutOfBoundsException: 1
>         at
> org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeRow(UpdateRowsWriter.java:87)
>         at
> org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.lambda$writeEvent$0(UpdateRowsWriter.java:57)
>         at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2570)
>         at
> org.apache.nifi.cdc.mysql.event.io.UpdateRowsWriter.writeEvent(UpdateRowsWriter.java:51)
>         at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.outputEvents(CaptureChangeMySQL.java:887)
>         at
> org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL.onTrigger(CaptureChangeMySQL.java:571)
>         at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1119)
>         at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147)
>         at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
>         at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:128)
>         at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>         at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>         at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>         at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at java.lang.Thread.run(Thread.java:748)
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: multi-row updates makes ArrayIndexOutOfBoundsException.

gywndi
Hi~

I check my binlog parameters.
We use with setting binlog_row_image to "MINIMAL", and it doesn't fit match
to the column count.
So I guess I've got the exception.

What I wanted to do is get the PK information from the binlog and expire
cache data(ex, redis/memcache) with the PK.

But, binlog_row_image is session variable, so anyone who has enough
privilege to change the paraemter can make the terrible errors. (The
Infinite loop and the only workaround is to remove CaptureChangeMySQL and
reconfigure it.)

At least it seems necessary to process this case.

Thanks.
Chan.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/