Get IOException occasionally

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Get IOException occasionally

David Yu
Hi all,
I need help finding the root cause on this issue.
I occasionally get the following exception from SelectHiveQL and PutHiveQL
processor.
Sometimes the queries executed without any issue but next minute this
exception could happen.
Once it happened, by restarting only the NiFi does not resolve the issue.
I have to reboot the Linux VM of the node and remove the flow.xml.gz file
then restart nifi node.

I am using NIFI version 1.7.1 in cluster mode with total 8 nodes

Here is the stacktrace
2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute
HiveQL select query select max(version) as max_partition, min(version) as
min_partition from Table_A for
StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0]
due to org.apache.nifi.processor.exception.ProcessException: Error during
database query or conversion of records.; routing to failure:
org.apache.nifi.processor.exception.ProcessException: Error during database
query or conversion of records.
org.apache.nifi.processor.exception.ProcessException: Error during database
query or conversion of records.
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407)
at
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown
Source)
at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395)
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285)
at
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown
Source)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.sql.SQLException: java.io.IOException:
java.lang.RuntimeException: native snappy library not available
at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at
org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
at
org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355)
at
org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322)
at
org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401)
... 18 common frames omitted
2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6]
o.a.nifi.processors.hive.SelectHiveQL
SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d]
org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d
failed to process due to
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not
specified; rolling back session:
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not specified
org.apache.nifi.processor.exception.FlowFileHandlingException:
StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1557687270864-64540,
container=default, section=28], offset=81777,
length=280],offset=0,name=done,size=280] transfer relationship not specified
at
org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264)
at
org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115)
at
org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
at
org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
at
org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)


Thanks
David
Reply | Threaded
Open this post in threaded view
|

Re: Get IOException occasionally

Shawn Weeks
There are several complications using Snappy but from what I can tell the SelectHiveQL Processor isn't trying to use it and instead the Hive JDBC Driver is trying to use it. Are you running a specific vendors distribution of NiFi? The reason I'm asking is when I go look at the source for the Hive JDBC Driver Class org.apache.hive.jdbc.Utils.verifySuccess where the error is being raised none of the line numbers are matching up with Apache. It may be that the error is actually coming from Hive not NiFi.

Thanks
Shawn

On 5/14/19, 6:37 PM, "David Yu" <[hidden email]> wrote:

    Hi all,
    I need help finding the root cause on this issue.
    I occasionally get the following exception from SelectHiveQL and PutHiveQL
    processor.
    Sometimes the queries executed without any issue but next minute this
    exception could happen.
    Once it happened, by restarting only the NiFi does not resolve the issue.
    I have to reboot the Linux VM of the node and remove the flow.xml.gz file
    then restart nifi node.
   
    I am using NIFI version 1.7.1 in cluster mode with total 8 nodes
   
    Here is the stacktrace
    2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6]
    o.a.nifi.processors.hive.SelectHiveQL
    SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute
    HiveQL select query select max(version) as max_partition, min(version) as
    min_partition from Table_A for
    StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0]
    due to org.apache.nifi.processor.exception.ProcessException: Error during
    database query or conversion of records.; routing to failure:
    org.apache.nifi.processor.exception.ProcessException: Error during database
    query or conversion of records.
    org.apache.nifi.processor.exception.ProcessException: Error during database
    query or conversion of records.
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407)
    at
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown
    Source)
    at
    org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285)
    at
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown
    Source)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
    at
    org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
    at
    org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
    at
    org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
    Caused by: java.sql.SQLException: java.io.IOException:
    java.lang.RuntimeException: native snappy library not available
    at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
    at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
    at org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330)
    at
    org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
    at
    org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
    at
    org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355)
    at
    org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401)
    ... 18 common frames omitted
    2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6]
    o.a.nifi.processors.hive.SelectHiveQL
    SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d]
    org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d
    failed to process due to
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not
    specified; rolling back session:
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not specified
    org.apache.nifi.processor.exception.FlowFileHandlingException:
    StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
    [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
    container=default, section=28], offset=81777,
    length=280],offset=0,name=done,size=280] transfer relationship not specified
    at
    org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264)
    at
    org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115)
    at
    org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
    at
    org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
    at
    org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
    at
    org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
    at
    org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at
    java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at
    java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
   
   
    Thanks
    David
   

Reply | Threaded
Open this post in threaded view
|

Re: Get IOException occasionally

Joe Witt
..another possible factor here is that if two different nars, or processors
which use per instance classloading (as the hadoop ones do), then only one
will have its native libs loaded.  Others will fail to load that same
native lib as this is a JVM wide enforced thing.  So if you have multiple
of these in your flow using the same native lib on some runs one processor
could work and on another a different one will work.  This is frustrating
for users for sure.  The solution is to leverage libraries and update
libraries and/or intercept and do it for them such that we given each
native lib a unique name in the JVM on a per classloader basis.  Just
something we need to knock out.

On Wed, May 15, 2019 at 8:54 AM Shawn Weeks <[hidden email]>
wrote:

> There are several complications using Snappy but from what I can tell the
> SelectHiveQL Processor isn't trying to use it and instead the Hive JDBC
> Driver is trying to use it. Are you running a specific vendors distribution
> of NiFi? The reason I'm asking is when I go look at the source for the Hive
> JDBC Driver Class org.apache.hive.jdbc.Utils.verifySuccess where the error
> is being raised none of the line numbers are matching up with Apache. It
> may be that the error is actually coming from Hive not NiFi.
>
> Thanks
> Shawn
>
> On 5/14/19, 6:37 PM, "David Yu" <[hidden email]> wrote:
>
>     Hi all,
>     I need help finding the root cause on this issue.
>     I occasionally get the following exception from SelectHiveQL and
> PutHiveQL
>     processor.
>     Sometimes the queries executed without any issue but next minute this
>     exception could happen.
>     Once it happened, by restarting only the NiFi does not resolve the
> issue.
>     I have to reboot the Linux VM of the node and remove the flow.xml.gz
> file
>     then restart nifi node.
>
>     I am using NIFI version 1.7.1 in cluster mode with total 8 nodes
>
>     Here is the stacktrace
>     2019-05-14 20:37:30,937 ERROR [Timer-Driven Process Thread-6]
>     o.a.nifi.processors.hive.SelectHiveQL
>     SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d] Unable to execute
>     HiveQL select query select max(version) as max_partition, min(version)
> as
>     min_partition from Table_A for
>
> StandardFlowFileRecord[uuid=9253b1ab-b4a2-4547-b695-4dbf5f964ca2,claim=,offset=0,name=done,size=0]
>     due to org.apache.nifi.processor.exception.ProcessException: Error
> during
>     database query or conversion of records.; routing to failure:
>     org.apache.nifi.processor.exception.ProcessException: Error during
> database
>     query or conversion of records.
>     org.apache.nifi.processor.exception.ProcessException: Error during
> database
>     query or conversion of records.
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:407)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$627/1348460623.process(Unknown
>     Source)
>     at
>
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:2625)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:395)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$0(SelectHiveQL.java:285)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376.execute(Unknown
>     Source)
>     at
>
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114)
>     at
>
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
>     at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
>     at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
>     at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
>     Caused by: java.sql.SQLException: java.io.IOException:
>     java.lang.RuntimeException: native snappy library not available
>     at org.apache.hive.jdbc.Utils.verifySuccess(Utils.java:121)
>     at org.apache.hive.jdbc.Utils.verifySuccessWithInfo(Utils.java:109)
>     at
> org.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:330)
>     at
>
> org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
>     at
>
> org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:207)
>     at
>
> org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:355)
>     at
>
> org.apache.nifi.util.hive.HiveJdbcCommon.convertToCsvStream(HiveJdbcCommon.java:322)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.lambda$onTrigger$2(SelectHiveQL.java:401)
>     ... 18 common frames omitted
>     2019-05-14 20:37:30,938 ERROR [Timer-Driven Process Thread-6]
>     o.a.nifi.processors.hive.SelectHiveQL
>     SelectHiveQL[id=365c2597-9483-302c-90d0-c7d2d7ce471d]
>
> org.apache.nifi.processors.hive.SelectHiveQL$$Lambda$622/285174376@54eb300d
>     failed to process due to
>     org.apache.nifi.processor.exception.FlowFileHandlingException:
>
> StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
>     [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
>     container=default, section=28], offset=81777,
>     length=280],offset=0,name=done,size=280] transfer relationship not
>     specified; rolling back session:
>     org.apache.nifi.processor.exception.FlowFileHandlingException:
>
> StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
>     [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
>     container=default, section=28], offset=81777,
>     length=280],offset=0,name=done,size=280] transfer relationship not
> specified
>     org.apache.nifi.processor.exception.FlowFileHandlingException:
>
> StandardFlowFileRecord[uuid=e2688819-edc9-406a-b5da-050ac072cd43,claim=StandardContentClaim
>     [resourceClaim=StandardResourceClaim[id=1557687270864-64540,
>     container=default, section=28], offset=81777,
>     length=280],offset=0,name=done,size=280] transfer relationship not
> specified
>     at
>
> org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:264)
>     at
>
> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:335)
>     at
>
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:115)
>     at
>
> org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:106)
>     at
>
> org.apache.nifi.processors.hive.SelectHiveQL.onTrigger(SelectHiveQL.java:285)
>     at
>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165)
>     at
>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203)
>     at
>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
>     at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>     at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>     at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>     at
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>     at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>     at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>     at java.lang.Thread.run(Thread.java:745)
>
>
>     Thanks
>     David
>
>
>