MergeContent processor - set demarcator as new line

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

MergeContent processor - set demarcator as new line

Igor Kravzov-2
I want to use MergeContent processor to merge tweets to bulk insert into Elasticsearch index. For this I need command and tweets (each) to be separated by \n.

This is how it should look like
{ action: { metadata }}\n
{ request body        }\n

Which will be
{ "index" )\n
{ tweet1 }\n
{ twee2 }\n

When I put \n as demarcator the processor actually adds \n as a string instead of new line separator. Is it possible to make it actual new line?
Also is it possible to leave or make footer empty?

Thanks in advance.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Aldrin Piri
You are correct in that it takes the literal representation of what is
entered. There are two ways to handle this:

   1. With delimiter strategy of 'Text': Hit shift + enter to provide a new
   line
   2. With delimiter strategy of 'Filename': Provide a path to a file that
   is only a newline character

With regards to your question on leaving the footer empty, it should work
as you anticipate for the Merge Format of 'Binary Concatenation.' Making a
quick test flow, I have verified this, but please let us know if it is not
working for your particular configuration.

On Sun, Dec 13, 2015 at 6:18 PM, Igor Kravzov <[hidden email]>
wrote:

> I want to use MergeContent processor to merge tweets to bulk insert into
> Elasticsearch index. For this I need command and tweets (each) to be
> separated by \n.
>
> This is how it should look like
> { action: { metadata }}\n
> { request body        }\n
>
> Which will be
> { "index" )\n
> { tweet1 }\n
> { twee2 }\n
>
> When I put \n as demarcator the processor actually adds \n as a string
> instead of new line separator. Is it possible to make it actual new line?
> Also is it possible to leave or make footer empty?
>
> Thanks in advance.
>
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Matt Burgess
Speaking of Elasticsearch, are you using InvokeHttp for the bulk insert, or something else? There is a Jira case (NIFI-1275) to add Elasticsearch processor(s) to NiFi, we welcome all thoughts and comments there on how to make things easier.

Regards,
Matt

Sent from my iPhone

> On Dec 13, 2015, at 6:57 PM, Aldrin Piri <[hidden email]> wrote:
>
> You are correct in that it takes the literal representation of what is
> entered. There are two ways to handle this:
>
>   1. With delimiter strategy of 'Text': Hit shift + enter to provide a new
>   line
>   2. With delimiter strategy of 'Filename': Provide a path to a file that
>   is only a newline character
>
> With regards to your question on leaving the footer empty, it should work
> as you anticipate for the Merge Format of 'Binary Concatenation.' Making a
> quick test flow, I have verified this, but please let us know if it is not
> working for your particular configuration.
>
> On Sun, Dec 13, 2015 at 6:18 PM, Igor Kravzov <[hidden email]>
> wrote:
>
>> I want to use MergeContent processor to merge tweets to bulk insert into
>> Elasticsearch index. For this I need command and tweets (each) to be
>> separated by \n.
>>
>> This is how it should look like
>> { action: { metadata }}\n
>> { request body        }\n
>>
>> Which will be
>> { "index" )\n
>> { tweet1 }\n
>> { twee2 }\n
>>
>> When I put \n as demarcator the processor actually adds \n as a string
>> instead of new line separator. Is it possible to make it actual new line?
>> Also is it possible to leave or make footer empty?
>>
>> Thanks in advance.
>>
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
Hi Matt,

I will try to use bot PostHTML and InvokeHTML and see what happens.
Regarding to make things easier... It would be nice to have the ability to insert one or multiple documents. The same with querying.
Not sure if it is possible to implement in one processor.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Aldrin Piri
Thanks Aldrin. I'll try that.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Aldrin Piri
HI Aldrin,

I just checked and if I leave footer empty I get NullPointerException. Checking Empty string as parameter makes processor invalid.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Joe Witt
Igor,

Bummer.  That was reported here
https://issues.apache.org/jira/browse/NIFI-982 and it doesn't look
like we've sorted it yet.

Can you please send the stacktrace of the NPE if it is present in the logs?

Thanks
Joe

On Sun, Dec 13, 2015 at 7:32 PM, Igor K. <[hidden email]> wrote:

> HI Aldrin,
>
> I just checked and if I leave footer empty I get NullPointerException.
> Checking Empty string as parameter makes processor invalid.
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5752.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
I probably should mention that I use NiFi packaged by Hortonworks as HDF. May it make a difference?
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Joe Witt
Here it is:

2015-12-13 20:44:54,848 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@202812b3 // Another save pending = false
2015-12-13 20:44:57,144 INFO [pool-28-thread-7] o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] to run with 1 threads
2015-12-13 20:44:57,146 ERROR [Timer-Driven Process Thread-10] o.a.n.processors.standard.MergeContent MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process session due to java.lang.NullPointerException: java.lang.NullPointerException
2015-12-13 20:44:57,150 WARN [Timer-Driven Process Thread-10] o.a.n.processors.standard.MergeContent MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor Administratively Yielded for 1 sec due to processing failure
2015-12-13 20:44:57,151 WARN [Timer-Driven Process Thread-10] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught Exception: java.lang.NullPointerException
2015-12-13 20:44:57,159 WARN [Timer-Driven Process Thread-10] o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.NullPointerException: null
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592) ~[na:na]
                at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453) ~[na:na]
                at org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225) ~[na:na]
                at org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182) ~[na:na]
                at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_66]
                at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
2015-12-13 20:44:57,375 INFO [Flow Service Tasks Thread-1] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@202812b3 // Another save pending = false
2015-12-13 20:44:59,168 ERROR [Timer-Driven Process Thread-7] o.a.n.processors.standard.MergeContent MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process session due to java.lang.NullPointerException: java.lang.NullPointerException
2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7] o.a.n.processors.standard.MergeContent MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor Administratively Yielded for 1 sec due to processing failure
2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught Exception: java.lang.NullPointerException
2015-12-13 20:44:59,171 WARN [Timer-Driven Process Thread-7] o.a.n.c.t.ContinuallyRunProcessorTask
java.lang.NullPointerException: null
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592) ~[na:na]
                at org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937) ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555) ~[na:na]
                at org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453) ~[na:na]
                at org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225) ~[na:na]
                at org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182) ~[na:na]
                at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077) ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119) [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
                at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.FutureTask.runAndReset(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) [na:1.8.0_66]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.8.0_66]
                at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Joe Witt
thanks.  updated the jira.  Will definitely tackle that for 0.4.1 or
0.5.0 whichever is next.

On Sun, Dec 13, 2015 at 8:07 PM, Igor K. <[hidden email]> wrote:

> Here it is:
>
> 2015-12-13 20:44:54,848 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@202812b3 // Another save pending =
> false
> 2015-12-13 20:44:57,144 INFO [pool-28-thread-7]
> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] to run with 1 threads
> 2015-12-13 20:44:57,146 ERROR [Timer-Driven Process Thread-10]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1]
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process
> session due to java.lang.NullPointerException:
> java.lang.NullPointerException
> 2015-12-13 20:44:57,150 WARN [Timer-Driven Process Thread-10]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2015-12-13 20:44:57,151 WARN [Timer-Driven Process Thread-10]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught
> Exception: java.lang.NullPointerException
> 2015-12-13 20:44:57,159 WARN [Timer-Driven Process Thread-10]
> o.a.n.c.t.ContinuallyRunProcessorTask
> java.lang.NullPointerException: null
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592)
> ~[na:na]
>                 at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937)
> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182)
> ~[na:na]
>                 at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_66]
>                 at java.util.concurrent.FutureTask.runAndReset(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_66]
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_66]
>                 at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
> 2015-12-13 20:44:57,375 INFO [Flow Service Tasks Thread-1]
> o.a.nifi.controller.StandardFlowService Saved flow controller
> org.apache.nifi.controller.FlowController@202812b3 // Another save pending =
> false
> 2015-12-13 20:44:59,168 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1]
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process
> session due to java.lang.NullPointerException:
> java.lang.NullPointerException
> 2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.MergeContent
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor
> Administratively Yielded for 1 sec due to processing failure
> 2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught
> Exception: java.lang.NullPointerException
> 2015-12-13 20:44:59,171 WARN [Timer-Driven Process Thread-7]
> o.a.n.c.t.ContinuallyRunProcessorTask
> java.lang.NullPointerException: null
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592)
> ~[na:na]
>                 at
> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937)
> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225)
> ~[na:na]
>                 at
> org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182)
> ~[na:na]
>                 at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>                 at
> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
> [na:1.8.0_66]
>                 at java.util.concurrent.FutureTask.runAndReset(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
> Source) [na:1.8.0_66]
>                 at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source) [na:1.8.0_66]
>                 at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> [na:1.8.0_66]
>                 at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5755.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Mark Payne
Igor,

As a workaround in the meantime, would using a space as a footer work? Or perhaps the expression ${literal('')}

Clearly not ideal but may help you get going until we get that bug sorted out.

Thanks
-Mark

Sent from my iPhone

> On Dec 13, 2015, at 8:56 PM, Joe Witt <[hidden email]> wrote:
>
> thanks.  updated the jira.  Will definitely tackle that for 0.4.1 or
> 0.5.0 whichever is next.
>
>> On Sun, Dec 13, 2015 at 8:07 PM, Igor K. <[hidden email]> wrote:
>> Here it is:
>>
>> 2015-12-13 20:44:54,848 INFO [Flow Service Tasks Thread-1]
>> o.a.nifi.controller.StandardFlowService Saved flow controller
>> org.apache.nifi.controller.FlowController@202812b3 // Another save pending =
>> false
>> 2015-12-13 20:44:57,144 INFO [pool-28-thread-7]
>> o.a.n.c.s.TimerDrivenSchedulingAgent Scheduled
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] to run with 1 threads
>> 2015-12-13 20:44:57,146 ERROR [Timer-Driven Process Thread-10]
>> o.a.n.processors.standard.MergeContent
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1]
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process
>> session due to java.lang.NullPointerException:
>> java.lang.NullPointerException
>> 2015-12-13 20:44:57,150 WARN [Timer-Driven Process Thread-10]
>> o.a.n.processors.standard.MergeContent
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor
>> Administratively Yielded for 1 sec due to processing failure
>> 2015-12-13 20:44:57,151 WARN [Timer-Driven Process Thread-10]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught
>> Exception: java.lang.NullPointerException
>> 2015-12-13 20:44:57,159 WARN [Timer-Driven Process Thread-10]
>> o.a.n.c.t.ContinuallyRunProcessorTask
>> java.lang.NullPointerException: null
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592)
>> ~[na:na]
>>                at
>> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937)
>> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182)
>> ~[na:na]
>>                at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
>> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> [na:1.8.0_66]
>>                at java.util.concurrent.FutureTask.runAndReset(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source) [na:1.8.0_66]
>>                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> [na:1.8.0_66]
>>                at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
>> 2015-12-13 20:44:57,375 INFO [Flow Service Tasks Thread-1]
>> o.a.nifi.controller.StandardFlowService Saved flow controller
>> org.apache.nifi.controller.FlowController@202812b3 // Another save pending =
>> false
>> 2015-12-13 20:44:59,168 ERROR [Timer-Driven Process Thread-7]
>> o.a.n.processors.standard.MergeContent
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1]
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] failed to process
>> session due to java.lang.NullPointerException:
>> java.lang.NullPointerException
>> 2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7]
>> o.a.n.processors.standard.MergeContent
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] Processor
>> Administratively Yielded for 1 sec due to processing failure
>> 2015-12-13 20:44:59,168 WARN [Timer-Driven Process Thread-7]
>> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding
>> MergeContent[id=f760a07d-be74-4adc-8de5-8f2d2ef345a1] due to uncaught
>> Exception: java.lang.NullPointerException
>> 2015-12-13 20:44:59,171 WARN [Timer-Driven Process Thread-7]
>> o.a.n.c.t.ContinuallyRunProcessorTask
>> java.lang.NullPointerException: null
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterTextContent(MergeContent.java:644)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.getDelimiterContent(MergeContent.java:614)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.access$500(MergeContent.java:539)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge$1.process(MergeContent.java:592)
>> ~[na:na]
>>                at
>> org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1937)
>> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.processors.standard.MergeContent$BinaryConcatenationMerge.merge(MergeContent.java:555)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.MergeContent.processBin(MergeContent.java:453)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.BinFiles.processBins(BinFiles.java:225)
>> ~[na:na]
>>                at
>> org.apache.nifi.processors.standard.BinFiles.onTrigger(BinFiles.java:182)
>> ~[na:na]
>>                at
>> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1077)
>> ~[nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:127)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:49)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:119)
>> [nifi-framework-core-1.0.0.0-7.jar:1.0.0.0-7]
>>                at
>> java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
>> [na:1.8.0_66]
>>                at java.util.concurrent.FutureTask.runAndReset(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown
>> Source) [na:1.8.0_66]
>>                at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
>> Source) [na:1.8.0_66]
>>                at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
>> [na:1.8.0_66]
>>                at java.lang.Thread.run(Unknown Source) [na:1.8.0_66]
>>
>>
>>
>> --
>> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5755.html
>> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Matt Burgess
In reply to this post by Igor K.
I'm thinking a PutElasticsearch processor to do bulk uploads of 1+ flow files (the batch size configurable), and another Get/Search Elasticsearch to use the search capability. Do you picture others, and/or more advanced options like specifying the index and creating it if it does not exist? Your experience and input are most welcome :)

Regards,
Matt

Sent from my iPhone

> On Dec 13, 2015, at 7:08 PM, Igor K. <[hidden email]> wrote:
>
> Hi Matt,
>
> I will try to use bot PostHTML and InvokeHTML and see what happens.
> Regarding to make things easier... It would be nice to have the ability to
> insert one or multiple documents. The same with querying.
> Not sure if it is possible to implement in one processor.
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5750.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Bryan Bende
I think the NullPointerException is resolved in 0.4.0 with this ticket:
https://issues.apache.org/jira/browse/NIFI-975

The stacktrace above coming from line 644 which used to be:

property =
context.getProperty(descriptor).evaluateAttributeExpressions(flowFile).getValue().getBytes();

and is now:

final String value =
context.getProperty(descriptor).evaluateAttributeExpressions(flowFile).getValue();
if (value != null) {
  property = value.getBytes();
}


On Sun, Dec 13, 2015 at 9:02 PM, Matt Burgess <[hidden email]> wrote:

> I'm thinking a PutElasticsearch processor to do bulk uploads of 1+ flow
> files (the batch size configurable), and another Get/Search Elasticsearch
> to use the search capability. Do you picture others, and/or more advanced
> options like specifying the index and creating it if it does not exist?
> Your experience and input are most welcome :)
>
> Regards,
> Matt
>
> Sent from my iPhone
>
> > On Dec 13, 2015, at 7:08 PM, Igor K. <[hidden email]> wrote:
> >
> > Hi Matt,
> >
> > I will try to use bot PostHTML and InvokeHTML and see what happens.
> > Regarding to make things easier... It would be nice to have the ability
> to
> > insert one or multiple documents. The same with querying.
> > Not sure if it is possible to implement in one processor.
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5750.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Joe Witt
Bryan,

If you find that it is can you link it to
https://issues.apache.org/jira/browse/NIFI-982 and mark it as a dupe
if it does indeed cover the bases of that ticket.

Thanks
Joe

On Sun, Dec 13, 2015 at 9:41 PM, Bryan Bende <[hidden email]> wrote:

> I think the NullPointerException is resolved in 0.4.0 with this ticket:
> https://issues.apache.org/jira/browse/NIFI-975
>
> The stacktrace above coming from line 644 which used to be:
>
> property =
> context.getProperty(descriptor).evaluateAttributeExpressions(flowFile).getValue().getBytes();
>
> and is now:
>
> final String value =
> context.getProperty(descriptor).evaluateAttributeExpressions(flowFile).getValue();
> if (value != null) {
>   property = value.getBytes();
> }
>
>
> On Sun, Dec 13, 2015 at 9:02 PM, Matt Burgess <[hidden email]> wrote:
>
>> I'm thinking a PutElasticsearch processor to do bulk uploads of 1+ flow
>> files (the batch size configurable), and another Get/Search Elasticsearch
>> to use the search capability. Do you picture others, and/or more advanced
>> options like specifying the index and creating it if it does not exist?
>> Your experience and input are most welcome :)
>>
>> Regards,
>> Matt
>>
>> Sent from my iPhone
>>
>> > On Dec 13, 2015, at 7:08 PM, Igor K. <[hidden email]> wrote:
>> >
>> > Hi Matt,
>> >
>> > I will try to use bot PostHTML and InvokeHTML and see what happens.
>> > Regarding to make things easier... It would be nice to have the ability
>> to
>> > insert one or multiple documents. The same with querying.
>> > Not sure if it is possible to implement in one processor.
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5750.html
>> > Sent from the Apache NiFi Developer List mailing list archive at
>> Nabble.com.
>>
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Matt Burgess
I would definitely make index and type as property with ability of expression language. I think there is no need to make option for creating an index because ES creates index automatically if does not exist.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Mark Payne
No Mark, it didn't help :(
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Joe Witt
In reply to this post by Igor K.
No.  Just means you're on apache nifi 0.3.0 though.  As Bryan pointed
out 0.4.0 (just released) this may be fixed.

Thanks
Joe

On Sun, Dec 13, 2015 at 7:45 PM, Igor K. <[hidden email]> wrote:
> I probably should mention that I use NiFi packaged by Hortonworks as HDF. May
> it make a difference?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5754.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
OH. Definitely I'll download 0.4.0 tomorrow and check :)

On Sun, Dec 13, 2015 at 10:25 PM, Joe Witt <[hidden email]> wrote:

> No.  Just means you're on apache nifi 0.3.0 though.  As Bryan pointed
> out 0.4.0 (just released) this may be fixed.
>
> Thanks
> Joe
>
> On Sun, Dec 13, 2015 at 7:45 PM, Igor K. <[hidden email]> wrote:
> > I probably should mention that I use NiFi packaged by Hortonworks as
> HDF. May
> > it make a difference?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/MergeContent-processor-set-demarcator-as-new-line-tp5747p5754.html
> > Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: MergeContent processor - set demarcator as new line

Igor K.
In reply to this post by Joe Witt
I downloaded 0.4.0 and looks like it works!!! Thank you.
Let's see now if ES will accept my file :)