ExecuteProcess Question

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ExecuteProcess Question

dale.chang13
So I have a bash script that I am able to run from the command line, and I want to be able to let NiFi call it using the ExecuteProcess processor.

The script itself runs fine from the command line, and it looks like the ExecuteProcess is executing the script as well (I have a LogAttribute processor as a downstream processor that verifies that no problems were executing the script), but neither the Bulletin nor the logs tell me if anything is wrong or successful.

I found that the echo commands in the script should write to the NiFi FlowFile Content, but I do not see anything show up.



The script is simply a java -jar xx.jar file, which happens to contain a java wrapper class with a main method that calls a scala main object that then performs Apache Spark operations.

Any ideas?
Reply | Threaded
Open this post in threaded view
|

Re: ExecuteProcess Question

Andy LoPresto-2
Hi Dale,

I just tried to replicate this and I’m not sure I fully understand the issue. 

You can see the actual contents of my Java class [1], Bash script [2], and command-line activity [3] in the Gists provided. I then set up a flow [4] which simply executed the script every 5 seconds and logged the output. I saw both the output from System.out.println() and System.err.println() as the content of the flowfile [5]. I did need to set the RedirectErrorStream property in ExecuteProcess to capture the error output as well. 

After doing this, I re-read your question and notice you mention that the “echo commands” are not showing up. Does this refer to echo output from the bash script itself? I also replicated this [6][7]. 

If you are referring to output from the Scala code, is this perhaps being indirected via logging mechanism? If you run the command directly from the command line, does this output appear in the standard output console?

Can you please clarify what I misunderstood from your question or let me know if I missed something. Thanks. 


Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

On Sep 27, 2016, at 2:40 PM, dale.chang13 <[hidden email]> wrote:

So I have a bash script that I am able to run from the command line, and I
want to be able to let NiFi call it using the ExecuteProcess processor.

The script itself runs fine from the command line, and it looks like the
ExecuteProcess is executing the script as well (I have a LogAttribute
processor as a downstream processor that verifies that no problems were
executing the script), but neither the Bulletin nor the logs tell me if
anything is wrong or successful.

I found that the echo commands in the script should write to the NiFi
FlowFile Content, but I do not see anything show up.



The script is simply a java -jar xx.jar file, which happens to contain a
java wrapper class with a main method that calls a scala main object that
then performs Apache Spark operations.

Any ideas?



--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/ExecuteProcess-Question-tp13471.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ExecuteProcess Question

leelaim
Hi Dale,

You could also check the permissions on the script.  I was able to echo
text from a bash script which placed the text in the flowfile content.

Thanks,
Lee





On Tue, Sep 27, 2016 at 6:32 PM, Andy LoPresto <[hidden email]> wrote:

> Hi Dale,
>
> I just tried to replicate this and I’m not sure I fully understand the
> issue.
>
> You can see the actual contents of my Java class [1], Bash script [2], and
> command-line activity [3] in the Gists provided. I then set up a flow [4]
> which simply executed the script every 5 seconds and logged the output. I
> saw both the output from System.out.println() and System.err.println() as
> the content of the flowfile [5]. I did need to set the RedirectErrorStream
> property in ExecuteProcess to capture the error output as well.
>
> After doing this, I re-read your question and notice you mention that the
> “echo commands” are not showing up. Does this refer to echo output from the
> bash script itself? I also replicated this [6][7].
>
> If you are referring to output from the Scala code, is this perhaps being
> indirected via logging mechanism? If you run the command directly from the
> command line, does this output appear in the standard output console?
>
> Can you please clarify what I misunderstood from your question or let me
> know if I missed something. Thanks.
>
> [1] https://gist.github.com/alopresto/f71a85793cabcb22917546b7e504fe00
> [2] https://gist.github.com/alopresto/b166a7a9ccc3347541aaa24410883d4b/
> c4f2d7e7cda5d40defafe3796f7587915612c5ac
> [3] https://gist.github.com/alopresto/1157967ec70da8a25c176c82911613c8
> [4] https://gist.github.com/alopresto/9ef34727a7e72bb81944c214f921078a
> [5] https://gist.github.com/alopresto/72ba8f56495813e3fd0f0d9b9165d0e7
> [6] https://gist.github.com/alopresto/b166a7a9ccc3347541aaa24410883d4b/
> ae888959fec099afc951dea23186b681329d4b22
> [7] https://gist.github.com/alopresto/9c93a5b3daed983219f43c7766450e90
>
> Andy LoPresto
> [hidden email]
> *[hidden email] <[hidden email]>*
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Sep 27, 2016, at 2:40 PM, dale.chang13 <[hidden email]>
> wrote:
>
> So I have a bash script that I am able to run from the command line, and I
> want to be able to let NiFi call it using the ExecuteProcess processor.
>
> The script itself runs fine from the command line, and it looks like the
> ExecuteProcess is executing the script as well (I have a LogAttribute
> processor as a downstream processor that verifies that no problems were
> executing the script), but neither the Bulletin nor the logs tell me if
> anything is wrong or successful.
>
> I found that the echo commands in the script should write to the NiFi
> FlowFile Content, but I do not see anything show up.
>
>
>
> The script is simply a java -jar xx.jar file, which happens to contain a
> java wrapper class with a main method that calls a scala main object that
> then performs Apache Spark operations.
>
> Any ideas?
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/ExecuteProcess-Question-tp13471.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com <http://nabble.com>.
>
>
>