How to get upstream connections or parent Processors

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

How to get upstream connections or parent Processors

Huang, Jie
Hi all,

I need to get the connection information or parents for further action in onTriger event.
Is there any way to get the upstream connections or parent processors from any customized Processor?

Thank you && Best Regards,
Grace (Huang Jie)

Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Rick Braddy
Me, too.

And I need the ability to determine which connection(s) are linked to each relationship, along with a way to auto-terminate unused relationships from my processor.

Thanks
Rick

> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
>
> Hi all,
>
> I need to get the connection information or parents for further action in onTriger event.
> Is there any way to get the upstream connections or parent processors from any customized Processor?
>
> Thank you && Best Regards,
> Grace (Huang Jie)
>
Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Joe Witt
Hello

Addressing Grace's question:

What you are looking for is 'context' and the way context is included
is on attributes of a given FlowFile.  You can, for example, use
UpdateAttribute to set that context in the proper part of the flow
preceding your custom processor.

Addressing Rick's comment:

I think we mentioned we could add a way to annotate relationships on
processors as being auto-terminated by default.  Not aware of a JIRA
for this yet though.  It simply provides a way for the processor to
developer to signal the default behavior is that a relationship is
unused and that is ok vs our current approach which says each
relationship must be chosen as terminated or connected.

Thanks
Joe

On Mon, Sep 21, 2015 at 11:26 PM, Rick Braddy <[hidden email]> wrote:

> Me, too.
>
> And I need the ability to determine which connection(s) are linked to each relationship, along with a way to auto-terminate unused relationships from my processor.
>
> Thanks
> Rick
>
>> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
>>
>> Hi all,
>>
>> I need to get the connection information or parents for further action in onTriger event.
>> Is there any way to get the upstream connections or parent processors from any customized Processor?
>>
>> Thank you && Best Regards,
>> Grace (Huang Jie)
>>
Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Bryan Bende
Not sure if this is exactly what everyone is looking for, but if you are
working against master (0.3.1-SNAPSHOT) then the ProcessContext interface
now has two new methods:

boolean hasIncomingConnection();
boolean hasConnection(Relationship relationship);

This way a processor can check hasIncomingConnection() to know if it should
expect incoming FlowFiles, and can check hasConnection(Relationship) to
know if it should even bother transferring a FlowFile to an outgoing
relationship.
These were added in support of making ExecuteSQL work with and without
incoming FlowFiles [1].

-Bryan

[1] https://issues.apache.org/jira/browse/NIFI-932


On Tue, Sep 22, 2015 at 6:32 AM, Joe Witt <[hidden email]> wrote:

> Hello
>
> Addressing Grace's question:
>
> What you are looking for is 'context' and the way context is included
> is on attributes of a given FlowFile.  You can, for example, use
> UpdateAttribute to set that context in the proper part of the flow
> preceding your custom processor.
>
> Addressing Rick's comment:
>
> I think we mentioned we could add a way to annotate relationships on
> processors as being auto-terminated by default.  Not aware of a JIRA
> for this yet though.  It simply provides a way for the processor to
> developer to signal the default behavior is that a relationship is
> unused and that is ok vs our current approach which says each
> relationship must be chosen as terminated or connected.
>
> Thanks
> Joe
>
> On Mon, Sep 21, 2015 at 11:26 PM, Rick Braddy <[hidden email]> wrote:
> > Me, too.
> >
> > And I need the ability to determine which connection(s) are linked to
> each relationship, along with a way to auto-terminate unused relationships
> from my processor.
> >
> > Thanks
> > Rick
> >
> >> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
> >>
> >> Hi all,
> >>
> >> I need to get the connection information or parents for further action
> in onTriger event.
> >> Is there any way to get the upstream connections or parent processors
> from any customized Processor?
> >>
> >> Thank you && Best Regards,
> >> Grace (Huang Jie)
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Corey Flowers
I would be a little weary of auto terminating relationships that aren't
defined. I would expect that to lead to unwanted data loss. If that is
implemented, can we make that a configuration option or something? I would
not want that to be the default in my graphs for sure.

On Tue, Sep 22, 2015 at 9:05 AM, Bryan Bende <[hidden email]> wrote:

> Not sure if this is exactly what everyone is looking for, but if you are
> working against master (0.3.1-SNAPSHOT) then the ProcessContext interface
> now has two new methods:
>
> boolean hasIncomingConnection();
> boolean hasConnection(Relationship relationship);
>
> This way a processor can check hasIncomingConnection() to know if it should
> expect incoming FlowFiles, and can check hasConnection(Relationship) to
> know if it should even bother transferring a FlowFile to an outgoing
> relationship.
> These were added in support of making ExecuteSQL work with and without
> incoming FlowFiles [1].
>
> -Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-932
>
>
> On Tue, Sep 22, 2015 at 6:32 AM, Joe Witt <[hidden email]> wrote:
>
> > Hello
> >
> > Addressing Grace's question:
> >
> > What you are looking for is 'context' and the way context is included
> > is on attributes of a given FlowFile.  You can, for example, use
> > UpdateAttribute to set that context in the proper part of the flow
> > preceding your custom processor.
> >
> > Addressing Rick's comment:
> >
> > I think we mentioned we could add a way to annotate relationships on
> > processors as being auto-terminated by default.  Not aware of a JIRA
> > for this yet though.  It simply provides a way for the processor to
> > developer to signal the default behavior is that a relationship is
> > unused and that is ok vs our current approach which says each
> > relationship must be chosen as terminated or connected.
> >
> > Thanks
> > Joe
> >
> > On Mon, Sep 21, 2015 at 11:26 PM, Rick Braddy <[hidden email]>
> wrote:
> > > Me, too.
> > >
> > > And I need the ability to determine which connection(s) are linked to
> > each relationship, along with a way to auto-terminate unused
> relationships
> > from my processor.
> > >
> > > Thanks
> > > Rick
> > >
> > >> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
> > >>
> > >> Hi all,
> > >>
> > >> I need to get the connection information or parents for further action
> > in onTriger event.
> > >> Is there any way to get the upstream connections or parent processors
> > from any customized Processor?
> > >>
> > >> Thank you && Best Regards,
> > >> Grace (Huang Jie)
> > >>
> >
>



--
Corey Flowers
Vice President, Onyx Point, Inc
(410) 541-6699
[hidden email]

-- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Rick Braddy
Yeah. I agree and only want to auto terminate at runtime in certain cases and not as a general rule or by default.

> On Sep 22, 2015, at 8:22 AM, Corey Flowers <[hidden email]> wrote:
>
> I would be a little weary of auto terminating relationships that aren't
> defined. I would expect that to lead to unwanted data loss. If that is
> implemented, can we make that a configuration option or something? I would
> not want that to be the default in my graphs for sure.
>
>> On Tue, Sep 22, 2015 at 9:05 AM, Bryan Bende <[hidden email]> wrote:
>>
>> Not sure if this is exactly what everyone is looking for, but if you are
>> working against master (0.3.1-SNAPSHOT) then the ProcessContext interface
>> now has two new methods:
>>
>> boolean hasIncomingConnection();
>> boolean hasConnection(Relationship relationship);
>>
>> This way a processor can check hasIncomingConnection() to know if it should
>> expect incoming FlowFiles, and can check hasConnection(Relationship) to
>> know if it should even bother transferring a FlowFile to an outgoing
>> relationship.
>> These were added in support of making ExecuteSQL work with and without
>> incoming FlowFiles [1].
>>
>> -Bryan
>>
>> [1] https://issues.apache.org/jira/browse/NIFI-932
>>
>>
>>> On Tue, Sep 22, 2015 at 6:32 AM, Joe Witt <[hidden email]> wrote:
>>>
>>> Hello
>>>
>>> Addressing Grace's question:
>>>
>>> What you are looking for is 'context' and the way context is included
>>> is on attributes of a given FlowFile.  You can, for example, use
>>> UpdateAttribute to set that context in the proper part of the flow
>>> preceding your custom processor.
>>>
>>> Addressing Rick's comment:
>>>
>>> I think we mentioned we could add a way to annotate relationships on
>>> processors as being auto-terminated by default.  Not aware of a JIRA
>>> for this yet though.  It simply provides a way for the processor to
>>> developer to signal the default behavior is that a relationship is
>>> unused and that is ok vs our current approach which says each
>>> relationship must be chosen as terminated or connected.
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mon, Sep 21, 2015 at 11:26 PM, Rick Braddy <[hidden email]>
>> wrote:
>>>> Me, too.
>>>>
>>>> And I need the ability to determine which connection(s) are linked to
>>> each relationship, along with a way to auto-terminate unused
>> relationships
>>> from my processor.
>>>>
>>>> Thanks
>>>> Rick
>>>>
>>>>> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I need to get the connection information or parents for further action
>>> in onTriger event.
>>>>> Is there any way to get the upstream connections or parent processors
>>> from any customized Processor?
>>>>>
>>>>> Thank you && Best Regards,
>>>>> Grace (Huang Jie)
>
>
>
> --
> Corey Flowers
> Vice President, Onyx Point, Inc
> (410) 541-6699
> [hidden email]
>
> -- This account not approved for unencrypted proprietary information --
Reply | Threaded
Open this post in threaded view
|

Re: How to get upstream connections or parent Processors

Rick Braddy
In reply to this post by Bryan Bende
Bryan,

Yes, this is what I need.

Also, need a call like terminateConnection( boolean autoTerminate, Relationship rel )

Need this at runtime as relationships are dynamically created.

Thank you.
Rick

> On Sep 22, 2015, at 8:06 AM, Bryan Bende <[hidden email]> wrote:
>
> Not sure if this is exactly what everyone is looking for, but if you are
> working against master (0.3.1-SNAPSHOT) then the ProcessContext interface
> now has two new methods:
>
> boolean hasIncomingConnection();
> boolean hasConnection(Relationship relationship);
>
> This way a processor can check hasIncomingConnection() to know if it should
> expect incoming FlowFiles, and can check hasConnection(Relationship) to
> know if it should even bother transferring a FlowFile to an outgoing
> relationship.
> These were added in support of making ExecuteSQL work with and without
> incoming FlowFiles [1].
>
> -Bryan
>
> [1] https://issues.apache.org/jira/browse/NIFI-932
>
>
>> On Tue, Sep 22, 2015 at 6:32 AM, Joe Witt <[hidden email]> wrote:
>>
>> Hello
>>
>> Addressing Grace's question:
>>
>> What you are looking for is 'context' and the way context is included
>> is on attributes of a given FlowFile.  You can, for example, use
>> UpdateAttribute to set that context in the proper part of the flow
>> preceding your custom processor.
>>
>> Addressing Rick's comment:
>>
>> I think we mentioned we could add a way to annotate relationships on
>> processors as being auto-terminated by default.  Not aware of a JIRA
>> for this yet though.  It simply provides a way for the processor to
>> developer to signal the default behavior is that a relationship is
>> unused and that is ok vs our current approach which says each
>> relationship must be chosen as terminated or connected.
>>
>> Thanks
>> Joe
>>
>>> On Mon, Sep 21, 2015 at 11:26 PM, Rick Braddy <[hidden email]> wrote:
>>> Me, too.
>>>
>>> And I need the ability to determine which connection(s) are linked to
>> each relationship, along with a way to auto-terminate unused relationships
>> from my processor.
>>>
>>> Thanks
>>> Rick
>>>
>>>> On Sep 21, 2015, at 10:22 PM, Huang, Jie <[hidden email]> wrote:
>>>>
>>>> Hi all,
>>>>
>>>> I need to get the connection information or parents for further action
>> in onTriger event.
>>>> Is there any way to get the upstream connections or parent processors
>> from any customized Processor?
>>>>
>>>> Thank you && Best Regards,
>>>> Grace (Huang Jie)
>>