Elasticsearch dynamic index name from data attribute possible?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Elasticsearch dynamic index name from data attribute possible?

shahbazatta
Hi,
We have dynamic indexes in elastic search like signal-YYYY.MM.DD => (which resolved as signal-2017.03.31)
I was try hard to find the way to specify the dynamic index in nifi processor.
I already have a field in my data which i want to use as dynamic index. for example "index_name". I just wanted to use that data field in the index parameter of nifi processor.
It might be simple, but i tried hard with different options like '${index_name}' , "${index_name}" , '${INDEX_NAME}' ,  $INDEX_NAME , $index_name

Can anyone just refer me how to use data field in the nifi's elasticsearch processor property of index name?
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch dynamic index name from data attribute possible?

Chad Zobrisky
Hello,

You need the index_name field to be a flow file attribute to be able to use
the expression language to set the index name dynamically.

I do not think there is a processor that currently takes flow file content
and adds it as an attribute, but someone correct me if I am wrong.  One way
to do so would be to use the ExecuteScript[1] processor to find the
index_name field in the data and set an attribute "index_name" on the flow
file.  Example on how to read in flow file content and set an attribute in
python for the ExceuteScript Processor [2] .

Then in the ElasticSearch processor you would have access to the index_name
attribute through the expression language, "${index_name}".

Hope this helps,
Chad

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/
[2] http://www.nifi.rocks/using-the-executescript-processor/

On Fri, Mar 31, 2017 at 2:33 PM shahbazatta <[hidden email]> wrote:

Hi,
We have dynamic indexes in elastic search like signal-YYYY.MM.DD => (which
resolved as signal-2017.03.31)
I was try hard to find the way to specify the dynamic index in nifi
processor.
I already have a field in my data which i want to use as dynamic index. for
example "index_name". I just wanted to use that data field in the index
parameter of nifi processor.
It might be simple, but i tried hard with different options like
'${index_name}' , "${index_name}" , '${INDEX_NAME}' ,  $INDEX_NAME ,
$index_name

Can anyone just refer me how to use data field in the nifi's elasticsearch
processor property of index name?



--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/Elasticsearch-dynamic-index-name-from-data-attribute-possible-tp15327.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch dynamic index name from data attribute possible?

Chad Zobrisky
Thought about it a little more and since the flowfile is json, you could
use the EvaluateJsonPath[1] processor to set the attribute.

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPath/

On Sat, Apr 1, 2017 at 4:32 PM Chad Zobrisky <[hidden email]> wrote:

> Hello,
>
> You need the index_name field to be a flow file attribute to be able to
> use the expression language to set the index name dynamically.
>
> I do not think there is a processor that currently takes flow file content
> and adds it as an attribute, but someone correct me if I am wrong.  One way
> to do so would be to use the ExecuteScript[1] processor to find the
> index_name field in the data and set an attribute "index_name" on the flow
> file.  Example on how to read in flow file content and set an attribute in
> python for the ExceuteScript Processor [2] .
>
> Then in the ElasticSearch processor you would have access to the
> index_name attribute through the expression language, "${index_name}".
>
> Hope this helps,
> Chad
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/
> [2] http://www.nifi.rocks/using-the-executescript-processor/
>
> On Fri, Mar 31, 2017 at 2:33 PM shahbazatta <[hidden email]>
> wrote:
>
> Hi,
> We have dynamic indexes in elastic search like signal-YYYY.MM.DD => (which
> resolved as signal-2017.03.31)
> I was try hard to find the way to specify the dynamic index in nifi
> processor.
> I already have a field in my data which i want to use as dynamic index. for
> example "index_name". I just wanted to use that data field in the index
> parameter of nifi processor.
> It might be simple, but i tried hard with different options like
> '${index_name}' , "${index_name}" , '${INDEX_NAME}' ,  $INDEX_NAME ,
> $index_name
>
> Can anyone just refer me how to use data field in the nifi's elasticsearch
> processor property of index name?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Elasticsearch-dynamic-index-name-from-data-attribute-possible-tp15327.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch dynamic index name from data attribute possible?

shahbazatta

Thanks alot Chad,
I will try to resolve through this...

On Apr 2, 2017 5:39 AM, "Chad Zobrisky [via Apache NiFi Developer List]" <[hidden email]> wrote:
Thought about it a little more and since the flowfile is json, you could
use the EvaluateJsonPath[1] processor to set the attribute.

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPath/

On Sat, Apr 1, 2017 at 4:32 PM Chad Zobrisky <[hidden email]> wrote:

> Hello,
>
> You need the index_name field to be a flow file attribute to be able to
> use the expression language to set the index name dynamically.
>
> I do not think there is a processor that currently takes flow file content
> and adds it as an attribute, but someone correct me if I am wrong.  One way
> to do so would be to use the ExecuteScript[1] processor to find the
> index_name field in the data and set an attribute "index_name" on the flow
> file.  Example on how to read in flow file content and set an attribute in
> python for the ExceuteScript Processor [2] .
>
> Then in the ElasticSearch processor you would have access to the
> index_name attribute through the expression language, "${index_name}".
>
> Hope this helps,
> Chad
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/
> [2] http://www.nifi.rocks/using-the-executescript-processor/
>
> On Fri, Mar 31, 2017 at 2:33 PM shahbazatta <[hidden email]>
> wrote:
>
> Hi,
> We have dynamic indexes in elastic search like signal-YYYY.MM.DD => (which
> resolved as signal-2017.03.31)
> I was try hard to find the way to specify the dynamic index in nifi
> processor.
> I already have a field in my data which i want to use as dynamic index. for
> example "index_name". I just wanted to use that data field in the index
> parameter of nifi processor.
> It might be simple, but i tried hard with different options like
> '${index_name}' , "${index_name}" , '${INDEX_NAME}' ,  $INDEX_NAME ,
> $index_name
>
> Can anyone just refer me how to use data field in the nifi's elasticsearch
> processor property of index name?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Elasticsearch-dynamic-index-name-from-data-attribute-possible-tp15327.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>



To unsubscribe from Elasticsearch dynamic index name from data attribute possible?, click here.
NAML
Reply | Threaded
Open this post in threaded view
|

Re: Elasticsearch dynamic index name from data attribute possible?

shahbazatta
In reply to this post by Chad Zobrisky
Thanks a lot Chad,
It works like charm...

On Sun, Apr 2, 2017 at 11:18 AM, Shahbaz Atta <[hidden email]> wrote:

Thanks alot Chad,
I will try to resolve through this...

On Apr 2, 2017 5:39 AM, "Chad Zobrisky [via Apache NiFi Developer List]" <[hidden email]> wrote:
Thought about it a little more and since the flowfile is json, you could
use the EvaluateJsonPath[1] processor to set the attribute.

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.standard.EvaluateJsonPath/

On Sat, Apr 1, 2017 at 4:32 PM Chad Zobrisky <[hidden email]> wrote:

> Hello,
>
> You need the index_name field to be a flow file attribute to be able to
> use the expression language to set the index name dynamically.
>
> I do not think there is a processor that currently takes flow file content
> and adds it as an attribute, but someone correct me if I am wrong.  One way
> to do so would be to use the ExecuteScript[1] processor to find the
> index_name field in the data and set an attribute "index_name" on the flow
> file.  Example on how to read in flow file content and set an attribute in
> python for the ExceuteScript Processor [2] .
>
> Then in the ElasticSearch processor you would have access to the
> index_name attribute through the expression language, "${index_name}".
>
> Hope this helps,
> Chad
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.script.ExecuteScript/
> [2] http://www.nifi.rocks/using-the-executescript-processor/
>
> On Fri, Mar 31, 2017 at 2:33 PM shahbazatta <[hidden email]>
> wrote:
>
> Hi,
> We have dynamic indexes in elastic search like signal-YYYY.MM.DD => (which
> resolved as signal-2017.03.31)
> I was try hard to find the way to specify the dynamic index in nifi
> processor.
> I already have a field in my data which i want to use as dynamic index. for
> example "index_name". I just wanted to use that data field in the index
> parameter of nifi processor.
> It might be simple, but i tried hard with different options like
> '${index_name}' , "${index_name}" , '${INDEX_NAME}' ,  $INDEX_NAME ,
> $index_name
>
> Can anyone just refer me how to use data field in the nifi's elasticsearch
> processor property of index name?
>
>
>
> --
> View this message in context:
> http://apache-nifi-developer-list.39713.n7.nabble.com/Elasticsearch-dynamic-index-name-from-data-attribute-possible-tp15327.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>
>



To unsubscribe from Elasticsearch dynamic index name from data attribute possible?, click here.
NAML