AttributesToJson Period

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

AttributesToJson Period

Ronnie Dove
Is there anyway to change all attribute names to exclude the period (.)?
For example, we have recently ran into an issue where ElasticSearch stopped
allowing users to put period/dot's (
https://discuss.elastic.co/t/field-name-cannot-contain/33251) in field
names (we were automatically using AttributesToJson to do this).  I came up
with a workaround by modifying the output of AttributesToJson with
ReplaceText but....I wish there was an easier way.

Additionally PutMongoDB's processor is trying to insert dotted field names
and bson's library does not allow that either.

Is there any advice or a solution available that would quickly convert my
attributes to have an underscore instead of dot.  Or would it require
changing of these processors to support a custom attribute field name
replacement.  Or is there a config for the delimiter to use in field names
of the attributes.

--
Ronnie Dove
Senior Game Engine Developer
http://www.techtunk.com
Reply | Threaded
Open this post in threaded view
|

Re: AttributesToJson Period

Bryan Bende
Could you use UpdateAttribute to rename attributes before getting to
AttributesToJson?

For example, if you had my.attribute then in UpdateAttribute make it
my_attribute = ${my.attribute}

An alternative solution for the future might be the TransformJson processor
that is coming, which I believe can rename fields in the JSON:
https://issues.apache.org/jira/browse/NIFI-361


On Wed, May 4, 2016 at 7:49 PM, Ronnie Dove <[hidden email]> wrote:

> Is there anyway to change all attribute names to exclude the period (.)?
> For example, we have recently ran into an issue where ElasticSearch stopped
> allowing users to put period/dot's (
> https://discuss.elastic.co/t/field-name-cannot-contain/33251) in field
> names (we were automatically using AttributesToJson to do this).  I came up
> with a workaround by modifying the output of AttributesToJson with
> ReplaceText but....I wish there was an easier way.
>
> Additionally PutMongoDB's processor is trying to insert dotted field names
> and bson's library does not allow that either.
>
> Is there any advice or a solution available that would quickly convert my
> attributes to have an underscore instead of dot.  Or would it require
> changing of these processors to support a custom attribute field name
> replacement.  Or is there a config for the delimiter to use in field names
> of the attributes.
>
> --
> Ronnie Dove
> Senior Game Engine Developer
> http://www.techtunk.com
>
Reply | Threaded
Open this post in threaded view
|

Re: AttributesToJson Period

Ronnie Dove
Thanks for the help.  We have a lot of attributes to keep track of so its
going to be rough but I will do that for now.  Another thing I think would
be nice to have is a way to tell JSON that one of the fields is an ISO Date
object on the insert.  For now doing some trickery with the data to get it
in the right format in JSON but means I am manipulating the content more
than I would like.  But would like to be able to define types to the Mongo
data for numbers, floats, strings, dates in the future.

{
   "S": "someString",
   "N": 123,
   "F": 12.3,
   "D": {"$date": 1352540684243}}
Reply | Threaded
Open this post in threaded view
|

Re: AttributesToJson Period

ttzhang
Ronnie Dove,
 i have a problem of ISODate to mongodb, i use JoltTransformJson processor
convert{"time":"2018-10-26","count":11} to
{"time":{"date":"2018-10-26"},"count":11},but i need a json structure is
{"time":{"$date":"2018-10-26"},"count":11},Do you know how I should fix this
or any other method I should try? Any advice is appreciated.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Reply | Threaded
Open this post in threaded view
|

Re: AttributesToJson Period

Mike Thomsen
If the end goal is to put that into Mongo, I would strongly recommend using
PutMongoRecord and use the record API's capabilities to insert a timestamp
instead of messing with Mongo Extended JSON (your $date issue from what I
can tell). You would set the date field's data type to be a long with, I
think timestamp-millis, as the logicalType (all of the syntax is documented
on the Avro site) and then set up the timestamp parser field in the JSON
reader to have the Java date format syntax for an ISO 8601 string.

On Fri, Oct 26, 2018 at 6:58 AM ttzhang <[hidden email]> wrote:

> Ronnie Dove,
>  i have a problem of ISODate to mongodb, i use JoltTransformJson processor
> convert{"time":"2018-10-26","count":11} to
> {"time":{"date":"2018-10-26"},"count":11},but i need a json structure is
> {"time":{"$date":"2018-10-26"},"count":11},Do you know how I should fix
> this
> or any other method I should try? Any advice is appreciated.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Reply | Threaded
Open this post in threaded view
|

Re: AttributesToJson Period

ttzhang
Thanks for your response. I tried this method,but i can not work out the
expected result. i use PutMongoRecord and set record reader is
JsonTreeReader, and my schema is
{
        "namespace": "nifi",
        "name": "userRecord",
        "type": "record",
        "fields": [
                { "name": "time", "type" : {
                                                "type" : "long",
    "logicalType" : "timestamp-millis"
    }
                },
                { "name": "count", "type": ["null","long"] }
                        ]
}
The error is: successfully parsed a json object from input but failed to
convert into a record object with the given schema.
what's wrong with this? thanks you in advance.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/