Issue with ValidateRecord processor and Avro map data type

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Issue with ValidateRecord processor and Avro map data type

Maxime Jattiot
Hello everyone,

I am having hard time to validate a simple JSON message with an Avro schema. My schema works with avro-tools but doesn't with ValidateRecord processor. I wish you could help me.
  • I am having the following debug log within Nifi : 
Record #1 is invalid due to:
MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is of type org.apache.nifi.serialization.record.MapRecord but was expected to be of type MAP[STRING] 

  • Here is my schema :
  • Here is the JSON content sent :


Does someone has any clue ? I have been struggling 2 days with this map data type ... :(

Thank you !

Maxime
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Bryan Bende
Hello,

The screenshots did not come through, maybe just paste the schema and JSON
as text in the email.

Thanks.

On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <[hidden email]>
wrote:

> Hello everyone,
>
> I am having hard time to validate a simple JSON message with an Avro
> schema. My schema works with avro-tools but doesn't with ValidateRecord
> processor. I wish you could help me.
>
>    - I am having the following debug log within Nifi :
>
> Record #1 is invalid due to:
> MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> of type org.apache.nifi.serialization.record.MapRecord but was expected to
> be of type MAP[STRING]
>
>
>    - Here is my schema :
>
> [image: Screen Shot 2018-10-10 at 11.21.41.png]
>
>    - Here is the JSON content sent :
>
> [image: Screen Shot 2018-10-10 at 11.22.10.png]
>
> Does someone has any clue ? I have been struggling 2 days with this map
> data type ... :(
>
> Thank you !
>
> Maxime
>
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Maxime Jattiot
Sure, here is the schema :
{
    "name" : "test",
    "type" : "record",
    "fields" : [ {
        "name" : "field1",
        "type" : {
            "type" : "map",
            "values" : "string"
        }
    } ]
}

And here is the JSON message :
[{
    "field1": {
        "toto" : "v1",
        "titi" : "v2"
    }
}]

On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <[hidden email]> wrote:

> Hello,
>
> The screenshots did not come through, maybe just paste the schema and JSON
> as text in the email.
>
> Thanks.
>
> On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <[hidden email]>
> wrote:
>
> > Hello everyone,
> >
> > I am having hard time to validate a simple JSON message with an Avro
> > schema. My schema works with avro-tools but doesn't with ValidateRecord
> > processor. I wish you could help me.
> >
> >    - I am having the following debug log within Nifi :
> >
> > Record #1 is invalid due to:
> > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> > of type org.apache.nifi.serialization.record.MapRecord but was expected
> to
> > be of type MAP[STRING]
> >
> >
> >    - Here is my schema :
> >
> > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> >
> >    - Here is the JSON content sent :
> >
> > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> >
> > Does someone has any clue ? I have been struggling 2 days with this map
> > data type ... :(
> >
> > Thank you !
> >
> > Maxime
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Bryan Bende
I was able to make it work using a nested record instead of a map....

{
    "type": "record",
    "name": "example",
    "fields" : [
      {
        "name": "field1",
        "type": {
          "name" : "field1",
          "type" : "record",
          "fields" : [
            { "name": "toto", "type": "string" },
            { "name": "titi", "type": "string" }
          ]
        }
      }
    ]
}
On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot <[hidden email]> wrote:

>
> Sure, here is the schema :
> {
>     "name" : "test",
>     "type" : "record",
>     "fields" : [ {
>         "name" : "field1",
>         "type" : {
>             "type" : "map",
>             "values" : "string"
>         }
>     } ]
> }
>
> And here is the JSON message :
> [{
>     "field1": {
>         "toto" : "v1",
>         "titi" : "v2"
>     }
> }]
>
> On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <[hidden email]> wrote:
>
> > Hello,
> >
> > The screenshots did not come through, maybe just paste the schema and JSON
> > as text in the email.
> >
> > Thanks.
> >
> > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <[hidden email]>
> > wrote:
> >
> > > Hello everyone,
> > >
> > > I am having hard time to validate a simple JSON message with an Avro
> > > schema. My schema works with avro-tools but doesn't with ValidateRecord
> > > processor. I wish you could help me.
> > >
> > >    - I am having the following debug log within Nifi :
> > >
> > > Record #1 is invalid due to:
> > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1: Value is
> > > of type org.apache.nifi.serialization.record.MapRecord but was expected
> > to
> > > be of type MAP[STRING]
> > >
> > >
> > >    - Here is my schema :
> > >
> > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > >
> > >    - Here is the JSON content sent :
> > >
> > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > >
> > > Does someone has any clue ? I have been struggling 2 days with this map
> > > data type ... :(
> > >
> > > Thank you !
> > >
> > > Maxime
> > >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Maxime Jattiot
Yes I made it worked with record type as well but I can't use a record
because my field names are dynamic.

In many JSON we use collections of keys that will be named at runtime. I
thought the map data type of Avro was specially designed for it ....

Any other ideas ?


On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende <[hidden email]> wrote:

> I was able to make it work using a nested record instead of a map....
>
> {
>     "type": "record",
>     "name": "example",
>     "fields" : [
>       {
>         "name": "field1",
>         "type": {
>           "name" : "field1",
>           "type" : "record",
>           "fields" : [
>             { "name": "toto", "type": "string" },
>             { "name": "titi", "type": "string" }
>           ]
>         }
>       }
>     ]
> }
> On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot <[hidden email]>
> wrote:
> >
> > Sure, here is the schema :
> > {
> >     "name" : "test",
> >     "type" : "record",
> >     "fields" : [ {
> >         "name" : "field1",
> >         "type" : {
> >             "type" : "map",
> >             "values" : "string"
> >         }
> >     } ]
> > }
> >
> > And here is the JSON message :
> > [{
> >     "field1": {
> >         "toto" : "v1",
> >         "titi" : "v2"
> >     }
> > }]
> >
> > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <[hidden email]> wrote:
> >
> > > Hello,
> > >
> > > The screenshots did not come through, maybe just paste the schema and
> JSON
> > > as text in the email.
> > >
> > > Thanks.
> > >
> > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <[hidden email]>
> > > wrote:
> > >
> > > > Hello everyone,
> > > >
> > > > I am having hard time to validate a simple JSON message with an Avro
> > > > schema. My schema works with avro-tools but doesn't with
> ValidateRecord
> > > > processor. I wish you could help me.
> > > >
> > > >    - I am having the following debug log within Nifi :
> > > >
> > > > Record #1 is invalid due to:
> > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> Value is
> > > > of type org.apache.nifi.serialization.record.MapRecord but was
> expected
> > > to
> > > > be of type MAP[STRING]
> > > >
> > > >
> > > >    - Here is my schema :
> > > >
> > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > >
> > > >    - Here is the JSON content sent :
> > > >
> > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > >
> > > > Does someone has any clue ? I have been struggling 2 days with this
> map
> > > > data type ... :(
> > > >
> > > > Thank you !
> > > >
> > > > Maxime
> > > >
> > >
>
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Matt Burgess-2
Maxime,

I believe this is a bug, specifically at [1]. It should support
MapRecord entries as well. I have written NIFI-5678 [2] to address
this. The workaround is the one you and Bryan mentioned, using a
record instead of a map.

Regards,
Matt

[1] https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java#L199
[2] https://issues.apache.org/jira/browse/NIFI-5678

On Wed, Oct 10, 2018 at 10:19 AM Maxime Jattiot <[hidden email]> wrote:

>
> Yes I made it worked with record type as well but I can't use a record
> because my field names are dynamic.
>
> In many JSON we use collections of keys that will be named at runtime. I
> thought the map data type of Avro was specially designed for it ....
>
> Any other ideas ?
>
>
> On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende <[hidden email]> wrote:
>
> > I was able to make it work using a nested record instead of a map....
> >
> > {
> >     "type": "record",
> >     "name": "example",
> >     "fields" : [
> >       {
> >         "name": "field1",
> >         "type": {
> >           "name" : "field1",
> >           "type" : "record",
> >           "fields" : [
> >             { "name": "toto", "type": "string" },
> >             { "name": "titi", "type": "string" }
> >           ]
> >         }
> >       }
> >     ]
> > }
> > On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot <[hidden email]>
> > wrote:
> > >
> > > Sure, here is the schema :
> > > {
> > >     "name" : "test",
> > >     "type" : "record",
> > >     "fields" : [ {
> > >         "name" : "field1",
> > >         "type" : {
> > >             "type" : "map",
> > >             "values" : "string"
> > >         }
> > >     } ]
> > > }
> > >
> > > And here is the JSON message :
> > > [{
> > >     "field1": {
> > >         "toto" : "v1",
> > >         "titi" : "v2"
> > >     }
> > > }]
> > >
> > > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <[hidden email]> wrote:
> > >
> > > > Hello,
> > > >
> > > > The screenshots did not come through, maybe just paste the schema and
> > JSON
> > > > as text in the email.
> > > >
> > > > Thanks.
> > > >
> > > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <[hidden email]>
> > > > wrote:
> > > >
> > > > > Hello everyone,
> > > > >
> > > > > I am having hard time to validate a simple JSON message with an Avro
> > > > > schema. My schema works with avro-tools but doesn't with
> > ValidateRecord
> > > > > processor. I wish you could help me.
> > > > >
> > > > >    - I am having the following debug log within Nifi :
> > > > >
> > > > > Record #1 is invalid due to:
> > > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> > Value is
> > > > > of type org.apache.nifi.serialization.record.MapRecord but was
> > expected
> > > > to
> > > > > be of type MAP[STRING]
> > > > >
> > > > >
> > > > >    - Here is my schema :
> > > > >
> > > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > > >
> > > > >    - Here is the JSON content sent :
> > > > >
> > > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > > >
> > > > > Does someone has any clue ? I have been struggling 2 days with this
> > map
> > > > > data type ... :(
> > > > >
> > > > > Thank you !
> > > > >
> > > > > Maxime
> > > > >
> > > >
> >
Reply | Threaded
Open this post in threaded view
|

Re: Issue with ValidateRecord processor and Avro map data type

Maxime Jattiot
I believe it's a bug too. Thank you Matt for opening the issue !

On Wed, Oct 10, 2018 at 5:15 PM Matt Burgess <[hidden email]> wrote:

> Maxime,
>
> I believe this is a bug, specifically at [1]. It should support
> MapRecord entries as well. I have written NIFI-5678 [2] to address
> this. The workaround is the one you and Bryan mentioned, using a
> record instead of a map.
>
> Regards,
> Matt
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-extension-utils/nifi-record-utils/nifi-standard-record-utils/src/main/java/org/apache/nifi/schema/validation/StandardSchemaValidator.java#L199
> [2] https://issues.apache.org/jira/browse/NIFI-5678
>
> On Wed, Oct 10, 2018 at 10:19 AM Maxime Jattiot <[hidden email]>
> wrote:
> >
> > Yes I made it worked with record type as well but I can't use a record
> > because my field names are dynamic.
> >
> > In many JSON we use collections of keys that will be named at runtime. I
> > thought the map data type of Avro was specially designed for it ....
> >
> > Any other ideas ?
> >
> >
> > On Wed, Oct 10, 2018 at 4:09 PM Bryan Bende <[hidden email]> wrote:
> >
> > > I was able to make it work using a nested record instead of a map....
> > >
> > > {
> > >     "type": "record",
> > >     "name": "example",
> > >     "fields" : [
> > >       {
> > >         "name": "field1",
> > >         "type": {
> > >           "name" : "field1",
> > >           "type" : "record",
> > >           "fields" : [
> > >             { "name": "toto", "type": "string" },
> > >             { "name": "titi", "type": "string" }
> > >           ]
> > >         }
> > >       }
> > >     ]
> > > }
> > > On Wed, Oct 10, 2018 at 10:00 AM Maxime Jattiot <[hidden email]
> >
> > > wrote:
> > > >
> > > > Sure, here is the schema :
> > > > {
> > > >     "name" : "test",
> > > >     "type" : "record",
> > > >     "fields" : [ {
> > > >         "name" : "field1",
> > > >         "type" : {
> > > >             "type" : "map",
> > > >             "values" : "string"
> > > >         }
> > > >     } ]
> > > > }
> > > >
> > > > And here is the JSON message :
> > > > [{
> > > >     "field1": {
> > > >         "toto" : "v1",
> > > >         "titi" : "v2"
> > > >     }
> > > > }]
> > > >
> > > > On Wed, Oct 10, 2018 at 2:48 PM Bryan Bende <[hidden email]>
> wrote:
> > > >
> > > > > Hello,
> > > > >
> > > > > The screenshots did not come through, maybe just paste the schema
> and
> > > JSON
> > > > > as text in the email.
> > > > >
> > > > > Thanks.
> > > > >
> > > > > On Wed, Oct 10, 2018 at 6:57 AM Maxime Jattiot <
> [hidden email]>
> > > > > wrote:
> > > > >
> > > > > > Hello everyone,
> > > > > >
> > > > > > I am having hard time to validate a simple JSON message with an
> Avro
> > > > > > schema. My schema works with avro-tools but doesn't with
> > > ValidateRecord
> > > > > > processor. I wish you could help me.
> > > > > >
> > > > > >    - I am having the following debug log within Nifi :
> > > > > >
> > > > > > Record #1 is invalid due to:
> > > > > > MapRecord[{toto=v1, titi=v2}] is not a valid value for /field1:
> > > Value is
> > > > > > of type org.apache.nifi.serialization.record.MapRecord but was
> > > expected
> > > > > to
> > > > > > be of type MAP[STRING]
> > > > > >
> > > > > >
> > > > > >    - Here is my schema :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.21.41.png]
> > > > > >
> > > > > >    - Here is the JSON content sent :
> > > > > >
> > > > > > [image: Screen Shot 2018-10-10 at 11.22.10.png]
> > > > > >
> > > > > > Does someone has any clue ? I have been struggling 2 days with
> this
> > > map
> > > > > > data type ... :(
> > > > > >
> > > > > > Thank you !
> > > > > >
> > > > > > Maxime
> > > > > >
> > > > >
> > >
>