Using record path API to remove field values

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Using record path API to remove field values

Mike Thomsen
Is there an example of how to use the record path API to remove an element
from a record? Ideally, if the field is nullable, I'd like to be able to
completely remove the value rather than send a null value (to Elastic in
this case).

Thanks,

Mike
Reply | Threaded
Open this post in threaded view
|

Re: Using record path API to remove field values

Pierre Villard
Hey Mike,

In the JSON Record Set Writer you have a "Suppress Null Values" parameter
to do just that.
Are you thinking about something else?

Pierre

Le mar. 30 oct. 2018 à 13:44, Mike Thomsen <[hidden email]> a
écrit :

> Is there an example of how to use the record path API to remove an element
> from a record? Ideally, if the field is nullable, I'd like to be able to
> completely remove the value rather than send a null value (to Elastic in
> this case).
>
> Thanks,
>
> Mike
>
Reply | Threaded
Open this post in threaded view
|

Re: Using record path API to remove field values

Mike Thomsen
Something more aggressive than that. I have a new ES bulk operation
processor that uses the record API to specify operations. I want to be able
to remove the record paths that relate to things like the index and type.

In StandardFieldValue it also looks like we ignore nulls altogether:

    @Override
    public void updateValue(final Object newValue) {
        final Optional<Record> parentRecord = getParentRecord();
        if (!parentRecord.isPresent()) {
            if (value instanceof Record) {
                ((Record) value).setValue(getField().getFieldName(),
newValue);
                return;
            } else if (value == null) {
                return; // value is null, nothing to update
            } else {
                throw new UnsupportedOperationException("Cannot update the
field value because the value is not associated with any record");
            }
        }

        parentRecord.get().setValue(getField().getFieldName(), newValue);
    }

Not sure what the rationale there is for not checking the value's field to
see if isNullable() is true or not and then acting from what. If no one
knows any gotchas, I can expand that out to allow null on nullable fields.
Then I could implement null suppression in a similar way as the json
writer, I suppose.

Thanks,

Mike

On Tue, Oct 30, 2018 at 8:51 AM Pierre Villard <[hidden email]>
wrote:

> Hey Mike,
>
> In the JSON Record Set Writer you have a "Suppress Null Values" parameter
> to do just that.
> Are you thinking about something else?
>
> Pierre
>
> Le mar. 30 oct. 2018 à 13:44, Mike Thomsen <[hidden email]> a
> écrit :
>
> > Is there an example of how to use the record path API to remove an
> element
> > from a record? Ideally, if the field is nullable, I'd like to be able to
> > completely remove the value rather than send a null value (to Elastic in
> > this case).
> >
> > Thanks,
> >
> > Mike
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Using record path API to remove field values

Bryan Bende
I'm not sure that removing a field from record path makes sense since
technically removing the field all together is a different schema.

You could easily make two versions of the schema, one with all fields,
and a second with a subset, then just use any record processor with a
reader using schema 1 and a writer using schema 2.
On Tue, Oct 30, 2018 at 9:28 AM Mike Thomsen <[hidden email]> wrote:

>
> Something more aggressive than that. I have a new ES bulk operation
> processor that uses the record API to specify operations. I want to be able
> to remove the record paths that relate to things like the index and type.
>
> In StandardFieldValue it also looks like we ignore nulls altogether:
>
>     @Override
>     public void updateValue(final Object newValue) {
>         final Optional<Record> parentRecord = getParentRecord();
>         if (!parentRecord.isPresent()) {
>             if (value instanceof Record) {
>                 ((Record) value).setValue(getField().getFieldName(),
> newValue);
>                 return;
>             } else if (value == null) {
>                 return; // value is null, nothing to update
>             } else {
>                 throw new UnsupportedOperationException("Cannot update the
> field value because the value is not associated with any record");
>             }
>         }
>
>         parentRecord.get().setValue(getField().getFieldName(), newValue);
>     }
>
> Not sure what the rationale there is for not checking the value's field to
> see if isNullable() is true or not and then acting from what. If no one
> knows any gotchas, I can expand that out to allow null on nullable fields.
> Then I could implement null suppression in a similar way as the json
> writer, I suppose.
>
> Thanks,
>
> Mike
>
> On Tue, Oct 30, 2018 at 8:51 AM Pierre Villard <[hidden email]>
> wrote:
>
> > Hey Mike,
> >
> > In the JSON Record Set Writer you have a "Suppress Null Values" parameter
> > to do just that.
> > Are you thinking about something else?
> >
> > Pierre
> >
> > Le mar. 30 oct. 2018 à 13:44, Mike Thomsen <[hidden email]> a
> > écrit :
> >
> > > Is there an example of how to use the record path API to remove an
> > element
> > > from a record? Ideally, if the field is nullable, I'd like to be able to
> > > completely remove the value rather than send a null value (to Elastic in
> > > this case).
> > >
> > > Thanks,
> > >
> > > Mike
> > >
> >