[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
GitHub user apiri opened a pull request:

    https://github.com/apache/incubator-nifi/pull/28

    NIFI-360 Create Processors to work against JSON data

    I have created two Processors, EvaluateJsonPath and SplitJson which are analogs of the functionality provided through EvaluateXPath and SpiltXML.
   
    Both are powered primarily around the usage of JsonPath by Jayway.
    Their capability descriptions are provided below:
   
    EvaluateJsonPath
    Evaluates one or more JsonPath expressions against the content of a FlowFile. The results of those expressions are assigned to FlowFile Attributes or are written to the content of the FlowFile itself, depending on configuration of the Processor. JsonPaths are entered by adding user-defined properties; the name of the property maps to the Attribute Name into which the result will be placed (if the Destination is flowfile-attribute; otherwise, the property name is ignored).
    The value of the property must be a valid JsonPath expression. If the JsonPath evaluates to a JSON array or JSON object and the Return Type is set to 'scalar' the FlowFile will be unmodified and will be routed to failure. A Return Type of JSON can return scalar values if the provided JsonPath evaluates to the specified value and will be routed as a match. If Destination is 'flowfile-content' and the JsonPath does not evaluate to a defined path, the FlowFile will be routed to 'unmatched' without having its contents modified. If Destination is flowfile-attribute and the expression matches nothing, attributes will be created with empty strings as the value, and the FlowFile will always be routed to 'matched.'
   
    SplitJson
    Splits a JSON File into multiple, separate FlowFiles for an array element specified by a JsonPath expression. Each generated FlowFile is comprised of an element of the specified array and transferred to relationship 'split, with the original file transferred to the 'original' relationship. If the specified JsonPath is not found or does not evaluate to an array element, the original file is routed to 'failure' and no files are generated.
   
    One item of note is the transitive dependency of ASM through Json-Smart through JsonPath.
    I have included, what I believe is needed to appropriately make use of this item in the LICENSE. Review of its correctness would is requested.
   
    Any feedback is appreciated. Thanks!

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apiri/incubator-nifi json-processors

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-nifi/pull/28.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #28
   
----
commit 446fc2ef8099faff786f11c7d640232191dac2b0
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-14T04:42:40Z

    Adding JSONPath as a managed dependency and including it in the standard processor pom

commit 5f03fb11d93791ea9cdde43ffb97bceaf22c0143
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-14T17:56:19Z

    Creating stub for an EvaluateJSONPath processor and providing configuration akin to EvaluateXPath

commit ec669e5b42e58335ab511dfa8dfd52bd0848f088
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-14T18:26:08Z

    Stubbing out validator, setting up relationships and properties.

commit e75213eead936baa7046d9c9ba5b86a4508f532a
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-14T18:35:53Z

    Adding EvaluateJsonPath to the Processor services file

commit da6b55f34ced99e3364aa382732a71d2d24b33f5
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-14T19:13:49Z

    Adding an implementation of validation for JsonPath, providing a sample JSON file, and creating an associated test class.

commit 40da65f1937089ea3935f21da384f4bbbc36d20d
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-15T20:25:22Z

    Adjusting filename of EvaluateJsonPath

commit c3c4d36944fa8a773d042edefa8cfa0f58edcd07
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T15:56:06Z

    Providing validation of the input FlowFile as JSON

commit b3328490c65d0b08bed4f711b6990d10262bbef2
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:03:24Z

    Completing initial functionality of EvaluateJsonPath and associated tests.

commit 974617d44ee41bd5cac9f238eca43cb9f81684a6
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:09:22Z

    Adding a test to verify the placement of a JSON path expression with a content destination.

commit 78ad0a3147cf1f3573f94f79685ebb868b5d190d
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:20:31Z

    Adding a test for multiple attribute paths

commit d4a94c37eea9c4a545e9f764c113a3d863257e57
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:24:57Z

    Adding a test for multiple attributes where neither evaluates to a found path.

commit 6897090771434fc1553b722b835c3b67e8dd14a9
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:26:35Z

    Adding a test for a destination of attributes and only one JsonPath expression evaluates to a found result.

commit 7e581307b7d65da79464914b09fe8a90dd5046e9
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:33:48Z

    Adding tests for indefinite results with and without the usage of operators.

commit 5b145e10e8508bec5293aab7318978fc3c9e2b57
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-16T23:34:23Z

    Adjusting error where the return type of string was omitted from the possible selections

commit bcebba6632165b6d69eb88ad12895e7678079267
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T03:44:06Z

    Refining logic of how errors are handled on a per destination basis.  Adding supporting tests to ensure contract is met.

commit 0a19ada0a3229b1c8a72fe638f286db33b6f7ba6
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T04:26:45Z

    Adjusting comments and providing a test to ensure a non-scalar value for a specified scalar return type is routed to failure.

commit 5a81f19b25bc073ec1089c115b6e6fcf01056f91
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T14:44:30Z

    Moving the JsonPath Validator to the JsonPathUtil class so that it can be reused by other processors.

commit 59ad1948519fa1f8acccfb6eafbda04684edbc96
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T15:26:18Z

    Refactoring common JSON/JsonPath functionality into JsonUtils

commit 2e05dcbbfdb37843456dff121d9878d8679d7067
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T20:15:53Z

    Providing a SplitJson processor which will break JSON Arrays into their individual elements.  Refactored supporting JsonUtils code and EvaluateJsonPath to reuse common functionality.

commit 7a74dd942008bf2178634746e70266f46657d4ef
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T20:50:42Z

    Adding a test case for SplitJson where the specified path cannot be found.  Adjusting documentation for additional detail.

commit 1a746212bf2e13680d71dd12dfb60df3a20784ab
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T21:04:21Z

    Removing unnessecary local variable

commit 627bd91fa719748b850b6c610a89b44be3e6f35e
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T21:15:33Z

    Adding an auto return type for EvaluateJsonPath to match the semantics of the EvaluateXPath processor.

commit a058fd0460dd5ca91560b1239dd733da37cfa782
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T21:58:04Z

    Adding license terms for asm dependency introduced via JsonPath -> Json-Smart

commit 408f636361cbad025752978e10775651b7daddf4
Author: Aldrin Piri <[hidden email]>
Date:   2015-02-17T22:01:06Z

    Preferring the constant empty string for clarity instead of quotes.

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
Github user joewitt commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74805392
 
    Aldrin - awesome!  I have created a branch and am reviewing the details now.  Regarding the update to nifi/LICENSE  - I think the correct place for the addition is nifi/nifi-assembly/LICENSE as this is a binary only dependency (as far as I can tell just yet).  The items in /nifi/LICENSE are only for source dependencies.  Will adjust that prior to merging.  Initial pass of all deps/licensing/notice looks good though.  Unit tests and sample data looks great too.    Really nice effort here - this will be very helpful.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user apiri commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74805553
 
    I think you are correct and that was my understanding of its usage when I was reviewing dependencies.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user joewitt commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74810190
 
    Review Findings:
   
    - Adjusted the LICENSE/NOTICE information for Standard Nar, NiFi source root, and NiFi assembly.
   
    - Can we cache the compiled JsonPaths [they are threadsafe, etc..]?
      - If so whenever onPropertyModified is triggers we know to invalidate the cache.  Can let it get refreshed during the next onTrigger.
     
    - EvaluateJsonPath: Instead of pulling some set batch size use the @SupportsBatching annotation and the framework can take care of this for you and will do so in a manner that favors user defined tolerances for latency vs throughput.  In addition this removes the need for the for loop, the flowfileloop goto, etc..  But it does make the previous suggestion more important (caching compiled JsonPaths)
   
    - Another contributor has provided an auto-documentation mechanism.  Not sure of its status for being pulled in just yet though. In the mean time please consider adding documentation pages in src/main/resources/docs....
   
    - EvaluateJsonPath: Line 207.  Is exception handling the only possible option for handling/detecting this?  Exceptions create a shocking amount of stress on GC and overall JVM performance at rate.  Having exception handling be key to the flow control will greatly limit the applicability of this processor for truly high volume processing.  If there is no alternative it is probably worth documenting this fact as a caveat for users to know in the docs for the processor.  If you'd like to see just how horrible exceptions are for throughput of the JVM do a simple test of Integer.parseInt to parse a String (when some of the inputs could be non-Integers) vs a regex to test first if it appears to be an Int then using Integer.parseInt.
   
    - EvaluateJsonPath: Consider telling the provenance event reporter if you modified the content of the flow file and providing some details about the change.
   
    - SplitJson: Exception based handling again to be avoided if possible.
   
    - Both Processors: It appears to be that the library will load the full input stream in memory then return results, splits, etc..  If so that provides important information about its applicability on large JSON documents.  If so please document this concern with the processor so users can consider this and its impact to the heap/memory space.
   
    - SplitJson: Consider telling the provenance event reporter that the splits are forks from the original.
   
    Summary: Really good stuff here.  This will be a very helpful processor.  Please consider making the suggested modifications.  I will push to origin this new branch NIFI-360.  Please modify off that and we can work from that PR.  Thanks!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user apiri commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74813229
 
    Great feedback, thanks.  I'll work on incorporating these changes over the next few days.
   
    Regarding performance issues, digging a bit deeper through the source, it looks like there may be a way around the exceptions.  Since the library in question is ASL/MIT, it's my understanding that it is okay to reproduce code segments as these fall in Category A, is this correct?
    Will definitely document the memory hit for loading items.  I was looking for a way to utilize output streams, but JsonPath does not currently provide this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

Re: [GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

Joe Witt
Aldrin:

I agree with your understanding for the ASL/MIT licensing.  If we do
that we will need to update our licensings to reflect the source
dependency but that is no problem at all.  I definitely think it is
worth it if we can bypass exception handling as a routine flow control
mechanism.

Thanks!
Joe

On Wed, Feb 18, 2015 at 12:08 AM, apiri <[hidden email]> wrote:

> Github user apiri commented on the pull request:
>
>     https://github.com/apache/incubator-nifi/pull/28#issuecomment-74813229
>
>     Great feedback, thanks.  I'll work on incorporating these changes over the next few days.
>
>     Regarding performance issues, digging a bit deeper through the source, it looks like there may be a way around the exceptions.  Since the library in question is ASL/MIT, it's my understanding that it is okay to reproduce code segments as these fall in Category A, is this correct?
>     Will definitely document the memory hit for loading items.  I was looking for a way to utilize output streams, but JsonPath does not currently provide this.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at [hidden email] or file a JIRA ticket
> with INFRA.
> ---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user danbress commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74858249
 
    Regarding generated processor documentation:I think everything is ready to
    go.  When I spoke to Mark about this last week he said he was keeping this
    on the side until the 0.0.2 release was cut.  So once 0.0.2 is cut, the
    auto doc feature will get merged into 0.1.0
   
    Dan
   
    On Wed, Feb 18, 2015 at 12:12 AM, asfbot <[hidden email]> wrote:
   
    > Joe Witt on [hidden email] replies:
    > Aldrin:
    >
    > I agree with your understanding for the ASL/MIT licensing. If we do
    > that we will need to update our licensings to reflect the source
    > dependency but that is no problem at all. I definitely think it is
    > worth it if we can bypass exception handling as a routine flow control
    > mechanism.
    >
    > Thanks!
    > Joe
    > 9
    > r the next few days.
    > , it looks like there may be a way around the exceptions. Since the librar=
    > y in question is ASL/MIT, it's my understanding that it is okay to
    > reproduc=
    > e code segments as these fall in Category A, is this correct?
    > king for a way to utilize output streams, but JsonPath does not currently
    > p=
    > rovide this.
    > ur
    > e
    > se
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/incubator-nifi/pull/28#issuecomment-74813584>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

Re: [GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

Mark Payne
In reply to this post by JPercivall
Right. In 0.0.2 we moved around the annotations and deprecated the old ones. We did this because they were all in a package called “org.apache.nifi.processor.annotations” or something like that, and the annotations really should apply to Reporting Tasks and Controller Services, as well, as they are becoming more first-class citizens of the framework.


The code that Dan wrote is intended to run against the new annotations. As such, I decided it probably made sense to go ahead and wait until the 0.1.0 release to pull that in. Code review looked good though, and I think it’s ready to go - just holding off because of the annotation changes.


THanks

-Mark








From: danbress
Sent: ‎Wednesday‎, ‎February‎ ‎18‎, ‎2015 ‎7‎:‎47‎ ‎AM
To: [hidden email]





Github user danbress commented on the pull request:

    https://github.com/apache/incubator-nifi/pull/28#issuecomment-74858249
 
    Regarding generated processor documentation:I think everything is ready to
    go.  When I spoke to Mark about this last week he said he was keeping this
    on the side until the 0.0.2 release was cut.  So once 0.0.2 is cut, the
    auto doc feature will get merged into 0.1.0
   
    Dan
   
    On Wed, Feb 18, 2015 at 12:12 AM, asfbot <[hidden email]> wrote:
   
    > Joe Witt on [hidden email] replies:
    > Aldrin:
    >
    > I agree with your understanding for the ASL/MIT licensing. If we do
    > that we will need to update our licensings to reflect the source
    > dependency but that is no problem at all. I definitely think it is
    > worth it if we can bypass exception handling as a routine flow control
    > mechanism.
    >
    > Thanks!
    > Joe
    > 9
    > r the next few days.
    > , it looks like there may be a way around the exceptions. Since the librar=
    > y in question is ASL/MIT, it's my understanding that it is okay to
    > reproduc=
    > e code segments as these fall in Category A, is this correct?
    > king for a way to utilize output streams, but JsonPath does not currently
    > p=
    > rovide this.
    > ur
    > e
    > se
    >
    > —
    > Reply to this email directly or view it on GitHub
    > <https://github.com/apache/incubator-nifi/pull/28#issuecomment-74813584>.
    >



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user markap14 commented on a diff in the pull request:

    https://github.com/apache/incubator-nifi/pull/28#discussion_r25540061
 
    --- Diff: nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JsonUtils.java ---
    @@ -0,0 +1,127 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nifi.processors.standard.util;
    +
    +import com.jayway.jsonpath.Configuration;
    +import com.jayway.jsonpath.DocumentContext;
    +import com.jayway.jsonpath.InvalidPathException;
    +import com.jayway.jsonpath.JsonPath;
    +import com.jayway.jsonpath.spi.json.JsonProvider;
    +import net.minidev.json.JSONValue;
    +import org.apache.nifi.components.ValidationContext;
    +import org.apache.nifi.components.ValidationResult;
    +import org.apache.nifi.components.Validator;
    +import org.apache.nifi.flowfile.FlowFile;
    +import org.apache.nifi.processor.ProcessSession;
    +import org.apache.nifi.processor.io.InputStreamCallback;
    +import org.apache.nifi.stream.io.BufferedInputStream;
    +import org.apache.nifi.util.BooleanHolder;
    +import org.apache.nifi.util.ObjectHolder;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.io.InputStreamReader;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Provides utilities for interacting with JSON elements and JsonPath expressions and results
    + *
    + * @see <a href="http://json.org">http://json.org</a>
    + * @see <a href="https://github.com/jayway/JsonPath">https://github.com/jayway/JsonPath</a>
    + */
    +public class JsonUtils {
    +
    +    static final JsonProvider JSON_PROVIDER = Configuration.defaultConfiguration().jsonProvider();
    +
    +    public static final Validator JSON_PATH_VALIDATOR = new Validator() {
    +        @Override
    +        public ValidationResult validate(final String subject, final String input, final ValidationContext context) {
    +            String error = null;
    +            try {
    +                JsonPath compile = JsonPath.compile(input);
    +            } catch (InvalidPathException ipe) {
    +                error = ipe.toString();
    +            }
    +            return new ValidationResult.Builder().valid(error == null).explanation(error).build();
    +        }
    +    };
    +
    +    public static DocumentContext validateAndEstablishJsonContext(ProcessSession processSession, FlowFile flowFile) {
    +
    +        final BooleanHolder validJsonHolder = new BooleanHolder(false);
    +        processSession.read(flowFile, new InputStreamCallback() {
    +            @Override
    +            public void process(InputStream in) throws IOException {
    +                validJsonHolder.set(JsonUtils.isValidJson(in));
    +            }
    +        });
    +
    +        // Parse the document once into an associated context to support multiple path evaluations if specified
    +        final ObjectHolder<DocumentContext> contextHolder = new ObjectHolder<>(null);
    +
    +        if (validJsonHolder.get()) {
    +            processSession.read(flowFile, new InputStreamCallback() {
    --- End diff --
   
    Do we need to read the content to validate it above and then read it again here? Can we just assume that it's valid unless an Exception is thrown?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user apiri commented on a diff in the pull request:

    https://github.com/apache/incubator-nifi/pull/28#discussion_r25563122
 
    --- Diff: nifi/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/util/JsonUtils.java ---
    @@ -0,0 +1,127 @@
    +/*
    + * Licensed to the Apache Software Foundation (ASF) under one or more
    + * contributor license agreements.  See the NOTICE file distributed with
    + * this work for additional information regarding copyright ownership.
    + * The ASF licenses this file to You under the Apache License, Version 2.0
    + * (the "License"); you may not use this file except in compliance with
    + * the License.  You may obtain a copy of the License at
    + *
    + *     http://www.apache.org/licenses/LICENSE-2.0
    + *
    + * Unless required by applicable law or agreed to in writing, software
    + * distributed under the License is distributed on an "AS IS" BASIS,
    + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    + * See the License for the specific language governing permissions and
    + * limitations under the License.
    + */
    +package org.apache.nifi.processors.standard.util;
    +
    +import com.jayway.jsonpath.Configuration;
    +import com.jayway.jsonpath.DocumentContext;
    +import com.jayway.jsonpath.InvalidPathException;
    +import com.jayway.jsonpath.JsonPath;
    +import com.jayway.jsonpath.spi.json.JsonProvider;
    +import net.minidev.json.JSONValue;
    +import org.apache.nifi.components.ValidationContext;
    +import org.apache.nifi.components.ValidationResult;
    +import org.apache.nifi.components.Validator;
    +import org.apache.nifi.flowfile.FlowFile;
    +import org.apache.nifi.processor.ProcessSession;
    +import org.apache.nifi.processor.io.InputStreamCallback;
    +import org.apache.nifi.stream.io.BufferedInputStream;
    +import org.apache.nifi.util.BooleanHolder;
    +import org.apache.nifi.util.ObjectHolder;
    +
    +import java.io.IOException;
    +import java.io.InputStream;
    +import java.io.InputStreamReader;
    +import java.util.List;
    +import java.util.Map;
    +
    +/**
    + * Provides utilities for interacting with JSON elements and JsonPath expressions and results
    + *
    + * @see <a href="http://json.org">http://json.org</a>
    + * @see <a href="https://github.com/jayway/JsonPath">https://github.com/jayway/JsonPath</a>
    + */
    +public class JsonUtils {
    +
    +    static final JsonProvider JSON_PROVIDER = Configuration.defaultConfiguration().jsonProvider();
    +
    +    public static final Validator JSON_PATH_VALIDATOR = new Validator() {
    +        @Override
    +        public ValidationResult validate(final String subject, final String input, final ValidationContext context) {
    +            String error = null;
    +            try {
    +                JsonPath compile = JsonPath.compile(input);
    +            } catch (InvalidPathException ipe) {
    +                error = ipe.toString();
    +            }
    +            return new ValidationResult.Builder().valid(error == null).explanation(error).build();
    +        }
    +    };
    +
    +    public static DocumentContext validateAndEstablishJsonContext(ProcessSession processSession, FlowFile flowFile) {
    +
    +        final BooleanHolder validJsonHolder = new BooleanHolder(false);
    +        processSession.read(flowFile, new InputStreamCallback() {
    +            @Override
    +            public void process(InputStream in) throws IOException {
    +                validJsonHolder.set(JsonUtils.isValidJson(in));
    +            }
    +        });
    +
    +        // Parse the document once into an associated context to support multiple path evaluations if specified
    +        final ObjectHolder<DocumentContext> contextHolder = new ObjectHolder<>(null);
    +
    +        if (validJsonHolder.get()) {
    +            processSession.read(flowFile, new InputStreamCallback() {
    --- End diff --
   
    Yeah, looking at what that method is actually doing is just catching an exception when parsing it instead of just inspecting.  Good catch.  Have to see what the API provides, but seems like it is not buying much.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---
Reply | Threaded
Open this post in threaded view
|

[GitHub] incubator-nifi pull request: NIFI-360 Create Processors to work ag...

JPercivall
In reply to this post by JPercivall
Github user asfgit closed the pull request at:

    https://github.com/apache/incubator-nifi/pull/28


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [hidden email] or file a JIRA ticket
with INFRA.
---