Url encoding with S3 processors

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Url encoding with S3 processors

Joe Gresock
Has anyone run into any URL encoding issues with the AWS nifi processors in
version 1.3.0?  My flow has GetSQS and then FetchS3Object, along with the
AWS feature of automated SQS events upon S3 file upload.  Some of my S3
objects apparently have characters in their object keys that need to be URL
encoded (e.g., %28 for open paren and + for space), and they come across
encoded in the SQS message.  However, FetchS3Object apparently doesn't like
this encoded URL, and it complains that the key does not exist.

I suppose I could add an UpdateAttributes processor to translate certain
urls in the keys, but this seems like a workaround.


--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*
Reply | Threaded
Open this post in threaded view
|

Re: Url encoding with S3 processors

James Wing
Joe,

The good news is that you can use expression language directly in
FetchS3Object's Object Key property to decode your keys.  For example,

${filename:urlDecode()}

However, I'm not sure that could be done without you specifying that the
key should be decoded. As I understand it, the following two keys:

file with space.txt
file+with+space.txt

are both valid, but distinctly separate keys. If we decoded by default, we
could create confusion with unexpected key changes.

Thanks,

James

On Thu, Sep 7, 2017 at 6:59 AM, Joe Gresock <[hidden email]> wrote:

> Has anyone run into any URL encoding issues with the AWS nifi processors in
> version 1.3.0?  My flow has GetSQS and then FetchS3Object, along with the
> AWS feature of automated SQS events upon S3 file upload.  Some of my S3
> objects apparently have characters in their object keys that need to be URL
> encoded (e.g., %28 for open paren and + for space), and they come across
> encoded in the SQS message.  However, FetchS3Object apparently doesn't like
> this encoded URL, and it complains that the key does not exist.
>
> I suppose I could add an UpdateAttributes processor to translate certain
> urls in the keys, but this seems like a workaround.
>
>
> --
> I know what it is to be in need, and I know what it is to have plenty.  I
> have learned the secret of being content in any and every situation,
> whether well fed or hungry, whether living in plenty or in want.  I can do
> all this through him who gives me strength.    *-Philippians 4:12-13*
>
Reply | Threaded
Open this post in threaded view
|

Re: Url encoding with S3 processors

Joe Gresock
Thanks James, I wasn't aware of the urlDecode() EL expression.

On Sun, Sep 10, 2017 at 5:31 PM, James Wing <[hidden email]> wrote:

> Joe,
>
> The good news is that you can use expression language directly in
> FetchS3Object's Object Key property to decode your keys.  For example,
>
> ${filename:urlDecode()}
>
> However, I'm not sure that could be done without you specifying that the
> key should be decoded. As I understand it, the following two keys:
>
> file with space.txt
> file+with+space.txt
>
> are both valid, but distinctly separate keys. If we decoded by default, we
> could create confusion with unexpected key changes.
>
> Thanks,
>
> James
>
> On Thu, Sep 7, 2017 at 6:59 AM, Joe Gresock <[hidden email]> wrote:
>
> > Has anyone run into any URL encoding issues with the AWS nifi processors
> in
> > version 1.3.0?  My flow has GetSQS and then FetchS3Object, along with the
> > AWS feature of automated SQS events upon S3 file upload.  Some of my S3
> > objects apparently have characters in their object keys that need to be
> URL
> > encoded (e.g., %28 for open paren and + for space), and they come across
> > encoded in the SQS message.  However, FetchS3Object apparently doesn't
> like
> > this encoded URL, and it complains that the key does not exist.
> >
> > I suppose I could add an UpdateAttributes processor to translate certain
> > urls in the keys, but this seems like a workaround.
> >
> >
> > --
> > I know what it is to be in need, and I know what it is to have plenty.  I
> > have learned the secret of being content in any and every situation,
> > whether well fed or hungry, whether living in plenty or in want.  I can
> do
> > all this through him who gives me strength.    *-Philippians 4:12-13*
> >
>



--
I know what it is to be in need, and I know what it is to have plenty.  I
have learned the secret of being content in any and every situation,
whether well fed or hungry, whether living in plenty or in want.  I can do
all this through him who gives me strength.    *-Philippians 4:12-13*