Suggestion required HTTP and S3

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Suggestion required HTTP and S3

Rajesh Biswas
Dear Experts,

I need your suggestion to design a requirement for my current project.

I would like to listen for HTTP post request thru NiFi which will have image
file attached.

I need to 1st extract the HTTP header information (to understand the type of
request), extract the image file and store the image file to S3 storage.

Would you please suggest if I require to write any custom processor or
existing Processors are sufficient.

Also I would like to pass the file URL in S3 to a Kafka queue, would you
please suggest what is the best way to get the S3 URL.

 

Thanks and Regards,

Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
www.bridgera.com

 



---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus
Reply | Threaded
Open this post in threaded view
|

Re: Suggestion required HTTP and S3

Bryan Bende
I think the existing processors such as HandleHttpRequest can be used.
The body of the POST will become the flow file content, and the
headers will become flow file attributes.

After HandleHttpRequest you can use RouteOnAttribute to make a
decision based on one of the headers (flow file attributes).

If it makes your criteria then you send it to a PutS3Object processor
to write the flow file contents to S3.

The success relationship of PutS3Object will have the flow file with
new attributes added for s3.bucket, s3.key, etc, which may be enough
for you to construct the URL.

You could then use ReplaceText to overwrite the content of the flow
with some dynamic expression like ${host}/${s3.bucket}/${s3.key} (or
whatever the URL is).

Then send to PublishKafka.

Thanks,

Bryan


On Tue, Sep 11, 2018 at 9:55 AM, Rajesh Biswas
<[hidden email]> wrote:

> Dear Experts,
>
> I need your suggestion to design a requirement for my current project.
>
> I would like to listen for HTTP post request thru NiFi which will have image
> file attached.
>
> I need to 1st extract the HTTP header information (to understand the type of
> request), extract the image file and store the image file to S3 storage.
>
> Would you please suggest if I require to write any custom processor or
> existing Processors are sufficient.
>
> Also I would like to pass the file URL in S3 to a Kafka queue, would you
> please suggest what is the best way to get the S3 URL.
>
>
>
> Thanks and Regards,
>
> Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
> www.bridgera.com
>
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
Reply | Threaded
Open this post in threaded view
|

RE: Suggestion required HTTP and S3

Rajesh Biswas
Hello Bryan,
We started implanting the requirement based on your suggestion, but we faced below issues:

1. Can we upload the actual image file instead of the flow-file that nifi creates?
(It is more like, can we convert the flow-file into its original format. i.e. jpg, jpeg,png etc)
2. Every time I try to make a upload request to S3 bucket, its making 2 requests simultaneously,  i.e 2 request for 1 image.

Thanks and Regards,
Rajesh Biswas | +91 9886433461 | www.bridgera.com


-----Original Message-----
From: Bryan Bende [mailto:[hidden email]]
Sent: Tuesday, September 11, 2018 7:34 PM
To: [hidden email]
Subject: Re: Suggestion required HTTP and S3

I think the existing processors such as HandleHttpRequest can be used.
The body of the POST will become the flow file content, and the
headers will become flow file attributes.

After HandleHttpRequest you can use RouteOnAttribute to make a
decision based on one of the headers (flow file attributes).

If it makes your criteria then you send it to a PutS3Object processor
to write the flow file contents to S3.

The success relationship of PutS3Object will have the flow file with
new attributes added for s3.bucket, s3.key, etc, which may be enough
for you to construct the URL.

You could then use ReplaceText to overwrite the content of the flow
with some dynamic expression like ${host}/${s3.bucket}/${s3.key} (or
whatever the URL is).

Then send to PublishKafka.

Thanks,

Bryan


On Tue, Sep 11, 2018 at 9:55 AM, Rajesh Biswas
<[hidden email]> wrote:

> Dear Experts,
>
> I need your suggestion to design a requirement for my current project.
>
> I would like to listen for HTTP post request thru NiFi which will have image
> file attached.
>
> I need to 1st extract the HTTP header information (to understand the type of
> request), extract the image file and store the image file to S3 storage.
>
> Would you please suggest if I require to write any custom processor or
> existing Processors are sufficient.
>
> Also I would like to pass the file URL in S3 to a Kafka queue, would you
> please suggest what is the best way to get the S3 URL.
>
>
>
> Thanks and Regards,
>
> Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
> www.bridgera.com
>
>
>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion required HTTP and S3

Bryan Bende
Hello,

I'm not sure what you mean by "upload the actual image file" ?

A flow file is made up of attributes and content, and the content is
just bytes which can be anything.

If you make a POST to an end-point provided by ListenHTTP or
HandleHttpRequest, and you send the image in the body of the POST,
then the image will be the content of the flow file.

I haven't used any S3 processors so I can't explain that behavior.

-Bryan

On Thu, Sep 20, 2018 at 8:00 AM Rajesh Biswas
<[hidden email]> wrote:

>
> Hello Bryan,
> We started implanting the requirement based on your suggestion, but we faced below issues:
>
> 1. Can we upload the actual image file instead of the flow-file that nifi creates?
> (It is more like, can we convert the flow-file into its original format. i.e. jpg, jpeg,png etc)
> 2. Every time I try to make a upload request to S3 bucket, its making 2 requests simultaneously,  i.e 2 request for 1 image.
>
> Thanks and Regards,
> Rajesh Biswas | +91 9886433461 | www.bridgera.com
>
>
> -----Original Message-----
> From: Bryan Bende [mailto:[hidden email]]
> Sent: Tuesday, September 11, 2018 7:34 PM
> To: [hidden email]
> Subject: Re: Suggestion required HTTP and S3
>
> I think the existing processors such as HandleHttpRequest can be used.
> The body of the POST will become the flow file content, and the
> headers will become flow file attributes.
>
> After HandleHttpRequest you can use RouteOnAttribute to make a
> decision based on one of the headers (flow file attributes).
>
> If it makes your criteria then you send it to a PutS3Object processor
> to write the flow file contents to S3.
>
> The success relationship of PutS3Object will have the flow file with
> new attributes added for s3.bucket, s3.key, etc, which may be enough
> for you to construct the URL.
>
> You could then use ReplaceText to overwrite the content of the flow
> with some dynamic expression like ${host}/${s3.bucket}/${s3.key} (or
> whatever the URL is).
>
> Then send to PublishKafka.
>
> Thanks,
>
> Bryan
>
>
> On Tue, Sep 11, 2018 at 9:55 AM, Rajesh Biswas
> <[hidden email]> wrote:
> > Dear Experts,
> >
> > I need your suggestion to design a requirement for my current project.
> >
> > I would like to listen for HTTP post request thru NiFi which will have image
> > file attached.
> >
> > I need to 1st extract the HTTP header information (to understand the type of
> > request), extract the image file and store the image file to S3 storage.
> >
> > Would you please suggest if I require to write any custom processor or
> > existing Processors are sufficient.
> >
> > Also I would like to pass the file URL in S3 to a Kafka queue, would you
> > please suggest what is the best way to get the S3 URL.
> >
> >
> >
> > Thanks and Regards,
> >
> > Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
> > www.bridgera.com
> >
> >
> >
> >
> >
> > ---
> > This email has been checked for viruses by Avast antivirus software.
> > https://www.avast.com/antivirus
>
Reply | Threaded
Open this post in threaded view
|

RE: Suggestion required HTTP and S3

Rajesh Biswas
Bryan,
I mean the file extension is lost.

Thanks and Regards,
Rajesh Biswas | +91 9886433461 | www.bridgera.com


-----Original Message-----
From: Bryan Bende [mailto:[hidden email]]
Sent: Thursday, September 20, 2018 7:08 PM
To: [hidden email]
Subject: Re: Suggestion required HTTP and S3

Hello,

I'm not sure what you mean by "upload the actual image file" ?

A flow file is made up of attributes and content, and the content is
just bytes which can be anything.

If you make a POST to an end-point provided by ListenHTTP or
HandleHttpRequest, and you send the image in the body of the POST,
then the image will be the content of the flow file.

I haven't used any S3 processors so I can't explain that behavior.

-Bryan

On Thu, Sep 20, 2018 at 8:00 AM Rajesh Biswas
<[hidden email]> wrote:

>
> Hello Bryan,
> We started implanting the requirement based on your suggestion, but we faced below issues:
>
> 1. Can we upload the actual image file instead of the flow-file that nifi creates?
> (It is more like, can we convert the flow-file into its original format. i.e. jpg, jpeg,png etc)
> 2. Every time I try to make a upload request to S3 bucket, its making 2 requests simultaneously,  i.e 2 request for 1 image.
>
> Thanks and Regards,
> Rajesh Biswas | +91 9886433461 | www.bridgera.com
>
>
> -----Original Message-----
> From: Bryan Bende [mailto:[hidden email]]
> Sent: Tuesday, September 11, 2018 7:34 PM
> To: [hidden email]
> Subject: Re: Suggestion required HTTP and S3
>
> I think the existing processors such as HandleHttpRequest can be used.
> The body of the POST will become the flow file content, and the
> headers will become flow file attributes.
>
> After HandleHttpRequest you can use RouteOnAttribute to make a
> decision based on one of the headers (flow file attributes).
>
> If it makes your criteria then you send it to a PutS3Object processor
> to write the flow file contents to S3.
>
> The success relationship of PutS3Object will have the flow file with
> new attributes added for s3.bucket, s3.key, etc, which may be enough
> for you to construct the URL.
>
> You could then use ReplaceText to overwrite the content of the flow
> with some dynamic expression like ${host}/${s3.bucket}/${s3.key} (or
> whatever the URL is).
>
> Then send to PublishKafka.
>
> Thanks,
>
> Bryan
>
>
> On Tue, Sep 11, 2018 at 9:55 AM, Rajesh Biswas
> <[hidden email]> wrote:
> > Dear Experts,
> >
> > I need your suggestion to design a requirement for my current project.
> >
> > I would like to listen for HTTP post request thru NiFi which will have image
> > file attached.
> >
> > I need to 1st extract the HTTP header information (to understand the type of
> > request), extract the image file and store the image file to S3 storage.
> >
> > Would you please suggest if I require to write any custom processor or
> > existing Processors are sufficient.
> >
> > Also I would like to pass the file URL in S3 to a Kafka queue, would you
> > please suggest what is the best way to get the S3 URL.
> >
> >
> >
> > Thanks and Regards,
> >
> > Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
> > www.bridgera.com
> >
> >
> >
> >
> >
> > ---
> > This email has been checked for viruses by Avast antivirus software.
> > https://www.avast.com/antivirus
>

Reply | Threaded
Open this post in threaded view
|

Re: Suggestion required HTTP and S3

Bryan Bende
It depends which processor you are using...

With ListenHttp you can send "filename" as a header.

With HandleHttpRequest all the headers get added as attributes
starting with "http.headers.<header-name>" so if you sent "filename"
as a header you'd have "http.headers.filename" and if you needed to
move it to the filename attribute you could just use an
UpdateAttribute with filename = ${http.headers.filename}
On Thu, Sep 20, 2018 at 10:12 AM Rajesh Biswas
<[hidden email]> wrote:

>
> Bryan,
> I mean the file extension is lost.
>
> Thanks and Regards,
> Rajesh Biswas | +91 9886433461 | www.bridgera.com
>
>
> -----Original Message-----
> From: Bryan Bende [mailto:[hidden email]]
> Sent: Thursday, September 20, 2018 7:08 PM
> To: [hidden email]
> Subject: Re: Suggestion required HTTP and S3
>
> Hello,
>
> I'm not sure what you mean by "upload the actual image file" ?
>
> A flow file is made up of attributes and content, and the content is
> just bytes which can be anything.
>
> If you make a POST to an end-point provided by ListenHTTP or
> HandleHttpRequest, and you send the image in the body of the POST,
> then the image will be the content of the flow file.
>
> I haven't used any S3 processors so I can't explain that behavior.
>
> -Bryan
>
> On Thu, Sep 20, 2018 at 8:00 AM Rajesh Biswas
> <[hidden email]> wrote:
> >
> > Hello Bryan,
> > We started implanting the requirement based on your suggestion, but we faced below issues:
> >
> > 1. Can we upload the actual image file instead of the flow-file that nifi creates?
> > (It is more like, can we convert the flow-file into its original format. i.e. jpg, jpeg,png etc)
> > 2. Every time I try to make a upload request to S3 bucket, its making 2 requests simultaneously,  i.e 2 request for 1 image.
> >
> > Thanks and Regards,
> > Rajesh Biswas | +91 9886433461 | www.bridgera.com
> >
> >
> > -----Original Message-----
> > From: Bryan Bende [mailto:[hidden email]]
> > Sent: Tuesday, September 11, 2018 7:34 PM
> > To: [hidden email]
> > Subject: Re: Suggestion required HTTP and S3
> >
> > I think the existing processors such as HandleHttpRequest can be used.
> > The body of the POST will become the flow file content, and the
> > headers will become flow file attributes.
> >
> > After HandleHttpRequest you can use RouteOnAttribute to make a
> > decision based on one of the headers (flow file attributes).
> >
> > If it makes your criteria then you send it to a PutS3Object processor
> > to write the flow file contents to S3.
> >
> > The success relationship of PutS3Object will have the flow file with
> > new attributes added for s3.bucket, s3.key, etc, which may be enough
> > for you to construct the URL.
> >
> > You could then use ReplaceText to overwrite the content of the flow
> > with some dynamic expression like ${host}/${s3.bucket}/${s3.key} (or
> > whatever the URL is).
> >
> > Then send to PublishKafka.
> >
> > Thanks,
> >
> > Bryan
> >
> >
> > On Tue, Sep 11, 2018 at 9:55 AM, Rajesh Biswas
> > <[hidden email]> wrote:
> > > Dear Experts,
> > >
> > > I need your suggestion to design a requirement for my current project.
> > >
> > > I would like to listen for HTTP post request thru NiFi which will have image
> > > file attached.
> > >
> > > I need to 1st extract the HTTP header information (to understand the type of
> > > request), extract the image file and store the image file to S3 storage.
> > >
> > > Would you please suggest if I require to write any custom processor or
> > > existing Processors are sufficient.
> > >
> > > Also I would like to pass the file URL in S3 to a Kafka queue, would you
> > > please suggest what is the best way to get the S3 URL.
> > >
> > >
> > >
> > > Thanks and Regards,
> > >
> > > Rajesh Biswas | +91 9886433461 |  <http://www.bridgera.com/>
> > > www.bridgera.com
> > >
> > >
> > >
> > >
> > >
> > > ---
> > > This email has been checked for viruses by Avast antivirus software.
> > > https://www.avast.com/antivirus
> >
>