Posting Multipart data

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Posting Multipart data

Chris Conklin-2
Hello,

Attempting to post to rest some multipart form data but running into
memory problems.  Copying the flowfile inputstream to memory is
obviously going to be bad but can't seem to come up with another way to
get the stream into the MultipartForm.  Any ideas?

I am using 0.7.3 version of nifi for this custom processor.

Here is a code snippet (the toBufferedInputStream works slightly better
then the new ByteArrayInputStream but eventually hits a memory issue
depending on size of heap):

         for (final FlowFile flowFile : flowFileList) {
             session.read(flowFile, new InputStreamCallback() {
             @Override
             public void process(final InputStream rawIn) throws
IOException {
                 MultipartEntityBuilder builder =
MultipartEntityBuilder.create();
                 builder.addTextBody("userName", "Chris");
                 builder.addTextBody("password", "password");

                 //InputStream in =IOUtils.toBufferedInputStream(rawIn);
                 //InputStream in = new
ByteArrayInputStream(IOUtils.toByteArray(rawIn));

                 builder.addBinaryBody("file",
IOUtils.toBufferedInputStream(rawIn), ContentType.DEFAULT_BINARY ,
"filename");
                 HttpEntity entity2 = builder.build();
                 post.setEntity(entity2);
                 post.setConfig(requestConfig);
             }
             });
         }


Thanks for any help anyone can offer.

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Posting Multipart data

Koji Kawamura-2
Hello Chris,

Did you find a solution for this? I looked at the underlying Apache
HttpComponents API, and found InputStreamBody class.
InputStreamBody doesn't copy whole byte array, but it streams data
from specified input stream to the output stream per 4kb.

You may find this example useful, the example uses MultipartEntity and
InputStreamBody together:
https://www.programcreek.com/java-api-examples/index.php?api=org.apache.http.entity.mime.content.InputStreamBody

Thanks,
Koji

On Mon, Sep 18, 2017 at 11:16 AM, Chris Conklin <[hidden email]> wrote:

> Hello,
>
> Attempting to post to rest some multipart form data but running into memory
> problems.  Copying the flowfile inputstream to memory is obviously going to
> be bad but can't seem to come up with another way to get the stream into the
> MultipartForm.  Any ideas?
>
> I am using 0.7.3 version of nifi for this custom processor.
>
> Here is a code snippet (the toBufferedInputStream works slightly better then
> the new ByteArrayInputStream but eventually hits a memory issue depending on
> size of heap):
>
>         for (final FlowFile flowFile : flowFileList) {
>             session.read(flowFile, new InputStreamCallback() {
>             @Override
>             public void process(final InputStream rawIn) throws IOException
> {
>                 MultipartEntityBuilder builder =
> MultipartEntityBuilder.create();
>                 builder.addTextBody("userName", "Chris");
>                 builder.addTextBody("password", "password");
>
>                 //InputStream in =IOUtils.toBufferedInputStream(rawIn);
>                 //InputStream in = new
> ByteArrayInputStream(IOUtils.toByteArray(rawIn));
>
>                 builder.addBinaryBody("file",
> IOUtils.toBufferedInputStream(rawIn), ContentType.DEFAULT_BINARY ,
> "filename");
>                 HttpEntity entity2 = builder.build();
>                 post.setEntity(entity2);
>                 post.setConfig(requestConfig);
>             }
>             });
>         }
>
>
> Thanks for any help anyone can offer.
>
> Chris
>