Ruby: How to post a file via HTTP as multipart/form-data?

I want to do an HTTP POST that looks like an HMTL form posted from a browser. Specifically, post some text fields and a file field.

Posting text fields is straightforward, there's an example right there in the net/http rdocs, but I can't figure out how to post a file along with it.

Net::HTTP doesn't look like the best idea. curb is looking good.

I like RestClient. It encapsulates net/http with cool features like multipart form data:

require 'rest_client'
RestClient.post('http://localhost:3000/foo', 
  :name_of_file_param => File.new('/path/to/file'))

It also supports streaming.

gem install rest-client will get you started.

I can't say enough good things about Nick Sieger's multipart-post library.

It adds support for multipart posting directly to Net::HTTP, removing your need to manually worry about boundaries or big libraries that may have different goals than your own.

Here is a little example on how to use it from the README:

require 'net/http/post/multipart'

url = URI.parse('http://www.example.com/upload')
File.open("./image.jpg") do |jpg|
  req = Net::HTTP::Post::Multipart.new url.path,
    "file" => UploadIO.new(jpg, "image/jpeg", "image.jpg")
  res = Net::HTTP.start(url.host, url.port) do |http|
    http.request(req)
  end
end

You can check out the library here: http://github.com/nicksieger/multipart-post

or install it with:

$ sudo gem install multipart-post

If you're connecting via SSL you need to start the connection like this:

n = Net::HTTP.new(url.host, url.port) 
n.use_ssl = true
# for debugging dev server
#n.verify_mode = OpenSSL::SSL::VERIFY_NONE
res = n.start do |http|

Another one using only standard libraries:

uri = URI('https://some.end.point/some/path')
request = Net::HTTP::Post.new(uri)
request['Authorization'] = 'If you need some headers'
form_data = [['photos', photo.tempfile]] # or File.open() in case of local file

request.set_form form_data, 'multipart/form-data'
response = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http| # pay attention to use_ssl if you need it
  http.request(request)
end

Tried a lot of approaches but only this was worked for me.

curb looks like a great solution, but in case it doesn't meet your needs, you can do it with Net::HTTP. A multipart form post is just a carefully-formatted string with some extra headers. It seems like every Ruby programmer who needs to do multipart posts ends up writing their own little library for it, which makes me wonder why this functionality isn't built-in. Maybe it is... Anyway, for your reading pleasure, I'll go ahead and give my solution here. This code is based off of examples I found on a couple of blogs, but I regret that I can't find the links anymore. So I guess I just have to take all the credit for myself...

The module I wrote for this contains one public class, for generating the form data and headers out of a hash of String and File objects. So for example, if you wanted to post a form with a string parameter named "title" and a file parameter named "document", you would do the following:

#prepare the query
data, headers = Multipart::Post.prepare_query("title" => my_string, "document" => my_file)

Then you just do a normal POST with Net::HTTP:

http = Net::HTTP.new(upload_uri.host, upload_uri.port)
res = http.start {|con| con.post(upload_uri.path, data, headers) }

Or however else you want to do the POST. The point is that Multipart returns the data and headers that you need to send. And that's it! Simple, right? Here's the code for the Multipart module (you need the mime-types gem):

# Takes a hash of string and file parameters and returns a string of text
# formatted to be sent as a multipart form post.
#
# Author:: Cody Brimhall <mailto:brimhall@somuchwit.com>
# Created:: 22 Feb 2008
# License:: Distributed under the terms of the WTFPL (http://www.wtfpl.net/txt/copying/)

require 'rubygems'
require 'mime/types'
require 'cgi'


module Multipart
  VERSION = "1.0.0"
  
  # Formats a given hash as a multipart form post
  # If a hash value responds to :string or :read messages, then it is
  # interpreted as a file and processed accordingly; otherwise, it is assumed
  # to be a string
  class Post
    # We have to pretend we're a web browser...
    USERAGENT = "Mozilla/5.0 (Macintosh; U; PPC Mac OS X; en-us) AppleWebKit/523.10.6 (KHTML, like Gecko) Version/3.0.4 Safari/523.10.6"
    BOUNDARY = "0123456789ABLEWASIEREISAWELBA9876543210"
    CONTENT_TYPE = "multipart/form-data; boundary=#{ BOUNDARY }"
    HEADER = { "Content-Type" => CONTENT_TYPE, "User-Agent" => USERAGENT }
    
    def self.prepare_query(params)
      fp = []
      
      params.each do |k, v|
        # Are we trying to make a file parameter?
        if v.respond_to?(:path) and v.respond_to?(:read) then
          fp.push(FileParam.new(k, v.path, v.read))
        # We must be trying to make a regular parameter
        else
          fp.push(StringParam.new(k, v))
        end
      end
      
      # Assemble the request body using the special multipart format
      query = fp.collect {|p| "--" + BOUNDARY + "\r\n" + p.to_multipart }.join("") + "--" + BOUNDARY + "--"
      return query, HEADER
    end
  end
  
  private
  
  # Formats a basic string key/value pair for inclusion with a multipart post
  class StringParam
    attr_accessor :k, :v
    
    def initialize(k, v)
      @k = k
      @v = v
    end
    
    def to_multipart
      return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"\r\n\r\n#{v}\r\n"
    end
  end
  
  # Formats the contents of a file or string for inclusion with a multipart
  # form post
  class FileParam
    attr_accessor :k, :filename, :content
    
    def initialize(k, filename, content)
      @k = k
      @filename = filename
      @content = content
    end
    
    def to_multipart
      # If we can tell the possible mime-type from the filename, use the
      # first in the list; otherwise, use "application/octet-stream"
      mime_type = MIME::Types.type_for(filename)[0] || MIME::Types["application/octet-stream"][0]
      return "Content-Disposition: form-data; name=\"#{CGI::escape(k)}\"; filename=\"#{ filename }\"\r\n" +
             "Content-Type: #{ mime_type.simplified }\r\n\r\n#{ content }\r\n"
    end
  end
end

Here is my solution after trying other ones available on this post, I'm using it to upload photo on TwitPic:

  def upload(photo)
    `curl -F media=@#{photo.path} -F username=#{@username} -F password=#{@password} -F message='#{photo.title}' http://twitpic.com/api/uploadAndPost`
  end

Fast forward to 2017, ruby stdlib net/http has this built-in since 1.9.3

> Net::HTTPRequest#set_form): Added to support both application/x-www-form-urlencoded and multipart/form-data.

https://ruby-doc.org/stdlib-2.3.1/libdoc/net/http/rdoc/Net/HTTPHeader.html#method-i-set_form

We can even use IO which does not support :size to stream the form data.

Hoping that this answer can really help someone :)

P.S. I only tested this in ruby 2.3.1

Ok, here's a simple example using curb.

require 'yaml'
require 'curb'

# prepare post data
post_data = fields_hash.map { |k, v| Curl::PostField.content(k, v.to_s) }
post_data << Curl::PostField.file('file', '/path/to/file'), 

# post
c = Curl::Easy.new('http://localhost:3000/foo')
c.multipart_form_post = true
c.http_post(post_data)

# print response
y [c.response_code, c.body_str]

restclient did not work for me until I overrode create_file_field in RestClient::Payload::Multipart.

It was creating a 'Content-Disposition: multipart/form-data' in each part where it should be ‘Content-Disposition: form-data’.

http://www.ietf.org/rfc/rfc2388.txt

My fork is here if you need it: [email protected]:kcrawford/rest-client.git

Well the solution with NetHttp has a drawback that is when posting big files it loads the whole file into memory first.

After playing a bit with it I came up with the following solution:

class Multipart
  
  def initialize( file_names )
    @file_names = file_names
  end
  
  def post( to_url )
    boundary = '----RubyMultipartClient' + rand(1000000).to_s + 'ZZZZZ'
    
    parts = []
    streams = []
    @file_names.each do |param_name, filepath|
      pos = filepath.rindex('/')
      filename = filepath[pos + 1, filepath.length - pos]
      parts << StringPart.new ( "--" + boundary + "\r\n" +
      "Content-Disposition: form-data; name=\"" + param_name.to_s + "\"; filename=\"" + filename + "\"\r\n" +
      "Content-Type: video/x-msvideo\r\n\r\n")
      stream = File.open(filepath, "rb")
      streams << stream
      parts << StreamPart.new (stream, File.size(filepath))
    end
    parts << StringPart.new ( "\r\n--" + boundary + "--\r\n" )
    
    post_stream = MultipartStream.new( parts )
    
    url = URI.parse( to_url )
    req = Net::HTTP::Post.new(url.path)
    req.content_length = post_stream.size
    req.content_type = 'multipart/form-data; boundary=' + boundary
    req.body_stream = post_stream
    res = Net::HTTP.new(url.host, url.port).start {|http| http.request(req) }

    streams.each do |stream|
      stream.close();
    end
    
    res
  end
  
end

class StreamPart
  def initialize( stream, size )
    @stream, @size = stream, size
  end
  
  def size
    @size
  end
  
  def read ( offset, how_much )
    @stream.read ( how_much )
  end
end

class StringPart
  def initialize ( str )
    @str = str
  end
  
  def size
    @str.length
  end
  
  def read ( offset, how_much )
    @str[offset, how_much]
  end
end

class MultipartStream
  def initialize( parts )
    @parts = parts
    @part_no = 0;
    @part_offset = 0;
  end
  
  def size
    total = 0
    @parts.each do |part|
      total += part.size
    end
    total
  end
  
  def read ( how_much )
    
    if @part_no >= @parts.size
      return nil;
    end
    
    how_much_current_part = @parts[@part_no].size - @part_offset
    
    how_much_current_part = if how_much_current_part > how_much
      how_much
    else
      how_much_current_part
    end
    
    how_much_next_part = how_much - how_much_current_part
    
    current_part = @parts[@part_no].read(@part_offset, how_much_current_part )

    if how_much_next_part > 0
      @part_no += 1
      @part_offset = 0
      next_part = read ( how_much_next_part  )
      current_part + if next_part
        next_part
      else
        ''
      end
    else
      @part_offset += how_much_current_part
      current_part
    end
  end
end

there's also nick sieger's multipart-post to add to the long list of possible solutions.

I had the same problem (need to post to jboss web server). Curb works fine for me, except that it caused ruby to crash (ruby 1.8.7 on ubuntu 8.10) when I use session variables in the code.

I dig into the rest-client docs, could not find indication of multipart support. I tried the rest-client examples above but jboss said the http post is not multipart.

The multipart-post gem works pretty well with Rails 4 Net::HTTP, no other special gem

def model_params
  require_params = params.require(:model).permit(:param_one, :param_two, :param_three, :avatar)
  require_params[:avatar] = model_params[:avatar].present? ? UploadIO.new(model_params[:avatar].tempfile, model_params[:avatar].content_type, model_params[:avatar].original_filename) : nil
  require_params
end

require 'net/http/post/multipart'

url = URI.parse('http://www.example.com/upload')
Net::HTTP.start(url.host, url.port) do |http|
  req = Net::HTTP::Post::Multipart.new(url, model_params)
  key = "authorization_key"
  req.add_field("Authorization", key) #add to Headers
  http.use_ssl = (url.scheme == "https")
  http.request(req)
end

https://github.com/Feuda/multipart-post/tree/patch-1

Using http.rb gem:

HTTP.post("https://here-you-go.com/upload",
          form: {
            file: HTTP::FormData::File.new(file_path)
          })

Details