Simple HTTP GET requests using Ruby

In order to be able to interact with web applications one needs to be able to write proper HTTP requests to access the API of the web application.

Ruby has a library called Net::HTTP that can provide the tools to write these call.

httpbin is an excellent web site to try to write HTTP requests against. You can even run it on your own computer if you'd like to avoid making external network calls.
It allows you to send various types of HTTP requests and will send you a response in a way that you can verify that your request was sent properly.

Part of the goal of this service is to allow you to generate error conditions and see how your web-client code handles a situation where the remote server returns an error code.

GET url

Let's start with a really simple example:

This is a plain GET requests

require 'net/http'

url = 'https://httpbin.org/get'
uri = URI(url)

response = Net::HTTP.get_response(uri)
# puts response.class  # Net::HTTPOK
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
end

The get_response method of the Net::HTTP class.

The response that we stored in a variable cleverly named response can be interrogated to see if the response was a success or failure.
It is an instance of a Net::HTTPResponse class.
In case it is a success we print the body of the response and see this:

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Accept-Encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3",
    "Host": "httpbin.org",
    "User-Agent": "Ruby",
    "X-Amzn-Trace-Id": "Root=1-6381d91a-6f521a3c0fa5af313924005b"
  },
  "origin": "46.120.8.206",
  "url": "https://httpbin.org/get"
}

For the other case see the next example.

GET URL that does not exist (404 error)

Here we deliberately created a URL that does not exist on the HTTPbin web server. In this case the response was not Net::HTTPOK, so Ruby executed the else part of our program. The code was 404 and the response message was NOT FOUND. I am sure you have already
encountered links that returned this error message. BTW you can try to visit the URLs using your regular browser as well to see the same response.

require 'net/http'

url = 'https://httpbin.org/qqrq'
uri = URI(url)

response = Net::HTTP.get_response(uri)
# puts response.class  # Net::HTTPNotFound
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
    puts response.msg
end

The output was:

404
NOT FOUND

GET URL that pretends not to exist (404)

Previously we got a 404 NOT FOUND HTTP response because we tried to access a page that really does not exist.
It was easy to generate a 404 error, but it would be a lot harder - effectively impossible to consistently get a web-site to return other HTTP error messages. eg. While we generally would want to avoid getting a 500 INTERNAL SERVER ERROR but for testing
purposes (for our client code) we might want to be able to consistently create it.

Luckily HTTPbin provide the service to fake any HTTP status code.

First let's see how does it generate 404 NOT FOUND message:

require 'net/http'

url = 'https://httpbin.org/status/404'
uri = URI(url)
# puts(uri)

response = Net::HTTP.get_response(uri)
# puts response.class  # Net::HTTPNotFound
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
    puts response.msg
end

Here, instead of the /get end-point we access the /status/CODE end-point replacing the CODE with the desired HTTP status code. HTTPbin will respond with that status code.

The output was:

404
NOT FOUND

GET URL that pretends to crash (500)

Generating 500 INTERNAL SERVER ERROR will be more fun, but we don't have to do anything special. Just send a GET request to the /status/500 end-point.

require 'net/http'

url = 'https://httpbin.org/status/500'
uri = URI(url)
# puts(uri)

response = Net::HTTP.get_response(uri)
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
    puts response.msg
end

The output was:

500
INTERNAL SERVER ERROR

Show request headers

Often you'll be interested to see if you managed to set the headers as expected by an API. For that HTTPbin provides the /headers end-point.

First let's see what happens if we send a plain request to this end-point?

require 'net/http'

url = 'https://httpbin.org/headers'
uri = URI(url)
# puts(uri)

response = Net::HTTP.get_response(uri)
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
end

This is the header as our Ruby code sent it. (Note the User Agent being Ruby)

{
  "headers": {
    "Accept": "*/*", 
    "Accept-Encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3", 
    "Host": "httpbin.org", 
    "User-Agent": "Ruby", 
    "X-Amzn-Trace-Id": "Root=1-6381e195-0fefc04b79ad1bc14b4688b0"
  }
}

Get request headers using Firefox

Just to compare, when I visited this url using Firefox, my default browser I get the
following results:

{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-US,en;q=0.5",
    "Host": "httpbin.org",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": "1",
    "User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0",
    "X-Amzn-Trace-Id": "Root=1-6394b5d7-395b7a2a377ea2df7dd4dbe8"
  }
}

Here the User-Agent is more detailed.

Get request headers using curl

Using curl

$ curl https://httpbin.org/headers

I got the following response:

{
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.81.0",
    "X-Amzn-Trace-Id": "Root=1-6394b68a-65bd2c145c2304274a314120"
  }
}

Set request headers

Finally we get to the point where we can set the headers in the request. Actually it is pretty easy to do so.
We just need to pass a new parameter called initheader with a hash of the header keys and values we would like to set.

This way we can add new fields to the header and we can replace existing ones. For example here we set the User-Agent
to Internet Explorer 6.0.

require 'net/http'

url = 'https://httpbin.org/headers'
uri = URI(url)
# puts(uri)

response = Net::HTTP.get_response(uri, initheader = {
    "Accept"     => "json",
    "Auth"       => "secret",
    "User-Agent" => "Internet Explorer 6.0",
  })
if response.is_a?(Net::HTTPSuccess)
    #puts "Headers:"
    #puts response.to_hash.inspect
    #puts '------'
    puts response.body
else
    puts response.code
end

In the response we can see that the fields were set as expected.

{
  "headers": {
    "Accept": "json", 
    "Accept-Encoding": "gzip;q=1.0,deflate;q=0.6,identity;q=0.3", 
    "Auth": "secret", 
    "Host": "httpbin.org", 
    "User-Agent": "Internet Explorer 6.0", 
    "X-Amzn-Trace-Id": "Root=1-6394bdbc-24b60d3376d56a493049d2e5"
  }
}

As you might know every time you access a web site it will log the request in a log file, usually including the User-Agent as well.
This way you could generate lots of request to a web site making their stats show that Internet Explorer 6.0 is back in popularity.

Don't do it!

End

This is it for now. The basics of using GET requests. I'll continue soon.