DEV Community

Colin Fay
Colin Fay

Posted on • Originally published at colinfay.me on

Automate the Creation of an API Wrapper package by Scraping its Online Documentation

#rstats #api #documentation

I’ve recently been building a package to interact with theRocket.chat API. One thing I don’t like about building API wrapper is that it’s very repetitive: you have to copy and paste things over and over again, for each endpoint of the API.Arguments, documentation…

But here’s a fun idea, how about scraping the help page of the API togenerate the whole thing?

General idea

So the global skeleton I need is the one for a .R file for a function, with {roxygen2} skeleton, and as much of the function as possible.Then, I need the arguments to the function, which are the available parameters of the API call.

If they are required by the API, I’ll leave them without a default value. If they are not, I’ll pass them NULL as a default parameter.Then, I’ll create a list with these params, remove the NULL with an internal fun, and turn them to JSON. Then I’ll be building the{httr} call.

One thing I know upfront is that I won’t be able to generate the whole thing, but let’s try to do as much as I can!

The code

I’ll try to build a function page for a call to"https://rocket.chat/docs/developer-guides/rest-api/users/create/"

library(rvest)


Loading required package: xml2


library(magrittr, warn.conflicts = FALSE)
library(purrr, warn.conflicts = FALSE)
page <- read_html("https://rocket.chat/docs/developer-guides/rest-api/users/create/")
Enter fullscreen mode Exit fullscreen mode

When on the page, we will find the args and their doc on the second table

fun_args <- page %>% 
  html_table() %>% 
  purrr::pluck(2)
fun_args


                Argument Example Required
1 email example@example.com Required
2 name Example User Required
3 password pass@w0rd Required
4 username example Required
5 active false Optional Default: true
6 roles ['bot'] Optional Default: ['user']
7 joinDefaultChannels false Optional Default: true
8 requirePasswordChange true Optional Default: false
9 sendWelcomeEmail true Optional Default: false
10 verified true Optional Default: false
11 customFields { twitter: '@example' } Optional Default: undefined
                                                              Description
1 The email address for the user.
2 The display name of the user.
3 The password for the user.
4 The username for the user.
5 Whether the user is active, which determines if they can login or not.
6 The roles the user has assigned to them on creation.
7 Whether the user should join the default channels when created.
8 Should the user be required to change their password when they login?
9 Should the user get a welcome email?
10 Should the user’s email address be verified when created?
11 Any custom fields the user should have on their account.
Enter fullscreen mode Exit fullscreen mode

Nice! We can start by creating some roxy tags out of this.

# Small util functions
tag_app <- function(var) {
  sprintf("#' %s", var)
}

ap_em_rox <- function(val){
  val %<>% c(
    "#' "
  )
  val
}
Enter fullscreen mode Exit fullscreen mode

The title of the function is at the second h2 of the page:

page %>% 
  html_nodes("h1") %>%
  pluck(2) %>%
  html_text()


[1] "Create"
Enter fullscreen mode Exit fullscreen mode

Let’s add it.

title <- page %>% 
  html_nodes("h1") %>%
  pluck(2) %>%
  html_text()
fun_code <- title %>%
  tag_app() %>% 
  ap_em_rox()
cat(fun_code, sep = "\n")


#' Create
#'
Enter fullscreen mode Exit fullscreen mode

And use the first paragraph as a function description.

fun_code %<>% c(
  page %>% 
    html_nodes("p") %>% 
    pluck(1)%>%
    html_text() %>%
    gsub("\n", ". ", .) %>%
    tag_app() %>% 
    ap_em_rox()
)

cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#'
Enter fullscreen mode Exit fullscreen mode

Now, something our API doc doesn’t have, the connection token we’ll be passing from the auth.

fun_code %<>% c("#' @param token The token to connect to the app.") 
styler::style_text(fun_code)


#' Create
#'
#' Create a new user. Requires create-user permission.
#'
#' @param token The token to connect to the app.
Enter fullscreen mode Exit fullscreen mode

Ok remember our table of parameters? Let’s us it now to create the roxygen tags.

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      sprintf(
        "#' @param %s %s %s", 
        thinkr::clean_vec(..1), ..4, ..3
      )
    }
  )
) %<>% ap_em_rox()
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#'
Enter fullscreen mode Exit fullscreen mode

Of course we’ll export the function:

fun_code %<>% c(
  tag_app("@export")
)
Enter fullscreen mode Exit fullscreen mode

Some deps

fun_code %<>% c(
  tag_app("@importFrom httr POST GET add_headers content stop_for_status"),
  tag_app("@importFrom jsonlite toJSON")

)
Enter fullscreen mode Exit fullscreen mode

Now, let’s create the function and its arguments.

fun_code %<>% c(
  sprintf(
    "%s <- function(tok,",
    thinkr::clean_vec(title)
  )
)

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      if (
        grepl("Optional", ..3)
      ){
        sprintf(
          " %s = NULL,", 
          thinkr::clean_vec(..1)
        )
      } else {
        sprintf(
          " %s,", 
          thinkr::clean_vec(..1)
        )
      }

    }
  )
)

fun_code[
  length(fun_code)
  ] %<>% gsub(",", "", .)
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#' 
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
  email,
  name,
  password,
  username,
  active = NULL,
  roles = NULL,
  joindefaultchannels = NULL,
  requirepasswordchange = NULL,
  sendwelcomeemail = NULL,
  verified = NULL,
  customfields = NULL
Enter fullscreen mode Exit fullscreen mode

Ok now a little bit of the internals:

fun_code %<>% c(
  "){", 
  " ", 
  " params <- list("
)

fun_code %<>% c(
  purrr::pmap_chr(
    fun_args,
    ~ {
      sprintf(
        " %s = %s,", 
        thinkr::clean_vec(..1), thinkr::clean_vec(..1)
      )
    }
  )
)

fun_code[
  length(fun_code)
  ] %<>% gsub(",", "", .)

fun_code %<>% c(
  ")", 
  "", 
  "params <- no_null(params)", 
  "", 
  "params <- toJSON(params, auto_unbox = TRUE)"
)
cat(fun_code, sep = "\n")


#' Create
#' 
#' Create a new user. Requires create-user permission.
#' 
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#' 
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
  email,
  name,
  password,
  username,
  active = NULL,
  roles = NULL,
  joindefaultchannels = NULL,
  requirepasswordchange = NULL,
  sendwelcomeemail = NULL,
  verified = NULL,
  customfields = NULL
){

  params <- list(
    email = email,
    name = name,
    password = password,
    username = username,
    active = active,
    roles = roles,
    joindefaultchannels = joindefaultchannels,
    requirepasswordchange = requirepasswordchange,
    sendwelcomeemail = sendwelcomeemail,
    verified = verified,
    customfields = customfields
)

params <- no_null(params)

params <- toJSON(params, auto_unbox = TRUE)
Enter fullscreen mode Exit fullscreen mode

A piece of the {httr} request. Note that our token contains a $url that contains the url of the chat, and the authToken and userId.

The endpoint is on the first table, as is the http method.

method <- page %>%
  html_table() %>% 
  pluck(1)
method["HTTP Method"] 


  HTTP Method
1 POST


method["URL"] 


                   URL
1 /api/v1/users.create


if ( method["HTTP Method"] == "POST"){
  fun_code %<>% c(
    sprintf(
      "res <- httr::%s(", 
      method["HTTP Method"]
    ), 
    "add_headers('Content-type' = 'application/json',",
    "'X-Auth-Token' = tok$data$authToken,",
    "'X-User-Id' = tok$data$userId",
    "),",
    sprintf("url = paste0(tok$url, '%s'),", method["URL"] ),
    "body = params"
  )
} else {
  fun_code %<>% c(
    sprintf(
      "res <- httr::%s(", 
      method["HTTP Method"]
    ), 
    "add_headers('Content-type' = 'application/json',",
    "'X-Auth-Token' = tok$data$authToken,",
    "'X-User-Id' = tok$data$userId",
    "),",
    sprintf("url = paste0(tok$url, '%s')", method["URL"] )
  )
}
Enter fullscreen mode Exit fullscreen mode

End this:

fun_code %<>% c(
  ")", 
  " ",
  "stop_for_status(res)", 
  "content(res)",
  "}"
)
Enter fullscreen mode Exit fullscreen mode

Let’s see what this looks like:

styler::style_text(fun_code)


#' Create
#'
#' Create a new user. Requires create-user permission.
#'
#' @param token The token to connect to the app.
#' @param email The email address for the user. Required
#' @param name The display name of the user. Required
#' @param password The password for the user. Required
#' @param username The username for the user. Required
#' @param active Whether the user is active, which determines if they can login or not. Optional Default: true
#' @param roles The roles the user has assigned to them on creation. Optional Default: ['user']
#' @param joindefaultchannels Whether the user should join the default channels when created. Optional Default: true
#' @param requirepasswordchange Should the user be required to change their password when they login? Optional Default: false
#' @param sendwelcomeemail Should the user get a welcome email? Optional Default: false
#' @param verified Should the user’s email address be verified when created? Optional Default: false
#' @param customfields Any custom fields the user should have on their account. Optional Default: undefined
#'
#' @export
#' @importFrom httr POST GET add_headers content stop_for_status
#' @importFrom jsonlite toJSON
create <- function(tok,
                   email,
                   name,
                   password,
                   username,
                   active = NULL,
                   roles = NULL,
                   joindefaultchannels = NULL,
                   requirepasswordchange = NULL,
                   sendwelcomeemail = NULL,
                   verified = NULL,
                   customfields = NULL) {
  params <- list(
    email = email,
    name = name,
    password = password,
    username = username,
    active = active,
    roles = roles,
    joindefaultchannels = joindefaultchannels,
    requirepasswordchange = requirepasswordchange,
    sendwelcomeemail = sendwelcomeemail,
    verified = verified,
    customfields = customfields
  )

  params <- no_null(params)

  params <- toJSON(params, auto_unbox = TRUE)
  res <- httr::POST(
    add_headers(
      "Content-type" = "application/json",
      "X-Auth-Token" = tok$data$authToken,
      "X-User-Id" = tok$data$userId
    ),
    url = paste0(tok$url, "/api/v1/users.create"),
    body = params
  )

  stop_for_status(res)
  content(res)
}
Enter fullscreen mode Exit fullscreen mode

And to automate even more:

usethis::use_r(title)
write(fun_code, sprintf("R/%s.R", title))
Enter fullscreen mode Exit fullscreen mode

Pretty cool! Seems like I have everything… or almost. Let’s automate it even more:

user_methods <- read_html("https://rocket.chat/docs/developer-guides/rest-api/users/")
user_methods %>% 
  html_nodes("a") %>%
  html_attr("href") %>% 
  grep("rest-api/users/.*/", ., value = TRUE)


 [1] "https://rocket.chat/docs/developer-guides/rest-api/users/presence/"                     
 [2] "https://rocket.chat/docs/developer-guides/rest-api/users/create/"                       
 [3] "https://rocket.chat/docs/developer-guides/rest-api/users/createtoken/"                  
 [4] "https://rocket.chat/docs/developer-guides/rest-api/users/delete/"                       
 [5] "https://rocket.chat/docs/developer-guides/rest-api/users/deleteownaccount/"             
 [6] "https://rocket.chat/docs/developer-guides/rest-api/users/forgotpassword/"               
 [7] "https://rocket.chat/docs/developer-guides/rest-api/users/generatepersonalaccesstoken/"  
 [8] "https://rocket.chat/docs/developer-guides/rest-api/users/getavatar/"                    
 [9] "https://rocket.chat/docs/developer-guides/rest-api/users/getpersonalaccesstokens/"      
[10] "https://rocket.chat/docs/developer-guides/rest-api/users/getpresence/"                  
[11] "https://rocket.chat/docs/developer-guides/rest-api/users/get-preferences/"              
[12] "https://rocket.chat/docs/developer-guides/rest-api/users/getusernamesuggestion/"        
[13] "https://rocket.chat/docs/developer-guides/rest-api/users/info/"                         
[14] "https://rocket.chat/docs/developer-guides/rest-api/users/list/"                         
[15] "https://rocket.chat/docs/developer-guides/rest-api/users/regeneratepersonalaccesstoken/"
[16] "https://rocket.chat/docs/developer-guides/rest-api/users/register/"                     
[17] "https://rocket.chat/docs/developer-guides/rest-api/users/removepersonalaccesstoken/"    
[18] "https://rocket.chat/docs/developer-guides/rest-api/users/requestdatadownload/"          
[19] "https://rocket.chat/docs/developer-guides/rest-api/users/resetavatar/"                  
[20] "https://rocket.chat/docs/developer-guides/rest-api/users/setavatar/"                    
[21] "https://rocket.chat/docs/developer-guides/rest-api/users/set-preferences/"              
[22] "https://rocket.chat/docs/developer-guides/rest-api/users/setactivestatus/"              
[23] "https://rocket.chat/docs/developer-guides/rest-api/users/update/"                       
[24] "https://rocket.chat/docs/developer-guides/rest-api/users/updateownbasicinfo/"
Enter fullscreen mode Exit fullscreen mode

🎉, now we have all the endpoints! Wrapping the previous code in a function, and we can automate even more.

Wanna see the created package? Here is the beginning:ColinFay/rrocketchat, thefunction that creates all other file can be found atdata-raw/DATASET.R

Top comments (0)

Read next

osmarmontesinos profile image

How to win a hackaton (spoiler: Power of Friendship)

Osmar Montesinos 🇦🇶 -

makowskid profile image

Introducing the Resume Parser/CV Parser for Laravel

Dawid Makowski -

apilover profile image

Top 10 API Load Testing Tools

Wanda -

juan_deto profile image

Configure Vitest, MSW and Playwright in a React project with Vite and TS - Part 2

Juan de Tomaso -