Forem has set a milestone to update our (v1) API documentation and we need YOUR help!
There are several endpoints that we would like to document in order to complete our v0 -> v1 upgrade. v0 will eventually be deprecated and removed (there aren't any breaking changes so existing endpoints will continue to work the same as before). If you’re looking to contribute to open source, these are awesome first issues to work on and this post will help guide you through them.
In this post, I’ll outline the details about our v1 API, we’ll discuss its documentation and then I’ll walk you through an example of an API endpoint that I had recently documented.
About the v1 API
The API is a REST (REpresentational State Transfer) API, which means that it follows a set of guiding principles that you can read more about here.
There are currently many resources that can be accessed via the API, however, it does not contain all of the resources that we have available on DEV. We are continuously adding more endpoints to the API.
It is important to note that all operations are scoped to a user at the moment, and one cannot interact with the API as an organization.
Headers
The API consists of both authenticated and non-authenticated endpoints.
Most read endpoints do not require authentication and can be accessed without API keys. However, we require authentication for most endpoints that can create, update or delete resources, and those that contain more private information.
CORS (Cross Origin Resource Sharing) is disabled on authenticated endpoints, however endpoints that do not require authentication have an open CORS Policy. With an open CORS policy you are able to access some endpoints from within a browser script without needing to connect via a server or backend.
Authentication is granted via a valid API Key. The API key can be generated by logging into your account at dev.to and clicking on Generate API Key on the Settings (Extensions) page.
Once you’ve generated your API key, you are ready to start interacting with the API. The API Key will be set as a header, namely api-key, on a request.
We require another header for accessing the v1 API - the Accept header. The Accept header needs to be set to application/vnd.forem.api-v1+json, where v1 is the version of the API. If you do not pass along an Accept header you will automatically be routed to v0 of the API. API (v0) will be deprecated soon and we encourage you to rather use v1.
Accessing the API
As mentioned above, there are some API endpoints that are authenticated and others that do not require authentication.
When interacting with an endpoint that does not require authentication, you can pass through a single header (the Accept header) that will set the version of the API that you are interacting with.
curl -X GET https://dev.to/api/articles\?page\=1 --header "accept: application/vnd.forem.api-v1+json"
An endpoint that requires authentication would need both the Accept and the api-key header to be set when making the request:
curl -X PUT http: //localhost :3000/api/users/1/unpublish
--header "api-key: <your-api-key>"
--header "accept: application/vnd.forem.api-v1+json" -V
You need to have the correct roles and permissions set on your user to be able to query certain data from the API. For example, only admins can read, create, update and delete Display Ad resources.
About our documentation
Our v1 API endpoints are documented according to the OpenAPI Specification.
The OpenAPI Specification
The OpenAPI Specification (previously known as a Swagger Specification) is a standard for defining RESTful interfaces. As per the definition on their website, it is a document (or set of documents) that defines or describes an API.
An OpenAPI definition uses and conforms to the OpenAPI Specification. The OpenAPI definition can be created within your codebase.
You can view the Open API Specification here. It describes API Versions, Formats, Document Structure, Data Types, Schemas and much more.
When an API adheres to the Open API specification, it allows opportunities to use document generation tools to display the API, code generation tools to generate servers and clients in various programming languages, access to testing tools etc. Some of these tools include Swagger UI, Redoc, DapperDox and RapidDoc.
Forem, which is a Ruby on Rails app, integrates the Open API Specification via a gem - the rswag gem. The rswag Ruby gem allows us to create a DSL for describing and testing our API operations. It also extends rspec-rails "request specs", hence, allowing our documentation to be a part of our test suite which allows us to make requests with test parameters and seed data that invoke different response codes. As a result, we are able to test what the requests and responses look like, however we do not test the business logic that drives the endpoint - that is tested elsewhere in the code.
Once we write the test, we are able to generate a JSON file that conforms to the Open API Specification thus allowing us to eventually use the document generation tools to format and beautify our documentation.
The OpenAPI definition in the form of api_v1.json is generated and used as input to Docusaurus to create our documentation. You can view our v1 documentation here.
Now that we’ve discussed how Open API and rswag fit together to create the API documentation let’s work through an example of adding documentation to an endpoint together.
Documenting a v1 endpoint
We’ll be working on this github issue together. The issue outlines a task to use rswag to document the /api/followers/users endpoint in the v1 API.
For reference, our v1 documentation lives here.
There is also a pull request for the code written in the example below which you can reference.
Skeleton
Let’s start by creating a file for the endpoint that we want to test and document.We can go ahead and create spec/requests/api/v1/docs/followers_spec.rb
There are some building blocks for each test - a skeleton that defines some standards, implementation details like generating the header values, seed data etc.
Below is a code snippet of the skeleton for this test:
require "rails_helper"
require "swagger_helper"
# rubocop:disable RSpec/EmptyExampleGroup
# rubocop:disable RSpec/VariableName
RSpec.describe "Api::V1::Docs::Followers" do
let(:Accept) { "application/vnd.forem.api-v1+json" }
let(:api_secret) { create(:api_secret) }
let(:user) { api_secret.user }
let(:follower1) { create(:user) }
let(:follower2) { create(:user) }
before do
follower1.follow(user)
follower2.follow(user)
user.reload
end
describe "GET /followers/users" do
path "/api/followers/users" do
get "Followers" do
end
end
end
end
# rubocop:enable RSpec/EmptyExampleGroup
# rubocop:enable RSpec/VariableName
We start off by importing the necessary libraries - in this case the rails_helper and the swagger_helper that will allow us to use the DSL to build out our definitions.
If you’re use RSpec before then the describe block will be familiar to you, it will create an example group.
let(:Accept) { "application/vnd.forem.api-v1+json" }
let(:api_secret) { create(:api_secret) }
Above, we define (but not yet set) the header values. The accept header will allow us to access the v1 API and since the /api/followers/users endpoint requires authentication we generate an API secret that we will use later on.
let(:user) { api_secret.user }
let(:follower1) { create(:user) }
let(:follower2) { create(:user) }
before do
follower1.follow(user)
follower2.follow(user)
user.reload
end
Above, we use RSpec to setup our data so that we can have example responses in our API documentation. We create a user and two follower users. With these models in the DB rswag will run the tests and display the example tags created by FactoryBot in the json file. In the before block, we then setup the two follows to follow the user.
describe "GET /followers/users" do
path "/api/followers/users" do
get "Followers" do
end
end
end
In a nested describe block we start by specifying the path for the endpoint that we’re testing which is /api/followers/users. You can read more about path operations in the rswag documentation.
In some circumstances, you may have an identifier or parameter in the path. These are surrounded by curly braces. For example: /api/articles/{id}.
Operation Ids
get "Followers" do
end
The above is considered our operation block. In this instance we define that we are implementing a GET.
The Operation Object has several fields that can be set to help define this endpoint. You can read more about operation objects here.
These are some of the fields that we want to define for api/followers/users:
get "Followers" do
tags "followers"
description(<<-DESCRIBE.strip)
This endpoint allows the client to retrieve a list of the followers they have.
"Followers" are users that are following other users on the website.
It supports pagination, each page will contain 80 followers by default.
DESCRIBE
operationId "getFollowers"
produces "application/json"
end
Below, I’ve taken some of the definitions from the specification and applied it to the code sample to help explain what each field does.
tags: A list of tags for API documentation control. They are used for logical grouping of operations by resources or any other qualifier. In this case, we want this endpoint to be grouped on its own and so we provide it with a new tag. In other circumstances, you may want to tag your crud operations for a single resource all with the same tag so that they can logically group together.
description: Provides a verbose explanation of the operation behavior. CommonMark syntax may be used for rich text representation. We try to describe what the endpoint does in a single sentence. Thereafter, we can provide additional context that we think will be useful to the user of the API at a glance.
operationId: This is a unique string used to identify the operation. The id must be unique among all operations described in the API. The operationId value is case-sensitive. Tools and libraries may use the operationId to uniquely identify an operation, therefore, it is recommended to follow common programming naming conventions. We try to structure the work with the CRUD operation as the prefix and the resource as the suffix.
produces: This field populates the response content type with the produces property of the OpenAPI definition. Our response type is usually JSON.
Another noteworthy field that we add to other endpoints but is not relevant to this particular endpoint is:
security: This is a declaration of which security mechanisms can be used across the API. Individual operations can override this definition.
We define a security scheme globally in swagger_helper.rb levelsecurity: [{ "api-key": [] }],. The security scheme that we utilize applies authentication via an api-key header.
However, remember earlier I mentioned that not all endpoints need authentication. Hence, for those that do not need authentication, we can override the "security" attribute at the operation level. To do this, we provide an empty security requirement ({}) to the array.
If you look at /api/articles in the article_spec you will see security [] which indicates that this endpoint does not need authentication.
However, in the case of the endpoint that we’re documenting /api/followers/users we do not need to provide a security field as we’ll use the one that is defined globally with the API key for authentication via an api key.
Now that we’ve set these operationIds, let’s have a look at how they would get generated in the JSON file.
The operationIds set the scene for how the API operates. Next, we’ll want to define the parameters that the API endpoint allows.
The following two sections, Parameters and Response Blocks, rely on schemas, so before we get into the details of these sections let’s first discuss what a schema is.
The OpenAPI Specification allows you to describe something called a "schema" which in its simplest form refers to some JSON structure. These can be defined either inline with your operation descriptions OR as referenced globals.
You can use a referenced global when you have a repeating schema in multiple operation specs. For example, an article resource may be returned in a create, read or update, hence instead of repeating this JSON across all these operations you could add it as a referenced global in the swagger_heper.rb and then use that $ref.
The global definitions section lets you define common data structures used in your API. They can be referenced via $ref: "a schema object" – both for request body and response body.
Another instance where you may want to use a referenced global is when multiple endpoints accept the same parameter schema and you do not want to repeat this JSON for multiple endpoints.
Parameters
If you look at the code for the api/followers/users endpoint, you’ll notice that it takes three optional parameters; page, per_page and sort.
You’ll notice that we use page and per_page across multiple endpoints for our pagination strategy hence that reusability makes it the ideal candidate for a referenced global.
Since it’s been used before, you’ll find it defined globally in the swagger_helper.rb. Hence, all we need to do is reference it in our spec.
parameter "$ref": "#/components/parameters/pageParam"
parameter "$ref": "#/components/parameters/perPageParam30to1000"
However, our next parameter - sort, has not been defined before and does not seem to be re-used in the same manner across any existing endpoints. Thus we can define it inline.
parameter name: :sort, in: :query, required: false,
description: "Default is 'created_at'. Specifies the sort order for the created_at param of the follow
relationship. To sort by newest followers first (descending order) specify
?sort=-created_at.",
schema: { type: :string },
example: "created_at"
You can read more about describing query parameters in the OpenAPI Guide here
This is what the final set of query parameters look like:
get "Followers" do
......
parameter "$ref": "#/components/parameters/pageParam"
parameter "$ref": "#/components/parameters/perPageParam30to1000"
parameter name: :sort, in: :query, required: false,
description: "Default is 'created_at'. Specifies the sort order for the created_at param of the follow
relationship. To sort by newest followers first (descending order) specify
?sort=-created_at.",
schema: { type: :string },
example: "created_at"
end
And this is what the corresponding generated JSON would look like:
Response Blocks
Once we’ve defined the operation Ids and Parameters, we can define what the response query looks like. We can create multiple response blocks in order to test the various responses a user of the API may receive. This includes testing when the API endpoint provides a response, when there is no content, when the user is not authorized etc.
In this case, we’ll test what a successful response looks like and what an unauthorized response looks like when we do not provide the correct api-key.
A successful response with status code 200
response "200", "A List of followers" do
let(:"api-key") { api_secret.secret }
schema type: :array,
items: {
description: "A user (follower)",
type: "object",
properties: {
type_of: { description: "user_follower by default", type: :string },
id: { type: :integer, format: :int32 },
user_id: { description: "The follower's user id", type: :integer, format: :int32 },
name: { description: "The follower's name", type: :string },
path: { description: "A path to the follower's profile", type: :string },
profile_image: { description: "Profile image (640x640)", type: :string }
}
add_examples
run_test!
end
The HTTP 200 OK success status response code indicates that the request has succeeded.
In order for the request to succeed, we first need to provide the necessary authentication. When this example runs, it will need the api-key for authentication, hence we set it in our RSpec test.
In this case, I’ve decided to define the schema object inline because it is a uniquely structured response that is not being shared with other endpoints. However, if more than one endpoint had the same schema it would have been beneficial to define it globally in the swagger_helper and then provide a reference to it in the various spec files.
schema type: :array,
items: {
description: "A user (follower)",
type: "object",
properties: {
type_of: { description: "user_follower by default", type: :string },
id: { type: :integer, format: :int32 },
user_id: { description: "The follower's user id", type: :integer, format: :int32 },
name: { description: "The follower's name", type: :string },
path: { description: "A path to the follower's profile", type: :string },
profile_image: { description: "Profile image (640x640)", type: :string }
}
}
The schema that we have defined above is an array of objects. The top level type is defined by an array type and each item is an object. Thereafter, we further define the properties that can be expected in each object. We advise that a description for a property is added where necessary.
If the need to re-use the schema object for multiple endpoints arose, we could have defined it as a Follower in the swagger_helper.spec and then referenced it in our spec like below:
schema type: :array,
items: { "$ref": "#/components/schemas/Follower" }
The add_examples method can be found in the swagger_helper and it is responsible for creating the Response Object.
It creates a map containing descriptions of potential response payloads.
The key is a media type or media type range like application/json and the value describes it.
Finally, the run_test! method is called within each response block. This tells rswag to create and execute a corresponding example. It builds and submits a request based on parameter descriptions and corresponding values that have been provided using the rspec "let" syntax. In order for our examples to add value, we want to give it a good set of seed data.
If you want to do additional validation on the response, you can pass a block to the run_test! method.
You can read more about how to use run_test! from the rswag documentation.
An unauthorized response with status code 401
response "401", "unauthorized" do
let(:"api-key") { nil }
add_examples
run_test!
end
The HyperText Transfer Protocol (HTTP) 401 Unauthorized response status code indicates that the client request has not been completed because it lacks valid authentication credentials for the requested resource. Hence, in this case if we provide an invalid api key, we should expect a 401.
To test this case, we simply provide an invalid API key which will not allow us to authenticate to the API.
The generated JSON for these two responses look as follows:
Generating the JSON
I’ve been referencing the JSON files above, but you must be wondering how do you access that JSON. Once you have written your spec, you can generate the JSON file for the API by running
SWAGGER_DRY_RUN=0 RAILS_ENV=test rails rswag PATTERN="spec/requests/api/v1/**/*_spec.rb"
Once you do this, you will see a newly generate file at https://github.com/forem/forem/blob/main/swagger/v1/api_v1.json.
Take the time to evaluate the generated content in this file, especially for the new spec. In order to view it you may paste the JSON into https://editor.swagger.io/. When you do this, it will display the data as documentation and also let you know if there are any errors.
If you have Visual Studio Code, we suggest you install the following extensions that enable validation and navigation within the spec file:
And that, my friends, is how we document API v1 endpoints at Forem.
You can find the code for this example here.
If you have any questions or feedback, please drop them in the comments below. If you’d like to contribute to our documentation please have a look through the ones that aren’t assigned in this milestone and raise your hand on the issue. We look forward to your contributions.
Top comments (11)
Are you interested in generating OpenAPI directly from your test cases? Here’s a post about doing exactly this for Mastodon:
dev.to/appmap/automatically-genera...
It uses your regular test suite - you don’t have to rewrite or rearrange tests like you do with RSwag.
Ooooh this looks interesting - thanks for dropping it here @kgilpin, I'll give it a read 👀.
Hey, I just wanted to let you know that we've prepared a PR showing how to generate OpenAPI for Forem with AppMap - github.com/forem/forem/pull/19041
In the PR description you'll find information about how it works, and some instances of how the generated OpenAPI can enhance the definitions that are already in the Forem repo.
Using OpenAI is a great tool to see how we can play with datas. Such a good idea! Can't wait when the version is released to updating some projects like
An example of getting data from the dev.to API to display its own articles.
Getting data from the API of DEV.TO
An example of getting data from the dev.to API to display its own articles. Work with VueJS, fetch and the API v0 of dev.to (this version will be DEPRECIATED. See the post on DEV).
How to get my data ?
Change this lines in the file src/components/devto.vue :
How to get my ID ?
Get your ID by using the website. Press F12, and on the
bodyelement, you have adata-userattribute. This is your ID.How to develop this project
Project setup
Compiles and hot-reloads for development
Compiles and minifies for production
Lints and fixes files
Customize configuration
See Configuration Reference.
Us too! 🔥
This is a great and very thorough article. I'm def interested in contributing. Thanks for sharing.
A couple things to clarify: Swagger has been renamed as OpenAPI since 2015. Any reference to the specification should use OpenAPI. Swagger remains many products offered by SmartBear. Unfortunately, many tools in the ecosystem used the swagger moniker in their package names and haven't been updated. I see you tried to explain it but the terms should not be used interchangeably.
smartbear.com/blog/what-is-the-dif...
Secondly, now would be a great time to move to OAS3.1.x to take advantage of the full JSON Schema support. This allows usage of many of JSON Schema's features from draft 2020-12 and beyond, without limitation. Whereas OAS 3.0.x has quite a few limitations to design proper JSON Schemas. Not to mention it's based on a modified draft-04 version which is quite old.
Yup, I came to comment the exact same thing - should be starting from OAS rather than Swagger / aka “OAS pre-OAS” 😌 happy to help with some of this work, so I’ll go ahead and talk a look at the related GitHub issue!
Thanks @andypiper. I went ahead an updated the article to make it a bit more clear and removed the confusing references where appropriate to the previous name (Swagger).
We're looking forward to your contribution ✨
Hi @jeremyfiel 👋🏽 Thanks so much for bringing that to my attention, I went ahead an updated the article to make it a bit more clear and removed the confusing references where appropriate to the previous name (Swagger).
That's a really good point, I'll bring it up to the team, we'll chat more about it and what it entails. Hopefully, we can upgrade soon!
Thanks again, we're looking forward to your contribution ✨
Interesting read and interesting approach to API documenting Ridhwana, thanks 👍
Talking about the OAS file you mentioned in the post, for me it was really interesting to audit it with 42Crunch audit tool (I work there but I'm a UI developer and not selling it just in case, full disclosure 😄 and it's free anyway). If you check out the API docs with it (either through UI or using the VS Code plugin), you may find that first, the audit is blocked by a structural JSON issue (schema error):
and second, if you fix that quickly, there are some issues of different criticality which would be not very hard to fix, provided with the decent remediation instructions:
Again, I'm not a dev advocate of any kind, I just personally use the tool for my pet project and it helps me a lot, as I'm not a BE dev. And I truly wish Forem to become better and better, being a contributer myself for a couple of times 😅
@ridhwana have you thought about adding SDKs for the Forem API? It could help let with usability and integration speed. For some ideas, check otu the Forem SDKs I created in Go, Python, TS, and Typescript from your OpenAPI spec.
If you're interested, I could help you set up automation so that every time a new version of your OpenAPI spec is published, a new version of the SDKs are published. Let me know! Really admire the work you and the forem team do.