StudySpy: Building the new PublicApi v2

How I built StudySpy's Public API from scratch: A solo developer's journey through tech choices, security challenges, and creating real value. Learn key strategies for your own API projects.

This year, I embarked on an exciting journey to build a new Public API system for StudySpy, one of my clients. The goal was to create a system that would allow StudySpy to sell access to their valuable dataset to external parties. This project wasn't just about opening up new revenue streams; it was about innovation and expanding the value StudySpy could offer to the New Zealand education sector.

APIs play a crucial role in modern software systems, facilitating the flow of essential data across various components and platforms. By enabling third parties to build products on top of StudySpy's infrastructure, we were set to significantly increase the platform's value and reach.

But our vision extended beyond external use. We aimed to create an API that would also serve as an internal standard for data access across the platform. This dual-purpose approach promised to streamline operations and set a new benchmark for efficiency within StudySpy.

Building the Infrastructure

Starting such a project can be daunting, but I've learned that the act of beginning is often the most crucial milestone. Once you have something tangible, it becomes much easier to iterate and improve. However, before diving into coding, I knew that designing the right infrastructure from the start would be half the battle. The old adage "measure twice, cut once" came to mind.

My focus was on three key areas:

1. Technologies: Choosing the right tech stack
2. Structure: Ensuring maintainability through proper project organization
3. Security: Implementing robust measures to protect the main platform

Technologies

Given that the main StudySpy platform runs on .NET 5/6+, I decided to leverage this technology as the foundation for the new API system. This decision immediately descoped a significant amount of effort – a critical consideration for a solo developer.

I also wanted to incorporate Swagger with it'OpenAPI specification, tools I'd had positive experiences with in the past. These two key technologies formed the base upon which I built the rest of the stack.

The final tech stack included:

• .NET
• Swagger + OpenAPI specification
• Microsoft SQL Server
• Docker
• Microsoft entity framework

Project Structure

For project structure, I opted to create a new area within the existing solution rather than a separate .sln file. This approach maintains a cohesive view of the entire codebase, which is particularly beneficial for smaller teams.

I started by defining the routes for the initial version of the API:

/api/v2/campuses
/api/v2/campuses/[id]
/api/v2/courses
/api/v2/courses/[id]
/api/v2/intakes
/api/v2/intakes/[id]
/api/v2/providers
/api/v2/providers/[id]

Based on these routes, I created the following directory structure:

PublicApi/
├── Controllers/
│   ├── CampusesController.cs
│   ├── CoursesController.cs
│   ├── IntakesController.cs
│   ├── ProviderController.cs
│   └── ScholarshipController.cs
├── Models/
│   ├── Campus.cs
│   ├── Course.cs
│   ├── Intake.cs
│   ├── Provider.cs
│   └── Scholarship.cs
└── Services/
    (initially empty, to be populated as needed)

I began by defining the models, which shaped the API's response payloads. With these in place, I could then set up basic GET endpoints for both list and individual resource retrieval, initially returning dummy data.

Here's a basic example of one of the domain-driven endpoints:

PublicApi/Models/Campus.cs

using System.Collections.Generic;

namespace StudySpy.Web.Areas.PublicApi.Models
{
    public class Course
    {
        public string Name { get; set; }
        public string SsId { get; set; }
        public string SsProviderId { get; set; }
        public string LevelOfStudy { get; set; }
        public List<string> AreasOfStudy { get; set; }
        public List<string> SubjectsOfStudy { get; set; }
        public string Details { get; set; }
        public string EntryRequirements { get; set; }
        public string Duration { get; set; }
        public bool Active { get; set; }
        public long LastUpdated { get; set; }
        public bool InternationalFlag { get; set; }
    }
}

PublicApi/Controllers/CoursesController.cs

using Microsoft.AspNetCore.Mvc;
using System.Collections.Generic;
using StudySpy.Web.Areas.PublicApi.Models;

namespace StudySpy.Web.Areas.PublicApi.Controllers
{
    [Area("PublicApi")]
    [ApiController]
    [Route("api/v2/[controller]")]
    [ApiExplorerSettings(GroupName = "PublicApi")]
    public class CoursesController : ControllerBase
    {
        [HttpGet]
        public ActionResult<IEnumerable<Course>> GetCourses()
        {
            // Dummy data for illustration
            var courses = new List<Course>
            {
                new Course 
                { 
                    Name = "Computer Science",
                    SsId = "CS101",
                    SsProviderId = "UNIV001",
                    LevelOfStudy = "Undergraduate",
                    AreasOfStudy = new List<string> { "Technology", "Mathematics" },
                    SubjectsOfStudy = new List<string> { "Programming", "Algorithms", "Data Structures" },
                    Details = "A comprehensive course covering the fundamentals of computer science.",
                    EntryRequirements = "High school diploma with strong mathematics background",
                    Duration = "4 years",
                    Active = true,
                    LastUpdated = 1630444800, // Unix timestamp for September 1, 2021
                    InternationalFlag = true
                },
                new Course 
                { 
                    Name = "Business Administration",
                    SsId = "BA201",
                    SsProviderId = "UNIV001",
                    LevelOfStudy = "Graduate",
                    AreasOfStudy = new List<string> { "Business", "Management" },
                    SubjectsOfStudy = new List<string> { "Finance", "Marketing", "Operations" },
                    Details = "An MBA program designed for aspiring business leaders.",
                    EntryRequirements = "Bachelor's degree and 2 years of work experience",
                    Duration = "2 years",
                    Active = true,
                    LastUpdated = 1641024000, // Unix timestamp for January 1, 2022
                    InternationalFlag = true
                }
            };

            return Ok(courses);
        }

        [HttpGet("{id}")]
        public ActionResult<Course> GetCourse(string id)
        {
            // Dummy data for illustration
            var course = new Course
            {
                Name = "Data Science",
                SsId = id,
                SsProviderId = "UNIV001",
                LevelOfStudy = "Graduate",
                AreasOfStudy = new List<string> { "Technology", "Statistics" },
                SubjectsOfStudy = new List<string> { "Machine Learning", "Big Data", "Statistical Analysis" },
                Details = "An advanced course in data science and analytics.",
                EntryRequirements = "Bachelor's degree in a quantitative field",
                Duration = "2 years",
                Active = true,
                LastUpdated = 1651363200, // Unix timestamp for May 1, 2022
                InternationalFlag = true
            };

            return Ok(course);
        }
    }
}

This setup allowed me to perform my first test using a Bruno client, and voila! I received my first response. Now we're getting somehwere 🎉

Security

Before shipping anything, implementing security measures was crucial. I focused on two main concerns:

1. Unauthorized access: I implemented a rough-cut API key authentication strategy and added our new API pages to the robots.txt file to prevent search engine indexing. This would mean no-one would really stumble across our API accidentally, and if they did, they wouldn't be able to access our system without a valid API key.
2. Protection against DoS: To mitigate the risk of database read operation overload (intentional or unintentional), I implemented API key rate limiting in combination with IP rate limits. I set a sensible limit of 120 requests per minute (2 requests per second) to maintain a reasonable SLA while protecting the system from accidental request floods.

With these security measures in place, I was able to release the initial implementation of the new API routes to a production environment and take them for a spin.

Making it a Product

While I had a basic API in production, transforming it into a valuable product required several additional steps:

Adding Business Logic

We needed to get the routes wired up so they could start returning something valuable to the user. I used Microsoft's Entity Framework ORM to pull records from the SQL database and map them to the API response payloads. This process involved creating data access layers and implementing the necessary business logic in each controller.

Creating Services for Reusable Logic

To promote code reuse and maintain a clean separation of concerns, I abstracted common business logic into services. By the end of the project, I had created several utility services:

Services/
├── AuthorityFormatter.cs
├── DateFormatCalculator.cs
├── DeliveryCalculator.cs
├── DurationCalculator.cs
├── HtmlHelper.cs
├── MacronRemover.cs
├── ProviderTypeFormatter.cs
├── RegionMapper.cs
├── StreetAddressFormatter.cs
├── StringFormatter.cs
└── SubjectTaxonomyMapper.cs

These provided me with a way I could share standarised business logic across various endpoints.

Implementing Pagination

Pagination was crucial for allowing third parties to navigate through records at the API level efficiently. I created a pagination model and service:

PublicApi/Models/Pagination.cs

using System.Collections.Generic;

namespace StudySpy.Web.Areas.PublicApi.Models
{
    public class ApiPaginatedResponse<T>
    {
        public List<T> Items { get; set; }
        public Pagination Pagination { get; set; }
    }

    public class Pagination
    {
        public int Skip { get; set; }
        public int Limit { get; set; }
        public int Count { get; set; }
        public string NextPage { get; set; }
        public int TotalCount { get; set; }
    }
}

I then wired this object up to all API response payloads:

var response = new ApiPaginatedResponse<Campus>
{
    Items = CoursesPayload,
    Pagination = pagination
};

return Ok(response);

Implementing Correct HTTP Response Codes

An often overlooked but crucial aspect of API design is the proper use of HTTP response codes. These codes provide immediate feedback to API consumers (and developers) about the status of their requests, making the API more intuitive and easier to work with.

I made sure to implement a range of appropriate status codes in our API responses:

• 200 OK: For successful GET, PUT, or PATCH requests
• 400 Bad Request: When the request is malformed or contains invalid parameters
• 401 Unauthorized: When authentication is required but not provided or is invalid
• 403 Forbidden: When the authenticated user doesn't have permission to access the requested resource
• 404 Not Found: When the requested resource doesn't exist
• 429 Too Many Requests: When the client has sent too many requests in a given amount of time (rate limiting)
• 500 Internal Server Error: For unexpected server errors

By consistently using these status codes across all endpoints, we ensure that our API communicates clearly with its consumers, delivering the best developer experience for uses of our API.

Customizing the UI

To enhance the user experience and bring our brand to the API, I customized the Swagger UI by replacing the generic branding with StudySpy's logo and applying clean, consistent styling.

I also added example response payloads for various endpoints:

Swagger/
└───Examples
    ├───CampusExamples
    │       CampusListResponseExample.cs
    │       CampusResponseExample.cs
    │
    ├───CourseExamples
    │       CourseListResponseExample.cs
    │       CourseResponseExample.cs
    │
    ├───IntakeExamplesse | Format-Tree                                                     [History]
    │       IntakeListResponseExample.cs
    │       IntakeResponseExample.cs
    │
    ├───ProviderExamples
    │       ProviderListResponseExample.cs
    │       ProviderResponseExample.cs
    │
    ├───ScholarshipExamples
    │       ScholarshipListResponseExample.cs
    │       ScholarshipResponseExample.cs
    │
    └───StatusCodes
            400ResponseExample.cs
            401ResponseExample.cs
            404ResponseExample.cs
            429ResponseExample.cs

Provisioning API Keys

Before sharing the new Swagger documentation, I generated and distributed API keys to the third parties who would be using the new API. This process involved creating a secure system for generating, storing, and managing these keys.

You don’t need to over-think this. We assigned API keys to a singleton and added code comments to identify the associated users. This simple system allowed for easy management and potential future revocation of access if needed.

Lessons Learned

Building this project was an exciting challenge that taught me several valuable lessons:

1. Start small and iterate: Beginning with a minimal viable product and improving it continuously proved to be an effective strategy.
2. Define first, then build: Having a clear understanding of the desired responses greatly simplified the process of writing business logic.
3. Don't stress the small stuff: Getting the service into users' hands quickly for feedback is crucial, even if it's not 100% polished.
4. Leverage existing resources: Choosing to use languages and frameworks already in play resulted in a more cohesive solution and easier future maintenance.

Future Horizons

The development of this API has not only expanded StudySpy's offerings but has also opened up new possibilities for innovation in the education sector for New Zealand. As more organizations recognize the power of APIs in driving growth and fostering ecosystems, projects like this will become increasingly vital.

Whether you're considering building an API for your own project or looking to leverage existing APIs in your organization, remember that the journey of a thousand miles begins with a single step. Start small, focus on delivering value, and don't be afraid to iterate and improve as you go.

If you’ve made it this far and are interested in viewing the swagger documentation on the live site, you can view it here:
https://studyspy.ac.nz/api/v2/documentation

Like you, I'm always interested in learning more. If you have any feedback or questions about this project or API development in general, feel free to reach out to me on Twitter.