Elizabeth Adeotun Adegbaju for AWS Community Builders

Posted on Feb 17

How to Supercharge Your AWS Well-Architected Reviews: A Step-by-Step Guide to Optimize Cloud Workloads

#wellarchitected #aws #programming #automation

You’ve likely found yourself here because you conduct numerous Well-Architected Reviews, Wilkommen! This article delves into one of my talks presented at AWS Community Day DACH 2023, where I shared expert insights on delivering the most effective Well-Architected Review you’ve ever done, leaving you feeling like a superstar.

Walk with me as I guide you through the key points we discussed and share additional insights to elevate your review process.

Introduction: The Reality of AWS Well-Architected Reviews

The AWS Well-Architected Framework is a powerful tool designed to help organizations build and maintain secure, reliable, performant, cost-optimized, operationally excellent, and sustainable applications in the AWS Cloud.

However, conducting a Well-Architected Review (WAR) can often feel like a daunting task. The Well-Architected Review process involves answering numerous questions, identifying risks, and generating reports that align with best practices. But what if there was a way to supercharge this process, making it more efficient, targeted, and insightful?

In this article, I will give you tips on how to customize and automate your AWS Well-Architected Reviews, leveraging the AWS WA Tool API and other services to create tailored reports that meet your organization’s specific needs.

The Need for Customization in Well-Architected Reviews

Why One-Size-Fits-All Doesn’t Work

The default AWS Well-Architected Review process is comprehensive, but it may not always align with the specific needs of your organization. A generic report might include information that isn’t relevant to your workload, or it might lack the depth required for your team to take actionable steps. Customizing your review process allows you to:

Target Specific Audiences: Tailor reports to different stakeholders, such as developers, managers, or executives.
Highlight Expertise: This is an opportunity to stand out from the sea of generic reports and deliver results with a touch of excellence.
Eliminate Bulky Reports: Do you really think your clients read the 76 pages of the black-and-white AWS report that is generated in the Well-Architected Tool?
Give Detailed Insights: Add custom recommendations in your reports or even add visuals like charts or tables.
Ensure Efficiency: Automate repetitive tasks or create a template that works for you, do something different to save time and resources.
Adhere to Company Standards: Align the review process with your organization’s internal standards and best practices, generate branded reports.
Be Prepared: Go into the review workshop well-prepared in a fraction of the time it would take you to get the same amount of information manually.

Key Stages of a Customized Well-Architected Review

1. Before the Review

Before diving into the review, it’s essential to prepare adequately. This stage involves:

Inspecting the Workload Environment: Understand the architecture, resources, and dependencies of the workload you’re reviewing.
Identifying Focus Areas: Determine which of the six pillars of AWS Well-Architected Framework (Operational Excellence, Security, Reliability, Performance Efficiency, Cost Optimization, and Sustainability) are most relevant to your workload.
Automating Custom Report Generation: Set up scripts or tools (like Steampipe, Former2) to automate the generation of reports or gather insights into the state of the workload.

2. During the Review

The review process itself can be optimized by:

Prioritizing Pillars: Focus on the most critical pillars for your workload, use the features available in the Well-Architected Tool (like using Profiles).
Taking Clear and Concise Notes: Document findings and recommendations in a manner that makes the review process more efficient and easy for you to organize your results.
Using Appropriate Lenses: AWS provides various lenses (e.g., SaaS Lens, Serverless Lens) that offer additional guidance tailored to specific types of workloads. Sometimes the default Well-Architected Lens is just not for you.

3. After the Review

Once the review is complete, the focus shifts to action:

Collating Recommended Action Items: Compile a list of actionable recommendations based on the review findings.
Building Custom Reports: Generate reports that are tailored to your needs, incorporating only relevant information.
Attaching the Default AWS Report (Optional): While custom reports are valuable, it’s also okay to include the default AWS report for reference or documentation purposes.

Automating the Review Process

Why Automate?

Automation is key to making the Well-Architected Review process more efficient. By automating tasks such as data collection, report generation, and integration with other tools (e.g., Confluence), you can save time and reduce the risk of human error.

Key Considerations for Automation

When automating your Well-Architected Reviews, consider the following:

Security: Always follow AWS best practices for IAM users and roles. Ensure that your automation scripts run in a secure environment.
Required Libraries: Choose the right programming language, open-source tools, and dependencies for your project.
API Integration: Use the AWS Well-Architected Tool API to programmatically access workload data, reviews, and improvement plans. You can access it in your code as part of the AWS SDK in different languages, here is the Python example.
Data Selection: Decide which data elements to include in your custom reports. Focus on the most relevant information for your audience.
Publication Format: Determine where and how the report will be published. For example, you might publish the report in Confluence or export it as a PDF.

Technical Implementation

I made an effort to avoid overwhelming you with excessive code. Instead, I took the time to explain the necessary steps and provided a snippet of the implementation for that specific stage. The actual implementation, however, is entirely up to you - after all, customization is at the heart of what we’re discussing here.

Exhibit 1: Setting Up and Fetching Workload

To get started, you’ll need to set up your environment. This involves configuring your AWS CLI profile and installing the necessary libraries. Then you can proceed to fetch the workloads from the AWS environment.

import os
from datetime import datetime
import boto3
import json
import pandas as pd
from atlassian import Confluence
from decouple import config

# Create a session using the AWS CLI profile
session = boto3.Session(
    profile_name=config("AWS_PROFILE", default="default")
)
wellarchitected = session.client("wellarchitected")

if not os.path.exists("report"):
    os.mkdir("report")

# Fetch the list of workloads and save it to a file
with open("report/workloads.json", "w") as file:
    workloads = wellarchitected.list_workloads()["WorkloadSummaries"]
    json.dump(workloads, file, indent=4, default=str)
    file.close()

print("Workloads fetched successfully. \nChoose a workload from the list below: ")

# Print the list of workloads and ask the user to select one
for index, workload in enumerate(workloads):
    print(f'{index}: {workload["WorkloadName"]}')

workload_index = input("Enter index of selected workload: ")
if not workload_index.isdigit() or int(workload_index) >= len(workloads):
    print("Error: Invalid workload index!")
    exit(1)
workload_id = workloads[int(workload_index)]["WorkloadId"]

Exhibit 2: Building an Overview Page

Once you’ve selected a workload, you can build an overview page using any method of your choice. This depends entirely on your planned output. I am publishing my report to confluence in the end so I am building my report using Confluence Wiki Markup.

# Build the Overview Page using Confluence Wiki Markup
overview = (
    "h1. Overview\n h2. Workloads properties\n || Property || Value ||\n"
)

with open(f"report/workload_{workload_id}.json", "r") as workload_file:
    workload = json.load(workload_file)
    date = datetime.fromisoformat(workload["UpdatedAt"])
    name = workload["WorkloadName"]
    overview += (
        f'|| Workload Name | {workload["WorkloadName"]} |\n'
        f'|| Description | {workload["Description"]} |\n'
        f'|| Workload Owner | {workload["ReviewOwner"]} |\n'
        f'|| Environment | {workload["Environment"]} |\n'
        f'|| AWS Regions | {str(workload["AwsRegions"])[1:-1]} |\n'
        f'|| Account IDs | {str(workload["AccountIds"])[1:-1]} |\n'
        f'|| Updated At | {date.strftime("%d %B %Y %H:%M:%S %Z")} |\n'
    )
    workload_file.close()

Exhibit 3 & 4: Data Selection and Data Wrangling

Ask yourself, "What specific data elements do I intend to incorporate in the customized report?" This is a sneakpeek of my script where I selected only high risk and medium risk items in my report with colour-coding for visual cues.

recommended_actions = f"h1. Recommended Actions \n\n {INTRO}\n\n"
# Load answer todos data
with open(f"/tmp/report/answer_todos_{workload_id}.json", "r") as file:
    answer_data = json.load(file)

# Create a dictionary to store questions for each pillar
pillar_questions = {pillar_id: [] for pillar_id in PILLAR_ABBR_IDS}

# Group questions by pillar
for answer in answer_data:
    pillar_id = answer["ChoicesToDo"][0]["ChoiceId"].split("_")[0]
    pillar_questions[pillar_id].append(answer)

# Loop through each pillar and print the questions
for pillar_id in PILLAR_ABBR_IDS:
    pillar_name = get_formatted_pillar(pillar_id)
    recommended_actions += f"h1. {pillar_name}\n\n----\n\n"

    for answer in pillar_questions[pillar_id]:
        title = answer["Title"].strip()
        risk = answer["Risk"]

        if risk == "HIGH":
            recommended_actions += "h2. {color:red}"
        elif risk == "MEDIUM":
            recommended_actions += "h2. {color:orange}"

        recommended_actions += title + "{color}\n\n"

        for choice in answer["ChoicesToDo"]:
            if choice["Title"] == "None of these":
                continue
            link = f"https://docs.aws.amazon.com/wellarchitected/latest/framework/{choice['ChoiceId']}.html"
            recommended_actions += f"* [{choice['Title'].strip()} | {link}]\n{choice['Description'].strip()}\n\n"

When you integrate additional tools into your review process, you can also incorporate their results into your automation. For example, the following code snippet demonstrates how to extract data from Steampipe results and include them in your automated workflow:

recommended_actions += (
    f"\n\nh1. AWS Environment Findings\n\n {FINDINGS_INTRO}\n\n "
)

data = pd.read_csv("./report/steampipe_data.csv")
filtered_data = data[data["status"] == "alarm"]
grouped_data = filtered_data.groupby("group_id") for group_id, group_df in grouped_data:
    # Find the start and end positions of the substring to extract
    start_index = group_id.find("well_architected_framework_") + len(
        "well_architected_framework_"
    )

    # Extract the desired substring
    extracted_text = group_id[start_index:-5]
    group_section = f"h2. {str.upper(extracted_text)} {group_df['title'].iloc[0][4:]}\n\n"

    # Group rows within the group by "control_id"
    control_grouped_data = group_df.groupby("control_id")

    # Loop through the control groups within the group
    for control_id, control_df in control_grouped_data:
        # Create a Markdown section for the control within the group
        control_section = (
            f"h3. {control_df['control_title'].iloc[0]}\n"
            f"|| Reason || Resource ||\n"
        )

        # Loop through the rows in the control group
        for index, row in control_df.iterrows():
            reason = row["reason"]
            resource = row["resource"]

            section = f"| {reason} | {resource} |\n"
            control_section += section

        group_section += control_section
    recommended_actions += group_section

Exhibit 5: Publishing the Report

Finally, don't forget the part where you package your report and export it to your desired format. I published to Confluence but you can select any platform (or output format) of your choice.

# Connect to Confluence API
base_url = config("CONFLUENCE_URL")  # Load from env secret
confluence = Confluence(
    url=base_url,
    token=config("CONFLUENCE_TOKEN"),  # Load from env secret
)

# Check if page already exists
page = confluence.get_page_by_title(
    space=config("CONFLUENCE_SPACE"),  # Load from env secret
    title=f"{workload_name} {updated_at.strftime('%d-%m-%Y')}",
)

if page:
    # Inform user that page already exists
    print(f"Existing_report found at {base_url + page['_links']['webui']}")
    update_report = input("Do you want to overwrite the existing report? (y/n): ")
    if update_report == "n":
        print("Exiting...")
        return
    elif update_report == "y":
        print("Updating report...")
        # Update existing page
        page = confluence.update_page(
            page_id=page["id"],
            title=f"{workload_name} {updated_at.strftime('%d-%m-%Y')}",
            body=report,
            representation="wixi",
        )

Conclusion: Supercharging Your Reviews

Customizing and automating your AWS Well-Architected Reviews can significantly enhance the efficiency and effectiveness of the process. By leveraging the Well-Architected Tool API and other AWS services, you can create tailored reports that provide actionable insights for your clients.

Further Learning

If you’re interested in diving deeper into customizing your AWS Well-Architected Reviews, check out these resources:

DEV Community