DEV Community

Ricardo Sueiras for AWS

Posted on

AWS open source newsletter, #207

Edition #207 - February 2025

Welcome to issue #207 of the AWS open source newsletter, the newsletter where we try and provide you the best open source on AWS content. As always, we have more great new projects to check out, which include real time analysis of your client calls to AWS APIs, a dashboard for all your AWS Health Events, an EBS analysis tool, a SIEM tool that allows you to query your AWS events, a really cool AI Coding Assistant powered by Amazon Bedrock, a really great demo of how to implement an MCP server, running WhispherX on AWS Lambda, and more!

The projects will keep you busy until next month for sure, but we also have plenty of reading material in this months newsletter. We feature projects including kro, AWS CDK, OpenTelemetry, Kubernetes, dstack, GraphRAG, FFMpeg, ArgoCD, Terragrunt, OpenTofu, Django, PostgreSQL, Apache Kafka, Amazon Linux, Apache Flink, Apache Airflow, Apache Spark, MySQL, Qonto, Prometheus, Amazon EMR, Apache Iceberg, OpenSearch, Valkey, AWS Amplify, Lustre, InfluxDB, Cedar, and Aider. Finally, don't forget to check out the videos as I have a couple of great ones for you in this edition.

As always, get in touch if you want me to feature your projects in this open source newsletter. Until the next time, I will leave you to dive into the good stuff!

Latest open source projects

The great thing about open source projects is that you can review the source code. If you like the look of these projects, make sure you that take a look at the code, and if it is useful to you, get in touch with the maintainer to provide feedback, suggestions or even submit a contribution. The projects mentioned here do not represent any formal recommendation or endorsement, I am just sharing for greater awareness as I think they look useful and interesting!

Tools

aws-client-monitor

aws-client-monitor is a new tool from Roman Tsypuk designed to analyse applications interacting with AWS services. It's particularly useful during local development, troubleshooting, or profiling third-party applications. It allows you to do real-time monitoring, capturing and displaying AWS API invocations in real time, and is easy to get started with. It comes with a gorgeous dashboard, so this is on my todo list to try out. Very nice Roman!

example dashboard for aws-client-monitor

aws-health-events-insight

aws-health-events-insight this project provides a very nice solution to centralise, store and then analyse your AWS Health events. Health Events Intelligence Dashboards and Insights (HEIDI) Data Collection Framework enables you to collect data from different accounts, regions and organisations. Check out the README for more details including how to deploy and configure this solution in your AWS environment.

ebsight

ebsight is a Python-based analysis tool developed by Stephen Jones in response to AWS's new EBS snapshot size reporting feature. This tool leverages the newly introduced FullSnapshotSizeInBytes field to provide comprehensive insights into EBS volume usage, performance metrics, and cost optimisation opportunities. After you have checked the repo out and gone through the README, why not read the supporting blog post, Unleash the Power of EBSight for Optimal AWS Storage Management

tailpipe

tailpipe is an open source SIEM for instant log insights from our friends at Turbot, powered by DuckDB. It allows you to analyse millions of events in seconds, right from your terminal. Check out the README that includes more info include a nice video demo of this in works. Bob Tordella (who we have featured many times over the years in this newsletter), has put together a blog post too, which is a must read. Go check it out - Query AWS CloudTrail Logs Locally with SQL

As the project says, select * from logs

vscode-iam-service-principal-snippets

vscode-iam-service-principal-snippets is the latest VSCode plugin from AWS Community Builder Danny Steenman (his fifth). This VS Code extension provides autocompletion of all AWS services that can be used as Service Principals in your IAM policies. Very hand indeed! Danny wrote:

The reason I created it is because when I write IAM policies or roles, I constantly found myself breaking out of my flow to hunt down the right AWS Service Principal from a GitHub gist maintained by the community.
It was effective, but having to toggle over to a browser, search, locate the specific principal, then switch back to VS Code was frustrating and inefficient.
Therefore I built this extension to keep me fully in the VS Code environment with context-aware auto-completion. Now, not only am I saving time, but the autocompletion suggestions are also automatically updated every week as new principals are released.

I've installed it, what are you waiting for...

bedrock-engineer

bedrock-engineer looks like an amazing project and one I am trying to find some time to play with. bedrock-engineer is Autonomous software development agent apps using Amazon Bedrock, capable of customise to create/edit files, execute commands, search the web, use knowledge base, use multi-agents, generative images and more. The project README has a short video that goes over some of the functionality and its pretty neat.

functionality of bedrock-engineer

StsSamlDriver

StsSamlDriver is A Python-based SAML authentication handler for AWS STS that allows you to get temporary credentials using SAML to the AWS CLI, or an application written using an AWS SDK without the need to screen scrape or emulate a browser.

Many tools exist today to get AWS credentials into an environment that can work with the AWS CLI or AWS SDK using SAML. However most of these scripts rely on parsing HTML or fully emulating a browser to do so which is difficult and may not be supported by various IDPs, especially on identity-as-a-service providers who may update the HTML on their login flows without notice.

This tool avoids interacting directly with a SAML IDP in any capacity by starting up a server on http://localhost:8090, which will receive a SAML assertion from your IDP on /saml from your browser. After receiving the assertion, the code will attempt to call AssumeRoleWithSAML to assume the role, and deliver the temporary credentials to your application and optionally launch an AWS console session. The server only listens for a single SAML assertion and will stop running after one is received.

You can use this tool by updating the ACS server of your SAML IDP (or target, or whatever your IDP's name is for where the SAML assertion is sent to) for a given application targeted at assuming a role on AWS to http://localhost:8090/saml , or doing similar with a userscript service like tampermoney, or custom browser plugins to override a SAML destination.

Demos, Samples, Solutions and Workshops

MCP2Lambda

MCP2Lambda is a project from my good friend Danilo Poccia and is a really great example of how Model Control Protocol (MCP) provides Large Language Model (LLM) with additional capabilities and flexibility. In this demo sample, an MCP server acts as a bridge between MCP clients and AWS Lambda functions, allowing generative AI models to access and run Lambda functions as tools. This is useful, for example, to access private resources such as internal applications and databases without the need to provide public network access. This approach allows the model to use other AWS services, private networks, and the public internet. I love that! As always, Danilos repos and code is top notch, so git this a go and don't forget to star it if you like it.

whisperx-on-aws-lambda

whisperx-on-aws-lambda is a project from Vincent Claes that shows you how you can run WhisperX (one of the most versatile and feature-rich Whisper variation that provides fast automatic speech recognition) on AWS Lambda - WhisperX goes serverless!

auth-pep-pdp

auth-pep-pdp is another solution from AWS Hero Jimmy Dahlqvist to add to the already great selection he shares in his repo. It provides code to help you implement PDP (Policy Decision Point) and a PEP (Policy Enforcement Point). We will build an simple API and use Amazon API Gateway and Lambda Authorizer as the PEP. It gets extra credit as its using Cedar for the policies. To help you get started with this project, you can check out the excellent README as well as the supporting blog post, PEP and PDP for Secure Authorization with AVP .

overview of architecture

draw-an-app

draw-an-app is a neat demo application that uses AWS Bedrock's Claude 3 Sonnet model to convert hand-drawn UI sketches into working web applications. It supports two modes of operation: 1/ Real-time webcam capture for immediate sketching and conversion, or 2/ Image upload through a user-friendly Streamlit interface. I have not tried this one yet, but if you do, let me know how you get on.

AWS and Community blog posts

This weeks essential reading

Here are the posts that I think are essential reads, so start here.

  • Kube Resource Orchestrator, From Experiment to Community Project - looks at a project that we launched at KubeCon in 2024, kro (a new experimental way to simplify and empower the use of custom APIs and resources with Kubernetes) and how the likes of Microsoft Azure, Google Cloud, and AWS are collaborating together

  • Announcing CDK Garbage Collection - is essential reading for folks (like me) who use AWS CDK and need help in managing those unused CDK Assets in your AWS account - tools like CDK Garbage Collection play a crucial role in maintaining clean, efficient, and cost-effective cloud environments [hands on]

decision flow of assets under gc

  • Announcing the general availability of AWS .NET OpenTelemetry libraries - in case you missed this announcement, the OpenTelemtry dotnet packages have been reworked for simplicity and to conform to the latest OpenTelemetry naming conventions, and include new features like support for enhanced observability in AWS SDK for .NET and additional instrumentation for AWS services

  • A deep dive into Amazon EKS Hybrid Nodes - dives deep into this new capability that was launched at re:Invent, that simplifies how you can deploy your workloads with Kubernetes on-premises or at the edge on Amazon EKS [hands on]

hybrid architecture with eks hybrid nodes

  • Introducing the GraphRAG Toolkit - describe how you can get started with the GraphRAG Toolkit (featured in #206 of the newsletter), a Python toolkit for building GraphRAG applications [hands on]

  • Efficient distributed training with AWS EFA - it is always great to hear from the open source community, and Andrey Cheptsov got in touch about this project, dstack, that provides a lightweight, open-source alternative to Kubernetes & Slurm, simplifying AI container orchestration with multi-cloud & on-prem support that also natively supports NVIDIA, AMD, TPU, and Intel accelerators - read the post to find out how this project leverages Amazon Elastic Fabric Adapter (EFA) to supercharge distributed training tasks [hands on]

Community round up

Each week I spent a lot of time reading posts from across the AWS community on open source topics. In this section I share what personally caught my eye and interest, and I hope that many of you will also find them interesting.

This weeks sees an old friend in the way of FFMpeg. It has been a long time since I have used FFMpeg in anger, but I was reminded about the happy times by Malte Reimann who posted How to install FFmpeg on Amazon Linux, which does exactly what it says.

AWS Community Builder Jonas Neumann is all about caring and sharing, specifically sharing tips he has learned from using AWS CDK in his post, AWS CDK tips and tricks for developers - some good stuff in here. AWS Community Builder Alejandro Velez continues with the infrastructure as code theme in his post, GitOps and IaC at Scale – AWS, ArgoCD, Terragrunt, and OpenTofu – Part 1 where he kicks off the first in a series of posts diving deep into GitOps. The final post in this mini IaC roundup is from AWS Hero Rehan van der Merwe who has put together CDK Constructs for connecting AWS Lambda to Tailscale where he explains how to connect AWS Lambda functions to a Tailscale network. Check out his other blog posts too if you get the time, they are all excellent.

I have been spending a lot of time exploring AI Coding Assistants like Amazon Q Developer to see how they can help open source developers, and Ricardo Tasso came up with a great example in how he is using Q Developer to generate deployment configurations for Kubernetes. He has written this up in his post, Creating Deployment Configurations for EKS with Amazon Q so go check it out (you can get started with Amazon Q Developer for free by registering your Builder ID here).
Langfuse is an open source LLM engineering platform that helps teams collaboratively develop, monitor, evaluate, and debug AI applications. AWS Community Builder Matsuda shows you how you can deploy this, using CDK to make it simple. If this sounds like something you are interested in, go read the post Self-Hosting Langfuse v3 on AWS Using CDK.

If you are a Django developer and are keen to get started with Amazon Aurora DSQL then AWS Community Builder Kevin Kiruri has you covered in his post, AWS Aurora DSQL for Django Developers: A Step-by-Step Guide. Nice!

If you were ever wondering what the performance characteristics of Apache Kafka producers (clients) was between languages, then I have something you might be interesting in reading. AWS Community Builder Mohammed has put together Rust vs Node — Kafka producer performance with AWS Lambda that dives into this topic. No spoilers, you will have to read the post to find out.

Cloud Native

pipeline overview

solution overview

Data and Analytics

overview of solution architecture

fall back and fall forward diagram

hybrid architecture with amazon emr on outposts

overview of Valkey metrics for measuring latency

Other posts to check out

architecture overview of implementing a high performance layer in front of Amazon S3

Quick updates

PostgreSQL

Over the past month we have had a few updates with regards to versions that Amazon Relational Database Service (RDS) for PostgreSQL now supports.

The latest minor versions 17.4, 16.8, 15.12, 14.17, and 13.20. Please note, this release supports the versions released by the PostgreSQL community on February, 20,2025 to address the regression that was part of the February 13, 2025 release. Also announced was support for 17.3, 16.7, 15.11, 14.16, and 13.19. The release also includes updates for PostgreSQL extensions such as pg_active 2.1.4, pg_cron 1.6.5, pg_partman 5.2.4, and others.

We recommend that you upgrade to the latest minor versions to fix known security vulnerabilities in prior versions of PostgreSQL, and to benefit from the bug fixes added by the PostgreSQL community. You can use automatic minor version upgrades to automatically upgrade your databases to more recent minor versions during scheduled maintenance windows. You can also use Amazon RDS Blue/Green deployments for RDS for PostgreSQL using physical replication for your minor version upgrades.

If you are using Amazon CloudWatch Database Insights, it now provides lock contention diagnostics for Aurora PostgreSQL instances. This feature helps you identify the root cause behind both ongoing and historical lock contention issues within minutes. The lock contention diagnostics feature is available exclusively in the Advanced mode of CloudWatch Database Insights. With this launch, you can visualise a locking condition in the Database Insights console, which shows the relationship between blocking and waiting sessions. The visualisation helps you quickly identify the dominating sessions, queries, or objects causing lock contention. Additionally, this feature persists historical locking data for 15 months, allowing you to analyse and investigate historical locking conditions. You no longer need to manually run custom queries or rely on application logs to diagnose lock contention issues, streamlining the troubleshooting process.

MySQL

Amazon Relational Database Service (Amazon RDS) for MySQL now supports MySQL minor versions 8.0.41 and 8.4.4. We recommend that you upgrade to the latest minor versions to fix known security vulnerabilities in prior versions of MySQL, and to benefit from the bug fixes, performance improvements, and new functionality added by the MySQL community. You can leverage automatic minor version upgrades to automatically upgrade your databases to more recent minor versions during scheduled maintenance windows. You can also leverage Amazon RDS Managed Blue/Green deployments for safer, simpler, and faster updates to your MySQL instances.

Amazon RDS for MySQL also announced Amazon RDS Extended Support for minor version 5.7.44-RDS.20250103. We recommend that you upgrade to this version to fix known security vulnerabilities and bugs in prior versions of MySQL. Learn more about the bug fixes and patches in this version in the Amazon RDS User Guide. Amazon RDS Extended Support provides you more time, up to three years, to upgrade to a new major version to help you meet your business requirements. During Extended Support, Amazon RDS will provide critical security and bug fixes for your RDS for MySQL databases after the community ends support for a major version. You can run your MySQL databases on Amazon RDS with Extended Support for up to three years beyond a major version’s end of standard support date.

Apache Kafka

Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports Apache Kafka version 3.8. You can now create new clusters using version 3.8 with either KRAFT or ZooKeeper mode for metadata management or upgrade your existing ZooKeeper based clusters to use version 3.8. Apache Kafka version 3.8 includes several bug fixes and new features that improve performance. Key new features include support for compression level configuration. This allows you to further optimize your performance when using compression types such as lz4, zstd and gzip, by allowing you to change the default compression level. For more details and a complete list of improvements and bug fixes, see the Apache Kafka release notes for version 3.8.

InfluxDb

If you are using Amazon Timestream for InfluxDB then we recently announced the launch of storage scaling functions for Amazon Timestream for InfluxDB, allowing you to scale your allocated storage and change your storage Tiers as needed. With Storage Scaling, in you few simple steps you have greater flexibility and control over your time-series data processing and analysis.

Timestream for InfluxDB is used in applications that require high-performance time-series data processing and analysis. You can quickly respond to changes in data ingestion rates, query volumes, or other workload fluctuations by moving to a faster more performant storage tier or extending your allocated storage capacity, ensuring that your Timestream for InfluxDB instances always have the necessary resources to handle your workload and cost effectively. This means you can focus on building and deploying your applications, rather than worrying about storage sizing and management.

Lustre

Amazon FSx for Lustre, a service that provides high-performance, cost-effective, and scalable file storage for compute workloads, now enables you to upgrade the Lustre version of your FSx for Lustre file systems. This feature allows you to benefit from the enhancements available in newer Lustre versions on your existing file systems. FSx for Lustre provides fully-managed file systems built on Lustre, the world's most popular open-source high performance file system. FSx for Lustre supports multiple long-term support Lustre versions released by the Lustre community. Newer Lustre versions provide benefits such as performance enhancements, new features, and support for the latest Linux kernel versions for your client instances.

Cedar

Amazon Verified Permissions is a permissions management and fine-grained authorisation service for the applications that you build. Using Cedar, an expressive and analysable open-source policy language, developers and admins can define policy-based access controls using roles and attributes for more granular, context-aware access control. For example, an HR application might call Amazon Verified Permissions (AVP) to determine if Alice is permitted to access Bob’s performance evaluation, given that she is in the HR Managers group. Customers can use Cedar JSON format to pass entity data describing the principal (Alice) and the resource (Bob’s performance evaluation).

Amazon Verified Permissions now supports the same JSON format for entity and context data, as the Cedar SDK. Developers can use this simpler format for authorisation requests. This aligns the Amazon Verified Permissions API more closely with the open source Cedar SDK, and simplifies moving from the SDK to Amazon Verified Permissions or vice versa.

Kubernetes

A few important updates this month.

AWS CodePipeline introduced a new action to deploy to Amazon Elastic Kubernetes Service (Amazon EKS). This action enables you to easily deploy your container applications to your EKS clusters, including those in private VPCs. Previously, if you wanted to deploy to a EKS cluster within a private network, you had to initialise and maintain a compute environment within the private network. Now, you can simply provide the name of the EKS cluster and add this action to your pipeline. The pipeline will automatically establish a connection into your private network to deploy your container application, without additional infrastructure needed. This streamlined approach reduces your operational overhead and simplifies your deployment process.

Kubernetes version 1.32 introduced several new features and bug fixes, and AWS is excited to announce that you can now use Amazon Elastic Kubernetes Service (EKS) and Amazon EKS Distro to run Kubernetes version 1.32. Starting today, you can create new EKS clusters using version 1.32 and upgrade existing clusters to version 1.32 using the EKS console, the eksctl command line interface, or through an infrastructure-as-code tool. Kubernetes version 1.32 introduces several improvements including stable support for custom resource field selectors and auto removal of persistent volume claims created by stateful sets. This release removes v1beta3 API version of FlowSchema and PriorityLevelConfiguration. To learn more about the changes in Kubernetes version 1.32, see our documentation and the Kubernetes project release notes.

AWS Secrets Manager also announced that AWS Secrets and Configuration Provider (ASCP) now integrates with Amazon Elastic Kubernetes Service (Amazon EKS) Pod Identity. This integration simplifies IAM authentication for Amazon EKS when retrieving secrets from AWS Secrets Manager or parameters from AWS Systems Manager Parameter Store. With this new capability, you can manage IAM permissions for Kubernetes applications more efficiently and securely, enabling granular access control through role session tags on secrets. ASCP is a plugin for the industry-standard Kubernetes Secrets Store CSI Driver. It enables applications running in Kubernetes pods to retrieve secrets from AWS Secrets Manager easily, without the need for custom code or restarting containers when secrets are rotated. The AWS EKS Pod Identity, streamlines the process of configuring IAM permissions for Kubernetes applications in a more efficient and secure way. This integration combines the strengths of both components, enhancing secret management in Amazon EKS environments. Previously, ASCP relied on IAM Roles for Service Accounts (IRSA) for authentication. Now, you can choose between IRSA and Pod Identity for IAM authentication using the new optional parameter "usePodIdentity". This flexibility allows you to adopt the authentication method that best suits your security requirements and operational needs.

AWS Elastic Beanstalk

AWS Elastic Beanstalk is a service that provides the ability to deploy and manage applications in AWS without worrying about the infrastructure that runs those applications. There were a couple of updates that will be of interest to open source folk.

First up is news that AWS Elastic Beanstalk now enables customers to build and deploy Python 3.13 applications on Amazon Linux 2023 (AL2023) platform. This latest platform support allows developers to leverage the newest features and improvements in Python while taking advantage of the enhanced security and performance of AL2023. Python 3.13 on AL2023 delivers enhanced interactive interpreter capabilities, improved error messages, and important security and API improvements. Developers can create Elastic Beanstalk environments running Python 3.13 on AL2023 through the Elastic Beanstalk Console, CLI, or API.

Following that is news that AWS Elastic Beanstalk now enables customers to build and deploy PHP 8.4 applications on Amazon Linux 2023 (AL2023) platform. This latest platform support allows developers to take advantage of the newest PHP features while leveraging the enhanced security and performance of AL2023. Developers can create Elastic Beanstalk environments running PHP 8.4 on AL2023 through the Elastic Beanstalk Console, CLI, or API.

Videos of the week

Deploy DeepSeek-R1 Models on Kubernetes with Amazon EKS

Those Containers from the Couch folks are back to mischief, this time looking at how you you can deploy DeepSeek-R1 on Kubernetes. In this video, Sai is joined by AWS Solutions Architects Lucas and Tiago to talk about why DeepSeek is different and show you a demo of deploying it on Amazon EKS Auto Mode. We use Amazon EKS Auto Mode for flexibility and scalability on the underlying compute, removing the need for you to manage infrastructure tasks.

Build Software with Amazon Bedrock AI and Aider Open Source

Aider is a super nice open source generative AI pair programming tool that I have been using on/off for a while. It provides a great tool for those of us who prefer text and command line environments. In this video, AWS Ambassador Trevor Sullivan from StratusGrid, takes you through the setup process of a containerized development environment with Aider, using Amazon Bedrock to provide access to Large Language Models (LLM)

Celebrating open source contributors

The articles and projects shared in this newsletter are only possible thanks to the many contributors in open source. I would like to shout out and thank those folks who really do power open source and enable us all to learn and build on top of what they have created.

So thank you to the following open source heroes: Roman Tsypuk, Stephen Jones, Bob Tordella, Danny Steenman, Danilo Poccia, Vincent Claes, Jimmy Dahlqvist, Andrey Cheptsov, Senaka Ariyasinghe, Deependra Shekhawat, Ranjini Menon, Rajesh Matkar, Wanchen Zhao, Wassim Benhallam, Qaisar Dar, Vikrant Telka, Tom McDonald, Kaizen Conroy, Adam Keller, Josh Hart, Nihilson Gnanadason, Deepmala Agarwal, Kashif Khan, Mandisa Nxumalo, Ravi Kumar Singh, Gaurav Sharma, Abhishek Nanda, Jayaprakash Alawala, Rajesh Singh, Rodrigo Bersa, Akshay Aggarwal, Stephanie Shen, Subham Rakshit, Srividya Parthasarathy, Dylan Qu, Kalyan Kumar Neelampudi, Aritra Gupta, Krishna Sarabu, Priyanka Sadhu, Pavankumar Kasani, Ty Augustine, Alexis Tekin, Jeremy Ber, Jagadeesh Chitikesi, Philip Pittle, Ulili Nhaga, Vivien de Saint Pern, Camille Hoarau, Damien Cupif, Vincent Mercier, Anil Maktala, Chris Splinter, Elamaran Shanmugam, Re Alvarez Parmar, Dylan Tong, Vamshi Vijay Nakkirtha, Yasha Jayaprakash, Mihir Mangalvedhekar, Rajbir Singh, Ian Robinson, Abdellah Ghassel, Jesse Butler, Shoukat Ghouse, Fernando Galves, Anand Komandooru, Anubhava Srivastava, Noorul Hasan, Li Liu, Subhash Dike, Mohammed, Kevin Kiruri, Matsuda, Rehan van der Merwe, Jonas Neumann, Malte Reimann, and Alejandro Velez.

Feedback

Please please please take 1 minute to complete this short survey.

Stay in touch with open source at AWS

Remember to check out the Open Source homepage for more open source goodness.

One of the pieces of feedback I received in 2023 was to create a repo where all the projects featured in this newsletter are listed. Where I can hear you all ask? Well as you ask so nicely, you can meander over to newsletter-oss-projects.

Made with ♥ from DevRel

Top comments (0)