Co-authored by Dustin Whited and Dakota Riley
Kubernetes adoption has grown, with managed services leading implementation according to a recent Cloud Native Computing Foundation (CNCF) survey. "79% of respondents use Certified Kubernetes Hosted platforms" with Amazon’s managed Kubernetes offering, Elastic Kubernetes Service (EKS), being the most popular of the hosted platforms.
The shared responsibility model means the customer is responsible for securing the workloads running on EKS. In January, 2022, AWS released new functionality for GuardDuty enabling threat detection on EKS clusters and aiding customers with securing their workloads. This new capability uses the Kubernetes Audit log to monitor for suspicious or malicious activity.
Falco, a CNCF project, is a common open source tool used to perform similar threat detection capabilities within Kubernetes clusters. Falco monitors system calls from the Linux kernel for the majority of its analysis. It is also preloaded with community maintained rule sets.
This blog will explore some commonalities and differences between Falco and the new GuardDuty feature.
Ease of implementation
GuardDuty Kubernetes protection is not enabled by default. Accounts that are in an AWS Organization can enable the feature through the GuardDuty delegated administrator account, or the detector can be updated on an individual account. Terraform support is pending at the time of writing, while Cloudformation and CDK support is fully implemented.
It is also worth noting that enabling EKS GuardDuty incurs additional pricing per one million audit events ingested. The cost is reasonable, but environments with many clusters or a high amount of activity may incur a surprise billing statement after the 30 day trial ends.
Falco can be installed with a chart as a daemonset in the cluster, or directly installed on the host operating system. Installing as a daemonset is the simplest way to get started with Falco, however installing on the OS insulates it from Kubernetes itself. Organizations that create hardened custom node AMIs may find it easiest to include Falco through the image build process.
Finding types
At a high level, EKS GuardDuty is ultimately more focused on detecting malicious activity against the EKS Control Plane, whereas by default Falco is better suited to detecting malicious activity on a container running on the cluster, or the host itself.
While there is some overlap in rules, GuardDuty and Falco detections are not congruent. At the time of writing, GuardDuty has 27 Kubernetes findings that leverage the Kubernetes Audit log. Importantly, regular GuardDuty findings for EC2 will also apply to the Kubernetes nodes. With over 68 community rules, Falco monitors syscalls from the Linux kernel by default and optionally can consume the Kubernetes Audit log - this requires configuring additional logic with a CloudWatch subscription filter on the EKS Audit Logs CloudWatch to feed it back into Falco.
Below we compare and contrast some notable detection capabilities of the two tools.
Anonymous Access:
EKS can be configured to allow anonymous access to its control plane (relevant GitHub Issue on AWS Containers Roadmap here). This configuration should almost never exist. This is an excellent "low-hanging fruit" detection provided by EKS GuardDuty upon that access being configured:
Policy:Kubernetes/AnonymousAccessGranted
It is also worth noting that EKS GuardDuty also includes findings for actual usage of that anonymous access - which can help inform your response, the findings themselves are categorized based on the specific APIs (which are not publicized by AWS) being called anonymously:
CredentialAccess:Kubernetes/SuccessfulAnonymousAccess
DefenseEvasion:Kubernetes/SuccessfulAnonymousAccess
Discovery:Kubernetes/SuccessfulAnonymousAccess
Impact:Kubernetes/SuccessfulAnonymousAccess
Persistence:Kubernetes/SuccessfulAnonymousAccess
Falco also has an anonymous access rule in its K8s Audit Logs ruleset, but requires the additional logic to feed into the engine:
Syscalls and Malicious Runtime Activity:
Where Falco shines is its ability to ingest syscalls from containers on the cluster to detect malicious activity on the container itself. EKS GuardDuty currently only alerts on activities against the K8s Control Plane of the cluster. Highlighting a few default Falco rules below:
Contact EC2 Instance Metadata Service From Container - Specific to AWS - this finding indicates when a container makes a call to the IMDS endpoint, which would by default provide access to credentials for the EKS Node Instance Role. In most cases, this would be an indicator of lateral movement/recon attempts on a compromised container (see also: IAM Roles for Service Accounts and Restrict Access to the Node Instance Profile).
Contact K8s API Server From Container - This finding can be an indicator of lateral movement/recon attempts on a compromised container. Allowlisting legitimate access by management tools may be required in order to be a high fidelity alert.
Known Malicious IP:
GuardDuty supports Malicious IP Caller detection which will alert on malicious IP addresses invoking Kubernetes API calls. Similar to the Anonymous Access findings, EKS GuardDuty has a number of Malicious IP Caller finding types that will trigger based on the Kubernetes API being called. For example, Discovery:Kubernetes/MaliciousIPCaller and Impact:Kubernetes/MaliciousIPCaller both trigger on malicious IPs, but are dependent on the specific API being called.
Falco requires a static deny list of malicious IPs or domain names, which is not optimal to maintain up to date threat intelligence.
Cryptomining:
While not part of EKS GuardDuty specifically, the EC2 finding CryptoCurrency:EC2/BitcoinTool.B!DNS will alert on resources within the cluster contacting Crypto Mining domains. GuardDuty is automatically updated with the latest threat intel feeds from Crowdstrike and ProofPoint so it will always have up to date information.
Falco has a rule to detect crypto miner domains but it is not enabled because Falco will invoke a DNS lookup to determine if it is in the list, which may trigger alerts external to Falco. This list is also not a true threat intel feed. One advantage to using Falco for this use case is that it will provide helpful metadata about the container in question making the calls to cryptoming domains, where GuardDuty will only provide the EC2 Instance and associated IP addresses and require additional enrichment to triage further.
Customizability
Customization can be a game changer for threat detection tooling, especially when it comes to fine tuning the fidelity of the findings. On the other hand, not all teams have the bandwidth to tune findings or implement custom detections. A look below at the customization ability of both tools:
GuardDuty findings cannot be customized. Some post-processing of the finding is possible with external integrations like Lambda – for example to reduce the severity level of the finding. While findings are not able to be customized, GuardDuty supports importing both custom threat lists or trusted IP lists, as well as native suppression rule capabilities.
Existing Falco rules can be disabled or modified. New supplemental rules can be written to detect environment specific use cases. Falco is not just limited to Kubernetes syscalls or audit logs - recently Falco released a plugin capability, allowing Falco to be extended to consume additional event sources. One such example of this is the AWS CloudTrail Plugin.
Integrations
Alerts need to have an action or notification method to bring value to a security organization. Integrating GuardDuty and Falco with external systems to notify a responder or take action is achieved in a couple different ways.
GuardDuty findings are sent to AWS services like SecurityHub and the default Eventbridge eventbus within the administrator account. Both allow consumption into a SIEM of choice, direct alerting with Chatbot, or even taking action with a Step Function workflow. For an example of auto-remediation using a Step Function workflow see our previous blog on this topic.
Falco can emit its findings to stdout, a file, syslog, or custom endpoints with "Program Output" using bash. Alternatively, Falcosidekick is an extra app that receives Falco alerts from multiple clusters and forwards them to a variety of outputs concurrently. Falcosidekick has a large number of custom destinations, so it may reduce the need to write custom integrations for existing destinations like a SIEM.
Overall, both GuardDuty and Falco will require some level of investment into integrating the alerts with existing systems. The only exception is if SecurityHub is the primary location as GuardDuty sends findings there by default.
Wrap up
GuardDuty EKS protection offers very quick centralized enablement and is managed by AWS for updates to findings. Enabling this functionality is a new best practice and should be enabled organization-wide as it only costs money when an EKS cluster is present and has activity.
Falco offers an increased level of customization and granularity that GuardDuty does not. Falco should be enabled in organizations with the maturity to tune and monitor the alerts, as well as create defined runbooks for responders.
In the end, it is not usually an either-or situation for GuardDuty EKS and Falco, but rather each is a milestone on a maturity roadmap. The two tools complement each other to supplement a defense in depth strategy securing container workloads on AWS.
The information presented was current as of date of publication 05/06/2022
Top comments (0)