HTTP/2’s multiplexing and binary framing make it significantly more efficient than HTTP/1.1, reducing latency and improving throughput. Connect RPC lets you harness these benefits while maintaining broad client compatibility for services that can’t yet support HTTP/2.

Connect RPC can be used to build both internal and external APIs, powering frontends, mobile apps, CLIs, agents and more. See the list of supported languages.

Core Features

Connect RPC provides a number of features out of the box, such as:

• Interceptors which make it easy to extend Connect RPC and are used to add authentication, logging, metrics, tracing and retries.
• Serialization & compression, with pluggable serializers, and support for asymmetric compression reducing the amount of data that needs to be transmitted, or received.
• Error handling, with a standard error format, with support for custom error codes to allow for more granular error handling.
• Observability, with in built support for OpenTelemetry enabling you to easily add tracing, or metrics to your APIs.
• Streaming, which provides a very efficient way to push or pull data without polling.
• Schemas, which enable you to define and validate your API schemas, and generate code from them.
• Code generation for Go, TypeScript, Kotlin, Swift and Java.

Ecosystem

In addition to these features, Connect RPC is built on top of the Buf ecosystem, which offers notable benefits:

• Connect RPC joins CNCF, entering the cloud-native ecosystem, which is great for the long term sustainability of the project.
• Buf Schema Registry, which is a great tool for managing, sharing and versioning your API schemas.
• Buf CLI, a handy all in one tool for managing your APIs, generating code and linting.

Recommended Interceptor Packages

Some handy Go packages that provide pre-built Connect RPC interceptors worth exploring or using as a starting point:

• authn-go, provides a rebuilt authentication middleware library for Go. It works with any authentication scheme (including HTTP basic authentication, cookies, bearer tokens, and mutual TLS).
• validate-go provides a Connect RPC interceptor that takes the tedium out of data validation. This package is powered by protovalidate and the Common Expression Language.
• rpclog provides a structured logging interceptor for Connect RPC with support for both unary and streaming RPCs.

Summary

1. Connect RPC provides a paved and well maintained path to building gRPC compatible APIs, while maintaining compatibility for HTTP/1.1 clients. This is invaluable for product teams that need to support multiple client types without building custom compatibility layers.

2. Using a mature library like Connect RPC, you get to benefit from all the prebuilt integrations, and the added capabilities of the Buf ecosystem. This makes publishing and consuming APIs a breeze.

3. Protobuf schemas, high performance serialisation and compression ensure you get robust and efficient APIs.

Conclusion

Connect RPC makes it easy to build high-performance, robust APIs with gRPC compatibility, while avoiding the complexity of building and maintaining custom compatibility layers.

In this model the CI provider acts as an identity provider, issuing tokens to the CI runner/agent which include a set of claims identifying the owner, pipeline, workflow and job that is being executed. This is then used to authenticate with the cloud service, and access the resources that the pipeline, workflow and job require.

In simple terms, this is a form of trust delegation, where the CI provider is trusted by the cloud service to issue tokens on behalf of the owner, pipeline, workflow and job.

How OIDC Works

The OIDC trust delegation flow is as follows:

sequenceDiagram
    participant CI as CI Provider<br/>(Identity Provider)
    participant Runner as CI Runner/Agent
    participant Cloud as Cloud Service<br/>(AWS/Azure/GCP)

    Note over CI,Cloud: OIDC Trust Delegation Flow

    CI->>Runner: Issue OIDC token with claims<br/>(pipeline, workflow, job)
    Runner->>Cloud: Request access with OIDC token
    Cloud->>Cloud: Verify token signature<br/>and validate claims
    Cloud->>Runner: Grant temporary credentials
    Runner->>Cloud: Access resources with credentials

    Note over CI,Cloud: Trust established via OIDC configuration

There are a few things to note:

• When using OIDC, the runner doesn’t need to be registered with the cloud service; it is granted access via the OIDC token.
• The OIDC token is cryptographically signed by the CI provider, and the cloud service verifies the signature to ensure the token is valid.
• In this model all three parties (CI provider, runner, and cloud service) are trusted to issue and verify tokens.

Limiting Cloud Access to the Agent/Runner

To ensure the CI provider can’t access the cloud service directly, you can add conditions which ensure only the runner/agent is allowed to access the cloud resources.

On top of this cloud providers such as AWS have conditions which can restrict access to a specific AWS network resource, such as a VPC. I recommend familiarizing yourself with the documentation for your cloud provider to understand how to lock down access to the runner/agent.

Benefits of OIDC

The reasons this is useful are:

• It provides a more secure and flexible approach to authentication and authorization
• It limits the scope of the token to the specific pipeline, workflow, and job
• It is tied to the lifecycle of the pipeline, workflow, and job, which means the token is limited to the duration of that execution
• It is more flexible than using machine identity for CI runners/agents as it allows for more granular control over the permissions granted to the runner/agent

Ephemeral Runners/Agents

Ephemeral runners/agents are short lived, single job, or single workflow runners/agents which are created before the workflow, or job is started. They provide a more secure and flexible approach to job execution as there is no need to worry about these environments being tainted by previous jobs or workflows.

When paired with OIDC these environments provide an extra layer of security as they are destroyed after the job or workflow is complete, further reducing the risk of cross job or workflow access.

Summary

So in summary, OIDC provides a more secure and flexible approach to access management for CI projects, and it is particularly useful when paired with ephemeral runners/agents.

The biggest advantage of this approach is that it allows engineers to focus on the access required by the pipeline, workflow, and job, rather than having to manage machine identities and permissions for each runner/agent.

One of the interesting things about this approach is that you’re not limited to using OIDC just with cloud providers; you can use it with your own services as well. By using OIDC libraries such as github.com/coreos/go-oidc, you can implement APIs which can use the identity of CI pipelines, workflows, and jobs. An example of this is Using OIDC With HashiCorp Vault and GitHub Actions.

Links

• OIDC for Buildkite
• OIDC for GitHub Actions
• OIDC for GitLab

The aim of this exercise is to help develop some intuition of how AI works, and how it can be used to help in your day-to-day tasks, while hopefully discovering ways to use it in future applications you build.

Getting Started

As the common saying that originated from a Chinese proverb says.

A journey of a thousand miles begins with a single step.

To kick off your understanding of AI I recommend you select a coding assistant and start using it on your personal, or side projects, this will provide you with a better understanding of how it succeeds, and sometimes fails. Building this knowledge up will help you develop an understanding of strengths and weaknesses as a user.

I personally recommend getting started with Cody as it is a great tool, and is free for personal use, while also being open source itself. The developers of Cody are very open and helpful, and have a great community of users, while also sharing their own experiences while building the tool.

Cody is more than just a code completion tool, you can ask it questions and get it to summarise and document your code, and even generate test cases. Make sure you explore all the options, again to build up more knowledge of how these AI tools work.

And most importantly be curios, and explore every corner of the tool.

Diving Into LLMs

Next, I recommend you start experimenting with some of the open source large language models (LLMs) using tools such as ollama to allow you to download, run and experiment with the software. To get started with this tool, you can follow the quick start in the README.md hosted at https://github.com/jmorganca/ollama. Also there is a great intro by Sam Witteveen Ollama - Local Models on your machine which I highly recommend.

What is a large language model?

Here is a quote from wikipedia on what a large language model is:

A large language model (LLM) is a large scale language model notable for its ability to achieve general-purpose language understanding and generation. LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly transformers and are (pre)trained using self-supervised learning and semi-supervised learning.

Why Open LLMs?

I prefer to learn from the open LLMs for the following reasons:

1. They have a great community of developers and users, who share information about the latest developments.
2. You get a broader range of models, and can try them out and see what they do.
3. You can run them locally with your data, and see what they do without some of the privacy concerns of cloud based services.
4. You have the potential to fine tune them to your data, and improve the performance.

I keep up with the latest developments I use hugging face open llm leader board, as they have been doing a lot of work on large language models, and have a great community of users. When the latest models are posted they are also sharing their experiences, and fine tuned versions via their blog, which is great resource. Notable models are normally added to ollama after a day or so, so you can try them out and see what they do.

There are a number of different types of LLMs, each with their own strengths and weaknesses. I personally like to experiment with the chat bot models, as they are very simple to use, and are easy to interface with via olloma. An example of one of these from the hugging face site is https://huggingface.co/mistralai/Mistral-7B-v0.1 which is a chat bot model trained by the Mistral AI team.

To get started with this model you can follow the instructions at https://ollama.ai/library/mistral, download and run the model locally.

Pick a Scenario To Test

My scenario relates to my current role, and covers questions which my team encounters on a day-to-day basis. As a team we are providing advice to a customers about how to improve the operational readiness and security posture for internally developed applications. This is a common scenario for many companies, where applications are developed to provide a proof of concept, and are then deployed to a production environment without the supporting processes in place.

What is approach is helpful as:

1. This is a scenario I can relate to, and can use my existing knowledge to review the results.
2. This is a scenario which is not too complex, and can be used to demonstrate the concepts.
3. This is a scenario which will provide me value while I am learning how to use the tools.

Building a list of questions

Once you have a scenario, you can draft a list of questions which you can start testing them with models, this will help you understand how the models work, and how they can be to support a team or business unit, while also learning how to use them.

The questions I am currently using mainly focus on DevOps, and SRE processes, paired with a dash of AWS security and terraform questions.

I need to create a secure environment in and AWS Account, where should I start?

This question is really common for developers starting out in AWS, it is quite broad and I am mostly expecting a high level overview of how to create a secure environment, and how to get started.

How would I create an encrypted secure s3 bucket using terraform?

This question is a bit more specific, focusing on a single AWS service, while also adding a few specific requirements. Models like Mistral will provide a step by step guide on how to achieve this, while others will provide the terraform code to achieve this.

I need to create an Application Risk Management Program, where should I start?

This question is quite common if your working in a company which doesn’t have a long history with internal software development, or a team that is trying to ensure they cover the risks of their applications.

What is a good SRE incident process for a business application?

This question is also quite broad, but includes Site Reliability Engineering (SRE) as a keyword, so I am expecting an answer which aligns with the principals of this movement.

What is a good checklist for a serverless developer who wants to improve the monitoring of their applications?

This is a common question asked by people who are just getting started with serverless and are interested in, or have been asked to improve the monitoring of their applications.

Whats Next?

So now that you have a scenario and a few questions I recommend you do the following:

1. Try a couple of other models, probably llama2 and orca are a good starting point.
2. Learn a bit about prompting by following A guide to prompting Llama 2 from the replicate blog.
3. Apply the prompts to your ollama model using a modelfile, which is similar to a Dockerfile.
4. Try out an uncensored model, something like llama2-uncensored and run through your questions, then ask about breaking into cars or killing processes, which can be a problematic question in some censored models. It is good to understand what censoring a model does, as it can be a useful tool for understanding the risks of using a model.
5. Start reading more about The State of Open Source AI (2023 Edition).

Further Research

Now that you are dabbling with LLMs, and AI, I recommend you try these models for the odd question in your day-to-day work, the local ones running in ollama are restively safe, and they can save you a lot of work.

Also try similar questions with services such as https://chat.openai.com/, hosted services are a powerful tool for adhoc testing and learning. Just be aware of data privacy, and security when using these services.

Once you have some experience you will hopefully even incorporate a model into work projects such as data cleansing, summarizations, or processing of user feedback to help you improve your applications. For this you can use services such as AWS Bedrock on AWS, or Generative AI Studio on Google cloud, while following the same methodology to evaluate and select a model for your use case.

If your intrigued and want to go even deeper than these APIs, I recommend you dive into some of the amazing resources on the web for learning how AI and LLMs work, and possibly even develop, or fine tune your own own models.

• fast.ia which provides some great online self paced learning on AI.
• A busy persons intro to LLMs great lecture on LLMs.
• MIT Introduction to Deep Learning for those who want to dive deeper and prefer more of a structured course.
• A Hackers’ Guide to Language Models, another great talk by Jeremy Howard of fast.ai.

If your using AWS_IAM authentication on an API Gateway, then make sure you set the default authorizer for all API resources. This will avoid accidental exposing an API if you mis-configure, or omit an authentication method for an API resource as the default is None.

In addition to this there is a way to apply a resource policy to an API Gateway, which will enforce a specific iam access check on all API requests. Combining the override to default authorizer, and the resource policy allows us to apply multiply layers of protection to our API, allowing us to follow the principle of defense in depth.

So to summarise, to protect your API with IAM authentication is as follows:

1. Enable a default authorizer method on the API Gateway resource.
2. Enable an authentication method on the API.
3. Assign an API resource policy which requires IAM authentication to access the API.

Doing this with AWS SAM is fairly straight forward, to read more about it see the SAM ApiAuth documentation.

  AthenaWorkflowApi:
    Type: AWS::Serverless::Api
    Properties:
      ...
      Auth:
        # Specify a default authorizer for the API Gateway API to protect against missing configuration
        DefaultAuthorizer: AWS_IAM
        # Configure Resource Policy for all methods and paths on an API as an extra layer of protection
        ResourcePolicy:
          # The AWS accounts to allow
          AwsAccountWhitelist:
           - !Ref AWS::AccountId

Through the magic of AWS SAM this results in a resource policy which looks like the following, this results in all the API methods being protected and only accessible by users authenticated to this account, and only where they are granted access via an IAM policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "execute-api:Invoke",
      "Resource": "arn:aws:execute-api:us-west-2:123456789012:abc123abc1/Prod/POST/athena/run_s3_query_template"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "execute-api:Invoke",
      "Resource": "arn:aws:execute-api:us-west-2:123456789012:abc123abc1/Prod/POST/athena/run_query_template"
    }
  ]
}

I typically use an openapi spec to define the API, using the extensions provided by AWS such as x-amazon-apigateway-auth to define the authorisation.

With the default authentication set to AWS_IAM hitting an API which is missing x-amazon-apigateway-auth using curl returns the following error.

{"message":"Missing Authentication Token"}

With default authentication disabled, and the resource policy enabled the API returns the following error, which illustrates the principle of defense in depth.

{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:********9012:abc123abc1/Prod/POST/athena/run_query_template"}

What is the problem with IAM User Credentials?

• IAM User Credentials are long lived, meaning once compromised they allow access for a long time
• They are static, so if leaked it is difficult to revoke access immediately

But there are better alternatives, the one I recommend is OpenID Connect (OIDC), which if you dig deep into the Terraform Cloud docs is a supported approach. This has a few benefits:

1. Credentials are dynamically created for each run, so if one set is compromised it does not affect other runs.
2. When Terraform Cloud authenticates with AWS using OIDC it will pass information about the project and run, so you can enforce IAM policies based on this context.
3. Credentials are short lived, expiring after the Terraform run completes.
4. You can immediately revoke access by removing the OIDC provider from AWS.
5. You don’t need to export credentials from AWS and manage their rotation.

Overall this allows for a more secure and scalable approach to integrating Terraform Cloud with AWS. If you are just starting out, I would recommend setting up OpenID Connect integration instead of using IAM credentials.

AWS Deployment

To setup the resources on the AWS side required to link AWS to Terraform Cloud we need to deploy some resources, in my case I am using a Cloudformation Template which deploy manually. You can find the source code to this template in my GitHub Repo along with a Terraform example to deploy the resources.

Using the Cloudformation template as the example for this post, it creates:

1. IAM Role, which assumed by Terraform Cloud when deploying
2. Open ID Connect Provider, which is used to connect Terraform Cloud to AWS

The Terraform Deployment role is as follows:

  TerraformDeploymentRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Ref TerraformOIDCProvider
            Condition:
              StringEquals:
                app.terraform.io:aud: "aws.workload.identity"
              StringLike:
                app.terraform.io:sub: !Sub organization:${OrganizationName}:project:${ProjectName}:workspace:${WorkspaceName}:run_phase:*

Note:

• The IAM role allows Terraform Cloud to assume the role using the OIDC provider, and limits it to the given organization, project and workspace names.
• The policy attached to this role, in my example, only allows Terraform to list s3 buckets; you should customise this based on your needs.

The Open ID Connect Provider is created as follows:

  TerraformOIDCProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://app.terraform.io
      ClientIdList:
        - aws.workload.identity
      ThumbprintList:
        - 9e99a48a9960b14926bb7f3b02e22da2b0ab7280

Once deployed this template will provide two outputs:

1. The role ARN for the Terraform Deployment role.
2. An Optional Audience value, this is only needed if you want to customise this value.

Terraform Cloud Configuration

You’ll need to set a couple of environment variables in your Terraform Cloud workspace in order to authenticate with AWS using OIDC. You can set these as workspace variables, or if you’d like to share one AWS role across multiple workspaces, you can use a variable set.

Note for more advanced configuration options please refer to Terraform Cloud - Dynamic Credentials with the AWS Provider.

That is it, your now ready to run plans in your Terraform Cloud workspace!

Auditing

Once you have setup both side of this solution you should be able to see events in AWS CloudTrail, filter by service sts.amazonaws.com and look at the AssumeRoleWithWebIdentity events. Each event will contain a record of the Terraform Cloud run, and the name of the project and workspace.

This is a cut down cloudtrail event showing the key information:

{
    "userIdentity": {
        "type": "WebIdentityUser",
        "principalId": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io:aws.workload.identity:organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "userName": "organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "identityProvider": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io"
    },
    "eventTime": "2023-07-18T00:08:34Z",
    "eventSource": "sts.amazonaws.com",
    "eventName": "AssumeRoleWithWebIdentity",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "x.x.x.x",
    "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.7.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.18.1 os/linux lang/go/1.20.5 md/GOOS/linux md/GOARCH/amd64 api/sts/1.19.2",
    "requestParameters": {
        "roleArn": "arn:aws:iam::12121212121212:role/terraform-cloud-oidc-acces-TerraformDeploymentRole-NOPE",
        "roleSessionName": "terraform-run-abc123"
    },
    "responseElements": {
        "subjectFromWebIdentityToken": "organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "assumedRoleUser": {
            "assumedRoleId": "CDE456:terraform-run-abc123",
            "arn": "arn:aws:sts::12121212121212:assumed-role/terraform-cloud-oidc-acces-TerraformDeploymentRole-NOPE/terraform-run-abc123"
        },
        "provider": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io",
        "audience": "aws.workload.identity"
    },
    "readOnly": true,
    "eventType": "AwsApiCall",
    "recipientAccountId": "12121212121212"
}

Links

• How to get rid of AWS access keys - Part 1: The easy wins
• Terraform Cloud - Dynamic Credentials with the AWS Provider
• AWS Partner Network (APN) Blog - Simplify and Secure Terraform Workflows on AWS with Dynamic Provider Credentials

So instead of using IAM User credentials, this approach uses IAM Roles and OpenID Connect to dynamically assign credentials to Terraform Cloud runs which is a big win from a security perspective!

So firstly what does automated security remediation for a cloud service do?

This is software which detects threats, more specifically misconfigurations of services, and automatically remediates problems.

How does automated security remediation work?

Typically, security remediation tools take a feed of events from a service such as AWS CloudTrail (audit logging service) and checks the configuration of the resources being modified. This is typically paired with regular scheduled scans to ensure nothing is missed in the case of dropped or missing events.

Can you use IAM to avoid security misconfigurations in the first place?

Cloud services, such as AWS, have fairly complex AWS Identity and Access Management (IAM) services which provide course grained security policy language called IAM policies. These policies are hard to fine tune for the myriad of security misconfigurations deployed by the people working on in these cloud services.

Everyone has seen something like the following administrator policy allowing all permissions for administrators of an AWS environments, this is fine for a “sandbox” learning account, but is far too permissive for production accounts.

Version: "2012-10-17"
Statement:
    - Sid: AdminAccess
    Effect: Allow
    Action: '*'
    Resource: '*'

That said, authoring IAM policies following the least privilege to cover current requirements, new services coming online and keeping up with emerging threats can be a significant cost in time and resources, and like at some point provide diminishing returns.

Can you use AWS service control polices (SCP) to avoid security misconfigurations?

In AWS there is another way to deny specific operations, this comes in the for of service control policies (SCP). These policies are a part of AWS Organizations and provide another layer of control above an account’s IAM policies, allowing administrators to target specific operations and protect common resources. Again, these are also very complex to configure and maintain, as they use the same course grained security layer.

Below is an example SCP which prevents any VPC that doesn’t already have internet access from getting it.

Version: '2012-10-17'
Statement:
- Effect: Deny
  Action:
  - ec2:AttachInternetGateway
  - ec2:CreateInternetGateway
  - ec2:CreateEgressOnlyInternetGateway
  - ec2:CreateVpcPeeringConnection
  - ec2:AcceptVpcPeeringConnection
  - globalaccelerator:Create*
  - globalaccelerator:Update*
  Resource: "*"

Investment in SCPs is important for higher level controls, such as disabling the modification of security services such as Amazon GuardDuty, AWS Config and AWS CloudTrail as changes to these services may result in data loss. That said, SCPs are still dependent on IAMs course grained policy language, which in turn is limited by the service’s integration with IAM.

A note about SCPs, often you will see exclusions for roles which enable administrators, or continuous integration and delivery (CI\CD) systems to bypass these policies. These should be used for exceptional situations, for example bootstrapping of services, or incidents. So, using these roles should be gated via some sort incident response process.

So why does automated security remediation exist?

Given the complexity of managing fine-grained security policies, organizations implement a more reactive solution, which is often in the form of automated security remediation services.

What are some of disadvantages of these automated security remediation tools?

• False positives and false negatives: They may generate false positives, where legitimate actions are flagged as security threats, or false negatives, where actual security issues are missed.
• Over-reliance on automation: Organizations may become over-reliant on tools, potentially leading to complacency or a lack of human oversight, which can create new risks and vulnerabilities.
• Limited scope: They may not be able to detect or remediate all types of security issues or vulnerabilities, especially those that are highly complex or require a more nuanced approach.
• Compliance and regulatory issues: Some compliance and regulatory frameworks may require manual security review or approval for certain types of security incidents, which can be challenging to reconcile with automated processes.
• Cultural resistance: Some organizations may experience cultural resistance to automated remediation, as it may be perceived as a threat to job security or the role of security professionals.
• Delayed or dropped trigger events: Automated remediation typically primarily depend on triggers from audit events provided, these events can be delayed in large AWS environments, or by a flood of activity.

What are some of the positive impacts automated remediation tools?

• Increased efficiency: Can reduce the time and resources required to respond to security incidents, allowing security teams to focus on higher-value tasks.
• Improved collaboration: Can help break down silos between different teams, as it often requires cross-functional collaboration between security, operations, and development teams.
• Reduced burnout: By automating repetitive and time-consuming tasks, automated remediation can help reduce burnout among security people, who may otherwise be overwhelmed by the volume of security incidents they need to respond to manually.
• Skills development: As organizations adopt these tools and processes, security teams may need to develop new skills and competencies in areas such as automation, scripting, and orchestration, which can have positive impacts on employee development and job satisfaction.
• Cultural shift towards proactive security: They can help shift the culture of security within an organization from reactive to proactive, by enabling security teams to identify and remediate potential security risks before they become actual security incidents.

Summary

Overall, while automated security remediation can have some cultural and productivity impacts that need to be managed, it can also bring significant benefits to organizations by enabling more efficient, collaborative, and proactive security practices.

That said, automated security remediation really needs to be part of a three-pronged approach:

1. Ensure people are working in cloud environments with only the privileges they require to do their work. There are of course exceptions to this, but they should be covered with a process which allows users to request more access when required.
2. SCPs should be used to protect security and governance services, and implement core restrictions within a collection of AWS accounts, depending on your business.
3. Automated security remediation should be used to cover all the edge cases, again this should be used only where necessary, and with the understanding it may take a period of time to fix.

One thing to note is we are working in an environment with a lot of smart and resourceful people, so organizations need to watch for situations complex workarounds evolve to mitigate ineffective or complex controls otherwise they may impact morale, onboarding of staff and overall success of a business.

Security works best when it balances threats and usability!

A supply chain attack targets less secure parts of the development process, this could be the tools and services you depend on, or the docker containers you host your software in. These attacks come in different forms but some examples are:

• Extract credentials from your CI services like the Codecov security incident.
• Seed malware for an attack down stream on your customers like the solarwinds breach.

In this post I am going to dive into an example of an attack that affected a lot of projects using GitHub Actions recently, but this could be applied more broadly to any CI tool or service relying on third party services or code.

Why is the Codecov security incident interesting?

The Codecov security incident illustrates a novel attack on a popular developer tool, which in turn exposed a number of CI integrations including the widely used GitHub Actions.

The initial attack happened in January when the Codecov Bash uploader script was modified in a cloud storage service.

This script provides a language-agnostic alternative for sending your coverage reports to Codecov and is used at least 5 of the Codecov continuos integration (CI) integrations.

The GitHub Action Codecov was one of them, it downloads and executes the script each time it is run and critically didn’t verify the checksum of this file against the release so it continued working.

The modified script extracted all environment variables in that workflow and uploaded them to a website operated by the attacker.

These variables are often used to pass credentials into the workflow for services such as Docker Hub, NPM, cloud storage buckets and other software distribution services.

The extraction of these credentials while this exploit was active could lead to the modifications of builds and other artifacts resulting in further exploits and extending the footprint of this attack.

Most concerning is this exploit was effectively sitting in the supply chain of 1000s of open source and proprietary workflows extracting data undetected for approximately 4 months.

Note: It is worth reading the security update posted by Codecov as it highlights some of the steps you need to take if you are effected by this sort of attack.

What can you do to mitigate these sorts of attacks?

To ensure your GitHub actions secure I recommend:

• Read the GitHub actions hardening documentation.
• Limit exposure of secrets to only the projects and repositories which need these values by implementing least privilege for secrets in GitHub Actions.
• Read about Keeping your GitHub Actions and workflows secure: Untrusted input.
• Regularly rotate the credentials used in your GitHub actions. this helps mitigate historical backups or logs being leaked by a service.
• If an action is supplied and supported by a vendor, ensure emails or advisories are sent to a shared email box, and not attached to a personal email. This will enable monitoring by more than one person, and enable you to go on holidays.

For actions which have access to important secrets, like those used to upload your software libraries and releases, or deploying your services, you may want to fork them and add security scanning. This is more important if there are no vendor supported alternatives, or it is a less widely supported technology.

Reviewing your actions

Given we all still want the benefits of services such as GitHub Actions while also managing the risks we need to maintain a balance between getting the most out of the service and limiting possible exploits.

The first step is to review the GitHub actions your using in your workflows, just like you would for open source libraries:

• How active are these projects? Are PRs merged / reviewed in a timely manner?
• Is the author known for building good quality software and build automation?
• Are these actions supported by a company or service?
• Does the project have a security policy?

When it comes to open source GitHub Actions you need to be aware that most open source licenses limit liability for the author, this means you as a consumer need to actively manage some of the risks running this software. Performing maintenance, bug fixes and contributing to the upkeep of the software is key to ensuring the risk of exploits is minimized.

Lastly run some internal training or workshops around supply chain attacks in your company, this could involve running a scenario like the Codecov incident as a table top exercise.

Case in point, the AWS CLI which a large number of engineers and developers rely on every day, the following command will create a bucket.

$ aws s3 mb s3://my-important-data

One would assume this commonly referenced example which is used in a lot of the resources provided by AWS would create a bucket following the best practices. But alas no…

The configuration which is considered best practice for security of an S3 bucket missing is:

• Enable Default Encryption
• Block Public access configuration
• Enforce encryption of data in transit (HTTPS)

Why is this a Problem?

I personally have a lot of experience teaching developers how to get started in AWS, and time and time again it is lax defaults which let this cohort down. Of course this happens a lot while they are just getting started.

Sure there are guard rails implemented using services such as AWS Security Hub, pointing out issues left right and center, but these typically identity problems which wouldn’t be there in the first place if the tools where providing better defaults.

Sure there is more advanced configuration but encryption and blocking public access by default seem like a good start, and would reduce the noise of these tools.

The key point here is it should be hard for new developers to avoid these recommended, and recognised best practices when creating an S3 bucket.

In addition to this, keeping up with the ever growing list of “best practice” configuration is really impacting both velocity and morale of both seasoned, and those new the platform. Providing some tools which help developers keep up, and provide some uplift when upgrading existing infrastructure would be a boon.

Now this is especially the case for developers building solutions using serverless as they tend to use more of the AWS native services, and in turn trigger more of these “guard rails”.

Lastly there are a lot of developers out there who just don’t have time to “harden” their environments, teams who have no choice but to ignore “best practices” and may benefit a lot from some uplift in this area.

What about Cloudformation?

To further demonstrate this issue this is s3 bucket creation in cloudformation, which is the baseline orchestration tool for building resources, provided free of charge by AWS. This is a very basic example, as seen in a lot of projects on GitHub, and the AWS cloudformation documentation.

      MyDataBucket:
        Type: AWS::S3::Bucket
          Properties:
            BucketName: MyDataBucket

Now you could argue that cloudformation is doing exactly what you tell it to do, it is just a primitive layer which translates YAML or JSON into API calls to AWS, but I think again this is really letting developers down.

Again this is missing default encryption, and public access safe guards. Now in addition to this a lot of quality tools also recommend the following:

• Explicit deny of Delete* operations, good practice for systems of record
• Enable Versioning, optional but good practice for systems of record
• Enable object access logging, which is omitted it to keep the example brief

So this is a basic example with most of these options enabled, this is quite a lot to fill in for yourself.

     MyDataBucket:
        Type: AWS::S3::Bucket
        DeletionPolicy: Retain
        UpdateReplacePolicy: Retain
        Properties:
          BucketName: !Ref BucketName
          BucketEncryption:
            ServerSideEncryptionConfiguration:
              - ServerSideEncryptionByDefault:
                  SSEAlgorithm: AES256
          VersioningConfiguration:
            Status: Enabled
          PublicAccessBlockConfiguration:
            BlockPublicAcls: True
            BlockPublicPolicy: True
            IgnorePublicAcls: True
            RestrictPublicBuckets: True
    
      MyDataBucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
          Bucket: !Ref MyDataBucket
          PolicyDocument:
            Id: AccessLogBucketPolicy
            Version: "2012-10-17"
            Statement:
              - Sid: AllowSSLRequestsOnly
                Action:
                  - s3:*
                Effect: Deny
                Resource:
                  - !Sub "arn:aws:s3:::${MyDataBucket}/*"
                  - !Sub "arn:aws:s3:::${MyDataBucket}"
                Condition:
                  Bool:
                    "aws:SecureTransport": "false"
                Principal: "*"
              - Sid: Restrict Delete* Actions
                Action: s3:Delete*
                Effect: Deny
                Principal: "*"
                Resource: !Sub "arn:aws:s3:::${MyDataBucket}/*"

To do this with the AWS CLI in one command would require quite a few flags, and options, rather than including that here I will leave that exercise up to the reader.

Now some may say this is a great opportunity for consulting companies to endlessly uplift customer infrastructure. But this again begs the questions:

1. Why is this the case for customers using the recommended tools?
2. What about developers getting started on their first application?
3. Wouldn’t be better to have these consultants building something new, rather than crafting reams of YAML?

Why Provide Resources which are Secure by Default?

So I have used S3 buckets as a very common example, but there is an ever growing list of services in the AWS that I think would benefit from better default configuration.

Just to summarise some of the points I have made above:

1. It would make it harder for those new to the cloud to do the wrong thing when following examples.
2. The cost of building and maintaining infrastructure would be reduced over time as safer defaults would remove the need for pages of code to deploy “secure” s3 buckets.
3. For new and busy developers things would be mostly right from the beginning, and likewise update that baseline even just for new applications, leaving them more time to do the actual work they should be doing.

So anyone who is old enough to remember Sun Solaris will recall the “secure by default” effort launched with Solaris 10 around 2005, this also came with “self healing” (stretch goal for AWS?), so security issues around defaults is not a new problem, but has been addressed before!

Follow Up Q&A

I have added some of the questions I received while reviewing this article, with some answers I put together.

Will CDK help with this problem of defaults?

So as it stands now I don’t believe the default s3 bucket construct has any special default settings, there is certainly room for someone to make “secure” versions of the constructs but developers would need to search for them and that kind of misses the point of helping wider AWS user community.

Why don’t you just write your own CLI to create buckets?

This is a good suggestion, however I already have my fair share of side projects, if I was to do this it would need to be championed by a orginisation, and team that got value from the effort. But again, needing to tell every new engineer to ignore the default AWS CLI as it isn’t “secure” seems to be less than ideal, I really want everyone to be “secure”.

How did you come up with this topic?

Well I am currently working through “retrofitting” best practices (the latest ones) on a bunch of aws serverless stacks which I helped build a year or so ago, this is when I asked the question why am I searching then helping to document what is “baseline” configuration for s3 buckets?!

Won’t this make the tools more complicated adding all these best practices?

I think any uplift at all would be a bonus at the moment, I don’t think it would be wise to take on every best practice out there, but surely the 80/20 rule would apply here. Anything to reduce the amount of retro fitting we need to do would be a good thing in my view.