Over the last few months, I have found myself going from working alone to working with an AI agent, such as Anthropic’s Claude, OpenAI’s Codex, amp or Google’s Gemini CLI.

This change has been both exciting and challenging. With the help of this AI agent, I have been able to delegate tasks and focus on the most important things.

With a shift in mindset, I’ve been able to delegate tasks more effectively and get predictable outcomes.

The death of the Chat Box

Over the last few months, I have found myself moving away from transactional interactions with AI agents via a chat box, to a more collaborative approach. Instead of asking the AI agent to fix an issue and then reviewing the results, I am now working in a more iterative way. This has led me to follow a more specification driven development process which is a great way to ensure more predictable and reliable results.

This process looks like:

1. Provide details of the problem, feature, or bug, then work with the agent to put together a plan.
2. Review the plan, remove any unnecessary steps and focus on the most important ones. I then ask the agent to export the plan to a specification in a markdown file in the codebase.
3. I then clear the context (/clear in claude code) and get the agent to review the specification and provide feedback. This typically highlights a few areas that need to be addressed.
4. If the specification looks good, and I am clear on the outcomes, then I instruct the agent to start work on the specification, this typically follows one or more phases.
5. I then do some testing, review the results and provide feedback.
6. I clear the context and get the agent to review the specification and the outcomes, then we update it with the results.
7. Finally, I clear the context and get the agent to review the outcomes and provide feedback, for code this is done using a code review skill or sub agent. Once we have completed this process I can commit the changes to the codebase.

This process is especially useful for most tasks, building new features, or refactoring existing ones, but I find a scaled back version of this process is even useful for small tasks, to ensure the agent is kept on track.

NOTE: During long conversations with the AI Agent it is important to keep the context clear otherwise it will fill up with irrelevant discussions which impacts the performance and distracts the agent.

Documentation is King

Documentation is the backbone of any software project and with the rise of AI, it is becoming even more important. The ability to quickly and easily create a specification, then iterate on it, and use it to drive the development process with an AI agent is key to ensuring work stays on track.

Why is maintaining a specification important?

• It helps you establish a clear goal and plan for what the AI agent is going to do.
• It provides a reference point for other developers and stakeholders.
• Once changes are complete the specification can be used to update the documentation used by customers.

The idea of documenting software isn’t new, this has been practiced since people started writing software. I am personally enjoying this renaissance of documentation as everyone wins, from the developers who are writing the code to the customers who are using it.

Conclusion

This is a new way of working. It won’t be perfect, especially while you’re figuring out how to work with the AI agent. But it is an opportunity to improve your productivity by embracing this new paradigm.

Key takeaways:

• Embrace specification driven development, as this is the foundation of good software development.
• Ensure specifications are reviewed and questioned before the AI agent starts work, this avoids wasting time reworking, or removing pointless changes.
• Be collaborative with your AI agent, ask questions, sweat the details and be patient as you learn to build up your intuition and confidence with these tools.

One big thing to understand is AI Agents are especially valuable when tackling tasks you aren’t familiar with. Tell the agent up front your goal is to solve a problem, and learn how it works, this will help the agent clearly understand the goals and provide the best possible outcomes.

Links

• A recent side project specification written with Claude Code
• Claude Code: Best practices for agentic coding
• Marc Brooker: On the success of ‘natural language programming’

HTTP/2’s multiplexing and binary framing make it significantly more efficient than HTTP/1.1, reducing latency and improving throughput. Connect RPC lets you harness these benefits while maintaining broad client compatibility for services that can’t yet support HTTP/2.

Connect RPC can be used to build both internal and external APIs, powering frontends, mobile apps, CLIs, agents and more. See the list of supported languages.

Core Features

Connect RPC provides a number of features out of the box, such as:

• Interceptors which make it easy to extend Connect RPC and are used to add authentication, logging, metrics, tracing and retries.
• Serialization & compression, with pluggable serializers, and support for asymmetric compression reducing the amount of data that needs to be transmitted, or received.
• Error handling, with a standard error format, with support for custom error codes to allow for more granular error handling.
• Observability, with in built support for OpenTelemetry enabling you to easily add tracing, or metrics to your APIs.
• Streaming, which provides a very efficient way to push or pull data without polling.
• Schemas, which enable you to define and validate your API schemas, and generate code from them.
• Code generation for Go, TypeScript, Kotlin, Swift and Java.

Ecosystem

In addition to these features, Connect RPC is built on top of the Buf ecosystem, which offers notable benefits:

• Connect RPC joins CNCF, entering the cloud-native ecosystem, which is great for the long term sustainability of the project.
• Buf Schema Registry, which is a great tool for managing, sharing and versioning your API schemas.
• Buf CLI, a handy all in one tool for managing your APIs, generating code and linting.

Recommended Interceptor Packages

Some handy Go packages that provide pre-built Connect RPC interceptors worth exploring or using as a starting point:

• authn-go, provides a rebuilt authentication middleware library for Go. It works with any authentication scheme (including HTTP basic authentication, cookies, bearer tokens, and mutual TLS).
• validate-go provides a Connect RPC interceptor that takes the tedium out of data validation. This package is powered by protovalidate and the Common Expression Language.
• rpclog provides a structured logging interceptor for Connect RPC with support for both unary and streaming RPCs.

Summary

1. Connect RPC provides a paved and well maintained path to building gRPC compatible APIs, while maintaining compatibility for HTTP/1.1 clients. This is invaluable for product teams that need to support multiple client types without building custom compatibility layers.

2. Using a mature library like Connect RPC, you get to benefit from all the prebuilt integrations, and the added capabilities of the Buf ecosystem. This makes publishing and consuming APIs a breeze.

3. Protobuf schemas, high performance serialisation and compression ensure you get robust and efficient APIs.

Conclusion

Connect RPC makes it easy to build high-performance, robust APIs with gRPC compatibility, while avoiding the complexity of building and maintaining custom compatibility layers.

In this model the CI provider acts as an identity provider, issuing tokens to the CI runner/agent which include a set of claims identifying the owner, pipeline, workflow and job that is being executed. This is then used to authenticate with the cloud service, and access the resources that the pipeline, workflow and job require.

In simple terms, this is a form of trust delegation, where the CI provider is trusted by the cloud service to issue tokens on behalf of the owner, pipeline, workflow and job.

How OIDC Works

The OIDC trust delegation flow is as follows:

sequenceDiagram
    participant CI as CI Provider<br/>(Identity Provider)
    participant Runner as CI Runner/Agent
    participant Cloud as Cloud Service<br/>(AWS/Azure/GCP)

    Note over CI,Cloud: OIDC Trust Delegation Flow

    CI->>Runner: Issue OIDC token with claims<br/>(pipeline, workflow, job)
    Runner->>Cloud: Request access with OIDC token
    Cloud->>Cloud: Verify token signature<br/>and validate claims
    Cloud->>Runner: Grant temporary credentials
    Runner->>Cloud: Access resources with credentials

    Note over CI,Cloud: Trust established via OIDC configuration

There are a few things to note:

• When using OIDC, the runner doesn’t need to be registered with the cloud service; it is granted access via the OIDC token.
• The OIDC token is cryptographically signed by the CI provider, and the cloud service verifies the signature to ensure the token is valid.
• In this model all three parties (CI provider, runner, and cloud service) are trusted to issue and verify tokens.

Limiting Cloud Access to the Agent/Runner

To ensure the CI provider can’t access the cloud service directly, you can add conditions which ensure only the runner/agent is allowed to access the cloud resources.

On top of this cloud providers such as AWS have conditions which can restrict access to a specific AWS network resource, such as a VPC. I recommend familiarizing yourself with the documentation for your cloud provider to understand how to lock down access to the runner/agent.

Benefits of OIDC

The reasons this is useful are:

• It provides a more secure and flexible approach to authentication and authorization
• It limits the scope of the token to the specific pipeline, workflow, and job
• It is tied to the lifecycle of the pipeline, workflow, and job, which means the token is limited to the duration of that execution
• It is more flexible than using machine identity for CI runners/agents as it allows for more granular control over the permissions granted to the runner/agent

Ephemeral Runners/Agents

Ephemeral runners/agents are short lived, single job, or single workflow runners/agents which are created before the workflow, or job is started. They provide a more secure and flexible approach to job execution as there is no need to worry about these environments being tainted by previous jobs or workflows.

When paired with OIDC these environments provide an extra layer of security as they are destroyed after the job or workflow is complete, further reducing the risk of cross job or workflow access.

Summary

So in summary, OIDC provides a more secure and flexible approach to access management for CI projects, and it is particularly useful when paired with ephemeral runners/agents.

The biggest advantage of this approach is that it allows engineers to focus on the access required by the pipeline, workflow, and job, rather than having to manage machine identities and permissions for each runner/agent.

One of the interesting things about this approach is that you’re not limited to using OIDC just with cloud providers; you can use it with your own services as well. By using OIDC libraries such as github.com/coreos/go-oidc, you can implement APIs which can use the identity of CI pipelines, workflows, and jobs. An example of this is Using OIDC With HashiCorp Vault and GitHub Actions.

Links

• OIDC for Buildkite
• OIDC for GitHub Actions
• OIDC for GitLab

This is especially a problem with two common packages I use:

1. Any HTTP adaptor package, which ships with integrations for multiple server packages, such as Gin, Echo, and others.
2. Any package which uses docker to test with containers.
3. Projects which include examples with their own dependencies.

To break this cycle in my own projects, and packages I publish privately in work projects, I have adopted the use of Go workspaces, which allows me to create a monorepo broken up into multiple packages, and then publish one or more of these packages.

So to understand how this helps, let’s provide an example. I have a project called s3iofs which provides an s3 based io/fs adaptor, and within this project I have integrations tests which use docker and minio server to test it.

Before I started using workspaces, if you added this package to your project you would have the docker client added to your dependencies, which in turn would add its dependencies, resulting in a lot of bloat in your project.

This is best illustrated by the following dependency count, which is from my github.com/wolfeidau/s3iofs package.

cat go.sum| wc -l
65

By comparison my github.com/wolfeidau/s3iofs/integration package has the following dependency count.

cat go.sum| wc -l
185

This is a rather simplistic comparison, but you can see that the integration tests have a lot more dependencies.

Because I have isolated the docker based integration tests in their own package, within this workspace, I can develop away happily, not needing to micromanage these modules, while you as the consumer of my package get a lean secure package.

How to use workspaces

To get started with workspaces I recommend a couple of tutorials, the Getting started with multi-module workspaces, then.

Once you have read through the getting started guide, you can publish your packages with the following commands.

• First we should initialise our go project, this is done from an empty folder with the same name as the project, this will create a go.mod file.

mkdir s3backend
cd s3backend
go mod init github.com/wolfeidau/s3backend

• Now once we have written some code and added some dependencies, we can set up some integration tests, to do this we will initialise another go project in a subfolder called integration. In the case of s3iofs the only code in this folder are test files.

mkdir integration
cd integration
go mod init github.com/wolfeidau/s3backend/integration

• Now we can add initialise our workspace and add the two packages, being our library in the root, and the integration tests.

go work init
go work use .
go work use ./integration

• Now we can run the tests in the integration folder, note I have included

cd integration
go test -covermode=atomic -coverpkg=github.com/wolfeidau/s3iofs -v ./...

• This will provide the test results as follows, note how I am able to provide test coverage across module boundaries using the -coverpkg flag which was introduced in Go 1.20 and is explained in Coverage profiling support for integration tests.

PASS
coverage: 70.2% of statements in github.com/wolfeidau/s3iofs
2023/12/28 12:58:43 code 0
ok  	github.com/wolfeidau/s3iofs/integration	1.225s	coverage: 70.2% of statements in github.com/wolfeidau/s3iofs

If you need this -coverpkg option to work in vscode, you will need to add the following to your .vscode/settings.json file in your project.

{
    "go.testFlags": [
        "-v",
        "-coverpkg=github.com/wolfeidau/s3iofs"
    ]
}

This is just one use case for using workspaces in a monorepo, but it is a very useful tool for managing dependencies you use in your project, and how you keep what you provide to others as lean and secure as possible.

I recommend you clone the s3iofs and dig into how it works locally, open it in your editor of choice and run the tests, then try it out in your own project.

The aim of this exercise is to help develop some intuition of how AI works, and how it can be used to help in your day-to-day tasks, while hopefully discovering ways to use it in future applications you build.

Getting Started

As the common saying that originated from a Chinese proverb says.

A journey of a thousand miles begins with a single step.

To kick off your understanding of AI I recommend you select a coding assistant and start using it on your personal, or side projects, this will provide you with a better understanding of how it succeeds, and sometimes fails. Building this knowledge up will help you develop an understanding of strengths and weaknesses as a user.

I personally recommend getting started with Cody as it is a great tool, and is free for personal use, while also being open source itself. The developers of Cody are very open and helpful, and have a great community of users, while also sharing their own experiences while building the tool.

Cody is more than just a code completion tool, you can ask it questions and get it to summarise and document your code, and even generate test cases. Make sure you explore all the options, again to build up more knowledge of how these AI tools work.

And most importantly be curios, and explore every corner of the tool.

Diving Into LLMs

Next, I recommend you start experimenting with some of the open source large language models (LLMs) using tools such as ollama to allow you to download, run and experiment with the software. To get started with this tool, you can follow the quick start in the README.md hosted at https://github.com/jmorganca/ollama. Also there is a great intro by Sam Witteveen Ollama - Local Models on your machine which I highly recommend.

What is a large language model?

Here is a quote from wikipedia on what a large language model is:

A large language model (LLM) is a large scale language model notable for its ability to achieve general-purpose language understanding and generation. LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly transformers and are (pre)trained using self-supervised learning and semi-supervised learning.

Why Open LLMs?

I prefer to learn from the open LLMs for the following reasons:

1. They have a great community of developers and users, who share information about the latest developments.
2. You get a broader range of models, and can try them out and see what they do.
3. You can run them locally with your data, and see what they do without some of the privacy concerns of cloud based services.
4. You have the potential to fine tune them to your data, and improve the performance.

I keep up with the latest developments I use hugging face open llm leader board, as they have been doing a lot of work on large language models, and have a great community of users. When the latest models are posted they are also sharing their experiences, and fine tuned versions via their blog, which is great resource. Notable models are normally added to ollama after a day or so, so you can try them out and see what they do.

There are a number of different types of LLMs, each with their own strengths and weaknesses. I personally like to experiment with the chat bot models, as they are very simple to use, and are easy to interface with via olloma. An example of one of these from the hugging face site is https://huggingface.co/mistralai/Mistral-7B-v0.1 which is a chat bot model trained by the Mistral AI team.

To get started with this model you can follow the instructions at https://ollama.ai/library/mistral, download and run the model locally.

Pick a Scenario To Test

My scenario relates to my current role, and covers questions which my team encounters on a day-to-day basis. As a team we are providing advice to a customers about how to improve the operational readiness and security posture for internally developed applications. This is a common scenario for many companies, where applications are developed to provide a proof of concept, and are then deployed to a production environment without the supporting processes in place.

What is approach is helpful as:

1. This is a scenario I can relate to, and can use my existing knowledge to review the results.
2. This is a scenario which is not too complex, and can be used to demonstrate the concepts.
3. This is a scenario which will provide me value while I am learning how to use the tools.

Building a list of questions

Once you have a scenario, you can draft a list of questions which you can start testing them with models, this will help you understand how the models work, and how they can be to support a team or business unit, while also learning how to use them.

The questions I am currently using mainly focus on DevOps, and SRE processes, paired with a dash of AWS security and terraform questions.

I need to create a secure environment in and AWS Account, where should I start?

This question is really common for developers starting out in AWS, it is quite broad and I am mostly expecting a high level overview of how to create a secure environment, and how to get started.

How would I create an encrypted secure s3 bucket using terraform?

This question is a bit more specific, focusing on a single AWS service, while also adding a few specific requirements. Models like Mistral will provide a step by step guide on how to achieve this, while others will provide the terraform code to achieve this.

I need to create an Application Risk Management Program, where should I start?

This question is quite common if your working in a company which doesn’t have a long history with internal software development, or a team that is trying to ensure they cover the risks of their applications.

What is a good SRE incident process for a business application?

This question is also quite broad, but includes Site Reliability Engineering (SRE) as a keyword, so I am expecting an answer which aligns with the principals of this movement.

What is a good checklist for a serverless developer who wants to improve the monitoring of their applications?

This is a common question asked by people who are just getting started with serverless and are interested in, or have been asked to improve the monitoring of their applications.

Whats Next?

So now that you have a scenario and a few questions I recommend you do the following:

1. Try a couple of other models, probably llama2 and orca are a good starting point.
2. Learn a bit about prompting by following A guide to prompting Llama 2 from the replicate blog.
3. Apply the prompts to your ollama model using a modelfile, which is similar to a Dockerfile.
4. Try out an uncensored model, something like llama2-uncensored and run through your questions, then ask about breaking into cars or killing processes, which can be a problematic question in some censored models. It is good to understand what censoring a model does, as it can be a useful tool for understanding the risks of using a model.
5. Start reading more about The State of Open Source AI (2023 Edition).

Further Research

Now that you are dabbling with LLMs, and AI, I recommend you try these models for the odd question in your day-to-day work, the local ones running in ollama are restively safe, and they can save you a lot of work.

Also try similar questions with services such as https://chat.openai.com/, hosted services are a powerful tool for adhoc testing and learning. Just be aware of data privacy, and security when using these services.

Once you have some experience you will hopefully even incorporate a model into work projects such as data cleansing, summarizations, or processing of user feedback to help you improve your applications. For this you can use services such as AWS Bedrock on AWS, or Generative AI Studio on Google cloud, while following the same methodology to evaluate and select a model for your use case.

If your intrigued and want to go even deeper than these APIs, I recommend you dive into some of the amazing resources on the web for learning how AI and LLMs work, and possibly even develop, or fine tune your own own models.

• fast.ia which provides some great online self paced learning on AI.
• A busy persons intro to LLMs great lecture on LLMs.
• MIT Introduction to Deep Learning for those who want to dive deeper and prefer more of a structured course.
• A Hackers’ Guide to Language Models, another great talk by Jeremy Howard of fast.ai.