Over the last few months, I have found myself going from working alone to working with an AI agent, such as Anthropic’s Claude, OpenAI’s Codex, amp or Google’s Gemini CLI.

This change has been both exciting and challenging. With the help of this AI agent, I have been able to delegate tasks and focus on the most important things.

With a shift in mindset, I’ve been able to delegate tasks more effectively and get predictable outcomes.

The death of the Chat Box

Over the last few months, I have found myself moving away from transactional interactions with AI agents via a chat box, to a more collaborative approach. Instead of asking the AI agent to fix an issue and then reviewing the results, I am now working in a more iterative way. This has led me to follow a more specification driven development process which is a great way to ensure more predictable and reliable results.

This process looks like:

1. Provide details of the problem, feature, or bug, then work with the agent to put together a plan.
2. Review the plan, remove any unnecessary steps and focus on the most important ones. I then ask the agent to export the plan to a specification in a markdown file in the codebase.
3. I then clear the context (/clear in claude code) and get the agent to review the specification and provide feedback. This typically highlights a few areas that need to be addressed.
4. If the specification looks good, and I am clear on the outcomes, then I instruct the agent to start work on the specification, this typically follows one or more phases.
5. I then do some testing, review the results and provide feedback.
6. I clear the context and get the agent to review the specification and the outcomes, then we update it with the results.
7. Finally, I clear the context and get the agent to review the outcomes and provide feedback, for code this is done using a code review skill or sub agent. Once we have completed this process I can commit the changes to the codebase.

This process is especially useful for most tasks, building new features, or refactoring existing ones, but I find a scaled back version of this process is even useful for small tasks, to ensure the agent is kept on track.

NOTE: During long conversations with the AI Agent it is important to keep the context clear otherwise it will fill up with irrelevant discussions which impacts the performance and distracts the agent.

Documentation is King

Documentation is the backbone of any software project and with the rise of AI, it is becoming even more important. The ability to quickly and easily create a specification, then iterate on it, and use it to drive the development process with an AI agent is key to ensuring work stays on track.

Why is maintaining a specification important?

• It helps you establish a clear goal and plan for what the AI agent is going to do.
• It provides a reference point for other developers and stakeholders.
• Once changes are complete the specification can be used to update the documentation used by customers.

The idea of documenting software isn’t new, this has been practiced since people started writing software. I am personally enjoying this renaissance of documentation as everyone wins, from the developers who are writing the code to the customers who are using it.

Conclusion

This is a new way of working. It won’t be perfect, especially while you’re figuring out how to work with the AI agent. But it is an opportunity to improve your productivity by embracing this new paradigm.

Key takeaways:

• Embrace specification driven development, as this is the foundation of good software development.
• Ensure specifications are reviewed and questioned before the AI agent starts work, this avoids wasting time reworking, or removing pointless changes.
• Be collaborative with your AI agent, ask questions, sweat the details and be patient as you learn to build up your intuition and confidence with these tools.

One big thing to understand is AI Agents are especially valuable when tackling tasks you aren’t familiar with. Tell the agent up front your goal is to solve a problem, and learn how it works, this will help the agent clearly understand the goals and provide the best possible outcomes.

Links

• A recent side project specification written with Claude Code
• Claude Code: Best practices for agentic coding
• Marc Brooker: On the success of ‘natural language programming’

HTTP/2’s multiplexing and binary framing make it significantly more efficient than HTTP/1.1, reducing latency and improving throughput. Connect RPC lets you harness these benefits while maintaining broad client compatibility for services that can’t yet support HTTP/2.

Connect RPC can be used to build both internal and external APIs, powering frontends, mobile apps, CLIs, agents and more. See the list of supported languages.

Core Features

Connect RPC provides a number of features out of the box, such as:

• Interceptors which make it easy to extend Connect RPC and are used to add authentication, logging, metrics, tracing and retries.
• Serialization & compression, with pluggable serializers, and support for asymmetric compression reducing the amount of data that needs to be transmitted, or received.
• Error handling, with a standard error format, with support for custom error codes to allow for more granular error handling.
• Observability, with in built support for OpenTelemetry enabling you to easily add tracing, or metrics to your APIs.
• Streaming, which provides a very efficient way to push or pull data without polling.
• Schemas, which enable you to define and validate your API schemas, and generate code from them.
• Code generation for Go, TypeScript, Kotlin, Swift and Java.

Ecosystem

In addition to these features, Connect RPC is built on top of the Buf ecosystem, which offers notable benefits:

• Connect RPC joins CNCF, entering the cloud-native ecosystem, which is great for the long term sustainability of the project.
• Buf Schema Registry, which is a great tool for managing, sharing and versioning your API schemas.
• Buf CLI, a handy all in one tool for managing your APIs, generating code and linting.

Recommended Interceptor Packages

Some handy Go packages that provide pre-built Connect RPC interceptors worth exploring or using as a starting point:

• authn-go, provides a rebuilt authentication middleware library for Go. It works with any authentication scheme (including HTTP basic authentication, cookies, bearer tokens, and mutual TLS).
• validate-go provides a Connect RPC interceptor that takes the tedium out of data validation. This package is powered by protovalidate and the Common Expression Language.
• rpclog provides a structured logging interceptor for Connect RPC with support for both unary and streaming RPCs.

Summary

1. Connect RPC provides a paved and well maintained path to building gRPC compatible APIs, while maintaining compatibility for HTTP/1.1 clients. This is invaluable for product teams that need to support multiple client types without building custom compatibility layers.

2. Using a mature library like Connect RPC, you get to benefit from all the prebuilt integrations, and the added capabilities of the Buf ecosystem. This makes publishing and consuming APIs a breeze.

3. Protobuf schemas, high performance serialisation and compression ensure you get robust and efficient APIs.

Conclusion

Connect RPC makes it easy to build high-performance, robust APIs with gRPC compatibility, while avoiding the complexity of building and maintaining custom compatibility layers.

In this model the CI provider acts as an identity provider, issuing tokens to the CI runner/agent which include a set of claims identifying the owner, pipeline, workflow and job that is being executed. This is then used to authenticate with the cloud service, and access the resources that the pipeline, workflow and job require.

In simple terms, this is a form of trust delegation, where the CI provider is trusted by the cloud service to issue tokens on behalf of the owner, pipeline, workflow and job.

How OIDC Works

The OIDC trust delegation flow is as follows:

sequenceDiagram
    participant CI as CI Provider<br/>(Identity Provider)
    participant Runner as CI Runner/Agent
    participant Cloud as Cloud Service<br/>(AWS/Azure/GCP)

    Note over CI,Cloud: OIDC Trust Delegation Flow

    CI->>Runner: Issue OIDC token with claims<br/>(pipeline, workflow, job)
    Runner->>Cloud: Request access with OIDC token
    Cloud->>Cloud: Verify token signature<br/>and validate claims
    Cloud->>Runner: Grant temporary credentials
    Runner->>Cloud: Access resources with credentials

    Note over CI,Cloud: Trust established via OIDC configuration

There are a few things to note:

• When using OIDC, the runner doesn’t need to be registered with the cloud service; it is granted access via the OIDC token.
• The OIDC token is cryptographically signed by the CI provider, and the cloud service verifies the signature to ensure the token is valid.
• In this model all three parties (CI provider, runner, and cloud service) are trusted to issue and verify tokens.

Limiting Cloud Access to the Agent/Runner

To ensure the CI provider can’t access the cloud service directly, you can add conditions which ensure only the runner/agent is allowed to access the cloud resources.

On top of this cloud providers such as AWS have conditions which can restrict access to a specific AWS network resource, such as a VPC. I recommend familiarizing yourself with the documentation for your cloud provider to understand how to lock down access to the runner/agent.

Benefits of OIDC

The reasons this is useful are:

• It provides a more secure and flexible approach to authentication and authorization
• It limits the scope of the token to the specific pipeline, workflow, and job
• It is tied to the lifecycle of the pipeline, workflow, and job, which means the token is limited to the duration of that execution
• It is more flexible than using machine identity for CI runners/agents as it allows for more granular control over the permissions granted to the runner/agent

Ephemeral Runners/Agents

Ephemeral runners/agents are short lived, single job, or single workflow runners/agents which are created before the workflow, or job is started. They provide a more secure and flexible approach to job execution as there is no need to worry about these environments being tainted by previous jobs or workflows.

When paired with OIDC these environments provide an extra layer of security as they are destroyed after the job or workflow is complete, further reducing the risk of cross job or workflow access.

Summary

So in summary, OIDC provides a more secure and flexible approach to access management for CI projects, and it is particularly useful when paired with ephemeral runners/agents.

The biggest advantage of this approach is that it allows engineers to focus on the access required by the pipeline, workflow, and job, rather than having to manage machine identities and permissions for each runner/agent.

One of the interesting things about this approach is that you’re not limited to using OIDC just with cloud providers; you can use it with your own services as well. By using OIDC libraries such as github.com/coreos/go-oidc, you can implement APIs which can use the identity of CI pipelines, workflows, and jobs. An example of this is Using OIDC With HashiCorp Vault and GitHub Actions.

Links

• OIDC for Buildkite
• OIDC for GitHub Actions
• OIDC for GitLab

Far too often I have seen datasets just dumped into one massive S3 bucket, and left for someone else to tidy up later, however with a little consideration, and empathy for those dealing with this in the future, we can do better than this.

Start By Asking a few questions

When I am planning to store a dataset in s3 I typically ask a few questions, one thing to note is I am focused on the semantics of the data, and the business, not just the bucket(s) technical configuration at this stage.

What will consume this information?

What I am trying to understand here is whether this dataset has any known consumers, with the AWS logs example, this may be an ingestion tool like Splunk, which is easier to integrate with if there are a few aggregate buckets.

For datasets which are exported from other systems, or transformed for use in an application, or with an integration it may be easier to combine them into one bucket, especially if other factors I cover in the next few questions aren’t a concern.

As you will see in my following questions, this is a trade off, and I would also review other points below to determine which is the best approach in the long run.

What is the classification for the data?

My goal here is to consider the sensitivity of the data, and how it could affect who is granted access.

Keeping sensitive datasets isolated in their own bucket makes it easier to add controls, and simplifies auditing as there is only one top level identifier, i.e., the bucket name.

One thing to avoid is mixing different data classifications in one bucket, as you then need to tag all data in that bucket at the highest classification, which could complicate granting access to the data.

For an example of data classifications, this is a five-tiered commercial data classification approach provided in this book CISSP Security Management and Practices:

• Sensitive
• Confidential
• Private
• Proprietary
• Public

These classifications would be assigned to a tag, such named Classification on your bucket, for more on this see Categorizing your storage using tags.

In general I recommend keeping different classifications of data separated, for example having raw data and anonymised, or cleaned data in the same bucket is not a good idea.

What are the cost implications for keeping this dataset?

The aim with this question is to understand how cost will be managed for a dataset, there are a couple of factors here, the size of the data, and how much churn will occur for the dataset, either in the form of reads or updates to the data.

For datasets which grow quickly, it may be easier to isolate them in their own bucket, as reporting cost for this dataset is easier, and cost control mechanisms such as lifecycle policies, or disabling/enabling versioning simpler to apply.

For more information on optimising storage costs, see Optimizing costs for Amazon S3.

What is the likely growth of this dataset in 6 to 12 months?

This question is related to the previous cost question, but I am trying to understand how challenging the dataset will be to handle over time. External factors such as traffic spikes for log datasets, which are often outside your control, should be taken into consideration as well.

There are two dimensions to this, the size of the data, and the number of objects in the dataset, both can have an impact on how difficult to wrangle the dataset will be in the future, and how much it will cost to move, or backup.

For more information on how to monitor and manage dataset growth in Amazon S3 I recommend digging into Amazon S3 Storage Lens.

Summary

As a general rule, I would recommend keeping datasets in separate buckets, with each bucket containing data of a single classification, and ideally a single purpose. This will help to simplify cost control, and make it easier to manage the data in the future.

Getting things right from the start will enable you to make the most of your datasets, which is a potential differentiator for your business in this new era of cloud computing, data engineering and AI.

This is especially a problem with two common packages I use:

1. Any HTTP adaptor package, which ships with integrations for multiple server packages, such as Gin, Echo, and others.
2. Any package which uses docker to test with containers.
3. Projects which include examples with their own dependencies.

To break this cycle in my own projects, and packages I publish privately in work projects, I have adopted the use of Go workspaces, which allows me to create a monorepo broken up into multiple packages, and then publish one or more of these packages.

So to understand how this helps, let’s provide an example. I have a project called s3iofs which provides an s3 based io/fs adaptor, and within this project I have integrations tests which use docker and minio server to test it.

Before I started using workspaces, if you added this package to your project you would have the docker client added to your dependencies, which in turn would add its dependencies, resulting in a lot of bloat in your project.

This is best illustrated by the following dependency count, which is from my github.com/wolfeidau/s3iofs package.

cat go.sum| wc -l
65

By comparison my github.com/wolfeidau/s3iofs/integration package has the following dependency count.

cat go.sum| wc -l
185

This is a rather simplistic comparison, but you can see that the integration tests have a lot more dependencies.

Because I have isolated the docker based integration tests in their own package, within this workspace, I can develop away happily, not needing to micromanage these modules, while you as the consumer of my package get a lean secure package.

How to use workspaces

To get started with workspaces I recommend a couple of tutorials, the Getting started with multi-module workspaces, then.

Once you have read through the getting started guide, you can publish your packages with the following commands.

• First we should initialise our go project, this is done from an empty folder with the same name as the project, this will create a go.mod file.

mkdir s3backend
cd s3backend
go mod init github.com/wolfeidau/s3backend

• Now once we have written some code and added some dependencies, we can set up some integration tests, to do this we will initialise another go project in a subfolder called integration. In the case of s3iofs the only code in this folder are test files.

mkdir integration
cd integration
go mod init github.com/wolfeidau/s3backend/integration

• Now we can add initialise our workspace and add the two packages, being our library in the root, and the integration tests.

go work init
go work use .
go work use ./integration

• Now we can run the tests in the integration folder, note I have included

cd integration
go test -covermode=atomic -coverpkg=github.com/wolfeidau/s3iofs -v ./...

• This will provide the test results as follows, note how I am able to provide test coverage across module boundaries using the -coverpkg flag which was introduced in Go 1.20 and is explained in Coverage profiling support for integration tests.

PASS
coverage: 70.2% of statements in github.com/wolfeidau/s3iofs
2023/12/28 12:58:43 code 0
ok  	github.com/wolfeidau/s3iofs/integration	1.225s	coverage: 70.2% of statements in github.com/wolfeidau/s3iofs

If you need this -coverpkg option to work in vscode, you will need to add the following to your .vscode/settings.json file in your project.

{
    "go.testFlags": [
        "-v",
        "-coverpkg=github.com/wolfeidau/s3iofs"
    ]
}

This is just one use case for using workspaces in a monorepo, but it is a very useful tool for managing dependencies you use in your project, and how you keep what you provide to others as lean and secure as possible.

I recommend you clone the s3iofs and dig into how it works locally, open it in your editor of choice and run the tests, then try it out in your own project.

The aim of this exercise is to help develop some intuition of how AI works, and how it can be used to help in your day-to-day tasks, while hopefully discovering ways to use it in future applications you build.

Getting Started

As the common saying that originated from a Chinese proverb says.

A journey of a thousand miles begins with a single step.

To kick off your understanding of AI I recommend you select a coding assistant and start using it on your personal, or side projects, this will provide you with a better understanding of how it succeeds, and sometimes fails. Building this knowledge up will help you develop an understanding of strengths and weaknesses as a user.

I personally recommend getting started with Cody as it is a great tool, and is free for personal use, while also being open source itself. The developers of Cody are very open and helpful, and have a great community of users, while also sharing their own experiences while building the tool.

Cody is more than just a code completion tool, you can ask it questions and get it to summarise and document your code, and even generate test cases. Make sure you explore all the options, again to build up more knowledge of how these AI tools work.

And most importantly be curios, and explore every corner of the tool.

Diving Into LLMs

Next, I recommend you start experimenting with some of the open source large language models (LLMs) using tools such as ollama to allow you to download, run and experiment with the software. To get started with this tool, you can follow the quick start in the README.md hosted at https://github.com/jmorganca/ollama. Also there is a great intro by Sam Witteveen Ollama - Local Models on your machine which I highly recommend.

What is a large language model?

Here is a quote from wikipedia on what a large language model is:

A large language model (LLM) is a large scale language model notable for its ability to achieve general-purpose language understanding and generation. LLMs acquire these abilities by using massive amounts of data to learn billions of parameters during training and consuming large computational resources during their training and operation. LLMs are artificial neural networks (mainly transformers and are (pre)trained using self-supervised learning and semi-supervised learning.

Why Open LLMs?

I prefer to learn from the open LLMs for the following reasons:

1. They have a great community of developers and users, who share information about the latest developments.
2. You get a broader range of models, and can try them out and see what they do.
3. You can run them locally with your data, and see what they do without some of the privacy concerns of cloud based services.
4. You have the potential to fine tune them to your data, and improve the performance.

I keep up with the latest developments I use hugging face open llm leader board, as they have been doing a lot of work on large language models, and have a great community of users. When the latest models are posted they are also sharing their experiences, and fine tuned versions via their blog, which is great resource. Notable models are normally added to ollama after a day or so, so you can try them out and see what they do.

There are a number of different types of LLMs, each with their own strengths and weaknesses. I personally like to experiment with the chat bot models, as they are very simple to use, and are easy to interface with via olloma. An example of one of these from the hugging face site is https://huggingface.co/mistralai/Mistral-7B-v0.1 which is a chat bot model trained by the Mistral AI team.

To get started with this model you can follow the instructions at https://ollama.ai/library/mistral, download and run the model locally.

Pick a Scenario To Test

My scenario relates to my current role, and covers questions which my team encounters on a day-to-day basis. As a team we are providing advice to a customers about how to improve the operational readiness and security posture for internally developed applications. This is a common scenario for many companies, where applications are developed to provide a proof of concept, and are then deployed to a production environment without the supporting processes in place.

What is approach is helpful as:

1. This is a scenario I can relate to, and can use my existing knowledge to review the results.
2. This is a scenario which is not too complex, and can be used to demonstrate the concepts.
3. This is a scenario which will provide me value while I am learning how to use the tools.

Building a list of questions

Once you have a scenario, you can draft a list of questions which you can start testing them with models, this will help you understand how the models work, and how they can be to support a team or business unit, while also learning how to use them.

The questions I am currently using mainly focus on DevOps, and SRE processes, paired with a dash of AWS security and terraform questions.

I need to create a secure environment in and AWS Account, where should I start?

This question is really common for developers starting out in AWS, it is quite broad and I am mostly expecting a high level overview of how to create a secure environment, and how to get started.

How would I create an encrypted secure s3 bucket using terraform?

This question is a bit more specific, focusing on a single AWS service, while also adding a few specific requirements. Models like Mistral will provide a step by step guide on how to achieve this, while others will provide the terraform code to achieve this.

I need to create an Application Risk Management Program, where should I start?

This question is quite common if your working in a company which doesn’t have a long history with internal software development, or a team that is trying to ensure they cover the risks of their applications.

What is a good SRE incident process for a business application?

This question is also quite broad, but includes Site Reliability Engineering (SRE) as a keyword, so I am expecting an answer which aligns with the principals of this movement.

What is a good checklist for a serverless developer who wants to improve the monitoring of their applications?

This is a common question asked by people who are just getting started with serverless and are interested in, or have been asked to improve the monitoring of their applications.

Whats Next?

So now that you have a scenario and a few questions I recommend you do the following:

1. Try a couple of other models, probably llama2 and orca are a good starting point.
2. Learn a bit about prompting by following A guide to prompting Llama 2 from the replicate blog.
3. Apply the prompts to your ollama model using a modelfile, which is similar to a Dockerfile.
4. Try out an uncensored model, something like llama2-uncensored and run through your questions, then ask about breaking into cars or killing processes, which can be a problematic question in some censored models. It is good to understand what censoring a model does, as it can be a useful tool for understanding the risks of using a model.
5. Start reading more about The State of Open Source AI (2023 Edition).

Further Research

Now that you are dabbling with LLMs, and AI, I recommend you try these models for the odd question in your day-to-day work, the local ones running in ollama are restively safe, and they can save you a lot of work.

Also try similar questions with services such as https://chat.openai.com/, hosted services are a powerful tool for adhoc testing and learning. Just be aware of data privacy, and security when using these services.

Once you have some experience you will hopefully even incorporate a model into work projects such as data cleansing, summarizations, or processing of user feedback to help you improve your applications. For this you can use services such as AWS Bedrock on AWS, or Generative AI Studio on Google cloud, while following the same methodology to evaluate and select a model for your use case.

If your intrigued and want to go even deeper than these APIs, I recommend you dive into some of the amazing resources on the web for learning how AI and LLMs work, and possibly even develop, or fine tune your own own models.

• fast.ia which provides some great online self paced learning on AI.
• A busy persons intro to LLMs great lecture on LLMs.
• MIT Introduction to Deep Learning for those who want to dive deeper and prefer more of a structured course.
• A Hackers’ Guide to Language Models, another great talk by Jeremy Howard of fast.ai.

If your using AWS_IAM authentication on an API Gateway, then make sure you set the default authorizer for all API resources. This will avoid accidental exposing an API if you mis-configure, or omit an authentication method for an API resource as the default is None.

In addition to this there is a way to apply a resource policy to an API Gateway, which will enforce a specific iam access check on all API requests. Combining the override to default authorizer, and the resource policy allows us to apply multiply layers of protection to our API, allowing us to follow the principle of defense in depth.

So to summarise, to protect your API with IAM authentication is as follows:

1. Enable a default authorizer method on the API Gateway resource.
2. Enable an authentication method on the API.
3. Assign an API resource policy which requires IAM authentication to access the API.

Doing this with AWS SAM is fairly straight forward, to read more about it see the SAM ApiAuth documentation.

  AthenaWorkflowApi:
    Type: AWS::Serverless::Api
    Properties:
      ...
      Auth:
        # Specify a default authorizer for the API Gateway API to protect against missing configuration
        DefaultAuthorizer: AWS_IAM
        # Configure Resource Policy for all methods and paths on an API as an extra layer of protection
        ResourcePolicy:
          # The AWS accounts to allow
          AwsAccountWhitelist:
           - !Ref AWS::AccountId

Through the magic of AWS SAM this results in a resource policy which looks like the following, this results in all the API methods being protected and only accessible by users authenticated to this account, and only where they are granted access via an IAM policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "execute-api:Invoke",
      "Resource": "arn:aws:execute-api:us-west-2:123456789012:abc123abc1/Prod/POST/athena/run_s3_query_template"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:root"
      },
      "Action": "execute-api:Invoke",
      "Resource": "arn:aws:execute-api:us-west-2:123456789012:abc123abc1/Prod/POST/athena/run_query_template"
    }
  ]
}

I typically use an openapi spec to define the API, using the extensions provided by AWS such as x-amazon-apigateway-auth to define the authorisation.

With the default authentication set to AWS_IAM hitting an API which is missing x-amazon-apigateway-auth using curl returns the following error.

{"message":"Missing Authentication Token"}

With default authentication disabled, and the resource policy enabled the API returns the following error, which illustrates the principle of defense in depth.

{"Message":"User: anonymous is not authorized to perform: execute-api:Invoke on resource: arn:aws:execute-api:us-east-1:********9012:abc123abc1/Prod/POST/athena/run_query_template"}

1. There is no automated migration path from existing Go Lambda functions to the new custom runtime. Customers will need to manually refactor and migrate each function to this new runtime, which this is time-consuming and error-prone.
2. This will remove Go1.x name from the lambda console, Go will now just be another “custom” runtime instead of a first class supported language. This makes Go development on Lambda seem less official/supported compared to other languages like Node, Python, Java etc.

Case in point, try searching for “al2.provided lambda” on Google and see how little documentation comes up compared to “go1.x lambda”. The migration essentially removes the branding and discoverability of Go as a Lambda language, I am sure this will improve over time, but it is still ambiguous.

There are articles on the advantages of the al2.provided runtime, including how to migrate functions over to it, such as https://www.capitalone.com/tech/cloud/custom-runtimes-for-go-based-lambda-functions/.

Why is this hard?

The main reason migrating Go Lambda functions to the new runtime is difficult is because:

1. Unlike the runtime provided for other languages, the custom runtime doesn’t use the Handler parameter to determine the function entry point, this value is ignored, but still required. This is a subtle difference can cause issues if developers are unaware or don’t read the documentation closely.
2. The lambda service doesn’t check if the bootstrap entry point exists in the archive, so customers may deploy broken functions if they don’t validate this. Sadly, this is NOT very intuitive, and often leads to confusion and errors.

Note: As pointed out by @Aidan W Steele some deployment tools upload empty archives, then later replace them with an updated archive containing the deployed code, so this could be problematic.

For those interested in what the error looks like if you’re missing the bootstrap file, it will return:

{"errorType":"Runtime.InvalidEntrypoint","errorMessage":"RequestId: d604d105-51be-49ce-8457-eee1641398eb Error: Couldn't find valid bootstrap(s): [/var/task/bootstrap /opt/bootstrap]"}

If you see this, you need to validate your deployment package contains the required bootstrap file in the root of the zip archive.

Why is removing Go 1.x a bad idea?

There will be no Go Lambda runtime available after this date, this will be more of an issue for developers who have never used AWS and expect lambda to have a Go runtime available out of the box. This is a change that will require some education and guidance for new developers.

Some of the drawbacks of this are:

1. You won’t be able to see Go functions directly in the Lambda console anymore. Go will just be another “custom” runtime instead of a first class supported language like Node, Python, Java etc. This makes Go development on Lambda seem less official/supported.
2. Developers will find samples or projects using the old Go 1.x runtime that no longer work out of the box. This will lead to confusion as they try to migrate those functions over to the new runtime.
3. Listing lambda functions by runtime used will no longer show “Go1.x” making it less clear if a function was written for Go or another language like Rust or Nim that also use the custom runtime.
4. Finding code samples for Go lambda functions on GitHub or tutorials will need to specify if they are using the old or new runtime. A lot of existing content will be outdated immediately.

What can AWS do better?

So what could AWS do to mitigate some of these issues? Here are a few suggestions:

1. Provide an updated go1.al2, which would match the pattern of Java update java8.al2 runtime announced a few years ago. This updated runtime would use the same entry point convention as the other languages like Node, Python etc and retain the existing user experience, avoiding the hard coded bootstrap file which is not very intuitive.
2. Add validation to the deployment process to check for the required bootstrap file, and prevent deployment of invalid archives. This would avoid broken functions being deployed.

I am disappointed that AWS did not invest a bit more time in listening to customers around the usability of the al2.provided runtime. Customers are used to compiling applications to a binary with a descriptive name, then deploying that binary to AWS, having to output a specific file called bootstrap is not very intuitive or discoverable.

Examples

To illustrate the differences I have included some examples, hopefully this helps those not familiar with lambda see the challenges.

Building

This is an example compile command for go based function prior to migration, this will build all commands using the name of the directory as their binary name, then zip up all the binaries with a name ending in -lambda.

build:
  CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w" -trimpath -o dist ./cmd/...

archive:
  cd dist && zip -X -9 ./deploy.zip *-lambda

With this migration, developers will need to package each of their functions to include a bootstrap file, and upload each archive to s3 individually rather than zipping multiple binaries together.

Deployment Configuration Examples

This is an example sam template for a go based function prior to migration:

  ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: go1.x
      # this is a example archive containing one or more go binary files
      CodeUri: ../../dist/deploy.zip  
      # note example is the name of the compiled go binary file
      Handler: example-lambda

This is an example sam template for a go based function after migration:

  ExampleFunction:
    Type: AWS::Serverless::Function
    Properties:
      Runtime: provided.al2
      # example archive which must contain a file named bootstrap, 
      # which is not referenced or checked during deploy.
      CodeUri: ../../dist/example_Linux_arm64.zip
      # unused by this runtime but still required and can 
      # cause some confusion with developers if not aware
      Handler: nope 

Closing Thoughts

While the custom runtime provides better performance, and an updated operating system, the change will require effort for many Go developers on AWS Lambda. Some automated assistance and validation from AWS could help reduce friction and issues from this change.

Personally I am sad to see AWS lambda remove Go as a first class language, as an early adopter of serverless it felt great to have Go supported out of the box. I will miss seeing the gopher logo when browsing functions! 😞🪦

Overall, I think this will negatively the adoption of Go in AWS lambda, at least in the short term, as a lot of developers will find the custom runtime requirements unfamiliar and confusing compared to other runtimes.

As is often the case, new developers will likely struggle most with the provided.al2, then most likely give up and use another language instead of taking the time to understand the custom runtime complexities.

What are your thoughts on the migration and how AWS could improve the experience?

Updates

Thanks to @Aidan W Steele for the feedback on my go2.x suggestion with a much better one of go1.al2 which would match the pattern of java8.al2, and reminder of the various empty zip file shenanigans used in some deployment tools.

What is the problem with IAM User Credentials?

• IAM User Credentials are long lived, meaning once compromised they allow access for a long time
• They are static, so if leaked it is difficult to revoke access immediately

But there are better alternatives, the one I recommend is OpenID Connect (OIDC), which if you dig deep into the Terraform Cloud docs is a supported approach. This has a few benefits:

1. Credentials are dynamically created for each run, so if one set is compromised it does not affect other runs.
2. When Terraform Cloud authenticates with AWS using OIDC it will pass information about the project and run, so you can enforce IAM policies based on this context.
3. Credentials are short lived, expiring after the Terraform run completes.
4. You can immediately revoke access by removing the OIDC provider from AWS.
5. You don’t need to export credentials from AWS and manage their rotation.

Overall this allows for a more secure and scalable approach to integrating Terraform Cloud with AWS. If you are just starting out, I would recommend setting up OpenID Connect integration instead of using IAM credentials.

AWS Deployment

To setup the resources on the AWS side required to link AWS to Terraform Cloud we need to deploy some resources, in my case I am using a Cloudformation Template which deploy manually. You can find the source code to this template in my GitHub Repo along with a Terraform example to deploy the resources.

Using the Cloudformation template as the example for this post, it creates:

1. IAM Role, which assumed by Terraform Cloud when deploying
2. Open ID Connect Provider, which is used to connect Terraform Cloud to AWS

The Terraform Deployment role is as follows:

  TerraformDeploymentRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Action: sts:AssumeRoleWithWebIdentity
            Principal:
              Federated: !Ref TerraformOIDCProvider
            Condition:
              StringEquals:
                app.terraform.io:aud: "aws.workload.identity"
              StringLike:
                app.terraform.io:sub: !Sub organization:${OrganizationName}:project:${ProjectName}:workspace:${WorkspaceName}:run_phase:*

Note:

• The IAM role allows Terraform Cloud to assume the role using the OIDC provider, and limits it to the given organization, project and workspace names.
• The policy attached to this role, in my example, only allows Terraform to list s3 buckets; you should customise this based on your needs.

The Open ID Connect Provider is created as follows:

  TerraformOIDCProvider:
    Type: AWS::IAM::OIDCProvider
    Properties:
      Url: https://app.terraform.io
      ClientIdList:
        - aws.workload.identity
      ThumbprintList:
        - 9e99a48a9960b14926bb7f3b02e22da2b0ab7280

Once deployed this template will provide two outputs:

1. The role ARN for the Terraform Deployment role.
2. An Optional Audience value, this is only needed if you want to customise this value.

Terraform Cloud Configuration

You’ll need to set a couple of environment variables in your Terraform Cloud workspace in order to authenticate with AWS using OIDC. You can set these as workspace variables, or if you’d like to share one AWS role across multiple workspaces, you can use a variable set.

Note for more advanced configuration options please refer to Terraform Cloud - Dynamic Credentials with the AWS Provider.

That is it, your now ready to run plans in your Terraform Cloud workspace!

Auditing

Once you have setup both side of this solution you should be able to see events in AWS CloudTrail, filter by service sts.amazonaws.com and look at the AssumeRoleWithWebIdentity events. Each event will contain a record of the Terraform Cloud run, and the name of the project and workspace.

This is a cut down cloudtrail event showing the key information:

{
    "userIdentity": {
        "type": "WebIdentityUser",
        "principalId": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io:aws.workload.identity:organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "userName": "organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "identityProvider": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io"
    },
    "eventTime": "2023-07-18T00:08:34Z",
    "eventSource": "sts.amazonaws.com",
    "eventName": "AssumeRoleWithWebIdentity",
    "awsRegion": "ap-southeast-2",
    "sourceIPAddress": "x.x.x.x",
    "userAgent": "APN/1.0 HashiCorp/1.0 Terraform/1.5.2 (+https://www.terraform.io) terraform-provider-aws/5.7.0 (+https://registry.terraform.io/providers/hashicorp/aws) aws-sdk-go-v2/1.18.1 os/linux lang/go/1.20.5 md/GOOS/linux md/GOARCH/amd64 api/sts/1.19.2",
    "requestParameters": {
        "roleArn": "arn:aws:iam::12121212121212:role/terraform-cloud-oidc-acces-TerraformDeploymentRole-NOPE",
        "roleSessionName": "terraform-run-abc123"
    },
    "responseElements": {
        "subjectFromWebIdentityToken": "organization:test-organization:project:Default Project:workspace:test-terraform-cloud:run_phase:plan",
        "assumedRoleUser": {
            "assumedRoleId": "CDE456:terraform-run-abc123",
            "arn": "arn:aws:sts::12121212121212:assumed-role/terraform-cloud-oidc-acces-TerraformDeploymentRole-NOPE/terraform-run-abc123"
        },
        "provider": "arn:aws:iam::12121212121212:oidc-provider/app.terraform.io",
        "audience": "aws.workload.identity"
    },
    "readOnly": true,
    "eventType": "AwsApiCall",
    "recipientAccountId": "12121212121212"
}

Links

• How to get rid of AWS access keys - Part 1: The easy wins
• Terraform Cloud - Dynamic Credentials with the AWS Provider
• AWS Partner Network (APN) Blog - Simplify and Secure Terraform Workflows on AWS with Dynamic Provider Credentials

So instead of using IAM User credentials, this approach uses IAM Roles and OpenID Connect to dynamically assign credentials to Terraform Cloud runs which is a big win from a security perspective!

So firstly what does automated security remediation for a cloud service do?

This is software which detects threats, more specifically misconfigurations of services, and automatically remediates problems.

How does automated security remediation work?

Typically, security remediation tools take a feed of events from a service such as AWS CloudTrail (audit logging service) and checks the configuration of the resources being modified. This is typically paired with regular scheduled scans to ensure nothing is missed in the case of dropped or missing events.

Can you use IAM to avoid security misconfigurations in the first place?

Cloud services, such as AWS, have fairly complex AWS Identity and Access Management (IAM) services which provide course grained security policy language called IAM policies. These policies are hard to fine tune for the myriad of security misconfigurations deployed by the people working on in these cloud services.

Everyone has seen something like the following administrator policy allowing all permissions for administrators of an AWS environments, this is fine for a “sandbox” learning account, but is far too permissive for production accounts.

Version: "2012-10-17"
Statement:
    - Sid: AdminAccess
    Effect: Allow
    Action: '*'
    Resource: '*'

That said, authoring IAM policies following the least privilege to cover current requirements, new services coming online and keeping up with emerging threats can be a significant cost in time and resources, and like at some point provide diminishing returns.

Can you use AWS service control polices (SCP) to avoid security misconfigurations?

In AWS there is another way to deny specific operations, this comes in the for of service control policies (SCP). These policies are a part of AWS Organizations and provide another layer of control above an account’s IAM policies, allowing administrators to target specific operations and protect common resources. Again, these are also very complex to configure and maintain, as they use the same course grained security layer.

Below is an example SCP which prevents any VPC that doesn’t already have internet access from getting it.

Version: '2012-10-17'
Statement:
- Effect: Deny
  Action:
  - ec2:AttachInternetGateway
  - ec2:CreateInternetGateway
  - ec2:CreateEgressOnlyInternetGateway
  - ec2:CreateVpcPeeringConnection
  - ec2:AcceptVpcPeeringConnection
  - globalaccelerator:Create*
  - globalaccelerator:Update*
  Resource: "*"

Investment in SCPs is important for higher level controls, such as disabling the modification of security services such as Amazon GuardDuty, AWS Config and AWS CloudTrail as changes to these services may result in data loss. That said, SCPs are still dependent on IAMs course grained policy language, which in turn is limited by the service’s integration with IAM.

A note about SCPs, often you will see exclusions for roles which enable administrators, or continuous integration and delivery (CI\CD) systems to bypass these policies. These should be used for exceptional situations, for example bootstrapping of services, or incidents. So, using these roles should be gated via some sort incident response process.

So why does automated security remediation exist?

Given the complexity of managing fine-grained security policies, organizations implement a more reactive solution, which is often in the form of automated security remediation services.

What are some of disadvantages of these automated security remediation tools?

• False positives and false negatives: They may generate false positives, where legitimate actions are flagged as security threats, or false negatives, where actual security issues are missed.
• Over-reliance on automation: Organizations may become over-reliant on tools, potentially leading to complacency or a lack of human oversight, which can create new risks and vulnerabilities.
• Limited scope: They may not be able to detect or remediate all types of security issues or vulnerabilities, especially those that are highly complex or require a more nuanced approach.
• Compliance and regulatory issues: Some compliance and regulatory frameworks may require manual security review or approval for certain types of security incidents, which can be challenging to reconcile with automated processes.
• Cultural resistance: Some organizations may experience cultural resistance to automated remediation, as it may be perceived as a threat to job security or the role of security professionals.
• Delayed or dropped trigger events: Automated remediation typically primarily depend on triggers from audit events provided, these events can be delayed in large AWS environments, or by a flood of activity.

What are some of the positive impacts automated remediation tools?

• Increased efficiency: Can reduce the time and resources required to respond to security incidents, allowing security teams to focus on higher-value tasks.
• Improved collaboration: Can help break down silos between different teams, as it often requires cross-functional collaboration between security, operations, and development teams.
• Reduced burnout: By automating repetitive and time-consuming tasks, automated remediation can help reduce burnout among security people, who may otherwise be overwhelmed by the volume of security incidents they need to respond to manually.
• Skills development: As organizations adopt these tools and processes, security teams may need to develop new skills and competencies in areas such as automation, scripting, and orchestration, which can have positive impacts on employee development and job satisfaction.
• Cultural shift towards proactive security: They can help shift the culture of security within an organization from reactive to proactive, by enabling security teams to identify and remediate potential security risks before they become actual security incidents.

Summary

Overall, while automated security remediation can have some cultural and productivity impacts that need to be managed, it can also bring significant benefits to organizations by enabling more efficient, collaborative, and proactive security practices.

That said, automated security remediation really needs to be part of a three-pronged approach:

1. Ensure people are working in cloud environments with only the privileges they require to do their work. There are of course exceptions to this, but they should be covered with a process which allows users to request more access when required.
2. SCPs should be used to protect security and governance services, and implement core restrictions within a collection of AWS accounts, depending on your business.
3. Automated security remediation should be used to cover all the edge cases, again this should be used only where necessary, and with the understanding it may take a period of time to fix.

One thing to note is we are working in an environment with a lot of smart and resourceful people, so organizations need to watch for situations complex workarounds evolve to mitigate ineffective or complex controls otherwise they may impact morale, onboarding of staff and overall success of a business.

Security works best when it balances threats and usability!

One thing to note is in my day job I work on an Apple Mac, but my personal machine is a Linux laptop running PopOS. I find using Linux as a desktop works as most software I use is web based or supported on linux. I also use it for IoT development as pretty much all the tool chains I use supports it.

On a whole over the years I have moved to a more minimal setup, primarily to keep things simple, less is easier to maintain, easier to share, and more likely to be adopted by others.

The stack I work with professionally is pretty varied, but can be summarized as:

• Amazon Web Services (AWS), I work primarily this cloud platform in my day job
• Cloudformation, native AWS infrastructure deployment
• Go, great language for building tools, apis, and backend services
• Python, used for cloud orchestration, scripting and machine learning
• NodeJS often using Typescript, for frontend development
• Git, used for all things source code

CLI Tools

I primarily use zsh as my shell, sticking to a pretty minimal setup tools wise.

• Docker for containers, which I mainly use for testing.
• direnv which is used to change environment settings in projects.
• The Silver Searcher a faster search tool for the cli, ag is my goto for locating stuff in files when developing.
• Git Hub CLI, makes working with GitHub from the CLI a dream.
• AWS CLI is used to write scripts and diagnosing what is up with my cloud.
• AWS SAM CLI for deploying cloudformation in a semi sane way.
• nvm, nodejs changes a lot so I often need a couple of versions installed to support both new and old software.
• Git Prompt for a dash more information in my shell about the current trees Git status.
• gnupg, which I mostly use for Signing of Git commits and software, and a bit of data encryption.

Most of my builds done using the good old Makefile so I always have make installed.

Editor

Currently I use vscode when developing, it is one of the first things I open each day. I was a vim user but moved to vscode as I prefer to use a more approachable editor, especially as I work with developers and “non tech” people and they find it less daunting to learn.

I am trying to help everyone code, so using an approachable editor is really helpful!

To support the stack I use the following plugins:

• Code Spell Checker, I really hate misspelling words in my code.
• EditorConfig for VS Code, handy way to keep things consistently formatted across editors when working in a team.
• GitLens — Git supercharged, helps me figure out what changed and who changed it without leaving my editor.
• Go, primary language I develop in.
• indent-rainbow, this addon keeps me sane when editing whitespace sensitive languages such as python and YAML!
• Python, tons of stuff uses this language so I always end up using it.
• vscode-cfn-lint, avoiding obvious errors and typos in my cloudformation templates saves a ton of time and frustration.
• TODO Highlight, I always try and add information and notes to my code, this helps highlight the important stuff.
• YAML, most of the tools I deploy with use it for configuration so I need a good linter.
• GitHub Theme, I use the dimmed dark mode which is really nice comfortable coding theme.

So why would you ever want to dive into AWS Billing data in the first place?

1. It is pretty easy for both novices, and experience developers to rack up a sizable bill in AWS, part of the learning experience is figuring out how this happened.
2. The billing data itself is available in parquet format, which is a great format to query and dig into with services such as Athena.
3. This billing data is the only way of figuring out how much a specific AWS resource costs, this again is helpful for the learning experience.
4. The Cost Explorer in AWS is great if you just want an overview, but having SQL access to the data is better for developers looking to dive a bit deeper.
5. The billing service has a feature which records created_by for resources, this is only available in the CUR data. If you have already you can enable it via Cost Allocation Tags.

These points paired with the fact that a basic understanding of data wrangling in AWS is an invaluable skill to have in your repertoire.

Suggested CUR Solution

I have put together an automated solution which uses AWS CloudFormation to create a Cost and Usage Reports (CUR) in your billing account with a Glue Table enabling querying of the latest data for each month in Amazon Athena. This project is on github at https://github.com/wolfeidau/aws-billing-store, follow the README.md to get it setup.

In summary it deploys:

1. Creates the CUR in the billing service and the bucket which receives the reports.
2. Configures a Glue Database and Table for use by Athena.
3. Deploys a Lambda function to manage the partitions using Amazon EventBridge S3 events.

Once deployed all you need to do is wait till AWS pushes the first report to the solution, this can take up to 8 hours in my experience, then you should be able to log into Athena and start querying the data.

One thing to note is designed to be a starting point, I have released it under Apache 2.0 license so your welcome to pull this solution apart and integrate it into your environment.

Next Steps

To test the solution you can start with a query which shows you AmazonS3 costs grouped by bucket name and aggregated using sum.

SELECT line_item_resource_id as bucket_name,
	round(sum(line_item_blended_cost), 4) AS cost,
	month
from "raw_cur_data"
WHERE year = '2022'
	and month = '7'
	AND line_item_product_code = 'AmazonS3'
GROUP BY line_item_resource_id,
	month
ORDER BY cost DESC;

There are some great resources with other more advanced queries which provide insights from your CUR data, one of the best is Level 300: AWS CUR Query Library from the The Well-Architected Labs website.

The standout queries for me are:

1. Amazon GuardDuty - This query provides daily unblended cost and usage information about Amazon GuardDuty Usage. The usage amount and cost will be summed.
2. Amazon S3 - This query provides daily unblended cost and usage information for Amazon S3. The output will include detailed information about the resource id (bucket name), operation, and usage type. The usage amount and cost will be summed, and rows will be sorted by day (ascending), then cost (descending).

Cost Allocation Tags

The Cost Allocation Tags in billing allows you to record data which is included in the CUR. This is a great resource for attributing cost to a user, role or service, or alternatively a cloudformation stack.

I enable the following AWS tags for collection and inclusion in the CUR.

• aws:cloudformation:stack-name
• aws:createdBy

I also enable some of my own custom tags for collection and inclusion in the CUR.

• application
• component
• branch
• environment

You can see how these are added in the https://github.com/wolfeidau/aws-billing-store project Makefile when the stacks are launched.

A supply chain attack targets less secure parts of the development process, this could be the tools and services you depend on, or the docker containers you host your software in. These attacks come in different forms but some examples are:

• Extract credentials from your CI services like the Codecov security incident.
• Seed malware for an attack down stream on your customers like the solarwinds breach.

In this post I am going to dive into an example of an attack that affected a lot of projects using GitHub Actions recently, but this could be applied more broadly to any CI tool or service relying on third party services or code.

Why is the Codecov security incident interesting?

The Codecov security incident illustrates a novel attack on a popular developer tool, which in turn exposed a number of CI integrations including the widely used GitHub Actions.

The initial attack happened in January when the Codecov Bash uploader script was modified in a cloud storage service.

This script provides a language-agnostic alternative for sending your coverage reports to Codecov and is used at least 5 of the Codecov continuos integration (CI) integrations.

The GitHub Action Codecov was one of them, it downloads and executes the script each time it is run and critically didn’t verify the checksum of this file against the release so it continued working.

The modified script extracted all environment variables in that workflow and uploaded them to a website operated by the attacker.

These variables are often used to pass credentials into the workflow for services such as Docker Hub, NPM, cloud storage buckets and other software distribution services.

The extraction of these credentials while this exploit was active could lead to the modifications of builds and other artifacts resulting in further exploits and extending the footprint of this attack.

Most concerning is this exploit was effectively sitting in the supply chain of 1000s of open source and proprietary workflows extracting data undetected for approximately 4 months.

Note: It is worth reading the security update posted by Codecov as it highlights some of the steps you need to take if you are effected by this sort of attack.

What can you do to mitigate these sorts of attacks?

To ensure your GitHub actions secure I recommend:

• Read the GitHub actions hardening documentation.
• Limit exposure of secrets to only the projects and repositories which need these values by implementing least privilege for secrets in GitHub Actions.
• Read about Keeping your GitHub Actions and workflows secure: Untrusted input.
• Regularly rotate the credentials used in your GitHub actions. this helps mitigate historical backups or logs being leaked by a service.
• If an action is supplied and supported by a vendor, ensure emails or advisories are sent to a shared email box, and not attached to a personal email. This will enable monitoring by more than one person, and enable you to go on holidays.

For actions which have access to important secrets, like those used to upload your software libraries and releases, or deploying your services, you may want to fork them and add security scanning. This is more important if there are no vendor supported alternatives, or it is a less widely supported technology.

Reviewing your actions

Given we all still want the benefits of services such as GitHub Actions while also managing the risks we need to maintain a balance between getting the most out of the service and limiting possible exploits.

The first step is to review the GitHub actions your using in your workflows, just like you would for open source libraries:

• How active are these projects? Are PRs merged / reviewed in a timely manner?
• Is the author known for building good quality software and build automation?
• Are these actions supported by a company or service?
• Does the project have a security policy?

When it comes to open source GitHub Actions you need to be aware that most open source licenses limit liability for the author, this means you as a consumer need to actively manage some of the risks running this software. Performing maintenance, bug fixes and contributing to the upkeep of the software is key to ensuring the risk of exploits is minimized.

Lastly run some internal training or workshops around supply chain attacks in your company, this could involve running a scenario like the Codecov incident as a table top exercise.

Case in point, the AWS CLI which a large number of engineers and developers rely on every day, the following command will create a bucket.

$ aws s3 mb s3://my-important-data

One would assume this commonly referenced example which is used in a lot of the resources provided by AWS would create a bucket following the best practices. But alas no…

The configuration which is considered best practice for security of an S3 bucket missing is:

• Enable Default Encryption
• Block Public access configuration
• Enforce encryption of data in transit (HTTPS)

Why is this a Problem?

I personally have a lot of experience teaching developers how to get started in AWS, and time and time again it is lax defaults which let this cohort down. Of course this happens a lot while they are just getting started.

Sure there are guard rails implemented using services such as AWS Security Hub, pointing out issues left right and center, but these typically identity problems which wouldn’t be there in the first place if the tools where providing better defaults.

Sure there is more advanced configuration but encryption and blocking public access by default seem like a good start, and would reduce the noise of these tools.

The key point here is it should be hard for new developers to avoid these recommended, and recognised best practices when creating an S3 bucket.

In addition to this, keeping up with the ever growing list of “best practice” configuration is really impacting both velocity and morale of both seasoned, and those new the platform. Providing some tools which help developers keep up, and provide some uplift when upgrading existing infrastructure would be a boon.

Now this is especially the case for developers building solutions using serverless as they tend to use more of the AWS native services, and in turn trigger more of these “guard rails”.

Lastly there are a lot of developers out there who just don’t have time to “harden” their environments, teams who have no choice but to ignore “best practices” and may benefit a lot from some uplift in this area.

What about Cloudformation?

To further demonstrate this issue this is s3 bucket creation in cloudformation, which is the baseline orchestration tool for building resources, provided free of charge by AWS. This is a very basic example, as seen in a lot of projects on GitHub, and the AWS cloudformation documentation.

      MyDataBucket:
        Type: AWS::S3::Bucket
          Properties:
            BucketName: MyDataBucket

Now you could argue that cloudformation is doing exactly what you tell it to do, it is just a primitive layer which translates YAML or JSON into API calls to AWS, but I think again this is really letting developers down.

Again this is missing default encryption, and public access safe guards. Now in addition to this a lot of quality tools also recommend the following:

• Explicit deny of Delete* operations, good practice for systems of record
• Enable Versioning, optional but good practice for systems of record
• Enable object access logging, which is omitted it to keep the example brief

So this is a basic example with most of these options enabled, this is quite a lot to fill in for yourself.

     MyDataBucket:
        Type: AWS::S3::Bucket
        DeletionPolicy: Retain
        UpdateReplacePolicy: Retain
        Properties:
          BucketName: !Ref BucketName
          BucketEncryption:
            ServerSideEncryptionConfiguration:
              - ServerSideEncryptionByDefault:
                  SSEAlgorithm: AES256
          VersioningConfiguration:
            Status: Enabled
          PublicAccessBlockConfiguration:
            BlockPublicAcls: True
            BlockPublicPolicy: True
            IgnorePublicAcls: True
            RestrictPublicBuckets: True
    
      MyDataBucketPolicy:
        Type: AWS::S3::BucketPolicy
        Properties:
          Bucket: !Ref MyDataBucket
          PolicyDocument:
            Id: AccessLogBucketPolicy
            Version: "2012-10-17"
            Statement:
              - Sid: AllowSSLRequestsOnly
                Action:
                  - s3:*
                Effect: Deny
                Resource:
                  - !Sub "arn:aws:s3:::${MyDataBucket}/*"
                  - !Sub "arn:aws:s3:::${MyDataBucket}"
                Condition:
                  Bool:
                    "aws:SecureTransport": "false"
                Principal: "*"
              - Sid: Restrict Delete* Actions
                Action: s3:Delete*
                Effect: Deny
                Principal: "*"
                Resource: !Sub "arn:aws:s3:::${MyDataBucket}/*"

To do this with the AWS CLI in one command would require quite a few flags, and options, rather than including that here I will leave that exercise up to the reader.

Now some may say this is a great opportunity for consulting companies to endlessly uplift customer infrastructure. But this again begs the questions:

1. Why is this the case for customers using the recommended tools?
2. What about developers getting started on their first application?
3. Wouldn’t be better to have these consultants building something new, rather than crafting reams of YAML?

Why Provide Resources which are Secure by Default?

So I have used S3 buckets as a very common example, but there is an ever growing list of services in the AWS that I think would benefit from better default configuration.

Just to summarise some of the points I have made above:

1. It would make it harder for those new to the cloud to do the wrong thing when following examples.
2. The cost of building and maintaining infrastructure would be reduced over time as safer defaults would remove the need for pages of code to deploy “secure” s3 buckets.
3. For new and busy developers things would be mostly right from the beginning, and likewise update that baseline even just for new applications, leaving them more time to do the actual work they should be doing.

So anyone who is old enough to remember Sun Solaris will recall the “secure by default” effort launched with Solaris 10 around 2005, this also came with “self healing” (stretch goal for AWS?), so security issues around defaults is not a new problem, but has been addressed before!

Follow Up Q&A

I have added some of the questions I received while reviewing this article, with some answers I put together.

Will CDK help with this problem of defaults?

So as it stands now I don’t believe the default s3 bucket construct has any special default settings, there is certainly room for someone to make “secure” versions of the constructs but developers would need to search for them and that kind of misses the point of helping wider AWS user community.

Why don’t you just write your own CLI to create buckets?

This is a good suggestion, however I already have my fair share of side projects, if I was to do this it would need to be championed by a orginisation, and team that got value from the effort. But again, needing to tell every new engineer to ignore the default AWS CLI as it isn’t “secure” seems to be less than ideal, I really want everyone to be “secure”.

How did you come up with this topic?

Well I am currently working through “retrofitting” best practices (the latest ones) on a bunch of aws serverless stacks which I helped build a year or so ago, this is when I asked the question why am I searching then helping to document what is “baseline” configuration for s3 buckets?!

Won’t this make the tools more complicated adding all these best practices?

I think any uplift at all would be a bonus at the moment, I don’t think it would be wise to take on every best practice out there, but surely the 80/20 rule would apply here. Anything to reduce the amount of retro fitting we need to do would be a good thing in my view.

To start out it is helpful to have an overview, this post and the associated talk Moving to event-driven architectures (SVS308-R1) are a good place to start.

• Tim Bray - Eventing Facets

Then for those that want to see some code, take a look at the analytics component in this project developed by the serverless team at AWS, there are tons of great infra examples in this project. Although the code is a bit complex there is a lot to garner even if your not a Java developer.

• awslabs/realworld-serverless-application

This project uses a great reusable component which takes a AWS DynamoDB stream and publishes it onto AWS Eventbridge, again if Java isn’t your language of choice there are still some gems in here, such as the logic used to retry submission of events to Eventbridge.

• awslabs/aws-dynamodb-stream-eventbridge-fanout

From the AWS Samples comes this project which is worth digging into, it has a bunch of simple examples with diagrams which are always a plus.

• aws-samples/aws-serverless-ecommerce-platform

To enable some experimentation and development this CLI tool is pretty handy.

• spezam/eventbridge-cli

As I go I will add links and happy to take suggestions.

• Having everything in one folder results in a lot of inter dependencies in the code.
• Reuse outside the project can be difficult as the code is only designed to be used in one package.
• It is impossible to have more than one binary, as you can have only one main method.

This post will provide an overview of the structure I follow in my Go projects when building web services.

Note: If your just building a library to use in your services, or share with others, it is OK to put everything in the base folder of your project, an example of this is my dynastore library.

/cmd

This folder contains the main application entry point files for the project, with the directory name matching the name for the binary. So for example cmd/simple-service meaning that the binary we publish will be simple-service.

/internal

This package holds the private library code used in your service, it is specific to the function of the service and not shared with other services. One thing to note is this privacy is enforced by the compiler itself, see the Go 1.4 release notes for more details.

/pkg

This folder contains code which is OK for other services to consume, this may include API clients, or utility functions which may be handy for other projects but don’t justify their own project. Personally I prefer to use this over internal, mainly as I like to keep things open for reuse in most of projects.

Project Structure

As you build out your project there are some very important goals you should consider when it comes to how you structure your packages:

• Keep things consistent
• Keep things as simple as possible, but no simpler
• Loosely couple sections of the service or application
• Aim to ensure it is easy to navigate your way around

Overall when getting started you should experiment a bit, try a few different ideas when building out your first and get some feedback on based on the above goals.

The number one objective is to you build easy to maintain, consistent and reliable software.

Example

I recommend taking a look at exitus to see how I structure my projects, most of the code is under the pkg folder with each sub folder having one or more files. From the top level it is pretty clear what each package relates to, and although lean on tests it has a few examples.

$ tree exitus/
 exitus/
├── cmd
│   ├── authtest
│   │   └── main.go
│   ├── backend
│   │   └── main.go
│   └── client
│       └── main.go
├── dev
│   ├── add_migration.sh
│   └── docker-compose.yml
├── Dockerfile
├── go.mod
├── go.sum
│   ├── 20190721131113_extensions.down.sql
│   ├── 20190721131113_extensions.up.sql
│   ├── 20190723044115_customer_projects.down.sql
│   ├── 20190723044115_customer_projects.up.sql
│   ├── 20190726175158_issues.down.sql
│   ├── 20190726175158_issues.up.sql
│   ├── 20190726201649_comments.down.sql
│   ├── 20190726201649_comments.up.sql
│   ├── bindata.go
│   ├── gen.go
│   ├── migrations_test.go
│   └── README.md
├── pkg
│   ├── api
│   │   ├── exitus.gen.go
│   │   ├── exitus.yml
│   │   └── gen.go
│   ├── auth
│   │   ├── scopes.go
│   │   └── user.go
│   ├── conf
│   │   ├── conf.go
│   │   └── conf_test.go
│   ├── db
│   │   ├── db.go
│   │   ├── dbtesting.go
│   │   ├── migrate.go
│   │   ├── sqlhooks.go
│   │   └── transactions.go
│   ├── env
│   │   └── env.go
│   ├── healthz
│   │   ├── healthz.go
│   │   └── healthz_test.go
│   ├── jwt
│   │   └── jwt.go
│   ├── metrics
│   │   └── metrics.go
│   ├── middleware
│   │   ├── jwt.go
│   │   └── middleware.go
│   ├── oidc
│   │   └── client.go
│   ├── server
│   │   ├── reflect.go
│   │   └── server.go
│   └── store
│       ├── comments.go
│       ├── comments_test.go
│       ├── customers.go
│       ├── customers_test.go
│       ├── issues.go
│       ├── issues_test.go
│       ├── migrate_test.go
│       ├── projects.go
│       ├── projects_test.go
│       └── store.go
└── README.md

The aim here is to illustrate how you grow your project from a couple of files, to a larger web service. I encourage you to trawl through github projects and dig into how other developers have structured theirs, and most of all try it out yourself!

References

• GopherCon EU 2018: Peter Bourgon - Best Practices for Industrial Programming
• Standard Go Project Layout
• GopherCon 2018: Kat Zien - How Do You Structure Your Go Apps

Before you start you will need to install Go, I recommend using homebrew or for ubuntu users Golang Backports, or as last resort grab it from the Go Downloads page.

So this looks like this for OSX.

brew install go

Or for ubuntu we add the PPA, then install golang 1.14 and update our path.

sudo add-apt-repository ppa:longsleep/golang-backports
sudo apt-get update
sudo install golang-1.14
echo 'export PATH=$PATH:/usr/lib/go-1.14/bin' >> ~/.bashrc
source ~/.bashrc

Now we should be able to run.

go version

Now navigate to where you build your projects, for me this is ~/Code/goprojects and make a folder. One thing to note here is that this goprojects folder is not in my $GOPATH as we are using modules.

cd ~/Code/goprojects
mkdir simple-go-service
cd simple-go-service

Before we start adding code lets initiliase our project, you should replace USERNAME with your github username, for me it is wolfeidau.

go mod init github.com/USERNAME/simple-go-service

Now for me I follow a pattern of storing the entry point in a cmd folder, this is done so I can easily customise the name of the binary as go uses the parent folder name executables.

mkdir -p cmd/simple-service
touch cmd/simple-service/main.go

Now add some code to the main.go you created in the previous command, this will listen on port :8000 for web requests.

package main

import (
	"io"
	"log"
	"net/http"
)

func main() {
	// Hello world, the web server

	helloHandler := func(w http.ResponseWriter, req *http.Request) {
		io.WriteString(w, "Hello, world!\n")
	}

	http.HandleFunc("/hello", helloHandler)
    log.Println("Listing for requests at http://localhost:8000/hello")
	log.Fatal(http.ListenAndServe(":8000", nil))
}

Now run this with the following command.

go run cmd/simple-service/main.go

This should print out a URL you can navigate to in your browser and see the classic Hello, world!.

Now from here you will want to setup an editor, I personnally use vscode which has really good support for golang once you add the go plugin.

From here I would recommend looking at something like echo which is a great web framework, it is well documented and has lots of great examples.

To add this library either just add the import in your editor, vscode will automatically trigger a download of libraries or run.

go get -u -v github.com/labstack/echo/v4

NOTE: We are using the v4 tagged import as we are using Go Modules, also I have also ensure this v4 tag is in the imports in the following example.

And update the main.go to use this great little REST crud example.

package main

import (
	"net/http"
	"strconv"

	"github.com/labstack/echo/v4"
	"github.com/labstack/echo/v4/middleware"
)

type (
	user struct {
		ID   int    `json:"id"`
		Name string `json:"name"`
	}
)

var (
	users = map[int]*user{}
	seq   = 1
)

//----------
// Handlers
//----------

func createUser(c echo.Context) error {
	u := &user{
		ID: seq,
	}
	if err := c.Bind(u); err != nil {
		return err
	}
	users[u.ID] = u
	seq++
	return c.JSON(http.StatusCreated, u)
}

func getUser(c echo.Context) error {
	id, _ := strconv.Atoi(c.Param("id"))
	return c.JSON(http.StatusOK, users[id])
}

func updateUser(c echo.Context) error {
	u := new(user)
	if err := c.Bind(u); err != nil {
		return err
	}
	id, _ := strconv.Atoi(c.Param("id"))
	users[id].Name = u.Name
	return c.JSON(http.StatusOK, users[id])
}

func deleteUser(c echo.Context) error {
	id, _ := strconv.Atoi(c.Param("id"))
	delete(users, id)
	return c.NoContent(http.StatusNoContent)
}

func main() {
	e := echo.New()

	// Middleware
	e.Use(middleware.Logger())
	e.Use(middleware.Recover())

	// Routes
	e.POST("/users", createUser)
	e.GET("/users/:id", getUser)
	e.PUT("/users/:id", updateUser)
	e.DELETE("/users/:id", deleteUser)

	// Start server
	e.Logger.Fatal(e.Start(":1323"))
}

Now run this with the following command.

go run cmd/simple-service/main.go

Hopefully you have managed to get this service running and started testing it with something like postman 🎉.

For next steps I recommend reading How do I Structure my Go Project?.

Starting Out

Probably the lion share of lessons, at least initially were learnt about how to setup a custom RC car, especially the power and drive train. I have put together a bunch of recommendations:

1. Always buy at least two servos, and check the number of teeth, force, and dimensions carefully.
2. Learn to solder bullet connectors, there are some good tutorials on youtube including on how to get the nice shrink wrapped look.
3. Always use a seperate power supply for the computer on your car, there are a number of reasons you can cause voltage drops during operation of the car, keeping things isolated will avoid the computer crashing and a car running into a wall.
4. Cable ties are a life saver when hiding or stashing cables, buy a good collection of sizes.
5. Start out with brushed motors, they are cheap and work better at low speeds.

The Car Build

For the base of my current build I chose the WLToys A979 4WD Truck, this was for a few reasons:

1. Same base as the original AWS Deepracer.
2. Good solid build and lots of spares available online.
3. Short wheelbase and good turning circle so great for use indoors.
4. Cheap and easy to purchase and ship to Australia.
5. Has a brushed motor which is great at low speeds, which is very important indoors and while learning to drive.

You can buy it from banggood for approx 90 AUD.

Then I replaced the Electronic Speed controller(ESC) with a Turnigy 20A BRUSHED ESC as the one provided was integrated with the radio and servo using some sort of proprietary setup.

Lastly I replaced the servo with a Turnigy™ S3101S Servo 2.4kg / 0.14sec / 17g 24T.

To connect all this stuff you will need some extra parts.

• Heat shrink such as Turnigy Heat Shrink Tube 4mm Black.
• Bullet connectors such as 4mm Easy Solder Gold Connectors (10 pairs).
• XT60 connectors which are used to connect 2s batteries such as Nylon XT60 Connectors Male/Female (5 pairs).

This will also be useful with any power upgrades such as the one listed down the bottom.

The Compute Build

To mount the compute on top of the car I used Dingo-Mount which was cut on a laser cutter at CCHS using some 3mm ply wood, which is light and strong.

The hardware I use on top of the car is as follows:

• Jetson Nano which you can buy from seeed studio.
• Intel 8265.NGWMG.DTX1 DUAL BAND Wireless-AC 8265 Desktop KIT.
• Raspberry Pi Camera Module V2 also buy from from seeed studio.
• An SSD to store all your data on, and because running this machine off sdcard is very slow. I used a Crucial BX500 240GB 3D NAND SATA 2.5-inch SSD.
• A USB to SATA cable such as Onvian USB 3.0 To SATA 22 Pin 15+7 Pin 2.5 Inch Hard Disk Driver SSD Adapter Data Power Cable.
• A USB powerbank such as 18W Power Bank, ROMOSS 20000mAh Portable Charger.

I used some brass standoffs, like these M2.5 Male Female Hex Brass Spacer Standoff which are similar to the ones you get when you buy a computer case, this enabled me to mount the SSD under my jetson nano, with some rubber acting avoiding the bottom of the board shorting out on the SSD.

To drive the servo and ESC I used Adafruit PCA9685 16-Channel Servo Driver which you can get from Core Electronics or purchase something similar from aliexpress.

Power Options

If you want to use the full power of the Jetson Nano compute then you may need to get your self a SBEC such as Turnigy 5A (8-40v) SBEC for Lipo and rig up an extra 2S1P LIPO such as Turnigy nano-tech 2200mah 2S 25~50C Lipo Pack. I use this setup currently to avoid drop outs when doing inference with higher resolution images (224px * 224px). Attached to this is a 2.1mm Barrel Cable I had laying around.

Driving Demonstration

This is my car in auto pilot running around a tight track in the CCHS workshop.

When setting up an applications authentication I try to keep in mind a few goals:

1. Keep my users data as safe as possible.
2. Try and find something which is standards based, or supports integrating with standard protocols such as openid, oauth2 and SAML.
3. Evaluate the authentication flows I need and avoid increasing scope and risk.
4. Try to use a service to start with, or secondarily, an opensource project with a good security process and a healthy community.
5. Limit any custom development to extensions, rather than throwing out the baby with the bath water.

As you can probably tell, my primary goal is to keep authentication out of my applications, I really don’t have the time or inclination to manage a handcrafted authentication solution.

What does AWS Cognito provide out of the box?

Lets look at what we get out of the box:

• Storing and protecting your users data with features such as Secure Remote Password Protocol (SRP).
• Signing up for an account with email / sms verification
• Signing in, optionally with Multi Factor Authentication (MFA)
• Password change and recovery
• A number of triggers which can be used to extend the product

What is great about Cognito?

Where AWS Cognito really shines is:

• Out of the box compliance to standards such as SOC and PCI
• Integration with a range of platform identity providers, such as Google, Apple and Amazon
• Support for integration with identity providers (IdPs) using OpenID and SAML.
• Really easy to integrate into your application using libraries such as AmplifyJS.
• AWS is managing it for a minimal cost

What is not so great about Cognito?

Where AWS Cognito can be a challenge for developers:

• Can be difficult to setup, and understand some of the settings which can only be updated during creation, changing these requires you to delete and recreate your pool.
• Per account quotas on API calls
• A lack of search
• No inbuilt to backup and restore the user data in your pool

So how do we address some of these challenges, while still getting the value provided and being able to capitalise on it’s security and compliance features.

What is the best way to setup Cognito?

To setup Cognito I typically use one of the many open source cloudformation templates on GitHub. I crafted this template some time ago cognito.yml, it supports login using email address, and domain white listing for sign ups.

As a follow on from this I built a serverless application serverless-cognito-auth which encapsulates a lot of the standard functionality I use in applications.

You can also use AWS Mobile Hub or AWS Amplify to bootstrap a Cognito pool for you.

Overall recommendations are:

1. If your new to Cognito and want things to just work then I recommend trying AWS Amplify.
2. If you are an old hand and just want Cognito the way you like it, then use one of the many prebuilt templates.

How do I avoid quota related issues?

Firstly I recommend familiarising yourself with the AWS Cognito Limits Page.

I haven’t seen an application hit request rate this more than a couple of times, and both those were related to UI bugs which continuously polled the Cognito API.

The one limit I have seen hit is the sign up emails per day limit, this can be a pain on launch days for apps, or when there is a spike in sign ups. If your planning to use Cognito in a startup you will need to integrate with SES.

How do I work around searching my user database?

Out of the box cognito will only allow you to list and filter users by a list of common attributes, this doesn’t include custom attributes, so if you add an attribute like customerId you won’t be able to find all users with a given value.

This limitation makes it difficult to replace an internal database driven authentication library just using the cognito service, so for this reason I recommend adding a DynamoDB table to your application and integrating this with cognito using lambda triggers to build your own global user store.

To simplify interacting with Cognito I wrote a CLI which provides some helpful commands to scan, filter, export and perform some common admin functions, you can find it at https://github.com/wolfeidau/cognito-cli.

How do I back up my user pool?

Backing up user accounts in any system is something you need to consider carefully as this information typically includes credentials as well as other sensitive data such as mobile number which is often used as a second factor for other services.

Currently Cognito doesn’t provide a simple way of exporting user data, the service does however have an import function which will import users in from a CSV file.

Note: AWS cognito doesn’t support export user passwords, these will need to be reset after restore.

For some examples of tooling see cognito-backup-restore.

Conclusion

If you really care about security and compliance then cognito is a great solution, it has some limitations, and gaps but these can be worked around if you want to focus your effort somewhere else.

Personally I think it is really important that as a developer I pick solutions which ensure my customers data is locked away as securely as possible, and ideally using a managed service.

You could totally roll your own authentication solution, and manage all the patching and challenges which go with that but that makes very little sense when you should probably be solving the original problem you had.

Authentication is a yak I am willing to let someone else shave manage, and so should you, if not for your own sanity, then that of your users.

Lastly if your building out a web application use amplify-js, this library makes it so easy to add Cognito authentication to your web application. I used it on cognito-vue-bootstrap which you can also check out.

The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation.

Before I go recommending this new project to anyone I most certainly need to road test it myself. This post provides a bit of background on where I work, and why I am looking into CDK, and what I love to see in the future.

Background

I have been building applications in Amazon Web Services (AWS) for a number of years using as many of the services as possible to keep things lean and online, that said it doesn’t come without some overhead and many lessons learned. While working in AWS I have always chosen to stick to native tools, such as Cloudformation, augmented by a range of deployment tools, this has means I get all the power with the inherent complexity, which grows every AWS reinvent conference.

Hiring into an organization which works very closely with AWS comes with some challenges. New hires will typically find themselves learning a lot of new services, while also grappling with Cloudformation. This can really impact a new team members productivity, and more importantly their confidence, especially when the first few PR reviews call out security issues, and subtle pitfalls around resource naming the examples they build find on the internet.

For this reason I have been looking at ways to reduce risk of issues in production without falling into the trap of isolating infrastructure development to a small number of “experts”, this is why CDK popped up on my radar. It promises to allow developers to manage to assemble stacks using reusable patterns either developed by AWS, or internally using code not YAML, which in my view is a big plus.

In short I care more about people than I care about technology, I want it to empower those who use it, not hold them back.

Road Testing

As I was starting to road test CDK I was fortunate enough to catch up with some of my peers from AWS Partner Community and get some good tips and anecdotes on what to dig into. Based on this I have put together some points, these are grouped into the good, the bad and the ugly.

The Good

• CDK enables developers to describe their infrastructure in code using an object model, then lets them synthesize it into Cloudformation templates.
  • VPC resources can be “connected” to each other, this automatically creates the required security groups, and entries in them.
  • Accessing a secret value will also update IAM policies, updating roles with the required policy changes.
• CDK automatically creates policies that narrow access down to the least privileges required to operate based on your model. This is a boon as because it is one of the most complex and time consuming aspects of crafting a Cloudformation template.
• The Cloudformation produced by CDK has sane defaults such as:
  • Enables deletion protection for Relational Database Service (RDS) instances to avoid accidental deletion during stack updates.
  • Enables orphaning of S3 buckets which leaves them behind when a stack update occurs, therefore avoiding deletion of all your data when messing with configuration of a resource in your stack.
• Includes patterns which incorporate a range of best practices, helpers and security enhancements.
  • An example of this is the LoadBalancedFargateService which can deploy a build and deploy a local service using a Dockerfile without ever having to delve into the finer points of Elastic Container Registry (ECR), Elastic Container Service (ECS) or Application Load balancers (ALB).
• Personally I feel a lot more productive with CDK, I am writing less code and producing more secure, consistent infrastructure.

The Bad

• Although amazing, the patterns feel like black boxes, there is no way to click through into the source code of an underlying pattern and dig into how it works.
  • Personally I think these should illustrate how amazing this model is, and act as a spring board into developing your own modules, currently it feels like a black box.
  • Yes I can clone and dig into repositories but the whole point of this is to be here for a good time, not a long time.
• It is really difficult to lock down the version of CDK in your NodeJS projects once a new release has come out. If there are changes I want to skip then I have to get a lock file from an older project, which breaks as soon as I add other CDK modules.
  • This is a less than ideal user experience for teams who aren’t moving as fast as the CDK development team.
  • Note work is happening to sort out semver usage in cdk packages issue #3711 which is great!
• The whole multi language, cross compiled thing seems very limited at the moment, especially around the lack of support for sharing modules developed in languages other than Typescript.
  • For more information on how CDK deliver polyglot libraries checkout the aws/jsii project on GitHub
  • Some background on Python experience requiring NodeJS tooling.

The Ugly

• In the current CDK I am encouraged, if not required to synthesize my templates in every AWS account I use, this is a big red flag for me.
  • If team member updates a service deployed a couple of month after it’s initial release there is NO guarantee the same code Cloudformation will be generated. To cover this operators will need to “stash” or archive templates for every account, before every deploy.
  • The NPM locking issues around pinning upgrades really restricts your power to ensure managed changes to Cloudformation.

This lack of reusable, easily reproducible artifacts is a bit of a show stopper for me, given the number of times I have been let down by tools which generate Cloudformation, I am loath to leap back into it for a production environment.

Summary

In short I will not be putting CDK between me and a production environment until some of the reproducible challenges are addressed. Like many of my peers, I have always advocated for solid, reproducible infrastructure tooling that is as simple as possible to recover and rollback.

That said I will most definitely be using CDK to quickly generate Cloudformation, especially generating IAM policies with least privileged, and harvesting some of the great ideas, and tricks from the patterns.

I would recommend using Typescript to develop CDK scripts, this will ensure you get the most reuse and enable harvesting directly from the CDK patterns!

Contributors

Thanks to Ashish Rajan @hashishrajan and Rowan Udell @elrowan for reviewing this post, Ian Mckay @iann0036 for starting a impromptu CDK discussion in Seattle and Aaron Walker @aaronwalker for being a great sounding board and walking me through some of his experience with CDK.

Example Project

My current work with CDK is mainly focused on providing infrastructure to a container based application called exitus which is hosted on GitHub with the CDK infra code exitus.ts. Ian Mckay

• complex customer on boarding processes jobs which provision resources then send a welcome email
• billing jobs where you may need wait for payment authorisation
• provisioning users and setup of any resources each user may need

pipeline

In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, functions, etc.), arranged so that the output of each element is the input of the next; the name is by analogy to a physical pipeline.

The term pipeline is used a lot in building of software, but can refer to any chain of tasks.

Over the last couple of years I have used Step Functions in a range of business applications, initially with mixed success due to service limitations and trying to fit complex “new requirements” into the model. Over time this changed as I better understand where step functions start and end.

I have put together a list of tips and recommendations for those using step functions.

Start Small

Practice with a few small workflows to get started, avoid building a Rube Goldberg machine. This means starting with something you already know and refactoring it to incorporate a step function, get used to tracing issues and make sure you have all the tools and experience to operate a serverless application.

Track Executions

Include a correlation id in all flow execution payloads, this could be seeded from Amazon correlation id included with all API gateway calls. This correlation id may be used for reruns of the state machine so don’t use it as the execution name.

Naming Things

Execution name should include some hints to why the flow is running, with a unique id or timestamp appended.

Step names should clearly indicate what this step does as this will enable devs or operations identify where errors or mistakes are occurring.

Exception Handling

When using Lambda functions make sure you use an exception tracker such as bugsnag or sentry to make fault finding easier. This allows you to track issues over time and avoids sifting through logs looking for errors.

Use the retry backoff built into each step to make your flows more robust.

Logging

Emit structured logs with key start and end information and use cloudwatch to search capture metrics, and trigger alerts based on them.

Infrastructure Automation

As an infrastructure engineer I use Step Functions to build and deploy a number of different applications, this is mainly where:

1. The task happens often
2. Someone owns the infrastructure and integration is required to orchestrate with external systems
3. There are a lot of “services” of a similar shape which need to be deployed the same way

When using cloudformation make sure you use change sets, this will allow you to:

1. Print a nice list of what will change before performing a change or create.
2. Rollback in a nicer way

When cloudformation changes fail try to collect the tail of the execution events to simplify fault finding.

When designing flows make sure they aren’t too generic, their structure should reflect what your automating, similar to an ansible playbook.

Build up a task modules to interface with cloudformation, and whenever possible just use that with custom cloud formation tasks.

Build up a library of common tasks, which can be used by lambdas. Test these thoroughly using unit and integration tests.

Use common sense when managing common code, don’t dump everything in there, keep to just the most important tasks. This just results in a teams or systems having a massive boat anchor holding back, and contributing to the fragility of the entire platform.

In the serverless space AWS Step Functions play a similar role to projects such as delayed job or resque in ruby, celery in python, but with the following differences:

• Built on a flexible flow definition language called Amazon States Language which is written in JSON
• Powered by lambda, with native integration to SNS, SQS, Kinesis and API Gateway
• Fully Managed by AWS

Step Functions?

AWS Step Functions provides a way of executing flows you have defined, and provides a visual representation, like a CI pipeline, showing the current state of the execution.

A simple task management example which polls the status of a task, and reports completion status can be seen as follows:

Why Step Functions?

So this is great but why should we decompose our workflows into functions and glue them together using a managed service?

There are a number of things to be gained by moving to Step Functions:

1. Testing, you will be able to test each element in the chain and make sure it performs it’s discreet task.
2. Decoupling, you will have broken things down into pieces of code which can be refactored, or replaced independent of each other.
3. Monitoring, given the visual nature of these pipelines you will be able to zero in on failures faster.

Step Functions aren’t the answer to every problem, but for multi step, long running jobs they are a great solution, if your fluent in the AWS ecosystem.

In this post I will show how to use the DeepRacer for ROS development.

So why would I use DeepRacer for ROS development?

1. Intel based so very easy to build and upload binaries / projects.
2. It works out of the box and is very well designed.
3. It will retail for 249 USD in march, given it’s high quality components it will be very useful.
4. It is based on Ubuntu 16.04.3, and ROS Kinetic which is a good solid starting point for ROS Development.

Below is a picture the RC car chassis which is a really nice size.

Logging into your DeepRacer

Logging into the DeepRacer for the first time is very similar to the DeepLens product, plug the compute module into a monitor using a HDMI cable and a USB Keyboard and mouse, then reset the users password to some value which satisfies the complexity requirements. I believe it is at least 8 characters, one upper, lower, number and a symbol.

Note: The username is deepracer.

Preparing your DeepRacer

In my case I also disabled x11, and enabled SSH.

To permit ssh through the firewall configured on the device I ran, the server was already installed and started.

sudo ufw allow ssh

To disable the desktop environment on the next restart run the following.

sudo systemctl disable lightdm.service

Note: I did this while the screen and keyboard where plugged in, then tested I could ssh to the DeepRacer from my laptop before restarting.

Now that you have ssh’d into the DeepRacer using your login details, and the TAG_HOSTNAME is on a tag on the bottom of the car.

ssh deepracer@TAG_HOSTNAME

Before you starting to mess around on the host you probably want to disable ufw to enable access to all the services from your laptop/desktop.

sudo ufw disable

Once you have disabled the firewall your should be able to access the video stream from ROS web_video_server using http://TAG_HOSTNAME.local:8080/stream_viewer?topic=/video_mjpeg where the TAG_HOSTNAME is on a tag on the bottom of the car. For more information on this service see http://wiki.ros.org/web_video_server.

Disable ROS Services

We are going to disable services which relate to the DeepRacer service, including the updates service so we can just use the control and media systems.

This is as simple as replacing /opt/aws/deepracer/share/deepracer_launcher/launch/deepracer.launch with content as follows, this simply comments out some of the AWS supplied services.

<?xml version="1.0"?>
<launch>
    <node pkg="web_video_server" type="web_video_server" name="web_video_server">
        <param name="ros_threads" value="1" />
    </node>
    <node name="servo_node" pkg="servo_pkg" type="servo_node" respawn="true" />
    <node name="media_engine" pkg="media_pkg" type="media_node" respawn="true" />

    <!-- <node name="inference_engine" pkg="inference_pkg" type="inference_node" respawn="true" output="screen"/> -->
    <!-- <node name="inference_probability" pkg="inference_pkg" type="inference_probability.py" respawn="true"/> -->
    <!-- <node name="model_optimizer" pkg="inference_pkg" type="model_optimizer_node.py" respawn="true" /> -->

    <node name="control_node" pkg="ctrl_pkg" type="ctrl_node" respawn="true" />
    <!-- <node name="navigation_node" pkg="ctrl_pkg" type="navigation_node.py" respawn="true" /> -->
    <!-- <node name="software_update" pkg="software_update_pkg" type="software_update_process.py" respawn="true" /> -->
    <!-- <node name="webserver" pkg="webserver_pkg" type="webserver.py" respawn="true" /> -->
</launch>

Now restart the DeepRacer service.

$ sudo systemctl restart deepracer-core.service

Is this thing on?

To test whether or not we can still drive the DeepRacer around, we will explore then interface with the control node provided.

Now we load the ROS environment, using the AWS setup.bash, this will populate variables holding names and paths for services, if you have never used ROS before you may want to run through some of the Tutorials.

$ source /opt/aws/deepracer/setup.bash

Now we should just have the services we need to start working with ROS.

$ rosnode list
/control_node
/media_engine
/rosout
/servo_node
/web_video_server

Lets look at the topics which are now accepting messages.

$ rostopic list
/auto_drive
/calibration_drive
/manual_drive
/rosout
/rosout_agg
/video_mjpeg

Now we are interested in /manual_drive to test out the throttle and steering.

$ rostopic info /manual_drive
Type: ctrl_pkg/ServoCtrlMsg

Publishers: None

Subscribers: 
 * /control_node (http://amss-n4lp:37837/)

So we need to post a message to this topic, but we need to know the format so lets print it’s structure.

$ rosmsg info ctrl_pkg/ServoCtrlMsg
float32 angle
float32 throttle

Now while the DeepRacer is off my desk, to stop it racing off, I run the following command which should trigger a throttle change. Note that the limit for this value is 0.7 by default, and 0 will stop it.

rostopic pub -1 /manual_drive ctrl_pkg/ServoCtrlMsg -- 0 0.3

Now we can stop the throttle by running.

rostopic pub -1 /manual_drive ctrl_pkg/ServoCtrlMsg -- 0 0

Likewise we can turn the DeepRacer to the left using the first value.

rostopic pub -1 /manual_drive ctrl_pkg/ServoCtrlMsg -- 0.9 0

And to the right.

rostopic pub -1 /manual_drive ctrl_pkg/ServoCtrlMsg -- -0.9 0

Then back to the center.

rostopic pub -1 /manual_drive ctrl_pkg/ServoCtrlMsg -- 0 0

I am really impressed with the DeepRacer so far, it is a great platform to start working with ROS at a great price, hopefully it spurs a whole raft of great robotics projects in the future. I would also love to see more detail, and hopefully source code for the services in this product as they seem to be really well thought out and could most certainly provide a spring board for future innovation.

In my next post we will write a ROS node which will use these services to drive the DeepRacer.

The highlights, and standout features for me are as follows:

• Adds intrinsic support for versioning into the go command.
• Includes a few new sub commands such as vendor and verify
• Incorporates a lot of new ideas around the storage and management of golang modules, which seems to correlate to something akin to a github project. Note there is support for more than one module in a repository but the general idea is one.
• Adds a new mechanism to retrieve and cache modules in zip files, which will supersede the current source repository.
• Adds a new proxy mechanism, enabling organisations to provide a mediated, verified module server to developers.

But probably the biggest change is the move away from the much maligned $GOPATH, this will as far as I can tell be deprecated over time. Developers will create their projects outside of the $GOPATH, using a file named go.mod to provide a pointer to the projects namespace.

If your interested the full background take a look at Go & Versioning, it is quite a long read.

So what does this look like in practice?

To illustrate how vgo works in practice lets create a project outside my $GOPATH, in my case I create a folder called ~/Code/hacking/golang-lambda-func and add a main.go containing code for a simple lambda API GW program. This code is copied directly from Announcing Go Support for AWS Lambda .

package main // import "github.com/wolfeidau/golang-lambda-func"

import (
	"log"

	"github.com/pkg/errors"

	"github.com/aws/aws-lambda-go/events"
	"github.com/aws/aws-lambda-go/lambda"
)

var (
	// ErrNameNotProvided is thrown when a name is not provided
	ErrNameNotProvided = errors.New("no name was provided in the HTTP body")
)

// Handler is your Lambda function handler
// It uses Amazon API Gateway request/responses provided by the aws-lambda-go/events package,
// However you could use other event sources (S3, Kinesis etc), or JSON-decoded primitive types such as 'string'.
func Handler(request events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) {

	// stdout and stderr are sent to AWS CloudWatch Logs
	log.Printf("Processing Lambda request %s\n", request.RequestContext.RequestID)

	// If no name is provided in the HTTP request body, throw an error
	if len(request.Body) < 1 {
		return events.APIGatewayProxyResponse{}, ErrNameNotProvided
	}

	return events.APIGatewayProxyResponse{
		Body:       "Hello " + request.Body,
		StatusCode: 200,
	}, nil

}

func main() {
	lambda.Start(Handler)
}

Firstly note the addition of // import "github.com/wolfeidau/golang-lambda-func" indicates where our module is located, being the current equivalent of creating that structure in our $GOPATH.

This depends on a bunch a few projects hosted on github, which translate into modules in vgo. To build the project we run vgo. If you want to try this just run go get -u golang.org/x/vgo to install it.

$ vgo build
vgo: resolving import "github.com/aws/aws-lambda-go/events"
vgo: finding github.com/aws/aws-lambda-go (latest)
vgo: adding github.com/aws/aws-lambda-go v1.1.0
vgo: resolving import "github.com/pkg/errors"
vgo: finding github.com/pkg/errors (latest)
vgo: adding github.com/pkg/errors v0.8.0
vgo: finding github.com/pkg/errors v0.8.0
vgo: finding github.com/aws/aws-lambda-go v1.1.0
vgo: finding github.com/urfave/cli v1.20.0
vgo: downloading github.com/aws/aws-lambda-go v1.1.0
vgo: downloading github.com/pkg/errors v0.8.0

Once completed a go.mod file is created which stores our modules package, and it’s dependencies. This content of this is as follows:

module "github.com/wolfeidau/golang-lambda-func"

require (
	"github.com/aws/aws-lambda-go" v1.1.0
	"github.com/pkg/errors" v0.8.0
)

Now what is also interesting is that vgo has created a cache in $GOPATH/src/v which looks something like. The following find command just lists the directories under this path.

$ find /Users/markw/go/src/v -type d
/Users/markw/go/src/v
/Users/markw/go/src/v/cache
/Users/markw/go/src/v/cache/github.com
/Users/markw/go/src/v/cache/github.com/urfave
/Users/markw/go/src/v/cache/github.com/urfave/cli
/Users/markw/go/src/v/cache/github.com/urfave/cli/@v
/Users/markw/go/src/v/cache/github.com/aws
/Users/markw/go/src/v/cache/github.com/aws/aws-lambda-go
/Users/markw/go/src/v/cache/github.com/aws/aws-lambda-go/@v
/Users/markw/go/src/v/cache/github.com/pkg
/Users/markw/go/src/v/cache/github.com/pkg/errors
/Users/markw/go/src/v/cache/github.com/pkg/errors/@v
/Users/markw/go/src/v/github.com
/Users/markw/go/src/v/github.com/aws
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/cmd
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/cmd/build-lambda-zip
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/lambda
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/lambda/messages
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/lambdacontext
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/events
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/events/testdata
/Users/markw/go/src/v/github.com/aws/aws-lambda-go@v1.1.0/events/test
/Users/markw/go/src/v/github.com/pkg
/Users/markw/go/src/v/github.com/pkg/errors@v0.8.0

So what it has created is a versioned cache under $GOPATH/src/v with a few notible features:

• No .git directories, this code is cloned and stored without version information.
• It has some interesting folders ending in @v which contain other files.

$ ls -1 pkg/errors/@v
v0.8.0.info
v0.8.0.mod
v0.8.0.zip
v0.8.0.ziphash

So as mentioned above this is an example of the new compressed module packaging which is being introduced.

Well hopefully that shines a bit more light on what vgo does behind the scenes, and some of the notable differences between it and how the go command currently works. I really like where vgo is going, overall it feels like a really good preview of things to come in a post $GOPATH world. I will endeavour to dig into some other the new features as I discover more.

If your after a better explanation of how to get up and running with vgo make sure you watch Using vgo for Go Dependency Management by @bketelsen.

This post will detail how I configured Keycloak with AWS SAML federation.

To demonstrate Keycloak I have setup a docker-compose project which can be cloned from https://github.com/wolfeidau/keycloak-docker-compose.

Assuming you have docker for mac installed you should be able to navigate to the project then run.

docker-compose up -d

Then to ensure it is all working you should be able to navigate to http://0.0.0.0:18080/auth/admin/master/console/#/realms/master.

Setup of the AWS SAML Client

To simplify the automated setup we can export a client configuration file containing the AWS SAML configuration, in my case I did this in the master realm then exported it.

First thing you need to do is download https://signin.aws.amazon.com/static/saml-metadata.xml, just put it in your Downloads folder.

Once you login navigate to clients http://0.0.0.0:18080/auth/admin/master/console/#/realms/master/clients then hit the create button and import the saml-metadata.xml file, then hit save.

Now configure:

• IDP Initiated SSO URL Name to amazon-aws
• Base URL to /auth/realms/wolfeidau/protocol/saml/clients/amazon-aws

Lastly under the Scope tab disable Full Scope Allowed, this will ensure we only pass through the roles configured in our client to AWS.

Now you can navigate back to http://0.0.0.0:18080/auth/admin/master/console/#/realms/master/clients and hit the export button next to the aws client.

Keycloak Setup Using Admin CLI

As a big proponent of automation I really wanted to illustrate, and indeed learn how to automate setup of keycloak, hence the CLI approach.

To get the tools we need for this guide download keycloak from Keycloak Downloads and extract this to say $HOME/Development/keycloak then add $HOME/Development/keycloak/bin to your $PATH as per Keycloak administration CLI docs.

export PATH=$PATH:$HOME/Development/keycloak/bin

Note: Commands which create new objects generate a unique GUID which looks like 6c684579-51a1-4bdf-a694-d641199874d8, you will need to adjust those values in the subsequent commands.

Now we can use the administration CLI program to configure our keycloak service.

To test it out and configure your account locally.

kcadm.sh config credentials --server http://0.0.0.0:18080/auth --realm master --user admin

Create a realm, in my case I am naming this wolfeidau.

$ kcadm.sh create realms -s realm=wolfeidau -s enabled=true

Import the keycloak client for AWS and add it to the wolfeidau realm we created, the JSON file is in the keycloak-docker-compose project.

$ kcadm.sh create clients -r wolfeidau -s clientId="urn:amazon:webservices" -s enabled=true -f urn-amazon-webservices.json
Created new client with id '6c684579-51a1-4bdf-a694-d641199874d8'

Create our AWS role under the AWS client, note this is an example name you will need to replace 123456789012 with your account id.

kcadm.sh create clients/6c684579-51a1-4bdf-a694-d641199874d8/roles -r wolfeidau -s 'name=arn:aws:iam::123456789012:role/wolfeidau-admin,arn:aws:iam::123456789012:saml-provider/docker-keycloak' -s 'description=AWS Administration Access'
Created new role with id 'docker-keycloak'

Create a group to grant AWS administration access.

$ kcadm.sh create groups -r wolfeidau -s name=aws-admins
Created new group with id 'dd02ed86-dd49-47c6-bd8a-5f74844b56d0'

Add a role to the group, note this is an example name you will need to replace 123456789012 with your account id.

$ kcadm.sh add-roles -r wolfeidau --gname 'aws-admins' --cclientid 'urn:amazon:webservices'  --rolename 'arn:aws:iam::123456789012:role/wolfeidau-admin,arn:aws:iam::123456789012:saml-provider/docker-keycloak'

Create a user for testing.

$ kcadm.sh create users -r wolfeidau -s username=wolfeidau -s email=mark@wolfe.id.au -s enabled=true
Created new user with id 'eb02cbfd-fa9c-4094-a437-3a218be53fe9'

Reset the users password and require update on login.

$ kcadm.sh update users/eb02cbfd-fa9c-4094-a437-3a218be53fe9/reset-password -r wolfeidau -s type=password -s value=NEWPASSWORD -s temporary=true -n

Add the user to our AWS administration group.

$ kcadm.sh update users/eb02cbfd-fa9c-4094-a437-3a218be53fe9/groups/dd02ed86-dd49-47c6-bd8a-5f74844b56d0 -r wolfeidau -s realm=wolfeidau -s userId=eb02cbfd-fa9c-4094-a437-3a218be53fe9 -s groupId=dd02ed86-dd49-47c6-bd8a-5f74844b56d0 -n

Export the metadata file required by AWS to setup the SAML provider.

$ kcadm.sh get -r wolfeidau clients/6c684579-51a1-4bdf-a694-d641199874d8/installation/providers/saml-idp-descriptor > client-tailored-saml-idp-metadata.xml

AWS Setup

Create the AWS SAML Provider in your account using the metadata file exported from keycloak.

aws iam create-saml-provider --saml-metadata-document file://client-tailored-saml-idp-metadata.xml --name docker-keycloak

Deploy the cloudformation template supplied in the keycloak-docker-compose project, this contains the SAML SSO IAM roles and saves clicking around in the UI.

aws cloudformation create-stack --capabilities CAPABILITY_IAM --stack-name sso-roles --template-body file://sso-roles-cfn.yaml

Note: You can just create the saml provider and launch the cloudformation from the AWS console.

Logging Into AWS

Now you should be ready to log into AWS using keycloak using the link http://0.0.0.0:18080/auth/realms/wolfeidau/protocol/saml/clients/amazon-aws.

Command Line SAML Authentication

To enable the use of SAML by command line tools such as ansible and the AWS CLI my colleagues and I developed saml2aws.

References

• Keycloak Admin CLI Docs
• Troubleshooting SAML 2.0 Federation with AWS
• Sign in to Amazon AWS using SAML protocol and Keycloak as Identity Provider

Why DynamoDB?

Some of the advantages DynamoDB offers:

• A Key/Value model where the values are any number of fields
• Simplified data access
• Low operational overhead
• Cost, a well tuned DynamoDB costs a few dollars a month to operate

As a developer getting starting with DynamoDB you need to know about:

1. Eventual consistency, this is integral to how DynamoDB achieves it’s cost, resilience and scalability. Simple things such as writing a record, then retrieving it straight afterwards require some logic to cope with records which aren’t visible yet.
2. Performing select * from Table is not recommended when working with DynamoDB, this will trigger scan operations which are less efficient than other operations in DynamoDB. It should be only used on small tables.

Using any key/value store can be tricky at first, especially if you’re used to relational databases. I have put together a list of recommendations and tips which will hopefully help those starting out with this product.

Retries

When you insert data into DynamoDB not every shard will immediately see your data, an attempt to read the data from the table may not get the value your looking for. If your inserting a new row, then attempting to read immediately afterwards you may get an empty response.

To mitigate this you will need to implement retries, ideally with a back off to avoid exhausting your provisioned throughput.

Global secondary indexes (GSIs) further complicate this, as these are also updated eventually, in our experience even more eventually than the base table. Again be aware when inserting rows you want to access straight afterwards you may need to check if a row is present in the index with a similar retry.

Data Modelling

The first big thing you need to understand is that DynamoDB doesn’t have relationships, in most cases it will be better to start by storing related data denormalised in a given record using the document feature of the client APIs. The reason we do this is it can be difficult keeping related data across tables in sync.

I recommend keeping everything in a single record for as long as you can.

Pagination

Although most of the clients provided by amazon have a concept of paging built in, this is really forward only, which makes building a classic paginated list quite a bit harder. This is best illustrated with some excerpts from the DynamoDB API.

Firstly QueryInput from the golang AWS SDK we have the ability to pass in an ExclusiveStartKey.

type QueryInput struct {
...
    // The primary key of the first item that this operation will evaluate. Use
    // the value that was returned for LastEvaluatedKey in the previous operation.
    //
    // The data type for ExclusiveStartKey must be String, Number or Binary. No
    // set data types are allowed.
    ExclusiveStartKey map[string]*AttributeValue `type:"map"`
...
}

And in the QueryOutput we have the LastEvaluatedKey.

type QueryOutput struct {
...
    // The primary key of the item where the operation stopped, inclusive of the
    // previous result set. Use this value to start a new operation, excluding this
    // value in the new request.
    //
    // If LastEvaluatedKey is empty, then the "last page" of results has been processed
    // and there is no more data to be retrieved.
    //
    // If LastEvaluatedKey is not empty, it does not necessarily mean that there
    // is more data in the result set. The only way to know when you have reached
    // the end of the result set is when LastEvaluatedKey is empty.
    LastEvaluatedKey map[string]*AttributeValue `type:"map"`
...
}

Given all we have are some keys, which may or may not be deleted it is very difficult to build a classic paged view.

So it has become clear to me we need to embrace a new strategy for displaying pages of results, luckily lots of others have run into this issue and the common pattern is to:

1. Use infinite scrolling, similar to twitter and other social media sites.
2. Maintain the state in the client with a cache of pages which have previously been loaded.

For more information on this see The End of Pagination.

Sorting

Just a small note on sorting, in more complicated tables you will need to add indexes purely to sort data in a specific way.

Also within the AWS api sorting uses the rather interestingly named ScanIndexForward.

    // If ScanIndexForward is true, DynamoDB returns the results in the order in
    // which they are stored (by sort key value). This is the default behavior.
    // If ScanIndexForward is false, DynamoDB reads the results in reverse order
    // by sort key value, and then returns the results to the client.
    ScanIndexForward *bool `type:"boolean"`

You are not alone

These challenges presented by DynamoDB not unique, NoSQL databases such as Riak, and Cassandra, share some of the same limitations, again to enable resilience and scalability. When searching for ideas, suggestions or strategies to deal with it you may find answers around these open source projects.

In Closing

I think it is important to note that learning DynamoDB will broaden your horizons, and in some ways change the way you look at persistence, this in my view is a good thing.

My general strategy for building apex applications is to build a standalone version of the functionality on my machine, typically in a way which makes the code reusable, then I import and use that in apex. This post will run through how I do this.

Firstly you will need to setup golang, I have documented how bootstrap a golang project.

Lets go ahead and make a project, this will hold our reusable code and a test command line tool. Note that wolfeidau needs to be changed to your github login.

mkdir -p $GOPATH/src/github.com/wolfeidau/shorten
cd !$

Now create a project folder and setup a sub folder for commands and create the main test program file.

mkdir -p cmds/shorten
touch cmds/shorten/main.go

Next we create a file in the base of project which will contain the reusable parts of our application. I normally use the project name for the first file I create and pull things out into other .go files as it grows.

touch shorten.go shorten_test.go

Add a package declaration file and some code as follows.

// Package shorten contains utility functions for shortening a URL.
package shorten

import (
    "fmt"
    "math/rand"
)

var letters = []rune("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_-.")

func init() {
    rand.Seed(time.Now().UnixNano())
}

func RandSeq(n int) string {
    b := make([]rune, n)
    for i := range b {
        b[i] = letters[rand.Intn(len(letters))]
    }
    return string(b)
}

Now copy the following test into the shorten_test.go file.

package shorten

import (
    "testing"

    "github.com/stretchr/testify/assert"
)

func TestRandSeq(t *testing.T) {
    v := RandSeq(12)

    assert.Equal(t, 12, len(v))
}

Now lets add some code to cmds/shorten/main.go to call this function. I am going to use github.com/alecthomas/kingpin to manage parsing command line flags.

package main

import (
  "fmt"

  "gopkg.in/alecthomas/kingpin.v2"
  "github.com/wolfeidau/shorten"
)

var (
  length = kingpin.Arg("length", "Length of random string").Int()
)

func main() {
  kingpin.Version("0.0.1")
  kingpin.Parse()
  fmt.Println("Random String:", shorten.RandSeq(*length))
}

Pull your depdencies into the local GOPATH using go get and run the test.

go get -uv ./...
go test -v ./...

Test output should be as follows.

=== RUN   TestRandSeq
--- PASS: TestRandSeq (0.00s)
PASS
ok      github.com/wolfeidau/shorten    0.011s
?       github.com/wolfeidau/shorten/cmd/shorten    [no test files]

Run the test program.

go run cmd/shorten/main.go 23

With the response something like.

Random String: YFynJVJYSJqsFejIGWGbEPF

Now we can build an apex project using the same method.

cd $GOPATH/src/github.com/wolfeidau
mkdir shorten-apex
cd !$

Next we initialise the apex project and remove the example function. Note this will setup an environment in AWS Sydney region using your default AWS profile.

apex init --region ap-southeast-2
rm -rf functions/hello

Create a function and go file.

mkdir functions/shorten
touch functions/shorten/main.go
echo '*.go' >> functions/shorten/.apexignore

Copy the following code into your main.go.

package main

import (
    "encoding/json"
    "fmt"
    "net/url"
    "time"

    "github.com/apex/go-apex"
    "github.com/wolfeidau/shorten"
)

const (
    domain = "https://s.example.com/"

    // ~83733937890625 should be enough random values
    // this assumes 55^8
    length = 8
)

type message struct {
    ShortURL  string `json:"shortUrl"`
    URL       string `json:"url"`
    Timestamp int64  `json:"timestamp"`
}

func main() {
    apex.HandleFunc(func(event json.RawMessage, ctx *apex.Context) (interface{}, error) {
        var m message

        if err := json.Unmarshal(event, &m); err != nil {
            return nil, err
        }

        m.ShortURL = domain + shorten.RandSeq(length)
        m.Timestamp = time.Now().UnixNano()

        return m, nil
    })
}

Now you can deploy your application.

apex deploy --region ap-southeast-2

To test out the deployed application create a sample event.json in the project directory containing the following JSON.

{"url": "https://www.wolfe.id.au/2016/08/12/bootstrap-an-apex-golang-project/"}

Then invoke your function.

apex invoke shorten --region ap-southeast-2 < event.json

You now have a modular lambda function deployed using apex and golang 🎉🚀.

The sample projects are located at:

• https://github.com/wolfeidau/shorten-apex
• https://github.com/wolfeidau/shorten

Firstly you will need to install golang and setup your GOPATH. If your on OSX you can just install homebrew and use it to install golang.

brew install go

Then in OSX I append a couple of lines to my $HOME/.bash_profile and source the file to update my current environment. On Linux you typically modify your .bashrc.

echo 'export GOPATH=$HOME/Code/go' >> ~/.bash_profile
echo 'export PATH=$PATH:$GOPATH/bin' >> ~/.bash_profile
source ~/.bash_profile

Now go env should look like.

go env | grep GOPATH
GOPATH="/Users/markw/Code/go"

We can now bootstrap your workspace to do this we are going to create a tree of folders which match the URL of your github projects. Note that you will need to change name_here to your github username.

mkdir -p $GOPATH/src/github.com/name_here
cd $GOPATH/src/github.com/name_here

Now in our workspace create a project folder and setup a sub folder for commands.

mkdir -p testproject/cmds/testproject
cd testproject

The cmds folder enables us to easily override the names of the golang applications we build, these inherit the name of the parent folder. So rather than just having a testproject command, I can make sub folders in the cmds directory for testproject-ui, testproject-server and so on.

Lets add a main entry point file to cmds.

touch testproject/cmds/testproject/main.go

Add the following code to our file using an editor.

package main

import fmt

func main() {
    fmt.Println("Hello World!")
}

Now you can create a README.md and init your git project.

touch README.md .gitignore
git init

Next steps I recommend getting yourself setup with an editor such as sublime text which I documented here Using Sublime Text for Go Development.

Once you have an editor you can dig into:

• How to Write Go Code
• Go Resources

So firstly why would we even do this?

The main aim of using docker for development is:

1. Portable build environment
2. Simplified on-boarding of new developers
3. Consistency between development and continuous integration (CI)

In summary tools like docker make it very easy to package up a development environment and share it among a team of developers.

Sharing folders

Within this setup we want the developer to choose what and how they code, on OSX we need an environment where they can just keep using their current editor.

The boot2docker service configures a shared folder which mounts /Users folder on OSX within the virtual machine it provisions which easy to mount your project all the way through from OSX to your docker container. For more information on how this works see VirtualBox Guest Additions.

Configuration

I typically start my webpack projects using macropodhq/webpack-skel by the people at Macropod.

Once i have my project setup I add the following fragment to the webpack configuration.

  watchOptions: {
    poll: 1000,
    aggregateTimeout: 1000
  },

Add a Dockerfile to the project which uses the official iojs images, note I am using 2.x as 3.x is still having issues with some native modules.

FROM iojs:2

# This will cd to the project root when docker starts in bash
CMD sh -c "cd ${PROJECT_PATH:-/}; exec /bin/bash"

Then build your docker container.

docker build -t iojsfsnotify .

Start the container passing in your current working directory as the path you want to change directory to when the container starts.

docker run -it -e PROJECT_PATH=$(pwd) -e DOCKER_IP=$(boot2docker ip) \
  -v "/Users:/Users" -p 8080:8080 -t iojsfsnotify

Then you can install your node modules.

npm install

Start the webpack-dev-server.

npm start

If you want to use the inline live reload mode you will need to use my fork of webpack-dev-server for the moment. I would love to get this change merged but I am guessing Tobias Koppers is pretty busy given how much of a runaway success webpack has been.

Below is the fragment from my package.json which points to my fork.

    "webpack-dev-server": "wolfeidau/webpack-dev-server"

A full example of this project is located at wolfeidau/webpack-docker-example.

Docker presents an opportunity to really make a big leap forward in providing a simple to deploy packaged SDK environments for hardware platforms. It is the first portable package format which works across operating systems, with consistent tooling and a mechanism to share changes with others.

For a more in depth run down on Docker and IoT see Rapidly develop Internet of Things apps with Docker Containers by @AnnaGerber

So how would this work?

To illustrate how docker simplifies delivery of these environments lets look at the Docker container I have built for development of esp8266 projects. This container is shared on docker hub esp8266-dev. To illustrate how this SDK environment is built take a look at the ansible-esp8266-role which is used to bootstrap it.

To get started you will need to setup docker on your system, in my case I am using boot2docker which works on OSX and Windows.

The first thing to understand with boot2docker is that your /User folder on OSX is configured as a shared folder in boot2docker virtual machine. This makes it easy to import data all the way through from OSX to your docker container. For more information on how this works see VirtualBox Guest Additions in the boot2docker project.

In my case I am using a project based on the esp8266/source-code-examples basic example.

So lets add a Dockerfile to this project, note that I have updated the WORKDIR to match the path to my ESP project.

FROM wolfeidau/esp8266-dev:1.1.0

# add all the SDK stuff to the PATH
ENV PATH=$PATH:/opt/Espressif/crosstool-NG/builds/xtensa-lx106-elf/bin

# Path which contains your esp8266 project source code
WORKDIR /Users/markw/Code/Systems/esp8266-led-lamp

# pass -v /Users:/Users to ensure your shared folder is available within 
# the container for builds.
VOLUME /Users

USER ubuntu

Then to build your container just run.

docker build -t esp8266-build .

And when you want to do a build run the following:

docker run -v /Users:/Users -i -t esp8266-build make

Now if you check in your firmware directory you should have the files required to flash your esp2866 project!

If we want to mess around inside the container we can run.

docker run -v /Users:/Users -i -t esp8266-build bash

This project is something I am working on at the moment esp8266-led-lamp.

So in summary we have installed boot2docker and built an esp8266 project with little or on messing around with complex SDK setup. In my view this is a big step forward in shortening the time to build for hardware projects, and simplifying delivery of complex SDK environments.

Hopefully the likes of Atmel and Texas Instruments look at using Docker in the future.

Two Factor Authentication

As soon as you start build things in Github for other people I recommend you enable two factor authentication, the process is pretty straight forward and services such as Authy and Google Authenticator make this even easier.

SSH Keys

Firstly please go to your Github SSH Keys Page and take a look at the number of SSH keys you have, either you can click the link or login to your Github account and select the settings icon in the top right hand corner.

• Are they all currently being used?
• Do you know where they all are?

Ideally use the key finger print to work out which ones you are using, and clear out any old ones. Most importantly, locate and verify where the all are, these keys are a gateway into all your stuff, make sure you keep them safe.

If your on OSX or Linux, you can see all the finger prints for your SSH keys by run the following command.

find ~/.ssh -name \*.pub | xargs -n1 ssh-keygen -lf

Third Party Applications

Next, lets talk about third party applications, I am not sure about you but as time goes on I seem to be getting access to an increasing tail of projects and organizations. Now in doing so, I also share that access with some third party services. This is especially the case with some of the early ones as their level of access was quite broad.

So how about we visit the Github Applications Page and clear out any you currently don’t use, I am especially talking about services that have access to ALL your repositories such as travis, drone.

Now there is nothing wrong with these specific services, however the level of access they require to operate is typically “all the things”.

If you do run a Github organisation you should also review the third-party application restrictions features.

Continious Integration Services

So if your using Github for a few personal projects, most of which are open source then services such as travis and drone are fantastic. But if your like me and work on Github for a living then you probably have access to code that belongs to someone else. If this is the case I recommend being a bit more careful about who and what you give access to all your repositories.

So how do we solve this problem, getting the most out of these services, while also controlling what they have access to see and possibly modify?

Rather than using your account as a gateway into the miriad of projects you have access too, how about creating a build user for that task? Given how easy it is to manage a variety of logins with services such as 1password and the like, maintaining an extra account is pretty simple.

So what other advantages are their to using a separate user for this?

• Lets you tightly control what services can access, for example only giving CI services read access to projects which need builds run.
• If you go on holidays, or change roles, other team members can take over responsibility for this account and things will continue to function.
• Enables you to centralise all ssh-keys used for builds or deployments.

The way this works is you setup a CI user add it to the organisation, then setup a CI team in Github which to manage this users access to projects. If necessary the credentials can be shared by a couple of key people within the team, so you can actually have a holiday in peace.

When keys are required for automation, or a service requires access to repositories, you login as that user and set them up, likewise all SSH keys used for automation go on this account.

Once setup you have essentially sandboxed access by external services to an account which has only what it needs to operate. This in turn leaves your account with as few links to external services as possible.

Summary

So by the end we should have:

• Enabled two factor authentication on your account.
• Done a cleanup of our SSH keys and worked out where they all are stored.
• Cleared out unused third party applications.
• Reconfigured continuous integration services to use a separate user in your organisation and restricted that to the bare minimum access.

Closing Thoughts

I hope I have at least encouraged you to review your Github account and give it a bit of a spring clean. I am sure there are a few other things you can do to tighten up access to the things you work on, your welcome to leave a comment if you have any other ways to improve the security of your Github account.

Also thanks to @michaelneale for proof reading, and @adam__brady for reminding me that not everyone has two factor authentication enabled, and @johnbarton for highlighting the third party restrictions feature I wasn’t aware existed.

The first thing to be clear on is the syslog logging driver enables you to relay the log messages written to stdout/stderr within a container, to syslog service on the host machine.

For this example I am going to do this for an agent of the buildkite continuous integration (CI) service running inside a Docker container.

To ensure these logs are only retained on the host for a short time logrotate will be imployed to rotate and delete aging logs.

Firstly to use the new log drivers you need Docker 1.6, in my case I used docker-machine to spin up a new system on digital ocean as follows.

docker-machine create -d digitalocean \
	--digitalocean-access-token XXXXX \ 
	--digitalocean-image 'ubuntu-14-04-x64' \
	--digitalocean-region 'sfo1' \
	--digitalocean-size '1gb' buildkite01

Note: You will need to grab your access token from the digital ocean dashboard.

Once the system is up and running use docker-machine to log into the host.

docker-machine ssh buildkite01

Configure rsyslog to isolate the Docker logs into their own file. To do this create /etc/rsyslog.d/10-docker.conf and copy the following content into the file using your favorite text editor.

# Docker logging
daemon.* {
 /var/log/docker.log
 stop
}

In summary this will write all logs for the daemon category to /var/log/docker.log then stop processing that log entry so it isn’t written to the systems default syslog file.

Now we configure logrotate to roll and archive these files. Create the /etc/logrotate.d/docker file and again copy the content as follows to it.

/var/log/docker.log {
    size 100M
    rotate 2
    missingok
    compress
}

This entry will roll the docker.log when it gets above 100MB and retain two of these files.

Note: The logrotate cron job only runs daily by default, so this check will only be done at the end of the day.

Now restart the rsyslog service.

service rsyslog restart

Now build your Docker container on your Docker machine from your laptop as follows. This will transfer the contents of . to the remote system and build your Docker container there, then tag it.

docker $(docker-machine config buildkite01) build -t buildkite/wolfeidau-golang:latest .

Then launch a named container, in this case we are passing --log-driver=syslog along with other buildkite agent specific information, and the tag we just built.

docker $(docker-machine config buildkite01) run \
	-d --rm --name buildkite-golang --log-driver=syslog \
	-e BUILDKITE_AGENT_TOKEN=XXX \
	-e BUILDKITE_AGENT_META_DATA="golang=1.4.2" \
	buildkite/wolfeidau-golang:latest

Then we ssh into the remote system and tail the /var/log/docker.log log file we can see output from the buildkite agent.

$ docker-machine ssh buildkite01
# tail /var/log/docker.log
May  3 05:53:56 buildkite01 docker/7ebdb2baff9c[3207]: 2015-05-03 09:53:56 INFO   Registering agent with Buildkite...
May  3 05:53:57 buildkite01 docker/7ebdb2baff9c[3207]: 2015-05-03 09:53:57 INFO   Successfully registred agent "7ebdb2baff9c" with meta-data [golang=1.4.2]

So at the moment some things are on the horizon for this driver including changing the tag and possibly remote server configuration.

That is the end of this brief post, hopefully it helps others get started with this new feature, and provides some tips on how to configure the rsyslog service to work with it.

Lastly I typically use ansible to automate this sort of thing, but it is nice to walk through it at least once.

So firstly if you’re new to golang then before you start setup your workspace, firstly watch this video Writing, building, installing, and testing Go code.

When I am setting a new system up I typically run the following commands in OSX or Linux, this example is of course for bash, if you use zsh I am sure you can adapt this where needed.

echo 'export GOPATH=$HOME/Code/go' >> ~/.bash_profile
echo 'export PATH=$PATH:$GOPATH/bin' >> ~/.bash_profile 
source ~/.bash_profile
mkdir -p ~/Code/go/src/github.com/wolfeidau
cd !$

Note: If you’re on OSX you should use .bash_profile, on Linux you typically use your .bashrc.

Once i have done this I can either clone a project from my github or make a directory for a new one.

Now that you have setup your environment setup you can install some tools.

go get -u golang.org/x/tools/cmd/goimports
go get -u golang.org/x/tools/cmd/vet
go get -u golang.org/x/tools/cmd/oracle
go get -u golang.org/x/tools/cmd/godoc

Then install package control in your Sublime editor and add the following plugins.

• GoSublime
• GoOracle

Then using update your GoSublime user configuration by opening Preferences -> Package Settings -> GoSublime -> Settings User which should open your GoSublime.sublime-settings file, below is the contents of mine.

{

	// you may set specific environment variables here
	// e.g "env": { "PATH": "$HOME/go/bin:$PATH" }
	// in values, $PATH and ${PATH} are replaced with
	// the corresponding environment(PATH) variable, if it exists.
	"env": {"GOPATH": "$HOME/Code/go", "PATH": "$GOPATH/bin:$PATH" },

  "fmt_cmd": ["goimports"],

	// enable comp-lint, this will effectively disable the live linter
	"comp_lint_enabled": true,

	// list of commands to run
	"comp_lint_commands": [
	    // run `golint` on all files in the package
	    // "shell":true is required in order to run the command through your shell (to expand `*.go`)
	    // also see: the documentation for the `shell` setting in the default settings file ctrl+dot,ctrl+4
	    {"cmd": ["golint *.go"], "shell": true},

	    // run go vet on the package
	    {"cmd": ["go", "vet"]},

	    // run `go install` on the package. GOBIN is set,
	    // so `main` packages shouldn't result in the installation of a binary
	    {"cmd": ["go", "install"]}
	],

	"on_save": [
	    // run comp-lint when you save,
	    // naturally, you can also bind this command `gs_comp_lint`
	    // to a key binding if you want
	    {"cmd": "gs_comp_lint"}
	]
}

Note: Ensure you update the GOPATH value to match the one configured earlier.

Once you restart sublime you should be ready to roll!

In addition to these plugins I also use GitGutter which provides some highlighting of changes for source code under git.

Lastly just another tip to enable spell check in markdown files.

Open any markdown file then go to Preferences -> Settings - More -> Syntax Specific - User which should open your Markdown.sublime-settings file, below is the contents of mine.

{
  "spell_check": true
}

Thanks to Marius Ursache for his feedback.

What does a package manager do?

A package manager’s core job is to:

• Provide a simple method of retrieving a snapshot of upstream libraries.
• Enable a user to maintain compatible versions of these upstream libraries therefore avoiding broken builds when one of these modules is changed.
• Let people maintaining libraries communicate differing levels of interface/module changes to downstream users via semantic versioning.

The current solution

Firstly go get is a simple solution to decentralised retrieval of dependencies, without much grief a developer can pull down and maintain a copy of upstream libraries. The main issue is that this currently only solves half of the problem, developers are left to manage these dependencies themselves, including vetting all changes upstream.

This is an issue because:

• Developers assume this is a solved problem, not something they have to deal with while also learning a new language.
• Those new to development get the double whammy of having no idea where to even start with this problem.

Now on the other side of the fence, as a library writer I have no standard way to communicate massive breaks in API compatibility. Again this disregards many years of learning/development of things like semantic versioning.

From what I have observed this has led most larger projects to naturally limit the number of external dependencies in use.

Lastly without a nice solution for modularising software most developers won’t bother.

Why add versions?

In my view the value that semantic versioning adds to dependency cannot be disregarded. As stated it gives both consumers and producers a way of communicating changes. In addition to this it provides people with a way of addressing a particular release of an API, therefore alleviating a confusion about “which” snapshot of a library a developer is having issues with.

The assumption that everyone is always up-to-date is disregarding the reality of how people maintain software in the greater software development community.

Can it be done using a decentralised model?

So this raises the question of whether or not versioning can be added to go without introducing a central package management system.

It is my opinion that simply encouraging people to manage their project using a standard versioning scheme, with a standard method for tagging their versions would be a good starting point.

At some point it would be helpful to have:

1. A central meta data store for download statistics, this is helpful to package maintainers to raise awareness of the impact of change on others. This would really fill the gaps in existing systems such as github around exposing download counts for a given project.
2. A distributed snapshot repository of packages, just to avoid the inevitable deletion or reorganisation of repositories (YES THIS HAPPENS).

Note: I don’t think github should have to provide this by the way.

What is the shortest path to easing pain for new users?

Based on my review tool just needs to do the following:

1. Simple command line tool, one entry point and minimal options.
2. The ability to manage versioning of my project using the same tool.
3. A method of retrieving different versions of a dependency with an option to save this to a file.
4. Automatic upgrades of dependencies based on a known standard versioning scheme such as semver.
5. Fall back to the existing go get model with an obvious report at the end indicating this dependency doesn’t provide versions.
6. Provide an idiomatic way to store this version information, based on reflection on existing features for maintaining meta data. My view is this should be a version.go in each project.

This is in response to a gist posted by Dave Cheney after a discussion on twitter with Mitchell Hashimoto.

Thanks to Conrad and Nicholas Faiz for their feedback and critique.

In our case the network looked as follows:

In this diagram we have macbook A which is connected to a MeshThing running contiki-os connected via serial over USB. The ipv6 connection is provided by slip6, which gives use a point to point link over the serial connection. The command to start tunslip6 is as follows:

sudo ./tunslip6 -B 38400 -s /dev/tty.usbmodem1411 aaaa::1/64
********SLIP started on ``/dev/tty.usbmodem1411''
opened tun device ``/dev/tun0''
ifconfig tun0 inet6 up
ifconfig tun0 inet6 aaaa::1/64 add
sysctl -w net.inet6.ip6.forwarding=1
net.inet6.ip6.forwarding: 1 -> 1
ifconfig tun0

tun0: flags=8851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	inet6 fe80::bae8:56ff:ffff:ffff%tun0 prefixlen 64 optimistic scopeid 0xc
	inet6 aaaa::1 prefixlen 64 tentative
	nd6 options=1<PERFORMNUD>
	open (pid 51393)
RPL started
Online
*** Address:aaaa::1 => aaaa:0000:0000:0000
Got configuration message of type P
Setting prefix aaaa::
Server IPv6 addresses:
 aaaa::11:22ff:ffff:ffff
 fe80::11:22ff:ffff:ffff

So this command, for those not aware is using my USB serial device, running at 38400 baud to establish a link to the device.

At the end of this command we are providing a prefix of aaaa::1/64, which enables the device to pick it’s own IPv6 address using Stateless Address Autoconfiguration (SLAAC). It is important to note that this means the prefix needs to use a /64 mask.

Now as we can see in our original diagram we now need to route packets between the aaaa::1/64 network assigned to our mesh.

Given we need to enable routing between macbook B and the MeshThing we need the local wireless network to provide an IPv6 prefix for auto configuration of this host.

As macbook B is going to be routing we will get it to advertise our prefix of bbbb::1/64 on the existing wireless lan.

This is done using rtadvd in BSD derivatives, and rdvd on linux. So to get this running on OSX edit your /etc/rtadvd.conf as root, and add the following line.

en1:\
 :addr="bbbb::1":prefixlen#64:

On linux we can use radvd, which can be installed via apt-get on ubuntu. This is configured via /etc/radvd.conf with the equivalent configuration below.

interface eth1 {
  ## (Send advertisement messages to other hosts)
  AdvSendAdvert on;
  ## IPv6 subnet prefix
  prefix bbbb::1::/64 {
    AdvOnLink on;
    AdvAutonomous on;
  };
};

So in this case we are using en1 on macbook A is the wireless interface. We start rtadvd, this will after about 4 seconds send out a route advertisement which will trigger auto-configuration of all hosts on wireless network, therefore providing a prefix to macbook B which it will used by SLAAC to generate an IPv6 address.

To check our configuration worked correctly, first thing you will notice is that all hosts on the lan now have an IPv6 address in the prefix range.

[~]$ ifconfig en0 | grep inet6
	inet6 fe80::bae8:56ff:ffff:ffff%en0 prefixlen 64 scopeid 0x4
	inet6 bbbb::bae8:56ff:ffff:ffff prefixlen 64 autoconf
	inet6 bbbb::615e:319:aaaa:aaaa prefixlen 64 autoconf temporary

So in order these adresses are:

• Link Local address, these always begin with fe80:: and also includes our MAC address of b8:e8:56:ff:ff:ff.
• SLAAC address, which is a combination of the prefix and our MAC.
• One temporary address, which hides the network card address, this is the one that should be “used by applications”, no idea what this means Wireshark says everything on the LAN is using the MAC based IPv6 addresses I need to RTFM (read the fine manual) more.

To see neighbours in OSX you can use ndp, the R flag in the example below indicates which host is a router.

[~]$ ndp -an
Neighbor                        Linklayer Address  Netif Expire    St Flgs Prbs
bbbb::bae8:56ff:ffff:ffff        b8:e8:56:ff:ff:ff     en0 permanent R

On linux the equivalent command is ip -6 neigh show.

So for Linux, IOS and Android this is all we need to do to provide get the routing to work, however on OSX clients we have one additional step.

As OSX doesn’t accept the route advertisements by default, a route solicitation daemon called rtsold needs to be running on all client machines aside from the router. Below is the command I ran on macbook B, note on this machine the wireless adapter was en0.

sudo rtsold en0

Once this is started you should be able to ping the MeshThing from macbook B on the wireless lan.

[~]$ ping6 aaaa::11:22ff:ffff:ffff
PING6(56=40+8+8 bytes) aaaa::1 --> aaaa::11:22ff:ffff:ffff
16 bytes from aaaa::11:22ff:ffff:ffff, icmp_seq=0 hlim=64 time=34.186 ms
16 bytes from aaaa::11:22ff:ffff:ffff, icmp_seq=1 hlim=64 time=33.553 ms
--- aaaa::11:22ff:fe33:4401 ping6 statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 33.553/33.870/34.186/0.316 ms

This is a part of my ongoing hardware hacking, for more details on how this started see Adding an ICSP header to the ATmega256RFR2 . As a note both my Atmel board and the MeshThing use the ATmega256RFR2 chips.

The next goal is to setup some name services, probably via MDNS, and test accessing the web site (yes on the 8 bit micro controller) from a mobile phone connected to the wireless network.

Update

Need to look into using site-local addresses in the fc00::/7 block rather than unallocated public space, as recommended by Julien Goodwin see unique local addresses.

Create your project directory.

mkdir npmtest

Change to the directory you just created.

cd !$

Tell git to make a repo.

git init

Pull down a preconfigured .gitignore file for node projects from github.

wget https://raw.githubusercontent.com/github/gitignore/master/Node.gitignore -O .gitignore

Pull down a basic Makefile I use for my projects.

wget https://gist.github.com/wolfeidau/8748317/raw/172a6adb79777676a8815da5719ef659fb66a35b/Makefile

This make file contains a few handy targets, these are:

• test runs tests in the test folder using the mocha test framework.
• jshint uses jshint to check over the code.
• skel generates a basic structure for my project creating index.js and, lib, example and test directories and installs mocha and chai the BDD / TDD assertion library.
• The default target which is invoked by just running make, this runs the jshint and test targets.

Note: You will need to install jshint globally using npm install -g jshint.

Now we will use the skel target to generate our project structure.

make skel

Create a project on github using hub, if your on osx you can install this with homebrew. We do this sooner rather than later so npm can pick this information up when building the package.json.

hub create

Now initialise your project.

npm init

This should ask for a bunch of information, note leave the version 0.0.0 we will change this later. For those interested this is driven by init-package-json.

...
name: (npmtest)
version: (0.0.0)
description: Some NPM test.
entry point: (index.js)
test command: make test
git repository: (git://github.com/wolfeidau/npmtest.git)
keywords: npm
author: Mark Wolfe <mark@wolfe.id.au>
license: (ISC) MIT
About to write to /Users/markw/Code/Javascript/npmtest/package.json:

{
  "name": "npmtest",
  "version": "0.0.0",
  "description": "Some NPM test.",
  "main": "index.js",
  "scripts": {
    "test": "make test"
  },
  "repository": {
    "type": "git",
    "url": "git://github.com/wolfeidau/npmtest.git"
  },
  "keywords": [
    "npm"
  ],
  "author": "Mark Wolfe <mark@wolfe.id.au>",
  "license": "MIT",
  "bugs": {
    "url": "https://github.com/wolfeidau/npmtest/issues"
  },
  "homepage": "https://github.com/wolfeidau/npmtest"
}

Is this ok? (yes) yes

Once you have added some code to index.js and some tests of course, add and commit your code.

git add .
git commit -a 'Initial release'

Now your ready to release use npm to update the version. There are three options for this command being major, minor and patch each of which increments the version numbers in that order. In the example below we should go from 0.0.0 to 0.1.0.

npm version minor

Run your tests!

npm test

Push to github, the version command automatically tags your project so we can check it out if we need!

git push origin master --tags

Ship it.

npm publish

When I started reading up on the kit I noticed it required windows to install some 600mb + more IDE (gasp), to hack on a small micro controller. This really didn’t gel with me, especially considering said IDE:

• ONLY works on windows
• Visual Studio is bloated piece of shit
• Has all this other “professional” features which i don’t give a shit about
• Is driving a 30mb included gcc tool chain..

So how do we ignore this and get started without this thing?

Well firstly we need an ICSP port, this will enable us to write a simple boot loader. This will enable use to load a firmware onto the board and get started doing some basic hacking.

The cable I used had the following coloured wires, to bind these into a six pin connector Andy Gelme used a hot glue gun. It is amazing how handy these things are..

The configuration of the header wire colours are:

BLUE	RED
GREEN	ORANGE
YELLOW	BROWN

The standard Arduino ICSP 6 pin header is as layer out is:

MISO	VCC
SCK     MOSI
RESET	GND

So to wire this up to the board I first had to solder 5 pins into the spare signals header, this enabled me to access the RSTN pin, which in the ICSP header is called RESET.

The wires were connected as follows:

• MISO (BLUE) PIN17 on the ext5 header
• VCC (RED) PIN20 on the ext5 header
• SCK (GREEN) PIN18 on the ext5 header
• MOSI (ORANGE) PIN16 on the ext5 header
• RESET (YELLOW) RSTN on the spare signals header
• GND (BROWN) to PIN19 on the ext5 header

The final product looks like:

So to write the firmware onto the board a USBasp ICSP Programmer for AVR / Arduino was used. The firmware I am currently running was built by some hackers at CCHS Melbourne. At the end of the day a fork of Contiki was up and running on the board! I am hoping this gets open sourced soon.

The toolset I am using at the moment is Crosspack AVR which works great on OSX. I also have a build environment on ubuntu in a virtual machine which I am using just to test things.

In addition to this fork of Contiki we also tested running some simple Arduino applications on my board using avrdude to upload the hex files.

The aim of this project is to get an MQTT client onto this board and transmit values from the various sensors to a gateway device running linux over 802.15.4 mesh using 6LoWPAN.

Thanks to the people at CCHS Melbourne for their help.

I am planning to follow up with some posts on each of the layers as I learn more and work my way up the stack.

1. It enables me to establish a baseline for performance and behavior of the system.
2. It lets me quickly zero on any areas I can help with.
3. It makes it easier for me to see when I get things wrong while I am still learning the ropes.

So when I talk about monitoring I am mainly talking about collecting some basic performance and health related metrics which fall into two different types:

• gauges, typically a measurement of the current CPU, Memory usage and latency of upstream APIs, which is typically averaged over a period.
• counters, typically things like a count requests serviced by a particular rest end point, or emails sent by a background job.

So when I have been managing systems in the past I have typically been very careful to follow a neat and clear naming standard for metrics such as:

• region, the physical location or region the data center hosting this server is located.
• environment, being either development staging or production.
• hostname, for host level metrics we include the hostname.
• application, which represents the service component of our system.
• label, a label for the data being gathered.
• metric, the metric which has been gathered over the configured time period.

So below we have an example of a gauge of the applications RSS memory captured on a production server hosted in us-east.

us-east.prod.a1ad1f1111a1111db111e1baab111e11.app.memory_rss.median

So armed with statsd and Librato I started adding monitoring to the system, unfortunately this is where things went a bit pear shaped. Having been used to monitoring static environments I was not prepared for the changing and in some ways obfuscated world of fully isolated process containers.

In these process containers things like hostnames and IP addresses change each time your service is moved within the environment, this makes correlating metrics emitted from statsd very difficult. On the outside of Heroku we get a picture of dynos web.1 and web.2, but inside the container your processes are unable to access these labels, the are essentially just anonymous process tasked with doing a particular job.

So this resulted in an endlessly growing list of metrics, every time your process is restarted, I would get a new hostname and a whole new group of metrics. One thing to note is Heroku restarts your process at least once per day.

So how do you monitor stuff housed in Heroku? The answer to this is logging, the Heroku gives you the ability to provision log drains, these endpoints are sent a copy of all messages written STDOUT and STDERR by your application. When configured the logdrain transports your logs via either syslog or HTTP(S) to an external server, running in the same AWS as Heroku. Most importantly, these logs include a context of which dyno is emitting the information. To read more about this feature take a look at Logplex documentation.

When I started my new role one of my coworkers mentioned that he was currently logging metrics using a feature called log-runtime-metrics available via the Heroku labs. Initially I thought this was a bit limited, however on further examination this is actually an extremely reliable measure of the processes resource utilisation. For more information on this feature Heroku Labs: log-runtime-metrics.

So just for clarity this is what these runtime metrics look like, note the reference to the dyno in the source along with the metric name, value and units.

source=heroku.2808254.web.1.d97d0ea7-cf3d-411b-b453-d2943a50b456 measure=memory_rss val=21.22 units=MB

So now that I had metrics, with context in a stream coming in via logs I needed a way of getting this data to Librato having already had contact with Nik Wekwerth I decided to ask whether he had any suggestions. After a few emails I was very fortunate enough to be passed onto the CTO Joseph Ruscio who after a few questions suggested I try out l2met, a project by Ryan Smith an engineer at Heroku. This service acts as a bridge between the two services filtering the metrics out of the log messages and send them off to librato at a configurable interval.

After learning a bit of golang I was able to add support for the Heroku labs runtime metrics feature and tune a few things for my requirements. This was helped along by some good high level tests and a few helpful tips from the author.

An example of my new log based metrics is as follows:

us-east.prod.app.load_avg_5m.median

The missing piece here is the dyno, fortunately l2met sends this in an optional source property enabling me to use aggregate, or per dyno level metrics in my dashboard. For more information on the attributes supported by Librato see their API documentation.

One bonus from this is that when dynos are added or removed the graphs in Librato automatically reflect this without requiring any configuration.

Now some key take points on this:

• Although Heroku appears to hide a lot information from you at first, it does provide some very good instrumentation as long as you understand how to tap into it.
• Logging is the primary source of information about the state of your hosted applications, use it as much as possible.
• Measure Anything, Measure Everything have a read of this if you haven’t before it is a great post.

Note, I am still using statsd for some of my counters, this is due to the frequency they change, I am hoping to implement a scheduled emitter for them in the future which I can pass through via logs. To remedy the amount of logging I am not including the hostname in the metrics anymore and just acquiring the aggregate value across all processes.

I plan to follow this up with a post containing some notes on how I configured l2met, hopefully I can get this out in the next couple of weeks.

Note that you MUST install xcode before installing anything, then install homebrew, rbenv, and lastly openssl.

ruby -e "$(curl -fsSL https://raw.github.com/mxcl/homebrew/go)"
brew install rbenv
brew install openssl

Next to overcome the fact that OSX doesn’t have an openssl ca certificate bundle, use the following brew to create and maintain one using the CA certs stored in your keychain.

brew tap raggi/ale && brew install openssl-osx-ca

Make a temporary directory to build the sources in, download the 2.0.0-p0 and extract it into this location, then navigate into the ruby-2.0.0-p0 directory containing the sources.

mkdir ~/temp && cd ~/temp
curl -L ftp://ftp.ruby-lang.org/pub/ruby/2.0/ruby-2.0.0-p0.tar.bz2 | tar xjf -
cd ruby-2.0.0-p0

Run configure file with the arguments as listed below.

./configure --prefix=$HOME/.rbenv/versions/2.0.0-p0 --enable-dtrace \
--with-opt-dir=`brew --prefix openssl`

Within this rather verbose output you should see the following, this indicates that dtrace has been included.

checking whether dtrace USDT is available... yes

Build ruby, note I am invoking make with the ‘-j’ or ‘–jobs’ option tells make to execute many recipes simultaneously, in my case I chose 9 as I have eight cores (n of cores + 1).

make -j9

Now install ruby into rbenv with the label 2.0.0-p0.

make install

To try it out we will alter our shell to use the 2.0.0-p0 version.

rbenv shell 2.0.0-p0

Running ruby -v should output the following.

ruby 2.0.0p0 (2013-02-24 revision 39474) [x86_64-darwin12.2.0]

You can remove the temporary directory you built ruby in now.

rm -r ~/temp

NOTE: Bundler has just been updated to cater for ruby 2.0.0 but you will need to retrieve the gem manually and install it as follows.

wget https://rubygems.org/downloads/bundler-1.3.0.gem
gem install bundler-1.3.0.gem
rm bundler-1.3.0.gem

So I did a bit of research on what hardware I could connect up to the Raspberry Pi with the least amount of circuitry this led me to the AdaFruit site and in particular this article DHT Humidity Sensing on Raspberry Pi with GDocs Logging. This was as almost what I was after but still had a little to much “construction”. After searching around I managed to deduce that a small pre-made Arduino compatible board existed with all the circuitry already assembled, all that remained was to solder 3 wires onto the board and get this connected to the Raspberry Pi. After some foraging around in my computer “junk” I found some old case wiring which had small 4 pin connectors which could be plugged into the gpio header and could have their pins rejigged into any combination of one, two or 4 pin headers for maximum flexibility.

So to build this project you will need the following items, I have included the approximate cost.

• 1 x Raspberry Pi ($42)
• 1 x 700mW power supply ($12)
• 1 x freetronics Humidity and Temperature Sensor Module ($20)
• 2 x 4 pin connectors, as found in old PC cases or CD drive audio cables

Total cost $74.

Assembly is quite simple.

Solder three wires into the freetronics board, in my case I had red which i soldered into the data pin, white which I soldered into the 3.3v pin and black which I soldered into the ground pin.

Move the ground wire into it’s own 4 pin connector, and put the 3.3v and data pins at either end of another 4 port connector.

Connect these as illustrated to the Raspberry Pi, being sure to triple check the location.

Bask in the glow of the little LED on the addon board which indicates you have powered it up.

First you will need to grab and install the bcm2835 library before building it. I grabbed the latest sources for Mike McCauley’s bcm2835 library and installed them on the pi.

pi@raspberrypi ~ $ wget http://www.open.com.au/mikem/bcm2835/bcm2835-1.14.tar.gz
pi@raspberrypi ~ $ tar xvzf bcm2835-1.14.tar.gz
pi@raspberrypi ~ $ cd bcm2835-1.14
pi@raspberrypi ~/bcm2835-1.14 $ ./configure
pi@raspberrypi ~/bcm2835-1.14 $ make
...
pi@raspberrypi ~/bcm2835-1.14 $ sudo make install

Download the software as instructed in the linked ADAFruit article, you will need git so install that first.

pi@raspberrypi ~ $ apt-get install git
pi@raspberrypi ~ $ git clone https://github.com/adafruit/Adafruit-Raspberry-Pi-Python-Code.git
pi@raspberrypi ~ $ cd Adafruit-Raspberry-Pi-Python-Code/
pi@raspberrypi ~/Adafruit-Raspberry-Pi-Python-Code $ cd Adafruit_DHT_Driver

Build the software.

pi@raspberrypi ~/Adafruit-Raspberry-Pi-Python-Code/Adafruit_DHT_Driver $ make

Run the Adafruit_DHT command.

pi@raspberrypi ~/Adafruit-Raspberry-Pi-Python-Code/Adafruit_DHT_Driver $ sudo ./Adafruit_DHT 2302 4
Adafruit_DHT 2302 4
Using pin #4
Data (40): 0x2 0x3e 0x0 0xde 0x1e
Temp =  22.2 *C, Hum = 57.4 %

I am working on another post with some details of how I am using this device, and the software I will be driving it with, this should go out in the next couple of weeks.

Hope others find this useful.

Update

2013-02-28 Removed the change to small change to the make file as it is not required anymore.

Recently I have been doing quite a bit of research and hacking in and around server APIs. Authentication for these type APIs really depends on the type of service, and falls into a couple of general categories:

• Consumer or personal applications, these typically use a simple username and password, OAuth is used in some cases however this is more for identity of an individuals authorisation session within a trusted third party.
• Infrastructure applications, these typically use a set of credentials which are different to the owners/admins credentials and provide some sort of automation API for business or devices to enhance the function or control something.

For infrastructure APIs I have had a look at a few options, these are explained in some detail below.

Basic Authentication

This is the simplest to implement and for some implementations can work well, however it requires transport level encryption as the user name and password are presented with ever request. For more information on this see Wikipedia Article.

Digest Authentication

This is actually quite a bit closer to HMAC than basic, it uses md5 to hash the authentication attributes in a way which makes it much more difficult to intercept and compromise the username and password attributes. Note I recommend reading over the Wikipedia page on the subject, in short it is more than secure than basic auth, however it is entirely dependent on how many of the safeguards are implemented in the client software and the complexity of the password is a factor.

Note unlike basic authentication, this does not require an SSL connection, that said make sure you read the Wikipedia article as there are some issues with man in the middle attacks.

HMAC Authentication

Hash-based message authentication code (HMAC) is a mechanism for calculating a message authentication code involving a hash function in combination with a secret key. This can be used to verify the integrity and authenticity of a a message.

Unlike the previous authentication methods there isn’t, as far as I can tell a standard way to do this within HTTP, that said as this is the main authentication method used by Amazon Web Services it is very well understood, and there are a number of libraries which implement it. To use this form of authentication you utilise a key identifier and a secret key, with both of these typically generated in an admin interface (more details below).

It is very important to note that one of the BIG difference with this type of authentication is it signs the entire request, if the content-md5 is included, this basically guarantees the authenticity of the action. If a party in the middle fiddles with the API call either for malicious reasons, or bug in a intermediary proxy that drops some important headers,the signature will not match.

The use HMAC authentication a digest is computed using a composite of the URI, request timestamp and some other headers (dependeing on the implementation) using the supplied secret key. The key identifier along with the digest, which is encoded using Base64 is combined and added to the authorisation header.

The following example is from Amazon S3 documentation.

"Authorization: AWS " + AWSAccessKeyId + ":"  + base64(hmac-sha1(VERB + "\n"
                   + CONTENT-MD5 + "\n"
                   + CONTENT-TYPE + "\n"
                   + DATE + "\n"
                   + CanonicalizedAmzHeaders + "\n"
                   + CanonicalizedResource))

Which results in a HTTP request, with headers which looks like this.

PUT /quotes/nelson HTTP/1.0
Authorization: AWS 44CF9590006BF252F707:jZNOcbfWmD/A/f3hSvVzXZjM2HU=
Content-Md5: c8fdb181845a4ca6b8fec737b3581d76
Content-Type: text/html
Date: Thu, 17 Nov 2005 18:49:58 GMT
X-Amz-Meta-Author: foo@bar.com
X-Amz-Magic: abracadabra

Note the AWS after the colon is sometimes known as the service label, most services I have seen follow the convention of changing this to an abbreviation of their name or just HMAC.

If we examine the Amazon implementation closely a few advantages become obvious, over normal user names and passwords:

1. As mentioned HMAC authentication guarantees the authenticity of the request by signing the headers, this is especially the case if content-md5 is signed and checked by the server AND the client.
2. An admin can generate any number of key pairs and utilise them independent of their Amazon credentials.
3. As noted before these are computed values and can be optimised to be as large as necessary, Amazon is using 40 character secrets for SHA-1, depending on the hash algorithm used.
4. This form of authentication can be used without the need for SSL as the secret is never actually transmitted, just the MAC.
5. As the key pairs are independent of admin credentials they can be deleted or disabled when systems are compromised therefor disabling their use.

As far as disadvantages, there are indeed some:

1. Not a lot of consistency in the implementations outside of the ones which interface with Amazon.
2. Server side implementations are few in number, and also very inconsistent.
3. If you do decide to build your own be advised Cryptographic APIs like OpenSSL can be hard to those who haven’t used them directly before, a single character difference will result in a completely different value.
4. In cases where all headers within a request are signed you need to be VERY careful at the server or client side to avoid headers being injected or modified by your libraries (more details below).

As I am currently developing, and indeed rewriting some of my existing implementations I thought I would put together a list of tips for library authors.

1. When writing the API ensure you check your request on the wire to ensure nothing has been changed or “tweaked” by the HTTP library you’re using, mine added a character encoding attribute to the Content-Type.
2. Test that order of your headers is correct on dispatch of the request as well, libraries my use an hash map (natural ordered), this may break your signature depending on the implementation. In the case of Amazon they require you to sort your “extra” headers alphabetically and lower case the header names before computing the signature.
3. Be careful of crazy Ruby libraries that snake case your header names (yes this is bad form) before presenting them to your code as the list of header names.
4. When debugging print the canonical string used to generate the signature, preferably using something like ruby inspect which shows ALL characters. This will help both debugging while developing, and to compare against what the server side actually relieves.
5. Observe how various client or server APIs introduce or indeed remove headers.

From a security stand point a couple of basic recommendations.

1. Use content MD5 at both ends of the conversation.
2. Sign all headers which could influence the result of the operation as a minimum.
3. Record the headers of every API call that may have side affects, on most web servers this can be enabled and added to the web logs (again ideally this would be encoded like what ruby inspect does).

So in closing I certainly recommend using HMAC authentication, but be prepared to learn a lot about how HTTP works and a little Cryptography, this in my view cant hurt either way if you’re building server side APIs.

Update

Based on some of the comments made when I submitted this to hacker news I have compiled some extra links and observations.

One interesting point made was the issue of replay attacks, which is where a valid message is maliciously or fraudulently repeated or delayed. This is either performed by the originator or by a man in the middle who retransmits the message, possibly as a part of a denial of service.

nonce

To protect from these types of attacks a Cryptographic nonce, which is an arbitrary number usable only once in a message exchange between a client and a server. These are in fact optionally used within Digest authentication mentioned previously.

One of the comments linked an article which suggested use of nonce with HMAC as described in RFC 5849 The OAuth 1.0 Protocol. In this specification an nonce is paired a timestamp and included with each message, the timestamp can be used to avoid the need to retain an infinite number of nonce values for future checks, the server can reject messages with timestamps older than window of time nonce values are retained.

Based on the original post I have developed a flexible hmac authentication library called Ofuda for nodejs, this currently contains a small routine to hmac sign a list of headers for a given request. In the near future I plan to add validation of a signature and an implementation of nonce based on the aforementioned strategy.

In the past with my plugin the administrator of the CI server had two options when managing the gems associated with a build:

1. Install all the gems required by the project prior to performing a build
2. Permit bundler to write to the local ruby installations gem directory

In my view neither of these options is ideal, so I decided I would do some research into staging gems within the working copy of a build. The aim here was to build a ruby project without tainting the local ruby installation, which is one of the key objectives of CI. After some reading I discovered that Bundler could help stage my gems, therefore saving me from doing so.

To take advantage of this feature in Bundler the user was required to pass a couple of switches to install command and then run Bundler’s exec command for all subsequent ruby executions.

To illustrate this using the Bamboo Ruby plugin I will run through configuring a build with a simple rails project.

Firstly we need a ruby, which I install using RVM, you could just use the system provided one if your on OSX.

$ rvm install 1.9.3

Then use this install.

$ rvm use 1.9.3

Now the only gem you need to install is bundler.

$ gem install bundler

Now within Bamboo install the Bamboo Ruby Plugin via the Universal Plugin Manager.

Now go to Server Capabilities in Administration section, and click the Detect Server Capabilities button, this should find your ruby installation as seen below.

Next setup a project and a release plan.

Then within your build Job add the Bundler Task, configuring the path option to vendor/bundle and ticking the binstubs option.

Next add a Rake task to run your database migrations.

Next we want to run the tests in our project, in my case I am using RSpec2 so I use the spec rake task.

As I like to see my test outputs in the CI server I have enabled xml output with my RSpec configuration file and added the rspec_junit_formatter gem to my Gemfile. This produces a JUnit XML report file named rspec.xml which Bamboo parses into test results report.

Now you should enable your build and run it, all going well you should have green, otherwise have a look at the build log and see what the issue was.

So that wraps up my demonstration of Bamboo Ruby Plugin using the awesome bundler gem to enable simple ruby build environments.

The aim of this post is to illustrate how this is done and how to craft an environment to run ruby once gems are “staged” within a working copy.

The aim of this post is to illustrate how a rails project is staged using bundler without installing any gems in the base ruby installation.

Firstly I will create a new rails project in an environment similar to that used when developing rails applications. I will be using an installation of ruby with bundler, rake and rails already installed, note I am passing the -T switch as I want to setup an alternate test framework.

rails new somenewrailsproj -T

Once created I navigate into the project and setup the test framework I am intending to use which is Rspec-2 for rails.

Add the following code to the end of the Gemfile.

group :test, :development do
    gem "rspec-rails", "~> 2.0"
end

Run bundle install.

bundle install

Run the Rspec-2 generator to plugin into the rails project.

rails generate rspec:install

Scaffold a sample model.

rails generate scaffold post title body:text published:boolean

Migrate this to the database.

rake db:migrate

Remove the pending specs as their not important

rm ./spec/helpers/posts_helper_spec.rb ./spec/models/post_spec.rb

Run Rspec and we should be all green.

bundle exec rake spec

Now to complete isolate our test environment we need a clean bash shell, to do this we run the following.

env -i bash

Next we export the PATH variable, in my case I only want my specific ruby version, and indeed the only thing I want is this versions bins.

export PATH=/Users/markw/.rbenv/versions/1.9.2-p320/bin

This version ruby has just been installed and therefore only has the base set of gems

$ gem list
minitest (1.6.0)
rake (0.8.7)
rdoc (2.5.8)

To simulate a build server the only gem we will add to this installation is bundler.

gem install bundler

Now we run gem adding the vendor/bundle/ruby/1.9.1 directory to the GEM_PATH this shows the gems we had previously bundled.

GEM_PATH=vendor/bundle/ruby/1.9.1 gem list

Now to illustrate how this will work inside a build environment we need augment our path a little, this will ensure gem install can find tools like compilers and such if required.

export PATH=/Users/markw/.rbenv/versions/1.9.2-p320/bin:/bin:/sbin:/usr/bin:/usr/sbin

Now run bundle install to recreate vendor/bundle and bin

bundle install --path vendor/bundle --binstubs

Now run our specs with the augmented GEM_PATH.

$ GEM_PATH=vendor/bundle/ruby/1.9.1 bundle exec rake spec                             
...
Finished in 0.32614 seconds
28 examples, 0 failures

So I have illustrated how I can stage gems for a rails application and run it’s tests without installing anything in the base ruby. This should work for any gem or project which uses bundler.

Some points to consider about this approach are:

• One should note that ruby has a notion of STD Library API compatibility which is reflected in the ruby/1.9.1 section of the path, this may vary for each release.
• For commercial projects I would recommend using a frozen set of packaged and quality assured gems and running tests with just this set.

That said for things like Octopress projects and gems / projects used internally this is quite a flexible way to run a CI build.

After considering the issue for a bit it dawned on me that this may be a good excuse to try out LessCSS JavaScript library. When incorporated into a website this library enables the developer to use a CSS like markup which significantly reduce the amount of duplication and redundancy in the style sheet. The markup is processed on the client using JavaScript and has an API to mess around with how the styles are loaded.

Initially I planned to invoke the reload function in less library using a timer, however after reading the docs I found this feature was already built into LessCSS. Simply add #!watch to the URL in the browser and LessCSS will poll the style sheet for updates.

On a wide screen monitor this means I can tweak the CSS and watch the changes appear, which in turn removes quite a bit of keyboard gymnastics while working on a design.

To take advantage of these features in your site simply add the following fragment to your html page.

<link rel="stylesheet/less" type="text/css" href="css/styles.less">
<script src="js/libs/less.js" type="text/javascript"></script>

Then move all your styles to css/styles.less within your mockup/site and reload the page. To enable auto reload of the styles.less file append #!watch to the URL and refresh the page.

One thing to note is you will need to serve the site using a web server of some sort otherwise you will get XHR issues, to do this on OSX I use a python one liner.

$ python -m SimpleHTTPServer

To illustrate this feature I have created a sample project up on github lesscss_watch_example_site.

I am currently working on a site which will also live reload page fragments using Ember and Handlebars.

So to start with there are some things you should learn before beginning:

Maven

The entire plugin development kit revolves around it so you need to understand it, have a read over the Maven Reference and add it to your bookmarks.

The first thing I do when coming back to maven is practice the release process, for most developers this is one of the most frustrating and complicated areas of maven so practice it.

Generate a test Java project using the basic archetype, and push it up to your version control site of choice, either bitbucket or github is fine, and work through the development cycle. Make a few changes check them in and then perform a release, this process normally takes me a few goes to get all the settings right in your maven project.

I recommend you use this approach to re-familiarise yourself with the release process after any long breaks as well, this will ensure maven hasn’t change since you last did it, and you don’t make a mess of your plugin project.

Once you have created your plugin project ensure you fill out all the relevant meta information in your maven pom file, as seen in the sample below. In addition to it being a good practice to do so this information can be used by maven plugins you may include in your project in the future.

    <description>This is the ruby rake plugin for Atlassian Bamboo.</description>

    <organization>
        <name>Mark Wolfe</name>
        <url>http://www.wolfe.id.au/</url>
    </organization>

    <developers>
        <developer>
            <name>Mark Wolfe</name>
            <email>mark@wolfe.id.au</email>
        </developer>
    </developers>

    <licenses>
        <license>
            <name>Apache 2</name>
            <url>http://www.apache.org/licenses/LICENSE-2.0.txt</url>
            <distribution>repo</distribution>
            <comments>A business-friendly OSS license</comments>
        </license>
    </licenses>

The Java Ecosystem

A small part of this ecosystem is downloaded to your system when you run maven to build your plugin, so I recommend you do a little bit of reading on some of them, the ones I have listed below are a few of my favourites. I myself are a big proponent of the old saying “When in Rome, do as the Romans do”, for this reason I will always try and use the libraries which are already in the SDK.

• Google Collections, in my view one of the core libraries which a Java developer should know.
• SLF4J, one of the many logging abstractions which are used in Java projects but the one I tend to prefer.
• Apache Commons Lang, this library has quite a few utility classes for manipulating strings as well as builders for toString and equals methods in classes.
• Spring Framework, most of the Atlassian products are built using this dependency injection framework so it is handy to understand a bit of how this works.
• JUnit, this unit testing framework has been around for a long time for good reason, learn how to use it.
• Mockito, because mocking is a BIG must when building something in a large application so learn this API and ensure it is included in your plugin project from the start.

Development process

This is one of the areas which is often left up to the developers themselves to manage so for this reason I typically follow a simple process, especially when I am working on open source projects.

1. Before you start write down what you want to achieve, keep things simple and don’t plan world domination at this stage.
2. Build a first release focusing on the goals more than the method, the goal is to prove the concept and most important ship it.
3. Do some research now that you know what you looking for, read other peoples code and hack on your initial release a bit.
4. Delete your code, start the whole thing again, this sounds nuts but your proof of concept code is probably best left behind (see Corey Haines Code Retreat).
5. Build a new release from scratch with more of a focus on structure, testing and extensibility, and again ship it.

The CLI tools I am referring to are:

• jps - JVM Process Status Tool
• jstat - JVM Statistics Monitoring Tool
• jhat - Java Heap Analysis Tool
• jstack - Java Stack Trace Tool

The tools I use most commonly are jps, jstat and jstack, the jhat tool is also very handy but really needs an entire blog post to itself as it crazy what you can do with it. In this post I have put together some tips, observations and sample outputs to illustrate how I use them.

As I am using ubuntu 11.10, which only installs the Java runtime environment (JRE) I will need to install the JDK. In my case I decided to give openjdk 7 a shot, but version 6 would work just fine.

root@oneric:~# apt-get install openjdk-7-jdk

To try out these commands I have installed tomcat7 this can be done through apt on ubuntu, again the previous version being tomcat 6 would be fine.

root@oneric:~# apt-get install tomcat7

Now that I have tomcat installed I want to list the Java processes, note that it is best to assume the same user account as the service when doing this. On ubuntu I would su to the user account, as the tomcat7 user is a system account I have to override the shell as it is /bin/nologin by default, I can then run jps as this user.

The jps command outputs the PID of the java process along with the main class name and the argument(s) passed to it on startup.

root@oneric:~# su - tomcat7 -s /bin/bash
tomcat7@oneric:~$ jps -ml
12728 org.apache.catalina.startup.Bootstrap start
13926 sun.tools.jps.Jps -ml
tomcat7@oneric:~$

Now that we have the PID of these processes we can run jstat, the first switch I use is -gcutil this gives me an overview of the heap use within the jvm. In cases where there are pauses or performance degradation I will look at the last two columns. These contain the garbage collection time (GCT) and full garbage collection time (FGCT). If the FGCT column is increasing every second then it is likely we have an issue.

The following example I am running jstat against the PID of tomcat. I have also instructed the command to display the table headers every 20 rows and print the statistics continuously with an interval of 1000 milliseconds, as normal control C with end the output.

This sample shows a newly started tomcat 7 with very little happening, this is clear from the values in the full garbage collection time(FGCT) and garbage collection time(GCT) columns.

Also of note is the permgen space (P) which is currently sitting at 70%. The permgen space is an important area of the heap as it holds user classes, method names and internal jvm objects. If you have used tomcat for a while you will have seen the java.lang.OutOfMemoryError: PermGen space error wich indicates when this space fills up and cannot be garbage collected. This frequently happened when redeploying large web applications.

Also in the sample we can see that the Survivor 0 (S0), Survivor 1 (S1), Eden and Old spaces have quite a bit of free space which is good.

tomcat7@oneric:~$ jstat -gcutil -h20 12728 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032
  0.00  17.90  32.12   4.81  71.41      5    0.009     1    0.023    0.032

To illustrate what a tomcat under load looks like in comparison we can install a tool called Apache bench.

root@oneric:~# apt-get install apache2-utils

And run the following command to hit the base page with a large number of requests concurrently.

markw@oneric:~$ ab -n 1000000 -c 100 http://localhost:8080/

Below is the output after this test was run for a bit, as we can see there has been considerable growth of the survivor 1, eden and old space, however the server hasn’t spent a lot of time doing full garbage collects as indicated by the value of the full garbage collection count(FGC) which is only 10, most of the work is in the young generation as seen by the increase in the young generation collection count (YGC).

Also to note here is that there wasn’t a lot of change in the permgen space, it actually went down, this was due to an increase in size of heap.

tomcat7@oneric:~$ jstat -gcutil -h20 12728 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250
  0.00 100.00  52.02  81.84  59.62    117    1.176    10    0.074    1.250

To look deeper into the cause of garbage collection we use the jstat command with the -gccause option, this displays the same columns as the previous command but with two extras which supply the reasons for GC.

In the following example we can see an example of an allocation failure, this indicates that a full gc is being performed because the heap is too small.

tomcat7@oneric:~$ jstat -gccause -h20 12728 1000
100.00   0.00   0.00  78.91  59.67    168    1.680    14    0.083    1.763 unknown GCCause      No GC
100.00   0.00  72.61  83.73  59.67    170    1.698    14    0.083    1.781 unknown GCCause      No GC
  0.00 100.00  46.24  91.83  59.67    173    1.729    14    0.083    1.811 unknown GCCause      No GC
100.00   0.00  11.39  29.80  59.67    176    1.759    16    0.086    1.846 unknown GCCause      No GC
100.00   0.00  92.41  35.30  59.67    179    1.777    16    0.086    1.864 unknown GCCause      Allocation Failure
  0.00 100.00  62.58  43.05  59.67    181    1.803    16    0.086    1.889 unknown GCCause      No GC

Another area which I like to look into when diagnosing performance issues is the threads running in the vm. This can help me undertand if any component is overloaded and therefore operating a lot of threads trying to catch up. This is mostly only applicable to async processes like messaging, or scheduling routines.

To dump a list of threads and their current stack us the jstack command as illustrated by the sample below, again I normally run this as the owner of the process.

tomcat7@oneric:~$ jstack 12728
2011-10-16 14:53:58
Full thread dump OpenJDK 64-Bit Server VM (20.0-b11 mixed mode):

"Attach Listener" daemon prio=10 tid=0x00000000015be800 nid=0x4004 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"http-bio-8080-exec-182" daemon prio=10 tid=0x00007f9d84274800 nid=0x3cd3 waiting on condition [0x00007f9d7a0df000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000000ef16da38> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:386)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
        at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:679)
...

I plan on doing a bit of work on some visualisation tools, in jruby of course, however that can be the focus of my next post. In the process of writing this post I located some interesting articles, these are linked below:

• Chuk-Munn Lee of Sun Microsystems Troubleshoots Java SE 6 Deployment
• Explaining java.lang.OutOfMemoryError: PermGen space

After watching this great presentation on chef by @benr from joyent, I decided it was time to roll up my sleeves and get started with chef. To do this I took some of his advice and my meager Unix knowledge and crafted a simple bootstrap method for my development Ubuntu systems, which I will describe in this post.

Firstly I get myself an ubuntu development system which I use to build my develop the profile I will use for this system in the future. If this is done locally I grab the ubuntu install CD for the server version and follow the default installation options with the only service I install being openssh. If your using a VPS this is typically what you get out of the box.

Because I am doing this on a local server I normally use ssh-copy-id to copy over my ssh public key the easy mode way.

ssh-copy-id admin@ubuntuserver

Next I run my bootstrap script to install my environment on the server, note the link below is retrieving a specific revision based on the raw link in the gist this may change based on my updates.

ssh admin@ubuntuserver -t -C 'curl https://raw.github.com/gist/3328844/2f4d74d49f8f7a2cd0b7a83a23fafe75d21241cf/gistfile1.sh | sudo bash'

In the next few steps I export my chef-solo template project to the system and build up a recipe for producing this type of system. You may do this differently, I have a script with all these commands in it but I have exploded it for this example.

I start my dev cycle by generating a new ssh key pair and upload that to github or bitbucket depending which one you use.

ssh admin@ubuntuserver -t -C "ssh-keygen -t rsa -b 4096 && cat ~/.ssh/id_rsa.pub"

Make the chef directory and chown it for my admin user.

ssh admin@ubuntuserver -t -C "sudo su - -c '(mkdir /var/chef && chown admin:admin /var/chef)'"

Clone my git project whilst retaining ownership of the files by my admin user, for those new to git see the awesome git book.

ssh admin@ubuntuserver -t -C "git clone git@github.com:wolfeidau/chef-solo-base.git /var/chef"

Initialise the sub-modules and update them, this is something I always forget unless I have a script to follow..

ssh admin@ubuntuserver -t -C "cd /var/chef && git submodule init && git submodule update"

Now you can run chef-solo just to ensure it is all running as expected.

ssh admin@ubuntuserver -t -C 'sudo chef-solo -c /var/chef/solo.rb -j /var/chef/node.json'

This template is comprised of:

• solo.rb - Glue code which loads various directories.
• node.json - This is the dna for your system and will be available to your recipes as meta.
• cookbooks - This directory holds the cookbooks. ** main - This cookbook is where my default recipe is located and all its associated templates. *** templates - This holds all my templates which I use to craft new or replacement configuration files. ** openssl - This is my first external cookbook, once I work out what I am doing I tend to externalise a few functions using other peoples cookbooks.

Using the handy tree command we can see the overall structure of the template.

markw@chefdev1204:/var/chef$ tree
.
├── cookbooks
│   ├── main
│   │   ├── recipes
│   │   │   └── default.rb
│   │   └── templates
│   │       └── default
│   │           ├── bambooxml.erb
│   │           ├── pg_hba_conf.erb
│   │           ├── screenrc.erb
│   │           ├── serverxml.erb
│   │           ├── tomcat7.erb
│   │           └── zshrc.erb
│   └── openssl
│       ├── CHANGELOG.md
│       ├── CONTRIBUTING
│       ├── libraries
│       │   └── secure_password.rb
│       ├── LICENSE
│       ├── metadata.rb
│       ├── README.md
│       └── recipes
│           └── default.rb
├── node.json
├── README.md
└── solo.rb

8 directories, 17 files

In my example I have commented out a whole section of example code which bootstraps my CI server using chef, this is gives me some starting points. Note that I am not an authority on either chef or ruby so my scrappy sysadmin code may make some people grimace but it is a starting point.

Once you’re ready to start hacking remove the remote repo and add your own.

git remote rm origin

Next some rules I try and live by when using chef:

1. Keep it simple, if a recipe looks complicated or you don’t understand it don’t use it.
2. Don’t use chef as an alternate package manager, make packages using fpm and install them.
3. Please reread #1.

Most importantly get started with this tool you will never look back once you have a few systems built using it.

To add more recipes from opscode-cookbooks account simply navigate to the base of your project and run something like the following example.

git submodule add https://github.com/opscode-cookbooks/openssl.git cookbooks/openssl

Once I have completed my chef project I check it out on my workstation and use rsync to push it to a clean target host for testing.

Again I run ssh-copy-id to copy my ssh key.

ssh-copy-id admin@ubuntuserver

Bootstrap chef onto the server.

ssh admin@ubuntuserver -t -C 'curl https://raw.github.com/gist/3328844/2f4d74d49f8f7a2cd0b7a83a23fafe75d21241cf/gistfile1.sh | sudo bash'

Create my chef directory.

ssh admin@ubuntuserver -t -C "sudo su - -c '(mkdir /var/chef && chown admin:admin /var/chef)'"

Using rsync and a locally checked out repo copy the files to the remote machine.

rsync -axvr -e ssh my-chef-project/ admin@ubuntuserver:/var/chef/

Since it’s recent release I have seen quite a few posts praising the new JIRA installer so I decided to give it a try at work. First thing that struck me when I went to download it was the linux version was a single file, no deb or RPM(s).

I copied the file to the Linux server and logged in as the jira user I wanted to run the service under, and ran the installer. The first thing the installer did was inform me I wasn’t running as an administrator, and as such it wouldn’t install a startup file for the JIRA. As this was a VM I took a snapshot and decided to try running it as root. When I did it asked me a couple of questions about where I would to put parts of the installation and then went off and did it’s thing. Once it completed I examined what it had done, these were the results:

1. Created a jira1 user.
2. Installed a startup script in /etc/init.d.
3. Installed a version of tomcat.
4. Installed a version of Java.

The first point is quite amusing as I already had a user called jira on the system and rather than ask me whether it should use it, it went and created jira1. Now most installers I have run recently at least had the decency of telling me before adding a user to my system, especially when they discover their preferred user id is currently in use. I also noted rather than set the user’s home directory to the location of the services working data it had just used the default add user leaving a pretty much unused home directory in /home. Overall user creation and use could be much better.

The second point was fine until I opened the script, when I did I was astounded to find a very brief script as follows:

#!/bin/bash

# JIRA Linux service controller script
cd "/opt/atlassian/jira/bin"

case "$1" in
    start)
        ./start-jira.sh
        ;;
    stop)
        ./stop-jira.sh
        ;;
    *)
        echo "Usage: $0 {start|stop}"
        exit 1
        ;;
esac

Now there are a numerous issues with this init script, firstly changing directory within a startup script and then running a script in the current working directory is a big no no, not only is it a potential security hole waiting to happen, it is also just plain bad form. In the case of tomcat itself there are a couple of system variables which instruct it where it’s base files are and where it’s working data is located. Using these variables removes the need for change directory (CD) in scripts and is much safer.

This script also not using any of the nice shell functions present in most linux distributions, for instances the linux standard base (LSB) functions which identify the distribution of Linux this script is running on, or the daemon functions which help run your service. Even worse is the fact this init script also calls another script which changes user context, then calls another script which starts the JVM. The tragedy of this multilayered shell abomination is further compounded by the fact the last script in the chain is just the default tomcat catalina.sh script.

The catalina.sh this script is a good script, however it is quite a generalised script primarily designed for use during development. All that is really required to start tomcat correctly is a small piece at the end of this script with paths configured based on the installation, along with some tuning for the application and the user it is to run under. As previously mentioned this is all typically available in shell functions within the init system.

The third point is a one of the ones I really have an issue with, the installer has put an unknown version of tomcat onto my system. As tomcat is a web server it is unfortunately targeted for exploitation, and does have the odd security issue. This would be ok if you ensured I had a method of updating said tomcat but alas you just dump it in your location of choice and leave me holding the baby. Note upon searching I noted tomcat bundled with JIRA, which is version 6.0.32, does indeed have a few security advisories, not a good start.

And lastly this installer has gone and dumped another copy of java on my system, again with no information on what version this is, and again no way of upgrading it if there is any security exploit for it. So overall not very impressed at all with this new installer.

Overall I think what astounds me most is that a company of this size is completely ignoring recognised best practices when packing software for Linux. Considering the customers who know least about linux are most likely to use this installer blissfully unaware of the traps it has dragged them into I am very disappointed.

Now as I am continually reminded by my boss, if I am going to take the time to present issues I should always accompany it with some suggested solutions to them. So my suggestions are as follows:

1. Work out what linux distributions your customers are running.
2. Review how each of these distributions package their software, in general be a good citizen in these operating systems.
3. Set up a couple of repositories for packages tuned for ease of deployment on the main couple of distributions. Essentially make it as easy as possible for a novice user to deploy and update your products using these distribution points.
4. Write a nice init script following the best practices promoted by each distribution.
5. Break your system into it’s distinct components, in a similar way to how you modularise your software, pull the system down to a few discreet packages, ideally with upstream version numbers.
6. Offer updates to these packages to ensure your customers data is secure.

One of things I respect about Microsoft and Apple is they have a keen focus on ensuring people who deploy their products look like wizards, a few clicks and everything is up and running. In my opinion the user experience for the person installing and maintaining this software is as important as the end user. In a lot of cases this person has the potential to be one of Atlassian’s greatest assets, so please don’t mess them around.

As Atlassian has had some quite high profile compromises, for example the Apache foundations JIRA incident I would have assumed that providing a secure method to install and operate their software would be at the top of their priority list.

Lastly I am always happy to sit down and have a chat with anyone from Atlassian, preferably over a cold beer. I have been using their products for 8 or 9 years, yes I even had a support request responded to by one of the founders. Atlassian’s products have served me and my customers well so this is the least I could do.

After quite a bitter fight with ant at work I decided it was time to give one of thes new contenders a try. As I had recently tinkered on a project which used Gradle and found it quite easy to use it seemed like as good a candidate as any.

My road test involved building a multi module project with a sample shared common library, a server library and a REST service. The results of my experiment are located in agent-manager up on github. Overall I found building this project quite enjoyable and using code, albeit groovy which I am not that familiar with, quite refreshing.

After a bit of consideration I would list the pros as follows:

• Good overall documentation
• Dependency management was a breeze
• Great support for generating IDEA projects, and some great options for customising them.
• Nice terse syntax with the ability to follow the Don’t Repeat Yourself Mantra(DRY).
• Very flexible build and test configuration, in my example I added a new integration test scope which was completely isolated from the existing unit tests.
• Very little code required to get a nice project build running.

The cons are as follows:

• Not a lot of plugins, in my case none of the major web service runtimes have a gradle plugin. To be honest I didn’t find a lot outside of this projects default ones.
• Some of the syntax errors can be cryptic, especially when you are missing a parameter to a closure.
• Not a lot of good examples.

So overall quite an interesting exercise, in my case I will keep at it as the cons were well and truly outweighed by the pros. I would be happy to recommend Gradle to anyone who is looking for something new, especially on simple java projects.

If your interested in learning more then have a look over my sample and make sure you checkout the following projects for inspiration.

• Hibernate Core
• Spring Integration
• Groovy

Install the development suite for Ubuntu.

sudo apt-get install build-essential

Install the Git version control package and curl http client utility

sudo apt-get install git curl

Install the development packages which ruby and it’s utilities depend on.

sudo apt-get install zlib1g-dev libssl-dev libreadline-dev

Install the packages required to build nokogiri, which is an xml library used by rails.

sudo apt-get install libxml2-dev libxslt1-dev

Install the command line SQLLite and development package, note the command line tools aren’t required but are very handy.

sudo apt-get install sqlite3 libsqlite3-dev

Install MySQL client, server and development packages.

sudo apt-get install mysql-client mysql-server libmysqlclient15-dev

Install the PostgeSQL client, server and development packages.

sudo apt-get install postgresql postgresql-client postgresql-contrib libpq-dev

Run the following command to install RVM

bash < <(curl -s https://rvm.beginrescueend.com/install/rvm)

Source the .bashrc to make put the RVM command(s) in your path

source .bashrc

Install ruby 1.9.2, this compiles the runtime and then installs it.

rvm install 1.9.2

Enable the ruby 1.9.2 runtime as the default.

rvm use 1.9.2 --default

Update your gem command.

gem update --system

Log into the MySQL command line interface using the password which was set for the root user at installation.

mysql -uroot -p

Set up a rails user with all permissions on the activerecord test database.

GRANT ALL PRIVILEGES ON activerecord_unittest.* to 'rails'@'localhost';
GRANT ALL PRIVILEGES ON activerecord_unittest2.* to 'rails'@'localhost';

Create a user with super user privileges in PostgreSQL for the login your currently authenticated under.

sudo -u postgres createuser --superuser $USER

Navigate to where you store the code your working on and clone the rails repository.

cd ~/Code/Ruby
git clone git://github.com/rails/rails.git
cd rails

Install the latest version of bundler.

gem install bundler

Install all dependencies including the MySQL and PostgreSQL drivers.

bundle install

Run the tests.

rake test

For more information on this process see Contributing to Ruby on Rails.

One of the first things that struck me was how dependant I was on xcode to get this up and running, and how damn confusing this was for the uninitiated. The myriad of windows you end up with in xcode 3.2.x was sending me crazy, this paired with the fact I a certain window was my project led me to upgrade to xcode 4.x.

So without further a do I have gathered together some screenshots on how I got my tests up and running once I had Xcode 4 installed.

Step 1: Create a project, in my case I am focused on Mac OS X desktop applications so I select Application from the options.

Step 2: Enter a project name and company, then disable unit tests as this will include OCUnit which I am not currently using. Save it where you keep your projects.

Step 3: Now add a new target, you should have a button down the bottom to Add Target, again click Mac OS X and select Application from the options and enter the details.

Step 4: Delete all the files which we don’t need.

Step 5: Remove the main nib file from the TestAppTests-Info.plist.

Step 6: Download GHUnitTestMain.m from git hub and then drag and drop it into the TestAppTests folder in your xcode. Then add it to the TestAppTests target.

Step 7: Download the GHUnit framework extract it, then copy it to your /Library/Frameworks. Then select the TestAppTests target and click the plus button. Search for the GHUnit.framework and add it.

Step 8: Create a new test by creating a new empty file named SampleTest.m in the TestAppTests folder in Xcode and add the following code. Also ensure it is added to the TestAppTests target.

//
//  SampleTest.m
//  TestApp
//
//  Created by Mark Wolfe on 14/06/11.
//  Copyright 2011 N/A. All rights reserved.
//

#import <Foundation/Foundation.h>
#import <GHUnit/GHUnit.h>


@interface SampleTest : GHTestCase {}
@end


@implementation SampleTest

- (void)testSomeStringsEqualEachOther{
    
    NSString *someString = @"Have me some objective c testing";
    
    GHAssertEquals(@"Have me some objective c testing", someString, @"The strings should match bro");
}

@end

Your project should look as follows.

Step 9: Now set the active scheme to TestAppTests and click the play button, all going well you should see the following application pop up with your test listed in it ready to run.

Now that is all setup you should be able to add tests to your hearts content. So far GHUnit has enabled me to test a wide array of routines and frameworks which my application will rely upon.

Prior to attending one of these events there are things you should bring:

1. A laptop set up with a range of tools installed, depending on your skill set and development stack.
2. A pad pen and some butchers paper for big diagrams.
3. If your a developer make sure you have played around with at least one online collaboration tool. I myself use github for source control and Wiki.
4. Make sure you have tested your laptop with an external display or projector, you will need to demo the stuff you build!
5. Ensure you have a copy of either Power Point or Keynote installed so you can build a presentation to pitch your ideas.

Having listened to the summary around each of the proposed problems and a bit of background on each we were instructed to divide ourselves into groups and begin work on each of the problems. This is an interesting time for each of the groups, with some people having never met before. Under the guidance of a mentor or problem owner the initial discussions begin.

For those new to this sort of situation I can offer the following tips:

1. Ask a few simple questions to break the silence and help trigger some discussion around the problem.
2. Do a role call and ask what role each member can fill whether it be designer, front end developer, backend developer or business analyst.
3. For those in the development arena see what stack they are comfortable developing in, whether it is Java, PHP, Rails or .NET. In this sort of situation look for ways to fit people in, a little bit of give and take can go a long way.
4. Bootstrap the project with some version control tools and a wiki before anyone gets to carried away. It is crucial all the assets created are out in the public domain for those that decide to continue hacking on the project. For me this means either setup a github repo or a google code project.
5. Lastly if you see anyone struggling to fit in go and have a chat to the other teams and see if you can find a place for them in another project.

Another interesting facet of the day was that each group was at different stages of development. one of the groups had already been established in a previous event and was looking to add / develop on the existing solution. While others were new problems with some prepared specifications and background documentation. And lastly the one I chose which was a problem which was completely new and had only preliminary investigation done. The only thing I can recommend here is do a bit of research around each of the problems presented for that RHOK event and choose something that interests you, then go with your gut on how to proceed. I our case we just had a good round table discussion about the problem then listened to some great insight offered by people who had experience in the problem area.

So what can I offer when trying to get started like me in a completely greenfield project:

• Google a lot to try and find ideas
• Talk to the event organisers and see what strings they can pull or who they can pull in to help, in my case this was VERY helpful.
• Start brain storming ideas as early as possible
• Draw a few concepts on paper
• Discuss where to get the data for these ideas and build a bit of a schema or structure.
• Get a mock up started as soon as possible
• Keep a note of any ideas and problems for later, this is very important as you will have to assemble some sort of presentation at the end of the event.

Lastly have fun and try new cool stuff, this will add to the pressure but also give a chance to test yourself.

So why start with html5boilerplate at all? Well the following things come with it:

• modenizer included and configured for legacy browser support.
• A cool favicon.ico to eliminate those 404 errors I always get in my logs, as well as apple touch icons.
• A base project structure with files originised into a nice layout.
• JQuery bundled in and ready to roll.
• A great starting point annotated css file with a whole raft of tips and tricks.
• Sample site compression build scripts in Apache Ant.
• A nice example of how to load the google analytics script asynchronously.

Even for those maintaining an existing site there are quite a few very handy lessons to learn from this project. Personally I like all the server configuration file examples located at html5-boilerplate-server-configs, this is a very important facet of any site and is often overlooked.

So in summary I have learnt a lot of good lessons while working with this project and I will be merging some of these into my own blog site.

So being a happy consumer of open source software, I had a browse over the sources to Apache CXF tools and discovered the method for retrieval of the WSDL files was using java.net.URL. To enable a proxy server for use by this class is as simple as passing some extra switches to the mvn command as in the following example.

$ mvn -Dhttp.proxyHost=proxy -Dhttp.proxyPort=8080 package

Once we had overcome this issue we hit another interesting hurdle. My integration tests in this Maven project were using the spring configuration method, these were also failing. Turns out we also needed to set the proxy in the Apache CXF configuration as well. This was done using a conduit as follows.

<http-conf:conduit name="*.http-conduit">
    <http-conf:client ProxyServer="squid.wolfe.id.au" ProxyServerPort="3128"/>
</http-conf:conduit>

So in summary if your working behind a proxy server building web services projects using Maven and Apache CXF you will need to do the following.

Configure a proxy in your Maven configuration so that assets can be retrieved, this is done as follows in your settings.xml.

     <proxy>
       <id>optional</id>
       <active>true</active>
       <protocol>http</protocol>
       <host>squid.wolfe.id.au</host>
       <port>3128</port>
       <nonProxyHosts></nonProxyHosts>
     </proxy>

Whenever you invoke any tests or calls which invoke wsdl2java you will need to pass the proxy settings in as previously described.

$ mvn -Dhttp.proxyHost=proxy -Dhttp.proxyPort=8080 package

When running any tests or routines which use Apache CXF driven web services you will need the conduit configured, in this example it is global and applies to all http connections.

<http-conf:conduit name="*.http-conduit">
    <http-conf:client ProxyServer="squid.wolfe.id.au" ProxyServerPort="3128"/>
</http-conf:conduit>

So there is no confusion about which libraries I am they are:

• Maven 2.2.1
• Apache CXF 2.4.0
• Sun\Oracle Java 1.6_24

So overall a frustrating day, but I won in the end, now all i need to do is incorporate all this information in my project so the onsite developer can work. Hopefully this post helps someone people that run into the same issues.

I then spun up a clean Ubuntu server running 10.04.02, yes old trusty Long Term Support release, and ran through the following steps.

First install the Sun Java development kit, I use this in preference over OpenJDK at the moment for QA as it is what I normally deploy to for customers. To install this package we need to modify the /etc/apt/sources.list and remove the comments at the start of the lines which include the partner repositories. These look as follows

deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner

Run apt-get to update the cache of packages.

$ sudo apt-get update

Then install the dpkg for Sun\Oracle JDK.

$ sudo apt-get install sun-java-jdk

I will be using PostgreSQL for my database follow the configuration process described in Ubuntu PostgreSQL Installation Howto.

Now create a PostgreSQL user and database, note you will be prompted to enter a password for the bamboo user.

$ sudo -u postgres createuser -R -D -S -P -e bamboo
$ sudo -u postgres createdb -O bamboo bamboo_db

To test this is login will work fine I use the psql command and pass it a hostname to connect to using TCP/IP, this is to ensure our JDBC driver will connect fine.

$ psql -h localhost -U bamboo bamboo_db
Password for user bamboo:
psql (9.0.4)
SSL connection (cipher: DHE-RSA-AES256-SHA, bits: 256)
Type "help" for help.

bamboo_db=> \q

Install tomcat6 using apt, the reason I am using this package rather than the all in from Atlassian is I like my tomcat maintained. Dumping a tomcat in your OS with no one watching your ass is unwise in my opinion.

$ sudo apt-get install tomcat6

Rather than just popping the war file into /var/lib/tomcat6/webapps, I prefer to externalise the application by extracting the archive into it’s own directory under /var/lib. I then configure tomcat to load the application from this location. This is done so that I can guarantee tomcat upgrade won’t nuke or otherwise disturb my Bamboo installation.

$ sudo mkdir /usr/share/atlassian-bamboo-3.1.1
$ cd /usr/share/atlassian-bamboo-3.1.1
$ sudo jar xvf ~/atlassian-bamboo-3.1.1.war

Before we start ensure tomcat is shut down, otherwise it mite go and deploy bamboo before we are ready!

$ sudo /etc/init.d/tomcat6 stop

Next we need to configure tomcat to use load this web application, to do this we create navigate to the tomcat configuration directory.

$ cd /var/lib/tomcat6/conf/Catalina/localhost

Backup the original ROOT.xml, being careful to preserve the permissions.

$ sudo cp -ipv ROOT.xml ROOT.bak

Edit file named ROOT.xml as root, I will be making my bamboo the ROOT application in this tomcat, in other words served at http://myserver.com/. Put the following content in this file. Note you also need to generate a password for your DB login and enter it where the XXXXXXXX is.

<Context path="/" docBase="/usr/share/atlassian-bamboo-3.1.1">

  <Resource name="jdbc/BambooDS" auth="Container" type="javax.sql.DataSource"
            username="bamboo"
            password="XXXXXXXX"
            driverClassName="org.postgresql.Driver"
            url="jdbc:postgresql://localhost:5432/bamboo_db"
            />

</Context>

Make the ROOT.xml file only readable by the tomcat user to protect the plain text password located within.

$ sudo chmod 600 ROOT.xml

Remove the current root web application from tomcats /var/lib/tomcat6/webapps directory.

$ sudo rm -r /var/lib/tomcat6/webapps/ROOT

Create a data location for bamboo, and change the file permissions so that tomcat6 can write to this location.

$ sudo mkdir /var/lib/atlassian-bamboo
$ sudo chown tomcat6:tomcat6 /var/lib/atlassian-bamboo

Now install the PostgreSQL driver in tomcat, to do this first download it to your home directory then copy it to /usr/share/tomcat6/lib as follows.

$ cd ~/
$ wget http://jdbc.postgresql.org/download/postgresql-9.0-801.jdbc4.jar
$ sudo cp postgresql-9.0-801.jdbc4.jar /usr/share/tomcat6/lib

Modify the /usr/share/atlassian-bamboo-3.1.1/WEB-INF/classes/log4j.properties as root, to correct the location of the log file. This in my opinion needs to be a MUST configure item during there installation process otherwise the log file could end up anywhere, for example the PWD of the executing start up script.

Navigate to the line that looks like this.

log4j.appender.filelog.File=atlassian-bamboo.log

And change it to this.

log4j.appender.filelog.File=${catalina.base}/logs/atlassian-bamboo.log

Also you will need to configure the bamboo.home in /usr/share/atlassian-bamboo-3.1.1/WEB-INF/classes/bamboo-init.properties. This should be changed to the following value. Again this will need to be done as root.

bamboo.home=/var/lib/atlassian-bamboo

Before we start tomcat we need to increase the amount of memory available to tomcat as well as some other params. This is done by editing /etc/default/tomcat6.

Open the file as root and edit to the following line.

JAVA_OPTS="-Djava.awt.headless=true -Xmx128m"

Change it to the following value.

JAVA_OPTS="-server -XX:MaxPermSize=256m -Djava.awt.headless=true -Xmx512m"

Now start tomcat and then tail the server log file.

$ sudo /etc/init.d/tomcat6 start
$ tail -f /var/log/tomcat6/atlassian-bamboo.log

Now open your browser and to http://servername:8080/ and follow the prompts.

When prompted for data base, select use datasource and specify the following value.

java:comp/env/jdbc/BambooDS

Now I have a nice new Atlassian Bamboo server ready to build my software, more on how that goes in future posts.

The basis for this issue revolves around “wrapping” in JSON and two different schools on what is correct way to encode it. As far as I can see there is the more verbose version which Jettison, the default JSON serialiser in Apache CXF produces, then there is the “unwrapped” version which the alternate serialiser Jackson produces.

In my case I chose Jackson the more terse version, this is good for a couple of reasons:

• It is compatible with ExtJS without any modifications
• It is smaller and therefore produces less data on the wire.

Also I like the annotations that Jackson comes with, and find it a bit easier to work with than Jettison.

So to enable Jackson I modify my projects Maven pom file I add the following dependency.

<dependency>
    <groupId>org.codehaus.jackson</groupId>
    <artifactId>jackson-jaxrs</artifactId>
    <version>1.5.7</version>
</dependency>

In addition to this some changes are required in the spring configuration which houses our RESTful services. In the following excerpt from my spring configuration, I have declared the jsonProvider then set it as one of the providers jaxrs:server.

<bean id="jsonProvider" class="org.codehaus.jackson.jaxrs.JacksonJsonProvider"/>

<jaxrs:server id="restServices" address="/">
        <jaxrs:serviceBeans>
            <ref bean="projectService"/>
        </jaxrs:serviceBeans>
        <jaxrs:providers>
            <ref bean="jsonProvider"/>
        </jaxrs:providers>
        <jaxrs:features>
            <cxf:logging/>
        </jaxrs:features>
</jaxrs:server>

Once Jackson was enabled my ExtJS JSON driven data stores were functioning perfectly, aside from dates. Jackson’s default behaviour for serialisation of a java.util.Date is to convert it to milliseconds since EPOC. To do this I used a feature in spring known as compound property names, this enabled me to instantiate an instance of the mapper, then override the serializationConfig.dateFormat to configure the mapper to produce ISO 8601 dates. This shown in the following excerpt which illustrates the updated jsonProvider using the reconfigured jacksonMapper.

<bean id="jacksonMapper" class="org.codehaus.jackson.map.ObjectMapper">
  <property name="serializationConfig.dateFormat">
    <bean class="java.text.SimpleDateFormat">
      <constructor-arg value="yyyy-MM-dd'T'HH:mm:ss.SZ"/>
    </bean>
  </property>
</bean>

<bean id="jsonProvider" class="org.codehaus.jackson.jaxrs.JacksonJsonProvider"
    p:mapper-ref="jacksonMapper"/>

The result of this is shown in the following JSON sample which illustrates how a project object containing some dates is encoded.

{
    success: true
    message: "Project found."
    data: {
        artifactId: "bobtheapp"
        groupId: "au.id.wolfe.bta"
        inceptionYear: "2011"
        organization: null
        developers: null
        dateAdded: "2011-05-21T20:34:15.862+1000"
        dateUpdated: "2011-05-21T20:34:15.862+1000"
        version: "1.0.0"
    }
}