Posts

Billing is an integral part of day to day AWS account operation, and to most it seems like a chore, however there is a lot to be learnt interacting with AWS Billing data. So why would you ever want to dive into AWS Billing data in the first place? It is pretty easy for both novices, and experience developers to rack up a sizable bill in AWS, part of the learning experience is figuring out how this happened. The billing data itself is available in parquet format, which is a great format to query and dig into with services such as Athena. This billing data is the only way of figuring out how much a specific AWS resource costs, this again is helpful for the learning experience. The Cost Explorer in AWS is great if you just want an overview, but having SQL access to the data is better for developers looking to dive a bit deeper. The billing service has a feature which records created_by for resources, this is only available in the CUR data. If you have already you can enable it via Cost Allocation Tags. These points paired with the fact that a basic understanding of data wrangling in AWS is an invaluable skill to have in your repertoire. ...

There has been a lot of press about supply chain attacks recently, these type of attacks are nothing new and understanding them is really important for developers using services such as GitHub Actions, given Continuos integration (CI) tools are a critical part of supply chain used in software projects. A supply chain attack targets less secure parts of the development process, this could be the tools and services you depend on, or the docker containers you host your software in. These attacks come in different forms but some examples are: ...

We have all read horror stories of Amazon Simple Storage Service (S3) buckets being “hacked” in the popular media, and we have seen lots of work by Amazon Web Services (AWS) to tighten up controls and messaging around best practices. So how do the amazon tools help you avoid some of the pitfalls with S3? Case in point, the AWS CLI which a large number of engineers and developers rely on every day, the following command will create a bucket. ...

For some time now I have been working on internal, and some product related services which use AWS events, some of this has been paired with AppSync subscriptions, slack and AWS SNS. To help everyone come up to speed with events, and async messaging in general in a world of REST and synchronous APIs I have been compiling a list of links, which I thought I would share in a post. To start out it is helpful to have an overview, this post and the associated talk Moving to event-driven architectures (SVS308-R1) are a good place to start. ...

Assuming you read my Starting a Go Project post you should have the starting point for a minimal go web service. For your first project it is easier to keep all your code in one folder, in the base of your project, but at some point you will want to restructure things, this is done for a few of reasons: Having everything in one folder results in a lot of inter dependencies in the code. Reuse outside the project can be difficult as the code is only designed to be used in one package. It is impossible to have more than one binary, as you can have only one main method. This post will provide an overview of the structure I follow in my Go projects when building web services. ...

Given the changes with Go Modules I wanted to document a brief getting started for Go projects, this will focus on building a minimal web service. Before you start you will need to install Go, I recommend using homebrew or for ubuntu users Golang Backports, or as last resort grab it from the Go Downloads page. So this looks like this for OSX. brew install go Or for ubuntu we add the PPA, then install golang 1.14 and update our path. ...

In my spare time I spend a bit of time building and working on a scaled down self driving RC project which uses an opensource project called donkeycar. I have been through a few generations of car, learning how both the hardware and software worked, hopefully I can provide some tips on how to avoid at least some of my mistakes in this post. Current Car Starting Out Probably the lion share of lessons, at least initially were learnt about how to setup a custom RC car, especially the power and drive train. I have put together a bunch of recommendations: ...

The AWS Cognito product enables developers to build web or API based applications without worrying about authentication and authorisation. When setting up an applications authentication I try to keep in mind a few goals: Keep my users data as safe as possible. Try and find something which is standards based, or supports integrating with standard protocols such as openid, oauth2 and SAML. Evaluate the authentication flows I need and avoid increasing scope and risk. Try to use a service to start with, or secondarily, an opensource project with a good security process and a healthy community. Limit any custom development to extensions, rather than throwing out the baby with the bath water. As you can probably tell, my primary goal is to keep authentication out of my applications, I really don’t have the time or inclination to manage a handcrafted authentication solution. ...

Early this year amazon web services released the Cloud Development Kit (CDK) which is best summed up by a quote from the GitHub project. The AWS Cloud Development Kit (AWS CDK) is an open-source software development framework to define cloud infrastructure in code and provision it through AWS CloudFormation. Before I go recommending this new project to anyone I most certainly need to road test it myself. This post provides a bit of background on where I work, and why I am looking into CDK, and what I love to see in the future. ...

Step Functions allow you to build pipelines involving one or more amazon, or external service. Some examples of this are: complex customer on boarding processes jobs which provision resources then send a welcome email billing jobs where you may need wait for payment authorisation provisioning users and setup of any resources each user may need pipeline In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, functions, etc.), arranged so that the output of each element is the input of the next; the name is by analogy to a physical pipeline. ...