In this post, we explore the extremes between a centralised model and a completely democratised model, and we suggest a possible nirvana between the two.

Platform Engineering is a hot topic right now - QCon London had an entire track devoted to it in 2023, PlatformCon 2023 saw a massive increase in attendees. It’s seen some huge success in places like Monzo , where the shape of a microservice is strongly defined by a platform team, and engineers use a single CLI command to generate their instance on said platform. This model is great… when it suits your organisation.

Paved Paths (or their similarly named counterparts) have been understandably embraced by the world of Platform Engineering. Using the Team Topologies platform topology, we know that the platform needs to provide a compelling internal product to accelerate delivery by Stream-aligned teams. For a platform to succeed, it needs some form of paved paths through it, which can be used to enable self-service by its users.

The problem with this, is that paved paths aren’t only valid if you’re following platform engineering principles. There’s a range of approaches when creating paved paths that your organisation could follow, each of which have different benefits and drawbacks. Let’s take a look at the spectrum…

The Spectrum

Platform Engineering

At one end of our spectrum, we have Platform Engineering. Engineering teams rely on clearly defined platforms to abstract away many of the ’things that aren’t code’. In a lot of cases, this covers the infrastructure containing the application (Kubernetes, Networking, Storage, Monitoring, Resilience etc) as well as common patterns like observability, cost management and security. The engineers focus on the application code, pump out features, and don’t bother themselves with things like crafting Helm or Terraform or figuring out observability.

You Build It, You Run It (YBIYRI)

At the other end of our spectrum, we have YBIYRI. Engineering teams own everything around their application. They write Terraform, or Bicep, or Helm Charts to define their infrastructure. They define the security configuration of their stack. Likewise, they own the resilience and scaling capabilities of their offering, and monitor all of it to ensure it’s running just how they expect it to.

Paved Paths

Sitting firmly in the middle of our extremes, are our paved paths. They’re not just a subset of Platform Engineering. They actually provide an intriguing and powerful middle-ground for engineering teams, where we can balance some benefits and drawbacks of the two extremes. This could suit your organisation better. Let’s get into why…

What do Paved Paths give us here?

YBIYRI has powerful implications on your people. Engineers will learn huge amounts about what it takes to serve digital products to clients. They’ll own their entire Dev(SecFin…)Ops lifecycle, and start driving the strategy of your technology in ways they never could. This sounds a bit like a beautiful theory, but we’ve seen it in practice. Engineers who took this opportunity by the horns (and accepted the surrounding challenges), and thrived. It’s not all sunshine and rainbows though. Delivery can be slow. Cognitive load is high. The risk of burnout is always just over your shoulder, lurking among the many unknowns being juggled.

You may think Platform Engineering is the only escape from these issues. Abstract it all away. Your engineers only need to care about the code, and the platforms will take care of everything else. However, we need not jump to that extreme. Paved Paths offer us a great helping hand, without hiding innovation opportunities and growth from our engineers. Build paved paths, almost as if you had a platform underneath, but following a YBIYRI model. Engineering teams still own everything, but their cognitive load is reduced by commonly trodden, high-quality paths. Engineers can break away and innovate where they see value, but they still just use a wheel when they need a wheel.

Risks

There are risks with this approach. Abstracting away complexity, without a platform team taking ownership of it, introduces a potential gap. At 2am, the engineering team is still expected to be able to debug an issue, they have no-one else to lean on. Mitigating this risk should mostly be a culture thing. A culture of continuous learning, where teams are encouraged and passionate about understanding their digital products, should fill this gap. Yes, engineers utilise paved paths to deliver faster, but they don’t live in ignorance of what happens ‘underneath’. Their architecture and tech stack is their own, paved paths simply accelerate their delivery. Obviously, the best culture in the world won’t solve this alone, great documentation and a great developer experience is vital. But as long as you’ve read our Part 4 , you should be in a solid position there!

Ownership

By sitting ourselves in the centre of this spectrum, we raise a new question over ownership of the Paved Paths. As we’ve discussed, no specific team owns them. There isn’t a platform team to rely on, and the engineering teams own their products. So how do we maintain them, and ensure they are still valuable to the community? Well, community engagement is a must. If your organisation is running quality production software, it already has the skills and subject-matter experts (SMEs) needed to maintain an incredible set of Paved Paths. They just need to be empowered and encouraged to own building blocks within them.

Engineering teams should contribute back to the Paved Paths they rely on, evolving them as they learn from production. They are the best informed on the state of the developer experience and are best placed to improve it. Encourage your teams to make time to contribute back and normalise taking time for continuous improvement.

Empower the SMEs in your organisation to contribute and own components of your paved paths. DBAs can pour their deep knowledge into sections relevant to databases, driving performance and cost-saving improvements across the organisation. Infrastructure experts can apply their expertise to areas like Infrastructure-as-Code, Networking, and Automation techniques. Security SMEs can take their standards and recommendations, and implement them within building blocks and pipeline tooling. For this to work, an innersourcing model is a great idea - make this part of your paved path contribution model, and make the contribution guidelines clear.

Organisational structures don’t need to change, if the right people are brought together and sponsored by leaders to contribute. As well as the organisational benefits, this is also incredibly attractive to SMEs. Rather than producing documents of standards, and then becoming a gate at some point in the delivery process, SMEs get to see their work ’live’, being part of the solution rather than being a gate that is seen as slowing teams down.

There’s one more group of people who can provide a huge amount of value towards Paved Paths when living on this spectrum. Enablement teams. We’re not going to go into detail of what enablement teams are (maybe a future post outside this series!), but a key area of their contribution is towards these Paved Paths. By being well-connected to their engineering teams, enablement folks are incredibly well-placed to inform on the requirements of all building blocks, whilst also encouraging alignment and fast feedback loops. The team should not own them alone, but instead integrate the engineering community and SMEs with a clear strategy of the paths the organisation requires at that point in time.

Up next, in part 6, we will discuss the important topic of measuring the success of your paved paths - coming soon!

Featured image background generated by Bing Image Creator , updated using excalidraw .

In this post we explore why organisations should care about paved paths.

What problems do paved paths solve?

There is a tension that is constantly at play in software engineering teams - increase their autonomy to enable them to build without hand-offs, while reducing the same teams’ cognitive load to allow them to focus on delivering value. With cloud computing, shifting left on everything, and maturing practices that enable continuous delivery, teams now have the potential to build, run and operate every aspect of their applications. But with this potential, comes the very real threat of burnout, skill gaps that lead to poor quality products, unwanted risks or slow time to market.

We’ve experienced both sides of this dichotomy - so let us quickly explore what each extreme looks like from our lived experience …

Centralised Capabilities

A centralised team (or many of them) abstract away areas of the tech stack from delivery teams. “Our engineers shouldn’t need to know where their app is hosted”. This reduces the cognitive load of the delivery team, which is great. Unfortunately, the delivery team now becomes hugely frustrated by a complete lack of autonomy, needing to log many tickets to the centralised team who have accidentally become a bottleneck. The delivery team also find that they struggle to make the centralised capabilities cover all of their requirements as these are constantly evolving. It’s rigid, but they can’t use anything else. Similarly, the centralised team becomes frustrated with an overwhelming number of tickets to complete and is not able to focus on creating self-service options.

Decentralised Capabilities

Delivery teams own their entire stack. Amazing! Their quest for maximum autonomy has been achieved, and now they can finally get down to delivering features, without blockers from those pesky centralised teams. Except, they now wade through increasing cognitive load, solving the same problems in multiple places, often unaware that 3 other teams solved the problem 6 months ago. Similarly, the centralised team is now disempowered and watches on as more and more inconsistency spreads.

Clearly, neither extreme is a great place to be as an organisation. Fortunately, this is a problem that Paved Paths can solve well - balancing team autonomy without overwhelming teams with too much cognitive load while enabling self-service capabilities.

Does this apply to all organisations?

Nope. Much like deciding when to build a monolith and when to split into microservices, building paved paths isn’t going to help some sizes and shapes of organisations. If you’re a startup with 5 engineers (and ChatGPT), you’re unlikely to see the benefits of paved paths. Your alignment and quality assurance comes from sitting together 12 hours a day, working on the same repository, and having solid PR reviews.

Even as you grow, and have 3 separate engineering teams working on separate microservices, you can probably live without paved paths.

So how do you know you’ve reached the size or shape where paved paths are going to be a valuable investment? To answer this, you need to have a pulse on your tech stack and engineering practices. You’ll start to see certain things popping up across your organisation. Sprawl of tooling. Inconsistent patterns. Engineer frustration. A slowing rate of change, or a higher rate of production failures. These are your signals. Your organisation has too many hikers fighting bears in the woods, and a paved path or two would really come in handy.

Why invest in paved paths?

Paved Paths provide a codified home for all of your learnt best practices, for the things you want to standardise on, for ensuring that things like security, observability and other common components are baked into solutions and only need to be solved once. They also need to be constantly evolving and the usage of them becomes a necessary feedback loop.

In modernisation projects that we’ve worked on before, there have always been reference implementations built that aim to show teams how something could work. They do provide some value. Teams can happily find them, copy-paste, and pick apart to suit their use case. The trouble with reference implementations is that they are generally only good for a specific point in time. If we were to breathe life into all of our documentation, reference implementations and written standards (with a bias towards enabling fast flow) we would start to get the essence of Paved Paths.

Paved paths are more than reference implementations, they are the building blocks in any internal developer platform - empowering teams to get up and running using well-understood design patterns with tried and tested technologies. They aim to provide a standardised experience that solves for common use cases, covering infrastructure, applications and operations.

Invest in paved paths if you would like

• To make doing the “right thing” also the easiest thing, enabling clean architecture
• To increase team autonomy while reducing cognitive load
• To provide self-service, composable building blocks
• To solve common problems once that the 80% are going to experience

Additionally,

• They should be optional for teams to use, to enable experimentation where appropriate
• They are designed to provide the best user experience for engineers (DX)
• They should be secure by design
• They help you implement guard rails, while reducing gates

In a perfect world, teams would never have to learn the same lesson more than once in your organisation. Paved paths form the home where this shared consciousness is stored, iterated on and shared.

Up next, in part 4, we will unpack the anatomy of Paved Paths and all the components we believe should go into them - read it here .

Featured image background by Megan Lee on Unsplash

Paved Paths. Golden Paths. Paved Roads. You’d be forgiven for thinking the industry already had these nailed down, a solved problem that has been spoken about for years in various guises. With the current spotlight on platform engineering and developer experience, their name is being comfortably thrown around more and more regularly. Unfortunately, descriptions of them are usually from 30,000ft. and if you try to delve deeper, looking for details and commonly experienced challenges, you’ll likely find very little. Maybe a few gems hidden in the clouds…

With this series of blog posts, we would like to change that.

Chris and I have worked together previously and are both passionate about engineering enablement. Recently we ran into each other at a conference and over a great lunch and a lot of conversation, we spoke about quantifying what paved paths are exactly, how we can make them better and what intentional steps can be taken to allow them to be successful. We realised, that in our professional capacities, this is a problem we’ve been in the detail of, learnt lessons and carried on without reflecting and writing them down. Through this series, we will hopefully demystify these incredibly useful concepts and add our insights with some real-world experience.

Our motivators for writing this series:

• Paved Paths are usually bundled in with a centralised platform engineering model. We’re both passionate about them being equally valuable with a decentralised approach.
• Paved Paths are hugely important to the industry with varying interpretations of what they are. They’re not all sunshine and rainbows, but we’d love to see more people apply them successfully.
• We are continuously asking our engineers to take on more and more responsibilities within the DevOps lifecycle as we shift left. This can ease their pain.
• Most examples are embedded in the Kubernetes ecosystem (where there is a lot of phenomenal tooling). We’d like to spotlight that Paved Paths are entirely possible outside of k8s, whether using PaaS or on-prem servers!

These are some topics we would like to explore in this series:

• The elusive one-pager that every CxO needs to kickstart your journey
• The problems we are solving for and why it is a good idea to invest time in paved paths
• What makes up a paved path?
• To centralise or decentralise?
• What is important to measure?
• Practical ways to get started

We hope you will enjoy these as much as we’ve enjoyed exploring them. Start a conversation with us on Twitter - I’m @rickroche_ and Chris is @ctaceygreen .

In part 2 of our Paved Path Series we start off with a one-pager to help you explain this to your CxO - read it here !

Images generated by Craiyon , updated using excalidraw .

We often build and deploy web applications specifically for users internal to our organisation. Azure Static Web Apps is proving to be an excellent replacement for Azure App Service in these scenarios.

At a high-level the service provides you with a great set of features (outlined in the Azure release notes )

• Globally distributed content for production apps
• Tailored CI/CD workflows from code to cloud
• Auto-provisioned preview environments
• Custom domain configuration and free SSL certificates
• Built-in access to a variety of authentication providers
• Route-based authorization
• Custom routing
• Integration with serverless APIs powered by Azure Functions
• A custom Visual Studio Code developer extension
• A feature-rich CLI for local development

The desired experience?

The experience I wanted to achieve was that if one of our internal users browsed to any of our internal apps, they would be able to use SSO across them provided they were a member of the AAD group needed to access the app (or just a member of our tenant for organisation-wide apps) - no login button, just a seamless logged-in user experience.

It turns out this was super easy to get right! Follow below!

Authentication options

Azure Static Web Apps makes authentication easy to enable across the three pre-configured identity providers

• Azure Active Directory (AAD)
• Github or
• Twitter

These options allow users to login using a login button linking to the desired provider.

Initially I tried to use the pre-configured AAD provider, and when trying to log in using my company account I was presented with this approval dialogue

Meaning if I was able to get an admin to grant the permission, all users from our tenant would be able to log in to all Azure Static Web Apps, regardless of who had deployed them making this a non-starter for me.

Fortunately we already deploy our static web apps using the Standard plan for $9/per app/month , giving us the ability to use custom authentication and use SSO with our organisations AAD tenant, internal app registrations and restrict access to groups / users in our tenants directory.

Getting it done!

To achieve the desired experience there are a number of components required

• the static web app (the website)
• an Azure App Registration for your app in your tenant
• an Azure resource group for your project (I’m working on the assumption you already have a subscription, if not read here )
  • an Azure Static Web App for the web app
  • an Azure Key Vault to safely store the secrets for your app

I will be showing you how to create, configure and deploy these using GitHub Actions , Bicep (for creating the Azure resources) and some once off scripts.

Initial setup and deployment

First we are going to scaffold our web app, define our infrastructure and deploy to Azure without any auth in place. Once done, we will move onto the configuration of the app with auth.

Scaffold the web app

For the purposes of this post, I needed a simple static web app and used the SvelteKit skeleton project with the static adapter .

1
2
3

npm init svelte@next webapp
cd webapp
npm install --save-dev @sveltejs/adapter-static@next

You need to update the default svelte.config.js to use the static adapter , see an example here .

Define the Azure Infrastructure

Resource Group

First, create an Azure resource group. For this tutorial I’m creating it manually from my terminal and Azure CLI

1

az group create -g rg-swa-sso -l northeurope

Resources

We will be deploying a Static Web App as well as a Key Vault using Bicep. I’ve split out the Bicep files to make them easier to work with as follows (filenames link to the code on GitHub)

• main.bicep - the main template that is deployed which makes use of the other templates
• get-kv-secrets-refs.bicep - a helper to build up the Key Vault secret references
• key-vault.bicep - defines the Key Vault resources and the role assignments needed
• static-sites.bicep - defines the Static Web App resources needed

In the main.bicep there are the following highlights to make note of

• Configuring the static web app
  • to have app settings that are Key Vault references (AAD_CLIENT_ID, AAD_CLIENT_SECRET)
  • to use the Standard sku

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

module swa 'static-sites.bicep' = {
  name: 'deploy-swa-${appName}'
  params: {
    appSettings: {
      AAD_CLIENT_ID: refs.outputs.aadClientIdRef
      AAD_CLIENT_SECRET: refs.outputs.aadClientSecretRef
    }
    ...
    sku: {
      name: 'Standard'
      tier: 'Standard'
    }
    ...
  }
}

• And configuring the key vault to allow the static web app permissions to read from it

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

// https://docs.microsoft.com/en-us/azure/key-vault/general/rbac-guide?tabs=azure-cli#azure-built-in-roles-for-key-vault-data-plane-operations
var keyVaultSecretsUserRole = '4633458b-17de-408a-b874-0445c86b69e6'

module kv 'key-vault.bicep' = {
  name: 'deploy-kv-${appName}'
  params: {
    ...
    roleAssignments: [
      {
        roleDefinitionId: keyVaultSecretsUserRole
        principalType: 'ServicePrincipal'
        principalId: swa.outputs.siteSystemAssignedIdentityId
      }
    ]
    ...
  }
}

Deploy using GitHub

Deployment credentials

For a GitHub Action to be able to deploy to your resource group you need to have some form of deployment credentials. You can read about the options available here . I’ll be using the Service Principal approach with Azure CLI .

1
2
3
4

az ad sp create-for-rbac --name "sp-swa-sso" \
    --role Owner \
    --scopes /subscriptions/{subscription-id}/resourceGroups/{resource-group-name} \
    --sdk-auth

You will get a JSON output from this that you can then save as a GitHub secret with the name AZURE_CREDENTIALS.

Workflow token

Azure Static Web Apps needs access to your workflow when deploying. For this we’ll set up a WORKFLOW_TOKEN secret using a GitHub personal access token with the workflow scope. Follow the instructions here to create the token.

Github Action Workflow

With the above secrets in place, we can now create a workflow (mines in .github/workflows/deploy.yaml ) which

• specifies some environment variables for names, tags and locations
• checks out the repo
• logs into Azure using ${{ secrets.AZURE_CREDENTIALS }}
• deploys to the resource group using the azure/CLI@v1 action
• gets the API from the deployed static web app (so that we can deploy code to it)
• deploys the webapp using the Azure/static-web-apps-deploy@v1 action

See the full workflow below.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

name: Deploy Infra and App

on:
  push:
    branches:
      - main
  pull_request:
    types: [opened, synchronize, reopened, closed]
    branches:
      - main

env:
  RESOURCE_GROUP: "rg-swa-sso"
  RESOURCE_TAGS: '{"owner":"rick.roche", "app":"azure-swa-sso", "repo":"https://github.com/rick-roche/azure-static-web-apps-sso" }'
  APP_NAME: "swa-sso"
  LOCATION: "westeurope"

jobs:
  deploy-infra:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - uses: actions/setup-node@v2
        with:
          node-version: "16"

      - name: Azure Login
        uses: azure/login@v1
        with:
          creds: ${{ secrets.AZURE_CREDENTIALS }}

      - name: Deploy Infra
        id: deploy_infra
        if: github.event_name != 'pull_request'
        uses: azure/CLI@v1
        with:
          inlineScript: |
            az deployment group create \
              --resource-group ${{ env.RESOURCE_GROUP }} \
              --template-file ./infra/main.bicep \
              --parameters \
                  appName='${{ env.APP_NAME }}' \
                  location='${{ env.LOCATION }}' \
                  repositoryUrl='https://github.com/rick-roche/azure-static-web-apps-sso' \
                  repositoryToken='${{ secrets.WORKFLOW_TOKEN }}' \
                  tags='${{ env.RESOURCE_TAGS }}'            

      - name: Get Static Web App API Key
        id: static_web_app_apikey
        if: github.event_name != 'pull_request'
        uses: azure/CLI@v1
        with:
          inlineScript: |
            APIKEY=$(az staticwebapp secrets list --name 'stapp-${{ env.APP_NAME }}' | jq -r '.properties.apiKey')
            echo "::set-output name=APIKEY::$APIKEY"            

      - name: Deploy WebApp to Static Web App
        id: static_web_app_deploy
        if: github.event_name != 'pull_request'
        uses: Azure/static-web-apps-deploy@v1
        with:
          azure_static_web_apps_api_token: ${{ steps.static_web_app_apikey.outputs.APIKEY }}
          repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for GitHub integrations (i.e. PR comments)
          action: "upload"
          # Build configuration for Azure Static Web Apps: https://aka.ms/swaworkflowconfig
          app_location: "webapp"
          api_location: ""
          output_location: "build" # relative to app_location

Finalising auth

At this stage you should have a GitHub workflow successfully deploying a Static Web App and a Key Vault to you resource group. You should also be able to browse to your web app using the generated URL (navigate into the Static Web App from the portal, and you will see your URL on the overview tab. e.g. https://gray-wave-03fb32a03.1.azurestaticapps.net ).

To enable auth our next steps will be to

• create an AAD app registration
• add it’s client ID and secret as secrets in your key vault
• configure the static web app to auto log you in if you aren’t already or your token has expired

AAD App Registration

An AAD app registration allows us to bind our application to a desired set of authentication flows and restrictions. Think of it as giving your application an identity inside AAD. Mark Foppen wrote a lovely article that may help demystify this a bit.

In this tutorial I’ll just be using Azure CLI to create one:

1
2
3
4
5

az ad app create --display-name aadapp-swa-sso \
    --available-to-other-tenants false \
    --identifier-uris api://stapp-swa-sso \
    --reply-urls 'https://gray-wave-03fb32a03.1.azurestaticapps.net/.auth/login/aad/callback' \
    --native-app false

Once created, we need to create an application secret for the application. For this tutorial you can do this via the portal following the instructions here .

Copy the value of the secret as well as the Application (client) ID of the AAD app (found under the Overview section of the app).

Setup Key Vault Secrets

Once again, I’ve used the portal for this tutorial. You can follow the guide here setting two secrets in your vault

• aadClientId - this should be set to the Application (client) ID of your app registration
• aadClientSecret - this should be set to the value of the application secret created above

Configuring the Static Web App

Configuration for Azure Static Web Apps is defined in the staticwebapp.config.json file, which controls, among other things, authentication and authorisation.

I’ve put mine in the root of the webapp folder and set it to do the following to enable the auto-login SSO magic

• enable custom auth with Azure Active Directory Version 2 using the app settings references from earlier (AAD_CLIENT_ID, AAD_CLIENT_SECRET)

  1 2 3 4 5 6 7 8 9 10 11

  "auth": { "identityProviders": { "azureActiveDirectory": { "registration": { "openIdIssuer": "https://login.microsoftonline.com/4af290c8-08df-440d-93db-cbac02bd9b19/v2.0", "clientIdSettingName": "AAD_CLIENT_ID", "clientSecretSettingName": "AAD_CLIENT_SECRET" } } } },

• a navigationFallback route to index.html

  1 2 3

  "navigationFallback": { "rewrite": "index.html" },

• specific rules for the apps routes

  • creates a /login route redirecting to AAD allowing anonymous access

    1 2 3 4 5

    { "route": "/login", "rewrite": "/.auth/login/aad", "allowedRoles": ["anonymous", "authenticated"] },

  • blocks all providers except AAD

    1 2 3 4 5 6 7 8

    { "route": "/.auth/login/github", "statusCode": 404 }, { "route": "/.auth/login/twitter", "statusCode": 404 },

  • creates a /logout route redirecting to AAD allowing anonymous access

    1 2 3 4 5

    { "route": "/logout", "redirect": "/.auth/logout", "allowedRoles": ["anonymous", "authenticated"] },

  • enforces auth for all other routes (/*)

    1 2 3 4

    { "route": "/*", "allowedRoles": ["authenticated"] }

• sets up a response override that if an unauthenticated user hits a page, they should be redirected to the /login route

  1 2 3 4 5 6

  "responseOverrides": { "401": { "redirect": "/login", "statusCode": 302 } }

The combination of all of the above means that anyone accessing the site that isn’t logged in will be routed to the /login route which will ensure that the AAD auth flow is completed!

Finishing up!

Commit all your outstanding changes, push to GitHub and watch your action deploy… once done, access your web app in the browser, and it should redirect you to login

Initially you will need to provide admin consent for your AD app registration to read the logged-in user details

If all has gone well you should see the default page after login completes

And that’s it! Hope you enjoyed this tutorial and that it helps you setup SSO for your Static Web Apps!

Featured image background by Hello I’m Nik on Unsplash

In my previous post I touched on the things I learnt while migrating ARM templates to Bicep . Bicep also introduces the concept of modules to enable template reuse. I took some time to refactor a composite application that had already been converted from using ARM to Bicep templates, to use Bicep modules. This post will cover the things that I learnt by working through that process.

Why modules?

From the Bicep documentation :

Bicep enables you to break down a complex solution into modules. A Bicep module is a set of one or more resources to be deployed together. Modules abstract away complex details of the raw resource declaration, which can increase readability. You can reuse these modules, and share them with other people. Bicep modules are transpiled into a single ARM template with nested templates for deployment.

One of the traps we fell into with ARM templates was duplicating templates to make composing and deploying the resources we need easier. Any opportunity to make the deployment of infrastructure more readable, more reusable, and more composable are excellent reasons for me to give it a go. My goal when doing this refactor was to

• get rid of any duplication by creating fine-grained modules, designed to be reused
• ensure that all main templates are super easy to use by composing modules together in a way that makes sense for the application
• enable reusability of the fine-grained modules in other projects going forward.

In short: let’s make the infra readable, composable and reusable.

Notable learnings

Reusable Modules

I went with the approach of trying to make a reusable module for each Azure resource type and putting the modules into a folder called modules and making sub folders for template groupings. For example,

1
2
3
4
5
6
7
8

modules/
    appInsights.bicep
    appServicePlan.bicep
    functionApp.bicep
    logAnalytics.bicep
    storageAccount/
        storageAccount.bicep
        tables.bicep

Personally I prefer to have one module to create a storage account and another to add tables to that storage account etc. This level of granularity felt like a good place to start.

Referencing existing resources inside a module

By making fine-grained modules, there were a number of use cases where I would need to reference an existing resource. For example, creating tables in a storage account requires an existing storage account. Referencing an existing resource is really easy – you only need to know its name and can reference it as follows,

1
2
3
4
5
6
7
8
9

// Lookup an existing resource
resource storageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: storageAccountName
  // Optional if the existing resource is in a different resource group or subscription
  scope: resourceGroup(subscriptionId, resourceGroupName)
}

// You can now use the resource as if you had created it. e.g.
outputs storageAccountResourceId = storageAccount.id

Loops

I really like the loops feature. This allows you to iterate over an array setting multiple properties or creating multiple resources etc. This came in super handy for a storage account tables module that can create multiple tables in one go. E.g.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

param storageAccountName string
param tables array = [
  {
    container: 'default'
    name: 'replace'
  }
]

resource storageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: storageAccountName
}

resource storageAccountTables 'Microsoft.Storage/storageAccounts/tableServices/tables@2021-02-01' = [for table in tables: {
  name: '${storageAccount.name}/${table.container}/${table.name}'
  dependsOn: [
    storageAccount
  ]
}]

output storageAccountTableNames array = [for (table, i) in tables: {
  name: storageAccountTables[i].name
}]

Note the use of the different styles of the loops in the storageAccountTables resource and the outputs.

Functions and Expressions

There are loads of functions and expressions that you can use in your Bicep files and I won’t go into all of them. There were a few that I used regularly, and it’s hopefully useful that I call them out.

Union

union(arg1, arg2, arg3, ...) Returns a single array or object with all elements from the parameters. Duplicate values or keys are only included once.

I used union everywhere I wanted to have fixed (opinionated) defaults in the module, but allow additional parameters to be merged in. For example, with function apps I defined base settings I want all function apps to have and still allow for additional app settings to be passed in and merged with the base.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

@description('Additional app settings for your function app')
param additionalAppSettings object = {}

var appSettingsBase = {
  FUNCTIONS_EXTENSION_VERSION: '~3'
  FUNCTIONS_WORKER_RUNTIME: 'dotnet'
  WEBSITE_RUN_FROM_PACKAGE: '1'
  AzureWebJobsStorage: 'DefaultEndpointsProtocol=https;AccountName=${storageAccount.name};EndpointSuffix=${environment().suffixes.storage};AccountKey=${listKeys(storageAccount.id, storageAccount.apiVersion).keys[0].value}'
}

var appSettings = union(appSettingsBase, additionalAppSettings)

We can also get rid of all the hardcoded schema API versions when looking up keys against a resource by using <resource>.apiVersion as well as hardcoded suffixes by using the environment function , in this case using the storage suffix: environment().suffixes.storage.

Another super useful call out would be to the listKeys function. It allows you to get connection strings or keys from your resources and is super handy (see the AzureWebJobsStorage example above). John Reilly went into detail on this over here , please have a read!

Ternary

I found that with Bicep I use the ternary operator a lot in my main templates when composing the modules. For example, only enabling a secondary region when the environment is nonprod or prod but not in dev can easily be described as,

1

var secondaryRegionEnabled = (contains(env, 'prod')) ? true : false

Much cleaner and readable without all the JSON around it.

Composing modules together

Every Bicep file can be consumed as a module which is an awesome feature. I chose to break my files up as follows:

1
2
3
4
5
6
7
8

infra/
  myApp/
    main.bicep - loads up the app.bicep and monitoring.bicep modules
    app.bicep - uses the fine grained modules - app service plans, functions, azure storage etc
    monitoring.bicep - uses the fine grained modules - app insights, log analytics etc
shared-infra/
  modules/
    <all the fine-grained modules>

In my main.bicep I reference the two local Bicep files as modules

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

module monitoring 'monitoring.bicep' = {
  name: '${appName}-monitoring'
  params: {
    tags: tags
    location: location
    appInsightsName: appInsightsName
    logAnalyticsWsName: logAnalyticsWsName
  }
}

module app 'app.bicep' = {
  name: '${appName}-app'
  params: {
    tags: tags
    location: location
    appName: appName
    storageAccountName: storageAccountName
    appInsightsName: monitoring.outputs.appInsightsName
    logAnalyticsWsName: monitoring.outputs.logAnalyticsWsName
    monitoringResourceGroup: monitoring.outputs.ResourceGroupName
  }
}

Note that the app module relies on the outputs of the monitoring module – this helps Bicep figure out the dependencies so that you don’t need to define dependsOn any more.

Then in monitoring.bicep I can reference fine-grained reusable modules that I’d like to share with other projects in much the same way

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

module log '../../shared-infra/modules/logAnalytics.bicep' = {
  name: '${appName}-log'
  params: {
    tags: tags
    location: location
    logAnalyticsWsName: logAnalyticsWsName
  }
}

module appi '../../shared-infra/modules/appInsights.bicep' = {
  name: '${appName}-appi'
  params: {
    tags: tags
    location: location
    appInsightsName: appInsightsName
    logAnalyticsWsName: log.outputs.logAnalyticsWsName
  }
}

Another thing to note is the name of your module is what is shown in the Deployments tab of the Azure Portal, so make these make sense to you for easy debugging if deployments are breaking.

Sharing modules

At this stage I’ve got different two styles of modules – app specific modules breaking up my main.bicep, making it easier to read and maintain and fine-grained templates in a modules folder that I can use across all the apps in my infra folder. Readable – check! Composable – check! Reusable – only inside this repo, so half a check!

The current version of Bicep (v0.4.63), does not have a native mechanism to externally share modules across projects. The good news is that this is being looked at and will hopefully be in the v0.5 release. The issues to watch are:

• [Story] Bicep Registry
• Ability to reference “external” modules
• Modules need versions

In the interim, I am using Git submodules to solve this, although this will not work with all CI/CD tooling when using private repos. To enable this, I moved the modules in shared-infra into its own repo and added it back to my project.

1
2

git submodule add <path-to-repo> shared-infra/
git submodule update --init

Testing locally

To quickly validate the individual modules and main templates on my local machine, I wrote a simple bash script to either

• build the Bicep file, outputting to the terminal rather than writing to file or
• validate the Bicep template against a resource group

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/bash
RG=replace-with-your-rg
SUB=replace-with-your-sub

op=$1 # lint, validate, create
main_shared=$2
if [ "$main_shared" == 'main' ]; then path='./infra' && filename='main.bicep'; fi
if [ "$main_shared" == 'shared' ]; then path='./shared-infra' && filename='*.bicep'; fi

lint () {
    az bicep build --file "$f" --stdout
}

validate () {
    az deployment group validate --resource-group "$RG" --subscription "$SUB" -f "$1" -p env=dev
}

for f in $(find "$path" -name "$filename") ; do
    echo "$f"
    $op "$f"
done

The script allows one to test either main or shared templates, linting them or validating them:

1
2
3

./infra.sh lint main
./infra.sh lint shared
./infra.sh validate main

Finishing up

After the refactor, my project is in a far cleaner state and the infrastructure is much easier to follow. A second plus is that we now have a separate repo of our shared modules that can become a shared asset across our various teams (using Git submodules for now and the Bicep registry in the future).

Featured image background by Michael Dziedzic on Unsplash

You may have heard of Bicep , and you may be wondering how much effort it is going to take to move all your ARM templates to this new way of deploying Azure resources.

I gave migrating from ARM to Bicep a go. This post will cover going from JSON ARM templates to shiny new Bicep templates that have no errors and don’t contain any warnings or linting issues!

Why should you migrate?

If you have ever deployed infra to Azure you have most likely used ARM templates before. They do the job, the docs aren’t bad and there are built in tasks for ADO which make deploying them super straightforward. However, working with JSON is always going to be verbose, ARM templates have a lot of boilerplate required, making reusable templates is clunky and deployments can get tricky.

Aiming to address these (and other) issues, Azure is working on a project called Bicep. From the projects overview :

Bicep is a domain-specific language (DSL) that uses declarative syntax to deploy Azure resources. It provides concise syntax, reliable type safety, and support for code reuse. We believe Bicep offers the best authoring experience for your Azure infrastructure as code solutions.

Essentially it is a DSL built on-top of ARM that makes the development of templates simpler and promotes reuse through modules. Judging by the release rate on their GitHub the team is working hard to make it awesome very quickly and according to them, as of v0.3, Bicep is now supported by Microsoft Support Plans and Bicep has 100% parity with what can be accomplished with ARM Templates. So Bicep is definitely prod ready (at time of writing v0.4.63 was the latest release)!

Getting started

Ensure you have the latest version of Bicep installed (excellent installation guide here ). I went with the az cli install option and all my examples will be using that variant. I also use Visual Studio Code and installed the Bicep VS Code Extension for enhanced editing.

Decompiling ARM to Bicep

First things first, it is possible to decompile an ARM template into Bicep by running the below command (all our ARM templates are in separate folders with an azuredeploy.json file for the template)

1

az bicep decompile --file azuredeploy.json

This creates a azuredeploy.bicep file in the same directory. Depending on your template, you will most likely get a stream of yellow warnings in the console, or potentially some red errors: don’t panic! Even a single error will result in all the console output being red (at least on macOS) and at the very least you will get this warning message:

WARNING: Decompilation is a best-effort process, as there is no guaranteed mapping from ARM JSON to Bicep. You may need to fix warnings and errors in the generated bicep file(s), or decompilation may fail entirely if an accurate conversion is not possible. If you would like to report any issues or inaccurate conversions, please see https://github.com/Azure/bicep/issues .

It is important to note that the migration from ARM to Bicep will highlight issues in your ARM templates. ARM was more forgiving in certain aspects, after the migration, your templates will be in a better state.

Errors come in the form of Error BCPXXX: Description and the descriptions generally let you get to the root of the problem quickly. E.g. Error BCP037: The property “location” is not allowed on objects of type “Microsoft.EventHub/namespaces/eventhubs”. Permissible properties include “dependsOn”.

If the decompilation gives you any errors, my preferred approach is to fix the ARM template and then run the decompilation again until you get a “clean” decompilation (warnings are fine, just ensure you decompile with no errors).

Warnings come in two flavours, ones that have a code (Warning BCPXXX) and ones that don’t have a code but have a link at the end. As of v0.4 there is a linter which helps you get your templates into great shape – these are the warnings with the links at the end. Both kinds of warnings are generally quick to fix and are sensible updates to start getting the benefits from Bicep.

Common errors and how to fix them

User defined functions are not supported (BCP007, BCP057)

If you have created user defined functions in ARM, these will not be decompiled (follow the issue ) and you will get two cryptic errors

Error BCP007: This declaration type is not recognized. Specify a parameter, variable, resource, or output declaration. Error BCP057: The name “functionName” does not exist in the current context.

To fix, move the function logic into your template (this may require duplication, but can be neatened up in the Bicep template later)

Nested dependsOn (BCP034)

Sometimes the use of nested dependsOn properties in your ARM templates gets weird with Bicep (dependsOn can often be removed entirely in Bicep). The ARM version would have validated and deployed perfectly fine however you will get the following error when decompiling to Bicep.

Error BCP034: The enclosing array expected an item of type “module[] | (resource | module) | resource[]”, but the provided item was of type “string”.

Fortunately Bicep handles dependencies in a much smarter way than ARM meaning you can safely remove your nested dependencies and run again.

For Bicep, you can set an explicit dependency but this approach isn’t recommended. Instead, rely on implicit dependencies. An implicit dependency is created when one resource declaration references the identifier of another resource.

To fix, delete your nested dependencies and validate (consider removing your dependsOn references entirely)

Strict schema validation (BCP037, BCP073)

Bicep validates the schema of each resource much more diligently than ARM. As a result you will probably find a bunch of places where you have added properties like location or tags to a resource that doesn’t support them and ARM didn’t mind this. These will be expressed as Error BCP037.

E.g. I had location on Microsoft.EventHub/namespaces/eventhubs

Error BCP037: The property “location” is not allowed on objects of type “Microsoft.EventHub/namespaces/eventhubs”. Permissible properties include “dependsOn”.

The one error I did run into was for Azure Logic Apps where the schema for Microsoft.Logic/workflows is missing the identity property needed for using Managed Identity with your Logic App:

Error BCP037: The property “identity” is not allowed on objects of type “Microsoft.Logic/workflows”. Permissible properties include “dependsOn”.

Another common error I encountered was having read-only parameters in my templates (these tend to come along if you have exported templates from the Azure Portal). E.g.

Error BCP073: The property “kind” is read-only. Expressions cannot be assigned to read-only properties.

To fix both types of errors, remove the properties that the error messages highlight and run the decompilation again.

Reserved words as parameters (BCP079)

In ARM templates you can have the same name for a parameter, variable, function etc as these are all addressed directly using parameters('name') or variables('name') etc. In Bicep the syntax is simplified and as such you will get errors if you have used a reserved word as a parameter. We had a couple of templates taking in a parameter called description resulting in a cryptic error message:

Error BCP079: This expression is referencing its own declaration, which is not allowed.

To fix, rename the parameter in your ARM template, update its usages and run the decompilation again.

Common warnings and how to fix them

Hopefully by now you are out of the error zone, for warnings you can now start editing the Bicep file itself. The Bicep VS Code Extension gives you great intellisense and highlights issues in the Bicep file.

To test an update on your Bicep file, run

1

az bicep build --file azuredeploy.bicep

Schema warnings, enums and types (BCP035, BCP036, BCP037, BCP073, BCP081, BCP174)

Data types in Bicep are stricter than in ARM. If you have used True or False for booleans these need to be updated to be true and false. When using enums, they need to match the enums in the schema exactly (case-sensitive). E.g. if you used ascending and the template schema defines Ascending this adjustment needs to be made. Examples of these warnings are shown below.

Warning BCP036: The property “kafkaEnabled” expected a value of type “bool | null” but the provided value is of type “‘False’ | ‘True’”. Warning BCP036: The property “order” expected a value of type “‘Ascending’ | ‘Descending’ | null” but the provided value is of type “‘ascending’”.

Similar to the errors thrown by BCP037 and BCP073 you will get warnings on schema mismatches using codes BCP035, BCP037 and BCP073. These highlight schema mismatches, missing properties and read-only properties used. Examples below.

Warning BCP035: The specified “object” declaration is missing the following required properties: “options”. Warning BCP037: The property “apiVersion” is not allowed on objects of type “Output”. No other properties are allowed. Warning BCP073: The property “status” is read-only. Expressions cannot be assigned to read-only properties.

I did find an instance where the Bicep template matches the schema perfectly however warnings are still thrown. I think this is due to this issue and fortunately doesn’t break anything.

To fix, update the offending property to match the schema or data type

String interpolation

concat gets to disappear in Bicep thanks to the lovely string interpolation option. I experienced two variants:

Warning prefer-interpolation: Use string interpolation instead of the concat function. [https://aka.ms/bicep/linter/prefer-interpolation]

To fix, search for all usages of concat and replace with the new syntax: 'string-${var}'.

Warning simplify-interpolation: Remove unnecessary string interpolation. [https://aka.ms/bicep/linter/simplify-interpolation]

To fix, remove the ${} and reference the variable directly. E.g. '${variable}' becomes variable.

Environment URLs

We had a lot of hardcoded URL’s in our templates for storage suffixes, front door etc. There is a much better way to do this by using the environment() function. Caveat here, if you had environment as a parameter to your ARM template, update this to be something else like env for example otherwise you won’t be able to use the function.

Warning no-hardcoded-env-urls: Environment URLs should not be hardcoded. Use the environment() function to ensure compatibility across clouds. Found this disallowed host: “core.windows.net” [https://aka.ms/bicep/linter/no-hardcoded-env-urls]

To fix, have a look at all the URLs that the environment function provides and replace your hard-coded version with environment().<property>.

E.g. for Azure Storage where core.windows.net had been hard coded, replacing with environment().suffixes.storage gives the desired result.

Scopes (BCP174)

Bicep introduces the concept of target scopes which dictates the scope that resources within that deployment are created in. This gives you a new way to define resources where you previously would have used the /providers/ syntax (role assignments, diagnostic settings etc). The warning you get looks as follows.

Warning BCP174: Type validation is not available for resource types declared containing a “/providers/” segment. Please instead use the “scope” property. [https://aka.ms/BicepScopes]

These are the most time-consuming to fix, essentially you need to

• use the schema reference of the actual resource (so for diagnostic settings us microsoft.insights/diagnosticSettings@2017-05-01-preview) instead of the parent resource /providers/
• add a scope referencing the parent resource
• rename so as not to reference the parent resource
• remove unnecessary properties and dependsOn

Before

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

resource functionAppLogAnalytics 'Microsoft.Web/sites/providers/diagnosticSettings@2017-05-01-preview' = {
  name: '${functionAppName}/Microsoft.Insights/LogAnalytics'
  tags: tagsVar
  properties: {
    name: 'LogAnalytics'
    workspaceId: resourceId(logAnalyticsResourceGroup, 'Microsoft.OperationalInsights/workspaces', logAnalyticsWsName)
    logs: [
      {
        category: 'FunctionAppLogs'
        enabled: true
      }
    ]
  }
  dependsOn: [
    functionApp
  ]
}

After

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

resource functionAppLogAnalytics 'microsoft.insights/diagnosticSettings@2017-05-01-preview' = {
  name: LogAnalytics
  scope: functionApp
  properties: {
    workspaceId: logAnalyticsResourceId
    logs: [
      {
        category: 'FunctionAppLogs'
        enabled: true
      }
    ]
  }
}

Should you migrate?

A resounding yes from me! The process above goes quite quickly, and I was on shiny new Bicep templates in less than an hour. Bicep is a much friendlier syntax to work with and the IDE support is great. What I also enjoyed is cleaning up all the errors that were present in my ARM templates that would have remained if not for the migration.

There is a neat comparison of ARM syntax vs Bicep syntax here which highlights a lot of the constructs that become simpler as well: https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/compare-template-syntax

Enjoy the migration! I will be playing around with modules and reuse next and will share what I find.

Featured image background by Nick Fewings on Unsplash .

Keeping your dependencies up to date in a project is a really easy way to try and keep the software secure. New releases of a dependency often include

• Patches for security vulnerabilities!
• Performance improvements!
• Awesome new features!
• Bug fixes!

It can also be quite a boring activity and can be time-consuming for a team maintaining the project to run updates regularly into production. Fortunately tools like Dependabot exist!

Dependabot?

Dependabot creates pull requests on your repos with the dependencies you should update. You can read about how it works ; but in a nutshell

• it checks for updates of your dependencies
• it opens up a pull request on your repo
• you review the PR and merge

This is an awesome tool to save you time and I find it makes keeping your dependencies up to date really easy.

Integration with Azure Pipelines

Dependabot is baked into the GitHub ecosystem and really easy to use there. Recently I needed to solve this problem on a project in Azure DevOps using Azure Pipelines and thought I would share my solution.

If you search the Azure DevOps Extension Marketplace for Dependabot you will find this extension made by Tingle Software . It is always great to find extensions to speed up integration and I gave this one a whirl.

The extension is feature rich and works really well with little configuration required. Simply add the below to an Azure Pipeline and run it, and you’ll end up with a host of PR’s!

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

stages:
  - stage: CheckDependencies
    displayName: "Check Dependencies"
    jobs:
      - job: Dependabot
        displayName: "Run Dependabot"
        pool:
          vmImage: "ubuntu-latest"
        steps:
          - task: dependabot@1
            displayName: "Run Dependabot"
            inputs:
              packageManager: "nuget" # Examples: nuget, maven, gradle, npm, etc. Add multiple tasks if multiple package managers are used in your solution
              targetBranch: "main"
              openPullRequestsLimit: 10 # Limits the number of PR's you get
              setAutoComplete: true # Saves us one click, once our PR policies pass, the update will merge

Have a look at all the Task Parameters that the extension supports to fine tune your implementation. The above checks for nuget dependencies on the main branch, limits the number of PR’s raise to be 10 and sets the PR to autocomplete (letting our branch policies and PR validation pipelines perform all their checks).

Work Item Linking

In the project I’m working in we have a policy set on all PR’s to ensure they are linked to a work item. When I ran the above pipeline, I ended up with 10 PR’s, none of which were linked to a work item so I couldn’t quickly review and merge each one. The extension handles this by allowing you to pass in a workItemId parameter.

The Microsoft team have an extension for creating work items which is really configurable. For my teams workflow, we wanted to have a User Story on our board with all the PR’s linked to it. We also didn’t want the pipeline creating duplicate User Stories every time it runs if we already had one open that we are working on. After a bit of trial and error, adding the below to the pipeline gave us the desired result.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

- task: CreateWorkItem@1
  displayName: "Create User Story for Dependabot"
  inputs:
    workItemType: "User Story" # We wanted a User Story created, but all work item types are available
    title: "Update Dependencies" # The title of the work item
    # Adding some tags to the work item
    fieldMappings: |
      Tags=dependabot; dependencies      
    areaPath: 'your\area'
    iterationPath: 'your\iteration'
    # Adding duplicate detection, using the area path, iteration path and title to match
    preventDuplicates: true
    keyFields: |
      System.AreaPath
      System.IterationPath
      System.Title      
    # Create output variables for the next task to be able use the work item ID
    createOutputs: true
    outputVariables: |
      workItemId=ID      

- task: dependabot@1
  displayName: "Run Dependabot"
  inputs:
    packageManager: "nuget"
    targetBranch: "main"
    openPullRequestsLimit: 10
    workItemId: $(workItemId) # This uses the output from the above as the work item to link the PR's to
    setAutoComplete: true

Docker Caching

The Dependabot extension depends on a Docker image hosted on Docker Hub . The image is around 4.4GB in size and can take some time to download in the pipeline.

In the docs it mentions

Since this task makes use of a docker image, it may take time to install the docker image. The user can choose to speed this up by using Caching for Docker in Azure Pipelines.

The caching tasks can look a bit confusing, breaking it down you need 3 parts:

1
2
3
4
5
6

- task: Cache@2
  inputs:
    key: docker | "${{ variables.imageToCache }}" # image to look for in the cache, e.g. tingle/dependabot-azure-devops:0.4
    path: $(Pipeline.Workspace)/docker # path of the cache
    cacheHitVar: DOCKER_CACHE_HIT # variable to set to true if a cache hit is found
  displayName: Cache Docker images

This checks if there is a cached version of the image to be used. If yes, DOCKER_CACHE_HIT is set to true, otherwise false.

1
2
3
4

- script: |
    docker load -i $(Pipeline.Workspace)/docker/cache.tar    
  displayName: Restore Docker image
  condition: and(not(canceled()), eq(variables.DOCKER_CACHE_HIT, 'true'))

If we have a cached version of the image, we need to pipeline to download it. This step only runs if DOCKER_CACHE_HIT is set to true and will download the cached archive and load it into the docker context.

1
2
3
4
5
6

- script: |
    mkdir -p $(Pipeline.Workspace)/docker
    docker pull -q ${{ parameters.imageToCache }}
    docker save -o $(Pipeline.Workspace)/docker/cache.tar ${{ parameters.imageToCache }}    
  displayName: Save Docker image
  condition: and(not(canceled()), or(failed(), ne(variables.DOCKER_CACHE_HIT, 'true')))

If we do not have a cached version of the image we need to pull it down from Docker Hub and save it as an archive to be cached.

Finishing up

We have a pipeline that runs on a schedule, running all the above steps together. A complete example can be found on GitHub here .

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66

schedules:
  - cron: "0 4 * * 1"
    displayName: "Weekly Run"
    always: true
    branches:
      include:
        - main

trigger: none

variables:
  - name: imageToCache
    value: tingle/dependabot-azure-devops:0.4

stages:
  - stage: CheckDependencies
    displayName: "Check Dependencies"
    jobs:
      - job: Dependabot
        displayName: "Run Dependabot"
        pool:
          vmImage: "ubuntu-latest"
        steps:
          - task: CreateWorkItem@1
            displayName: "Create User Story for Dependabot"
            inputs:
              workItemType: "User Story"
              title: "Update Dependencies"
              fieldMappings: |
                Tags=dependabot; dependencies                
              areaPath: 'your\area'
              iterationPath: 'your\iteration'
              preventDuplicates: true
              keyFields: |
                System.AreaPath
                System.IterationPath
                System.Title                
              createOutputs: true
              outputVariables: |
                workItemId=ID                

          - task: Cache@2
            inputs:
              key: docker | "${{ variables.imageToCache }}"
              path: $(Pipeline.Workspace)/docker
              cacheHitVar: DOCKER_CACHE_HIT
            displayName: Cache Docker images
          - script: |
              docker load -i $(Pipeline.Workspace)/docker/cache.tar              
            displayName: Restore Docker image
            condition: and(not(canceled()), eq(variables.DOCKER_CACHE_HIT, 'true'))
          - script: |
              mkdir -p $(Pipeline.Workspace)/docker
              docker pull -q ${{ variables.imageToCache }}
              docker save -o $(Pipeline.Workspace)/docker/cache.tar ${{ variables.imageToCache }}              
            displayName: Save Docker image
            condition: and(not(canceled()), or(failed(), ne(variables.DOCKER_CACHE_HIT, 'true')))

          - task: dependabot@1
            displayName: "Run Dependabot"
            inputs:
              packageManager: "nuget"
              targetBranch: "main"
              openPullRequestsLimit: 10
              workItemId: $(workItemId)
              setAutoComplete: true

In summary the pipeline does the following for you

• Runs on a weekly schedule (update the cron to suite your needs)
• Creates a new work item, with the tags we would like (avoiding duplicates)
• Tries to use a cached version of the image for faster builds (creating a cached version if not found)
• Run Dependabot, limiting the number of open PR’s to 10, linking the PR’s to the created work item and completing the PR once all the policies pass

Hopefully this will help you to get your projects running in Azure DevOps nice and up to date!

I wanted to create a super simple website that is easy to maintain and easy to add content to. You are reading this on the finished product, here is what makes it tick.

My searches started looking around for static website generators (I was hoping that I could run this without a database and all that jazz that normally gets dragged along) and stumbled upon Hugo whose GitHub README describes it as

A Fast and Flexible Static Site Generator built with love by bep , spf13 and friends in Go .

This immediately ticked a few boxes for me being a Go fan, being able to keep everything in version control and write my posts in markdown. I dived right in by following the Quick Start and was up and running in a couple of minutes. Very awesome developer experience.

After reading How to start a blog using Hugo I fiddled around with a few of the themes available, initially choosing anatole to get something up and running.

The basics

With my initial investigations running on my local machine, the time for continuous deployment had arrived! First step: Get the code into version control… For me this means creating repo on GitHub and pushing my initial code to main .

For the deployment of the website I decided to use Netlify . It’s awesome and has a great developer experience - essentially add the site, linking your repo on GitHub (or other supported version control), Hugo gets picked up automagically and after hitting Deploy Site! you will be up and running in less than a minute. There is a Step-by-Step Guide to get you on your way if you get stuck.

I gave my site a nice name (rick-roche) to test using rick-roche.netlify.app immediately and pointed my domain to the Netlify name servers (DNS propagation time can take up to 72 hours) for this site to be live.

Finishing up

There were a few features I was looking for such as search, extended markdown for adding copyable code snippets and the ability to add tips and notes like I am used to doing when using Confluence for work. I found the LoveIt theme while browsing through the Hugo Themes which had everything I was looking for.

A few updates later (LoveIt has a lot of configurable options), code pushed to GitHub and an automatic deployment thanks to Netlify and you are looking at the results.

Allowing Hugo and Netlify to do the heavy lifting I have ended up with a website that

• uses an open-source web framework (I like to be able to see the code)
• allows me to easily write content using markdown
• has continuous deployment for rapid publishing
• my entire website can live in version control

References

I googled and read a bunch as I hacked my way through getting this site up. Here are links to the content that stood out!

• Hugo
• Hugo Themes (I used the awesome LoveIt theme
• Start a blog with Hugo
• Hugo best practices
• A Step-by-Step Guide: Deploying on Netlify

Featured image background by NihoNorway graphy on Unsplash .