How to utilize reusable components in Azure DevOps pipelines?

I worked recently in a project where most of Build and Release pipelines were refactored to utilize YAML and Bicep instead of using classic Azure pipelines and ARM templates. Refactoring was needed because some of components used in the Azure pipelines weren't anymore supported by Azure DevOps.

This blog post covers a few techniques what can be used to create reusable components (YAML and Bicep) between multiple Azure DevOps pipelines. Blog post assumes that you have a basic knowledge about YAML and Bicep.

Environment overview

Environment followed typical micro service architecture approach with multiple API endpoints and a few UI applications. Micro services were coherently built and chosen technology stack was identical. Starting point was really good to start thinking reusability of DevOps components.

Reusable YAML-templates 

YAML based pipelines are used to orchestrate Build and Release process in Azure DevOps.

YAML-templates enables that you can separate Build and Release related functionalities to another file which can be reused to prevent "code" duplication. Operations which are consumed multiple times inside main "the orchestrator" pipeline should be templated. 

Locally shared YAML-templates

Locally shared YAML-template means in this context that template is shared only inside specific Azure DevOps repository.

Application deployment job is an one good example of operation which must be repeated for many environments ("deploy-pipeline.yml" in the sample below). Template can be consumed in each environment with different parameters and this effectively prevents duplication.

This approach assumes that YAML-templates are located in the same repository where main pipeline "the orchestrator" exists. This is not optimal way because deploy-pipeline.yml template is wanted to use also from another Azure pipelines.

stages:
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'Test'
    azureSubscription: $(azureSubscriptionTest)
    deploymentResourceGroupName: 'rg-test-application'
    appserviceResourceName: 'app-test-my-application'
    location: $(location)
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'QA'
    azureSubscription: $(azureSubscriptionQa)
    deploymentResourceGroupName: 'rg-qa-application'
    appserviceResourceName: 'app-qa-my-application'
    location: $(location)
- template: pipelines/deploy-pipeline.yml
  parameters:
    stageTechnicalName: 'Prod'
    azureSubscription: $(azureSubscriptionProd)
    deploymentResourceGroupName: 'rg-prod-application'
    appserviceResourceName: 'app-prod-my-application'
    location: $(location)

Globally shared YAML-templates

Globally shared YAML-template enables that template is located to centralized Azure DevOps repository where application specific Azure pipelines can consume templates. Diagram below illustrates that pipelines for Application X and Y are consuming globally shared templates from separate YAML-repository.

Consumption of globally shared YAML templates is easy to configure:

1) Create a new Azure DevOps repository for globally shared YAML-templates. I have used repository name called "Yaml.Templates" in the sample below.

2) Commit YAML templates to this new repository

3) Open application specific main pipeline and add Resources (multi-repository configuration)

4) Add multi-repository name to the template reference ('pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates')

resources:
  repositories:
  - repository: GloballySharedYamlTemplates
    type: git
    name: AdoProject/Yaml.Templates
    ref: main

stages:
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'Test'
    azureSubscription: $(azureSubscriptionTest)
    deploymentResourceGroupName: 'rg-test-application'
    appserviceResourceName: 'app-test-my-application'
    location: $(location)
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'QA'
    azureSubscription: $(azureSubscriptionQa)
    deploymentResourceGroupName: 'rg-qa-application'
    appserviceResourceName: 'app-qa-my-application'
    location: $(location)
- template: 'pipelines/deploy-pipeline.yml@GloballySharedYamlTemplates'
  parameters:
    stageTechnicalName: 'Prod'
    azureSubscription: $(azureSubscriptionProd)
    deploymentResourceGroupName: 'rg-prod-application'
    appserviceResourceName: 'app-prod-my-application'
    location: $(location)

Reusable Bicep-modules

Bicep is used to configure/determine Azure infrastructure which is provisioned during deployment. Bicep-module is a similar concept like YAML-templates which enable encapsulate details to another files.

Bicep enables you to organize deployments into modules. A module is a Bicep file (or an ARM JSON template) that is deployed from another Bicep file. With modules, you improve the readability of your Bicep files by encapsulating complex details of your deployment. You can also easily reuse modules for different deployments. Source

Locally shared Bicep-modules

Locally shared Bicep-modules are files inside specific Azure DevOps repository. This is a sample how local App Service related Bicep-module is consumed from main file. This App Service module is used to createa Azure App Service resource with preferred network settings. 

module appServiceModule './appservice.bicep' = {
  name: 'appServiceModule'
  params: {
      appServiceName: appServiceName
      appServicePlanName: appServicePlanName
      appServicePlanPlatform: appServicePlanPlatform
      appServicePlanSku: appServicePlanSku
      tags:defaultTags
      appSettings: appSettings
      alwaysOnEnabled: alwaysOnEnabled
      linuxFxVersion: 'DOTNETCORE|6.0'
      httpsRedirectionEnabled: false
  }
}

We want to consume this same App Service Bicep-module from all Azure pipelines so we need to make it globally shared.

Globally shared Bicep-modules

Bicep supports that global Bicep-modules can be shared via Azure Container Registry which acts as a centralized repository. You can find detailed information about how to configure Azure Container Registry from here.

Bicep-modules can be published to Azure Container Registry with the following command. 

az bicep publish --file appservice.bicep --target br:mycontainerreqistry.azurecr.io/bicep/modules/appservice:v1

After publish Bicep-modules from Azure Container Registry can be consumed like this

module appServiceModule 'br:mycontainerreqistry.azurecr.io/bicep/modules/appservice:v1' = {
  name: 'appServiceModule'
  params: {
      appServiceName: appServiceName
      appServicePlanName: appServicePlanName
      appServicePlanPlatform: appServicePlanPlatform
      appServicePlanSku: appServicePlanSku
      tags:defaultTags
      appSettings: appSettings
      alwaysOnEnabled: alwaysOnEnabled
      linuxFxVersion: 'DOTNETCORE|6.0'
      httpsRedirectionEnabled: false
  }
}

To publish modules to a registry, you must have permission to push an image. To deploy a module from a registry, you must have permission to pull the image. For more information about the roles that grant adequate access, see Azure Container Registry roles and permissions.

Summary

Globally shared YAML-templates and Bicep-modules enable a powerful way to optimize and improve maintenance of Azure pipeline (YAML) and Infrastructure of code files (Bicep). Especially if environment is coherent and similar technology stack is widely used then consumption of global component creates lot of value.

During this project the following functionalities were created global.

YAML-templates:

- Build template for .NET Framework based applications

- Deploy template for Azure App Service application

- Deploy template for Azure Infrastructure

- Database migration template

- Unit tests execution template

Bicep-modules:

- Create App Service with preferred network settings

- Key Vault policy module which assigns access policies