I had a discussion with a colleague before summer holiday about how to schedule background processes in Azure. There are lot of options available for many use cases. This blog posts introduces some Azure services which can be used to execute scheduled background processes.
A scheduled background process refers to a task or operation which is automatically executed in specific times or intervals and typically it doesn't require user intervention. Scheduled processes are useful when system need to execute something "behind the scenes" which is invisible for the end-user. Typically scheduled background processes are related to maintenance or operational tasks. Scheduled background processes are typically part of the almost any systems because especially maintenance related tasks or operations can decrease infrastructure costs.
A few samples of maintenance related scheduled background processes:
- Cleaning unused user sessions from the database.
- Cleaning Azure resources from test environments and re-provision resources by schedule.
- Execute health checks
Operations tasks which can be executed in scheduled background processes:
- Aggregate data from multiple sources and generate reports.
- Train AI model.
- Data syncronization between different internal or external systems.
Scheduling services provided by Azure
Azure Automation Account
Azure Automation Account is a process automation service which enables to run Python and PowerShell scripts (Runbooks) in Azure by a schedule. Azure Automation Account is useful especially for people who maintain the IT infrastructure and are already familiar with PowerShell. PowerShell and Python basically enable to create almost anything in scripts if necessary. Azure Automation Account supports Managed Identity and provides role-based access control and secure store for variables & credentials & cerfiticates.
Execution of process automation (=running scheduled scripts) jobs are billed per minute. Currently (7/23) 500 minutes per month are provided by free which is nice amount of time to execute some basic light tasks or operations.
Azure Automation Account is a good choice if you want to create scheduled operations or tasks with PowerShell and you're running long running tasks (see Checkpoints).
Use cases for scheduling:
- Scaling down SQL server during night and weekends in test environment to save costs
- Execute SQL database maintenance tasks
Azure Function
Azure Function is a serverless computing platform provided by Azure which enables you to write less code, maintain less infrastructure and save costs. Azure Function runtime supports multiple languages i.e. C#, Javascript, F#, Java, Powershell, Python and Typescript. Azure Function widely supports event triggers from multiple Azure services i.e. Service Bus and Storage Account. Timer Trigger is also supported which enables to execute Function on a schedule. When using Azure Functions in Consumption plan you need to remember that default timeout is 5 minutes and it can be extended to 10 minutes. More expensive Premium plan provides a longer timeout.
Azure Function is billed by the time and total amount of executions.
Azure Function has wide support for multiple languages so there is a suitable language available for all to create scheduled background processes. Azure Function is a good choice especially when creating something more complex and process is more advanced than just script with a few lines of code. Serveless platform enables great scalability, but you should know limits of timeouts.
Use cases for scheduling:
- Execute fraud detection model twice a day
- Execute complex business rules to detect which users should be notified by email
[FunctionName("ScheduledBackgroundTrigger")]
public static void Run([TimerTrigger("0 */5 * * * *")]TimerInfo myTimer, ILogger log)
{
// Business logic
}Azure Logic Apps
Azure Logic Apps is an integration platform as a service (iPaaS) where workflows can be created with small amount of coding. Powerfull visual designer of Azure Logic Apps enables quick creation of workflows using drag-and-drop type of working. Azure Logic Apps platform supports large amount of connectors which provide integrations to multiple services provided by Microsoft or external (ex. Salesforce). Azure Logic Apps support recurrence trigger which enables to create varied scheduling rules i.e. repeat weekly on specific days etc.
Azure Logic Apps is a powerful tool to create scheduled background processes via visual designer and take advance of great scalability. Azure Logic App is great match for simpler scheduled tasks or operations because many scenario can be created without writing any code. If business logic becomes large then the maintenance of Azure Logic App might become hard and other solutions i.e. Azure Functions might be better.
Use cases for scheduling:
- Send email notification for admin users about failed orders in Service Bus Dead-Letter
Azure App Service with scheduled WebJobs
WebJobs are tighly coupled with Azure App Service. WebJobs supports scheduling tasks or operations based on CRON expression. With version 3.x of the Azure WebJobs SDK, you can create and publish WebJobs as .NET Core console apps. If you deploy WebJobs alongside with another application in App Service, you should remember that WebJob consumes the same CPU and memory capacity. WebJobs also has limits with timeout and timeout can be adjusted via WEBJOBS_IDLE_TIMEOUT configuration in App Settings.
There has been some discussion that WebJobs will be depracated, but this hasn't confirmed by the Microsoft. I would prefer Azure Functions or Azure App Service with .NET hosted service for all new scheduled solutions which require more advanced handling and business logic.
public class ScheduledBackgroundWebJob
{
public async Task TimerTriggerOperation([TimerTrigger("0 */5 * * * *", RunOnStartup = true)] TimerInfo timerInfo, ILogger logger, CancellationToken cancellationToken)
{
// Business logic
}
}
Azure App Service with .NET hosted service
.NET supports to create and run background operations or tasks using Hosted Services. Basically a hosted service is a class which implements the IHostedService interface. IHostedService requires you to implement StartAsync() method which is called at startup and StopAsync() method which executes in shutdown. Hosted Services are registered in the dependency injection. You can schedule background operation in StartUp() using i.e. Timer class.
Hosted service is a good scheduling solution like Azure Function for more advanced use cases which require more business logic and handling. Hosted service like WebJobs consume same
Use cases for scheduling:
- Clean-up expired user sessions from the database
- Process order queue
public class ScheduledBackgroundHostedService : IHostedService
{
private readonly ILogger _logger;
private Timer _timer;
public ScheduledBackgroundHostedService(ILogger<ScheduledBackgroundHostedService> logger)
{
_logger = logger;
}
public Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Hosted Service running.");
_timer = new Timer(DoWork, null, TimeSpan.Zero,
TimeSpan.FromSeconds(5));
return Task.CompletedTask;
}
private void DoWork(object state)
{
_logger.LogInformation("Timed Hosted Service is working...");
// Business logic
}
public Task StopAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("Timed Hosted Service is stopping...");
_timer?.Change(Timeout.Infinite, 0);
return Task.CompletedTask;
}
}Azure Batch
Azure Batch enables to run large-scale parallel and high-performance computing batch jobs in Azure. Azure Batch is a good solution i.e. nighly data processing, testing or analysis operations which requires high-performance and scalability. Behind the scenes Azure Batch creates and manages a pool of compute nodes (virtual machines), installs the applications you want to run, and schedules jobs to run on the nodes.
Use cases for scheduling:
- AI model training
- Processing large volume of image data
Azure Data Factory
Azure Data Factory is a cloud based ETL (Extract Transform Load) and data integration service. It natively integrates with large number of Azure services and enables creating workflows of orchestrating the data movement and tranformation. Azure Data Factory supports scheduled triggers for pipelines which enables the scheduling based on your rules.
Use cases for scheduling:
- Moving audit log data from Blob Storage to another long-term Blob Account (archieve)
- Create AVRO files from latest order rows and send them to SFTP server
Jobs in Azure Container Apps
Jobs in Azure Container Apps (preview) enables to create containerized tasks which are executed by the schedule (CRON expression). Jobs supports Manual, Schedule and Event type of triggers. If you're already working in containerized environment, Jobs are good solution to consider for scheduled background processes.
Use cases for scheduling:
- Notify users by email about abandoned shopping carts
Scheduled pipelines in Azure DevOps
Pipelines in Azure DevOps support executing different kind of scripts (Powershell etc.) which can do i.e. maintenance type of operations. Pipelines can be scheduled to execute in right moment of time.
Use cases for scheduling:
- Nightly Security check of environment
- Nightly execution of UI automation tests
- Scaling down Azure resources during night and weekends in test environment
- Scaling up Azure resource before office hours in test environment
- Cleaning up sandbox Azure environments
Comments