Bringing down ASP.NET deployment interruptions from minutes to blip of a time with Azure DevOps

This is going to be about improving deployment process from classic file copy to something more effective, automated and low friction for developers and bringing no interruptions to users. Maximizing Developer Effectiveness is actually competitive advantage after all.

Pain points

Our customer, a national school management platform was experiencing inconvenience when deploying a fairly big classic .NET Framework application to 22 Windows IIS servers:

1. Deployments would be taken at night
2. If a hotfix had to be issued at day with high load - it could take up to 30 minutes of complete system halt as IIS server farm compiles resources while pile of requests waits in a queue.
3. The deployment is not atomic - that is, while files are being copied between v1 and v2, we would get v1.314 - while some files are updated, the others not.
4. If we ever wanted a rollback - occasionally files would be locked, exacerbating already bad situation.
5. The website itself consists of ~220MB of data across ~2500 number of files. And small file copy is SLOW.

The root cause of this is that .NET resources (.aspx pages, razor views) get compiled to assemblies on first request whenever they change. When IIS is constantly bombarded with requests to all possible resources, it all must be compiled at the same time on every IIS server. On the other hand, when deployed at night, delay was minor as not many requests came in, and those that did, compiling was handled fairly quickly.

ASP.NET Requests Queued is a good performance counter to monitor to identify when user browsers are just waiting for response and how quickly situation resolves after an update. A healthy value is 0 or something close to that. Here you see cumulative request queue for all IIS servers in evening, when load actually is not high. 4 minutes of interruptions:

We thought about using robocopy to copy changed files based on timestamp, but that wouldn't work for us - as there could be multiple persons that would deploy the code. When Visual Studio Publishes the project, files between machines have different timestamps. Rsync would probably help greatly, but then again - some challenge to get that on developers windows machines and windows share server. And if different hosts build files, little differences in build tools may end up with different checksums. So we didn't go down this path.

The webroot (and IIS Shared Configuration) is actually served from a single Windows share, which at least provided a low effort way to conveniently copy files to a single location and keep them, along with the IIS configuration, in sync.

That's how we lived for years and it was clear that something had to change.

Deployment strategy

We came to a decision we would like to stop depending on a file share. We already started using Azure DevOps for some other stuff. So we got an idea 💡

1. Keep files on IIS local disk (Duh)
2. Azure DevOps Agent would be the one compiling code.
3. We would precompile views in advance. Locking wouldn't be an issue as we will copy within a new folder. 💎
4. Orchestrate deployment to all IIS servers via Azure DevOps Release Pipelines
5. Run all .dll files through ngen. That would eliminate additional delay when assembly is first loaded. As C# code compiled is actually MSIL - an intermediate language that CPU doesn't understand any of that. JIT compiler is the one that translates that to bytecode for the processor architecture at hand. And it does so usually Just-In-Time or On-The-Fly. Or, in our case, ngen (Native Image Generator)
6. A deployment task would have to copy files within a new folder and then we would change IIS Physical path from where to serve site files. 💎

So there are some gems there that are enough to solve our pain points. Now let's see some practical ways on achieving all of this!

Implementing build

Build is the process of generating files we must copy onto IIS servers.

Before we talk about build & deployment, let's get this straight: We are using Azure DevOps online offering. However this could as well be On-Premises installation or any other CI/CD platform of your choice. We are, however, using self-hosted agents that perform the build tasks.

We are talking about solution, which consists of multiple .NET projects. Our main interest is deploying ....Web.Application and ..Web.Application.Payments. The others are mostly dependencies which must also be built. So, the developers used to build app within Visual Studio. When deploying web application, you had to right click on particular project and choose publish. From there, some settings could be configured like whether we want precompilation or not, whether we deploy straight to IIS or filesystem. In our case, it was deployed to filesystem and then copied to production. More about Publishing an ASP.NET web app.

When writing Azure DevOps pipeline, I have to figure out what kind of MSBuild properties I have to use to invoke the "Publish" process. So, after searching the web, reading MSBuild diagnostic output logs, reading MSBuild .target files, I'v come up with properties I need.

By the way, here is a quick tip on how to find relevant .target files by some keyword:

> gci -Path "${env:ProgramFiles(x86)}\Microsoft Visual Studio\" -Recurse -Filter "*.targets" | sls "MvcBuildViews" -SimpleMatch -List
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\MSBuild\Microsoft\VisualStudio\v15.0\Web\Microsoft.Web.Publishing.targets:849:  <Target Name="CleanupForBuildMvcViews" Condition=" '$(_EnableCleanOnBuildForMvcViews)'=='true' and '$(MVCBuildViews)'=='true' " BeforeTargets="MvcBuildViews">
C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\MSBuild\Microsoft\VisualStudio\v16.0\Web\Microsoft.Web.Publishing.targets:847:  <Target Name="CleanupForBuildMvcViews" Condition=" '$(_EnableCleanOnBuildForMvcViews)'=='true' and '$(MVCBuildViews)'=='true' " BeforeTargets="MvcBuildViews">

Or better yet, use Project System Tools extension with MSBuild Binary and Structured Log Viewer to sneak peek into what MSBuild is doing - you won't regret it.

So, the relevant "Publish" command within Azure Devops build pipeline:

- task: VSBuild@1
  displayName: Publish Web.Application  
  inputs:
    solution: 'Web.Application'
    msbuildArgs: >
      /t:Build,GatherAllFilesToPublish
      /p:PublishProfileName="$(publishProfileName)"
      /p:WebPublishMethod=FileSystem
      /p:DeleteExistingFiles=true
      /p:DeployOnBuild=true
      /p:MvcBuildViews="${{ parameters.Precompile }}"
      /p:PrecompileBeforePublish="${{ parameters.Precompile }}"
      /p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
      /p:SingleAssemblyName="Web.Application.Precompiled"
      /p:UseMerge="true"
      /p:DebugSymbols="True"
      /p:EnableUpdateable="False"
      /p:PublishUrl="$(Build.BinariesDirectory)\my"
      /p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\my"
      /p:RunNpmScripts="$(runNpmScripts)"
      /p:AutoParameterizationWebConfigConnectionStrings="false"
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    msbuildArchitecture: x64
    logFileVerbosity: detailed

• Note that specifying publishProfileName, I use the actual publish profile used by Visual Studio Publish process and then override some properties by passing /p msbuild arguments.
• The WebPublishMethod ensures build files are put within PublishUrl folder.
• RunNpmScripts is our own custom build property used within .pubxml to run some npm build process.
• MvcBuildViews, PrecompileBeforePublish, SingleAssemblyName, UseMerge, EnableUpdateable all relate to precompilation. Before build, I can choose to disable precompilation if I want the build to happen much faster.
• AutoParameterizationWebConfigConnectionStrings - required for copy/paste if publish profile contains connection string replacements. Without this, there will be a placeholder value that must be replaced afterwards. More on StackOverflow.

The full build can be seen here:

# ASP.NET
# Build and test ASP.NET projects.
# Add steps that publish symbols, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/apps/aspnet/build-aspnet-4
trigger: none
name: $(Build.SourceBranchName) $(Date:yyyyMMdd)$(Rev:.r)

pool:
  name: win-dev-pool

parameters:
  - name: Precompile
    default: false
    type: boolean
    displayName: Precompile

variables:
  webApplicationProject: 'Web.Application'
  solution: 'solution.sln'
  buildPlatform: 'x64'
  buildConfiguration: 'Release'
  releaseArchiveFilename: 'webapp.7z'
  releasePaymentsArchiveFilename: 'payments.7z'
  # which .pubxml file to use. Don't append .pubxml
  publishProfileName: STAGING
  runNpmScripts: true
  artifactShareName: \\MY-APP-AGENT2\agentpublishedfiles

steps:
- task: NuGetToolInstaller@1
- task: NuGetCommand@2
  inputs:
    command: 'restore'
    restoreSolution: '$(solution)'
    feedsToUse: 'config'
    nugetConfigPath: 'NuGet.Config'

# npm script prerequisites
- task: NodeTool@0
  inputs:
    versionSpec: '14.x'

# https://docs.microsoft.com/en-us/azure/devops/pipelines/release/caching?view=azure-devops#nodejsnpm
- task: Cache@2
  displayName: Cache npm
  inputs:
    key: 'v2 | npm | "$(Agent.OS)" | $(webApplicationProject)/package.json'
    path: '$(webApplicationProject)/node_modules'
    restoreKeys: 'v2 | npm | "$(Agent.OS)"'
    cacheHitVar: NPM_CACHE_RESTORED
  condition: and(succeeded(), variables.runNpmScripts)

- task: npmAuthenticate@0
  inputs:
    workingFile: '$(webApplicationProject)/.npmrc'
- task: Npm@1
  displayName: 'npm install'
  inputs:
    command: 'install'
    workingDir: '$(webApplicationProject)/'
  condition: and(succeeded(), variables.runNpmScripts, ne(variables.NPM_CACHE_RESTORED, 'true'))

# Workaround for AspNetPrecompile to skip scanning node_modules directory and finding .c,.cpp,.h files... https://stackoverflow.com/a/20963170/50173
- task: CmdLine@2
  displayName: Hide node_modules.
  inputs:
    script: 'attrib +H $(webApplicationProject)/node_modules'

# Specificually pass PublishProfileName as empty. Because otherwise in consequent runs, this task will run npm run-script that is specified as BeforeBuild target within publish profile.
# We already run publish actions further.
- task: VSBuild@1
  displayName: Build $(solution)
  inputs:
    solution: '$(solution)'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    msbuildArchitecture: x64
    logFileVerbosity: detailed
    msbuildArgs: >
      /p:PublishProfileName=""

- task: VSTest@2
  displayName: Test $(solution)
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
  condition: ne(variables.NoTests, true)

# Target also Build, otherwise BeforeBuild target won't execute. BeforeBuild specifies npm run-script commands. So, this project gets built twice, but project build by itself is fast.
# PublishUrl actually unused as we won't use msdeploy to deploy stuff, just simple copy.
# Precompilation decision matrix: https://docs.microsoft.com/en-us/previous-versions/aspnet/bb398860(v=vs.100)

- task: VSBuild@1
  displayName: Publish Web.Application  
  inputs:
    solution: 'Web.Application'
    msbuildArgs: >
      /t:Build,GatherAllFilesToPublish
      /p:PublishProfileName="$(publishProfileName)"
      /p:WebPublishMethod=FileSystem
      /p:DeleteExistingFiles=true
      /p:DeployOnBuild=true
      /p:MvcBuildViews="${{ parameters.Precompile }}"
      /p:PrecompileBeforePublish="${{ parameters.Precompile }}"
      /p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
      /p:SingleAssemblyName="Web.Application.Precompiled"
      /p:UseMerge="true"
      /p:DebugSymbols="True"
      /p:EnableUpdateable="False"
      /p:PublishUrl="$(Build.BinariesDirectory)\my"
      /p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\my"
      /p:RunNpmScripts="$(runNpmScripts)"
      /p:AutoParameterizationWebConfigConnectionStrings="false"
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    msbuildArchitecture: x64
    logFileVerbosity: detailed


- task: VSBuild@1
  displayName: Publish Web.Application.Payments  
  inputs:
    solution: 'Web.Application.Payments'
    msbuildArgs: >
      /t:Build,GatherAllFilesToPublish
      /p:PublishProfileName="$(publishProfileName)"
      /p:WebPublishMethod=FileSystem
      /p:DeleteExistingFiles=true
      /p:DeployOnBuild=true
      /p:MvcBuildViews="${{ parameters.Precompile }}"
      /p:PrecompileBeforePublish="${{ parameters.Precompile }}"
      /p:WDPMergeOption="MergeAllOutputsToASingleAssembly"
      /p:SingleAssemblyName="Web.Application.Payments.Precompiled"
      /p:UseMerge="true"
      /p:DebugSymbols="True"
      /p:EnableUpdateable="False"
      /p:PublishUrl="$(Build.BinariesDirectory)\payments"
      /p:WPPAllFilesInSingleFolder="$(Build.BinariesDirectory)\payments"
      /p:AutoParameterizationWebConfigConnectionStrings="false"
    platform: 'AnyCPU'
    configuration: '$(buildConfiguration)'
    msbuildArchitecture: x64
    logFileVerbosity: detailed

# Artifacts
- task: ArchiveFiles@2
  displayName: Archive $(releaseArchiveFilename)
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)\my'
    includeRootFolder: false
    archiveType: '7z'
    sevenZipCompression: 'fastest'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(releaseArchiveFilename)'
    replaceExistingArchive: true
    verbose: true
- task: ArchiveFiles@2
  displayName: Archive $(releasePaymentsArchiveFilename)
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)\payments'
    includeRootFolder: false
    archiveType: '7z'
    sevenZipCompression: 'fastest'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(releasePaymentsArchiveFilename)'
    replaceExistingArchive: true
    verbose: true

- task: PublishBuildArtifacts@1
  displayName: Publish website deployment Artifacts
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'FilePath'
    TargetPath: '$(artifactShareName)\my'

1. What this does, is builds solution (dependency .dlls)
2. Builds JavaScript SPA (Single Page App). Commands are buried within MSBuild project.
3. Publishes 2 applications (Just generates files on disk)
4. 7zips those files
5. Publish artifact (Azure Devops thingie - makes them available for release pipeline). In this case, they are copied to a share.

Implementing deployment

Once we have the artifacts ready (archive of website files), we are ready to implement Release. The release must copy given files to some directory and instruct IIS to change base path from where it will server files. We had 2 options to choose from:

1. Make Build agent connect to IIS servers and perform the deployment on each IIS server. In this case, we have to write some script that will copy files onto each server and issue some commands to IIS. Moreover, we must control/see whether deployment is successful or not. And the build agent could actually be in another domain, with no direct access to production.
2. Install Deployment agent on all target IIS servers. Every host then receives deployment job and does whatever it is instructed to do. We don't have to bother about any other remote communication channel other than Deployment Agent with Azure DevOps server over 443/TCP. Plus we get nice UI of seeing whether deployment succeeded, partially succeeded (if partially, which hosts failed) and on which step it failed. Neat.

We went for the second option. Deployment agent is actually almost same as Build agent, just carries on deployment tasks.

When creating release pipeline, you get to play with the UI which is nice.

As you see, I'v split some operations into multiple steps.

1. Root App Pool - ensures appropriate application pool is created on IIS. It is actually a one-time step and may have been created beforehand.
2. Copy my files - PowerShell script to extract 7z files:

Expand-7Zip -ArchiveFileName "$(System.DefaultWorkingDirectory)\_Web.Application\drop\webapp.7z" -TargetPath "$(DeployTargetFolder)"

1. Ngen my - runs ngen.exe on all .dll files found within deployment folder. Also using custom condition so this step which may take a little more than a minute, could be turned off: and(succeeded(), eq(variables.Ngen, 'true'))

Set-Alias ngen -Value (Get-ChildItem -Recurse $env:windir\Microsoft.NET\Framework64\ -Filter ngen.exe | ? Length -gt 0 | select -first 1).Fullname

Get-ChildItem $(DeployTargetFolder) -Recurse -Filter *.dll | % { ngen install $_.Fullname /nologo /verbose }

1. Deploy my - Switchers virtual directory, issues some IIS configuration commands. When this stage is completed, IIS servers new code.

The great thing is that if any of steps fail for ANY IIS server before Deploy step, deploy won't run for ANY IIS server and they will all be still consistent.

In case of a rollback, we can open appropriate release and for "Deploy my" stage, press Redeploy. IIS will immediately switch back to appropriate folder.

Drawbacks

Using cloud hosted solution, if issues do happen, we won't be able to run our deployment pipeline. And they do happen. However, this may impact us on the rare case of rushing some kind of hotfix out. There are 2 steps we can take in this case:

1. Change IIS path manually to previous deployment folder (rollback)
2. Build via developer Visual Studio and copy appropriate DLL manually. Hotfix usually doesn't involve much code changes and is probably contained within a single or few files.

Otherwise we just wait for while DevOps issue gets resolved.

End result

Everything is actually configured that, when production branch gets new code, build is run automatically. After build, deployment is run automatically but stops for an approval. With Azure Pipelines Slack app we just get a message within channel where upon Approve button press, code goes into production.

This doesn't include database updates, which still must be performed manually, but eventually DACPAC deployment can be incorporated within a pipeline too. But luckily, database updates are more rare than backend code/frontend updates.

Judging from the Request Queue size, guess where did deployment happen?

Yeah, that little spike just before 09:00. Except, vertical axis shows max 30 instead of 3k and horizontally, just a blip of a time.

For a fair picture I should share another deployment request queue graphic:

Queue size spiked to almost 300. However it happens on some IIS servers, in this case 3 out of 10. After 60-90 seconds, queue size dropped, so fairly short period of time compared to what we experienced before. But it's not like those IIS completely stopped processing GET/POST requests - actually only minority of requests queued up. Currently I don't know the cause. Maybe if you have any thoughts on what may cause it, leave down in the comments.

In the end, the results leave everyone happy!

On an upcoming post, I'd like to share how these IIS server/Windows OS settings can be installed, managed and kept in sync with PowerShell DSC. And you don't have to keep a separate documentation file somewhere that may drift and be outdated in time. Stay tuned! 👋