In this blog, you will learn how to leverage Azure OpenAI’s large language models to generate pull request descriptions based on the changes in your code.
Overview
The goal is to create seamless automation that summarises the code changes in a pull request using natural language. The text generated by the large language model (LLM) is placed in the description field of the pull request.
To leverage large language models, we will use Azure OpenAI. For a frictionless experience, we will integrate Azure DevOps build validations. A build validation is an Azure Pipeline that runs when a pull request is created or updated. The only action required from the developer is to create a pull request, which triggers the process. The build validation will use predefined Azure DevOps variables to enable frictionless automation.
Below is a high-level overview of the process:
- An actor creates a pull request.
- The pull request creation automatically triggers the configured build validation.
- The build validation Azure Pipeline calls the Chat Completion Azure OpenAI API with the data.
- The LLM creates the description based on the prompts from step 3. Human readable natural text is returned to the Azure Pipeline.
- The Azure Pipeline calls the Azure DevOps API to set the pull request description.
Azure DevOps Permissions
To update pull request descriptions, the service principal (service connection) requires the following permissions and configuration:
- The service principal requires a Basic Azure DevOps licence, as it interacts with code (Azure Repos).
- Contributor permissions within the project where the pull requests are created.
Azure AI Foundry
This blog does not cover the deployment of Azure OpenAI or the deployment of the large language model. If you are looking for an Azure Bicep template to deploy Azure OpenAI, see Azure Verified Modules.
Azure AI Foundry is the platform that enables the deployment of large language models, including OpenAI’s GPT-4o, o1, and many more. For this blog, the model used to generate pull request descriptions is GPT-4o.
In Azure AI Foundry, the deployment looks like this:
Azure OpenAI key
An Azure OpenAI key is required to authenticate with the APIs. The key can be found in the Azure Portal:
Automate pull request description
To automate pull request descriptions, three PowerShell functions and an orchestrator Azure Pipeline are used:
• Set-PullRequestDescription.ps1
• Invoke-PullRequestGeneration.ps1
• Get-Repository.ps1
• azure-pipelines.yml (build validation pipeline)
Set-PullRequestDescription function
The function below gathers information about the repository by invoking Get-Repository, and it also calls Invoke-PullRequestGeneration. More details on these functions are covered later in this blog post.
Finally, it makes a PATCH API call to Azure DevOps with the newly generated description in natural language, produced by the Azure OpenAI model.
| function Set-PullRequestDescription { | |
| param ( | |
| [parameter(Mandatory)] | |
| [string] | |
| $SourceBranch, | |
| [parameter(Mandatory)] | |
| [string] | |
| $TargetBranch, | |
| [parameter(Mandatory)] | |
| [string] | |
| $PullRequestId, | |
| [parameter(Mandatory)] | |
| [string] | |
| $ProjectName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $RepositoryName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $OrgName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $OpenAIKey | |
| ) | |
| $apiUrl = "https://dev.azure.com/$OrgName/$ProjectName/_apis/git/repositories/$RepositoryName/pullRequests/$($PullRequestId)?api-version=7.1" | |
| $formattedSourceBranchName = $SourceBranch -replace "refs/heads/", "" | |
| Get-Repository -ProjectName $ProjectName -RepositoryName $RepositoryName -OrgName $OrgName -SourceBranch $formattedSourceBranchName | |
| $newDescription = Invoke-PullRequestDescriptionGeneration -SourceBranch $formattedSourceBranchName -TargetBranch $TargetBranch -OpenAIKey $OpenAIKey | |
| $body = @{ | |
| description = $newDescription | |
| } | ConvertTo-Json | |
| $headers = @{ | |
| Authorization = "Bearer $(Get-AzAccessToken | Select-Object -ExpandProperty Token)" | |
| "Content-Type" = "application/json" | |
| } | |
| Invoke-RestMethod -Uri $apiUrl -Method PATCH -Headers $headers -Body $body | |
| } |
Get-Repository function
This function clones the Azure Repo where the pull request is created. When a checkout is performed during a pipeline run, it is in a “detached HEAD” state, meaning the checkout is done on a commit instead of a branch.
After cloning the Azure Repo, we gain access to both the main (target) branch and the feature (source) branch.
| function Get-Repository { | |
| param ( | |
| [parameter(Mandatory)] | |
| [string] | |
| $ProjectName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $RepositoryName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $OrgName, | |
| [parameter(Mandatory)] | |
| [string] | |
| $SourceBranch | |
| ) | |
| git clone "https://$(Get-AzAccessToken | Select-Object -ExpandProperty Token)@dev.azure.com/$($OrgName)/$($ProjectName)/_git/$($RepositoryName)" | |
| Set-Location $RepositoryName | |
| git checkout $SourceBranch | |
| } |
Invoke-PullRequestGeneration function
This function calls the Chat Completion API in Azure OpenAI. However, some configuration is required before making the API call.
| function Invoke-PullRequestDescriptionGeneration { | |
| param ( | |
| [parameter(Mandatory)] | |
| [string] | |
| $SourceBranch, | |
| [parameter(Mandatory)] | |
| [string] | |
| $TargetBranch, | |
| [parameter(Mandatory)] | |
| [string] | |
| $OpenAIKey | |
| ) | |
| $openai = @{ | |
| api_key = $OpenAIKey | |
| api_base = 'https://openai-devops-jlo.openai.azure.com/' | |
| api_version = '2024-08-01-preview' | |
| name = 'openai-devops-jlo-gpt4o' | |
| } | |
| # Completion text | |
| $messages = @() | |
| $messages += @{ | |
| role = 'system' | |
| content = 'Start with an overall summary of the changes before summarizing the git log content using bullet points. Combine the commits into one set of bulletpoints. Make sure the summarization is clear and easy to read. ' | |
| } | |
| $messages += @{ | |
| role = 'user' | |
| content = $(git log "$TargetBranch..$SourceBranch" -p) | |
| } | |
| # Header for authentication | |
| $headers = [ordered]@{ | |
| 'api-key' = $openai.api_key | |
| } | |
| # Adjust these values to fine-tune completions | |
| $body = [ordered]@{ | |
| messages = $messages | |
| } | ConvertTo-Json | |
| $url = "$($openai.api_base)/openai/deployments/$($openai.name)/chat/completions?api-version=$($openai.api_version)" | |
| $response = Invoke-RestMethod -Uri $url -Headers $headers -Body $body -Method Post -ContentType 'application/json' | |
| return $response.choices.message.content | |
| } |
The $openai variable stores key configuration details:
| Parameter | Description |
|---|---|
| API_KEY | The secret key used to authenticate with Azure OpenAI |
| API_BASE | The URL of Azure OpenAI |
| API_VERSION | The version of the deployment model |
| name | The name of the deployment model, which can be found in the Azure AI Foundry portal |
Completion text
The completion text contains two different roles:
| Role | Description |
|---|---|
| System role | The system role is used to provide initial instructions to the model. For example, you can give a brief description on how the assistant should behave. |
| User role | The user role is for sending questions or context to the assistant. |
In this case, the system message is:
“Start with an overall summary of the changes before summarizing the git log content using bullet points. Combine the commits into one set of bulletpoints. Make sure the summarization is clear and easy to read.”
For the user message, the following command is used:
git log "$TargetBranch..$SourceBranch" -p
This Git command displays the commit log history between two branches. The -p option includes the code changes, which is useful as additional context for the large language model. This allows the model to access both the commit messages and the actual code changes.
The command output looks like this:
git log "$TargetBranch..$SourceBranch" -p commandFinally, the headers for authentication and the message body with the prompts are set, and the Chat Completion API is called. The output is then returned and used in the Set-PullRequestDescription function to update the Azure DevOps pull request description.
Orchestration pipeline
The Azure Pipeline uses the AzurePowerShell task, leveraging a service connection with the necessary permissions to make API calls to Azure DevOps. The task calls the Set-PullRequestDescription function, which uses predefined Azure DevOps variables as input parameters. The OpenAI key is loaded from an Azure DevOps library.
| trigger: none | |
| pool: | |
| vmImage: ubuntu-latest | |
| variables: | |
| - group: vg-openai | |
| steps: | |
| - task: AzurePowerShell@5 | |
| displayName: 'Generate PR Description using GPT4o' | |
| inputs: | |
| azureSubscription: sc-devops-openai | |
| ScriptType: 'InlineScript' | |
| azurePowerShellVersion: LatestVersion | |
| pwsh: true | |
| Inline: | | |
| . ./Invoke-PullRequestDescriptionGeneration.ps1 | |
| . ./Get-Repository.ps1 | |
| . ./Set-PullRequestDescription.ps1 | |
| Set-PullRequestDescription ` | |
| -SourceBranch $(System.PullRequest.SourceBranch) ` | |
| -TargetBranch $(System.PullRequest.TargetBranchName) ` | |
| -PullRequestId $(System.PullRequest.PullRequestId) ` | |
| -ProjectName $(System.TeamProject) ` | |
| -RepositoryName $(Build.Repository.Name) ` | |
| -OrgName ("$(System.CollectionUri)" -split "/" | Select-Object -index 3) ` | |
| -OpenAIKey $(openai-key) |
Result
When a pull request is created, the build validation will automatically run. During this run, the code changes are analysed, and the model is instructed to start with a summary of the changes. After that, a summary of the Git log must be generated in bulletpoints:
Let’s take a look at some code changes. The pull request description states: ‘Previous sample content replaced with a motivational message: “You are awesome!”’ which is correct. The README has been updated as follows:
Another change is that scripts have been moved, and their references are now in the azure-pipelines.yml file:
Personalisation
This is just one possible format for automating pull request descriptions. If you want to adjust the format, you can customise the prompt to your preference. Keep in mind that it may take a few attempts to find the perfect summary format.
Conclusion
By leveraging Azure OpenAI, Azure DevOps, and PowerShell, you can automate the generation of pull request descriptions, saving time and reducing the manual effort involved in documenting changes.
Azure Cloud | John Lokerse 