Automate project documentation from Azure DevOps to SharePoint on Microsoft 365

Photo by matthew Feeney on Unsplash

We recently had a need to create a knowledge repository, which seems like a straight forward request when you’re dealing with SharePoint. The ask started as, “We would like for people to submit documents, review and approve these documents, and have this library be the be-all, end-all place for approved, sanctioned documentation that people can search.”

Easy, sounds like a SharePoint library. It gives you everything you need. Approvals, filtering, grouping, and searching.

This is great until you get more requirements:

• We have links to other documentation [You can still have Shortcut files in a library. Not an ideal solution, but could still work. Suggestion is to start curating pages that would contain these links or add them to the navigation of the site.]
• We want to provide both in-house and external training material as links, videos, or articles [Again, this sounds like modern pages could handle this]
• We have documentation that we don’t want to update in two places. The source files live in Azure DevOps and these are mostly markdown (md) files. [This is what we are going to solve]

Two days before I started writing this, Yannick Reekmans put out an article on how to connect M365 CLI securely within Azure Devops: Secure app-only authentication with CLI for M365/PnP PowerShell using Variable Groups in Azure DevOps — Yannick Reekmans — Building things on Office 365, Dynamics 365, Power Platform and Azure

Quick thanks to Yannick for doing the difficult work.

Enough with the intro. After following Yannick’s article I was able to connect M365 CLI through Azure DevOps. Then it was a simple means of setting up the file movement. I had a couple options here:

1. Just push the files to a document library with metadata.
2. Add a Markdown web part to a page and insert the text.
3. Add a File viewer web part to a page and view the file.

Here’s the basic architecture of the system, no matter which option we’re using.

Push to Document Library (1 of 3)

Benefits:

• Easy to setup
• Get filtering, grouping, and searching based on metadata within the destination library

Downfalls:

• Currently, SharePoint search does not support indexing the contents of markdown files, so you need to rely on metadata (Vote to include this)
• Viewing of the actually content is not the best. It will feel disjointed from the rest of your experience

Implementation:

m365 spo file add -u https://contoso.sharepoint.com/sites/mytestsite -f 'Shared Documents/AzureAutomation' -p 'sample-markdown.md'

Where -u is the URL to your site, -f is the relative path to your library/folder, and -p is the file in your repo relative to your YAML pipeline file.

Markdown Web Part (2 of 3)

Benefits:

• Can augment the markdown file with more web parts
• Contents are now searchable

Downfalls:

• Little more difficult to set up
• Markdown web part needs HTML in order to function properly which means we need to translate markdown to HTML which involves another library

Implementation:

# Make sure we have pip updated for the 'markdown' package
python -m pip install --upgrade pip
python -m pip install markdown
# Convert the Markdown to HTML, for the web part properties to work correctly
$md = Get-Content sample-markdown.md
$md = $md -join "\n"
$htmlFromMd = python -m markdown sample-markdown.md
$htmlFromMd = $htmlFromMd -join ''
# Set the web part data from the html and md. The data needs both to publish correctly
$webPartData = '{""title"": ""ADO-Markdown"",""description"": ""Auto-generated from ADO pipeline"",""serverProcessedContent"": {""htmlStrings"": {""html"": ""' + $htmlFromMd + '""},""searchablePlainTexts"": {""code"": ""' + $md + '""},""imageSources"": {},""links"": {}},""dataVersion"": ""2.0"",""properties"": {""displayPreview"": true,""lineWrapping"": true,""miniMap"": {""enabled"": false},""previewState"": ""Show"",""theme"": ""Monokai""}}'
# Find the first existing markdown webpart on the page
$pageControlId = m365 spo page control list --webUrl https://contoso.sharepoint.com/sites/mytestsite --name Sample-Markdown-from-ADO.aspx --query "[?title=='ADO-Markdown'] | [0].id" -o json
if ($pageControlId -eq 'null') {
  m365 spo page clientsidewebpart add -u https://contoso.sharepoint.com/sites/mytestsite -n Sample-Markdown-from-ADO.aspx --webPartId 1ef5ed11-ce7b-44be-bc5e-4abd55101d16 --webPartData $webPartData
}
else {
  # Add the webpart to an existing page, and existing web part, if found
  m365 spo page control set -i $pageControlId -u https://contoso.sharepoint.com/sites/mytestsite -n Sample-Markdown-from-ADO.aspx --webPartData $webPartData
}
# Publish the page
m365 spo file checkin --fileUrl /sites/mytestsite/SitePages/Sample-Markdown-from-ADO.aspx --webUrl https://contoso.sharepoint.com/sites/mytestsite

I could have used variables for some of the parameters, but this shows what format is expected for each of the calls. The flow is as follows:

1. Get the markdown python library (I’m sure there are other libraries/technologies that could be used here, but this seemed the easiest)
2. Generate HTML from the markdown file
3. Store the webPartData into a variable, setting “html” and “code” to the generated html, and the original markdown, respectively. I also set the title of the web part so that I can differentiate this web part from other markdown web parts that may be added to the page
4. Try to find an existing markdown web part on the page, so that we don’t just keep adding web parts for each change to the file
5. Add/Update a web part with the new data (Note: When adding the web part, the webPartId should always be 1ef5ed11-ce7b-44be-bc5e-4abd55101d16)
6. Publish the page

File Viewer Web Part (3 of 3)

Benefits

• Can be added to a page, which allows for augmentation with more web parts
• Once the web part is on a page, any update to the file will be reflected on the page.
• If there are multiple pages showing this file, they will all be updated

Downfalls

• The contents of the markdown file are not searchable
• File viewer web part makes it feel disjointed from the rest of your experience

Implementation

I was struggling to find a good reason to automate this. Adding a file viewer web part to a page is trivial, and if you are going this route so that you can augment the markdown contents with other web parts, you’ll be editing the page anyway.

Put it all together

Here’s the look at my solution structure:

markdown-mover
- ado-pipeline.yaml
- move-files.ps1
- sample-markdown.md

ado-pipeline.yaml

1. I’m getting the cert from the Secure Files rather than through Key Vault in this example.
2. appId, m365TenantId, and certThumbprint are coming from variables I set up through the pipeline interface.
3. Rather than inline the commands, I split them into a PowerShell file.

name: Markdown Mover
jobs:
- job: Move_Files
  steps:
  - task: NodeTool@0
  inputs:
    versionSpec: '12.x'
  - task: DownloadSecureFile@1
  name: encodedCert
  inputs:
    secureFile: privateKeyWithPassphrase.pem
  - task: PowerShell@2
  inputs:
    targetType: 'filePath'
    filePath: move-files.ps1
    arguments: -appId $(appId) -m365TenantId $(m365TenantId) -secureFilePath $(encodedCert.secureFilePath) -certThumbprint $(certThumbprint)
    pwsh: true

move-files.ps1

Added some output to understand where we are in the pipeline.

param (
  $appId,
  $m365TenantId,
  $secureFilePath,
  $certThumbprint
)
# Make sure we have pip updated for the 'markdown' package
python -m pip install --upgrade pip
python -m pip install markdown
#install the latest m365 cli
sudo npm i -g @pnp/cli-microsoft365
# Login
$env:CLIMICROSOFT365_AADAPPID = "$appId"
$env:CLIMICROSOFT365_TENANT = "$m365TenantId"
m365 login --authType certificate --certificateFile $secureFilePath --thumbprint $certThumbprint
Write-Host 'login complete'
# Add the files to the library
m365 spo file add -u https://contoso.sharepoint.com/sites/mytestsite -f 'Shared Documents/AzureAutomation' -p 'sample-markdown.md'
Write-Host 'file added'
# Convert the Markdown to HTML, for the web part properties to work correctly
$md = Get-Content sample-markdown.md
$md = $md -join "\n"
$htmlFromMd = python -m markdown sample-markdown.md
$htmlFromMd = $htmlFromMd -join ''
# Set the web part data from the html and md. The data needs both to publish correctly
$webPartData = '{""title"": ""ADO-Markdown"",""description"": ""Auto-generated from ADO pipeline"",""serverProcessedContent"": {""htmlStrings"": {""html"": ""' + $htmlFromMd + '""},""searchablePlainTexts"": {""code"": ""' + $md + '""},""imageSources"": {},""links"": {}},""dataVersion"": ""2.0"",""properties"": {""displayPreview"": true,""lineWrapping"": true,""miniMap"": {""enabled"": false},""previewState"": ""Show"",""theme"": ""Monokai""}}'
# Find the first existing markdown webpart on the page
$pageControlId = m365 spo page control list --webUrl https://contoso.sharepoint.com/sites/mytestsite --name Sample-Markdown-from-ADO.aspx --query "[?title=='ADO-Markdown'] | [0].id" -o json
if ($pageControlId -eq 'null') {
  Write-Host 'Did not find an existing web part on the page :-('
  
  m365 spo page clientsidewebpart add -u https://contoso.sharepoint.com/sites/mytestsite -n Sample-Markdown-from-ADO.aspx --webPartId 1ef5ed11-ce7b-44be-bc5e-4abd55101d16 --webPartData $webPartData
  Write-Host 'webpart added'
}
else {
  # Add the webpart to an existing page, and existing web part, if found
  Write-Host 'Found existing web part! ID:' + $pageControlId
  m365 spo page control set -i $pageControlId -u https://contoso.sharepoint.com/sites/mytestsite -n Sample-Markdown-from-ADO.aspx --webPartData $webPartData
  Write-Host 'webpart updated'
}
# Publish the page
m365 spo file checkin --fileUrl /sites/TestingComms/SitePages/Sample-Markdown-from-ADO.aspx --webUrl https://contoso.sharepoint.com/sites/mytestsite
Write-Host 'published page'
# Log out from the M365 CLI
m365 logout
Write-Host 'logout'