Unlike the subscription level, you cannot just uses Az DevOps command with a management group parameter. This does not appear to be available, the answer is to pass in a json template.
On the above picture, you can see next to the words Tenant Root Group a link (details). You probably do not have the details link clickable at this time. This is because although you have been able to create the initial Tenant Root Group – Management Group, you need to promote your account access to it.
Note: You can only do this as a Global Administrator.
In the left hand navigation under Manage, click Properties
Under Access management for Azure resources switch the button to Yes.
Now if you go back to the Tenant Root Group – Management Group, you will be able to click the details link and have access to the Management group, see deployments made at that level, modify access for others etc.
Switching the button to No will then remove your access.
Programatically
Using Az Cli, log in with you account first az login. The below snippet, on line 2 shows how to give yourself access. Where line 5 & 6 would remove the account.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Running of this code, will create a DevOps project if it doesn’t exist, and then create a Management Group level Service Connection to the Tenant Root Management Group. To apply to a different level management group would require modification to the code to grab the name and ID of the management group you wish to use and pass into the JSON template.
Your account is now set up to run, you will need to first be logged into AZ Cli.
Note: This can be a Service Principal, as long as the account being used is able to list App Registrations, and has ‘User Access Administrator’ RBAC on the Tenant Root Group – Management Group.
You will need to create a ‘management-group.json’ file which is used as a template, and key tokens will be replaced within the script.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
(Line 32) – How the authentication works with DevOps. The Personal Access Token is added to the $Env: variable “AZURE_DEVOPS_EXT_PAT”
(Line 61 – 74) – Updating the json template, saving the file as a temp file, and then creating the Service Connection passing in the json template. The json template is the same template used by Azure DevOps when you set up the Management Group Service connection manually, you can see this by watching the network traffic.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To run the above code, you will need to put in your parameters. Replace with your values then run the below script, this will call the script above.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
A previous post of mine Connecting to Azure Devops with a Service Principal has been popular since I have written it. Therefore, I’ve decided to extend on the topic and show how you can do it programatically with AZ DevOps.
Sign in and click on User settings -> Personal access tokens
Click New Token
Give it a meaningful name so you know what the PAT token is for in the future. (E.g, Devops Service Connection)
Select your Organization
Select the Expiration date for as long as you need. Maximum 1 Year
Select Scopes at Full access (You might want to tighten your permission in a production environment, for this demo Full access is fine).
Click Create
Once you have clicked Create this is the only chance to grab a copy of the token. Please take a copy of this token as you will require it later.
The Code
You will need to first be logged into Az Cli. You can sign in using a service principal as you might with a pipeline, as long as the account being used is able to list App Registrations, and ‘User Access Administrator’ RBAC role to be able to apply contribute access to the DevOps service principal on the subscription (Line 43) .
The important part to note in the code is how the authentication works with Devops. The Personal Access Token is added to the $Env: variable “AZURE_DEVOPS_EXT_PAT”. (Line 32)
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To run the above code, you will need to put in your parameters. Replace with your values then run the script, this will call the script above.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This blog post came about because I wanted a way to create new Application Registrations and grant consent for the tenant, all programmatically. This is so I can use Devops pipelines to create and deploy my code without any human interaction or using a person account.
By reading through Sam’s post it helped me understand the connection between Application Registrations, Service Principals and Oauth2permission, and helped me on the quest of understand how to grant the Application permissions through appRoleAssignments.
At my Github project https://github.com/pmatthews05/CFAppOnlyGrantPermissions the README.md will walk you through how to set up and run the code. At the end of the README.md file you should have 2 Application Registration, where the Azure API Registration app would have created the second app (in my case CFCodeApp) for you. This code is idempotent. You can change the permissions for an existing Application Registration by providing it a different Permission.json file.
As the README.md file gives the instruction on how to run the code, I will not replicate it here. I will use the rest of this post to explain how the code works.
Permissions required for ‘Azure API Registration’
To allow the Azure API Registration to create new Application Registrations using AZ cli it requires to use both the legacy Azure Active Directory Graph and Microsoft Graph permissions. It seems that some of the commands in the az cli still points to https://graph.windows.net when it makes calls, according to some issue notes in the az cli git hub repository, it looks like this is in the process of being changed.
With Azure Active Directory Graph we need 2 permissions
Application.ReadWrite.All – This allows us to read and write the Application Registrations.
Directory.ReadWrite.All – This allows us to read the application registration permission list, and service principal information.
With Microsoft Graph we also need permission.
AppRoleAssignment.ReadWrite.All – This allows us to call the REST API to grant permissions and assign Role assignment permissions.
Steps in the code
Set-AppRegistration
Set-AppCredentials
Set-ServicePrincipalForAppId
Remove-CurrentAppPermissions
Set-DelegatePermissions
Remove-CurrentOauth2PermissionGrants
Set-ApplicationPermissions
Remove-CurrentServicePrincipalGrants
Please note, the snippets of code I am showing here in the blog, are showing the command(s) that are performing the main action, not the full function.
Set-AppRegistration
We need an App Registration to be created first. If the name already exists it just returns the existing App Registration
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This will create a secret for the App Registration with a random secret with the description set to Registration. There are a couple of override parameters that I am not using, where you can give it your own Description, and provide your own SecureString secret. This returns the appCredentials that supply the appId, name, password and tenantId. In the script it outputs this to screen at the end, however, if using in production environment, you would probably want to put the secret value in a keyvault, without displaying to the user what the value is.
Set-ServicePrincipalForAppId
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
All App Registrations require a Service Principal behind them. When you manually create an App Registration and assign permissions, it automatically creates a service principal for you. When you create an App Registration programmatically, it is your responsibility to also create the Service Principal. It is the Service Principal that defines the access policy and permissions for the user/application in the Azure AD tenant. A multi-tenant App Registration would have the same app Id in all tenants, but all have a different Service Principal which allows them access within that tenant. For example, in all tenants the AppId for the Microsoft Graph API is ‘00000003-0000-0000-c000-000000000000’ and in your tenant it has an associated Service Principal, which is a different object Id in your tenant compared to mine. When I finally understood this, it made more sense how this all ties together.
The above az command is not idempotent, and therefore a check to see if it already exists is required.
Remove-CurrentAppPermissions
To allow idempotency of my script, I wanted to ensure that it removes all existing permissions before adding them back in. This piece of code does not remove the permission from the service principal, and if you stop the code after this command, you will see that your API Permissions in the GUI would look like this.
Seeing the permissions separated at the bottom of the screen, now understanding the relationship between Application Registrations and Service Principal, it makes a lot more sense to me now. The service principal still has access at this point and calls to these API’s will still work.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The code gets a list of all the permissions assigned to the Application Registration, then loops through each resourceAppId (the objectId value of the API permission service principal e.g, Microsoft Graph, SharePoint)*2 and deletes the permission.
Set-DelegatePermissions
To ensure the code is idempotent the first thing I am doing is removing the delegate permission from the Service Principal. See the next section on how this works.
Now we need to assign the Application Registration permissions for the delegate permissions. We do this by providing the AppID of our Application Registration, the API Permission AppID (the appID of the API Permission service principal e.g, Microsoft Graph)*2 and the oauth2Permissions scope Id*4
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
clientId: This is the Service Principal Object ID that is tied to your App Registration
consentType: Set to AllPrincipals when granted to the entire tenant, or Principal when granted to an individual user
id: The ID of the oauth2permissiongrants
principalId: This is set to null if using AllPrincipals, otherwise it will contain the objectID of the User that has been granted the permission
resourceId: This is the Serivce Principal Object ID value of the API Permission*2
scope: This is a string array of granted scope values for the given ResourceId. (e.g User.Read Directory.Read.All etc)
If the oauth2permissiongrants with the App Registration Service Principal Object ID and API Permission Service Principal Object ID (clientId and resourceId) doesn’t exist in your tenant, then you will need to POST a new oauth2permissiongrants, otherwise you will require to PATCH an existing oauth2permissiongrants/<id> with the new string array of scope values.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
You must add a startTime and expiryTime, it does not matter what the datetime is, as long as expiryTime is later than the startTime.
Remove-CurrentOauth2PermissionGrants
To remove the permissions from the Service Principal for the Delegate Permissions, we need to remove the Oauth2PermissionGrants.
Unfortunately, with App Only permissions you cannot delete an oauth2permissiongrants. You require to access the directory as a person to delete. I found that by setting the scope to empty string, gives the same desired effect as removing them.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Please Note: I am using Invoke-RestMethod instead of az rest because I have not been able to get it to work without an error message.
Set-ApplicationPermissions
To ensure the code is idempotent the first thing I am doing is removing the application permission grants from the Service Principal. See the next section on how this works.
Now we need to assign the Application Registration permissions for the application permissions. We do this by providing the AppID of our Application Registration, the API Permission AppID (the appID of the API Permission service principal e.g, Microsoft Graph)*2 and the AppRoles scope Id*3
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Next, we need to assign the Application Registration associated Service Principal AppRoleAssignments to grant these permissions to the tenant.
Using Graph API explorer, you can view all the Application Role Grants for your Application Registration. You will need the Service Principal Object ID*1 and then use the following URL:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Unlike the Oauth2PermissionGrants, where there is only one entry per clientid and resourceId which contains all the scopes, with AppRoleAssignments there is an entry for each scope, and it uses the appRoleId scope Id*3 instead of the scope string value.
id: The ID of the appRoleAssignment
principalId: The Service Principal Object ID that is tied to your App Registration
resourceId: This is the Serivce Principal Object ID value of the API Permission*2
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To remove the permission from the Service Principal for the Application Role permissions, we need to remove the AppRoleAssignments ID’s for the service principal
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Please Note: I am using Invoke-RestMethod instead of az rest because I have not been able to get it to work without an error message.
Conclusion
That is it. In the Data folder of the github project there is an examplePermission.json file. As you can see that the JSON format is very flexible to add more or remove permissions. The name can be the appDisplayName or the AppId of the API Permission.
Run the Add-RegistrationAndGrantPermissions.ps1 script passing in the name of your App Registration you wish to create / update and your custom permission file. The script will run fine with user logged in, or an App Only with the correct permissions.
Please feel free to use/enhance the github project.
Footnotes
*1 How to find your Service Principal Object ID of your Application Registration
Using the GUI, the quickest way to find your Service Principal Object ID is to first go to the overview of the Application Registration. Then on the right-hand side of the screen, click the name of your Application Registration where Manage application in local directory is.
This will take you to the Service Principal information in your tenant. It is here you can get the ObjectID.
*2 How to find the Service Principal Object Id for the Permission API.
You may already know that Microsoft Graph API appId is 00000003-0000-0000-c000-000000000000, this is the same on all tenants, but the service principal object ID is different in each tenant. This is the resourceId. To find out what the objectID is in your tenant run the following script.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
az ad sp list --query "[?appId=='00000003-0000-0000-c000-000000000000'].{appDisplayName:appDisplayName,appId:appId,objectId:objectId}" --all --output table
Remove “?appId== ‘00000003-0000-0000-c000-000000000000′” between the [ ] and it will list all for your tenant.
*3 How to find the Application Role Id of a Permission API Scope.
For each Permission API such as Microsoft Graph API, there are Application Role. No matter what tenant you are in, the id of them is always the same. The below example gets the Application Scope of Directory.Read.All from the Permission API Microsoft Graph.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*4 How to find the oAuth2Permission Id of a Permission API Scope.
For each Permission API such as Microsoft Graph API, there are oauth2Permissions. No matter what tenant you are in, the id of them is always the same. The below example gets the Delegation Scope of Directory.Read.All from the Permission API Microsoft Graph.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In my past two blogs I have shown you how to obtain all the users policies and output to a csv file, and how to create a new policy. In this blog post, I’m going to show you a couple of ways of setting users to new policies.
Change policy for an individual user
Change policy for a group of users
Change policies for group of users using a csv file
Change policy for an individual user
In this example I’m going to stick to just the messaging policy.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
It can take a while before the change is reflected in the Teams Administration.
Change policy for a group of users
There are a couple of ways you can do this. If you have your people data filled in correctly, such as Department, Office, City etc, you could assign all the people from one of these areas to a given policy. For example the following script grabs everyone from the sales department and assign them the sales policy for messaging:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Alternatively you could do it based on a AD Group, you first need to connect to grab the members of the given group, I would connect with AZ cli.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
Change policies for a group of users using a csv file
In a previous blog post, I showed you how to obtain all the valid users from the tenant with their policies. We are going to use the csv it produces to change users policies.
The screen shot below shows my users with licenses, originally when I ran my script I only had DisplayName, UserPrincipalName and SipAddress showing, everything else was blank because my users were all in the global policies.
I have now filled in the CSV file with either SalesPolicy, HRPolicy, or NoGiphyOrStickersMessagePolicy. Left my account as global. These policy have already been created in my environment, help with doing that can be found in this blog post.
The following script requires your MS Teams Administrator username, and the path to the csv file. It loops through each item and then sets the polices for each user.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
If you have a lot of users to update, the session might timeout. On line 52 – 57 there is a check to see if the session has timed out and then gets you to reauthenticate. If anyone else knows a better way to do this, please add a comment below, or get in touch.
In my previous blog I showed you how to obtain all the policies for users. This blog post, I’m going to show an example of how to create a new Policy for Meeting.
The script I’m providing below is a simple script that will:
Name the policy – Line 1
Import the SkypeOnlineConnector module – Line 4-7
Create the policy if it doesn’t exist – Lines 9 – 12
Set the properties of the Policy – Lines 15 – 31
This policy that is being created will prevent users of this policy creating Giphy, Memes or Stickers in their chats. (Basically, taking the fun out of teams. [insert wicked laugh])
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters
As you can see from the above screen shot, I now have a policy called NoGiphyOrStickersMessagePolicy, and below is a screen shot of the settings within Teams Administration.
My previous blog showed you how to get 13 different policies for each user. Below are the links to those PowerShell Set policy commands with the different parameters.
With everyone working at home at the moment, you might need to grab a report of the User Policies for MS Teams. These few steps will show you how to grab all the users and display in a csv file.
Installing the SkypeOnlineConnector and creating a session
Once you have installed the PowerShell module you will need to import the module and create a PowerShell session.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
You will be prompted for MS Teams administrator username and password, you can pass your credentials in with the -credential parameter at the end of New-CsOnlineSession, however this doesn’t work with MFA.
Once connected you will be able to grab all users using the Get-CsOnlineUser cmdlet, or grab one user by providing the users Identity.
If you call the above for a single user you will see that there are loads of policies that can be set. Not all are MS Teams related. The ones I will be focusing on are the same 12 policies you see when you view Assigned policies for a user in Teams Administration.
These policies are named slightly different in the results compared to the display name shown above.
Display Name
Policy Name
Meeting policy
TeamsMeetingPolicy
Messaging policy
TeamsMessagingPoliy
Live events policy
TeamsMeetingBroadcastPolicy
App permission policy
TeamsAppPermissionPolicy
App setup policy
TeamsAppSetupPolicy
Call park policy
TeamsCallParkPolicy
Calling policy
TeamsCallingPolicy
Caller ID policy
CallerIdPolicy
Teams policy
TeamsChannelsPolicy
Emergency calling policy
TeamsEmergencyCallingPolicy
Emergency call routing policy
TeamsEmergencyCallRoutingPolicy
Dial plan
TenantDialPlan
Teams Upgrade
TeamsUpgradePolicy
The following script will grab all users and their current policy for the above polices, with the provided path it will output to csv file.
Please note: Anything that is set to Global policy will be blank.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In a later blog, I will be using the csv file to update user policies.
Bonus: If you want to get all users for just a single policy you can perform a filter on the Get-CsOnlineUser
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This blog post has all come about as the client I was working for was having problems sharing documents in SharePoint with some external users. It turned out that the user was already in Azure AD as a Contact which is part of Exchange. This meant when an internal person attempted to share/Invite into SharePoint/MSTeams it all appeared to work correctly for the external user, but sometimes it didn’t. When looking at external users through the Admin portal, this external user was showing, but their email address was blank. After speaking with Microsoft, it turns out, because the email address was already found within the tenancy, it creates a unique violation when adding the external user to the Active Directory.
I have been working with Microsoft support regarding this, and the resolution was that this is as design!!??! Only by feeding back on the Office 365 uservoice this issue “might” looked at and fixed. See resolution notes below:
Symptom:
When you invite external users who exist as contacts in your environment, their email does not get populated in their guest user ID which results in them not being able to login to your environment and access the shared data. Cause:
The issue is coming from a conflict caused by the email address which is already populated for the mail contact. Resolution:
This is behavior by design as all objects in Azure AD have to be unique.
You cannot have 2 objects with the same email address.
When you invite one of your contacts to your content in O365, it actually creates a completely new guest user object in your environment and since the email address which is supposed to be populated in the email attribute is already in use by the contact, the email address does not get populated.
The only way to resolve this issue at the moment is to eliminate any conflicts that are in place, by removing the conflicting email contact and re-invite the user to your content. More information:
The best thing I can offer to you is the following:
Please go to our UserVoice portal where other people are facing the same behavior and up-vote it, comment and have the whole IT department do the same as well.
This led me to working on a process and script that would remove the users from everywhere.
Locations to remove the External User from:
Contacts
Azure AD Guest Users
Azure AD Deleted Users
All SharePoint Sites
All SharePoint Hidden User lists
SharePoint User Profile
Contacts
To remove the External User from the contacts you will need to use the MSOL PowerShell module.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Or you can manually do this by going to admin.microsoft.com and under Users -> Contacts select the user and click Delete contacts.
Azure AD
To remove the External User from Azure AD you will still require using the MSOL PowerShell module. In fact, this script and the above script could be merged.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
To do this manually, in admin.microsoft.com under Users -> Guest Users, select the user and click delete.
Then go into Users -> Deleted users and remove them from there.
Remove from SharePoint
To remove from SharePoint, if you have a large tenancy and you don’t know all the places where the external user could have been shared with, then you will have to use the following script. This script will remove the external user from the SharePoint Site, ensure that they are removed from the User Information list, and then lastly it will clear the person from the SharePoint User Profile.
I discovered that if I didn’t remove them from the User Profile, when attempted to reshare a document with that user, the people picker would grab the internal userprincipalname (<ExternalUserEmail>#EXT#@<Tenant>.onmicrosoft.com) as the email address and then prevent me clicking the Sharing button. This is because the people picker uses Graph API /Me/People and grabs the value from there. Once removed from everywhere, including the User Profile this no longer happens.
The following script uses SPO PowerShell Module and you will need to connect first using Connect-SPOService. The account that you use, needs to be a SharePoint Global Administrator.
The script checks if it can find the ExternalUser, and if it can remove the user using Remove-SPOExternalUser.
Then it loops through every site collection and looks for the user using Get-SPOUser with the internal userprincipalname. If found it removes the user using Remove-SPOUser. Once it has looped through all SharePoint sites, it then checks the SharePoint User Profile and removes the user from UserProfile Remove-SPOUserProfile. This command will remove a user from the UserProfile if they in the “Active Profiles” or the “Profiles Missing from Import”
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loops through the SharePoint sites of the tenant, looking for the external user and removing them.
You need to have already connected to the Tenant as a SharePoint Global Adminstrator using Connect-SPOService -url:https://<tenant>-admin.sharepoint.com
If the plan is to add the external person back into your tenant, once the script has run, you will need to wait at least a few hours (maybe leave it for a day to be sure) to ensure all back end processes of Microsoft have completed.
When you share a document/folder with the external user they will get the invited link and enter a code experience, this way they do not turn up inside you Azure AD. However, if you share a site with them, or add them to a MS Teams, they will appear in your Azure AD correctly.
I’ve had a script for a while that allows you to view all the items in the Recycle Bin for a Site Collection and prints out to a CSV file. Recently the environment I’ve been running this in has been throwing an error saying;
“The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator“.
Getting all items out of the recycle bin.
Originally, I used the PNP Powershell command Get-PnPRecycleBinItem and it was only when I did a Google search for this issue, I found that other people were also having this problem. The PnP team have solved this issue now by adding -RowLimit parameter. If you set the RowLimit high enough you can return all items, as internally, it seems to implement a paging mechanism.
I now use the below script to export the result to a CSV file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Once I have the CSV file, I’m able to filter further in excel and save back to CSV to use to either Restore / Delete the items out of the recycle bin.
Restoring Deleted Items using a csv file.
It seemed that now that I can use RowLimit with Get-PnPRecycleBinItem I should be able to call Restore-PnpRecycleBinItem to restore the item. However, this isn’t the case. Even just passing the Identity of one item within the Recycle Bin, you get the same error message.
“The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator“.
There is no RowLimit option on the Restore-PnpRecycleBinItem. The code must internally make a call to get all RecycleBin Items first without using RowLimit. Interestingly though, a user could go to a recycle bin, see items, and restore them if they wanted to. By looking through the network traffic, I was able to see that the GUI uses the following API to Restore Items.
POST /_api/site/RecycleBin/RestoreByIds
Passing in the following JSON body.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The trouble with using REST API you need an accessToken. Using the Invoke-PnPSPRestMethod it automatically provides the AccessToken in the call.
This is what I do in the below code. Loop through every item in a CSV file to restore, and call “/_api/sites/RecycleBin/RestoreByIds” using Invoke-PnPSPRestMethod.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Again, I was able to do this in the GUI, and looking at the network traffic there is an API to delete the items.
POST /_api/site/RecycleBin/DeleteByIds
The JSON body is same format as the RestoreByIds, where it passes in one or many Ids.
The code below is almost identical to the Restore-RecycleBinItems.ps1. Passing in a CSV file with the IDs of files to delete permanently.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Dive into the code for O365 Audit Log webhooks to see how it works – (This Post)
The previous blog post showed how to get you up and running with O365 Audit logs and webhooks. In this blog post I’m going to show and explain parts of the code that ties everything together.
PowerShell to initialize the Webhook to the Audit logs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
From inside the PowerShell folder (.\O365AuditWebhook\PowerShell) there is a PowerShell file called Set-AuditLogs.ps1 This PowerShell file Starts a subscription to the given Audit Content Type. This is done by calling:
The above call is a POST call and uses the ClientID and Secret to authenticate against the tenant. The body is a Json object
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
authId – Optional string that will be included as the WebHook-AuthID header in notifiations sent to the webhook as a means of identifying and authorizing the source of the request to the webhook
expiration – Optional datetime that indicates the datatime after which notifications should no longer be sent to the webhook. By leaving it empty, indicates the subscription will be active for the next 180 days.
address – Required HTTPS endpoint that can receive notifications. A test message will be sent to the webhook to validate the webhook before creating the subscription.
When the /start operation is called, the webhook URL specified in the address will be sent a validation notification to validate that an active listener can accept and process notifications.
The Azure Function AuditWebhook found in the O365AuditWebhook.cs file has two parts to it.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The first part, as shown above, handles the validation. It looks for a validation code within the content, and if found it response back with a 200 status (OK) and includes the validation code.
If an OK is not received back, then the webhook will not be added and the subscription will remain unchanged.
The second part of the AuditWebhook Azure function is explained in the next section.
Webhook handling O365 notifications
After the initial validation, notifications will be sent to the webhook as the content logs become available.
From the first part of the AuditWebHook Azure Function, notifications do not have the validationCode, this allows us to determine that notifications have been sent, instead of a new subscription.
The content of these notifications contains an array of one or more JSON objects that represent the available content blobs.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
On line 5 of the above code, show where I handle the content of deserialize json object (notifications) to a list of AuditContentEntity.
The notification/AuditContentEntity contains the following:
tenantId – The GUID of the tenant to which the content belongs
clientId – The GUID of your application that created the subscription
contentType – Indicates the content type
contentId – An opaque string that uniquely identifies the content
contentUri – The URL to use when retrieving the content
contentCreated – The datetime when the content was made available
contentExpiration – The datetime after which the content will no longer be available for retrieval.
At this point you do not have any log information, you just have a collection of contentUri which when called will provide you with the logs. To ensure that the webhook response quickly so that it can continue to handle incoming requests, we place the contentUri, contentType, and TenantId onto an Azure Storage Queue. This allows a different Azure function to handle getting the actual logs.
Lines 9-16 will set up the storage queue if it doesn’t exist.
Lines 19-26 prepares my queue object and serialize it to a json string.
Line 28 adds the message to the Azure Storage Queue.
Once all notifications/AuditContentEntity have been processed, a 200 status (OK) is passed back. The subscription that calls our webhook is waiting for an OK response. If it encounters failure, it has a built in retry mechanism that will exponentially increase the time between retries. If the subscription continues to receive failure response, the subscription can disable the webhook and stop sending notifications. The subscription will need to be started again to re-enable the disabled webook.
Processing the Storage Queue AuditContentUri
As items are put on the Storage Queue the Azure Function AuditContentUri found in the O365AuditWebhook.cs file fires.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
First you need an authorization token to read the audit logs, we do this with AcquireTokenForApplication method. This uses the Tenant Name, ClientId and Secret that is stored within your Azure configurations. See ‘How to acquire token for application?’ below.
It grabs the ContentUri and then goes into a do loop. This is because the logs that come back, if it is a very busy tenant, not all the logs will be returned, and there will be a NextPageUri value in the header of the response to allow you to obtain the next page of logs.
Line 7 – This adds your tenantID to the end of the URI as a PublisherIdentifier. This parameter is used for throttling the request rate. Make sure this parameter is specified in all issued requests to get a dedicated quota. All requests received without this parameter will share the same quota. The IF statement ensures it is added to the end of the URI correctly.
Line 9 – This calls the ContentUri and gets a results and request headers. You can see the file .\O365AuditWebHook\AuditWebHook\Utilities\RestAPI.cs The Method GetRestDataAsync is very similar to the GetRestData call you find within PNP Core code. Creates a HttpWebRequest, passing in Authorization Token, and calling the ContentUri. Only difference in my code is that I’m grabbing the response.Headers to find out if there are additional logs, and passes them back with the results.
Line 10 – This parse the results into a JArray. (Json Array object). Here you can manipulate what comes back. For example, instead of grabbing all results and then displaying them out, you can query the results for a particular log type.
In the example code below, this would be using the Audit.General logs, and it will grab any logs that are of RecordType 25 (Indicates Microsoft Teams event) where the operation is creating a new channel, and the Channel type is Private. I then convert the JArray to an object list of AuditGeneralEntity.
Line 17 – If there are any additional pages, then this will return a value, and the loop will loop until no more pages are found.
How to acquire token for application?
In the previous section, I called a method AcquireTokenForApplication. This is a helper class and method that I use quite often, when I need to obtain an AccessToken. You can find this in the repo at .\O365AuditWebHook\AuditWebHook\Utilities\AuthenticationHelper.cs. This solution has a cut down version of the helper class I use. It is cut down as it just gets an access token for Audit Logs using AppId and Secret.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Above is a snippet, as you can see it is wrapped in a retry method in case there is throttling.
PowerShell to stop the Audit logs
Within the PowerShell folder I have also included a file called Remove-AuditLogs.ps1
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Once the subscription is stopped, no notifications will be sent to your webhook, and you will not be able to retrieve available content. Please note, if you decide to start the subscription again later using the Set-AuditLogs.ps1 you will not receive any content that was available between the stop and start time of the subscription.
Â
This is quite a heavy post; I hope it has helped you in some way. It is just a starter, as you will probably want to do something with the logs instead of just writing them out to the Azure Logs. Maybe capturing a given process to then implement some logic to react. You might also want to put different Audit content types ContentUri onto different Azure Storage queue, so that different Azure Functions can process the ContentUri.
In this blog post I’m going to show you how to get the O365 Audit logs using WebHooks. The full code can be found at my Github repo https://github.com/pmatthews05/O365AuditWebHook. My post will show you how to set up with screenshots and the expected results. In my next blog post I will dive into the important parts of the code to get this Audit WebHook connected and working.
Set up – Walkthrough
Creating an App Only Token
Once you have downloaded a copy from my Repo you will need to set up your environment. First thing we are going to do is create an App Only Token that will be able to read the Audit Logs.
Click Grant Admin Consent for [tenant] and accept the permissions.
Click on Certificates & Secrets
Click New Client Secret
Description: Audit Web Hook
Expires: Never
Take a copy of the Secret value
Setting up Azure
You will need to set up your Azure Environment, this will consist of the following:
Resource Group
Azure Function V1
Applications Insights
Storage Account
I like to automate where I can, also it saves me creating loads of screenshots which are probably all out of date after 2 months. I have written an Az CLI PowerShell script that will create the above for you in your Azure Environment. In the next blog post I will explain the code.
Using a PowerShell window – Sign into your Azure Environment using ‘az login’
If you have multiple subscriptions, ensure you are pointing to the correct subscription ‘az account set –subscription [SubscriptionName]‘
Change the directory to .\O365AuditWebhook\powershell
Run the following: ‘.\Install-AzureEnvironment.ps1 -Environment “[Environment]” -Name:”AuditWebHook”‘ replacing the [Environment] with your tenant name. For example, I’ve used cfcodedev.
Once the script has run, you will have the basic template Azure resources you need within the Resource group named [Environment]-AuditWebHook
Deploying Azure Function from Visual Studio 2019
Firstly, you don’t have to deploy this way. If you prefer to use Visual Studio code, create an AZ install script or manually deploy using Kudu, that is your choice, and all are valid. My choice of doing this is simplicity for screen shots and steps.
Open the solution using Visual Studio Code 2019
Right click on the project AuditWebHook and select Publish
From the Pick a publish target dialog (click Start if you are not seeing a dialog), and under Azure Functions Consumption Plan click Select Existing, and select Create Profile.
Sign into your account if you need to, then pick your subscription, resource group, and then you can either search, or just pick the Azure Function. Click OK.
This takes you back to the Summary page. Under Actions click Edit Azure App Service settings
The Application Settings dialog will show you the values Local and what is found within Azure Function in the cloud. You will need to update the Remote value for the following:
FUNCTIONS_EXTENSION_VERSION: ~1
You will need to add the following Settings, by clicking on Add Setting creating the setting name, and put the value in afterwards. Repeat for each setting below.
Tenant: [Name of your Tenant, do not include .onmicrosoft.com]
ClientId: [Client ID created in step ‘Creating an App Only Token’ earlier]
AppSecret: [Secret Value created in step ‘Creating an App Only Token’ earlier]
Click OK
Back on the Publish screen, click the Publish button. This will push the code to your environment, with the correct Application Settings.
By going to your Azure Function at portal.azure.com, you will see 2 Azure Functions
Then clicking on Configuration, it will take you to the Application settings page, click Show Values and you will see your values.
At this point you just have the Azure function as a Webhook in place. Next steps are to tie the O365 Audit log to the WebHook.
Connecting O365 Audit Logs to your webhook
The last step is tying the Audit logs to your webhook. The webhook can be used for the different Audit logs. There are 5 different types of logs.
Audit.AzureActiveDirectory
Audit.Exchange
Audit.SharePoint
Audit.General
DLP.All -Note: DLP sensitive data is only available in the activity feed API to users that have been granted “Read DLP Sensitive Data” permission.
I have written a PowerShell script for you that will register the webhook for you. You will find this in the repo.
Open PowerShell
Change the directory to .\O365AuditWebhook\powershell
Run the following PowerShell script (Run on one line), change the parameters to match your environment. I’ve picked Audit.SharePoint, but you can use any listed above, and run the PowerShell script multiple times to connect all logs to the webhook.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
“When a subscription is created, it can take up to 12 hours for the first content blobs to become available for that subscription. The content blobs are created by collecting and aggregating actions and events across multiple servers and datacenters. As a result of this distributed process, the actions and events contained in the content blobs will not necessarily appear in the order in which they occurred. One content blob can contain actions and events that occurred prior to the actions and events contained in an earlier content blob. We are working to decrease the latency between the occurrence of actions and events and their availability within a content blob, but we can’t guarantee that they appear sequentially.“
If you are using a Development environment – like myself – and setup the Audit.SharePoint content type then I suggest you go into SharePoint, and start using SharePoint. Just so the logs start to fill.
Please note, it can take up to 30 minutes or up to 24 hours after an event occurs for the corresponding audit log entry to be displayed in the search results, depending on the service of Office 365. See the table at the bottom of this section Search the audit log in security and compliance – Before you begin
Viewing the AuditWebHook azure function, you will see that it has fired more times since your initial setup.
If you look at your latest call, (note: logs can display out of order in azure functions) you will see that it attempts to find the validation code, which is what it needs to set up the webhook. When it is unable to find the validation code, the code assumes that content contains log information. It grabs the URI of the log that has been created and then it adds it to our Azure Storage Queue for our other azure function to process. Depending on how busy your environment is, this request could hold multiple URL’s to logs. A webhook has to respond quickly back to the calling code with a 200 status code. Therefore we are adding the URI’s of the logs directly to a Storage Queue to allow a different process to interrogate the logs.
The second Azure Function (AuditContentUri) will fire every time an item lands on the Storage Queue. This will grab the information from within the log file by calling the URI.
If we select one of the calls and view the logs of that Azure Function call, every entry within that Audit log file URI will be displayed in a JSON format. Clicking on a row in the logs, will display the full details of the line. At this point in the code, would be where you process the line and do whatever you need to do with the Audit log. I’m just printing it out to the Azure Function Logs.
Remove O365 Audit Logs from your webhook
To remove the webhook from the Audit log just run the following PowerShell script. You will find this in the repo.
Open PowerShell
Change the directory to .\O365AuditWebhook\powershell
Run the following PowerShell script (Run on one line), change the parameters to match your environment. I’ve picked Audit.SharePoint, but you can use any listed above, and run the PowerShell script multiple times to remove all logs to the webhook.
The below codes login with the ClientID and Secret and stops the subscription of the given ContentType audit.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Hopefully, if you have followed this correctly, (and I have written decent enough instructions for you), you should have a basic Audit Log Webhook working in your environment. This isn’t anywhere near production ready code, but it gives you an idea where to start. In my next blog post I will be going though parts of the code, to explain how it all fits together.
SharePoint and other geeky stuff 