Adding to Property Bag and Index using PowerShell in SharePoint Online


The other day, I very lazily, used SharePoint Designer to add a few items to the property bag. Soon as I done it, someone then said they wanted to search upon the values inside the property bag. Unfortunately this isn’t something that can be done via SharePoint Designer. After spending 5 minutes or so searching online for a pre-made PowerShell code, I could only find the solution for On-Prem.

The only piece of code I found to do this to SharePoint Online was through the Office Developer PNP CSOM code. I really didn’t want to create a Visual Studio project just to use the PNP Core. (See http://dev.office.com/patterns-and-practices for more information Office Dev PNP, there is so much information and cool videos there, there really is no need to re-hash anything in my own blog.) So I decided to just create PowerShell script which is pretty close to being like for like copy of the PNP Core method:

Web.AddIndexedPropertyBagKey(this Web web, string key)

Before I go into the code, I will explain how SharePoint knows which property bag items need to be indexed for search. Just by adding the item doesn’t make it visible to search, it has to be added to another property bag item called “vti_indexedpropertykeys“. The value of this property bag item is a pipe delimited Base64String. (Example: RABhAHUAZwBoAHQAZQByAA==|UwBvAG4A|cwBvAG4A| ). As you can see, it’s not just as simple as adding the string name to the vti_indexedpropertykeys value.

To encode a single value in PowerShell the following line of code works:

[System.Convert]::ToBase64String([System.Text.Encoding]::Unicode.GetBytes($value))

To decode a single value from Base64String to text the following PowerShell code works:

[System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($EncodedValue))

The below code will ask you for the Web Url, your username, password (Secure string), propertybag key name and value. The code will first add or update the value in the property bag, and if it hasn’t been added to the index it will add it.

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client") | Out-Null
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")| Out-Null
$indexPropertyKeyConst = "vti_indexedpropertykeys"

function GetIndexedPropertyBagKeys($ctx)
{
  $results = @()
  $web = $ctx.Web;
  $ctx.Load($web.AllProperties)

  try
  {
  $ctx.ExecuteQuery();
  }
  catch{
   Write-host "Error accessing property bag " $_.Exception.Message -ForegroundColor Red
   exit 1
  }

  $indexPropertyBase64 = $web.AllProperties[$indexPropertyKeyConst];
  $separator = "|"
  $option = [System.StringSplitOptions]::RemoveEmptyEntries

  if(![string]::IsNullOrEmpty($indexPropertyBase64))
  {
    $resultsBase64 = $indexPropertyBase64.Split($separator, $option)

    foreach($r in $resultsBase64)
    {
     $results += [System.Text.Encoding]::Unicode.GetString([System.Convert]::FromBase64String($r))
    }
  }
    #comma not a mistake, required to ensure correct type is returned on empty or single array value.
    return ,$results
}

function GetEncodedValueForSearchIndexProperty($keysArray)
{
  $encode64Keys = [String]::Empty
  foreach($key in $keysArray)
  {
   $encode64Keys += [System.Convert]::ToBase64String([System.Text.Encoding]::Unicode.GetBytes($key))
   $encode64Keys += "|"
  }

  return $encode64Keys;
}

function AddIndexedPropertyBagKey($ctx, $propertyKey)
{
  [bool]$result = $false;
  [bool]$addValue = $false;
  $keys = GetIndexedPropertyBagKeys($ctx)

  if($keys -cnotcontains $propertyKey)
  {
     $addValue = $true;
  }

 if($addValue)
 {
   try
   {
   $keys += $propertyKey
   $keysBase64String = GetEncodedValueForSearchIndexProperty($keys)
   $ctx.Web.AllProperties[$indexPropertyKeyConst] = $keysBase64String
   $ctx.Web.Update()
   $ctx.ExecuteQuery()
   $result = $true
   }
   catch
   {
    Write-host "Error adding $propertyKey to index. " $_.Exception.Message -ForegroundColor Red
    exit 1
   }
 }

  return $result
}

function CreateUpdatePropertyBag($ctx, $propertyKey, $propValue)
{
  $web = $ctx.Web;
  $ctx.Load($web.AllProperties)
  try
  {
    $ctx.ExecuteQuery();
    $web.AllProperties[$propertyKey] = $propValue;
    $web.Update();
    $ctx.ExecuteQuery();
  }
  catch{
   Write-host "Error adding $propertyKey to property bag " $_.Exception.Message -ForegroundColor Red
   exit 1
  }
}

$webUrl = Read-Host -Prompt "Enter the WebUrl"
$username = Read-Host -Prompt "Enter your Email login"
$password = Read-Host -Prompt "Password for $username" -AsSecureString
$propKeyToIndex = Read-Host -Prompt "Enter the property key name to index"
$propValue = Read-Host -Prompt "Enter the $propKeyToIndex value"

$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($webUrl)
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)

CreateUpdatePropertyBag $ctx $propKeyToIndex $propValue;
AddIndexedPropertyBagKey $ctx $propKeyToIndex;

Write-Host "Complete" -ForegroundColor Green

You can download the source code directly from my OneDrive.

Thank you to my colleague Paul Perry with his help around PowerShell arrays.

Setting the landing page for a wiki library


When you create a new wiki library, just by clicking on the library takes you directly to the home page of the wiki. Not to the list of pages like you would get if you clicked on the document library, or even the Site Pages library (Which is also a wiki library). In wiki libraries, this happens because there is a value in the property bag of the list called “vti_welcomepage”. This value is set to a page within the library, typically “home.aspx”. This can be change, unfortunately it can only be changed in code. I haven’t found a way to do this in the GUI.

UPDATE: As pointed out to me by Ronnie Holm (Thanks), you shouldn’t modify the vti_welcomepage value in the property bag. There is a method on the RootFolder called “WelcomePage” that you can get or set this value. I wasn’t aware of this method originally. When changing the WelcomePage via this method it automatically changes vti_welcomepage in the property bag.

The below PowerShell will be able to make the change for you, just update the page name to the name you wish to use.

 
$UserName = Read-Host -Prompt "UserName" 
$Password = Read-Host -Prompt "Password" -AsSecureString 
$Url = "https://mySharePoint365.sharepoint.com/sites/TestSite" 

Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll" 
Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll" 

$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($Url) 
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $Password) 
$ctx.ExecuteQuery(); 

$list = $ctx.Web.Lists.GetByTitle("My Wiki") 
$list.RootFolder.WelcomePage = "What you should know about wiki's.aspx" 
$list.RootFolder.Update(); 
$ctx.ExecuteQuery(); 

$ctx.Load($list.RootFolder)
$ctx.ExecuteQuery();

Write-host("Current Welcome Page : " + $list.RootFolder.WelcomePage);
write-host("Complete");

Now when you click on your wiki library or navigate to the URL of the wiki library, it will redirect and display the wiki page you have set as the home page.

Updated Pages link in Wiki Pages missing



When creating a Wiki page library, every page you create/modify appears in the Updated Pages section above your quick launch navigation. Now, out of the box a team site has site pages. Site pages is a Wiki page library, and therefore it make sense just to use the Site pages as your wiki library. However, when you are using the Site Pages library, the Updated Pages link is missing.

There are lots of blog post out there showing you how to hide the Updated Pages link. Easiest way is using a bit of css :

.ms-quicklaunchouter{
 display:none;
}

.ms-core-listMenu-separatorLine{
 border-style:none;
}

However, I was struggling to find anywhere that showed you how show it. (Reversing the CSS didn’t make any difference, as the Updated Pages link wasn’t in the HTML in the first place.

Showing the Updated Pages in SitePages library.

What’s in a URL? Quite a lot it seems in this case. There must be some server side code that states, if the URL is /SitePages then don’t display Updated Pages. So the solution is to change the URL of the SitePages list. (Or don’t use the Site pages list in the first place, however, we already have content in our Site Pages list).

UPDATE: Do not do the below, if you do, when you go to “add a page”, SharePoint will ask you to create the default Wiki Library which is Site Pages. Therefore, it would be best to just create a new Wiki Library with the name you want to give, and move the existing wiki pages from Site Pages to your new Wiki Library. As long as you are on a page within the new Wiki Library, when you select “Add a Page” from the Cog, the page will be added to the Wiki Library.

There is 3 ways you could change the url.

  1. Using SharePoint designer. Find the SitePages library in the All Files section. Right click it and rename it.
  2. Using File Explorer. Find a list that you can open with Explorer (as Site Pages doesn’t allow you to) and then navigate to your site structure. Rename the Site Pages Folder. (I don’t recommend doing this way)
  3. PowerShell. I would recommend this way over the other two.
$UserName = Read-Host -Prompt "UserName"
$Password = Read-Host -Prompt "Password" -AsSecureString

$Url = "https://mysharepoint365Url.sharepoint.com/sites/TestSite"

Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.dll"
Add-Type -Path "C:\Program Files\Common Files\microsoft shared\Web Server Extensions\16\ISAPI\Microsoft.SharePoint.Client.Runtime.dll"

$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($Url)
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName, $Password)
$ctx.ExecuteQuery();

$list = $ctx.Web.Lists.GetByTitle("Site Pages")
$list.RootFolder.MoveTo("NewLocation");
$ctx.ExecuteQuery();

write-host("Complete");

As soon as the URL has changed, the Updated Pages link will show.

NOTE: The only time it will never show up, is if you make the page your home page to your site. It doesn’t matter what the URL is, it will not show.

UPDATE: Do not move the Site Pages library, if you do, when you go to “add a page”, SharePoint will ask you to create the default Wiki Library which is Site Pages. Therefore, it would be best to just create a new Wiki Library with the name you want to give, and move the existing wiki pages from Site Pages to your new Wiki Library. As long as you are on a page within the new Wiki Library, when you select “Add a Page” from the Cog, the page will be added to the Wiki Library.

PowerShell – Loop through Sites, Content Types, Lists.


Many times before I have had to write a bit of PowerShell to loop through a SharePoint site fixing something up. When I start writing this script I normally start from scratch each time. Basically I need to perform the same basic looping functionality, and then either at the List level, content type level maybe even at the item level I need to perform some change.

For example, I might need to delete a certain field from every list of type event, or I need to set the default value on a column, or perhaps you just want to loop through the site to see what is where. Whatever the reason is, if the change needs to be made in many places, I will need to loop through the environment.

The order typically I loop through would be:

  • Web
  • Content Type
  • Lists
  • Content Types in Lists
  • Fields in Content Types
  • Items

Depending on what I was trying to achieve, I would either add, or remove extra sections going forward.

The code below is the basic template I would use to get me going.

[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client")
[System.Reflection.Assembly]::LoadWithPartialName("Microsoft.SharePoint.Client.Runtime")

#Functions
#Process the web Site, checking the Content Type at the web first before checking the Content Types at the list level.

function processWeb($web){
 #web
 $ctx.load($web)
 #Content Type
 $cts = $web.ContentTypes
 $ctx.load($cts)
 #Lists
 $lists = $web.Lists
 $ctx.Load($lists)
 $ctx.ExecuteQuery()
 #Display Web 
 write-host "Web Name:" $web.Title 
 processContentTypes($cts)
 processLists($lists)
}

#Loop through each content type
function processContentTypes($cts)
{
  foreach($ct in $cts)
  {
    Write-Host "Content Type:" $ct.Name             
    $ctx.Load($ct.FieldLinks);
    $ctx.ExecuteQuery();

    processFieldsLinks($ct.FieldLinks)
  }
}

#Loop through fields
function processFieldsLinks($fieldsLinks)
{
  foreach($f in $ct.FieldLinks)
  {
      Write-Host "Field " $f.Name 
  }
}

#Loop through each item
function processItems($items)
{
 foreach($item in $items)
 {
  if($item.FieldValues["Title"] -eq $null)
  {
   write-host "Item FileLeafRef: " $item.FieldValues["FileLeafRef"]
  }
  else
  {
   write-host "Item Title: " $item.FieldValues["Title"]
  }
 }
}

#Loop through each list in the site
function processLists($lists){
 foreach($list in $lists)
 {
  $ctx.load($list)
  $ctx.load($list.ContentTypes)
  $query = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery(10000, 'UniqueId','ID','Created','Modified','FileLeafRef','Title')

  $listItems = $list.GetItems($query);
  $ctx.load($listItems)
  $ctx.ExecuteQuery();

  write-host "BaseTemplate: " $list.BaseTemplate " List: "$list.Title
  processContentTypes($list.ContentTypes)
  processItems($listItems)
 }
}

#Get the URL, and an account to log in with. (Needs permissions across the sites to make changes)
$webUrl = Read-Host -Prompt "Enter the URL of your Root site" 
$username = Read-Host -Prompt "Enter your Email login"
$password = Read-Host -Prompt "Password for $username" –AsSecureString

#Create Context and load sites.
$ctx = New-Object Microsoft.SharePoint.Client.ClientContext($webUrl)
$ctx.Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($username, $password)

$rootWeb = $ctx.Web
$childWebs = $rootWeb.Webs
$ctx.Load($rootWeb)
$ctx.Load($childWebs)
$ctx.ExecuteQuery();

#Process the rootweb
processWeb($rootWeb)

#process the child webs
foreach($childWeb in $childWebs)
{
 processWeb($childWeb);
}

#complete
Write-Host "******Completed*********" -ForegroundColor Green

Just to give an example of how I might adjust this, if I wanted to only loop through lists that are Site Pages, with BaseTemplate of 119, I would change the processLists() function to only process the list if the BaseTemplate is 119.

#Loop through each list in the site
function processLists($lists){
 foreach($list in $lists)
 {
   $ctx.load($list)
   $ctx.ExecuteQuery()

   if($list.BaseTemplate -eq 119)
   {
     $ctx.load($list.ContentTypes)
     $query = [Microsoft.SharePoint.Client.CamlQuery]::CreateAllItemsQuery(10000, 'UniqueId','ID','Created','Modified','FileLeafRef','Title')
     $listItems = $list.GetItems($query);
     $ctx.load($listItems)
     $ctx.ExecuteQuery();

     write-host "BaseTemplate: " $list.BaseTemplate " List: "$list.Title
     processContentTypes($list.ContentTypes)
     processItems($listItems)
   }
 }
}

Link to file in OneDrive: http://1drv.ms/1V5Q8j8

Basic Logging within Web Jobs


When you are a developer it’s very easy to debug code, step through it and understand exactly what is happening at different parts of the code. When the code is finally shipped to a production environment, you are virtually blind to what is happening within your code unless you put sufficient logging within the code. (Or you are lucky enough to use Visual Studio to debug production environment, something that you probably shouldn’t do).

The basic way to log within WebJobs is using Console.Writeline or Trace from the System.Diagnostics.

Console.Writeline() logs

By using the Console.Writeline you only write to the Web Job logging window. These do not display as errors, warning or information. They are just plain print out to screen.

To get to the web job logging screen head to https://portal.azure.com then select your web app. On the All settings blade, select Web Jobs, then on the Web Jobs blade it will list all the web jobs currently assigned to this web app. The logs URL is stored on the right hand side of the screen. Click this link and it will take you through to your logs. Normally the URL is similar to the following https:// <WebAppName> .scm.azurewebsites.net/azurejobs/#/jobs/ <ContinuousOrTriggered> / <WebJobname>

Trace logs

The trace logs are stored within Azure Storage Blobs or Tables, which will need to be set up and configured. (This is shown later in this blog post). They do not show up in the Web Job window as described above for the Console.Writeline(). Please note that the Trace logs are not just for Web Jobs, you can use them in your web application, and most of this information will work for Web Apps.

Within your code, all you need to do is add a using statement to System.Diagnostics. Then you have a choice of logging an information, a warning, an error, or verbose message log.

System.Diagnostics.Trace.TraceInformation("This creates an information log");
System.Diagnostics.Trace.TraceWarning("This creates a warning log");
System.Diagnostics.Trace.TraceError("This creates an error log");
System.Diagnostics.Trace.WriteLine("This creates a verbose log");

Configuring Blob and Table logging.

Unfortunately I’m unable to find any way of doing this in the new Azure Portal. Therefore for these set of instructions I’m using the old Azure Portal https://manage.windowsazure.com

First you will need to create a Storage if you don’t already have one to use. In the old portal, you can do this just by clicking the New button at the bottom of the screen, and then selecting Data Services > Storage > Quick Create

Now head to your Web App, and click on the CONFIGURE tab, and on the CONFIGURE tab, scroll down until you reach the application diagnostics section.

In the Application Diagnostics section there are 3 different places you can set up logging.

  1. File System – This is written to the Web App file system, you can access these logs from the FTP share for this Web App. Please note: Application Logging is only enable for 12 hours. (Not part of this post)
  2. Table Storage – The logs are collected in the table storage that is specified. You can access the traces from the specified table in the storage account. Please note: These logs will remain in the Storage table increasing in size until someone clears the logs.
  3. Blob Storage – The logs are collected in the blob contain that is specified. You can access the traces from the specified container in the storage account. Please note: By default, application diagnostic logs are never deleted, however you have an option to set a retention period between 1 and 99999 days.

Configuring Blob Storage

Following on from the last section, to configure the Blob Storage click the ON button of the Application Logging (Blob Storage). Once you have clicked ON you will be presented with some other values.

The logging level options are Verbose, Error, Warning and Information. By setting the level at a given status, will depend which trace appear in the logs.

  • Verbose – Shows all logs.
  • Information – Shows Information, Warnings and Errors.
  • Warnings – Shows only Warnings and Errors.
  • Errors – Shows just Errors.

Next Click on manage blob storage, and you will be presented with a dialog. Select the Storage Account you created previously, and then Create a new blob container. Give your container a name, and then click the tick button.

Next you can set the retention in days to store the logs. Click Save on the page and the Blob Storage has been set up.

Configuring Table Storage

Configuring the Table Storage isn’t much different from configuring the Blob Storage. The only difference really is you are creating a new table instead of a new blob container. Click the ON button of the Application Logging (Table Storage). Once you have clicked ON you will be presented with some other values.

The logging level options are Verbose, Error, Warning and Information. By setting the level at a given status, will depend which trace appear in the logs.

  • Verbose – Shows all logs.
  • Information – Shows Information, Warnings and Errors.
  • Warnings – Shows only Warnings and Errors.
  • Errors – Shows just Errors.

Next Click on manage table storage, and you will be presented with a dialog. Select the Storage account you created previously, and then Create a new table. Give your table a name, and then click the tick button.

How to view the application diagnostic logs.

The biggest issue with Blob and Table storage in Azure is that there isn’t a simple way within Azure to view the information stored within them. There are plenty of 3rd party tools out there, that allow you to view blob and table storage for free. Azure Storage Explorer 6 is a free Windows application that you can use. It’s available on CodePlex https://azurestorageexplorer.codeplex.com/ however as useful as it can be, it is a bit painful, as you need download blob file to view once found, or filter the table to find the logs you are looking for. Also it is a windows application viewer looking at your Azure storage. Meaning if you use multiple PC’s, you need to ensure this is installed on all PC’s you use.

I try not to suggest 3rd party apps/extensions in my blog post, however I do like the Azure Web Site Log Browser written by Amit Apple. Amit Apple blog site http://blog.amitapple.com/ seems to just live and breathe Web Jobs, and I have learnt many things about Web Jobs from his blog. To install his extension you can do this directly from the new Portal or going to Site Extensions within you website scm. I will show both ways below how to do this.

Installing Azure Web Site Log Browser via Azure Portal

  • Head to your Web App in the Azure new portal. https://portal.azure.com
  • In the Web-App blade, along the menu buttons, select Tools.
  • On the Tools blade, select Extensions from the bottom of the Develop section.
  • Then in the Installed web app extensions blade, click Add.

  • On the Add Web Extension blade, you can choose an extension from the Choose web app extension blade. At time of writing this, Azure Web Site logs Browser is the 5th extension from the list. Click Azure Web Site Logs Browser.

  • Then click the OK button to accept terms. Then lastly, click the OK button to add the extension to your site. It takes a few moments to install into your Web App. It will only be installed on this Web Application.

  • Lastly on the Web App blade Click the restart button. This will ensure the Azure Web Site Logs Browser has fully installed.
  • To ensure it has worked. Click on the extension within the Installed web app extensions blade, and then click on Browse button on the Azure Web Site Logs Browser blade.
  • This will take you to https://<YourWebApp>.scm.azurewebsites.net/websitelogs/#

Installing Azure Web Site Logs Browser via SCM website.

  • Click on the Gallery tab, this will display all available site extensions.

  • As you can see from the screen shot, Azure Web Site Logs Browser is the first item in the second row at the time of writing this. You could search for it, by typing Azure Web Site Logs in search box.
  • Click the + button and this will install Azure Web Site Logs Browser to your Web App.
  • After it has successfully install, click the Restart Site button in the top right hand corner of the screen. This will ensure everything has been loaded up correctly.
  • By clicking on the Installed tab of Site extensions and then clicking the play button.
  • This will take you to https://<YourWebApp>.scm.azurewebsites.net/websitelogs/#

Viewing Application Logs – Blob Storage.

In my web job which I have set up logs for both Blob and Table reporting, I have a simple loop that displays some trace informations, warnings and errors, with some sleep threading in between to make sure the webjob lasts more than 10 seconds.

Trace.WriteLine("We are about to loop."); 

for (int i = 0; i < 20; i++)
{
   Trace.TraceInformation(String.Format("This is an information message brought to you by BasicLogging Web Job, looping for the {0} time", i)); // Write an information message
   Thread.Sleep(2000);

   Trace.TraceWarning("I'm warning you, don't get me angry.");
   Thread.Sleep(2000);

   Trace.TraceError("That's it, I'm annoyed now!");
   Thread.Sleep(2000);

   Trace.TraceInformation("Ok, I'm sorry, I got a little mad I'm OK now.");
   Thread.Sleep(4000);
}
Trace.WriteLine("End of the loop.");

I have then run this web job once. Now I wish to view this in the Blob storage within my site extension. This you would think would be easy to find. Unfortunately it’s not, this is down to Azure, not the site extension. The location of the blob file is different depending on the name of the application, name of the web job, date/time, website instance and website process ID of the web job. Please note: If the web job runs from one hour into the next, it will create the same named file in two different hour folders.

Typical path would look like:

/blobapplication/<WebSiteName>-<TriggeredOrContinous>-<WebJobName>/<CurrentYear>/<CurrentMonth>/<CurrentDay>/<CurrentHour>/<InstanceIdFirst6Char>-<ProcessId>.applicationLog.csv

If you have a web job that runs every hour on the hour, then it’s not that difficult to find, but a web job that runs on demand can be difficult to find. Therefore I write to the WebJob Log using Console.WriteLine the location of the log, using code that calculates the location of the file. This way if there is a reason I need to look into the trace logs of a given web job instance, I just view the web job to get the link to the file.

var instanceID = Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID").Substring(0, 6);
var pid = Process.GetCurrentProcess().Id;
var currentTime = DateTime.UtcNow;
var filename = instanceID + "-" + pid + ".applicationLog.csv";

//Location of the blob file path within the Azure Web Site logs extension
var filePath = "https://" + Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") + 
                             ".scm.azurewebsites.net/WebSiteLogs/#/blobapplication/" + 
                             Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") + 
                             "-triggered-" + Environment.GetEnvironmentVariable("WEBJOBS_NAME") + "/" +
                            currentTime.Year + "/" + currentTime.Month.ToString().PadLeft(2,'0') + "/"  + 
                           currentTime.Day.ToString().PadLeft(2, '0') + "/" +
                           currentTime.Hour.ToString().PadLeft(2, '0') + "/" ;

//Location of the blob file to download within the Azure Web Site log Extension
var linkToDownload = "https://" + Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") + 
                     ".scm.azurewebsites.net/WebSiteLogs/api/log?path=/blobapplication/" +
                     Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") + 
                     "-triggered-" + Environment.GetEnvironmentVariable("WEBJOBS_NAME") + "/" +
                     currentTime.Year + "/" + currentTime.Month.ToString().PadLeft(2,'0') + "/"  +
                     currentTime.Day.ToString().PadLeft(2, '0') + "/" +
                     currentTime.Hour.ToString().PadLeft(2, '0') + "/" + filename + "&download=true";

Console.WriteLine(String.Format("FilePath Link: {0}", filepath));
Console.WriteLine(String.Format("Download csv file: {0}", linkToDownload));

Using the actual Azure Web Site Log Browser, you can view the csv log file directly in the browser. Using the filePath link that I’m displaying in the Web Job log window, this takes you to the Azure Web Site Log Browser.

By clicking on the file it opens the CSV in the browser.

Using this browser, you can view the logs. You can use Search to find key words within your application.

The columns available within a blob storage is defined by Azure, you cannot add or remove these columns. These columns provide you more granular information about the event. The following properties are used for each row in the CSV:

  • Date – The date and time that the event occurred
  • Level – The event Level (Verbose, Error, Warning, Information)
  • Application Name – The web app name, in our case the WebApp – TypeOfWebJob – WebJobName.
  • InstanceId – Instance of the Web app that the event occurred on.
  • EventTickCount – The date and time that the event occurred, in Tick format
  • EventId – The event ID of this event, defaults to 0 if none specified.
  • Pid – Process ID of the web job
  • Tid – The thread ID of the thread that produced the event.
  • Message – The event detail message
  • ActivityID – This is only included if you set up the Activity ID in the code. This can be useful especially for continuous webjobs as all the jobs processed will be entered into the same csv file.

Setting the Activity ID in code

//Add to the start of the code
var correlationGuid = Guid.NewGuid();
Trace.CorrelationManager.ActivityId = correlationGuid;
Console.WriteLine(String.Format("ActivityID for this job: {0}, correlationGuid"));

Viewing Application Logs – Table Storage.

The URL for table logs within the Azure Web Site Log Browser is https://<WebSiteName>.scm.azurewebsites.net/websitelogs/viewtable.cshtml

On landing on the page, it shows the last 20 items found within the table. However you can change the date, number of items, or what to search for using search. There is also sorting within the columns. I have put the ActivityID in search (see end of last section how to assign Activity ID in code), and this has brought back all log items for that web job.

The columns available within a table storage is defined by Azure, you cannot add or remove these columns. These columns provide you more granular information about the event. The following properties are used for each row in the table:

  • Date – The date and time that the event occurred
  • Level – The event Level (Verbose, Error, Warning, Information)
  • Instance – Instance of the Web app that the event occurred on.
  • Activity – This is only included if you set up the Activity ID in the code.
  • Message – The event detail message

If you open the row of information on the table using Visual Studio/Azure Storage 6 or another tool, there are additional columns that contains information that isn’t shown within the Azure Web Site Log Browser.

  • PartitionKey – The Date Time of the event in yyyyMMddHH format
  • RowKey – A GUID value that uniquely identifies this entity
  • Timestamp – The date and time that the event occurred
  • EventTickCount – The date and time that the event occurred, in Tick format
  • Application Name – The Web App Name
  • EventId – The event ID of this event, defaults to 0 if none specified.
  • InstanceId – Instance of the Web app that the event occurred on.
  • Pid – Process ID of the web job
  • Tid – The thread ID of the thread that produced the event.

References:

Amit Apple Blog – http://blog.amitapple.com/ (Creator of the Azure Web Site Log Browser)

Microsoft Azure – Enable diagnostics logging for web apps in Azure App Service – https://azure.microsoft.com/en-us/documentation/articles/web-sites-enable-diagnostic-log/

Azure Table Storage and pessimistic concurrency using an Azure blob


In my previous two blog posts, I have spoken about concurrency using Azure Blobs and Azure Table Storage.

Azure Blob Storage and managing concurrency

Azure Table Storage and managing concurrency

In the Azure Table Storage and managing concurrency, I state that they only option you have for concurrency is optimistic concurrency. This is when performing an update, it will verify if the data has changed since the application last read that data. However there is a way to perform pessimistic locking on an Azure table entity by assigning a designated blob for each table, and try to take a lease on the blob before operating the table.

This blog post will walk you through creating a solution that allows you to perform pessimistic locking for Azure table entity. My solution will show two methods. The first method is a single thread application that will try to update a number in the Azure table. If we run the program twice at the same time, one of the programs will be blocked and receive an HttpsStatusCode of 409 (Conflict).

You will need to following NuGet Packages installed for the code to work:

  • Microsoft.WindowsAzure.ConfigurationManager
  • WindowsAzure.Storage

The SingleThreadTableStorageUpdate() method will first obtain the values from the app.config. These values are:

  • BlobStorageFileName – The filename of the blob that will be assigned a lease.
  • BlobStorageContainerReference – The Blob container, that will hold the blob file.
  • TableStorageReference – The name of the Table within Azure Storage.
  • StorageConnectionString – The connection string to Azure Storage.
//Obtain the BlobStorage information
String filename = System.Configuration.ConfigurationManager.AppSettings["BlobStorageFileName"];
String blobStorageContainerRef = System.Configuration.ConfigurationManager.AppSettings["BlobStorageContainerReference"];
String blobStorageTableRef = System.Configuration.ConfigurationManager.AppSettings["TableStorageReference"];
String connectionString  = CloudConfigurationManager.GetSettings("StorageConnectionString");

//Instantiate the NextNumber class
NextNumber nextNumber = new NextNumber(filename, blobStorageContainerRef, blobStorageTableRef, connectionString);

Within the NextNumber Class when it is instantiated, it will check or create the Blob file and table entity.

class NextNumber
{
 private readonly CloudBlobContainer _leaseContainer;
 private readonly CloudTable _table;
 private readonly String _filename;

 public NextNumber(string filename, string blobContainerReference, string tableStorageReference, string storageConnectionString)
 {
   //Get Connection to Storage.
   CloudStorageAccount storageAccount = CloudStorageAccount.Parse(storageConnectionString);
   _filename = filename;

   //This creates a Blob
   CloudBlobClient blobClient = storageAccount.CreateCloudBlobClient();
   _leaseContainer = blobClient.GetContainerReference(blobContainerReference);
   _leaseContainer.CreateIfNotExists();

   //This creates a table.
   CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
   _table = tableClient.GetTableReference(tableStorageReference);
   _table.CreateIfNotExists();

   try
   {
       //Get a reference to the blob.
       CloudBlockBlob blob = _leaseContainer.GetBlockBlobReference(String.Format("{0}.lck", _filename));
       if (!blob.Exists())
       {
           //Blob doesn't exist therefore create table entity.
           NextNumberEntity entity = new NextNumberEntity
          {
             PartitionKey = _filename,
             RowKey = "",
             NumberValue = 0
          };

          //Upload blob with some information in file
          blob.UploadText("Created on " + DateTime.UtcNow);
          //Insert entity into table.
         _table.Execute(TableOperation.Insert(entity));
       }
   }
   catch (Exception ex)
   {
       Console.WriteLine("Error happened " + ex.Message);
   }
 }
}

Once I know that the blob file and table actually exists, I’m then able to get the next number and update the entity in the table.

nextNumber.GetNextNumber()

Below is the code for the GetNextNumer() method. Within the code I have grabbed the blob property lease information and displayed it within the console. This is useful to see the state of the current blob object and see if there is a lease on it.

internal int GetNextNumber()
{
   //Get blob reference and display current lease information
   CloudBlockBlob blob = _leaseContainer.GetBlockBlobReference(String.Format("{0}.lck", _filename));
   blob.FetchAttributes();

   Console.WriteLine(String.Format("LeaseDuration = {0}", blob.Properties.LeaseDuration));
   Console.WriteLine(String.Format("LeaseState = {0}", blob.Properties.LeaseState));
   Console.WriteLine(String.Format("LeaseStatus = {0}", blob.Properties.LeaseStatus));

   //Acquire the lease for 30 seconds.
   string leaseId = blob.AcquireLease(TimeSpan.FromSeconds(30), Guid.NewGuid().ToString());
   var nextNumber = 0;
   Console.WriteLine();

   Console.WriteLine(String.Format("Aquired lease on blob ID: {0}", leaseId));
   Console.WriteLine();

   try
   {
        //Get and display current lease information
        blob.FetchAttributes();
        Console.WriteLine(String.Format("LeaseDuration = {0}", blob.Properties.LeaseDuration));
        Console.WriteLine(String.Format("LeaseState = {0}", blob.Properties.LeaseState));
        Console.WriteLine(String.Format("LeaseStatus = {0}", blob.Properties.LeaseStatus));

        //Retrieve the entity out of the Azure table.
        TableResult tableResult = _table.Execute(TableOperation.Retrieve<NextNumberEntity>(_filename, ""));
        NextNumberEntity entity = (NextNumberEntity)tableResult.Result;
        //Update the number
        entity.NumberValue++;
        //Add back into Azure table.
        _table.Execute(TableOperation.Replace(entity));
        nextNumber = entity.NumberValue;
        //Wait to extend the time this calling code hold the lease for (demo purposes)
        Thread.Sleep(TimeSpan.FromSeconds(10));
    }
    catch (Exception ex)
    {
        Console.Write("An error: " + ex.Message);
    }
    finally
    {
       //Release the blob.
       blob.ReleaseLease(AccessCondition.GenerateLeaseCondition(leaseId));
    }
    return nextNumber;
  }

Lastly I have a class that is my NextNumberEntity. This inherits Microsoft.WindowsAzure.Storage.Table.TableEntity.

class NextNumberEntity : TableEntity
{
     public int NumberValue { get; set; }
}

If I run the above code, it creates the blob file, the table, and updates the number from 0 to 1 in the table.

Above shows the leaseobject blob container, and the nextnumbers table.

Above shows the Cann0nF0dderDemo.lck file created within the leaseobject blob container.

Above shows the nextnumber table with the number value set to one.

Above shows the console window. You can see how before the lease was acquired, the LeaseState for the blob is Available with the LeaseStatus set as Unlocked. Soon as a lease was acquired, the LeaseState for the blob is Leased, with the LeaseStatus set to Locked. Lastly the console displays the next number which is one.

If I run two instances of the program and get them trying to acquire the lease at the same time, one errors with a conflict.

So how can I ensure that both instances run and eventually both get a number? I’ve written another method that is similar to the GetNextNumber method within the NextNumber class, called GetNextNumberWithDelay(). If a conflict is discovered then it retries the method, until I have obtained the next number.

internal int GetNextNumberWithDelay()
{
   //Get the blob reference
   CloudBlockBlob blob = _leaseContainer.GetBlockBlobReference(String.Format("{0}.lck", _filename));
   var nextNumber = 0;
   bool gotNumber = false;

   while (!gotNumber)
   {
       try
       {
         //Acquire the lease for 60 seconds.
         string leaseId = blob.AcquireLease(TimeSpan.FromSeconds(60), Guid.NewGuid().ToString());
         Console.WriteLine("Acquired Lease to update number");

         try
         {
              //Retrieve the entity out of the Azure table.
              TableResult tableResult = _table.Execute(TableOperation.Retrieve<NextNumberEntity>(_filename, ""));
              //Wait to extend the time this calling code hold the lease for (demo purposes)
              Thread.Sleep(TimeSpan.FromSeconds(10));

              NextNumberEntity entity = (NextNumberEntity)tableResult.Result;
              //Update the number
              entity.NumberValue++;

              //Add back into Azure table.
              _table.Execute(TableOperation.Replace(entity));
              nextNumber = entity.NumberValue;
              Console.WriteLine();

              Console.WriteLine(String.Format("The next number is: {0}", nextNumber));
          }
          catch (Exception inner)
          {
              Console.WriteLine("Another Error: " + inner.Message);
          }
          finally
          {
              //Release the blob
              blob.ReleaseLease(AccessCondition.GenerateLeaseCondition(leaseId));
              gotNumber = true;
           }
       }
       catch (Microsoft.WindowsAzure.Storage.StorageException se)
       {
           var response = se.RequestInformation.HttpStatusCode;
           if (response != null && (response == (int)HttpStatusCode.Conflict))
           {
               //A Conflict has been found, lease is being used by another process, wait and try again.
               Console.Write(".");
               Thread.Sleep(TimeSpan.FromSeconds(2));
           }
           else
           {
              throw se;
           }
        }
    }

   Thread.Sleep(TimeSpan.FromSeconds(3));
   return nextNumber;
}

Just to really test my code, instead of just calling the GetNumberWithDelay() just once, I’m going to uses Task threading and call it 5 times at once. This way if I run two instances of the program, I’m requesting 10 different numbers on 2 instances, 10 different threads. The lease on the blob will only allow one thread at a time to request a number, making all the other threads wait. Before I ran this, I reset the NumberValue in the Storage table back to 0.

//Run the GetNextNumber code 5 times at once. 
Task.WaitAll(new[]{
             Task.Run(() => nextNumber.GetNextNumberWithDelay()),
             Task.Run(() => nextNumber.GetNextNumberWithDelay()),
             Task.Run(() => nextNumber.GetNextNumberWithDelay()),
             Task.Run(() => nextNumber.GetNextNumberWithDelay()),
             Task.Run(() => nextNumber.GetNextNumberWithDelay())
});

When the code runs, one thread on one instance will obtain the lease, every other thread (on both instances) will have to wait, and will display a dot “.”

As you can see from above, both instances obtained 5 different numbers, however each instance didn’t receive 5 sequential numbers, and together, they never grabbed the same number twice.

This concludes my blog posts on Azure Blob/Table concurrency.

Code: http://1drv.ms/1JBoIek

Azure Table Storage and managing concurrency


In my previous post, Azure Blob Storage and managing concurrency, I wrote about storing a blob and then using either:

  • Optimistic concurrency – When performing an update, it will verify if the data has changed since the application last read that data.
  • Pessimistic concurrency – When performing an update, it will acquire a lock, preventing other processes from trying to update it.

When using Azure Table storage you only have the option of using optimistic concurrency. If pessimistic locking is needed, one approach is to assign a designated blob for each table, and try to take a lease on the blob before operating the table. I discuss this in my next post, Azure Table Storage and pessimistic concurrency using an Azure blob.

Optimistic concurrency

Every entity that is added to the storage is assigned an ETag. Every time the entity changes, the ETag will also change. It is this ETag that is used to calculate if the entity has changed since the process has last read it. The steps are:

  • Read the entity from the table storage, and grab the ETag header value.
  • Update the entity, passing in the ETag value from reading the entity from previous step.
  • The ETag passed in, matches the ETag of the current entity in the storage table. The entity is updated, and a new ETag assigned.
  • If the ETag passed in, doesn’t match the ETag of the current entity in the storage table, due to another process changes it, then an HttpStatusCode of 412 (PreconditionFailed) is returned and the file isn’t updated.

The below code (taken partly from Managing Concurrency using Azure Storage – Sample Application) shows an example of optimistic concurrency. It also shows how to ignore concurrency completely to simulate a different process updating the blob. You will need to following NuGet Packages installed for the code to work:

  • Microsoft.WindowsAzure.ConfigurationManager
  • WindowsAzure.Storage
internal void DemonstrateOptimisticConcurrencyUsingEntity()
{
    Console.WriteLine("Demo - Demonstrate optimistic concurrency using a table entity");
    CloudStorageAccount storageAccount = CloudStorageAccount.Parse(CloudConfigurationManager.GetSetting("StorageConnectionString"));
    string tableStorageReference = System.Configuration.ConfigurationManager.AppSettings["TableStorageReference"];

   CloudTableClient tableClient = storageAccount.CreateCloudTableClient();
   CloudTable nextNumberTable = tableClient.GetTableReference(tableStorageReference);
   nextNumberTable.CreateIfNotExists();

   //Add new entity to table (Requires PartitionKey and RowsKey to work)
   NextNumberEntity originalNumber = new NextNumberEntity()
   {
        PartitionKey = "Numbers",
        RowKey = "Next",
        NumberValue = 0
   };

   TableOperation insert = TableOperation.InsertOrReplace(originalNumber);
   nextNumberTable.Execute(insert);
   Console.WriteLine("Entity added. Original ETag = {0}", originalNumber.ETag);

   //Simulate an update by different process
   NextNumberEntity updatedNumber = new NextNumberEntity()
   {
       PartitionKey = "Numbers",
       RowKey = "Next",
       NumberValue = 1
   };

   insert = TableOperation.InsertOrReplace(updatedNumber);
   nextNumberTable.Execute(insert);
   Console.WriteLine("Entity updated. Updated ETag = {0}", updatedNumber.ETag);

   //Try updating originalNumber. Etag is cached within originalNumber and passed by default.
   originalNumber.NumberValue = 2;
   insert = TableOperation.Merge(originalNumber);

   try
   {
       Console.WriteLine("Trying to update original entity");
       nextNumberTable.Execute(insert);
    }
    catch(StorageException ex)
    {
        if (ex.RequestInformation.HttpStatusCode == (int)HttpStatusCode.PreconditionFailed)
        {
            Console.WriteLine("Precondition failure as expected. Entities orignal etag does not match");
        }
        else
        {
            throw;
        }
    }

    Console.WriteLine("Press enter to exit");
    Console.ReadLine();
}

My app.config file for my console application has the following in it for the above to work.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
    <startup> 
        <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
    </startup>
  <appSettings>
    <add key=" TableStorageReference" value="nextnumbers"/>
    <add key="StorageConnectionString" value="[Insert your own DefaultEndpointsProtocol to your Azure Storage] "/>
  </appSettings>
</configuration>

The results from running DemonstrateOptimisticConcurrencyUsingEntity()

If you want to see the table you could use visual studio. In the Server explorer, if you sign into your Azure account, under storage you should see the table you created. Mine is called NextNumbers, which I defined in my app.config “TableStorageReference”.

Within our table, we have the data we placed in from the update.

References: https://azure.microsoft.com/en-gb/blog/managing-concurrency-in-microsoft-azure-storage-2/

You can download my code from here.

Code: http://1drv.ms/1FlbDUw