Lessons learned from designing and building a modern SharePoint Server 2016 platform architecture for a Government of Canada agency and some advanced troubleshooting scenarios that arose. We will look in particular at web and service applications, host-named site collections, search, and security.

1. SharePoint 2016
Platform Adoption
Lessons Learned and Advanced
Troubleshooting
Level 300 / Technical
SHAREPOINT SATURDAY OTTAWA
October 28, 2017

2. Thank you Sponsors!!
Platinum
Gold
Silver
Bronze Breakfast
SharePint

3. Salon A
@4:45
pm
Please drink responsibly . We will be happy to call a cab for you
Sponsored By

4. About Me
• SharePoint / .NET solution and technical architect
• Over 20 years experience developing business
solutions for private industry & government
• Recent clients include DND, StatCan, HoC, Justice,
NRC, NSERC, DFAIT, CFPSA, MCC, OSFI
• Specialize in Microsoft technologies
• Speaker at user groups and conferences

5. Thank you to my sponsor
• CloudShare – Environments Made Easy
• http://www.cloudshare.com/

6. Overview
• Lessons Learned
• Adoption
• Features
• Security
• Host names
• Advanced Troubleshooting
• Central Admin on the Blink
• BDC Service App Association Failure
• Resource failed to load
• Host Named SC in iFrame
• HTTP Protocol of New Site Collection

7. Adoption
• Very simple / smooth
• No major differences in project core workloads
• Publishing Portal site collection with EN / FR
variations
• Design Manager package
• WET 4
• BDC .NET Connector
• Search service app
• Display templates

8. What Is New and Sort-Of Useful
• Smaller MinRole farm
• New capability of Feature Pack 1 / Nov 2016 public
update
• Front-end with Distributed Cache
• Application with Search
• Admin Actions Logging (Central Admin and
PowerShell)
• New capability of Feature Pack 1 / Nov 2016 public
update
• No GUI or browse tooling available, access via
PowerShell
• Retained for a maximum of 31 days

9. What Is New and Sort-Of Useful
• SharePoint Framework
• New capability of Feature Pack 2 / Sept 2017 public
update
• YADeF4SP – Yet Another Development Framework For
SharePoint, hope this one sticks
• Highlights include
• New page and web part model
• Easy integration with SharePoint data
• Responsive and mobile-ready experiences and apps
• Modern web client-side development

10. What Has Improved?
• PSConfig (PU)
• Recovery from cancel or abort/error
• Restarting of app pools
• Synonyms weighted the same as the original query
terms (PU)
• Ampersand and other special characters in file
name (RTM)
• Both drag & drop to library in browser and Explorer View

11. What Got Worse?
• Retrieve user crawled properties with PowerShell
• Only OOTB crawled properties are retrieved by
PowerShell
• Get-SPEnterpriseSearchService
Application -Identity "Search" |
Get-SPEnterpriseSearchMetadata
CrawledProperty -Category 'Business
Data' | ft
Name Category Name Is Mapped To
Contents
Is Name
Enum
Schema
Id
Variant
Type
docaclmeta Business Data False False 0 0
EntityName Business Data False False 0 0
EntityNamespace Business Data False False 0 0

12. MinRole / Service Instances
• Enforces predefined set of service instances per
server role
• Attempts to auto-restart service instance if it stops
• Does not repair service instance if it won’t restart
or is corrupted
• Use Install-SPService to repair / re-
provision service instance
• Per server node not entire farm
• https://technet.microsoft.com/en-
us/library/ff607705(v=office.16).aspx

13. Admin vs Farm Account
• Admin account is not the same as Farm account
• Admin account
• Farm setup and patching
• PSCONFIG
• Configure and manage farm and servers
• Farm account
• Central Admin app pool identity
• Timer Service identity

14. Admin vs Farm Account
• Account logged in when SharePoint farm is created
(psconfig) becomes db_owner for farm config and
central admin databases
• Account logged in when service apps and web apps
are created in PowerShell becomes db_owner for
their databases, with limited exceptions
• However, Farm account is automatically owner for
some of these

15. Admin vs Farm Account
• Important when SQL is hosted and SP admins have
limited access / permissions to SQL Server
• Important when installing and configuring those
service apps that have Farm account as dbo

16. Admin vs Farm Account
Services with Farm Account as dbo no matter what:
• Business Connectivity
• Secure Store
• Why?

17. Admin vs Farm Account
• Even reputable authorities confound Admin and
Farm accounts
• Eg well-known Vlad Catrinescu blog post,
SharePoint 2013 Service Accounts Best Practices
Explained
• As of Mar 27, 2017, more than four years after
originally published

18. FQDN versus Non-FQDN
• Modern DNS config resolves both to same host
• However HTTP request still carries original
hostname
• SharePoint and IIS may need AAM / bindings
defined for both otherwise won’t respond
• Do you really want / need to support both?
• If not, consider a rewrite rule to canonical form
• Use generic bindings

19. FQDN versus Non-FQDN
• Agnostic bindings for IIS are most flexible
• If multiple web apps IP Address could be specific IP
or All Unassigned

20. FQDN versus Non-FQDN
• Rewrite Non-FQDN to canonical FQDN
• CACHE_URL is to capture protocol, no server variable for
that
<rule name="Non-FQDN to FQDN" enabled="true"
stopProcessing="true">
<match url=".*" />
<conditions logicalGrouping="MatchAll"
trackAllCaptures="false">
<add input="{SERVER_NAME}" pattern="^hnsc-group$"
negate="true" />
<add input="{SERVER_NAME}" pattern="^[^.]*$" />
<add input="{CACHE_URL}" pattern="^(.+)://" />
</conditions>
<action type="Redirect“
url="{C:1}://{SERVER_NAME}.XXX.ca:{SERVER_PORT}{PATH_INFO
}" />
</rule>

21. Advanced Troubleshooting Tools
• Google! + Experience + Intuition
• ULS Viewer
• PowerShell
• F12 Developer Tools in browser / Developer
Dashboard
• SharePoint Manager 2013
• SharePoint 2013 Search Tool
• dotPeek
• Fiddler and Wireshark

22. #1 Troubleshooting Technique
• Google Search !
• + Judicious keywords !

23. Central Admin On The Blink
• Refresh Central Admin but page is blank!?
• No error msg
• No correlation Id
• Entirely blank browser window

24. Central Admin On The Blink
• Try again in new browser tab!?
• Now standard browser 404 response
• But how does Central Admin go away so
completely?!

25. Central Admin On The Blink
• Check and restart Central Admin IIS website and
app pool
• Hostname and server ping succeeds

26. Central Admin On The Blink
• Check ULS, filter to Show Errors Only, a sea of red

27. Central Admin On The Blink
SQL database login for
‘XXX_DEV_SharePoint_Conf
ig' on instance
‘XXX_DEV_SP16' failed.
Additional error
information from SQL
Server is included
below. Login failed for
user ‘YYYXXX-D_Farm'.
Cannot continue the
execution because the
session is in the kill
state. A severe error
occurred on the current
command. The results,
if any, should be
discarded.

28. Central Admin On The Blink
• Check SQL Server Management Studio
• Farm account is Deny access to database engine

29. Central Admin On The Blink
• There were numerous legacy service accounts
• In the process of disabling these we went one too
far!
• Denied the Farm account access to the farm SQL
server
• Reset Farm account to:
• Permission to connect to database engine: Grant
• Central Admin came back online

30. Conclusion
• ULS log had a simple and straightforward error msg
• Sometimes you get lucky with ULS!
• But not often!

31. ULS – View Logs From All Server
Nodes!

32. BDC Service App Issue
• Dev team reports BDC .NET Connector deployment
fails
• Dev team workaround:
• Switch web app service app associations from Default to
Custom and include BDC service app
• Even though Default and Custom groups show the
same service app associations

33. Service App Associations in Central
Admin
• Default associations vs custom associations
• BDC actions in Central Admin and PowerShell fail
with Default associations assigned, succeed with
Custom associations

34. BDC: Configure for Profile Page Host
site Issue
• Possibly related issue in BDC service app Configure
command for Profile Page Host site
• The default BDC Service on site http://informatics-
pot16-dev.XXX.ca/ does not match current BDC Service.

35. BDC: Configure for Profile Page Host
site Issue
• ULS logs show a divergence but no clear leads for
troubleshooting

36. PowerShell Discrepancy vs Central
Admin
• Discrepancy in default proxy group: PowerShell
compared to Central Admin
• SPWebApplication.ServiceApplicationProxy
Group.DefaultProxies
• Picasso BDC Proxy is missing!
Display Name Type Name
Managed Metadata Connection Managed Metadata Service Connection
Picasso Search Proxy Search Service Application Proxy
Picasso User Profile Service User Profile Service Application Proxy
Secure Store Proxy Secure Store Service Application Proxy
State Service Proxy State Service Proxy
Usage and Health Usage and Health Data Collection Proxy

37. Fix: Force Add Service App Proxy to
Default Group
• PowerShell to add Picasso BDC Proxy to Default
associations group
• Add-
SPServiceApplicationProxyGroupMember
$pg -Member 5b29056d-2206-426a-b9db-
096a3a43fd60 # Picasso BDC Proxy
• No change in how Central Admin displays Default
service app associations list
• Resolves issue in BDC service app Configure command
for Profile Page Host site
• True fix versus using Custom association group work
around

38. Root Cause – Solution Deployment Script
• BDC service app deployed by solution PowerShell
script
• New-SPBusinessDataCatalogServiceApplication
cmdlet also creates a proxy but does not permit
assigning it’s name
• New-
SPBusinessDataCatalogServiceApplicationProxy
cmdlet permits assigning the proxy name
• Solution deployment script combined these and
Remove-SPServiceApplicationProxy cmdlet
• In a funky way that left the farm confused about
the state of the Default association group

39. Conclusion
• Inspect farm config with PowerShell as it may be
more accurate than Central Admin

40. Resources Failed to Load
• “Test User” / Restricted Reader experience
• No correlation id for ULS lookup

41. Resources Failed to Load
• Network Request and Response clearly shows
which file

42. Resources Failed to Load
• Various files not published with major version

43. SharePoint Developer Dashboard
• Similar to F12 Dev Tools, but server-side focus
• Deep ASP.NET / SharePoint page info
• Enable with PowerShell

44. Conclusion
• Use browser F12 Developer Tools and other tools
• Inspect HTTP requests, responses, content, and
browser errors
• Rapidly zero in on page or item related problems
• Inspect ASP.NET / SharePoint page lifecycle and
Server Object Model (SSOM) API calls

45. SharePoint Manager 2013

46. Search Query Tool

47. Search Query Tool

48. Host Named SC in iFrame
• Dev team reports that BDC service app
Create/Upgrade profile pages screen started having
an issue
• This content cannot be displayed in a frame

49. Host Named SC in iFrame
• But that it opens fine a new tab or window

50. Host Named SC in iFrame
• You might recognize the issue immediately:
• Same Origin policy for framing
• Or Google
• Request domain is sps-adds-d01.XXX.ca
• Target domain is informatics-pot16-dev.XXX.ca
• But can F12 Dev Tools tell us more about root cause
/ resolution?

51. Host Named SC in iFrame
• Console errors are not relevant
• /_layouts/15/CreateProfileDialog.aspx appears to
be the culprit
• Response aborts after that

52. Host Named SC in iFrame
• Response Headers for
/_layouts/15/CreateProfileDialog.aspx

53. Host Named SC in iFrame
• Where does the X-FRAME-OPTIONS setting come
from?
• Not IIS website HTTP Response Headers

54. Host Named SC in iFrame
• Try inserting our own HTTP Response Header, but
creates duplicate

55. Host Named SC in iFrame
• Add control to master page:
• <WebPartPages:AllowFraming
runat=”server” />
• But what is going on under the covers?

56. Host Named SC in iFrame

57. Host Named SC in iFrame

58. Host Named SC in iFrame

59. Conclusion
• SharePoint forces X-FRAME-OPTIONS: SAMEORIGIN
• No farm or IIS config will override this
• HttpModule can override
• Or just live with browser framing warning msg

60. HTTP Protocol of New Site Collection
• How to control the protocol of the URL of a new
site collection?
• 3rd party Create Host-Named Site Collection
• OOTB Create Site Collection

61. Analyze Internals of an Application
Page
• /_admin/SharePointHNSC/createhostnamedsite.as
px

62. Analyze Internals of an Application
Page
• C:Program FilesCommon Filesmicrosoft
sharedWeb Server
Extensions16TEMPLATEADMINSharePointHNSC

63. Analyze Internals of an Application
Page
• 300 Lines of ASP.NET and JavaScript: How to find
protocol element?

64. Analyze Internals of an Application
Page
• Browser > F12 Developer Tools > DOM Explorer

65. Analyze Internals of an Application
Page
• HTML element in createhostnamedsite.aspx

66. Analyze Internals of an Application
Page
• Identify page class’ code behind assembly

67. Analyze Internals of an Application
Page
• Identify page class’ code behind – from source code

68. Analyze Internals of an Application
Page
• Identify page class’ code behind assembly

69. Analyze Internals of an Application
Page
• Open assembly in dotPeek from JetBrains
(ReSharper)

70. Analyze Internals of an Application
Page
• Locate references to HTML element by its Id

71. Analyze Internals of an Application
Page
• Analyze markup and code to determine source of
element InnerText

72. Analyze Internals of an Application
Page
• Analyze markup and code to determine source of
element InnerText
• <SharePoint:WebApplicationSelector
id="Selector" runat="server“
OnContextChange="OnContextChange"
AllowAdministrationWebApplication="fa
lse" />
• SPWebApplication currentItem =
this.Selector.CurrentItem;
• this.SpanUrlProtocol.InnerText =
currentItem.GetResponseUri(SPUrlZone.
Default).Scheme +
Uri.SchemeDelimiter;

73. HTTP Protocol of New Site Collection
• Central Admin force default zone protocol for new
site collection to match default zone protocol for
web app container
• AAM of web app container needs to satisfy
platform workloads HTTP vs HTTPS

74. HTTP Protocol of New Site Collection
• Swap HTTP / HTTPS AAMs of web app container

75. HTTP Protocol of New Site Collection
• Swap HTTP / HTTPS AAMs of web app container

76. HTTP Protocol of New Site Collection
• Success: Switched default protocol for new site
collection
• 3rd party Create Host-Named Site Collection
• OOTB Create Site Collection

77. Conclusion
• Get comfortable with reading disassembled
Microsoft assemblies to delve into actual logic of
Central Admin application pages
• Also works for PowerShell cmdlets which are often
written in C# / .NET

78. Fiddler and Wireshark
• Show a degree of detail no available in F12 Dev
Tools and SP Dev Dashboard

79. Next Steps – Try Out These Tools
• ULS Viewer
• PowerShell
• F12 Developer Tools in browser / Developer
Dashboard
• SharePoint Manager 2013
• SharePoint 2016 Client Browser
• SharePoint 2013 Search Tool
• dotPeek
• Fiddler and Wireshark

80. Contact Me
• John Calvert, Chief Architect
• Software Craft, Inc.
• John (a) softwarecraft dot ca
• Softwarecraft dot ca
• (a) softwarecraft99