A side-by-side approach for upgrading SharePoint search to FAST

Possibly the shortest path to upgrading SharePoint built-in search to FAST
Page Content

The scheduled downtime solution

Upgrading your search solution to FAST for SharePoint (FS4SP) involves creating two new Search Service Applications, reindexing your content, and then at some point switching out the Standard SSA with the FAST Query SSA in your application proxy group as seen in fig. 1. (You can read more about the FS4SP SSA architecture on the TechNet Wiki)

Figure 1 – Configure Service Application Associations

In addition you will have to create a new search center for your FS4SP results in order to verify that you actually get results back and that they look the way you expected. Depending on how much content you have indexed this can usually only be done in the production environment. When testing your new search solution against FS4SP you must switch the default SSA in the application proxy group from the SharePoint one to the FS4SP one. This renders the built-in search in a non-working state while you test FS4SP. And once you switch back, the FAST search center will not work.

This all boils down to scheduled down time for testing and switching your search solution over to FS4SP.

You can work around this limitation by creating a new web application and a new application proxy group for this web application. You will also have to create a new site collection and a search center in this site collection. But why all the extra work, when you can have both the built-in search and FAST search running side by side in the same site collection?

Mikael’s no-downtime solution

The approach is to “upgrade” your current SSA to take the role of the FAST Query SSA. You will still have to reindex all your content (except people search) and most likely customize a new search center, but you can do it without touching the proxy application group or creating new web applications or site collections. And there is no down-time!

How is this possible?

Figure 2 – Create new Search Service Application

If you look under the covers on the search service applications, you will see that the FAST Query SSA is merely a regular SharePoint SSA with some extra properties. The first clue as to how alike they are is that people search with FS4SP is set up on the FAST Query SSA, using all the search bits from SharePoint itself and not FS4SP.

The second clue appears when you look at <14 Hive>\TEMPLATE\FEATURES\SearchAdminWebParts\searchadministration.aspx which has a conditional setting to display a link to the “FAST Search Administration” page as seen below.

01 <% if (IsSearchApplicationFast) { %>
02 <li class="static">
03     <a  class="static menu-item" href="/_admin/search/extendedsearchadministration.aspx?appid=<%=SearchApplicationId%>" title='<asp:Literal runat="server" Text="<%$Resources:Microsoft.Office.Server.Search,FSAdmin_Central_Admin_Title%>"/>'
04         id="S2LeftNav_Fastsearch"> 
05         <span class="additional-background">
06         <span class="static menu-item" >
07         <asp:Literal runat="server" Text="<%$Resources:Microsoft.Office.Server.Search, FSAdmin_Central_Admin_Title%>"/>
08         </span
09         </span
10     </a>
11 </li>
12 <%}%>

And thirdly when looking at the PowerShell command to script both a FAST Query SSA and a SharePoint SSA, the likeness is pretty obvious.

1 New-SPEnterpriseSearchServiceApplication "Search SSA" -SearchApplicationType "Regular" -DatabaseServer "intranet.contoso.com" -DatabaseName "SearchApplicationDB" -ApplicationPool $AppPool
2
3 New-SPEnterpriseSearchServiceApplication "FAST Query SSA" -SearchApplicationType "Regular" -DatabaseServer "intranet.contoso.com" -DatabaseName "FAST Query DB" -ApplicationPool $AppPool

Both lines create a search application of type “Regular”. The difference with the FAST Query SSA is five additonal PowerShell commands. First change the default search provider from SharePoint to FASTSearch.

1 Set-SPEnterpriseSearchServiceApplication "FAST Query SSA" -DefaultSearchProvider "FASTSearch"

Then set four extended properties, which associates the SSA with services on the FS4SP farm (included in the upgrade script at the end).

What this means is that you can take your existing search service application, change the default search provider, set the extended properties and your conversion is complete.

Any request from your old search center will have “Local Search Results” as the search location, and will get results from the SharePoint search index, while your new search center based on the FAST Enterprise Search site template will set the search location to “Local FAST Search Results” and the requests are redirected over to FS4SP.

Extended Query Properties before and after the “upgrade”

01 PS C:\> Get-SPEnterpriseSearchExtendedQueryProperty -SearchApplication "Search Service Application"
02
03 PropertyKey                               Value
04 -----------                               -----
05 FASTSearchContextProperties               SPS-Location,SPS-Responsibility
06 FASTSearchAdminServiceAuthenticationUser
07 FASTSearchAdminServiceLocation
08 FASTSearchQueryServiceLocation
09 FASTSearchContextCacheTimeout
10 FASTSearchManagedPropertyResultMapping
11 FASTSearchDisableUserContext
12 FASTSearchResourceStoreLocation
13 FASTSearchAlternateAccessMapProperties
14
15
16 PS C:\> .\upgrade_ssa.ps1
17 PS C:\> Get-SPEnterpriseSearchExtendedQueryProperty -SearchApplication "Search Service Application"
18
19 PropertyKey                               Value
20 -----------                               -----
21 FASTSearchContextProperties               SPS-Location,SPS-Responsibility
22 FASTSearchAdminServiceAuthenticationUser  comp\svc_spadmin
23 FASTSearchAdminServiceLocation            http://intranet.contoso.com:13257
24 FASTSearchQueryServiceLocation            http://intranet.contoso.com:13287
25 FASTSearchContextCacheTimeout
26 FASTSearchManagedPropertyResultMapping
27 FASTSearchDisableUserContext
28 FASTSearchResourceStoreLocation           http://intranet.contoso.com:13255
29 FASTSearchAlternateAccessMapProperties

Note that if you are using the KeywordQuery class in parts of your solution with the ResultsProvider property set to Default, you will have to change this to “SharePointSearch” if you want results from the SharePoint index and not FS4SP, as we changed the default provider with PowerShell.

Conclusion

You will still have to create the FAST Content SSA as per the FS4SP deployment documentation and migrate all your content sources and crawl rules from your existing SSA to the FAST Content SSA. But you don’t have to schedule any downtime, and you can test both search solutions side by side before shutting down the old one.

Once you have results from FS4SP up and running you can delete all content sources except the sps3://server entry which has to be kept for people search to function.

Upgrade script

01 # upgrade_ssa.ps1
02 #
03 # The below script is based on a script found in "Microsoft® SharePoint® 2010 Administrator's Companion", but fixed for bugs.
04 # The port numbers used assume you are using 13000 as the FS4SP base port, and use http and not https for the query traffic.
05 # Edit $SrvName, $ResLoctV and $ssa to match your system
06
07 Function Set-FASTProp
08 param ($SSA, [string] $Cmd, [string] $Prop, [string] $PropValue
09 # Initialise $obj to Extended property object 
10 $obj = "SPEnterpriseSearchExtended" + $cmd + "Property"
11 $GetProp = &("Get-" + $obj) -Sea $ssa -id $Prop -ErrorAction SilentlyContinue
12  
13   if ($GetProp
14  
15     # Property exists 
16     &("Set-" + $obj) -SearchApplication $ssa -ID $Prop -Value $PropValue
17   } else { 
18     # Property does not exist 
19     &("New-" + $obj) -SearchApplication $ssa -Name $Prop -Value $PropValue
20  
21 } # End of Function Set-FASTProp
22
23 $ssa = "Standard Search SSA"
24 $SrvName = "://intranet.contoso.com:"
25 $AdminUserV = "contoso\svc_spadmin"
26 $QueryLoc = "FASTSearchQueryServiceLocation"
27 $QueryLocV = "http" + $SrvName +"13287"
28 $AdminLoc = "FASTSearchAdminServiceLocation"
29 $AdminLocV = "http" + $SrvName + "13257"
30 $AdminUser = "FASTSearchAdminServiceAuthenticationUser"
31 $ResLoc = "FASTSearchResourceStoreLocation"
32 $ResLoctV = "http" + $SrvName + "13255"
33
34 Set-SPEnterpriseSearchServiceApplication $ssa -DefaultSearchProvider "FASTSearch"
35
36 Set-FASTProp $ssa "Query" $QueryLoc $QueryLocV
37 Set-FASTProp $ssa "Query" $AdminLoc $AdminLocV
38 Set-FASTProp $ssa "Query" $AdminUser $AdminUserV
39 Set-FASTProp $ssa "Query" $ResLoc $ResLoctV
Advertisements

Got REST? Querying SharePoint List data using REST services client-side: Part 1

 

So you need to get some data out of SharePoint, show it to your users in a non-utilitarian way, but don’t have time to build a full .NET Web Part? Find a List DataView too limiting and XSLT too painful? SharePoint (with JavaScript/jQuery) has your back.

Since SharePoint 2003 we have always had client-side ASP.NET Web Services we can query to pull List data out to show as we wish (http://Server04:50000/Examples/_vti_bin/Lists.asmx), but no matter the query language we used, you’ve always had to create and receive SOAP (Simple Object Access Protocol) packets using XML-based CAML (Collaborative Application Markup Language) over the wire.

Below is just a simple example of a packet we would need to send to SharePoint after ensuring its validity using a 3rd party toolset. If the CAML query is wrong SharePoint will only return a vague CAML error.

<soapenv:Envelope xmlns:soapenv='http://schemas.xmlsoap.org/soap/envelope/'>
	<soapenv:Body>
		 <GetListItems xmlns='http://schemas.microsoft.com/sharepoint/soap/'>
			<listName>" + listGuidCG + "</listName>
			<viewFields><ViewFields>
			   <FieldRef Name='FileRef' />
			   <FieldRef Name='DocIcon' />
			   <FieldRef Name='ContentType' />
			   <FieldRef Name='Document_x0020_Type' />
			</ViewFields></viewFields>
			<query>
				<Query>
					<OrderBy>
						<FieldRef Name='LinkFilenameNoMenu' Ascending='True' />
					</OrderBy>
				</Query>
			</query>
			<queryOptions>
				<QueryOptions>
					<RowLimit>150</RowLimit>
					<IncludeMandatoryColumns>FALSE</IncludeMandatoryColumns>
					<DateInUtc>TRUE</DateInUtc>
				</QueryOptions>
			</queryOptions>
		</GetListItems>
	</soapenv:Body>
</soapenv:Envelope>

Looks like fun, huh? I thought not, and that’s just the initial POST request!

To make querying easier, Microsoft implemented REST (Representational State Transfer plus HTTP) into the SharePoint 2010 platform using ADO.NET Data Services to make client-side data retrieval easier and more lightweight. What is REST, you ask? It is the Web itself as it was originally intentioned, a client either in transition between application states or “at rest“, able to interact with its user, but creates no load and consumes no per-client storage. It is also security-trimmed just like everything else in the SharePoint platform.

So what does that really mean? It is simply SharePoint data, or any oData-compliant (Open Data Protocol) information, output to the client like an RSS feed.

2011-12-27-GotRest-01.png

Note: if you are not seeing the SharePoint List information as XML in the browser then you’ll need to turn off RSS feed reading. In Internet Explorer 7 or greater, go to Tools–>Internet Options–>Content tab–>Feeds and Web Slices-settings

2011-12-27-GotRest-02.png

Here’s an example from NetFlix’s oData feed of their movie catalog. Note the QueryString variables and XML format of the data returned.
http://odata.netflix.com/Catalog/Titles?$top=10&$orderby=ReleaseYear desc

2011-12-27-GotRest-03.png

Well that looks pretty simple, doesn’t it? We can just view it in any current browser without any custom tools. So how do we access it? For SharePoint 2010, we change our Site URL to access the always-on service.
http://Server04:50000/Examples/_vti_bin/ListData.svc

2011-12-27-GotRest-04.png

So what are we seeing here? By loading that Site URL with the service extension we are automatically calling SharePoint’s REST service for that specific site (without a SOAP packet) and receiving a quick, light-weight output of all the Lists for that site with little effort.

Note: interacting with the REST service is case-sensitive for the List name and the QueryString variables. If you aren’t receiving any data back, check your values for typos as this is the ‘error screen’ you will see.

2011-12-27-GotRest-05.png

Now we’ll target a specific List by copying the name of our target List from the page (to avoid typos) and appending it to our URL, telling the REST service to ‘target’ that list for output. This will then show details of the List itself and all the entries we’ve previously entered (10 in my example).
http://Server04:50000/Examples/_vti_bin/ListData.svc/EmployeeDirectory

2011-12-27-GotRest-06.png

That’s great and all, but your users probably want to see only a subset of the records, maybe even with a filter. Using the ADO.NET Data Services guide (search for ‘Structure of Web Data Services URLs’), we’ll begin passing some QueryString variables against the List. The primary ones you will probably use are $orderby, $top, and $filter with some Logical Operators.

Note: Interacting with the QueryString requires putting a ‘?’ before the first variable passed and ‘&’ before each subsequent variable after that.

Let’s first only pull the top 2 of the records back as that’s what the business is looking for.
http://Server04:50000/Examples/_vti_bin/ListData.svc/EmployeeDirectory?$top=2

2011-12-27-GotRest-07.png

That’s good, but we forgot to sort by anything to control which records are returned. We’ll now add an order by clause to correct that.
http://Server04:50000/Examples/_vti_bin/ListData.svc/EmployeeDirectory?$top=2&$orderby=Modified desc

2011-12-27-GotRest-08.png

And finally the business decided they don’t need to see certain individuals, in this case Inactive Status, so we’ll filter by that.
http://Server04:50000/Examples/_vti_bin/ListData.svc/EmployeeDirectory?$top=2&$orderby=Modified desc&$filter=StatusValue eq ‘Active’

2011-12-27-GotRest-09.png

And that’s all that’s needed to get the data back, with all the ‘data source debugging’ done in the browser. In the next part of this series, we’ll take our querying of the SharePoint data to the next level and begin consuming it with JavaScript/jQuery to display on a Site for our users.