Learn how you can get the most out of this new search technology
FAST Search Server 2010 for SharePoint is a new product from Microsoft that brings together the high-end search technology developed by FAST—a company Microsoft acquired in 2008—and SharePoint technology. The result, also known as “FAST Search for SharePoint” or “FS4SP,” offers a lot to SharePoint developers and IT pros. Let’s take a look at what this product is and how you can get the most out of it.
Going Beyond the Search Box
Most people understand intuitively that search is important, especially as the volume and diversity of information grows. But few people understand how search works or what makes one type of search different or better than another. Search technology is simple on the outside, but complex on the inside. To understand why you might care about FAST Search for SharePoint, it’s useful to understand what Enterprise Search is, how it differs from web search, and how one search experience can be better than another.
Enterprise Search makes an organization’s content accessible to employees, customers, and partners. Most people use web search (like Bing or Google) regularly at home, and expect search inside the firewall to be the same. However, Enterprise Search is different. Rather than just serving up public information in web pages and a handful of other formats, Enterprise Search must connect to dozens of different systems, parse hundreds of different types of files, and provide airtight security and access control. Instead of getting tens of thousands of pages relevant to almost any search on the web, corporate searchers prefer fewer but highly relevant results, and often there is only one "right" document. The structure and metadata of corporate data is important, as are company-specific vocabularies and taxonomies. The algorithms used for Enterprise Search are somewhat different, because they can’t depend as heavily on popularity and inter-web-page link structure.
The "box, button, and result list" paradigm used for the past 25 years in web search is limiting. With high-end search it's possible to deliver more effective and compelling search experiences. FAST Search for SharePoint delivers a better search experience out of the box. The box, button, and search results are still there, but the overall experience is quite different. The search experience provides better results faster by being visual, conversational, social, and contextual (Figure 1).
Humans are great at visual pattern recognition, so visual cues in the search experience help people quickly find information, identify patterns, and discover new insights. Visual capabilities in FAST Search for SharePoint include Visual Best Bets, document thumbnails and scrolling previews, people photos, and other capabilities that help users quickly explore and recognize information. Visual features on the main search page also tie to the "open in browser" capabilities of Office 2010, so a user can quickly edit and work with contents directly from the search results.
Conversational capabilities change how users interact with information for better results. Rather than typing a query, reading results, and then typing another query, the searcher can click to explore information. The conversational capabilities in FAST Search for SharePoint include deep refiners, query suggestions, sorting, similarity search, and multiple relevance ranking. Refiners let users drill down for more detailed information, and explore different facets of the result set. Deep refiners include counts across the entire result set, so that searchers can see the distribution of information within each facet. By exploring the demographics of information this way, you can gain insight even before you find a specific document.
Search is Social
A very common challenge in enterprises is locating people and expertise. FAST Search for SharePoint includes people search with name and expertise matching, making it easy to find people by name, title, expertise, and organizational structure. This includes phonetic name matching that will return names that sound similar to what the user has typed in a query. It will also return all variations of common names, including nicknames.
Refiners are also provided with people search results—exploring results via name, title, and various fields in a user's profile enables quick browsing and selection of people. People search results include real-time presence through Office Communication Server, making it easy to immediately connect with people once they are found through search.
Social behavior also improves search. In particular, common queries become suggestions, and relevance adapts and gets better with use as users click-through on search results.
Using Context to Deliver Audience-Specific Search
Because different audiences care about different things, IT can’t please everyone without providing audience-specific experiences. Typically, IT is forced to create a search experience that serves only a few groups well, at the expense of others. Some companies found that to ensure search worked for everyone, they needed multiple search implementations, sometimes even one for each department.
Contextual search is the way to solve this issue. FAST Search for SharePoint provides contextual search via an easy-to-administer construct called ‘"user context." This means search can be tailored to the user role, task, and perspective, and search administrators can provide great search for each audience. IT can let groups administer their own user contexts and individuals can participate in different contexts simply by updating their SharePoint user profile. An organization can deliver targeted results, role-specific relevance, and even task-specific relevance—from a single system.
Eliminating Compromise for IT
If high-end Enterprise Search is so much better, why isn’t it more prevalent? Historically, IT professionals have had to choose between high-end search and out-of-the-box search. High-end search products, such as FAST ESP, have been highly capable, with lots of power, flexibility, and sophisticated search technology, but they can be complex to install and manage and might require extensive services and specialized tools and training to customize. Many IT professionals love what they can do with high-end search technology, but they need it to be easier to use and less expensive.
There has also been a class of out-of-the-box Enterprise Search products, which are easy to manage and are designed to install and operate without any special customization—but they offer limited capability. Designed to be simple to download, install, and use, Microsoft’s SearchServer Express—available as a free download—is a case in point.
The need to choose between capability and ease of management in search products has forced IT to compromise. FAST Search for SharePoint eliminates the need to compromise by combining the best of high-end search technology, largely from FAST ESP’s heritage, and the best of SharePoint search technology—with the advantages of a mainstream product from Microsoft. By using the SharePoint 2010 architecture and administrative framework, (and investing a lot of development work into ease of management), FAST Search for SharePoint is the first high-end search product that you can use out of the box. It's a high-end search technology that delivers dynamic user experiences, advanced content processing, broad repository connectivity, extreme scale and performance, and an open and extensible platform—as well as out-of-the-box intranet search, people search, and SharePoint platform integration.
In addition, FAST Search for SharePoint offers simplicity and the low total cost of ownership users have come to expect from Microsoft. Users are able to work with standard, familiar, integrated developer tools, and to realize the benefits of a robust partner and platform ecosystem.
FAST Search Extends SharePoint Server
From an architecture standpoint, FAST Search for SharePoint builds on SharePoint Server 2010 and integrates tightly. SharePoint Server 2010 offers many new capabilities for IT pros and developers, including a new search center, unsealed web parts, deployment in sandboxes, a new connector infrastructure, and a new set of tools to make customizing SharePoint and search easy. SharePoint 2010 introduces shared service applications (SSAs) to serve common functions across multiple site collections and farms.
FAST Search for SharePoint adds an additional FAST Farm (this can be a virtual machine, a dedicated Server or multiple servers, depending on the required capacity). Content search is served by redirecting content after the crawlers into the FAST Farm, and by federating all queries to FAST Search for SharePoint into the content search. This means that FAST Search for SharePoint uses the same crawler and query handlers and provides the same people and expertise search as SharePoint Server 2010. It also uses the same OMs and the same administrative framework. There are extensions to accommodate the additional functionality in FAST Search for SharePoint, such as FAST Query Language (FQL). There are also a few extra connectors that are FAST specific, notably a more intelligent, high-capacity FAST web crawler. But for the most part, administration and development with FAST Search for SharePoint will be very familiar to those used to SharePoint (Figure 2).
FAST Search for SharePoint includes advanced content processing, which is applied to all content. This is configured and managed from the FAST Content SSA. It provides very powerful text analytics technology that prepares content for findability. This includes web link analysis, linguistics support for 84 languages, document format support for over 400 types, and property extraction (which creates metadata by machine, based on the content of each document).
FAST Search for SharePoint uses two SSAs: the FAST Query SSA and the FAST Content SSA. These SSAs govern all the processing on the SharePoint Farm. The FAST Farm itself is connected to both of these SSAs. Understanding what processing happens in which farm and which SSA can be confusing at first, but is pretty straightforward. Remembering the hybrid approach with common crawlers and query OM, but separate people and content search, is key to understanding the system configuration.
Scaling with FAST Search for SharePoint involves scaling both the FAST Farm and the components that support the two SSAs. Most of the processing is done by the FAST Farm, which scales linearly to extreme scale using a row and column architecture from FAST ESP which has been proven across thousands deployments. The content index itself is kept on the file system, not in a database. The search center itself and the bulk of content crawling resides on the SharePoint farm, and scales out linearly as well.
The scaling approach and two-farm architecture used for FAST Search for SharePoint means that organizations can work with a large, centralized search service. Multiple FAST Content SSAs and FAST Query SSAs can share the same FAST Farm, and they in turn can work across site collections and SharePoint farms. A centralized search service with distributed control means lower operational costs. The architectural approach also means that IT staff and developers trained in SharePoint can work with FAST Search for SharePoint immediately.
Management of High-End Search Made Mainstream
For the end user, FAST Search for SharePoint means great search, whether people search, content search, or any combination of the two. End users also realize the benefits of a high-end search experience—dynamic, engaging, and effective search.
From an administration standpoint, FAST Search for SharePoint is different from any other high-end search. There are two types of administration: search administration and system administration. Both of these are much easier than other high-end search such as FAST ESP.
A key advantage of FAST Search for SharePoint is that it works out of the box and that it's easy to control. With FAST Search for SharePoint, search administration is directly a part of site administration. Visual “best bets,” user context parameters and administration, relevancy profiles and parameters, and promotion/demotion of documents and sites are the same paradigm as what a SharePoint site administrator is used to.
There are capabilities that can be placed directly into the end-user’s control—sorting, ranking, navigation, and administration-enabled controls—as well as some pretty sophisticated linguistics that work out of the box and which can be added to or enhanced by the system administrator. For example, if a user has a SharePoint library with an author, date, or terms, the refiners that come from metadata drill down. The same is true for metadata that comes from an ERP system, CRM system, or file share, which is picked up from the crawler and is available directly for refinement.
Frequently, metadata is inconsistent or incomplete and can be enriched through entity extraction. The system administrator can create custom refiners by enabling entity extraction and configuring the confinement Web Part to show a new refiner. Or, the system administrator can give control to a super user in the organization with a specialized need who can be authorized to control the creation of custom refiners themselves.
Easily administered through a central console by the IT pro (rather than by a site administrator), the key to content processing is the content processing pipeline, which historically was a major strength of FAST. The content processing pipeline’s text analytics bring in text and read it, detecting the language, finding the core forms of that language, and completing a number of tasks including property extraction. This means that out of the box, the technology can recognize a number of qualities, such as people name, place name, and organization—a challenge when you consider, for example, that “Dow Jones” is a company but “Bob Jones” is a person and “Albert Hall” is a place.
IT pros can also add their own company “language,” such as a list of product names, to create a custom entity extractor that is usable for refinement, sorting, or relevance. The entity extractor of company, location, product, and people names is only a piece of the picture, however. It's extremely powerful and also offers capabilities that normalize dates and times that might be in different formats into information that is consistent for the purposes of search, as well as vectorized content that identifies the common terms and key concepts in a given document to be used for similarity searching.
FAST Search for SharePoint also provides a pipeline extensibility API that runs managed code in a sandbox so that the operations of the whole system aren’t threatened. This allows very flexible processing, which is especially valuable in some specialized installations—chemical or biological analysis, for example—where specialized property or entity extractors are required for chemical or genome names.
Historically, high-end search has been a very powerful, complex, technical undertaking that required special handling, special skills, and special tools. FAST Search for SharePoint retains the power but places it in a much more manageable context using tools from SharePoint and technology from Microsoft. It's natively 64-bit and supports full virtualization (this is relatively new for search, which is very I/O intensive and I/O latency sensitive), taking advantage of Windows 2008 and Hyper-V to make performance strong and system management simpler.
FAST Search for SharePoint uses a consolidated administration dashboard, which is the same dashboard used for SharePoint search. With the Administrative Web Parts, users can then tweak the dashboard as needed. Everything is scriptable through PowerShell. If an IT pro needs to complete common operations, he simply strings together PowerShell comdlets. In addition, FAST Search for Share Point comes with a Systems Center Operations Manager (SCOM) management pack that provides added visibility for IT pros and people in the NOC.
Developing Search-Driven Applications
For developers, FAST Search for SharePoint offers the ability to do more with search, including creating applications that can work across Microsoft search products, and search-driven applications will benefit from the high end capabilities that come with FAST.
Many people are just scratching the surface of what can be done with search, and FAST Search for SharePoint gives developers the opportunity to work across a range of configuration and development options. Many that once would have required code can now be accomplished through basic configuration, whether directly in Web Parts or through simple PowerShell scripts.
Developers also benefit from familiar, integrated development tools and a recognizable framework. For example, there is tooling built into SharePoint Designer that allows developers to extend Web Parts or create new connectors without writing code. This interoperates with Microsoft Visual Studio, so if a developer begins a project in SharePoint Designer, he can also work in Microsoft Visual Studio. Or, he can work directly in Microsoft Visual Studio. All Web Parts are public so a developer can extend them, and with the power of FAST Search for SharePoint, a developer can create new search experiences easily.
Although most organizations have general-purpose search needs such as intranet or site search, many organizations have special-purpose search requirements; roles such as marketing, sales, or support might have unique requirements to complete a particular task. The applications that allow users to accomplish these tasks can be very valuable and important to those who use them.
Historically, organizations used different technology to meet each of these requirements, choosing whichever vendor had developed an application specific to each requirement. With FAST Search for SharePoint, it's now possible to meet all the search application needs across a business. FAST Search for SharePoint has a deep understanding of an organization’s information, flexible relevance to meet diverse needs, and a customizable UI to increase user efficiency, and while search-driven applications cannot be developed out of the box, they can be developed with straightforward extensions.
Products for Every Search Need
Today, Microsoft can deliver search solutions to meet every need. There are now three main Enterprise Search offerings. Search Server 2010 express is a free, downloadable search offering for basic departmental or site search. SharePoint Server 2010 has search capabilities built in, which provides a great out-of-the-box search experience, allows for light customization, and scales up to 100 Million searchable items. FAST Search for SharePoint provides highly customizable productivity search and is the platform for search-driven applications.
FAST Search for SharePoint is a high-end search offering brought to the mainstream. It has an out-of-the-box Search Center with a search experience that is visual, dynamic, conversational, and social. It also provides extreme scalability and flexibility while being much simpler to license, deploy, and manage than traditional high-end search.