SQL 2012 Full-Text Catalog not populating after a Rebuild

January 9, 2014 Leave a comment

Problem

I work for a Software Development Company and we use our own CRM software that was designed in house, which leverages a SQL backend. It is a decent size database for a small company (e.g. ~ 7GB). This database has a Full-Text Catalog that is used by a Search GUI much like the Bing Desktop search toolbar or Google’s.

Anyway, I was getting reports that this was not finding specific records that previously it would have found. To cut a long story short, it appeared that the Full-Text Catalog was corrupted.

So I was asked to simply Rebuild the Catalog outside of office hours to fix the problem. The problem was that when I checked first thing in the morning, the Re-population of data appeared to be finished (e.g. the progress was showing Processing Notificiations, which occurs after the Population). If I looked at the Properties of the Full-Text Catalog the Unique Key Count = 33 and the Item Count = 11131. I was expecting Unique Key Count = ~3682890 and Item Count = ~531018. Something was wrong….

I retried this several times including recreating the Full-Text Catalog from scratch but it always seemed to stop around the same point.

Off to the trusty log files.

The log file for the Full-Text Catalog (yes, it has one…phew) is located here (C:\Program Files\Microsoft SQL Server\MSSQL.10\MSSQL\LOG) or similar as the MSSQL.10 is dependent on the version of SQL you are running). In this file I found the following error (lots of them);

Error ’0x80040e97′ occurred during full-text index population for table or indexed view ‘[DataBaseName].[dbo].[TableName]‘ (table or indexed view ID ’1641772906′, database ID ’6′), full-text key value ’463051′. Attempt will be made to reindex it.

In the SQL Server Logs themselves I found the following messages;

To View the SQL Server Error Log

  1. In Object Explorer, expand a server, expand Management, and then expand SQL Server Logs.
  2. Right-click a log and click View SQL Server Log.

Message: A new instance of the full-text filter daemon host process has been successfully started.
Message: The full-text filter daemon host process has stopped normally. The process will be automatically restarted if necessary.

Explanation

When you index large or complex documents by using integrated full-text search in Microsoft SQL Server 2008, Microsoft SQL Server 2008 R2, or Microsoft SQL Server 2012, you may receive time-out errors that resemble the following:

2010-06-07 15:02:44.64 spid10s Error ’0x80040e97′ occurred during full-text index population for table or indexed view ‘[DatabaseName].[dbo].[TableName]‘ (table or indexed view ID ’622625261′, database ID ’171′), full-text key value ’2375057′. Attempt will be made to reindex it.

Additionally, SQL Server may restart the FDHOST.exe process. When this issue occurs, an error message that resembles the following is displayed in the SQL Server error log:

2012-07-10 20:18:39.92 spid14s A new instance of the full-text filter daemon host process has been successfully started.
2012-07-10 20:30:00.02 spid18s The full-text filter daemon host process has stopped normally. The process will be automatically restarted if necessary.

When SQL Server indexes data types such as varbinary, varbinary(max), image, or xml, SQL Server sends data to the filter daemon process (FDHOST.exe). At any point during the indexing process, SQL Server will not wait for more than 60 seconds for the FDHOST.exe process to respond.

For large XML documents, the XML Filter that is hosted by the FDHOST.exe process reads in all the data from SQL Server and stores the data in a temporary file. Then FDHOST.exe processes the XML content for filtering and word-breaking. If this process takes more than 60 seconds, SQL Server stops the batch and retries the operation by using a smaller batch size. If an XML document is large, and the FDHOST.exe process takes more than 60 seconds to filter and break the words, then you can experience the issue above.

Resolution

To resolve this problem, increase the full-text indexing time-out value. To do this, call the sp_fulltext_service stored procedure that has the ‘ft_timeout’ parameter. For example, the following increases the full-text indexing time-out to 20 minutes for any document:

Exec sp_fulltext_service ‘ft_timeout’, 1200000

The second parameter is full-indexing time-out in milliseconds.
Restart your SQL Server for the new setting to take effect.

System.ServiceModel.ServiceActivationException: The service ‘/SecurityTokenServiceApplication/securitytoken.svc’ cannot be activated due to an exception during compilation.

January 7, 2014 Leave a comment

Problem
I had performed an in-place upgrade of a Team Foundation Server from Windows Server 2012 to Windows Server 2012 R2.
Overall, no issues detected until a couple of weeks later when all the Developers came back to work (after XMas break) and informed me that Documents were not available
via the Visual Studio 2012 application. The following error was occurring in Visual Studio;
Please contact your administrator. There was an error contacting the server.Technical information (for administrator):  HTTP code 200: OK

So off to the TFS Server it was….

Research
First stop was the Event Viewer and there were two errors that I believe were related and occurring.
Error 1 – Event ID 3 System.ServiceModel 4.0.0.0
WebHost failed to process a request.
Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/4032828
Exception: System.ServiceModel.ServiceActivationException: The service ‘/SecurityTokenServiceApplication/securitytoken.svc’ cannot be activated due to an exception during compilation.  The exception message is: Exception has been thrown by the target of an invocation.. —> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. —> System.ArgumentNullException: Value cannot be null.
Parameter name: certificate
at System.IdentityModel.Tokens.X509SecurityToken..ctor(X509Certificate2 certificate, String id, Boolean clone, Boolean disposable)
at System.IdentityModel.Tokens.X509SecurityToken..ctor(X509Certificate2 certificate)
at Microsoft.SharePoint.Administration.Claims.SPSecurityTokenServiceManager.ConfigureTokenHandlerCollection(SPSecurityTokenServiceManager manager, SecurityTokenHandlerCollectionManager collectionManager, String key, SecurityTokenHandlerCollection& tokenHandlerCollection) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenServiceConfiguration.ConfigureTokenHandlerCollectionForLocalIssuer(SPSecurityTokenServiceManager manager, SecurityTokenHandlerCollectionManager collectionManager, String key) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenServiceConfiguration..ctor()
— End of inner exception stack trace —
at System.RuntimeTypeHandle.CreateInstance(RuntimeType type, Boolean publicOnly, Boolean noCheck, Boolean& canBeCached, RuntimeMethodHandleInternal& ctor, Boolean&bNeedSecurityCheck)
at System.RuntimeType.CreateInstanceSlow(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean skipCheckThis, Boolean fillCache, StackCrawlMark& stackMark) at System.Activator.CreateInstance(Type type, Boolean nonPublic) at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark) at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes) at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceHostFactory.CreateSecurityTokenServiceConfiguration(String constructorString) at Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses) at Microsoft.SharePoint.IdentityModel.SPSecurityTokenServiceHostFactory.CreateServiceHost(String constructorString, Uri[] baseAddresses) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.CreateService(String normalizedVirtualPath, EventTraceActivity eventTraceActivity) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.ActivateService(ServiceActivationInfo serviceActivationInfo, EventTraceActivity eventTraceActivity) at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity)
— End of inner exception stack trace —
at System.ServiceModel.ServiceHostingEnvironment.HostingManager.EnsureServiceAvailable(String normalizedVirtualPath, EventTraceActivity eventTraceActivity) at System.ServiceModel.ServiceHostingEnvironment.EnsureServiceAvailableFast(String relativeVirtualPath, EventTraceActivity eventTraceActivity)
Process Name: w3wp
Process ID: 5664

Error 2 – Error ID 8306 SharePoint Foundation
An exception occurred when trying to issue security token: The requested service, ‘http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc’ could not be activated. See the server’s diagnostic trace logs for more information..

Error 3 – Error ID 6398
The Execute method of job definition Microsoft.Office.Server.UserProfiles.LMTRepopulationJob (ID 1b0c4725-fbcf-476d-af60-3aeabbdbd35c) threw an exception. More information is included below.
System.ServiceModel.ServiceActivationException

The common problem here appeared to me to be with relation to the SecurityTokenServiceApplication, which can be sussed out within IIS…
First I check that the Application Pool was configured with correct TFS Account and started…check
Next went to browse the SecurityTokenServiceApplication web page itself (IIS Manager –> Sites –> SharePoint WebServices –> SecurityTokenServiceApplication, click on ‘Content View’ down at the bottom, right click on Securitytoken.svc and click Browse)…ERROR
Basically you get a ‘Server Error in ‘/..’ Application + Error 1 above, or Internet Explorer cannot display the webpage etc.

From here I knew that the only way to fix this was to focus on the SecurityTokenServiceApplication web service but I wasn’t really sure where to start except that I knew this would be easy with PowerShell. :)

Thanks to gurus such as Syed and Abhishek Saigal, this is what fixed my issue.

Resolution
The below PowerShell commands re-provisions all the SharePoint Web Services.

Don’t worry about losing any data/applications on SharePoint, all will remain intact.

Run the following commands one by one on SharePoint PowerShell:
$h = Get-SPServiceHostconfig
$h.Provision()
$services = Get-SPServiceApplication
foreach ($service in $services) { $service.provision(); write-host $service.name}

The output will take a little time and display each service one after another, patiently wait until it finishes.

Perform an IIS Reset and give another shot to browsing ‘http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc’
This page then displayed correctly, e.g. no error messages
Documents then worked within Visual Studio.

NOTE: If there is a problem running the PowerShell commands the following may assist..
With the SharePoint Management Shell, you do not have to register the snap-in that contains the cmdlets. Registration of the Microsoft.SharePoint.PowerShell.dll module for

SharePoint 2013 cmdlets is automatic, as a result of the Add-PSSnapin Microsoft.SharePoint.PowerShell line in the SharePoint.ps1 file that is located in %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\15\Config\PowerShell\Registration. To use the Windows PowerShell console, you must register this snap-in manually.

IIS problem after inplace upgrade to Windows 8.1 – HTTP Error 503. The service is unavailable.

January 2, 2014 2 comments

Problem

After upgrading a machine with Windows 8.1 I had troubles with an ASP.NET application running in IIS. The error was as follows;

Service Unavailable
HTTP Error 503. The service is unavailable.

Explanation

The AppPool died and showed a status of Stopped.
The IIS log had nothing.
In the Windows Application log was the following error;

The Module DLL C:\WINDOWS\system32\inetsrv\rewrite.dll failed to load. The data is the error.

Resolution

An uninstall and reinstall of the rewrite module was required.

Go to Programs and Features and uninstall IIS URL Rewrite Module 2

Reinstall it from http://www.iis.net/downloads/microsoft/url-rewrite

Restart the IIS Service.

Server 2012 R2 / Windows 8.1 – Refresh completed with one or more warning messages

December 6, 2013 2 comments

Upgraded a Server 2012 to Server 2012 R2 (in-place upgrade).
In Server Manager there was a persistent error;
“Refresh completed with one or more warning messages”.

When I double click the Automatic Refresh item to look into the details I get the message;
“%ServerName%: Configuration refresh message: The system cannot access one or more event logs because of insufficient access rights, file corruption or other reasons. For more information, see the Operational channel in the ServerManager-ManagementProvider error log on target server.”

So far as Microsoft are concerned this could be one of the following 3 issues;
1. Insufficient Access Rights;
Please make sure you are the member of the local group administrator. Please right click the icon of the server manager then select run as administrator.
2. File Corruption
Yeah, ok, nothing really specific here
3. Other Reasons
What a cop out that is :/

I was Domain Administrator and ran Server Manager with administrative priveleges however it still happened.

I then went into Event Viewer, specifically to ‘Custom View’ -> ‘Administrative Events’ which threw a query error;
“microsoft-windows-dxptaskringtone/analytic the system cannot find the file specified”

Resolution:

Open regedit: press Win+r and input regedit
Backup Up registry first!

Go to: [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WINEVT\Channels] Delete “Microsoft-Windows-DxpTaskRingtone/Analytic”

Restart  “Windows Event Log” service.

Note:

This is the same for Windows 8.1.

This registry setting is deprecated in Server 2012 R2 and Windows 8.1 so this error occurs when doing in-place upgrades. A fresh install of either operating systems will not run into this issue.

Windows 8.1 – Sysprep Preparation Tool 3.14. A fatal error occurred while trying to sysprep the machine.

November 21, 2013 Leave a comment

I was using Windows Server 2012 WDS (Windows Deployment Services) to create a capture image that I will be using for future client installs, one that has all required updates, drivers, printers, applications etc. required by our business.

I was doing all of this in  VMware and had built the perfect reference machine with Windows 8.1, Office 2013 etc.

The final step in the process is to sysprep the vm and then boot it from the capture (boot) image to take a copy however I got the following error towards the end of the sysprep process; “Sysprep Preparation Tool 3.14. A fatal error occurred while trying to sysprep the machine”.

Now this stuffs up the reference machine, so luckily I had taken a snapshot prior to sysprep and could easily revert and troubleshoot the sysprep issue. (don’t make the mistake of not snapshotting).

This appears to be a problem with relation to the Windows 8.1 Metro Apps and I resolved by doing the following (NB. critical to observe the sysprep log file (setuperr.log);

  1. Take Snapshot
  2. Run Sysprep: sysprep /generalize /oobe /shutdown
  3. When the error occurs, check the error log and then query the problem package using PowerShell (insert name of package into the “”). Get-AppxPackage -AllUsers | Where Name -Like “”
  4. Once you have it run the following to remove the package; Get-AppxPackage -AllUsers | Where Name -Like “” | Remove-AppxPackage
  5. Repeat steps 3 and 4 until successful.

If you don’t care whatsoever for Metro Apps (which is the case in our environment) you could just run the following and remove all packages; Get-AppxPackage | Remove-AppxPackage

ShoreTel Voicemail stopped emailing when new Exchange Server installed

November 21, 2013 Leave a comment

I installed a new Exchange 2013 Server into our network and decommissioned the old Exchange server, the new server had a different IP address.

Obviously voicemail and mailing of recorded phonecalls stopped working so I changed the IP address within Director and IIS. Still not working.

Besides the obvious of having the correct Exchange IP in Director and IIS, the SMTP service itself needs to be configured with correct IP. This has only become an issue due to the Exchange Server changing its IP and the SMTP Service being configured initially with old IP.
To rectify I had to do the following;
•    From the Run prompt, run mmc.
•    Add the snap-in ‘Internet Information Services (IIS) 6.0 Manager.
•    Drill down and right-click the SMTP Virtual Server (e.g. [SMTP Virtual Server #1]) and select properties.
•    Click on the Delivery tab and then the Advanced button.
•    Change the Smart Host details to correct IP.

Voila! I got a shitload of emails.

VMware vCenter Converter Standalone tweak for error (converter.fault.FileIOFault) and performance

October 17, 2013 1 comment

Problem

I have had some with regards to converting Hyper-V VM’s to VMware Infrastructure VM’s using the VMware vCenter Converter Standalone, and one of the errors (that I remember) was FAILED: An error occurred during the conversion:
‘converter.fault.FileIOFault’.

I have also experienced very slow conversions of these Hyper-V VM’s, for example 2-3 days.

Explanation

Converter uses NFC (a proprietary VMware protocol) for cloning to managed destinations. Security was enhanced by encrypting the data, however this has a significant performance affect and in turn can cause errors with regards to IO at various stages of the conversion.

Resolution

Switching off SSL encryption has prevented all errors and performance is vastly better (e.g. instead of 2-3 days now conversions are taking 2-3 hours). Just do the following;

Open the converter-worker.xml configuration file. It is located in “%ALLUSERSPROFILE%\VMware\VMware vCenter Converter Standalone” folder for Windows Vista or newer or in “%ALLUSERSPROFILE%\Application Data\VMware\VMware vCenter Converter Standalone” for older Windows versions.

Search for the block of code starting with <ssl> and comment out the following code block;

<ssl>
<!– Private key file –>
<privateKey>ssl/rui.key</privateKey>
<!– Certificate file –>
<certificate>ssl/rui.crt</certificate>
</ssl>

Also change the following;

<nfc>
<readTimeoutMs>120000</readTimeoutMs>
<useSsl>false</useSsl>

Restart the ‘VMware vCenter Converter Standalone Worker’ service.

Restart the conversion :)

Follow

Get every new post delivered to your Inbox.