Pages

Wednesday, May 23, 2018

Running Test-ExchangeServerHealth.ps1 Exchange Server health check script returns services as being down

Problem

You’ve been using the Test-ExchangeServerHealth.ps1 Exchange Server health check script for sometime but noticed that it has began falsely reporting the Client Access Server Role, Hub Transport Server Role, Mailbox Server Role and Mailbox Server Role services as being in a Fail state:

image

Solution

One of the possible causes of the incorrect report of the status for the services is if WinRM is not started on the Exchange server.  A way of determining this is to execute the Test-ServiceHealth <serverName> then review the output:

image

Note how the last line for each service displays {ServicesNotRunning: WinRM}

WinRM will be reported as not running regardless of whether the Test-ServiceHealth cmdlet is executed remotely or locally on the server.

To correct the issue, log onto the affected Exchange server and ensure that the Windows Remote Management (WS-Management) service is started:

imageimage

Restarting the service and rerunning the health check PowerShell should now report the services as being up:

imageimage

Sunday, May 20, 2018

Configuring an on-premise Exchange 2016 OWA with SecurEnvoy for 2fa causes webpage to load with the error: "HTTP Error 403.18 - Forbidden"

Problem

You’ve downloaded the latest SecurEnvoy Version 9.1.501 package as of May 2018 from:

https://www.securenvoy.com/support/downloads.shtm

Then used the following guide to configure your on-premise Exchange 2016 OWA access for 2FA:

Microsoft Outlook Web Access 2013 - SecurEnvoy
https://www.securenvoy.com/IntegrationGuides/Microsoft/Outlook-Web-Access-2013.pdf

… but receive the following error when attempting to access the Outlook Web App page after enabling SecurEnvoy 2FA:

HTTP Error 403.18 - Forbidden
The specified request cannot be processed in the application pool that is configured for this resource on the Web server.
Most likely causes:

· An ISAPI filter or custom module changed the URL to run in a different application pool than the original URL.

· An ISAPI extension (or custom module) used ExecuteURL (or ExecuteRequest) to run in a different application pool than the original URL.

· You have a custom error page that is located in one application pool but is referenced by a Web site in another application pool. When the URL is processed, it is determined by IIS that that it should have been processed in the first application pool, not the other pool.

· The Web site has multiple applications configured. The application this request is configured to run in is set to run in an application pool that does not exist.

Things you can try:

· If you have an application that is trying to process a URL in another application pool (such as trying to process a custom error), ensure that they both run in the same application pool if appropriate.

· If you are trying to process a custom error URL that is located in another application pool, enable the custom errors Redirect feature.

· Verify that the application pool for the application exists.

· Create a tracing rule to track failed requests for this HTTP status code and see if ExecuteURL is being called. For more information about creating a tracing rule for failed requests, click here.

Detailed Error Information:

Module

   IIS Web Core

Notification

   BeginRequest

Handler

   SecurEnvoy MS Server Agent

Error Code

   0x00000000

Requested URL

   https://<webmailURL>:443/securenvoyauth/webauth.exe?action=auth&dir=WEBAUTHTEMPLATE&ip=7C91BFF7D8EBAB9B9879278A1F44F11D92&redirect=https://tmrbmexmb02/owa/

Physical Path

   C:\Program Files (x86)\SecurEnvoy\Microsoft Server Agent\WEB\webauth.exe

Logon Method

   Not yet determined

Logon User

   Not yet determined

More Information:

This error occurs if the application pool for the request does not exist, or if an ISAPI filter, ISAPI extension or HTTP module calls the ExecuteURL server support function (or ExecuteRequest) with a URL that is configured in a different application pool. Due to security reasons, a Web site in one application pool cannot make ExecuteURL requests against a URL in another application pool. If you have an application that is trying to process a URL in another application pool, ensure that they both run in the same application pool if appropriate.

View more information »

image

Server Error

403 - Forbidden: Access is denied.

You do not have permission to view this directory or page using the credentials that you supplied.

image

Solution

One of the possible causes of this error is if the MSExchangeOWAAppPool for the IIS server on the Exchange 2016 server is configured incorrectly. I’ve only configured SecurEnvoy 2FA with OWA 2016 once so I am unsure as to whether this is a common issue because the deployment guide (https://www.securenvoy.com/IntegrationGuides/Microsoft/Outlook-Web-Access-2013.pdf) does indicate this as a requirement but it is labeled as a note:

image

To verify that the parameter is configured correctly, launch the Internet Information (IIS) Manager on the Exchange server, navigate to the SecurEnvoyAuth virtual directory:

image

Right click on the SecurEnvoyAuth node, navigate to Manage Application and then select Advance Settings…:

image

If the Application Pool is configured as DefaultAppPool then change it to MSExchangeOWAAppPool:

image

imageimageimage

The page should now load with the SecurEnvoy customizations:

image

Note that the above screenshot shows that the images are missing, which is another issue I will blog about in another post.

Thursday, May 17, 2018

Attempting to remote desktop to Windows server fails with the error: "An authentication error has occurred. The function requested is not supported"

Problem

You attempt to use a Windows 10 workstation to remote desktop to a server but notice that the connection fails with the following error message:

An authentication error has occurred.

The function requested is not supported

Remote computer: <computerName>

This could be due to CredSSP encryption oracle remediation.

For more information, see https://go.microsoft.com/fwlink/?linkid=866660

image

Reviewing the System logs on the client will show that the following error is logged:

A CredSSP authentication to TERMSRV/server.fqdn.com failed to negotiate a common protocol version. The remote host offered version 3 which is not permitted by Encryption Oracle Remediation.

See https://go.microsoft.com/fwlink/?linkid=866660 for more information.

image

Using a Windows 7 workstation to perform the same operation would display the following error message:

An authentication error has occurred.

The function requested is not supported

Remote computer: <computerName>

image

Solution

The cause of this error is explained in detail in the following TechNet blog post: https://blogs.technet.microsoft.com/mckittrick/unable-to-rdp-to-virtual-machine-credssp-encryption-oracle-remediation/

The short answer is that a patch was released in May 2018 that addresses a vulnerability issue with the Credential Security Support Provider protocol (CredSSP) and if you have patched your workstation with this patch but have not done the same for the server then this error would be displayed. Note that patching the server but not patching the workstation would not cause this issue.

There are several ways to work around this and they are:

Workaround #1 – Disable NLA on Server

Disable Allow connections only from computers running Remote Desktop with Network Level Authentication (recommended) for the server:

image

Workaround #2 – Uninstall Patch

Another way is to uninstall the patch from the workstations. The patch to uninstall for Windows 10 is:

KB4103723

image

The patch to uninstall for Windows 7 is:

KB4103712

image

You can use the following cmdlets to search:

Get-HotFix | Where HotfixID -match "4103712"

… or with the following to uninstall the patch:

wusa.exe /uninstall /kb:<KB Number>

**Refer to this blog post for using PowerShell to search for installed hotfixes: http://terenceluk.blogspot.com/2014/10/handy-get-hotfix-windows-powershell.html

Workaround #3 – Adjust Encryption Oracle Remediation

The new configuration that causes this error can be located on the patched workstation’s Computer Configuration / Administrative Templates / System / Credentials Delegation:

image

You can temporarily disable this by changing Encryption Oracle Remediation policy to Enabled, and Protection Level to Vulnerable:

image

Recommended Solution

The three of these work arounds is *not* recommended as they are temporary measures and does not address the vulnerability. The recommended way of addressing this would be to install the corresponding update on the server:

KB4103725

imageimage

Wednesday, May 16, 2018

Skype for Business Peer-to-Peer Session Detail Report reports: "No media quality data is available." for the "Media Quality Report"

Problem

You attempt to retrieve information about a bad call reported by a user so you launch the Skype for Business Monitoring Reports, drill down to the Peer-to-Peer Session Detail Report reports, expand the Media Quality Report section but noticed that No media quality data is available. Is displayed and there is no data logged:

image

Reviewing the event logs on the Skype for Business Front-End server reveals that the following error is logged:

Log Name: Lync Server

Source: LS Data Collection

Event ID: 56407

Level: Error

Failed to execute a stored procedure on the back-end.

Component: QoE Adaptor

Stored Procedure: QoeInsertSessionReport2

Error: System.Data.SqlClient.SqlException (0x80131904): Trying to pass a table-valued parameter with 109 column(s) where the corresponding user-defined table type requires 101 column(s).

at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)

at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)

at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)

at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption)

at System.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean async, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)

at System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

at System.Data.SqlClient.SqlCommand.InternalExecuteNonQuery(TaskCompletionSource`1 completion, String methodName, Boolean sendToPipe, Int32 timeout, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry)

at System.Data.SqlClient.SqlCommand.ExecuteNonQuery()

at Microsoft.Rtc.Common.Data.DBCore.Execute(SprocContext sprocContext, SqlConnection sqlConnection, SqlTransaction sqlTransaction)

ClientConnectionId:26e8fcf1-c35d-4284-adf6-7bfa82b60d24

Error Number:500,State:1,Class:16

Cause: Configuration issues, an unreachable back-end or an unexpected condition has resulted in the error.

Resolution:

Verify the back-end is up and this Skype for Business Server has connectivity to it. If the problem persists, notify your organization's support team with the relevant details.

image

Solution

One of the possible causes to this would be if you have a version mismatch between the ExpectedVersion and InstalledVersion for the QoEMetrics database. To determine whether this is the cause, execute the following cmdlet:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn <SQLserverHostingMonitoringDatabase>

Note the difference between the ExpectedVersion and InstalledVersion for the QoEMetrics database:

ExpectedVersion: 62.93.12

InstalledVersion: 62.93.8

image

In the event that the environment does have a mismatched version for SQL, execute the following cmdlet:

Install-CsDatabase -DatabaseType Monitoring -SqlServerFqdn <SQLserverHostingMonitoringDatabase> -DatabasePaths "Z:\Data\MonitoringStore\(default)\DbPath","Y:\Logs\MonitoringStore\(default)\LogPath"

**Replace the paths with the appropriate paths to the database and logs

A similar output will be displayed:

image

Execute the following cmdlet again to confirm that the database no longer has a mismatched version:

Test-CsDatabase -ConfiguredDatabases -SqlServerFqdn <SQLserverHostingMonitoringDatabase>

Note the matching versions between the ExpectedVersion and InstalledVersion for the QoEMetrics database:

ExpectedVersion: 62.93.12

InstalledVersion: 62.93.12

image

With the mismatched database version corrected, the Media Quality Report section will now have data recorded:

image

Monday, May 14, 2018

Dialing into Polycom hosted meeting with Skype for Business Server fails with: "Previous hop server component did not report diagnostic information";Domain-"

Problem

You’ve configured Skype for Business integration with Polycom RealPresence Collaboration Server 1800 to allow Skype for Business clients to join into scheduled Polycom meetings but notice that an attempt to dial into the meeting would ring but the Polycom does not answer. Performing a logging session reveals the following entries:

SIP/2.0 500 Server Internal Error

Previous hop server component did not report diagnostic information";Domain=”internalPolycomDomain.com”;PeerServer=”yourPolycomDMAserver.com”;source=”yourSfBFEserver.com”

image

image

Solution

I encountered this error after several components of the Polycom conferencing server were moved from one datacenter to another and the error messages provided by the trace did not help because it did not point me to the right direction. What did was logging into the RMX Manager and reviewing the Signaling Monitor status, which revealed the following:

Relay Server UDP Not Available

Relay Server TCP Not Available

image

What ended up causing this issue was that the relocated RMX server was no longer able to reach the Skype for Business Edge Server (not the front-end server) and reconfiguring the firewall to allow the required ports corrected the issue.

Friday, May 11, 2018

Attempting to move a mailbox from one mailbox database to another in Exchange 2016 stops with the StatusDetail: StalledDueToSource_MailboxCapacityExceeded

Problem

You’ve initiated a mailbox move request from one mailbox database to another (cross a WAN link in this example) but noticed that the migration stops / halts with the StatusDetail StalledDueToSource_MailboxCapacityExceeded:

image

image

Suspending and resuming the migration does not restart the process.

Solution

While there are various reasons why the mailbox migration would halt, one of the things to try to restart the process is to identify the source server that hosts the mailbox and restart it.  Once the source server has been restarted with the services back up for a while, review the status again and verify it is either displaying InitialSeeding or CopyingMessages:

image

Thursday, May 10, 2018

Attempting to use the Invoke-Command PowerShell cmdlet to remotely install an application hangs indefinitely without starting the installation

Problem

You’ve enabled PSRemoting on a remote server or desktop and attempt to use the Invoke-Command PowerShell cmdlet to remotely install an application with the following cmdlet:

Invoke-Command -computername wkst-122 -ScriptBlock {Start-Process -FilePath 'C:\temp\agent\VMware-viewagent-x86_64-6.2.1-3284564.exe' -ArgumentList '/S /V"/qn' -Wait}

image

… but the cmdlet executes and hangs indefinitely without starting the installation on the remote computer.  You do notice that the msiexec.exe process is briefly displayed on the target computer’s task manager when the cmdlet is executed but it disappears after a second.

Solution

The first two obvious items to check are:

  1. Ensure PSRemoting is enabled
  2. The installation file is not on a UNC share as it must be copied locally on the target computer

The one item that is not as obvious and had me stumped for an hour is to ensure that the executable is not blocked. 

If you are copying the executable from a file server then executing the install command then ensure the executable on the file server is unblocked and if you already have the file on the target computer then ensure that the file is unblocked.

image