Monday, January 23, 2017

Lync Server 2013 / Skype for Business Server 2015 federated contacts listed as “Presence unknown”

Problem

Companies federated with your Lync / Skype for Business environment have recently noticed that the status contacts in your organization briefly show as Updating…:

image

… then eventually changes to Presence unknown:

image

You’ve confirmed that the services on the Edge server are all Running:

image

However, reviewing the event logs show numerous errors logged in reference to the Lync Standard Server:

image

Log Name: Lync Server

Source LS Web Conferencing Edge Server

Event ID: 41987

Level: Error

Web Conferencing Server connection failed to establish.

Over the past 30 minutes Lync Server has experienced incoming TLS connection failures 120 time(s). The error code of the last failure is 0x80096004 (The signature of the certificate cannot be verified.

) and the last connection was from the host "".

Cause: This can occur if this box is not properly configured for TLS communications with remote Web Conferencing Server.

Resolution:

Check your topology configuration to ensure that both this host and remote Web Conferencing Server can validate each other TLS certificates and are otherwise trusted for communications.

image

Log Name: Lync Server

Source LS Protocol Stack

Event ID: 14428

Level: Error

TLS outgoing connection failures.

Over the past 86 minutes, Lync Server has experienced TLS outgoing connection failures 15 time(s). The error code of the last failure is 0x80096004(TRUST_E_CERT_SIGNATURE) while trying to connect to the server "svrlyncstd02.domain.internal" at address [10.1.1.66:5061], and the display name in the peer certificate is "Unavailable".

Cause: Most often a problem with the peer certificate or perhaps the host name (DNS) record used to reach the peer server. Target principal name is incorrect means that the peer certificate does not contain the name that the local server used to connect. Certificate root not trusted error means that the peer certificate was issued by a remote CA that is not trusted by the local machine.

Resolution:

Check that the address and port matches the FQDN used to connect, and that the peer certificate contains this FQDN somewhere in its subject or SAN fields. If the FQDN refers to a DNS load balanced pool then check that all addresses returned by DNS refer to a server in the same pool. For untrusted root errors, ensure that the remote CA certificate chain is installed locally. If you have already installed the remote CA certificate chain, then try rebooting the local machine.

image

Log Name: Lync Server

Source LS Protocol Stack

Event ID: 14366

Level: Error

Multiple invalid incoming certificates.

In the past 480 minutes the server received 30 invalid incoming certificates. The last one was from host 10.1.1.66.

Cause: This can happen if a remote server presents an invalid certificate due to an incorrect configuration or an attacker.

Resolution:

No action needed unless the number of failures is large. Contact the administrator of the host sending the invalid certificate and resolve this problem.

image

Attempting to browse the the FQDN of the Lync Standard Server from the Edge server displays the following webpage with a certificate warning:

· If you arrived at this page by clicking a link, check the website address in the address bar to be sure that it is the address you were expecting.

· When going to a website with an address such as https://example.com, try adding the 'www' to the address, https://www.example.com.

For more information, see "Certificate Errors" in Internet Explorer Help.

image

Solution

One of the reasons why presence would stop working and the event ID errors above to be thrown is if the Lync Edge server’s internal network interface certificate was recently updated by the issuing Root certificate issuing the updated certificate is not installed onto the Lync Edge server.  Note that the Lync Edge server is never joined to the domain so if the internal network interface certificate is issued by an internal CA then the root CA certificate along with the chain must be manually imported into the Trusted Root Certification Authorities certificate store on the Edge server:

image

Once this has been completed, proceed to restarting the Lync Edge server’s services and confirm that the following informational event logs are written:

image

Verify that the following page is now displayed when you browse the the FQDN of the Lync Standard Server from the Edge server:

Server Error

403 - Forbidden: Access is denied.

You do not have permission to view this directory or page using the credentials that you supplied.

image

Thursday, January 19, 2017

HTTP 500 Internal Server Error is thrown when accessing the App Volumes Manager webpage after upgrading VMware App Volumes Manager from 2.6.0.1226 to 2.12.0.70

Problem

You’ve completed an upgrade of VMware App Volumes Manager from 2.6.0.1226 to 2.12.0.70:

clip_image002[4]

clip_image002

… by uninstalling the old version then reinstalling the new version using the same database but notice that launching the Apps Volume Manager now display the following HTTP 500 Internal Server Error:

image

clip_image002[7]

Reviewing the logs on the App Volumes Manager server in the folder:

C:\Program Files (x86)\CloudVolumes\Manager\log

Reveals quite a few large 200MB+ logs:

clip_image002[9]

Opening these logs file in Notepad++ shows the following error repeatedly logged:

[2017-01-17 00:25:04 UTC] P2236R698508  INFO Started GET "/cv_api/version" for 127.0.0.1 at 2017-01-16 20:25:04 -0400

[2017-01-17 00:25:04 UTC] P2236R698508  INFO Processing by CvApi::VersionsController#show as */*

[2017-01-17 00:25:04 UTC] P2236R698508 ERROR    Manager: Unhandled request exception: ODBC::Error: 42S02 (208) [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'ldap_domains'.: EXEC sp_executesql N'SELECT TOP (1) 1 AS one FROM [ldap_domains]'

[2017-01-17 00:25:04 UTC] P2236R698508 ERROR    Manager: Inspecting Array (21028620) (from log block)

clip_image002[11]

Solution

I’m not much of an expert with App Volumes so I called support to have an engineer review the logs and was told that the line:

[2017-01-17 00:25:04 UTC] P2236R698508 ERROR    Manager: Unhandled request exception: ODBC::Error: 42S02 (208) [Microsoft][ODBC SQL Server Driver][SQL Server]Invalid object name 'ldap_domains'.: EXEC sp_executesql N'SELECT TOP (1) 1 AS one FROM [ldap_domains]'

… indicates that a table in the SQL database named ldap_domains was missing.  Reviewing the database via Microsoft SQL Server Management Studio confirms the missing table:

image

The engineer classified this as a catastrophic error during the upgrade and said that it looks like many of the other tables were not created during the upgrade.  He wasn’t completely sure but the issue may have been due to the major jump in version and the schema changes between the versions.  At this point, the suggestion was to restore a database backup and attempt to reinstall or reinstall with a new database and reconfigure the configuration with the existing AppStacks.  This environment was only used by 20 users so I decided to perform a reinstall and reconfigure.

To ensure that the previous created AppStacks are imported, ensure that you select the same datastore containing all the AppStacks during the initial configuration after the new install:

image

image

image

image

image

Not exactly the best solution but since I’ve configured Active Directory groups to grant access to the applications, it was easier for me to go this route than try to perform a restore and then upgrade again:

image

Wednesday, January 18, 2017

Upgrading a Citrix NetScaler VPX HA pair via command line

Those who are familiar with the Citrix NetScaler’s administrative console would be familiar with the upgrade button in the Systems menu that allows the administrator to upload the upgrade package and have the appliance automatically apply the firmware update:

image

While this feature makes the upgrade process quite easy, I’ve also found that it is sometimes unreliable because the upgrade progress status window could freeze at a certain step and not update which then leaves us wondering if we should close and start over or continue waiting.  What I’ve typically done in the past is manually apply the update by uploading the package onto the appliance and using command line to execute the upgrade either via the console or SSH session.

Step #1 – Download and upload firmware package

Begin by logging onto the the Citrix website and download the upgrade package normally named similar to the following:

build-11.1-51.21_nc.tgz

image

Launch your preferred SFTP client such as WinSCP, connect to the secondary appliance and navigate to the following directory:

/var/nsinstall

image

Create a directory for the new package and copy the firmware into the directory:

image

image

Repeat the same for the active appliance.

Step #2 – Backup and save the NetScaler configuration

Log onto the active NetScaler’s administration console and proceed to backup and save the configuration:

image

image

The command save config could be used to save the configuration via the console or SSH session.

A hypervisor snapshot could also be created as well.

Step #3 – Unpack and install firmware upgrade

With the NetScaler backed up and upgrade firmware package uploaded, proceed with accessing the console or opening an SSH session to the secondary node, enter the shell mode by executing shell, navigate to the /var/nsinstall/<firmwareUpdate> directory then execute the following to extract the package:

tar -zxvf ns-x.0-xx.x-doc.tgz

For this example, the command to execute would be:

tar -zxvf build-11.1-51.21_nc.tgz

image

Once the files have been extracted, proceed to install by executing:

./installns

image

Continue and restart the appliance once the installation has completed.

image

Step #4 – Confirm HA status, confirm synchronization is disabled and force failover

With the secondary appliance upgraded and restarted, log back into the console and execute show ver to confirm that the version has been upgraded:

image

Continue and review the HA status by executing:

show ha node

… to confirm that this node is listed as secondary and synchronization is disabled:

image

**Note that both Sync State and Propagation is currently configured as AUTO DISABLED because build 51.21 automatically disables these settings during the upgrade.

In the event that synchronization is not disabled, execute the following to disable it:

set node -hasync disable

Execute show ha node again to confirm the status then force the failover:

force failover

image

image

Step #5 – Upgrade primary NetScaler node

Repeat the steps outlined in #3:

With the NetScaler backed up and upgrade firmware package uploaded, proceed with accessing the console or opening an SSH session to the secondary node, navigate to the /var/nsinstall/<firmwareUpdate> directory then execute the following to extract the package:

tar -zxvf ns-x.0-xx.x-doc.tgz

For this example, the command to execute would be:

tar -zxvf build-11.1-51.21_nc.tgz

Once the files have been extracted, proceed to install by executing:

./installns

Continue and restart the appliance once the installation has completed.

Step #6 – Verify upgrade of appliances and failover to original primary node

With the second appliance upgraded, proceed by logging onto the appliance and execute show ver to confirm that the version has been upgraded:

image

Execute the following command to check the status:

show ha node

image

Proceed by failing back the primary role back to the previous primary appliance with the command:

force failover

image

Step #7 – Enable synchronization on secondary appliance

Log onto the secondary appliance and execute the following to verify it is in secondary state:

show node

image

**Note that both Sync State is labeled as SUCCESS and Propagation is labelled as ENABLED because build 51.21 automatically enables these settings after the upgrade.

If Sync State and Propagation is not enabled then execute the following command to enable synchronization:

set node -hasync enable

Execute the following command to verify that the configuration of the secondary appliance is synchronized with that of the primary appliance

show ns runningconfig

Tuesday, January 17, 2017

Configuring Citrix NetScaler to load balance Exchange SMTP inbound connections

I’ve recently been involved with configuring a client’s Citrix NetScalers to load balance inbound SMTP connections to Exchange and thought I’d take this opportunity to blog the process.

#1 – Configure Exchange Server Objects

Begin by creating the Exchange Server objects in Traffic Management > Load Balancing > Servers:

image

#2 – Create SMTP Monitor

Create an SMTP monitor object by navigating to Traffic Management > Load Balancing > Monitors:

Name: EXCH_MONITOR_SMTP

Type: SMTP

Interval: 5 second

Response Time-out: 2 second

Destination Port: Bound Service

Down Time: 30 second

image

image

Click on the Special Parameters tab and configure the following:

Script Name: nssmtp.pl

Dispatcher IP: 127.0.0.1

Dispatcher Port: 3013

image

Note that the nssmtp.pl script bundled with the NetScaler will go as far as attempting to open a connection to confirm that the service is up.  The script and the actual code can be found in the following directory of the NetScaler:

/netscaler/monitors

image

#!/usr/bin/perl -w

################################################################

##

## Copyright 1998-2016 Citrix Systems, Inc. All rights reserved.

## This software and documentation contain valuable trade

## secrets and proprietary property belonging to Citrix Systems, Inc.

## None of this software and documentation may be copied,

## duplicated or disclosed without the express

## written permission of Citrix Systems, Inc.

##

################################################################

## This is a netscaler supplied script. Please dont modify this as it will be overwritten during

## reboot. If you want to modify, please make a copy of this script and modify.

## This script is used to do smtp monitoring using KAS feature.

use strict;

use Net::SMTP;

use Net::SMTP6;

use Netscaler::KAS;

sub is_ipv4_address

{

my $address = $_[0];

if ($address =~ m/^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)/) {

return 1;

}

return 0;

}

## This function is a handler for performing smtp probe in KAS mode

sub smtp_probe

{

## There must be at least 4 arguments to this function.

## 1. First argument is the IP that has to be probed.

## 2. Second argument is the port to connect to.

## 3. Timeout, it is present in index 3

if(scalar(@_) < 2)

{

return (1,"Invalid number of arguments");

}

## Try to connect to the server

my $smtp;

if (is_ipv4_address($_[0])) {

$smtp=Net::SMTP->new($_[0].":".$_[1],Timeout=>$_[3])

or return (1,"Unable to connect to server - $!");

## Probe succeeded

$smtp->quit;

return 0;

}

else { #IPV6 adress

$smtp=Net::SMTP6->new($_[0], PeerPort => $_[1], Timeout=>$_[3])

or return (1,"Unable to connect to server - $!");

## Probe succeeded

$smtp->quit;

return 0;

}

}

## Register smtp probe handler, to the KAS module.

probe(\&smtp_probe);

image

#3 – Configure Service Group

Proceed to configure the Service Group object by navigating to Traffic Management > Load Balancing > Service Groups:

Name: Exchange_2016_SMTP

Protocol: TCP

Cache Type: SERVER

image

image

Click on the No Service Group Member to add the Exchange server objects that were created in Step #1:

image

Click on the Monitors option on the right to add the SMTP monitor created in Step #2:

image

image

Complete the creation of the Service Group and you should now see the group listed with the State and Effective State as being up:

image

Step #4 – Create the Load Balancing Virtual Server

Continue to configure the load balancing virtual server object by navigating to Traffic Management > Load Balancing > Virtual Servers:

image

image

Add the Service Group created in Step #3:

image

Complete the creation of the load balancing virtual server and you should see State and Effective State listed as being up:

image

Step #5 – Lockdown Open Relay for Exchange Receive Connector

One of the common mistakes often overlooked when configuring SMTP load balancing via the NetScaler is inadvertently allowing open relay on the Exchange Server’s receive connector traffic coming from the NetScaler would appear to be an internal IP to the Exchange server.  One of the ways to test whether the receive connector allows for open relay is to execute the following commands via telnet:

telnet exchangeServerFQDN 25

220 EXMB02.contoso.com Microsoft ESMTP MAIL Service ready at Thu, 1
2 Jan 2017 14:20:03 -0400
ehlo bogus.com
250-EXMB02.contoso.com Hello [10.21.1.32]
250-SIZE 37748736
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250 CHUNKING
mail from:bogus@bogus.com
250 2.1.0 Sender OK
rcpt to:validperson@domain.com
250 2.1.5 Recipient OK

image

Note that the mail from email address has a domain that is not hosted on the Exchange server and the rcpt to address is meant to be an email address that is also not hosted on the Exchange server.  If the response to these commands is Recipient OK then your receive connect is allowing open relay.  To correct this, ensure that the receiving connector has the Externally secured (for example, with IPsec) setting disabled:

image

Once the connect has been locked down, the following response is what the telnet commands would yield:

220 EXMB01.contoso.com Microsoft ESMTP MAIL Service ready at Thu, 1
2 Jan 2017 14:15:47 -0400
ehlo bogus.com
250-EXMB01.contoso.com Hello [10.21.1.32]
250-SIZE 37748736
250-PIPELINING
250-DSN
250-ENHANCEDSTATUSCODES
250-STARTTLS
250-8BITMIME
250-BINARYMIME
250 CHUNKING
mail from:bogus@bogus.com
250 2.1.0 Sender OK
rcpt to:validperson@google.com
550 5.7.54 SMTP; Unable to relay recipient in non-accepted domain

image

Step #6 – Lockdown SMTP Load Balancing Virtual Server Connectivity

Another often overlooked issue that load balancing SMTP requests through a NetScaler creates is that the Exchange server’s receive connectors no longer see the true source IP address because all of the requests now originate form the NetScaler’s NSIP address which means a malicious or unauthenticated internal device could potentially relay mail off of the load balancing virtual server and be able to successfully have the Exchange server deliver the email.  This could be addressed by either configuring the Direct Server Return (DSR) feature on the NetScaler or simply locking down which IP addresses can connect to the load balancing virtual server.  I won’t cover the configuration of DSR and will point to one of my previous blog posts to demonstrate how to lock down the load balancing virtual server:

Filtering a Citrix NetScaler load balancing virtual server access based on source IP address
http://terenceluk.blogspot.com/2017/01/filtering-citrix-netscaler-load.html