Kerberos Event ID 4, KRB_AP_ERR_MODIFIED

Long time since last post, but this one is something I feel like I have to share.

Recently experienced a issue after upgrading en environment from Exchange Server 2013 to Exchange Server 2019.
After installing the new server, some users startet to experience the Outlook client asking for password repeatedly. The problem was not consistent accross all devices/users, so it was somewhat troublesome to troubleshoot.

The key to the solution appeared after a client repeatedly reported Event ID 4 from Security-Kerberos service:

The kerberos client received a KRB_AP_ERR_MODIFIED error from the server %1. The target name used was %3. This indicates that the password used to encrypt the kerberos service ticket is different than that on the target server. Commonly, this is due to identically named server accounts in the target realm (%2), and the client realm (%4). Please contact your system administrator.

Turns out that the SPN of the e-mail service and autodiscover in the local domain was not updated automatically to the new Exchange 2019 Server.

By using this article from Microsoft, I was able to locate the SPN pointing to the old Exchange server, remove the SPN and replace it with the correct server address. When completed, all Outlook clients startet functioning as normal.
https://learn.microsoft.com/en-us/troubleshoot/windows-server/windows-security/kerberos-event-4-access-denied
The syntax for the setspn command is as follows:
setspn -Q HTTP/mail.domain.com (alt. HTTP/autodiscover.domain.com)
setspn -D <SPN> LOCALDOMAIN\OLD_ServerName$
setspn -A <SPN> LOCALDOMAIN\NEW_ServerName$

Skype for Business Server 2015 Updates and Build Numbers.

In this post I will try to keep you updated with the current releases and build numbers for Skype for Business 2015 Server.

As of now, there has been released eighteen updates for Skype for Business Server 2015. The Microsoft download page is updated with the latest SkypeServerUpdateInstaller, so the link will always point to the most recent CU.

Server updates and Build Numbers:

August 2021 Update – Build number 6.0.9319.619 (CU12)

July 2020 Update – Build number 6.0.9319.591(CU11HF1)

May 2020 Update – Build number 6.0.9319.580(CU11)

August 2019 Update – Build number 6.0.9319.562(CU10HF1)

July 2019 Update – Build number 6.0.9319.559(CU10)

May 2019 Update – Build number 6.0.9319.548(CU9)

Januar 2019 Update – Build number 6.0.9319.537(CU8)

March 2018 Update – Build number 6.0.9319.519(CU7)

January 2018 Update – Build number 6.0.9319.514(CU6 HF1)

December 2017 Update – Build number 6.0.9319.510(CU6)

May 2017 Update – Build number 6.0.9319.281(CU5)
Be adviced: CU6 has been reported to break Web Conferencing Edge service due to updates to the .Net Framework.
Check out this TechNet article if you experience problems.

February 2017 Update – Build number 6.0.9319.277(CU4 HF1)

November 2016 Update – Build number 6.0.9319.272(CU4)

June 2016 Update – Build number 6.0.9319.239(CU3)

March 2016 Update – Build number 6.0.9319.235(CU2)

November 2015 Update – Build number 6.0.9319.102 (CU1)

This post will update as new CU’s are released.

Unable to set Global Teams Upgrade mode to UpgradeToTeams.

When you are moving from an on-premise Skype for Business organization to Teams Only in M365, the final step on the way would be to switch your organizational setting to Teams Only in Coexistence mode.

This is done after you move all users from Skype for Business on-prem and decommission the Skype for Business server infrastructure and remove all your on-prem servers. So, there is nothing left on-premise whatsoever(make sure that this is the case).

If you are hosting multiple custom domains in your tenant, you have to make sure that public DNS is pointed to M365 and Skype for Business Online/Teams.

The following lists the records you need to change:

Modify	Sip	CNAME	sipdir.online.lync.com
Modify	lyncdiscover	CNAME	webdir.online.lync.com
Modify	_sipfederationtls._tcp	SRV	0 0 5061 sipfed.online.lync.com
Modify	_sip._tls	SRV	0 0 443 sipdir.online.lync.com

When you try to change the Coexistence mode in the Teams Admin portal, you might experience that the setting will not save. No further info is displayed.

The remote powershell command for setting the Coexistence mode will display the following error message:

“This organization cannot be upgraded to TeamsOnly at the tenant level because there is an on-premise deployment of Skype for Business detected in 1 or more of it sip domains”.

Check the public DNS for the domains in question and make sure that the DNS points to M365. When all domains are pointed to M365, you should be able to set Coexistence mode to Teams Only either in the Teams Admin portal or by running the powershell command in a remote powershell session:  

Grant-CsTeamsUpgradePolicy -PolicyName UpgradeToTeams -Global 

The end result showing in this image:

Coexistence mode set to Teams only for the entire organization

Skype for Business server 2015 and Event ID 30009

Sometimes the user database in Skype for Business can get out of sync for different reasons. When this happens, users can experience different issues like not being able to sign in to Skype for Business for new users etc.

A typical error message for the user indicates something wrong with DNS in the domain for instance.

In the Event log on the FrontEnd server, EventID 30009 starts showing up telling you there is something wrong with user replication.

The solution to this problem is quite simple, but still can be challenging to figure out.

When you run the Get-CsUserReplicatorConfigurator, the cmdlet returns the ADDomainNamingContextList. This parameter tells Skype For Business in which domain to search for users. If you Skype For Business SIP domain is the same as your internal domain, that’s the naming context that should be listed here.

However, if your SIP domain is different from your internal domain, you must make sure that it’s the internal domain that is listed under this parameter. Any other domains should be removed.

The PowerShell commands to clean up and get it right:

Set-CsUserReplicatorConfiguration -Identity global -ADDomainNamingContextList @{remove=”dc=domain, dc=com”}

Set-CsUserReplicatorConfiguration -Identity global -ADDomainNamingContextList @{add=”DC=domain,DC=internal”}

The changes should pick up automatically and stuff should start working again.

Microsoft Teams and Realtime Traffic, How VPN is affecting user Experience.

In these challenging times with a lot of users WFH, there is one thing that comes up as an issue in many cases.

Why are our users experiencing bad Audio and Video quality in Teams (or Skype for Business)?

Many companies has designed their remote access solutions with VPN dependencies, and this has been a perfect solution in many years to secure access to company data when users are on the move. However, times are changing, and with the increasing use of Microsoft Teams (replacing Skype for Business in many cases) problems arise.

This is Microsoft’s recommendations regarding realtime traffic and VPN connections:

VPN

VPNs are typically not designed or configured to support real-time media. We recommend you provide an alternate path that bypasses the VPN for Teams traffic. This is commonly known as split-tunnel VPN. Split tunneling means that traffic won’t traverse the VPN but will go directly to Teams. This change will have a positive impact on quality, but also provides the secondary benefit of reducing load from the VPN devices and the organization’s network.

To implement a split-tunnel, consult with your VPN vendor for the configuration details.

These recommendations also apply for Skype for Business where in use.

Troubleshooting Exchange Event ID 4002 from MSExchange Avalability.

This blogpost is about a strange incident I had with a fresh Exchange 2016 two node DAG.

The environment is a virtual VmWare environment. The case was a customer case where I was hired to migrate from a working Exchange 2013 environment to a new Exchange 2016 deployment. The customer had a relatively simple setup with a single AD site and nothing more.

I installed the new Exchange servers, and configured the environment accordingly setting up DAG and configuring mailflow etc. Proceeded with the pilot users and did some testing to confirm the environment was Ok. Everything checked out Ok, and the customer moved all users from Exchange 2013 to 2016. The 2013 servers were decommissioned and everything was Ok.

After a couple of months we suddenly experienced free-busy problems. The users with mailbox on one node would not be able to see free-busy from users on the other node.. This started happening out of the blue without no changes being done in the environment. We also started to see Event ID 4002 in the server logs on the server trying to do free-busy lookup.

Process 17932: ProxyWebRequest CrossSite from S-1-5-21-1409082233-1343024091-725345543-35887 to https://dagmember02.domain.com:444/EWS/Exchange.asmx failed. Caller SIDs: NetworkCredentials. The exception returned is Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequestProcessingException: Proxy web request failed. —> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. —> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. —> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

— End of inner exception stack trace —

at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)

at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)

— End of inner exception stack trace —

at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)

at Microsoft.Exchange.InfoWorker.Common.Availability.Proxy.RestService.EndGetUserPhoto(IAsyncResult asyncResult)

at Microsoft.Exchange.InfoWorker.Common.UserPhotos.UserPhotoApplication.EndProxyWebRequest(ProxyWebRequest proxyWebRequest, QueryList queryList, IService service, IAsyncResult asyncResult)

at Microsoft.Exchange.InfoWorker.Common.Availability.ProxyWebRequest.EndInvoke(IAsyncResult asyncResult)

at Microsoft.Exchange.InfoWorker.Common.Availability.AsyncWebRequest.EndInvokeWithErrorHandling()

— End of inner exception stack trace —

. Name of the server where exception originated: dagmember01. LID: 43532. Make sure that the Active Directory site/forest that contain the user’s mailbox has at least one local Exchange 2010 server running the Availability service. Turn up logging for the Availability service and test basic network connectivity.

I immediately started to search for a possible solution to this strange behaviour, but there was no working solution to be found anywhere(read through most of the posts on this event ID on the internet :)) All other services on the Exchange environment were working fine, there were no other error messages in the logs indicating something wrong, just this Event ID 4002 from time to time when people where trying to add someone to a meeting using Scheduling assistant.

After quite a while of research and asking a couple of colleagues, the solution suddenly appeared.

A colleague of mine asked me wether or not the customer used templates to create the VM’s. After checking this with the customer, we could confirm this. He then told me that he had been experiencing some strange similar behaviour on Exchange 2007 some years ago, and asked me to check if the servers had unique SID’s. I did so, and discovered that both the new Exchange 2016 servers had identical SID’s. The tool used was the PsGetSID from Microsoft Sysinternals.

Turned out that the servers were created from a template in VmWare and not sysprepped. After removing one of the servers and reinstalling it, everything started working fine again.

Bottom line is:

If your Exchange servers start acting weird and there doesn’t seem to be a logical explanation to the problem, check the server SID’s. They have to be unique, or obviously, strange things can start to happen in your environment. In my case, there was no obvious reason to the problems that suddenly started to appear and the server setup was made in good faith 🙂

This might bee a noob fault, but I can imagine that someone else but me would have experienced this or other strange problems with no logical explanation, so I think the tip would be useful in case everything else leads nowhere.

The weird part here is the servers functioning 100% ok for a couple of months before the problems started. I’ve never experienced this before, so for all I know that’s how Exchange handles this kind of misconfiguration?

Skype for Business not able to share PPT presentation with federated participants.

When you experience problems sharing PPT content in a Skype for Business meeting, the reason could be a variety of different factors not beeing implemented in the correct way.

There are many ways to publish the Office Web App server necessary to enable this feature. One of those are the IIS ARR web proxy.

In this case the proxy is set up and everything is working fine with all web services, and the officewebapp URL test(https://officewebapp.yourdomain.com/hosting/discovery) is also returning the correct XML page. However, when an internal user uploads a PPT to the meeting, the federated participant experiences just a spinning “Loading” and the presentastion does not load correctly in the client. Other internal users are able to see the presentation in the meeting.

Solution:
Remove response header added from IIS on Revers Proxy server, Open Internet Information Services (IIS) Manager on IIS ARR server.

• In the Connections pane on the left side, expand the Sites folder and select the site(Default Web Site).
• Double-click the HTTP Response Headers icon in the feature list in the middle.
• Highlight “X-Frame-Options” In the Actions pane on the right side, click Remove.
• Click OK to save your changes.
• Run iisreset from an elevated commandprompt.

After Removing the X-Frame-Options the presentations renders perfectly from the OfficeWebApp server for external participants as well.

Unable to log in to O365 using ADFS and sub-domains.

I recently came across a problem in a customer tenant regarding sign in for sub-domain users.

ADFS is enabled and sign-in for regular users with primary domain as UPN functions as expected. When a user with a UPN of a sub-domain tries to sign in, the following error message appears in the browser(my apologies for the norwegian text):

AADSTS50107: Requested federation realm object ´{Email hidden}´ does not exist.

The solution involves editing a regular expression in an ADFS claim rule which is used to build the IssuerUri associated with the user’s UPN. It’s described in this article on TechNet and boils down to changing a regEx value.

This is how I did it:

In the AD FS Console:

• Trust Relationships\Relying Party Trusts
• Right-click on ‘Microsoft Office 365 Identity Platform’ and choose “Edit Claim Rules…”
• In the “Issuance Transform Rules” tab, highlight “Issue issuerid when it is not a computer account” then choose “Edit Rule…”
• Take a copy of the existing rule as a backup and then change to this value:

  c:[Type == "http://schemas.xmlsoap.org/claims/UPN"] => issue(Type = "http://schemas.microsoft.com/ws/2008/06/identity/claims/issuerid", Value = regexreplace(c.Value, "^((.*)([.|@]))?(?<domain>[^.]*[.].*)$", "http://${domain}/adfs/services/trust/"));

I found that replacing the entire expression with the one listed above solved the situation immediately, letting the sub-domain users sign in as they were supposed to.
Be aware that this approach could possibly break other stuff in the ADFS service, so you should have a backup of your existing settings to be restored if everything goes south.

Microsoft Teams Direct Routing GA

msunified.net

Today Microsoft Teams Direct Routing was announced as General Available. This is the means for you to bring your own SIP trunk to Microsoft Teams using only a standard SBC. Today AudioCodes and Ribbon are certified SBC’s for Direct Routing and more are in the works. There are three flavors to Direct Routing

Hosted in Azure!

Yes you read correct. AudioCodes has a certified SBC that now is supported in Azure, which means you can run you Direct Routing SBC in Azure as an appliance.

• AudioCodes Virtual Edition SBC is certified for Direct Routing
• Check out the AudioCodes whitepaper on how to install their SBC in Azure.
  • https://www.audiocodes.com/media/13250/mediant-virtual-edition-sbc-for-microsoft-azure-installation-manual-ver-72.pdf
• As long as the SBC is certified and supported by the vendor to run in Azure, it is implied that Microsoft will support it as well, no official Microsoft link at the moment

Installed in your datacenter connected to your PBX or SIP…

View original post 237 more words

Skype for Business, “SIP/2.0 504 Server time-out” when trying to federate.

I recently came across a small chalenge which maybe is nothing to write about, but I choose to anyway as I came across a few solutions to this error message while I was investigating it.

The problem is the following message in the client log when trying to federate in a fresh Skype for Business on-prem environment.

Message:
SIP/2.0 504 Server time-out

ms-diagnostics: 1034;reason=”Previous hop federated peer did not report diagnostic information”;Domain=”partnerDomain.com”;PeerServer=”accessedgeFQDN.partnerDomain.com”;source=”accessedgeFQDN.yourdomain.com”

After some back and forth and checks of firewall rules and port openings, in addition to going over the topology a few times, I stumbled across the solution(which should have been pretty obvious to start with). It turned out that the SRV records for the domain had been registered with typo’s.

Solution:

Make sure you have the correct DNS entries registered in public DNS for your domain zone.
SRV records should be in the format of _sipfederationTLS._tcp.yourdomain.com weight 0 priority 0 port 5061 host accessedge.yourdomain.com.
Make sure you enter just the _sipfederationTLS._tcp part if you do this manually, as the domain name will be appended automatically in the DNS zone.

When you do a Nslookup -q=srv _sipfederationTLS._tcp.yourdomain.com, it should resolve to your access Edge FQDN in public DNS.