Always On availability group(s) – Rolling upgrades

Aim: To upgrade/migrate (side-side) SQL Server 2014 Availability group(s) running on Windows Server 2012 R2 to SQL Server 2019 running on Win 2016 with the least amount of downtime.

Couple of years ago, I wrote a blog post explaining how to upgrade Windows OS from 2012 R2 to 2016 on nodes participating in fail over clustering with minimal downtime using rolling upgrade technique. In this blog post I will be sharing something similar but throwing SQL Server availability groups into the mix. So let me briefly explain what we are trying to achieve here.

Current environment:
I’ve a two node Failover cluster (Windows Server 2012 R2) hosting SQL Serevr 2014 Always On Availability Group with synchronous commit mode. I have a listener configured for my applications to connect. These replicas are running on the latest build of SQL 2014 as of the date this post is published.

As you can see, W12SQL2016A/B are my two replicas(Nodes) which are running Win2012R2+SQL 2014.

Originally I thought of Installing SQL 2016(hence the host names), but ended up installing SQL 2014 for now based on our specific requirement. I didn’t want to change the host names as I had my windows Fail over cluster all setup by this time and I really don’t want to deal with fixing any annoying errors that might popup because of messing up my host names of my nodes. Anyways…the bottom line is I have SQL 2014 AG running on Win 2012R2 which needs to be upgraded/migrated to SQL 2019 running on Windows 2016.

Listener Name: RestAGL

To upgrade these SQL Instances to 2019  running on windows server 2016 with a very minimal downtime and no configuration changes for the App teams, assuming In-place upgrades are not allowed.

What’s the high level plan:

Take Full Backups.

  • Add W16SQL2019A and W16SQL2019B nodes to the same windows cluster leveraging mixed mode.
  • Install SQL 2019 and add these two nodes as replicas at SQL Server AOAG layer.
  • Join the databases and let the magic happen.
  • During the final cutover date/time, failover to SQL 2019 and remove the old replicas from AG.
  • Evict both windows 2012R2 nodes from the cluster and raise the functionality level to 2016.

Now, let’s see this in action one step at a time.

Below is the screenshot of all my SQL Instances which I will be working on. To begin with I have two brand new SQL Server 2019 standalone Instances(W16SQL2019A and W16SQL2019B), on which I just enabled HADR feature.

Let’s go, I added the new Win2019 nodes to the existing windows failover cluster which is running on 2012 R2 functionality level.

Note: You don’t want to run in mixed mode of WSFC for long periods. Microsoft might not support if you stay in mixed mode for more than 4 weeks. This is only to perform rolling upgrades to make your systems really highly available. Wrap up the entire process in a day or two and be done with it.

This is expected.  For more details on this, hop on to the blog post that I provided in the beginning of this blog post.

Now it’s time to jump into SQL Server to add these servers as replicas into our AG.

Awesome, so far so good 🙂

Let’s move on….Connecting to one of the SQL 2019 instances, below is what I have. Oops!!

I had to join the individual databases manually to the AG from both the newly added SQL 2019 replicas. Once that is done, below is how my dashboard looked like.

Also, I changed the failover mode to manual just to make sure cluster has no control over failing over my AG. I want to have total control over how and when to failover my AG till the entire upgrade process is complete. Hey BTW, did you take Full Backups?

Did I mention, I have a table called “McD” in “American” database with one row in it? See below…

Now comes the fun part. Set one of the SQL 2019 Instances availability mode to Synchronous commit and perform a controlled manual failover. In my case, I selected W16SQL2019A on which I changed it to Synchronous mode and failed over my AG from W12SQL2016A(Which is my current primary) to W16SQL2019A .

Awesome, At this point, W16SQL2019A took over the primary role all your databases participating in your AG have been upgraded to SQL 2019 and the other SQL 2019(W16SQL2019B in my case) Instance will be in sync from now on, but the two SQL 2014 Instances will be in unhealthy state, In fact those databases become inaccessible at this time, since Logs can’t be shipped from higher(2019) to lower(2014) version. Duh!!!!!….

Perfectoo! Also, I have my table and data intact, double perfectoo!

If you are curious, this is how the error log looked like. You can clearly see, the internal database version is getting upgraded from SQL 2014 all the way to 2019.

Below is a screenshot showing what to expect on old SQL instances after failing over AG to newer version.

Time to do some clean up now. I removed both SQL 2014 Instances from AG as replicas and boom……PRESTO!

The only thing left now is to take care of WSFC by evicting old windows server 2012 R2 nodes and raise the functional level of the cluster to come out of mixed mode.

That’s it folks. Hope this is helpful, Cheers!

Installing SQL 2019 on LInux (RHEL/CENTOS)

Wish you all a very happy new year folks!  Been a while since I posted something, so here I am 🙂  I would like to begin my blogging in year 2020 with Linux. Long ago, I wrote an article on how to setup a VM with CentOS and install SQL 2017 on top of it. I am not going to walk through those steps again., Instead let’s see if the setup experience is any different from SQL 2017 on RHEL.

Before begin installing, Let me examine repos on my linux machine. you can do that by navigating to /etc/yum.repos.d folder.

Well, this is a brand new installation. That explains all the repositories I have on my machine! Now, It’s time to download SQL Server repo from microsoft.
sudo curl -o /etc/yum.repos.d/mssql-server.repo

Once this part is done, it’s time for actual installation.
sudo yum install -y mssql-server

Alright, let’s perform the configuration portion(EULA,Edition,sa pwd) as instructed by setup program.
sudo /opt/mssql/bin/mssql-conf setup

Oops! My configuration failed. you can clearly see the reason why my installation failed. I was way too conservative in terms of granting resources to my VM. Once I bumped up my RAM to 4 GB, all went well.

Check status using either ps-ef | grep mssql-server or systemctl command as shown below.

Now it is time to install client tools on the server. Microsoft has provided a separate repo for client tools which we need to download similar to what we did in step 1.
sudo curl -o /etc/yum.repos.d/msprod.repo
Note: Check RHEL version (7 or 8) and download appropriate repos.

Don’t forget to add sqlcmd and bcp tools to path to avoid providing literal path every time we need to access sqlcmd. I hope microsoft will take care of this as part of post installation in future releases instead of manual intervention. ( Symbolic links may be??)

Testing what we have done so far…..

That’s it for this blog post folks! As you can see the installation experience hasn’t been changed much from the last version of SQL on Linux!

Note: This is online setup experience, assuming your Linux machine can talk to internet. Microsoft did a great job on documenting online and offline installation methods. I would definitely recommend anyone to go through the documentation for more details.

Happy learning!

SQL Server on Docker containers – Welcome to the present!

In this article let’s see how to install a SQL Server Instance using a Docker container in Windows server 2016.

“Containers” is probably one of the hottest buzz words talked about these days in IT world, especially Docker containers. So, what exactly is Docker? Well, Docker is one of the few companies who develops, builds and distributes container technologies. It’s a pioneer in container technologies, similar to VMWare in virtualization technologies. I won’t be going into internals and architecture of how docker container works under the hoods in this article, instead lets just focus on installing docker containers inside windows server 2016 and then setup the latest version of SQL Server Instance on top of it.

One can install dockers in windows 10 (should be enterprise or professional with anniversary update or later and Hyper-V enabled.) or in Windows server 2016, In our case we will be using Windows Server 2016 for this purpose.

Installing Docker services on windows server 2016

In order to install Docker suite, first we have to enable containers feature in windows server 2016 and install Microsoft management package provider for docker. Okay, let’s get into action. To begin with, I created a Windows server 2016 Virtual machine and enabled direct internet access to the VM.
For complete article, see below…


Simulating a Multi Subnet cluster for setting up SQL Server Always On Availability Groups – lab setup

In this article, we are going to see how to create a multi subnet cluster spanning across multiple subnets for lab purposes. Creating such an environment should help creating Availability groups simulating a far replica residing in a different Data Center (Subnet) acting as a disaster recovery site in your lab for learning/experimenting real world scenarios.

Let’s get into action, below is what we are going to setup for our lab purposes to simulate a multi subnet cluster environment and create an Always On Availability group.

  • Two nodes (Replicas) residing in my production/primary Data center (Subnet 192.168.1.x) – Let’s say these two replicas are used for High availability, assuming we have low latency, since they are residing in the same data center (Probably in the same rack), I will set them up with Synchronous Mode with Automatic Failover
  • Third node (Replica) sitting in my disaster recovery data center which is geographically dispersed (Subnet 192.168.2.x) – Assuming it’s connected over WAN and is used just for DR purposes, I would set this replica in Asynchronous Mode with Manual Failover

For more reading…go to below link.

SSMS 18.0 – SQL Server Priority Boost

I’ve been playing with SQL Server 2019/SSMS 18.0 since past few days and noticed something really awesome, there’s no more “Boost SQL Server Priority” option is SSMS 18.0 !!!! It’s probably one of the most requested feature in SSMS by DBA community.

The name “Boost SQL Server Priority” sounds like an amazing switch which makes SQL run on steroids, but in reality it’s the other way around.

So, what exactly does this option do?
Ans: It doesn’t matter in year 2019, Let’s just pretend we never had this option 🙂  Well, If you are still curious what this is about, enabling this setting will make SQL Server process run in “HIGH_PRIORITY_CLASS”. Almost every process in windows run in “NORMAL_EXECUTION_CONTEXT” including network drivers, Storage drivers, I/O drivers and etc etc etc. In other words, you are making SQL Server run in higher execution context on CPU than almost any other process on your server.

So, when do we need to enable this on our SQL Servers?
Ans: Never. There’s a reason why microsoft removed this option from SSMS.

Enough background, see what I am talking about in the below screenshots.


SQL Server 2019 CTP 2.3 is here…

Microsoft announced SQL Server 2019 CTP 2.3 in SQL bits (If you are not aware of what SQL bits is, it’s European version of SQL PASS) Keynote on Friday, first of March 2019. I really didn’t got chance to Install SQL 2019 initial release of CTP when it got released sometime around  Sep 2018, In this blog post let’s take a quick Screenshot tour of Installing SQL Server 2019 CTP 2.3 on Windows Server 2016.

First thing first, we need to download media from Microsoft.

In this case, I am installing on Win Server 2016, So I picked Option 1.

As highlighted in the above screenshot, I opted to download ISO file and have a copy for myself so that I can use this to Install on a different machine in future without any hassles.  Well, It’s all done in few mins…

Now, let’s go to the actual Installation part.



4. I am interested in just the DB Engine for now.

5. Provided my service accounts

6. Time to configure Data Directories and don’t forget to visit TempDB tab.

7. Next—-> Next and Done!

Nothing fancy here with Installation. Now, I downloaded and Installed latest release preview of SSMS which supports SQL Server 2019 and below is the output of @@Version.

Note: If you have previous releases of 2019 CTP installed on the same machine, you can’t perform an Inplace upgrade to CTP 2.3, However you can have them side by side without any issues. In other words, It’s not required to uninstall previous release of CTP in order to Install 2019 CTP 2.3 as a different Instance.

BTW, my favorite feature of this release is “Accelerated DB Recovery and Instant Rollback for long running transactions”. There are few other new features added related to SSAS and Big data Clusters as well, but I am really not interested in those at this moment.


Merry Christmas – From SQL Server

Below is the script which you can run directly in your SSMS and SQL Server (With the power of  Spatial results) sends you Christmas wishes directly on your screen 🙂

USE tempdb

-- Prepare the scene

CREATE TABLE #ChristmasScene


item VARCHAR(32)

) ;

--Put up the tree and star

INSERT INTO #ChristmasScene

VALUES ( 'Tree',

'POLYGON((4 0, 0 0, 3 2, 1 2, 3 4, 1 4, 3 6, 2 6, 4 8, 6 6, 5 6, 7 4, 5 4, 7 2, 5 2, 8 0, 4 0))' ),

( 'Base', 'POLYGON((2.5 0, 3 -1, 5 -1, 5.5 0, 2.5 0))' ),

( 'Star',

'POLYGON((4 7.5, 3.5 7.25, 3.6 7.9, 3.1 8.2, 3.8 8.2, 4 8.9, 4.2 8.2, 4.9 8.2, 4.4 7.9, 4.5 7.25, 4 7.5))' )

--Decorate the tree


,@x INT

,@y INT ;

WHILE ( @i < 20 )


INSERT INTO #ChristmasScene

VALUES ( 'Bauble' + CAST(@i AS VARCHAR(8)),

GEOMETRY::Point(RAND() * 5 + 1.5, RAND() * 6, 0).STBuffer(0.3) )

SET @i = @i + 1 ;


--Christmas Greeting

INSERT INTO #ChristmasScene


'POLYGON((0 10, 0 11, 0.25 11, 0.5 10.5, 0.75 11, 1 11, 1 10, 0.75 10, 0.75 10.7, 0.5 10.2, 0.25 10.7, 0.25 10, 0 10))' ),

( 'E',

'POLYGON((1 10, 1 11, 2 11, 2 10.8, 1.25 10.8, 1.25 10.6, 1.75 10.6, 1.75 10.4, 1.25 10.4, 1.25 10.2, 2 10.2, 2 10, 1 10))' ),

( 'R',

'POLYGON((2 10, 2 11, 3 11, 3 10.5, 2.4 10.5, 3 10, 2.7 10, 2.2 10.4, 2.2 10, 2 10),(2.2 10.8, 2.8 10.8, 2.8 10.7, 2.2 10.7, 2.2 10.8))' ),

( 'R',

'POLYGON((3 10, 3 11, 4 11, 4 10.5, 3.4 10.5, 4 10, 3.7 10, 3.2 10.4, 3.2 10, 3 10),(3.2 10.8, 3.8 10.8, 3.8 10.7, 3.2 10.7, 3.2 10.8))' ),

( 'Y',

'POLYGON((4 11, 4.2 11, 4.5 10.6, 4.8 11, 5 11, 4.6 10.5, 4.6 10, 4.4 10, 4.4 10.5, 4 11))' ),

( 'C',

'POLYGON((0 9, 0 10, 1 10, 1 9.8, 0.2 9.8, 0.2 9.2, 1 9.2, 1 9, 0 9))' ),

( 'H',

'POLYGON((1 9, 1 10, 1.2 10, 1.2 9.6, 1.8 9.6, 1.8 10, 2 10, 2 9, 1.8 9, 1.8 9.4, 1.2 9.4, 1.2 9, 1 9))' ),

( 'R',

'POLYGON((2 9, 2 10, 3 10, 3 9.5, 2.4 9.5, 3 9, 2.7 9, 2.2 9.4, 2.2 9, 2 9),(2.2 9.8, 2.8 9.8, 2.8 9.7, 2.2 9.7, 2.2 9.8))' ),

( 'I',

'POLYGON((3.2 9, 3.2 9.2, 3.4 9.2, 3.4 9.8, 3.2 9.8, 3.2 10, 3.8 10, 3.8 9.8, 3.6 9.8, 3.6 9.2, 3.8 9.2, 3.8 9, 3.2 9))' ),

( 'S',

'POLYGON((4 9, 4 9.2, 4.8 9.2, 4.8 9.4, 4 9.4, 4 10, 5 10, 5 9.8, 4.2 9.8, 4.2 9.6, 5 9.6, 5 9, 4 9))' ),

( 'T',

'POLYGON((5 9.8, 5 10, 6 10, 6 9.8, 5.6 9.8, 5.6 9, 5.4 9, 5.4 9.8, 5 9.8))' ),

( 'M',

'POLYGON((6 9, 6 10, 6.25 10, 6.5 9.5, 6.75 10, 7 10, 7 9, 6.75 9, 6.75 9.7, 6.5 9.2, 6.25 9.7, 6.25 9, 6 9))' ),

( 'A',

'POLYGON((7 9, 7 10, 8 10, 8 9, 7.75 9, 7.75 9.3, 7.25 9.3, 7.25 9, 7 9),(7.25 9.5, 7.25 9.8, 7.75 9.8, 7.75 9.5, 7.25 9.5))' ),

( 'S',

'POLYGON((8 9, 8 9.2, 8.8 9.2, 8.8 9.4, 8 9.4, 8 10, 9 10, 9 9.8, 8.2 9.8, 8.2 9.6, 9 9.6, 9 9, 8 9))' ) ;

--Admire the scene


FROM #ChristmasScene -- Tidy up the pine needles and put away the decorations

DROP TABLE #ChristmasScene

After letting it run (Shouldn’t take more than a second), navigate to ‘spatial results’ tab in your results pane and voila…..

The total credit goes to the original author of this script. Unfortunately, I don’t have his/her name, I saved this script years ago in my tool set.

Wishing you all Merry Christmas and a very happy new year 🙂