AlwaysOn Availability Groups

How to move AlwaysON AG Databases ?

Back in 2011, we have seen how to move a database which is participating in DB Mirroring here. In this blog post, let’s see how to move a database which is participating in a AG to a new drive(location). In my AG setup,I’ve three replicas(2 near Replicas-Sync mode and 1 far replica sitting in a different data center-Async mode). The database which I will be moving to a new location is “sales”. See below for current paths.


Now we shall move these files to below mentioned new location (In this post, let’s see method two mentioned below):


Method one:

Remove the database from AG.
Detach the database.
Move the files physically to new location(s).
Attach the database.
Rejoin the database in your AG.

Method Two:
In this method we shall see how to move files without removing the database from AG.

Steps to perform in sequence:

Suspend data movement for the database which you are working on to all replicas.
Logically move the database files(On all the Replicas).
Stop the SQL Server services. – This step will create an outage for all other databases residing on this instance.
Now move the physical MDF and LDF files to your new location.
Start the SQL service from config manager.
Resume data movement.

Before proceeding any further, I made sure the new path exists on of my  replicas.



2. Run this on all the replicas.


Now…I stopped SQL Services and moved physical files to new locations and started SQL Server.

3. Now resume data movement.


Voila…Now it’s all set as per my requirement.


That is it folks for today! Have fun…

Lab setup – AlwaysOn AGs in a Multi Subnet Cluster – Part 2

In part1, I’ve shown how to create a windows cluster in a multi subnet setup.  In this post let’s see how to create a AG and corresponding listener.

In my lab, I will be creating two AGs and two corresponding listeners.

I’ve two databases – > sales and customers.
Two AGs – > Sales_AG and Customers_AG.Two Listners – > sqllst_Sales and sqllst_Cust.
For sales AG, I’ve disabled “Database level health detection”, a new feature introduced in SQL 2016.



Now, for Listener two IPs from both subnets have been provided.



Same process has been followed for creating customers AG and listener as well(But this time I’ve enabled Database level health detection).


Since this is a multi-subnet setup two entries(one from each subnet) will be created in DNS for each listener name as shown below.


That’s about it folks.

Lab setup – AlwaysOn AGs in a Multi Subnet Cluster – Part 1

Let’s see how to setup an AG(SQL 2016) in a multi subnet cluster(Geo cluster) in a lab environment.

Below is my lab setup:

Two Replicas sitting in my Production Data center.(Subnet 192.168.1.x) – Sync Mode Automatic Failover.
Third(Far) Replica sitting in my DR Data Center.(Subnet 192.168.2.x) – Asynch Mode Manual Failover.

So, What do we need to be able to setup multiple subnets and routing in a lab environment? Answer is “Routing and Remote Access“. Have that installed by going to Add Roles/Features on your AD/DNS server.

Pre-req Step: Created 2 NICs on my SANDC machine with IPs and

Open Routing and remote access config tool; right click on the root node and select “Enable and Configure Routing and remote access”.

Now…under IPv4, under General right click and select new routing protocol and select “RIP Version 2 for Internet Protocol”.


Now right click on RIP and select new interface and select your NIC1 and hit okay and next repeat the same step and select NIC2 this time and click okay. You are done with routing…That’s all you need for routing to work(As long as you got all the IPs and DNS details right).





Now, I’ve setup 3 nodes(two nodes(Prd) in 1.x and one(DR) in 2.x) and installed Failover Cluster feature on all the nodes and disabled all firewalls.


Do find all my NIC settings from all my nodes at the very end of this post.

Now, Let’s create Windows Cluster:

Please refer to my earlier posts on how to create a cluster under “Clustering category” for detailed steps. Below are the steps at a high level.




Now…My cluster is ready, but it’s missing Quorum which is very critical for a cluster to be healthy. For that I’ve created a File share witness as Quorum.



FYI, NIC settings from Cluster manager are shown below.



NIC settings on all of my nodes:



STLSQLAG2: ( 1 NIC Card)


AZSQLAG3(DR Server): 1 NIC Card


AD/DNS Server: ( 2 NICs one for 1.x and other for 2.x)



In this post we’ve seen how to setup a geo cluster in a lab environment. So, this completes the prep work needed from Windows stand point…Let’s see how to create AGs and Listeners in our Multi Subnet environment in next part of this series.

SQL Server 2016 AlwaysON AG setup – Part 2

Now that I’ve created a windows cluster and installed SQL server on all the three nodes(Part 1), I am ready to configure AGs. Below is the screenshots tour.


You can see  a new option “Database Level Health Protection” (In CTP releases It had “Basic AG”and “Per Database DTC Support” options as well.). Now, on Enterprise/Eval/Developer editions, all we get is above option. If you are trying to setup AG on standard edition(Wait till RTM gets released), then You should be seeing an option for Basic AG(Well, only Basic AG can be configured on standard edition)

So, what’s DB level health protection? If one of your databases in your AG goes offline or becomes unavailable for some reason, entire AG doesn’t get impacted.




Here’s the bonus, Now we get 3 Synchronous(Automatic) replicas and Active secondaries are Load balanced with your routing list and also distributed transactions are now supported in 2016 AGs 🙂

Now, pick your backup preferences.


And finally configure Listener:


Below are the corresponding CNOs/VCOs/DNS records created in my AD.

From my SSMS:


That’s it folks! Happy weekend…

SQL Server 2016 AlwaysON AG setup – Part 1

This is a quick guide on how to configure AlwaysON AGs on SQL Server 2016. This blog post isn’t a thorough walk through, My goal in this post is to show you guys what’s different in SQL 2016 comparing with 2012 or 2014. (For detailed information on how to setup AGs in your own lab, please refer to my previous posts on SQL 2012).

First and foremost, I’ve built a windows cluster with 3 nodes(No shared Storage) for setting up AGs(See below).


Starting from Node SQLTrekAG1, Inserted SQL Server 2016 RC2 media, hit setup.exe…basic sanity checks…next next…blah blah.


One thing you might have already noticed, all our management tools and Data tools has a separate link to install. Well, This is a huge change(I would say improvement, YMMV), in earlier versions we had option to select Client tools when selecting features and IIRC data tools was a separate download all together in SQL 2014.



As mentioned earlier we no longer see client tools(basic and complete) in features selection.



One more amazing enhancement, now it always remind DBAs about IFI 🙂



Choose your dirs and if you go to tempdb tab, you can see that now SQL Server is smart enough to identify number of CPUs on the box and will choose number of data files for your tempdb.

Everything else should be similar to what we had in previous versions. Now after performing a standalone installation successfully, you can see SSMS hasn’t been installed.


Now let’s install SSMS on this machine:

All I had to do was double click setup.exe and on Installation tab, select “Install SQL Server Management Tools”and……Oops!Bummer….! Did I already mention, you need to have Internet connection to be able to do this. Well, see below screenshot carefully again.


It downloads an 815 MB file(SSMS-Setup-enu.exe), now that’s the media for installing SSMS and other SQL utilities! Store it in your media library, from now on we don’t need internet connection.

Note: You can directly go to for downloading the same(As of 2016 RC2).

Run that executable…I got a warning about shared components already installed(I am good with that) and next to Install. That’s it!

It installs .NET framework 4.6.1(took a while), Report viewer, batch parser, SSMS, SQLCMD, BCP, SMOs, Visual studio 2015 Shell(took its sweet time), SQLPS,Profiler,DTA and all those goodies. Restart if required.

Now, I enabled AGs from SQL Config manager.


I will install SQL by performing the same on all the three nodes and will come up with part-2 later where I setup an actual AG. Cheers!

AlwaysON Secondary database going to “Not Synchronizing/ Suspect” State!

In this blog post I will share an issue we had with a database which is configured with AlwaysON. Before proceeding any further, the environment which we’ve got is:

Each node has Windows Server 2008R2(With all the service packs and hot fixes recommended for AlwaysON)
Running on top of VMware VShpere 5.1
SQL Server 2012(SP1) Enterprise Edition
RAM: 10 GB (8 GB assigned to SQL Server).
2 VCPU’s.
Availability Mode- Synchronous Commit

Issue: Daily around 5 AM, the secondary database is going to “Not Synchronizing/Suspect” state and until we fix this the T-Log on primary grows and all that normal jazz once the AlwaysON databases get out of Sync…(See below)

So, what’s happening?
The App team is performing data load daily around 4.30 AM. Okay…So what’s bad about that? They are loading ~30 Million Records daily, in a single transaction. Oops!!!…
From SQL Server error logs, we see the below message:

AlwaysOn Availability Groups data movement for database ‘Test_DB’ has been suspended for the following reason: “system” (Source ID 2; Source string: ‘SUSPEND_FROM_REDO‘). To resume data movement on the database, you will need to resume the database manually. For information about how to resume an availability database, see SQL Server Books Online.
This message is always accommodated with another message(Shown below):
The instance of the SQL Server Database Engine cannot obtain a LOCK resource at this time. Rerun your statement when there are fewer active users. Ask the database administrator to check the lock and memory configuration for this instance, or to check for long-running transactions.

Ummm…This doesn’t looks good. If you are thinking, what Locks have to do with AlwaysON secondaries, let me tell you this. With Database Mirroring and AlwaysON Rollbacks/Redo thread will also take a lock on the secondary side to avoid any other transaction to interrupt REDO process, thus guaranteeing consistency. If for some reason SQL Server is not able to acquire locks for redo thread it won’t synchronize the database starting that point. (It’s by design).

In our case what’s happening was SQL Server was running out of memory and was not able to acquire any further locks(Remember, each lock structure in SQL Server will need certain amount of memory).Basically, it says “Since I wasn’t able to acquire a lock during the REDO, I don’t know what else happened at that time and I can’t guarantee the database to be consistent. So…am not going to synchronize from this point and I will suspend the data movement and also take the database to Suspect state”).

From AlwaysON standpoint, Suspending Synchronization when the REDO thread encounters any error is by design and is done on purpose by SQL Server.

To avoid this, all they(App team) have to do is optimize their load process to better manage lock acquisition.(We are not being granted any more memory on these boxes unfortunately).

Bottom Line: Avoid huge transactions on tiny SQL Servers. Try to split the transactions into multiple chunks especially when dealing with millions/billions of rows.  That helps in general many ways, not just in this particular scenario.

Have a safe and happy long weekend guys!


SQL Server AlwaysOn Availability Groups Terminology….!

In SQL Server 2012, with AlwaysOn being introduced there are lot of new terms/words which we need to get used to as we support SQL Sever 2012. Well, Just getting used to those terms is not enough…we’ve to understand the terminology. In this short Post let me define what I’ve understood so far with these new fancy terms.

Availability Groups(AG) : – Group of Databases which move together from one Instance to other. Each Instance can have multiple AGs, where each AG contains multiple Databases.

Listener :- It’s a virtual entity which moves around to the current Primary Server for a given AG.

Replicas :- All the SQL Servers involved in your AG are considered as Replicas. Even current Primary Server is treated as a Replica, not just the secondaries…! So, replicas are not Just secondary SQL Instances. They are differentiated by using Primary Replica and Secondary Replica.

Note: AGs run on top of Windows Clustering. So is it a new clustering?? Nope! Same old Windows Clustering, but with a flavor of no Shared Storage. FYI, SQL Instances in my lab are 3 Standalones which are built on top of a 3 Node Windows Cluster with no shared Storage!

Note: your SQL Server Instances can be clustered as well, which adds more complexity, but is needed for some customers based on their own business needs. Typically, MSFT calls this scenario as AlwaysOn Failover Clustered Instance.

AlwaysOn Failover Cluster is not same as AlwaysOn Availability Groups!