SQL Won’t come up after Installing Service pack!!

Hello folks! In this blog post, I would like to share a strange experience which I’ve never experienced till today while Installing Service packs for SQL Server.

Well, I’m in the process of Installing new SQL Server(2012) on Windows Server 2012. Installed RTM…Everything went well. Now I kicked off applying Service pack 1 for SQL Server 2012. Erkk…..SQL services won’t come up and the event viewer says “Cannot recover the master database. SQL Server is unable to run. Restore master from a full backup, repair it, or rebuild it. For more information about how to rebuild the master database, see SQL Server Books Online.” and the one which is shown below!!!…

1

 “Script level upgrade for database ‘master’ failed because upgrade step ‘u_tables.sql’ encountered error 25641, state 0, severity 16. This is a serious error condition which might interfere with regular operation and the database will be taken offline. If the error happened during upgrade of the ‘master’ database, it will prevent the entire SQL Server instance from starting. Examine the previous errorlog entries for errors, take the appropriate corrective actions and re-start the database so that the script upgrade steps run to completion.”

Whatttt….?? Really? Nothing useful found in the SQL Error logs as well :( My immediate thought was to run Repair and I kicked off the repair wizard. It took its own sweet time. 25 mins passed and I got this stupid message:(

2

Agian, Nothing useful found in the SQL Error logs :( End result is it couldn’t fix it :(

3

This is not good…!

So what’s the work around/Fix for this issue which worked for me in this case?

I changed the Service account for SQL Server to Local System and Boom…it worked :) SQL Server DB Engine Service started and the build number has been updated as well! Once am done with Installing Service pack and the CU, I changed it back to the appropriate domain account(as it was earlier).

So, this sounds to me like some missing permissions issue but am not 100% sure what exactly it was looking for. Please let me know if you’ve encountered this issue in the past and what was the resolution.

SQL Server 2014 RTM released!

SQL Server 2014, MSFT’s cloud first data platform, released to manufacturing today(18th of march 2014) and will be generally available on 1st of April 2014 as per the SQL Server blog.

Alright, the wait is almost over and am waiting to get my hands dirty, exploring all the features 2014 has to offer, in real world work loads :)

Cheers!

Importance of reviewing your SQL Server Error Logs.

Few weeks ago I posted a poll here asking for how often do you review your Error logs. As promised, here comes the follow up post:)

Unfortunately, 45.45% out of 11 people said “Never and I’ll open Error log only If I want to investigate something” . IMHO, this is not a good practice(Especially if you want to be an outstanding DBA). Please don’t get me wrong…Am not saying that you are not an awesome and outstanding DBA! ( Well, People call you a DBA because you are already outstanding :) )

Okay…….let  me say this guys, Start reviewing your Error logs daily as the first thing when you get into your cubicle/desk/office. Trust me, there will be lot of surprises(at least a few) based on number of Servers(Instances I should say) you manage. SQL Server Error log is really one of the best buddies a SQL DBA has got for free besides default trace :) . I simply love these two things in SQL Server which provide so much useful information in case of investigating something(At no additional cost/overhead).

A simple real life use case: SQL error log records all the failed login attempts by default, if you keep an eye(or even Just glance over it) you should know if there are any unwanted/unsafe login attempts(Which are failed) hitting your SQL Instance. For example…I see quite a few times, incoming connections from a particular host attempting to connect as “sa”. Error Log records IP address of the machine and the time stamp of course. I can just do a NSLOOKUP and check with respective owner of the machine on what’s happening. This is just one useful scenario, there are tons of use cases which you can come up with!

Believe me guys, there will be lot of surprises for you and lot of things you’ll discover about your own SQL Server Instance which you don’t know, just by reviewing your error logs on daily basis for a week or two!

Most of the times, I hear people saying ” I am responsible for managing multiple Instances, I really don’t have time to get into each of my SQL Server and review error log. It will eat up my entire day“. Well, you really don’t have to do this. All you have to do is just setup a job and automate which basically reads all the error logs from all of your servers and generates a fancy report and sends you an email.(All you need is to just create a simple SSIS package and SSRS for fancy reports) or may be Powershell could help you.

If you just have handful of SQL Instances to manage, see this blog post.

Basically, This will allow us to be “Proactive” and not being “reactive”.  Preventing something bad happening to your database as opposed to fixing will really make you stand tall in front of your management and your peers. Am not saying, by reviewing Error logs daily you can literally prevent any issue from being happening. But you will be definitely able to reduce number of “OOPS…” moments!
IMHO, Even though part of DBA job is to troubleshoot/fix database related issues, our prime time goal as a DBA should be  trying to prevent those issues from occurring at the first place!
Let me know if you need any help automating this process. I will be glad to point you in the right direction.

Cheers! Happy friday :)

SQL Server Database Snapshots – Things to remember!

In this blog post, let’s see what SQL Server database Snapshots are and how they can be created and how to revert to a DB Snapshot and few important things to keep in mind before making use of them.

Microsoft introduced database snapshots in SQL Server 2005 as an enterprise only feature(I believe, that’s true even with SQL 2014…but I could be wrong). The basic concept of a database snapshot is to create a read-only, transactionally consistent(as of the moment of snapshot creation) static view of a given database. Even though they are named as “DB Snapshots”, they have got nothing to deal with Snapshot replication or Snapshot Isolation.

When could a DB Snapshot be useful?
Most common use case:  Let’s say your DEV/Testers are doing some testing(which makes whole bunch of changes to your database) in your test environment and they want the database to be reverted to a state how it was prior to the testing once they are done. A Snapshot works awesome in this case. Reverting is waaaay faster than restoring your database(especially when you are dealing with huge databases).

See this for more info on how they work behind the scenes, when will they grow in size, other use cases and Gotchas to remember while using Snapshots. Let me brief a few important things here.

  • DB Snapshots are not substitutes for your DB backups. Irrespective of whether you’ve snapshots or not, you should have healthy restorable backups in place. Don’t treat DB Snapshots as DR or HA solution.
  • Snapshots will introduce IO overhead, depending on workload on your Source Database.
  • You can’t backup a DB Snapshot and can’t attach/detach a snapshot; you can’t create snapshot for System Databases.
  • No GUI for creating Snapshot(Only T-SQL).
  • You can’t drop Original Source Database as long as a referring snapshot exists for that database.
  • No FAT32 support(Snapshots work on NTFS Sparse files technology).
  • Since they are read-only, static copies…new users can’t be added.
  • Make sure you’ve enough space on the drive holding your Snapshot file. If the drive runs out of space, Snapshot will be marked as “SUSPECT” by SQL Server.
  • Reverting to a Snapshot will break Log chain. Take either a Full or a Diff Backup(assuming you’ve got your latest Full backup) to bridge the Gap in your log chain) immediately after you revert. If not, all the log backups will fail from now on…
  • If your DB is not in SIMPLE recovery mode, Take a log backup before you perform Reversion to secure everything since your last log backup.
  • Full Text Catalogs will be lost once you revert your database. Be careful, if your database uses Full Text features as this could be an unpleasant surprise.
  • Both the source DB and the snapshot will be unavailable when the actual reversion process is in progress.
  • Source DB should be Online,No read only file groups should be present in your Source DB to be able to revert from a snapshot.
  • If your source DB gets corrupted, You can’t use Snapshot for reverting. ( See, they simply can’t replace your Backups)

Now it’s Demo time :)
I’ve a database called “Tst_restore” which has 3 tables and this will be my source database for this demo purpose. I’ve created Snapshot as shown in the below screenshot.

1

T-SQL:

IF EXISTS (SELECT name FROM sys.databases WHERE name = N'Tst_Restore_SS')
DROP DATABASE Tst_Restore_SS
GO
CREATE DATABASE Tst_Restore_SS ON (
NAME = Tst,
FILENAME = 'C:\Program Files\Microsoft SQL Server\MSSQL11.SREE\MSSQL\DATA\Tst_Restore_SS.ss')
AS SNAPSHOT OF Tst_Restore;

Now you can see I’ve a snapshot created.

2

See below for the Properties(files section).

3

As you can see, it has got no ldf file. (If you’ve noticed, while creating the Snapshot I didn’t provided ldf file in my syntax. Yes, that correct…you should provide only (all of) the data files, not the log file while creating DB Snapshot)

Revert a Database to a snapshot:

Now, let’s see how to revert a database to a Snapshot. Assuming something went wrong and you want to revert to snapshot which you’ve taken in the above step, you can use below syntax.

USE master
GO
ALTER DATABASE Tst_Restore SET OFFLINE WITH ROLLBACK IMMEDIATE
GO
ALTER DATABASE Tst_Restore SET ONLINE
GO

-- Reverting Tst_Restore to Tst_Restore_SS

RESTORE DATABASE Tst_Restore
FROM
 DATABASE_SNAPSHOT = 'Tst_Restore_SS';
 GO

As you can see above, to be able to revert we need exclusive access to the database(same as restore). Hence I am taking DB offline and bringing it back online immediately.
Note: Don’t forget to bridge the gap in backup chain right after you revert to a Snapshot. As I’ve already mentioned reverting to a Snapshot will break Log chain. Log backups will fail if you forget this step.

Hope this is informative…!

SQL Server Error Logs – My First Poll.

Hello folks…Thought of creating a poll for the first time :) I hope I picked something which makes sense and…….this polling will be open for the next three days. I will try come up with a follow up blog post If I get at least 10 votes. (Am keeping the results visible for this time)

Cheers-Sree!

Buffer Pool Extension – SQL Server 2014

With SQL Server 2014, Microsoft is bringing a lot of features/enhancements into the wild, one of them is “Buffer Pool Extension” AKA BPE. ( Enterprise edition Only)

so, what is BPE?  The name itself is pretty much self explanatory…It’s an extension for your Buffer pool. yes, with this you can use any non volatile storage device(SSD’s preferred) as an extension for your SQL Server Buffer Pool. Basic idea here is to eliminate requirement of “adding more physical RAM”, Instead you can mount a new Drive and tell SQL Server to treat that newly created drive as Buffer Pool. Sounds very interesting right? But there are few limitations Per Microsoft SQL Server Team’s Blog which we should keep in mind.

Limitations:
Enterprise Only Feature.
Might not be useful for OLAP/DSS Work loads.
Even in OLTP workloads, improvements are very limited for Write-heavy work loads.
If your server got more than 64 GB of RAM, don’t expect much from this.

Personally, for me these limitations are really frustrating. On one hand, this is Enterprise only feature and on the other hand, they are saying don’t expect much from this if your SQL Server has got already more than 64 GB of RAM allocated. What are the chances, any DBA would build a new SQL Server 2014(Enterprise) with less than or equal to 64 Gigs in 2014 or 2015? I would say almost “NIL”.

It Offers:
Performance Gain on Read heavy OLTP Work Loads.

MSFT recommendations:  (From SQL Server team Blog post)
Use High throughput SSD for better results.
Start from 4x-10x of the Memory available to SQL Server.

How to Enable BPE :
Step 1:
I’ve created a new Drive(BPE) for this demo purpose as shown below.

1

Step 2:
Before enabling BPE, Let’s check max memory setting I’ve got on this Instance.(512 MB as shown below)

2

Step 3: 
I’ve tried enabling BPE with 500 MB which actually failed. Reason – Buffer Pool extension size must be larger than current memory allocation threshold.(See below screenshot)

3

Well, now I increased the size to 600MB and now I got a different Error message(See below)!

4

Am not sure why this happened, but I’ve created a new Drive (F$) with 2 GB size and it worked this time without any issues as you can see below!!!…I will investigate further why this happened some time later.

5

6

Examine from DMV:

7

How to disable BPE: 

8

Can We alter the size of BPE?
Nope, we can’t. We have to basically disable and recreate it with new size.

Caution: When you disable BPE, chances are….you will get into significant Memory Pressure situation  and hence  increased IO Pressure.

Bottom line: IMHO, this idea sounds interesting, but I don’t think I would implement this on a production SQL Server considering the limitations it has(YMMV). I would rather convince  my management for purchasing real Memory which has no hand cuffs and no limitations!

After all….What are the chances my client doesn’t have money to purchase more DRAM for the sake of true performance boost when they’ve enough money to purchase High Performance SSD’s just for the sake of BPE(which promises me only limited performance gains)? . Remember folks…..now a days physical RAM is very cheap.

Reviewing SQL Server Error Log – A Better approach!

How many of you have a habit of reviewing Error Log(s) of the SQL Server(s) which you support on a daily basis(At least on regular basis, if not daily)? If you are not doing that, go ahead and start making it as a practice from “Right now”. You can thank me later for this suggestion :D

If you’re already doing that,then you are awesome…! But, let me ask this. How many times you get annoyed reviewing your error log(s), especially if you are looking for a specific error. Well, you can apply filter or you can simply query your error log from T-SQL. But what if you are only looking for all the severe Errors, total error count, Errors by frequency all at one place? Yes, you can write your own query and use SSRS for generating a fancy report. But Wait……What if I say SSMS already has this cool report inbuilt for us?

In this blog post I will show you a better way to decipher your current Error Log for Errors by using that hidden report.
It’s called “Number Of Errors” Report and you can get to this report by right clicking on “Management” Node -> Reports -> Standard Reports->Number of Errors in your SSMS as shown below.a

How it looks? See below…

1
As you can see, at a glance I know I’ve got 6 Severe Errors(Sev 18 or above) and 5 Moderate errors in my current error log. See below screenshots for how neat the info is once I expand those nodes.

Severe Errors:
2

You can see I’ve got the same 824 Error repeated 6 times. ( Well, I tried attaching the same corrupted ldf file 6 times for this demo)

Moderate Errors:
3

All errors by Frequency:
4

Is in’t nice? Personally, I liked this report…and thought of sharing what I’ve discovered with you folks :D
Please ignore if you are already aware of this nifty report, if not…well, you know it now!