A rebuttal to “Is Your DBA Lying To You?”

Thomas Larock recently wrote a piece regarding his idea of the realities of being a DBA in “Is Your DBA Lying To You?” I have some serious issues with his perspective.

Let me first say that it’s possible that I missed his tone. It’s possible the piece was written as satire, or as an indication of things that happen in DBA land without necessarily endorsing them. My other disclaimer is that it depends on the relationship between the DBA and the database user. If his perspective is regarding a database as a service, then sure – all that is guaranteed is the availability of table space and uptime. However, some of the tips certainly sound like we’re talking about organic database assets and administrators, and if that’s the case, we have a problem.

Let’s go point by point…

Lie 1: “Your database server is still on physical hardware”

The assertion here is that the database staff will virtualize your database server without telling you, and that’s okay. It’s not okay. First of all, there can be definite performance implications of running in a shared environment. For example, Oracle has specific requirements regarding supportability on virtualized hardware.

More importantly, there may be support issues if a server is running in a virtual space. Apart from the database server itself, it may be running as the back end for a commercial application that has specific requirements for support. Exchange 2010 expressly specifies the storage requirements with respect to virtualization.

The application owner is ultimately responsible for supportability and performance of their application – so if the DBA lies about whether or not the storage is virtualized, they are now part of the problem. (I wonder how the DBA would feel if the operations team lied to them about their database server being virtualized…)

 

Lie 2: Everything is fine, nothing to see here.

Hiding problems is never the answer

“No, really – everything is fine”

Here Thomas advocates hiding problems with production servers from application owners. He asserts that as soon as an application owner hears about a problem, they will start blaming everything on that problem. While it’s true that this is an observed phenomenon, it doesn’t justify lying to application owners or hiding the truth from them. While it’s possible that the application owner doesn’t understand how problems may or may not affect their application, it’s equally true that the DBAs may not understand the ramifications of their actions on applications.

While it’s nice to believe that DBAs will be all-knowing about their servers and the applications that are running on them, the honest truth is that in many large organizations, there are simply too many applications to keep track of. That’s the job of the application owner, and why they need to be kept informed of any data center incidents that may affect their application.

 

Lie 3: It’s the network

“Admit nothing, deny everything, make counter-accusations.” We’re on that last suggestion. The idea here is that when there’s something wrong and the DBA doesn’t know what the problem is, blame the sysadmins. There are so many things wrong with this I’m not sure where to start. First, the DBA is going to deprive the application owner of options – if a server is unresponsive, the application owner may want it failed over; or may be able to shift to another server that’s not current, but can at least continue operations. By blaming the network, they won’t consider any of these options, instead feeling that the network problems need to be addressed before they can do anything.

Then there’s the evilness of blaming your own problems on someone else…

By spreading misinformation, the DBA is also depriving themselves of necessary information that might help them resolve the issue. For one thing, when the word gets to the sysadmins that there’s a network problem, they’re going to start troubleshooting, which is going to complicate the DBA’s troubleshooting efforts.

And finally, let’s talk “Boy who cried wolf” – what happens when it really is a network problem?

 

Lie 4: Of course your server is dedicated.

Overcrowding a server is never a good idea.

“Your application is the only one on the server. I don’t know why it’s running slow.”

This is a variation of Lie 1 – “we tell everyone they get a dedicated server, but in reality we have four physical servers for a hundred applications.” Here the problem is that if an application owner is dealing with performance issues, and their application is on an oversold virtual server, they will waste time looking for potential optimizations or bottlenecks when the actual performance problem is an oversold server or even misconfigured throttles or limits.
Lying about overselling virtual server capacity also sabotages the organization’s ability to plan and budget for hardware expansion. Five critical applications thrashing on an overcrowded virtual server may be plenty of justification for buying new hardware, but nobody’s going to know if the DBA is sitting on the truth.

Lie 5: Yes I gave you the permissions you asked for.

More hiding the truth because “the DBA is smarter than you.” If an application owner asks for permissions, either grant the permissions they request, or deny the request. This is yet another place where company time will be wasted on pointless troubleshooting. The application says it needs sysadmin permissions to install, the DBA thinks he or she knows better and grants superuser permissions, but tells the application owner that the installation account is SA.

Unfortunately, the application installer explicitly checks for sysadmin permissions, and when the account doesn’t have them, it fails with some obscure error. Since the DBA has told the owner that the account has SA permissions, then the problem must be something else. (Of course – it’s the network…)

 

Here is the only lie you need to understand is false:

It’s Okay To Lie To A Colleague

DBAs are not smarter by default, they do not magically know more about the application than the application owner. A culture of lying to others creates an “us vs. them” mentality that is only destructive and wastes resources and time. And of course there’s the serious problem of creating the impression that you cannot trust the DBA group.

Be honest, and be prepared to explain the reasons behind why you did what you did. If you’re afraid to (or cannot) explain decisions you’ve made, perhaps the problem isn’t the application owners.