Check out this DevOps Reactions animated GIF and caption.
It’s funny on multiple levels, but it also makes me both mad and disappointed.
I get mad because it’s 2015. Surely by now most of us, especially those who have worked in the enterprise with development teams, know that the old 1970s vision of a walled off data center with DBAs in white lab coats acting as gatekeepers to the data is long discredited. As DBAs, even if you’re not working with development teams at all, you’re just offering a service to the business. This whole, a DBAs favorite word is “NO”, meme needs to die a quick, hard, death. All those “Technology X” is going to eliminate the DBA articles that come out every six months like a comet with a small orbit are partially predicated around many teams attempting to get around, over or through the DBA because we still think our purpose in life is to say “No” to every single request. Come on people. You’re part of a team. That team has a very simple goal, keep the company afloat so it can make money. That’s it. If your goals are narrower than that, if you’re a road block to progress and motion, if you’re trying to stop every development proposition that comes down the road, well then, heck yes, your dev team is going to try out “Technology X” so maybe they can get rid of you. Deservedly so. You have it coming and you brought it on yourself.
I get disappointed because this whole attitude is just going to make things harder for data professionals down the line. Why do I say that? Because, inevitably, data management is going to fall back under specialized knowledge sets because it’s actually a pretty specialized skill. And I’m not talking about running backups. I’m talking about 1) Optimization, even if you’re on Hadoop and are luvin life with NoSQL, you can still do it wrong and hurt performance, 2) Emergencies, recovery from corruption or point in time recoveries or proper management of whatever failover system is set up (assuming one is) requires specialized knowledge that most developers just don’t have the time or inclination to learn (BIG NOTE: not talking ability). Those two areas are where everyone I know who makes big money consulting is working. Why? Because that’s where things break down. And this entire attitude of “NO” from DBAs mean we’re going to be excluded by development teams (and I don’t blame them, in fact, I encourage it if your DBA is still saying “NO”) so that they can get their jobs done. But… Later, stuff will go wonky because of load or the SAN getting switched off or a limited Zombiepocalypse and suddenly, businesses and dev teams, and probably the one or two data pros still in the company, are playing catch up in ways that could have been avoided.
This has to stop.
DBAs must be a part of DevOps because we’re part of Dev and we’re part of Ops. I’ll put up another post on this topic shortly.
Dude…spot…f’ing…on.
–dave
I’ll admit that I laughed at that graphic, but mostly because the times I tend to think like that are “in the next two week sprint, we want to …” followed by something that the system just isn’t built to handle (yet). Could we get there? Of course, but in the meantime, let’s work with what we have or work out how to get there and build something that the users won’t hate when we turn it on because it performs poorly. That’s not to say that project “x” can’t be done, but sometimes the timeframe is a bit off.
I’m actually a big fan of the DevOps concept. Working as a dev w/ our Ops team made life better overall. We could see what was coming up, plan server build-outs appropriately, tune code to work better or servers to handle the code better, and streamline our release processes. Before that there was a lot of finger-pointing because it couldn’t possibly be the (code/servers), it’s obviously the (server/code)’s fault. 🙂
DBA’s should be part of the Dev cycle, if only because you’re less likely to be surprised when someone releases new code to production that has an adverse impact on performance. Good reminder that we’re not supposed to be the last line of defense, but part of the team pushing out software to our users.
[…] DevOps, the DBA, and the word “Noâ€Â – Grant Fritchey (Blog|Twitter) […]
Spot on.
I’m pretty sure I got an interview for my current job in part because of the deliberately attention-grabbing first paragraph of my cover letter, which was a single question: “Why would you want to hire a yes-man?”
I then went on to explain that I had developed a reputation at my then-existing job to saying “yes, I think we can do that” whenever managers came to me with some new kind of data request. This had not been the previous norm, and people really appreciated the attitude. Even when I was thinking “no”, I would re-cast it as “I’m not sure that’s possible with our existing data structure, but if we did this instead, would it get you close?”
It’s pretty simple. When you say “yes”, people want to work with you, and you’re in demand. When you say “no”, people work around you.
All in all, I’m pretty disappointed by this article, Grant. There wasn’t anything here for me to be scared of.
😉
This may sound strange, but in my experience you have to earn the right to say no. To accomplish that, follow three simple rules: Be Honest. Be Direct. Be Professional.
Be Honest: This means your primary conditioned reflex when a new idea is proposed should be to educate and inform rather than dictate. It’s okay to say “I’m not really sure” or “I don’t know, let me run some tests and get back to you.”
Be Direct: Stick to the issue. Lay out what the impact could be. Maybe the timeframes are unrealistic, maybe performance would suffer, maybe other systems would be adversely affected. Just lay it out without prejorative wording. Other times it may mean saying, “Sure, we can do X, but A, B, and C need to happen first.” Or maybe all you need do is lay out some alternatives that will accomplish the same thing, just maybe not exactly how everyone else wants to do it.
Be Professional: DON’T TAKE IT PERSONALLY. It’s a job. Treat others with respect and be willing to say “I was wrong, that actually worked.” And as much as we hate to hear this, sometimes you have to let the train wreck happen but keep the mop and bucket handy to clean up the mess without saying “I told you so.”
By sticking those simple rules, over time the people I’ve worked with have come to know that I don’t say NO lightly. I’ve earned that right.
Rich,
That’s a great approach. I like it. In fact, I’ll be stealing it next time I’m applying for a job. Thanks!
Larry,
No actual disagreements with you. I do believe there are places where we should say no, for any number of reasons. But the issue is, for many data professionals, and I’d argue, most DBAs, the default position is “NO.” I think that default has to change. Think of it like “Cost Threshold for Parallelism.” That default should just be changed immediately, everywhere. Same thing with saying “No.”
I’m the team lead for a small SQL group and we sit with a similar Oracle group; both servicing a huge corporation.
I hate to generalise but there was definitely a difference in the way our two teams approached users. I made sure my team was the, “We are here to help” voice. We need that support because one day, yes, we are going to have to trim back access and need people on our side. However we draw the line at where that impacts people doing their jobs – if they need sysadmin and we have no easy alternative then they’re damn well getting it!
Oracle team on the other hand was far more confrontational. It was a matter of, “We are taking your access because management says so and this is how professionals work.” Were they wrong? Absolutely not. And they were fighting worse outages caused by sysadmin users than us. But… in the long run perception of them was very poor (and sadly, undeservedly so, considering their excellent work).
Anyhow I feel this post is a little preaching to the choir, but, maybe there are still some holdouts. I wrote a funny video a few years ago as a developer DBA being unable to get reliable access to dev, and being refused access to Prod for the temporary purposes of deploying a solution (and where I wasn’t provided any alternatives for testing and documenting how someone else should deploy it). Most people were supportive but there was one person who left a comment that they would have forbidden that access also.
Really, a dev cannot have sysadmin to dev? That is a rule we don’t implement at our company (though Oracle team has a different view). Devs can have full access in dev, and we have daily checks to ensure that they havent interfered with parts that are our responsibility (backups, integrity checks, index maintenance, and disk space), and also that they are keeping a clean house (recurring scheduled jobs don’t fail, if they do they have to review and fix or disable them – it’s just good practice… some devs want no backups, leave crap littered everywhere, and sitting on zero disk space, and think that’s totally okay – we don’t).
Some DBAs (either highly paid consultants who’ve forgotten how hard it is to work with your hands tied behind your back, or IT staff masquerading as DBAs) are definitely too hard nosed.
Here’s the crux of the problem, which has existed in some form since at least IMS (ask your Grand Pappy what that is; yes, not was): app coders want to silo all the data, and its control, in their client code. Now, this was fine back in the early days of COBOL (and PL/1 and …), since all there was, was client code. Then along came Dr. Codd and Larry, and the war started.
DBAs, at least those who take normal form schemas seriously, see that the database should be agnostic to application code (and that there’s no reason to duplicate data to multiple stores to satisfy the silo imperative of multiple application teams). Moreover, data integrity has to be done at the engine, since otherwise no guarantees can be made. “We” never know what nonsense may go on in client code, yet we get blamed when it goes south.
This approach does impact development in a number of ways. First, it strips client coders of absolute authority over data design and implementation. “We must denormalize for speed!!!” Second, it adds authority to the responsibility of the database folk for data integrity. Third, it reduces client coding largely to I/O duties. This last is the crux of the war. Fourth, it requires that the database folk actually deal with user/stakeholder to determine what the data integrity issues are; ewww, users!!
Yes, teamwork can work. But, no, it won’t so long as dev shops still adhere to the COBOL/VSAM model that our Grand Pappies lived by. It’s been 45 years since Dr. Codd’s first paper made its way out of the IBM labs, and normal form datastores are still viewed as stupid and obstructionist by app coders. For their hegemony, it is. The roles of the team’s groups have to morph in order for all that money spent on database engines to be worth it. A RDBMS isn’t just a common query syntax parser and simpler backup process storing the same silly flatfiles from 1970.
[…] this goes back to my most recent post on the word “NO” and DBAs. Two, it’s just […]
Grant,
I think we’re saying kind of the same thing from opposite point of view. You can say no, but you have to get to the point where when you DO say no, people understand it isn’t just your primary conditioned reflex. In general I’ve done the same thing as Rich mentioned: Try to do what you can to make it work but when you can’t, be upfront and honest about it.
Cody,
Leadership through service. It’s the way to do it. Well done.
Robert,
No arguments, but what I hear from the devs is that we’re the ones stuck in the past.
Larry,
We do have to be able to say “No,” absolutely. But the reflexive “no” has to be a thing of the past.
The Dude abides! One of my favorite movies.
In the ’90s I was working at a police department. The City wanted to deploy Peoplesoft and arranged a meeting between us and their PS people, along with City IT, to explain our payroll pre-process so they could implement it in PS. It was a complicated process involving OCR (which was abandoned) and very complex relationships and processing to accommodate something like four layers of contracts with unions (I’ve never seen fishhook relationships before or since), and the deeper we explained it the more terrified the PS people looked as they continued to say ‘We can’t do that.’ We were running SQL Server 4.21a, ObjectView, and VB 4(?). And Peoplesoft was allegedly the cream of the crop?
I got my current job last October because one consultant said he could do the entire project without any code, another said it couldn’t be done as specified. I spent the first two months refining the data model and gathering more information (I’d never worked at the school level in the field of education before), one month doing development, and I’ve got a VERY functional alpha release right now. By myself.
There’s always Clarke’s Law: When a distinguished but elderly scientist says something can be done, he’s probably right. When a distinguished but elderly scientist says something can’t be done, he’s probably wrong.
Yes. We must say yes. It might take time and effort, but it’s usually doable. I also believe in under-promise and over-deliver.
I’m not speaking as a DBA, although I can find my way around a database and have been known to do database development at many levels. Rather, I am one of ‘those’ developers who would rather think outside the box and attack a problem creatively to provide that value added solution. I will say this, “Not yet”, in whatever way, shape, or form it takes, be it what, how, when, or even who or why, is usually heard as “No” but key stake holders. The default, “yes, we’re here for you to help make the business money,” needs to be communicated, a lot.
I just saw a blog post the other day by a DBA who didn’t like what developers did and published scripts to show how to go around them and create policies to stop them from doing things he didn’t like… so I commented and asked if he was going to explain why he didn’t like those things and better ways of coding instead.
If you see something in code that is not good, talk to the developer. It might be that they are not aware of the pitfalls of a particular method of coding. From their perspective, it works and usually very well. Share your perspective with them.
This is the never-ending cycle. For me I’ve been on both sides as a dba. I used to work hand in hand with dev teams, and it worked well after we agreed to certain terms. But often, dev teams don’t understand what causes sql performance to tank, and the dba is spending evenenings and weekends t
[…] and requirements still get “thrown over the wall” at the last minute. All of the old misperceptions remained in […]
[…] few weeks ago Grant Fritchey (b|t) posted about DevOps, the DBA , and the word “No”. Go ahead and read it, I’ll wait right […]