I had a fun support call I need to share. A developer called up to tell me that a particular dev instance was offline. He informed me that the server SQL08\DEV01 (the names have been changed to protect the innocent) server was completely inaccessible. I knew that multiple development teams would shortly be calling and that I’d better get on this issue most riki-tik. I quickly typed the connection string into Management Studio and watched in confusion as the server instance popped up on my screen. It was fine. I did a number of checks, looking for active connections, recent connections, errors in the log, indications of a recent reboot… Nothing.
I called the developer back and told him that the server was fine. He called me again in two minutes to tell me that it was still offline, or was offline again. My connection was still open so I checked everything a second time, with the guy on the phone. I told him, no, it’s still there. Then he said, “Wait, which SQL08\DEV01 are you looking at? Is it the one with database X or database Y.” Now I was confused, but looked at the database list and told him that database X was on the server. “I want the other SQL08\DEV01” he then told me. The other one? What the…
Then it hit me. We have quite a few instances of development servers all running on the same server. So I started checking instances, looking at the databases on them and finally found that SQL08\DEV04 was what the guy needed. I called back and told him that he had the wrong instance and needed to change his connection string. I was then given a lecture that he had been using different copies of SQL08\DEV01 for quite some time and could I please find someone who knew what they were talking about to help him… Yeah, one of THOSE developers.
After a couple of deep, cleansing breaths, and one more fervent wish that the company would let me hang a heavy bag in my cubicle as a means of further relaxation during trying episodes (I promised not to put people’s faces, likenesses, caricatures or even names on the bag), I got back on the phone with him. I asked him to send me over his connection string so I could see what was going on.
When I got it I saw an interesting point. He had the server name & instance, but he also listed a port. It just so happens that we’d had an issue with the DEV04 instance and had recently recreated it from scratch. I guess whoever did it let it get a different port in the process. Mr. Knowledgable had been connecting to the wrong instance because, and this is the kicker, when you supply a port to the connection string, it ignores the instance name. This is why he was able to connect to the “other” DEV01 instance. Just something else to watch for.
[…] This post was mentioned on Twitter by Grant Fritchey, Jen & Sean McCown. Jen & Sean McCown said: RT @GFritchey: The Other Server: I had a fun support call I need to share… http://bit.ly/7MpkTW < Not sure why, but this makes me happy. […]
Welcome to my world. I am just lucky that I don’t have that many developers to deal with. Some people have just enough knowledge to be very dangerous.
Great post. I’ve lived through this many times from the developer side of things. You had to do this over the phone? It is too bad you are not co-located with the developers you support. If you were sitting near them I imagine this would have been a completely different experience. Developers and DBAs often have an antagonistic relationship, so why do companies make it worse by physically separating them? If they work together, on the same team, in the same room, devs and DBAs stop seeing themselves as warring groups. They will still bicker on the small issues, but the small issues will not snowball into big issues and, who knows, they may even begin to develop some respect for each other!
The only reason my company doesn’t put us in the same room with the developers is bacause we support multiple development teams. Where do I sit if I’m on multiple projects with multiple people? With the DBA’s.
I’ve often been surprised by the need to have people in the room with you in order to work on technology. I just don’t think it’s required. It does highlight the need for communication, but it needn’t eliminate it.
Developers are just people, like DBA’s. We can be wonderful or nasty.
Fair point, but if you need to work with multiple teams, why not walk over to each team’s area when you need to work with them? A little face-to-face relationship building goes a long way. Once you have a good repoire with a team then maybe you can back off a bit on the co-location.
The most challenging aspect of building systems is not the technology–those problems tend to be easily solvable. It is always the people that make things difficult. There is no substitute for building trusting relationships within and among teams.
Absolutely. We do spend time with them. Probably not nearly enough, but it does happen. Phone & email & IM only get you so far. Actual, direct conversations with human beings really do help. I just don’t think it’s a constant requirement.
I agree. Trust is key and there does seem to be far too little of it between DBA’s and developers sometimes.