"Working" or "Working As Intended": How to handle desk-side support

Lyniaer

Guest
Nov 2, 2010
68
0
18,630
Hi guys,

Really tough topic as far as I'm concerned. I don't wanna throw around terms like "OCD" without there being an actual condition tied to it, but I like my IT environment "Just So". If it's not tidy and uniform it means you're only one small change away from total disaster.

Your users may say "it WORKS, so leave it ALONE" but is it Working As Intended? They may be able to Send and Receive email, but they may not know that there are port issues in the background that may cause conflicts later if you switch from Office to your own In-House email server. Or, all of your printers may have sporadic and random host names and have been that way for years - your users are used to those host names - if you change them to bring them in line with a naming convention to make it easier to identify (and faster to rectify) issues - will your users care?

I work for a Global Corporation - a recognized brand, but I won't name it here - and am a Systems Administrator for multiple locations in my area and serve roughly 300 users. We just replaced our printers - the job having been done by contractors rather than ME much to my chagrin - and are soon going to update all users to Windows 10. However, the printers have been a big thorn in people's sides and are going to be a problem when we change up from 8.1 to 10.

We just moved several printers around the building and doing so caused a massive headache for everybody. Not to mention we have users running software that operates on a VM in Corporate Headquarters and it does NOT like to play well with our printers.
As I'm between projects, I'm going around changing all the host names and IP addresses of all of our printers and making them static as well a uniformed naming convention, but here's the problem:

My users are getting triggered by this interruption. It's a minor inconvenience and every so often I have to deal with an IP conflict or just a straight-up silly configuration that makes no sense. I try to explain to them "Sure. It WORKS, but to put it in a way you understand, the Wires are Tangled. I'm just unknotting a mess you can't see that will cause us all headaches in the future."

"But it WORKS" they chant. "Don't mess it up or you'll mess ME up." "I LIKE my printer being named UNQFWPSLP4546; it's been like that since I started here 18 years ago!" Yeah, back when we used Okidata and the walls were wired with BNC. It wouldn't surprise me if some of these host names are actually serial numbers of printers that haven't been manufactured since I was born.

For those of you in IT. How do you handle end users like this? I have one right now who steadfastly refuses to allow me to touch her computer anymore because "after I messed with it" now her "Outlook email doesn't print correctly" when, in reality, it now prints correctly - she got used to it doing what it wasn't supposed to do.
It's like your car mechanic friend randomly tightening up a coolant line on your car and you getting mad because you planned out your Tuesdays as "needlessly replace the coolant in your car" day.

"But I USE that slow leak to find my way home!"
"Now the temp needle isn't perfectly straight up and down!"
Worse yet.
"The garage says I have a blown head gasket! Everything was FINE until YOU messed with it!"

Maybe I'm in the wrong line of work. But, in a decade or so, the technologically passive will retire and the next generation of workers will be more willing to let us do what we have to do.
 
first of all - about the printers -

Why are you doing it in person? Why not jump on over a weekend remotely, install the printers to the server (your choice); with your chosen names, and push them out to the company via the active directory?

Make the names intuitive for the location/department. coordinate with maintenance to put labels on <mod edit>, and leave the old name in parenthesis in the printer name, old to remove at a later date. this way you can manage driver updates from the server, and not make so much busy work for yourself.

--customer service --

Finally, the problem you're having is a classic one. In a large IT environment, you need to do 2 things to get along.
1) announce your projects to those who will be affected WELL IN ADVANCE
---and follow up with a reminder the day before
2) don't give them a choice.

What you're failing to do, is communicate in advance. What you should have done is sent an email out to the affected that via corporate IT policy the names of the printers would change on such and such date, but not to worry, because this is the name they're changing to. Then remind them before hand and bam you're done.

by doing as much of your job from your office, you keep them from having to vent at you personally. Now, I make it a policy to run by any potential work interruptions to the managers in charge of the area before sending anything out in a email so the MANAGERS know before hand, so that when those busybodies complain he'll know what the heck this is all about, and he won't be blindsided because you coordinated with the manager before hand about ideal times and the like.

I do almost all my work remote-AND by appointment when I can get away with it. the only time my clients will see me is if they have an emergency, in which case, since I'm doing things remotely and scheduled, I usually have plenty of time to be on the spot immediately whenever there is a real emergency. This results in you developing a positive impression with staff. Remember, they have to work too. some of them might be under a lot of stress, and frankly, I wouldn't want you crawling under my desk in the middle of the day unanounced either if I was in the middle of something.

SO TO REVIEW
1) schedule in advance with an authority figure, make sure they understand why you want to do it, and what's entailed (work disruption) let them have a say in when and how it gets done
2) send out an email to those effected WELL in advance; describing what you're going to do, and what is changing. You don't have to get too technical here, most people won't understand it, and if you get in too deep, those who have some knowledge will argue if it's necessary so keep it vague enough they know what you'll do and what type of work disruption it might be, but no real need to justify it clearly (you already did that with their management)
3) send a reminder the day before
4) do as much as possible remote - be polite when you must be in person, but remind them they were warned it would happen today around this time, and just like they're busy you have a lot of this to do as well.
5) if you make a change in person - ALWAYS make them verify it works before you leave. This way you know you did your job, and any complaints later are USER ERROR.
--if you make a change remotely, like I suggested with the printers, make yourself availible the following day to "assist" anyone having issues. There will be some, but if you have control over your IT environment (and a ticket system) this shouldn't be a big deal.


 

Lyniaer

Guest
Nov 2, 2010
68
0
18,630
This sounds lovely.
Sadly, I've tried to no avail to get access to our print server. Well, I can go over to the server room and physically slap it, but there's no interface there. I can't do anything with it. I can't push out changes via Group Policy and I have to clean up printers that haven't been in the building for five years by doing so manually as I come across them with different users.

Mind you, this is a global corporation. They're so inefficient that when they changed their internal branding "They went from 'CompanyName Ambiguous Noun' to 'CompanyName North America' and because of some contract decision all the old computers (they were only a year old) have to be switched out with new ones. The difference in computers? The asset tags. Well, and serial numbers. Literally nothing but paperwork was shuffled and now all the computers have to be changed.

If I want something to be pushed down through AD, I have to put a ticket into one of the 37 different server groups - figuring out which one is half the fun - but almost all of them are in India or other infamous outsourced locations. If it gets done or done correctly is usually a crapshoot.

So, yes, I am a small cog in a very big machine that runs about as well as most farm pickups, but so long as it can make it across the field with a bed full of pig crap - good enough for a multi-billion dollar corporation!
However, I can, without any managerial approval, buy a $2000 laptop. Sure, maybe a few months down the road someone might ask "Did you buy a laptop?" and the only answer they want is "Yes, I did."

But access to the print server is just asking too much.

I used to be a Network Admin for the State Government. Surprisingly less red tape than this company which makes...we'll just say "Beverages." The restrictions were reversed. I could access anything and everything remotely and do whatever I wanted to do "within reason", but GOD HELP ME IF I NEEDED A NEW STAPLER. Buy it yourself if you need it and MAYBE the state would reimburse you. Maybe! I once bought a deskjet for a user. Never got my money back, but the state happily purchased ink for it that easily cost as much as the printer did.

For once I would like an IT job where things made sense.

But, for now, I have to go around and manually - or via Web Interface - change all the printers to static IP addresses, change host names to a proper naming convention, deal with IP conflicts as they arise or just pick another IP because I have no way of knowing what's in use because the ability to ping stuff is blocked at the server level. Everything I ping comes back as no response (ICMP is blocked or something), so I just have to guess or remote shut-down the computer that's tying it up. Most of the printers are not even targeted to the same gateway. Whichever IDF it reaches first it treats as the gateway and damn you if you try to give it a gateway that's further up the line. After that, I have to go to each computer and create a new IP Port As Local so that it propagates to all users on that workstation - but still doesn't delete whatever nonsense has piled up in their devices.

With the state, I didn't deal with IDF's and square-footage issues. Every drop ran to the server room with switches stacked on switches, so I'm a little out of knowledge with what they've done here. The IDF's mostly handle the security cameras, but some workstations/MFPs out in the far corners of the warehouse are connected to them. Shipping, Receiving and Maintenance all have their own IDFs and then the front office sits with the main server room and the Trunks come in on the opposite side of the building and we're still waiting for a fiber connection from the main to the server room so everything is ungodly slow despite 10 T3 lines.

They wanna role out windows 10 fairly soon and on top of that our file server is migrating from the old server to the new one the printers are running on. I'm trying to get as many users off WSD and onto direct-to-IP as possible and will be asking to lock down their ability to add printers as they see fit. I've seen users with about 15 printers in their devices - many of which are configured wrong or are for printers that haven't been in the building for 5 years. This way when we roll out W10 I can import their cleaned up profiles instead of making them save all their documents elsewhere and wiping their stuff out - which would make me have to get new VPN tokens for them and a bunch of other problems that would come from a complete format-install that changes the computer name. And, no, I can't just use the old computer name. It sits around in the global domain for about 3 months and won't let it be reused.

This job is like 8 hours of Brian trying to get passed Muscular Stewie. "Oop. Whoop. Oop. Whoop. Oop. Whoop."
 
Sounds like a headache. I am a subcontracter; I have a number of clients; but my big one is a defense contractor.

They’re a multinational too; and as a subcontractor I have a lot more access then it looks like your company has given you. Shame it’s like that.
 

Lyniaer

Guest
Nov 2, 2010
68
0
18,630
They want to get the most out of their ServiceNow contract. Turned out, a couple printers were set up with MAC reservation. All the rest had reserved IP addresses with the print server, but those IP addresses followed the host name and NOT the printer itself. How they managed to do that I have no idea but after a series of conference calls today it looks like they're first going to give their outsourcers a shot at dealing with it and if that goes over like a lead balloon then I get my access after all.
 


I speak as an end user with absolutely zero input into IT, and I work in a field where we are completely and utterly dependent on IT (healthcare.) We are forced to run an EMR and EVERY SINGLE FRIGGING THING WE DO is using that EMR program. If the network is down, we close up shop and send everybody home as we can't connect to the EMR server farm. If we have a power failure at an outlying site, once the BBUs die, we close up shop and send everybody home as we can't access the EMR server farm. If we have problems with the EMR sever farm, EVERYTHING shuts down and we send EVERYBODY home, losing about a quarter million bucks per hour and people in the hospital have died as a result. All information flow stops and there is VERY limited access to an outdated read-only database of past information. We are essentially prohibited from maintaining hard copy records, else we receive massive fines and penalties from the federal government that would more than bankrupt even the largest monopoly vertically integrated healthcare delivery system.

We end users only care that we can get SOMETHING done, as we are functionally contractors being paid on piecework. No work = no pay, and for most of us, we have clawbacks and actually have to write a check to our employer if we don't work (we PAY to not work) so there is a big push to try to do something to keep open and running to some extent. Those of us with technical knowledge will use as much as we can use given the fact we don't have any privileged access to try to keep going. If the printer's hostname changes and if we know what it changes to, we re-add the printer under the new hostname as we really need to print. I've even run Linux live CDs/USB sticks to troubleshoot problems using a sane, familiar OS to do things like run Wireshark to passively pick up pings to figure out hostnames and such.

We ignore background issues as unless they absolutely prevent us from doing our job. One old cruddy printer doesn't work? We'll print to another one 50 yards down the hall. Yes, we file a ticket, and in 6 months, the IT guy will replace the working printer with a new one (and completely miss the fact it was the OTHER printer that was broken, even though we gave them the asset tag #). We have to keep going- remember we lose money (not just don't get paid) in not working.

We cuss any modifications/updates as they are poorly tested if they are even tested, and essentially any update to the giant spaghetti codebase of the EMR program breaks 50 things for every one unrequested gee-whiz change that the vendor made. I cannot name our vendor due to NDA requirements in their contract not allowing us to publicly comment on (lack of) performance or (huge presence of) bugs, but suffice to say they do billions in business per year and are one of the largest in the industry. "Hey, we made the refresh button border shape a 1 px border rounded-corner square instead of a 2 px border circle!" "Yeah, well, now it locks up and crashes every time I push the "sign note" button and displays an error message instead of the lab results I need to see to treat the patient! I can't do my job!" "Yeah, but our refresh buttons are really pretty! We'll address your concern with the next full codebase update in two years, your sequentially-numbered ticket number is 457,132,567,781. Thank you for your patience."

The IT people somewhat tolerate me as they know I know more than they do about their own work, despite being hired to do something completely differently with a grossly different skill set (that they obviously know NOTHING about). One mentioned that I was an oddball because I am as much against the current state of health IT as the old curmudgeons, but unlike the old curmudgeons, I understand what is going on and simply reject the EMR as grossly inferior after specifically and categorically recognizing its enormous shortcomings and how they are not being addressed rather than just "it's different and I don't want to change." I'm typical age for a THG reader, 30-something, and unless the economy of the last year persists rather than the economy of the preceding 15 years, I'm going to be working for the next 40+ years. The Baby Boomers who hunt and peck and say "who moved my button?!" have pretty well all retired as they made their money before the Second Great Depression and could afford to retire before their 70s.
 

Lyniaer

Guest
Nov 2, 2010
68
0
18,630


It's the same problem as replacing one tire at a time on your car. The tread will never be even and that kind of uneven tread pattern easily leads to other mechanical difficulties over time. Most likely because of budgetary reasons, IT equipment is never "refreshed" evenly. They will buy all new computers that are 2 years behind the curve. Then update some servers but not others. Then realize that THIS server or THAT operating system doesn't work with THAT vendors software or VMware, or it works but there's an unfixable bug that causes THIS or THAT problem, but they paid millions for that software or millions for thousands of licenses and can't get their money back so they have to wait until it's absolutely ready to buckle before they throw more money at it. They update THAT to the latest and greatest but, UH OH it doesn't work on this operating system. Gotta update that. Now the printers are too old and there's no driver support with the new OS so there's another thing that needs refreshed - and each of these take time. It makes the company feel that every time they turn around they need to buy new equipment or software.

The poor IT department gets stuck in the middle because they have less than 0 say in what goes on. IT (be they Deskside Support or Offsite Help Desk) is there simply to translate the end users frustrations into something that can be deciphered by IT Management further up the corporate ladder and then THEY have to translate THAT into something the bean counters can understand and would be willing to write a check for.

They all know there's a problem and typically this stuff is Most Definitely tested before being deployed for production. The problem is, this stuff can't sit in development Hell forever. Once the contracts are signed the company has a finite amount of time to build what they claimed was usable or they lose a crap ton of money and depending on the circumstances it could cost them their whole business - but they won the bid and they gotta put out.

We're facing a dilemma right now where we're having a new program practically forced upon us (not "us" us, but, well, you know - big company).. It has to do with a database software that maintains customer information and SKU numbers and some such. But it doesn't work. Like.. At all. But because they're so close to the deadline they're forcing us to Go Live with it and they'll work out all the problems WHILE IT'S IN PRODUCTION.

So, with mismanaged technology refreshes on one hand and lowest bid contracts for production software on the other - you wind up in this No Win situation and there's really nothing to blame but Capitalist Exceptionalism. There is such a thing as Too Big To Function. Things would be much more manageable if corporations were satisfied with being smaller.
Imagine being the father of 500 kids, half of them are adopted, most of them with different women. You can't handle all that, so you hire a bunch of surrogate fathers and issue them instructions on how you want them to raise your children - regardless of the fact that each of them is in a different environment with different influences and highly varying needs. You pressure the surrogate fathers to stay the course with the current plan, but many of the children are just not prospering in the way you imagined. So you sit there and wonder what it is you're doing wrong. You give up ownership of several of them and let the surrogates raise them themselves. Things get better. But now you have this surplus of time and energy and instead of spreading that around to your existing projects - you adopt more kids to fill the gap. And things get worse again.. And you still don't understand what the problem is..

In short; Bad IT is often just the reflection of a Mismanaged Company .
 


^ This sums it up beautifully. And in healthcare, you have not only "too big to succeed" monopoly/duopoly "health system" but you also have the even larger and far more dysfunctional federal government forcing terrible IT down our throats.

 

Ralston18

Titan
Moderator
Human nature opposes change.

I have been on both sides of the issues described. And some of that time has been simultaneous.

The only solution that ever worked was that someone with enough power and the applicable "SOB" rating came in and said "you will" and/or "you will not". Period. End of discussion. Do it. Change.

Sometimes such folks were not popular people but they got things done and it did tend to work out well for everyone for most part. The lazy, incompetent, foolish, biased, etc. people will never be satisfied or willing to make changes. Always seemed very difficult to fire them for some reason.

Yes. Leadership is needed. Especially when things are mismanaged or simply in need of change.


 

Lyniaer

Guest
Nov 2, 2010
68
0
18,630


I used to work in IT for my State. Recycled Paper and Remanufactured Toner. I would blow a portion of my budget just buying up every photoconducting unit I could find because they only lasted about 5,000 pages.
Let's get me started on the hilariously ill-advised Dell contract. I recommend them for enterprise because they are inexpensive, easy to repair and KACE made PXE deployment a breeze. We use Lenovo where I'm at now and while they last hardware-wise they can't stand up to abuse or wear-and-tear, are very expensive and the models expire rapidly so the model you bought six months ago won't exist when you need to buy another one and you'll need a completely different docking station and power adapters to go with it. This was when the switched from the 20v to the weird square one that people try to jam into the USB slot and cause shorts and damage.
 


But human nature and self-interest paradoxically mandates it. If some software you made does an excellent job, does all the tasks required of it, and is reliable, then everybody buys one and then you go out of business as you essentially never sell any more of them. So you continue to tweak your product or if the market will tolerate it, change to a subscription model, so you can continue to stay in business to forever milk what you did a very long time ago. This is best illustrated by Microsoft. They finally got the worst of the bugs out of their OSes and office suites in the 2000s, and added little more than window dressing since. Businesses stuck with XP/Office XP or 2003 despite a slew of newer releases being sold. MSFT hated not being paid, so they changed their licensing and you can no longer actually buy Office any more, you simply rent it.

The only solution that ever worked was that someone with enough power and the applicable "SOB" rating came in and said "you will" and/or "you will not". Period. End of discussion. Do it. Change.

Sometimes such folks were not popular people but they got things done and it did tend to work out well for everyone for most part. The lazy, incompetent, foolish, biased, etc. people will never be satisfied or willing to make changes. Always seemed very difficult to fire them for some reason.

In my field, that SOB was the previous U.S. President, and he forced healthcare record-keeping to change from a slew of various options individually implemented by practices that overall worked well but not perfectly to a cookie-cutter electronic chart recipe (via the HITECH Act and Meaningful Use regulation in Obamacare) that works very poorly for just about everybody. Today it is essentially "pick one of 3-4 programs that because of the regs, are nearly identical." Nationwide and locally, the number of patients the physicians can see has dropped to about 60% of previous and stayed there despite the EMRs being mandated about a decade ago. Notably, it affects the younger physicians just as much as the older ones, so it's not just the "old crusty bastards" that are affected. Practice costs have skyrocketed, overhead in a typical office is up 50% directly due to the EMR costs and the large amount of ancillary staff required for their implementation, maintenance, security, and certification. This is also true nationwide as well as locally. The terrible state in health IT has been shown to cause large numbers of errors that didn't previously exist, widespread early retirement of physicians, and also is a major factor in physicians reducing hours. All of that decreases patient access and is exactly the wrong thing when you already have a notable shortage of physicians.

So sometimes IT change that is forced on an organization IS horrible. Health IT is a little different than some other fields as it was the government forcing something on the field, and unlike if the suits made the decision, we cannot ever get away from it unless laws change. But it is still a great example of why "change for change's sake" is something to be appropriately skeptical of.
 

Ralston18

Titan
Moderator
No disagreement per se.

However, to abide with the "no GRAPES' rules all I will say is that sizable campaign contributions via special interests/lobbyists have scuttled many good plans and legislation.

In all fields and Market sectors. Seldom an issue of doing it right.... Mostly a matter of who is going to profit.




 


No GRAPES topics were implied. The mention of legislation being involved was to illustrate how bad it is when bad IT is forced on a business (or entire industry) and we can do absolutely nothing to change it. Most of us responded by quitting if we could or cutting back hours. I don't have enough saved up yet to quit, so I cut back from 65-70 hours a week to about 45-50. The sad or perhaps fortunate thing is that my paycheck barely budged by cutting back that much. The fortunate thing is that my paycheck didn't change much, but the sad thing is that I made somebody else a bunch of money, just not me, when I was making myself miserable by trying to really work hard to make some money.
 

Ralston18

Titan
Moderator
Can certainly agree.

The recent missile warning in Hawaii is a good example of bad IT.

No common sense it seems - and in many disciplines and areas. Not just IT.

Seems to be a growing gap between proper development procedures and fielding a product.

If the powers that be spent as much time on making things as they should be versus generating lots of feel good (for themselves I think) PR we would all be better off.

Yes. Bad legislation is a plague on us all. Does not bode well for anyone.

The reasons and results behind it all is far more than I know and/or could type. Probably would need a dissertation to even start on it all.

Hope you can find a meaningful and viable balance.








 


The fake ballistic missile warning CMAS message was absolutely political in nature. The official had to click through a bunch of "are you really sure you want to do this" messages and knew absolutely what he was doing. A discussion of "why" would violate the GRAPES rule but suffice to say it involved North Korea's recent long-range ballistic missile tests and many Hawaiians, particularly government officials in HI, being of a different political persuasion than the current US President.

I have found a balance, I have simply pulled back and only devoted four 12 hour shifts per week to my job instead of 5 1/2-6. Much more time spent at home with my family and who cares if various others don't benefit as much from me doing less work. Again, they don't pay me hardly any less now, so no reason to work any more.