Aaron N. Tubbs bio photo

Aaron N. Tubbs

Dragon chaser.

Twitter Facebook Google+ LinkedIn Github

My primary job role is application support. That means a lot of things, but in theory, these are my core job responsibilities:

  • End-user application support for proprietary trading applications
  • Liaison with other IT teams and business units during testing and coordination of infrastructure upgrades
  • Performance management and load balancing of trading applications and infrastructure
  • Implementing and following through change management processes
  • Coordination of strategic, tactical, and system integration projects between business and development teams

If only it were that simple. In the situation where you have an established system that is mature, robust, well documented, and changes infrequently, that is roughly what an application support role requires. Monkeys could do that job, and it’s the type of situation that screams something like HBR Offshore Labor Case Study #13652.

There are, unfortunately, hypothetical systems that do not fall into this category, and instead tend to involve words like fledgling, disorganized, incomplete, chaotic, unspecified, untested, emergency, patch, bugfix, and so forth. We are talking about systems where “we don’t have any idea, we’ll see what happens in production, because it’s too hard to model” is the answer so “what is the performance impact of this change?” We’re talking about systems with rolling implementation windows, where production decisions are made day of, rather than according to a project plan. We’re talking about systems where “the library is broken, we don’t have the time to interface with them, so you’ll need to develop a work-around to pre-process the data” is a typical situation. We’re talking about systems that wake people up at night and cancel their vacation plans. We’re talking about users that will not accept somebody who speaks even slightly poor English. We’re talking about users that require somebody who has both a technical understanding of the system and an understanding of and intuition for the underlying financial details. And yes, before things get out of hand, we’re talking about situations where one’s ability to fix all of the above problems is severely limited due to various management edicts, no matter how hard one attempts to kick, scream, and change things without permission.

In this hypothetical scenario, we have entered application support hell, and trained monkeys are no longer sufficient. Instead, you need somebody with a different set of requirements:

  1. They must be software developers. Without that skill set, they’re lost when working with the development team, and will be a victim of their decisions, rather than an agent for process improvement. What’s worse, they need to be better developers than the developers, or else they will not have the respect of the developers. Additionally, if they are not developers, they will not be able to understand why software breaks, and what needs to be done to resolve it. Finally, when confronted with a situation where a work-around is necessary, if they are not developers, the only way they will know to work around this process is through constant manual intervention, which is unreliable, mistake-prone, and a waste of time.
  2. They must be problem solvers, and know how to solve a problem without a pre-defined plan to follow. I also do not think it is enough to be a problem solver, but they must be motivated to actually solve the problems, and have the drive to follow them through to resolution. By leveraging their past experience and knowledge of other problems, they need to recognize signs that point them to the root cause of the new problem, and ultimately its resolution. In completely undefined situations, they still need the ability to reason about what they are seeing, and try to infer a reasonable explanation for what is happening, even if they cannot pinpoint the exact cause.
  3. It is necessary that they have the motivation to make their job easier. If all they do is accept their situation, do the same repetitive tasks, and watch the same processes fail in the same way day after day, they cannot be an agent for change, and the system will never improve.
  4. They must be able to stay motivated to keep doing application support (to a certain extent this is the responsibility of the manager, but finding good managers in application support is even harder), despite the fact that they have the skill set to do something that is much more interesting.

I used to think that you could approach this problem by ignoring one of the above requirements. I no longer believe this is realistic for the aforementioned situation. I now think this sort of approach, complete with front-line monkeys (Level 1), intermediate slightly-skilled chimps (Level 2) and people that fit the above requirements (Level 3) really only works for mature supportable systems.

The bitch of it all is that finding people with these requirements that will fill application support roles instead of core development or business roles is just about impossible, and keeping people that have even one or two of the above motivated and satisfied in this sort of job is even more difficult. The upside to being in an application support role is a potentially faster route to middle management and in the ideal situation greater visibility; the downside is vast, at least in terms of somebody that would typically come to the job with the above skill set. In fact, the most unintuitive aspect of the job is that if one does it to the fullest possibility, they eliminate their job. All told, this is a terribly difficult sell.

The thing is, this sort of hypothetical system is all over the place, and people have to hired to support it, which leads to a number of compromises. I’m not sure how to solve this problem. Based on some previous thoughts, perhaps starting a firm with exceptional branded application support folks for tactical outsourcing is the answer. Even then though, retaining this sort talent in the long term would be difficult, because the only real motivation that can be offered is compensatory, and the drive provided by bonuses and more money starts to lose its ability to compensate for the pain eventually.

Then again, maybe I just have a skewed perspective and this is all melancholy hogwash.