Wednesday 17 November 2010

CAB and TAB: a waste of time for everyone



I've been trying for weeks to write a piece on ITIL that did not sounded vindictive or bitter. I really wanted to be fair on ITIL, or to be more precise, I'm all in favor of having some kind of change control and IT management in place.

But having to go through the ITIL recommended change management processes a couple of times has revealed me the true costs and overhead of ITIL. I'm currently immersed in an organization that drunk the ITIL kool aid a few years ago. And so far, my experience with the ITIL implementation is very, very negative.

A couple of stellar examples of this are the Change Approval Board (CAB) and the Technical Approval board (TAB). According to ITIL, these boards should review and approve every change made on production environments. In theory they are composed of the people with the most extensive technology and business knowledge. This is because they are the last line of defence before anyone can touch live systems and wreak havoc.

So far, theory is good. This small but selected team of individuals can detect and sort out incompatible changes, and block them if they step into critical business periods, even deny them completely if they don't fit with the technology standards. To make things more convenient for everybody, and unless there is an emergency, the team meets on a weekly basis. Otherwise, they would be unable to do any other work than review and approve changes.

Let's try to determine the composition of this CAB and TAB teams. For a big organization, you'll have attending CAB and TAB meetings someone with knowledge of the ERP system, right? Oh, yes, and put in some networking guys. And some storage people. With luck, your organization has also a CRM application, so add one to the mix. And is far too common, especially in big multinationals, to have their own localized versions of ERP and CRM packages. Well, also you probably have some customer facing web sites, do you? Another one to add. Done? Not yet, then there is the desktop support people and the security and incident response teams probably have something to say about changes. And don't forget the server maintenance team. Ooops, sorry, if your CRM and ERP have a database component, let's add one DBA to the mix, possibly one for each database flavor.

Done? Wait, are there systems related to production facilities? Add one more. And so on. The point is: in complex and big organizations, the technology environment is complex and the knowledge is fragmented across many different teams. Each and everyone of them needs to be represented in CAB and TAB meetings.

I see a few problems emerging from all this:

One, while the size of the team is by itself something to worry, that's nothing compared with the different degrees of connection between them. The networking team usually do not talk very much with the ERP team, while the DBA team is very close to the storage team but almost completely ignorant of what the networking team does. And so on. For something so focused on productivity as ITIL, it is surprising that they dictate allocating a lot of valuable time from valuable people just to approve changes. More so when a fair amount of those changes are tangentially, if at all, interesting for them. Seriously, if the networking guy wants to add a firewall rule, do you really think the ERP guy is going to stop him and ask if he's using the right subnet mask?

Second, these approvals are holding off changes to a weekly cycle. Rare is the project or change that can be executed ahead of schedule, but if that happens, forget about applying your changes earlier than planned unless you've managed to save a whole week of time and can catch up the prior approval meeting.

Third, the cost of validating the change could be very well higher than the cost impact of the change going wrong. ITIL is full of "everything has to be managed" in the name of being able to find efficiencies, but little is said of the cost of this management.

Clearly ITIL does not see those things as problems. And reading around ITIL, I think I've found why. Picture this, when you last saw a small group of highly skilled individuals that could have visibility and knowledge of a big production environment, whose cost was already sunk in other activities and working in organizations with long, long, change cycles?

The mainframe world, of course. ITIL was created for the mainframe shops, and it is likely to be a much better fit for those environments.

For the rest of today's world, CAB and TAB are a waste of time. Any half baked web based workflow management system can do better, and provide the same level of change control. Plus, it allows the DBA guy to skim over the network changes, and the network guy to just surface scan the ERP module changes. The problem? It would not be ITIL. The consecuences of that are the subject of another post, probably.

But I don't have time to write that post, I need to attend a CAB meeting now....

7 comments:

  1. My experience with CAB and TAB is even worse! It is bad enough when CAB/TAB members are technically competent, but imagine them being populated with process managers who couldn't even spell TCP/IP let alone understand what its purpose is. Usually, changes are denied until the change requests (I refuse to call them RFCs, because anyone with a basic understanding of IT knows RFC means Request for Comments!) are forwarded by the process managers to the IT department heads, and then (maybe) on to someone who knows what they are doing. Add the time taken to reverse this procedure, and months can go by for even simple changes.

    ReplyDelete
  2. This is a cool review I have just started working in these environments and my head is fried with sitting through these meeting waiting for a possible issue that is relevant to me.

    ReplyDelete
  3. i am reading this while attending a CAB call.

    ReplyDelete
  4. Reading this while TAB call....and waiting for my CR

    ReplyDelete
  5. Well... I can understand why technical people find it hard to channel their activities as per a given process. Anyways, here is what i think:
    CAB/TAB is basically a platform for you to announce your changes to the different business functions on a single forum - this looks like the best way to give your changes the required visibility - hence a CAB is required.

    For the weekly cycle thing - if you really think a change is really minor and does not need evaluation on a CAB call - propose it as a standard change. Once its acknowledged as a standard activity - you can by pass CAB.

    Also, 90% of the times changes might not require deep questioning or probing and are generally well understood. But for 5-10% changes are risky and may need proper visibility - Its better to be proactive in communication than saying SORRY afterwards. Also, not all issue caused by changes might be critical - but sometimes it takes 1 change which was not properly reviewed and that might cause millions of dollar's of revenue impact. Now, we see that the chances of a change causing issues or impacting business are not that high + IT Teams also put in a lot of effort to make sure everything is tested, communicated, approvals are in place - but still someone will make a BOO-BOO somewhere. Hence as a business function, you must ensure that whatever measures we can take to evaluate and assess a change from all perspective - we should do that. That's where CAB comes into picture.

    From my personal experience, If your changes are ready and planning is proper - CAB/TAB can't hold your changes. And if there are any eyebrow raisers about your change on a CAB/TAB - It means you need to learn and plan better next time - that how you become more efficient with your change planning.


    ReplyDelete
  6. Addition to above - looks like you guys like to work in Silos and don't want to be questioned on anything. If you want to grow - you need a more holistic approach to the planned changes and should respect CAB/TAB discussions. Generally, no one would have a question for you because the scope you are managing is small, but once you are at SMT level - you want to be aware of everything happening in your Infrastructure - So try to learn from these discussions and grow as a professional.

    ReplyDelete