Customer constraints may affect your ability to tune

Any tuning effort will touch parts of a system. Some of them touch the database, others the client application, others the middleware layer, and some change them all. One element thus that measures the effort required for tuning is the number of system elements that have been changed. It is reasonable to assume that the more components you touch, the more complex the tuning is.

But I've recently realized that there is a portion of the complexity that is hidden in a tuning job: the number of elements that you would want to change but you cannot. Say, for example, that you could make the client go twice as fast by changing it, and you could make the whole thing run three times faster by changing the server. That's a six times speed-up factor. You prepare a plan for the changes and a proposal to your client. Everything looks great: you're heading to complete another successful engagement and your customer is going to be delighted with the results. Right?

No, not right. When you present your findings to your customer, shock and horror, you learn from the client that X part of the system cannot be changed. If you're thinking about this happening because of some "closed source" choice, think again. This can happen even when all the stack from OS to client is open source. People factors (specially training) and political factors can play a big role, to the point of not being able to touch some components.

Now that makes your job much harder. Following the example, you were going to provide a six fold performance improvement. Suppose that you cannot touch the client component and have to provide the same six fold improvement. Now you have to find those gains only in the server component.

If you've read other articles here or elsewhere, performance improvements follow the classic diminishing returns model, where at each successive step, more efforts in tuning derives less improvements in each iteration. This means that your job of getting six times more performance has become much more than twice as hard if you need to get all of it from the same component.

All this comes from a recent experience. I had conversations with the customer prior to start the baseline analysis stage, and I understood (wrongly) that he was so desperate for performance that he was willing to change anything in the system. When I revealed my findings on a specific component, it turned out that this component could not be touched. My customer had being assuming all the time that this specific component was not part of the problem, so he did not thought of "changing any parts of the system" as including this piece.

Lesson learned. Now, as part of my preliminary conversations with potential customers, I'm going to deep dive into the exact amount of change they are willing to have in each system component. Even if they think that the problem is not in the component that they will absolutely reject to change.