Explaining why "Just add another column to the DB" is a bad idea, to non programmers

I have sales people and bean counters who are trying to sell customizations to clients, which is fine. But when a complex change request comes in that I send back a large estimate for, they get confused. Often they come back at me with "Why can't you just add another column?" which by another, they mean a dozen or so custom columns PER client.

So far all I can come back with is "We are trying to keep the database well normalized" which means nothing to them. I tell them I can create a system of tables that allows each client to define their own set of custom fields, but of course that takes more time and money than "just adding a few columns". And of course they want to have their cake and eat it too.

So how can I make them understand?

> I tell them I can create a system of tables that allows each client to define their own set of custom fields, but of course that takes more time and money than "just adding a few columns".

I think you should push this option to your bosses since customizability is obviously a feature much in demand. Emphasize that an individually customized (rather than generalized, limited customizability) system for each client means that patches and updates will have to be created for each individual client (leading to longer roll-out times and higher costs); that non-standardized installations mean that HelpDesk tickets will take much longer to close (leading to dissatisfied clients and higher costs); etc.

In other words sell short term pain for long term gain by showing that the cost of their solution far outweighs the cost of your solution.

Salespeople are focussed on making the sale. That's what gets them their commission. They don't care about what comes after. Bosses, however, are focussed on cost. Sell to your bosses and your bosses can sell to the salespeople.

The best way I've found is to show how you can create a new feature out of what they're asking for that you couldn't add with just a couple customized columns. Features are better than customizations, especially when you can charge someone for it.

Try to make a good business case for your side before you get into the technical stuff.

Ah.. a little knowledge is a dangerous thing.

Try this one:

You: Which companies did we fail to sell to?
Sales: Acme Industries, OCP Corp, blah blah blah
You: Well.... why can't you just make a couple of more phonecalls?

The answer of course is sales isn't that simple. Neither is software development. Unless they really want hours of explanation in regards to architecture and maintenance I suggest they trust your judgement as a software developer.

This is the issue here, trust. You should explain to them they are displaying a lack of trust in your abilities by making these statements.

You can tell them that a poorly designed database means that in the long term:

• it will take longer for them to retrieve their data - do they really want to wait and wait?

• it will be harder and take longer to design queries to generate reports - again, if they need that query tomorrow, do they want to be told that it's still being worked on?

• it will be a nightmare to maintain, with error prone queries more likely to be written.

If they're sales people and bean counters, then they will definitely understand the almighty dollar (pound, euro, etc.). Can you demonstrate that the time spent to maintain these extra columns doesn't justify the added sales?

Be very careful here and make sure your argument makes sense. I've found myself resistant in the past to doing customizations more because I didn't want to ugly up my pretty little domain model than because it would really be that difficult to maintain. A decent analysis will help you determine why you're resisting the customization.

Remember - the bottom line is that you need to keep clients happy in order to stay in business. We thoughtful developers can sometimes lose sight of that in our quest to make things maintainable and simple.

Google "technical debt"; Show them the results.

If you're in the business of selling a product along with customizations, the product needs to support customizations without having to fork the build each time they sell it.

Sounds like you've tried explaining that, to no avail. Instead, try estimating the cost of adding your "customization the right way" for one table with maintaining, say, half a dozen versions of the product with different customizations, and fixing a bug across off of them. My bet is that they will see that they're pretty soon money ahead having a unified codebase and schema. And a developer who isn't driven insane.

Tell them that when people make a car and then they want a model that goes faster and does more than the previous, they usually don't add another engine.

The problem is that "We are trying to keep the database well normalized" is almost certainly the wrong answer - it plays back the ball into the court of mistrust and cross-purposes.

You have got to turn focus back onto the end goal, how best to meet that end goal (perhaps in several releases) and what it will cost in the short and long term. I've seen mention of technical debt in answers and cost estimates should take those factors into consideration.

It might not be a bad idea to "just add another column?". You really haven't given the entire business case. On the other hand, they have challenged your negative response with an ignorant technical question. That doesn't get to the heart of helping you understand the requirement because they didn't like hearing "no". (I'd like to know what the original statement of the problem was.)

Making the database normalized is a technical problem and has no bearing on the requirements the system must satisfy - it is a system design principle which you use to deliver systems with certain properties like maintainability. But a maintainable systems which don't meet user needs have zero value, while unmaintainable systems which do meet user needs have non-zero value (which might be exceeded by the cost of maintenance - which is a business problem). Whether EAV or some other mechanism is required is not really the point either - that just causes system complexity or cost to increase.

If the requirements are too expensive to ever implement, then that's part of the business case. You haven't told us enough about the architecture or the type of entities these tables model. Say you have 100 clients. There may be overlap in columns in a particular entity. Just as many as 95% of clients will never use the optional Billing-Address or a Middle-Name column, that doesn't mean those columns are left out - not only that, they are often in an original design! Alternatively, if this is a Products table and every client wants many special columns and there is no overlap, you might need a user-defined field system (EAV/XML/tag - the design will have to match the requirements) instead in order to maintain a cohesive system design.

I have rarely found business to ignore a technical debt argument - particularly if a proposed solution can be shown to meet the user needs and flexibility can become a selling point. What I have found is that business will often prefer if you present solution choices as quickly and thoroughly as possible without spending more time explaining why something can't be done or how much it's going to cost than it would take to buckle down in a couple afternoons and actually getting the work done.

I've never tried this myself, but I've thought about it: draw an analogy to the legal system. Legal loopholes exist because law makers try to patch the system with lazy kludges. The software equivalent is bugs, security holes, etc. The only way around these problems is careful planning and hard work.

Make them understand how much that costs in development time, will this change require 1 or two developers time? what about testing? if complex requests cost more then the company as whole is making less on the job. The account / project manager should be the middleman who's job it is to buffer these type of requests.

You won't get anywhere explaining it to them in technical terms. You need a metaphor. Tailor it to the person you're talking to, if you can. If he/she is a car freak, get them to think in terms of engine modifications. How much would it cost Ford to offer three different motors in the Taurus, or custom mods on demand?

Once they accept that comparison, even if they don't fully understand it, you can begin to get into why the metaphor applies.

There's another great way to help them see it your way- take some time to see it their way as well. When your paycheck depends on giving the customer what they want, you don't care what the propellerhead in Engineering tells you. If you're getting a lot of requests for customization, you should consider the architectural and strategic approaches to delivering those customizations, wherever possible.

...I tell them I can create a system of tables that allows each client to define their own set of custom fields, but of course that takes more time and money....

Looks like you want to build some kind of generic data model? Entity-attribute-value...?

Those generic models are often real slow, they can't be indexed properly and confuse the query optimizer. It is often better to just add some columns.

Do some very thorough benchmarking before going the generic road.

Maybe it is db vendor dependent but if you use Oracle, I would prefer the 'just add some columns' road above the entity-attribute-value-road.

To expand on tuinstoel's suggestion (avoid generic entity-attribute-value structures): While I generally like this structure for light use, excessive (whatever that means) usage will degrade performance as noted. Such structures cannot be well indexed. I wrote and supported one such system. By the time we had 50,000 "entities" each with 10-100 keys it was SLOW even on midrange hardware).

However, they are very useful and fairly easy to implement. So if there's a need for many arbitrary "extra fields" to be added on a per customer basis, then it may make the most sense.

Another possible option might be to add a number of unused generic column in appropriate tables to be used by clients for their own purposes. Some enterprisy applications do just this. A Sales table might have ten or twenty CUSTCODE01 to CUSTCODE10 columns which each deployment of the application can use in different, wholly custom way.

This may at first look horrid, it may also be a happy medium. There is a fair amount of room to customize on a per-customer basis without a) "just adding a column" and disrupting the application or development process, or b) implementing a potentially slow generic system. You only get a limited amount of felxablity, though, and there is a lack of self-documenting column names (but column descriptions can be customized as needed).

You can explain this problem drawing a comparison with a library. There are many books. Small one and big one, thin and thick ones - everybody can imagine that. Now if you want to store more information somewhere it would be rather simpler to add some new pages to a book than enlarge some single pages - if there are several pages of a book larger than the others, this not very robust and how would one find this information if it has no entry in the index of contens? Maybe it is better to store the new additional information in a further book, a new one with a particular structur. Imagine how one may get a information if the whole contens of a library would be written in one big thick book? Nobody else could find anything until you find what you want and set the book back at its place...if you are able to carry this enormous book. Why retrieving the whole Livestory if you only want to know the birthdate of a person?

The mentioned people don't have to understand the architecture of a database but they should trust you. And you organize it so that they can throw their information in this big hole of database and get it back when ever they want it - fast and reliable.