Concrete symptoms of over-engineering

I have recently found myself in the position of explaining an (In-House) application I have written to two candidates my company likes to hire in order to assist in maintenance and adding minor features.

It is the first "production" application I have written, it has 45k LOCs and I spent almost two years of "solo" development on it. I am fairly young (18) and wrote the application from scratch while being contracted as stand-in for a former developer who left the company. Unexperienced in designing applications of this size, I tried to use common architecture- and design-patterns.

Today I know I have done some serious over-engineering, e.g. using a disconnected change tracking architecture instead of the Unit Of Work pattern, which the chosen ORM has already implemented. I will probably never have to go "real" three tiers.

Both candidates have 10 years+ background in In-House Application Development with the relevant platform. Being half their age and having little experience I do respect their opinion. When I was explaining the application architecture to them, comments were along the lines of:

• Jeez, no one would pay me to do stuff like that, I have to get things done
• Stick with what the framework does, don't use fancy libraries/technologies
• Don't wrap framework code. On a team, everyone will write his own wrapper code anyway.
• You're using .NET 3.5? Well, we are using 2.0.
• What does that LINQ stuff buy me? All this query composition and projection seems too complicated.

Now I am asking myself:
Am I an architecture astronaut? How do I know I am going too far with architecture? What are common symptoms of over-engineering?

> What are common symptoms of > over-engineering?

Code that solves problems you don't have.

One very strong warning sign of overengineering is when everything goes through so much indirection that it's hard to find the piece of code that actually implements some concrete, domain-level piece of functionality. If you find that most of your functions do very little concrete work and just call other virtual functions, you may have a problem.

Boredom

Boredom is good precursor to over-engineered code. I'll admit, when I got my first job, I felt so underutilized. I was just bored. And when I got bored, I wrote code. Not just any code -- CATHEDRALS OF CODE.

No seriously, I had a mental picture of my code and abstractions as large towers with golden jutting spires, flying buttresses of glassy onyx, a wonderful vault supporting by arched domes topped with beautiful geometrical tracery, etc etc etc.

It was really fascinating to see the patterns working together for myself, but in retrospect, I am completely ashamed of the ungodly mess I left behind.

If you're writing your own frameworks and DSLs code to while away the less stimulating hours at work, just stop. Time is better spent reading http://www.c2.com/cgi/wiki?WelcomeVisitors">Wards Wiki, or writing http://en.wikibooks.org/wiki/Main_Page">an open source book, or you may just want to ask management for more work.

As for the question about if you an architecture astronaut: If you are aware of the danger that puts you ahead of a lot of people. You don't want to go the way of your cow-orkers either, it sounds like some of them have become crusty old whiners.

Over-engineering is the result of a problem with prioritization that resulted in some part of the system getting too much attention. So the most apparent symptom of over-engineering would be that you can see all around other parts of the system that are hurting for lack of attention.

(There is also a tendency for over-engineering to expose the system to increased risks of bad design, because of increased complication and the amount of error-prone speculation involved in deciding what aspects to over-engineer, but as a comment points out, that doesn't automatically follow.)

Writing your own Framework

Odds are, someone's already done it. More than that, they've already done it 1000x better than you ever could. More than that, whatever they've done is probably already an industry standard, so that learning the technology will make you more competitive at other jobs.

At the last company where I worked, a programmer had worked solo on his projects for most of his tenure. He wrote one of the company's more popular apps and was widely regarded as the best on the team -- but in my opinion, he had a nasty habit of writing everything he needed from scratch.

He'd written his own dependency injection framework, his own ORM, a unit testing framework (which, inexplicably, looked and acted very similar to NUnit -- why didn't he used NUnit?), a framework for creating factory objects (a "http://discuss.joelonsoftware.com/default.asp?joel.3.219431.12">factory factory" I'd call it).

Mind you, the code was actually remarkable, but what was the point?

Writing a better Core Library

At my current company, it always seemed like programmers writing useless amounts of code to replicate features already present in the .NET framework.

Among other things, they wrote:

• An active-directory framework for forms authentication in ASP.NET webforms -- inexplicable because ASP.NET has this function built-in.
• Hot-swappable themes and skins for websites -- also inexplicable, since the code was less functional than built-in ASP.NET themes and required 1000% more bloat.
• They inexplicably wrote their own typed data sets and data adapters. These objects provided less functionality than typed datasets which VS will autogenerate for you, while simultaneously requiring more boilerplate code than NHibernate domain objects.

Either they don't know the framework very well, or they think its notoriously inadequate.

There are only preciously few examples I can think of where the re-implemented library is better than the original (see http://ocaml.janestreet.com/?q=node/13">Jane Street Core Library, http://www.itu.dk/research/c5/">C5 Generic Collections for .NET, http://thinkarchitecture.blogspot.com/2008/07/making-money-with-c.html">a real currency class), but odds are, you won't write a better standard library.

For most in-house business applications, most of your code should be concerned with implementing business concerns, and not technical concerns unrelated to the business (like your "disconnected change tracking architecture"). The currently available frameworks are pretty mature and support most common use cases. If you're inventing new technology or (in the context of business-application development) just wrapping some other existing framework or library just for the sake of wrapping, you're probably doing it wrong. Ideally every piece of architecture you build should be traceable back to some business requirement. Keep it simple.

IMHO most of the comments you got about your application aren't really about over-engineering because over-engineering is not about technology. It's about architecture. New technologies can be learned and understood in a reasonable amount of time. Understanding an over-engineered application is usually much more difficult and sometimes even impossible. This makes the points 2, 4 and 5 invalid. The first point is not really valid because you obviously got paid for writing the application as it is and if it works you got no problem here.

This is my "quick test" to find out if an application tends to be over-engineered:

• Wrappers for "everything": Wrappers are useful but it's easy overdoing it. Check if you only wrap things that really need to be wrapped. (I basically wrapped my own wrapper once. I know what I'm talking about ;-) .)
• Reinventing the wheel: A classic. This is very common and you already mentioned it. Did you implement some functionality because you needed to your wanted to? What does your framework do what other available libraries don't?
• "Feels" over-engineered: This is the most important point but also the hardest point to see. Take a look at your code and look which parts feel overly complicated. Ask your self then if there is a easier way to implement it and why you didn't choose this way. If you got no good answer this part is probably over-engineered.

These are just quick tips which I use for my applications. They are not guaranteed to be the be-all and end-all of "over-engineering detection".

When a co-worker’s computer comes flying past your head because they’ve spent the last 6 hours trying and failing to make a freaking dialog box appear using your ridiculous framework, that’s a pretty concrete symptom right there.

Avoiding any use of YAGNI, DRY, and KISS come to mind in looking at things that are over-engineered. If there are many parts that seem to be partially completed and many parts of code that seem to have a, "What if this happens? What if that happens?" feel to it, that would be another point. Ignoring good principles of OO design or SOLID principles would be another to note. If you think you have written the perfect code, that would be another sign of trouble as it is extremely rare for anyone to write something that can't be improved in one way or another.

IMO, beware that some people may be overly critical of your work as part of any code base can involve people liking things a certain way, e.g. naming conventions on methods, tests and variables. That's just the way it is. Now, what you may have to figure out is how to handle people in siutations such as conflict or persuasion/influence, where there are tools that can help.

Plugins which provide intrinsic functionality to your app

Let's face it, pluggable architectures are just damn sexy and fun to write. However, this is another one of those areas where you need to ask yourself, "do I really need to do it this way?".

If you don't have a need for ad-hoc additions to your application, don't expect anyone to write third-party extensions, and the scope of your application fairly well-defined, you don't need a pluggable architecture.

You shouldn't write plugins to support intrinsic functionality in your app. Let's say you were writing a Paint program; you'd probably support plugins to save files in multiple formats, but you wouldn't need a pluggable undo manager or file browse dialog.

You're not an architecture astronaut. LINQ is pretty simple and basic and useful, for one. Same goes for .NET 3.5.

At the same time, you're the newbie on the team and going to get some kind of ribbing, even if they like what you did.

Take it all with a grain of salt. Just accept their criticism, nod, and have a beer with them afterwards.

If they ask you to change it, then your comment is "jeez .. I know i did it wrong, but it works and it's going to be too much trouble to change".

Try to evaluate if you have done things to minimize work over the expected life of the code. This includes maintenance as well as development.

In the code lifecycle, remember that the life of the code is not the same as the life of the application. It maybe better to write a quick prototype first, then re-engineer/refactor after better understanding of the domain. In this situation, the lifecycle is very short, so keep it simple.

In addition, different applications require different amouhts of engineering. Is this application going to cost lives if it fails? I.e. is it a controller for a mechanical heart, or for the space shuttle navigation rockets? Or is it a contact list for bored teens?

On a more generic and high level approach,

I feel that when there's a gap between the complexity of the problem and
the complexity of the solution, then you have a clear case of overengineering. 

What are the ways to achieve that? Solve problems that you don't have, seeing the problem more complex that it really is, trying to forecast too much in the future, building too generic stuff, among others.

Did you implement your project within a reasonable time frame? Is it working right now? If so, you can probably relax a little bit, as one of the worst problems with over engineering is never actually getting anything useful done.

The guys you talked too may have some good suggestions, but it doesn't mean you need to take everything they say as gospel. For example, using a recent version of .NET on a fresh project does not strike me as anything to worry about. Are they really complaining about that?

half of it is old men (i'm in their league) sticking to the tried and true steam engine they've known. the other half is true, but not really telling you anything new.

other than that, Jeff Sternal's answer is stellar.

As someone who has been developing software professionally for 24 years, all I can say is that this issue - making the call on the level of abstraction required for a particular problem - is the hardest part of my daily design & programming work.

In the "old" days, under-engineering was rife, which led to all the structured methodologies and practices. Now, it seems we've gone the other way. There is no easy answer, and sometimes you will only know if you under or over engineered in hindsight ;)

I've had multiple hindsight experiences where I said to myself: "I wish I didn't complicate this so much", but also "If only I listened to the little voice in the back of my head, and made it a bit more generic here".

I've yet to distill absolute rules for making the call...

I suspect I'm prone to over-engineering, so I've made this picture my PC's wallpaper

Kind of want to offer a different perspective, but I think this term should be struck out and replaced with a more appropriate description. It places a stigma on "engineering" which should be about simplicity and maintainability and practicality above all, when a lot of what is described as "over-engineering" has the precise opposite qualities (as though engineering too much, or taking it too seriously, is supposed to yield the most convoluted solutions). It is common, for example, to see a correlation between over-engineering and poor testing procedure (at least I have often seen these two go hand-in-hand), and what kind of excess in engineering performs the minimal amount of formal testing?

I say that coming from a standpoint where I often heard software managers and occasional dinosaurs who weren't so SE-savvy to throw this term around at anyone attempting to, say, reduce the maintenance costs in a system or propose more sound testing procedures and safety standards. It's too easy for the layman to think that any kind of serious focus on sound engineering, testing, and not producing code at the sprint-like snap-snap-snap pace they want is "over-engineering".

There are plenty who genuinely recognize the true smells of "over-engineering" but the term is at least arguably misleading.

As for the symptoms, besides the excellent ones already provided, to me it's a blindspot in being able to properly evaluate and reevaluate the work in front of you. Programmers can build worlds in their imagination, get obsessed and absorbed with towering concepts. While it's kind of a boring and generic answer, that can lead to a danger of no longer seeing the actual problems and priorities right under our nose. To conceptualize too deep is to often develop a blindspot as to what is right in front of us, to overcomplicate problems, to lose touch with reality and the actual user-end needs. To me the biggest symptoms to watch out for revolve around psychology.

All kinds of professions revolving around production are vulnerable to this basic trend where one loses sight of the actual and immediate problems to solve as result of getting too obsessed about concepts (ex: a film director). With programming in general, a way to combat it is to favor plainness, simplicity, practicality, coding sooner before conceptualizing something to the nth degree (buying us more time to reevaluate before we fall too deeply in love with any ideas). One can't go too far wrong embracing these ideals, but the key to me is to avoid developing that blindspot which prevents us from being able to properly evaluate our own work. An overengineered codebase doesn't come about overnight -- it takes a long amount of dedicated work in the wrong direction to yield it, and to me the biggest preventable issue with that is not foresight-related but often developing the proper hindsight too late.