Thursday
Jan172008
Moving old to new. Do not be afraid of the re-write -- but take some help
Thursday, January 17, 2008 at 6:10AM
Recently I had to help users on one of my opensource project ISPMan. http://ispman.net
This project started in 2001 as I was too unwilling to take care of the DNS and VitualHosting stuff as it was a side-thing to the company I worked for (so i wrote a software that took care of all these little details)
Summary: A large project that needs a rewrite can be done in a matter of day.
I will not give you a full case study about a project that went through a re-write but a case study about how easy it is to re-write something.
Details:
My boss was cool enough to let me open-source the project and obviously, I got a lot of cool-cred out of it.
Later on I also did some support and implementation and earned quiet some money with it.
Eventually I had to let the project go out of my hand to the community as I only did it to facilitate a job that wasnt williing to do. (Setup DNS zones of multiple servers, find out which host should host the website and put VirtualHost section there, find out which mail server should take care of the mailbox and create a mailbox, etc)
The project was quiet successful and there are a number of users who are using it.
One of the project members took himself to be the project manager and has been running the project since.
I have been out of this project almost 5 years (not much time, I have had 2 kids 4.5 years and 8 months old, and my job was very demanding). The stress from my job has weakened a bit now (It took me really 3.5 years to bring them to a stable "actually an oxymoron when we are talking about high scalability" state).
Back to the topic. We have had having complains about not having this feature and that for this opensource product.
I tried to put in this feature... "I was the main authour. How complicated would it be for me to add this new feature.." and eventuall I went "What the ...".
Yup, I hated my code, I hated everything about it. (as a programmer, as a sysadmin, as an op).
Yes it was me coded it 7 years ago, it was me who insisted it to be like that... etc.
But times have changes.. Its not 2001 any more.
So I want to re-write it.
Its not the first time that the idea of re-write was in place.
Couple of years the senior members (any one who had to deal with the code) wanted to re-write.
Its not extensible, its not pluggable, blah, blah, blah. Yup it was not written to do so. I was just written so I dont have to do the work which I did not wanted to and it served it welll... So if you want more extend it, re-write it, fork-it, you get my point (from the guy who wrote an app and opensourced it).
I understand that point of view of the developers too "Why do I have to know how you LDAP scheme works, or how does
Cyrus mail server works, etc"
The re-write part:
Using a PHP framework ( http://framework.zend.com ) ( something that almost did not exist in 2001 ) I was able to get something up and running in a couple of hours. http://ispdirector.net
This whole thing was not possible without.
* PHP5 (we moved from perl to php, in 2001 PHP was really Pretty Home Pages)
* Zend framework ( opensource frameworks were scarce in 2001)
* Some experience that LDAP as great it is, should not be where you put all your eggs
This particular example does not say that perl(or any other language ) is bad and php is good or ldap(or any other directory) is bad and mysql is good , its just how we did it for this particuliar project.
Oh and forgot about the "Get Some Help"
Tellling your colleagues that they have to move to a new API can be more difficult then to hear a few blah and blah by the secreteries moved from XP to Vista
This was about really moving from Perl to PHP, from PHP to Java, from Java to Perl, from Perl to Ruby.
The whole point is that it does not matter.
If you can do X faster than Y than I take you (In compute intensive scenario)
If you do all your calculations this way, it might go somewhere.
This project started in 2001 as I was too unwilling to take care of the DNS and VitualHosting stuff as it was a side-thing to the company I worked for (so i wrote a software that took care of all these little details)
Summary: A large project that needs a rewrite can be done in a matter of day.
I will not give you a full case study about a project that went through a re-write but a case study about how easy it is to re-write something.
Details:
My boss was cool enough to let me open-source the project and obviously, I got a lot of cool-cred out of it.
Later on I also did some support and implementation and earned quiet some money with it.
Eventually I had to let the project go out of my hand to the community as I only did it to facilitate a job that wasnt williing to do. (Setup DNS zones of multiple servers, find out which host should host the website and put VirtualHost section there, find out which mail server should take care of the mailbox and create a mailbox, etc)
The project was quiet successful and there are a number of users who are using it.
One of the project members took himself to be the project manager and has been running the project since.
I have been out of this project almost 5 years (not much time, I have had 2 kids 4.5 years and 8 months old, and my job was very demanding). The stress from my job has weakened a bit now (It took me really 3.5 years to bring them to a stable "actually an oxymoron when we are talking about high scalability" state).
Back to the topic. We have had having complains about not having this feature and that for this opensource product.
I tried to put in this feature... "I was the main authour. How complicated would it be for me to add this new feature.." and eventuall I went "What the ...".
Yup, I hated my code, I hated everything about it. (as a programmer, as a sysadmin, as an op).
Yes it was me coded it 7 years ago, it was me who insisted it to be like that... etc.
But times have changes.. Its not 2001 any more.
So I want to re-write it.
Its not the first time that the idea of re-write was in place.
Couple of years the senior members (any one who had to deal with the code) wanted to re-write.
Its not extensible, its not pluggable, blah, blah, blah. Yup it was not written to do so. I was just written so I dont have to do the work which I did not wanted to and it served it welll... So if you want more extend it, re-write it, fork-it, you get my point (from the guy who wrote an app and opensourced it).
I understand that point of view of the developers too "Why do I have to know how you LDAP scheme works, or how does
Cyrus mail server works, etc"
The re-write part:
Using a PHP framework ( http://framework.zend.com ) ( something that almost did not exist in 2001 ) I was able to get something up and running in a couple of hours. http://ispdirector.net
This whole thing was not possible without.
* PHP5 (we moved from perl to php, in 2001 PHP was really Pretty Home Pages)
* Zend framework ( opensource frameworks were scarce in 2001)
* Some experience that LDAP as great it is, should not be where you put all your eggs
This particular example does not say that perl(or any other language ) is bad and php is good or ldap(or any other directory) is bad and mysql is good , its just how we did it for this particuliar project.
Oh and forgot about the "Get Some Help"
Tellling your colleagues that they have to move to a new API can be more difficult then to hear a few blah and blah by the secreteries moved from XP to Vista
This was about really moving from Perl to PHP, from PHP to Java, from Java to Perl, from Perl to Ruby.
The whole point is that it does not matter.
If you can do X faster than Y than I take you (In compute intensive scenario)
If you do all your calculations this way, it might go somewhere.
Reader Comments (14)
A total rewrite, build from the stratch almost makes things clearer, better, as the previous revisions - since the world, the surrounding technologies also changing and you just simply cannot separate your code from them anymore.
That's great for side projects and open source things that don't have deadlines, but you'll never see this sort of behavior in a corporate environment (or at least, you shouldn't). 99% of the time, rewriting something, unless the rewrite gives you something you couldn't have gotten by just editing the existing code, will take way longer than you want it to. Time = money, and you're not going to see a real company dump money because "you don't like your code".
If you want to do this inside a corporate environment, make sure you can justify your actions with new features and new ways to make revenue, or they're going to deny you on the spot.
Love your Avatar!
Worst article so far....
Rewriting something trivial isn't a problem
Anonymous,
Its not about rewrittin something trivial or complex.
Its about scalability. Sometime to achive scalability, you have to stop patching and do a clean re-write.
So, you are talking about choosing the right language to re-write?
@UltimateBrent
I disagree. I've been in touch with way too many (former) startup-companies that
are losing real money because they are and were too scared to start over from scratch
in time.
Unless you are google then your first shot at a problem is almost never "right".
You will make design mistakes and fixing those on a living and breathing codebase
while trying to develop features against this very codebase is, in many cases,
easily the most inefficient way to go about it.
The same applies for transforming your initial hacks (time is money, especially when you're on angel money...) into something that can scale if your business becomes successful. I have seen many businesses where this constant state of "refactoring"
has eaten all available ressources up to the point of effective stasis. This is especially true when scalability is a concern because you can't just sprinkle "magic scalability dust"
over your existing spaghetti design.
These companies often live off an edge that they gained in the early days but
are very vulnerable to all kinds of attacks - because they are literally unable to
respond to a threat or rebuild any kind of momentum, even in a live-or-death situation.
"Rewrite is less effective than refactoring" is a deadly myth.
If you look at the big internet names of today then *all* of them have
rewritten and re-architected major parts of their codebase not once but
multiple times in the past.
Its still a trivial task. Rewriting something trivial is easy and quick. The website is about google, flickr and other large scale projects and not private hobby project
Regardig the rewrite .. As long as you
a) are in business (and stay because the old release is still available until you release the new one ) you can
b) rewrite your app, if you want to or you have to
again, I see no correlation between an ebay application/website and your own private project
Anonymous,
I used my side project as an example here because I cannot talk about my employer's or client's projects.
Small or Big, Sometimes a rewrite can save the project. A rewrite does not only mean "code for 6 months". There are other techniques of doing it for non-trivial projects.
Anyway, if you think, its irrelevant, then sorry to have wasted your time.
Ultimate Brent,
The rewrite should NOT be done because "You dont like your code". I agree that a very hobby reason to do so.
It should be done for reasons such as
* You might not be able to handle the next doubling of your customers
* you might not be able to handle the next doubling of the machines etc
In these cases, there are not too many choices but to either hack/patch the code or go for a complete fresh approach.
My point is not that everytime there is an old project that starts to show its age, It should be re-written.
My point is that sometimes re-writting is a choice and should be kept open as a choice.
@Moe
>"Rewrite is less effective than refactoring" is a deadly myth.
>If you look at the big internet names of today then *all* of them have
>rewritten and re-architected major parts of their codebase not once but
>multiple times in the past.
Again, the main point is whether you can remain in business while rewriting. If rewriting means your business stops and you cannot do any business, then of course you do not rewrite, but refactor => desktop apps.
If you can stay in business while rewriting you can do it.. Also once you are really in business with a site you have the budget and the overall capabilites to refactor the whole thing to enable scaling.
The whole point about creating ultra-high-scalability sites from scratch is a bit ridiculous because its more important to roll out and get sume customers first. Time to market.
Here's the obligatory "Joel On Software" article on how rewriting from scratch is a huge mistake:
http://www.joelonsoftware.com/articles/fog0000000069.html
Hello Atif,
Long time! I hope you are well!
I for one agree with you. Rewriting is not something to be taken lightly, but to try to avoid it when it becomes obvious that it is necessary is to risk the entire project.
Refactoring != rewriting. Refactoring is important, but refactoring is an iterative process. You should really never stop refactoring.
A complete, ground-up rewrite can reap HUGE gains in performance, productivity, and scalability. Simply put, if you are considering moving to a totally different architecture (i.e., cloud/grid architecture rather than a single writes blocker such as SQL or LDAP), a rewrite is simply required, and the sooner the better. Don't put it off. Or what if you want to change what platform or language your system is designed for? This is a strategic decision. A language change is a legitimate reason to change.
(I've heard it contended that Java is as capable as any other language. Clearly, that's a self-defeating statement. No language can be as capable as any other language. Let us say that BASIC is as capable as LISP. True? of course not. Let us say that Java is as capable as Haskell or Erlang. True? In some ways. False in many others. Many people don't like the argument that "my language can beat up your language" but just don't make claims that they are all the same, because it only exposes limited experience.)
On the other hand, if you can limp along for a while on your current codebase, maybe you should. At this point it becomes a business rather than a technical decision.
To insist that rewrites are a huge mistake, or that refactoring can somehow fix a misdesigned product... that's just absurd. You are not the same person you were five years ago, and neither is your code base. The benefits of time and experience are on your side.
A big problem is that you often still want to take advantage of libraries that you have already lots of time and money in. This becomes much harder to do when a language change is necessary. Sometimes the old system can become a client for the new system, with the new system calling the old system for some things. Sometimes you can shell out or use web services or RPC to access the old stuff. And, sometimes, you just have to throw away the old stuff and write from scratch.
There are legitimate arguments to be made on both sides. To argue that Atif's simple statement "don't be afraid to rewrite" is wrong reflects as unwillingness to think outside the box -- to realize that conventional wisdom is just that and doesn't apply across the board.
Regards,
Jamie
"The whole point about creating ultra-high-scalability sites from scratch is a bit ridiculous because its more important to roll out and get sume customers first. Time to market."
This is what Friendster thought. In some ways, it's true, but in many ways it's not. If you do not have a solid foundation, then you will HAVE to rewrite later.
That's why it's called scaling, right? ;-)