How to get started building a web browser?

I decided to put some effort in building a web browser from scratch. What are the common functions, architectures, and features of modern web browsers that I should know before getting started?

Any recommendations are highly appreciated!

Well break it down into pieces. What is a Web browser? What does it do? It:

• Fetches external content. So you need a HTTP library or (not recommended) write this yourself. There's a lot of complexity/subtlety to the HTTP protocol eg handling of expires headers, different versions (although it's mostly 1.1 these days), etc;
• Handles different content types. Theres a Windos registry for this kind of thing that you can piggyback. I'm talking interpreting content based on MIME type here;
• Parses HTML and XML: to create a DOM (Document Object Model);
• Parses and applies CSS: this entails understanding all the properties, all the units of measure and all the ways values can be specified (eg "border: 1px solid black" vs the separate border-width, etc properties);
• Implements the W3C visual model (and this is the real kicker); and
• Has a Javascript engine.

And that's basically a Web browser in a nutshell. Now some of these tasks are incredibly complex. Even the easy sounding ones can be hard. Take fetching external content. You need to deal with use cases like:

• How many concurrent connections to use?
• Error reporting to the user;
• Proxies;
• User options;
• etc.

The reason I and others are colletively raising our eyebrows is the rendering engine is hard (and, as someone noted, man years have gone into their development). The major rendering engines around are:

• Trident: developed by Microsoft for Internet Explorer;
• Gecko: used in Firefox;
• Webkit: used in Safari and Chrome 0-27;
• KHTML: used in the KDE desktop environment. Webkit forked from KHTML some years ago;
• Elektra: used in Opera 4-6;
• Presto: used in Opera 7-12;
• Blink: used in Chrome 28+, Opera 15+, webkit fork;

The top three have to be considered the major rendering engines used today.

Javascript engines are also hard. There are several of these that tend to be tied to the particular rendering engine:

• SpiderMonkey: used in Gecko/Firefox;
• TraceMonkey: will replace SpiderMonkey in Firefox 3.1 and introduces JIT (just-in-time) compilation;
• KJS: used by Konqueror, tied to KHTML;
• JScript: the Javascript engine of Trident, used in Internet Explorer;
• JavascriptCore: used in Webkit by the Safari browser;
• SquirrelFish: will be used in Webkit and adds JIT like TraceMonkey;
• V8: Google's Javascript engine used in Chrome and Opera;
• Opera (12.X and less) also used its own.

And of course there's all the user interface stuff: navigation between pages, page history, clearing temporary files, typing in a URL, autocompleting URLs and so on.

That is a lot of work.

Sounds like a really interesting project, but it will require you to invest an enormous effort.

It's no easy thing, but from an academic point of view, you could learn so much from it.

Some resources that you could check:

• HTMLayout.NET: fast, lightweight and embeddable HTML/CSS renderer and layout manager component.
• GeckoFX: Windows Forms control that embeds the Mozilla Gecko browser control in any Windows Forms Application.
• SwiftDotNet: A Browser based on Webkit in C#
• Gecko DotNetEmbed
• Gecko#
• Rendering a web page - step by step

But seeing it from a realistic point of view, the huge effort needed to code it from scratch reminded me this comic:

_{(source: geekherocomic.com)}

Good Luck :-)

Most modern web browsers are giant beasts, and probably fairly poorly designed because they (and the web itself) evolved in a rather haphazard way.

You need to start by first making the goals of your project (and what you hope to achieve) very explicit. Is this something you're just doing for fun, or do you expect other people to use your browser? If you expect others to use it, what will the incentive for them be? It is unrealistic to expect that you will develop a new browser from scratch that everyone will be able to use as a replacement for Chrome, Safari, Firefox, IE, Opera, etc. All of those projects have a 10-15 year head start on you, and by the time you've caught up to them, they will be another 10-15 years ahead of you. Plus they have a lot more man power behind them, and so if you want your project to be successful, you will need that man power at some point.

This is the reason that Apple and Google, big companies with lots of resources, did not start from scratch. Not even Microsoft started from scratch. The original IE was based on Mosaic. The only significant browsers still around today that were started from scratch are Opera, Konqueror and Lynx, which unfortunately all have minuscule market share. Let's forget about Lynx for the moment, since it's a text-only browser and presumably the only reason it's still around is because it serves that specific niche. Opera is arguably one of the best browsers ever made, and yet it's never had a great market share, so remember that success and innovation are not the same thing. KHTML is the engine behind Konqueror, which never itself became very successful, but is the basis of WebKit that both Apple and Google use. I think one could definitely argue that if KHTML had never been made, neither Safari or Chrome would exist. Interestingly enough, both KHTML and Opera were largely produced by Norwegian programmers working in the same building in Oslo.

You need to look at building a web browser like building an operating system, because that's essentially what a browser is -- it's an operating system for running web apps. And like an operating system, a web browser is a very complex piece of software with many components. Of course, people have been successful at creating new operating systems from scratch. Linus Torvalds comes to mind. He made Linux, one of the most successful operating systems ever.

Of course, you face an additional challenge, which makes building a new successful browser harder than building a new successful OS. Browsers are expected to flawlessly run all the legacy code floating around on the web. Now suppose that Linus Torvalds had been told that his new OS wouldn't matter unless it was perfectly backwards compatible with UNIX or some existing OS. I doubt he would have bothered, and Linux probably wouldn't exist today. Realistically, of course, the only reason Linux became popular was because it was designed well and the GNU project was able to make tools for porting huge amounts of existing code to Linux. Without GNU's ideological support for Linux, it never would have had a chance.

So assuming you really are ambitious (or foolhardy) enough to try to make a new successful browser, the thing you should be focusing on is architecture and design. There is no practical reason to build a new browser from scratch unless you are sure you can improve upon the design of existing browsers in some way. That means you need to familiarize yourself enough with the code of WebKit and Gecko enough to understand the design decisions they made, but you shouldn't be attempting to copy their design because otherwise you might as well just use their code.

My personal thoughts (without having done enough research) is that today's browsers are not modular enough. If I were going to make a new browser, I would find a way to make it easy to swap things in and out (like replace one JavaScript engine with another), and give the user a lot more control than they currently have with existing browsers. Modern browsers and web designers have taken almost all control away from the user. Why can't I, the user, tell the web browser how I want it to render content being displayed on my machine? The original HTML only gave guidelines for how to structure content, and over time, newer standards have become more and more dogmatic, to the point where the user is now at the total mercy of the web designer. The appeal of Linux was that it gave back control to the user, and that's why so many geeks supported it and made it into a successful OS.

The other thing I would spend time researching, if I were you, is operating system design principles. Designing a good browser, should at least in theory, require the same principles as designing a good OS -- especially in regards to concurrent processes, security models, etc.

Finally, after having done lots and lots of research, this is where you should start coding I think:

1. Re-engineer Mosaic, but with your own design ideas. This is also what I would suggest if your are just doing it for fun or your own educational benefit. Read the original HTML 1.0 and HTML 2.0 specs, as well as the HTTP 1.1 specs and the current URI specs, and make sure your browser adheres to all those specs. You can of course download existing software that already handles the transport protocols, URI conventions, etc. but if you're serious about designing your own browser, I think it is a good exercise to do these things from scratch as well, so you get a good sense of how all the puzzle pieces fit together. At the end of step 0, you should have a browser that is at least comparable to what was state-of-the-art in the 90's. This is a good first milestone. And you can actually download the original Mosaic at ftp://ftp.ncsa.uiuc.edu/Mosaic/ and see how it compares with your browser. It's also a good exercise to see how current websites render in an ancient browser like Mosaic.

2. Add support for the DOM to your browser. Focus on W3C DOM Level 1 and Level 2 first, since pretty much all current browsers support those completely. Then look at Level 3 and Level 4. The DOM is extremely fundamental to web programming, and so if you're going to actually build a modern web browser, it's entire design has to take this into consideration. Since you are writing the browser in C# you may want to take into consideration how you could leverage the existing .NET object model to your advantage.

3. Look at existing scripting engines and see if you can port them to your project. I'd discourage you from writing your own JavaScript interpreter, not only because that's a very large project in itself, but because so much work has already been put into optimizing JS compilers (e.g. V8). So unless you're a guru in compiler design, your hand-built JS interpreter will likely be inferior to what's already out there, even if it follows the EMCAScript specs flawlessly. Again, I think the scripting engine should be something that is a completely separate module from the actual browser anyway, so I think it would be much more useful to have a framework that allows you to substitute any scripting engine, rather than build a scripting engine that only works with your browser.

4. Look at the HTML / CSS / JS source code for the top 10-20 websites in North America (Google, Facebook, YouTube, Twitter, Wikipedia, Amazon, popular blogging platforms, etc.) and engineer your browser to work well with these sites. This is a somewhat more tractable problem to solve than making a browser that adheres to all the existing standards (something that current browsers still don't do perfectly) much less making a browser that correctly renders all the web sites on the web (nobody can do that). People will complain that your browser breaks standards and so-forth, but that's not as big of a problem as people complaining that they can't access Google or Facebook with your browser. I can't think of any browser that correctly followed all (or even most) standards on its first release, so I say don't even bother trying. If you can make something that people will want to use enough that there will ever be a 2nd or 3rd version, then you can worry about standards at that point.

You mean as in writing your own rendering engine?

I can only say good luck. Many man years have gone into the current generation of the various browsers, If you want to do better than either of them you will need some serious skills. If you have to ask where to start, you probably have more than a few years of study to go before it would make any sense to attempt such a task.

That said, here are some (obvious) pointers:

1. write lots of code that does small things, like solve all the projecteuler.net problems
2. learn everything you can about your toolkit and its community standards
3. write lots more code
4. get a real solid grasp of finite state machines
5. write yet more code
6. learn all about the tcp/ip stack and how it's used for http
7. learn all you can about http
8. learn the standards (html, xml, sgml, css)
9. celebrate your 150th birthday.
10. get started on the actual browser project.

edit below here

I didn't mean for it to be either motivating or demotivating, just an attempt to show you that a browser is a really big project and that really big projects require a whole lot of thought. Blunt honesty sprinkled with humour.

I've been programming for over two thirds of my life and I like to think that I am a pretty decent programmer, but it would be foolish of me to think that I'd stand half a chance at writing a decent web browser from scratch.

Ofcourse, if this is what you want to do, don't let my comment stand in your way. You can probably do better than Internet Explorer.

It's an insanely ambitious project (especially for a single developer) but something I'd love to do someday - you could learn so much from it.

I don't know a lot about how the protocols work (something that you definitely need to research) or much about what goes on in a browser but a great place to start would be the source of the open-source browsers, primarily Chrome and Firefox. Chrome is an especially good project to look at as they only do what I'd expect you to start with: the chrome and the backend of the browser. Forget creating a rendering engine at first - use Webkit or Gekko.

As everyone else has already said, a web browser is a huge project. You've got to worry about tcp/ip&sockets, rendering html, using css, creating a DOM model, executing javascript, dealing with malformed markup and code, and handling all types of files before you can even think about all the things people expect from a browser (ie bookmarks, history, private browsing, security, etc.) It's a huge project.

That being said, it can be done. My suggestion would be to go look at the source of Firefox. I know that you said you want to build a browser from scratch, but it would be very helpful to learn from an open-source project, first.

I would download the Firefox source, and slowly strip it down. In other words, I would take the source and remove all bookmarking functionality. Then, I'd remove the ability to handle addons. Then, I'd delete all code regarding saving files. I would continue this process until I got a very basic web browser. I'd look over that code.

Then, I'd start building my own. I'd take the knowledge I'd gained from taking apart Firefox, and I'd put it into building a new browser.

A whole lot of luck to you!

You could start with well-formed and valid XHTML, which should be easier than the tag soup your browser will encounter in real "life".

Then you must find a way to bend the real HTML from the web to your needs.

But don't kid yourself: A browser isn't a small project.

...then start worrying about security

(non-functional and cross cutting concerns should be generally considered up front though :) )

very ambitious project but one developer can't do this alone you need a team(project manager , testers ...) and maybe you should review your choise of language c# works only on windows(i know mono on linux but it is not the same) anyway i wish you good luck and i ll be happy to use your browser :D

You really have a lot of free time in your hand, haven't you? AFAIK, most browsers were written in C++, not all users have the .NET framework installed on their computers and if they do it might not be the version you need.

This could take you years but anyway, there are many open source browsers out there, FireFox, Google Chrome .. etc, you could start by having a look on the code, good luck with that :)