Update 2: The official response from the folks at mozilla.org can be found here.  Their results match our testing, that we were able to make it take a long time for Firefox to start, but were not able to make it crash.  Further, there doesn't seem to be any credible evidence at this time that this could be exploited to execute arbitrary code.

When Firefox 1.5 was officially released I wondered when the first security vulnerability would be announced.  To be fair, it's taken longer than I thought it would.  Packetstorm Security has released proof of concept code that causes a buffer overflow and denial of service on the Firefox browser.  Long and short of it is, history.dat stores various pieces of information on websites you've visited.  If the topic of a page is crafted to be long enough, it will crash the browser each time it is started after going to such a page.  This vulnerability has been tested and does work, and no known patches are available at this time.  Once this happens, firefox will be unable to be started until you erase the history.dat file manually.  Presumably, if the topic was more tightly crafted than in the proof-of-concept code, a more malicious attack could be crafted that would install malware on the machine with the extra fun step of being reinstalled after each restart of firefox (unless you erase history.dat).

As we research this more, details will be added on to this post.

UPDATES:

The machine I was testing this on has McAfee Enterprise 8, and Firefox would not crash.  Despite my valiant efforts in disabling the protection, I couldn't get it to crash.  While annoyed that I couldn't (short of uninstalling) get the protection disabled, it probablly is a good thing.  I'll test more when I get in the office tomorrow and have more machines to play with.

This seems to be more of a denial of service than a true buffer overflow.  It looks like Firefox just chokes on page topics that are too long.  Some people it hangs, other people it crashes.

WORKAROUNDS:

However, the following is a workaround that should work (if it doesn't let me know).  Go to Tools -> Options.

Select the Privacy Icon, and then the History tab.  Set the number of days to save pages at 0.  This will disable writing anything to history.dat as far as I can tell, and should nullify the exploit.  Readers have confirmed that this workaround does prevent the buffer overflow.  You can also change your privacy settings to delete personal info when you close Firefox.

Another workaround is to modify prefs.js while Firefox has not been started and put in the line:

user_pref("capability.policy.default.HTMLDocument.title.set","noAccess");

Lastly, you can also run the NoScript extension, found here.  (Which I have not looked at in depth.)  However, there are other ways of exploiting this where NoScript might not work.

Some users have reported being unable to reproduce this error.  I will test more to try to establish what makes this work and not.  So far it appears Mac users are not affected by this.

HOW TO LOCATE THE PROFILE FOLDER:

If you need to delete your history.dat file (in case you tested this PoC code), it can be difficult to locate where exactly this file is.
You can find instructions for locating the profile folder at the following URL: http://www.mozilla.org/support/firefox/edit#profile.

--
John Bambenek, bambenek *at* gmail *dot* com