Tag: firefox

Doubling the Number of Content Processes in Firefox

Over the past year, the Fission MemShrink project has been working tirelessly to reduce the memory overhead of Firefox. The goal is to allow us to start spinning up more processes while still maintaining a reasonable memory footprint. I’m happy to announce that we’ve seen the fruits of this labor: as of version 66 we’re doubling the default number of content processes from 4 to 8.

Doubling the number of content processes is the logical extension of the e10s-multi project. Back when that project wrapped up we chose to limit the default number of processes to 4 in order to balance the benefits of multiple content processes — fewer crashes, better site isolation, improved performance when loading multiple pages — with the impact on memory usage for our users.

Our telemetry has looked really good: if we compare beta 59 (roughly when this project started) with beta 66, where we decided to let the increase be shipped to our regular users, we see a virtually unchanged total memory usage for our 25th, median, and 75th percentile and a modest 9% increase for the 95th percentile on Windows 64-bit.

Doubling the number of content processes and not seeing a huge jump is quite impressive. Even on our worst-case-scenario stress test — AWSY which loads 100 pages in 30 tabs, repeated 3 times — we only saw a 6% increase in memory usage when turning on 8 content processes when compared to when we started the project.

This is a huge accomplishment and I’m very proud of the loose-knit team of contributors who have done some phenomenal feats to get us to this point. There have been some big wins, but really it’s the myriad of minor improvements that compounded into a large impact. This has ranged from delay-loading browser JavaScript code until it’s needed (or not at all), to low-level changes to packing C++ data structures more efficiently, to large system-wide changes to how we generate bindings that glue together our JavaScript and C++ code. You can read more about the background of this project and many of the changes in our initial newsletter and the follow-up.

While I’m pleased with where we are now, we still have a way to go to get our overhead down even further. Fear not, for we have a quite a few changes in the pipeline including a fork server to help further reduce memory usage on Linux and macOS, work to share font data between processes, and work to share more CSS data between processes. In addition to reducing overhead we now have a tab unloading feature in Nightly 67 that will proactively unload tabs when it looks like you’re about to run out of memory. So far the results in reducing the number of out-of-memory crashes are looking really good and we’re hoping to get that released to a wider audience in the near future.

Firefox Memory Usage in the Quantum Era

This is a continuation of my Are They Slim Yet series. For background see my previous installment.

Firefox’s upcoming release 57 has a huge focus on performance. We’ve quantum-ed all the things but we haven’t really talked about memory usage, which is something that often falls by the wayside in the pursuit of performance. Luckily since we brought AWSY in tree it’s been pretty easy to track memory usage and regressions even on separate development branches. The Stylo team was a big user of this and it shows, we flipped the switch to enable Stylo by default around the 7th and you can see a fairly large regression, but by the 16th it was mostly gone:

Hopefully I’ve convinced you we’ve put a lot of work into performance, now let’s see how we’re doing memory-wise compared to other browsers.

The methodology for the test is the same as previous runs: I used the ATSY project to load 30 pages and measure memory usage of the various processes that each browser spawns during that time.

The results

Browser Memory Usage.
Memory usage of browsers across operating systems.

Edge has the highest memory usage on Windows, Chrome comes in with 1.4X the memory usage of Firefox 64-bit on Windows, about 2X Firefox on Linux. On macOS Safari is now by far the worst offender in memory usage, Chrome and Firefox are about even with Firefox memory usage having gone up a fair amount since the last time I measured.

Overall I’m pretty happy with where we’re at, but now that our big performance push is over I’d like to see us focus more on dropping memory usage so we can start pushing up the number of content processes. I’d also like to take a closer look into what’s going on on macOS as that’s been our biggest regression.

Browsers included are Edge 38 on Windows 10, Chrome Beta 62 on all platforms, Firefox Beta 57 on all platforms, and Safari Technology Preview 40 on macOS 10.12.6.

Note: I had to run the test for Safari manually again, they seem to have made some changes that cause all of the pages from my test to be loaded in the same content process.

MemShrink’s 6th Birthday

MemShrink, it’s still a thing

Although not as active, we still have a MemShrink group at Mozilla. We’ve transitioned from an all out assault on memory usage to mostly just attempting to keep memory usage sane. I wasn’t around when things started, but when I joined there were at least seven people actively attending our MemShrink triage meetings, now we’re down to two. Some members have moved on, others have transitioned through, but really it comes down to the fact that we did a pretty good job of getting memory under control and with limited resources there were more important tasks to look at.

Fear not, we haven’t abandoned the project. We’re just in a bit of a lull. With big pushes for multiple content processes and the Quantum project I think we’re going to see the need to ramp up MemShrink again. In the meantime rest assured we’re still chugging along, just at a slower pace.

Big Ticket Items – 2014

Three years ago Nicholas Nethercote wrote a blog post celebrating MemShrink’s 3rd birthday and put together a list of important work we saw coming up. Lets see how those projects went.

Better regression detection

AWSY has moved into our testing automation system and we are now have automated regression detection through perfherder. I think we can declare victory here.

Devtools

The devtools team added a memory tab. Dan Callahan and Nick Fitzgerald put together a nice writeup of the new memory tool. There’s more work that can be done, but most of the devtools team’s focus is on performance profiling these days. It sounds like it could become a priority again next year.

GC Arena Fragmentation

Jon Coppeard did some heroic work (64 patches!) and got compacting GC landed. Initial measurements showed an 8% reduction in JS memory usage which is quite impressive. You can read more details in a blog post by Jon about [compacting garbage collection in SpiderMonkey].(https://hacks.mozilla.org/2015/07/compacting-garbage-collection-in-spidermonkey/)

Tarako

We actually shipped the 128MB phone! It never took off in it’s target market and eventually the entire FirefoxOS project was shut down, but I’m still super impressed we achieved such a feat.

Windows OOM crashes

This is an ongoing problem. We still think the push to 64-bit Windows builds will be a huge win. We have a plan to upgrade users from 32-bit to 64-bit if their system can handle it and will make 64-bit the default in Firefox 55.

In the meantime the JS engine is now smarter about requesting memory on Windows and multi-process Firefox has shipped.

We had hopes that upgrading our memory allocator would help as well, but we’ve since abandoned that effort.

Big Ticket Items – 2017

That was a nice trip down memory lane, but now we need to look forward. Let’s take a look at some of what I see as our next big ticket items.

Reduce JS memory usage and increase sharing of data across processes

The JavaScript engine is probably our biggest target coming up for reducing memory usage, particularly with multiple content processes enabled. There’s some impressive work going on to have our core JavaScript modules share a single global. Initial testing has shown some pretty big wins for this.

In general we need think about ways to share more data across processes.

Improved devtools for memory analysis

The devtools team did a great job with their initial iteration of memory profiling, but it would be great to see a more refined UI and tie in information from our cycle collector on the C++ side.

Expanded testing

I’d like to get the ATSY project automated so that we can get consistent numbers on how we fare against other browsers. This has been a boon for JavaScript performance, I can see it being a good motivator for improving memory usage as well. An updated test corpus that uses modern web features would be a big improvement. Making it easier to track the memory impact of WebExtensions would also be great.

Conclusions

We ticked off 4 out of 5 of our big ticket items. 64-bit builds on Windows by default is just around the corner so lets just go ahead and count that as 5 out of 5. I see plenty of future challenges for the MemShrink group particularly once the dust settles from enabling multiple content processes and the various Quantum projects.

Let me know if I missed any big improvements, I’m sure there are plenty!

Are we slim yet is dead, all hail are we slim yet

Aside from some pangs of nostalgia, it is with great pleasure that I announce the retirement of areweslimyet.com, the areweslimyet github project, and its associated infrastructure (a sad computer in Mountain View under dvander’s desk and a possibly less sad computer running the website that’s owned by the former maintainer).

Wait, what?

Don’t worry! Are we slim yet, aka AWSY, lives on, it’s just moved in-tree and is run within Mozilla’s automated testing infrastructure.

For equivalent graphs check out:
– Explicit
– RSS
– Miscellaneous

You can build your own graph from Perfherder. Just choose ‘+ Add test data’, ‘awsy’ for the framework and the tests and platforms you care about.

Wait, why?

I spent a few years maintaining and updating AWSY and some folks spent a fair amount of time before me. It was an ad hoc system that had bits and pieces bolted on over time. I brought it into the modern age from using the mozmill framework over to marionette, added support for e10s, and cleaned up some old slightly busted code. I tried to reuse packages developed by Mozilla to make things a bit easier (mozdownload and friends).

This was all pretty good, but things kept breaking. We weren’t in-tree, so breaking changes to marionette, mozdownload, etc would cause failures for us and it would take a while to figure out what happened. Sometimes the hard drive filled up. Sometimes the status file would get corrupted due to a poorly timed shutdown. It just had a lot of maintenance for a project with nobody dedicated to it.

The final straw was the retirement of archive.mozilla.org for what we call tinderbox builds, builds that are done more or less per push. This completely broke AWSY back in January and we decided it was just better to give in and go in-tree.

So is this a good thing?

It is a great thing. We’ve gone from 18,000 lines of code to 1,000 lines of code. That is not a typo. We now run on linux64, win32, and win64. Mac is coming soon. We turned on e10s. We have results on mozilla-inbound, autoland, try, mozilla-central, and mozilla-beta. We’re going to have automated crash analysis soon. We were able to use the project to give the greenlight for the e10s-multi project on memory usage.

Oh and guess what? Developers can run AWSY locally via mach. That’s right, try this out:

mach awsy-test --quick

Big thanks go out to Paul Yang and Bob Clary who pulled all this together — all I did was do a quick draft of an awsy-lite implementation — they did the heavy lifting getting it in tree, integrated with task cluster, and integrated with mach.

What’s next?

Now that we’re in-tree we can easily add new tests. Imagine getting data points for running the AWSY test with a specific add-on enabled to see if it regresses memory across revisions. And anyone can do this, no crazy local setup. Just mach awsy-test.

A Rust-based XML parser for Firefox

Goal: Replace Gecko’s XML parser, libexpat, with a Rust-based XML parser

Firefox currently uses an old, trimmed down, and slightly modified version of libexpat, a library written in C, to support parsing of XML documents. These files include plain old XML on the web, XSLT documents, SVG images, XHTML documents, RDF, and our own XUL UI format. While it’s served it’s job well it has long been unmaintained and has been a source of many security vulnerabilities, a few of which I’ve had the pleasure of looking into. It’s 13,000 lines of rather hard to understand code and tracing through everything when looking into security vulnerabilities can take days at a time.

It’s time for a change. I’d like us to switch over to a Rust-based XML parser to help improve our memory safety. We’ve done this already with at least two other projects: an mp4 parser, and a url parser. This seems to fit well into that mold: a standalone component with past security issues that can be easily swapped out.

There have been suggestions adding full XML 1.0 v5 support, there’s a 6-year old proposal to rewrite our XML stack which doesn’t include replacing expat, there’s talk of the latest and greatest, but not quite fully speced, XML5. These are all interesting projects, but they’re large efforts. I’d like to see us make a reasonable change now.

What do we want?

In order to avoid scope creep and actually implement something in the short term I just want a library we can drop in that has parity with the features of libexpat that we currently use. That means:

  • A streaming, sax-like interface that generates events as we feed it a stream of data
  • Support for DTDs and external entities
  • XML 1.0 v4 (possibly v5) support
  • A UTF-16 interface. This isn’t a firm requirement; we could convert from UTF-16 -> UTF-8 -> UTF-16, but that’s clearly sub-optimal
  • As fast as expat with a low memory footprint

Why do we need UTF-16?

Short answer: That’s how our current XML parser stack works.

Slightly longer answer: In Firefox libexpat is wrapped by nsExpatDriver which implements nsITokenizer. nsITokenizer uses nsScanner which exposes the data it wraps as UTF-16 and takes in nsAString, which as you may have guessed is a wide string. It can also read in c-strings, but internally it performs a character conversion to UTF-16. On the other side all tokenized data is emitted as UTF-16 so all consumers would need to be updated as well. This extends further out, but hopefully that’s enough to explain that for a drop-in replacement it should support UTF-16.

What don’t we need?

We can drop the complexity of our parser by excluding parts of expat or more modern parsers that we don’t need. In particular:

  • Character conversion (other parts of our engine take care of this)
  • XML 1.1 and XML5 support
  • Output serialization
  • A full rewrite of our XML handling stack

What are our options?

There are three Rust-based parsers that I know of, none of which quite fit our needs:

  • xml-rs
    • StAX based, we prefer SAX
    • Doesn’t support DTD, entities
    • UTF-8 only
    • Doesn’t seem very active
  • RustyXML
    • Is SAX-like
    • Doesn’t support DTD, entities
    • Seems to only support UTF-8
    • Doesn’t seem to be actively developed
  • xml5ever
    • Used in Servo
    • Only aims to support XML5
    • Permissive about malformed XML
    • Doesn’t support DTD, entities

Where do we go from here?

My recommendation is to implement our own parser that fits the needs and use cases of Firefox specifically. I’m not saying we’d necessarily start from scratch, it’s possible we could fork one of the existing libraries or just take inspiration from a little bit of all of them, but we have rather specific requirements that need to be met.

Firefox memory usage with multiple content processes

This is a continuation of my Are They Slim Yet series, for background see my previous installment.

With Firefox’s next release, 54, we plan to enable multiple content processes — internally referred to as the e10s-multi project — by default. That means if you have e10s enabled we’ll use up to four processes to manage web content instead of just one.

My previous measurements found that four content processes are a sweet spot for both memory usage and performance. As a follow up we wanted to run the tests again to confirm my conclusions and make sure that we’re testing on what we plan to release. Additionally I was able to work around our issues testing Microsoft Edge and have included both 32-bit and 64-bit versions of Firefox on Windows; 32-bit is currently our default, 64-bit is a few releases out.

The methodology for the test is the same as previous runs, I used the atsy project to load 30 pages and measure memory usage of the various processes that each browser spawns during that time.

Without further ado, the results:

Graph of browser memory usage, Chrome uses a lot.

So we continue to see Chrome leading the pack in memory usage across the board: 2.4X the memory as Firefox 32-bit and 1.7X 64-bit on Windows. IE 11 does well, in fact it was the only one to beat Firefox. It’s successor Edge, the default browser on Windows 10, appears to be striving for Chrome level consumption. On macOS 10.12 we see Safari going the Chrome route as well.

Browsers included are the default versions of IE 11 and Edge 38 on Windows 10, Chrome Beta 59 on all platforms, Firefox Beta 54 on all platforms, and Safari Technology Preview 29 on macOS 10.12.4.

Note: For Safari I had to run the test manually, they seem to have made some changes that cause all the pages from my test to be loaded in the same content process.

Are they slim yet, round 2

A year later let’s see how Firefox fares on Windows, Linux, and OSX with multiple content processes enabled.

Results

Graph comparing memory usage, chrome is still quite high

We can see that Firefox with four content processes fares better than Chrome on all platforms which is reassuring; Chrome is still about 2X worse on Windows and Linux. Our current plan is to only move up to four content processes, so this is great news.

Two content processes is still better than IE, with four we’re a bit worse. This is pretty impressive given last year we were in the same position with one content process.

Surprisingly on Mac Firefox is better than Safari with two content processes, compared with last year where we used 2X the memory with just one process, now we’re on par with four content processes.

I included Firefox with eight content processes to keep us honest. As you can see we actually do pretty well, but I don’t think it’s realistic to ship with that many nor do we currently plan to. We already have or are adding additional processes such as the plugin process for Flash and the GPU process. These need to be taken into consideration when choosing how many content processes to enable and pushing to eight doesn’t give us much breathing room. Making sure we have measurements now is important; it’s good to know where we can improve.

Overall I feel solid about these numbers, especially considering where we were just a year ago. This bodes well for the e10s-multi project.

Test setup

This is the same setup as last year. I load the first 30 pages of the tp5 page set (a snapshot of Alexa top 100 websites from a few years ago), each in its own tab, with 10 seconds in between loads and 60 seconds of settle time at the end.

Note: There was a minor change to the setup to give each page a unique domain. At least Safari and Chrome are roughly doing process per domain, so just using different ports on localhost was not enough. A simple solution was to modify my /etc/hosts file to add localhost-<1-30> aliases.

Methodology

Measuring multiprocess browser memory usage is tricky. I’ve settled with a somewhat simple formula of:

total_memory = sum_uss(content processes) + sum_rss(parent processes);

Where a parent process is defined as anything that is not a content process (I’ll explain in a moment). Historically there was just one parent process that manages all other processes, this is still somewhat the case but each browser still has other executables they may run in addition to content processes. A content process has a slightly different definition per browser, but is generally “where the pages are loaded” — this is an oversimplification, but it’s good enough for now.

My definitions:

Browser Content Definition Example “parent”
Firefox firefox processes launched with the -contentproc command line. firefox without the -contentproc command line, plugin-process which is used for Flash, etc.
Chrome chrome processes launched with the --type command line. chrome without out the --type command line, nacl_helper, etc.
Safari WebContent processes. Safari, SafariServices, SafariHistory, Webkit.Networking, etc.
IE iexplore.exe process launched with the /prefetch command line. iexplore without the /prefetch command line.
Edge MicrosoftEdgeCP.exe processes. MicrosoftEdge.exe, etc.

For Firefox this is a reasonable and fair measurement, for other browsers we might be under counting memory by a bit. For example Edge has a parent executable, MicrosoftEdge.exe, and a different content executable, MicrosoftEdgeCP.exe, arguably we should measure the RSS of one the MicrosoftEdgeCP.exe processes, and USS for the rest, so we’re probably under counting. On the other hand we might end up over counting if the parent and content processes are sharing dynamic libraries. In future measurements I may tweak how we sum the memory, but for now I’d rather possibly under count rather then worry about being unfair to other browsers.

Raw numbers

OS Browser Total Memory
Ubuntu 16.04 LTS Chrome 54 (see note) 1,478 MB
Ubuntu 16.04 LTS Firefox 55 – 2 CP 765 MB
Ubuntu 16.04 LTS Firefox 55 – 4 CP 817 MB
Ubuntu 16.04 LTS Firefox 55 – 8 CP 990 MB
macOS 10.12.3 Chrome 59 1,365 MB
macOS 10.12.3 Firefox 55 – 2 CP 1,113 MB
macOS 10.12.3 Firefox 55 – 4 CP 1,215 MB
macOS 10.12.3 Firefox 55 – 8 CP 1,399 MB
macOS 10.12.3 Safari 10.2 (see note) 1,203 MB
Windows 10 Chrome 59 1,382 MB
Windows 10 Edge (see note) N/A
Windows 10 Firefox 55 – 2 CP 587 MB
Windows 10 Firefox 55 – 4 CP 839 MB
Windows 10 Firefox 55 – 8 CP 905 MB
Windows 10 IE 11 660 MB

Browser Version Notes

  • Chrome 54 — aka chrome-unstable — was used on Ubuntu 16.04 LTS as that’s the latest branded version available (rather than Chromium)
  • Firefox Nightly 55 – 2 CP is Firefox with 2 content processes and one parent process, the default configuration for Nightly.
  • Firefox Nightly 55 – 4 CP is Firefox with 4 content processes and one parent process, this is a longer term goal.
  • Firefox Nightly 55 – 8 CP is Firefox with 8 content processes and one parent process, this is aspirational, a good sanity check.
  • Safari Technology Preview 10.2 release 25 was used on macOS as that’s the latest branded version available (rather than Webkit nightly)
  • Edge was disqualified because it seemed to bypass the hosts file and wouldn’t load pages from unique domains. I can do measurements so I might revisit this, but it wouldn’t have been a fair comparison as-is.

Are they slim yet?

In my previous post I focused on how Firefox compares against itself with multiple content processes. In this post I’d like to take a look at how Firefox compares to other browsers.

For this task I automated as much as I could, the code is available as the atsy project on github. My goal here is to allow others to repeat my work, point out flaws, push fixes, etc. I’d love for this to be a standardized test for comparing browsers on a fixed set of pages.

As with my previous measurements, I’m going with:

total_memory = RSS(parent) + sum(USS(children))

An aside on the state of WebDriver and my hacky workarounds

When various WebDriver implementations get fixed we can make a cleaner test available. I had a dream of automating the tests across browsers using the WebDriver framework, alas, trying to do anything with tabs and WebDriver across browsers and platforms is a fruitless endeavor. Chrome’s actually the only one I could get somewhat working with WebDriver.

Luckily Chrome and Firefox are completely automated. I had to do some trickery to get Chrome working, filed a bug, doesn’t sound like they’re interested in fixing it. I also had to do some trickery to get Firefox to work (I ended up using our marionette framework directly instead), there are some bugs, not much traction there either.

IE and Safari are semi-automated, in that I launch a browser for you, you click a button, and then hit enter when it’s done. Safari’s WebDriver extension is completely broken, nobody seems to care. IE’s WebDriver completely failed at tabs (among other things), I’m not sure where to a file a bug for that.

Edge is mostly manual, its WebDriver implementation doesn’t support what I need (yet), but it’s new so I’ll give it a pass. Also you can’t just launch the browser with a file path, so there’s that. Also note I was stuck running it in a VM from modern.ie which was pretty old (they don’t have a newer one). I’d prefer not to do that, but I couldn’t upgrade my Windows 7 machine to 10 because Microsoft, Linux, bootloaders and sadness.

I didn’t test Opera, sorry. It uses blink so hopefully the Chrome coverage is good enough.

The big picture

Browser memory compared

The numbers

OS Browser Version RSS + USS
OSX 10.10.5 Chrome Canary 50.0.2627.0 1,354 MiB
OSX 10.10.5 Firefox Nightly (e10s) 46.0a1 20160122030244 1,065 MiB
OSX 10.10.5 Safari 9.0.3 (10601.4.4) 451 MiB
Ubuntu 14.04 Google Chrome Unstable 49.0.2618.8 dev (64-bit) 944 MiB
Ubuntu 14.04 Firefox Nightly (e10s) 46.0a1 20160122030244 (64-bit) 525 MiB
Windows 7 Chrome Canary 50.0.2631.0 canary (64-bit) 1,132 MiB
Windows 7 Firefox Nightly (e10s) 47.0a1 20160126030244 (64-bit) 512 MiB
Windows 7 IE 11.0.9600.18163 523 MiB
Windows 10 Edge 20.10240.16384.0 795 MiB

So yeah, Chrome’s using about 2X the memory of Firefox on Windows and Linux. Lets just read that again. That gives us a bit of breathing room.

It needs to be noted that Chrome is essentially doing 1 process per page in this test. In theory it’s configurable and I would have tried limiting its process count, but as far as I can tell they’ve let that feature decay and it no longer works. I should also note that Chrome has it’s own version of memshrink, Project TRIM, so memory usage is an area they’re actively working on.

Safari does creepily well. We could attribute this to close OS integration, but I would guess I’ve missed some processes. If you take it at face value, Safari is using 1/3 the memory of Chrome, 1/2 the memory of Firefox. Even if I’m miscounting, I’d guess they still outperform both browsers.

IE was actually on par with Firefox which I found impressive. Edge is using about 50% more memory than IE, but I wouldn’t read too much into that as I’m comparing running IE on Windows 7 to Edge on an outdated Windows 10 VM.