[PATCH 3 of 3 stable] changegroup: sparsely populate fnodes

[Gregory Szorc]

On 11/7/14 12:12 AM, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc at gmail.com>
> # Date 1415342900 28800
> #      Thu Nov 06 22:48:20 2014 -0800
> # Node ID 1c646d31cdd16a0f7dddaf869bc0ea59d7c0c25c
> # Parent  87945ee4a50d8e639a737d72f8e5c1e8a45441da
> changegroup: sparsely populate fnodes
>
> Previously, fnodes had a key and empty dict value for every element in
> changedfiles. This is somewhat wasteful. Empty dicts in CPython consume
> a lot more memory than you would expect - 280 bytes.
>
> On mozilla-central, which has ~190,000 files/fnodes keys, the previous
> loop populating fnodes allocated 91,924 KB of memory, most of that for
> the empty dicts.
>
> With this patch in place, our peak RSS during mozilla-central clone
> drops:
>
> before:  364,356 KB
> after:   326,008 KB
> delta:   -38,348 KB
>
> When combined with the previous patch, total peak RSS decrease is now
> 190,116 KB.

As nice as the memory savings of these 2 patches are, we're still not as 
lean as 2.5.4. With 2.5.4, I'm consistently able to clone 
mozilla-central with ~250 MB RSS with a few forays into ~300 MB 
territory. With these patches applied, we're pretty much always using 
320-350 MB during clone.

Also, add tracemalloc to the list of nice things Python 3 support would 
allow us to use.