Gdb-heap

gdb-heap@lists.stg.fedorahosted.org

2 discussions

Trying to analyse yum's memory usage
by David Malcolm 10 Aug '10

10 Aug '10

I've been working on a tool named "gdb-heap" [1] which extends gdb with command to analyse the dynamic memory usage of user-space processes. It has lots of heuristics for dealing with CPython. As an experiment, I tried attaching it to a "yum update" process. This is on a 64-bit Fedora 13 machine. The yum process is showing up in "top" as: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7815 root 20 0 402m 98m 8556 t 0.0 1.3 0:03.00 yum I believe I attached to it during depsolving: yum was printing messages of the form: ---> Package python-twisted-core-zsh.x86_64 0:8.2.0-6.fc13 set to be updated ---> Package python-webhelpers.noarch 0:1.0-0.2.b7.fc13 set to be updated etc. Method: Using git HEAD of gdb-heap (0c6b3506098e21f42c43ccac6796e847e86033ea), with: yum-3.2.27-4.fc13.noarch rpm-4.8.0-14.fc13.x86_64 python-2.6.5-9.fc14.x86_64 (local build, but nothing special) $ sudo yum update $ sudo PYTHONPATH=$(pwd) gdb --eval-command="python import gdbheap" attach $(pidof -x yum) (gdb) heap It took a couple of minutes for gdb-heap to scrape data from the yum process and analyse it. The top entries in the result were: Domain Kind Detail Count Allocated size ------------- ---------------------------- ----------------------------------- ------- ------------------------- python tuple 216,816 17,845,416 uncategorized 1296 bytes 11,999 15,550,704 python str 163,082 9,619,016 cpython PyDictEntry table 822 7,938,032 rpm Header blob 97 7,722,368 python list 73,138 5,265,936 cpython PyListObject ob_item table 66,186 3,633,520 uncategorized 1778224 bytes 2 3,556,448 python dict 10,844 3,124,832 cpython PyDictEntry table YumAvailablePackageSqlite.__dict__ 3,082 2,450,896 uncategorized 1781760 bytes 1 1,781,760 cpython PyDictEntry table RPMInstalledPackage.__dict__ 511 1,577,984 cpython PySetObject setentry table 137 1,236,672 cpython PyDictEntry table TransactionMember.__dict__ 343 1,059,184 python dict YumAvailablePackageSqlite.__dict__ 3,082 887,984 cpython PyDictEntry table interned 1 790,528 python str bytecode 4,169 708,520 python code 4,169 500,280 python function 4,027 483,240 pyarena pool_header overhead 8,642 414,816 uncategorized 52016 bytes 6 312,096 pyarena alignment wastage 138 293,936 python type 262 253,184 cpython PyUnicodeObject buffer 2,938 238,712 python YumAvailablePackageSqlite 3,082 197,248 ...which isn't necessarily very illuminating: 17MB of tuples, 15MB of 1296-byte blocks of unknown type (these could actually be 1280-byte mallocs, with 16 byte padding), 9MB of strings. I notice 97 rpm "Header blobs" taking 7MB of ram; does yum open up rpm files? This is data referenced by rpm.hdr python objects. Do these get cleaned up immediately after use, or do they need to stick around for a while? Not sure if savings are possible. Moving down the profile, locating the 17778224-byte allocations (I've implemented a mini query language for locating blocks by criteria): (gdb) heap select size == 1778224 Start End Domain Kind Detail Hexdump ------------------ ------------------ ------------- ---- ------------- ---------------------------------------------------------------------------------- 0x000000000360a810 0x00000000037bca3f uncategorized 1778224 bytes 00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........| 0x00000000068596c0 0x0000000006a0b8ef uncategorized 1778224 bytes 00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc |...C...`...?........| and using gdb-heap's "hexdump" command on those blocks shows a buffer with some uint32 values, then what looks like rpm metadata; perhaps it's all rpm metadata? Berkeley DB? Not sure. (gdb) hexdump 0x000000000360a810 0x000000000360a810 -> 0x000000000360a82f 00 00 00 43 00 00 86 60 00 00 00 3f 00 00 00 07 00 00 80 fc 00 00 00 10 00 00 00 64 00 00 00 08 |...C...`...?...............d....| 0x000000000360a830 -> 0x000000000360a84f 00 00 00 00 00 00 00 01 00 00 03 e8 00 00 00 06 00 00 00 02 00 00 00 01 00 00 03 e9 00 00 00 06 |................................| 0x000000000360a850 -> 0x000000000360a86f 00 00 00 10 00 00 00 01 00 00 03 ea 00 00 00 06 00 00 00 16 00 00 00 01 00 00 03 ec 00 00 00 09 |................................| (snip) 0x000000000360ac50 -> 0x000000000360ac6f 2d 70 79 6c 6f 6e 73 00 30 2e 39 2e 37 00 32 2e 66 63 31 32 00 50 79 6c 6f 6e 73 20 77 65 62 20 |-pylons.0.9.7.2.fc12.Pylons web | 0x000000000360ac70 -> 0x000000000360ac8f 66 72 61 6d 65 77 6f 72 6b 00 54 68 65 20 50 79 6c 6f 6e 73 20 77 65 62 20 66 72 61 6d 65 77 6f |framework.The Pylons web framewo| 0x000000000360ac90 -> 0x000000000360acaf 72 6b 20 69 73 20 61 69 6d 65 64 20 61 74 20 6d 61 6b 69 6e 67 20 77 65 62 61 70 70 73 20 61 6e |rk is aimed at making webapps an| 0x000000000360acb0 -> 0x000000000360accf 64 20 6c 61 72 67 65 20 70 72 6f 67 72 61 6d 6d 61 74 69 63 0a 77 65 62 73 69 74 65 20 64 65 76 |d large programmatic.website dev| 0x000000000360acd0 -> 0x000000000360acef 65 6c 6f 70 6d 65 6e 74 20 69 6e 20 50 79 74 68 6f 6e 20 65 61 73 79 2e 20 53 65 76 65 72 61 6c |elopment in Python easy. Several| 0x000000000360acf0 -> 0x000000000360ad0f 20 6b 65 79 20 70 6f 69 6e 74 73 3a 0a 0a 2a 20 41 20 66 72 61 6d 65 77 6f 72 6b 20 74 6f 20 6d | key points:..* A framework to m| I was a little disappointed with this result; I was hoping that gdb-heap might suggest some substantial memory-saving wins, but nothing jumps out from the top entries in the memory profile. Having said that, I notice about 5.7MB of PyDictEntry tables for YumAvailablePackageSqlite.__dict__ etc. Have you considered using the "__slots__" optimization here? [2] You provide something like: class Foo(object) __slots__ = ('name', 'version', 'epoch', 'etc') and this locks down the attributes of instances to just the ones listed. This typically will save half the memory, since the table of attribute name pointers can be stored per-class, rather than on the left-hand-side of every instance dictionary. This assumes that the set of attributes is limited. Unfortunately, it's only a 2-3% saving in this case. (There's also a saving from not storing the PyDictObjects themselves, but this is much less; about 288 bytes per object on 64bit = 1.7MB in this case). I'm attaching the full report on heap usage by category from the process. There's clearly a bug in the final lines: uncategorized -559571704885180256 bytes 1 -,559,571,704,885,180,256 TOTAL 605,352 -,559,571,704,793,262,616 which I'm looking into; I believe the correct lines should have read: TOTAL 605,352 91,917,640 (i.e roughly 6*10^6 allocated blocks of memory, occupying 91*10^6 bytes; it doesn't currently report on fragmentation within glibc's heap, which may be one reason for the discrepancy relative to the 98MB figure given by "top". Unfortunately "gdb-heap" consumes a large amount of memory itself (about 6GB to do the above analysis). I'm looking at fixing this (pointing it at itself, naturally). Hope this is helpful Dave [1] https://fedorahosted.org/gdb-heap/ [2] http://docs.python.org/reference/datamodel.html#slots

2 1

testing
by David Malcolm 07 Jul '10

07 Jul '10

This is a test message

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

Gdb-heap