gcc-python-plugin July 2011

gcc-python-plugin@lists.stg.fedorahosted.org

6 participants
8 discussions

Start a nNew thread

test errors with gcc 4.6.1
by Denys Duchier 28 Jul '11

28 Jul '11

Here are the diffs I used on Gentoo to test with gcc 4.6.1: and here are the messages from the tests: Cheers, --Denys

2 11

ANN: gcc-python-plugin 0.5
by David Malcolm 27 Jul '11

27 Jul '11

gcc-python-plugin is a plugin for GCC 4.6 onwards which embeds the CPython interpreter within GCC, allowing you to write new compiler warnings in Python, generate code visualizations, etc. Tarball releases are available at: https://fedorahosted.org/releases/g/c/gcc-python-plugin/ Prebuilt-documentation can be seen at: http://readthedocs.org/docs/gcc-python-plugin/en/latest/index.html Project homepage: https://fedorahosted.org/gcc-python-plugin/ High level summary of the changes since the initial announcement: - new contributors - lots of bug fixes and compatibility fixes (e.g. for Python 3): the selftest suite now works for me with all eight different combinations of: - optimized vs debug builds of Python - Python 2.7 vs Python 3.2 - i686 and x86_64 building against gcc-4.6.1 (also tested with gcc-4.6.0) - new example scripts; see: http://readthedocs.org/docs/gcc-python-plugin/en/latest/examples.html - if PLUGIN_PYTHONPATH is defined at build time, hardcode the value into the plugin's sys.path, allowing multiple builds to be independently packaged - more documentation - work-in-progress on detecting reference-count errors in C Python extension code. Although this usage example can now detect errors, it isn't yet ready for general use. It can generate HTML visualizations of those errors; see http://dmalcolm.livejournal.com/6560.html for examples. - numerous other improvements (see below) - new dependency: the "six" module is required at both build time and run-time, to smooth over Python 2 vs Python 3 differences: http://pypi.python.org/pypi/six/ I've also packaged the plugin in RPM form for Fedora 16 onwards; see: https://fedoraproject.org/wiki/Features/GccPythonPlugin Enjoy! Dave Detailed change notes follow Version 0.5 =========== David Malcolm (7): Override all locale information with LC_ALL=C when running selftests Revamp support for options in selftests Add note about ccache Improvements to the example scripts Split up examples within the docs Fix gcc.Pass.__repr__ Version 0.4 =========== David Malcolm (10): add explicit BR on gmp-devel (rhbz#725569) Make the test suites be locale-independent Suppress buffering of output in Python 3 run-test-suite.py: support excluding tests from a run Python 3 fixes to testcpychecker.py Add 'str_no_uid' attribute to gcc.Tree and gcc.Gimple; use it to fix a selftest Fix segfault seen on i686 due to erroneous implementation of 'pointer' attribute of gcc.TypeDecl Selftest fixes and exclusions for 32-bit builds Fix the test for 32/64-bit in selftests so that it works with both Python 2 and 3 Version 0.3 =========== David Malcolm (3): If PLUGIN_PYTHONPATH is defined at build time, hardcode it into the plugin's sys.path Python 3 fixes Add the beginnings of a manpage for gcc-with-python Version 0.2 =========== Alexandre Lissy (2): fix: permerror() misusage Using Freedesktop standard for image viewing David Malcolm (98): Introduce gcc.Parameter and gcc.get_parameters() Introduce a compatibility header file Document the 'basic_blocks' attribute of gcc.Cfg Fix a mismatch between gccutils.pformat() and the API docs Add note about debugging Move the debugging information to be more prominent, and reword Automatically supply the correct header search directory for selftests that #include <Python.h> Set up various things in sys, including sys.path Cope with calls to function pointers in the arg checker Remove stray import Fix issue with PyArg_ParseTuple("K") seen compiling gdb Fix erroneous error messages for the various "s" and "z" format codes Format codes "U" and "S" can support several different argument types Add a way of turning of const-correctness for "const char*" checking Fix breakage of the various "es" and "et" format codes introduced in last commit Implement verification of the "O&" format code (converter callback, followed by appropriate arg) Use newlines and indentation to try to make the PyArg_ error messages more readable Add the example from my blog post ( http://dmalcolm.livejournal.com/6364.html ) Remove redundant (and non-functioning) selftest for "O&" format code Add 'local_decls', 'start', 'end', 'funcdef_no' to gcc.Function Add 'arguments' and 'result' to gcc.FunctionDecl Fix typos in docs Add 'operand' to gcc.Unary; check that the keywords table to PyArg_ has a NULL terminator Add 'location' to more tcc types; fill out more documentation Start building out examples of C syntax vs how it's seen by the Python API Fix the behavior of the various "#" format codes. Add Alexandre Lissy to contributors The various "e" codes can accept NULL as the encoding Add support for PyNumberMethods; use this to support int() on gcc.IntegerCst Introduce gcc.Option, wrapping GCC's command-line options Add gcc.warning() and gcc.error() Implement gcc.Option.is_enabled; allow -Wno-format to work, adding related workaround to gcc.warning() Explicitly set -Wno-format in the selftest that requires it If pygments is available, use it to colorize the source code in graphviz visualizations. Add line numbers. Add (disabled) option to generate CFGs as .svg rather than .png Minor tweaks to dot generation Support edge connections to individual statements in CFG renderings (via graphviz 'port=') Add an optional name to CFG renderings, along with other tweaks Work-in-progress on refcount checker Disable the checker by default again (accidentally enabled in last commit) Disable non-functioning tests for C stdio checker Remove the now-redundant SSA support from location-tracking Move generic interp code from refcounts.py to absinterp.py Eliminate superfluous StateEdge class, adding 'src' field to Transition Add test coverage for gcc.inform(); clarify docs (the output is to stderr) CFG renderer: cope with pygments styles that have a "bold" prefix Implement gcc.Tree.__hash__ Add a high-level overview of the insides of GCC Add "operand" field to gcc.MemRef; document gcc.MemRef, along with gcc.FieldDecl, gcc.ComponentReference Add a Table class to gccutils, to make it easier to generate tabular data Make gccutils.Table's output be a valid .rst table Expose more of the pass machinery, and use this to include the full pass tree within the docs Fix broken image link for cfg docs Add more information on the different representations Finish the table of gcc.PROP_ flags Add gcc.get_gcc_version() and gcc.get_plugin_gcc_version() Add a note about the GCC version to the pass documentation Add an autogenerated SVG diagram of GCC's passes Syntax-highlight Gimple within CFG dumps Rewrite of state tracking within libcpychecker (work-in-progress) Eliminate the DataState class, merging it all into State Report erroneous refcounts returned from functions, and errors Highlight new and changing values in State Graph visualizations Fix lookup of ob_refcnt on PyObject* for instances that were never modified Suggest the use of Py_RETURN_NONE when detecting _Py_NoneStruct.ob_refcnt is too low Detect reference leaks for values that aren't the return value Ensure that the checker can cope with calls to functions it doesn't know about Fix the folding of gcc.GimpleCond where we can know the value of the boolean Remove dead code Set ob_size in PyList_New Set up ob_type on objects within the analyser Add "dump_traces" option to check_refcounts, to make it easier to write selftests Avoid hardware dependent results in the tests/plugin/parameters selftest Improvements to error reporting Add support for stolen references Report on where objects were allocated when reporting on an ob_refcnt problem Generate HTML reports on errors (in addition to text on stderr) Try to make the plugin compile against gcc 4.5 libcpychecker cleanups Warn when a function returns NULL without setting the per-thread exception state Beginning implementing Py_DECREF Warn about code that accesses deallocated memory Warn about functions that return pointers to deallocated memory Don't analyse traces with loops, as a primitive means of ensuring that the analysis terminates Improve error handling within the checker Rigidly distinguish between l-values and r-values in cpychecker Add 'array' and 'index' fields to gcc.ArrayRef minor docs tweak Support arrays, and more pointer handling Fix the build (and many selftests) against Python 3; tweak exception handling Better support for building against debug builds of Python Comment out the "Other callback events" section for now Add explicit requirements section to the documentation Clean the docs during a 'make clean' Minor tweak to Makefile Tom Tromey (2): make html from top-level expose 'inform' function

1 0

New dependency: "six"
by David Malcolm 24 Jul '11

24 Jul '11

I've been fixing python 3 support within the plugin, so that it's buildable and usable with both Python 2 and Python 3. [1] The plugin now builds against Python 3, and most of the selftests now pass (I'm testing against 3.2, fwiw). (It still builds against Python 2, and all of the selftests pass, on my machine, at least, with 2.7). My aim is to support both Python 2 and Python 3 from a single source tree, without needing to run 2to3 on the .py sources. (The plugin will need to be built twice, once for each python version, and only one build of the plugin can be loaded into gcc at once). To make this easier, I'm using the compatibility module named "six": http://pypi.python.org/pypi/six/ This is now needed at both buildtime, and at runtime. I'm assuming this isn't too onerous a dependency (it's just one .py file; fwiw, I've packaged it for Fedora) Dave [1] http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=commitdiff;h=74b…

1 0

compilation problems with gcc 4.5.2
by Denys Duchier 18 Jul '11

18 Jul '11

I cloned the git repo today and tried to compile the plugin. Apparently, includes have changed a bit: by adding in <coretypes.h> and <toplev.h> I went a little further, but now I am stuck on an error about all_translation_units (indeed, it does seem to be undeclared). any idea? Cheers, --Denys

2 5

gcc-python-plugin status update: static analysis, and HTML reports
by David Malcolm 15 Jul '11

15 Jul '11

I've written up a status update on the gcc-python-plugin on my blog here: http://dmalcolm.livejournal.com/6560.html Summarizing that blog post, I've revamped the internals of how my checker script so that it can detect various kinds of CPython reference count bug, and it can now render bug reports as both regular text on stderr, and in HTML form: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/refcount_demo-refcount-er… (you may need a modern browser to see the control flow annotations in the HTML, as it uses JavaScript and the <canvas> element; there are screenshots in the blog post) Detecting the all-too-common: "return Py_None;" bug: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/losing_refcnt_of_none-ref… Detecting a (contrived) reference leak: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/object_leak-refcount-erro… Detecting a stray Py_INCREF that makes the reference count too high, or segfaults python, depending on what happened earlier within the function: http://fedorapeople.org/~dmalcolm/blog/2011-07-15/too_many_increfs-refcount… Some of that rendering code might be of use to other users of the plugin; has anyone else rendered HTML reports from within GCC? I didn't discuss implementation details in the blog post, but given the audience on these lists, it's probably on-topic to mention here: It works by a rather crude implementation of abstract interpretation, tracking all possible paths through the function, generating traces of execution. (It currently will fail horribly when it encounters a loop; I suspect I need widening/narrowing operators, and to read a lot more papers on the subject :) ) Internally, the analyser models the memory state using a method similar to the one used by LLVM's clang static analyser, as described in: "A Memory Model for Static Analysis of C Programs" (Zhongxing Xu, Ted Kremenek, and Jian Zhang) thus handling pointers and dereferences (though my implementation is very much an early prototype as this stage). I've added a new (as far as I know) class of abstract value representing "all reference counts that the function owns" (i.e. an integer value, relative to the unknown refcount owned by all of the rest of the program). Hope this is helpful/interesting Dave

1 0

Fixed a hardware dependency in the selftests
by David Malcolm 13 Jul '11

13 Jul '11

When I implemented gcc.Parameter, out of all the parameters I could have chosen for the selftest [1] I managed to pick the two parameters that vary based on the hardware on the test machine, thus embedding an expected value that only worked with a specific certain amount of RAM. Specifically, 'ggc-min-expand' and 'ggc-min-heapsize' are set up by gcc's init_ggc_heuristics() to values based on the amount of physical memory on the test machine. So the selftest suite has been working fine for me, but probably failing for anyone else. I've pushed a fix for this, so hopefully the test suite is no longer hardware dependent, and "make" should now run to completion. Sorry about this. Please let me know if you see any other test failures running "make". Dave [1] "tests/plugin/parameters"

1 0

A visualization of GCC's passes, as a subway map
by David Malcolm 12 Jul '11

12 Jul '11

For fun over the weekend I wrote a python script (using my gcc-python-plugin[1]) to render an SVG diagram of GCC's optimization passes (or, at least, based on my understanding of them). This diagram shows the various GCC optimization passes, arranged vertically, showing child passes via indentation. The lifetime of the various properties that they maintain is shown, giving the pass that initially creates the data (if any), the pass that destroys it (if any), and each pass that requires a particular property (based on the PROP_* flags). I've attempted to make it look a little like a subway train schematic map, where the properties are train lines, and the passes are the stations they stop at; the initial/final passes are the termini. The resulting SVG can be seen at the top of: http://readthedocs.org/docs/gcc-python-plugin/en/latest/tables-of-passes.ht… Direct link to SVG, in case the above gets mangled on your browser: http://readthedocs.org/docs/gcc-python-plugin/en/latest/_images/passes.svg (550 x 3302 pixels; about 250k in size). The script can be seen here (it uses pycairo to abstract the drawing operations): http://git.fedorahosted.org/git/?p=gcc-python-plugin.git;a=blob;f=generate-… Hope this is fun/helpful (and that I'm correctly interpreting the data!) Dave [1] https://fedorahosted.org/gcc-python-plugin/

5 6

More documentation
by David Malcolm 07 Jul '11

07 Jul '11

I wrote a high-level guide to the insides of GCC, from the perspective (I hope) of a Python programmer hoping to write a script. The built HTML version can be seen at: http://readthedocs.org/docs/gcc-python-plugin/en/latest/gcc-overview.html I've also exposed more of the pass-handling machinery, and used this to generate a dump of all of the passes (in my version of gcc, 4.6.0, at any rate). This can be seen at: http://readthedocs.org/docs/gcc-python-plugin/en/latest/passes.html and: http://readthedocs.org/docs/gcc-python-plugin/en/latest/tables-of-passes.ht… I hope I've understood things correctly! Please let me know if you see any significant errors. Dave

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

gcc-python-plugin July 2011