gcc-python-plugin February 2012

gcc-python-plugin@lists.stg.fedorahosted.org

5 participants
9 discussions

Functions setting exceptions
by Daniele Varrazzo 01 Mar '12

01 Mar '12

Hello, attached there is a patch to implement PySequence_SetItem, which *doesn't* steal a reference as the PyList/PyTuple variants do, and returns 0 on success, -1 on error. After cleaning many genuine errors in psycopg, mostly about unchecked return values, I'm left with several false positives related exactly to this type of functions, returning 0 on success, -1 on failure and setting an exception. Each of these functions caller is reported as "returning (PyObject*)NULL without setting an exception". Is there any way to flag the behaviour of these functions? An __attribute__? I'm trying implementing it but time is out for tonight. I've find how to register the attribute callback but haven't found yet where and how to use it. -- Daniele

3 10

analyzing psycopg2
by Daniele Varrazzo 24 Feb '12

24 Feb '12

Hello, I'm using the CPython static analyzer on psycopg2 and it's being quite helpful: so far it has discovered several quirks. They are mostly relative to unlikely code paths, such as out of memory during module import, nonetheless I'm trying to fix everything. Here is something to report: 1. Running the repos head out-of-the-box, the program stops quite early with an error. The attached steal-ref.patch resolves it, but I don't know if the fix is correct. 2. The library uses PyWeakref_GetObject, returning a borrowed ref: I've implemented it, with tests, in the other attached patch. 3. The run still doesn't complete: after a lot of output it tracebacks with: .... File "/home/piro/fs/gcc-python-plugin/libcpychecker/absinterp.py", line 2451, in eval_rhs return self.eval_rvalue(rhs[0], stmt.loc) File "/home/piro/fs/gcc-python-plugin/libcpychecker/absinterp.py", line 1498, in eval_rvalue region = self.get_field_region(expr, loc)#.target, expr.field.name) File "/home/piro/fs/gcc-python-plugin/libcpychecker/absinterp.py", line 1660, in get_field_region self.raise_split_value(ptr) File "/home/piro/fs/gcc-python-plugin/libcpychecker/absinterp.py", line 1984, in raise_split_value check_isinstance(ptr_rvalue.gcctype, gcc.PointerType) File "/home/piro/fs/gcc-python-plugin/gccutils.py", line 629, in check_isinstance raise TypeError('%s / %r is not an instance of %s' % (obj, obj, types)) TypeError: None / None is not an instance of <type 'gcc.PointerType'> I haven't debugged it yet: will try when I arrive to that point, for the moment I'm fixing psycopg instead. 4. Another weird function psycopg uses is _PyString_Resize (_PyBytes_Resize in Python 3): the function looks no more public (it was in the API docs for Python 2.0), and is used in the reimplementation of the % operator for bytes that psycopg uses internally (see [1], as _Bytes_Resize). This weird function takes a PyString as input and returns 0 on success, else returns -1 *and steals a reference*. The quirkness is reported in [2]. I guess it would take a "CPython.make_transitions_for_success_or_steal()" method, but haven't seen yet how to implement it. 5. While at it, make_transitions_for_borrowed_ref_or_fail docstring says it takes an optional name: it's probably a pasto from the above docstring. 6. Finally, haven't been able to create a ticket to https://fedorahosted.org/gcc-python-plugin/: I should probably log in but can't find how to create an account, unless opening a project on fedora. Maybe I'm just sleepy. Will try to provide further patches as I go ahead cleaning up psycopg. The branch where I'm cleaning up is at [3] (in case you want to reproduce the reported issues). Thank you very much for the tool: it seems very useful. -- Daniele [1] https://github.com/dvarrazzo/psycopg/blob/devel/psycopg/bytes_format.c [2] http://mail.python.org/pipermail/python-dev/2002-April/023719.html [3] https://github.com/dvarrazzo/psycopg/commits/gcc-python-plugin

2 1

Triaging script for gcc-with-cpychecker
by David Malcolm 10 Feb '12

10 Feb '12

I've been working on my primary use-case for the gcc-python-plugin: finding bugs in C Python extension code. To that end, I've been attempting to automate a mass rebuild of all of the C Python extension code in Fedora, building it with the cpychecker code [1]. The source tree has recently grown a "misc/fedora" subdirectory, and within it there's a mass-rebuild.py script which attempts to rebuild a set of source packages (within chroots), injecting the gcc python plugin and the libcpychecker code into the build. It then slurps out all of the refcount-error.html files from the chroot into a results directory. On doing so, I'm running into the very real issue of what to do with all the errors that the plugin finds: some packages have dozens of errors. I want to provide something that an upstream maintainer of a python package can look at and understand, without subjecting them to a wall of noise. To that end, I've built a triaging script which walks over the results, and tries to triage them into high-severity through low-severity categories, and writes out an index.html indexing them all. You can see an example of all of this here, where the cpychecker code has analyzed the python bindings for GStreamer: http://people.fedoraproject.org/~dmalcolm/gcc-python-plugin/2012-02-10/gstr… Thoughts? Hope this looks sane Dave [1] I'm tracking this for Fedora here: http://fedoraproject.org/wiki/Features/StaticAnalysisOfPythonRefcounts

1 0

Dominator Analysis
by Emílio Wuerges 08 Feb '12

08 Feb '12

Hello, I've made a small patch including my dominator analysis. It is ugly. It is the first time I did something with python embedded in C, so I'm aware I did not use the best practices. I inserted all my changes inside the gcc-python.c If you could check my patch and suggest some changes I'd be happy to do them before re-submitting my patch. -- Emilio Wuerges ECL - Embedded Computing Lab UFSC - Universidade Federal de Santa Catarina Brasil

2 2

ANN: gcc-python-plugin 0.9
by David Malcolm 06 Feb '12

06 Feb '12

gcc-python-plugin is a plugin for GCC 4.6 onwards which embeds the CPython interpreter within GCC, allowing you to write new compiler warnings in Python, generate code visualizations, etc. It ships with "gcc-with-cpychecker", which implements static analysis passes for GCC aimed at finding bugs in CPython extensions. In particular, it can automatically detect reference-counting errors: http://gcc-python-plugin.readthedocs.org/en/latest/cpychecker.html This release (0.9) is mostly about internal fixes: * support for gcc 4.7 prereleases * the plugin is now properly integrated with GCC's garbage collector, fixing segfaults that could happen when compiling large files (the Python wrapper objects no longer have their underlying GCC objects swept away from under them) * gcc-with-cpychecker's analysis logic has been reworked, fixing numerous bugs, and extending the scope of the checker enough to find 8 previously-missed memory-leak bugs when run upon itself. There are many other improvements. Detailed release notes can be seen at: http://gcc-python-plugin.readthedocs.org/en/latest/0.9.html Tarball releases are available at: https://fedorahosted.org/releases/g/c/gcc-python-plugin/ Prebuilt-documentation can be seen at: http://gcc-python-plugin.readthedocs.org/en/latest/index.html The project's homepage is: https://fedorahosted.org/gcc-python-plugin/ The plugin and checker are Free Software, licensed under the GPLv3 or later. Enjoy! Dave Malcolm

1 0

Small patch for x86_64 compatibility
by Emílio Wuerges 03 Feb '12

03 Feb '12

Hello, I starting to use the python plugin today. In order to build it, I had to 2 two things: 1) add -fPIC to testcpybuilder.py compile flags. (in the patch) 2) to use the plugin, I have to add a LD_PRELOAD=libpython2.7.so I don't know if that was intended, but maybe it would be nice to add a hint to the README. I'm using ubuntu oneiric ocelot 64bit. Also I got some failures in some tests: 253 successes; 4 failures; 1 skipped Failed tests: tests/plugin/gc/_gc_selftest tests/plugin/gimple-walk-tree/dump-all tests/plugin/gimple-walk-tree/exceptions tests/plugin/gimple-walk-tree/find-one make: *** [test-suite] Error 1 3 of them had the same reason, just some warnings: --- Expected stderr +++ Actual stderr @@ -0,0 +1,3 @@ +tests/plugin/gc/_gc_selftest/input.c: In function 'main': +tests/plugin/gc/_gc_selftest/input.c:41:9: warning: format '%i' expects argument of type 'int', but argument 2 has type 'char *' [-Wformat] +tests/plugin/gc/_gc_selftest/input.c:41:9: warning: format '%s' expects a matching 'char *' argument [-Wformat] 1 of them had this reason, with an actual error: --- Expected stderr +++ Actual stderr @@ -1,4 +1,6 @@ tests/plugin/gimple-walk-tree/exceptions/input.c: In function 'main': +tests/plugin/gimple-walk-tree/exceptions/input.c:41:9: warning: format '%i' expects argument of type 'int', but argument 2 has type 'char *' [-Wformat] +tests/plugin/gimple-walk-tree/exceptions/input.c:41:9: warning: format '%s' expects a matching 'char *' argument [-Wformat] tests/plugin/gimple-walk-tree/exceptions/input.c:35:1: error: Unhandled Python exception raised calling 'execute' method Traceback (most recent call last): File "tests/plugin/gimple-walk-tree/exceptions/script.py", line 30, in execute tests/plugin/gimple-walk-tree/find-one: FAIL -- Emilio Wuerges ECL - Embedded Computing Lab UFSC - Universidade Federal de Santa Catarina Brasil

3 8

Initial support for GCC 4.7
by David Malcolm 03 Feb '12

03 Feb '12

I've committed some fixes that make the plugin work with GCC 4.7 (or, at least, the gcc-4.7.0-0.2.fc17.x86_64 on my test box). Doing this [1] requires a configuration mechanism, so that we can detect the features of the gcc we're building against at the beginning of the build, and conditionalize things accordingly. So the Makefile has gained a new step in which an autogenerated-config.h is created; the output deliberately mimics that of an autoconf-generated "configure" script, but the configurator is written in Python, rather than m4. Hopefully this doesn't break the build for anyone. Dave [1] gory details: gcc.Block.vars no longer contains unused globals in gcc 4.7 (see https://fedorahosted.org/gcc-python-plugin/ticket/21 ), which broke the cpychecker code's usage of various CPython globals during its analyses (type objects such as "PyList_Type" and exception objects such as "PyExc_MemoryError"). It now works around this by using the new gcc.PLUGIN_FINISH_DECL event to get at these as they're parsed, if that event is available, otherwise doing it the old way.

1 0

fix issue #31
by Tom Tromey 01 Feb '12

01 Feb '12

This fixes issue #31 by documenting a couple of fields of IntegerType. The bug is just that min_value and max_value are not documented. I looked and all the other fields are documented. It would be nice if there were a way to verify that each field or method is also documented, so that "make" would fail if something were missing. Offhand I am not sure how best to do this. Tom >From 51de57a5b9ddb150d5eceee9332538caca87c7b1 Mon Sep 17 00:00:00 2001 From: Tom Tromey <tromey(a)redhat.com> Date: Tue, 24 Jan 2012 13:49:36 -0700 Subject: [PATCH 2/2] Fix issue #31 --- docs/tree.rst | 8 ++++++++ 1 files changed, 8 insertions(+), 0 deletions(-) diff --git a/docs/tree.rst b/docs/tree.rst index cd3090f..d3798ad 100644 --- a/docs/tree.rst +++ b/docs/tree.rst @@ -425,6 +425,14 @@ Types The gcc.IntegerType for the unsigned version of this type + .. py:attribute:: max_value + + The maximum possible value for this type, as a `gcc.IntegerCst` + + .. py:attribute:: min_value + + The minimum possible value for this type, as a `gcc.IntegerCst` + .. py:class:: gcc.FloatType Subclass of :py:class:`gcc.Type` representing C's `float` and `double` types -- 1.7.6.5

2 1

fix issue #6
by Tom Tromey 01 Feb '12

01 Feb '12

This "fixes" issue #6 by documenting the current state. The bug requests a way to call define_macro with two arguments, but David pointed out that define_macro("NAME=VALUE") already works. Tom >From 4977934000d92c107d8d779476a2bfa2d84f9977 Mon Sep 17 00:00:00 2001 From: Tom Tromey <tromey(a)redhat.com> Date: Tue, 24 Jan 2012 13:29:16 -0700 Subject: [PATCH 1/2] Fix issue #6 by documenting that define_macro accepts a definition as well. --- docs/preprocessor.rst | 13 ++++++++++--- 1 files changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/preprocessor.rst b/docs/preprocessor.rst index 7e37f03..e4110f3 100644 --- a/docs/preprocessor.rst +++ b/docs/preprocessor.rst @@ -7,10 +7,17 @@ For languages that support a preprocessor, it's possible to inject new The motivation for this is to better support the creation of custom attributes, by creating preprocessor names that can be tested against. -.. py:function:: gcc.define_macro(name) +.. py:function:: gcc.define_macro(argument) - Defines a preprocessor macro with the given name, which may be of use - for code that needs to test for the presence of your script. + Defines a preprocessor macro with the given argument, which may be + of use for code that needs to test for the presence of your script. + The argument can either be a simple name, or a name with a + definition: + + .. code-block:: python + + gcc.define_macro("SOMETHING") # define as the empty string + gcc.define_macro("SOMETHING=72") This function can only be called from within specific event callbacks, since it manipulates the state of the preprocessor for a given source -- 1.7.6.5

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

gcc-python-plugin February 2012