gcc-python-plugin-commits October 2011

gcc-python-plugin-commits@lists.fedorahosted.org

1 participants
65 discussions

[gcc-python-plugin] cpychecker: fix handling of loops
by David Hugh Malcolm 27 Oct '11

27 Oct '11

commit 67b72a903cf4da9ab552cc980d34999beb5d1c9f Author: David Malcolm <dmalcolm(a)redhat.com> Date: Thu Oct 27 20:48:48 2011 -0400 cpychecker: fix handling of loops Previously, we stopped analyzing a trace if we reached a Location that we had been to before, with some special-casing for split states, and for calls to functions that don't return, like exit(), with the aim of analyzing zero and one times through each loop. This turned out not to be working as desired: it was silently discarding all traces containing a loop. For example, given /* (A): before loop */ for (i = 0; i < n; i++) { /* (B): body of loop */ if (some_condition) { goto error; } } /* (C): after loop */ error: /* (D): cleanup */ the checker would cover these traces: (A) -> (C) (zero iterations) (A) -> (B) -> (D) (one iteration, interrupted by the goto) but the trace (A) -> (B) -> (C) would be discarded, since the checker would reach the i < n conditional for a second time, and throw away that trace. This patch changes the loop detection to instead work on transitions, similar to edges in the CFG, but covering pairs of (src, dest) Location instances. We now detect a loop and terminate the analysis for a trace that has followed a (src, dest) path for a second time. In the above example, this means that it's now OK to execute the (i<n) conditional a second time, but the checker will stop for the trace where we go from that conditional into the loop body a second time. This means that the checker _will_ now cover the (A) -> (B) -> (C) trace (i.e. once through the loop). So in theory the checker ought to now analyze the Trace instances where a loop body is executed zero and one times. libcpychecker/absinterp.py | 34 +++- .../refcounts/PyList_Append/incorrect-loop/input.c | 68 +++++++ .../PyList_Append/incorrect-loop/script.py | 22 ++ .../PyList_Append/incorrect-loop/stderr.txt | 24 +++ .../PyList_Append/incorrect-loop/stdout.txt | 205 ++++++++++++++++++++ 5 files changed, 347 insertions(+), 6 deletions(-) --- diff --git a/libcpychecker/absinterp.py b/libcpychecker/absinterp.py index 248102c..0060b39 100644 --- a/libcpychecker/absinterp.py +++ b/libcpychecker/absinterp.py @@ -852,6 +852,10 @@ class Location(object): self.bb = bb self.idx = idx + def __repr__(self): + return ('Location(bb=%i, idx=%i)' + % (self.bb.index, self.idx)) + def __str__(self): stmt = self.get_stmt() return ('block %i stmt:%i : %20r : %s' @@ -2258,7 +2262,7 @@ class Transition(object): self.dest.log(logger) class Trace(object): - __slots__ = ('states', 'transitions', 'err', ) + __slots__ = ('states', 'transitions', 'err', 'paths_taken') """A sequence of States and Transitions""" def __init__(self): @@ -2266,10 +2270,15 @@ class Trace(object): self.transitions = [] self.err = None + # A list of (src Location, dest Location) pairs + self.paths_taken = [] + def add(self, transition): check_isinstance(transition, Transition) self.states.append(transition.dest) self.transitions.append(transition) + self.paths_taken.append( (transition.src.loc, + transition.dest.loc) ) return self def add_error(self, err): @@ -2280,6 +2289,7 @@ class Trace(object): t.states = self.states[:] t.transitions = self.transitions[:] t.err = self.err # FIXME: should this be a copy? + t.paths_taken = self.paths_taken[:] return t def log(self, logger, name): @@ -2298,7 +2308,7 @@ class Trace(object): def has_looped(self): """ - Is the tail state of the Trace at a location where it's been before? + Is the tail transition a path we've followed before? """ endstate = self.states[-1] if hasattr(endstate, 'fromsplit'): @@ -2310,9 +2320,18 @@ class Trace(object): # The handler not "exit" etc leads to a transition that has a # repeated location: return False - for state in self.states[0:-1]: - if state.loc == endstate.loc: - return True + + endtransition = self.transitions[-1] + if 0: + gcc.inform(endstate.get_gcc_loc(endstate.fun), + ('paths_taken: %s' + % (self.paths_taken,))) + gcc.inform(endstate.get_gcc_loc(endstate.fun), + 'src, loc: %s' % ((endtransition.src.loc, endtransition.dest.loc),)) + + # Is this a path we've followed before? + if (endtransition.src.loc, endtransition.dest.loc) in self.paths_taken[0:-1]: + return True def get_all_var_region_pairs(self): """ @@ -2479,10 +2498,13 @@ def iter_traces(fun, facets, prefix=None, limits=None): # Stop interpreting when you see a loop, to ensure termination: if prefix.has_looped(): log('loop detected; stopping iteration') + if 0: + gcc.inform(curstate.get_gcc_loc(fun), + 'loop detected; stopping iteration') # Don't return the prefix so far: it is not a complete trace return [] - # We need the prevstate to handle Phi nodes + # We need the prevstate in order to handle Phi nodes if len(prefix.states) > 1: prevstate = prefix.states[-2] else: diff --git a/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c new file mode 100644 index 0000000..c19b319 --- /dev/null +++ b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c @@ -0,0 +1,68 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +#include <Python.h> + +/* + Test of incorrect reference-handling in a call to PyList_Append within + a loop +*/ +extern PyObject *item_ctor(void); + +PyObject * +test(int n) +{ + PyObject *list; + int i; + + list = PyList_New(0); + if (!list) { + return NULL; + } + for (i = 0; i < n; i++) { + PyObject *item; + + item = item_ctor(); + if (!item) { + Py_DECREF(list); + return NULL; + } + + if (PyList_Append(list, item)) { + Py_DECREF(list); + Py_DECREF(item); + return NULL; + } + + /* + PyList_Append adds a ref to the item, but we still own a reference to + it, so each time through the loop leaks a reference to the item + */ + } + + return list; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/script.py b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/script.py new file mode 100644 index 0000000..fdd5ba3 --- /dev/null +++ b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/script.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main +main(verify_refcounting=True, + dump_traces=True, + show_traces=False) diff --git a/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stderr.txt b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stderr.txt new file mode 100644 index 0000000..5207b21 --- /dev/null +++ b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stderr.txt @@ -0,0 +1,24 @@ +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c: In function 'test': +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:60:1: error: ob_refcnt of '*item' is 1 too high +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:60:1: note: was expecting final ob_refcnt to be N + 1 (for some unknown N) +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:60:1: note: due to object being referenced by: PyListObject.ob_item[0] +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:60:1: note: but final ob_refcnt is N + 2 +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41:14: note: new ref from (unknown) item_ctor allocated at: item = item_ctor(); +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34:10: note: when PyList_New() succeeds at: list = PyList_New(0); +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:35:8: note: taking False path at: if (!list) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:38:12: note: reaching: for (i = 0; i < n; i++) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:38:5: note: when taking True path at: for (i = 0; i < n; i++) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41:14: note: reaching: item = item_ctor(); +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41:14: note: when item_ctor() succeeds at: item = item_ctor(); +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41:14: note: ob_refcnt is now refs: 1 + N where N >= 0 +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:42:12: note: taking False path at: if (!item) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47:26: note: reaching: if (PyList_Append(list, item)) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47:26: note: when PyList_Append() succeeds at: if (PyList_Append(list, item)) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47:26: note: ob_refcnt is now refs: 2 + N where N >= 0 +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47:26: note: '*item' is now referenced by 1 non-stack value(s): PyListObject.ob_item[0] +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47:12: note: taking False path at: if (PyList_Append(list, item)) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:38:25: note: reaching: for (i = 0; i < n; i++) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:38:5: note: when taking False path at: for (i = 0; i < n; i++) { +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:59:5: note: reaching: return list; +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:60:1: note: returning +tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:30:1: note: graphical error report for function 'test' written out to 'tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c.test-refcount-errors.html' diff --git a/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stdout.txt b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stdout.txt new file mode 100644 index 0000000..c9ce4d3 --- /dev/null +++ b/tests/cpychecker/refcounts/PyList_Append/incorrect-loop/stdout.txt @@ -0,0 +1,205 @@ +Trace 0: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() succeeds' + 'taking False path' + 'when PyList_Append() succeeds' + 'taking False path' + 'when taking False path' + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34))) + str(): (struct PyObject *)&RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: refs: 1 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionForGlobal(gcc.VarDecl('PyList_Type'))) + new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41: + repr(): RegionOnHeap('new ref from (unknown) item_ctor', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41)) + str(): new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + r->ob_refcnt: refs: 2 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41), region=Region('PyTypeObject for new ref from (unknown) item_ctor')) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:30 + +Trace 1: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() succeeds' + 'taking False path' + 'when PyList_Append() fails' + 'taking True path' + 'when taking True path' + 'when taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=50), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:50 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: refs: 0 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionForGlobal(gcc.VarDecl('PyList_Type'))) + new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41: + repr(): RegionOnHeap('new ref from (unknown) item_ctor', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41)) + str(): new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + r->ob_refcnt: refs: 0 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41), region=Region('PyTypeObject for new ref from (unknown) item_ctor')) + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47 + +Trace 2: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() succeeds' + 'taking False path' + 'when PyList_Append() fails' + 'taking True path' + 'when taking True path' + 'when taking False path' + 'calling tp_dealloc on new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=50), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:50 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: refs: 0 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionForGlobal(gcc.VarDecl('PyList_Type'))) + new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41: + repr(): RegionOnHeap('new ref from (unknown) item_ctor', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41)) + str(): new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + r->ob_refcnt: None + r->ob_type: None + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47 + +Trace 3: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() succeeds' + 'taking False path' + 'when PyList_Append() fails' + 'taking True path' + 'when taking False path' + 'calling tp_dealloc on PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34' + 'when taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=50), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:50 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: None + r->ob_type: None + new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41: + repr(): RegionOnHeap('new ref from (unknown) item_ctor', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41)) + str(): new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + r->ob_refcnt: refs: 0 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41), region=Region('PyTypeObject for new ref from (unknown) item_ctor')) + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47 + +Trace 4: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() succeeds' + 'taking False path' + 'when PyList_Append() fails' + 'taking True path' + 'when taking False path' + 'calling tp_dealloc on PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34' + 'when taking False path' + 'calling tp_dealloc on new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=50), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:50 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: None + r->ob_type: None + new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41: + repr(): RegionOnHeap('new ref from (unknown) item_ctor', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=41)) + str(): new ref from (unknown) item_ctor allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + r->ob_refcnt: None + r->ob_type: None + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:47 + +Trace 5: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() fails' + 'taking True path' + 'when taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=44), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:44 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: refs: 0 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionForGlobal(gcc.VarDecl('PyList_Type'))) + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + +Trace 6: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking True path' + 'when item_ctor() fails' + 'taking True path' + 'when taking False path' + 'calling tp_dealloc on PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=44), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:44 + PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34: + repr(): RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) + str(): PyListObject allocated at tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: None + r->ob_type: None + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:41 + +Trace 7: + Transitions: + 'when PyList_New() succeeds' + 'taking False path' + 'when taking False path' + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34))) + str(): (struct PyObject *)&RegionOnHeap('PyListObject', gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34)) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34 + r->ob_refcnt: refs: 1 + N where N >= 0 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=34), region=RegionForGlobal(gcc.VarDecl('PyList_Type'))) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:30 + +Trace 8: + Transitions: + 'when PyList_New() fails' + 'taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c', line=36), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:36 + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/PyList_Append/incorrect-loop/input.c:34

1 0

[gcc-python-plugin] cpychecker: Py_BuildValue("s") expects a (char*)
by David Hugh Malcolm 27 Oct '11

27 Oct '11

commit eaf9b8665be224c47bb2371e3748232fdd4a8b07 Author: David Malcolm <dmalcolm(a)redhat.com> Date: Thu Oct 27 10:46:22 2011 -0400 cpychecker: Py_BuildValue("s") expects a (char*) Py_BuildValue("s") expects a (char*), but can also accept a (const char*) libcpychecker/Py_BuildValue.py | 4 +- libcpychecker/formatstrings.py | 17 ++++++- .../code_s/incorrect-constness/input.c | 48 ++++++++++++++++++++ .../code_s/incorrect-constness/script.py | 19 ++++++++ .../code_s/incorrect-constness/stderr.txt | 7 +++ .../refcounts/Py_BuildValue/correct-code-s/input.c | 2 +- 6 files changed, 92 insertions(+), 5 deletions(-) --- diff --git a/libcpychecker/Py_BuildValue.py b/libcpychecker/Py_BuildValue.py index 508ee56..79afcd3 100644 --- a/libcpychecker/Py_BuildValue.py +++ b/libcpychecker/Py_BuildValue.py @@ -205,12 +205,12 @@ class PyBuildValueFmt(ParsedFormatString): if c in 'sz': if next == '#': result.add_argument(c + '#', - [get_const_char_ptr(), + [get_char_ptr(), get_hash_size_type(with_size_t).pointer]) i += 1 else: result.add_argument(c, - [get_const_char_ptr()]) + [get_char_ptr()]) continue if c in 'NSO': diff --git a/libcpychecker/formatstrings.py b/libcpychecker/formatstrings.py index 11dfb1c..3037f6d 100644 --- a/libcpychecker/formatstrings.py +++ b/libcpychecker/formatstrings.py @@ -24,6 +24,9 @@ from libcpychecker.utils import log const_correctness = True +def get_char_ptr(): + return gcc.Type.char().pointer + def get_const_char_ptr(): return gcc.Type.char().const_equivalent.pointer @@ -272,6 +275,16 @@ def compatible_type(exp_type, actual_type, actualarg=None): if compatible_type(exp_type.dereference, actual_type.dereference): return True + # Support (const char*) vs (char*) + # Somewhat counter-intuitively, the APIs that expect a char* are those that + # read the string data (Py_BuildValue); those that expect a const char* are + # those that write back a const char* value (PyArg_ParseTuple) + # + # Hence it's OK to accept a (const char*) when a (char*) was expected: + if str(exp_type) == 'char *': + if str(actual_type) == 'const char *': + return True + # Don't be too fussy about typedefs to integer types # For instance: # typedef unsigned PY_LONG_LONG gdb_py_ulongest; @@ -288,8 +301,8 @@ def compatible_type(exp_type, actual_type, actualarg=None): if compare_int_types(): return True - # Support character arrays vs const char*: - if str(exp_type) == 'const char *': + # Support character arrays vs char*: + if str(exp_type) == 'char *': if isinstance(actual_type, gcc.PointerType): if isinstance(actual_type.dereference, gcc.ArrayType): if actual_type.dereference.dereference == gcc.Type.char(): diff --git a/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/input.c b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/input.c new file mode 100644 index 0000000..6e56419 --- /dev/null +++ b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/input.c @@ -0,0 +1,48 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +/* + Verify that the checker emits a warning for code "s" with a non-const + char* + + Code "s" writes a pointer back to the insides of an object's representation, + so it really should be a (const char*), not just a (char*) +*/ +#include <Python.h> + +PyObject * +test(PyObject *self, PyObject *args) +{ + char *str; + + if (!PyArg_ParseTuple(args, "s", + &str)) { + return NULL; + } + + Py_RETURN_NONE; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/script.py b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/script.py new file mode 100644 index 0000000..e120c53 --- /dev/null +++ b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/script.py @@ -0,0 +1,19 @@ +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main +main() diff --git a/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/stderr.txt b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/stderr.txt new file mode 100644 index 0000000..7aa7320 --- /dev/null +++ b/tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/stderr.txt @@ -0,0 +1,7 @@ +tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/input.c: In function 'test': +tests/cpychecker/PyArg_ParseTuple/code_s/incorrect-constness/input.c:34:26: error: Mismatching type in call to PyArg_ParseTuple with format code "s" [-fpermissive] + argument 3 ("&str") had type + "char * *" + but was expecting + "const char * *" + for format code "s" diff --git a/tests/cpychecker/refcounts/Py_BuildValue/correct-code-s/input.c b/tests/cpychecker/refcounts/Py_BuildValue/correct-code-s/input.c index 2c2c2c6..3c4ed33 100644 --- a/tests/cpychecker/refcounts/Py_BuildValue/correct-code-s/input.c +++ b/tests/cpychecker/refcounts/Py_BuildValue/correct-code-s/input.c @@ -24,7 +24,7 @@ */ PyObject * -test(const char *foo, const char *bar) +test(const char *foo, char *bar) { return Py_BuildValue("(s, s)", foo, bar);

1 0

[gcc-python-plugin] cpychecker: fix impl_PyErr_SetFromErrnoWithFilename
by David Hugh Malcolm 27 Oct '11

27 Oct '11

commit ea53f2eed7fecc5394ea35519ea3b8c42ac6cacf Author: David Malcolm <dmalcolm(a)redhat.com> Date: Thu Oct 27 10:00:45 2011 -0400 cpychecker: fix impl_PyErr_SetFromErrnoWithFilename libcpychecker/refcounts.py | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-) --- diff --git a/libcpychecker/refcounts.py b/libcpychecker/refcounts.py index 57ecd31..453c087 100644 --- a/libcpychecker/refcounts.py +++ b/libcpychecker/refcounts.py @@ -1017,7 +1017,14 @@ class CPython(Facet): t_next.dest.cpython.exception_rvalue = v_exc return [t_next] - def impl_PyErr_SetFromErrnoWithFilename(self, stmt, v_exc): + def impl_PyErr_SetFromErrnoWithFilename(self, stmt, v_exc, v_filename): + # http://docs.python.org/c-api/exceptions.html#PyErr_SetFromErrnoWithFilename + # Defined in Python/errors.c: + # PyObject * + # PyErr_SetFromErrnoWithFilename(PyObject *exc, const char *filename) + # + # "filename" can be NULL. + # # It always returns NULL: t_next = self.state.mktrans_assignment(stmt.lhs, make_null_pyobject_ptr(stmt),

1 0

[gcc-python-plugin] use gcc-with-cpychecker's attributes on ourselves
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit 181b59d44462a266279391798ff3dee6cea8cf1d Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 20:46:13 2011 -0400 use gcc-with-cpychecker's attributes on ourselves Mark "gcc_python_finish_invoking_callback" as stealing a reference so that when we compile the plugin with itself (with gcc-with-cpychecker), we don't get false warnings about reference-counting issues. gcc-python.c | 6 ++++++ gcc-python.h | 19 +++++++++++++++++++ 2 files changed, 25 insertions(+), 0 deletions(-) --- diff --git a/gcc-python.c b/gcc-python.c index 7c39fee..d327169 100644 --- a/gcc-python.c +++ b/gcc-python.c @@ -170,6 +170,12 @@ static void gcc_python_finish_invoking_callback(PyGILState_STATE gstate, int expect_wrapped_data, PyObject *wrapped_gcc_data, void *user_data) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(3) /* wrapped_gcc_data */ ; + +static void +gcc_python_finish_invoking_callback(PyGILState_STATE gstate, + int expect_wrapped_data, PyObject *wrapped_gcc_data, + void *user_data) { struct callback_closure *closure = (struct callback_closure *)user_data; PyObject *args = NULL; diff --git a/gcc-python.h b/gcc-python.h index 55af6fa..99d552b 100644 --- a/gcc-python.h +++ b/gcc-python.h @@ -29,6 +29,25 @@ #define GCC_PYTHON_PLUGIN_BAD_EVENT (0xffff) /* + Define some macros to allow us to use cpychecker's custom attributes when + compiling the plugin using itself (gcc-with-cpychecker), but also to turn + them off when compiling with vanilla gcc (and other compilers). +*/ +#if defined(WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE) + #define CPYCHECKER_RETURNS_BORROWED_REF \ + __attribute__((cpychecker_returns_borrowed_ref)) +#else + #define CPYCHECKER_RETURNS_BORROWED_REF +#endif + +#if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) +#else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) +#endif + +/* Macro DECLARE_SIMPLE_WRAPPER(): ARG_structname: the struct tag for the resulting python wrapper class,

1 0

[gcc-python-plugin] cpychecker: add cpychecker_steals_reference_to_arg(n) attribute
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit e95fb39f025e5bfdb68c45822980a373a09d2859 Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 20:28:26 2011 -0400 cpychecker: add cpychecker_steals_reference_to_arg(n) attribute docs/cpychecker.rst | 66 +++++++++++++++++- libcpychecker/absinterp.py | 45 +++++++++++- libcpychecker/attributes.py | 38 +++++++++-- libcpychecker/refcounts.py | 14 ++++- .../correct-marking/input.c | 60 ++++++++++++++++ .../correct-marking/script.py | 22 ++++++ .../correct-marking/stdout.txt | 20 ++++++ .../steals-reference-to-arg/correct-usage/input.c | 68 ++++++++++++++++++ .../correct-usage/script.py | 22 ++++++ .../correct-usage/stdout.txt | 48 +++++++++++++ .../incorrect-marking/input.c | 55 +++++++++++++++ .../incorrect-marking/script.py | 22 ++++++ .../incorrect-marking/stderr.txt | 10 +++ .../incorrect-marking/stdout.txt | 20 ++++++ .../incorrect-usage/input.c | 72 ++++++++++++++++++++ .../incorrect-usage/script.py | 22 ++++++ .../incorrect-usage/stderr.txt | 13 ++++ .../incorrect-usage/stdout.txt | 43 ++++++++++++ 18 files changed, 645 insertions(+), 15 deletions(-) --- diff --git a/docs/cpychecker.rst b/docs/cpychecker.rst index a520ae9..5ced382 100644 --- a/docs/cpychecker.rst +++ b/docs/cpychecker.rst @@ -151,7 +151,19 @@ function when the function terminates. It will assume this behavior for any function (or call through a function pointer) that returns a PyObject*. -It is possible to override this behavior using custom compiler attributes: +It is possible to override this behavior using custom compiler attributes as +follows: + +Marking functions that return borrowed references +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The checker provides a custom GCC attribute: + +.. code-block:: c + + __attribute__((cpychecker_returns_borrowed_ref)) + +which can be used to mark function declarations: .. code-block:: c @@ -168,9 +180,55 @@ It is possible to override this behavior using custom compiler attributes: CPYCHECKER_RETURNS_BORROWED_REF; Given the above, the checker will assume that invocations of ``foo()`` are -returning a borrowed reference (or NULL), rather than a new reference, and -will apply the same policy when verifying the implementation of ``foo()`` -itself. +returning a borrowed reference (or NULL), rather than a new reference. It +will also check that this is that case when verifying the implementation of +``foo()`` itself. + +Marking functions that steal references to their arguments +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The checker provides a custom GCC attribute: + +.. code-block:: c + + __attribute__((cpychecker_steals_reference_to_arg(n))) + +which can be used to mark function declarations: + +.. code-block:: c + + /* The checker automatically defines this preprocessor name when creating + the custom attribute: */ + #if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) + #else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) + #endif + + extern void foo(PyObject *obj) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(1); + +Given the above, the checker will assume that invocations of ``foo()`` steal +a reference to the first argument (``obj``). It will also verify that this is +the case when analyzing the implementation of ``foo()`` itself. + +More then one argument can be marked: + +.. code-block:: c + + extern void bar(int i, PyObject *obj, int j, PyObject *other) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(2) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(4); + +The argument indices are 1-based (the above example is thus referring to +``obj`` and to ``other``). + +All such arguments to the attribute should be ``PyObject*`` (or a pointer to a +derived structure type). + +It is assumed that such references are stolen for all possible outcomes of the +function - if a function can either succeed or fail, the reference is stolen in +both possible worlds. Error-handling checking ----------------------- diff --git a/libcpychecker/absinterp.py b/libcpychecker/absinterp.py index 0117bc5..248102c 100644 --- a/libcpychecker/absinterp.py +++ b/libcpychecker/absinterp.py @@ -1865,10 +1865,14 @@ class State(object): if fnname in fnnames_returning_borrowed_refs: # The function being called was marked as returning a # borrowed ref, rather than a new ref: - return self.cpython.make_transitions_for_borrowed_ref_or_fail(stmt, - 'borrowed ref from %s()' % fnname) - return self.cpython.make_transitions_for_new_ref_or_fail(stmt, - 'new ref from (unknown) %s' % fnname) + return self.apply_fncall_side_effects( + self.cpython.make_transitions_for_borrowed_ref_or_fail(stmt, + 'borrowed ref from %s()' % fnname), + stmt) + return self.apply_fncall_side_effects( + self.cpython.make_transitions_for_new_ref_or_fail(stmt, + 'new ref from (unknown) %s' % fnname), + stmt) # Unknown function of other type: log('Invocation of unknown function: %r', fnname) @@ -1882,6 +1886,25 @@ class State(object): for i, arg in enumerate(stmt.args): log('args[%i]: %s %r', i, arg, arg) + def get_function_name(self, stmt): + """ + Try to get the function name for a gcc.GimpleCall statement as a + string, or None if we're unable to determine it. + + For a simple function invocation this is easy, but if we're + calling through a function pointer we may or may not know. + """ + check_isinstance(stmt, gcc.GimpleCall) + + v_fn = self.eval_rvalue(stmt.fn, stmt.loc) + if isinstance(v_fn, PointerToRegion): + if isinstance(v_fn.region, RegionForGlobal): + if isinstance(v_fn.region.vardecl, gcc.FunctionDecl): + return v_fn.region.vardecl.name + + # Unable to determine it: + return None + def apply_fncall_side_effects(self, transitions, stmt): """ Given a list of Transition instances for a call to a function with @@ -1895,6 +1918,20 @@ class State(object): check_isinstance(stmt, gcc.GimpleCall) args = self.eval_stmt_args(stmt) + + fnname = self.get_function_name(stmt) + + # cpython: handle functions marked as stealing references to their + # arguments: + from libcpychecker.attributes import stolen_refs_by_fnname + if fnname in stolen_refs_by_fnname: + for t_iter in transitions: + check_isinstance(t_iter, Transition) + for argindex in stolen_refs_by_fnname[stmt.fn.operand.name]: + v_arg = args[argindex-1] + if isinstance(v_arg, PointerToRegion): + t_iter.dest.cpython.steal_reference(v_arg.region) + for t_iter in transitions: check_isinstance(t_iter, Transition) for v_arg in args: diff --git a/libcpychecker/attributes.py b/libcpychecker/attributes.py index 5207c5b..b87cc8d 100644 --- a/libcpychecker/attributes.py +++ b/libcpychecker/attributes.py @@ -21,18 +21,44 @@ from gccutils import check_isinstance # Recorded attribute data: fnnames_returning_borrowed_refs = set() -def attribute_callback_for_returns_borrowed_ref(*args): - if 0: - print('attribute_callback_for_returns_borrowed_ref(%r)' % args) - check_isinstance(args[0], gcc.FunctionDecl) - fnname = args[0].name - fnnames_returning_borrowed_refs.add(fnname) +# A dictionary mapping from fnname to set of argument indices: +stolen_refs_by_fnname = {} def register_our_attributes(): # Callback, called by the gcc.PLUGIN_ATTRIBUTES event + + # Handler for __attribute__((cpychecker_returns_borrowed_ref)) + # and #ifdef WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE + def attribute_callback_for_returns_borrowed_ref(*args): + if 0: + print('attribute_callback_for_returns_borrowed_ref(%r)' % args) + check_isinstance(args[0], gcc.FunctionDecl) + fnname = args[0].name + fnnames_returning_borrowed_refs.add(fnname) + gcc.register_attribute('cpychecker_returns_borrowed_ref', 0, 0, False, False, False, attribute_callback_for_returns_borrowed_ref) gcc.define_macro('WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE') + # Handler for __attribute__((cpychecker_steals_reference_to_arg(n))) + # and #ifdef WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE + def attribute_callback_for_steals_reference_to_arg(*args): + if 0: + print('attribute_callback_for_steals_reference_to_arg(%r)' % (args, )) + check_isinstance(args[0], gcc.FunctionDecl) + check_isinstance(args[1], gcc.IntegerCst) + fnname = args[0].name + argindex = int(args[1].constant) + + if fnname in stolen_refs_by_fnname: + stolen_refs_by_fnname[fnname].add(argindex) + else: + stolen_refs_by_fnname[fnname] = set([argindex]) + + gcc.register_attribute('cpychecker_steals_reference_to_arg', + 1, 1, + False, False, False, + attribute_callback_for_steals_reference_to_arg) + gcc.define_macro('WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE') diff --git a/libcpychecker/refcounts.py b/libcpychecker/refcounts.py index 3bfa382..57ecd31 100644 --- a/libcpychecker/refcounts.py +++ b/libcpychecker/refcounts.py @@ -26,7 +26,8 @@ import gcc from gccutils import cfg_to_dot, invoke_dot, get_src_for_loc, check_isinstance from libcpychecker.absinterp import * -from libcpychecker.attributes import fnnames_returning_borrowed_refs +from libcpychecker.attributes import fnnames_returning_borrowed_refs, \ + stolen_refs_by_fnname from libcpychecker.diagnostics import Reporter, Annotator, Note from libcpychecker.PyArg_ParseTuple import PyArgParseFmt, FormatStringError, \ TypeCheckCheckerType, TypeCheckResultType @@ -2271,6 +2272,17 @@ def check_refcounts(fun, dump_traces=False, show_traces=False, exp_refcnt = len(exp_refs) log('exp_refs: %r', exp_refs) + if fun.decl.name in stolen_refs_by_fnname: + # Then this function is marked as stealing references to one or + # more of its arguments: + for argindex in stolen_refs_by_fnname[fun.decl.name]: + # Get argument's value (in initial state of trace): + parm = fun.decl.arguments[argindex - 1] + v_parm = trace.states[0].eval_rvalue(parm, None) + if isinstance(v_parm, PointerToRegion): + if region == v_parm.region: + exp_refcnt -= 1 + # Helper function for when ob_refcnt is wrong: def emit_refcount_error(msg): err = rep.make_error(fun, endstate.get_gcc_loc(fun), msg) diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c new file mode 100644 index 0000000..202d11d --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c @@ -0,0 +1,60 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +#include <Python.h> + +/* + Test of compiling a function that's been (correctly) marked as stealing + a reference +*/ + +#if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) +#else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) + #error (This should have been defined) +#endif + +extern PyObject *test(PyObject *foo, PyObject *bar) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(1) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(2); + +PyObject *saved_objA; +PyObject *saved_objB; + +PyObject * +test(PyObject *foo, PyObject *bar) +{ + /* + This code steals references to both of its arguments + */ + saved_objA = foo; + saved_objB = bar; + + Py_RETURN_NONE; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/script.py b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/script.py new file mode 100644 index 0000000..6bb62fd --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/script.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main + +main(verify_refcounting=True, + dump_traces=True) diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/stdout.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/stdout.txt new file mode 100644 index 0000000..15cad05 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/stdout.txt @@ -0,0 +1,20 @@ +Trace 0: + Transitions: + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c', line=51), region=RegionForGlobal(gcc.VarDecl('_Py_NoneStruct'))) + str(): (struct PyObject *)&RegionForGlobal(gcc.VarDecl('_Py_NoneStruct')) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c:51 + r->ob_refcnt: refs: 1 + N where N >= 1 + r->ob_type: None + Region("region-for-arg-gcc.ParmDecl('foo')"): + repr(): Region("region-for-arg-gcc.ParmDecl('foo')") + str(): Region("region-for-arg-gcc.ParmDecl('foo')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c', line=43), region=Region("region-for-type-of-arg-gcc.ParmDecl('foo')")) + Region("region-for-arg-gcc.ParmDecl('bar')"): + repr(): Region("region-for-arg-gcc.ParmDecl('bar')") + str(): Region("region-for-arg-gcc.ParmDecl('bar')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c', line=43), region=Region("region-for-type-of-arg-gcc.ParmDecl('bar')")) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-marking/input.c:44 diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c new file mode 100644 index 0000000..4670c44 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c @@ -0,0 +1,68 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +#include <Python.h> + +/* + Test of correctly calling a function that's been marked as stealing a + reference to its arguments +*/ + +#if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) +#else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) + #error (This should have been defined) +#endif + +extern void foo(PyObject *obj) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(1); + +PyObject * +test(PyObject *self, PyObject *args) +{ + PyObject *obj; + + obj = PyLong_FromLong(42); + if (!obj) { + return NULL; + } + + /* + We now own a reference to "obj" + + Pass it to foo(), which steals it. + + Given that foo has been marked with + __attribute__((cpychecker_steals_reference_to_arg(1))) + the checker should not complain. + */ + foo(obj); + + Py_RETURN_NONE; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/script.py b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/script.py new file mode 100644 index 0000000..6bb62fd --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/script.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main + +main(verify_refcounting=True, + dump_traces=True) diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/stdout.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/stdout.txt new file mode 100644 index 0000000..4a08c84 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/stdout.txt @@ -0,0 +1,48 @@ +Trace 0: + Transitions: + 'when PyLong_FromLong() succeeds' + 'taking False path' + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=59), region=RegionForGlobal(gcc.VarDecl('_Py_NoneStruct'))) + str(): (struct PyObject *)&RegionForGlobal(gcc.VarDecl('_Py_NoneStruct')) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:59 + r->ob_refcnt: refs: 1 + N where N >= 1 + r->ob_type: None + Region("region-for-arg-gcc.ParmDecl('self')"): + repr(): Region("region-for-arg-gcc.ParmDecl('self')") + str(): Region("region-for-arg-gcc.ParmDecl('self')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=39), region=Region("region-for-type-of-arg-gcc.ParmDecl('self')")) + Region("region-for-arg-gcc.ParmDecl('args')"): + repr(): Region("region-for-arg-gcc.ParmDecl('args')") + str(): Region("region-for-arg-gcc.ParmDecl('args')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=39), region=Region("region-for-type-of-arg-gcc.ParmDecl('args')")) + PyLongObject allocated at tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:43: + repr(): RegionOnHeap('PyLongObject', gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=43)) + str(): PyLongObject allocated at tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:43 + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=43), region=RegionForGlobal(gcc.VarDecl('PyLong_Type'))) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:40 + +Trace 1: + Transitions: + 'when PyLong_FromLong() fails' + 'taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=45), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:45 + Region("region-for-arg-gcc.ParmDecl('self')"): + repr(): Region("region-for-arg-gcc.ParmDecl('self')") + str(): Region("region-for-arg-gcc.ParmDecl('self')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=39), region=Region("region-for-type-of-arg-gcc.ParmDecl('self')")) + Region("region-for-arg-gcc.ParmDecl('args')"): + repr(): Region("region-for-arg-gcc.ParmDecl('args')") + str(): Region("region-for-arg-gcc.ParmDecl('args')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c', line=39), region=Region("region-for-type-of-arg-gcc.ParmDecl('args')")) + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/correct-usage/input.c:43 diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c new file mode 100644 index 0000000..976bcc0 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c @@ -0,0 +1,55 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +#include <Python.h> + +/* + Test of compiling a function that's been incorrectly marked as stealing + a reference, when it does not. +*/ + +#if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) +#else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) + #error (This should have been defined) +#endif + +extern PyObject *test(PyObject *foo, PyObject *bar) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(1) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(2); + +PyObject * +test(PyObject *foo, PyObject *bar) +{ + /* + This code doesn't steals references to its arguments; the marking is + incorrect, and this should be flagged by the checker. + */ + Py_RETURN_NONE; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/script.py b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/script.py new file mode 100644 index 0000000..6bb62fd --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/script.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main + +main(verify_refcounting=True, + dump_traces=True) diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stderr.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stderr.txt new file mode 100644 index 0000000..18ea731 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stderr.txt @@ -0,0 +1,10 @@ +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c: In function 'test': +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: error: ob_refcnt of '*foo' is 1 too high +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: was expecting final ob_refcnt to be N + -1 (for some unknown N) +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: but final ob_refcnt is N + 0 +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: returning at: Py_RETURN_NONE; +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: error: ob_refcnt of '*bar' is 1 too high +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: was expecting final ob_refcnt to be N + -1 (for some unknown N) +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: but final ob_refcnt is N + 0 +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46:5: note: returning at: Py_RETURN_NONE; +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:41:1: note: graphical error report for function 'test' written out to 'tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c.test-refcount-errors.html' diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stdout.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stdout.txt new file mode 100644 index 0000000..a4ca8ec --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/stdout.txt @@ -0,0 +1,20 @@ +Trace 0: + Transitions: + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c', line=46), region=RegionForGlobal(gcc.VarDecl('_Py_NoneStruct'))) + str(): (struct PyObject *)&RegionForGlobal(gcc.VarDecl('_Py_NoneStruct')) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:46 + r->ob_refcnt: refs: 1 + N where N >= 1 + r->ob_type: None + Region("region-for-arg-gcc.ParmDecl('foo')"): + repr(): Region("region-for-arg-gcc.ParmDecl('foo')") + str(): Region("region-for-arg-gcc.ParmDecl('foo')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('foo')")) + Region("region-for-arg-gcc.ParmDecl('bar')"): + repr(): Region("region-for-arg-gcc.ParmDecl('bar')") + str(): Region("region-for-arg-gcc.ParmDecl('bar')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('bar')")) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-marking/input.c:41 diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c new file mode 100644 index 0000000..4ce5d0a --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c @@ -0,0 +1,72 @@ +/* + Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> + Copyright 2011 Red Hat, Inc. + + This is free software: you can redistribute it and/or modify it + under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, but + WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see + <http://www.gnu.org/licenses/>. +*/ + +#include <Python.h> + +/* + Test of calling a function that's been marked as stealing a + reference to its arguments, but incorrectly treating the argument as + still being owned by us. +*/ + +#if defined(WITH_CPYCHECKER_STEALS_REFERENCE_TO_ARG_ATTRIBUTE) + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) \ + __attribute__((cpychecker_steals_reference_to_arg(n))) +#else + #define CPYCHECKER_STEALS_REFERENCE_TO_ARG(n) + #error (This should have been defined) +#endif + +extern void foo(PyObject *obj) + CPYCHECKER_STEALS_REFERENCE_TO_ARG(1); + +PyObject * +test(PyObject *self, PyObject *args) +{ + PyObject *obj; + + obj = PyLong_FromLong(42); + if (!obj) { + return NULL; + } + + /* + We now own a reference to "obj" + + Pass it to foo(), which steals it. + + */ + foo(obj); + + /* + Given that foo has been marked with + __attribute__((cpychecker_steals_reference_to_arg(1))) + this is an error: we no longer own the reference to "obj" + that's expected of us when returning a non-NULL PyObject*: + */ + return obj; +} + +/* + PEP-7 +Local variables: +c-basic-offset: 4 +indent-tabs-mode: nil +End: +*/ diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/script.py b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/script.py new file mode 100644 index 0000000..6bb62fd --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/script.py @@ -0,0 +1,22 @@ +# -*- coding: utf-8 -*- +# Copyright 2011 David Malcolm <dmalcolm(a)redhat.com> +# Copyright 2011 Red Hat, Inc. +# +# This is free software: you can redistribute it and/or modify it +# under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, but +# WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program. If not, see +# <http://www.gnu.org/licenses/>. + +from libcpychecker import main + +main(verify_refcounting=True, + dump_traces=True) diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stderr.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stderr.txt new file mode 100644 index 0000000..4add37a --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stderr.txt @@ -0,0 +1,13 @@ +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c: In function 'test': +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:64:1: error: ob_refcnt of return value is 1 too low +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:64:1: note: was expecting final ob_refcnt to be N + 1 (for some unknown N) +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:64:1: note: due to object being referenced by: return value +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:64:1: note: but final ob_refcnt is N + 0 +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:44:9: note: PyLongObject allocated at: obj = PyLong_FromLong(42); +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:44:9: note: when PyLong_FromLong() succeeds at: obj = PyLong_FromLong(42); +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:44:9: note: ob_refcnt is now refs: 1 + N where N >= 0 +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:45:8: note: taking False path at: if (!obj) { +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:55:8: note: reaching: foo(obj); +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:55:8: note: ob_refcnt is now refs: 0 + N where N >= 1 +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:64:1: note: returning +tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:41:1: note: graphical error report for function 'test' written out to 'tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c.test-refcount-errors.html' diff --git a/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stdout.txt b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stdout.txt new file mode 100644 index 0000000..829f7d1 --- /dev/null +++ b/tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/stdout.txt @@ -0,0 +1,43 @@ +Trace 0: + Transitions: + 'when PyLong_FromLong() succeeds' + 'taking False path' + 'returning' + Return value: + repr(): PointerToRegion(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=44), region=RegionOnHeap('PyLongObject', gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=44))) + str(): (struct PyObject *)&RegionOnHeap('PyLongObject', gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=44)) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:44 + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=44), region=RegionForGlobal(gcc.VarDecl('PyLong_Type'))) + Region("region-for-arg-gcc.ParmDecl('self')"): + repr(): Region("region-for-arg-gcc.ParmDecl('self')") + str(): Region("region-for-arg-gcc.ParmDecl('self')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('self')")) + Region("region-for-arg-gcc.ParmDecl('args')"): + repr(): Region("region-for-arg-gcc.ParmDecl('args')") + str(): Region("region-for-arg-gcc.ParmDecl('args')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('args')")) + Exception: + (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:41 + +Trace 1: + Transitions: + 'when PyLong_FromLong() fails' + 'taking True path' + 'returning' + Return value: + repr(): ConcreteValue(gcctype='struct PyObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=46), value=0) + str(): (struct PyObject *)0 from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:46 + Region("region-for-arg-gcc.ParmDecl('self')"): + repr(): Region("region-for-arg-gcc.ParmDecl('self')") + str(): Region("region-for-arg-gcc.ParmDecl('self')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('self')")) + Region("region-for-arg-gcc.ParmDecl('args')"): + repr(): Region("region-for-arg-gcc.ParmDecl('args')") + str(): Region("region-for-arg-gcc.ParmDecl('args')") + r->ob_refcnt: refs: 0 + N where N >= 1 + r->ob_type: PointerToRegion(gcctype='struct PyTypeObject *', loc=gcc.Location(file='tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c', line=40), region=Region("region-for-type-of-arg-gcc.ParmDecl('args')")) + Exception: + (struct PyObject *)&RegionForGlobal(gcc.VarDecl('PyExc_MemoryError')) from tests/cpychecker/refcounts/attributes/steals-reference-to-arg/incorrect-usage/input.c:44

1 0

[gcc-python-plugin] docs: add another bug found in gdb
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit c2a57529b75ecfdacade84c0c6757d35ced08006 Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 18:25:30 2011 -0400 docs: add another bug found in gdb docs/success.rst | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) --- diff --git a/docs/success.rst b/docs/success.rst index 0afa83a..8dbb44d 100644 --- a/docs/success.rst +++ b/docs/success.rst @@ -33,6 +33,7 @@ Bugs found in gdb by running the cpychecker script when compiling it: * http://sourceware.org/bugzilla/show_bug.cgi?id=13316 * http://sourceware.org/ml/gdb-patches/2011-06/msg00376.html * http://sourceware.org/ml/gdb-patches/2011-10/msg00391.html + * http://sourceware.org/bugzilla/show_bug.cgi?id=13331 Tom Tromey also wrote a specialized Python script to analyze gdb's resource-management code, which found some resource leaks and a possible

1 0

[gcc-python-plugin] docs: start addding bugs found in the plugin itself to the success stories
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit 4063c24bc5c4498d1c03842e4054d359c67eb427 Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 18:22:57 2011 -0400 docs: start addding bugs found in the plugin itself to the success stories docs/success.rst | 9 +++++++++ 1 files changed, 9 insertions(+), 0 deletions(-) --- diff --git a/docs/success.rst b/docs/success.rst index 8d5f668..0afa83a 100644 --- a/docs/success.rst +++ b/docs/success.rst @@ -37,4 +37,13 @@ Bugs found in gdb by running the cpychecker script when compiling it: Tom Tromey also wrote a specialized Python script to analyze gdb's resource-management code, which found some resource leaks and a possible crasher: + * http://sourceware.org/ml/gdb-patches/2011-06/msg00408.html + + +Bugs found in itself +-------------------- +Bugs found in the gcc Python plugin itself, by running the cpychecker script +when compiling another copy: + + * bad format string: https://fedorahosted.org/pipermail/gcc-python-plugin/2011-August/000065.html

1 0

[gcc-python-plugin] cpychecker: further fixes to wording in docs
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit dea65059deeb2e3bc709c58ed943f64dcdfb9f1d Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 17:14:30 2011 -0400 cpychecker: further fixes to wording in docs docs/cpychecker.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) --- diff --git a/docs/cpychecker.rst b/docs/cpychecker.rst index a3355e1..a520ae9 100644 --- a/docs/cpychecker.rst +++ b/docs/cpychecker.rst @@ -112,8 +112,8 @@ along with this HTML report (as referred to by the final line on stderr): :alt: screenshot of the HTML report The HTML report is intended to be relatively self-contained, and thus easy to -attached to bug tracking systems: it embeds its own CSS inline, and references -the JavaScript it uses via URLs to the web. +attach to bug tracking systems (it embeds its own CSS inline, and references +the JavaScript it uses via URLs to the web). .. note:: The arrow graphics in the HTML form of the report are added by using the JSPlumb JavaScript library to generate HTML 5 <canvas> elements. You @@ -190,8 +190,8 @@ The checker will emit errors for various events: * if it detects a dereferencing of a ``NULL`` value * if a ``NULL`` value is erroneously passed to various CPython API - entrypoints which implicitly deference those arguments (which would lead - to a segmentation fault if that code path were executed). + entrypoints which are known to implicitly dereference those arguments + (which would lead to a segmentation fault if that code path were executed). * if it detects that an uninitialized local variable has been used

1 0

[gcc-python-plugin] cpychecker: fix wording in docs
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit 4859a4382a8e04dbaf2588631c5a5f6ae17b580e Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 17:09:07 2011 -0400 cpychecker: fix wording in docs docs/cpychecker.rst | 8 ++++---- 1 files changed, 4 insertions(+), 4 deletions(-) --- diff --git a/docs/cpychecker.rst b/docs/cpychecker.rst index 9c4a4de..a3355e1 100644 --- a/docs/cpychecker.rst +++ b/docs/cpychecker.rst @@ -64,9 +64,10 @@ reference count ought to be at the end of the function, issuing an error report for any that are incorrect. The error report is in two forms: the classic textual output to GCC's standard -error stream. +error stream, together with an HTML report indicating the flow through the +function, in graphical form. -For example, given: +For example, given this buggy C code: .. code-block:: c @@ -105,8 +106,7 @@ the checker emits these messages to stderr:: input.c:38:1: note: returning input.c:24:1: note: graphical error report for function 'test' written out to 'input.c.test-refcount-errors.html' -and also an HTML report indicating the flow through the function, in graphical -form: +along with this HTML report (as referred to by the final line on stderr): .. figure:: sample-html-error-report.png :alt: screenshot of the HTML report

1 0

[gcc-python-plugin] cpychecker: revamp of the documentation
by David Hugh Malcolm 26 Oct '11

26 Oct '11

commit 3eaf0eb6a7ed791c2e7c478aba6263d94f0848bf Author: David Malcolm <dmalcolm(a)redhat.com> Date: Wed Oct 26 16:59:16 2011 -0400 cpychecker: revamp of the documentation docs/cpychecker.rst | 320 +++++++++++++++++++++++++++++++++---- docs/sample-html-error-report.png | Bin 0 -> 72902 bytes 2 files changed, 292 insertions(+), 28 deletions(-) --- diff --git a/docs/cpychecker.rst b/docs/cpychecker.rst index 2503c66..9c4a4de 100644 --- a/docs/cpychecker.rst +++ b/docs/cpychecker.rst @@ -18,59 +18,315 @@ Usage example: a static analysis tool for CPython extension code ================================================================ +.. note:: This code is under heavy development, and still contains bugs. It + is not unusual to see Python tracebacks when running the checker. You + should verify what the checker reports before acting on it: it could be + wrong. + An example of using the plugin is a static analysis tool I'm working on which checks the C source of CPython extension modules for common coding errors. -This code is under heavy development, and is not yet ready to be used. - This was one of my main motivations for writing the GCC plugin, and I often need to extend the plugin to support this use case. For this reason, the checker is embedded within the gcc-python source tree itself for now: - * `cpychecker.py` is a short top-level script, which will eventually be - invoked like this:: - - gcc -fplugin=python.so -fplugin-arg-python-script=cpychecker.py OTHER_ARGS + * `gcc-with-cpychecker` is a harness script, which invokes GCC, adding + the arguments necessary to use the Python plugin, using the + `libcpychecker` Python code * the `libcpychecker` subdirectory contains the code that does the actual work - * various test cases + * various test cases (in the source tree, below `tests/cpychecker`) -Here's a list of C coding bugs I intend for the tool to detect (each one -followed by its current status): +So it should be possible to use the checker on arbitrary CPython extension +code by replacing "gcc" with "gcc-with-cpychecker" in your build with +something like:: - * lack of error handling (i.e. assuming that calls to the Python API - succeed) [AT EARLY, BUGGY, STAGE] + make CC=/path/to/built/plugin/gcc-with-cpychecker - * reference-counting errors: [AT EARLY, BUGGY, STAGE] +You may need to supply an absolute path, especially if the "make" recursively +invokes "make" within subdirectories (thus having a different working +directory). - * missing Py_INCREF/Py_DECREF +(TODO: some way of integrating the checker with `distutils +<http://docs.python.org/library/distutils.html>`_) - * leaking references to objects, not having enough references to an object +Reference-count checking +------------------------ +The checker attempts to analyze all possible paths through each function, +tracking the various PyObject* objects encountered. - * using an object after a point where it might have been deleted +For each path through the function and PyObject*, it determines what the +reference count ought to be at the end of the function, issuing an error report +for any that are incorrect. - * tp_traverse errors (which can mess up the garbage collector); missing it - altogether, or omitting fields [NOT IMPLEMENTED YET] +The error report is in two forms: the classic textual output to GCC's standard +error stream. + +For example, given: + +.. code-block:: c + + PyObject * + test(PyObject *self, PyObject *args) + { + PyObject *list; + PyObject *item; + list = PyList_New(1); + if (!list) + return NULL; + item = PyLong_FromLong(42); + /* This error handling is incorrect: it's missing an + invocation of Py_DECREF(list): */ + if (!item) + return NULL; + /* This steals a reference to item; item is not leaked when we get here: */ + PyList_SetItem(list, 0, item); + return list; + } + +the checker emits these messages to stderr:: + + input.c: In function 'test': + input.c:38:1: error: ob_refcnt of '*list' is 1 too high + input.c:38:1: note: was expecting final ob_refcnt to be N + 0 (for some unknown N) + input.c:38:1: note: but final ob_refcnt is N + 1 + input.c:27:10: note: PyListObject allocated at: list = PyList_New(1); + input.c:27:10: note: when PyList_New() succeeds at: list = PyList_New(1); + input.c:27:10: note: ob_refcnt is now refs: 1 + N where N >= 0 + input.c:28:8: note: taking False path at: if (!list) + input.c:30:10: note: reaching: item = PyLong_FromLong(42); + input.c:30:10: note: when PyLong_FromLong() fails at: item = PyLong_FromLong(42); + input.c:33:8: note: taking True path at: if (!item) + input.c:34:9: note: reaching: return NULL; + input.c:38:1: note: returning + input.c:24:1: note: graphical error report for function 'test' written out to 'input.c.test-refcount-errors.html' + +and also an HTML report indicating the flow through the function, in graphical +form: + + .. figure:: sample-html-error-report.png + :alt: screenshot of the HTML report + +The HTML report is intended to be relatively self-contained, and thus easy to +attached to bug tracking systems: it embeds its own CSS inline, and references +the JavaScript it uses via URLs to the web. + +.. note:: The arrow graphics in the HTML form of the report are added by using + the JSPlumb JavaScript library to generate HTML 5 <canvas> elements. You + may need a relatively modern browser to see them. + +.. note:: The checker tracks reference counts in an abstract way, in two parts: + a part of the reference count that it knows about within the context of the + function, along with a second part: all of the other references held by the + rest of the program. + + For example, in a call to PyInt_FromLong(0), it is assumed that if the call + succeeds, the object has a reference count of 1 + N, where N is some unknown + amount of other references held by the rest of the program. The checker + knows that N >= 0. + + If the object is then stored in an opaque container which is known to + increment the reference count, the checker can say that the reference count + is then 1 + (N+1). + + If the function then decrements the reference count (to finish transferring + the reference to the opaque container), the checker now treats the object as + having a reference count of 0 + (N+1): it no longer owns any references on + the object, but the reference count is actually unchanged relative to the + original 1 + N amount. It also knows, given that N >= 0 that the actual + reference count is >= 1, and thus the object won't (yet) be deallocated. + +Assumptions and configuration +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +For any function returning a PyObject*, it assumes that the PyObject* should be +either a new reference to an object, or NULL (with an exception set) - the +function's caller should "own" a reference to that object. For all +other PyObject*, it assumes that there should be no references owned by the +function when the function terminates. + +It will assume this behavior for any function (or call through a function +pointer) that returns a PyObject*. + +It is possible to override this behavior using custom compiler attributes: + +.. code-block:: c + + /* The checker automatically defines this preprocessor name when creating + the custom attribute: */ + #if defined(WITH_CPYCHECKER_RETURNS_BORROWED_REF_ATTRIBUTE) + #define CPYCHECKER_RETURNS_BORROWED_REF \ + __attribute__((cpychecker_returns_borrowed_ref)) + #else + #define CPYCHECKER_RETURNS_BORROWED_REF + #endif + + PyObject *foo(void) + CPYCHECKER_RETURNS_BORROWED_REF; + +Given the above, the checker will assume that invocations of ``foo()`` are +returning a borrowed reference (or NULL), rather than a new reference, and +will apply the same policy when verifying the implementation of ``foo()`` +itself. + +Error-handling checking +----------------------- +The checker has knowledge of much of the CPython C API, and will generate +a trace tree containing many of the possible error paths. It will issue +error reports for code that appears to not gracefully handle an error. + +(TODO: show example) + +As noted above, it assumes that any function that returns a PyObject* can +return can either NULL (setting an exception), or a new reference. It knows +about much of the other parts of the CPython C API, including many other +functions that can fail. + +The checker will emit errors for various events: + + * if it detects a dereferencing of a ``NULL`` value + + * if a ``NULL`` value is erroneously passed to various CPython API + entrypoints which implicitly deference those arguments (which would lead + to a segmentation fault if that code path were executed). + + * if it detects that an uninitialized local variable has been used + + * if it detects access to an object that has been deallocated, or such an + object being returned:: + + input.c: In function 'test': + input.c:43:1: error: returning pointer to deallocated memory + input.c:29:15: note: when PyLong_FromLong() succeeds at: PyObject *tmp = PyLong_FromLong(0x1000); + input.c:31:8: note: taking False path at: if (!tmp) { + input.c:39:5: note: reaching: Py_DECREF(tmp); + input.c:39:5: note: when taking False path at: Py_DECREF(tmp); + input.c:39:5: note: reaching: Py_DECREF(tmp); + input.c:39:5: note: calling tp_dealloc on PyLongObject allocated at input.c:29 at: Py_DECREF(tmp); + input.c:42:5: note: reaching: return tmp; + input.c:43:1: note: returning + input.c:39:5: note: memory deallocated here + input.c:27:1: note: graphical error report for function 'returning_dead_object' written out to 'input.c.test.html' - * errors in PyArg_ParseTuple and friends: +Errors in exception-handling +---------------------------- +The checker keeps track of the per-thread exception state. It will issue a +warning about any paths through functions returning a PyObject* that return +NULL for which the per-thread exception state has not been set:: - * type mismatches e.g. `int` vs `long` (often leads to flaws on big-endian - 64-bit architectures, where sizeof(int) != sizeof(long)) - [MOSTLY IMPLEMENTED FOR PyArg_Parse*, NOT IMPLEMENTED FOR Py_BuildValue] + input.c: In function 'test': + input.c:32:5: error: returning (PyObject*)NULL without setting an exception - * reference-counting errors, for the cases where objects are returned with - either new or borrowed references (depending on the format codes). - [NOT IMPLEMENTED YET] +(TODO: provide a way to mark a function as setting this state) - * errors in exception handling, so that we can issue errors about e.g. not - returning NULL if an exception is set, or returning NULL if an exception is - not set [NOT IMPLEMENTED YET] - * errors in GIL-handling [NOT IMPLEMENTED YET] +Format string checking +---------------------- + +The checker will analyze some `Python APIs that take format strings +<http://docs.python.org/c-api/arg.html>`_ and detect mismatches between the +number and types of arguments that are passed in, as compared with those +described by the format string. + +It currently verifies the arguments to the following API entrypoints: + + * `PyArg_ParseTuple + <http://docs.python.org/c-api/arg.html#PyArg_ParseTuple>`_ + + * `PyArg_ParseTupleAndKeywords + <http://docs.python.org/c-api/arg.html#PyArg_ParseTupleAndKeywords>`_ + + * `Py_BuildValue + <http://docs.python.org/c-api/arg.html#Py_BuildValue>`_ + +along with the variants that occur if you define `PY_SSIZE_T_CLEAN` before +`#include <Python.h>`. + +For example, type mismatches between ``int`` vs ``long`` can lead to flaws +when the code is compiled on big-endian 64-bit architectures, where +``sizeof(int) != sizeof(long)`` and the in-memory layout of those types differs +from what you might expect. + +The checker will also issue a warning if the list of keyword arguments in a +call to PyArg_ParseTupleAndKeywords is not NULL-terminated. + +.. note:: All of the various "#" codes in these format strings are affected by + the presence of the macro `PY_SSIZE_T_CLEAN`. If the macro was defined + before including Python.h, the various lengths for these format codes are of + C type `Py_ssize_t` rather than `int`. + + This behavior was clarified in the Python 3 version of the C API + documentation, though the Python 2 version of the API docs leave the matter + of which codes are affected somewhat ambiguous. + + Nevertheless, the API *does* work this way in Python 2: all format codes + with a "#" do work this way. + + Internally, the C preprocessor converts such function calls into invocations + of: + + * `_PyArg_ParseTuple_SizeT` + * `_PyArg_ParseTupleAndKeywords_SizeT` + + The checker handles this behavior correctly, by checking "#" codes in the + regular functions against `int` and those in the modified functions against + `Py_ssize_t`. + + +Limitations and caveats +----------------------- + +Compiling with the checker is significantly slower than with "vanilla" gcc. +I have been focussing on correctness and features, rather than optimization. +I hope that it will be possible to greatly speed up the checker via +ahead-of-time compilation of the Python code (e.g. using Cython). + +The checker does not yet fully implement all of C: expect to see Python +tracebacks when it encounters less common parts of the language. (We'll fix +those bugs as we come to them) + +The checker has a rather simplistic way of tracking the flow through a +function: it builds a tree of all possible traces of execution through a +function. This brings with it some shortcomings: + + * In order to guarantee that the analysis terminates, the checker will only + track the first time through any loop, and stop analysing that trace for + subsequent iterations. This appears to be good enough for detecting many + kinds of reference leaks, especially in simple wrapper code, but is clearly + suboptimal. + + * In order to avoid combinatorial explosion, the checker will stop analyzing + a function once the trace tree gets sufficiently large. When it reaches + this cutoff, a warning is issued:: + + input.c: In function 'add_module_objects': + input.c:31:1: note: this function is too complicated for the reference-count checker to analyze + + * The checker doesn't yet match up similar traces, and so a single bug that + affects multiple traces in the trace tree can lead to duplicate error + reports. + +Only a subset of the CPython API has been modelled so far. + + +Ideas for future tests +---------------------- + +Here's a list of some other C coding bugs I intend for the tool to detect: + + * type-checking for PyMethodDef tables: the callbacks are typically cast + to PyCFunction, but the exact type needs to correspond to the flags + given (e.g. ``(METH_VARARGS | METH_KEYWORDS)`` implies a different function + signature to the default, and the C compiler has currently no way of + verifying that these agree with each other). + + * tp_traverse errors (which can mess up the garbage collector); missing it + altogether, or omitting fields + + * errors in GIL-handling * lock/release mismatches @@ -82,3 +338,11 @@ Ideas for other tests are most welcome (patches even more so!) We will probably need various fallbacks and suppression modes for turning off individual tests (perhaps pragmas, perhaps compile-line flags, etc) + +Reusing this code for other projects +------------------------------------ +It may be possible to reuse the analysis engine from cpychecker for other +kinds of analysis - hopefully the python-specific parts are relatively +self-contained. Email the `gcc-python-plugin's mailing list +<https://fedorahosted.org/mailman/listinfo/gcc-python-plugin/>`_ if you're +interested in adding verifiers for other kinds of code. diff --git a/docs/sample-html-error-report.png b/docs/sample-html-error-report.png new file mode 100644 index 0000000..add7291 Binary files /dev/null and b/docs/sample-html-error-report.png differ

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

gcc-python-plugin-commits October 2011