gach-dev November 2009

gach-dev@lists.fedorahosted.org

1 participants
10 discussions

for reference [Fwd: regarding spotting unowned directories in fedora]
by Rakesh Pandit 29 Nov '09

29 Nov '09

---------- Forwarded message ---------- From: Michael Schwendt <mschwendt(a)gmail.com> Date: 2009/11/29 Subject: Re: regarding spotting unowned directories in fedora To: Rakesh Pandit <rakesh.pandit(a)gmail.com> On Fri, 27 Nov 2009 14:18:35 +0530, Rakesh wrote: > Hello Michael, > > I believe you have a script which can figure out unowned directories > in a package. May you post it to me. I want to have a look at it and > develop a check which can go into gach[1]. I also request you to fill > up some tickets for all guideline checks which you consider can be > automated into out trac instance[2], in case you feel like doing it :) > > I haven't been able to invest time in this project for long now, but > these days I am fairly active and am planning release near new year > eve. It would be great if you can help in figuring out what all checks > can be automated. > > As you might already know I plan to try implementing checks in rpmlint > .. but in case they don't go there and are hybrid checks as in require > inspection of srpms, logs, source in mix, or interaction with koji, or > other build systems will do that. > > Thanks, > > [1] https://fedorahosted.org/gach > [2] https://fedorahosted.org/gach/newticket Hi! The tool is called "dircheck-remote" and has not been developed further since Jan/Feb. As I've pointed to it in bugzilla, it's still found in a public place, though: http://mschwendt.fedorapeople.org/dircheck-remote.py Yum API has changed a few months ago once more and required a small update. By default, it evaluates yum.conf and enabled repos, but it can also be told to look at specific repos or specific packages only, such as: $ dircheck-remote.py -n ^ypserv [...] => ypserv-2.19-13.fc12.src.rpm => ypserv-2.19-13.fc12.i686 (fedora) /usr/include/rpcsvc provided by: 1:quota-devel-3.17-8.fc12.i686 provided by: glibc-headers-2.11-2.i686 Any path it prints, is "unowned". The "provided by" lines are hints about what other packages own a directory. Note that one big problem caused by unowned directories is gone as of RPM 4.4.2.3: https://fedoraproject.org/wiki/Packaging:UnownedDirectories#Inaccessible_Di… With that item no longer being an issue, the problem of unowned directories is less interesting. What's left is that -- regularly -- discovering unowned directories leads to discovering major mistakes made in a package. e.g. misplaced files, bad subpackage dependencies. The most recent RPM (announced on fedora-devel-list some days ago) will do ordered package erasures. To include a directory in the _right_ package instead of in an arbitrary package of a dependency-chain is something that hasn't been checked before. Basically, with a weird dependency-chain, it is possible to own a directory in a package that would be installed _last_ and erased _first_. That would lead to unremoved directories _after_ erasing packages. A related problem, but would require a check that simulates in which order RPM will erase a set of packages. -- Michael Schwendt <mschwendt(a)fedoraproject.org> Fedora release 12 (Constantine) - Linux 2.6.31.6-145.fc12.i686.PAE loadavg: 1.59 1.75 1.83 -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first

1 0

some adjustment to Log check
by Rakesh Pandit 26 Nov '09

26 Nov '09

some adjustment to Log check -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first From: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Thu, 26 Nov 2009 06:18:01 +0000 (+0530) Subject: BuildLog.py: Fixed log file exception. X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h=d… BuildLog.py: Fixed log file exception. LogWarningCheck.py: Adjusted report for no log. --- diff --git a/BuildLog.py b/BuildLog.py index 866215d..e95a79d 100644 --- a/BuildLog.py +++ b/BuildLog.py @@ -27,7 +27,14 @@ class BuildLog(object): self.file = file self.filename = os.path.basename(file) self.path = os.path.dirname(file) - fobj = open(self.file, "r") + self.log_lines = [] + + fobj = None + try: + fobj = open(self.file, "r") + except IOError: + # Return in case no log found + return try: self.log_lines = fobj.readlines() finally: diff --git a/LogWarningCheck.py b/LogWarningCheck.py index c257ab3..cbe4713 100644 --- a/LogWarningCheck.py +++ b/LogWarningCheck.py @@ -41,11 +41,14 @@ class LogWarningCheck(Check.Check): """ log_file = details["log"] log = BuildLog(log_file) - reporter.printDividerInfo("Log info %s" %(log_file)) - # Grep all lines lines = log.searchstring(["warning", "Warning", "WARNING"]) + # Return if no warnings found + if not lines: + return + + reporter.printDividerInfo("Log info %s" %(log_file)) for line in lines: reporter.printLogWarnings(line)

1 0

Check for warning .. whether a doc subpackage is required
by Rakesh Pandit 24 Nov '09

24 Nov '09

ExcessDocFileCheck: New check -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first From: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Tue, 24 Nov 2009 10:52:07 +0000 (+0530) Subject: ExcessDocFileCheck.py: New file. X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h=4… ExcessDocFileCheck.py: New file. GachApplication.py, GachRPMReviewer.py, GachReviewer.py, GachUtils.py: Adjusted accordingly. --- diff --git a/ExcessDocFileCheck.py b/ExcessDocFileCheck.py new file mode 100644 index 0000000..2b1146d --- /dev/null +++ b/ExcessDocFileCheck.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python +# +# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with translate; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +# +# Check for warning about files in /usr/share/doc more then 1024KB or +# 100 files and would be nice to go for a separate subpackage. +# + +import os + +import RPMCheck +from GachUtils import extract_rpm, rm_rf, get_folder_size +from GachReporter import GachReporter + +reporter = GachReporter() + +class ExcessDocFileCheck(RPMCheck.RPMCheck): + """Check whether Doc files are in excess + """ + + def __init__(self): + """ + """ + RPMCheck.RPMCheck.__init__(self, "ExcessDocFileCheck") + + def check(self, **details): + """ + """ + rpm_file = details["rpm"] + extract_outdir = extract_rpm(rpm_file) + docdir = extract_outdir + "/usr/share/doc/" + if os.path.exists(docdir): + folder_size = get_folder_size(docdir) + # If size greater then 1024K and 100 files + if int(folder_size) >= 1024: + reporter.printWarning("excess-doc-files-in-docdir", Size=folder_size, File=rpm_file) + + files = [] + for dir, subdir, fl in os.walk(docdir): + files.append(fl) + # Check whether number of files >= 100 + lg = len(files) + if lg >= 100: + reporter.printWarning("excess-doc-files-in-docdir", Numberoffiles=lg, File=rpm_file) + + rm_rf(extract_outdir) + return + +check = ExcessDocFileCheck() + +reporter.addDetails( +'excess-doc-files-in-docdir', +'''Doc files in /usr/share/doc more then 1024KB or file count >= 100.''' +) diff --git a/GachApplication.py b/GachApplication.py index fd5459d..a9ea5aa 100644 --- a/GachApplication.py +++ b/GachApplication.py @@ -131,10 +131,11 @@ class GachApplication: # Need to check later why this is needed print reporter.getReport() - # Load all src rpm checks checks reviewer = GachReviewer() srpmreviewer = GachSrcRPMReviewer() + rpmreviewer = GachRPMReviewer() + # Load all src rpm checks checks for file in downloaded_files: if file.endswith("src.rpm"): srpmreviewer.review(file) @@ -143,6 +144,11 @@ class GachApplication: for file in downloaded_files: if file.endswith("build.log"): reviewer.review(log=file) + reporter.printDividerInfo("Log check ends") + + # Load all rpm checks + for file in downloaded_files: + if file.endswith(".rpm") and not file.endswith("src.rpm"): print reporter.getReport() rm_rf(srpm_spec_downdir) diff --git a/GachRPMReviewer.py b/GachRPMReviewer.py index 7a820cb..b41ead4 100644 --- a/GachRPMReviewer.py +++ b/GachRPMReviewer.py @@ -28,14 +28,16 @@ class GachRPMReviewer(GachReviewer.GachReviewer): def __init__(self): """ """ - pass + # All Checks are loaded here. + self.default_guidelines = ("ExcessDocFileCheck",) + GachReviewer.GachReviewer.__init__(self, guidelines=self.default_guidelines) def review(self, rpm): """ Implements reviewer for rpm. """ - for guide in RPMCheck.RPMCheck.checks: - guide.check(rpm) + for guide in RPMCheck.RPMCheck.rpm_checks: + guide.check(rpm=rpm) return diff --git a/GachReviewer.py b/GachReviewer.py index f6beaca..3c7fd98 100644 --- a/GachReviewer.py +++ b/GachReviewer.py @@ -64,9 +64,9 @@ class GachReviewer(object): """ Implements the loading module. """ - print "Guidelines:", self.default_guidelines + # print "Guidelines:", self.default_guidelines for name in self.default_guidelines: - print "Loaded: ", name + # print "Loaded: ", name file = None path = None description = None diff --git a/GachUtils.py b/GachUtils.py index b93dad8..b876081 100644 --- a/GachUtils.py +++ b/GachUtils.py @@ -21,6 +21,7 @@ # import os +import stat import fnmatch import glob import sys @@ -98,3 +99,20 @@ def istext(s): t = s.translate(_null_trans, text_characters) return float(len(t))/len(s) >= 0.30 + +def get_folder_size(folder): + """Return folder size + """ + folder_size = commands.getoutput("du -s " + folder) +# FIXME: Current method does not ok, but below one even does not +# work fine. Need to get it right. + +# folder_size = 0 +# for (path, dirs, files) in os.walk(folder): +# for file in files: +# filename = os.path.join(path, file) +# if os.path.islink(filename) and not os.path.exists(os.readlink(filename)): +# continue +# folder_size += os.stat(filename)[stat.ST_SIZE] + return folder_size.split("\t")[0] +

1 0

fix to Koji build URL
by Rakesh Pandit 23 Nov '09

23 Nov '09

$subect -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first commit aca8c665e40a2ca3e5cbf7f02f3ad6810fca5ce5 Author: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Mon Nov 23 14:33:10 2009 +0530 GachKojiBuilder.py: Add Koji URL diff --git a/GachKojiBuilder.py b/GachKojiBuilder.py index 4f63806..720f265 100644 --- a/GachKojiBuilder.py +++ b/GachKojiBuilder.py @@ -85,10 +85,10 @@ class GachKojiBuilder(GachBuilder.GachBuilder): print "done." server_source = server_dir + "/" + os.path.basename(source) build_opts = {"scratch": True, "skip_tag": None} - print "Started koji build with", + print "Started koji build" self.taskId = self._session.build(server_source, target, build_opts) self.__logout_session() - print "taskid", self.taskId, ".. waiting ..." + print "Koji Build URL: http://koji.fedoraproject.org/koji/taskinfo?taskID=" + str(self.taskId), ".. waiting ..." if wait: while True: state = self.getState()

1 0

Just pushed new Log warning Check
by Rakesh Pandit 23 Nov '09

23 Nov '09

Just pushed Log Warning Check -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first From: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Mon, 23 Nov 2009 11:00:47 +0000 (+0530) Subject: BuildLog.py: New file X-Git-Url: http://git.fedorahosted.org/git/gach.git/?p=gach.git;a=commitdiff_plain;h=c… BuildLog.py: New file LogWarningCheck.py: New file Check.py, GachApplication.py, GachReporter.py, GachReviewer.py, SampleCheck.py: Minor adjustments to work with new check. --- diff --git a/BuildLog.py b/BuildLog.py new file mode 100644 index 0000000..866215d --- /dev/null +++ b/BuildLog.py @@ -0,0 +1,48 @@ +#!/usr/bin/env python +# +# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with translate; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +import os + + +class BuildLog(object): + """Log file class""" + + def __init__(self, file): + """ """ + self.file = file + self.filename = os.path.basename(file) + self.path = os.path.dirname(file) + fobj = open(self.file, "r") + try: + self.log_lines = fobj.readlines() + finally: + fobj.close() + + def searchstring(self, strglist): + """Search string in log""" + lines = [] + for line in self.log_lines: + for strg in strglist: + flag = 0 + if strg in line: + lines.append(line) + flag = 1 + # Prevents line to be included twice + if flag: + break + return lines diff --git a/Check.py b/Check.py index b97006e..722c977 100644 --- a/Check.py +++ b/Check.py @@ -32,9 +32,9 @@ class Check(object): def __init__(self, name): """ """ - if name not in SpecCheck.check_names_seen: - SpecCheck.checks.append(self) - SpecCheck.check_names_seen.append(name) + if name not in Check.check_names_seen: + Check.checks.append(self) + Check.check_names_seen.append(name) self.name = name def check(self, **details): diff --git a/GachApplication.py b/GachApplication.py index fbde2d2..fd5459d 100644 --- a/GachApplication.py +++ b/GachApplication.py @@ -126,7 +126,11 @@ class GachApplication: if file.endswith(".rpm"): reporter.printDividerInfo("Koji build rpm %s output" %(file)) runRpmlint(reporter, Pkg(file)) - + + # Prints rpmlint report + # Need to check later why this is needed + print reporter.getReport() + # Load all src rpm checks checks reviewer = GachReviewer() srpmreviewer = GachSrcRPMReviewer() @@ -135,6 +139,11 @@ class GachApplication: if file.endswith("src.rpm"): srpmreviewer.review(file) + # Load all build log checks + for file in downloaded_files: + if file.endswith("build.log"): + reviewer.review(log=file) + print reporter.getReport() rm_rf(srpm_spec_downdir) diff --git a/GachReporter.py b/GachReporter.py index 0a5f162..51c06c8 100644 --- a/GachReporter.py +++ b/GachReporter.py @@ -144,5 +144,10 @@ class GachReporter(object): """ self.report += "\n" + "Rpmlint Messages:" + "\n" + messages + "\n" + def printLogWarnings(self, messages): + """ + Adds log messages + """ + self.report += "Log warnings:" + messages # Abstract reporter interface ends here diff --git a/GachReviewer.py b/GachReviewer.py index cbc9451..f6beaca 100644 --- a/GachReviewer.py +++ b/GachReviewer.py @@ -55,7 +55,7 @@ class GachReviewer(object): if guidelines: self.default_guidelines = guidelines else: - self.default_guidelines = ("SampleCheck",) + self.default_guidelines = ("SampleCheck", "LogWarningCheck") # Load all checks. self.loadChecks() @@ -66,6 +66,7 @@ class GachReviewer(object): """ print "Guidelines:", self.default_guidelines for name in self.default_guidelines: + print "Loaded: ", name file = None path = None description = None @@ -86,7 +87,10 @@ class GachReviewer(object): obects. """ for guide in Check.Check.checks: - guide.check(details) + if "log" in details: + guide.check(log=details["log"]) + else: + guide.check(details) return diff --git a/LogWarningCheck.py b/LogWarningCheck.py new file mode 100644 index 0000000..e49435e --- /dev/null +++ b/LogWarningCheck.py @@ -0,0 +1,56 @@ +#!/usr/bin/env python +# +# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with translate; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +# +# Check: Print warnings from log +# + +import Check +from BuildLog import BuildLog +from GachReporter import GachReporter + +reporter = GachReporter() + +class LogWarningCheck(Check.Check): + """Print log warnings + """ + + def __init__(self): + """ + """ + Check.Check.__init__(self, "LogWarningCheck") + + def check(self, **details): + """ + """ + log_file = details["log"] + log = BuildLog(log_file) + reporter.printDividerInfo("Log info %s:" %(log_file)) + + # Grep all lines + lines = log.searchstring(["warning", "Warning", "WARNING"]) + + for line in lines: + reporter.printLogWarnings(line) + + return + +check = LogWarningCheck() +# Ends here + diff --git a/SampleCheck.py b/SampleCheck.py index ddf0f61..36fbb2b 100644 --- a/SampleCheck.py +++ b/SampleCheck.py @@ -30,9 +30,9 @@ class SampleCheck(SpecCheck.SpecCheck): """Sample constructor""" SpecCheck.SpecCheck.__init__(self, "SampleCheck") - def check(self, spec): + def check(self, **details): """Sample Check""" - reporter.printInfo(spec, "sample-testing") + reporter.printInfo("sample-testing") # An object to autoregister the check check=SampleCheck()

1 0

regarding checks
by Rakesh Pandit 18 Nov '09

18 Nov '09

In case you see any guideline which can be automated and is not already done in rpmlint .. please file a ticket here. I will see where it should go later. Right now working on https://fedorahosted.org/gach/ticket/6 -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first

1 0

regarding BuildRootCheck
by Rakesh Pandit 17 Nov '09

17 Nov '09

I will keep BuildRootCheck (submitted by oget) and twist it a bit as it is still useful for EPEL branches. -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first

1 0

few fixes along with binary file check
by Rakesh Pandit 17 Nov '09

17 Nov '09

Few fixes which I missed committing with Binary file check -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first commit 7f9b708e1e7e3de11b29ac048cfdef40dd1d666f Author: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Tue Nov 17 14:20:41 2009 +0530 GachReporter.py (locate, istextfile, istext): New functions. GachUtils.py (_print): fixed details key value printing. diff --git a/GachReporter.py b/GachReporter.py index 090dd6c..0a5f162 100644 --- a/GachReporter.py +++ b/GachReporter.py @@ -86,7 +86,7 @@ class GachReporter(object): self.report += "%s: %s" % (type, reason) for d in details: - self.report += " %s" %d + self.report += " %s: %s" %(d, details[d]) self.report +="\n" diff --git a/GachUtils.py b/GachUtils.py index 768b675..b93dad8 100644 --- a/GachUtils.py +++ b/GachUtils.py @@ -21,11 +21,18 @@ # import os +import fnmatch +import glob import sys +import string import commands import tempfile +text_characters = "".join(map(chr, range(32, 127)) + list("\n\r\t\b")) +_null_trans = string.maketrans("", "") + + def rm_rf(d): """Remove directory recursively """ @@ -64,3 +71,30 @@ def get_source_from_srpm(srpm_file, output_dir): print "rpmbuild failed to build source from srpm" sys.exit(1) return + + +def locate(pattern, root=os.curdir): + """Locate all files matching supplied filename pattern in and below + supplied root directory. + """ + matches = [] + for dir, subdir, files in os.walk(root): + for file in files: + if glob.fnmatch.fnmatch(file, pattern): + matches.append(file) + + return matches + + +def istextfile(filename, blocksize = 512): + return istext(open(filename).read(blocksize)) + + +def istext(s): + if "\0" in s: + return 0 + if not s: + return 1 + t = s.translate(_null_trans, text_characters) + return float(len(t))/len(s) >= 0.30 +

1 0

Check: Whether source contains binary files
by Rakesh Pandit 17 Nov '09

17 Nov '09

Pushed initial prototype check for following: +# Check: Whether source contains binary files. +# URL: https://fedoraproject.org/wiki/Packaging/Guidelines#No_inclusion_of_pre-bui… -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first commit 37aafbe670cd4fec768b7982ce216bfa37e35873 Author: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Tue Nov 17 14:16:47 2009 +0530 BinaryFilesCheck.py: New Check diff --git a/BinaryFilesCheck.py b/BinaryFilesCheck.py new file mode 100644 index 0000000..892b01d --- /dev/null +++ b/BinaryFilesCheck.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python +# +# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with translate; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + + +# +# Check: Whether source contains binary files. +# URL: https://fedoraproject.org/wiki/Packaging/Guidelines#No_inclusion_of_pre-bui… +# + +import commands +import tempfile +import os +import glob +import stat + +import SrcRPMCheck +from GachUtils import rm_rf, get_source_from_srpm, locate +from GachReporter import GachReporter +from GachUtils import istextfile + +reporter = GachReporter() + +# *.class, *.dll, *.DS_Store, *.exe, *.jar, *.o, *.pyc, *.pyo, *.so +list_of_binary_extensions = ['*.class', '*.dll', '*.DS_Store', '*.exe', '*.jar', '*.o', '*.pyc', '*.pyo', '*.so'] + +class BinaryFilesCheck(SrcRPMCheck.SrcRPMCheck): + """Binary files present in source files + """ + + def __init__(self): + """ + """ + SrcRPMCheck.SrcRPMCheck.__init__(self, "BinaryFilesCheck") + + def check(self, **details): + """ + """ + srpm_file = details["srpm"] + srpm_extract_dir = tempfile.mkdtemp(prefix="gach") + build_dir = srpm_extract_dir + "/BUILD/" + get_source_from_srpm(srpm_file, srpm_extract_dir) + + binary_files_with_extension = {} + for extension in list_of_binary_extensions: + matches = locate(extension, root=build_dir) + if matches: + binary_files_with_extension[extension] = matches + + for extension in binary_files_with_extension: + for file in binary_files_with_extension[extension]: + reporter.printWarning("binary-extension-file-in-source", File=file) + + binary_files_nontext_chars = [] + empty_files = [] + for dir, subdir, files in os.walk(build_dir): + for file in files: + fullpath = os.path.join(dir, file) + # Knock out empty files + if os.stat(fullpath)[stat.ST_SIZE] == 0: + empty_files.append(fullpath) + continue + if istextfile(fullpath): + binary_files_nontext_chars.append(fullpath) + + if binary_files_nontext_chars: + for binary in binary_files_nontext_chars: + reporter.printWarning("binary-file-with-30%-plus-non-text-chars", File=binary) + + if empty_files: + for empty_file in empty_files: + reporter.printWarning("empty-file", File=empty_file) + + rm_rf(srpm_extract_dir) + return + +check = BinaryFilesCheck() + +reporter.addDetails( +'binary-extension-file-in-source', +'''Files with *.class, *.dll, *.DS_Store, *.exe, *.jar, *.o, *.pyc, *.pyo, *.so extensions are most probably binary files.''', +'binary-file-with-30%-plus-non-text-chars', +'''Binary file with 30% plus no text characters has been considered as binary file''', +'empty-file', +'''Empty file''' +)

1 0

fixing structure to make src checks work
by Rakesh Pandit 16 Nov '09

16 Nov '09

These fixes I made while writing binary file checks over packages. -- Rakesh Pandit https://fedoraproject.org/ freedom, friends, features, first >From c26b5a7ce9c12c6ed4f9785f5dfb8b7389b891e2 Mon Sep 17 00:00:00 2001 From: Rakesh Pandit <rakesh(a)fedoraproject.org> Date: Tue, 17 Nov 2009 02:11:20 +0530 Subject: [PATCH] GachApplication, GachReviewer.py, GachSrcRPMReviewer.py: Fixed for writing src checks. GachUtils.py: New file. --- GachApplication.py | 41 +++++++++++++++++++----------- GachReviewer.py | 18 +++++++++--- GachSrcRPMReviewer.py | 9 +++--- GachUtils.py | 66 +++++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 110 insertions(+), 24 deletions(-) create mode 100644 GachUtils.py diff --git a/GachApplication.py b/GachApplication.py index 5e6fcc3..fbde2d2 100644 --- a/GachApplication.py +++ b/GachApplication.py @@ -25,6 +25,9 @@ import tempfile from Spec import Spec from Pkg import Pkg +from RPMCheck import RPMCheck +from SrcRPMCheck import SrcRPMCheck + from GachBugzillaFeeder import GachBugzillaFeeder from GachRPMReviewer import GachRPMReviewer from GachSpecReviewer import GachSpecReviewer @@ -32,16 +35,9 @@ from GachSrcRPMReviewer import GachSrcRPMReviewer from GachBugzillaReporter import GachBugzillaReporter from GachKojiBuilder import GachKojiBuilder from GachReporter import GachReporter -from GachReviewer import runRpmlint - - -def rm_rf(d): - for path in (os.path.join(d,f) for f in os.listdir(d)): - if os.path.isdir(path): - rm_rf(path) - else: - os.unlink(path) - os.rmdir(d) +from GachReviewer import runRpmlint, GachReviewer +from GachSrcRPMReviewer import GachSrcRPMReviewer +from GachUtils import rm_rf class NoBugId(Exception): @@ -107,24 +103,39 @@ class GachApplication: print "Downloaded files are at: " for file in file_path: print file - + reporter = GachReporter() spec = Spec(file_path[0]) pkg = Pkg(file_path[1]) # TODO Test expecution and building via builders - reporter.printDividerInfo("Spec file rpmlint output") + reporter.printDividerInfo("Spec file rpmlint output (one from bug) ") runRpmlint(reporter, spec) - reporter.printDividerInfo("Pkg rpmlint output") + reporter.printDividerInfo("Pkg rpmlint output (one from bug) ") runRpmlint(reporter, pkg) builder = GachKojiBuilder() - builder.build(file_path[1], 'dist-f12', wait=True) + + builder.build(file_path[1], 'dist-f13', wait=True) downloaded_files = builder.retrieve(koji_downdir) - print reporter.getReport() + reporter.printDividerInfo("Lint output on koji build rpms") + + for file in downloaded_files: + if file.endswith(".rpm"): + reporter.printDividerInfo("Koji build rpm %s output" %(file)) + runRpmlint(reporter, Pkg(file)) + + # Load all src rpm checks checks + reviewer = GachReviewer() + srpmreviewer = GachSrcRPMReviewer() + + for file in downloaded_files: + if file.endswith("src.rpm"): + srpmreviewer.review(file) + print reporter.getReport() rm_rf(srpm_spec_downdir) return diff --git a/GachReviewer.py b/GachReviewer.py index f79ca81..cbc9451 100644 --- a/GachReviewer.py +++ b/GachReviewer.py @@ -47,24 +47,32 @@ class GachReviewer(object): derived class. """ - def __init__(self): + def __init__(self, guidelines=None): """ """ # Need to update it for each guideline. - self.default_guidelines = ("SampleCheck") + self.default_guidelines = None + if guidelines: + self.default_guidelines = guidelines + else: + self.default_guidelines = ("SampleCheck",) # Load all checks. - self.__loadChecks() + self.loadChecks() - def __loadChecks(self): + def loadChecks(self): """ Implements the loading module. """ + print "Guidelines:", self.default_guidelines for name in self.default_guidelines: + file = None + path = None + description = None try: # Later on we need to generalize the second parameter # depending on where guideline files go after installation. - (file, path, description) = imp.find_module(name, ".") + (file, path, description) = imp.find_module(name, ["."]) except ImportError: print "Guideline import error: ", name sys.exit(1) diff --git a/GachSrcRPMReviewer.py b/GachSrcRPMReviewer.py index 9e91ad1..3f00ae8 100644 --- a/GachSrcRPMReviewer.py +++ b/GachSrcRPMReviewer.py @@ -28,15 +28,16 @@ class GachSrcRPMReviewer(GachReviewer.GachReviewer): def __init__(self): """ """ - pass + # All Checks are loaded here. + self.default_guidelines = ("BinaryFilesCheck",) + GachReviewer.GachReviewer.__init__(self, guidelines=self.default_guidelines) def review(self, srpm): """ Implements reviewer for srpm. """ - for guide in SrcRPMCheck.SrcRPMCheck.checks: - guide.check(srpm) + for guide in SrcRPMCheck.SrcRPMCheck.srpm_checks: + guide.check(srpm=srpm) return - # GachSrcRPMReviewer ends here diff --git a/GachUtils.py b/GachUtils.py new file mode 100644 index 0000000..768b675 --- /dev/null +++ b/GachUtils.py @@ -0,0 +1,66 @@ +#!/usr/bin/env python +# +# Copyright 2009 Rakesh Pandit <rakesh(a)fedoraproject.org> +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with translate; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA + +# +# Misc functions go in here. +# + +import os +import sys +import commands +import tempfile + + +def rm_rf(d): + """Remove directory recursively + """ + for path in (os.path.join(d,f) for f in os.listdir(d)): + if os.path.isdir(path): + rm_rf(path) + else: + os.unlink(path) + os.rmdir(d) + + +def extract_rpm(file): + """Extract rpm into output_dir + """ + curdir = os.curdir + srpm_extract_dir = tempfile.mkdtemp(prefix="gach") + os.chdir(srpm_extract_dir) + status = commands.getstatusoutput("rpm2cpio " + file + "| cpio -idmv") + os.chdir(curdir) + return srpm_extract_dir + + +def get_source_from_srpm(srpm_file, output_dir): + """build source from srpm file and return path to build directory + """ + if commands.getstatusoutput("which rpmbuild")[0] != 0: + print "rpmbuild command not present!" + sys.exit(1) + + name = commands.getoutput("rpm -qp --qf \"%{NAME}\" " + srpm_file) + if commands.getstatusoutput("rpm -i --define \"_topdir " + output_dir + "\" " + srpm_file)[0] != 0: + print "Installation of srpm file failed." + sys.exit(1) + + if commands.getstatusoutput("rpmbuild -bp --quiet --define \"_topdir " + output_dir + "\" " + output_dir + "/SPECS/" + name + ".spec")[0] != 0: + print "rpmbuild failed to build source from srpm" + sys.exit(1) + return -- 1.6.2.5

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

gach-dev November 2009