python-devel November 2011

python-devel@lists.fedoraproject.org

2 participants
3 discussions

Re: More Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein 27 Nov '11

27 Nov '11

I developed a fix that can byte compile only the .py files in site-packages. It involved creating my own version of brp-python-bytecompile, preventing brp-python-bytecompile from running, and running my-python-bytecompile instead. The relevant statements in the spec file are as follows: Before the description insert (from the Fedora packaging python page): # Turn off the brp-python-bytecompile script %global __os_install_post %(echo '%{__os_install_post}' | sed -e 's!/usr/lib[^[:space:]]*/brp-python-bytecompile[[:space:]].*$!!g') At the end of the %install (following copying of the doc directories into the buildroot) insert: ~/my-rpm-macros/my-python-bytecompile The code for my-python-bytecompile is simply a copy of brp-python-bytecompile with the final section commented out. That section does files in directories other than site-packages. The code is below. Stan Klein ---------------------------------------------------------------------- my-python-bytecompile #!/bin/bash errors_terminate=$2 # If using normal root, avoid changing anything. if [ -z "$RPM_BUILD_ROOT" -o "$RPM_BUILD_ROOT" = "/" ]; then exit 0 fi # If we don't have a python interpreter, avoid changing anything. default_python=${1:-/usr/bin/python} if [ ! -x "$default_python" ]; then exit 0 fi # Figure out how deep we need to descend. We could pick an insanely high # number and hope it's enough, but somewhere, somebody's sure to run into it. depth=`(find $RPM_BUILD_ROOT -type f -name "*.py" -print0 ; echo /) | \ xargs -0 -n 1 dirname | sed 's,[^/],,g' | sort -u | tail -n 1 | wc -c` if [ -z "$depth" -o "$depth" -le "1" ]; then exit 0 fi # .pyc/.pyo files embed a "magic" value, identifying the ABI version of Python # bytecode that they are for. # # The files below RPM_BUILD_ROOT could be targetting multiple versions of # python (e.g. a single build that emits several subpackages e.g. a # python26-foo subpackage, a python31-foo subpackage etc) # # Support this by assuming that below each /usr/lib/python$VERSION/, all # .pyc/.pyo files are to be compiled for /usr/bin/python$VERSION. # # For example, below /usr/lib/python2.6/, we're targetting /usr/bin/python2.6 # and below /usr/lib/python3.1/, we're targetting /usr/bin/python3.1 shopt -s nullglob for python_libdir in $RPM_BUILD_ROOT/usr/lib{,64}/python[0-9].[0-9]/ ; do python_binary=/usr/bin/$(basename $python_libdir) real_libdir=${python_libdir/$RPM_BUILD_ROOT/} echo "Bytecompiling .py files below $python_libdir using $python_binary" # Generate normal (.pyc) byte-compiled files. $python_binary -c 'import compileall, sys; sys.exit(not compileall.compile_dir("'"$python_libdir"'", '"$depth"', "'"$real_libdir"'", force=1, quiet=1))' if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then # One or more of the files had a syntax error exit 1 fi # Generate optimized (.pyo) byte-compiled files. $python_binary -O -c 'import compileall, sys; sys.exit(not compileall.compile_dir("'"$python_libdir"'", '"$depth"', "'"$real_libdir"'", force=1, quiet=1))' if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then # One or more of the files had a syntax error exit 1 fi done # Handle other locations in the filesystem using the default python # implementation: # Generate normal (.pyc) byte-compiled files. #$default_python -c 'import compileall, re, sys; sys.exit (not compileall.compile_dir("'"$RPM_BUILD_ROOT"'", '"$depth"', "/", 1, re.compile(r"'"/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]"'"), quiet=1))' #if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then # # One or more of the files had a syntax error # exit 1 #fi # # Generate optimized (.pyo) byte-compiled files. #$default_python -O -c 'import compileall, re, sys; sys.exit(not compileall.compile_dir("'"$RPM_BUILD_ROOT"'", '"$depth"', "/", 1, re.compile(r"'"/bin/|/sbin/|/usr/lib(64)?/python[0-9]\.[0-9]"'"), quiet=1))' > /dev/null #if [ $? -ne 0 -a 0$errors_terminate -ne 0 ]; then # # One or more of the files had a syntax error # exit 1 #fi exit 0

1 0

More Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein 27 Nov '11

27 Nov '11

One additional issue: The system produces pyo and pyc files for all the .py files it finds. That is good for the files that go into site-packages because they are intended to be executed from there, but might not be so good for documentation files such as examples and code-snippets that are intended to be run or otherwise used in user-space. The commands: find . -type f -name *.pyc -exec rm -f {} \; find . -type f -name *.pyo -exec rm -f {} \; executed at some point in the process at the root of the default documentation directory after the .pyc and .pyo files have been created can remove them. However, I can't seem to figure out where to put the statements. Also, might there be a way to prevent the byte compiling of documentation files? Stan Klein

1 0

Re: [Distutils] Compatibility of bdist_rpm with Fedora packaging instructions
by Stanley A. Klein 26 Nov '11

26 Nov '11

Tarek - Thanks. I got it working and want to document some of my findings. I'm cc:'ing the Fedora Python list so they can take these issues into account in their Python packaging instructions. My findings are as follows: 1. The source code management system was git. I needed to install setuptools-git to get files recognized that were being maintained under git. 2. I also needed to establish a MANIFEST.in file to ensure all relevant files were included. 3. The setup.cfg statement under [bdist_rpm] of "doc_files =" doesn't work if there are directories involved. This is a documented "gotcha" in http://fedoraproject.org/wiki/How_to_create_an_RPM_package They also advise avoiding use of INSTALLED_FILES. Here is what I used in the spec file (it had to be edited for the directories to be included in docs): %install python setup.py install --root=$RPM_BUILD_ROOT cd $RPM_BUILD_ROOT mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/ mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/docs mkdir -p %{buildroot}%{_defaultdocdir}/%{name}-%{version}/examples cd %{_builddir}/%{name}-%{version} cp -p *.txt %{buildroot}%{_defaultdocdir}/%{name}-%{version}/ cp -rp docs/ %{buildroot}%{_defaultdocdir}/%{name}-%{version}/docs cp -rp examples/ %{buildroot}%{_defaultdocdir}/%{name}-%{version}/examples %clean rm -rf $RPM_BUILD_ROOT %files %defattr(-,root,root) %{_defaultdocdir}/%{name}-%{version}/ %{python_sitelib}/%{name}/ %{python_sitelib}/%{name}-%{version}-py%{python_version}.egg-info/ 4. The overall approach I used was to run $ python setup.py --command-packages=pypi2rpm.command bdist_rpm2 --spec-only That brought in items such as the description specified in setup.py. I then moved the spec file up a directory level from dist, edited in the above parts relevant to the doc files, and finally ran $ python setup.py --command-packages=pypi2rpm.command bdist_rpm2 > --spec-file=SPECFILE I also found it very useful to actually run python setup.py --command-packages=pypi2rpm.command bdist_rpm2 --spec-file SPECFILE >buildout 2>&1 Recording the build output enabled me to find details of errors that occurred during the build. There were a number of those, and the output flies by too quickly to notice them for diagnosis of problems that need to be fixed. (an approach I've also used in the past is to tee the output, so I can see if the build was successful when it completes). Also, the -k option came in useful in figuring out the documentation-related fixes to the spec file, because it helped seeing what was included in the build and what was available in the build but not getting copied to the install. 5. I inquired about how the project produced its sdist files for pypi, and was told that they had so many problems related to their switch to git that they simply did a gzipped tar of their repository. 6. The project used a different extension on their README file from the expected values of README and README.txt. I needed to change that to README.txt to get it to process properly. Again, thanks. Stan Klein On Fri, November 11, 2011 6:00 am, Tarek Ziad? <ziade.tarek(a)gmail.com> wrote: > > Message: 3 > Date: Thu, 10 Nov 2011 18:42:08 +0100 > From: Tarek Ziad? <ziade.tarek(a)gmail.com> > To: "Stanley A. Klein" <sklein(a)cpcug.org> > Cc: distutils-sig(a)python.org > Subject: Re: [Distutils] Compatibility of bdist_rpm with Fedora > packaging instructions > > On Thu, Nov 10, 2011 at 6:11 PM, Stanley A. Klein <sklein(a)cpcug.org> > wrote: >> Tarek - >> >> I downloaded pypi2rpm and built it using bdist_rpm. > > you built it ? pypi2rpm is a script and a bidist_rpm2 command, no need > to build, just install > > >> ?The instructions on how to use it are rather thin. > > Yes there's no doc, as its mostly use in our own build tools for now. > >> My best guess is to run it in spec-only >> mode, modify the spec, and then run it with the modified spec specified. >> I tried to get a help listing but that was also think and I couldn't >> quite >> figure out how to run it. >> >> Could you provide further information on how to use it. >> >> Thanks. > > Once it's installed you can build a rpm in your project, using: > > $ python setup.py --command-packages=pypi2rpm.command bdist_rpm2 > --spec-file=SPECFILE > > (there are other options you can find with --help) > > >> >> Stan Klein >> >> >> >> On Thu, November 10, 2011 5:06 am, Tarek Ziad? wrote: >>> On Thu, Nov 10, 2011 at 10:04 AM, Paul Nasrat <pnasrat(a)gmail.com> >>> wrote: >>>> I don't think bdist_rpm should track vendor packaging requirements, >>>> purely as those recommendations may change faster than the release >>>> process of distutils. I also believe bdist_rpm may be going away in >>>> the future: >>> >>> Yes I confirm this. We removed it in packaging because we believe it >>> should be maintained by the RPM communities -- with their own release >>> cycles etc. >>> >>> FWIW I have a custom version in the pypi2rpm project where I just feed >>> a .spec file to the bdist_rpm command, so I can do proper RHEL or >>> Fedora packaging. >>> >>>> For Fedora have you considered rpmdev-newspec which can creates a >>>> templated python spec file for your packages. >>>> >>>> http://fedoraproject.org/wiki/How_to_create_an_RPM_package >>>> >>>> Paul >>>> >>>> On 8 November 2011 21:40, Stanley A. Klein <sklein(a)cpcug.org> wrote: >>>>> I will need to build some Python packages for Fedora and Centos. ?The >>>>> spec >>>>> file produced by bdist_rpm automatically includes the statement >>>>> %files -f INSTALLED_FILES >>>>> >>>>> The Fedora Python packaging instruction includes a recommendation to >>>>> avoid >>>>> use of INSTALLED_FILES and provides some alternatives. ?That is the >>>>> first >>>>> incompatibility I've encountered, but there may be more. >>>>> >>>>> The bdist_rpm code probably should be changed to enable >>>>> compatibility. >>>>> Meanwhile, is there a workaround? >>>>> >>>>> >>>>> Stan Klein

2 2

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

python-devel November 2011