Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
Parag.
Parag N(पराग़) píše v Pá 14. 12. 2012 v 21:14 +0530:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#...
Dan
On Fri, Dec 14, 2012 at 04:48:48PM +0100, Dan Horák wrote:
Parag N(पराग़) píše v Pá 14. 12. 2012 v 21:14 +0530:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
https://fedoraproject.org/wiki/Packaging:Guidelines?rd=Packaging/Guidelines#...
Random thoughts on parallel builds:
In cases where parallel builds fail, I try to fix the problem, but sometimes this is not possible. I try to add a comment about this in the spec file, and I think that's a good idea for everyone to do that. Can we amend the above guidelines to say so?
Also, I have MAKEFLAGS=-j9 set on my development machine. This is passed through to rpmbuild, and it does break 'fedpkg local' on some packages. One way to avoid this is to unset MAKEFLAGS in %build, but better ways are:
(1) Fix the build so it works in parallel.
(2) If (1) is not possible, push a patch upstream to add
.NOTPARALLEL:
to the troublesome Makefile (can also be added to Makefile.am etc).
We use .NOTPARALLEL in a couple of libguestfs subdirectories to cope with certain hard-to-fix cases, or where we haven't had the time yet to work out precisely what dependencies are missing. [The vast majority of libguestfs builds in parallel]. Using .NOTPARALLEL is ideal for upstream if they can't / don't want to fix parallel builds, and want to ensure that even if MAKEFLAGS is set it won't break things.
Finally, weird build systems. A few OCaml packages have this, where it's just not obvious *how* one could do a parallel build, particularly where it's not even using 'make'. (eg. anything using ocamlbuild, I really wish ocamlbuild would die)
Rich.
On Fri, Dec 14, 2012 at 4:44 PM, Parag N(पराग़) panemade@gmail.com wrote:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
If this is the case we should work with upstream to fix the build system, as we do with code.
Hi, On Sat, Dec 15, 2012 at 12:51 AM, Gianluca Sforna giallu@gmail.com wrote:
On Fri, Dec 14, 2012 at 4:44 PM, Parag N(पराग़) panemade@gmail.com wrote:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
If this is the case we should work with upstream to fix the build system, as we do with code.
I actually want to work on it as upstream is unresponsive since a year. That is why I need help here. Is there any document that gives examples on how can one fix broken Makefile?
Thanks, Parag.
On 12/15/2012 06:19 AM, Parag N(पराग़) wrote:
Hi, On Sat, Dec 15, 2012 at 12:51 AM, Gianluca Sforna giallu@gmail.com wrote:
On Fri, Dec 14, 2012 at 4:44 PM, Parag N(पराग़) panemade@gmail.com wrote:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
If this is the case we should work with upstream to fix the build system, as we do with code.
I actually want to work on it as upstream is unresponsive since a year. That is why I need help here. Is there any document that gives examples on how can one fix broken Makefile?
In general, broken parallel makes mean "missing make dependencies". How and where to fix these, depends a lot on a package's implementation details - There simply is no "golden rule"/"simple howto".
That said, which package are you referring to? May-be somebody else can spot something obvious.
Ralf
Hi, On Sat, Dec 15, 2012 at 11:25 AM, Ralf Corsepius rc040203@freenet.de wrote:
On 12/15/2012 06:19 AM, Parag N(पराग़) wrote:
Hi, On Sat, Dec 15, 2012 at 12:51 AM, Gianluca Sforna giallu@gmail.com wrote:
On Fri, Dec 14, 2012 at 4:44 PM, Parag N(पराग़) panemade@gmail.com wrote:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
If this is the case we should work with upstream to fix the build system, as we do with code.
I actually want to work on it as upstream is unresponsive since a year. That is why I need help here. Is there any document that gives examples on how can one fix broken Makefile?
In general, broken parallel makes mean "missing make dependencies". How and where to fix these, depends a lot on a package's implementation details - There simply is no "golden rule"/"simple howto".
That said, which package are you referring to? May-be somebody else can spot something obvious.
Here is the whole story: I am working on w3m package cleanup. Its old package and upstream is non-responsive on w3m bug tracker. Parallel make was previously commented in spec since long time by me only I guess. Now, last week while working on package cleanup, I found parallel make is working fine. I prepared new update for w3m. Committed in master. Did F19 scratch build, it worked. Did F19 official build, it failed. Did F18 scratch build, it worked. Did successful official build for F18. Attempted again official F19 build, it failed. Did scratch F19 build, it worked. I then third time attempted w3m build for F19 and it worked.
I then come to know that if I comment parallel make then build always succeeds. Now, Can someone tell what used to go wrong when build failed in F19 and how now official build succeed with parallel make enabled?
Regards, Parag.
On Dec 15, 2012 2:07 AM, "Parag N(पराग़)" panemade@gmail.com wrote:
Hi, On Sat, Dec 15, 2012 at 11:25 AM, Ralf Corsepius rc040203@freenet.de
wrote:
On 12/15/2012 06:19 AM, Parag N(पराग़) wrote:
Hi, On Sat, Dec 15, 2012 at 12:51 AM, Gianluca Sforna giallu@gmail.com wrote:
On Fri, Dec 14, 2012 at 4:44 PM, Parag N(पराग़) panemade@gmail.com wrote:
Hi, I want to know is there any documentation available on parallel make? If a package is not building with parallel make then can we fix Makefile?
If this is the case we should work with upstream to fix the build system, as we do with code.
I actually want to work on it as upstream is unresponsive since a year. That is why I need help here. Is there any document that gives examples on how can one fix broken Makefile?
In general, broken parallel makes mean "missing make dependencies". How
and
where to fix these, depends a lot on a package's implementation details
-
There simply is no "golden rule"/"simple howto".
That said, which package are you referring to? May-be somebody else can
spot
something obvious.
Here is the whole story: I am working on w3m package cleanup. Its old package and upstream is non-responsive on w3m bug tracker. Parallel make was previously commented in spec since long time by me only I guess. Now, last week while working on package cleanup, I found parallel make is working fine. I prepared new update for w3m. Committed in master. Did F19 scratch build, it worked. Did F19 official build, it failed. Did F18 scratch build, it worked. Did successful official build for F18. Attempted again official F19 build, it failed. Did scratch F19 build, it worked. I then third time attempted w3m build for F19 and it worked.
I then come to know that if I comment parallel make then build always succeeds. Now, Can someone tell what used to go wrong when build failed in F19 and how now official build succeed with parallel make enabled?
The reason the parallel makes fail is effectively a race condition between dependent targets. In Makefile where there are missing dependencies between targets, a parallel make won't ALWAYS fail. If the parallel make regularly fails some percentage of attempts, then it's broken. The fact that it worked by chance a few times doesn't change that.
A simple approach to fixing is to look at:
1) what part of the build fails when there is a failure. 2) what was missing (a .o file, a library, etc.) that made it fail.
Then determine how to introduce a dependency in the makefile between what was missing and what needed it. It will probably be an iterative process.
HTH,
Andy
Regards, Parag. -- packaging mailing list packaging@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/packaging
Hi, I also see that "fedpkg local" on F18 never used to fail for me but koji builds fails sometimes. Is there any way to reproduce the parallel make failure in local build?
Regards, Parag
On 12/17/2012 05:21 AM, Parag N(पराग़) wrote:
Hi, I also see that "fedpkg local" on F18 never used to fail for me but koji builds fails sometimes. Is there any way to reproduce the parallel make failure in local build?
The koji builders are very highly parallelized, which increases the chances of hitting race conditions. You could improve your own chances in a similar way, but no guarantees; edit ~/.rpmmacros to include a line:
%_smp_mflags -j16
Paul.
packaging@lists.fedoraproject.org