--- This will become necessary if we want to upload the results of a mash task, as I am currently working on.
Should I have created a new uploadDirRecursively() method, instead of changing uploadDir()?
builder/kojid | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-)
diff --git a/builder/kojid b/builder/kojid index ca73cad..fbd4e73 100755 --- a/builder/kojid +++ b/builder/kojid @@ -371,15 +371,34 @@ class BuildRoot(object): append '.' + suffix to the filenames, so that successive uploads of the same directory won't overwrite each other, if the files have the same name but different contents.""" + def walker(arg, dirname, filenames): + # `arg` is completely ignored here + if dirname == dirpath: + uploadpath = rootuploadpath + + else: + dirrelpath = os.path.relpath(dirname, start=dirpath) + uploadpath = os.path.join(rootuploadpath, dirrelpath) + + for filename in filenames: + filepath = os.path.join(dirname, filename) + + if os.path.isdir(filepath): + # TODO: Should we do something about this? + continue + + if os.stat(filepath).st_size > 0: + if suffix: + filename = '%s.%s' % (filename, suffix) + + self.session.uploadWrapper(filepath, uploadpath, filename) + if not os.path.isdir(dirpath): return - uploadpath = self.getUploadPath() - for filename in os.listdir(dirpath): - filepath = os.path.join(dirpath, filename) - if os.stat(filepath).st_size > 0: - if suffix: - filename = '%s.%s' % (filename, suffix) - self.session.uploadWrapper(filepath, uploadpath, filename) + + rootuploadpath = self.getUploadPath() + + os.path.walk(dirpath, walker, None)
def init(self): rv = self.mock(['--init'])
On 10/02/2013 04:00 AM, Mathieu Bridon wrote:
This will become necessary if we want to upload the results of a mash task, as I am currently working on.
Should I have created a new uploadDirRecursively() method, instead of changing uploadDir()?
It should probably be an optional flag rather than a change to default behavior.
It might be simpler if you used os.walk instead of os.path.walk.
I don't think you need to do anything in the os.path.isdir case, unless you intend to prune the walk.
builder/kojid | 33 ++++++++++++++++++++++++++------- 1 file changed, 26 insertions(+), 7 deletions(-)
diff --git a/builder/kojid b/builder/kojid index ca73cad..fbd4e73 100755 --- a/builder/kojid +++ b/builder/kojid @@ -371,15 +371,34 @@ class BuildRoot(object): append '.' + suffix to the filenames, so that successive uploads of the same directory won't overwrite each other, if the files have the same name but different contents."""
def walker(arg, dirname, filenames):# `arg` is completely ignored hereif dirname == dirpath:uploadpath = rootuploadpathelse:dirrelpath = os.path.relpath(dirname, start=dirpath)uploadpath = os.path.join(rootuploadpath, dirrelpath)for filename in filenames:filepath = os.path.join(dirname, filename)if os.path.isdir(filepath):# TODO: Should we do something about this?continueif os.stat(filepath).st_size > 0:if suffix:filename = '%s.%s' % (filename, suffix)self.session.uploadWrapper(filepath, uploadpath, filename)if not os.path.isdir(dirpath): return
uploadpath = self.getUploadPath()for filename in os.listdir(dirpath):filepath = os.path.join(dirpath, filename)if os.stat(filepath).st_size > 0:if suffix:filename = '%s.%s' % (filename, suffix)self.session.uploadWrapper(filepath, uploadpath, filename)
rootuploadpath = self.getUploadPath()os.path.walk(dirpath, walker, None) def init(self): rv = self.mock(['--init'])
On Wed, 2013-10-02 at 19:16 -0400, Mike McLean wrote:
On 10/02/2013 04:00 AM, Mathieu Bridon wrote:
This will become necessary if we want to upload the results of a mash task, as I am currently working on.
Should I have created a new uploadDirRecursively() method, instead of changing uploadDir()?
It should probably be an optional flag rather than a change to default behavior.
I wonder, uploadDir seems to not be used anywhere else?
It might be simpler if you used os.walk instead of os.path.walk.
Slightly, but not that much.
However, I just realized that os.path.walk is considered deprecated (in favour of os.walk) and has been removed from Python 3.
So yes, I'll replace that in the next version of the patch.
I don't think you need to do anything in the os.path.isdir case, unless you intend to prune the walk.
Hah, that TODO is a leftover from when I was writing the code, I forgot to remove it. Sorry about that.
On 10/03/2013 01:29 AM, Mathieu Bridon wrote:
I wonder, uploadDir seems to not be used anywhere else?
Heh, indeed you are right. It predates the public release of Koji. Once upon a time it was used in init() and build(), but that use was dropped when we added real-time log uploads.
Still, the name "uploadDir" doesn't really indicate recursive behavior, so I'm still inclined to think it should be an option.
Hi,
Sorry for the long time without any news, I've been pretty busy at $dayjob.
On Thu, 2013-10-10 at 18:13 -0400, Mike McLean wrote:
On 10/03/2013 01:29 AM, Mathieu Bridon wrote:
I wonder, uploadDir seems to not be used anywhere else?
Heh, indeed you are right. It predates the public release of Koji. Once upon a time it was used in init() and build(), but that use was dropped when we added real-time log uploads.
Still, the name "uploadDir" doesn't really indicate recursive behavior, so I'm still inclined to think it should be an option.
No problem, I could do that.
But before I send a new patch, I was wondering if that was still useful.
I started this because it was needed to run Mash in Koji, as the assumption was that the mash results would need to be uploaded back to the hub.
But that proved too costly, and now Dennis told me that the way to go would be a plugin, and mounting /mnt/koji rw.
So, running Mash in Koji won't need the recursive upload feature any more.
Is that still a desired feature anyway?
If it still is, I'll send a patch soon. :)
On 11/18/2013 12:07 AM, Mathieu Bridon wrote:
On Thu, 2013-10-10 at 18:13 -0400, Mike McLean wrote:
Still, the name "uploadDir" doesn't really indicate recursive behavior, so I'm still inclined to think it should be an option.
No problem, I could do that.
But before I send a new patch, I was wondering if that was still useful.
I started this because it was needed to run Mash in Koji, as the assumption was that the mash results would need to be uploaded back to the hub.
But that proved too costly, and now Dennis told me that the way to go would be a plugin, and mounting /mnt/koji rw.
A few recommendations: Be exceptionally careful with that mount. Be extremely diligent about unmounting it as well (a try..finally is in order here). Only mount the smallest necessary subtree rw.
So, running Mash in Koji won't need the recursive upload feature any more.
Is that still a desired feature anyway?
If it still is, I'll send a patch soon. :)
I guess not. We don't even use the current version of it.
buildsys@lists.fedoraproject.org
-
Mathieu Bridon -
Mike McLean