On Sat, 2007-01-13 at 19:42 +0200, Ahmed Kamal wrote:
not sure I understand your question, but they way things should go is, the client downloads the drpm, client side code combines client side files from older rpm + the drpm into a new rpm. Then that new rpm is installed. The required bandwidth should drop to 20%, the numbers need some testing ofcourse, but I can imagine such savings.
SuSE did a lot of work on distributing the files as less than a whole rpm. Their original tool works as you say, with an "rpm patch file" that combines with an rpm on the client's system which is then installed. The newer tool that they were pushing a couple years ago was also able to upgrade files on the filesystem rather than going through the intermediate step of creating an rpm. I consider this method to be less desirable from a security standpoint than the first.
-Toshio
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm + (either ondisk files, or old rpm). Either way, a new rpm is created, then installed. Never does it replace files directly. Not sure why this would be bad security wise
Hey, why do emails appear to come from the sender, not the FI list. I always hit reply, and end up replying to the original author only! :)
On 1/13/07, Toshio Kuratomi a.badger@gmail.com wrote:
On Sat, 2007-01-13 at 19:42 +0200, Ahmed Kamal wrote:
not sure I understand your question, but they way things should go is, the client downloads the drpm, client side code combines client side files from older rpm + the drpm into a new rpm. Then that new rpm is installed. The required bandwidth should drop to 20%, the numbers need some testing ofcourse, but I can imagine such savings.
SuSE did a lot of work on distributing the files as less than a whole rpm. Their original tool works as you say, with an "rpm patch file" that combines with an rpm on the client's system which is then installed. The newer tool that they were pushing a couple years ago was also able to upgrade files on the filesystem rather than going through the intermediate step of creating an rpm. I consider this method to be less desirable from a security standpoint than the first.
-Toshio
On Sat, 13 Jan 2007 20:40:38 +0200, Ahmed Kamal wrote:
Hey, why do emails appear to come from the sender, not the FI list. I always hit reply, and end up replying to the original author only! :)
Because ReplyTo is not set, and with Google Mail you then need to click "Reply to all".
On Saturday 13 January 2007 12:40 pm, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm + (either ondisk files, or old rpm). Either way, a new rpm is created, then installed. Never does it replace files directly. Not sure why this would be bad security wise
I personally don't like the idea of binary delta's there are too many variables with them and too much overhead. for instance say we update cups 4 times during the life of a release. that means we need to create 4 delta's as the end user can have 4 possible states of the package.
Windows has always done delta's and for the longest time they only provided delta's from the latest version. which is the reason you had to update a ton of times to get to the latest version. that has changed but its not http://www.directionsonmicrosoft.com/sample/DOMIS/update/2005/02feb/0205fsut...
Apple also provides delta's but they do only deltas from the previous version to latest if you you have an older version you have to get the full version.
most packages are so small that i don't think the overhead is worth the pain. OOo and a couple of others i could see maybe, but otherwise I personally don't think its a good idea. It means mirrors need to carry more data.
I don't like tracking different update states too, which is why I was suggesting previous-to-latest only drpms. Guess like what apple does.
It's not easy for me to determine whether users will like such a feature. Most casual linux users I meet, are not keen on updating their systems. And when they finally decide to, they don't want to download lots of megabytes.
One more scenario I could think of, is users in developing countries like mine, where broadband is rare. Deltas make a lot of sense on a modem (tell me about it a few years ago). Anyway, if most of you don't think this is worth the effort, let me know about it.
Any idea if OLPC has implemented implemented something similar ?
On 1/13/07, Dennis Gilmore dennis@ausil.us wrote:
On Saturday 13 January 2007 12:40 pm, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package
that
is made by suse. This suse package can create new rpms from drpm +
(either
ondisk files, or old rpm). Either way, a new rpm is created, then installed. Never does it replace files directly. Not sure why this would
be
bad security wise
I personally don't like the idea of binary delta's there are too many variables with them and too much overhead. for instance say we update cups 4 times during the life of a release. that means we need to create 4 delta's as the end user can have 4 possible states of the package.
Windows has always done delta's and for the longest time they only provided delta's from the latest version. which is the reason you had to update a ton of times to get to the latest version. that has changed but its not
http://www.directionsonmicrosoft.com/sample/DOMIS/update/2005/02feb/0205fsut...
Apple also provides delta's but they do only deltas from the previous version to latest if you you have an older version you have to get the full version.
most packages are so small that i don't think the overhead is worth the pain. OOo and a couple of others i could see maybe, but otherwise I personally don't think its a good idea. It means mirrors need to carry more data. -- Dennis Gilmore, RHCE Proud Australian
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Saturday 13 January 2007 17:03, Ahmed Kamal wrote:
Any idea if OLPC has implemented implemented something similar ?
OLPC doesn't use rpm as a package management system. It is used somewhat to create the deployment images, but after that it is treated as firmware almost for the XOs. When new "firmwares" are available, they will update over a mesh with other users, almost like a virus or worm.
Dennis Gilmore wrote:
On Saturday 13 January 2007 12:40 pm, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm + (either ondisk files, or old rpm). Either way, a new rpm is created, then installed. Never does it replace files directly. Not sure why this would be bad security wise
I personally don't like the idea of binary delta's there are too many variables with them and too much overhead. for instance say we update cups 4 times during the life of a release. that means we need to create 4 delta's as the end user can have 4 possible states of the package.
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
most packages are so small that i don't think the overhead is worth the pain. OOo and a couple of others i could see maybe, but otherwise I personally don't think its a good idea. It means mirrors need to carry more data.
Then provide deltas only for the biggest things that would benefit the most from delta patching. All else download the original RPM.
Some algorithm could attempt to make deltas, and offer deltas where the download savings are something like > 80% and > 2MB.
Warren Togami wtogami@redhat.com
On 1/17/07, Warren Togami wtogami@redhat.com wrote:
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
-Mike
On Wed, 2007-01-17 at 09:29 -0600, Mike McGrath wrote:
On 1/17/07, Warren Togami wtogami@redhat.com wrote:
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it with repoquery. Just compare the changes in 'updates-released' from one day to the next.
Keep in mind the package doesn't always go up. Sometimes old pkgs get removed. So it could remain the same number but new pkgs are released.
-sv
seth vidal (skvidal@linux.duke.edu) said:
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it with repoquery. Just compare the changes in 'updates-released' from one day to the next.
Couldn't you also just poll the RSS and parse that?
Bill
On Wed, 2007-01-17 at 10:34 -0500, Bill Nottingham wrote:
seth vidal (skvidal@linux.duke.edu) said:
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it with repoquery. Just compare the changes in 'updates-released' from one day to the next.
Couldn't you also just poll the RSS and parse that?
so you're parsing the parsing of the parsed metadata?
aaaaaahh :)
-sv
On 1/17/07, seth vidal skvidal@linux.duke.edu wrote:
On Wed, 2007-01-17 at 09:29 -0600, Mike McGrath wrote:
On 1/17/07, Warren Togami wtogami@redhat.com wrote:
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it with repoquery. Just compare the changes in 'updates-released' from one day to the next.
Keep in mind the package doesn't always go up. Sometimes old pkgs get removed. So it could remain the same number but new pkgs are released.
-sv
i think a lot of this has been discussed before in Fedora. Anyone know of any threads we can point Ahmed at? Looks like there's a lot of work to be done :-D
-Mike
On Wed, 2007-01-17 at 09:34 -0600, Mike McGrath wrote:
On 1/17/07, seth vidal skvidal@linux.duke.edu wrote:
On Wed, 2007-01-17 at 09:29 -0600, Mike McGrath wrote:
On 1/17/07, Warren Togami wtogami@redhat.com wrote:
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
Does anyone track how many updates are released / day? I should start tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it with repoquery. Just compare the changes in 'updates-released' from one day to the next.
Keep in mind the package doesn't always go up. Sometimes old pkgs get removed. So it could remain the same number but new pkgs are released.
-sv
i think a lot of this has been discussed before in Fedora. Anyone know of any threads we can point Ahmed at? Looks like there's a lot of work to be done :-D
The first thread I'm aware of was in 1998 and involved xdelta'ing the rpm's. Google can't locate it so I'm betting it was on contrib-list (whose archives have disappeared from redhat.com)
Some of the more recent discussions:: http://www.redhat.com/archives/fedora-devel-list/2005-December/msg00404.html http://www.redhat.com/archives/fedora-devel-list/2005-June/msg00018.html
... and google knows of others as well. Try:: site:redhat.com delta rpm
-Toshio
Appreciating everyone's help, it seems others have attempted this before. Anyway, let's put this through the test of time :) Also, I totally agree with keeping drpms only if they meet certain criteria, i.e. provide >50% savings or similar.
Right now, I am trying to figure out how/where the server side will store the drpm metadata. Other.xml.gz seems like a good place, or maybe a new drpm.xml.gz, but I am not sure how such file should be written. Should I just write code that will generate drpms, and that xml metadata file too ? Must the xml file be written according to a specific form, I only need to attach a hash to each drpm, which the clients will use to know whether using the drpm will be successful.
On 1/18/07, Toshio Kuratomi a.badger@gmail.com wrote:
On Wed, 2007-01-17 at 09:34 -0600, Mike McGrath wrote:
On 1/17/07, seth vidal skvidal@linux.duke.edu wrote:
On Wed, 2007-01-17 at 09:29 -0600, Mike McGrath wrote:
On 1/17/07, Warren Togami wtogami@redhat.com wrote:
Then limit the delta to the most common update paths. If the
desired
delta doesn't exist when the user tries, it can fall back to
download
the full RPM. No big loss.
Does anyone track how many updates are released / day? I should
start
tracking that, I bet its significant.
If you're willing to work with per-day granularity - you can do it
with
repoquery. Just compare the changes in 'updates-released' from one day to the next.
Keep in mind the package doesn't always go up. Sometimes old pkgs get removed. So it could remain the same number but new pkgs are released.
-sv
i think a lot of this has been discussed before in Fedora. Anyone know of any threads we can point Ahmed at? Looks like there's a lot of work to be done :-D
The first thread I'm aware of was in 1998 and involved xdelta'ing the rpm's. Google can't locate it so I'm betting it was on contrib-list (whose archives have disappeared from redhat.com)
Some of the more recent discussions::
http://www.redhat.com/archives/fedora-devel-list/2005-December/msg00404.html http://www.redhat.com/archives/fedora-devel-list/2005-June/msg00018.html
... and google knows of others as well. Try:: site:redhat.com delta rpm
-Toshio
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Thu, 2007-01-18 at 09:53 +0200, Ahmed Kamal wrote:
Appreciating everyone's help, it seems others have attempted this before. Anyway, let's put this through the test of time :) Also, I totally agree with keeping drpms only if they meet certain criteria, i.e. provide >50% savings or similar.
Right now, I am trying to figure out how/where the server side will store the drpm metadata. Other.xml.gz seems like a good place, or maybe a new drpm.xml.gz, but I am not sure how such file should be written.
I seem to recall that primary.xml.gz can list arbitrary xml files which the depsolvers will ignore if they don't need them but Seth would know better.
Should I just write code that will generate drpms, and that xml metadata file too ?
That would probably work but you might want to break the steps into two parts. Just remember to think about how the drpms are getting to the repo. Are you going to do the updates on a separate machine and then push them over? Are you going to push the new drpms onto the server, regenerate the metadata, then push the new metadata over?
Must the xml file be written according to a specific form, I only need to attach a hash to each drpm, which the clients will use to know whether using the drpm will be successful.
Are you writing this from scratch? If it's using the yum plugin that already exists it might be expecting the metadata in a specific format already.
-Toshio
On Thu, 2007-01-18 at 09:39 -0800, Toshio Kuratomi wrote:
On Thu, 2007-01-18 at 09:53 +0200, Ahmed Kamal wrote:
Appreciating everyone's help, it seems others have attempted this before. Anyway, let's put this through the test of time :) Also, I totally agree with keeping drpms only if they meet certain criteria, i.e. provide >50% savings or similar.
Right now, I am trying to figure out how/where the server side will store the drpm metadata. Other.xml.gz seems like a good place, or maybe a new drpm.xml.gz, but I am not sure how such file should be written.
I seem to recall that primary.xml.gz can list arbitrary xml files which the depsolvers will ignore if they don't need them but Seth would know better.
repomd.xml can list additional files, not primary - but the result is the same.
You'd want to tie knowledge of drpms into createrepo.
Are you writing this from scratch? If it's using the yum plugin that already exists it might be expecting the metadata in a specific format already.
plugins can do whatever, pretty much, when it comes to what kind of metadata they want to deal with.
-sv
I've been hacking on the code yesterday, and it seems I am hitting some sort of a bug. Basically, what happens is that the plugin starts, does its work, exits, yum tries installing the new rpm, it goes through the "updating" progress bar, then hangs before doing "cleanup" part?!
The thing is, after the hang, all rpm based tools "rpm -q" "rpm -e" "rpm -i" anything, would just sit there stuck! Stracing this, and it is stopped at futex(0xb7988c3c, FUTEX_WAIT, 1, NULL <unfinished ...> I did find some bug reports about that https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=145021 http://lists.centos.org/pipermail/centos-devel/2005-June/001890.html
But it should be rare! In my case, this happens everytime. And I have to reboot to clear the lock! I'm probably not releasing the rpmDB lock somehow, anyone faced something similar before ? This is "yum -d 10" output
Running Transaction Test Member: ImageMagick.i386 0-6.2.8.0-3.fc6.1 - u Adding Package ImageMagick - 6.2.8.0-3.fc6.1.i386 in mode u Finished Transaction Test Transaction Test Succeeded Member: ImageMagick.i386 0-6.2.8.0-3.fc6.1 - u Adding Package ImageMagick - 6.2.8.0-3.fc6.1.i386 in mode u Running Transaction Updating : ImageMagick ######################### [1/2]
On 1/18/07, seth vidal skvidal@linux.duke.edu wrote:
On Thu, 2007-01-18 at 09:39 -0800, Toshio Kuratomi wrote:
On Thu, 2007-01-18 at 09:53 +0200, Ahmed Kamal wrote:
Appreciating everyone's help, it seems others have attempted this before. Anyway, let's put this through the test of time :) Also, I totally agree with keeping drpms only if they meet certain criteria, i.e. provide >50% savings or similar.
Right now, I am trying to figure out how/where the server side will store the drpm metadata. Other.xml.gz seems like a good place, or maybe a new drpm.xml.gz, but I am not sure how such file should be written.
I seem to recall that primary.xml.gz can list arbitrary xml files which the depsolvers will ignore if they don't need them but Seth would know better.
repomd.xml can list additional files, not primary - but the result is the same.
You'd want to tie knowledge of drpms into createrepo.
Are you writing this from scratch? If it's using the yum plugin that already exists it might be expecting the metadata in a specific format already.
plugins can do whatever, pretty much, when it comes to what kind of metadata they want to deal with.
-sv
Ahmed Kamal wrote:
I've been hacking on the code yesterday, and it seems I am hitting some sort of a bug.
...
But it should be rare! In my case, this happens everytime. And I have to reboot to clear the lock!
rm -f /var/lib/rpm/__db.* doesn't work?
-- Rex
that does help, I can clear the lock without rebooting anymore! but, the rpm hanging still happens every time. Not sure what's causing the issue though.
On 1/24/07, Rex Dieter rdieter@math.unl.edu wrote:
Ahmed Kamal wrote:
I've been hacking on the code yesterday, and it seems I am hitting some sort of a bug.
...
But it should be rare! In my case, this happens everytime. And I have to reboot to clear the lock!
rm -f /var/lib/rpm/__db.* doesn't work?
-- Rex
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Wed, 2007-01-24 at 23:58 +0200, Ahmed Kamal wrote:
that does help, I can clear the lock without rebooting anymore! but, the rpm hanging still happens every time. Not sure what's causing the issue though.
Do your scripts leave the constructed rpms around somewhere? You might want to rpm -K <RPM> to see if the checksums match. Then rpm -qpi <RPM> to make sure it was signed with the correct key.
-Toshio
RPM's DB backend gets wedged sometimes. It's become a lot less common than it used to be, but it just happened to me again recently. A solution (maybe not the best) is to rebuild the database. I have a script that does the following: $ cat fixrpmdb.sh #!/bin/sh rm -f /var/lib/rpm/__db* rpm -vv --rebuilddb
The '-vv' makes it take longer, but I like it so that I can be sure it's doing something.
Hope that helps, Matt
Ahmed Kamal wrote:
that does help, I can clear the lock without rebooting anymore! but, the rpm hanging still happens every time. Not sure what's causing the issue though.
On 1/24/07, * Rex Dieter* <rdieter@math.unl.edu mailto:rdieter@math.unl.edu> wrote:
Ahmed Kamal wrote: > I've been hacking on the code yesterday, and it seems I am hitting > some sort of a bug. ... > But it should be rare! In my case, this happens everytime. And I have > to reboot to clear the lock! rm -f /var/lib/rpm/__db.* doesn't work? -- Rex _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com <mailto:Fedora-infrastructure-list@redhat.com> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Thanks guys, I finally came to the conclusion that the problem was not the python code, but rather the applydeltarpm command that was touching the DB in some wrong way causing it to lockup. I confirmed this by writing a trivial yum plugin that only ran the command using direct on disk files, and yum still hung! BTW, yes, the constructed rpm does pass all signature checks. I'm currently investigating what's causing the issue inside applydeltarpm.
On 1/25/07, Matthew Gillen matt@gillens.us wrote:
RPM's DB backend gets wedged sometimes. It's become a lot less common than it used to be, but it just happened to me again recently. A solution (maybe not the best) is to rebuild the database. I have a script that does the following: $ cat fixrpmdb.sh #!/bin/sh rm -f /var/lib/rpm/__db* rpm -vv --rebuilddb
The '-vv' makes it take longer, but I like it so that I can be sure it's doing something.
Hope that helps, Matt
Ahmed Kamal wrote:
that does help, I can clear the lock without rebooting anymore! but, the rpm hanging still happens every time. Not sure what's causing the issue though.
On 1/24/07, * Rex Dieter* <rdieter@math.unl.edu mailto:rdieter@math.unl.edu> wrote:
Ahmed Kamal wrote: > I've been hacking on the code yesterday, and it seems I am hitting > some sort of a bug. ... > But it should be rare! In my case, this happens everytime. And Ihave
> to reboot to clear the lock! rm -f /var/lib/rpm/__db.* doesn't work? -- Rex _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com <mailto:Fedora-infrastructure-list@redhat.com> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
That rpm DB lock up issue has been resolved with help from the deltarpm package developer. Basically, the code was not freeing the DB transaction. Suse was not facing that issue, as they do not use BDB locking mechanisms. Now, that this issue was fixed. I've completed a basic prototype of the system. Currently, the yum plugin will detect the user is upgrading a rpm, will download the delta rpm, construct the new rpm, and leave that for yum to install. On the server side of things (code to generate drpms), I found it very helpful to integrate the drpm generation code into RepoPrune script, as they both share of a lot of functionality. RepoPrune depends on some variables available on fedora servers, which is why I have made a stripped down version so I could use that script on my laptop, but ideally, it should be integrated.
One tiny thing I am facing is the naming convention to be used for the resulting drpms. This name needs to reflect both versions for which this delta was made. I am thinking I need to use something like: "name-VerNew-RelNew-VerOld-RelOld.arch.drpm" Not sure if this would cause any issues, I mean not using the standard rpm naming scheme. But then again, this is not a rpm, which is why I chose suffix .drpm instead of say .delta.rpm Let me know what you guys think
On 1/25/07, Ahmed Kamal email.ahmedkamal@googlemail.com wrote:
Thanks guys, I finally came to the conclusion that the problem was not the python code, but rather the applydeltarpm command that was touching the DB in some wrong way causing it to lockup. I confirmed this by writing a trivial yum plugin that only ran the command using direct on disk files, and yum still hung! BTW, yes, the constructed rpm does pass all signature checks. I'm currently investigating what's causing the issue inside applydeltarpm.
On 1/25/07, Matthew Gillen matt@gillens.us wrote:
RPM's DB backend gets wedged sometimes. It's become a lot less common than it used to be, but it just happened to me again recently. A solution (maybe not the best) is to rebuild the database. I have a script that does the
following: $ cat fixrpmdb.sh #!/bin/sh rm -f /var/lib/rpm/__db* rpm -vv --rebuilddb
The '-vv' makes it take longer, but I like it so that I can be sure it's doing something.
Hope that helps, Matt
Ahmed Kamal wrote:
that does help, I can clear the lock without rebooting anymore! but,
the
rpm hanging still happens every time. Not sure what's causing the
issue
though.
On 1/24/07, * Rex Dieter* <rdieter@math.unl.edu mailto:rdieter@math.unl.edu> wrote:
Ahmed Kamal wrote: > I've been hacking on the code yesterday, and it seems I amhitting
> some sort of a bug. ... > But it should be rare! In my case, this happens everytime. And Ihave
> to reboot to clear the lock! rm -f /var/lib/rpm/__db.* doesn't work? -- Rex _______________________________________________ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com <mailto: Fedora-infrastructure-list@redhat.com> https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
On Fri, 2007-02-02 at 19:32 +0200, Ahmed Kamal wrote:
One tiny thing I am facing is the naming convention to be used for the resulting drpms. This name needs to reflect both versions for which this delta was made. I am thinking I need to use something like: " name-VerNew-RelNew-VerOld-RelOld.arch.drpm" Not sure if this would cause any issues, I mean not using the standard rpm naming scheme. But then again, this is not a rpm, which is why I chose suffix .drpm instead of say .delta.rpm Let me know what you guys think
What does SuSE use when they generate delata-rpms? (I do like drpm better than delta.rpm).
-Toshio
AFAIK, suse does use .delta.rpm as can be found here http://www.filewatcher.com/b/ftp/ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1...
Tried to guess their naming convention, it seems something like "newVer_oldVer-release", not sure why there is a single release, and it's not even consistent. I'm trying to lookup some info on their naming convention.
On 2/3/07, Toshio Kuratomi a.badger@gmail.com wrote:
On Fri, 2007-02-02 at 19:32 +0200, Ahmed Kamal wrote:
One tiny thing I am facing is the naming convention to be used for the resulting drpms. This name needs to reflect both versions for which this delta was made. I am thinking I need to use something like: " name-VerNew-RelNew-VerOld-RelOld.arch.drpm" Not sure if this would cause any issues, I mean not using the standard rpm naming scheme. But then again, this is not a rpm, which is why I chose suffix .drpm instead of say .delta.rpm Let me know what you guys think
What does SuSE use when they generate delata-rpms? (I do like drpm better than delta.rpm).
-Toshio
It seems suse uses "_" as a separator, and will mention the version once if it is the same across the 2 rpms. For example kernel-xenpae-2.6.16.13_2.6.16.21-4_0.21.i586.delta.rpmftp://ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1/rpm/i586/kernel-xenpae-2.6.16.13_2.6.16.21-4_0.21.i586.delta.rpm That would be name-verOld_verNew-relOld_relNew.arch.delta.rpm and libextractor-0.5.10-12_12.2.i586.delta.rpmftp://ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1/rpm/i586/libextractor-0.5.10-12_12.2.i586.delta.rpmshows using a single version number
On 2/3/07, Ahmed Kamal email.ahmedkamal@googlemail.com wrote:
AFAIK, suse does use .delta.rpm as can be found here http://www.filewatcher.com/b/ftp/ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1...
Tried to guess their naming convention, it seems something like "newVer_oldVer-release", not sure why there is a single release, and it's not even consistent. I'm trying to lookup some info on their naming convention.
On 2/3/07, Toshio Kuratomi a.badger@gmail.com wrote:
On Fri, 2007-02-02 at 19:32 +0200, Ahmed Kamal wrote:
One tiny thing I am facing is the naming convention to be used for the resulting drpms. This name needs to reflect both versions for which this delta was made. I am thinking I need to use something like: " name-VerNew-RelNew-VerOld-RelOld.arch.drpm" Not sure if this would cause any issues, I mean not using the standard rpm naming scheme. But then again, this is not a rpm, which is why I chose suffix .drpm instead of say .delta.rpm Let me know what you guys think
What does SuSE use when they generate delata-rpms? (I do like drpm better than delta.rpm).
-Toshio
Hi, The delta rpm support now looks pretty functional (at least for me). The "presto" project is now up at https://hosted.fedoraproject.org/projects/presto Feel free to grab the code and test.
The most important missing piece of the puzzle is more testing! I would like any mirror of "updates" for FC6 to host delta rpms, so that people can start testing, reporting bugs, and we can work to fix them. Unfortunately, I don't have the means to host such a mirror! I mailed the mirror admins list, but no volunteers till now. If anyone is interested in helping with this, please contact me. The disk space required shouldn't be too much, because we could mirror just the FC6 release (only i386 as well for now), also the bandwidth required would be minimal (am I gonna get that much testers :)
cheers
On 2/3/07, Ahmed Kamal email.ahmedkamal@googlemail.com wrote:
It seems suse uses "_" as a separator, and will mention the version once if it is the same across the 2 rpms. For example kernel-xenpae-2.6.16.13_2.6.16.21-4_0.21.i586.delta.rpmftp://ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1/rpm/i586/kernel-xenpae-2.6.16.13_2.6.16.21-4_0.21.i586.delta.rpm That would be name-verOld_verNew-relOld_relNew.arch.delta.rpm and libextractor-0.5.10-12_12.2.i586.delta.rpmftp://ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1/rpm/i586/libextractor-0.5.10-12_12.2.i586.delta.rpmshows using a single version number
On 2/3/07, Ahmed Kamal < email.ahmedkamal@googlemail.com> wrote:
AFAIK, suse does use .delta.rpm as can be found here http://www.filewatcher.com/b/ftp/ftp.cs.pu.edu.tw/pub2/SuSE/suse/update/10.1...
Tried to guess their naming convention, it seems something like "newVer_oldVer-release", not sure why there is a single release, and it's not even consistent. I'm trying to lookup some info on their naming convention.
On 2/3/07, Toshio Kuratomi < a.badger@gmail.com> wrote:
On Fri, 2007-02-02 at 19:32 +0200, Ahmed Kamal wrote:
One tiny thing I am facing is the naming convention to be used for
the
resulting drpms. This name needs to reflect both versions for which this delta was made. I am thinking I need to use something like: " name-VerNew-RelNew-VerOld-RelOld.arch.drpm" Not sure if this would cause any issues, I mean not using the
standard
rpm naming scheme. But then again, this is not a rpm, which is why I
chose suffix .drpm instead of say .delta.rpm Let me know what you guys think
What does SuSE use when they generate delata-rpms? (I do like drpm better than delta.rpm).
-Toshio
On Wednesday 17 January 2007 09:26, Warren Togami wrote:
Dennis Gilmore wrote:
On Saturday 13 January 2007 12:40 pm, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm + (either ondisk files, or old rpm). Either way, a new rpm is created, then installed. Never does it replace files directly. Not sure why this would be bad security wise
I personally don't like the idea of binary delta's there are too many variables with them and too much overhead. for instance say we update cups 4 times during the life of a release. that means we need to create 4 delta's as the end user can have 4 possible states of the package.
Then limit the delta to the most common update paths. If the desired delta doesn't exist when the user tries, it can fall back to download the full RPM. No big loss.
i could see for OOo the need and maybe firefox but that is really about it. X in the old days but no now. If the user is not really wanting to update because of to many updates chances are that they dont have the latest versions installed. Im not saying that we should not do it. Just that personally im not a fan, and I dont think the gains will be worth the pain.
Please prove me wrong if i am.
On Sat, 2007-01-13 at 20:40 +0200, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm
- (either ondisk files, or old rpm). Either way, a new rpm is created,
then installed. Never does it replace files directly. Not sure why this would be bad security wise
It's the construction of the rpm from ondisk files that I don't like. You lose the ability to sign the rpm that you're installing. Patching an older rpm is a safer transformation.
I just googled and found an August post to yum-devel about what I think is this plugin:
https://lists.dulug.duke.edu/pipermail/yum-devel/2006-August/002385.html
Is this right? Is there more recent code? It looks like that code is tied into up2date so it wouldn't help Fedora users much. It also needs a server side which implies the mirrors would have to run additional software to make it functional....
-Toshio
The most recent code is at: http://people.redhat.com/~herrmann/deltarpm/
We should be able to sign the drpms (not sure yet!) Reconstructing the new rpm from ondisk files, doesn't look bad security wise, because the new data is signed. If the on disk files are not trusted, this means the system is already compromised! It's not really the drpm's fault.
Yes the old code is focused on up2date. It will take some cleaning. And I am all for dropping the server side code completely. drpms are to be generated by a cron-job and *not* on the fly as is the current implementation. So, the servers should not be running additional software.
On 1/13/07, Toshio Kuratomi a.badger@gmail.com wrote:
On Sat, 2007-01-13 at 20:40 +0200, Ahmed Kamal wrote:
FYI, this yum deltarpm support, is based on that same deltarpm package that is made by suse. This suse package can create new rpms from drpm
- (either ondisk files, or old rpm). Either way, a new rpm is created,
then installed. Never does it replace files directly. Not sure why this would be bad security wise
It's the construction of the rpm from ondisk files that I don't like. You lose the ability to sign the rpm that you're installing. Patching an older rpm is a safer transformation.
I just googled and found an August post to yum-devel about what I think is this plugin:
https://lists.dulug.duke.edu/pipermail/yum-devel/2006-August/002385.html
Is this right? Is there more recent code? It looks like that code is tied into up2date so it wouldn't help Fedora users much. It also needs a server side which implies the mirrors would have to run additional software to make it functional....
-Toshio
On Jan 13, 2007, at 5:11 PM, Ahmed Kamal wrote:
We should be able to sign the drpms (not sure yet!) Reconstructing the new rpm from ondisk files, doesn't look bad security wise, because the new data is signed. If the on disk files are not trusted, this means the system is already compromised!
Installed files get modified for reasons other than a hacked system. Think about config files that the sysadmin edits after a package is installed. Think about documentation files, whose may not be installed at all. Think about dealing with file conflicts between installed packages. Run 'rpm -Va' on a sample of Fedora systems and tell me that all those changes just don't matter... And make sure to talk to a sysadmin who has had to recover from a rootkit-ed system, and tell them that the rootkit'd files will get rolled into their newly installed packages if drpm is enabled during recovery.
Relying on the integrity of installed files when generating and applying rpm diffs is just a bad idea, period. It's a hack that relies on hope instead of best practices, and it gives up the guarantees that are a substantial part of rpm's value. Any rpm delta solution must produce results that are identical to the original desired file, down to the last byte.
Maybe there is a clever way to use a network server and local installed files, along with the rsync algorithm, to generate a .rpm file that is guaranteed to be byte-for-byte identical to the desired file. Mix BitTorrent technology in there, and there is plenty of room for innovation without resorting to a really bad hack. :)
Best, -- Elliot
Thanks for the reply. The points raised are important and must be validated, before we implement such a system. I will now try to debunk the FUD ;)
basically, drpm does construct a byte-for-byte rpm that is equivalent to the new rpm, then it installs that. If constructing such rpm fails for whatever reason, the drpm operation is aborted, and the system falls back to full rpms. One more interesting point to note, is that checking whether constructing a new rpm will be successful, is done prior to downloading the drpm, so bandwidth usage is kept minimal.
While researching the topic, I found the mandriva folks were having similar discussions. http://qa.mandriva.com/show_bug.cgi?id=24535 Comment #13 is from the deltarpm author
I also stumbled upon a forum poll, about adding deltarpm support for 2007 in Mandriva http://forum.club.mandriva.com/viewtopic.php?t=52030 http://forum.club.mandriva.com/viewtopic.php?t=52029 Surprisingly most users do think it's nice feature to have. Although not too many people voted, but that's what we have.
BTW, using bittorrent does not seem like a good idea for yum: http://wiki.linux.duke.edu/YumTodont
On 1/14/07, Elliot Lee sopwith@gmail.com wrote:
On Jan 13, 2007, at 5:11 PM, Ahmed Kamal wrote:
We should be able to sign the drpms (not sure yet!) Reconstructing the new rpm from ondisk files, doesn't look bad security wise, because the new data is signed. If the on disk files are not trusted, this means the system is already compromised!
Installed files get modified for reasons other than a hacked system. Think about config files that the sysadmin edits after a package is installed. Think about documentation files, whose may not be installed at all. Think about dealing with file conflicts between installed packages. Run 'rpm -Va' on a sample of Fedora systems and tell me that all those changes just don't matter... And make sure to talk to a sysadmin who has had to recover from a rootkit-ed system, and tell them that the rootkit'd files will get rolled into their newly installed packages if drpm is enabled during recovery.
Relying on the integrity of installed files when generating and applying rpm diffs is just a bad idea, period. It's a hack that relies on hope instead of best practices, and it gives up the guarantees that are a substantial part of rpm's value. Any rpm delta solution must produce results that are identical to the original desired file, down to the last byte.
Maybe there is a clever way to use a network server and local installed files, along with the rsync algorithm, to generate a .rpm file that is guaranteed to be byte-for-byte identical to the desired file. Mix BitTorrent technology in there, and there is plenty of room for innovation without resorting to a really bad hack. :)
Best, -- Elliot
Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
One thing I forgot to mention, is that this will be implemented as a yum plugin. So, it will not affect anyone who doesn't really want to use it.
On 1/15/07, Elliot Lee sopwith@gmail.com wrote:
On Jan 14, 2007, at 3:43 PM, Ahmed Kamal wrote:
basically, drpm does construct a byte-for-byte rpm that is equivalent to the new rpm, then it installs that.
Cool beans - that's all I wanted to know :)
Happy hacking, -- Elliot
infrastructure@lists.fedoraproject.org
-
Ahmed Kamal -
Bill Nottingham -
Dennis Gilmore -
Elliot Lee -
Jesse Keating -
Matthew Gillen -
Michael Schwendt -
Mike McGrath -
Rex Dieter -
seth vidal -
Toshio Kuratomi -
Warren Togami