I'm attaching a patch which uses DBManager for interacting with the user database rather than always relying upon sqlite. This way, if you use pgsql or mysql for your database back-end, user information will be stored there as well.
The problem is, this will screw up current installations which are using pgsql/mysql because the old code assumes sqlite is used for user info, so suddenly your user database will be empty... I guess the easiest way around this is to write a quick script to pull everything out of the sqlite database and put it into mysql/pgsql. Also, I'll look at updating the user-manager.py script. In fact, I guess it may be easiest to add export/import functions to that for migrating the user info.
Let me know what you think.
-Jeff
On Oct 26, 2005, at 2:48 PM, Jeff Sheltren wrote:
I'm attaching a patch which uses DBManager for interacting with the user database rather than always relying upon sqlite. This way, if you use pgsql or mysql for your database back-end, user information will be stored there as well.
Whoops, I forgot to include the diff for DBManager.py in that file. I'm attaching an updated patch which also changes DBManager.py so that the users table will be created as needed.
-Jeff
Here is a patch for user-manager.py to use DBManager for its database access. This allows user-manager.py to work with multiple db types.
-Jeff
On 10/28/05, Jeff Sheltren sheltren@cs.ucsb.edu wrote:
Here is a patch for user-manager.py to use DBManager for its database access. This allows user-manager.py to work with multiple db types.
I haven't had a chance to take a look at the patch yet, but does it provide for a way to keep the userdb & jobdb separate? I'm thinking (and might be wrong here <grin>) that if it's easy to do, it might be useful to allow for, e.g, jobdb in mysql but userdb in sqlite, especially in terms of upgrades. Maybe a flag, e.g. "job_db_engine" or somesuch in the server config, that if left out, defaults to the overall db being used.
-Chris
On Oct 28, 2005, at 4:41 PM, Chris Weyl wrote:
On 10/28/05, Jeff Sheltren sheltren@cs.ucsb.edu wrote:
Here is a patch for user-manager.py to use DBManager for its database access. This allows user-manager.py to work with multiple db types.
I haven't had a chance to take a look at the patch yet, but does it provide for a way to keep the userdb & jobdb separate? I'm thinking (and might be wrong here <grin>) that if it's easy to do, it might be useful to allow for, e.g, jobdb in mysql but userdb in sqlite, especially in terms of upgrades. Maybe a flag, e.g. "job_db_engine" or somesuch in the server config, that if left out, defaults to the overall db being used.
-Chris
No it doesn't. The idea was that all database information should be stored in the same location. Off the top of my head, I think that using separate database types would complicate DBManager.py quite a bit. Currently it just assumes that everything is in the same DB, so there would need to be extra checks to see which DB holds which information, and then use the appropriate DB for each case. In fact, this would probably mean using two DBManager objects, one for jobs and one for users (currently only one DBManager object is used by plague-server). I don't know if that is a bad or good thing, I just think it is pretty complicated compared to the current situation.
I think the easier thing to do is to write a tool for extracting all user info from sqlite in order to import it into mysql/pgsql. I'll try to work on that this weekend.
-Jeff
On 10/28/05, Jeff Sheltren sheltren@cs.ucsb.edu wrote:
No it doesn't. The idea was that all database information should be stored in the same location. Off the top of my head, I think that using separate database types would complicate DBManager.py quite a bit. Currently it just assumes that everything is in the same DB, so there would need to be extra checks to see which DB holds which information, and then use the appropriate DB for each case. In fact, this would probably mean using two DBManager objects, one for jobs and one for users (currently only one DBManager object is used by plague-server). I don't know if that is a bad or good thing, I just think it is pretty complicated compared to the current situation.
Sounds... messy. Hm.
I think the easier thing to do is to write a tool for extracting all user info from sqlite in order to import it into mysql/pgsql. I'll try to work on that this weekend.
Gotcha. I think you're right... just trying to keep the future 0.4 -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :)
-Chris
On Oct 28, 2005, at 7:08 PM, Chris Weyl wrote:
Gotcha. I think you're right... just trying to keep the future 0.4 -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :)
-Chris
Yeah, I agree with that; I hate to break stuff with a version upgrade. But personally I think that may be the best solution instead of creating a bunch of workarounds to keep older setups working. I guess that ultimately it's up to Dan - so, what do you think? :)
-Jeff
On Fri, 2005-10-28 at 19:39 -0400, Jeff Sheltren wrote:
On Oct 28, 2005, at 7:08 PM, Chris Weyl wrote:
Gotcha. I think you're right... just trying to keep the future 0.4 -> 0.5 migration pain at a lower level than 0.3 -> 0.4. :)
-Chris
Yeah, I agree with that; I hate to break stuff with a version upgrade. But personally I think that may be the best solution instead of creating a bunch of workarounds to keep older setups working. I guess that ultimately it's up to Dan - so, what do you think? :)
So here's my take... I think unified database is fine, but we do need to provide an easy migration path IMHO. Whether that's just a one-off tool that does transfer the users for you, or whatever, I think that would save people quite a bit of pain later.
On the other hand, I don't think it would be _that_ much work to pull job and user data from different databases, but that strikes me as more clutter in the code for little obvious benefit...
So my vote is for consolidating databases for 0.5, and providing a small tool to migrate for you. Anyone want to do that tool? :)
Dan
On Nov 1, 2005, at 2:43 PM, Jeff Sheltren wrote:
On Nov 1, 2005, at 10:50 AM, Dan Williams wrote:
So my vote is for consolidating databases for 0.5, and providing a small tool to migrate for you. Anyone want to do that tool? :)
Sure, I'll work on that.
-Jeff
OK, I don't know how pretty this is, I'm new to this whole python thing :) But I'm attaching a script for migrating the user data from sqlite to pgsql/mysql. I've tested it with MySQL and it seems to work great. Let me know if you notice any bugs or something that needs to be changed. Usage is like this: # ./plague-user-migration.py /etc/plague/server/userdb /etc/plague/ server/plague-server.cfg Using database engine mysql. Added user: sheltren@host.domain.com Added user: apache@host.domain.com Done!
-Jeff
On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote:
<plague-user-migration.py>
I realized there are a couple of things that I didn't like, so I've made an updated script, located here:
http://www.cs.ucsb.edu/~jeff/plague-user-migration.py
Now it will not attempt to migrate from sqlite to sqlite :) Also, all errors are output to stderr - before some were going to stdout.
-Jeff
On Thu, 2005-11-03 at 07:30 -0400, Jeff Sheltren wrote:
On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote:
<plague-user-migration.py>
I realized there are a couple of things that I didn't like, so I've made an updated script, located here:
http://www.cs.ucsb.edu/~jeff/plague-user-migration.py
Now it will not attempt to migrate from sqlite to sqlite :) Also, all errors are output to stderr - before some were going to stdout.
Added to HEAD, thanks!
Dan
On Sun, 2005-11-20 at 13:43 -0500, Dan Williams wrote:
On Thu, 2005-11-03 at 07:30 -0400, Jeff Sheltren wrote:
On Nov 2, 2005, at 9:57 AM, Jeff Sheltren wrote:
<plague-user-migration.py>
I realized there are a couple of things that I didn't like, so I've made an updated script, located here:
http://www.cs.ucsb.edu/~jeff/plague-user-migration.py
Now it will not attempt to migrate from sqlite to sqlite :) Also, all errors are output to stderr - before some were going to stdout.
Added to HEAD, thanks!
Jeff,
Could you retest the user migration script? It turns out that postgres doesn't like double-quotes in the INSERT INTO statement. Since all the rest of the stuff in BuildMaster.py uses single quotes in the SQL statements, I switched that bit of the migration script to use single-quotes too.
Also, Postgres evidently uses "True" and "False" (probably case-insensitive) for fields of type BOOLEAN, so I added a conversion function for postgres to do that, and converted the %d format options in the INSERT INTO statement to %s instead.
These shouldn't affect MySQL at all, but I just want to be sure...
Thanks! Dan
On Nov 25, 2005, at 6:13 PM, Dan Williams wrote:
Jeff,
Could you retest the user migration script? It turns out that postgres doesn't like double-quotes in the INSERT INTO statement. Since all the rest of the stuff in BuildMaster.py uses single quotes in the SQL statements, I switched that bit of the migration script to use single-quotes too.
Also, Postgres evidently uses "True" and "False" (probably case-insensitive) for fields of type BOOLEAN, so I added a conversion function for postgres to do that, and converted the %d format options in the INSERT INTO statement to %s instead.
These shouldn't affect MySQL at all, but I just want to be sure...
Thanks! Dan
Oops, guess I should have tested it against postgres also. Your new version works fine with mysql.
-Jeff
On Sun, November 20, 2005 2:43 pm, Dan Williams said:
On Fri, 2005-10-28 at 14:34 -0400, Jeff Sheltren wrote:
Here is a patch for user-manager.py to use DBManager for its database access. This allows user-manager.py to work with multiple db types.
Committed to HEAD, thanks!
Dan
Cool, thanks for adding it!
-Jeff
On Wed, 2005-10-26 at 15:20 -0400, Jeff Sheltren wrote:
On Oct 26, 2005, at 2:48 PM, Jeff Sheltren wrote:
I'm attaching a patch which uses DBManager for interacting with the user database rather than always relying upon sqlite. This way, if you use pgsql or mysql for your database back-end, user information will be stored there as well.
Whoops, I forgot to include the diff for DBManager.py in that file. I'm attaching an updated patch which also changes DBManager.py so that the users table will be created as needed.
Committed to HEAD, thanks!
Dan
buildsys@lists.fedoraproject.org