I have been operating a game server for the better part of a year that runs a very popular mod of BattleField 1942 called Desert Combat. From the start, I have been using Fedora Core 1 on a hyper-threaded P4 2.8GHz system using the SMP kernel and have enjoyed fantastic stability and performance. Just yesterday I upgraded the system to Fedora Core 2 and have noticed something strange. All of a sudden, every F-16 fighter that spawns on any map has its right rear wheel sunk into the ground. The game files are the same exact ones from the system when it was running FC1 so that can't be the cause.
The game itself has two versions of the executable. One is statically linked and one is dynamically linked. I have tried both and the problem still occurs. This seems to indicate to me that the problem lies with operating system files that both versions require. I did a little research and ran "ldd" against both the static and dynamic versions of the executable. Here are the results:
Static build ====================== libdl.so.2 => /lib/libdl.so.2 (0x00c7e000) libm.so.6 => /lib/tls/libm.so.6 (0x00c59000) libncurses.so.5 => /usr/lib/libncurses.so.5 (0x03798000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00d71000) libc.so.6 => /lib/tls/libc.so.6 (0x00b3c000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00b1f000)
Dynamic build ====================== libdl.so.2 => /lib/libdl.so.2 (0x00c7e000) libncurses.so.5 => /usr/lib/libncurses.so.5 (0x03798000) libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x0023a000) libm.so.6 => /lib/tls/libm.so.6 (0x00c59000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00191000) libpthread.so.0 => /lib/tls/libpthread.so.0 (0x00d71000) libc.so.6 => /lib/tls/libc.so.6 (0x00b3c000) /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x00b1f000)
The only lines that differ between the two are these:
libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x0023a000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x00191000)
While it is possible that only the lines in common are to blame, it is also logical to assume that it may be related to an interface between them and other system components. Not having a tremendous amount of experience with C under Linux, I'm not quite sure where to go next but I am certainly willing to roll up my sleeves and dive in head first.
If determining the specific cause of this problem by someone who isn't a developer of the game itself is not feasible, then my question is this: Is it possible to isolate the pieces specific to FC1 and use them in some sort of artificial environment on FC2? If that is possible, perhaps the game server could be tricked into believing that the system was actually FC1. That assumes, of course, that this isn't some kernel related issue which certainly cannot be circumvented.
The gaming community tends to be fairly picky about this sort of thing and might not feel comfortable playing on our server for matches if this bug persists. That would mean I would need to go back to FC1 and lose the benefits of the advances in FC2. That's something I would obviously like to avoid. I and others have posted about this problem for some time now and the developers of the mod probably won't respond because they have moved on to bigger and better things. If anyone could point me in the right direction, I would greatly appreciate it.
On Tue, Aug 31, 2004 at 10:58:10AM -0400, Shockwave wrote:
have noticed something strange. All of a sudden, every F-16 fighter that spawns on any map has its right rear wheel sunk into the ground.
Softer sand ;)
bug persists. That would mean I would need to go back to FC1 and lose the benefits of the advances in FC2. That's something I would obviously like to avoid. I and others have posted about this problem for some time now and the developers of the mod probably won't respond because they have moved on to bigger and better things. If anyone could point me in the right direction, I would greatly appreciate it.
There was a post recently about FPU setup corner cases on 2.6.x being different. Not being an FPU person I didn't pay much attention. Something like that is more likely I suspect to be relevant.
On Tue, 2004-08-31 at 10:56, Alan Cox wrote:
Softer sand ;)
LMAO!
There was a post recently about FPU setup corner cases on 2.6.x being different. Not being an FPU person I didn't pay much attention. Something like that is more likely I suspect to be relevant.
Is there anything I can do about it if the FPU is to blame short of abandoning the 2.6 kernel? Do you know of anywhere I could look to get some more information? The FPU isn't something I know a lot about either. ;)
On Tue, Aug 31, 2004 at 02:06:07PM -0400, Shockwave wrote:
Is there anything I can do about it if the FPU is to blame short of abandoning the 2.6 kernel? Do you know of anywhere I could look to get some more information? The FPU isn't something I know a lot about either. ;)
First tests to run are the ones Jakub suggested
On Tue, Aug 31, 2004 at 10:58:10AM -0400, Shockwave wrote:
If determining the specific cause of this problem by someone who isn't a developer of the game itself is not feasible, then my question is this: Is it possible to isolate the pieces specific to FC1 and use them in some sort of artificial environment on FC2? If that is possible, perhaps the game server could be tricked into believing that the system was actually FC1. That assumes, of course, that this isn't some kernel related issue which certainly cannot be circumvented.
You can e.g. unpack FC1 glibc into some subtree and run the game against that glibc instead. 1) make sure vdso=0 is passed on the kernel command line 2) mkdir ~/fc1glibc; cd ~/fc1glibc; rpm2cpio ~/glibc-2.3.2-101.4.i686.rpm | cpio -id 3) run the game with ~/fc1glibc/lib/ld-linux.so.2 --library-path ~/fc1glibc/lib /the/game arguments
You can also try booting 2.4 kernel instead of 2.6 one.
Jakub
On Tue, Aug 31, 2004 at 11:03:06AM -0400, Jakub Jelinek wrote:
You can e.g. unpack FC1 glibc into some subtree and run the game against that glibc instead.
- make sure vdso=0 is passed on the kernel command line
Is "sysctl -w kernel.vdso=0" good enough?
I ask because I've noticed that even with that, mach doesn't seem to want to let me rebuild FC1 packages. (It always gives me "Return value: 127".) Could be a mach bug though, I suppose.
Steve
On Tue, 2004-08-31 at 11:03, Jakub Jelinek wrote:
You can e.g. unpack FC1 glibc into some subtree and run the game against that glibc instead.
- make sure vdso=0 is passed on the kernel command line
- mkdir ~/fc1glibc; cd ~/fc1glibc; rpm2cpio ~/glibc-2.3.2-101.4.i686.rpm | cpio -id
- run the game with
~/fc1glibc/lib/ld-linux.so.2 --library-path ~/fc1glibc/lib /the/game arguments
You can also try booting 2.4 kernel instead of 2.6 one.
Jakub
Thank you very much for the information. I'm certainly learning a lot along the way. :)
I edited grub.conf to use the new parameter, rebooted, then used sysctl to verify it took effect. Then I performed the rest of the steps you mentioned, started the server, and unfortunately the problem is still there. I made sure to try both builds of the executable as well. Aside from the possibility the kernel is to blame, could there be other libraries native to FC1 that I may need? Perhaps the reason it still isn't fixed is because the virtual FC1 environment is not complete in some way.
Shockwave wrote:
On Tue, 2004-08-31 at 11:03, Jakub Jelinek wrote:
You can e.g. unpack FC1 glibc into some subtree and run the game against that glibc instead.
- make sure vdso=0 is passed on the kernel command line
- mkdir ~/fc1glibc; cd ~/fc1glibc; rpm2cpio ~/glibc-2.3.2-101.4.i686.rpm | cpio -id
- run the game with
~/fc1glibc/lib/ld-linux.so.2 --library-path ~/fc1glibc/lib /the/game arguments
You can also try booting 2.4 kernel instead of 2.6 one.
Jakub
Thank you very much for the information. I'm certainly learning a lot along the way. :)
I edited grub.conf to use the new parameter, rebooted, then used sysctl to verify it took effect. Then I performed the rest of the steps you mentioned, started the server, and unfortunately the problem is still there. I made sure to try both builds of the executable as well. Aside from the possibility the kernel is to blame, could there be other libraries native to FC1 that I may need? Perhaps the reason it still isn't fixed is because the virtual FC1 environment is not complete in some way.
Looking at the other libraries you are going to need the ncurses rpm also installed in that tree.
I would also check via lsof that the program is running with just those libraries. I would also try the last FC1 2.4.x kernel.
On Tue, 2004-08-31 at 14:20, Stephen J Smoogen wrote:
Looking at the other libraries you are going to need the ncurses rpm also installed in that tree.
I would also check via lsof that the program is running with just those libraries. I would also try the last FC1 2.4.x kernel.
-- Stephen John Smoogen smoogen@lanl.gov Los Alamos National Lab CCN-5 | PH: 4-0645 Ta-03 SM-1498 MailStop B255 DP 10S Los Alamos, NM 87545
Here is the relevant output from lsof when the dynamic version of the game executable is running:
/lib/ld-2.3.3.so /lib/libdl-2.3.3.so /lib/libgcc_s-3.3.3-20040413.so.1 /lib/libnss_dns-2.3.3.so /lib/libnss_files-2.3.3.so /lib/libresolv-2.3.3.so /lib/tls/libc-2.3.3.so /lib/tls/libm-2.3.3.so /lib/tls/libpthread-0.61.so /usr/lib/libncurses.so.5.4 /usr/lib/libstdc++.so.5.0.5
Here is the output when the static build is running:
/lib/ld-2.3.3.so /lib/libdl-2.3.3.so /lib/libnss_dns-2.3.3.so /lib/libnss_files-2.3.3.so /lib/libresolv-2.3.3.so /lib/tls/libc-2.3.3.so /lib/tls/libm-2.3.3.so /lib/tls/libpthread-0.61.so /usr/lib/libncurses.so.5.4
The only difference between the two are the following lines:
/lib/libgcc_s-3.3.3-20040413.so.1 /usr/lib/libstdc++.so.5.0.5
Using the FC1 glibc libraries with the dynamic build yields this:
/usr/lib/libncurses.so.5.4 /usr/lib/libstdc++.so.5.0.5 /usr/local/games/bf1942/bf1942_lnxded.dynamic /usr/local/games/fc1glibc/lib/ld-2.3.2.so /usr/local/games/fc1glibc/lib/libdl-2.3.2.so /usr/local/games/fc1glibc/lib/libnss_dns-2.3.2.so /usr/local/games/fc1glibc/lib/libnss_files-2.3.2.so /usr/local/games/fc1glibc/lib/libresolv-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libc-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libm-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libpthread-0.60.so
And using the FC1 glibc libraries with the static build shows this:
/usr/lib/libncurses.so.5.4 /usr/local/games/bf1942/bf1942_lnxded.static /usr/local/games/fc1glibc/lib/ld-2.3.2.so /usr/local/games/fc1glibc/lib/libdl-2.3.2.so /usr/local/games/fc1glibc/lib/libnss_dns-2.3.2.so /usr/local/games/fc1glibc/lib/libnss_files-2.3.2.so /usr/local/games/fc1glibc/lib/libresolv-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libc-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libm-2.3.2.so /usr/local/games/fc1glibc/lib/tls/libpthread-0.60.so
This time, the only difference is this line:
/usr/lib/libstdc++.so.5.0.5
Aside from ncurses in the case of the static build using FC1 libraries, there don't appear to be any additional open files that have been missed. With respect to ncurses, I believe it is only used to create the status display of the server output in the shell from which it was launched.
I matched all this against the output from ldd and I couldn't find a smoking gun. It looks as if Jakub's trick worked flawlessly and everything that would've been FC1 specific has been captured and put to work. Could this indicate that the kernel is the problem or more specifically the implementation of the FPU as you have suggested Alan?
I did try to install the most recent FC1 kernel rpms for 2.4.22-1.2199.nptl as well, but it told me a newer kernel version is already installed and subsequently aborted the installation. Does anyone have any suggestions?
Shockwave wrote:
On Tue, 2004-08-31 at 14:20, Stephen J Smoogen wrote:
/usr/lib/libstdc++.so.5.0.5
I would add the c++ and the ncurses from FC1 'just-in-case' to your /usr/local/games/fc1glibc tree also. use the directions jakub gave in the previous email using rpm2cpio.
I am guessing that these will NOT fix the problem.. but it would be best to get those out of the way right now. Nothing better to avoid a glibc/kernel finger pointing match as quickly as possible. ;).
Aside from ncurses in the case of the static build using FC1 libraries, there don't appear to be any additional open files that have been missed. With respect to ncurses, I believe it is only used to create the status display of the server output in the shell from which it was launched.
I matched all this against the output from ldd and I couldn't find a smoking gun. It looks as if Jakub's trick worked flawlessly and everything that would've been FC1 specific has been captured and put to work. Could this indicate that the kernel is the problem or more specifically the implementation of the FPU as you have suggested Alan?
I am heading towards that in my case. My view will be that the server code is written expecting something that 2.4.xx gives FPU wise and 2.6.xx gives differently. The code is closed sourced? If it isnt it would be interesting to find out where it is goofy now.. because it might affect other people doing scientific code who assume that their code works on 2.4 it will work smae on 2.6
I did try to install the most recent FC1 kernel rpms for 2.4.22-1.2199.nptl as well, but it told me a newer kernel version is already installed and subsequently aborted the installation. Does anyone have any suggestions?
rpm -ivh --force <kernel.rpm>
Check grub.conf to make sure it didnt break anything.
On Wed, 2004-09-01 at 12:58, Stephen J Smoogen wrote:
I would add the c++ and the ncurses from FC1 'just-in-case' to your /usr/local/games/fc1glibc tree also. use the directions jakub gave in the previous email using rpm2cpio.
I am guessing that these will NOT fix the problem.. but it would be best to get those out of the way right now. Nothing better to avoid a glibc/kernel finger pointing match as quickly as possible. ;).
Done. Now all libraries in use are the ones in my special directory. Unfortunately, you were right. The problem remains.
I am heading towards that in my case. My view will be that the server code is written expecting something that 2.4.xx gives FPU wise and 2.6.xx gives differently. The code is closed sourced? If it isnt it would be interesting to find out where it is goofy now.. because it might affect other people doing scientific code who assume that their code works on 2.4 it will work smae on 2.6
I agree that this has implications in areas other than this game server code, but the game engine code is closed sourced. I could write Electronic Arts and ask them, however I don't expect they will care to listen. They never replied to any of my email when I wrote them asking for non-proprietary information to develop some anti-cheat code to help the gaming community so I don't expect them to care now when their proprietary code is involved. I can try, but I think I would need some fairly strong arguments and perhaps the backing of someone else who isn't the leader of a competitive gaming team. If it had an impact on homeland security...maybe they'd help...maybe. ;)
rpm -ivh --force <kernel.rpm>
Check grub.conf to make sure it didnt break anything.
Since the server is co-located in a downtown facility, I'm going to wait until tomorrow morning to mess with the kernel. If I do something that hoses the system, it will not only take down the game server but our TeamSpeak server as well. Considering we have a practice tonight, I don't think I should risk it.
In the meantime, if anyone is interested in seeing what I am talking about with the F-16, here are some links to a few images:
http://www.clan-tf20.com/uploads/ps/F-16_bug1.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug2.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug3.JPG
When the plane spawns, the entire wheel is underground. If you jump in and hit the thrusters, it will level itself as you gain speed and start to act normally. However, as soon as you land and start to come to a halt, the strut under the right wing sinks down again. If you look closely, you'll see the strut isn't connected to the wheel on the right as it is on the left. It's as if the calculation that places the strut in the center of the wheel is wrong.
By the way, I just want to take a moment to thank everyone who has taken the time to weigh in on this issue. I know it's not very important to anyone outside of our gaming community, but I've received some top-notch help regardless from some very cool people. The Fedora community is truly a wonderful group of individuals that don't get nearly the amount of praise they deserve. Thanks again!
As soon as I install the kernel files tomorrow, I'll post my results.
Shockwave wrote:
I am heading towards that in my case. My view will be that the server code is written expecting something that 2.4.xx gives FPU wise and 2.6.xx gives differently. The code is closed sourced? If it isnt it would be interesting to find out where it is goofy now.. because it might affect other people doing scientific code who assume that their code works on 2.4 it will work smae on 2.6
I agree that this has implications in areas other than this game server code, but the game engine code is closed sourced. I could write Electronic Arts and ask them, however I don't expect they will care to listen. They never replied to any of my email when I wrote them asking for non-proprietary information to develop some anti-cheat code to help the gaming community so I don't expect them to care now when their proprietary code is involved. I can try, but I think I would need some fairly strong arguments and perhaps the backing of someone else who isn't the leader of a competitive gaming team. If it had an impact on homeland security...maybe they'd help...maybe. ;)
I think that in that case, you will be wanting to maintain a FC1/ 2.4.xx box until EA decides that a 2.6.xx version is needed. I would drop them an email letting them know about it anyway.
In the meantime, if anyone is interested in seeing what I am talking about with the F-16, here are some links to a few images:
http://www.clan-tf20.com/uploads/ps/F-16_bug1.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug2.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug3.JPG
Looks like soft sand to me ;). The F-16 was never known to deal with anything but hard-top.
=
In the meantime, if anyone is interested in seeing what I am talking about with the F-16, here are some links to a few images:
http://www.clan-tf20.com/uploads/ps/F-16_bug1.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug2.JPG http://www.clan-tf20.com/uploads/ps/F-16_bug3.JPG
Looks like soft sand to me ;). The F-16 was never known to deal with anything but hard-top.
Looks more like a feature than a bug then, 2.6 offers a more realistic gaming experience =)
-HK
On Thu, 2004-09-02 at 09:53, Stephen J Smoogen wrote:
I think that in that case, you will be wanting to maintain a FC1/ 2.4.xx box until EA decides that a 2.6.xx version is needed. I would drop them an email letting them know about it anyway.
I really hate the idea of reloading the box with FC1 although it would certainly fix this problem. I was really hoping that I could make the FC1 kernel work. Unfortunately, my attempt to force the install met with disaster. I installed it this morning, rebooted, then found out it wouldn't respond to my pings. I drove into the city and hooked up a monitor to it and noticed that it was running but the network didn't start. After I started the network manually, I noticed that the routing table was hosed. I then did a quick check of the system log and found that there were some serious problems during boot. I'm not sure I know enough about the kernel to go messing around with trying to make the 2.4.x one work on FC2 and I don't want to be any more of a burden than I already have been. I want to be sure I don't become someone who keeps sucking resources from the great people who post on this list. I feel bad enough for taking up so much of everyone's time as it is.
One thing I think I'm going to do is research what it takes to mod the BattleField 1942 engine so I can hopefully take a peek at the code used to model the F-16. It might be a dead end, but it's worth a shot. The way I look at it is if the left rear wheel works, perhaps there is a way to "borrow" the code and modify it for use with the right rear wheel. There must be something about the right rear wheel that is causing this. With any luck, I'll find it.
I've already taken the liberty of posting on the official BF1942 server administrator list and in the forums where the developers visit from time to time. So far I haven't heard a peep from anyone about this but I'm going to try to remain optimistic. In the meantime, I'm going to go hit some of those modding web sites. If anyone comes up with anything else I can try, I'm all for it. Thanks again for all the excellent help! You guys are the best!
I've been successful in crafting a fix for this problem and wanted to say thanks to everyone who helped me track down the cause. As it turns out, I don't think it was the 2.6.x kernel, but rather how it responded to an incorrectly formed set of parameters given to it by the game engine. For a more complete discussion of this topic, you can view this post I made in the BF1942 Lightcubed forums:
http://bf1942.lightcubed.com/forum/viewtopic.php?t=1866
Thanks again to everyone for the outstanding help!
devel@lists.stg.fedoraproject.org
-
Alan Cox -
Hans Kristian Rosbach -
Jakub Jelinek -
Shockwave -
Stephen Smoogen -
Steven Pritchard