In an effort to create a set of benchmarks for gauging desktop performance I have written up a procedure for gnome-terminal to test the speed that text is sent to the terminal window and written a script to perform the test. I have attached the writeup for the procedure (bench_terminal.txt) and the script to run the test (cattest) to this email.
I am interested in hearing people's comments on this test. I know this is only one test and doesn't address desktop issues like program startup time, but we have to start somewhere.
We can start to assemble the benchmarks and put them in
http://fedora.redhat.com/projects/additional-projects/benchmarks/
-Will
Benchmark text output performance of gnome-terminal
Frequently output is sent to a terminal window. In Gnome the program gnome-terminal handles the display of text on a terminal window. The, one simple benchmark is to determine the amount of time required to output a large text file to a gnome-terminal window.
gnome-terminal is a little tricky to benchmark because has a server. When you start gnome-terminal it connects to the server rather than starting a new child process. Thus, timing the gnome-terminal command will count the time required to communicate information to the gnome-terminal server rather than the time required to actually perform the task. To work around this problem a script is executed within the new gnome-terminal window and the information is saved to a file.
Procedure:
0) Get system configuration information hardware and software: CPU: cat /proc/cpuinfo Memory: cat /proc/meminfo Kernel: uname -a gnome-terminal: rpm -qa gnome-terminal xserver: rpm -qa xorg-x11
1) Get the test file, the jargon file from Project Gutenberg:
http://www.gutenberg.net/etext02/jarg422.txt.
2) Verify that the file is the same with md5sum $ md5sum jarg422.txt ef9b53f52312ee266c98c8e206d9e823 jarg422.txt
3) Place the cattest script in the same directory as jarg422.txt and make it executable.
4) Run the test on the console of the machine with the command below. The script will generate a file "cattime in the directory with the amount of time required to cat the file to the terminal window.
gnome-terminal -e "./cattest"
4a) If the system is set up to run oprofile and you have root access, you can run the same script as root with command below to get some additional profiling information in the cattime file:
gnome-terminal -e "./cattest --profile"
Additional analysis on the oprofile data can be performed after the benchmark completes.
#! /bin/bash # # Simple test to gather data on where gnome-terminal spends # time This is compilicated by the terminal server model of # gnome-terminal. Time taken for benchmark written to cattime. # # When optional --profile on commandline, oprofile used to get an # overall view of what is happening on the system. PROFILING is only # going to work with kernel that have oprofile support (Red Hat SMP # kernels). # # Will Cohen # 5/27/2004 #
BENCHMARK="cattest" VERSION=0.0
OPCONTROL=/usr/bin/opcontrol OPREPORT=/usr/bin/opreport RM=/bin/rm RESULTS_FILE=cattime
if test "$1" = "--profile"; then PROFILING=yes else PROFILING=no fi
# Setup default oprofile. if test "$PROFILING" = "yes"; then $OPCONTROL --deinit $OPCONTROL --reset # FIXME Command below may use previous event settings for oprofile. $OPCONTROL --setup --no-vmlinux --separate=library $OPCONTROL --start fi
# Run the actual experiment $RM -rf $RESULTS_FILE echo "Benchmark: " $BENCHMARK " Version: " $VERSION >> $RESULTS_FILE date >> $RESULTS_FILE
# The actual benchmark being timed is below. /usr/bin/time /bin/cat `pwd`/jarg422.txt 2>> $RESULTS_FILE
# Shutdown oprofile. if test "$PROFILING" = "yes"; then $OPCONTROL --dump $OPCONTROL --shutdown # If PROFILING, need to do analysis with oprerport after running the test. # May need more details than what is provided by command below. $OPREPORT --threshold 2 --long-filenames >> $RESULTS_FILE fi
This is what I got with the benchmark. I'd like to compare it with others. Mine was tested on a thoshiba 1400-503 (1.3Ghz) running fc2, 256 MB Ram.
Benchmark: cattest Version: 0.0 Do Mai 27 22:49:22 CEST 2004 0.00user 0.09system 0:15.82elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+122minor)pagefaults 0swaps
Will Cohen wrote:
In an effort to create a set of benchmarks for gauging desktop performance I have written up a procedure for gnome-terminal to test the speed that text is sent to the terminal window and written a script to perform the test. I have attached the writeup for the procedure (bench_terminal.txt) and the script to run the test (cattest) to this email.
I am interested in hearing people's comments on this test. I know this is only one test and doesn't address desktop issues like program startup time, but we have to start somewhere.
We can start to assemble the benchmarks and put them in
http://fedora.redhat.com/projects/additional-projects/benchmarks/
-Will
Benchmark text output performance of gnome-terminal
Frequently output is sent to a terminal window. In Gnome the program gnome-terminal handles the display of text on a terminal window. The, one simple benchmark is to determine the amount of time required to output a large text file to a gnome-terminal window.
gnome-terminal is a little tricky to benchmark because has a server. When you start gnome-terminal it connects to the server rather than starting a new child process. Thus, timing the gnome-terminal command will count the time required to communicate information to the gnome-terminal server rather than the time required to actually perform the task. To work around this problem a script is executed within the new gnome-terminal window and the information is saved to a file.
Procedure:
- Get system configuration information hardware and software:
CPU: cat /proc/cpuinfo Memory: cat /proc/meminfo Kernel: uname -a gnome-terminal: rpm -qa gnome-terminal xserver: rpm -qa xorg-x11
- Get the test file, the jargon file from Project Gutenberg:
http://www.gutenberg.net/etext02/jarg422.txt.
- Verify that the file is the same with md5sum
$ md5sum jarg422.txt ef9b53f52312ee266c98c8e206d9e823 jarg422.txt
- Place the cattest script in the same directory as jarg422.txt and
make it executable.
- Run the test on the console of the machine with the command
below. The script will generate a file "cattime in the directory with the amount of time required to cat the file to the terminal window.
gnome-terminal -e "./cattest"
4a) If the system is set up to run oprofile and you have root access, you can run the same script as root with command below to get some additional profiling information in the cattime file:
gnome-terminal -e "./cattest --profile"
Additional analysis on the oprofile data can be performed after the benchmark completes.
#! /bin/bash # # Simple test to gather data on where gnome-terminal spends # time This is compilicated by the terminal server model of # gnome-terminal. Time taken for benchmark written to cattime. # # When optional --profile on commandline, oprofile used to get an # overall view of what is happening on the system. PROFILING is only # going to work with kernel that have oprofile support (Red Hat SMP # kernels). # # Will Cohen # 5/27/2004 #
BENCHMARK="cattest" VERSION=0.0
OPCONTROL=/usr/bin/opcontrol OPREPORT=/usr/bin/opreport RM=/bin/rm RESULTS_FILE=cattime
if test "$1" = "--profile"; then PROFILING=yes else PROFILING=no fi
# Setup default oprofile. if test "$PROFILING" = "yes"; then $OPCONTROL --deinit $OPCONTROL --reset # FIXME Command below may use previous event settings for oprofile. $OPCONTROL --setup --no-vmlinux --separate=library $OPCONTROL --start fi
# Run the actual experiment $RM -rf $RESULTS_FILE echo "Benchmark: " $BENCHMARK " Version: " $VERSION >> $RESULTS_FILE date >> $RESULTS_FILE
# The actual benchmark being timed is below. /usr/bin/time /bin/cat `pwd`/jarg422.txt 2>> $RESULTS_FILE
# Shutdown oprofile. if test "$PROFILING" = "yes"; then $OPCONTROL --dump $OPCONTROL --shutdown # If PROFILING, need to do analysis with oprerport after running the test. # May need more details than what is provided by command below. $OPREPORT --threshold 2 --long-filenames >> $RESULTS_FILE fi
On Thu, May 27, 2004 at 04:21:38PM -0400, Will Cohen wrote:
# The actual benchmark being timed is below. /usr/bin/time /bin/cat `pwd`/jarg422.txt 2>> $RESULTS_FILE
You're profiling cat under gnome-terminal, not gnome-terminal + cat.
Wouldn't be better to run instead: /usr/bin/time gnome-terminal -x cat $PWD/jarg422.txt
Regards, Luciano Rocha
Luciano Miguel Ferreira Rocha wrote:
On Thu, May 27, 2004 at 04:21:38PM -0400, Will Cohen wrote:
# The actual benchmark being timed is below. /usr/bin/time /bin/cat `pwd`/jarg422.txt 2>> $RESULTS_FILE
You're profiling cat under gnome-terminal, not gnome-terminal + cat.
Wouldn't be better to run instead: /usr/bin/time gnome-terminal -x cat $PWD/jarg422.txt
You are correct that this also includes the time for the cat. I tried that earlier there are comments on that in the procedure. The net result with the suggested change on gnome-terminal is that you get the amount of time it takes to fire off the command to the gnome-terminal server, not the amount of time to complete the task. The /usr/bin/time will finish long before the cat is actually done. Doing something like that on xterm you will get gnome-terminal + cat time.
The goal of the benchmark was to make something that could provide some indication about the amount of time required to push a lot of text to a terminal window, exercise some of the gnome-terminal code, be reasonably easy to run, and have some chance at being repeatable. That the exeperiment includes time for cat is not that big an issue, so long the amount of time for cat stays the same and cat times don't totally dominate the time. If you have oprofile setup and run the benchmark with "--profile" you can see that xterm dominates the cpu by a large margin, about 75% of the samples.
Just to get an idea of the time spent on cat for the test I did the following on a 2.4GHz P4 with 512M memory running fc2:
$ /usr/bin/time /bin/cat /home/wcohen/jarg422.txt > /dev/null 0.00user 0.00system 0:00.57elapsed 0%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+106minor)pagefaults 0swaps $ /usr/bin/time /bin/cat /home/wcohen/jarg422.txt > /dev/null 0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+105minor)pagefaults 0swaps $ /usr/bin/time /bin/cat /home/wcohen/jarg422.txt > /dev/null 0.00user 0.00system 0:00.00elapsed 100%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+104minor)pagefaults 0swaps $ /usr/bin/time /bin/cat /home/wcohen/jarg422.txt > /dev/null
It probably would be a good idea to have the script run the test multiple times to make sure that jarg422.txt is pulled in and that the results are consistent.
-Will
On Fri, May 28, 2004 at 09:16:18AM -0400, Will Cohen wrote:
Luciano Miguel Ferreira Rocha wrote:
On Thu, May 27, 2004 at 04:21:38PM -0400, Will Cohen wrote:
# The actual benchmark being timed is below. /usr/bin/time /bin/cat `pwd`/jarg422.txt 2>> $RESULTS_FILE
You're profiling cat under gnome-terminal, not gnome-terminal + cat.
Wouldn't be better to run instead: /usr/bin/time gnome-terminal -x cat $PWD/jarg422.txt
You are correct that this also includes the time for the cat. I tried that earlier there are comments on that in the procedure.
I have no problem with cat being included in the calculated time. It shouldn't consume much.
The net result with the suggested change on gnome-terminal is that you get the amount of time it takes to fire off the command to the gnome-terminal server, not the amount of time to complete the task. The /usr/bin/time will finish long before the cat is actually done.
Oh, yes, I forgot. I usually run gnome-terminal with the options --disable-factory --sm-disable. They should prevent the use of a gnome-terminal server, giving more meaningful results.
Doing something like that on xterm you will get gnome-terminal + cat time.
The goal of the benchmark was to make something that could provide some indication about the amount of time required to push a lot of text to a terminal window, exercise some of the gnome-terminal code, be reasonably easy to run, and have some chance at being repeatable. That the exeperiment includes time for cat is not that big an issue, so long the amount of time for cat stays the same and cat times don't totally dominate the time. If you have oprofile setup and run the benchmark with "--profile" you can see that xterm dominates the cpu by a large margin, about 75% of the samples.
I wasn't sure about oprofile also recording other applications. Guess I should read a good documentation about it.
Regards, Luciano Rocha
On Fri, 2004-05-28 at 09:52, Luciano Miguel Ferreira Rocha wrote:
The net result with the suggested change on gnome-terminal is that you get the amount of time it takes to fire off the command to the gnome-terminal server, not the amount of time to complete the task. The /usr/bin/time will finish long before the cat is actually done.
Oh, yes, I forgot. I usually run gnome-terminal with the options --disable-factory --sm-disable. They should prevent the use of a gnome-terminal server, giving more meaningful results.
But is it interesting to time the startup time of gnome-terminal? While that's perhaps an interesting benchmark as well, it has little correlation with questions like "how fast will my kernel compile go when run in gnome-terminal"?
If we are measuring wall clock time, timing the cat command seems to be pretty appropriate.
If we are measuring CPU usage, then we need to find CPU usage from the terminal *and* from the X server, so we already need to use a tool like oprofile that can get a global view.
Regards, Owen
On Fri, May 28, 2004 at 02:19:42PM -0400, Owen Taylor wrote:
But is it interesting to time the startup time of gnome-terminal? While that's perhaps an interesting benchmark as well, it has little correlation with questions like "how fast will my kernel compile go when run in gnome-terminal"?
But will cat end after all output has been displayed or before? (Due to pipe buffers.)
If we are measuring wall clock time, timing the cat command seems to be pretty appropriate.
If we are measuring CPU usage, then we need to find CPU usage from the terminal *and* from the X server, so we already need to use a tool like oprofile that can get a global view.
I agree.
Regards, Luciano Rocha
Luciano Miguel Ferreira Rocha wrote:
On Fri, May 28, 2004 at 02:19:42PM -0400, Owen Taylor wrote:
But is it interesting to time the startup time of gnome-terminal? While that's perhaps an interesting benchmark as well, it has little correlation with questions like "how fast will my kernel compile go when run in gnome-terminal"?
But will cat end after all output has been displayed or before? (Due to pipe buffers.)
I don't know. That is a good question. I don't know enough about the gnome-terminal internals to say.
If we are measuring wall clock time, timing the cat command seems to be pretty appropriate.
If we are measuring CPU usage, then we need to find CPU usage from the terminal *and* from the X server, so we already need to use a tool like oprofile that can get a global view.
I agree.
Regards, Luciano Rocha
desktop@lists.fedoraproject.org
-
Florian Idelberger -
Luciano Miguel Ferreira Rocha -
Owen Taylor -
William Cohen