Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid-rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
On Fri, 2018-05-25 at 01:13 +0200, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid-rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
If I'm not wrong, Dell XPS 13 have AlpsPS/2 ALPS like my Dell Latitude E6410 .
I installed xorg-x11-drv-synaptics-legacy-1.9.0-6.fc27.x86_64 and set /etc/X11/xorg.conf.d/01-touchpad.conf Section "InputClass" Identifier "tap-by-default and other custom settings" MatchIsTouchpad "on" Option "TapButton1" "1" Option "TapButton2" "1" Option "TapButton3" "1" Option "VertEdgeScroll" "1" Option "HorizEdgeScroll" "1" Option "VertTwoFingerScroll" "1" Option "HorizTwoFingerScroll" "1" EndSection
cat /var/log/Xorg.0.log| grep -i syna [ 73.402] (II) LoadModule: "synaptics" (...) [ 73.415] (II) Using input driver 'synaptics' for 'AlpsPS/2 ALPS DualPoint TouchPad'
On Fri, 2018-05-25 at 03:32 +0100, Sérgio Basto wrote:
If I'm not wrong, Dell XPS 13 have AlpsPS/2 ALPS like my Dell Latitude E6410 .
Hi Sérgio,
The touchpad works well without any extra configuration. The problem here is not that I couldn't configure my touchpad but that Linux detects two touchpads on this laptop instead of one.
In any case I tried what you suggested but it didn't fix the issue.
Tim
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid-rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
On Thu, 2018-05-31 at 13:44 -0700, Laura Abbott wrote:
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid- rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
Looking into this a bit more, the root cause of this issue is that the touchpad is wired up in such a way that it is accessible on both PS/2 and I2C. I2C is preferred and used by i2c-hid but at the same time the PS/2 interface is also picked up by psmouse. Thus, Xorg sees the two interfaces as two different devices.
Solution would be to somehow remove the psmouse-detected device node from the system when i2c-hid detects the same device. Is this possible?
Best regards, Timur
On 1/6/18 22:50 , Timur Kristóf wrote:
On Thu, 2018-05-31 at 13:44 -0700, Laura Abbott wrote:
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid- rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
Looking into this a bit more, the root cause of this issue is that the touchpad is wired up in such a way that it is accessible on both PS/2 and I2C. I2C is preferred and used by i2c-hid but at the same time the PS/2 interface is also picked up by psmouse. Thus, Xorg sees the two interfaces as two different devices.
That is fairly standard and doesn't matter. This was also the case for earlier SMBus implementation but the common thing was that the PS/2 device never actually sent events. So it was only an issue for some tools that expected only one touchpad to exist (e.g. synclient). Check with evemu-record whether the device sends events. if not, then Xorg is definitely not confused by it. If it does and both event nodes send events then yes, we have a kernel bug there.
Solution would be to somehow remove the psmouse-detected device node from the system when i2c-hid detects the same device. Is this possible?
This is more an upstream discussion, CC-ing benjamin for details.
I do recommend filing a bug with an evemu recording of a freezing/jittering interaction. Lots of touchpad processing happens in userspace and this could be the cause of pressure thresholds set too high. CC me on that bug and I can have a look. sudo libinput debug-events --verbose will also show extra information including things when a touch ended based on its pressure information.
Cheers, Peter
On Sat, 2018-06-02 at 14:14 +1000, Peter Hutterer wrote:
On 1/6/18 22:50 , Timur Kristóf wrote:
On Thu, 2018-05-31 at 13:44 -0700, Laura Abbott wrote:
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid- rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
Looking into this a bit more, the root cause of this issue is that the touchpad is wired up in such a way that it is accessible on both PS/2 and I2C. I2C is preferred and used by i2c-hid but at the same time the PS/2 interface is also picked up by psmouse. Thus, Xorg sees the two interfaces as two different devices.
That is fairly standard and doesn't matter. This was also the case for earlier SMBus implementation but the common thing was that the PS/2 device never actually sent events. So it was only an issue for some tools that expected only one touchpad to exist (e.g. synclient). Check with evemu-record whether the device sends events. if not, then Xorg is definitely not confused by it. If it does and both event nodes send events then yes, we have a kernel bug there.
I've talked to Mario, a developer at Dell, and he pointed me to this freedesktop bug report: https://bugs.freedesktop.org/show_bug.cgi?id=101470
According to his description, there is actually no input coming from PS/2 but it is still (wrongly) picked up by syndaemon. To be honest, I'm not sure if Fedora uses syndaemon (I assume libinput is used for everything now?), but I can tell that the touchpad does jitter when the PS/2 device is there.
I can also confirm that evemu-record doesn't see any input coming from the PS/2 device.
Solution would be to somehow remove the psmouse-detected device node from the system when i2c-hid detects the same device. Is this possible?
This is more an upstream discussion, CC-ing benjamin for details.
I do recommend filing a bug with an evemu recording of a freezing/jittering interaction. Lots of touchpad processing happens in userspace and this could be the cause of pressure thresholds set too high. CC me on that bug and I can have a look. sudo libinput debug-events --verbose will also show extra information including things when a touch ended based on its pressure information.
I can try to reproduce the problem and see if I can make an evemu recording. What should I file the bug against (and where)? Unfortunately I'm not familiar with the components involved.
Searching for libinput in the journal reveals a bunch of touch jump messages, which seem to be related to the problem: https://paste.fedoraproject.org/paste/MoZGqf25VpHjCLSYzTxdfw
Best regards, Tim
On 2/6/18 19:53 , Timur Kristóf wrote:
On Sat, 2018-06-02 at 14:14 +1000, Peter Hutterer wrote:
On 1/6/18 22:50 , Timur Kristóf wrote:
On Thu, 2018-05-31 at 13:44 -0700, Laura Abbott wrote:
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid- rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
Looking into this a bit more, the root cause of this issue is that the touchpad is wired up in such a way that it is accessible on both PS/2 and I2C. I2C is preferred and used by i2c-hid but at the same time the PS/2 interface is also picked up by psmouse. Thus, Xorg sees the two interfaces as two different devices.
That is fairly standard and doesn't matter. This was also the case for earlier SMBus implementation but the common thing was that the PS/2 device never actually sent events. So it was only an issue for some tools that expected only one touchpad to exist (e.g. synclient). Check with evemu-record whether the device sends events. if not, then Xorg is definitely not confused by it. If it does and both event nodes send events then yes, we have a kernel bug there.
I've talked to Mario, a developer at Dell, and he pointed me to this freedesktop bug report: https://bugs.freedesktop.org/show_bug.cgi?id=101470
According to his description, there is actually no input coming from PS/2 but it is still (wrongly) picked up by syndaemon. To be honest, I'm not sure if Fedora uses syndaemon (I assume libinput is used for everything now?), but I can tell that the touchpad does jitter when the PS/2 device is there.
syndaemon works in that it monitors the keyboard state (or attaches to XRECORD events) and then sets a synaptics-specific property on the device it attaches to. This means that unless the synaptics driver is used, it cannot do anything. Do you have xorg-x11-drv-synaptics-legacy installed? If not, then libinput is being used (also: xinput list-props <device name> shows the driver name in the property names).
Even if you do have the synaptics driver installed, disabling a kernel device that doesn't send events ... well, you can't get less than "no events" :)
I can also confirm that evemu-record doesn't see any input coming from the PS/2 device.
Solution would be to somehow remove the psmouse-detected device node from the system when i2c-hid detects the same device. Is this possible?
This is more an upstream discussion, CC-ing benjamin for details.
I do recommend filing a bug with an evemu recording of a freezing/jittering interaction. Lots of touchpad processing happens in userspace and this could be the cause of pressure thresholds set too high. CC me on that bug and I can have a look. sudo libinput debug-events --verbose will also show extra information including things when a touch ended based on its pressure information.
I can try to reproduce the problem and see if I can make an evemu recording. What should I file the bug against (and where)? Unfortunately I'm not familiar with the components involved.
For upstream, this link has everything you need: https://wayland.freedesktop.org/libinput/doc/latest/reporting_bugs.html In the Fedora bugzilla it's the libinput component. Either bugzilla will work, it's the same person answering you anyway (i.e. me :)
Searching for libinput in the journal reveals a bunch of touch jump messages, which seem to be related to the problem: https://paste.fedoraproject.org/paste/MoZGqf25VpHjCLSYzTxdfw
there are a few issues here. The cursor jump complaint could be the cause of the jerkiness but we only trigger that when you move >20mm within one event - not something that can be triggered easily (or at all) in real life. All the timer offset negative bugs indicate that your compositor isn't rendering fast enough and libinput is starved for attention. That is currently relatively common under Wayland but I haven't seen this under Xorg yet.
IOW I think we may have a multitude of bugs here that need untangling before we can pin down psmouse as real culprit.
Cheers, Peter
On Sun, 2018-06-03 at 09:35 +1000, Peter Hutterer wrote:
On 2/6/18 19:53 , Timur Kristóf wrote:
On Sat, 2018-06-02 at 14:14 +1000, Peter Hutterer wrote:
On 1/6/18 22:50 , Timur Kristóf wrote:
On Thu, 2018-05-31 at 13:44 -0700, Laura Abbott wrote:
On 05/24/2018 04:13 PM, Timur Kristóf wrote:
Hi,
On the XPS 13 9360 and 9370 xinput sees two touchpads instead of one: ⎡ Virtual core pointer id=2 [master pointer (3)] ⎜ ↳ Virtual core XTEST pointer id=4 [slave pointer (2)] ⎜ ↳ DELL07E6:00 06CB:76AF Touchpad id=12 [slave pointer (2)] ⎜ ↳ SynPS/2 Synaptics TouchPad id=17 [slave pointer (2)]
dmesg also gives me the following:
[ 1.429811] psmouse serio1: synaptics: Your touchpad (PNP: DLL07e6 PNP0f13) says it can support a different bus. If i2c-hid and hid- rmi are not used, you might want to try setting psmouse.synaptics_intertouch to 1 and report this to linux-input@vger.kernel.org.
The Dell one is the real touchpad, the other one is an artifact of the psmouse driver. Weird touchpad freezes and jittering issues can be observed when both of these devices are there. The general advice on the web is to blacklist the psmouse driver. (This is also published by Dell as a .deb package which contains a config file doing just this.)
However, on Fedora I cannot blacklist the psmouse driver because it is compiled built-in instead of as a module. Could you guys change the Fedora kernel config to compile it as a module instead?
Thanks & best regards, Timur
So assuming nobody else has objections, I think it's okay to at least try this on rawhide and see if it uncovers any other problems.
Thanks, Laura
Looking into this a bit more, the root cause of this issue is that the touchpad is wired up in such a way that it is accessible on both PS/2 and I2C. I2C is preferred and used by i2c-hid but at the same time the PS/2 interface is also picked up by psmouse. Thus, Xorg sees the two interfaces as two different devices.
That is fairly standard and doesn't matter. This was also the case for earlier SMBus implementation but the common thing was that the PS/2 device never actually sent events. So it was only an issue for some tools that expected only one touchpad to exist (e.g. synclient). Check with evemu-record whether the device sends events. if not, then Xorg is definitely not confused by it. If it does and both event nodes send events then yes, we have a kernel bug there.
I've talked to Mario, a developer at Dell, and he pointed me to this freedesktop bug report: https://bugs.freedesktop.org/show_bug.cgi?id=101470
According to his description, there is actually no input coming from PS/2 but it is still (wrongly) picked up by syndaemon. To be honest, I'm not sure if Fedora uses syndaemon (I assume libinput is used for everything now?), but I can tell that the touchpad does jitter when the PS/2 device is there.
syndaemon works in that it monitors the keyboard state (or attaches to XRECORD events) and then sets a synaptics-specific property on the device it attaches to. This means that unless the synaptics driver is used, it cannot do anything. Do you have xorg-x11-drv-synaptics- legacy installed? If not, then libinput is being used (also: xinput list- props <device name> shows the driver name in the property names).
No, I don't have the synaptics driver installed. Probably the reason why I was pointed to that bug report is that this laptop is shipped with and old Ubuntu version which still uses it.
So it seems that this is a different problem with the same symptoms.
Even if you do have the synaptics driver installed, disabling a kernel device that doesn't send events ... well, you can't get less than "no events" :)
I was pretty convinced that the issue went away after disabling that device, but I might be wrong and it might have just been a coincidence.
I can also confirm that evemu-record doesn't see any input coming from the PS/2 device.
Solution would be to somehow remove the psmouse-detected device node from the system when i2c-hid detects the same device. Is this possible?
This is more an upstream discussion, CC-ing benjamin for details.
I do recommend filing a bug with an evemu recording of a freezing/jittering interaction. Lots of touchpad processing happens in userspace and this could be the cause of pressure thresholds set too high. CC me on that bug and I can have a look. sudo libinput debug-events --verbose will also show extra information including things when a touch ended based on its pressure information.
I can try to reproduce the problem and see if I can make an evemu recording. What should I file the bug against (and where)? Unfortunately I'm not familiar with the components involved.
For upstream, this link has everything you need: https://wayland.freedesktop.org/libinput/doc/latest/reporting_bugs.ht ml In the Fedora bugzilla it's the libinput component. Either bugzilla will work, it's the same person answering you anyway (i.e. me :)
Thanks!
Searching for libinput in the journal reveals a bunch of touch jump messages, which seem to be related to the problem: https://paste.fedoraproject.org/paste/MoZGqf25VpHjCLSYzTxdfw
there are a few issues here. The cursor jump complaint could be the cause of the jerkiness but we only trigger that when you move >20mm within one event - not something that can be triggered easily (or at all) in real life. All the timer offset negative bugs indicate that your compositor isn't rendering fast enough and libinput is starved for attention. That is currently relatively common under Wayland but I haven't seen this under Xorg yet.
That is just a copy-paste from my journal from the past couple of days, and I admit I was running both X and Wayland during that time. (Mostly to investigate whether this-or-that issue is reproducible on X or Wayland.) So some of those messages do come from Wayland actually.
IOW I think we may have a multitude of bugs here that need untangling before we can pin down psmouse as real culprit.
Looks like this turns out more complicated than I anticipated. What do you think should be the next step in untangling it?
Interestingly, the problem seems to be less noticable when running kernel 4.17, compared to 4.16 - but just because I haven't noticed it doesn't mean it's not there.
Best regards, Tim
On Sun, Jun 03, 2018 at 02:06:14AM +0200, Timur Kristóf wrote: [...]
For upstream, this link has everything you need: https://wayland.freedesktop.org/libinput/doc/latest/reporting_bugs.ht ml In the Fedora bugzilla it's the libinput component. Either bugzilla will work, it's the same person answering you anyway (i.e. me :)
Thanks!
Searching for libinput in the journal reveals a bunch of touch jump messages, which seem to be related to the problem: https://paste.fedoraproject.org/paste/MoZGqf25VpHjCLSYzTxdfw
there are a few issues here. The cursor jump complaint could be the cause of the jerkiness but we only trigger that when you move >20mm within one event - not something that can be triggered easily (or at all) in real life. All the timer offset negative bugs indicate that your compositor isn't rendering fast enough and libinput is starved for attention. That is currently relatively common under Wayland but I haven't seen this under Xorg yet.
That is just a copy-paste from my journal from the past couple of days, and I admit I was running both X and Wayland during that time. (Mostly to investigate whether this-or-that issue is reproducible on X or Wayland.) So some of those messages do come from Wayland actually.
IOW I think we may have a multitude of bugs here that need untangling before we can pin down psmouse as real culprit.
Looks like this turns out more complicated than I anticipated. What do you think should be the next step in untangling it?
file a bug :)
libinput (and kernel input) is at a point where most issues are well beyond quick fixes on a mailing list and we need to look at multiple complete logs, all too large for mailing lists. pastebins sort-of work but are too transient and don't work with tooling set up for bugzilla.
Cheers, Peter
kernel@lists.fedoraproject.org
-
Laura Abbott -
Peter Hutterer -
Sérgio Basto -
Timur Kristóf