Are there any Fedora Xen kernel gurus reading this list?
I'm trying to get my head around a problem reported by one of our atl1
(network driver) users. I can duplicate his problem. It happens
when we load the atl1 module while running 2.6.20-2925.5.fc7xen. Here's
the oops:
Unable to handle kernel paging request at ffff880382180ae8 RIP:
[<ffffffff80323985>] swiotlb_map_page+0x54/0x125
PGD 1100067 PUD 0
Oops: 0000 [1] SMP
last sysfs file: /class/net/eth0/address
CPU 0
Modules linked in: atl1 mii i915 drm netloop netbk blktap blkbk ipt_MASQUERADE iptable_nat nf_nat xt_physdev bridge w83627ehf hwmon i2c_isa eeprom nf_conntrack_netbios_ns ipt_REJECT nf_conntrack_ipv4 ipt_LOG ipt_recent iptable_filter ip_tables ip6t_LOG nf_conntrack_ipv6 xt_state nf_conntrack nfnetlink xt_tcpudp ip6table_filter ip6_tables x_tables ipv6 video sbs i2c_ec dock button battery asus_acpi backlight ac parport_pc lp parport snd_hda_intel snd_hda_codec snd_seq_dummy snd_seq_oss snd_seq_midi_event snd_seq snd_seq_device snd_pcm_oss snd_mixer_oss i2c_i801 i2c_core snd_pcm serio_raw snd_timer snd iTCO_wdt iTCO_vendor_support pcspkr soundcore snd_page_alloc shpchp sr_mod cdrom floppy sg dm_snapshot dm_zero dm_mirror dm_mod ata_piix ata_generic libata sd_mod scsi_mod ext3 jbd mbcache ehci_hcd ohci_hcd uhci_hcd
Pid: 2710, comm: ip Not tainted 2.6.20-2925.5.fc7xen #1
RIP: e030:[<ffffffff80323985>] [<ffffffff80323985>] swiotlb_map_page+0x54/0x125
RSP: e02b:ffff880019f2bd28 EFLAGS: 00010246
RAX: 7fffffffffffffff RBX: ffff880000f4c070 RCX: 000000007003975d
RDX: ffff880001fb5000 RSI: ffff881881f6ec58 RDI: 0000000000000012
RBP: 0000000000000002 R08: 0000000000000002 R09: ffff88002ba8a580
R10: ffff880038383780 R11: 00000000000000d0 R12: 00000000000005f0
R13: ffff88002ba8a880 R14: 0000000000000000 R15: ffff88002ba8a580
FS: 00002aaaaaac6820(0000) GS:ffffffff80579000(0000) knlGS:0000000000000000
CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000019d24000 CR4: 0000000000002620
Process ip (pid: 2710, threadinfo ffff880019f2a000, task ffff880026a73080)
Stack: ffff88002ba8a880 0000000000000000 ffff880019f99000 ffff880019db1800
0000000000000001 ffffffff883d2be0 ffff880019f99000 ffff88002ba8a000
ffff880000f4c000 00000000fffffff4 ffff88002ba8a580 ffff88002ba8a000
Call Trace:
[<ffffffff883d2be0>] :atl1:atl1_alloc_rx_buffers+0x180/0x220
[<ffffffff883d2caf>] :atl1:atl1_up+0x2f/0x660
[<ffffffff883d37eb>] :atl1:atl1_open+0x2b/0x60
[<ffffffff803d5050>] dev_open+0x2f/0x6e
[<ffffffff803d3548>] dev_change_flags+0x5a/0x11a
[<ffffffff80407f20>] devinet_ioctl+0x235/0x59c
[<ffffffff8020ba24>] __might_sleep+0x26/0xd0
[<ffffffff803cbf9b>] sock_ioctl+0x1c8/0x1e5
[<ffffffff80240762>] do_ioctl+0x21/0x6b
[<ffffffff802304db>] vfs_ioctl+0x25c/0x275
[<ffffffff8024a735>] sys_ioctl+0x59/0x78
[<ffffffff8025b300>] tracesys+0xb2/0xb7
Code: 48 8b 14 ca 48 21 c2 48 c1 e2 0c 48 85 db 48 8d 04 3a 74 11
RIP [<ffffffff80323985>] swiotlb_map_page+0x54/0x125
RSP <ffff880019f2bd28>
CR2: ffff880382180ae8
<3>BUG: sleeping function called from invalid context at kernel/rwsem.c:20
in_atomic():0, irqs_disabled():1
Call Trace:
[<ffffffff80295804>] down_read+0x15/0x23
[<ffffffff8029e75f>] acct_collect+0x42/0x18e
[<ffffffff80215467>] do_exit+0x200/0x809
[<ffffffff80262c8c>] do_page_fault+0x129c/0x134b
[<ffffffff8020e887>] link_path_walk+0xc5/0xd7
[<ffffffff802b07bf>] __rmqueue+0x50/0xf9
[<ffffffff80260447>] error_exit+0x0/0x6e
[<ffffffff80323985>] swiotlb_map_page+0x54/0x125
[<ffffffff883d2be0>] :atl1:atl1_alloc_rx_buffers+0x180/0x220
[<ffffffff883d2caf>] :atl1:atl1_up+0x2f/0x660
[<ffffffff883d37eb>] :atl1:atl1_open+0x2b/0x60
[<ffffffff803d5050>] dev_open+0x2f/0x6e
[<ffffffff803d3548>] dev_change_flags+0x5a/0x11a
[<ffffffff80407f20>] devinet_ioctl+0x235/0x59c
[<ffffffff8020ba24>] __might_sleep+0x26/0xd0
[<ffffffff803cbf9b>] sock_ioctl+0x1c8/0x1e5
[<ffffffff80240762>] do_ioctl+0x21/0x6b
[<ffffffff802304db>] vfs_ioctl+0x25c/0x275
[<ffffffff8024a735>] sys_ioctl+0x59/0x78
[<ffffffff8025b300>] tracesys+0xb2/0xb7
I'm pretty sure what's happening is that swiotlb -- apparently used by Xen to do IO memory mapping -- doesn't like pci_map_page(). Here's the driver code snippet that does the dma mapping for our receive buffers:
static u16 atl1_alloc_rx_buffers(struct atl1_adapter *adapter)
{
[snip}
skb_reserve(skb, NET_IP_ALIGN);
skb->dev = netdev;
buffer_info->alloced = 1;
buffer_info->skb = skb;
buffer_info->length = (u16) adapter->rx_buffer_len;
page = virt_to_page(skb->data);
offset = (unsigned long)skb->data & ~PAGE_MASK;
buffer_info->dma = pci_map_page(pdev, page, offset,
adapter->rx_buffer_len,
PCI_DMA_FROMDEVICE);
rfd_desc->buffer_addr = cpu_to_le64(buffer_info->dma);
rfd_desc->buf_len = cpu_to_le16(adapter->rx_buffer_len);
rfd_desc->coalese = 0;
[snip]
}
The Xen kernel seems to work okay if I change pci_map_page() to pci_map_single(), but there are other dma mappings in the code that seem to /need/ pci_map_page(). (We inherited this driver from the vendor, BTW.)
Should swiotlb_map_page() work? I can't even /find/ the function in 2.6.21-rc5 (non-xen).
Thanks,
Jay
