[Ntop-misc] PF_RING/TNAPI hang
George Wynes
george.wynes at gmail.com
Sat Dec 5 00:01:36 CET 2009
Long mail -- apologies.
I've run into problems trying to implement PF_RING/TNAPI to to capture
traffic off a 10GbE link. My end goal is to run multiple instances of
snort. I need to be able to capture over 1Mpps for it to be any use.
I've been making slow progress but now i'm stuck.
Here's the relevant spec from my environment - I can provide more if
necessary:
4 x 4 Core Xeon E7338 @ 2.40GHz
Ubuntu 9.4 (kernel 2.6.28-11-server)
Intel 7300 Chipset
With PF_RING alone and enabling ioatdma and dca (many thanks to this
list), I am able to capture about 250K - 400K pps depending on the system
tuning performed but I whatever knobs I twist I always hit the single
CPU core bottleneck.
With TNAPI included though, pfcount no longer works. It immediately hangs
after
the first line of output...
[...snipped...]
gwynes at host-01:/var/tmp/PF_RING/userland/examples$ sudo ./pfcount -i
eth14 at 12
Capturing from eth14 at 12
[...snipped...]
...and eventually I get messages written to the console...
[...snipped...]
[ 1554.779998] BUG: soft lockup - CPU#0 stuck for 61s! [tnapi(eth14.0):4851]
[...snipped...]
I get the same result from both the parent device and the individual
queues. I've also tried the different example applications and tcpdump.
While this is going on there are tnapi processes running on all CPUs at
about 50% except for the one on CPU0 which is at 100%. Also, ethtool -S
eth14 shows traffic being received on all of the RX channels while this is
going
on.
Any ideas where to look? I've included the output i'm getting in
/var/log/messages. I think this is showing me that dca, ioatdma and
ixgbe are set up correctly. I've tried newer/older kernels, CentOS 5.4
(TNAPI wouldn't compile) and Ubuntu 9.10 (same problem).
Any pointers appreciated.
Thanks.
-g
[...snipped...]
Dec 4 08:13:35 host-01 kernel: [ 1235.765771] ixgbe: Intel(R) 10 Gigabit
PCI Express Network Driver - version 2.0.44.14-NAPI
Dec 4 08:13:35 host-01 kernel: [ 1235.765775] Copyright (c) 1999-2009 Intel
Corporation.
Dec 4 08:13:35 host-01 kernel: [ 1235.765878] ixgbe 0000:0f:00.0: PCI INT A
-> GSI 16 (level, low) -> IRQ 16
Dec 4 08:13:35 host-01 kernel: [ 1235.865886] ixgbe: 0000:0f:00.0:
ixgbe_init_interrupt_scheme: Multiqueue Enabled: Rx Queue count = 16, Tx
Queue count = 16
Dec 4 08:13:35 host-01 kernel: [ 1235.867748] ixgbe: eth14: ixgbe_probe:
(PCI Express:2.5Gb/s:Width x8) 00:1b:21:20:dd:6b
Dec 4 08:13:35 host-01 kernel: [ 1235.867832] ixgbe: eth14: ixgbe_probe:
MAC: 1, PHY: 5, PBA No: e31879-002
Dec 4 08:13:35 host-01 kernel: [ 1235.867834] ixgbe: eth14: ixgbe_probe:
Internal LRO is enabled
Dec 4 08:13:35 host-01 kernel: [ 1235.867836] ixgbe: eth14: ixgbe_probe:
Intel(R) 10 Gigabit Network Connection
Dec 4 08:13:35 host-01 kernel: [ 1235.867864] TNAPI: (C) 2006-09 ntop.org
Dec 4 08:13:35 host-01 kernel: [ 1235.867866] TNAPI: using kernel threads
for packet polling
Dec 4 08:14:15 host-01 kernel: [ 1276.162906] TNAPI: init_tnapi(eth14)
Dec 4 08:14:15 host-01 kernel: [ 1276.162915] ---- TNAPI: eth14 added at
device slot 0
Dec 4 08:14:15 host-01 kernel: [ 1276.163007] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163061] TNAPI: spawn thread [eth14]
on CPU 10
Dec 4 08:14:15 host-01 kernel: [ 1276.163107] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163160] TNAPI: spawn thread [eth14]
on CPU 10
Dec 4 08:14:15 host-01 kernel: [ 1276.163206] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163256] TNAPI: spawn thread [eth14]
on CPU 13
Dec 4 08:14:15 host-01 kernel: [ 1276.163304] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163349] TNAPI: spawn thread [eth14]
on CPU 13
Dec 4 08:14:15 host-01 kernel: [ 1276.163403] TNAPI: spawn thread [eth14]
on CPU 9
Dec 4 08:14:15 host-01 kernel: [ 1276.163449] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163496] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163539] TNAPI: spawn thread [eth14]
on CPU 13
Dec 4 08:14:15 host-01 kernel: [ 1276.163583] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163626] TNAPI: spawn thread [eth14]
on CPU 13
Dec 4 08:14:15 host-01 kernel: [ 1276.163669] TNAPI: spawn thread [eth14]
on CPU 7
Dec 4 08:14:15 host-01 kernel: [ 1276.163716] TNAPI: spawn thread [eth14]
on CPU 13
Dec 4 08:14:15 host-01 kernel: [ 1276.226313] ADDRCONF(NETDEV_UP): eth14:
link is not ready
Dec 4 08:14:15 host-01 kernel: [ 1276.227861] ixgbe: eth14:
ixgbe_watchdog_task: NIC Link is Up 10 Gbps, Flow Control: RX/TX
Dec 4 08:14:15 host-01 kernel: [ 1276.234144] ADDRCONF(NETDEV_CHANGE):
eth14: link becomes ready
Dec 4 08:17:49 host-01 kernel: [ 1489.974496] [PF_RING] Welcome to PF_RING
4.1.0 ($Revision: 3982 $)
Dec 4 08:17:49 host-01 kernel: [ 1489.974498] (C) 2004-09 L.Deri <
deri at ntop.org>
Dec 4 08:17:49 host-01 kernel: [ 1489.974502] NET: Registered protocol
family 27
Dec 4 08:17:49 host-01 kernel: [ 1489.974513] [PF_RING] Ring slots
4096
Dec 4 08:17:49 host-01 kernel: [ 1489.974514] [PF_RING] Slot version 10
Dec 4 08:17:49 host-01 kernel: [ 1489.974516] [PF_RING] Capture TX
Yes [RX+TX]
Dec 4 08:17:49 host-01 kernel: [ 1489.974517] [PF_RING] Transparent Mode 0
Dec 4 08:17:49 host-01 kernel: [ 1489.974519] [PF_RING] IP Defragment No
Dec 4 08:17:49 host-01 kernel: [ 1489.974520] [PF_RING] Initialized
correctly
Dec 4 08:17:49 host-01 kernel: [ 1489.974535] [PF_RING] registered
/proc/net/pf_ring/
Dec 4 08:17:49 host-01 kernel: [ 1489.975093] [PF_RING] successfully
allocated 995328 bytes at 0xffffc20013562000
Dec 4 08:17:49 host-01 kernel: [ 1489.975096] [PF_RING] allocated 4111
slots [slot_len=242][tot_mem=995328]
Dec 4 08:17:49 host-01 kernel: [ 1489.975226] device eth14 entered
promiscuous mode
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] Modules linked in: pf_ring
ixgbe ioatdma msr isofs udf crc_itu_t video output input_polldev lp parport
iTCO_wdt iTCO_vendor_support psmouse pcspkr serio_raw tpm_infineon tpm
i7300_idle joydev tpm_bios shpchp dca ses enclosure usbhid usb_storage
qla2xxx scsi_transport_fc scsi_tgt mptsas niu mptscsih mptbase e1000e
scsi_transport_sas fbcon tileblit font bitblit softcursor [last unloaded:
ixgbe]
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] CPU 0:
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] Modules linked in: pf_ring
ixgbe ioatdma msr isofs udf crc_itu_t video output input_polldev lp parport
iTCO_wdt iTCO_vendor_support psmouse pcspkr serio_raw tpm_infineon tpm
i7300_idle joydev tpm_bios shpchp dca ses enclosure usbhid usb_storage
qla2xxx scsi_transport_fc scsi_tgt mptsas niu mptscsih mptbase e1000e
scsi_transport_sas fbcon tileblit font bitblit softcursor [last unloaded:
ixgbe]
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] Pid: 4851, comm:
tnapi(eth14.0) Not tainted 2.6.28-11-server #42-Ubuntu
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] RIP:
0010:[<ffffffffa0255c92>] [<ffffffffa0255c92>]
ixgbe_clean_rxtx_many+0x232/0x2d0 [ixgbe]
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] RSP: 0018:ffffffff80a9ae40
EFLAGS: 00000202
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] RAX: ffff88201ac82320 RBX:
ffffffff80a9aea0 RCX: ffff88201ac82310
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] RDX: ffff88201ac82490 RSI:
0000000000000010 RDI: 0000000000000202
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] RBP: ffffffff80a9adc0 R08:
0000000000000010 R09: ffff88201ac82358
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] R10: ffff8800a75dd000 R11:
0000000000000000 R12: ffffffff80213668
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] R13: ffffffff80a9adc0 R14:
0000000000000001 R15: 0000000000000009
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] FS: 0000000000000000(0000)
GS:ffffffff80aa3000(0000) knlGS:0000000000000000
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] CS: 0010 DS: 0018 ES: 0018
CR0: 000000008005003b
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] CR2: 00007f8a00d25540 CR3:
00000020140b8000 CR4: 00000000000006a0
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] DR0: 0000000000000000 DR1:
0000000000000000 DR2: 0000000000000000
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] DR3: 0000000000000000 DR6:
00000000ffff0ff0 DR7: 0000000000000400
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] Call Trace:
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] <IRQ> [<ffffffff805b6ae4>]
? net_rx_action+0x104/0x240
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80256a5c>] ?
__do_softirq+0x9c/0x170
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80213d8c>] ?
call_softirq+0x1c/0x30
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80214ffd>] ?
do_softirq+0x5d/0xa0
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff802567dd>] ?
irq_exit+0x8d/0xa0
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff802152c5>] ?
do_IRQ+0xc5/0x110
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80212bf3>] ?
ret_from_intr+0x0/0x29
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] <EOI> [<ffffffff805c8411>]
? eth_type_trans+0x1/0x170
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffffa0252123>] ?
pkt_poll_thread+0x1d3/0x5d0 [ixgbe]
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80268890>] ?
autoremove_wake_function+0x0/0x40
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffffa0251f50>] ?
pkt_poll_thread+0x0/0x5d0 [ixgbe]
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80268429>] ?
kthread+0x49/0x90
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff80213979>] ?
child_rip+0xa/0x11
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff802683e0>] ?
kthread+0x0/0x90
Dec 4 08:18:54 host-01 kernel: [ 1554.780009] [<ffffffff8021396f>] ?
child_rip+0x0/0x11
[...snipped...]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listgateway.unipi.it/pipermail/ntop-misc/attachments/20091204/965061e4/attachment.htm>
More information about the Ntop-misc
mailing list