[Ntop-misc] why is PF_RING limited to only allocating up to
4MBofring slots
Rob Willis
rob.willis at quest.com
Tue Jun 27 19:46:52 CEST 2006
Hi Luca,
Thanks for the response. I agree that an application has to keep up. In our app, we maintain ongoing performance and volume statistics. Normal operation mode is usually no problem. Where we hit difficulties is when we dump aggregate information on 5 minute intervals. Which then gets stored in a long-term database. On those intervals, lots of other processes kick in and our data collector suddenly starts dropping packets. Seems like the easy solution is to just allocate much larger buffers to cope with these periodic high CPU utilization bursts. We also have problems when someone runs a complex query and the database starts churning. Larger buffers would buy us more time for those queries to complete.
Also, seems like most folks are just doing basic packet counts and such so they only need the first 120 or so bytes of each packet. We are doing full TCP packet reassembly and SSL decryption to extract HTTP header information. So we have to keep the entire packet. So that 4 MB block gets used up fast when you are looking at a gigabit stream of 1500 byte packets.
So I'm definitely interested in working on or help testing any kind of change that would extend PF_RING to use more than a 4 MB buffer.
-Rob
----------------------------------------------
Hi all,
the free slots are used to store packets until an application polls it.
You can increase it of course (we can find a solution to the 4MB
problem) but the point is that this is temporarely storage. It means
that your application must fetch packets anyway and that the buffer is
used just to give enough time to the application to do the job. So,
increasing the number of slots can help sometimes but in general if you
app is faster than the network, increasing the buffer size won't help much.
Cheers, Luca
Rob Willis wrote:
>>I think you may be speaking of the 4/gb/ limit instead of /mb/.
>>
>>
>
>Nope. This kernel has the 64 GB option enabled.
>
>I'm definitely refering to the max amount of memory that will
>be allocated for ring slots. I've modified the # of slots via
>modprobe.conf. That definitely works for me. However, after a
>certain point, the amount of memory (and hence the # of slots
>available for buffering) is capped at 4 MB due to __get_free_pages
>not returning anything larger than 4 MB.
>
>Since that is a low level kernel call, probably not much you can do
>to change its behaviour. However, PF_RING could probably be modified
>to create multiple large chunks of memory. Rings slots within each chunk
>would be contiguous but at chunk boundaries there would be a gap.
>
>This would mean pf_ring would make multiple calls to __get_free_pages and
>have to manage these multiple chunks - tracking which ring slots are on
>which chunks. Not sure if this is even possible since the device driver
>is writing to this memory... Hummm... Maybe not so easy unless it is
>possible to make the device driver aware of multiple blocks of memory.
>
>Anyways, I'm not a kernel hacker... Hoping for some guidance on this
>point (can it be done?). If someone with more knowledge thinks it can
>be done, I might try my hand at it...
>
>-Rob
>
>
>------------------------------------------------------------------
>Re: [Ntop-misc] why is PF_RING limited to only allocating up to 4MB ofring
>slots
>
>Nick Whitehill
>Mon, 26 Jun 2006 19:50:27 -0700
>
>Hi,
>
>I think you may be speaking of the 4/gb/ limit instead
>of /mb/.
>
>You can add additional ring slots in modprobe.conf. I
>cannot remember the syntax but will report back when I
>get to work tomorrow. I get the slotsize vs ring size
>confused but I know that once we moved to 16384k
>ringsize we no longer dropped packets. We are now
>running multiple instances of snort on several
>interfaces so we were able to remove the 16384 entry
>in modprobe.conf as each instance of a libpcap'd
>program calls an additional "ring buffer".
>
>Luca, please correct me if the above statement is off
>the mark. I have been using PF_RING with tcpdump and
>snort for over a year now and I am only reporting my
>experience.
>
>Thanks again for the wiki and thanks to Luca for
>PF_RING!!
>
>Cheers,
>Nick
>
>--- Amitava Biswas <[EMAIL PROTECTED]> wrote:
>
>
>
>>why is PF_RING limited to only allocating up to 4MB
>>of ring slotsWell I do not know much about 2.6
>>kernels,
>>but a few observations which might be the cause-
>>1) The get free pages routine has non deterministic
>>behavior, I mean its a best effort process, when I
>>wanted to get very big chunks of contiguous blocks,
>>the process failed because there wasnt any such
>>bloocks left. however it runs ok for smaller chunks,
>>its expected as bigger contiguos blocks may not be
>>available at all,
>>
>>2) There may be a limit programed in the process
>>itself, because 4 Mb was the kernel memory limit for
>>older kernels, though it should be taken care off if
>>you enable larger memory support option during
>>kernel compile.
>>
>>and by the way you can use a different option to
>>reduce memory requirement, try my posting on DMA
>>ring, i wrote couple of papers on it and notified
>>the user group about that approach, this approach
>>cut down the memory requirement drasticaly and
>>improved the performance on the other hand.
>>
>>allocating big memory blocks might create problem to
>>other processes or kernel activities, so should take
>>care about it.
>> ----- Original Message -----
>> From: Rob Willis
>> To: ntop-misc at listgateway.unipi.it
>> Sent: Monday, June 26, 2006 6:24 PM
>> Subject: [Ntop-misc] why is PF_RING limited to
>>only allocating up to 4MB ofring slots
>>
>>
>>
>>
>> (Hopefully, this doesnt turn into a double post
>>since I sent an earlier email via my home email
>>account by accident).
>>
>> I've been playing with PF_RING (with positive
>>results). However it seems to be limited to only
>>allocating enough slots up to 4MB of memory. I'm on
>>FedoraCore4, 2.6.12.
>>
>> I took a look at the RF_RING code and it is
>>calling __get_free_pages(priority,order) where order
>>represents the number of kernel memory pages to
>>return: 2^order. So pages returned:
>>
>> order=0: 1 page
>> order=1: 2 pages
>> order=2: 4 pages
>> ...
>> And in my case, PF_RING decides on an order of 10:
>> order=10: 1024 pages
>>
>> Since each page is 4kb, that means it returns 4 MB
>>of memory for use as ring slots. My ring slot size
>>is set to 1518 (just big enough for an 802.1q
>>packet).
>>
>> Anyways, even when I request 8MB or 16 MB, this
>>function always returns 4 MB (ie it always uses an
>>order of 10). Is there another option to get
>>PF_RING to allocate more memory? I've got 8 GB on
>>this system and need to do some fairly indepth
>>analysis of packets. So would like to have a very
>>large buffer allocated.
>>
>> If there is some reason that __get_free_pages can
>>only return 4 MB blocks, perhaps PF_RING could be
>>reworked to manage multiple large blocks. Then
>>could call __get_free_pages multiple times and put
>>all of the blocks in a array. Would mean an extra
>>level of indirection when referencing a ring slot,
>>but that doesnt seem like that much more overhead.
>>
>> I might even be willing to dive in and come up
>>with a patch to do just that if someone told me that
>>was even possible. And if they said there wasnt an
>>easier way to get PF_RING to allocate more memory.
>>
>> -Rob
>>
>>
>>
>>
>>
>>
>>
>------------------------------------------------------------------------------
>
>
>> _______________________________________________
>> Ntop-misc mailing list
>> Ntop-misc at listgateway.unipi.it
>>
>>
>>
>>
>http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>>>_______________________________________________
>>>
>>>
>>Ntop-misc mailing list
>>Ntop-misc at listgateway.unipi.it
>>
>>
>>
>http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
>
>__________________________________________________
>Do You Yahoo!?
>Tired of spam? Yahoo! Mail has the best spam protection around
>http://mail.yahoo.com
>_______________________________________________
>Ntop-misc mailing list
>Ntop-misc at listgateway.unipi.it
>http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Ntop-misc mailing list
>Ntop-misc at listgateway.unipi.it
>http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>
--
Luca Deri <[EMAIL PROTECTED]> http://luca.ntop.org/
skype://lucaderi/
Don't be encumbered by past history. Go off and do
something wonderful - Robert Noyce
_______________________________________________
Ntop-misc mailing list
Ntop-misc at listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listgateway.unipi.it/pipermail/ntop-misc/attachments/20060627/07957b51/attachment-0001.htm
More information about the Ntop-misc
mailing list