[Ntop-misc] why is PF_RING limited to only allocating up to 4MB ofring slots

Tue Jun 27 14:51:34 CEST 2006

> I think you may be speaking of the 4/gb/ limit instead of /mb/.

Nope.  This kernel has the 64 GB option enabled.

I'm definitely refering to the max amount of memory that will 
be allocated for ring slots.  I've modified the # of slots via 
modprobe.conf.  That definitely works for me.  However, after a
certain point, the amount of memory (and hence the # of slots 
available for buffering) is capped at 4 MB due to __get_free_pages
not returning anything larger than 4 MB.

Since that is a low level kernel call, probably not much you can do
to change its behaviour.  However, PF_RING could  probably be modified
to create multiple large chunks of memory.  Rings slots within each chunk
would be contiguous but at chunk boundaries there would be a gap.  

This would mean pf_ring would make multiple calls to __get_free_pages and 
have to manage these multiple chunks - tracking which ring slots are on
which chunks.  Not sure if this is even possible since the device driver 
is writing to this memory...  Hummm...  Maybe not so easy unless it is 
possible to make the device driver aware of multiple blocks of memory.

Anyways, I'm not a kernel hacker...  Hoping for some guidance on this 
point (can it be done?).  If someone with more knowledge thinks it can 
be done, I might try my hand at it...

-Rob

------------------------------------------------------------------
Re: [Ntop-misc] why is PF_RING limited to only allocating up to 4MB ofring slots

Nick Whitehill
Mon, 26 Jun 2006 19:50:27 -0700

Hi,

I think you may be speaking of the 4/gb/ limit instead
of /mb/.

You can add additional ring slots in modprobe.conf. I
cannot remember the syntax but will report back when I
get to work tomorrow. I get the slotsize vs ring size
confused but I know that once we moved to 16384k
ringsize we no longer dropped packets. We are now
running multiple instances of snort on several
interfaces so we were able to remove the 16384 entry
in modprobe.conf as each instance of a libpcap'd
program calls an additional "ring buffer". 

Luca, please correct me if the above statement is off
the mark. I have been using PF_RING with tcpdump and
snort for over a year now and I am only reporting my
experience. 

Thanks again for the wiki and thanks to Luca for
PF_RING!!

Cheers,
Nick

--- Amitava Biswas <[EMAIL PROTECTED]> wrote:

> why is PF_RING limited to only allocating up to 4MB
> of ring slotsWell I do not know much about 2.6
> kernels,
> but a few observations which might be the cause-
> 1) The get free pages routine has non deterministic
> behavior,  I mean its a best effort process, when I
> wanted to get very big chunks of contiguous blocks,
> the process failed because there wasnt any such
> bloocks left. however it runs ok for smaller chunks,
> its expected as bigger contiguos blocks may not be
> available at all, 
> 
> 2) There may be a limit programed in the process
> itself, because 4 Mb was the kernel memory limit for
> older kernels, though it should be taken care off if
> you enable larger memory support option during
> kernel compile.
> 
> and by the way you can use a different option to
> reduce memory requirement, try my posting on DMA
> ring, i wrote couple of papers on it and notified
> the user group about that approach, this approach
> cut down the memory requirement drasticaly and
> improved the performance on the other hand.
> 
> allocating big memory blocks might create problem to
> other processes or kernel activities, so should take
> care about it.
>   ----- Original Message ----- 
>   From: Rob Willis 
>   To: ntop-misc at listgateway.unipi.it 
>   Sent: Monday, June 26, 2006 6:24 PM
>   Subject: [Ntop-misc] why is PF_RING limited to
> only allocating up to 4MB ofring slots
> 
> 
> 
> 
>   (Hopefully, this doesnt turn into a double post
> since I sent an earlier email via my home email
> account by accident).
> 
>   I've been playing with PF_RING (with positive
> results).  However it seems to be limited to only
> allocating enough slots up to 4MB of memory.  I'm on
> FedoraCore4, 2.6.12.
> 
>   I took a look at the RF_RING code and it is
> calling __get_free_pages(priority,order) where order
> represents the number of kernel memory pages to
> return: 2^order.  So pages returned:
> 
>   order=0:  1 page
>   order=1:  2 pages
>   order=2:  4 pages
>   ...
>   And in my case, PF_RING decides on an order of 10:
>   order=10: 1024 pages
> 
>   Since each page is 4kb, that means it returns 4 MB
> of memory for use as ring slots.  My ring slot size
> is set to 1518 (just big enough for an 802.1q
> packet).
> 
>   Anyways, even when I request 8MB or 16 MB, this
> function always returns 4 MB (ie it always uses an
> order of 10).  Is there another option to get
> PF_RING to allocate more memory?  I've got 8 GB on
> this system and need to do some fairly indepth
> analysis of packets.  So would like to have a very
> large buffer allocated.
> 
>   If there is some reason that __get_free_pages can
> only return 4 MB blocks, perhaps PF_RING could be
> reworked to manage multiple large blocks.  Then
> could call __get_free_pages multiple times and put
> all of the blocks in a array.  Would mean an extra
> level of indirection when referencing a ring slot,
> but that doesnt seem like that much more overhead. 
> 
>   I might even be willing to dive in and come up
> with a patch to do just that if someone told me that
> was even possible.  And if they said there wasnt an
> easier way to get PF_RING to allocate more memory.
> 
>   -Rob
> 
> 
> 
> 
>
------------------------------------------------------------------------------
> 
> 
>   _______________________________________________
>   Ntop-misc mailing list
>   Ntop-misc at listgateway.unipi.it
>  
>
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> 
> > _______________________________________________
> Ntop-misc mailing list
> Ntop-misc at listgateway.unipi.it
>
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
> 

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 
_______________________________________________
Ntop-misc mailing list
Ntop-misc at listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listgateway.unipi.it/pipermail/ntop-misc/attachments/20060627/dd19af8c/attachment.htm