[Ntop] how to monitor http and https only

Gary Gatten Ggatten at waddell.com
Tue Apr 27 23:08:23 CEST 2010 

Ah, I see....  You just want to see if the users are "surfing" or actually working?  Not sure if nTop will give you this.  The Domain report will have some of this info, and rrd may actually store this as well.  I'm just not sure it will provide exactly what you seek.  What about "IP -> Summary -> Internet Domain"?, then drill down from there?  If this report will work for you, maybe run a script with several "wget" on the appropriate URL's and save those each night?  Perhaps you could enable sticky hosts and then run a cron job that restarts nTop at midnight (or whenever) each night?

Are you wanting something like "WebSense" - that records every url visited, the time of day, the time spent at each site, etc.?  You may want to check out "OpenDNS".  They offer similar service for tracking this type of info and it's not "too" expensive for small number of users.  I'm sure there is Open Source stuff that will do this as well - I just don't know of any.

nTop may be able to get what you want - it for sure will capture the data, I just don't know of a predefined "report" that will show exactly what you want.  NTop is good at lots of things, but isn't a perfect fit for everything.

Maybe someone else will have other ideas.  In the mean time I recommend you play with nTop's options a little and see if you can get what you need without being too convoluted.  

-----Original Message-----
From: ntop-bounces at listgateway.unipi.it [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
Sent: Tuesday, April 27, 2010 3:47 PM
To: ntop at unipi.it
Subject: Re: [Ntop] how to monitor http and https only

The restrictions are done with iptables. There are only 4 hosts with 
internet access(http and https only), 1 with full access(the manager) 
and 1 test machine with full access. The rest are limited to antivirus 
updates.

I want to save which sites were visited by the users each day. I need 2 
type of reports: by local IP and by remote hosts.

Istvan

On 27.04.2010 23:35, Gary Gatten wrote:
> With Sticky hosts, idle hosts are never purged from memory.  Therefore, every new host will take more and more until it runs out.  Depending on the number of hosts, I can't tell you if 256MB will be enough or not.  My guess is not.
>
> Maybe Wireshark is all you need?  A capture filter will limit your traffic to http (or whatever) and you can tell it to create a new file every hour / 100MB / whatever.  Then, some of the summary reports may give the info you need.  If you don't capture DNS traffic you may have a hard time reconciling host ip's to urls, so keep that in mind.
>
> If you're trying to solve a specific problem or answer a specific question, perhaps post that?
>
> G
>
>
>
>
> -----Original Message-----
> From: ntop-bounces at listgateway.unipi.it [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
> Sent: Tuesday, April 27, 2010 3:29 PM
> To: ntop at unipi.it
> Subject: Re: [Ntop] how to monitor http and https only
>
> After all I don't even need graphs, but everywhere I looked, everybody
> is suggesting ntop, or maybe I'm not asking the right questions.
>
> What do you mean by "ntop memory usage continue to grow". The system
> running ntop is a piece of junk, with 256MB ram. Will it crash within 24h?
>
> Istvan
>
> On 27.04.2010 23:05, Gary Gatten wrote:
>    
>> Sounds right. Beware: enabling sticky hosts will cause ntop memory usage to continue to grow until: ntop is restarted, or ntop crashes from a malloc error.
>>
>> There is probably a way to use "wget" and / or other tools to "download" reports from ntop and save them somewhere.  Then maybe you could set idle purge for say... 70 minutes, and run this batch report every hour?
>>
>> I think I understand what you're trying to do as I often need the same thing.  You may want to spend a few minutes looking at the "rrd" settings.  There may be some combination of "Data to Dump" and "RRD Detail" that will do what you wish.  I've played with these settings some, but it's been a long time so can't offer much guidance.  There are several good docs on the web that give details on what these settings do.  If you can get RRD to store the data you wish, you can then use the "Arbitrary Graph" option to fetch / display that data.  My initial thought is rrd will NOT store "conversation" level info, but who knows - maybe somewhere in there you'll get what you need?  You could start be enabling all rrd data sets at the "high" level.
>>
>> G
>>
>>
>> -----Original Message-----
>> From: ntop-bounces at listgateway.unipi.it [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>> Sent: Tuesday, April 27, 2010 2:47 PM
>> To: ntop at unipi.it
>> Subject: Re: [Ntop] how to monitor http and https only
>>
>> This means, if I want to see what web pages were opened by one specific
>> user(local IP), I need to enable "sticky hosts" or I need to increase
>> purge hosts to 12 hours, right?
>>
>> I'll try with sticky hosts. That seems to be the closest to what I need.
>>
>> Istvan
>>
>> On 27.04.2010 18:27, Gary Gatten wrote:
>>
>>      
>>> You may be speaking of two different issues:
>>> 1.) How nTop determines which hosts are local and which are remote
>>> 2.) Idle host purge timers
>>>
>>> First, please make sure you specify "-m all your local network ranges" on the command line.  Or add via the GUI.  This is the only way ntop knows local from remote.  Anything not defined as local is considered remote.
>>>
>>> Next, the default idle host purge is 5 minutes.  You have two options that I know of:
>>> 	1.) Enable "sticky hosts" - which as implies hosts will never go away until you restart nTop.  Only recommended in unique environments.
>>> 	2.) Change the idle purge time in "globals-defines.h" and recompile 	nTop.
>>>
>>> Not sure which settings over ride which.  If you make a change to the startup options, you must restart ntop and most/all recorded traffic will be lost.  If done by the GUI, some settings are dynamic, I can't say for sure which ones.  I think the GUI settings are saved in the prefsCache.db file.
>>>
>>>
>>>
>>> -----Original Message----
>>> From: ntop-bounces at listgateway.unipi.it [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>> Sent: Tuesday, April 27, 2010 10:06 AM
>>> To: ntop at unipi.it
>>> Subject: Re: [Ntop] how to monitor http and https only
>>>
>>> Even if I choose All protocols -->    Traffic . I choose Hosts: All , I
>>> can't see all the remote hosts. But for a while I could see some remote
>>> hosts which than disappeared. What is the effective time range for All
>>> protocols -->    Traffic ?
>>> Where are the parameters saved if I use the web interface for changing
>>> the configuration(Admin-->Configure-->Startup options)?
>>> I noticed that if I modify /etc/ntop.conf it overrides the web config
>>> settings. Is that right?
>>> If I modify the /etc/ntop.conf, how can I make the settings effective
>>> without losing the recorded traffic?
>>>
>>> On 27.04.2010 17:45, Gary Gatten wrote:
>>>
>>>
>>>        
>>>> There's a startup arg to specify which network ranges are local, it might be -b? Check the man and make sure you have this configured correctly for your environment.
>>>>
>>>> ----- Original Message -----
>>>> From: ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>> To: ntop at unipi.it<ntop at unipi.it>
>>>> Sent: Tue Apr 27 09:38:42 2010
>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>
>>>> Ok, I got confused. Ntop is set on my Centos router. All the internet
>>>> traffic goes through it.
>>>> I go on the web interface All protocols -->     Traffic . I choose Hosts:
>>>> Remote only and I see only some of the remote hosts. I don't understand.
>>>> Where can I see all the remote hosts which were accessed today?
>>>>
>>>> Istvan
>>>>
>>>> On 26.04.2010 18:34, Gary Gatten wrote:
>>>>
>>>>
>>>>
>>>>          
>>>>> You can't disable "everything", but with packet and protocol filters, and by viewing specific reports - you can get pretty close to what you need.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>>> To: ntop at unipi.it<ntop at unipi.it>
>>>>> Sent: Mon Apr 26 09:31:35 2010
>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>
>>>>> Thanks for the hints. But there is still too much information.
>>>>> All I want is:
>>>>> - 192.168.0.xxx, between 08:00-14:00, accessed the following sites: ...
>>>>> - www.facebook.com, between 08:00-14:00, was accessed by the following
>>>>> local IP-s: ...
>>>>>
>>>>> I don't need the:
>>>>> - Host Traffic Stats
>>>>> - Packet Statistics
>>>>> - Protocol Distribution
>>>>> - TCP/UDP Recently Used Ports
>>>>> - IP Service Stats: Client Role
>>>>> - TCP/UDP - Traffic on Other Ports
>>>>>
>>>>> How can I do all these?
>>>>>
>>>>> Istvan
>>>>>
>>>>> On 26.04.2010 17:12, Gary Gatten wrote:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>            
>>>>>> Good call. One can also restrict the displayed protocols with -p, all remaining traffic will be displayed as "other"
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> From: ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>>>> To: ntop at unipi.it<ntop at unipi.it>; ntop at listgateway.unipi.it<ntop at listgateway.unipi.it>
>>>>>> Sent: Mon Apr 26 08:44:04 2010
>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>
>>>>>> Have you taken a look at the  manpages for ntop? On a unix system, the "-B" switch followed by a pcap expression will give you want you want.
>>>>>>
>>>>>> e.g
>>>>>>
>>>>>> ntop -d -w 8080 -B "port 80 or 443"
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: ntop-bounces at listgateway.unipi.it [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>>>>> Sent: Monday, April 26, 2010 9:40 AM
>>>>>> To: ntop at listgateway.unipi.it
>>>>>> Subject: [Ntop] how to monitor http and https only
>>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I just installed ntop and it gives me much more information I need. I
>>>>>> would like to see only the traffic on ports 80 and 443.
>>>>>> How can I do that?
>>>>>>
>>>>>> Istvan
>>>>>> _______________________________________________
>>>>>> Ntop mailing list
>>>>>> Ntop at listgateway.unipi.it
>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>> _______________________________________________
>>>>>> Ntop mailing list
>>>>>> Ntop at listgateway.unipi.it
>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>> _______________________________________________
>>>>>> Ntop mailing list
>>>>>> Ntop at listgateway.unipi.it
>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>              
>>>>> _______________________________________________
>>>>> Ntop mailing list
>>>>> Ntop at listgateway.unipi.it
>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>> _______________________________________________
>>>>> Ntop mailing list
>>>>> Ntop at listgateway.unipi.it
>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>            
>>>> _______________________________________________
>>>> Ntop mailing list
>>>> Ntop at listgateway.unipi.it
>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>> _______________________________________________
>>>> Ntop mailing list
>>>> Ntop at listgateway.unipi.it
>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>
>>>>
>>>>
>>>>          
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop at listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop at listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>>
>>>
>>>        
>> _______________________________________________
>> Ntop mailing list
>> Ntop at listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>> _______________________________________________
>> Ntop mailing list
>> Ntop at listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
>>
>>      
> _______________________________________________
> Ntop mailing list
> Ntop at listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
> _______________________________________________
> Ntop mailing list
> Ntop at listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
>    

_______________________________________________
Ntop mailing list
Ntop at listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop

More information about the Ntop mailing list