[Ntop] how to monitor http and https only
Istvan Köpe
istvan at advancetech.ro
Wed Apr 28 17:48:01 CEST 2010
Maybe, but not as transparent proxy
On 28.04.2010 17:42, Fabio Pardi wrote:
> you need squid 2.5+ i guess
>
> http://www.comfsm.fm/computing/squid/FAQ-1.html#ss1.12
>
>
> Istvan Köpe wrote:
>
>> I just learned that squid doesn't handle https. So is not good for what
>> I need.
>>
>> On 28.04.2010 15:51, Gary Gatten wrote:
>>
>>> I'm sure iptables can log most everything as well. Someone has
>>> probably written an app to format the logs and summarize the data.
>>>
>>> ----- Original Message -----
>>> From:
>>> ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>> To: ntop at unipi.it<ntop at unipi.it>
>>> Sent: Wed Apr 28 07:37:52 2010
>>> Subject: Re: [Ntop] how to monitor http and https only
>>>
>>> Once I had contact with squid and than I realized that squid is a whole
>>> chapter in Linux... If is possible I prefer to not install any proxy.
>>>
>>> I.
>>>
>>> On 28.04.2010 13:55, Steve Clark wrote:
>>>
>>>
>>>> Or maybe something like squid proxy. I am pretty sure it keeps an
>>>> access.log that could provide
>>>> the info you are looking for.
>>>>
>>>> On 04/27/2010 05:08 PM, Gary Gatten wrote:
>>>>
>>>>
>>>>> Ah, I see.... You just want to see if the users are "surfing" or
>>>>> actually working? Not sure if nTop will give you this. The Domain
>>>>> report will have some of this info, and rrd may actually store this
>>>>> as well. I'm just not sure it will provide exactly what you seek.
>>>>> What about "IP -> Summary -> Internet Domain"?, then drill down
>>>>> from there? If this report will work for you, maybe run a script
>>>>> with several "wget" on the appropriate URL's and save those each
>>>>> night? Perhaps you could enable sticky hosts and then run a cron job
>>>>> that restarts nTop at midnight (or whenever) each night?
>>>>>
>>>>> Are you wanting something like "WebSense" - that records every url
>>>>> visited, the time of day, the time spent at each site, etc.? You may
>>>>> want to check out "OpenDNS". They offer similar service for tracking
>>>>> this type of info and it's not "too" expensive for small number of
>>>>> users. I'm sure there is Open Source stuff that will do this as well
>>>>> - I just don't know of any.
>>>>>
>>>>> nTop may be able to get what you want - it for sure will capture the
>>>>> data, I just don't know of a predefined "report" that will show
>>>>> exactly what you want. NTop is good at lots of things, but isn't a
>>>>> perfect fit for everything.
>>>>>
>>>>> Maybe someone else will have other ideas. In the mean time I
>>>>> recommend you play with nTop's options a little and see if you can
>>>>> get what you need without being too convoluted.
>>>>>
>>>>> -----Original Message-----
>>>>> From: ntop-bounces at listgateway.unipi.it
>>>>> [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>>>> Sent: Tuesday, April 27, 2010 3:47 PM
>>>>> To: ntop at unipi.it
>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>
>>>>> The restrictions are done with iptables. There are only 4 hosts with
>>>>> internet access(http and https only), 1 with full access(the manager)
>>>>> and 1 test machine with full access. The rest are limited to antivirus
>>>>> updates.
>>>>>
>>>>> I want to save which sites were visited by the users each day. I need 2
>>>>> type of reports: by local IP and by remote hosts.
>>>>>
>>>>> Istvan
>>>>>
>>>>> On 27.04.2010 23:35, Gary Gatten wrote:
>>>>>
>>>>>
>>>>>> With Sticky hosts, idle hosts are never purged from memory.
>>>>>> Therefore, every new host will take more and more until it runs
>>>>>> out. Depending on the number of hosts, I can't tell you if 256MB
>>>>>> will be enough or not. My guess is not.
>>>>>>
>>>>>> Maybe Wireshark is all you need? A capture filter will limit your
>>>>>> traffic to http (or whatever) and you can tell it to create a new
>>>>>> file every hour / 100MB / whatever. Then, some of the summary
>>>>>> reports may give the info you need. If you don't capture DNS
>>>>>> traffic you may have a hard time reconciling host ip's to urls, so
>>>>>> keep that in mind.
>>>>>>
>>>>>> If you're trying to solve a specific problem or answer a specific
>>>>>> question, perhaps post that?
>>>>>>
>>>>>> G
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: ntop-bounces at listgateway.unipi.it
>>>>>> [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>>>>> Sent: Tuesday, April 27, 2010 3:29 PM
>>>>>> To: ntop at unipi.it
>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>
>>>>>> After all I don't even need graphs, but everywhere I looked, everybody
>>>>>> is suggesting ntop, or maybe I'm not asking the right questions.
>>>>>>
>>>>>> What do you mean by "ntop memory usage continue to grow". The system
>>>>>> running ntop is a piece of junk, with 256MB ram. Will it crash
>>>>>> within 24h?
>>>>>>
>>>>>> Istvan
>>>>>>
>>>>>> On 27.04.2010 23:05, Gary Gatten wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Sounds right. Beware: enabling sticky hosts will cause ntop memory
>>>>>>> usage to continue to grow until: ntop is restarted, or ntop crashes
>>>>>>> from a malloc error.
>>>>>>>
>>>>>>> There is probably a way to use "wget" and / or other tools to
>>>>>>> "download" reports from ntop and save them somewhere. Then maybe
>>>>>>> you could set idle purge for say... 70 minutes, and run this batch
>>>>>>> report every hour?
>>>>>>>
>>>>>>> I think I understand what you're trying to do as I often need the
>>>>>>> same thing. You may want to spend a few minutes looking at the
>>>>>>> "rrd" settings. There may be some combination of "Data to Dump"
>>>>>>> and "RRD Detail" that will do what you wish. I've played with
>>>>>>> these settings some, but it's been a long time so can't offer much
>>>>>>> guidance. There are several good docs on the web that give details
>>>>>>> on what these settings do. If you can get RRD to store the data
>>>>>>> you wish, you can then use the "Arbitrary Graph" option to fetch /
>>>>>>> display that data. My initial thought is rrd will NOT store
>>>>>>> "conversation" level info, but who knows - maybe somewhere in there
>>>>>>> you'll get what you need? You could start be enabling all rrd data
>>>>>>> sets at the "high" level.
>>>>>>>
>>>>>>> G
>>>>>>>
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: ntop-bounces at listgateway.unipi.it
>>>>>>> [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>>>>>> Sent: Tuesday, April 27, 2010 2:47 PM
>>>>>>> To: ntop at unipi.it
>>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>>
>>>>>>> This means, if I want to see what web pages were opened by one
>>>>>>> specific
>>>>>>> user(local IP), I need to enable "sticky hosts" or I need to increase
>>>>>>> purge hosts to 12 hours, right?
>>>>>>>
>>>>>>> I'll try with sticky hosts. That seems to be the closest to what I
>>>>>>> need.
>>>>>>>
>>>>>>> Istvan
>>>>>>>
>>>>>>> On 27.04.2010 18:27, Gary Gatten wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> You may be speaking of two different issues:
>>>>>>>> 1.) How nTop determines which hosts are local and which are remote
>>>>>>>> 2.) Idle host purge timers
>>>>>>>>
>>>>>>>> First, please make sure you specify "-m all your local network
>>>>>>>> ranges" on the command line. Or add via the GUI. This is the
>>>>>>>> only way ntop knows local from remote. Anything not defined as
>>>>>>>> local is considered remote.
>>>>>>>>
>>>>>>>> Next, the default idle host purge is 5 minutes. You have two
>>>>>>>> options that I know of:
>>>>>>>> 1.) Enable "sticky hosts" - which as implies hosts will never
>>>>>>>> go away until you restart nTop. Only recommended in unique
>>>>>>>> environments.
>>>>>>>> 2.) Change the idle purge time in "globals-defines.h" and
>>>>>>>> recompile nTop.
>>>>>>>>
>>>>>>>> Not sure which settings over ride which. If you make a change to
>>>>>>>> the startup options, you must restart ntop and most/all recorded
>>>>>>>> traffic will be lost. If done by the GUI, some settings are
>>>>>>>> dynamic, I can't say for sure which ones. I think the GUI
>>>>>>>> settings are saved in the prefsCache.db file.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> -----Original Message----
>>>>>>>> From: ntop-bounces at listgateway.unipi.it
>>>>>>>> [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan Köpe
>>>>>>>> Sent: Tuesday, April 27, 2010 10:06 AM
>>>>>>>> To: ntop at unipi.it
>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>>>
>>>>>>>> Even if I choose All protocols --> Traffic . I choose Hosts:
>>>>>>>> All , I
>>>>>>>> can't see all the remote hosts. But for a while I could see some
>>>>>>>> remote
>>>>>>>> hosts which than disappeared. What is the effective time range for
>>>>>>>> All
>>>>>>>> protocols --> Traffic ?
>>>>>>>> Where are the parameters saved if I use the web interface for
>>>>>>>> changing
>>>>>>>> the configuration(Admin-->Configure-->Startup options)?
>>>>>>>> I noticed that if I modify /etc/ntop.conf it overrides the web
>>>>>>>> config
>>>>>>>> settings. Is that right?
>>>>>>>> If I modify the /etc/ntop.conf, how can I make the settings
>>>>>>>> effective
>>>>>>>> without losing the recorded traffic?
>>>>>>>>
>>>>>>>> On 27.04.2010 17:45, Gary Gatten wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>> There's a startup arg to specify which network ranges are local,
>>>>>>>>> it might be -b? Check the man and make sure you have this
>>>>>>>>> configured correctly for your environment.
>>>>>>>>>
>>>>>>>>> ----- Original Message -----
>>>>>>>>> From:
>>>>>>>>> ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>>>>>>>
>>>>>>>>> To: ntop at unipi.it<ntop at unipi.it>
>>>>>>>>> Sent: Tue Apr 27 09:38:42 2010
>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>>>>
>>>>>>>>> Ok, I got confused. Ntop is set on my Centos router. All the
>>>>>>>>> internet
>>>>>>>>> traffic goes through it.
>>>>>>>>> I go on the web interface All protocols --> Traffic . I
>>>>>>>>> choose Hosts:
>>>>>>>>> Remote only and I see only some of the remote hosts. I don't
>>>>>>>>> understand.
>>>>>>>>> Where can I see all the remote hosts which were accessed today?
>>>>>>>>>
>>>>>>>>> Istvan
>>>>>>>>>
>>>>>>>>> On 26.04.2010 18:34, Gary Gatten wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>> You can't disable "everything", but with packet and protocol
>>>>>>>>>> filters, and by viewing specific reports - you can get pretty
>>>>>>>>>> close to what you need.
>>>>>>>>>>
>>>>>>>>>> ----- Original Message -----
>>>>>>>>>> From:
>>>>>>>>>> ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> To: ntop at unipi.it<ntop at unipi.it>
>>>>>>>>>> Sent: Mon Apr 26 09:31:35 2010
>>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>>>>>
>>>>>>>>>> Thanks for the hints. But there is still too much information.
>>>>>>>>>> All I want is:
>>>>>>>>>> - 192.168.0.xxx, between 08:00-14:00, accessed the following
>>>>>>>>>> sites: ...
>>>>>>>>>> - www.facebook.com, between 08:00-14:00, was accessed by the
>>>>>>>>>> following
>>>>>>>>>> local IP-s: ...
>>>>>>>>>>
>>>>>>>>>> I don't need the:
>>>>>>>>>> - Host Traffic Stats
>>>>>>>>>> - Packet Statistics
>>>>>>>>>> - Protocol Distribution
>>>>>>>>>> - TCP/UDP Recently Used Ports
>>>>>>>>>> - IP Service Stats: Client Role
>>>>>>>>>> - TCP/UDP - Traffic on Other Ports
>>>>>>>>>>
>>>>>>>>>> How can I do all these?
>>>>>>>>>>
>>>>>>>>>> Istvan
>>>>>>>>>>
>>>>>>>>>> On 26.04.2010 17:12, Gary Gatten wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> Good call. One can also restrict the displayed protocols with
>>>>>>>>>>> -p, all remaining traffic will be displayed as "other"
>>>>>>>>>>>
>>>>>>>>>>> ----- Original Message -----
>>>>>>>>>>> From:
>>>>>>>>>>> ntop-bounces at listgateway.unipi.it<ntop-bounces at listgateway.unipi.it>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> To: ntop at unipi.it<ntop at unipi.it>;
>>>>>>>>>>> ntop at listgateway.unipi.it<ntop at listgateway.unipi.it>
>>>>>>>>>>> Sent: Mon Apr 26 08:44:04 2010
>>>>>>>>>>> Subject: Re: [Ntop] how to monitor http and https only
>>>>>>>>>>>
>>>>>>>>>>> Have you taken a look at the manpages for ntop? On a unix
>>>>>>>>>>> system, the "-B" switch followed by a pcap expression will give
>>>>>>>>>>> you want you want.
>>>>>>>>>>>
>>>>>>>>>>> e.g
>>>>>>>>>>>
>>>>>>>>>>> ntop -d -w 8080 -B "port 80 or 443"
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: ntop-bounces at listgateway.unipi.it
>>>>>>>>>>> [mailto:ntop-bounces at listgateway.unipi.it] On Behalf Of Istvan
>>>>>>>>>>> Köpe
>>>>>>>>>>> Sent: Monday, April 26, 2010 9:40 AM
>>>>>>>>>>> To: ntop at listgateway.unipi.it
>>>>>>>>>>> Subject: [Ntop] how to monitor http and https only
>>>>>>>>>>>
>>>>>>>>>>> Hello,
>>>>>>>>>>>
>>>>>>>>>>> I just installed ntop and it gives me much more information I
>>>>>>>>>>> need. I
>>>>>>>>>>> would like to see only the traffic on ports 80 and 443.
>>>>>>>>>>> How can I do that?
>>>>>>>>>>>
>>>>>>>>>>> Istvan
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ntop mailing list
>>>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ntop mailing list
>>>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ntop mailing list
>>>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Ntop mailing list
>>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Ntop mailing list
>>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Ntop mailing list
>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>> _______________________________________________
>>>>>>>>> Ntop mailing list
>>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ntop mailing list
>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>> _______________________________________________
>>>>>>>> Ntop mailing list
>>>>>>>> Ntop at listgateway.unipi.it
>>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ntop mailing list
>>>>>>> Ntop at listgateway.unipi.it
>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>> _______________________________________________
>>>>>>> Ntop mailing list
>>>>>>> Ntop at listgateway.unipi.it
>>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>>>> Ntop mailing list
>>>>>> Ntop at listgateway.unipi.it
>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>> _______________________________________________
>>>>>> Ntop mailing list
>>>>>> Ntop at listgateway.unipi.it
>>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Ntop mailing list
>>>>> Ntop at listgateway.unipi.it
>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>> _______________________________________________
>>>>> Ntop mailing list
>>>>> Ntop at listgateway.unipi.it
>>>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop at listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>> _______________________________________________
>>> Ntop mailing list
>>> Ntop at listgateway.unipi.it
>>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>>
>>>
>> _______________________________________________
>> Ntop mailing list
>> Ntop at listgateway.unipi.it
>> http://listgateway.unipi.it/mailman/listinfo/ntop
>>
> _______________________________________________
> Ntop mailing list
> Ntop at listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
>
More information about the Ntop
mailing list