netflowanalyzer | Enterprise IT Management, Network performance management, IT Servicedesk, Desktop Management, Datacenter Management, Server Management, Log Analysis and Security Management, Network Tools, ManageEngine Blogs

NetFlow Analyzer, though the name says NetFlow, can work with quite a number of flow formats like sFlow, jFlow, NetStream, IPFIX etc. This blog will give you a brief idea on sFlow technology and also guide you on how to use NetFlow Analyzer with sFlow from HP Procurve devices.

What is sFlow?

sFlow is a monitoring technology which allows you to capture the traffic data from a switched or routed network to give complete visibility into the use of network bandwidth. This data helps in performance optimization, accounting/billing for usage, defense against security threats, capacity planning and much more.

sFlow datagrams are exported based on sampling due to which impact on the device CPU/Memory and available bandwidth is minimal. Based on a defined sampling rate, 1 out of N packets (where N is the sampling rate) is captured and sent to the NetFlow Analyzer for traffic analysis by the device. Though, this type of sampling does not provide 100% accurate statistics, it does provide a result with quantifiable accuracy.

sFlow analysis with NetFlow Analyzer:

NetFlow Analyzer can work with any devices which are capable of exporting NetFlow, sFlow and other compatible flow which are completely vendor dependent. You can check out the list of flow formats and devices with which NetFlow Analyzer can work from here.

HP Procurve and sFlow:

Just like Cisco has NetFlow and other vendors have thier flow formarts, some vendors use a technolgy called sFlow. HP Procurve devices are capable of exporting sFlow datagrams which can be used for bandwidth monitoring and traffic analysis. NetFlow Analyzer is capable of analyzing the sFlow datagram exported from the HP Procurve to give you the traffic statiscs on each active ports.

sFlow export on the HP procuve device can be configured using two different methods, We can enable sFlow on the HP device either by logging in to the router and configuring them for sFlow export. But this is available only in the older device models or OS.

On the new HP devices, sFlow can be enabled only through SNMP. To make the sFlow configuration on HP device a simple task, NetFlow Analyzer provides scripts to enable and disable the sFlow export. So, lets see how we can use the script and enable sFlow.

sFlow Enable utility:

The script to enable sFlow, named as sFlowEnable.bat (for Windows and .sh for Linux),  is present under <\AdventNet\ME\NetFlow\troubleshooting> directory.

The usage for the script is as follows:

SFlowEnable.bat switchIp snmpPort snmpWriteCommunity collectorIP collectorPort samplingRate

Example:-

C:\AdventNet\ME\NetFlow\troubleshooting>sFlowEnable.bat 192.168.188.30 161 private 192.168.133.1 9996 4096    

Once sFlow is enabled on the HP devices, NetFlow Analyzer server will receive the packets and the product will capture the packets to automatically generate the reports. You also need to ensure that no access control lists (ACLs) or firewalls block the NetFlow packets (on UDP 9996) and that even the software firewalls on the server are allowing the packets to reach the NetFlow Analyzer installation.

After enabling the sFlow on the HP devices, we need ensure a few points to get the accurate traffic statistics about the device in NetFlow Analyzer.

The first and foremost is the sampling rate. We suggest setting the sampling rate to 4096. We have observed from various setups and from our existing customers feedback that the sampling rate of 4096 gives the most accurate traffic statistics in NetFlow Analyzer.Most of the other sFlow collectors in the market suggest the sampling rate to 256 which means more number of exported sFlow datagrams. With a sampling rate of 4096, you get the additional benefit that the device is not being overloaded by sampling large number of datagrams and exporting to the NetFlow Analyzer.

Next point we need verify is the "sFlow receiver timeout". This determines how long sFlow remains active on the exporting device. When the value has expired, sFlow also gets disabled on the device forcing you to re-enable sFlow export. Due to this, we recommend setting the sFlow Receiver Timeout to the maximum possible value, which is 2147483647 seconds which is 68 years ! The command to be used on the HP device for setting the sFlow receiver timeout is:

setmib sFlowRcvrOwner.1 -D NetFlow Analyzer IP sFlowRcvrTimeout.1 -i 2147483647

sFlow Disable Utility:

Of course. We have thought about that too. Just in case you want to export sFlow to different server or stop the flows for some time or whatever be the reason, NetFlow Analyzer provides you the script to disable sFlow export on the HP device.

The disable can be done using the script sFlowDisable.bat (for Windows and .sh for Linux) and the file is present under <\AdventNet\ME\NetFlow\troubleshooting > directory. The usage of the script is as below:

SFlowDisable.bat switchIp snmpPort snmpWriteCommunity

Example :-

C:\AdventNet\ME\NetFlow\troubleshooting>sFlowDisable.bat 192.168.188.30 161 private

Go ahead and try our 30 day trial to see for yourself on how well NetFlow Analyzer works with sFlow and HP devices.

Thanks

Praveen Kumar

Download | Interactive Demo | Product overview video | Twitter | Customers

Couple of days back, we had an interesting conversation going on in our forums. One of our privileged ManageEngine customer wanted to have speed based alerting mechanism and gave us a real good reason to have this feature. Please find the conversation on the below link. 

http://forums.manageengine.com/#Topic/49000003700030

I just wanted to check how the UI should look like and input configuration. Please share us your views and inputs to add the speed based alert feature. 

Please write your technical questions to netflowanalyzer-support@manageengine.com. We are happy to assist you at any moment.

Thanks
Raj

Download | Interactive Demo | Product overview video | Twitter | Customers

 Some tools claim to be free and some are free AND useful. Talking with relation to the so many free network traffic analysis tools available online. The main objective of a traffic monitoring and analysis tool is to be able to see the history of threats, threshold violations, bandwidth utilization and extrapolate it to the future for taking better informed capacity planning decisions. All this analysis is carried out with the data (from NetFlow, sFlow, IPFIX, jfLow and more) available (stored) with the tool. One should be able to compare traffic through a particular device various time periods to see the effectiveness of the policies that have been recently changed / set.

Free tool with no data storage is as needless!

At the end of the day, "relative results" matter. To be able to show that one has made certain changes and how it has affected the network for good, hopefully! All this is possible only if a large amount of data is available for analysis. There are free tools which offer to store data for up to one wHOLE day. All a user will find the next day is a hole in the previous day data. A clean data base and a blank look on one's face. For analysis, data size is very critical. And it doesn't take a genius to say that one day data does not contribute to any analyzable data. Time and data are somethings that cannot be got back once lost (data can be, if you have fail-over, but, hey! how many free tools have that!).

Even when you are going for a free tool, you have a choice to make. To make the choice between something that is going to cost your time and data or the one that is useful-AND-free, which can store the data forever, carry out the necessary analysis.

NetFlow Analyzer free edition lets you monitor two most critical interfaces in your network and the data can be stored forever - that is absolutely free AND useful. An useful solution which gives better analysis with the data that can be stored forever. You can see the history of security threats, the trend of bandwidth requirement growth over a period of time, answers questions such as "who are the top talkers?, is the bandwidth used for the business critical applications ?" and much more.

So you want a "free" tool or a free AND useful tool?

Cheers

Joe

Follow NetFlow Analyzer on twitter!

Download | Interactive Demo | Product overview video | Twitter | Customers

Regards,
Don Thomas Jacob

        This blog may need prior reading of my first blog about Flexible NetFlow. We have already discussed about the advantages of Flexible NetFlow and migration from traditional NetFlow versions to FNF. To make this transition smooth Cisco provides the option of pre-defined flow records which can be used to configure Flexible NetFlow without investing a lot of time. And as I mentioned earlier it also helps your existing NetFlow V9 collector to parse exported data. However to use Flexible NetFlow to its fullest potential or to monitor a specific network behavior, you should create your own customized records. 

        Let’s see how to configure Flexible NetFlow to export flow statistics. Flexible NetFlow export can be configured in three easy steps.

1. Configure the exporter

2. Configure the Flow Monitor with the pre-defined Flow Record and Flow Exporter attached to the monitor.

3. Add the Flow Monitor to the interface to monitor either ingress (input) or egress (output traffic).

1. Configuring Exporter

                    Flow exporter can be configured with a unique name. Multiple Flow exporter profiles can be configured. Below is the configuration to configure Flow Exporter.

flow exporter <exporter name>

destination <ip address of ME NFA>

transport udp <port number>

Example configuration:

flow exporter me_nfa_analyzer

destination 192.168.1.1

transport udp 9996   

2. Flow Monitor and Flow record configuration

Flow record configuration defines the fields exported via NetFlow protocol. Flexible pre-defined flow records are based on the original NetFlow ingress or egress caches. Cisco provides a unique keyword to identify the pre-defined records and these records can associated with a Flexible NetFlow Flow record configuration. The Flexible NetFlow "netflow-original" and netflow ipv4 original-input are predefined records and these two records can be used interchangeably to export the basic key fields and time stamp fields. Flow monitors can also include packet sampling information if sampling is required.

flow monitor <monitor name>

record netflow-original

exporter <exporter name>

cache timeout active <seconds>

cache timeout inactive <seconds>

Example Configuration:

flow monitor me_nfa_monitor

record netflow-original

exporter me_nfa_analyzer

cache timeout active 60

3. Adding Flow Monitor to the interface

Flow Monitor has to be attached to a specific physical or logical interface to export flow statistics for that particular interface. Below is the configuration to attach flow monitor to a specific interface.

interface <interface name>

ip flow monitor <monitor_name> input

Example Configuration:

interface serial0/0

ip flow monitor me_nfa_monitor input

   And the above configuration can be verified by "show flow monitor" command. As I mentioned earlier Flexible NetFlow has numerous advantages and has the power of supporting new performance monitoring statistics as soon as they are available.  Flexible NetFlow is an evolving technology available in Cisco devices to help with visibility into how network assets are being used and the network behavior. 

Please find more information on FNF here.

   ManageEngine constantly studies the market and user demands to support new technologies. In fact ManageEngine NetFlow Analyzer is the first tool to support multiple bandwidth and performance monitoring technologies like NetFlow, NBAR and CBQoS in the market. And currently ManageEngine NetFlow Analyzer supports Flexible NetFlow without any issues. Please write your questions to netflowanalyzer-support@manageengine.com. We are happy to assist you at any moment.

Thanks

Raj 

Download | Interactive Demo | Product overview video | Twitter | Customers

Released!

NetFlow Analyzer Enterprise Edition 7.0 is packed with a load of amazing features. The official PR is available here.

And happy to announce that NetFlow Analyzer Enterprise Edition supports Cisco NetFlow (and other flows), Cisco NBAR and Cisco CBQoS out–of–the–box. Download the 30-day free trial and try it out in your network setup.

Following are some of the new features added in 7.0.

• Validating QoS policies with Cisco CBQoS - Enterprise edition now supports Cisco CBQoS and provides report on the per-class pre policy, post policy drops and queues. This new feature complements the already existing support for Cisco's Network based application recognition (NBAR), helping in application mapping and providing better quality of service. Read more...
  


• User based dashboard page for guests / Operators - Each user can have their own dashboard, only viewing devices that need to be monitored by them, which can be sorted based on utilization, speed etc.


• Business hour alerts - makes sure that the users do not have to worry about the alerts that might be generated during non-business hours. With the new version of NetFlow Analyzer, business hours can be preset as per the enterprise's need and the alerts can be activated only during that period.


• Exclude IP address(es) option in IP groups - During creations of IP groups, the exclude option makes it much easier to exclude only particular addresses from a network as the requirement may be.


• Radius authentication - Radius Server is useful in centralised management of user credential details. Once the user roles are defined in the User Management feature of NetFlow Analyzer, subsequent authentication of the user profiles can be done from the Radius Server.


• Exclude encrypted applications - Enabling NetFlow on cryptomap tunnel interfaces double counts the ESP / GRE traffic. That can be prevented by applying this filter on cryptomap tunnel interfaces.


• Output interface suppression - WAN optimizers compress the packets and therefore the flow size varies. The size of the packet going in and coming out is not the same, and the readings can be misleading and confusing, to say the least. To avoid this, "Output Interface Suppression" can be used. The interface in which the compression takes place (destination/output interface) can be suppressed.


• ACL related drops - Access control filter drops the flow information which contains data pertaining to dropped traffic due to Access Control List.

 Flexible NetFlow is the next generation flow export technique promoted by Cisco Systems. As the word depicts it is highly flexible based on user requirements and to monitor specific network behaviour. Traditional NetFlow used a fixed seven tupple of IP information to identify a flow most of the time. Advantages of Flexible NetFlow 

1. Flexibility to choose the desired export fields. 

2. Reduce the number of flows and allows CPU to perform efficient routing and switching

3. Convergence of multiple accounting technologies into one accounting mechanism

Flexible NetFlow and NetFlow V9

  The export protocol of choice for Flexible NetFlow is the NetFlow Version 9 export protocol, but unfortunately and to date, NetFlow Version 5 has been a much more widely used protocol because of the legacy Cisco IOS® Software images that are still around that supported the NetFlow v5 export protocol only and worked very well. However Cisco claims the future is going to be Flexible NetFlow. And believe it this migration is going to very smooth since Flexible NetFlow can also be configured to export some predefined flow records using the NetFlow Version 5 protocol format for backward compatibility. This helps your existing collectors can work with Flexible NetFlow until you find a real requirement to use additional fields offered by Flexible NetFlow.

Flexible NetFlow Configuration

    Traditional NetFlow configuration is pretty much straight forward. Flexible NetFlow consists of components that can be used together in several variations to perform traffic analysis and data export, and the new command-line interface (CLI) configuration follows the same traditional logic.In this user-defined flow records and the component structure of Flexible NetFlow make it easy to create various configurations for traffic analysis and data export on a networking device with a minimum number of configuration commands. 

    Flexible NetFlow consists of components that can be used together in several variations to perform traffic analysis and data export, and the new command-line interface configuration follows the same traditional logic.

 Let's see this components in detail

Flow Monitor:

    A Flexible NetFlow Flow Monitor describes the NetFlow cache or information stored in the cache. The Flow Monitor contains the Flow Records or key and non-key fields within the cache. Also, part of the Flow Monitor is the Flow Exporter which contains information about the export of NetFlow information including the destination address of the NetFlow collector. The Flow Monitor includes various cache characteristics including the timers for exporting, the size of the cache and if required, the packet sampling rate.

Flow Record:

    A Flow Record is a set of key and non-key NetFlow field values used to characterize flows in the NetFlow cache. Flow Records may be pre-defined for ease of use or customized and user defined. A typical pre-defined record will aggregate flow data and allow users to target common applications for NetFlow. User defined records will allow selection of specific key or non-key fields in the Flow Record. The user defined field is the key to Flexible NetFlow allowing a wide range of information to be characterized and exported by NetFlow. It is expected that different network management applications will support specific user defined and pre-defined Flow Records based on what they are monitoring (ie: security detection, traffic analysis, capacity planning).

Flow Exporter:

    The Flexible NetFlow Exporter allows the user to define where the export can be sent, the type of transport for the export and properties for the export. Multiple exporters can be configured per Flow Monitor or the same exporter can be used by multiple monitors.

The following figure shows the flow monitor and it components.

 In our next blog we are going to use a pre-defined (defined in IOS itself) flow record to export netflow records using Flexible Netflow. In the meanwhile if you have any queries. please write to netflowanalyzer-eesupport@manageengine.com

Thanks

Raj

Download | Interactive Demo | Product overview video

We have posted a number of blogs to share information on how to use NetFlow technology and NetFlow Analyzer to manage your network better. Those blogs will definitely continue to give you more ideas to put the product to better usage but we will also discuss about some of the common issues that you may have come across in the product and how they can be resolved.

NetFlow Analyzer generates traffic reports based on the NetFlow packets exported from the router. Based on the information in the NetFlow packets, the product displays the traffic passing through the interfaces of the exporting device.

One issue that is frequently reported is that the traffic utilization shown in NetFlow Analyzer is more than the actual traffic on the interface. Reports showing more than actual utilization or more than 100 % utilization can be resolved quickly by checking a few points on the exporting device and the product.

Incorrect active timeout:

The traffic reports in NetFlow Analyzer is shown with a 1 minute granularity, ie. NetFlow Analyzer shows details of the traffic for each minute. By default, the active timeout on the NetFlow exporting devices is 30 minutes, which means that the information about the traffic that passed through the interface in the previous 30 minutes is exported at the 30th minute.

Since NetFlow Analyzer reports traffic every minute, the export of 30 minutes information all at once leads to the product's reports showing a spike every 30 minutes. The incorrect traffic details for that minute leads to showing incorrect speed which thus leads to worng utilization calculation. To avoid this, simply check if the active timeout on the router is set to 1 minute using the command "ip flow-cache timeout active 1""

Multiple NetFlow commands:

NetFlow can be enabled on the router using any one of the three commands:

ip route-cache flow   : -  This command can be applied on all main interfaces and will automatically enable NetFlow on the sub interfaces too. This command accounts for the IN traffic across an interface.

ip flow ingress           :-  Some of the newer IOS supports this command which also accounts for the IN traffic across an interface. The difference is that this command needs to be applied on a sub-interface level

ip flow egress            :-  The same as 'ip flow ingress' but this command accounts for the OUT traffic across an interface.

NetFlow can be enabled on the interfaces of the router by applying any one of the above mentioned command, but most of the netwrok admin  enable either "ip flow ingress" or "ip route-cache flow" on the interfaces for traffic accounting. When all these commands are applied on the interfaces, it causes the same traffic to be counted multiple times again causing the product to show incorrect traffic stats and thus incorrect utilization reports.

Incorrect link speed in NetFlow Analyzer:

NetFlow Analyzer calculates the utilization based on the link speed. For example, if the link has capability to handle 1 Mbps and the actual traffic passing through an interface is about 512 Kbps, the utilization graph in NetFlow Analyzer displays the traffic percentage as 50 %. Here is the  formula which explains the utilization calculation on NetFlow Analyzer.

Utilization = Actual Speed/Link Speed * 100

So, if the link speed is not updated properly in NetFlow Analyzer, the utilization shown in NetFlow Analyzer will be different than the actual. NetFlow  Analyzer can determine the interface speed if you set the appropriate SNMP Port and Community for the router on NetFlow Analyzer. This can be  done from the 'Set SNMP Parameters' icon on the 'Interface View' right next to the router name or you can set the interface speed  manually for each interface on NetFlow Analyzer (from the Edit Settings icon on the 'Interface View' next to the interface name). You can refer to this blog for more details.

Non dedicated burstable bandwidth:

Certain ISPs allows you to use over the allocated bandwidth depending on the other customers sharing that link. So, even though the max bandwidth is 2Mbps, the ISP may allow you to use even more based on availability. This also affects the accurate reporting on NetFlow Analyzer causing incorrect bandwidth utilization values and even more than 100%.

ESP and GRE traffic:

This is another reason for traffic to get double counted in NetFlow Analyzer. With NetFlow data, the tunnel traffic will be accounted as the normal traffic before encryption and again as the encrypted traffic. NetFlow Analyzer have an option to filter this kind of encrypted  tunnel traffic from the reports. This option is availble under Product Settings - Advance Settings - ESP or GRE Filter.

To know more about the about ESP and GRE traffic double count, check this link.

If none of the above resolves the issue, please find the technical explanation on what could still be causing this:

Any analyzer tools calculates the OUT traffic of an interface based on the IN traffic of the interface that sends traffic to it. When traffic is passing from higher speed interface to lower speed interface, the calculation of OUT traffic from a higher speed IN traffic causes incorrect traffic utilization to be shown on the OUT traffic.

The above reason for more than 100 % utilization on OUT traffic can be resolved by enabling only "ip flow egress" on all the interfaces.

If you have any further queries on this, kindly send us a email at netflowanalyzer-support@manageengine.com.

Thanks
Praveen

Download | Interactive Demo | Product overview video

 Being a niche player in the SAAS market, Zoho brings an amazing level of engineering expertise to ManageEngine in building highly secure and scalable distributed applications. And hopefully you know, Adventnet has recently changed its name to Zoho Corp and formed three divisions namely ManageEngine, Zoho, and WebNMS.

Many of the NetFlow Analyzers available in the market are not truely distributed in nature. They parse and store the flow records in the same collector and cannot give you the unified view of all the collection points. And there is no automatic crash recovery of data is possible. Unlike in ManageEngine, it involves individual backup and upgrade procedures which requires lot of maintenance activities. All these procedures are automated in ManageEngine NetFlow Analyzer Enterprise Engine via failover and smart upgrade manager technologies. And this is why we call ManageEngine NetFlow Analyzer is a Enterprise class distributed NetFlow collection and reporting engine suitable for any large organisations. And when we say distributed we mean it.

Before you start evaluating a distributed and scalable netflow monitoring solution, please ensure that you have the following Enterprise class features are available.

1. Distributed flow collection capability and optimized bandwidth usage between collectors and central reporting server.

2. Scales upto 20000 interface with 15000 flows per second. Any number of collectors can be added without any additional license.

3. Support for NetFlow V5,V7,V9 /sFlow, JFlow, NetStream, IPFIX.

4. Support for Cisco NBAR and correlate NBAR data with NetFlow data.

5. Support for CB-QoS (Class Based - Quality of Service) monitoring. Identify Pre and Post policy metrics and fine tune your QoS configurations.

6. Failover support - automatic crash recovery and data replication. Please visit this link for more information.

7. Ability to use your existing SAN (Storage Area Network).

8. Compatible with VM ware.

9. No data loss even after a link failure between Collectors and Central Server.

10. Ensure separate 64 bit binaries are available for increased flow handling and reporting performance.

11. Secure data transfer - https mode between collector and central server

12. Smart upgrade manager. Upgrade patchs are pushed automatically from the central console to collectors. 

12. User defined dashboards and views.

13. Group devices based on their location and build tree view for easy access and troubleshooting.

14. Ability to work in multiple time zones

15. Network Forensics using raw data

Sample screen shot from ManageEngine NetFlow Analyzer Enterprise Edition:

 And remember thousands of users like Cisco,Adobe, Ferrari and many fortune companies cannot be wrong.

 Please download and try our 30 day full featured trial edition in the following link

http://www.manageengine.com/products/netflow/distributed-monitoring/download.html

Full Feature List is available in the following link

http://www.manageengine.com/products/netflow/distributed-monitoring/new-features.html?temp

  Kindly write your questions to netflowanalyzer-eesupport@manageengine.com. We are happy to assist you at any moment. 

Thanks

Raj