Don Thomas Jacob | netflowanalyzer | Enterprise IT Management, Network performance management, IT Servicedesk, Desktop Management, Datacenter Management, Server Management, Log Analysis and Security Management, Network Tools, ManageEngine Blogs

Bandwidth monitoring and traffic analysis is turning out to be more important than ever with growing advances in networking technologies and advent of Web 2.0. It is no more possible to simply let the organization's traffic network pass through the WAN links, pushing each other for bandwidth. Prioritizing traffic, so that mission-critical applications receive the bandwidth they need, is the key word today.

There is a little feature called NBAR available in many Cisco devices, which lets you do a lot more than it spells and can play a great role in defining the network's traffic policies.

NBAR or Network-Based Application Recognition is a feature available in Cisco IOS that does a deep packet inspection of traffic passing through an interface and can recognize a wide variety of applications, including applications that dynamically assigns TCP or UDP port numbers or even undesired applications that uses well known port numbers to mask itself.

NBAR will show the details of the applications used on an interface basis. The feature can identify even peer to peer applications like Bit Torrent or applications like Skype which uses random port numbers for connectivity and hogs the organizational bandwidth. The results available from NBAR can also be used to define your QoS policies in a much better manner blocking out the unwanted applications.

NetFlow Analyzer, which uses NetFlow data and other similar flow data to give reports on bandwidth usage by host, port, protocol, applications, DiffServ and conversations, can also report on NBAR statistics from the your devices, making reporting an easy task.

NBAR with its deep packet inspection capability is a great feature for security analysis also. An example is how NBAR helped to identify CODE-RED worm and the related Cisco information can be seen from here. You can even make use of the AutoQoS for the Enterprise feature available in some Cisco devices which can use NBAR data for prioritizing traffic. Do check out how to do this from here.

Since NBAR data help define CBQoS policies, NetFlow Analyzer can also report on the Class Based QoS policies and its pre and post policy traffic usage and drops. Get a first hand experience of the features in NetFlow Analyzer using the 30 day trail.

Download | Interactive Demo | Product overview video | Twitter | Customers

Regards,
Don Thomas Jacob

With Internet bandwidth being costly and transmission of business critical data being a priority, tracking of bandwidth taken by fun and entertainment sites is an essential in bandwidth management. Such tracking helps ensures that bandwidth taken for traffic to fun sites does not affect business critical applications traversing over the Internet links.

NetFlow Analyzer and NetFlow technology can be used for detailed traffic and bandwidth analysis to identify the applications used, find the hosts involved with the traffic and trace their QoS markings among many other reporting capabilities. But, how exactly would you distinguish between normal HTTP traffic and the traffic to sites such as facebook, myspace, youtube, sports sites and so on?

NetFlow Analyzer provides multiple options to track the traffic to specific sites or departments, separating them from the normal traffic for easier view and analysis. One is through the capability to combine application mapping with IP Address, network or range, helping categorize applications which use the same port but have different hosts involved as separate applications.

Such a mapping will show the traffic to this certain site in the list of total applications for an interface, thus giving you an idea on how much of the total traffic was taken by users connecting to the social site.

If this sounds good, check the next option we have. The IP Group option in NetFlow Analyzer lets you group together IP Address, network or range, applications or a combination of all these as a separate category and see their specific reports. Such a grouping helps categorize the complete network traffic to fun sites, lets call them social sites, see the hosts involved and how much each are using every hour/day and even custom time periods. Sounds better ?

Both these features can be used to quickly categorize applications based on their source and destination or to categorize traffic separately with a combination of criteria.

The feature is not limited to just classifying social site traffic, but can be used for traffic to a specific branch or office, traffic related to any business critical applications, and so on. Do let us know your suggestions on the product and its features and what more you would like to see in the future.

Download | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

Great News for all who were looking for monitoring NetFlow data from Cisco ASA devices. ManageEngine NetFlow Analyzer now provides preliminary support for NetFlow data from ASA devices.

For those who have not caught up on this news, a couple of months back, Cisco released a new IOS which brings support for NetFlow capabilities to ASA devices. The NetFlow feature from ASA devices, termed as NetFlow Secure Event Logging (NSEL), is based on NetFlow version 9 flow format and can give real time bandwidth reports.

Ever since this release, we have had a huge demand to start supporting the new flow format. Working with some customers who provided packet captures from their ASA devices, our engineering team has successfully developed a patch which would provide support for these flows. The patch has to be applied on top of the latest version of NetFlow Analyzer.

This patch enables NetFlow Analyzer to report on traffic and bandwidth information using the NetFlow packets from ASA devices when exported in the same format as NetFlow version 5. We will be extending our support to the new fields in our next release.

You can find the recommended configuration for ASA NetFlow from this post in our forum. Please contact our technical support at netflowanalyzer-support@manageengine.com / +1 925 965 9435 to get more information.

Regards,
Don Thomas Jacob

When wondering what to write about for our blog, came this question from a user. They needed to get alerts when the hosts in his network communicated with a set of blacklisted IP Address. Felt this could be useful for a number of users which is why we now have this blog here.

For his requirement, the user could have opted for an expensive flow based anomaly detection solution and achieve it the costly way. The cost effective method was to work with features already available in the easy to use, all software bandwidth monitoring solution from ManageEngine, which is NetFlow Analyzer.

Now in detail about what was wanted and how this can be done.

There is a set of IP Addresses with which the hosts in a company's network is not expected to communicate. If there is traffic either to or from these blacklisted IPs, the network administrator needs to be alerted, find the violating host and then carry out cautionary steps.

NetFlow (or any similar flows), with its capabilities for in-depth reports, is the only technology that can tell you about the application used, source and destination of traffic, priority of the traffic and much more. NetFlow Analyzer, which supports all the major flow formats, has an IP Group feature with which you can group together IP Address/Network or Range and monitor the traffic to and from it. Making use of this, one can create an IP Group and associate all the blacklisted IP Addresses with it. When creating the IP Group, the speed which is taken for utilization calculation is set at the lowest possible value, 1bps. This way, even a single conversation will account for more than 1 percentage utilization.

After creating the IP Group, we can use the alert profiles to receive alerts when the traffic utilization exceeds 1% in the IP Group. The alerts can be emailed to the email address specified and you can even give multiple threshold actions in the same alert.

With this, you will be able to ensure that no traffic passed to or from the blacklisted IP Addresses and even if there was traffic, you are alerted. Drilling down on the IP Group to the conversation tab shows the hosts involved thus helping you take your cautionary measures.

A combination of simple features for proactive troubleshooting !

Download | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

In this blog I am going to talk about how NetFlow Analyzer reports speed and utilization and what is the use of bandwidth command used on the router.

ManageEngine NetFlow Analyzer is the first NetFlow solution to offer one minute granularity for speed, volume and utilization reports. Additionally, the product even lets you drill down from any minute and find the conversations causing this traffic giving you sub minute visibility.

We will now see how speed and utilization is calculated for a particular minute using the NetFlow exports.

Consider any report from NetFlow Analyzer with a reporting granularity of 1 minute. Here the speed value of any particular minute is calculated by adding the transmission volume of all the conversations happening on that particular minute from the interface and the value is divided by 60 seconds to derive speed.

Lets take an example scenario for better understanding.

Speed Calculation:

Let say at 10:10 - total IN traffic is 50 MB (Volume is always represented in Bytes).

Now the speed is 50 MB * 8 / 60 (The multiplication by 8 is to convert the volume to bits and division by 60 (seconds) to get the speed.)

So, from the above calculation data rate at 10:10 is 6.67 Mbps (Speed is always represented in bits per second)

Utilization calculation:

Now that the link usage at 10:10 is 6.67 Mbps, utilization can be defined as how much of the actual link capability was used for transmission of traffic.

Utilization = (Usage rate /Actual link speed) * 100. So for an 8Mbps link, the utilization is: (6.67 Mbps /8 Mbps) * 100 = 83 %

This is how the utilization is calculated in NetFlow Analyzer every minute

What is bandwidth command in router?

Often users complain about not getting the speed set at the interface in the product or about incorrect utilization reports. User data rate depends on the operating link speed/bandwidth and not the bandwidth configured in the interface using the IOS bandwidth command. The bandwidth configured on the interface is to help routing protocols choose the best route to transmit the packets. We recommend our users to set the link speed as the bandwidth in the interface for efficient routing.

NetFlow Analyzer reports the actual data rate on the link and is not based on the bandwidth set on the interface. However, since NetFlow Analyzer fetches the link speed via SNMP from the router based on the bandwidth speed set, the utilization calculation will be effected.

So it is important to set the right speed using the bandwidth command to get better routing efficiency and correct reports in NetFlow Analyzer. And it is not possible to get the faster throughput from any interface by configuring a faster speed using bandwidth command.

With this information users should be in a better position to see the right utilization reports in NetFlow Analyzer for real time traffic and also get the efficient routing for network traffic.

Download | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

NetFlow technology and ManageEngine NetFlow Analyzer makes bandwidth monitoring and traffic analysis easy by giving you detailed reports about the traffic, applications used, source and destination of traffic, traffic conversations, DSCP based QoS values and so on. All these reports, available for time periods ranging from the last minute to the time at which the product was installed, can show the hosts participating in the traffic making further analysis easier and aiding in proactive network troubleshooting.

But there is a dark side to this coin. The number of hosts involved in the traffic can be quite large and in case of huge networks, a daily or weekly reports will involve a few thousands of IP Addresses. Identifying the top 10 or so hosts from their IP Address could be easy, but what if there are more than hundred conversations in each of your main interfaces? Here, you can make use of the 'Resolve DNS' option available to resolve the IP Address to the DNS name making identification easier.

The DNS name resolution makes use of the mapping available in your primary or secondary DNS server to resolve the DNS names for IP Address. The names displayed using this resolution will be based on your organization's standard naming policy and may not be easiest of the names to remember ! This is why NetFlow Analyzer has an option for custom DNS names. Custom or user defined DNS names can be set in the product for any number of IP Addresses and the next time you opt for showing the DNS names, it is these names that will be shown overriding the system resolved DNS names.

To set your own DNS names, navigate to Product Settings - Server Settings, and here under the DNS Settings option click on the 'Add/Modify' option next to the 'User defined DNS names' option. You can also set user defined DNS names for IP Address from the Source or Destination tab using the edit icon next to an IP Address shown in the reports. A small feature that is a big help to the large enterprises !

Download | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

Having discussed about what are the traffic patterns to be checked for to identify bot behavior, we will now look at how NetFlow / cFlowd data with ManageEngine NetFlow Analyzer can make the tracking much more easier. In case you have not read the PART 1 of this blog which outlines the traffic patterns related to bot behavior in a network, do catch up with it.

It is not easy to track the traffic behavior with just SNMP based or other tools as no technology provides an in depth information as NetFlow. To use NetFlow or cFlowd for traffic analysis, all you need is a device that is capable of exporting NetFlow / sFlow or similar flow packets. The sweet part of this is that NetFlow or similar flows are supported on most of the devices from the major vendors thus removing the need for additional equipment and hardware purchases. NetFlow Analyzer, which can be installed on a server in your network would receive the exported packets to generate reports about bandwidth and traffic. With no additional configuration to get the reports, setting up and reporting all happens in a matter of minutes.

Lets now delve into how the traffic patterns outlined in the last blog can be analyzed easily with NetFlow Analyzer using NetFlow data.

The first point we had talked about is the necessity of the bots to locate the C&C servers for updates and how this is done through DNS requests. Thus, seeing a large volume of DNS requests from the network is something to be concerned about.
NetFlow data can report in detail about the traffic and show you the applications passing through the interfaces. Monitoring the OUT traffic on the WAN link will show if there are any DNS requests to the Internet from the network and how much of the total traffic was taken up by these requests. You can also drill down on the application to find the hosts involved with the traffic. NetFlow Analyzer can also generate alerts if the DNS requests leaving the network exceeds an expected percentage. This can be done from 'Alert Profiles' under 'Admin Operations' where the alert for IN or OUT traffic with specific criteria can be set and have the alerts send as an email or SNMP trap.

After having located the C&C server, the bots communicate with them over IRC to receive the update commands on attacks to be performed or for updating the bot itself. If see your internal hosts communicating a lot over IRC, this should definitely be checked. To track this, you can take a look at the LAN or WAN interface and look at the applications being used. The 'Application' tab will show you if IRC is being used and the hosts involved with IRC traffic. Here again, you can make use of alerts to alert you or if you are expecting zero percent IRC traffic from the network, create an IP Group associating the IRC application and this grouping will show you even the smallest volume of IRC traffic in an easy to view category removing the needs for drill down and searches.

Another work of the bots is to spread further and this is done by scanning hosts in the subnet for vulnerable hosts. The scanning is done by sending small burst of packets in the subnet checking for host vulnerabilities. You can easily track the traffic based on packets for each interface from the Traffic - Packets tab. This should show you easily if there is an increase in the number of packets in the LAN without a large increase in traffic volume.

After locating the C&C server and after having received the updates, the botnets take part in a DDoS attack. One method of DDoS attack is by sending a high volume of outbound TCP SYN requests with an invalid source IP Address. Our blog on tracking TCP/SYN attack will help in finding an attack on your server and looking at the TCP/SYN requests exiting your WAN link will let you know if there are DDoS attacks originating from your network.

The last point we talked about was on how the botnets involved with email spamming can be identified. Since such bots sends millions of emails to all sort of email addresses, keep a watch on the SMTP traffic to the outside and get alerted for an unusually large volume of SMTP traffic. Here too you can make use of the IP Group option for tracking specific application behavior and this blog should help you on application tracking with IP Groups.

The features and capabilities of NetFlow Analyzer does not end here. You can make use of various options like customizable dashboard to track the top applications from the network, use IP Groups and alerts to inform you on traffic behavior, have reports exported to PDF, CSV or even instantly emailed and much more. Try ManageEngine NetFlow Analyzer 30 day trial with free technical support to have a hands on experience on what more you can do with NetFlow Analyzer.

Download | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

With Twitter being the new big site brought down through a DDoS attack, botnets and DDoS is making news. So, what does DDoS attacks have to to do with bots (botnet) or bots have to do with DDoS ? DDoS or Distributed Denial of Service attacks involve flooding of a host with continuous communication requests by numerous distributed computers which leads to a bandwidth choke for the hosted service, denying legitimate users the access to a service.

BotNet, which means a network of computer robots or bots, is a set of compromised computers controlled by a bot herder or bot master (the one who manages the bots) through a Command and Control (C&C) Server and is used for performing malicious activities like DDoS attacks, email spamming, click fraud, spreading malware and etc.

With bots being used for malicious activities, no organization can let their computers to be a part of a botnet. Due to this, it is important to identify the bots and quickly remove them from the network or clean them, thus preventing further rot and attacks. One of the best methodologies that can be adopted to identify botnets is to analyze network traffic for common botnet behavior patterns and find the infected hosts.

We will outline some of the common traffic patterns to keep an eye on to identify botnets and what these patterns mean in terms of botnet activity.

One of the main character of a bot is its need for communication with the C&C server and this is a must for maintaining control and update of the botnet by the bot herder. Most of the botnets today use DNS service to find the location of a C&C server from which it has to receive the updates.

To avoid detection and shut down, the C&C servers uses different methodologies like IP flux and Domain flux to change their DNS name or the IP Addresses associated with FQDN. Due to this, the botnets cannot connect to a specific C&C server as and when needed. Instead, it has to do a large number of DNS lookups, scanning a large volume of addresses to find the C&C server for receiving the update. This turns out to be the best way to track the botnet. If you find a lot more DNS lookup in your network than ever expected or DNS queries from hosts to improper DNS names, the chances that it is an infected host trying to find its C&C server is as high as probability can be !

What does the bot do after locating the C&C server ? In most cases the C&C server could also be an IRC server, and once it has been discovered, the bots receives updates from the master about what type of action has to be performed. The action can be anything from sending spam emails to mounting a DDoS attack. Since most C&C servers is an IRC server, the communication takes place via IRC and so seeing unexpected IRC traffic to and from your internal hosts where IRC traffic is not allowed is definitely a case of concern.

Bots also needs to spread further and add more bots into its botnet which finally helps increase the strength of a botnet and thus that of the attack carried out too. An infected bot will scan for other hosts in its network for vulnerabilities and when such a host is found, will attack it to compromise the host. When scanning the network for possible hosts to infect, bots generate a burst of small packets. So, if you see a sudden increase in the number of packets without a major increase in the traffic volume, what you are possibly seeing is a bot scanning the subnet for other hosts to infect and add to the botnet.

Another common option that can be used is to track outbound TCP SYN packets having an invalid source IP Address. The reason for these large number of TCP SYN packets could mean that some of the internal hosts in your network is part of a botnet and are participating in a DDoS attack at the moment.

One of the functions other than DDoS attack for which botnets are used is for email spamming. Email spamming involves sending huge volume of spam emails advertising fake products intended at financial gains. When the hosts in a network are part of a botnet involved with spamming, they send huge number of emails to the outside world and mostly using some external email server. So, unusual SMTP activity from your network to the outside is another significant network activity that needs to be tracked. Steps can also be taken to forward emails only through the organization's mail server and prevent the use of any external or public mail servers.

Now that you have an idea on what kind of information needs to be tracked, how does one do this easily?

The best technology that lets you keep track of such detailed network activity is NetFlow / cFlowd with its capability for exporting in-depth information. With NetFlow Analyzer's capability to group and classify traffic and analyze in detail the flow records, the job is made even easier. In our next blog, we will discuss on how we can use NetFlow Analyzer to track the mentioned traffic behavior within minutes of set up.

Download (30 day trial) | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

Article References:
Taxonomy of Botnet
Book Excerpt: Virtual Honeypots: From Botnet Tracking to Intrusion Detection by Niels Provos and Thorsten Holz
Botnet Communication Topologies
Wikipedia

Failover Blog 1 discussed about the need for failover and Blog 2 about the architecture and enabling of the failover setup in NetFlow Analyzer Enterprise edition. So, what happens after the setup of failover and how does it work when the primary server goes down is what will be explained in this final blog.

Once the setup is complete, the hotstandby server watches the health of the primary server and syncs with it for updated information on the data and configurations through HTTPS and MySql replication. In case the primary server crashes or is shutdown, the collectors will try to contact the hotstandby server for sending the collected data. There is no data loss from collectors as the collected data is not discarded. The collectors will store the data till either the primary server is back up or until the hotstandby server becomes the new primary server.

So, when does the hotstandby server become the primary server ? While the collectors try sending data to the hotstandby server becuase the primary server is unavailable, the hoststandby server will check if the primary server is still up. When the time set for the failover on the primary server has elapsed, the hotstandby server becomes the new primary server and thus starts accepting the new data and configurations from the collectors.

Now, the collectors and the central server continues to work normally through a hassle free swicth over, thus enabling users to get continuous access to their data and reports. When the primary server does come up later, it automatically becomes the hotstandby server whle the new primary server continues to run so. This setup will continue and if at any later point the new primary server goes down, the process repeats so that reporting is never effected.

With Failover also acting as a data backup mechanism and seamless switchover ensuring 100% up time, NetFlow Analyzer Enterprise edition is a class above other similar solutions

Download (30 day trial) | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

Having outlined the need for failover which also substitutes as a data backup mechanism in NetFlow Analyzer Enterprise edition through our first blog, we will tell you in detail about the architecture of the failover feature in our product and how you can enable and make use of the feature.

The Enterprise edition of NetFlow Analyzer is based on a collector - central server architecture. In, the distributed architecture data collection to be done by collectors which sends the collected data to the central server from where all the reporting takes place. This provides higher scalability and performance to the Enterprise edition by making it capable to handle up to 20,000 interfaces and each collector to handle 10,000 flows per second.

To use failover feature, all you need is a second server identical to the primary central server in configurations which acts as the hot standby server.The machine can be located in the same network or even in a geographically separated DR center. You only need to ensure that both the primary and hot standby server (failover machine) can communicate with each other and that the collectors can reach the failover machine if the primary server is down.

In this setup, the primary server is the central machine which receives all the data from the collectors located at various locations.The collectors are configured with the primary server IP Address so that the collected data is sent across in real time. To enable failover, you can navigate to 'Settings' under 'Admin Operations' and click on the the 'Failover Settings' tab. From here, you can enable failover by simply clicking on the 'Enable' radio button.

After enabling failover, shut down the primary server and copy the data folder from <NetFlow_Central>/mysql/ directory and tmp folder from <NetFlow_Central> directory to a safe backup location. Start the central after the this step is complete and you will be prompted with the message "Replication is enabled. How would you like to start your server?". Click "primary server". This will start your existing central server as the primary server in the failover setup.

Now download and install the Enterprise edition central server on the machine you intend to use as the hotstandby server. You need to ensure the following:

1. The Operating System and hardware on both the primary server and hotstandby server should be identical.

2. The time and timezone on the primary server and hotstandby server should be same.

After the installation is complete, do not run the central server installation. Before running the installation, copy the data and tmp folder which were backed up earlier to the <NetFlow_Central>/mysql/ directory and<NetFlow_Central> directory respectively. You will be prompted to overwrite the existing file and you can proceed with this step. This is needed so that the data and configurations which have been stored in the primary server is available in the hotstandby server. Once the copying of database is complete, start the hotstandby server from 'All Programs > ManageEngine NetFlow Analyzer EE - Central server > Replication > Start hotstandby server'. You will be prompted to enter the host name/ IP Address of the primary server and HTTPS port (by default it is 443).

You can change the port used for communication whichever port you prefer. At this step, the hotstandby server communicates with the primary server and updates its details to the primary server database. The primary server will update all the collectors with the information about the hotstandby server thus relieving the user from the task of updating all the collectors with this information.

Guess you can take a break now ! Part 3 of this blog will outline how the hotstandby server becomes the primary server. For those who would like to have a go at it now, do try the NetFlow Analyzer Enterprise edition.

Download (30 day trial) | Interactive Demo | Product overview video

Regards,
Don Thomas Jacob

NetFlow Analyzer and NBAR – A step closer to better Traffic Management

How NetFlow and NetFlow Analyzer helps in Social sites traffic monitoring

Cisco ASA NetFlow - Now supported by ManageEngine NetFlow Analyzer

Alert ! - Traffic to a blacklisted IP

Interface/Link Speed Vs Interface Bandwidth Command - What NetFlow Analyzer Reports

Custom DNS - Making Host Identification Easier

Stopping the bots – With NetFlow and NetFlow Analyzer - Part 2

Stopping bots – Tracking Botnet Behavior with NetFlow

Failover - Part 3 - From Hotstandby to Primary

Failover - Part 2 - Architecture and Enabling

ManageEngine Blogs