Improving Network Insight Through Local Context Gathering and Analysis

Identity of network traffic is becoming increasingly important in the definition and enforcement of security policies in an enterprise network. Network management and auditing require a finer granularity to be associated with the traffic flows in addition to the host level. Unfortunately, the inability of current architecture results in the local context of the connectivity in terms of the user and application being inferred from the packet content, such as IP address and port numbers. It is this inference that frequently results in overly coarse rules for the firewall in the interest of performance or simply enabling connectivity. While there are mechanisms proposed in the literature that purport to address this issue, the reality of deployment often negates adoption of such techniques. To that end, this work proposes a distributed network data collection and analysis system, eXpsicor, that pro-actively gathers the missing characteristics (local context) for the purpose of enterprise network management. By combining the full visibility at the end hosts (through such simple tools as netstat, ps, and lsof) and the global aggregate view at the central management server, it is demonstrated how local context improves network insight, and can be used for security auditing, finer network management and better policy mapping without costly deployment overhead. The system has been built and evaluated, and tools for visualizing analyzing data have been developed. The system has been deployed on over 150+ machines in our department. Many interesting analysis have been performed and shown in this thesis based on the data that has been collected since April 2007.