LISA 2010 - Lawrence's attendee notes

LISA '10: November 7–12, San Jose CA

Training Sessions Attended

Nagios: Advanced Topics

Instructor: John Sellens (note: former MFCF staff member)

• Description: http://www.usenix.org/events/lisa10/training/tutonefile.html#s4

Nagios is a very widely used tool for monitoring hosts and services on a network. It's very flexible, configurable, and can be extended in many ways, using home-grown or already existing extensions.

This tutorial will cover the advanced features and abilities of Nagios and related tools, which are especially useful in larger or more complex environments, or for higher degrees of automation or integration with other systems.

• My "take-aways"
  • John's "religious viewpoint" - "use SNMP as much as you can ..." - he feels it's not used as much as it should be
  • while some systems are "all-in-one", Nagios design is actually very good being a core system and separate plugins - very extensible
    • there are a rich collection of plugins available
  • while the web-interface is "older looking", it's quite functional and "considered optional"
  • he recommends learning/using PERL for maximum benefit for creating/modifying plug-ins
  • you can have event handlers to automate problem resolution
  • the most common way of doing checks is using "check_by_ssh" - he suggested using ssh keys
  • he reviewed some of the other web-interfaces for Nagios (eg: Icinga and Nagios XI ("Nagios Ex Eye"), a commercial Product)
  • it was clear to me that many attendees at LISA do use Nagios

Have a Drink from the Network Services Firehose

Instructor: Gerald Carter

• Description: http://www.usenix.org/events/lisa10/training/tutonefile.html#s10

There is a collection of network services we use every day when performing basic tasks such as checking email, sharing documents, and browsing the Web.

For each topic, we'll answer the questions:

• What does it do?
• Why do I have it or need it on my network?
• What are the popular (or at least common) applications I might run into?
• How does it work on Solaris, Linux, and *BSD systems?
• What is in the configuration files and where do they live?

• My "take-aways"
  • good over-view of most common network services - DNS, DHCP, NFS, LDAP, E-Mail, & HTTP
  • he illustrated common configuration files and locations, plus log files for troubleshooting
    • subtle differences between OS'es/distributions (Solaris, Ubuntu, RedHat, FreeBSD)
  • commonly used ports

Real-World Insights on How to Secure and Route Your Linux Network

Instructor: Jason Faulkner

• Description: http://www.usenix.org/events/lisa10/training/tutonefile.html#m5

We will be talking about how to configure IP addresses and static routes, securing Linux servers on a network level, using the iproute2 utilities for advanced routing and IP configurations, and some things to look out for with Linux networks. You'll learn why you should throw away "ifconfig" on Linux in favor of "ip," how to create and optimize iptables rule sets, and even how to do some basic load balancing using ipvsadm.

• My "take-aways"
  • used the term "Good Practices" rather than the more presumptious "Best Practices"!
  • highlighted the need to get ready for IPv6!! "~30 weeks from today, ICANN runs out of of IPv4 addresses. Please investigate IPv6. The internet needs you"
  • reviewed basic terminology, eg: NAT, VLAN, ARP, OSI model, TCP, UDP, IP
  • we set up a sample network configuration (simple LAMP server, couple of workstations

Advanced Time Management: Team Efficiency

Instructor: Thomas A. Limoncelli

• Description: http://www.usenix.org/events/lisa10/training/tutonefile.html#m8

Techniques to help your IT team work better, faster, and more transparently. Topics include:

• Efficient meetings
  • Why meetings are a big waste of time
  • How to improve bad meetings, and how to skip unfixable meetings
  • Being a better meeting facilitator using shared documents
  • Taking detailed meeting notes using multi-user text editors
  • Lab: Using Google Wave to host an online meeting with detailed notes
  • Using Google Moderator to manage Q&A sessions
  • How to get people to volunteer for tasks
  • Lab: Planning a weekly meeting

• Eliminate email overload
  • What are the causes?
  • Gmail searching and filtering tips (demo)
  • Three rules for more productive mailing list operation
• How to use collaborative document systems such as wikis and Google Docs
  • To coordinate a system upgrade (with a large team)
  • To maintain common reference material (emergency contact lists, etc.)
  • To cooperatively write a budget or presentation slides
  • Lab: Group budget writing
  • Lab: Group presentation editing
• Collect data and surveys using Web survey tools
  • Lab: Gathering sign-up information for a team BBQ
  • Lab: Using a spreadsheet to manage a pool of test machines
• Communicate more effectively with users; create screen-casts to help users
• Improved "institutional memory" using wikis
  • Document processes to trick other people into doing your work
  • Creating a simple home page and FAQ system for your users
  • Creating a department home page easily

• My "take-aways"
  • our CSCF and RSG staff meetings are very much the way he suggested running meetings - using agendas that can be viewed and modified by all attendees, listed in reverse chronological order
  • Email - use filters to move less important emails (eg: mailing lists) to separate folders
  • Documentation - use wikis to more easily create docs
  • effective use of check-lists to more efficiently train new staff or do other routine tasks (eg: setting up a system, etc.)

SANS Security 464 - Hacker Detection for System Administrators: Catching the Wily Hacker (Day 1 and 2)

Instructor: Seth Misenar

• Description: http://www.usenix.org/events/lisa10/training/tutonefile.html#sans

Topics include:

• Why bad things happen to good system administrators: five common misconfigurations and mistakes that lead to a system being compromised
• Security methodology and thought process in daily system administration activities
• A sysadmin's view of what matters in systems architectures
• Security monitoring: not knowing makes the auditors and hackers happy
• The hard part: knowing what is normal for Windows and UNIX systems
• The harder part: knowing what is abnormal for Windows and UNIX systems
• Hardening Windows and UNIX systems is easier than you thought
• Command line kung fu for UNIX and Windows
• Understanding network traffic for system administrators
• Malware: why it is still effective in your environment

• My "take-aways"
  • You can't depend on your anti-virus - remarkably easy to get-around - in-class excercise - create a trojan that most A/V tools can't recognize
  • don't treat your network like a Tootsie Roll with a hard exterior shell and a soft squishy inside!
  • WMIC - really useful command-line tool in Windows for getting configuration information
  • OSSIM - Bundle of security and monitoring tools, including Nagios

Tech Sessions attended

• Full list of tech sessions - most include the slides and/or video

Rethinking Passwords

Instructor: William Cheswick, AT&T Labs—Research

Passwords and PINs are used everywhere these days, but their use is often painful. Traditional password advice and rules are seldom appropriate for today's threats, yet we labor with the password rules and servers of yesteryear. Strong passwords are weakening our security, and it is time to fix that.

There are numerous proposals for new password solutions. I will present a few half-baked ideas. But there are good solutions available now.

We are facing much more worrisome security challenges: we ought to get this easy stuff right.

• Slides
• Video (requires Usenix login)

• My "take-aways"
  • "Eye of Newt" passwords (ie: complex rules) actually reduce security
  • "It is simply poor engineering to expect people to select and remember passwords that are resistant to dictionary attacks"
  • allowing longer phrases of simple text (including spaces) is easier to remember, essentially uncrackable and less likely to be written down (ie: improves security)
  • Less painful account locking - increasing time increments
  • two-factor authentication using apps on iPhones or Blackberries (RSA SoftKey)
  • many interesting ideas for alternative login methods, including Google Maps, faces, Mandelbrot

System Administrators in the Wild: An Outsider's View of Your World and Work

Speaker: Eben M. Haber, IBM Research—Almaden

You understand the work of system administration, but how do you explain it to others? Since 2002 a group at IBM has been studying sysadmins in the wild to better understand how they work, both to inspire improvements in tools and practices and to explain the ever-growing human costs in enterprise IT. As outsiders we were fascinated by what we learned, so we've written a book on the subject to explain your work to the rest of the world. This talk provides a summary of our most important findings, supported by real-life footage of sysadmins at work.

• My "take-aways"
  • amusing session that celebrates the work that Sysadmins do!
  • emphasized the realization how collaborative our work is

The Path to Senior Sysadmin

Speaker: Adam Moskowitz

Being a senior system administrator is about more than knowing all the options to mount(8) or that modprobe is what's used to replace that buggy kernel module with the latest version. Rather, a good senior sysadmin will have a wide knowledge of relevant technical topics, in-depth knowledge of one or more technologies, good interpersonal skills, and the ability to manage "problem users" and will be comfortable making presentations to and negotiating with mid- and upper-level management. This talk will cover the skills a senior sysadmin needs and why they are necessary and will provide some suggestions for how to acquire these skills.

• Slides

• My "take-aways"
  • Senior sysadmins should be providing high-level solutions to management
  • do not "trash-talk" others in your organization
  • some professional development may have to happen on your own time - not enough time in the day to learn or experiment everything that may be useful to you or your organization

Panel: Legal and Privacy Issues in Cloud Computing

Speakers: Stephen Beck, VMware; Richard Goldberg, Attorney at Law, Washington, DC; Bill Mooz, VMware

• My "take-aways"
  • may need to insist that your data stay in your own country

Birds-of-a-Feather (BoF) sessions

Oracle Solaris 10 and Beyond Vendor BoF

Oracle Solaris Engineer Panel Moderator: Dan Roberts, Oracle Bart Smaalders and Dan Price, Oracle

This session covers the current state of the Oracle Solaris 10 and gives insights into the future direction of Oracle Solaris technologies and products. Come learn how Oracle Solaris can benefit your environment today and what will continue to make Oracle Solaris the most advanced mission critical enterprise OS going forward, with even more features for performance, serviceability, and ease of administration in both small- and large-scale deployments.

• My "take-aways"
  • active on-going development of Solaris 11

Amanda Backup BoF

Paddy Sreenivasan and Chander Kant, Amanda Project

Amanda is the leading open source backup software in the world. During this BoF we invite current and future Amanda users to share their experiences. We would also discuss future directions of Amanda and get your input.

• My "take-aways"
  • looks worth investigating for possible backup of our grad PCs and research machines

Splunk, the Future: 4.2 and Beyond Vendor BoF

Gaurav Gupta, Archana Ganapathi, and Jim Hansen, Splunk

Expected early next year, Splunk 4.2 promises some key features and benefits for system administrators, developers and IT managers. Included in this release will be real-time alerting, to go with the real-time search capability that debuted in 4.1. There will also be the ability to manage Splunk deployments and some nice solutions apps, including VMware and ESS apps.

At the end of the product roadmap, you'll also hear about Splunk's community initiatives over the last year and what to expect in the coming year.

The presentations will be followed by a lively Q and A session.

• My "Take-aways"

Zenoss Open Source Monitoring BoF

Today's data centers are more dynamic than ever before. New hardware appears and disappears regularly, servers are virtualized and move in and out of the data center. Fortunately there is an open source tool that makes this all manageable and allows you to monitor your network, wherever it is. Zenoss Core is a free and open source (GPLv2) IT monitoring solution written in Python that has been downloaded over 1 million times and used in over 25,000 organizations worldwide. Zenoss delivers the functionality to monitor the health and performance of networks, servers and applications through a single, integrated software package. At the heart of Zenoss is a dynamic unified model of the entire IT environment, which allows system administrators to manage and monitor the ever-increasing complexity of their environments. The demonstration will cover Zenoss' capabilities and discuss how the large and active community around Zenoss gives it an advantage over closed-source alternatives. More information about Zenoss Core can be found at: http://community.zenoss.org

Vendors met

Oracle/Sun

• My "take-aways"
  • We had a long discussion with a senior Oracle/Sun rep about our concerns about their handling of the Canadian market and support and maintenance of existing servers

QualStar

• My "take-aways"
  • CSCF uses QualStar for our backup systems

Ksplice

• My "take-aways"
  • Looks interesting for not having to reboot when doing Linux kernel updates