四处找的文章之Scott Barber

Scott Barber is the person behind www.perftestplus.com（阅读地图“www.perftestplus.com”）, co-founder of WOPR (the Workshop On Performance and Reliability) and a prolific writer. In this interview he explains what performance testing is, how one should go about learning it, various resources that can help you and many more things.

If you want to ask him more questions or you want to send any feedback on this interview, please send it to webmaster @ whatistesting.com.

1. Scott, please tell us something about yourself.
I was born in Woodbury, NJ USA in 1971 and grew up in a typical American small town. As a child I was very active in school, sports and Boy Scouts. Growing up, the goals I had for myself in adulthood included:
Getting to travel and “see the world”
Get a good college education
Get out of small-town America
Be recognized as a success in my career (as with most young people in the USA my specific career choice changed many times as I grew up)
Be a father who is active in his children’s lives and activities.
Design and build my own home.
I left “small-town America” to attend Virginia Tech on a U.S. Army ROTC Scholarship to pursue a degree in Civil Engineering. After four years, I completed a Bachelor’s of Science in Civil Engineering and started my 4-year active duty commitment in the U.S. Army as a Second Lieutenant (2LT). I later went back to school at night to earn a Master’s Degree in Information Technology. My career has taken a rather wandering, but in retrospect an extremely beneficial, path to finally land in the area of Software Testing.

I spent 4 successful, but difficult years in the U.S. Army culminating as a Company Commander for the 101st Airborne Division, Air Assault Headquarters and Headquarters Company approximately 8 years before I was technically qualified to hold the position. As a junior Captain at the end of my 4 years, I was recruited into a government consulting company to be an Information Engineer on a team that was designing and building a system to replace the computer system I most often complained about while I was on active duty. I held this position for about 1½ years while becoming increasingly more technical before I went to a small private software development company with the intent of becoming an Oracle Database Administrator, but actually ended up spending 6 months as a Configuration Manager. During that time, I learned a ton about Databases, Configuration Management, development and networking. Most importantly, I learned that none of those were what I wanted to do day in and day out. 6 months later, that company was bought by a huge, publicly traded conglomerate that resulted in a significant reduction in pay, leading me to seek a new job. This is when I was recruited into a Performance Test Consulting job by an old college friend where I spent the next 4 years. This time as a consultant is where most of my performance testing experience comes from. After 4 years of consulting, my family and I decided it was time to move. As it turned out, we moved to Florida about the same time the consulting company went through a major re-organization leading me to re-think what I wanted to do next in my career. After a long and difficult 4 months of soul searching, I ended up taking a wonderful position as a Systems Test Engineer and Software Test Manager at my current company AuthenTec, Inc.

As far as my childhood goals are concerned, I have had a chance to travel more than I ever thought I would, but have plenty left to do. I achieved my educational goals, but have a passive interest in earning a Doctorate degree someday. After living in a wide variety of places, I have finally settled in a place that feel like “home” and am very content with my career progression so far.

I have two young boys who are the greatest joy in my life. As I write this, it is with thoughts of my older boy starting school tomorrow morning and my youngest learning to talk. I guess that means that the only childhood goal I have left is to design and build that house. Luckily, I continue adding goals, so I’m far from bored.

2. You specialize in performance testing. What drew you to it? How did you get started, and what led you to start writing about it?

One Sunday evening I got a call from my manager…
“Scott, be at the Marriott Hotel Conference Room at 8am tomorrow. Our CEO has an announcement to make.”

Being a good employee, I did as I was told and went to the meeting. As it turns out, our software development company was “merging” with a huge media company. We were going to become their “development branch”. As a result, we were giving up our bonuses, overtime, and our base pay was being reduced, but we were going to get stock options.

As one might imagine, I was less than enthusiastic about this. I called my friend of 10 years before I even left the parking lot of the hotel. I told him the circumstances and asked him if he’d help me update my resume, to which he responded:

“Dude, that sucks. Send me your resume I’ll take a look, but, we need performance engineers. You’d be a perfect fit.”

I said “Performance engineer? What’s that?!?”

He replied “Don’t worry, you’ll like it.”

He was right, I’ve loved every minute of it! Since then I have dedicated my professional career to this field. It has been a wonderful ride so far.

After a few years, people started approaching me to write about what I do. I never though of myself as much of a writer, so I hesitated and stalled. (Mind you, I was the Civil Engineer in college who avoided writing classes at ALL COSTS. When they finally told me that if I didn’t take a writing course I wouldn’t graduate, I went to the smallest, least reputable, community college I could find, took the course and transferred it!) Eventually, I agreed to create the Performance Engineering section of my company’s Best Practices. Some folks thought it was good and started selling me to other companies to write their procedures. Finally, my boss talked me into trying my hand at writing articles. I really didn’t think anyone would read them, but I wrote a few anyway. As it turns out, people did read them, and liked them.

3. What fascinates you in performance testing?
Most simply, I am fascinated by the diversity of it. As my meandering career path let me to this field, I had a lot of “false starts” in other branches of software development, I learned that regardless of my degree of success in those more narrow branches, I found myself becoming easily bored. As a technical tester, specifically as a performance tester of multi-user distributed systems, I get to apply aspects of all of those branches and more in new and unique ways on every project I do. This diversity from project to project is also what led me to embrace the Context-Driven School of software Testing and ultimately become friends with such software testing thought leaders as Cem Kaner and James Bach.

4. If one wants to gain expertise in performance testing what all should one do? What are the tools, books and sites that can be helpful?
There is very little consolidated information out there about the type of performance testing that most organizations need. There is no doubt that my site (http://www.perfetsplus.com) and articles are one of the most complete references for “Practical Performance Testing”. However, it is by no means complete. Some of the other places I have learned valuable lessons that I have applied to performance testing are:（以下标红色的全是阅读地图的线路）
Everything I can find written by Alberto Savoia
Lessons Learned in Software Testing, John Wiley & Sons, 2002 by Cem Kaner, James Bach and Brett Pettichord.
Testing Computer Software (2nd Ed.), International Thomson Computer Press, 1993 by Cem Kaner, Jack Falk, & Hung Quoc Nguyen.
Select Chapters from The Web Testing Handbook, STQE Publishers, 2001 by Steve Splaine and Stefan P. Jaskiel.
Research published by performance tool vendors.
Research published by testing and research organizations such as Nielsen//NetRatings, Keynote Systems, Inc., and IEEE
Online forums and magazines such as StickyMinds.com, QAForums.com and stpmag.com and performancetester.com.
Performance Engineering of Software Systems, Addison-Wesley, 1990 by Connie U. Smith, Ph.D.
Scaling for E-Business: technologies, models, performance, and capacity planning, Prentice Hall, 2000 by Daniel A. Menascé, Ph.D.
Speed Up Your Site, New Riders, 2003 by Andrew B. King
Anything by Cem Kaner, James Bach, Jerry Weinburg and/or Edward Tufte.
These days, I continue learning through networking with experts, being a technical editor for new performance related publications, beta testing tools, hosting and/or attending workshops and conferences and assisting with research being led by other experts and students in the field.
To really grow your career as a performance tester, I believe there are several things you can do that will gain you largest benefit.

Become a “Mid-Level Everything” – Developer, DBA, Network Admin, Systems Admin, Architect, Business Analyst, etc.
Become an expert in the following – your tools, performance analysis, relevant aspects of operational research and statistics, consolidating, interpreting, sorting, and graphically presenting complex information in an intuitive format.
Become skilled/knowledgeable in the following areas – group facilitation, product management and training.
Find a mentor. There are several exceptional performance testers out there who believe inJerry Weinburg’s consulting principle（阅读地图线路）“Give your best work away for free.” Befriending one (or more) of them to ask questions of, share ideas and brainstorm with is amazingly useful… to everyone involved.

5. What is your opinion about the state of performance testing tools? Are there some open source or free tools available?
Most tools on the market aren’t actually Performance Testing tools, they are Load Generation tools. No matter what the tool vendors claim, they do not make performance testing simple, they do not analyze results, and they do not pinpoint performance issues. They are virtually all grossly overpriced, and never believe the demo. That said, I have a favorite tool that I support heavily… with a long list of caveats.My favorite is my favorite because it allows me to create realistic loads by writing custom functions and procedures in C – rather than trying to guess how to make the silly GUI buttons duplicate the workload I have constructed.（算不算断章取意？）

The best OpenSource tool for general web based load generation is OpenSTA. As long as you are only testing websites, it is competitive with the major (pay) tools in terms of actual ability to generate load and collect data. There are plenty of other free niche tools for specific platforms and protocols, but they are only useful to small groups of people doing very specific types of testing.

6. How do you combine performance testing with load and stress testing
Actually, I see Performance Testing as a superset of testing that includes load and stress testing. I often refer to it as “Performance Related Testing”. In general, load testing is testing your application under expected load conditions to determine the overall performance. Stress testing is testing your application under unexpected or extreme conditions to determine failure modes so you can protect against them.（Scott Barber说的，一种关于“Performance Testing、load testing、Stress testing”的定义？）In my presentation“Introduction to Performance Testing; The Who, What, Where, When and Why.”（阅读地图线路） I describe various components of Performance Testing, and the process in general.

7. What is WOPR? How did it happen? Who all are involved in it and what is your role?
WOPR is the Workshop On Performance and Reliability (http://www.performance-workshop.org). Ross Collard and I co-founded this workshop about a year ago with the purpose of bringing together both Performance Testing Experts and “Enthusiastic Beginners” in a forum where we can share our experiences together in a format that encourages questions, critical thinking and learning. So far, we have had three very successful workshops and plan to continue having them in the spring and fall of every year. You can find out more from the website. The workshop is limited to 20 people per meeting and is by invitation only, but you can apply for an invitation by filling out the form on the website.

8. How is performance testing for consumer applications different from enterprise/server based applications?
There is actually less difference between consumer applications and enterprise/server based applications than there can be between any two individual applications in the same category. The real difference centers around performance expectations. One of the keys to effective performance testing is the ability to collect and classify requirements and then develop tests that verify those requirements.

9. Where do you think performance testing fits in the testing life cycle?
Should it be done early or should it be done late? What are the pros and cons and how do you decide?
There are actually at least 3 separate, but highly related, disciplines related to performance that should ideally be included in the Software Development Lifecycle – Software Performance Engineering (SPE), Performance Testing (what we have been talking about) and Capacity Planning. The only book I am aware of that addresses these three discipline as part of an integrated process is Improving .NET Application Performance and Scalability, Microsoft Press, 2004. In this book, Senior Author J.D. Meier does a fantastic job of taking the reader through the entire process. The only drawback is that much of the book is extremely Microsoft/.NET specific.

Performance Test planning should actually begin before the architecture has been determined or any code has been written. It is especially important to finalize performance testing planning before any machines have been configured for testing and the plan should continue to be revised as necessary until the application has reached it’s target load in production several times. Very few organizations do this. Most organizations create a plan shortly before load generation is scheduled to commence and stick to the letter of the plan regardless of the actual results or detected issues. By working this way, much of the value that can be gained through performance testing is lost.

The bottom line is this, Performance Testing late in the process alone, as most organizations do, can be highly effective, but only if all identified issues are relatively easy to resolve. Saving the testing for the end of the development cycle can be disastrous if the identified issues point back to poor architectural or design decisions.

10. You have written a lot of papers on performance testing. Do you plan to write a book on this subject?
I have written quite a number of papers, presentations and tried to tie them all together on my website. When I started doing performance testing, I found very little written on the topic and had to learn through trial and error, seeking out mentors and applying principles from other disciplines. I became frustrated with the lack of valuable information available and found many others who were looking to improve their performance testing. As a result I tried to share what I learned through these mediums so that others will have an easier time getting started than I did.

I have tentative plans for two books. The first is simply a consolidation and expansion of my existing articles. There are two main reasons that I would like to do this. First, I’d like to retrofit the articles to be even more generally applicable, include examples and code samples for various tools since they were initially written on contract for IBM Rational. The second reason is because many readers have requested hard copies, which is unfortunately prohibitively expensive on a case-by-case basis.

While there are some very good books on performance tuning, and other performance related topics and some very good chapters in various testing books about performance testing, there is no single “good” book for practical performance testing lessons. Writing a book is much more difficult than writing a book’s worth of articles, but I keep hoping that I’ll find the time, energy, resources and support to get it done in the not too distant future. While I certainly don’t think I know everything there is to know about performance testing, I do think sharing what I do know would be helpful to many performance testers both now and in the future.

11. What are the things you are currently working on?
Outside of the books and some “clean-up” articles, I have actually moved in a new direction with my new job. I am now in charge of testing fingerprint sensors and their associated software (API, drivers, etc) on both PC and embedded platforms. This is extremely challenging as the testing is a blend of four very different industries (Hardware, Software, Embedded/RealTime and Biometrics) that, to the best of my knowledge, has never been written about and as far as I can tell has rarely, if ever, been tried. The knowledge I have gained as a Performance Tester/Analyst has been extremely helpful in the endeavor, as has my relationships with various industry experts such as Cem Kaner, James Bach, and Alan Jorgensen, not to mention my Embedded Software Development Instructor friends at MIT.

Additionally, I am trying to take my User Community Modeling Language (UCML™) to the next level. It was created specifically to model Performance Workloads and the data associated to those workloads, but I have found it to be personally useful in modeling other application characteristics by simply adding a few symbols and thinking about it from a slightly different perspective. Soon, I hope to publish UCML™ v2.0 to see what other folks think.