Tuesday, May 31, 2011

My Journey From Developer To Tester

I've been thinking about this post for a while, and was inspired to complete it after just returning from a local tester group meeting discussing the difficulties of finding good testers.  Hopefully my story will help out, at least from one perspective.
I graduated with a computer science degree from Campbell University.  Like most CS graduates, I expected to be a software developer when I graduated, and my major classes were in programming and theory.  A career as a tester was never discussed, and there were no classes in testing.

Due to a scheduling error my last semester I was only taking one class at night, so I took my first career job doing telephone tech support for a large personal computer company.  I learned a lot about the way people
 think (or don't, in many cases!) and how to translate technical speak for the average user.  I was also doing a lot of testing.  Forget black box, this was troubleshooting a computer I couldn't even see via a phone conversation with a non technical user!  It was frequently frustrating, and I looked forward to escaping to a real programming job when I graduated.  Like many such experiences, I didn't appreciate some of the skills I was learning until after I left.

Once I got my degree I happily took my first job writing code for a large telecom company, back when the industry was booming.  My job was writing software fixes that would be deployed via patches to communications switches.  The actual coding was a relatively small part of the job, most of it involved reproducing the problem, configuring test environments, and testing the fix to ensure the problem was gone.  I enjoyed the coding and the thrill of making something work that was broken before, but the actual testing was often challenging and interesting to me also.

A few years later I moved to a position working on a similar product, but where I got to do a few pure development projects.  It was satisfying at times, but difficult, and didn't really seem like a good fit for me.  I did OK and managed to finish my projects in time, but I was uncomfortable with my job and performance level.  It was about this time that the industry tanked and layoffs finally caught up with me.  Having seen them coming, I had already been taking a good look at my skill set and what I really wanted to do.  I realized that throughout my career, breaking things (and to a lesser extent fixing them) had always been more easy and satisfying for me than creating them from scratch.  I was a better tester than a developer!

Armed with this newfound realization, my job search got a lot easier.  I was able to find lots of ways in which my previous positions had involved testing and supporting skill sets like reporting and communication with end users.  I was also able to easily look outside of the telecom industry for jobs, a challenging problem for development positions as the languages and platforms I had worked on were very industry specific.

Through a family connection I found a small company that would take me on to fill a tester position, a job which they didn't even have at the time.
It was interesting interviewing for an opportunity to create a new role in a completely different company and industry.  While my testing experience at various projects was useful, I emphasized my interest in learning.  Perhaps more importantly from an attitude perspective, I distilled all the customer facing experiences I had during my career to talk about my focus on customer quality, and being able to represent the customer's interests.
It also helped that I had taken the time to research the company as much as possible, and think in advance about how I could work on their product.

I was hired, and it turned out to be a great fit:  While learning the product and industry I had time to find out what was necessary to MAKE the position. I found and read the testing blogs and periodicals, and got involved with the local testing group for support.  I researched and implemented an automated testing tool the 'right' way, with a data driven framework that was easy to maintain, reviewed and optimized their defect workflow, and performed a lot of exploratory testing.  As you can imagine this all took a couple of years to get running smoothly, especially with the learning curve of a new industry and product.

Do I miss the development world?  Not a bit.  Writing and maintaining the automated test suite and supporting scripts satisfies my coding urge and keeps my skills up, and occasionally I use my free time to tinker with other automation tools and languages.   A steady stream of new functionality and fielding defects and questions from the customer base gives me enough challenge to keep my manual testing skills up to snuff, as well as spec reviews and critical thinking.
I'm part of the development group, so I can still 'tech talk' with the developers and do a little white box testing now and then.

Would I go back to being a developer?  Not me, I've had enough experience to know that testing is my passion and strength.  I do know others who have gone back and forth between the two positions, but in my experience it's not that common.  I think that there is enough of a difference in the 'fix it' versus 'break it' mindset that most people have a preference and aptitude one way or the other.

Whether you're a developer or a tester, or one of the rare folks who enjoy both, the important thing is to know yourself.  Even if you're fresh out of college, think about your experiences with people and technology, in terms of what you enjoyed and what you were good at (be honest).  It may change over time.  But if you find your passion it's a lot more enjoyable to go to work on Monday morning!



Monday, May 23, 2011

Testing and the Art of Auto Repair

I do a good bit of basic work on my car, from oil changes all the way to intake manifold cleaning and suspension work.  It's an 8 year old Volkswagen diesel, and an online community helps keep me informed and encouraged about how to work on it.
I'm often struck by the similarities with software testing.  And yes, a lot of software diagnosis is involved with modern cars also.

In my most recent example,  I had recently cleaned out my intake manifold, a several hour job on a Saturday morning (a major software release with significant changes to the source code).  Performance was greatly improved, but there was still a nagging sensation that it wasn't performing as it should (customers seem happy with the new functionality, but occasional non-reproducible bug reports and logs filter in).

After parking the car for a couple of days, I went to start it on Monday morning to go to work, and it wouldn't stay running.  It would idle for 10-15 seconds before quitting, and refused to rev over 1000 rpm (a crippling system failure just as the customers need to use the system).

I had seen very similar symptoms a few months before, and a new fuel filter fixed the problem.  So my wife and I pushed the car out of the way, and I took another car to work (reverted to the earlier release to provide needed functionality during the day).    After I got home, I installed the new fuel filter with high hopes of success, but in spite of priming and many starting attempts the car still would not remain running (previous fix doesn't work for the current problem).

I called the mechanic to give him a heads up, and called the tow truck to take it to the expert (call subject matter experts and architects).  But when the tow truck arrived two hours later to pick up the car, after some initial hesitation it started up an ran fine (this never happens with software - just kidding!).  I'm left to assume that some air in the fuel line managed to finally purge itself after being able to sit for a while (external influences on the software/server or varying system loads affecting an installation routine finally settled down).

A test drive shows that the car is back to its old self with even more power than before, now that it has plenty of both air and fuel.  But without ever conclusively proving the source of the problem, I'll still be crossing my fingers when I start it in the morning.
And that, too, is often just like software.

Monday, May 16, 2011

Large data set testing

One testing scenario that I've been thinking about recently is testing large data sets.  Our software runs large furniture manufacturing plants, and the combination of all of the different options available in some screens can get quite large and cause a performance issue.  More importantly, each customer is capable of configuring their data in many different ways.  

Take a recent example.  A customer had configured two features with the same name, but one contained alphanumeric data and one had integer data.  A  poor data setup from our perspective, so we disregarded this possibility when considering a schema change. When upgrading to a newer software version which put features in a common pool by name, the SQL upgrade script tried to merge the two data sets as one, and some data was lost because of the unexpected data type difference.  Luckily this was caught in a test upgrade by customer support and no live data was lost, but the point is the same:  How do you test for data compatibility issues when each customer is capable of creating data combinations in new and 'interesting' ways?  You may not agree with a data setup, but if it's allowed there's always the chance that someone somewhere will try it, and you can't always get away with calling it a 'data problem' (not if you want to keep your customers for long).

Large data sets like this are one of those good examples of the testing maxim that it isn't possible to test everything.  In any kind of large scale data based solution, there are going to be too many possible ways for customers to create and combine their data.  So how do we get the best test coverage?  Here are a few ways I currently go about it,  I'm sure you can think of more.

1) Focus on the most important data scenarios.
This may fall under the heading of 'duh', but is always worth restating.  It doesn't do your customer any good if you catch a problem with a board being cut 2 mm too long, but the procurement screen for ordering new material is broken.  There is a mantra in aviation that in an emergency situation, 'First, fly the airplane'.  Don't get distracted by details and forget the big stuff.  There is plenty of material already published on techniques for determining what's most important to you and your customers, I won't restate it here.

2) Restrict possible data entries and combinations.
While this is a development function based on lots of spec discussion and user input, you as the tester also have important input as a representative of the end user.   Sure, we all know to test edge cases and weird data inputs.  But we also need to slow down occasionally and think about whether
we can or should allow the end user to enter that data if it doesn't make sense.  Speak up and suggest a restriction if you think one is needed.  If it's shot down (hopefully with an explanation) you've learned something about the data and at least planted the idea.  If not, and the user later decides they want that functionality, now it's a discrete feature to test. 
Once the restrictions are in place, don't forget to test them.  More importantly, test data migration from versions without the restrictions to versions with the restriction to catch any data that has already strayed.

3) Keep expanding your regression test suite as defects are reported.
This is an easy one, you don't even have to find your own bugs!  As data problems are found and reported, try to add them to your suite of regression tests.  While automated tests are preferable for this, even a list of data combinations to try in manual testing is better than nothing.
The most important thing is to add the necessary data to whatever database you are using for testing, and use it.  Take the time to do a little exploratory testing around the defect scenario to see if you can catch any other issues revealed by that data combination.

4) Exploratory testing.
This one can be difficult, especially if you've been looking at a lot of the same program functionality over and over again for years like I have.
The idea is to try different ways to break out of the 'normal' success path testing mindset, and test the software in different ways and data
combinations.  As I mentioned before, it it's allowed there's a chance someone will try it, so try to get away from the 'nobody would ever do that!' mindset for at least a few minutes.  Talk to customer support and see if any end users are known for pushing the boundaries of the software, then take a look at their data if you can.  Read industry publications, competitor news, or occasionally even your own corporate website to see what marketing is trying to tell people your software will do.

5) Use customer data.
This one overlaps with the exploratory testing point.  If you can get your hands on some representative customer databases that use a lot of the functionality your software offers, spend some time getting to know their data.  Test the same test cases you're familiar with on a different data set, see if they are handled differently.  Stop and think: "What mindset does this data represent?  Can I apply it back to my familiar data?"
If you have customer representatives who work directly with the customer they can be a big help here.
Another test to try here is to upgrade several sets of customer data between software versions.  As seen in the example I first mentioned, this often
finds data scenarios that the person writing the upgrade scripts didn't think of.

6) Pairwise testing
http://www.pairwise.org/
This is a particularly useful tool to have in your back pocket when dealing with large sets of data that can be combined.
Any time I have more than 3 discrete sets of data being combined into a single scenario, I consider this approach.
However, if the individual data sets are very large, and/or have interdepencies and restrictions, I find that it loses its usefulness.

7) Fuzzing and semi-random data sets
Don't have enough data?  Make some up!  Free tools such as http://www.generatedata.com/#about can be used to generate data sets within certain parameters that you can use to test your application with (e.g. addresses, names, integers, strings).  The TestComplete testing tool that I use comes with its own data generator also, or if you're handy with scripting you can write your own.
Fuzzing is a similar principle, but automatically applies large numbers of semi random inputs to the program.  
I must admit, though, while both of these can be useful for stress tests, I haven't had much luck in applying them to create realistic scenarios.

Hopefully this has given you a few ideas to start with.  One thing I can guarantee you is that customers will continue to find ways to break your software that you could never have imagined.  Don't feel too bad about it, add each problem to your testing repertoire, and make up your mind that that's one bug that won't slip by you again!

Saturday, May 14, 2011

Welcome to my blog!  I just returned from an inspiring week at the STAREast 2011 conference, which I highly recommend.  One piece of advice I often hear at events such as this is to distinguish yourself online, via Twitter, blogs etc.
Since I already have a Twitter account (follow me at @allenjfly), I decided it's finally time to get a 'Round TUIT' (ha) and test my blogging skills.

I have a full time job and plenty to do outside of work, so don't expect too much, but I'll try to keep posting when I can.

I started as a developer with a Computer Science degree, working on software patches for telecommunications systems. This involved a lot of testing to reproduce issues and to verify that the fix worked.  When the telecom market went downhill and I got laid off,
I decided that the testing part was really what I was best at, and have been doing it ever since.

I landed at a small company with no tester through a personal connection, and convinced them they would be better off with some quality control in the development department.  6 years later I'm still there, and still the lone tester.   I've created and maintain an automated test suite, test fixes on every  nightly build, oversee the defect tracking process and filter incoming defects, serve as 3rd level tech support, and perform any other testing task I can think of in my spare time: Performance testing, exploratory testing, etc.  

Thanks for reading, and I welcome your comments and discussion!