Test Engineering in the NI Ecosystem (Part 1)

Owen Search

This post is intended to be the first of a series of posts targeted at exploring some of the development challenges I have faced during my time as a test engineer primarily leveraging the NI toolchain to do my work.  I hope the series will serve as an informative and insightful look at the discipline of test engineering, it's challenges and how I have muddled through solving some of those challenges (or at least worked within their constraints).  My experience as a (fairly) recent grad and new to the discipline is that it can be hard to navigate due to it's relatively tribal nature in most organizations.  Combine that with the seemingly small number of formal educational programs in test engineering and you end up with a line of work that seems to mean many different things to many different people and organizations.  My own goal for these posts is to attempt to make some sense of it all.

I should mention that most of me experience has been spent on the software side of things, although I am no stranger to sticking my head inside a 19" rack to go looking for the source of a short.  Please provide feedback and insight (and corrections) in the comments! ​

​Part 1:  Test Engineering

Test engineering?
Test engineering is the discipline of engineering hardware and software systems whose goal is to verify and validate the quality of a product throughout it's life cycle.  The field is highly inter-disciplinary and test engineering groups are typically working closely with design, quality and manufacturing teams to get the job done. 

What is a test?
On one end of the spectrum stands validation testing, where we attempt to validate that a widget was designed correctly.  I.E. Did we build the right thing?  On the other end we have verification or production testing where we attempt to verify that the widget was built correctly.  I.E.  Did we build it right?  The "ends" of this spectrum parallel the ends of the product V&V curve.  Validation is typically more of a design related activity while verification tends to be more of a production related activity.  There can be many various stages of test placed throughout the development cycle that blur the lines of these two classifications.

Why do we do it?
So that our widgets don't break on our customers, or worse, become self aware and do all sorts of other nasty things.  Incorporating and planning for test in the product development life cycle is key for achieving schedule and budget goals in an engineering project (Or more specifically one that spits out a widget.  Caveat: software systems are not exempt!  They entail different considerations though).  As the old saying goes, "test early and often".  Incorporating a sound, unified test strategy throughout a development cycle leads to a number of things that make leadership happy AND a number of things that make engineers happy.  Let's take a look:

Leadership smiles when

• Products are delivered on time
• Delivered products are of a high quality
• Investigating product escapes* and/or test failures does not negatively impact development of widget 2.0

*Escapes are when a bad product ends up in the hands of the customer.

Engineers smile when

• They have confidence their design works as expected
• They are not crunched on a tight schedule
• They are not redesigning the wheel
• They are given the tools and the opportunities to solve problems now vs fight fires later
• They have the visibility into their process necessary to correct problems that do occur
• And of course, when leadership smiles

Test keeps everyone smiling by ensuring

• It is easier and much less costly (schedule and budget) to correct problems earlier in the design process
• Design defects are caught early and often, resulting in a more robust design
• Fewer design defects make it into production, resulting in a more fully understood product
• A pre-existing test strategy developed in design validation can migrate to verification (production) and minimize repeated engineering efforts
• A better understood product and test results in less burden on test engineering during the deployment of the test solution
• Less engineering burden on the test team results in less schedule impact when trying to kickoff and ramp production
• Sound product verification strategy and a robust product result in fewer product quality issues in the field
• A unified test solution that was borne in validation and trialed in production results in highly effective and far reaching product traceability for escapes that do occur
• High traceability results in less impact on engineering productivity from managing unexpected product failures over the product's lifetime

Why should I care?
You should care because test engineering can be pretty darn tough and as a result some really interesting problem spaces and corresponding solutions pop up quite often.  Test engineering is inherently difficult for a number of reasons.  In my mind, some of the largest are:

• Test engineering schedules are commonly squeezed between the end of the design group's schedule and the beginning of the production group's schedule.  This is often one of the worst places to be (stuck between a rock and a hard place)
• Test engineering requires the development of many "one off" designs.  Typically software and hardware systems are built for the test of a particular product and may not translate to the test of others
• Test engineering requires designing systems that are a highly diverse mix of technologies.  Technologies typically include a mix of:
  1. Off-the-shelf and custom electronic components
  2. Software modules that range from unit-test-scale functions to enterprise-resource-planning-scale integration frameworks
  3. Custom mechanical interconnects, harnessing, housings and/or fixturing

​
These operational constraints force the need for test engineering groups to leverage some of the following unique types of engineering solutions in order to stay ahead of the curve and deliver on time and on budget:

• Highly flexible hardware platforms that
  • Can accommodate a diverse mix of products
  • Support the ability to do high throughput production testing as well as low throughput, high accuracy, high precision, deterministic validation testing
  • Provide interconnect to products under test as well test hardware
  • Provide mechanical interfaces to exotic product form factors
  • Simulate external hardware components not capable of being present during test
  • Etc.
• Software platforms that
  • May need to execute deterministically
  • Interact with a high variety of hardware products used in test
  • Interact with a high variety of products under test
  • Integrate enterprise resource management services with test software
  • Integrate commercial off the shelf software products where appropriate
  • Log exhaustive test data to databases or other data stores
  • Provided data analysis capabilities for functionality verification/validation or higher level process control
  • Etc.

There is much to talk about concerning both hardware and software platforms used in test engineering.  My goal with the next post will be to break down an effective approach to meeting the challenge of these numerous and varied requirements.

---

Owen Search is an Avionics Test Engineer at SpaceX.  He is a Certified LabVIEW Architect, Certified TestStand Architect, Certified LabVIEW Embedded Systems Developer, Certified LabWindows/CVI Developer and NI Certified Professional Instructor.  C# is his current language of choice but he has a soft spot for C/C++ and a love of LabVIEW.  He holds a BS in Biomedical Engineering from the University of Connecticut.