A Better Vocabulary for Testing

(alperenkeles.com)

56 points | by alpaylan 21 hours ago ago

15 comments

  • ozim 2 hours ago

    I would like "non-functional testing" to be called "cross-functional testing".

  • Blowoffvalve an hour ago

    I loved this article. I find that the most important part of engineering projects is defining a shared vocabulary for discussing and aligning with other parties.

    The vocabulary proposed that clear delineates across the various concerns you're testing against is particularly helpful.

  • FabCH an hour ago

    I find that the biggest problem with all these classifications is that at the end of the day, developers will write whatever works for them in the moment, and not care if it fits your theoretical box.

    I did a paper on it once, to see what tests devs actually write: https://ieeexplore.ieee.org/document/11081450

  • jruohonen 3 hours ago

    I think the alleged dictum between functional and non-functional is the biggest conceptual, theoretical, or even philosophical issue.

    • weinzierl 3 hours ago

      This was also my first thought when reading OP. It is not restricted to testing even. The whole development process has this distinction (e.g. functional vs non-functional requirements) and it hurts more than it helps.

    • gls2ro 3 hours ago

      I think this is an important distinction that is also real when thinking about verification: verifying function can be boolean or lets say limited to a simple set while verifying non functional properties means mapping to a more continous interval and so it is a qualitative decision.

  • g-b-r 2 hours ago

    I recall finding different definitions in every book about testing I read.

    I think most of the confusion came from people suddenly deciding that a unit test can't test less than a whole object, in OOP.

    • smitty1e 2 hours ago

      Mad Men meme:

      "Testing, but applied to terminology across publications."

  • gzell 2 hours ago

    Always enjoyable for us aged s/w engineers to see the new generations relearn the fundamentals.

    • wreath 2 hours ago

      I dont think this will last much longer. Looking around, i see most engineers not being interested in anything other than AI related stuff. Certainly true for the junior ones

      • andsoitis 2 hours ago

        AI opens up whole new ways of validating that systems behave correctly or to find issues.

        • gls2ro an hour ago

          Maybe or maybe now we can do proper testing.

          Most of the testing techniques are baed on groups/set and graph theories while test data is based on sampling.

          So now we really have the chance to model verification and validation even better

  • jerf an hour ago

    I think the problem with testing vocabulary is that the field is just too continuous. We have all sorts of situations in the world where there is no sharp line, yet we have terminology for various chunks of the possibility space even so because there are still some sort of clump in the space that we can identify. For example, there isn't a true sharp line between any of the primary colors, but we do not generally have a problem labelling things as "red" or "green" or whatever even so. There is a certain personality type frequently encountered online that this bothers no end, who can't really deal with the idea of words that have fuzzy boundaries around them and struggle with the concept of a "fuzzy line", but most of us get on even so.

    However, when there is no natural clumping, there isn't even a place to put the fuzzy boundary necessarily. I think a lot of testing has this problem. I've disliked "unit" versus "integration" terminology for at least a decade now, because to me there's just a continuum. Yes, obviously, tests can test more or less of the code base at a time, and can draw in fewer or more dependencies in the process, and arguably there is an important distinction between 0 other dependencies included and 1, but beyond that I've thought for a long time there aren't any other distinctions worth putting a label on.

    There is value to tests that mock out all the other dependencies. There is value to writing stubs. There is value to bringing up those dependencies and mocking or stubbing out their dependencies. There is value to having a full staging environment and running acceptance tests by hand, and that staging environment itself can have any number of mocks or stubs in it itself. And there is no rule that says you rigidly must make this decision based on the exact number of dependencies away something is from your code, to the extent that that is even a well-defined concept, nor are "mock" and "stub" themselves some sort of bright shining line as one can mix and match them freely, nor is that the complete list of all possibilities. The result is a fairly smooth continuum of possible tests. In my opinion, any attempt to put labels on very particular bits of it raises the risk of overprivileging certain parts of the space just because they get a name and creating the streetlight effect on the named bits and causing the others to be missed.

    Even "performance testing" versus "correctness testing" gets awfully fuzzy; again, there is arguably some value in saying this test has performance concerns and this other test is just related to correctness, but even "performance testing" still has to worry about a lot of other things too and I don't find it anywhere near a binary distinction. Anyone who has spent any time with performance testing has certainly learned the risk of worrying about "performance" without also worrying about all the other things we test for at the same time, or to put it another way, I can make my performance testing look really good if I let the code it is testing be incorrect.

    It is unreasonable to expect a universal nomenclature catalog of such a smooth space. We can't even as a species completely agree on color delineations in the spectrum, which is a simpler problem in most ways.

  • sfeizi an hour ago

    does vocabulary matter anymore as coding agents can make interpretations?

  • gfalcao 2 hours ago

    great infographic