Generating Test Cases From Application Behaviour

Your test suite tells you what your code does. Behaviour-driven test generation tells you what your code actually does, including the parts nobody wrote specs for.

---

Most testing workflows start with a human writing a test case based on what they think the code should do. That assumption is the problem. Requirements drift, undocumented edge cases accumulate, and the delta between "what the spec says" and "what the system does in production" grows quietly until something breaks in front of a user.

Generating test cases from observed application behaviour inverts the process. You instrument the running system, record what it does, and derive tests from that ground truth. The tests reflect reality rather than intention.

What Does "Behaviour-Driven Test Generation" Actually Mean?

The phrase gets used loosely, so it's worth being precise about what we're talking about here.

Behaviour-Driven Development (BDD) in the Cucumber/Gherkin sense is about collaborative specification. That's not this. What we're describing is closer to what researchers call specification mining or automatic test generation from execution traces. You observe the system at runtime, capture inputs, outputs, and state transitions, and use those observations to synthesise test cases programmatically.

There are three distinct approaches:

• Execution trace recording: Capture real HTTP requests and responses, database reads and writes, and queue messages. Tools like Pact (consumer-driven contract testing) and Hoverfly sit in this category. You replay real traffic as regression tests.
• Fuzzing with feedback: Tools like AFL++ and libFuzzer mutate inputs based on code coverage signals. They discover paths through the system that humans wouldn't think to test.
• Model inference: Tools like Daikon analyse execution traces to infer invariants ("this field is always positive", "this list is never empty before this function returns"). Those invariants become assertions.

Each approach has different failure modes and different infrastructure requirements. Execution trace recording is the lowest-friction starting point for most production systems. Fuzzing gives you the highest confidence in correctness at the input boundary. Model inference is the most powerful but requires the most investment to interpret correctly.

How Do You Instrument a System Without Wrecking It?

This is the engineering question that kills most attempts at runtime-derived testing before they start.

At the HTTP boundary

For HTTP services, recording is straightforward. A middleware layer captures request/response pairs and writes them to a store. In Python, a simple WSGI middleware can do this. In Go, wrapping http.Handler adds minimal overhead. The key concern is payload size: if you're logging every request body in a high-throughput API, you will fill your storage quickly. Sample at 1–5% in production, or replay traffic from a staging environment that mirrors production load.

Sanitise PII before storing. This is not optional. If you're in a regulated sector, your test fixtures will inherit the compliance obligations of your production data unless you anonymise aggressively at the point of capture.

At the function level

For internal logic, you need something closer to the code. Python's sys.settrace and the unittest.mock.patch approach can capture function call arguments and return values, but the overhead is significant and it's not suitable for production use. Better to use this in integration test environments where you run production-like workloads and harvest the traces.

Java has ByteBuddy for runtime instrumentation. Node has v8-inspector and async hooks. The specifics matter because the instrumentation approach determines what granularity of behaviour you can capture.

What you cannot easily capture

Non-deterministic behaviour is genuinely hard. If a function returns different results for the same input depending on wall-clock time, external API state, or random seeds, your recorded test cases will be flaky from the start. Before you harvest traces, audit your system for hidden non-determinism. This is often the most valuable output of the exercise, independent of whether you end up generating tests.

What Happens to the Captured Traces?

Raw traces are not test cases. They need transformation.

For HTTP-level tests, the pipeline typically looks like this:

1. Deserialise the recorded request/response pairs.
2. Group by endpoint and response status code.
3. Deduplicate by request shape (not just exact value, but structural signature).
4. Parameterise values that vary (user IDs, timestamps, session tokens) and replace them with fixtures or factories.
5. Emit test code in the target framework: pytest, JUnit, RSpec, whatever the project uses.

Step 4 is where most automation breaks down. Distinguishing "this value varies because it's a natural parameter" from "this value varies because of non-determinism" requires heuristics or human review. Tools like Differ and Karate have partial answers, but no tool fully automates this step without producing noise.

For invariant-based assertions (the Daikon approach), you get output like:

user.balance >= 0
order.items.size() >= 1
response.status in {200, 201, 400}

These become property-based test assertions. Hypothesis in Python is a natural home for them: you write a strategy that generates inputs satisfying the preconditions you've inferred, and assert the postconditions hold.

Is This Worth Doing If You Already Have Good Coverage?

It depends on where your coverage actually is.

Line coverage at 80–90% sounds healthy, but line coverage does not measure whether your assertions are meaningful. A test that calls every line without asserting anything useful is a test that passes when the system is broken. This is sometimes called weak mutation killing performance, measured by mutation testing tools like Mutmut (Python) or PIT (Java). Projects with 85% line coverage routinely kill fewer than 50% of mutations.

Behaviour-derived tests, because they come from real outputs, tend to be stronger on assertion quality. They assert what the system actually returned, not what someone thought it should return. That specificity is the point.

The trade-off is maintenance. A test suite generated from behaviour grows with the system but can also encode bugs. If the system was wrong when you recorded the traces, your tests will assert that wrong behaviour. This is why behaviour-derived test generation works best as a complement to human-written tests, not a replacement for them.

Approach	Coverage type	Assertion quality	Maintenance cost	Risk
Hand-written unit tests	High on logic, low on integration paths	High (intentional)	Medium	Misses undocumented paths
Behaviour-derived (trace replay)	High on real paths	High (empirical)	High (traces go stale)	Encodes existing bugs
Fuzzing	High on input boundary	Variable	Low	Misses semantic correctness
Model inference (Daikon)	High on invariants	Medium (requires validation)	Medium	Over-generalises

Conclusion

Start with HTTP-level trace recording if you're adding behaviour-derived tests to an existing system. The infrastructure is shallow, the output is immediately useful, and the feedback loop is fast. Instrument a staging environment, run a representative workload, and pipe the output through a deduplication and parameterisation step.

The concrete next step: pick one API endpoint that has poor test coverage and high production traffic. Set up trace recording in staging for a week. Count how many distinct request shapes you see that your existing tests don't cover. That number will tell you whether this investment is worth scaling.

FAQ

Does behaviour-derived test generation replace writing tests manually?

No. It supplements manual tests by covering paths that humans don't think to write. Manual tests encode intention; behaviour-derived tests encode reality. You need both. Systems with only auto-generated tests tend to have good empirical coverage but poor detection of semantic regressions, because the tests don't know what the system should do.

What if the recorded production traffic contains personally identifiable information?

Sanitise at the point of capture before traces ever reach your test data store. Use deterministic anonymisation (consistent fake values per real value) so tests that involve related records stay coherent. In regulated environments, treat this as a compliance requirement, not an engineering convenience.

How does this interact with microservices and async messaging?

Contract testing tools like Pact handle the HTTP case well. For async messaging (Kafka, RabbitMQ), the equivalent is recording message schemas and payloads from consumer perspectives and verifying producers against those schemas. Schema registries (Confluent, AWS Glue) make this tractable, but the tooling is less mature than HTTP contract testing.

How much overhead does runtime instrumentation add?

At the HTTP boundary, sampling at 5% with async writes to a queue adds under 2ms per request in typical configurations. At the function level with full tracing enabled, overhead can be 20–40%, which is why function-level instrumentation is restricted to test environments. Never run Daikon-style invariant inference against production traffic.

When is behaviour-derived test generation the wrong choice?

When the system's current behaviour is significantly wrong. Generating tests from a broken system encodes the bugs. Fix known correctness issues first, then instrument. Also avoid this approach for systems with extreme non-determinism where you cannot isolate and control sources of randomness, because the resulting tests will be unreliable from the start.

Have a project in mind? Contact Sodio Technologies to discuss your requirements and explore the right technology solution for your business.