Conformance and testing

Both upstreams ship golden corpora, and both transfer to this port unchanged. They are the definition of "correct" for their half, not a smoke test around it.

The corpora

Suite	Source	Shape
Shell	Bashkit's 2,521 cases	a language-agnostic text format: script, expected stdout, optional exit code
Python	Monty's 568 fixtures	ordinary Python whose body is `assert` statements; a file passes when it runs to completion
Joined	operations taken from real sessions	read a file, search a tree, patch a source file with a heredoc Python program, check the result
Extensions	fixtures for what this port adds beyond Monty	kept in their own folder on purpose

Some Python fixtures instead pin an expected traceback or exception, which makes them error-message conformance tests: the message a model reads back is part of the contract.

Current state

Suite	Passing
Shell	2,521 / 2,521 (27 skipped by upstream directive)
Python	557 / 558
Joined	210 / 210
Extensions	11 / 11

The one Python failure asserts that a temporary's id() is handed to the next object of the same shape, an artifact of upstream's slot-recycling heap. Object identity here is the host runtime's.

Running them

dotnet build
dotnet test

Both work with no arguments from the repository root, and keeping it that way is deliberate: one solution, every project building from a clean clone, xunit on the standard test SDK, so nothing needs a runner selection.

dotnet test tests/Computerwelt.Emulation.Bash.SpecTests     # shell conformance only
dotnet test tests/Computerwelt.Emulation.Python.SpecTests   # python conformance only

The ratchet

Both conformance suites are ratchet-based. A per-file baseline records how many cases each file currently passes, and a run fails if any file regresses below it.

COMPUTERWELT_UPDATE_BASH_BASELINE=1   dotnet test tests/Computerwelt.Emulation.Bash.SpecTests
COMPUTERWELT_UPDATE_PYTHON_BASELINE=1 dotnet test tests/Computerwelt.Emulation.Python.SpecTests

Raise a baseline after making cases pass. Never lower one to make a build green.

The joined suite is absolute rather than ratcheted: every case must pass. It is also the suite that catches what the other two structurally cannot, because it tests operations rather than features, and every defect it has found so far was in code both corpora already covered.

Checking against upstream alone

The extension fixtures exercise things upstream Monty does not have, so running one against upstream fails at the import. To check that the port still stands on upstream's corpus by itself:

COMPUTERWELT_SKIP_EXTENSIONS=1 dotnet test

Nothing in the extension folder may contradict the upstream corpus. Where the two would disagree, upstream wins and the difference is recorded as a limitation instead.

Benchmarks

Upstream's benchmark corpus is carried over case for case, alongside micro-benchmarks for session construction, parsing, the filesystem and the Python half.

dotnet run -c Release --project bench/Computerwelt.Benchmarks -- --report
dotnet run -c Release --project bench/Computerwelt.Benchmarks -- --filter '*Session*'

--report prints microseconds and bytes per operation per case, and checks each case against the output upstream recorded, because a case that stopped producing the right answer is not a faster case. The BenchmarkDotNet path is for before you believe a number. Neither is a ratchet: these are measurements, not acceptance criteria.