Testing Methodology
1. Introduction
1.1 Purpose
This document outlines the testing methodology employed by the development team for the Automata Editor and Visualiser project. It details the testing strategies, tools, processes, and standards that ensure the product meets the required quality benchmarks as defined in the project brief and Sprint 2 milestone criteria.
1.2 Scope
This methodology covers:
- Automated engine unit testing, API endpoint testing, and UI component testing
- Coverage measurement and reporting
- Manual and exploratory testing procedures
- User acceptance testing at sprint milestones
1.3 Project Context
The Automata Editor and Visualiser is a web-based tool designed to help students learn Turing Machines through a dual editor (visual state diagram and code editor), an IO test panel, and a computation visualiser. The project follows a three-sprint development cycle with specific testing requirements at each milestone.
2. Testing Strategy
2.1 Testing Pyramid
Our testing approach follows the testing pyramid model, prioritising fast, isolated unit tests at the base, integration and API tests in the middle, and slower manual and user acceptance tests at the top. The engine layer (parser, test parser, marker runner, simulator, and utilities) is pure, framework-free TypeScript, which makes it the cheapest and most reliable place to catch defects — and it carries the largest share of our automated suite. Component and endpoint tests build on that foundation, and everything that jsdom cannot meaningfully exercise (canvas interaction, CodeMirror surfaces) is pushed to the manual layer at the top.
2.2 Testing Types
| Testing Type | Description | Frequency | Owner |
|---|---|---|---|
| Unit (engine) | Parser, test parser, marker runner, simulator, and TM utilities tested in isolation with hand-built machine fixtures | On every engine change (watch mode during development) | Developers |
| Integration | Endpoint chaining (machine object returned by the parse endpoint fed into the run endpoint) and parser-to-simulator flows | When contracts change | Developers |
| API Testing | Route handlers for the public TM endpoints — validation, error contract, CORS behaviour | On endpoint changes | Backend Developers |
| UI Testing | Render-level component tests in jsdom using Testing Library queries | On UI changes | Frontend Developers |
All automated suites run locally through the test script before merges; the coverage script produces the report described in section 4.
3. Automated Testing with Vitest
3.1 Why Vitest?
We chose Vitest as our primary testing framework for engine, API, and UI testing for the following reasons:
| Reason | Explanation |
|---|---|
| Unified Framework | One tool for engine, endpoint, and component testing, reducing cognitive overhead and configuration complexity |
| Speed | Vitest is significantly faster than Jest, especially for large test suites, thanks to its Vite-powered transform pipeline |
| Vite Integration | The app is built with Vite; Vitest shares that ecosystem while its configuration lives in a separate vitest.config.ts, keeping app plugins (Solid, Nitro, Tailwind) out of the test run |
| TypeScript Support | First-class TypeScript support out of the box |
| Coverage | Built-in coverage reporting through the V8 coverage provider with text, HTML, and JSON reporters |
3.2 Engine Unit Testing
The engine layer is where the majority of our tests live (159 tests across six files). All five modules are pure functions with no DOM, network, or framework dependencies, so tests construct machine fixtures directly and assert on returned values — no mocks or stubs are needed.
The suites cover: DSL parsing (state construction, transitions in both single and tuple syntax, alphabets, variants, error handling), the makeiotest syntax (result aliases, wildcards, cutoff values, malformed lines), the test runner (accept/reject/halt/cutoff classification, tape comparison, boundary behaviour at exactly the step limit), the simulator (stepping, halting semantics, multi-tape machines, complexity metrics, nondeterministic computation trees), and the shared utilities (move validation, variant normalisation, transition normalisation and formatting).
These tests also serve as executable documentation of engine quirks discovered during development: argument whitespace is trimmed when splitting, the tests marker terminates machine parsing, a halt occurring on the last allowed step is not a cutoff, and state references may arrive by id or by label depending on whether the source is an editor snapshot or parser output.
3.3 API Testing with Vitest
3.3.1 How We Test the API
Route handlers are exported as plain asynchronous functions that accept a web-standard Request and return a Response. Tests invoke the handler directly with a constructed request and assert on the response — status code, JSON body, and headers. This means no server needs to boot, no HTTP client harness is required, and each test runs in milliseconds. Route test files opt into the node environment individually since no DOM is needed.
Both endpoints currently under automated test — the parse endpoint (DSL code to machine object) and the run endpoint (machine object plus input to execution result) — are stateless pure functions of the request body. There is no database involvement, so no seeding or cleanup is required, and tests are trivially isolated and repeatable. The authenticated file and auth routes are outside the current automated suite and are covered by the manual regression checklist.
Because the endpoints are public by design (external documentation-site consumers cannot hold sessions), there are no token-based authorisation tests. Instead, the error contract is tested exhaustively: malformed JSON bodies, missing or non-string fields, machines with duplicate state ids, missing start states, transitions referencing unknown states, non-array symbol fields, out-of-range tape counts and step limits, and oversized inputs each return the documented client-error status with a user-readable error message. The preflight handler is tested directly to verify CORS headers are present on every response, including errors.
An integration test chains the two endpoints: the machine object returned by the parse endpoint is passed unmodified into the run endpoint. This pins the shared contract between them, including the label-based state references and multi-tape tape count semantics that a naive consumer would get wrong.
3.3.2 Why This Approach
Testing handlers as functions keeps the suite fast and deterministic, and a failing test points directly at the handler rather than requiring investigation across a running stack. Because the public endpoints are the contract between this backend and the external documentation site, thorough contract testing gives us confidence that both sides can evolve independently as long as the contract is upheld.
3.4 UI Testing with Vitest and Testing Library
We use Vitest together with the Solid flavour of Testing Library in a jsdom environment to test UI components from the user's perspective: components are rendered with minimal props, elements are located by their accessible role, label, or visible text, and assertions are made on what the user should see. This keeps the tests resilient to refactoring because they only care about rendered output, not internal state or component methods.
The current component suite is deliberately render-level: it verifies that each presentational surface mounts and displays the right content rather than simulating full interaction flows. Interaction-level testing (dispatching events to drive tabs, buttons, and inputs) is available through Testing Library's event helpers and will grow alongside the interaction test cases tracked in section 6.
3.4.1 What We Test in Our UI
| Area | Current automated coverage |
|---|---|
| Code editor shell | Mode tabs (code, transitions, tests, complexity) render correctly |
| Playback controls | Toolbar renders input field, play/pause/step controls, speed, and dynamic toggle |
| Computation visualiser | Tape cells and current-state badge render in the DOM |
| Transitions panel | Transition list renders |
| Complexity views | Metrics display and complexity plot render |
| IO test panel | Parsing and execution covered at the engine level; component surface verified manually |
Canvas-based diagram rendering (pan, zoom, fit-to-view), the CodeMirror editor surfaces, the test modal, and machine thumbnails are intentionally outside the automated suite: jsdom cannot meaningfully exercise canvas or CodeMirror internals, so automated tests there would measure mocking rather than behaviour. These areas are owned by the manual regression checklist and are excluded from coverage metrics, as described in section 4.
4. Coverage
Coverage is produced on demand with the coverage script using the V8 provider, with text, HTML, and JSON reporters written to the coverage directory.
The measurement scope covers all engine TypeScript and all component TSX, excluding the test files themselves and six components that are manually tested by design (Toolbar, DiagramView, Editor, MachineThumbnail, TestModal, TestPanel). This exclusion list mirrors the automated/manual split in section 3.4: files where jsdom line coverage would dilute the signal without adding real assurance are left out of the metric rather than dragging the average down misleadingly.
Current state: 202 tests across 16 files, all passing. Overall coverage sits around 80% of statements and 84% of lines, with the engine layer at roughly 94% of statements — the parser and test parser near or at full statement coverage, the utilities at 100%, and the simulator at 85%. A known gap is that route handlers are asserted by their own 35 tests but are not yet included in the coverage report's include list, so their lines are not counted; folding them in is a tracked improvement item.
5. Manual Testing
5.1 Exploratory Testing
Exploratory testing sessions are time-boxed and focused on a specific area of the application. Each session has a charter that defines what to explore, and findings are documented and triaged alongside automated test failures.
This approach has proven its worth during Sprint 2: the sync bug where a malformed test line was silently deleted while the user was still editing it, machines overflowing the diagram canvas before pan and zoom existed, and a duplicated complexity readout introduced by a merge were all found by manual exploration — none were reachable by the automated suites, since each lived in the interaction between user timing and component behaviour.
5.2 Regression Testing
A manual regression checklist covers the critical user paths — signing in, creating and opening a machine, parsing code into the visual editor and regenerating code from visual edits, running a machine on an input with play, pause, step, and speed controls, authoring and running IO tests and reviewing the pass/fail cards, saving and reviewing version history, and smoke-testing the public API endpoints. This checklist is run before each milestone submission.
5.3 Cross-Browser and Accessibility Testing
The application is checked across major Chromium-based and Firefox browsers at typical screen sizes. Accessibility verification is currently manual and pragmatic: keyboard shortcuts are gated so they only fire when their panel is visible, and interactive controls remain keyboard-operable. Formal screen reader audits and automated accessibility scans are planned before the final milestone.
6. Continuous Improvement
After each sprint, the team reflects on what testing went well, what could be improved, and what should change. Testing metrics — coverage percentage, test count, bug discovery rate, resolution time, and flaky test rate — are reviewed to identify trends.
Improvement items currently identified: fold route handlers into the coverage include list so endpoint code is measured; reduce jsdom environment creation cost, which dominates the suite's wall-clock time; extend component tests from render-level smoke checks toward interaction-level coverage; and close the known simulator reporting gap where a machine looping forever in an accept state is reported as accepted rather than cut off.