You have been given access to a new repository with the API designed by another team. Your task is to write tests for the public API and setup CI for the project. This is an individual task, so that every team receives two sets of tests for their own API.

API tests

The purpose of this task is to practice writing tests against a public API without access to the implementation (i.e. black-box style testing). Your tests should cover the functionality provided by the library — review the API requirements from task 1, check whether the library provides some extra functionality and write the tests.

The character of your tests will depend on the API provided by the library. Libraries providing a declarative API may require setting up different situations using data classes and checking that the library had the expected effect after parsing a specific command line. Tests for libraries providing an imperative interface may look just like a set of unit tests, because the API might provide you with a set of classes full of public methods.

Keep in mind, though, that your tests are not supposed to be unit tests that focus on methods in specific classes. You should test the API from the perspective of the client code, focusing on the features provided by the API. This also means that your test classes should be in a different package (to make package-private elements inaccessible), and should not rely on “friend” access (in languages where that applies).

Use the API requirements together with the explicit (documentation) or implicit (sample code) contract offered by the API to derive the expected behavior and write tests for that. If in doubt, define a sensible expected behavior yourself using the tests. Do not forget to test failure cases, i.e., situations involving invalid parser configurations and invalid inputs on the command line.

Because the API has no implementation (yet), both the tests your will write and the tests you will receive will be failing. While that can be somewhat discouraging, a failing test is also a driving force for the implementation, because it makes the expected behavior explicit and thus makes it clear what needs to be implemented.

It is difficult to specify the number of tests that your solution needs to provide, but experience from past years shows that the overall quality of the solution typically correlates with the number of test scenarios. Solutions with 30+ scenarios were markedly better than solutions with less than 20. While there is no upper limit and bonus points were awarded for high number of test scenarios (high-tier), there appears to be a sweet spot somewhere around 25 (common-tier). Solutions with less than 20 scenarios are generally considered low-tier.

When it comes to the actual number of test methods, keep in mind that not all test methods are created equal:

  • A single parametric test method can capture one scenario along with many different inputs to be tested against. A solution with 20 such methods can easily produce a high number (e.g., 100+) of actual “test instances”.

  • A single non-parametric test method usually tests a single scenario with a single data input and testing with different inputs requires multiple test methods. A solution with 30 such methods produces only 30 “test instances” over very few scenarios.

You are expected to primarily use parametric test methods to capture scenarios and use the parameters to test the scenarios with different data inputs. Solutions with 20 parametric test methods are rarely considered inadequate, unlike solutions with zero or very few parametric methods.

In general, strive to provide a reasonable set of tests that you believe would help your colleagues drive their implementation and that you would expect from the other students creating tests for your team’s API.

Finally, use a well-known unit testing framework that nicely integrates with the language; avoid inventing your own. Just keep in mind that even though you will be using a unit testing framework, your tests are not really unit tests.

CI in GitLab

The second part of this task is to setup a simple CI for the project. Simply put, configure the project so that tests are executed automatically on each commit (push) to GitLab.

If you have never configured CI in GitLab, we suggest to consult the reference manual and examples for unit testing.

When setting up the CI, use separate stages for build/compilation and for running the tests.

Suggestions

The following is a short summary of things to watch out for in your tests, because they keep appearing over and over in the submitted tests.

Naming

  • Test method names should reflect the situation being tested and the expected outcome.

    • StackTest.popFailsOnEmptyStack() is a better name than StackTest.testPop().

    • Avoid test prefix in the name if you language supports annotations, e.g., @Test (Java) or [Test] (C#), or similar.

    • Avoid should in the test method name, just describe the expected outcome.

    • Like in any other names, avoid sequential numbering of methods, e.g., testFoo1(), testFoo2(), etc.

  • Prefer the language of the problem, not the implementation. There are even libraries (e.g., catch for C++) where you don’t have to deal with test method names and just use plain English descriptions.

Coverage

  • Cover different situations in a systematic fashion, based on the requirements from task-1, and organize the tests accordingly (i.e., not necessarily by classes, but by scenarios).

  • Don’t forget to test robustness with respect to invalid inputs. The API should be explicit in how it reacts to such inputs — error handling IS part of design.

Implementation

  • Make the test code obvious for the reader.

    • The reader must be able to see that you are arranging the right situation, executing the right action, and performing the right asserts to determine the test outcome.

    • In other words, strive for the Arrange-Act-Assert (AAA) structure in your test code.

  • Keep tests small and test one situation at a time.

    • This also means that any testable conditions from the arrange phase should be tested using separate tests (not asserts). The assert phase should only concern the particular situation being tested.

  • Make test inputs “visible”.

    • Unlike non-test code, constant literals are tolerated and often preferable in tests, because they are part of the scenario being tested (e.g., invalid inputs, boundary conditions).

Exceptions

  • Declare expected exceptions using test annotations, don’t catch them.

    • Unexpected exception should crash (fail) the test.

    • If multiple statements may throw the expected exception, use the assert throws facility of your testing framework, which typically provides an assertThrows() method which takes code to execute as a lambda and an exception type to expect. This may also indicate that the API does not provide exception types corresponding to the provided level of abstractions.

Parametrization

  • Use parametric (theory) tests for cases where everything but the input data changes and provide the data in a declarative way.

    • Strive to keep the tests obvious even when using parametric (theory) tests. This typically requires capturing the expected outcomes in the provided data. You can always comment on the data, or include an extra (dummy) string in the data containing a textual description.

  • Avoid code duplication (while keeping the code obvious).

    • Extract common parts into test setup phase or create parametric helper methods (i.e., build a “vocabulary”) to setup various scenarios. Just make sure the code remains readable.

Sources

  • Treat test code with the same care as your other (non-test) code.

    • Put the code in the right package, split the code into multiple files where applicable, use white space (empty lines) to separate elements (methods), etc.

Use of generative AIs

Using AI assistants is discouraged, but not prohibited. In general, you may use AI to improve your own work but not to do the work for you.

Paraphrasing an excerpt from Ethical use of AI: Guidelines for Caltech applicants: If you are wondering where the boundary is, ask yourself whether it would be ethical to have, e.g., a teacher perform the task (you are asking of the AI) for you. Would a teacher be able to review your code and suggest improvements? Of course! Would that same teacher write most of the code for you to tweak and then submit? Definitely not.

That said, you MUST disclose any use of AI along with a summary of how it has been used (to help shape the policy on AI use in the coming years).

Submission

When you are satisfied with your solution to this task, make sure it is in the master branch of your task-4 repository and tag the commit with the task-4-submission tag.

Note on tagging

Use the git tag task-4-submission command instead of putting the tag name (or a hashtag) into a commit message — that is not a proper Git tag. Then push the tag to GitLab using the git push --tags command.