Automatically marked question. Question where the Candidates enter a code snippet that is automatically run and validated.

Content

• Introduction
• Useful Terminology
  • Code Editor
  • Test Cases
• Options
  • Programming Language
  • Edit Default code snippet
• Question Settings
  • Marks & Mark Scheme
  • Feedback, Main Illustration, PDF panel, and Labels
  • Marking test cases
  • Sample test cases
• How to use Input and Expected Output
• Testing the code as the Candidate
• Testing the code as the Author
• Known Limitations

Introduction

This is available for Authors.

Code Compile is a question type in which the Candidate can write code, which is automatically compiled (if it’s a compilable language) and run and validated against test cases defined by the Author.

Useful Terminology

Code Editor

The Candidate writes the code using a code editor, with syntax highlight and line numbering like in the Programming question type. The Candidate does not need to open a new tab to use this editor, as it’s presented as an input box. For some languages, some degree of auto-correction is also supported. As an Author, you can choose whether this should be available or not. 

Test Cases

A test case consists of a pair of input and expected output, i.e., when the Candidate's code is run against the provided input, the test passes if the output of the Candidate's code matches exactly the expected output.

In the example of the above image, when the input “hallo” is inserted, then the correct output should be “hllo”, i.e., the “a” was removed.

There are two types of test cases in a Code Compile Question:

• Marking Test Cases: These will be used during the marking of the question and are not accessible by the Candidate;
• Sample Test Cases: These are accessible by the Candidate. They can run their code against these test cases and see which ones pass and which ones fail

Options

Upon clicking on the answer template the Author can define the question Prompt, which by default is Fill in your answer here and to some options specific to the Code Compile question type.

Programming Language

The Author can choose from a wide variety of code languages and compiler versions.

Edit Default code snippet

After choosing the Programming Language (and closing and reopening the Code Settings), the Author can write some code to be presented to the Candidate alongside the question text. This code is always visible and editable by the Candidate.

The code submitted by the Candidate needs to contain all it needs to be run, including any imports, includes, etc. For instance, in languages like C++, there needs to be a main function and in Java, the class must be called Main. The Author can provide this code ahead of time to reduce the chances of error by the Candidate.

It’s also useful to use this to help the Candidate using the Input and Output Stream.

Question Settings

Upon clicking on the question text, which by default is “Replace with question text”, the Author can edit the question text as they want, using all of the text editor options available.

The Author also has access to some options related to the question:

Mark & Mark scheme

These define how the marking of the question should be done.

The Candidate's code is run against the Marking test cases. If all the cases pass, then the Candidate obtains the total score. If at least one of the cases fails, then the question is marked as 0.

As such, only the Marks for all correct settings are used when scoring the question. For more information on Marking, please see Question Scoring - Basic.

Feedback, Main illustration, PDF panel, and Labels

These are settings common to all question types. For more information on each of them, please check the General Options for Questions.

Marking Test Cases

The Candidate's code is run against the marking test cases to evaluate its correctness and mark the question accordingly. The Author can define as many Marking test cases as they wish. The Candidate has no access to the Marking test cases.

Each test case consists of an input and an expected output. The top right cross can be used to delete that specific marking test case. It’s important to note that if any marking test case fails, the question will be marked as 0 (see Mark & Mark scheme section).

Sample Test Cases

Sample test cases work similarly to the Marking test cases but with a different purpose. These test cases are visible to the Candidate and the Candidate can run their code against them to test it before submitting the question. No marks are associated with these test cases.

How to use Input and Expected Output

For a test case to correctly run, the Candidate needs to read the input from the Input Stream and send the output to the Output Stream. For instance, in C++, the input would be obtained from std::in and the output would be sent through std::out. The output and the expected output must be an exact match, so the Candidate should not send any other information through the output stream, otherwise, they will not match.

Testing the code as the Candidate

Before submitting the question, the Candidate can test their code against the Sample test cases by clicking the “Test Code” Button.

After some seconds the Candidate will see the result of each of the sample cases if the code compiled successfully. For each test case, the Candidate will see the Input, Expected output, Status (Correct or Wrong), and how long it took to run each of them.

If a compilation error occurs an error message is shown to the Candidate.

By clicking on Close (both if the compilation was successful or not), the Candidate goes back to the Question, where they can continue editing it or move to the next question.

Testing the code as the Author

While previewing the question, the Author has the same experience as the Candidate. However, the Author can choose to run the code against the Sample test cases or the Marking test cases, by clicking on the “Check validation of evaluation test cases” button, which is not available for Candidates.

After running the code against the Marking test cases, the Author can run it again against Sample test cases by clicking on “Check validation of sample test cases”.

Known limitations

• The Candidate can only test their code against the sample test cases defined by the Author.

• Neither the Candidate nor the Author can see the Actual Output of the Marking/Sample Test Cases, only the Input, the Expected Output, the Status (Correct or Wrong), and the execution time.

• The code submitted by the Candidate needs to contain all it needs to be run, including any imports, includes, etc. For instance:

  • For languages like C++, there needs to be a main function

  • In Java, the class must be called the Main

• This question type does not have offline support.

• Service-level agreements do not include this question type.
• There may be instances of delay or lag experienced by users.
• In the event of code compilation failure, there will not be any automatic retries and the Candidate will need to manually attempt again.
• We do not recommend using this feature on large tests i.e. over 500 Candidates.
• The question is not compatible with the design setting "Show warning when navigating away from pages with incomplete responses". For more information, refer to the Design settings in question sets article.