Automatically marked question. Question where the candidates enter a code snippet that is automatically run and validated.

The Code Compile question type in Inspera Assessment allows candidates to enter a code snippet that is automatically run and validated. This question type is currently in Open Beta access and requires an active Internet connection to function. 

To use it, the question type must be activated for your Inspera Assessment environment; please contact your primary contact for more information.

This article provides an introduction to the Code Compile question type, explains useful terminology for authors, details how to author these questions, clarifies the role of test cases, outlines the testing process for both authors and candidates, and lists known limitations.

Overview and key terminology

The Code Compile question type is designed for automatically marking programming questions. Candidates write their code within a dedicated editor, which is then compiled (for compilable languages) and run against test cases predefined by the Author.

• Candidates write their code using a specialized code editor that features syntax highlighting and line numbering, similar to the Programming question type. This editor is presented as an input box directly within the question, eliminating the need for candidates to open a new tab. For some languages, a degree of auto-correction is also supported. Authors can choose whether this auto-correction should be available.

• You define test cases to validate the candidate's code. Each test case consists of an input and an expected output. For a test case to run correctly, the candidate's code needs to read input from the Input Stream and send output to the Output Stream. 

  

  As shown in the example image, if the input is “hallo,” the correct output should be “hllo,” indicating that the letter “a” was removed.

  For instance, in C++, input is obtained from std::in and output is sent through std::out. The output and the expected output must be an exact match, so the candidate should not send any other information through the output stream.

Authoring a Code Compile question

When authoring a Code Compile question, you will set up the question text, define the programming environment, and provide any default code.

You can configure general options for this question type in the side panel. These options include settings for marks, mark scheme, feedback, and more. For more information on the general options, refer to the Overview of Question Options in Inspera Assessment article.

The following general options are specific to the Code Compile Question Type:

• These define how you should do the marking of the question

  The candidate's code is run against the Marking test case If all the cases pass, then the candidate obtains the total score. If at least one of the cases fails, then the question is marked as 0.

  As such, you only use the Marks for all correct settings when scoring the question. For more information on Marking, please see Basic method for question scoring.

• The candidate's code is run against the marking test cases to evaluate its correctness and mark the question accordingly. The Author can define as many Marking test cases as they wish. The candidate has no access to the Marking test cases.

  Each test case consists of an input and an expected output. You can use the top right X icon to delete that specific marking test case. It’s important to note that if any marking test case fails, the question will be marked as 0 (see Mark and Mark Scheme section). 

• Sample test cases work similarly to the Marking test cases but with a different purpose. Candidates can see these test cases and can run their code against them to test it before submitting the question. No marks are associated with these test cases.

Specific options

Upon clicking on the answer template the Author can define the question Prompt, which by default is Fill in your answer here and to some options specific to the Code Compile question type.

• Authors can choose from a wide variety of code languages and compiler versions. This selection automatically provides syntax highlighting adapted to the chosen language.
• After choosing the Programming Language (and closing and reopening the Code Settings), Authors can write code that will be visible and editable by the candidate when they start the test. This code appears alongside the question text.
  • The code submitted by the candidate needs to be fully runnable, including any necessary imports or includes. For instance, in languages like C++, there must be a main function, and in Java, the class must be called Main.
  • The Author can provide this code ahead of time to reduce the chances of error by the candidate and guide them in using the input and output streams.

Testing the code

•  

  1. Before submitting the question, the candidate can test their code against the Sample test cases by clicking the Test Code button. 

     

  2. After a few seconds, the candidate will see the result of each of the sample cases if the code compiled successfully. For each test case, the candidate will see the Input, Expected output, Status (Correct or Wrong), and how long it took to run each of them. 

     

  3. If a compilation error occurs, an error message is shown to the candidate. 

     

  4. By clicking on Close (whether the compilation was successful or not), the candidate goes back to the question, where they can continue editing it or move to the next question.
•  

  1. While previewing the question, the Author has the same experience as the candidate. However, the Author can choose to run the code against the Sample test cases or the Marking test cases by clicking on the Check validation of evaluation test cases button, which is not available for candidates. 

     

  2. After running the code against the Marking test cases, the Author can run it again against Sample test cases by clicking on the Check validation of sample test cases.

     

Known limitations

• The candidate can only test their code against the sample test cases defined by the Author.
• Neither the candidate nor the Author can see the Actual Output of the Marking/Sample Test Cases, only the Input, the Expected Output, the Status (Correct or Wrong), and the execution time.
• The code submitted by the candidate needs to contain all it needs to be run, including any imports, includes, etc. For instance:
  • For languages like C++, there needs to be a main function
  • In Java, the class must be called the Main
• This question type does not have offline support.
• Service-level agreements do not include this question type.
• There may be instances of delay or lag experienced by users.
• In the event of code compilation failure, there will not be any automatic retries and the candidate will need to manually attempt again.
• We do not recommend using this feature on large tests i.e. over 500 candidates.
• The question is not compatible with the design setting Show warning when navigating away from pages with incomplete responses. For more information, refer to the Design Settings in Question Sets article.