Numerical simulation

  • Updated
The Numerical Simulation question type icon

Automatically marked question.

Question with a numeric input field and a dynamically calculated correct answer.

The Numerical Simulation question type in Inspera Assessment is a programmatic, automatically marked question type. Candidates answer using numeric values generated by a program model (for equation responses, see Symbolic Math). Questions and answers are randomized each time the question is generated for a candidate. 

This article provides an introduction to the Numerical Simulation question type, explains its core authoring elements, details how to configure response outcomes, outlines testing and previewing procedures, clarifies how marking works, and lists known limitations.

Understanding the question type

Numerical Simulation utilizes a Program Model that enables simulations using an expressive programming language called Maxima, which is based on Lisp. By setting variables, their parameters, and equations within this Program Model, values within the question and their answers are randomized each time the question is generated for a candidate. The allocation of values to the Program Variables happens when the test is started, ensuring unique variables for each test attempt.

The “Pump Water” example used in this document is shared with permission from the creator - the UNSW School of Banking and Finance, University of New South Wales, Sydney, Australia.

Click to Download the QTI package of the question shown in the video

Authoring a Numerical Simulation question

There are three core elements when authoring a Numeric Simulation question: the Question Text, the Program Model, and the Response Outcome.

  • The Question Text: These are the fixed elements of your question. Only the values of the program variables you insert within it will change.
  • The Program Model: Define all variables, their parameters, and calculations here.
  • The Response Outcome: Decide here how the system will evaluate the candidate’s response.

Instructions

  1.  
    1. First start by selecting Author > Questions.
    2. Click Create new.
    3. Under Automatically marked, click Numerical Simulation. 
  2. The Question Text contains the fixed elements of the question. The only parts that change are the values of the program variables you insert within the question text.
    1. Author the question text in the main authoring area
    2. To insert a program variable, click + Insert followed by Program Variable wherever you'd like the randomized value to appear. This will insert a variable placeholder.

    3. To add a candidate input field, click + Insert followed by Candidate Response. This will insert a response placeholder. 

      The question editor interface when inserting a candidate response.

  3. To set all the variables and their parameters, you must edit the Program Model. The Program Model uses MAXIMA for its operations. In MAXIMA, variables are declared by first writing the variable name, followed by a colon. After the colon, you specify the value or parameters for the variable. Remember, each line of code must end with a semicolon to be correctly interpreted.

    1. From the general options panel on the right-hans siden, click Program Model > Edit Program Model.

      The Program model general option

    2. Write the Program Model. When finished, click Compile to verify your syntax and ensure the example values are being generated as expected. 

    3. Click Save. 

      The program model editing interface.

    For a step-by-step example of a question being authored, see the video at the top of this article.

    For further detail on the Program Model syntax, see The Program Model article.

  4. Once the program model is complete, return to the question text. Click on each Program Variable placeholder you inserted. From the Program Variables dropdown, select which variable you would like to display. 

    the Program Variables dropdown options

  5. To set the type of input you expect from the candidate, as well as the different ways you would like to validate them, you must set the Response Outcome for each Candidate Response.
    1. Click on each Candidate Response field and select Edit response outcome.
    2. Select how you would like it to be evaluated from the options below:
      • Equal: The candidate input is exactly the value of the variable as set-out in the programme model.
      • Abs. Tolerance: The candidate input can be between a set value lower and higher than the variable as set-out in the programme model. See the article Absolute and relative tolerance for more information.
      • Rel. Tolerance: The candidate’s input can be between a higher and lower range relative to a percentage of variable as set-out in the program model. See the article Absolute and relative tolerance for more information.
      • Unequal: The candidate input is any value other than the value of the variable as set-out in the programme model
      • Greater than: The candidate input is greater than the value of the variable as set-out in the programme model
      • Less than: The candidate input is less than the value of the variable as set-out in the programme model
    3. Select the variable to be evaluated against
    4. Select the outcome: Correct, Partially Correct, or Incorrect

    5. Set the Response-Spesific Feedback.

      You can provide response-specific feedback within the Response Outcome. This allows you to tailor automatic feedback based on whether a candidate's answer was correct, incorrect, or perhaps a common error you factored into your Program Model. 

      This feedback can also include the variable values themselves from the Program Model, so candidates see the exact numbers they were presented with when they attempted the question, rather than just variable names.

      See the article Response feedback for further information.

  6. You can further customise the response outcome by visiting the Options menu.

    1. Select the Options drop-down.
    2. From here, you have the following options for input validation which restricts what the candidate can type:
      1. Expected length.
      2. Expand input field automatically.

      3. Restrict number of characters to input width.

        The response outcome options

    Note: The formatting of the Program Variable is not affected by the input field setting. It must for now be defined in the Program Model.

  7. Once you've finished setting the Question Text, Program Model, the Response Outcomes, simply click Save.

    If you'd like to preview the question, click Preview (eye icon) in the top right.

    As an author, each time you refresh your browser the variables will be recalculated. This is not functionality available to candidates. For candidates, once variables have been calculated, they remain fixed for that attempt.

How to preview the question

You can preview your question during the authoring process to see what it looks like for the candidate.

  1. Within the Author editor, in the upper right-hand corner, click Preview.
    • Program variables are displayed with their numeric value in the preview pane.
  2. Enter an answer within the Candidate input field and click Check answer. To enter a different Candidate input, click Try again.
    • Wrong answers are given the message Wrong.
    • Correct answers are given the message Correct.

Question preview

While you are previewing the question, please make sure that the decimal separator is behaving as you want. 

  • The default behaviour of decimal separators is that both comma and point are allowed, but comma will be transformed to a point when the response is stored. Candidates will see the decimal separator as a point when they navigate back to a question they have already answered, or when they view their submission after the test. 

    Example where comma and point are allowed in Numerical Simulation.

    If your organisation wants to enforce decimal point instead of comma as a decimal separator in this question type, please contact the Service Desk for activation.

    Please be aware of the following when enforcing decimal point as decimal separator:

    • Enabling this will also add specific information about decimal points to both Authors and Candidates.

      Enforcement of decimal point in Numerical Simulation.

    • This will also apply to the Numeric Entry question type

Override automatically calculated marks

Numerical Simulation is an automatically marked question. Automatically marked questions are automatically marked by Inspera Assessment once the candidate has submitted the test. As the Grader, you do not need to mark an Automatically marked question, unless you intend to override the mark. Automatically marked questions can be found within the Marks dropdown, and are identified with Automatically calculated.

Numerical simulation question types that contain Error carry forward are rewarded with partial marks.

Marking workspace view of a Numerical simulation question.

As a Planner or Grader, you can override the automatically calculated marks. 

  1. Within the Marks dropdown, Click Override.
  2. Within the text field, enter your marks (up to 2 decimal places).

  3. Click Apply.

    Marking workspace view for overriding automatic scores.
  4. Once you have overridden automatically calculated marks, the status of the question changes to Overridden. 

Error Carry Forward

Numerical Simulation has Error Carry Forward functionality, which means that an error in an earlier part of a candidate's calculation might not penalize them in subsequent steps if those subsequent steps are logically correct based on their earlier incorrect answer.  See Error Carry Forward for further information.

Known limitations

  • Feedback per response outcome is not displayed in Candidate report.
  • Correct answer not included in downloaded question set PDF.

Was this article helpful?

2 out of 2 found this helpful