The Steps

Each phase consists of up to 4 steps.

Allocation defines how the submissions in the phase are distributed, how many Graders will mark them, and whether the whole submission will be marked by a grader or if different questions will be allocated to different graders

Marking controls how each question and submission is marked and what other grading information is displayed to the Grader. As a Grader’s responsibilities and requirements can change from test to test, access permissions cannot be set at a user-level. Instead, what graders can see and do is controlled at the phase level of each test. This provides greater flexibility and control.

Calculations are used to mathematically resolve differences in marks between multiple graders, or if the difference is too great to apply agreed calculations then mark them as a conflict. Marking submissions as conflicts is a way of flagging them to be moved to another phase so they can be evaluated again.

It is possible to set up to eight (8) calculations per Phase. If more are required, use Rules to move the remaining submissions to a new phase where a new set of calculations can be set-up.

Rules are used to decide where the submission goes next or if the grading is complete.

It is possible to set up to eight (8) rules per Phase. If more are required, create a Rule to move the remaining submissions to a new phase where another set of rules can be set-up.

Allocation

Start

Manually: The phase will not start until a Planner has manually selected for the phase to start. 

Candidate to Mark

All submissions in phase: Every submission allocated to this phase will proceed through the subsequent steps

Randomly select n submissions: If a random sample is to be selected for this phase, input the number of submissions required and move all other submissions to the next phase by selecting their destination from the dropdown. The submissions that have not been randomly selected will bypass the remainder of this phase.

Distribution

Random: All submissions in the phase will be randomly allocated to groups (and therefore graders) based on the workload percentage set for each group.

Explicit: When the phase opens, a list of all submissions will be available and the Planner can explicitly state which submissions and going to which groups (and therefore which graders).

Graders per submission

Single grader: Each submission in the phase will be marked by a single grader.

Multiple graders: Each submission in the phase will be marked by more than one grader, this includes when questions will be allocated to different graders. When multiple graders are marking the same question, any differences will be handled in the Calculations step.

Allocate Graders by

Question: [Available November 2026] It will be possible to allocate different questions to different graders to mark. This includes multiple graders marking the same questions as blind marking.

• Allow Graders to view (read-only) questions not allocated to them. Selecting this option will allow graders to view all questions in the submission but they will only be able to mark those assigned to them. If this option is not selected, graders will only be able to view and mark questions assigned to them.

Submission (Candidate): Each grader will mark the entirety of the submissions assigned to them.

Marking

No marking in this phase

If no marking is required for this phase, select the checkbox. This is to be used when a phase only requires rules to reroute submissions.

Evaluation type

Mark/rubric set on question: The grader will have the ability to give a question a mark based on the values set by the Author on the question. This includes any value from 0 to the maximum value of the question

Approve/Do Not Approve: This cannot be used in the first phase as there is nothing to review. This option is to be used when read-only access is required and the Grader simply needs to acknowledge they have reviewed the submissions. No mark will be assigned in this phase. 

Currently the level of approval is for the entire phase. The Grader will have a single Approve and Do Not Approve option which will cover all submissions in the phase. 

Grader’s visibility of other grading

The following options control what each grader can see in the phase.

Each grader can:

• View Marks: Marks given by other graders will be viewable in the summary table under the marking area of each question. This will include submission totals.
• View Grader Names: Graders’ names will be included alongside the marks they awarded for each question in the summary table

Of:

• This phase: the selected information of other graders in this phase will be available for the grader to see. If other graders in the phase have not completed their marking, those values will display as ‘Pending’ for the relevant questions and totals.
• Previous phases: the selected information from all previous phases will be available to the Grader in this phase

No visibility of other grading: Graders will see no other grading information.

Calculations

How calculations are prioritised

With the first implementation of Grading Workflows, Calculations are processed in order. Therefore if you have conditions that overlap, the first sequential calculation where the condition is TRUE will be the one that is applied.

eg.

Scenario A

Calculation 1: Difference is equal to or greater than 40 then -Calculation 2: Difference is equal to or greater than 50 then -

Scenario B

Calculation 1: Difference is equal to or greater than 50 then -Calculation 2: Difference is equal to or greater than 40 then -

In Scenario A, Calculation 2 becomes redundant because all differences greater than 40 will be captured by Calculation 1

In Scenario B, Calculation 1 will capture all differences greater than 50, and Calculation 2 will capture all differences between 40 < 50.

Source for Calculation

Select which phases will be included as part of the calculation. By default the current phase will be the only one selected. If previous phases are selected as well, you must select whether you want to use the individual Grader marks or, if calculations have been applied in that phase, the resolved marks (ie. the mark after a calculation was applied). 

For example: 

• Phase A
  • Grader1= 72 marks
  • Grader2 = 80 marks
  • Calculation is raw average
  • Resolved Mark = 78
• Phase B
  • Grader3 = 82 marks
  • Calculation is raw average including Phase A

If Grader marks from Phase A is selected, then the Phase B resolved mark would be Phase A Grader marks + Phase B Grader mark divided by number of values. (72+80+82) / 3 = 78

If Resolved marks from Phase A is selected, then the Phase B resolved mark would be Phase A Resolved Mark + Phase B Grader mark divided by number of values. (78 + 82) / 2 = 80

Calculation based on

Submission: The only mark that will be used in the calculation is the total (ie. the sum of all question marks).

Calculation condition

The difference is based on the marks at a question or submission level, depending on the selection of the previous setting.

Any difference: Unless the marks are of equal value, this condition becomes TRUE. If Any difference is selected, no other difference conditions can be selected and this covers all eventualities.

Difference is equal to or greater than: If the difference between the lowest and highest marks being evaluated is equal to or greater than (≥ n) the inputted value, this condition becomes TRUE.

Difference is equal to or less than: If the difference between the lowest and highest marks being evaluated is lower than (< n) the inputted value, this condition becomes TRUE.

Between: If the difference between the lowest and high marks being evaluated is equal to or greater than the first number (≥ n1) but less than the second (< n2), this condition becomes TRUE

Difference of grades: If the provisional grade for the submission is different between any graders, this condition becomes TRUE. This option is only available when calculations are based on the submission and grading thresholds have been enabled for the test.

If condition met, then 

Any question or submission (depending on based on setting) that matches a TRUE condition will have its resolve mark calculated as follows:

Flag as conflict: No calculation applied.

Calculate as Average:

• Raw: The resolved mark will be set at the average rounded to the nearest 3 decimal places
• Round up to nearest: The resolved mark will be averaged and the rounded up to the nearest multiple of n*
• Round down to nearest: The resolved mark will be averaged and the rounded down to the nearest multiple of n*

* n can be no greater than 100. It can also be up to 3 decimal places.

Set as highest mark: The resolved mark will be the highest mark of any of the graders

Set as lowest mark: The resolved mark will be the lowest mark of any of the graders

Rules

Applies to

Individual submission: Each submission in the phase will be processed based on its individual mark or grade.

Based on

Mark: Rules in the phase will be applied based on the numerical total mark of the submission

Grade: Rules in the phase will be applied based on the grade awarded to the submission

Each Rule:

Rule Condition

Entered: If any resolved mark or grade as been decided in the phase, this condition is TRUE

Equal to: If the resolved mark or grade is equal to the value inputted, this condition is TRUE

Equal to or greater than: If the resolved mark is equal to or greater than (≥ n) the inputted value, this condition is TRUE

Equal to or lower than: If the resolved mark is lesser than (< n) the inputted value, this condition is TRUE

Between: If the resolved mark is equal to or greater than the first number (≥ n1) but less than the second (< n2), this condition is TRUE

Conflict: If the submission was flagged as a conflict in Calculations, this condition is TRUE

Approved: If Approved / Not approved was enabled in the Marking step and the phase as Approved, this condition is TRUE

Not approved: If Approved / Not approved was enabled in the Marking step and the phase was Not Approved, this condition is TRUE

If condition met, then

Any submission that returns a TRUE condition will either continue to the next phase or have reached the end of the grading workflow.

Move to Phase: This will route the submission to whatever phase is selected from the dropdown

Grading Complete: This last resolved mark will be treated as the final mark and grade (if applicable) and the submission’s status will be considered complete.

All other submissions (Else)

All other submissions (Else) is a catch-all for the submissions that do not match any of the Rule conditions. If follows the same logic as the Then settings.

Move to Phase: This will route the submission to whatever phase is selected from the dropdown

Grading Complete: This last resolved mark will be treated as the final mark and grade (if applicable) and the submission’s status will be considered complete.