Mathematical modelling for academic performance status reports in learning analytics

The fast changing nature of the educational environment and the subsequent increase in the volumes of generated learner data, have found existing data analysis techniques lacking in certain fields. These techniques form part of the analysis and reporting phases of learning analytics and need to adapt to accommodate the changing face of education. In this paper, a set of interrelated algorithmic solutions that utilise mathematical programming models to generate and provide learning feedback in the form of academic performance status reports, is presented. Three existing mathematical models, more specifically the benchMark program, an outputs–only data envelopment analysis and a traditional analytic hierarchy process were evaluated for providing the information required to assist students in improving their academic achievement. The requirements include providing students with their current academic performance status, setting interim improvement goals and calculating improvement targets towards reaching those goals. The evaluated models did not address the requirements satisfactorily. The solution proposed in this paper consists of an algorithm that implements a linear programming model to generate performance status reports based on the current assessment scores of a group of students in a module. The output is used in a second algorithm that utilises the remaining improvement opportunities available to generate a participation future time perspective. The resulting schedule together with each individual student’s current assessment scores, is used to calculate discrete improvement goals for each student as well as targets towards reaching those goals. A third algorithm provides a lecturer with some insight into the mastering of module content.


Introduction
The pace at which technology is advancing necessitates role players in education to adapt their teaching and assessment techniques [24].Most of the activities relevant to education can now be performed by means of a computer.This makes online learning, where students do not need to be at a campus to interact with a lecturer, possible.The subsequent increase in student numbers results in an increase in the learner data being generated which includes the use of resources, academic performance, and even effectiveness of teaching tools.Analytical methods used to process student data consequently need to adapt as well.Many tools implemented as educational aids already perform some analytical functions on institutional, regional, national, and international level [39].
Predictive analytics incorporates statistics, machine learning and data mining to discover meaningful patterns in data [25].The focus of learning analytics is, however, on the learning process rather than on an institutional level [39].It presents the potential to predict and improve student success and resulting retention by incorporating the measurement, collection, analysis and reporting of learner data [8].Although many methods exist with which these actions can be performed, measurement and collection of learner data are mostly context dependent.Various methods for analysing and reporting such data are commonly implemented in academic environments by means of learning management systems (LMSs), which consider learning activities, course administration, and tracking [40].Examples like PACE (Progress and Course Engagement), GPS (Grade Performance System) [31], and E2Coach (Expert Electronic Coach) [27] are only some LMSs that determine and provide alerts regarding academic issues.Other systems that can furthermore present interventions include the Student Advice Recommender Agent (SARA) [17] and the MiGen teacher assistance tool, which is a system that was developed with some visualization and notification capabilities [26].However, Stantchev et al. [40] argue that LMSs are not yet optimally contributing towards the improvement of learning because they mostly address institutional needs rather than those of the students.Students need specific information relating to their progress in a course to enable change [13].
Learning analytics can change the value of higher education by assisting in the identification of students who are at risk of failing and to provide them with interactive solutions on how to improve during the semester (term).Barker and Garvin-Doxas [4] proposed the use of a hierarchical structure for the purpose of progress feedback to create a communicative environment between a lecturer and the students.Studies show that ranking is an important determinant of student behaviour [14,42].Therefore, a solution in the form of a student ranking incorporated into the analysis and reporting phases of learning analytics, is proposed.
Ranking models have been applied in the ranking of NBA basketball players [28], college sport coaches [19], alternatives in sports tournaments, products, political candidates [6], and others such as the PageRank algorithm introduced by Google [21].The main aim of this study is to evaluate existing ranking models that may be used for student feedback and to develop an improved ranking feedback model by implementing mathematical programming techniques in three interrelated algorithms.Three models were chosen for evaluation in this specific research project.They include benchMark [11], an outputs-only based data envelopment analysis (DEA) model [22], and the traditional analytic hierarchy process (AHP) [2].The benchMark program was selected because it is an empirical student ranking model that ranks students according to their academic progress in a module.It was successfully implemented in a software application and was evaluated in an academic environment with positive results.Furthermore, the benchMark program is directly related to this study.The outputs-only based DEA was chosen because in general, no measurable inputs (e.g.numbers of hours studied) are available on students.It allows the calculation of outcome improvement targets and it is an established ranking method.The AHP is another technique that ranks items influenced by several factors, or in this study, to rank students according to their formative assessment outcomes like class test scores, practical assignments, etc. Appropriate adjustments and pairwise comparisons were made in this study to eliminate traditional inconsistency-and normalisation problems.This process also allows for the allocation of weights (as predetermined by a lecturer) to the factors.The AHP was applied in this context to operate as an additional means of appraising the rankings obtained in other methods.
The motivation for the study is founded in the repeated observation that student attendance in some courses at tertiary institutions, seems to wane after the initial contact session.Students think that because the semester had just started their absence cannot yet affect their academic progress and when they do return to class, they do not know what to do to improve their performance [11].There is a demand for an academic performance feedback system that will continuously provide students with information on their current academic status in a module as well as how to improve from an early stage in the semester [18].
The remainder of this paper is organised as follows: Section 2 provides some background on a typical teaching-learning environment and a short discussion on the requirements for effective feedback.Subsequently, the three chosen ranking models are discussed in Section 3. Finally, an improved mathematical ranking model that provides effective feedback to students is proposed in Section 4. This is followed by a discussion in Section 5 and conclusions in Section 6.

The teaching-learning environment
In a typical classroom environment, the role of a lecturer ranges from teaching and facilitation to assessment and grading.Students are expected to prepare for and attend contact sessions where lecturers will discuss previously completed work, explain how it relates to new concepts, and introduce and discuss new material.Often during contact sessions students are expected to complete class assignments and may also be given homework related to topics discussed during class.Regular formative assessments in the form of class tests, practical assignments or informal tuition tests among others, can be taken without prior notice.This form of assessment provides lecturers with valuable information on the success of the teaching techniques employed when introducing new material to students.
Lecturers grade formative assessments and use them to compile an overall semester mark for each student.This mark, called a participation mark (p-mark), is the measurement by which students are granted admission to the final examination.Some lecturers also reward students for class attendance by letting it contribute a small quantity towards the p-mark.In the calculation of the p-mark a lecturer has the responsibility to decide which factors he/she wants to consider, and how the averages in each of these factors are to be calculated.The p-marks can then be calculated as the weighted sum of the average marks obtained for each factor according to where p is the participation mark, y 1 , • • • , y n are the respective average marks calculated, w 1 , • • • , w n are the respective weights, and n is the number of factors.A lecturer normally decides on applicable weights for each of the respective factors.In some environments, if the p-mark is larger than a certain minimum threshold the student is allowed to write the final examination.The final module mark is then calculated as the average of the pmark and the examination mark.This means that students have the opportunity to start building half of their module mark from as early as the first scheduled contact session.Students are, however, not always aware of how they are progressing towards their pmarks.Regular and interactive feedback is needed to keep students informed on how their p-marks are growing.
Ranking is considered a feedback format that will effectively inform students on their positions in a hierarchy based on their outputs.The proposed solution will provide feedback in the form of performance status reports (PSRs) throughout the semester.For a PSR to be effective it must address three key issues [18].Firstly, students must know what the goal is that they are working towards.This requirement is referred to as feed-up.At the start of a semester in an academic environment such a goal can be to pass a module or get a distinction.However, as the semester advances and students start to progress or regress, interim goals based on their performance can help them reach their ultimate goals.Secondly, they must know what progress they are making towards reaching the goals, or feed-back.Lastly, students must be informed as to which activities they have to perform in order to enhance their progress, or feed-forward.Furthermore, Wiggins [45] provided some attributes to which the information in a PSR must conform to.Apart from having to provide feed-back, feed-forward and feed-up, the reports generated by the solution in this paper will also be evaluated for referring to a goal, being actionable, user-friendly, timely, ongoing and for being provided consistently.In the next section, different methods for presenting student progress in the form of academic rankings, are evaluated.

Ranking as feedback in learning
Ranking as a hierarchical structure is used in the academic environment for many purposes, among others for evaluating quality in higher education [9,16], for determining the success of different departments within a university [5], for student selection [15] and for ranking specific student attributes to determine the probability of success [35,37].In this section, the three selected models will be discussed and applied to the same data set.This set contains marks for 26 students in four factors that were achieved in a fourth-year module presented at a tertiary institution.The factors (denoted F 1 to F 4 respectively) are a semester test, class tests, theoretical assignments and attendance.The data set is provided in Appendix A.

The benchMark model
A software application called benchMark was developed by Du Toit [11] for the purpose of timeously calculating p-marks and rankings for a group of students.The program creates a participation profile for each student which is continuously provided to them as PSRs throughout the semester.
The model utilised in this program is a standardised form of equation (1).After completion of each assessment the new set of marks is used to calculate a new average for each factor.Each student's current factor averages are then used to create a p-mark according to where w j is the weight assigned to factor j, y αj is the calculated average for factor j of student α, and n is the number of factors.A unique attribute of benchMark is that instead of requiring a specific known weight for each factor beforehand, it empirically calculates a set of possible p-marks for each student using a range of permissible weights per factor.Permissible weights have minimum and maximum values, with their sum constrained to be 1.A drawback to this approach is the exponential increase in the number of possible p-marks that need to be calculated as the number of factors increase.Initially it was found that the calculation of all possible combinations of weights would result in an intractable problem [11].To cater for this, a priority sequence constraint was implemented on the weights and they were constrained to be in multiples of a predetermined number k.The priority sequence constraint requires the lecturer to predetermine the order of importance for each of the factors.It was empirically determined that by enforcing these constraints the search space was reduced considerably, resulting in a tractable problem.
The program also calculates the average of all the possible p-marks for every student, including the minimum and maximum.After each execution of the program the maximum, minimum and average p-mark, and a student ranking according to the average p-marks, are available.Statistics relating to the current pass/fail rate, the number of distinctions and the number of students without admission to the examination, are also available.
During the semester students are frequently given feedback on their latest factor averages, their minimum p-marks, and ranking.Instead of informing the students what the respective weights for the factors are going to be at the start of a semester, the option exists to provide them with the order of the priority of the factors.This affords some freedom in selecting a specific factor weight distribution closer to the examination and students have a discrete measuring tool with which they can gauge their progress and/or improvement in the module.The program therefore guarantees that the final weight distribution selected at the close of a semester, will not cause any student's final p-mark to drop lower than the minimum mark provided to them after having processed all relevant assessments.
The model was implemented and tested with k = 0.05.The priority sequence applied in the evaluations was w 1 ≥ w 2 ≥ w 3 ≥ w 4 with the weights applying to F 1 to F 4 respectively.Based on these constraints the program calculated a set of 23 possible p-marks for each student from which each respective student's average p-mark was calculated.For illustration purposes the participation profiles of the top seven students are presented in Table 1 (the profiles of all the participants are provided in Apart from providing student p-marks and ranking status, other advantages of the bench-Mark program include the following: • Performance status can be provided to students as frequently as assessments are graded; and • Additional throughput-and performance information is available to the lecturer.
Although benchMark has been successfully implemented, it uses all weight combinations allowed within the model constraints to empirically calculate the p-marks and ranking.
The disadvantages that were identified are listed below.
• Although some students (especially those at the top of the ranking) prefer to know their ranking, others exhibit a rather indifferent attitude towards their progress and feel that they need discrete information on how to improve to higher ranking positions.The benchMark program does not provide improvement targets that will assist in improving students' ranking.• The program uses empirical calculations in a sequential algorithm to create a set of p-marks for each student and to set up statistical information.It was found that for less than 80 students the execution time is reasonable and the program can be executed a few minutes prior to a contact session [11].However, for large student numbers program execution time does not scale linearly and it may be more practical to perform the calculations only once a week.• The benchMark program initially has a steep learning curve, both for students and lecturers.• Execution of the benchMark program needs continuous human involvement making importing of marks and calculating new p-marks rather labour intensive.• Without the priority sequence-and the multiples-of-k constraint sets the search space is so large that the problem is intractable.
To address some of the disadvantages identified in evaluating benchMark (the human factor, the iterative and sequential nature of the program, and its intractability), the model was formalised in the non-linear mathematical model [43] that follows.
Maximise/minimise p α = n j=1 y αj w j , subject to the constraints with k ∈ R + , l ∈ R + , u ∈ R + and a small positive number.
The objective function is similar to the formulation in (2).The constraint set in (5) ensures that each w j is a multiple of k while (7) represents the sequence ordering requirement of benchMark.Weights are constrained to be between an upper and lower limit in (6).
The model is non-linear (due to constraint set ( 5)) and may be solved using any nonlinear mathematical programming software.For the purpose of this study it was decided to obtain a heuristic solution using the Evolutionary Solver in Excel.Evolutionary solvers are designed to mimic the process of biological evolution in certain ways.A full discussion of heuristic solutions and the optimisation of non-smooth models is beyond the scope of this paper and detailed explanations can be found in Baker [3].The choice of a solver was based on the fact that the solution generated in this manner does not rely on the mathematical structure of the model but rather on solutions generated by combining previous solutions along with some randomness [33].It should be noted that there is no guarantee of a global optimum and human judgement is required when solving a model using an evolutionary approach.The implementation of a heuristic solution using an evolutionary solver is simple and allows for easy use of mathematical functions such as the modulus function in constraint set (5).
The same data set used to illustrate benchMark, was used to evaluate the non-linear model in ( 3)-( 8).The model was solved again with k = 0.05, and the same priority sequence applied to the factors as in benchMark.The maximum and minimum p-marks were calculated accordingly, and the average p-mark was determined from these two values only.The students were again ranked according to the average p-marks as produced by the model in ( 3)-( 8).
Table 2 shows the calculated p-marks, the top seven of the subsequent rankings, and the associated weights generated by this heuristic solution.
The ranking for all 26 students (all student rankings can be found in Appendix B, Table 10) differs marginally from that produced by benchMark.Student 10, for example, is ranked 4 th whereas this student was ranked 5 th by benchMark.The reason for this difference is the fact that benchMark calculated the average p-mark for each respective student from all 23  of the empirically calculated possible p-marks as opposed to the non-linear model which used only the calculated maximum and minimum values.To produce a more structured comparison between the two rankings, a Spearman's rank correlation coefficient [7] was calculated.Spearman's rank correlation coefficient (ρ) is a nonparametric assessment of the statistical relationship between two different rankings which expresses the relationship between them in terms of a monotonic function.It is expressed as where d i is the difference between the two ranks at the i th position, and m is the number of ranking positions.The rank correlation coefficient between the two rankings was calculated as 0.99, proving a very strong positive association between them.This is to be expected considering the similar constraints applied in both methods.
Formalisation of the benchMark model in a non-linear mathematical model yielded improvements on some of the disadvantages identified earlier: • The need for continuous human intervention is reduced as the model parameters are specified at the onset of execution and the calculations are repeated for each student by means of an algorithm.• The evolutionary solver is easy to implement and converges to acceptable solutions for the entire data set, faster than benchMark did.This model can therefore be used more frequently and for large student numbers.• Although a heuristic solution does not always guarantee to be optimal, the model does find acceptable solutions.This was proven by the high Spearman's rank correlation coefficient calculated for the respective rankings resulting from the benchMark program and the non-linear mathematical model.
The multiples-of-k constraint set (5) was enforced in the non-linear model so that the resulting ranking could be compared with that obtained in benchMark.However, this constraint creates a non-linear relationship between the decision variables and some predetermined constant.
Enforcing this constraint set was justified in benchMark, but its exclusion will result in a linear mathematical model which, under certain conditions, can be solved in polynomial time [30].The model was therefore modified and solved as a linear model, which is discussed in more detail in Section 4.1.
Another drawback to the benchMark and non-linear modelling approaches is that no discrete improvement targets are calculated with which students can attempt to improve their rankings.This means that although these methods address the feed-back requirement for effective PSRs, no feed-forward or feed-up is provided.In a pilot study done by Du Toit [11] it was noted that although students at the top of the ranking felt motivated to improve, those in the middle and lower parts of the ranking indicated that they wanted more information on exactly how to improve.One of the attributes PSRs must have, is that they must be actionable.In other words, students have to be provided with some form of incentive or target that shows them what they can do to improve their rankings or more specifically, feed-forward.This problem was partly addressed by implementing an outputs-only based data envelopment analysis (DEA) on the student mark data set [44], and is discussed next.

Class-ranking by means of an outputs-only based DEA
DEA is a mathematical programming method commonly employed to compare service units called decision making units (DMUs) in a specified environment, based on a calculated ratio between their inputs and outputs [29].However, student input data such as study environment, hours of study or lifestyle aspects are considered either unavailable or unreliable.Therefore, a model that utilises inputs was not included in this study.Outputs-only based DEA effectively categorises DMUs considered equally efficient into groups or classes.Examples of such analyses include the class-ranking of management colleges [22], and of secondary schools [23].To be able to compare the student rankings, an outputs-only DEA was performed on the marks (factor averages) obtained in certain outputs (factors) [44].The students were treated as DMUs and the factor averages were considered the outputs.This method effectively searches for the most favourable weights and calculates an efficiency score for each student [22] by solving the model in ( 10)- (12).
subject to the constraints where E α is the composite index for student α, y ij is the output measure of student i in factor j, w j is the weight for factor j, n is the number of factors, m is the number of students to be evaluated, and is a small positive number ensuring all factors are considered.
This model was used to create a class-ranking that effectively sorted the students into groups of similar efficiency.The principle of Pareto optimality applies wherein the efficiency score (E α ) represents the ratio of the current student's performance to that of all the other students in that specific factor.If E α = 1, then student α is considered Pareto optimal which means that no output can increase without decreasing another [41].Such a student would then be categorised into the highest available class, lie on the Pareto frontier, and is said to dominate those in lower classes.If E α < 1, then student α is Pareto non-optimal and is said to be dominated by those in higher classes.Detailed discussions of Pareto optimality and the dominance principle can be found in Kao and Lin [22], and Seiford and Zhu [38].
The outputs-only DEA was also performed on the same data set containing marks for 26 students, in which the four factors as described in the previous sections, are considered as outputs.This resulted in a class-ranking that categorised the students in the data set into 8 different classes of efficiency (Figure 1). Figure 1 shows the class-ranking: one student was classified into class 1, followed by two students in class 2 and so forth.The student class-ranking cannot directly be compared to the rankings obtained in Section 3.1, for a few reasons.The outputs-only based DEA method calculates the efficiency scores by determining the weights that will benefit each student most, with no predetermined factor priority sequence included.Also, the basis for calculating E α is determined by the outputs of the students on the Pareto frontier and is therefore not necessarily the same for each dominated student [16].Although the class-ranking produced by this model cannot be compared to the rankings resulting from the methods in Section 3.1, the students classified into the first three classes, correspond with the top seven from Section 3.1.
One of the disadvantages identified in Section 3.1, was that neither the benchMark program nor the non-linear model offered students any actionable targets with which they could attempt to improve their rankings.A major advantage presented by the outputs-only DEA model formulated in ( 10)- (12), is that by implementing its dual formulation, discrete improvement targets per student per factor could be calculated.The dual formulation [22] is expressed in ( 13)- (15).
subject to the constraints The variable definitions are the same as for the model in ( 10)- (12).Furthermore, let θ = m i=0 λ i , then the target for factor j can be calculated as (y αj +s j )/θ and j = 1, . . ., n.For each student not in class 1 (c = 1), this model would calculate n(c − 1) improvement targets, one per factor for every class higher than the current.Consider, for instance, Table 3   If Student 1 wanted to progress to class 3, the semester test (F 1 ) must improve from 45% to 49%, class tests (F 2 ) from 55% to 58% and so forth.Likewise, for each student not categorised into the highest class, targets were calculated in each factor for improvement into the next higher class.Unfortunately the model does not cater for factors that have been completed.A semester test for example, is written once in the middle of the term and the mark cannot change after that.Also, any changes in student marks will have a ripple effect on the entire class-ranking as follows: • the model would need to be solved again and new optimal weight sets will be determined for each student; • new weights per student could mean that the Pareto frontier for each class could change; and • the outputs of all students are used in calculating the improvement targets.This means that even if a student reaches a target, the class-ranking might not change if other students improved in the same factor.
Another drawback to the outputs-only based DEA is that because the output of a student is compared to that of every other student, the size of the problem increases considerably when performing this analysis for modules with large student numbers.A solution to this problem is proposed through the construction of an algorithmic approach which is discussed in detail in Section 4.2.
The outputs-only DEA model can be used to calculate individual improvement targets for each student, but it has certain disadvantages.Another student ranking was generated in this context by means of the AHP for comparative purposes, and is discussed in the next section.

An AHP model for ranking students
The analytic hierarchy process (AHP) is a mathematical method used to make decisions when multiple objectives or criteria need to be considered [34].The method is used to find the best contender among a certain number of alternatives (students), based on a number of criteria (factors).
Each one of the students is compared to the others in terms of each of the factors.This is followed by a comparison of the importance of the factors resulting in an average rating for every student.
Typically, the following steps are performed [29]: 1. Develop a ranking for each student for each factor: • develop a pairwise comparison matrix for the students in each factor; • normalise the resulting matrices; • find the normalised principal eigenvector, or priority vector; and • check the consistency of the pairwise comparisons.
2. Develop the weights for the factors: • develop a pairwise comparison matrix for the factors; • normalise the resulting matrix; • find the priority vector; and • check the consistency of the pairwise comparisons.
3. Calculate the weighted average rating for each student.Rank them according to their average ratings.
Pairwise comparisons are based on a preference scale which usually ranges from 1 to 9 [1].The ratings show how much one alternative is preferred above another, and vice versa.In this application pairwise comparisons were made with student factor averages and the rating scale was modified as follows: At tertiary level an admission point score (APS) is used to determine whether a learner who completed the South African National Senior Certificate, may study at a South African university [12].An adaptation of the APS system was therefore used for generating the pairwise comparison ratings of student averages where the difference between the mark for factor j of student α and student β is calculated as y αj − y βj , α, β ∈ {1, . . ., m} and α = β, with m representing the number of students.These calculations are then associated with the normal AHP preference scale of 1-9.Table 4 shows this association for y αj ≥ y βj .For cases where y αj < y βj , the inverse of the rating is used.If for example y αj < y βj for the interval 20%-29% then the corresponding rating would be 1/4.The non-linear model discussed in Section 3.1 yielded a set of weights used for calculating the minimum and maximum p-marks of each student (Table 2).To enable comparison between the AHP ranking and those obtained in previous sections, the weights were pre-set to averages calculated as wj = ( wj(max) + wj(min where wj is the average weight for factor j, wj(max) is the average of all the weights calculated for factor j for the maximum p-marks and wj(min) is the average of all the weights calculated for factor j for the minimum p-marks.The weights used in the AHP are 0.35, 0.30, 0.20 and 0.15 for w1 to w4 , respectively.The weights can be varied at the lecturer's discretion but lecturers do not always know what the expected class throughput resulting from specific weight combinations might be.This problem was addressed by formulation of a non-linear weight selection model and is discussed in more detail in Section 4.3.
Consistency checks on pairwise comparisons are usually required because with a large number of decision alternatives inconsistent comparisons occur frequently.A consistency ratio is calculated [10] using the formula where is the consistency index with λ max the principal eigenvalue, m the number of students, and RI is a random consistency index [36].Generally, the degree of consistency is acceptable if CR < 0.10.The scale used for pairwise comparison of student marks in this study is considered consistent because it compares each pair of marks by means of a mathematical equation (Table 4).Therefore, it was not necessary to perform consistency checks for these pairwise comparison matrices.
The number of comparisons (s m ) to be made for m students and n factors, is expressed as Accordingly for the mark set of 26 students and four factors, a total of 1300 pairwise comparisons were made.
The process was implemented on the same mark set used previously.The marks of the students were rated according to Table 4 by calculating the difference between each pair of marks and setting up the pairwise comparison matrix based on the given preference scale.The matrix was completed and normalised for all of the 26 students in the set after which the averages were calculated to determine each factor priority vector.This step was performed for all four of the factors.Using the preference weights as discussed previously, a composite rating r αj , was calculated for each student by finding for student α, with γ αj the average priority rating of student α in factor j, w j the pre-set weight for factor j, m the number of students, and n the number of factors.
The students were then ranked according to the calculated composite ratings.Table 5 shows the top seven rankings determined by the AHP (the complete set is available in Table 5: Top seven of the student ranking resulting from the AHP. The use of the AHP on student marks in different factors, is an unconventional approach to student ranking.It creates a ranking without the calculation of p-marks and can be used to provide an alternative form of feedback.The Spearman's rank correlation coefficient, ρ (Section 3.1) was calculated to compare the AHP ranking to those obtained previously.
The AHP ranking correlated with the benchMark ranking by ρ = 0.97, and with that of the non-linear model, by ρ = 0.98.
One of the problems with using the AHP is that although the resulting ranking highly correlated with those from other methods, a different normalisation order can lead to different rankings [32].Another disadvantage is that the AHP is difficult to use with large volumes of data because the number of pairwise comparisons that need to be made is directly related to the number of alternatives to consider [20].This method also did not address the feed-up or feed-forward requirements.Despite these difficulties, the results showed the AHP to be another technique which can be used for ranking students.For this study, the AHP also acted as an efficient control measure, by which the rankings obtained in the other methods could be evaluated according to the Spearman's rank correlation coefficient.
The existing mathematical models discussed in Section 3 were implemented on the same data set to generate different student rankings.The rankings were evaluated according to specific criteria essential to a PSR, as identified in literature and practice.Although some of the criteria were addressed, other issues remain unattended.To this end, a solution consisting of three different but related algorithms, is proposed and is discussed in Section 4.

Proposed mathematical model for effective PSRs
The requirements for effective PSRs in an academic environment were identified in Section 2 and used as evaluation guidelines for the three methods evaluated in Section 3.An effective PSR needs to address these important aspects: it must feed-up or provide a goal to work towards, feed-back or present current performance, and feed-forward or offer actions to perform towards reaching the goal.The evaluated methods were implemented as mathematical models which were chosen for specific reasons and applied on the same data set for comparative purposes.Although each one of those methods offered a ranking solution and addressed some of the requirements, none of them satisfied all of the criteria for PSRs.The benchMark program, non-linear model and the AHP model provided feedback but neither feed-forward nor feed-up.The outputs-only based DEA model addressed the feed-forward criteria by providing factor improvement targets and partially addressed the feed-back criteria in the form of a class-ranking.However, numerous disadvantages were identified that makes this method unsuitable for the use in PSRs.Evaluation of these methods is illustrated in Figure 2. As Figure 2 illustrates, none of the methods comprehensively addressed the feed-up requirement which demands that students be continuously presented with new goals aimed at progression.The requirements for an effective PSR has therefore not been addressed completely.Evaluation of the methods in Section 3 presented some valuable insights that led to the proposal and development of an algorithmic solution that is illustrated in Figure 3 and discussed in subsequent sections.An important issue to consider when generating the information required for PSRs, is the fact that as time passes by, so does a student's opportunity for improvement.At the start of a semester when only a few assessments have been completed, the factor averages are calculated from the only scores available.Each assessment therefore has a greater influence on the calculated p-mark at an early stage.As a semester elapses and the number of assessments remaining in each factor decreases, the contribution of each individual assessment towards the p-mark diminishes and the opportunities to improve decreases.Figure 3 illustrates how the requirements for PSRs relate to one another in the proposed solution.A linear model for p-mark calculation and its associated algorithm are discussed in Section 4.1.This is followed by the algorithmic approach to calculate improvement scenarios in Section 4.2.The algorithms in Sections 4.1 and 4.2 provide the information required for a student PSR.Section 4 will conclude with a weight selection model that provides feed-back to a lecturer, which is discussed in Section 4.3.

Linear model for p-mark calculation
To address the feed-back requirement, p-mark calculation is performed by means of a linear model according to the framework proposed in Figure 3.The linear model is based directly on the non-linear model ( 3)-( 8) discussed in Section 3.1.It was converted to a linear model by omitting constraint set (5) which stipulated that the weights be in multiples of a number k.This constraint set can be excluded because no empirical calculations are required to find a solution.The model was implemented by means of Algorithm 1.
The linear model was solved for the data set in Appendix A. For the purpose of validating the model the p-marks were used for compiling a student ranking.This ranking correlated favourably with that of the benchMark program (ρ = 0.96) as well as with that of the non-linear model (ρ = 0.97).It may therefore be accepted that the proposed linear model produces acceptable results and may be used in the proposed framework of Figure 3 to address the feed-back requirement.The results of this model are attached in Appendix C and are used as input to the algorithm implemented to address the feed-forward and feed-up requirements, which is discussed in the next section.

Improvement target calculation and potential participation plan
At the start of a semester a lecturer typically plans a certain number of assessments to be taken by the students in each of the factors.As the semester progresses and assessments Algorithm 1: Linear modelling of p-mark calculation Input : Y = {ȳ 1 , . . ., ȳm } is an array representing the current factor averages for a group of m students in n factors.Output: A set of arrays P = {p 1 , . . ., pm } with the minimum and maximum p-marks, their associated weights, and the average p-marks for m students.are given by the lecturer, the remaining assessments in each of the factors decrease.Feedforward is therefore dependent on the point in time at which it is generated and must consider completed as well as remaining assessments in each factor.This time dependency means that different scenarios exist according to which students can complete a semester but with limited combinations of assessments available in each factor.An algorithmic approach was followed to determine the available scenarios at any specific point in time.
The purpose of the algorithm is to provide a student with a future time perspective based on assessments planned for the remainder of the semester, the student's current factor averages and minimum p-mark.This perspective will provide students with insight into the effect that their continued participation or lack thereof, can have on their minimum p-marks.
The approach takes as input the current factor averages for a student at a specific time during the semester, the number and layout of the assessments remaining in each one of the appropriate factors and the required percentage improvement (δ).A table, based on truth table layout principles, is developed containing all the possible participation scenarios for the remainder of the semester.If there are assessment opportunities remaining, there would be a total of 2 s possible participation scenarios.The assessment plan is then used to eliminate participation scenarios that are impossible within the time constraints stipulated.This approach effectively caters for students engaging in, or missing out on any number of the remaining assessments.The algorithm provides output in the form of a set of participation scenarios based on the remaining possibilities and shows corresponding scores that a student must achieve in each scenario to affect the desired improvement in each factor.The projected change in the minimum p-mark is also shown for each scenario.The algorithmic approach is based on mathematical modelling principles and is shown below.
The algorithm starts by developing the participation scenarios in lines 1-4 and continues to calculate the scores required in upcoming assessments to produce the desired improvement Algorithm 2: Potential participation plan for a student Input : ȳ = {ȳ 1 , . . ., ȳn } is an array representing the current factor averages for a student in n factors, A = {ā 1 , . . ., ās } and array denoting the s remaining assessment sessions and δ the required percentage factor improvement.Output: An array xr , with r possible improvement scenarios.
1 declare q(2 s , n); // to store all scenarios for s sessions 2 for c = 1 to 2 s do 3 populate q with participation scenario c; 4 end 5 remove duplicate rows in q; 6 r ← number of rows(q); 7 declare marks(r, n); // scores required for a required improvement of δ 8 declare min f assessments(n); // minimum assessments per factor to cause a required improvement of δ 9 for j = 1 to n do 10 determine e j from āj ; // remaining number of assessments in factor j 11 determine t j from āj ; // 'total number of assessments in factor j 12 for u = 1 to e j do 13 read y u from ȳ; where marks j is the score the student must achieve in the next η j assessments of factor j, t j is the total number of assessments in the semester for factor j, e j is the remaining number of assessments for factor j and y αj is the current factor average in factor j for student α.In lines 15-22, these scores are used to determine the minimum number of assessments to be completed to achieve an improvement of δ in the factors for which it will be possible.In lines 23-32 the possible scenarios are reduced further by comparing the number of assessments in the remaining scenarios to the minimum required in each factor.
For a number of s remaining assessment sessions, this approach can be illustrated according to Figure 4.As an illustrative example, consider the student data set in Appendix A with four factors and 26 students and an example set of five remaining assessment sessions as shown in Table 6.Table 6: Example of a future assessment plan for the remainder of a semester.
In session 1, a student can participate in one attendance (F 4 ).In session 2, a student can participate in one attendance (F 4 ) and one theoretical assignment (F 3 ), and so forth.
In this example, assessments can be taken in more than one factor per session and it is assumed that if a student participates in one session, all assessments given during that session are completed.By attending class in the second session for example, a student can obtain a mark for attendance (F 4 ) as well as a theoretical assignment (F 3 ).Assume that each factor is represented by a number that increases when a student participates in an assessment.In the example, a set of four numbers will then represent assessment participation in F 1 , . . ., F 4 respectively.The possible participation options available for the assessment plan given in Table 6 were generated by means of a truth table in Algorithm 2 and is illustrated in Figure 5.
Each node represents a unique participation scenario for the semester, with self-loops indicating no further participation for the remaining assessment sessions.The set {0, 1, 0, 2}, Figure 5: Possible participation scenarios for a student with the assessment plan as laid out in Table 6.
for example, represents a scenario in which the student will have completed only one assessment in F 2 and two assessments in F 4 at the close of the semester.
The total number of possible participation scenarios for this assessment plan is 2 5 = 32.
The algorithm was implemented for the assessment plan in Table 6, for Student 22. Table 7 shows a potential participation plan with an improvement target of δ = 5%.
The table shows the current factor averages and minimum p-mark for Student 22, followed by the scenarios possible for a 5% improvement in the relevant factors.Note that for attendance (F 4 ) a student can either get 100% for being in class or 0% for being absent.The scores for F 4 were, therefore, calculated as with similar variables as in (22).The layout of the assessments in Table 6 and their time Improvement of a maximum of 5% on factor averages for Student 22 Current performance status report: Factor averages(%): 40.00 54.07 42.67 88.89 45.48 Number of remaining assessments at current stage in the semester: Total number of assessments for the entire semester: Improvement plan: The scenarios for which improvement is possible in F 2 , F 3 and F 4 Scores (%) required for a 5% improvement: dependency were used to establish which scenarios can lead to an improvement of 5%.As the assessment plan included no more available assessments in F 1 , the improvement plan shows all the possible scenarios to produce an improvement of 5% on F 2 and F 3 .
Consider Scenario 3 in Table 7: to increase the current factor averages of F 2 and F 3 by 5%, the student will need to complete at least one assessment in F 2 with a score of 74.07%and at least two assessments in F 3 with an average score of 65.17%.The average for F 4 will increase to 92.59% if the student participates in all four assessments available.If all of these conditions are met, the increase in minimum participation mark will be 2.96%.Although this increase in minimum p-mark seems small, it is important to note that it is dependent on the required improvement and the point in time at which these scenarios are calculated.Early in the semester, the number of remaining assessments per factor will have a greater influence on the extent to which the minimum p-mark can change than at a later stage.Furthermore, due to the small value of δ the resulting change in minimum p-mark is also small.The algorithm was also implemented for the same student with δ = 10%, and resulted in the same three scenarios shown in Table 7.The maximum possible increase in minimum p-mark for Scenario 3 in this instance was 5.9%.Should a higher improvement percentage be required, it may be possible that the targets cannot be reached because the remaining number of assessments is not enough.In the example a required improvement of 15% under the same conditions, will be impossible.
An advantage of using this algorithm is that by setting δ = 0% and consequently excluding any improvement, it can be used to generate a projected report for a student who does not plan to participate in any number of further assessments.Students can build their factor averages up to a certain stage and then fail to participate in any further assessments for numerous reasons, for example illness.Providing them with a projected status record that shows how much their minimum p-marks can decrease, can provide the motivation for their continued participation.Scenario 0 = {0, 0, 0, 0} was generated for the current factor averages of Student 22 and the assessment plan in Table 6 and showed that the student's minimum p-mark can decrease by 7% if there is no further participation in any further assessments.This scenario will have more serious consequences earlier in the semester when there are many assessments to be completed for each factor.
Algorithm 2 effectively addresses the feed-forward as well as feed-up requirements.Feedup is provided in the form of the predicted increase or decrease in minimum p-mark.
Feed-forward is presented in the form of target scores that a student must obtain to reach said change in the minimum p-mark.
Although the combination of Algorithms 1 and 2 effectively and efficiently addresses all three requirements for student PSRs, lecturers sometimes require additional information before deciding on the weights to use for final p-mark calculation.To assist a lecturer in deciding which weight combination to select for calculation of the final p-marks, a nonlinear model was formulated and implemented in an iterative algorithm.This algorithm was developed to provide feed-back solely to the lecturer and is discussed next.

Non-linear programming model in a weight selection algorithm
The linear model in Section 4.1 produced not only the p-marks and subsequent student ranking, but also the weights used to calculate the maximum and minimum p-marks per student.A lecturer decides only at the close of a semester which factor weights to use for the calculation of the final p-marks.Generally, lecturers have certain guideline throughput objectives (e.g. a 70% pass rate or p-mark class average) according to which they can measure the success of a group of students.It will, however, be useful to know throughout the course of a module whether a certain objective is reachable with the marks available at a certain point in time.To this end, a non-linear model was formulated to determine the required weights that will produce a p-mark class average as close as possible to some predetermined target.The model is expressed in (24). Minimise subject to constraints ( 4) and ( 6)-( 8) in Section 3.1.In this model, p t represents the target p-mark class average, p a is the current p-mark class average calculated with fixed weights as with m the number of students and n the number of factors.Due to the non-linearity of the model, it can be solved using a heuristic approach.As with the non-linear model in Section 3.1, the Evolutionary Solver in Excel was used.The approach for determining the weights by means of the model in (24) was implemented according to Algorithm 3.
p-marks obtained in Section 4.1 as input to determine an estimate of the weights required to reach a class average as close as possible to a specific target.

Discussion
None of the existing models evaluated in Section 3 fully addressed all of the requirements for effective PSRs.In Section 4, three algorithms were discussed that not only address the requirements but also provide additional feed-back to a lecturer.The three algorithms are closely related because the feed-back is used to generate the feed-forward and feed-up which are included in a PSR.This is illustrated in Figure 6.In Figure 6, the arrows represent the passing of time and show how feed-back is required to set interim goals, or feed-up, as well as to calculate improvement targets as feed-forward.
As time passes, the number of completed assessments grows and the number of remaining assessments decreases.Feed-back therefore progressively gets more accurate and closer to the final marks that will be used for examination admission.The decreasing size of the feed-back circle is therefore an indication of the decreasing number of remaining assessments.The interim goal(s) as well as the related improvement targets, are calculated based on completed and remaining assessments and therefore get stricter as time passes and the remaining assessments decrease.This means that closer to the end of a semester, certain improvements may not be possible to achieve within the remaining available assessments.The algorithmic solutions offer specific as well as general contributions towards PSRs in the academic environment.These are discussed below.

Specific remarks
The ranking produced by Algorithm 1 proves to be acceptable because of the high Spearman's ranking correlation between it and those obtained from benchMark and the nonlinear model discussed in Section 3.1.The algorithm eliminates the need for iterative empirical calculations.Furthermore, the weights calculated by Algorithm 1 are considered more accurate than those obtained by benchMark because they are not restricted to be integer or in multiples of a predetermined number.
Algorithm 2 uses the minimum p-mark and its associated weights to develop a future time participation perspective for a student.The scenarios according to which the required improvement can be reached are determined and provided.The algorithm can be used to generate a participation plan at any point in time during the semester.
Algorithm 3 provides additional progress information to a lecturer which can be used to determine the weights required to reach a target p-mark class average.The calculated weights may also provide a lecturer with some insight into how successful students were in each factor.A specific factor weight which is much higher than all the others would for example indicate that the students overall performed much better in that factor than in all the others.
Although the proposed algorithmic solutions provide the information essential to PSRs, some disadvantages were identified and are listed accordingly.Algorithm 2 was developed to generate participation scenarios for an individual student.The algorithm can be used iteratively to generate improvement plans for an entire class.However, depending on the marks, the same improvement percentage will not be possible for all students.Therefore every time the lecturer generates an improvement plan for a student, an improvement percentage related to the student's current performance is required.This makes execution of the algorithm for an entire class rather labour intensive.
If the upper limit weight constraints in Algorithm 3 need to be relaxed because a target p-mark class average cannot be reached, the resulting weights may not conform to the guidelines set by the lecturer at the start of the semester.

General remarks
The purpose of the proposed algorithmic solutions was to address the demand for a system that will continuously provide students with academic performance status reports in a module, at any point in time during a semester.The discussed algorithmic solutions address the feed-back, feed-forward and feed-up requirements and furthermore exhibit the attributes necessary for it to be constructive and to motivate students to improve.
The proposed framework provides ongoing feed-back that is timely because it can be used as frequently as required.The results are readily available and can be made available to students instantly.The approach is user-friendly and consistently uses an updated set of the students' current marks to obtain the information needed for feed-back.Furthermore, the suggested approach provides a student with personalised and goal-referenced feedup which is based on the student's current performance.The feed-forward improvement targets are presented in the form of actionable scores that a student needs to acquire in a specified number of upcoming assessments, in each factor.
The collective algorithmic approach sufficiently generates feed-back, feed-up and feedforward for academic PSRs based on a set of students' current performance.Implementation of the framework could also be useful in environments where PSRs are required for motivation to participate in specific learning opportunities or assessments or also, for example, in sports activities or tournaments.

Conclusions
The purpose of this study was to evaluate existing mathematical models against the requirements for effective PSRs and develop an improved algorithmic approach for that purpose.An example data set containing the marks 26 students in four factors (Appendix A) was used throughout the study to evaluate the methods and solutions.
Three existing models were evaluated.The benchMark program was assessed and formalised in a non-linear mathematical model.Although these options did address feedback in PSRs, other disadvantages made them impractical to use regularly and under certain conditions.An outputs-only DEA was performed to group students into classes of similar efficiency.Improvement targets (feed-forward) were then calculated to help students advance to one class higher.The interim goal (feed-up) for each student was not clear because it was not linked to the p-mark.The method was independent of weight constraints and impractical to use for classes with many students.The analytic hierarchy process provided a means to rank students without calculating their p-marks but did not address the feed-up or feed-forward requirements.
To address the deficiencies, a set of related algorithmic approaches in which alternative mathematical models are implemented, was developed.The reports generated were consequently assessed in terms of the required attributes for effective PSRs.Three algorithms were developed to attend to the requirements for PSRs.Firstly, a linear programming model was implemented in an algorithm to address the requirements of feed-back in the form of a student ranking.The next algorithm generated a future participation and improvement plan for each student and addressed the feed-up and feed-forward requirements for PSRs.A non-linear model was implemented in a third algorithm to provide additional feed-back to a lecturer during the course of a semester.Some aspects that are currently receiving attention to improve the proposed algorithmic approach include the calculation of the maximum possible improvement per individual student in a class and automatic generation of the future perspective plans; and development of an interactive user-friendly graphical interface that combines the different algorithmic activities.

Appendices
A. Data set used in the mathematical models

Figure 2 :
Figure 2: Evaluation of selected methods for PSRs.

Figure 3 :
Figure 3: Proposed solution to address PSR criteria.

1 for i = 1 to m do 2 read ȳi from Y ; 3 calculate
minimum p-mark and associated weights for student i; 4 calculate maximum p-mark and associated weights for student i; 5 calculate average p-mark for student i; 6 populate pi with the values calculated in lines 3-5; 7 end 8 output P ;

Figure 4 :
Figure 4: Expansion diagram for potential participation.

Table 9 in
Appendix B).

Table 1 :
Participation profiles created by benchMark for the top seven students.F 1 -F 4 are the factor averages arranged according to weight priority.The columns Max, Min and Avg, are the maximum, the minimum and the average participation marks, respectively.The column Rank is the student ranking according to average participation marks.All marks are indicated in percentages.

Table 2 :
Results of the top seven students, obtained by the non-linear model.
that shows Student 1 who was categorised into class 4.

Table 3 :
Improvement targets for Student 1.

Table 4 :
Preference rating for pairwise comparison of academic marks.

Table 7 :
Improvement plan for Student 22 with an improvement of δ = 5% on the current factor averages.

Table 8
presents the data set containing the factor averages of 26 students in four factors, where the factors (denoted F 1 to F 4 respectively) are a semester test, class tests, theoretical 14 assignments and attendance.This set was used in the paper to evaluate the relevant ranking methods.

Table 8 :
Data set utilised in the mathematical models in this study.B.Participation profiles of the 26 participants as calculated by benchMarkParticipation profiles of the 26 participants as calculated by benchMark.

Table 9 :
Participation profiles created by benchMark in Section 3.1, for the 26 students.F 1 -F 4 are the factor averages arranged according to weight priority.The columns Max, Min and Avg, are the maximum, the minimum and the average participation marks, respectively.The column Rank is the student ranking according to average participation marks.All marks are indicated in percentages.The maximum, minimum and average p-marks, student ranking, and corresponding weights for calculating the minimum and maximum p-marks for 26 students using the non-linear model in Section 3.1.p-marks(%)Weightsfor minimum p-mark Weights for maximum p-mark

Table 10 :
Resulting maximum, minimum, and average p-marks, ranking and weights calculated by means of the non-linear model in Section 3.1.The student ranking of the 26 students, as determined by the AHP discussed in Section 3.3.

Table 11 :
Ranking of the 26 students produced by the AHP in Section 3.3.