A PARTITIONING SCHEME FOR SOLVING THE 0-1 KNAPSACK PROBLEM

The application of valid inequalities to provide relaxations which can produce tight bounds, is now common practice in Combinatorial Optimisation. This paper attempts to complement current theoretical investigations in this regard. We experimentally search for "valid" equalities which have the potential of strengthening the problem's formulation. Recently, Martello and Toth [13] included cardinality constraints to derive tight upper bounds for the 0-1 Knapsack Problem. Encouraged by their results, we partition the search space by using equality cardinality constraints. Instead of solving the original problem, an equivalent problem, which consists of one or more 0-1 Knapsack Problem with an exact cardinality bound, is solved. By explicitly including a bound on the cardinality, one is able to reduce the size of each subproblem and compute tight upper bounds. Good feasible solutions found along the way are employed to reduce the computational effort by reducing the number of trees searched and the size of the subsequent search trees. We give a brief description of two Lagrangian-based Branch-and-Bound algorithms proposed in Kruger [9] for solving the exact cardinality bounded subproblems and report on results of numerical experiments with a sequential implementation. Implications for and strategies towards parallel implementation are also given.

1 max , 0,1 for 1, 2, , , n n IP j j j j j j j z c x a x b x j n (1) where all data are integers, and x j = 1 if item j is included in the knapsack and 0 otherwise.
Without loss in generality we may assume that the a j > 0, c j > 0, a j b, for j = 1, 2, ..., n and where z is the largest integer z.If the weights and profits are sorted according to nonincreasing profit density, i.e. (4) We will call the above LP-solution with x t set to 0, the break solution and t the break item.
The bound z is called the Dantzig-bound.If the items are already sorted the computation of LP z is done in O(n) time.

Historical Notes on Exact Algorithms
The 0-1 Knapsack Problem has received considerable attention, not only because it has several important applications in itself, but also as a substructure in many discrete optimisation problems.The techniques for exactly solving the 0-1 Knapsack Problem can be classified into three groups: Dynamic Programming based algorithms, Branch-and-Bound based algorithms, and hybrid approaches.
Early papers which specialized Dynamic Programming for the 0-1 Knapsack Problem include Bellman [2], Dantzig [4] and Bellman and Dreyfus [3].The idea is to first fill a small knapsack optimally and then, using this information, fill larger a knapsack optimally.This process is repeated until the original problem is solved completely.Some computational improvements were proposed by Toth in [17].
In his award winning PhD-thesis Pisinger [16,Chapter 4] devised a dynamic programming recursion, which, although the worst-case time complexity is still O(bn) as for the Bellman recursion, solves most relatively large problem instances without enumerating too many variables.The algorithm starts from the break solution and at each stage either inserts or removes an item.Strong upper bounds are used to limit the number states in the recursion.
The enumeration process terminates due to some bounding tests, in which case it is possible to prove that the current incumbent solution is optimal.
Recently, Martello, Pisinger and Toth [10] incorporated cardinality constraints into a very efficient Dynamic Programming algorithm.Although the worst-case time complexity of their algorithm is still O(bn), they solved most instances quite quickly due to the tight bounds produced by the cardinality constraints.
One of the first Branch-and-Bound algorithms for the 0-1 Knapsack Problem was proposed by Kolesar [8].It was a straightforward specialization of the Land and Doig algorithm for general integer programming problems.The algorithm used the best bound selection rule and branching was done on the fractional variable.The large computer memory requirements of this algorithm led to the development of other Branch-and-Bound algorithms by Horowitz and Sahni [7], Nauss [14], Fayard and Plateau [6] and Martello and Toth [11], to name but a few.
These algorithms use a depth-first selection rule and branching is done on the free variable with the smallest index (assuming that the items are sorted according to non-increasing profit densities).The depth-first enumeration limits the space consumption of these algorithms.
Many of these algorithms include a reduction or preprocessing phase in which some of the extremal variables are fixed to 0 or 1.
The performance of a Branch-and-Bound algorithm very much depends on the tightness and application of upper bounds.Several bounds were proposed in the literature of which Martello and Toth [11,12,13] were some of the main contributors.Bounds are generally based on Lagrangian relaxation, partial enumeration, the adding of valid inequalities, and relaxations of these valid inequalities.
Martello and Toth [11] compared some of these algorithms.Their results showed that "easy" problem instances can be solved efficiently.
Hybrid approaches to the solving of the 0-1 Knapsack Problem are mainly devised by combining Branch-and-Bound with Dynamic Programming.One approach is to do a Branchand-Bound search up to a certain time limit and thereafter automatically switch over to Dynamic Programming (see Dudzi ski and Walukiewicz [5]).An approach recently proposed by Martello and Toth [13] , is to do a Branch-and-Bound search down to a certain level in the search tree or while the residual capacity is above a certain threshold value.Dynamic Programming is then used to solve the subproblem to optimality.

Aspects of Search Models
In order to avoid enumerating all feasible solutions a good search strategy and a bounding procedure are crucial.Bounding procedures are usually based on the exact solution of a relaxed problem which is obtained from the original problem by relaxing or dropping some of the constraints.The application of valid inequalities to provide relaxations which can produce tight bounds, is now common practice.With this approach one tries to find a polyhedral description of the convex hull of the feasible points.This paper attempts to complement current theoretical investigations in this regard.We experimentally search for "valid" equalities which have the potential of strengthening the problem's formulation.The inclusion of these equalities in the problem formulation partitions the search space.
If one partitions the search space before it is searched and concentrate the search only on those parts that have the potential of delivering feasible solutions better than the current best, one has more than one search tree.The best solution found amongst these trees (forest search) is then the optimal solution of the original problem.This approach lends itself naturally to a high-level parallel implementation in which each processing node is given a tree to search.It is a well-known fact that the branching in a search tree can be reduced significantly if tight lower and upper bounds are known.If we partition the search space into N different parts, the formulation of the original problem must be specialized for each different part by adding extra constraints.In general, this will lead to tighter upper bounds.The maximum of the upper bounds taken over all parts of the partition will produce a global upper bound which, in general, will be tighter than upper bounds derived for the original problem.
The availability of a tight global upper bound early in the solution process has the added advantage that the "quality" of any incumbent solution (the best feasible found so far) can be appraised much more reliably.This means that if one wants to curtail the search one has a better estimate of how "good" the incumbent solution really is.This is the main reason why we chose the Branch-and-Bound algorithm and not Dynamic Programming.One of the drawbacks of Dynamic Programming is that a solution is only available at the end of the process.
Recently, Martello and Toth [13] included cardinality constraints (i.e. on the number of items in the knapsack) to derive tight upper bounds in their Branch-and-Bound algorithm.
Encouraged by their results, we partition the search space by using equality cardinality constraints.So, instead of solving the original problem an equivalent problem which, in general, consists of one or more separate problems, is solved.Each separate problem is a 0-1 Knapsack Problem with an exact bound (an equality constraint) on the cardinality.These subproblems will be called Exact k-item 0-1 Knapsack Problems.Pandit and Ravi Kumar [15] used a similar approach for the solution of strongly-correlated knapsack problem instances.
By explicitly including a bound on the cardinality, one is able to reduce the size of each subproblem and compute tight upper bounds.Furthermore, if a good feasible solution is found along the way it may reduce the computational effort by reducing the number of trees searched and the size of the subsequent search trees.
In the following section we will illustrate some of the properties of the cardinality constrained 0-1 Knapsack Problem.In Section 3 we define an equivalent problem and give a high-level description of our partitioning scheme.The inclusion of the extra equality constraint can be handled by using Lagrangian Relaxation or by using an LP-solver to provide upper bounds in a Branch-and-Bound type algorithm.In order to compete favourably with current exact algorithms for the 0-1 Knapsack Problem, one needs an algorithm that is efficient.Section 4 gives a brief description of two Lagrangian-based algorithms, proposed in Kruger [9], for solving the Exact k-item 0-1 Knapsack Problems.Some techniques for reducing the size of the subproblems are also given.Computational experiments are presented in Section 5, followed by a conclusion.

The Maximum k-item Knapsack Problem
Assume that the items are sorted according to non-decreasing weights, i.e. 1 , j j a a for j = 1, 2, ..., n -1.Suppose that we fill our knapsack according to non-decreasing weights, i.e. , then it is clear that in any integer solution to (1), the number of items in our knapsack is bounded from above by k u and hence we may introduce the canonical inequality 1 .
Note that k u can be found without sorting by using a partitioning scheme similar to the one used by Balas and Zemel [1] to find the break item.

Definition 1 (Maximum k-item 0-1 Knapsack Problem) Given k = k u , we define the
Maximum k-item 0-1 Knapsack Problem (MCP) as follows Solving the continuous case of the above problem using an LP-solver can be computationally expensive.Martello and Toth [13] solved the continuous relaxation of ( 7) without an LPsolver by using Lagragian relaxation to add the cardinality constraint to the objective function.
More recently, Martello et al. [10] solved the continuous case of ( 7) by surrogate relaxing the cardinality constraint with the knapsack constraint.Since we have only two constraints, we can limit the explicit number of surrogate multipliers to one multiplier.Martello et al. [10] prove some monotonicity properties and derive a special binary search, similar to that of Martello and Toth [13], which considers only a limited number of integer surrogate multipliers.They report that since the continuous bounds are generally tight, the transformed problem tends to be solved much easier.According to them, their approach has the additional advantage that if the optimal solution to the LP-relaxed problem is correct, i.e. the cardinality constraint is satisfied, one also obtains a feasible solution to the original problem, thus solving the problem to optimality.

The Minimum k-item Knapsack Problem
Assume that the items are sorted according to non-increasing profits, i.e.
Suppose that z best is our current best lower bound for the 0-1 Knapsack Problem and that we fill our knapsack according to non-increasing profits, i.e. , then it is clear that in any integer solution to the 0-1 Knapsack Problem with solution value better than z best , the number of items in our knapsack is bounded from below by k l and hence we may introduce the canonical inequality 1 .
Note again that k l can be found without sorting by using a partitioning scheme similar to the one used by Balas and Zemel [1] to find the break item.Martello and Toth [13] and Martello et al. [10] solved the continuously relaxed version of problem (10) with algorithms similar to the ones used to solve the Maximum k-item 0-1 Knapsack Problem and the reader is referred to the relevant articles and technical reports for details.
In our partitioning scheme we solve the closely related cardinality constrained 0-1 Knapsack Problem, in which the constraint is an equality constraint, repeatedly.

The Exact k-item Knapsack Problem
Suppose that we know beforehand that the number of items in an optimal solution to the 0-1 Knapsack Problem is, k e , then we can solve the original problem by solving the following equivalent problem.

LEKP z k
Kruger [9] recently proposed two Lagragian-based Branch-and-Bound algorithms which solve problem (11) exactly (see Section 4 for a brief description).

A PARTITIONING SCHEME
It is now easy to see that one can solve the original 0-1 Knapsack Problem by solving EKP(k) for each k in the cardinality range, .This gives rise to the following equivalent formulation: We are now ready to state our algorithm.

An Exact Algorithm for the 0-1 Knapsack Problem
Given an algorithm, ECardKnap, which can solve any instance of the Exact k-item 0-1 Knapsack Problem exactly, the algorithm is as follows: Algorithm 1 CardKnap 1: Determine the LP-solution for the original (unsorted) problem by using the partitioning scheme of Balas and Zemel [1]; 2: Construct an incumbent solution with solution value, z best ; 3: Try to fix some variables at their upper or lower bounds, by using reduced costs; 4: Sort the reduced problem according to non-increasing profit densities; 5: Compute the break solution, x b , for the reduced problem; 6: Try and improve on z best by filling the remaining capacity in a greedy fashion; 7: Reduce the problem by using probing to fix some variables at 0 or 1; 8: Call CardRange to calculate the cardinality range, , for the reduced problem and where is convex.
The proof of the following Lemma is given in Kruger [9].This proposition can be used in the following way; one can solve LEKP(k) and note that if we set x i = 1 and x j = 0 in (iii) of the above proposition and adjust the slack accordingly, we get an integer feasible solution.Moreover, sometimes it is possible to construct an even better integer feasible solution by setting x r = 1 where * \{ } max : , 0 .
In our algorithm we use this technique repeatedly to construct our first incumbent solution.

EXPERIMENTAL RESULTS
In this section we present some results of an experimental testing of the ideas proposed in this paper.For a more detailed report see Kruger [9].

Data Generation
The data instances used in the literature to test and compare knapsack algorithms are usually classified as (i) Uncorrelated (uc): weights a j and profits c j are uniformly distributed in [1, R].
(ii) Subset-sum (ss): weights a j are uniformly distributed in [1, R], and profits are set to c j = a j .
(iii) Weakly correlated (wc): weights a j are uniformly distributed in [1, R], and profits c j in [a j -, a j + ] such that j 1.
(iv) Strongly correlated (sc): weights a j are uniformly distributed in [1, R], and profits are set to c j = a j + .
(v) Inverse strongly correlated (isc): profits c j are uniformly distributed in [1, R], and weights are set to a j = c j + .
(vi) Almost strongly correlated (asc): weights a j are uniformly distributed in [1, R], and profits c j in [a j + 0.99 , a j + 1.01 ].
Of these classes, the strongly correlated (sc), almost strongly correlated (asc) and inverse strongly correlated (isc) classes are regarded by many researchers in this field as the difficult classes.
In the results that follow we do not report on uncorrelated instances with nearly similar weights (ucsw) instances, because the integers involved for problems of high dimension could not be handled by our (current) computer code.
We used an algorithm (similar to that given in Pisinger [16, pages 105-106]) to generate test instances.The algorithm uses the standard lrand48 generator of the C library for generating pseudo-random profits and weights.A seed for the algorithm is given by the C library procedure srand48.

Experiments
The proposed algorithm, Algorithm 1, was coded in FORTRAN90 and tested on an IBM RISC6000 with 128Mb of memory.We did one set of tests by using the partition-based algorithm proposed in Kruger [9,Chapter 7] to solve the resulting Exact k-item 0-1 Knapsack Problems.
Each value in the tables and graphs, shown below, is the average taken over 10 problem instances (seed = 100, 200, ..., 1000).The values in the tables given between the parentheses is the standard deviation for these 10 problems.

Results
(a) The reduction by the first reduction step was very good for the "easy" problem instances but poor for the "difficult" instances.For the subset-sum instances this reduction should be zero, but for large problems it often happens that the constructed greedy solution is optimal.
(b) The reduction by the second reduction step was good for almost all problem instances.This is partly due to the fact that good solutions are often found during the reduction process itself.
(c) The reduction by the third reduction step was good for almost all problem instances.
For the subset-sum and strongly correlated instances this reduction should be zero, but for large problems it often happens that an optimal greedy solution can be constructed.
What often happens in this case is that as soon as the maximum (equality) cardinality constraint is enforced, an optimal solution is produced.Table 1 give an upper bound on the (average) number of trees that have to be searched (potentially), while Table 2 report on the number of trees actually searched.) sc 1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)isc 1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)1.00(0.00)asc 1.10(0.30)1.00(0.00)1.00(0.00)1.10(0.30)1.00(0.00)1.00(0.00)1.00(0.00)1.20(0.40) Note: A zero value in the table means that all the problems were solved prior to the cardinality calculations.(f) To demonstrate that the search strategies proposed in this paper show good promise we have also done some comparative experiments.We obtained the C-code (combo) of the algorithm proposed in Martello et al. [10] from Pisinger (http://www.diku.dk/~pisinger/codes.html) and compared it with the proposed algorithm.The experimental results showed (see for instance Fig. 2) that the proposed search strategy is competitive with the best algorithms currently known, especially for the so-called hard instances (strongly correlated, inverse strongly correlated, almost strongly correlated).This must also be evaluated against the background that combo is the product of more than two decades of research done by the architects of the code, while our code is of experimental nature.(b) The effectiveness of our approach was demonstrated by producing an experimental code which compares very favourably with some of best computer codes currently available.The competitiveness of our approach can mainly be attributed to the pruning power by the inclusion of the cardinality constraints.Another major factor is the efficiency of the algorithms used to solve the Exact k-item Knapsack Problems.This can be attributed to the fact that no searching (in the normal sense) is needed to complete partial solutions containing k -1 items in the branch-and-bound process.
Furthermore, since these completed solutions are integer, the bounds they produce are exact.
(c) Although the number of cardinalities in the cardinality range may be large, we have shown that good feasible solutions found along the way, reduce the computational effort by reducing the number of trees searched and the size of the search trees.
(d) Further investigations into the use of cardinality constraints to partition the search space for general 0-1 Integer Linear Programs with the aim of deriving (novel) highlevel parallel algorithms now seem feasible.

6.2
Directions for Future Research

Parallelization
The algorithm proposed in this paper lend itself naturally to high-level parallel implementation in which each processing node is given a tree to search.We have done some preliminary studies on an IBM SP2 machine with 7 nodes where we have used a master-slave model.To devise a competitive master-slave the following issues need to be addressed to keep the message passing overhead at an acceptable level: Is the master also going to search a tree?
How is the process at each node going to get subproblem data?
o Is the subproblem data to be written to a file and then read by each process?
o Is the data going to be packed into a message buffer and passed to the node by some message passing interface like PVM or MPI?
If a good solution is found by one process, is it to be broadcasted to the other processes?
If good solutions are to be broadcast to other processes, how often will this be done?

A Partitioning Approach to the Multi-dimensional Knapsack Problems
The LP-solution of the 0-1 Knapsack Problem can be written down analytically as soon as the items have been sorted according to non-decreasing profit densities.We think that the variables of the Multi-dimensional Knapsack Problem formulation can also be sorted according to some criteria used in LP column selection.A partitioning scheme like the proposed one can then be devised which may produce tight bounds.A successful partitioning scheme will certainly have parallelization potential.
An upper bound z , for IP z , can be produced by relaxing the integrality constraints on the variables x j and solving the Linear Programming relaxation problem and setting LP z z ,

Definition 2 (
Minimum k-item 0-1 Knapsack Problem) Given k = k l , we define the Minimum k-item 0-1 Knapsack Problem as follows

Definition 3 (
Exact k-item 0-1 Knapsack Problem) Given k = k e , we define the Exact k-item 0-1 Knapsack Problem (EKP(k)) as follows We denote the LP relaxation of EKP(k) by LEKP(k) and the corresponding objective value by ( ).
such that a i a j , then a i b red a j , {1,2,...,n}.

( d )Figure 1 :
Fig.1.It, therefore, seems that putting more effort into obtaining better feasible solutions is the only way to further improve the performance in these instances.

Figure 2 :
Figure 2: GoalPart vs combo, Strongly Correlated (sc) Knapsack Problem is equivalent to solving the following problem: , can be computed by Algorithm 2 which first tries to find the largest k 2 {1, 2, ..., The time complexity of the above algorithm is dominated by the calculation of z LEKP (k), but recall that we use a tailor-made LP solver to solve it.