The Squared Slacks Transformation in Nonlinear Programming

: In this short paper, we recall the use of squared slacks used to transform inequality constraints into equalities and several reasons why their introduction may be harmful in many algorithmic frameworks routinely used in nonlinear programming. Numerical examples performed with the sequential quadratic programming method illustrate those reasons. Our results are reproducible with state-of-the-art implementations of the methods concerned and mostly serve a pedagogical purpose, which we believe will be useful not only to practitioners and students, but also to researchers

where f and c are 2 C functions from n R into R. To simplify, we assume that there is a single inequality constraint, but our arguments still hold in the general case. A simple technique to convert the inequality constraint into an equality constraint is to add a new variable yR  that appears squared in the reformulation     2 , minimize subject to 0 , hence the name, squared slack. The two problems are equivalent in the sense that * x solves (1) if and only if ** ( , ( )) x c x  solves (2). This transformation can be used to derive optimality conditions of an inequality constrained problem, (see for example (Bertsekas, 1999, §3.3.2) and an interesting discussion on this subject in (Nash, 1998)). The use of squared slacks has also been advocated for algorithmic purposes (see for example (Tapia, 1980)). In (Gill et al., 1981, §7.4), the authors give theoretical reasons why converting (1) into (2) may be a bad idea, but the literature does not appear to state the numerical drawbacks of the approach clearly.
The purpose of this note is to demonstrate why many algorithmic frameworks of nonlinear programming will fail on problems of the form (2), arising from a squared slack transformation or in their own right. Among other reasons, we wish to communicate this fact to potential practitioners, who are not necessarily aware of the arcane corners of optimization methods. We wish to show that such simple, innocuous-looking transformations can be disastrous. This note is mainly motivated by the fact that some users, and some researchers, know that the usage of squared slacks is discouraged in practice, but the reason why remains unclear.
The frameworks that are concerned are the sequential quadratic programming (SQP) method, the augmented Lagrangian method, and conjugate-gradient-based methods, among others. It is important to realize that (2) satisfies the same regularity conditions as (1) and does not defeat the convergence theory of those methods, but illustrates cases where things can, and do, go wrong. We show that the main difficulty comes from the linearization of the first order conditions and a bad choice of the starting point.
The rest of the paper is organized as follows. Section 2 presents a parameterized two-dimensional problem that illustrates the major shortcomings of SQP-type methods. Section 3 presents numerical experiments with a standard implementation of the SQP method to illustrate our point. However, the results are reproducible with state-of-the-art implementations. We elected to not choose a particular existing solver for two main reasons. The first is to ensure that we are running the plain SQP method, with no bells and whistles often found in state-ofthe-art software. The second is that the central point of this paper concerns algorithms, not particular implementations. We leave to the interested reader the possibility to make his or her own opinion by reproducing the numerical tests performed in this paper with his or her favorite implementation of the SQP method. Finally, Section 4 describes other well-known families of algorithms subject to similar shortcomings. Conclusions are discussed in Section 5.
We believe that the examples given in this note can serve as pedagogical tools to expose the somewhat surprising behavior of some minimization methods for the solution of nonlinear problems. Hopefully, they will also be useful to design better methods that do not suffer the same shortcomings.

Shortcomings of some current methods
In this section, we present a few example problems on which several of the most widely used nonlinear optimization algorithms are bound to fail. The reason for this failure is that the problems, without being complex or artificial, cause a certain behaviour of the algorithm which corresponds precisely to a case that is usually 'ruled out' of the theory by means of assumptions. By way of example, we cite Boggs and Tolle (1995), where the authors mention various types of solution that an SQP algorithm can produce and that do not correspond to a local constrained minimizer of (1). Those cases are when the iterates 1. are unbounded (in violation of Assumption C1 in (Boggs and Tolle, 1995)), 2. have a limit point that is a critical point of (1) but not a local minimizer, or 3. have a limit point that is a critical point of the measure of infeasibility 2 ( , ) | ( ) | . x y c x y  We will use the following simple parametrized example to expose those problematic cases. It is based on the addition of squared slacks to convert an inequality constraint into an equality.

Example 1 Consider the problem
where aR  is a parameter. Various values of a will illustrate each of the three shortcomings listed above.  The point indicated by a red square is a local maximizer. Starting from 0 ( ,0), x an SQP method will converge to the local maximizer.
Figures 1, 2 and 3 show the various situations graphically. Note that for all , aR  Example 1 satisfies the linear independence constraint qualification at all feasible points. One of its features is that the inequality constraint is not active at the minimum of each inequality constrained problem. an SQP method will converge towards the red square.
For the purpose of reference, note that the first-order optimality conditions of (1) are given by The main reason for the failure on Example 1 is the following. Assume that Problem (1) is regular in the sense that * ( ) 0 cx  at any critical point * .
x Problem (2) is then regular as well and the Karush-Kuhn-Tucker conditions are necessary for first-order optimality. The Lagrangian for problem (2) is where  is the Lagrange multiplier associated to the equality constraint, and at a first-order critical point, we Note that in (5), the complementarity condition of (4) is only apparent as the equality = 0, y  so that the sign restrictions on the multiplier and on the constraints in the second part of (4) do not appear. As it turns out, the sign of the multiplier can be recovered from the second-order optimality conditions (see (Bertsekas, 1999, §3.3.2)). This pseudo-complementarity condition and the loss of sign restriction creates a difficulty in linearization-based methods. Indeed, any method that linearizes conditions (5)

Numerical experiments
In order to eliminate side effects sometimes found in sophisticated implementations, we implemented a basic SQP method for equality-constrained problems, as given in (Nocedal and Wright, 1999, Algorithm 18.3), for example. The method is globalized with a backtracking linesearch on the 1  is a penalty parameter, and the Lagrange multipliers are updated using the least-squares estimates. In all cases, the starting point is chosen as (0,0) with the initial multiplier set to 1 2. In the following tables, k is   Table 1 gives the detail of the iterations in the case where = 0. a As expected, we see that =0 k y for all k. Because the least-squares estimates happen to yield the exact multipliers in the present case, the gradient of the Lagrangian always vanishes. In order to satisfy the first-order optimality conditions, there thus only remains attaining feasibility, which is achieved by having k x converge to .  Note also that || k  converges to .  This behaviour is that of the first shortcoming of Section 2. Some backtracking linesearch iterations were only necessary at the first iteration, the unit step was always accepted at the other iterations. Note that the fact that k x 'escapes' to  can be observed in practice by tightening the stopping tolerance. Table 2 gives the detail of the iterations in the case where = 1.
a  The results are representative of any value < 0.
a Here, k x converges to a value which is a local maximizer. This illustrates the second shortcoming of Section 2. Again, no backtracking was necessary on this problem, except at the first iteration.  Table 3 gives the detail of the iterations in the case where = 2, a but again, the results are representative of any value (0, ). ae  This situation is that given in the third shortcoming of Section 2. Backtracking was used in this case and the algorithm stopped claiming that the steplength was too small.
As a side note, we remark that when =, a e there is a unique feasible point for (2) that has = 0. y The SQP algorithm converges towards that point, which is in fact a saddle point. For >, ae the feasible set is made of two disconnected curves. Each one intersects the axis = 0. y One of those intersection points is a local maximizer while the other one is a local minimizer, but neither of them has = 0.
x Depending on the value of the starting point, the SQP algorithm converges to one or the other. We decided to not report those results here since they do not add new elements to the present analysis.
Finally, the above numerical results hold not only for an initial 0 =0 y but of course, also for 0 y sufficiently close to 0. This is an effect of finite precision however, and not a shortcoming of the SQP method.

Other algorithmic frameworks
As we showed in (6), any traditional SQP-type method will necessarily generate iterates of the form ( ,0) k x when started from 0 ( ,0) x with a nonzero Lagrange multiplier. In this section, we show that similar conclusions hold for a variety of families of algorithms for nonlinear programming.
Augmented-Lagrangian-based methods fail for a reason similar to that given in Section 2. Note that the squared slacks transformation can again be used to derive the proper form of the augmented Lagrangian for inequality-constrained problems (Bertsekas, 1996, §3.1-3.2;Bertsekas, 1999, §4.2). Minimization of the augmented Lagrangian with squared slacks with respect to the slacks only yields the usual form for inequalityconstrained problems. As we illustrate below, the direct application of the augmented Lagrangian algorithm to the formulation involving the slacks suffers the same pitfalls as the SQP method.
Consider the augmented Lagrangian for (2) iterate has the form ( ,0). k  This method will therefore also be unable to depart from the hyperplane = 0. y

Discussion
When the constraint of (1) is inactive, the optimal multiplier is = 0.  However, in (2) we will frequently observe ||    when starting with 0 =0 y to reach dual feasibility. Looking for instance at the results of Table 1, the final iterate ( 19.4,0)  is feasible to within the prescribed tolerance. To compensate in dual feasibility, we need to have a large multiplier. In this sense, the addition of squared slacks has created a critical point at infinity.
For lack of a better initial value, implementations often set their variables to zero prior to starting the optimization, provided the objective and constraints are well defined at the origin. Modeling languages also often set the variables to zero unless the user specifies otherwise. The difficulties exposed in this paper thus certainly occur in practice and illustrate another reason why starting points should not be chosen to be zero.
Unfortunately, the problem is difficult to avoid. Of course, it seems that we should not add squared slacks to convert inequality constraints into equality constraints. However problems having the form (2) where the variable y does not appear in the objective may arise in their own right. If at all feasible, it is recommended that equality constraints involving squared slack variables be converted to inequality constraints.
It is worth noting that any problem of the form