Open Access

Adaptive Modulation with Smoothed Flow Utility

EURASIP Journal on Wireless Communications and Networking20102010:815213

DOI: 10.1155/2010/815213

Received: 6 May 2010

Accepted: 14 September 2010

Published: 21 September 2010

Abstract

We consider the problem of choosing the data flow rate on a wireless link with randomly varying channel gain, to optimally trade off average transmit power and the average utility of the smoothed data flow rate. The smoothing allows us to model the demands of an application that can tolerate variations in flow over a certain time interval; we will see that this smoothing leads to a substantially different optimal data flow rate policy than without smoothing. We pose the problem as a convex stochastic control problem. For the case of a single flow, the optimal data flow rate policy can be numerically computed using stochastic dynamic programming. For the case of multiple flows on a single link, we propose an approximate dynamic programming approach to obtain suboptimal data flow rate policies. We illustrate, through numerical examples, that these approximate policies can perform very well.

1. Introduction

We consider the flow rate assignment problem on a wireless link with randomly varying channel gain, to optimally trade off average transmit power and the average utility of the smoothed flow data rate. We pose the multiperiod problem as an infinite-horizon stochastic control problem with linear dynamics and convex objective. For the case of a single flow, the optimal policy is easily found using stochastic dynamic programming (DP) and gridding. For the case of multiple flows, DP becomes intractable, and we propose instead an approximate dynamic programming approach using suboptimal policies developed in the single-flow case. Simulations show that these suboptimal policies perform very well.

In the wireless communications literature, varying a link's transmit rate (and power) depending on channel conditions is called adaptive modulation (AM); see, for example, [15]. One drawback of AM is that it is a physical layer optimization technique with no knowledge of upper layer optimization protocols. Maximizing a total utility function is also very common in various communications and networking problem formulations, where it is referred to as network utility maximization (NUM); see, for example, [610]. In the NUM framework, performance of an upper layer protocol (e.g., TCP) is determined by utility of flow attributes, for example, utility of link flow rate.

Our setup involves both adaptive modulation and utility maximization but is nonstandard in several respects. We consider the utility of the smoothed flows, and we consider multiple flows over the same wireless link [11].

2. Problem Setup

2.1. Average Smoothed Flow Utility

A wireless communication link supports https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq1_HTML.gif data flows in a channel that varies with time, which we model using discrete-time intervals https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq2_HTML.gif . We let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq3_HTML.gif be the data flow rate vector on the link, where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq4_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq5_HTML.gif , is the https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq6_HTML.gif th flow's data rate at time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq7_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq8_HTML.gif denotes the set of nonnegative numbers. We let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq9_HTML.gif denote the total flow rate over all flows, where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq10_HTML.gif is the vector with all entries one. The flows, and the total flow rate, will depend on the random channel gain (through the flow policy, described below) and so are random variables.

We will work with a smoothed version of the flow rates, which is meant to capture the tolerance of the applications using the data flows to time variations in data rate. This was introduced in [12] using delivery contracts, in which the utility is a function of the total flow over a given time interval; here, we use instead a very simple first-order linear smoothing. At each time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq11_HTML.gif , the smoothed data flow rate vector https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq12_HTML.gif is given by
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ1_HTML.gif
(1)
where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq13_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq14_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq15_HTML.gif , is the smoothing parameter for the https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq16_HTML.gif th flow, and we take https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq17_HTML.gif . Thus, we have
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ2_HTML.gif
(2)

where at time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq18_HTML.gif , each smoothed flow rate https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq19_HTML.gif is the exponentially weighted average of previous flow rates.

The smoothing parameter https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq20_HTML.gif determines the level of smoothing on flow https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq21_HTML.gif . Small smoothing parameter values ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq22_HTML.gif close to zero) correspond to light smoothing; large values ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq23_HTML.gif close to one) correspond to heavy smoothing. (Note that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq24_HTML.gif means that flow https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq25_HTML.gif is not smoothed; we have https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq26_HTML.gif .) The level of smoothing can be related to the time scale over which the smoothing occurs. We define https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq27_HTML.gif to be the smoothing time associated with flow https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq28_HTML.gif . Roughly speaking, the smoothing time is the time interval over which the effect of a flow on the smoothed flow decays by a factor https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq29_HTML.gif . Light smoothing corresponds to short smoothing times, while heavy smoothing corresponds to longer smoothing times.

We associate with each smoothed flow rate https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq30_HTML.gif a strictly concave nondecreasing differentiable utility function https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq31_HTML.gif , where the utility of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq32_HTML.gif is https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq33_HTML.gif . The average utility derived over all flows, over all time, is
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ3_HTML.gif
(3)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq34_HTML.gif . Here, the expectation is over the smoothed flows https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq35_HTML.gif , and we are assuming that the expectations and limit above exist.

While most of our results will hold for more general utilities, we will focus on the family of power utility functions, defined for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq36_HTML.gif as
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ4_HTML.gif
(4)

parameterized by https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq37_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq38_HTML.gif . The parameter https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq39_HTML.gif sets the curvature (or risk aversion), while https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq40_HTML.gif sets the overall weight of the utility. (For small values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq41_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq42_HTML.gif approaches a log utility.)

Before proceeding, we make some general comments on our use of smoothed flows. The smoothing can be considered as a type of time averaging; then we apply a concave utility function; finally, we average this utility. The time averaging and utility function operations do not commute, except in the case when the utility is linear (or affine). Jensen's inequality tells us that average smoothed utility is greater than or equal to the average utility applied directly to the flow rates, that is,
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ5_HTML.gif
(5)

So the time smoothing step does affect our average utility; we will see later that it has a dramatic effect on the optimal flow policy.

2.2. Average Power

We model the wireless channel with time-varying positive gain parameters https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq43_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq44_HTML.gif , which we assume are independent identically distributed (IID), with known distribution. At each time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq45_HTML.gif , the gain parameter affects the power https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq46_HTML.gif required to support the total data flow rate https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq47_HTML.gif . The power https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq48_HTML.gif is given by
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ6_HTML.gif
(6)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq49_HTML.gif is increasing and strictly convex in https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq50_HTML.gif for each value of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq51_HTML.gif ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq52_HTML.gif is the set of positive numbers).

While our results will hold for the more general case, we will focus on the more specific power function described here. We suppose that the signal-to-interference-and-noise ratio (SINR) of the channel is given by https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq53_HTML.gif . (Here https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq54_HTML.gif includes the effect of time-varying channel gain, noise, and interference.) The channel capacity is then https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq55_HTML.gif , where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq56_HTML.gif is a constant; this must equal at least the total flow rate https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq57_HTML.gif , so we obtain
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ7_HTML.gif
(7)
The total average power is
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ8_HTML.gif
(8)

where, again, we are assuming that the expectations and limit exist.

2.3. Flow Rate Control Problem

The overall objective is to maximize a weighted difference between average utility and average power,
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ9_HTML.gif
(9)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq58_HTML.gif is used to trade off average utility and power.

We require that the flow policy is causal; that is, when https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq59_HTML.gif is chosen, we know the previous and current values of the flows, smoothed flows, and channel gains. Standard arguments in stochastic control (see, e.g., [1317]) can be used to conclude that, without loss of generality, we can assume that the flow control policy has the form
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ10_HTML.gif
(10)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq60_HTML.gif . In other words, the policy depends only on the current smoothed flows and the current channel gain value.

The flow rate control problem is to choose the flow rate policy https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq61_HTML.gif to maximize the overall objective in (9). This is a standard convex stochastic control problem, with linear dynamics.

2.4. Our Results

We let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq62_HTML.gif be the optimal overall objective value and let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq63_HTML.gif be an optimal policy. We will show that in the general (multiple-flow) case, the optimal policy includes a "no-transmit" zone, that is, a region in the https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq64_HTML.gif space in which the optimal flow rate is zero. Not surprisingly, the optimal flow policy can be roughly described as waiting until the channel gain is large, or until the smoothed flow has fallen to a low level, at which point we transmit (i.e., choose nonzero https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq65_HTML.gif ). Roughly speaking, the higher the level of smoothing, the longer we can afford to wait for a large channel gain before transmitting. The average power required to support a given utility level decreases, sometimes dramatically, as the level of smoothing increases.

We show that the optimal policy for the case of a single flow is readily computed numerically, working from Bellman's characterization of the optimal policy, and is not particularly sensitive to the details of the utility functions, smoothing levels, or power functions.

For the case of multiple flows, we cannot easily compute (or even represent) the optimal policy. For this case we propose an approximate policy, based on approximate dynamic programming [18, 19]. By computing an upper bound on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq66_HTML.gif , by allowing the flow control policy to use future values of channel gain (i.e., relaxing the causality requirement [20]), we show in numerical experiments that such policies are nearly optimal.

3. Optimal Policy Characterization

3.1. No Smoothing

We first consider the special case https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq67_HTML.gif , in which there is no smoothing. Then we have https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq68_HTML.gif , so the average smoothed utility is then the same as the average utility, that is,
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ11_HTML.gif
(11)
In this case the optimal policy is trivial, since the stochastic control problem reduces to a simple optimization problem at each time step. At time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq69_HTML.gif , we simply choose https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq70_HTML.gif to maximize https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq71_HTML.gif . Thus, we have
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ12_HTML.gif
(12)

which does not depend on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq72_HTML.gif . A simple and effective approach is to presolve this problem for a suitably large set of values of the channel gain https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq73_HTML.gif and store the resulting tables of individual flow rates https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq74_HTML.gif versus https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq75_HTML.gif ; online we can interpolate between points in the table to find the (nearly) optimal policy. Another option is to fit a simple function to the optimal flow rate data and use this function as our (nearly) optimal policy.

For future reference, we note that the problem can also be solved using a waterfilling method (see, e.g., [21,Section https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq76_HTML.gif ]). Dropping the time index https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq77_HTML.gif and using https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq78_HTML.gif to denote the flow index, we must solve the problem
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ13_HTML.gif
(13)
with variables https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq79_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq80_HTML.gif . Introducing a Lagrange multiplier https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq81_HTML.gif for the equality constraint (which we can show must be nonnegative, using monotonicity of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq82_HTML.gif with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq83_HTML.gif ), we are to maximize
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ14_HTML.gif
(14)
over https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq84_HTML.gif . This problem is separable in https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq85_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq86_HTML.gif , so we can maximize over https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq87_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq88_HTML.gif separately. We find that
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ15_HTML.gif
(15)

(Each of these can be expressed in terms of conjugate functions; (see, e.g., [21,Section https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq89_HTML.gif ].) We then adjust https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq90_HTML.gif (say, using bisection) so that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq91_HTML.gif . An alternative is to carry out bisection on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq92_HTML.gif , defining https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq93_HTML.gif in terms of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq94_HTML.gif as above, until https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq95_HTML.gif , where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq96_HTML.gif refers to the derivative with respect to https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq97_HTML.gif .

For our particular power law utility functions (4), we can give an explicit formula for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq98_HTML.gif in terms of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq99_HTML.gif :
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ16_HTML.gif
(16)
For our particular power function (7), we use bisection to find the value of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq100_HTML.gif that yields
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ17_HTML.gif
(17)

where the flow values come from the equation above. (The left-hand side is decreasing in https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq101_HTML.gif , while the right-hand side is increasing.)

3.2. General Case

We now consider the more general case, with smoothing. We can characterize the optimal flow rate policy https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq102_HTML.gif using stochastic dynamic programming [2225] and a form of Bellman's equation [26]. The optimal flow rate policy has the form
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ18_HTML.gif
(18)
where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq103_HTML.gif is the Bellman (relative) value function. The value function (and optimal value) is characterized via the fixed point equation
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ19_HTML.gif
(19)
where, for any function https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq104_HTML.gif , the Bellman operator https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq105_HTML.gif is given by
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ20_HTML.gif
(20)

where the expectation is over https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq106_HTML.gif . The fixed point equation and Bellman operator are invariant under adding a constant; that is, we have https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq107_HTML.gif , for any constant (function) https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq108_HTML.gif , and, similarly, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq109_HTML.gif satisfies the fixed point equation if and only if https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq110_HTML.gif does. So without loss of generality we can assume that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq111_HTML.gif .

The value function can be found (in principle) by value iteration [14, 26]. We take https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq112_HTML.gif and repeat the following iteration, for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq113_HTML.gif .
  1. (1)

    https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq114_HTML.gif (apply Bellman operator).

     
  2. (2)

    https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq115_HTML.gif (estimate optimal value).

     
  3. (3)

    https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq116_HTML.gif (normalize).

     

For technical conditions under which the value function exists and can be obtained via value iteration, see, for example, [2729]. We will simply assume here that the value function exists, and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq117_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq118_HTML.gif converge to https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq119_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq120_HTML.gif , respectively.

The iterations above preserve several attributes of the iterates, which we can then conclude holds for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq121_HTML.gif . First of all, concavity of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq122_HTML.gif is preserved; that is, if https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq123_HTML.gif is concave, so is https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq124_HTML.gif . It is clear that normalization does not affect concavity, since we simply add a constant to the function. The Bellman operator https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq125_HTML.gif preserves concavity since partial maximization of a function concave in two sets of variables results in a concave function (see, i.e.,[21, Section https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq126_HTML.gif ]) and expectation over a family of concave functions yields a concave function; finally, addition (of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq127_HTML.gif ) preserves concavity. So we can conclude that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq128_HTML.gif is concave.

Another attribute that is preserved in value iteration is monotonicity; if https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq129_HTML.gif is monotone increasing (in each component of its argument), then so is https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq130_HTML.gif . We conclude that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq131_HTML.gif is monotone increasing.

3.3. No-Transmit Region

From the form of the optimal policy, we see that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq132_HTML.gif if and only if https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq133_HTML.gif is optimal for the (convex) problem
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ21_HTML.gif
(21)
with variable https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq134_HTML.gif . This is the case if and only if
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ22_HTML.gif
(22)
(see, e.g., [21,page 142]). We can rewrite this as
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ23_HTML.gif
(23)
Using the specific power function (7) associated with the log capacity formula, we obtain
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ24_HTML.gif
(24)

as the necessary and sufficient condition under which https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq135_HTML.gif . Since https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq136_HTML.gif is decreasing (by concavity of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq137_HTML.gif ), we can interpret (24) roughly as follows: do not transmit if the channel is bad ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq138_HTML.gif small) or if the smoothed flows are large ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq139_HTML.gif large).

4. Single-Flow Case

4.1. Optimal Policy

In the case of a single flow (i.e., https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq140_HTML.gif ) we can easily carry out value iteration numerically, by discretizing the argument https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq141_HTML.gif and values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq142_HTML.gif and computing the expectation and maximization numerically. For the single-flow case, then we can compute the optimal policy and optimal performance (up to small numerical integration errors).

4.2. Power Law Suboptimal Policy

We replace the optimal value function (in the above optimal flow policy expression) with a simple analytic approximation of the value function to get the approximate policy
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ25_HTML.gif
(25)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq143_HTML.gif is an approximation of the value function.

Since https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq144_HTML.gif is increasing, concave, and satisfies https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq145_HTML.gif , it is reasonable to fit it with a power law function as well, say https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq146_HTML.gif , with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq147_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq148_HTML.gif . For example, we can find the minimax (Chebyshev fit) by varying https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq149_HTML.gif ; for each https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq150_HTML.gif we choose https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq151_HTML.gif to minimize
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ26_HTML.gif
(26)

where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq152_HTML.gif are the discretized values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq153_HTML.gif , with associated value function values https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq154_HTML.gif . We do this by bisection on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq155_HTML.gif .

Experiments show that these power law approximate functions are, in general, reasonable approximations for the value function. For our power law utilities, these approximations yield very good matches to the true value function. For other concave utilities, the approximation is not as accurate, but experiments show that the associated approximate policies still yield nearly optimal performance.

We can derive an explicit expression for the approximate policy (25) for our power function:
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ27_HTML.gif
(27)
where
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ28_HTML.gif
(28)

and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq156_HTML.gif is the Lambert function; that is, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq157_HTML.gif is the solution of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq158_HTML.gif [30].

Note that this suboptimal policy is not needed in the single-flow case since we can obtain the optimal policy numerically. However, we found that the difference between our power law policy and the optimal policy (see the example of value functions below) is small enough that in practice they are virtually the same. This approximate policy is needed in the case of multiple flows.

4.3. Numerical Example

In this section we give simple numerical examples to illustrate the effect of smoothing on the resulting flow rate policy in the single-flow case. We consider two examples, with different levels of smoothing. The first flow is lightly smoothed ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq159_HTML.gif ; https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq160_HTML.gif ), while the second flow is heavily smoothed ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq161_HTML.gif ; https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq162_HTML.gif ). We use utility function https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq163_HTML.gif , that is, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq164_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq165_HTML.gif in our utility (4). The channel gains https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq166_HTML.gif are IID exponential variables with mean https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq167_HTML.gif . We use the power function (7), with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq168_HTML.gif .

We first consider the case https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq169_HTML.gif . The value functions are shown in Figure 1, together with the power law approximations, which are https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq170_HTML.gif (light smoothing) and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq171_HTML.gif (heavy smoothing). Figure 2 shows the optimal policies for the lightly smoothed flow ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq172_HTML.gif ), and the heavily smoothed flow ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq173_HTML.gif ). We can see that the optimal policies are quite different. As expected, the lightly smoothed flow transmits more often, that is, has a smaller no-transmit region.
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Fig1_HTML.jpg
Figure 1

(a) Comparing https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq174_HTML.gif (blue) with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq175_HTML.gif (red, dashed) for the lightly smoothed flow. (b) Comparing https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq176_HTML.gif (blue) with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq177_HTML.gif (red, dashed) for the heavily smoothed flow.

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Fig2_HTML.jpg
Figure 2

(a) Optimal policy https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq178_HTML.gif for smoothing time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq179_HTML.gif ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq180_HTML.gif ). (b) Optimal policy for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq181_HTML.gif ( https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq182_HTML.gif

Average Power versus Average Utility.

Figure 3 further illustrates the difference between the two flow rate policies. Using values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq183_HTML.gif , we computed (via simulation) the average power-average utility tradeoff curve for each flow. As expected, we can see that the heavily smoothed flow achieves more average utility, for a given average power, than the lightly smoothed flow. (The heavily smoothed flow requires less average power to achieve a target average utility.)
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Fig3_HTML.jpg
Figure 3

Average utility versus average power: heavily smoothed flow (top, dashed), and lightly smoothed flow (bottom).

Comparing Average Power.

We compare the average power required by each flow to generate a given average utility. Given a target average utility, we can estimate the average power required roughly from Figure 3, or more precisely via simulation as follows: choose a target average utility, and then run each controller, adjusting https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq184_HTML.gif separately, until we reach the target utility. In our example, we chose https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq185_HTML.gif and found https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq186_HTML.gif for the lightly smoothed flow, and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq187_HTML.gif for the heavily smoothed flow. Figure 4 shows the associated power trajectories for each flow, along with the corresponding flow and smoothed flow trajectories. The dashed (horizontal) line indicates the average power, average flow, and averaged smoothed flow for each trajectory. Clearly the lightly smoothed flow requires more power than the heavily smoothed flow, by around https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq188_HTML.gif : the heavily smoothed flow requires https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq189_HTML.gif , compared to https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq190_HTML.gif for the lightly smoothed flow.
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Fig4_HTML.jpg
Figure 4

Sample power, flow, and smoothed flow trajectories; lightly smoothed flow (a, c, e), heavily smoothed flow (b, d, f).

Utility Curvature.

Table 1 shows results from similar experiments using different values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq191_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq192_HTML.gif . We see that for each https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq193_HTML.gif value, as expected, the heavily smoothed flow requires less power. Note also that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq194_HTML.gif decreases as https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq195_HTML.gif increases. This is not surprising as lower curvature (higher https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq196_HTML.gif ) corresponds to lower risk aversion.
Table 1

Average power required for target https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq197_HTML.gif , lightly smoothed flow https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq198_HTML.gif , heavily smoothed flow https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq199_HTML.gif .

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq200_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq201_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq202_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq203_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq204_HTML.gif

0.032

0.013

59 https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq205_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq206_HTML.gif

0.59

0.39

34 https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq207_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq208_HTML.gif

0.93

0.70

25 https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq209_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq210_HTML.gif

1.15

0.97

16 https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq211_HTML.gif

https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq212_HTML.gif

1.22

1.08

11 https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq213_HTML.gif

5. A Suboptimal Policy for the Multiple-Flow Case

5.1. Approximate Dynamic Programming (ADP) Policy

In this section we describe a suboptimal policy that can be used in the multiple-flow case. Our proposed policy has the same form as the optimal policy, with the true value function https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq214_HTML.gif replaced with an approximation or surrogate https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq215_HTML.gif :
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ29_HTML.gif
(29)

A policy obtained by replacing https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq216_HTML.gif with an approximation is called an approximate dynamic programming (ADP) policy [18, 19, 31]. (Note that by this definition (25) is an ADP policy for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq217_HTML.gif .)

We construct https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq218_HTML.gif in a simple way. Let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq219_HTML.gif denote the power law approximate function for the associated single-flow problem with only the https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq220_HTML.gif th flow. (This can be obtained numerically as described above.) We then take
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ30_HTML.gif
(30)

This approximate value function is separable, that is, a sum of functions of the individual flows, whereas the exact value function is (in general) not. The approximate policy, however, is not separable; the optimization problem solving to assign flow rates couples the different flow rates.

In the literature on approximate dynamic programming, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq221_HTML.gif would be considered basis functions [3234]; however, we fix the coefficients of the basis functions as one. (We have found that very little improvement in the policy is obtained by optimizing over the coefficients.)

Evaluating the approximate policy, that is, solving (29), reduces to solving the resource allocation problem
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ31_HTML.gif
(31)

with optimization variables https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq222_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq223_HTML.gif . This is a convex optimization problem; its special structure allows it to be solved extremely efficiently, via waterfilling.

5.2. Solution via Waterfilling

We can solve (31) using the waterfilling method (described earlier). At each time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq224_HTML.gif , we are to maximize
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ32_HTML.gif
(32)
over variables https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq225_HTML.gif , whereas before, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq226_HTML.gif is a Lagrange multiplier associated with the equality constraint. For our particular power law approximate functions we can express https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq227_HTML.gif in terms of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq228_HTML.gif :
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ33_HTML.gif
(33)
We then use bisection on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq229_HTML.gif to find the value of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq230_HTML.gif for which
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ34_HTML.gif
(34)

Since our surrogate value function is only approximate, there is no reason to solve this to great accuracy; experiments show that around 5–10 bisection iterations are more than enough.

Each iteration of the waterfilling algorithm has a cost that is https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq231_HTML.gif which means that we can solve (31) very fast. An interior point method that exploits the structure would also yield a very efficient method; see, for example, [35].

5.3. Upper-Bound Policies

In this section we describe two heuristic data flow rate policies: a steady-state flow policy and a prescient flow policy. We show that both policies result in upper bounds on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq232_HTML.gif (the optimal objective value). These upper bounds give us a way to measure the performance of our suboptimal flow policy https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq233_HTML.gif : if we obtain a https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq234_HTML.gif from https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq235_HTML.gif that is close to an upper bound, then we know that our suboptimal flow policy is nearly optimal.

5.3.1. Steady-State Policy

The steady-state policy is given by
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ35_HTML.gif
(35)
where https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq236_HTML.gif is channel gain at time https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq237_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq238_HTML.gif is the steady-state flow rate vector (independent of time) obtained by solving the optimization problem
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ36_HTML.gif
(36)

with optimization variable https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq239_HTML.gif , and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq240_HTML.gif being known. Let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq241_HTML.gif be our steady-state upper bound on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq242_HTML.gif obtained using the policy (35) to solve (9). Note that in the above optimization problem, we ignore time (and hence, smoothing) and variations in channel gains, and so, for each https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq243_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq244_HTML.gif is the optimal (steady-state) flow vector. (This is sometimes called the certainty equivalent problem associated with the stochastic programming problem [36, 37].)

By Jensen's inequality (and convexity of the max) it is easy to see that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq245_HTML.gif is an upper bound on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq246_HTML.gif . Note that once https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq247_HTML.gif is determined, we can evaluate (35) using the waterfilling algorithm described earlier.

5.3.2. Prescient Policy

To obtain a prescient upper bound on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq248_HTML.gif , we relax the causality requirement imposed earlier on the flow policy in (10) and assume complete knowledge of the channel gains for all https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq249_HTML.gif . (For more on prescient bounds, see, e.g., [20].) For each realization of channel gains, the flow rate control problem reduces to the optimization problem
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ37_HTML.gif
(37)

where the optimization variables are the flow rates https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq250_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq251_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq252_HTML.gif and smoothed flow rates https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq253_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq254_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq255_HTML.gif . (The problem data are https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq256_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq257_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq258_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq259_HTML.gif .) The optimal value of (37) is a random variable parameterized by https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq260_HTML.gif . Let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq261_HTML.gif denote our prescient upper bound on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq262_HTML.gif . We obtain https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq263_HTML.gif by using Monte Carlo simulation: we take https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq264_HTML.gif large and solve (37) for independent realizations of the channel gains. The mean is our prescient upper bound.

5.4. Numerical Example

In this section we compare the performance of our ADP policy to the above prescient policy using a numerical example.

We construct a simple two-flow problem using the previous problem instance from Section 4.3 with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq265_HTML.gif , where, now, both flows share the single link, that is, https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq266_HTML.gif . Our approximate value function is
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Equ38_HTML.gif
(38)

(Note that this is easily extended to a problem with more than two flows.)

Let https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq267_HTML.gif denote the objective obtained using our ADP policy. Each https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq268_HTML.gif obtains an ADP controller, a point https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq269_HTML.gif in the https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq270_HTML.gif plane. Using the same https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq271_HTML.gif , we can compute the corresponding prescient bound giving the point https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq272_HTML.gif . (Every feasible controller must lie on or below the line, with slope https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq273_HTML.gif , that passes through https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq274_HTML.gif .)

We carried out Monte Carlo simulation (100 realizations, each with https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq275_HTML.gif time steps) for several values of https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq276_HTML.gif , computing https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq277_HTML.gif as described in Section 5.2 and our prescient upper bound as described above.

Figure 5 shows our ADP controllers and the associated upper bounds. We can see that the ADP controllers are clearly feasible and perform very well depending on https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq278_HTML.gif . For example, for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq279_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq280_HTML.gif https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq281_HTML.gif and https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq282_HTML.gif https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq283_HTML.gif , so we know that https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq284_HTML.gif . So in this example, for https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq285_HTML.gif , https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq286_HTML.gif is not more than https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_IEq287_HTML.gif suboptimal.
https://static-content.springer.com/image/art%3A10.1155%2F2010%2F815213/MediaObjects/13638_2010_Article_2032_Fig5_HTML.jpg
Figure 5

ADP controllers (red), and prescient upper bound (blue).

6. Conclusion

In this paper we present a variation on a multiperiod stochastic network utility maximization problem as a constrained convex stochastic control problem. We show that judging flow utilities dynamically, that is, with a utility function and a smoothing time scale, is a good way to account for network applications with heterogenous rate demands.

For the case of a single flow, our numerically computed value functions obtain flow policies that optimally trad off average utility and average power. We show that simple power law functions are reasonable approximations of the optimal value functions and that these simple functions obtain near optimal performance.

For the case of multiple flows on a single link (where the value function is not practically computable using dynamic programming), we approximate the value function with a combination of the simple one-dimensional power law functions. Simulations, and comparison with upper bounds on the optimal value, show that the resulting ADP policy can obtain very good performance.

Declarations

Acknowledgments

This material is based upon work supported by AFOSR Grant FA9550-09-0130 and by Army contract W911NF-07-1-0029. The authors thank Yang Wang and Dan O'Neill for helpful discussions.

Authors’ Affiliations

(1)
Information Systems Laboratory, Department of Electrical Engineering, Stanford University

References

  1. Hayes J: Adaptive feedback communications. IEEE Transactions on Communication Technology 1968, 16(1):29-34. 10.1109/TCOM.1968.1089811View Article
  2. Cavers J: Variable-rate transmission for Rayleigh fading channels. IEEE Transactions on Communications 1972, 20(1):15-22. 10.1109/TCOM.1972.1091106MathSciNetView Article
  3. Hentinen VO: Error performance for adaptive transmission on fading channels. IEEE Transactions on Communications 1974, 22(9):1331-1337. 10.1109/TCOM.1974.1092383View Article
  4. Webb WT, Steele R: Variable rate QAM for mobile radio. IEEE Transactions on Communications 1995, 43(7):2223-2230. 10.1109/26.392965View Article
  5. Soon-Ghee C, Goldsmith AJ: Variable-rate variable-power MQAM for fading channels. IEEE Transactions on Communications 1997, 45(10):1218-1230. 10.1109/26.634685View Article
  6. Kelly FP, Maulloo AK, Tan D: Rate control for communication networks: shadow prices, proportional fairness and stability. Journal of the Operational Research Society 1997, 49(3):237-252.View ArticleMATH
  7. Low SH, Lapsley DE: Optimization flow control—I: basic algorithm and convergence. IEEE/ACM Transactions on Networking 1999, 7(6):861-874. 10.1109/90.811451View Article
  8. Chiang M, Low SH, Calderbank AR, Doyle JC: Layering as optimization decomposition: a mathematical theory of network architectures. Proceedings of the IEEE 2007, 95(1):255-312.View Article
  9. Neely MJ, Modiano E, Li C-P: Fairness and optimal stochastic control for heterogeneous networks. IEEE/ACM Transactions on Networking 2008, 16(2):396-409.View Article
  10. Chen J, Xu W, He S, Sun Y, Thulasiraman P, Shen X: Utility-based asynchronous flow control algorithm for wireless sensor networks. IEEE Journal on Selected Areas in Communications 2010, 28(7):1116-1126.View Article
  11. O'Neill D, Akuiyibo E, Boyd S, Goldsmith AJ: Optimizing adaptive modulation in wireless networks via multi-period network utility maximization. Proceedings of the IEEE International Conference on Communications, 2010
  12. Trichakis N, Zymnis A, Boyd S: Dynamic network utility maximization with delivery contracts. Proceedings of the IFAC World Congress, 2008 2907-2912.
  13. Bertsekas D: Dynamic Programming and Optimal Control: Volume 1. Athena Scientific; 2005.MATH
  14. Bertsekas D: Dynamic Programming and Optimal Control: Volume 2. Athena Scientific; 2007.
  15. Åström K: Introduction to Stochastic Control Theory. Dover, New York, NY, USA; 1970.MATH
  16. Whittle P: Optimization Over Time: Dynamic Programming and Stochastic Control. John Wiley & Sons, New York, NY, USA; 1982.MATH
  17. Bertsekas D, Shreve S: Stochastic Optimal Control: The Discrete-Time Case. Athena Scientific; 1996.MATH
  18. Bertsekas D, Tsitsiklis J: Neuro-Dynamic Programming. Athena Scientific; 1996.MATH
  19. Powell W: Approximate Dynamic Programming: Solving the Curses of Dimensionality. John Wiley & Sons, New York, NY, USA; 2007.View ArticleMATH
  20. Brown DB, Smith JE, Sun P: Information relaxations and duality in stochastic dynamic programs. Operations Research 2010, 58(4):785-801. 10.1287/opre.1090.0796MathSciNetView ArticleMATH
  21. Boyd S, Vandenberghe L: Convex Optimization. Cambridge University Press, Cambridge, UK; 2004.View ArticleMATH
  22. Puterman M: Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, New York, NY, USA; 1994.View ArticleMATH
  23. Ross S: Introduction to Stochastic Dynamic Programming: Probability and Mathematical. Academic Press; 1983.
  24. Denardo E: Dynamic Programming: Models and Applications. Prentice-Hall, New York, NY, USA; 1982.MATH
  25. Wang Y, Boyd S: Performance bounds for linear stochastic control. Systems and Control Letters 2009, 58(3):178-182. 10.1016/j.sysconle.2008.10.004MathSciNetView ArticleMATH
  26. Bellman R: Dynamic Programming. Courier Dover, New York, NY, USA; 1957.MATH
  27. Derman C: Finite State Markovian Decision Processes. Academic Press; 1970.MATH
  28. Blackwell D: Discrete dynamic programming. The Annals of Mathematical Statistics 1962, 33: 719-726. 10.1214/aoms/1177704593MathSciNetView ArticleMATH
  29. Arapostathis A, Borkar V, Fernández-Gaucherand E, Ghosh MK, Marcus SI: Discrete-time controlled Markov processes with average cost criterion: a survey. SIAM Journal on Control and Optimization 1993, 31(2):282-344. 10.1137/0331018MathSciNetView ArticleMATH
  30. Corless RM, Gonnet GH, Hare DEG, Jeffrey DJ, Knuth DE: On the Lambert W function. Advances in Computational Mathematics 1996, 5(4):329-359.MathSciNetView ArticleMATH
  31. Manne A: Linear programming and sequential decisions. Management Science 1960, 6(3):259-267. 10.1287/mnsc.6.3.259MathSciNetView ArticleMATH
  32. Schweitzer PJ, Seidmann A: Generalized polynomial approximations in Markovian decision processes. Journal of Mathematical Analysis and Applications 1985, 110(2):568-582. 10.1016/0022-247X(85)90317-8MathSciNetView ArticleMATH
  33. Trick MA, Zin SE: Spline approximations to value functions: linear programming approach. Macroeconomic Dynamics 1997, 1(1):255-277.View ArticleMATH
  34. De Farias DP, Van Roy B: The linear programming approach to approximate dynamic programming. Operations Research 2003, 51(6):850-865. 10.1287/opre.51.6.850.24925MathSciNetView ArticleMATH
  35. Madan R, Boyd SP, Lall S: Fast algorithms for resource allocation in wireless cellular networks. IEEE/ACM Transactions on Networking 2010, 18(3):973-984.View Article
  36. Birge J, Louveaux F: Introduction to Stochastic Programming. Springer, New York, NY, USA; 1997.MATH
  37. Prekopa A: Stochastic Programming. Kluwer Academic Publishers, New York, NY, USA; 1995.View ArticleMATH

Copyright

© E. Akuiyibo and S. Boyd. 2010

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.