1

LlNll‘uWWWWNWWWkMl

   

 

 

 

 

‘Iritiiiiiiiiiiw‘

dissertation entitled

Set Compound Decision Estimation
Under Entropy Loss In Exponential Families

presented by
Zhihui Liu

has been accepted towards fulfillment
of the requirements for

Ph.D. degreein Statistics

 

at; /§%4/

Major professor

Date June 17. 1997 %n 5:: M
Major Professor

MS U i: an Affirmative Action/Eq ual Opportunity Institution 0-12771

 

 

 

LIBRARY

Mlchlgen State
Unlverelty

 

 

 

PLACE ll RETURN BOX to remove thin checkout trom your record.
TO AVOID FINES return on or More due due.

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

|| |

MSU le An Affirmative ActionlEcpel Oppommlty Institution
W

 

 

 

M‘

SET COMPOUND DECISION ESTIMATION
UNDER ENTROPY LOSS IN EXPONENTIAL FAMILIES

By

Zhihui Liu

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Department of Statistics and Probability

1997

Copyright by
Zhihui Liu
1997

ABSTRACT

SET COMPOUND DECISION ESTIMATION
UNDER ENTROPY LOSS IN EXPONENTIAL FAMILIES

By

Zhihui Liu

Set compound decision estimation has been studied for half a century, starting
with finite state examples.

Set compound estimation under the entropy (Kullback-Leibler, same through the
thesis) loss for a k-dimensional standard exponential family with a compact parameter
space is discussed here. The entropy loss with the exponential family and related

properties are investigated in detail. Asymptotically optimal set compound estimators

with rates 0(n’i) under this loss are established for one dimensional discrete
exponential families including Poisson and negative binomial by being able to view the
Bayes estimators as a ratio of two power series and representing them in form of mixture
density and using Singh-Datta Lemma. Some remarks related to squared error loss and
noncompact state case are also given. Generalization to a higher dimensional discrete
exponential family is illustrated with a two dimensional example. Secondly going from
cumulant generating functions and using kernel density estimation, continuous
exponential families are studied under the same setting. Normal and Gamma distribution

families are examples.

To my husband, my son and my daughter.

ACKNOWLEDGMENTS

First the author would like to take this opportunity to thank Professors Dennis
Gilliland and James Harman for suggesting the problems, for valuable recommendations
and for their criticism through which the author has gained research abilities. The author
has benefited from their knowledge both in statistics and language.

The author likes to express her thanks to the department for the financial support
and consideration during her Ph. D. study here. She wants to thank Professor Salehi,
Professor Stapleton, Professor LePage and other committee members for their help.

At the end, the author thanks her parents for their encouragement and her husband

for his understanding and support in many ways.

TABLE OF CONTENTS

CHAPTER 0

INTRODUCTION ............................................................................................................... 1

CHAPTER 1

SET COMPOUND DECISION PROBLEM UNDER ENTROPY LOSS IN

EXPONENTIALFAMILIES............ .............................................................................. 4
1.1. The Component Decision Problem4
1.2. The Corresponding Set Compound Problem .................................................. .8
1.3. Bounds for the Regret Dn(Q,t) 10
1.4. Summary ............................................................................... 12

CHAPTER 2

SET COMPOUND DECISION ESTIMATION UNDER ENTROPY LOSS IN
DISCRETE EXPONENTIAL FAMILIESIS

2.1 . Introduction ..................................................................................................... 1 5
2.2. An Asymptotically Optimal Set Compound Estimator for
a One Dimensional Family .......................................................... 18
2.3. Poisson and Negative Binomial Distributions ............................................. .22
2.4. A Two Dimensional Example ....................................................... 24
CHAPTER 3
SET COMPOUND DECISION ESTIMATION UNDER ENTROPY LOSS IN
CONTINUOUS ONE-DIMENSIONAL EXPONENTIAL FAMILIES .................... 30
3.1. Kernel Density Estimation .............................................................................. 31
k
3.2. Case w(9) = Z:aq6q (k 22 and {aq} are constants)... ............................... .34
q=0

k
3.3. Case me) = Zatqe-q + bln(—0) (-00 < a 5.93 p < o , b s o and
q=l

sign{aq}=(—l)q for 1 quk) ......................................................... 37

BIBLIOGRAPHY .............................................................................................................. 47

vi

Chapter 0

INTRODUCTION

Compound decision theory was introduced by Robbins (1951) through an
example of decision between N(-1,l) and N(1,l). In his paper he proposed a compound
procedure that was optimal in the sense of asymptotic subminimaxity and showed the
necessity of this kind of procedure. The theory was greatly developed and in much
general finite state situation by Hannan (1956) and (1957). He used a randomization
technique to overcome the difficulty caused by discontinuity of a Bayes response with
respect to priors in his (1957) paper. Since then many works have been done in this area.
Gilliland (1968) further demonstrated the necessity of compound procedures by showing
that the supremum of regrets over both a parameter space and stages for any simple
procedure is positive. Vardeman (1982) applied Hannan (l957)’s randomization
technique to extended sequence compound problems for finite state case. Gilliland and
Harman (1986) considered set compound problems in a setting of restricted risk
components that avoided action space and loss function. In their paper a large class of
asymptotic solutions to the set compound decision problems for finite state case were

established. Singh (1974) considered sequence compound problems for exponential

families with compact parameter spaces for parameter functions in three cases: 9, e9 and

2

G" , and yielded some rates. Datta (1991a) generalized and strengthened Gilliland,
Hannan and Huang (1976)’s work from finite state case to compact state case and
established an asymptotically optimal Bayes compound estimator based on a hyperprior
for a continuous parameter function of an exponential family with a compact parameter
space under squared error loss. Mashayekhi (1991 and 1995) strengthened Datta
(1991a)’s result to the equivariant envelope. He also successfully extended Hannan and
Huang (1972)’s results on the stability of symmetrization of product measures to the
compact state case for exponential families. Zhu (1992) developed Datta’s work (1991) to
multidimensional case. In addition he considered a nonregular family - two dimensional
truncation family, and got some rate for this family. Majumdar (1993) developed Datta
(1991a) and (l99lb)’s work to Hilbert spaces.

The present work considers the problem of set compound estimation of the natural

parameter in an exponential family with entropy loss. The families considered here had

been the focus of sequence compound estimation of 9, e6 and 9" with squared error loss
[see Samuel (1965), Gilliland (1968), Swain (1965), Yu (1971) and Singh (1974)].
Estimation with entropy loss was also studied extensively. [see M. Ghosh and M. C.
Yang (1988), D. K. Dey, M. Ghosh and C. Srinivasan (1987)]. The work draws on the
earlier work, but the results are not immediate extensions of that work. Compound
estimation in the Hannan’s sense with entropy loss is first attempted apparently. Terms
not previously considered in the compound literature have been analyzed.

Chapter 1 conducts a general discussion about component and set compound

problems for exponential families with compact parameter spaces under entropy loss.

3

Chapter 2 obtains asymptotically optimal set compound estimators with rates O(n”4) for
some discrete exponential families with compact parameter spaces by representing the
Bayes estimators in terms of mixture density under entropy loss. Chapter 3 discusses set
compound estimation with continuous exponential families under entropy loss through

using the kernel density estimation considered in Singh (1974).

Chapter 1

SET COMPOUND DECISION PROBLEM UNDER ENTROPY LOSS
IN EXPONENTIAL FAMILIES

1.1 The Component Decision Problem

The component statistical decision problem considered has a standard exponential

family {Peze GO} , where P9 has density
(1.1) r5.<x>= e°'*“"‘°’

with respect to a measure u on R" and 9' x denotes the inner product of 9 and x in R".

(9 is a subset of the natural parameter space

(1.2) N = {9: jee'Xdp < 00}
and
(1.3) w(9) = ln( Icelxdp.) , GEN

is the cumulant generating function (see Brown (1986, page 1)). Of course, in this thesis,
we consider only the case where G), and therefore, N are nonempty.

Define
(1.4) N~ = {0 eN: E9|X| <oo}

where I - | is the Euclidean norm and

5
(1.5) n(9)=E9X,6c—:N‘.

For future reference, note that the interior No of the natural parameter space is a
subset of N~ and that 111 has derivatives of all orders on N0 with
41(9) = E,(X) = 11(9) , (13(9) = Vare(X),
the variance-covariance matrix of X ~ P9 (see Brown (1986, Theorem 2.2)).
Throughout this thesis we assume that P8 is identified by 9. Note that P9 is
equivalent to p. for all 9 e N, so that for every distribution G on O , the mixture PG is

equivalent to u.
The action space A is a nonempty subset of N and the loss function is the entropy

loss function:
_ 59 ~
(1.6) L(e,a) _ [an 3*)pedtt , e e N, a e N.

Note that L(9,a)=K( P9,Pa), the Kullback-Leibler information of Pal at P6 and that
L(6,a) 2 O and = 0 if and only if P0 = P, (see Brown (1986, page 174)). Because of our
identifiability assumption, L(G,a) =0 if and only if 9 = a. It follows from (1.1) and (1.6)
that
(1.7) L(9,a)=(0—a)’n(9)-w(9)+w(a),9 e N‘,aeN.

A non-randomized decision rule t is a mapping from the underlying measure
space X into A such that L(G, t) is a measurable function of X for each 9 e (9. The risk

oftatGis

R(9,t) = jL(e,t)dP, .

6
If G is a probability distribution on O and R(-,t) is measurable, then the Bayes risk of t at

G is R(G, t) = IR(-,t)dG. The infimum Bayes risk over all decision rules is denoted by

R(G). A decision rule To such that R(G,TG) = inf‘ R(G,t) is called a Bayes decision
rule versus G or simply Bayes versus G. The terms non-randomized decision rule and
estimator are synonymous in this thesis.

The following notations are used in this thesis. Let f " stand for the inverse of a

function f if it exists. Let fIO] be the image of f on a domain (9. “:=” is used as “denote”

or “is denoted”. PG(B): = IP9(B)dG(9) for prior probability G and measurable subsets B.

We denote the vector and matrix of the first and second derivatives by dot and double dot
notations. For example, \i/ and (I) . Let E be the joint expectation operator of x and 6 in
the rest of the chapter unless otherwise noted.

Bayes estimators under the entropy loss (1.6) are different from that under
squared error loss. The following propositions demonstrate the unique (a.e. 11) Bayes
estimator under certain conditions and entropy loss. We will use the following lemma in

the proof of Proposition 1.2.

Lemma 1.1 n is a 1-1 function on N”.

Proof Let 9,,92 e N~ and note that by (1.7),

(1.8) u0.,e.)+ 1492.9.) = (e. -92)'(n(9.)- «9.».

Thus, n(9,) = 7(6),) implies that the nonnegative quantities L(9l ,9, ), L(92,9,) are both

zero from which 9, = 62. O

7

Proposition 1.1 Suppose that (9 and A are measurable subsets of N” with G) g A, A is
compact and n is continuous on A. Let G be a distribution on O. Suppose that 1: is an

estimator satisfying
(1-9) n(t(X)) = 1301(9)! X) M u -
Then for any estimator t
R(G,t) 2 R(G,t)
with equality iff t = 'c a.e. p. Thus, 1: is the unique Bayes estimator.
Proof Using (1.7),
(1.10) R(G, t) - R(G, t)= EL(9, t) - EL(9,I‘) =
EKT - t111(9) + W“) - W(T)] = EL(1:, t) + E{(T - 0171(9) - TIMI}-
It follows by taking conditional expectation given x that the second term on the right
hand side (1.10) = O by the hypothesis (1.9). The first term on RHS(1.10) is 2 0 since L 2
0 and furthermore it equals to 0 if and only if t = t a.e. PG. As noted previously, PG is
equivalent to p, which completes the proof. 0
Proposition 1.2 Suppose that (9 and A are measurable subsets of N‘ with (9 g A, A is
compact and n is continuous on A. Let G be a prior on (9. Suppose that
(1.11) convex hull {n[(9]} c_: n [A] .
Then there is an estimator T = “CG satisfying (1.9) ( a.e. u ) and
(1-12) T(3(X) = 11"IE(TI(9)IX)] 3.6- 11
Proof In the beginning we show that RHS (1.12) is well defined. Since 1) is continuous
and A is compact, n[A] is compact. Thus by (1.11) the closure of the convex hull of

n[(9] is compact. It follows from Theorem 3.27 of Rudin (1973) that there is a version of

8
the conditional expectation E(n(9)|x) that takes values in n[A]. n is 1-1 by Lemma 1.1

so that with this version 1G:= n”'[E(n(0)lx)] takes values in A and is seen to satisfy
(1.9). 0

Remark 1.1 Suppose that n(-) is continuous on (9 which is compact. Then n[®] is

compact, Therefore the convex hull of G) is a compact subset of N” by Theorem 3.25(b)

'

of Rudin (1973).

Remark 1.2 Propositions 1.1 and 1.2 develop the unique a.e. p. Bayes response 1G with
respect to a prior distribution G on (9 when the loss function is entropy loss for estimation
of 9 in an exponential family. The assumptions are G) g A g N” where A is compact, n
is continuous on A and (1.11) is valid. These assumptions are satisfied if (9 g A g N0
where A is compact and (1.11) holds. This is the case since N0 g N” and u: is infinitely

differentiable on N0 with q) = 7].

Remark 1.3 The condition (1.11) holds in the one-dimensional case if (9 g A g N” , A
= [a, b] and n is continuous on A. When k = 1, n is seen to be strictly increasing by (1.8)
so that n[A]=[n(a),n(b)]. Hence n[®] ; n[A] which implies (1.11).
Remark 1.4 The condition (1.11) holds in the one-dimensional case if G) g A g N” , (9
= [a, b] and n is continuous on (9. Here convex hull{ n[®]} = n[(9]<;_ n[A].
Remark 1.5 If G) = A , (1.11) is equivalent to n[®] convex.
1.2 The Corresponding Set Compound Problem

The set compound decision problem with 11 components is defined for each n = l,

2,--- as follows: for given 11 it consists of n independent repetitions of the component

9

decision problem with compound loss equal to the average loss across the component
decisions. The observation X = (X,,--~,Xn) has distribution Fez: ”LP": with

Q=(Ol,---,9n)e®". A compound estimator g: (t.(_X_).'" ,t,(L)) has compound

risk

(1.13) R.(9.1)=n"Zf=,E,L(e.,t.(>_<»
and regret

(1.14) D.(9.1)=R.<Q.s)-R(G.)

where R is the Bayes envelope in the component decision problem and Gn denotes the
empirical distribution of 9|,-",9n. (Note that the notation hides the dependence of the
component decision rules ti(_)§) on n, i = 1, , 11). If a sequence of compound
estimators {t } satisfies

(1.15) 511139 D..(9,I)=0(1)(0(0t.,)),

we say it is asymptotically optimal (with rate an). ((1.15) is equivalent to
supg Dn(Q,t)+ =o(l)(O(0tn)) used by Datta (1988, page 3)). In this thesis our main
concern is the construction of compound estimators {t} satisfying (1.15) with
demonstrated rate.

Remark 1.6 The property (1.15) is uniform in parameter sequences {9 } so that results
for given 6) with (9_c_ A are a corollary to (1.15) for the case (9 = A. Since the
construction of rules {t} satisfying (1.15) is the focus of this thesis, we will henceforth

take (9=A.

10
1.3 Bounds for the Regret Dn(9,t)

Proposition 1.3 Suppose G) = A g No where G) is compact. Suppose that 11[(9] is

convex. Then there exists a constant B0 (not depending on Q and n) such that

(1.16) |Dn(Q,t)|s 130 n”' ZI~:._,|1i —an(x,)| for all g and n.

i=1
Proof By Remark 1.5, (1.11) is satisfied. By Remark 1.2 ton exists where G" is the

empirical distribution of 01,- - 39" . Note that

R(G,, )= [134(1),an )dG,,(e) = flag: L(9i,th(Xi )).

.
I81

Substituting this along with (1.13) into (1.14) and using (1.7) results in

1 n
(1.17) Dn(§9i)=;ZEglll(ei)'(Ton(Xi)"ti(_)_(.))+W(ti(X_))-W(To_(xi))l-

i=1
Thus, (1.16) follows from (1.17) and the mean value theorem with B0 = sup { 171(911'
”(9211391 , 92 e convex hull{(9}}. 0

The form of the Bayes response (1.12) and the bound RHS (1.16) motivate

compound rules of the form

(1.18) ti(X)=n”'[si(X)] ,i =1,2,...,n,

where si is an estimator for the conditional expectation

(1.19) 12.,(X.):= E.<n(e)IX.),

where En denotes expectation on (9, Xi) when 9 ~ Gn and given 6, Xi ~ P9. Note that
to. (X) is a component Bayes response for estimating 11(6) with squared error loss and

prior Gn .

11

Proposition 1.4 Suppose G): A _c_: N0 where (9 is compact. Suppose that n[(9] is

convex and n”' is a Lipschitz function on n[(9] with constant C0. Then 1 defined by

(1.18) and (1.19) satisfies

(1.20) |Dn(Q,t)|s Bocoxr' 213,13, — 12,. (x,)| , for all 9 and n.

Proof The inequality (1.20) follows from (1.16), (1.18), (1.12), (1.19) and the
assumption |n”'(a) — n”'(b)ls Cola — bl for all a, b e 11[A]. 0
Remark 1.7 It is easy to show that RHS(1.20) with C0 = 1 is a bound for the absolute
regret of the set compound estimator s in the estimation of n(9) with squared error loss.
In subsequent chapters we show that RHS(1.20) converges to zero uniformly in Q with
rates under various conditions. Hence, all of the results for the set compound estimator t
for Q under entropy loss transfer to corresponding results for the set compound estimator
_s_ of 11(6) under squared error loss. (For examples of estimators t and § sec (2.14) and
(2.15)).

Proposition 1.4 serves as the starting point for finding asymptotically optimal set
compound estimators. For the exponential families considered in Chapters 2 and 3, the

conditional expectations 1'0" (X) are expressed in terms of ratios that are estimated. For

bounding the error in ratio estimates, we will use Singh-Datta Lemma (see page 40 of
Datta (1988)).

Singh-Datta Lemma For real numbers a, b, A, B, and D, with b i O < D, O/O:=0,

12

a A _
lbl{lg--I-3-IAD} s1 b1 'IaB-bAI+D(IbI-IBI)+

(1.21)
Sla — A|+(|%|+D)| b — Bl.

1.4 Summary

In the previous section, as is typical of the analysis of compound risk, the excess
compound risk over the simple envelope is bounded by a Cesaro mean estimation error.

In Proposition 1.3 the estimation is of the Bayes estimator 70,,(X1) of 9 under entropy
loss and the Cesaro mean is RHS(1.16). In Proposition 1.4, the estimation is of the Bayes

estimator 1.0 (X) for 11(8) under squared error loss and the Cesaro mean is RHS(1.20).

In subsequent chapters, the estimation of 12;“ is accomplished in both discrete and

continuous cases under assumptions on w and with compact parameter sets.

In Theorem 2.1 of Chapter 2, for certain one-dimensional discrete exponential
families, set compound estimators {t } are established for which RHS (1.20) is O(n’% )

uniform in Q. The proof starts by expressing 11 as a power series in e9,
11(9) = qu(e9 )j. Then interchanging the order of summation and conditional
expectation in 13:3“(Xi) (see (1.19)), it is found that the terms can be consistently

estimated. Substituting the estimates into the form of 123"(Xi) (see (1.19)) results in a

compound estimator that is consistent in the Cesaro mean with rate. Poisson and negative
binomial families with compact parameter spaces interior to their natural parameter
spaces are covered. The proof can be extended to higher dimensional families, which is

illustrated by working out the details for a particular 2-dimensional family.

13

In Chapter 3, for certain one-dimensional continuous exponential families, set
compound rules {t} are established for which RHS (1.20) is O(n”’) uniform in 9

where y is smaller than 1/2 (y can be made arbitrarily close to 1/2 by choice of kernels in
the estimation process).

In Theorem 3.1, the proof starts by expressing w as a finite power series in 9,
w(9) = ZaqGq . Then interchanging the order of summation and the conditional
expectation of (11(9): n(9), one finds that the terms can be consistently estimated using

kernel estimates for densities and derivatives of densities. (These kernel estimates have

been studied by other authors and applied in sequence compound decision theory by Yu
(1971) and Singh (1974)). Substituting the estimates into the form of 1'6“ (X) results in a
compound estimator that is consistent in the Cesaro mean with rate. A family of normal
distributions with fixed variance is an example since \1/(9)=92 /2 (when 02:1). In
addition, cumulant generating functions of the form w(9) = W where j Z 2 are

investigated. It is shown that the integer part of j/2 is necessarily odd.

For Theorem 3.2, the proof starts for u: that can be expressed as a finite sum
w(9) = Z aq9”q + bln(—9). Then interchanging the order of summation and the
conditional expectation of 111(9) = 11(9) and using a representation of E[9”“|Xi] in a
form that can be estimated consistently, the same result as in the first case is essentially

derived. A family of Gamma distributions serves as an example. In addition, cumulant

generating functions of the form (11(9) = Z aq9”q + b ln(—9) are investigated and some

l4

sufficient conditions for such u; to be a cumulant generating function for some

exponential family are given.

Chapter 2

SET COMPOUND DECISION ESTIMATION UNDER ENTROPY
LOSS IN DISCRETE EXPONENTIAL FAMILIES

We discussed the set compound decision problem under entropy loss for
exponential families in Chapter 1. In this chapter, we specialize this set compound
decision problem to a one dimensional discrete exponential family. Under certain

conditions, a sequence of set compound decision rules 1 is shown to be asymptotically

optimal with rate n”% . Poisson and negative binomial families serve as examples. A two
dimensional example ends the chapter to illustrate how the ideas extend to

multidimensional cases.
2.1 Introduction

Consider the exponential family (1.1) with respect to a discrete measure u on the
nonnegative integers such that g(x):= u({x}) > 0, x = O,1,2,---. It follows that P9 has the
probability mass function

(2.1) pe(x) = g(x)e°"'*"‘°’, x = 0,1,2,---,e e e.

16

x=0

Clearly, 111(9) = ln[z e9“ g(x):l is strictly increasing in 9. Here the natural parameter space

is an interval (-oo, v}, open or closed on the right with —oo < v 3 +00. (From display (9) of

Rainville (1967) on page 111, v = -ln[lim sup Vg(k) ].) Consider

k—>+oo
(2.2) o=A=[v,,v,] c N0.
Remark 2.1 If (2.2) holds, then the hypotheses of Propositions 1.1 - 1.4 are satisfied so
that for any probability distribution G on G), the unique Bayes response is given by (1 .12),
that is , by
(2.3) rem: n”‘[E(n(9)IX)l, x = 0,1,2.---,
Moreover, in Proposition 1.4, the Lipschitz constant C0 can be taken to be 1/ r'1(vl ) , that
is,
(2.4) |n”'(a) — n”‘(b)| Sla — b|/1'1(v,) for all a, b 6A.
Proof Since G)=A=[ vl, v2] is a compact subset of N0 g N” and (p is differentiable of
all orders on NO, n = \1! is continuous on (9. Therefore, E(n(9)| x) exists and is

integrable. Also n[(9] = [11(vl), n(v2)] is a convex set so that (1.11) is satisfied. Also

note that @1140): 1/n(n”'(y)). Slnce n(9)=\p(9)=Var9(X)>0 on N0, n 18

d . .
strictly increasing. Thus, Enfiy) s 1/ n(n”'(n(vl)))= 1/ n(vl). (2.4) follows from the

mean value theorem. 0

We will use the bounds in (2.5) in proofs that follow.

17
Remark 2.2 If (2.2) holds, then

(2.5) c”1pvl(x) S p9(x) S cpvz(x), x =0, l, 2, ..., 9 e [vl,v2]

where c = e‘V‘V’W‘V". (This is an analog of the inequality on page 1894 of Gilliland
(1968).)
Proof (2.5) follows directly from (2.1) and (2.2). 0

Suppose that n has the power series representation
(2.6) 11(9) = quej", 9 66 = [v,,v,]
j=1

and that the series converges uniformly on (9. It follows from (2.1) and (2.6) that for any

probability distribution G on (9,

 

' ,_ _ )6 _w Po(x+j) g(x) _
(2.7) 10(x)... E(n(9)|x)—§j:qu(e IX)-;qj pe(X) g(“DJ-0,1,2,

where pG(x)= Ipe(x)dG(9). (2.7) expresses the conditional expectation in terms of
ratios. In view of the bound (1.20), we turn our attention to the estimation of
1:3 (Xi):= En(n(9)IXi), i=1, 2, ---,n through estimation of the probabilities in the

ratios, namely, the pan (x) , x = O, l, 2,---. For this consider

(2'8) I3(X)=n—IZ:[X1=X]sx=0,1, 2;"
j=1
and, fori = 1,2, ---,n,n22
(2.9) 6.6) =(n—1)"ZIX,- = x1,x=o,1, 2,-~;
jaei

i=1

where square brackets denote indicator functions.

18

Lemma 2.1 With c = e“’(v””“’”",
(2.10) Eglmx) — p0,, (x)l s 1cp., (x) I n1”, n 2 1

(2.11) E9|P1(X)‘Pon(x)l s 21013., (x)/(n-1>1” , n22.

Proof Since E9(p(x)) = pG“(x) , LHS (2.10) is no greater than the standard deviation

of p(x) , that is,

16

(2.12) LHS (2.10) s all: p,1(x)(1 — p9j(x))] .
(2.10) follows from weakening (2.12) and applying the second inequality of (2.5).

To prove (2.11) we triangulate in LHS (2.11) about E9(pi(x)) to get
0-”) LHS (2-11) 5 59113.0() - Eg(l3i(X))|+l Eg(§.(X)) - P6,, 001-
The first term in RHS (2.13) is bounded by [cpV2 (x) / (n - ”1% by applying (2.10) to the
set with 9i deleted. The second term in RHS (2.13) is equal to IV)? - wil/(n - l) where
wj:= p9j(x), j= 1, 2,---,n. Since the wj 2 0, j=1,2,---,n, IW—wjl/(n—1)s
mjax wj /(n — 1), so that the second term in RHS (2.13) is bounded by
max p9j(x) / (n — 1) for n 2 2. Bounding this by its square root and applying the second
inequality of (2.5) to the W]. completes the proof of (2.1 1). 0
2.2 An Asymptotically Optimal Set Compound Estimator for a One

Dimensional Family

19

With the results of Chapter 1 and the previous section, we are prepared to

demonstrate a compound estimator that satisfies (1.15) with rate O(n”%) uniform in 9.
Let (-)[a'b] denote retraction to the interval [a, b], that is (-) [ab] = (av-)Ab.

Consider the compound estimator {t} defined by

 

(2-14) t.QS) = n"(S.(X))
where
Xi+ gxi .
(2.15) sI(X)=(;:qu)iIfE(i +1.1)(FE.+)J'))["”""“V’"’l =1,2,...,n

where the q j, p, are defined by (2.6) and (2.9) and 0/0 :=0. SAX) is an estimator of
12;“ (X) defined by (2.7) with G = G".
Theorem 2.1 Let {Pez9 66)} be the discrete exponential family defined by (2.1) and

suppose (2.2) holds. Also suppose that (2.6) holds with uniform convergence on

®=[vl, v2] and that

2.16 0° .. 21pg(x) ’2
( > gglqjlkrp (mg—mp

Let {t} be defined by (2.14) and (2.15). Then

(2.17) supl D.(9,t)l = 0(n‘” ).
13
Proof By Remark 2.1 and Proposition 1.4,

3 (B0 / f](v, ))n"z Eglsi - 1.0,(X1)| , for all 9 and n.

(2.18) ID.(9,:)

 

The conditional expectation 1'0" (X) = a/ b , where

20

a3: ZQJPG,,(X1 +j)[g(xi)/g(xi + 1)] and b3: pG"(Xi)a takes values in ln(Vt), 7105)]-

i=1

Letting A := 2%.;eri + j)[g(xi )/ g(xi + j)] and 13:==fa,(xi ) we see that
j=1

. A a
(2.19) ISi —TGn(Xi)lSIE—BIAD

where D = 11(v2 ) — 11(vI ). Applying Singh-Datta Lemma (1.21) and weakening the result

 

wehave
(2.20) ISi-T.6“(Xi)|
1 °° ,. . . g(x.)
S . . X. — X _____.___
pGn(x,){§qu”p'( .+1) pe,( '+J)lg(Xi+j)

+ (2710/2) — T1(V1 ))l P1(Xi ) ’ Po, (x1 )1}-
We first consider the Cesaro mean expectation of the last term in RHS(2.20).

Applying (2.11) to each summand shows that for n22,

 

 

n-IZEQ lPii-(X) PG(X1)l< 20y ii“ WX)

2.21
( ) pe_(X.) S:/————n-1n_9‘pG(X)

1 “ °° . . .
Since “ZEGi = E0" , RHS(2.21) 5 2c” lef?2 (x)/ dn —1 . Since v2 is 1n the 1nterior of
x=0

i=1
the natural parameter space, pf: (x) is summable with respect to x as indicated in

Gilliland (1966, page 24). (This is easily seen to be the case because there is a number b

“11": 1’2

larger than v2 such that Zeb‘g(x) < 00. Thus, sup[eb "g(x)] < oo , so that e pé(x)=

x=0

21
e‘v2 b”"2[eb"g(x)]’% < {sup[e""g(x)]}%elvz”b”"2 which is the summable.) Therefore we

X

have RHS(2.21) =0(n"V2) uniform in 9.
We now consider the Cesaro mean expectation of the first term 1n RHS(2 20)

Applying (2 11) to each summand, using an argument similar to that used above and
using pv (x + j) = evl’pv2 (x)g(x + j)/ g(x) shows that for n 22,

_ lpii(X H)" PG..1(X +1) g(X
2.22 ' E _2 ”LHS 2.16 / -1
( ) n2 eZIq-I paw) g(x, i)j)< c ( )J—n

Us1ng (2 16) we see that RHS(2.22) is O(n”%) uniform in 9 completing the proof 0

Remark 2.3 (2 16) does not follow from (2.2). Consider the example with g(2k) — e
g(2k+1) =(2k+1)”3, k=0,1,2,--- Then, N=(-oo, -1). Take v2 = —% which is smaller

than -1, but

__g__(2k)l)] =62 ——(2k)(2k+1)§ezk = 00.
k=0

212342101 g(2k

Remark 2 4 If \i/() can be represented as a series of powers of exponential functlon e

i.e., 41(9) = Zq e39 with q J 2 0 , then the dominating measure p. indicated 1n (1 1) has to
J:

k—l
eq.“"‘°°’ . em = 1812 g(j)q...-.,1, by the

be discrete. In fact g(0) = 6‘4””), g(1)=
j=0

argument in Theorem 51 of Rainville (1967) (see page 129). But if q ' < 0 for some

2

9
-— 'nN9,1,
21 ( )

1nteger j0 >0 the conclusion is not true. For example, w(9)

22

ac . 0 _ j
01(9) = 9 = 2(4)J+l E—r—l—L, for -oo<9sln2, p. is not discrete, actually u is absolutely
i=1

continuous with respect to the Lebesgue measure.

2.3 Poisson and Negative Binomial Distributions

Poisson Case. Consider the Poisson family with the mean 11. in a closed interval

[1», , 1.2] g (0, 00). It is transformed to standard form by letting 9 = ln 1. resulting in

611-111(9)

(2.23) p9(x)= x' x=0, 1, 2,---; 9 e[ln1t,,ln1tz]

 

where \y(9) = e9. In this case 01(9) = 11(9) = e6 and the loss function (1.7) is
L(9,a) = (9 — a)ee - e6 + e“.

In the power series representation (2.6), qI = 1 and q j = 0 for j 2 2. Thus,

LHS(2.16) = X§Z[(x+l)pvz(x)]% which is easily shown to be finite. Thus, this
x=0

Poisson case is covered by Theorem 2.1.

Negative Binomial Case. Consider the family with probability mass functions

F(x +01)
F(x +1)r(a)’

 

(1-p)“p" xe0,1,2,---;O<p,$p$p2<1

where a>0 is fixed. It is transformed into standard form by letting 9 = ln p resulting in

 

F
ex-wte) (“'00 -O,1,2,---; lnp. 595111132-

(2.24) p9(x) = e F(X +1)F(a) , x—

where (41(9) = —0t ln(l — e9). In this case 01(9) = n(9) = one9 / (1 — e”) and the loss

function (1 .7) is

23

 

ore“
L(9,a)=(9—a)1_ee +aln(1—e9)—aln(1—e“).

In the power series representation (2.6) , q J. = 01 , j = 1,2, . -- . Thus

 

LHS(2.16)= a2 2632, p;2(x)=

x=0 j=l P2 x=0

iiZU—p )Zp :J RH“)

r(x +1)F(a)

I”(x+oc)
x°F(x+ 1)

which is finite since -> O as x —>oo. Hence this negative binomial case is

covered by Theorem 2.1.
An interesting observation is that the Bayes estimator with geometric distribution
(01=1) has the simple form

PG(X2x+1)

(2.25) TG(X)=1n( PG(X 2 x)

 

)

by Remark 1.4, Proposition 1.2 and the following fact:
ICNNHG = PG(X _>_ x +1).
It is not difficult to prove the above fact. First note that
pG(x) = je°*+'"“-°">do = Jede — [6’9“le

so that by telescoping,

Ie°(“*‘)dG =1— 2130(1) = PG(X 2. x +1).

i=0
One can estimate the ratio in th(x) using natural estimates of the tail probabilities in
constructing asymptotically optimal compound estimators in the geometric case.
Remark 2.5 (i) No one has done the set compound problem with discrete exponential

families and continuous parameter spaces. The achievement in this chapter is that instead

24

of the usual squared error loss, the entropy loss is used and with this loss and a different

proof, set compound asymptotic estimators with rate n”% are established for some

discrete exponential families.

(ii) For Poisson exponential family (2.23), squared error loss (x—e°)2,

G) = (—oo, 1n kl] (which is unbounded) and compound estimator t with

_ p (X +1)
t.<X)—1<X.+ 1) mm ———]A?»2,
we can show
(226) suplD (0 t)I=supl—ZE_ 10 (X) e“) —t‘< e°i )le=0(n’i),

i=1
where t. is the Bayes estimator E(e°|x) corresponding to squared error loss and prior

G". This is a stronger result than what follows from Remark 1.7 since 6) is unbounded.

(iii) For geometric exponential family, squared error loss (x -— ee )2, noncompact
= (—00, In p,] and compound estimator t with

13,,(X +1)

tiX()=-p'..——[ I3at(X)

]Apza

(2.26) can be shown. This is a stronger result than what follows from Remark 1.7 since (9

is unbounded.

2.4 A Two Dimensional Example

The following is a two dimensional example.
Let X be distributed as Poisson (1t) and conditional on X = x , let Y be distributed
as x+Z, where Z has the negative binomial distribution (x+1, p). Then (X, Y) has a 2-

dimensional exponential distribution

25

 

—1. x
1»
(2.27) ( :—)(1 p)"+'p H e X. , forny,
0<p,_<_p_<_p2<1, O<1tls1ts1tz<oo.
Let 9 =ln [_1p_:_)_1t] and 9 =ln p. Then the distribution above can be transformed into

the standard form:

91"92

pg 9 (x, y) = exp[9lx +92y— 16 9 +ln(1— e92 )(33 / xl, ny.
1" 2 _e 2

 

The natural parameter space N is the open set Rx(-oo,0). Let g(x,y)= (ii) / x! for ny

throughout the following.

 

Define T: (0,1)x R, —> R2 by T(p, 1.) = (ln (1 _ p)?» , ln p). Then T is continuous,
P

which yields the compactness of (9:: T{[pI , p2]x[1t,, 12]}.

In this example let the action space A=(9. Now

9+0 m
. el 2 +.
111(9) = Me) = 13,00: 2. =1—ee. = Zea. n,

 

and
. 9+1 00 -0+)‘0 )0
n2(9)=w2(9)=Ea(Y)=—_p =lee- 2+e 2].
i=1

___€_+l

n[®]= [(Q,,§2 ):1t1 SCI 5 12,—— ] —£l-pl <§2< _ ——], which is a polytope in R2. Thus

n[(9] is convex. By using Remark 1.5 we see that the hypotheses of Propositions 1.1 and

26

1.2 are satisfied. By Proposition 1.2 the unique Bayes estimator of 9 with the entropy loss
is

1:(i("V)3'-” (710, 1720) = 714137le W1],
where w := (x, y). Interchange the summation and conditional expectation, obtain

°° p6(X+l,y+i) g(x,y)
E 0 = ,
(M W) ,2; pe(x,y) g(x +l,y+j)

E<n2(e)lw)=iljpo(x+l’“j) go”) 309””) 89"”

, ,],fory2x.
p. pe(x.y) g(X+l.y+l) pe(X.y) g(X+l.y+l)

Let
fi(x,y) = “le,- = XX,- = Y] and 131(X,Y) = (n -1)"Z[Xj = erj = Y]
i=1 pal

for i = 1, - - - , n, where as before square brackets denote the indicator function.

More widely, let {vk =T(1.0k,p0k):k=1,-~,m}c N and (9 c convex
hull {vk}c N0. Similar to one dimensional case the following inequalities can be
verified:

\p‘—\p.

(228) 139103-190" )(x,y)ls He 2

 

NI—

Vk (x3 y) ’

m
29
k=l

1)).—111.

13,103. — pa, )0. ms 2n‘3e 2

 

m l
Zpivxx. y)
k=l

by using (17) on page 13 of Zhu (1992) and (a + b); S ai + bi for all a, b 20. Establish a
set compound estimator based on observations of W: = [(XI , Y1),--- ,(Xn ,Yn )] for i= 1 ,

---,n as follows:

27
._ _ —1 0° 13i(Xi+l,Yi+j) g(Xi,Yi)
ti<w_).-(tr.(_w>.tn(w))-n (<1; mm“) g(Xi+l,Yi+j)’
0° “1(Xi‘l‘laYi‘l'l) g(xlaYi) +Pi(X1,Yi+j) g(XisYi)

. P
[J A O A e
j=l pi(xi’Yi) g(Xi +1’Yi +1) pi(Xi’Yi) g(xiaYi + J)

 

 

])[n[@]]}

and _t_:=(t,,t,,---,t ).
Then {t} is an asymptotically optimal set compound estimator with (2.8) satisfied under

the condition that

1‘ \lVPOk
(2.29) VKOR < ————.
1+ ,lvpo,‘

The proof follows.

From (1.16)

an(9, 01s Ban-{Z Enum- ra,(X..Y. )I
(2.30) f‘
s Bon“ZE._.{I my!) — rianlxifli )|+| any) — rm, (X.,Y.)l}.
i=1

Let C be a constant in the following. Use (2.28), Mean Value Theorem to the

functions 11”] and n“ and Singh-Datta Lemma (see Section 1.3),

(2.31)

28

 

 

 

 

 

 

LHS(1.16)
n - d.) X Y')
sC " E ' X-,Y- *. X2+1,Y.+' g( " '
co - g(xi’Yi) A
- X LY _ I C 1 Xl’Yl — Xl’Yl
§p0.( 1+ 1+1) gi(X +19Yi +J)|+ lp( ) pGn( )l
g(xi9Yi) 0° - 2 g(xi’Yi)
+1 JP1(X1+1:Yi+J) J (Xi+1aYi+J) .'
; g(X +1, Y +j) g1: p9» g(x,+l,Y,+))'
Go A 2 gg(X i, Yi) - gg(xi9Yi)
' X'sY . — X'9Yi .

 

sen'iZleZpgivxxnmfl) g(x’” +Zva.(x.y+ +1)—~———-—— 8‘ —-—-——’—-——”.l

y2x j=l k=l g(x+1,y+j) j=1 k: 1 g(x y+j)

'_ _ l
(y+Jx1).<

Furthermoreb (2.27) and , __ ,
y (m)!

 

Zijipjivk(x+l,y+j) g(X, Y)

 

y2x j=l k=l g(x'l'laY‘l‘j)
][yliJ/
00 -m 3"” x+2p +'x—-le lkk__o__0__kx+l
= J [( )(1- -p k) 1"“ ’
E: El: E “I O (“1)! Wk +1):

HI)

/ .
°°. y / (1-p0k)>"0k) [ij (y+J—x—1)!%
21" W 2‘1”)?" 21 x‘ 0+1)! )

j=l x=0 p0k

 

 

 

j=l y=0

 

Z
k=l
5C2 Zip OkyZ(1+Y)p0k% [J(1’P0k)7~0k/P0k +lly
lt=l
C2

2' n’Z<I+y)1F—pn)ln NR1
j=l

k=l
which is finite by the fact that ,/(1—p0k)1tOk + p0k < 1 for k=1,-~, m from the

condition (2.29).

29

In the same way, the second sum in RHS(2.31) is finite. Combined with above,
the proof is finished.

At last take a look at the condition (2.29). If vpok = r'2 (r an integer), then
vko,‘ S (r —1)/(r + 1) .

Remark 2.6 If ®=T{[p,,p2]x[ll,kz]}, then we can take v,=T(7tl,p2),

v2 =(an-T—B-Z—ELLJnpl),v3 =T()tz,p,) and v4 =(ln(1—_Efl1,lnp2) and(~) c convex

P2 P1

hull{vl,v2,v3,v4}. Actually v2 =T(?Cl,p,) and v4 =T(7t'2,p2) with 7U, =%§3%11
P2 “P1

and X2 = Q—Lp—Jp—z k2 . Condition (2.31) becomes
P1 (1 "' P2)

 

Chapter 3

SET COMPOUND DECISION ESTIMATION
UNDER ENTROPY LOSS IN CONTINUOUS ONE-DIMENSIONAL
EXPONENTIAL FAMILIES

Set compound estimation under entropy loss with continuous one-dimensional
exponential families is pursued in this chapter. Before we get into details, let us look at
(1.12). If we can represent E(n(6)|x) in some form of pG(x), as we did in Chapter 2,
then we will be able to give a compound estimator. In continuous case, we use the kernel
density estimation considered in Singh (1974) to construct compound estimators. We
start our work with some notations that will be used in the chapter. Sections 3.2 and 3.3

discuss continuous exponential families with cumulant generating functions of the form

I: k
Zaqe“ and zaqe-q + bln(—9) respectively. Normal and Gamma distributions are
q=0 q=l

covered by these cases.
Recall that our distribution has density (1.1) with k=1 in the one-dimensional
case. For simplicity we will drop the ~ and let
(3.1) Po(x) = e""“"‘°’
denote the density of P9 with respect to p. In the chapter we assume that p. is dominated

by Lebesgue measure A with density denoted by u. Furthermore, we assume that u is

30

31

positive iff x is larger than c for some c 2 —oo. Without loss of generality we assume that
the variable x and the random variables X,,X2,---take values in (c, 00) in what follows.
In addition we assume throughout that

(3.2) G) = [a,B].

We continue the convention 0/0 :=O.

Remark 3.1
(3.3) pg s e‘""“’°(p.. + p1)
where w’ = sup{w(9)l9 so} and w. = inf{\p(9)|9 so}.
Proof (3.3) follows directly from the inequality: e8" 5 ea" + eBx for all x when 6
e [a,B].
3.1 Kernel Density Estimation
The kernel functions K satisfy K(s) =0 if s 55 [0,1] and IK2(s)ds < 00. We will
consider K that satisfy
(3.4) (in-1 IsiK(s)ds=[i=o] , i=O,-~,r—l
for given r and o < r. The kernels satisfying (3.4) for o = O are used in the estimation of
the density and those for u > O are used in the estimation of its uth derivative.
Such K(s) exists since there is a linear functional on L2 [0,1] which satisfies (3.4)

(see Rudin (1973), Theorem 3.5). Let

 

 

1 “ 1 X- -x 1
“M :=— K J

32

x > c for o = O, 1, ..., r-l, n 21 and 0 < h < 1. We let p§”’(x) denote the (n-l) term
average with the ith term deleted, i = l, 2, ..., n. The upper index (1)) will be omitted in

case 1) = O.

In the following let T=e‘”‘ [K2(s)ds, T,=[(r—1)!]"s{;‘[|K(s)|s'ds and

(I)

1 .
so=1alv1131. Also let p€“’(x)=;-_-;Zpé',"(x), p"(x)=pe,(x), p.(x)=p§°’<x),
jaei

 

 

_ Tpi(x) s0 _ r—k _ Tpon(x)
m”) ‘ \/(n—1)h“*'u<x,h) + “e ”h m") and D“) ' \[(n—1)h“‘*'u(x,h)

+ T,(e"’ —1)h""pGn(x). The upper index (k) indicates the kth derivative with respect to x

 

 

)

there. The next lemma concerns the variance and bias of 13‘" as an estimator for the kth

. . . . . 1 "
derivative p‘G‘” of the emplrlcal mixture pG = —ZP9. .
I) n n 5:]

Lemma 3.1 Let u(x,h) = essinf{u(t): x < t S x+h } with respect to A. Then

Tpe,(x)
nh2u+lu(x, h) ’

 

(i) Var9(f>‘"’ (X)) S

(ii) lng)‘°’(X) — p’(x>| s mes" —1)h"°pe,<x>,
(3-6) (iii) Eglf)‘°’(X)-P2’,’(X)IS D001)-
Proof Note that ElWlS [Var(W)]%+|EW| by triangulation at E(W) and moment
inequality for ElW—EWI . Thus, (iii) follows from (i) and (ii). Singh (1974, page 59)
proved ( i ) and (ii). The bound in (ii) is slightly different from the Singh’s bound because
of use of the inequality le" —1|s|e' —1||x/ a| if lxls a instead of the Mean Value

Theorem. 0

33

Lemma 3.2 For n .>_ 2,
(3.7) E91P10)(X)—P(U)(x)I-<- D.(n) + 28.‘.’¢‘".‘”'(P..(X) + p.(X)) / (n -1)-

Proof Triangulate LHS(3.7) at p§°)(x). Use the fact |W —wi|/(n - l) s Zmaxle-l/(n — l)
J

for W]: =p(")(x) and (3. 6) for the following second inequality and (3. 3) for the third

inequality below

LHS(3 7)— < EelP(°)(X)“ P. (U)(X)I+IP. M00 P2:)(X)|
< D .(n) + 2maxp‘”’(x)/(n - l)

(3.8) s D.(n)+ 2s::e* " (p.(x)+ p.(x»/(n -1)
The proof is finished. 0

Lemma 3.3 Assume for a positive number ho with ho < 1,

(3'9) = Cé‘ifidC—j—fl V(x x,xh,)

Let Uu(x) be RHS(3.7) with u < r. Then

IUu (x)d “(X)<4 S(,__e___“’.1")‘"(n +zcfie%(w'-w.)(n_1)_%h_(u+%)

+ mes” —1)h""

(3.10)

forO<hS h0 ananZ.
Proof The first and last terms in RHS(3.10) follow directly from integration with

respect to u. The middle term in RHS(3.10) follows from integration with respect to 11

applied to the first term in Di(n) after noting that

p.(x) < eW-v , (p. (X) + p. (x))
U(X,h) — uy(X,ho)

 

9

34
which follows from (3.3), the fact that u(x,h) is monotone in h, and the inequality

(a+b)y13a%+by’ fora,b20. O

k
3.2 Case w(9)= Xaqe“ (k 22 and {aq} are constants)

q=0
2
Example 3.1 For the normal distribution family N(9,1), 111(9): .9?
We first establish a set compound estimator. We assume
(3.11) ®=[a,B]g N".

By Proposition 1.4, for any compound estimator

(3.12) t.(X)=n"[S.(X)]. i=1. ...n.

(3.13) lD..(9.t)ls B.C.n"ZE.ls. — r;,(X.)l.

Because 111(6): Zaq 9" and 11:11), {g(x):= E(n(9)|x)= an E[9q 'lx]. With density
q: —0

(3. 1) it is easy to show that E[€)q lIx]: p‘Gq l’(x)/ pG(x) so that

. " p‘qf”(X)
(3.14) taunt)”..Lquqaq :0_(x) .

 

We consider

It fim- ”(X )

(3.15) s(X):=[a +anp' 1’5..(X) ———-]Wm,,,i i=1,2,---,n.

1:
Theorem 3.1 Suppose that \p(9) = Zaqeq and that (3.11) and (3.9) hold. Then {t}
q=0

defined by (3.12) and (3.15) with choice h = n""2”" is asymptotically optimal with rate

r-k+1
2r+1 '

 

n"’ where y =

35
Proof From (3.13) and the Singh-Datta inequality (Section 1.3) with

k k
a==anqp$."’(X. ). bz=po, (X. ). A==anq13§"‘”(x.) and B==l3.(Xi)
q=1 q=|

. A a 1 a
(3-16) ISi—TG"(X1)ISIE_EIAD S EHA -al+(lgl+D)lB-bl}

where D=n(B)-n(a)- Here We ln(B), M00],

k
(3.17) lA-al qulaqllfiE“"’(X.)—- p‘o‘,"’(Xi)I
q=2
and
(3-18) lB'bl zlPi(Xi)" Po,(xi)|-

By applying (3.13), (3.16)-(3.18), Lemmas 3.2 and 3.3,

 

k 4sq'le‘VO’W' . . l i
SUpI Dn(ga 91 S BoCo {Z {QI aq|[—-—9——-—-——— + 2CJ’1—‘efl‘1’ ‘W.)(n __.1)—2 hi—
9 q=2 (11 " 1)
l 4eW°-w.
(“9) + Hes" -1>h""* 1} + 11 n(a)IVI n(13)l+n(B) — nth)” (n _1)

+ 2cfiei“*’"‘"~’(n -1)‘ih‘% + mes" — l)h']).

1

For the choice of h = n“5 with 8 = ,
2r+1

 

RHS(3.7) 5 Chi-1 ,

 

r—k+1

where Cl is a constant and y =
2r+1

1:
Remark 3.2 (i) If w(9) = Zaqe“ (k 22 and {aq} are constants), then N is closed

(i=0

9‘!

because J‘ee‘dtt=eza“ If 9“ —>a, a finite boundary point, Iliflee'xdps

36

28.93.

flme =e}:"“°‘q <oo by Fatou Lemma. Note that E9(X"‘)=(eza“eq)‘"", E9(X‘“)

It

exists and is continuous of 9 on N (m> O) by the same argument before. And also 0 e N
if a2 = 0. If 111(6): a9k (k > 1), then a > O for all even integers k >1. Ifk is odd, a9 is

nonnegative for all 9 belonging to N if N0 is not empty.
(ii) In the above case, i.p.(k/2) (integer part) can not be even. The reason is as

follows: assume p. is the corresponding measure to the cumulant generating function

111(9) == a9k . Take 90 6 N3 , the interior of the natural parameter space for u and define

du' = e°"‘du. Then 11’ is a finite measure. a(9 + 90 )k will be the corresponding cumulant

generating fimction. First let k be even and k/2 be even, for any tl , t2 6 R,

689:: ea[i(t.-tz)+9ttlk

ealiit.-t.>+e.l"

2.19:; 2a93-k(k-l)a03"(t.—t,)’+.-.+2a(t.-t,)"

=6 '6 ,

eat}:

which is smaller than 0 for large (t| — t2 ). This means came")k

is not a Laplace-Stieljes
transform, contradicting to the argument of Brown (1986) (see page 42).
Now let k be odd and (k-1)/2 be even. In this case, a90 is nonnegative. For any

t,,tze R,

net a[i(t -t you)"
(1 t 0 e I 2 =62”: _eaBu[2e:"_k(k_|)93"((|—:2)1+-~-+2k(t.-t2)b']
ennui-wen" e393 ’

which can be negative for large (tl — t2) if 90 ¢ 0.
In case 90 = 0, take t,,t2,t3e R such that
a[(tl_t2 )k + “3 — t1)k + “2 " t3)k] ‘7‘ 2m”

for any integer m,

37

1 emu-ti)" cam-1.)"
cal-(‘z-Il )k 1 eai(I1‘t3)k

cams-ti)" 631“)": )IK 1

= -2+2eos{a[(t,—t,)“ +(t, -t.)k +(t2 -t.)"]} <0.

a(it+6., )k

Therefore e is not positive definite, and is not a Laplace-Stieljes transform. So

a(6 + 90)" is not a cumulant generating function, neither is aG" in case that k is odd and
(k-1)/2 is even.

(iii) If k is odd, the dominating measure for 111(9) = a9k (k > 1) can not be finite
from the latest argument above.

(iv) In Theorem 2 on page 62 of Singh (1974), we can reduce the condition

(A2.1) there to

Jfldu<oo

u%(x,ho)

for some 0 < h() <1 because of the fact that LHS(A2.1) is increasing with respect to h,

where
C1. = —_\J P9 1 -
(p... A p. )’
An interesting thing is that we can get a better rate with this weaker and simpler
condition.
3.3 Case
(3.20) time) = iaqe-i + b ln(—G),

q=l

(-oo<a$9$B<0,b_<.OandSign{aq}=(-1)q forlSqSk)

38

Some common distribution families have this kind of cumulant generating
functions. We will see through the following examples.

Example 3.2 If aq = 0 for all q in (3.9), 111 is actually the cumulant generating function
of a Gamma distribution family F(—0,—b) which has density:

(_e)—beex-bln(-0)
with respect to u, where it has density x‘b"I(0m)(x) with respect to the Lebesgue

measure.
Example 3.3 Consider another example (see page 3 of Brown (1986) for further

reference ). Let Yl , - ~ - , Yn be i.i.d. coming from normal distribution family :

¢,.,.(y) = (2ne2)-%exp(_(y_..)2 Mal).

Let

x] =‘i?:=n"ZYi , s2 = it"Zoti —?)2 and x, =32 +Y2 = n"ZYi2.

i=1 in] i8]

Then the joint density of Y = (Y1; --,Yn) is

(3.21) fp'o,(y) = (27:02 )-g exp((nu / oz)xl + (—n / 262 )x2 )exp(—np.2 / 2c:2 ).

Let 0I = 2% and 02 = _2n_2. (3.12) becomes
6 o

fo..e, (x1, x2) = exP(91xl + 92x2 " Wm»

2
with w(0) = — 296'— — glog(— 23—2) , and the natural parameter space
2

N ={(91,92)2916R, 0, <0}.

39

For fixed 0, = 9.0 , define a measure 1(A) = I €910X1 du for any measurable set
RxA

A in R. Then the marginal distribution of (3.12) becomes an exponential family with

02 n 29
9 =__I.0___10 ___2
w( 2) 492 2 g n

as the cumulant generating function and r as the dominating measure.
Proposition 3.1 111 in (3.9) is a cumulant generating function of some exponential

family and now the natural parameter space N = (-oo, v}, open or closed for some v _<_ 00.

Proof Let

f(W) ___ ew-w) = [H eI-q|W"' ]wb , W 6 (0,00) .
q=l

I:
Then f(w)s H e"“' for w e[1,oo).
«i=1

1

Let Cl=l, C~=‘.—._
’ J!(J-2)!

(i=2, 3, ° ' °), and let

a)

jaw [w2"'e""’] , forj=1,2,

Q,(t,w) = c
According to Theorem 7.3 of Boas and Widder (1940), if we can verify the following
inequality:

(3.22) Aj(c,t):= e°'tj" [Qj(t,w)f(w + c)dw 2 0

for c > 1, j = 1, 2, ..., and 0 < t < 00, (which is (7.6) there), then the proposition will
follow.

First, let us show that

40

. . 1
(3.23) f‘”<w>= (—1)Jf(w>P.(;v—),

where

j(k+l)

1 1 .
P,(;V-)= 29“?

with C1120 for 1 Sisj(k+1).

f‘”(w) = f(w)[zk: (—|am|mw‘""l ) + b / w]

= (—1)f(w>l§ (IamlmW'"‘" ) + (—b)/ wl

= (—1)f(w)P.(-:;),

where P,(;1v-) = Z (Iamlmw‘m")+ (—b)/ w.

Assume (3.23) is true for j.

j(k+1)

 

j+l) _ _ j+1 __1_ i _ j+l i
f‘ (w)—( 1) f(w>P.(w)P,(w)+( 1) f(w)(§C..w...)

j(k+1)

. 1 1
= (-1>’*'f(w)tP.(;)P,(;) + ;C.

W
:= (-1)"" f(W)P,-..(-v1;)
(3.23) is verified.
Using the argument on page 4 of Boas and Widder (1940),
Aj(c,t) = (~1)’cjtj"e°‘ fw“"f‘”(w + c)dw 2 0,

foranyc>1,j21and0<t<oo.
The proof is complete.

The derivative of 111(0) is

41

k
time) = —anqe'°" + b0".

q=l

According to Proposition 1.2,

k
To(X) = \i'"[E(\ll(9)l X)] = ‘11-'1—anqE(9“'-'|X)+ bE(9"| X)]-

q=|

Let pIG'Wx) = £301“- pe(x)dG(8) and pG(x) = fpe(x)dG(9). Then with these

k MW» {-11
notations 10(X)= \iI"[-anq £§___(_x) + b Po (X)

q=l pG(x) pG(x) ]

The following lemma generalizes a result of Singh (1974) for k = 1 (see page 69)
to k > 1 case.

Lemma 3.4 Let xe R , G a prior on [a,B]. Then for any integer k > 0,

(3.24) p‘o’“(x) = (—1)“ Es... 1:... [Boom

Proof Induction on k 2 1 will achieve the proof. Observe that the integral of pG(s) with
respect to the Lebesgue measure ds on [x, 00) is the integral of —-(1;pe(x) with respect to

G0 on [01,13] . Assume (3.24) is true for k = n-l. Now consider k = 11 case.
RHS(324) = H) I ds.-. f 9"""’p.(s.-.>dG<e)
= [(4) 0"""’e““‘°’dG(0) f ee’"'dsn_, = LHS(3.24).

We have used the assumption for the first equation above. The lemma follows by

induction. 0

Let L=log(h')/B and

42

A _ +L “+1. 2+1, |+L A
p: “(>0 = (-1)“ I ask“. 1:. ds,_,--- 1: as, 1: p,(s)ds
Lemma 3.5 In the context above,

2

n -1
LkTPi(X)

(n -1)hu(x,kL + h)| m"

 

E, | fil‘“ - plink’IS—IBI“" {kh'pa (x) + eW‘W' (p.00 + ppm)

 

(3.25)

 

+ T,(e’°L —1)h'pi(x)} +J

Proof Triangulate the LHS(3.25) about k iterated integrals of p0_(s) changing each

iterated integral’s interval at a time and apply the subadditivity of the absolute value,

LHS(325)
+L k_|+L 2+L I+L A
S dSk-l I»: dSk—2”°£ dslf' Egilpa(S)—po. (S)IdS
an k-I*L 2+L .+L
+ I+LdSk—l [H dSk—2H'J'2 dslfl p0,, (S)dS
m m 1+L ”L
+ Ids,“I £_I+Ldsk_2---£ dslfl pGn(s)ds+-~
+ Ids“ £_|dsk_2...[ dsl[+LpGn(s)ds
Consider the second one of k+1 iterated integrals above, denoted as [2. Replace
the upper limits of k-l inner integrals of I2 by +00 and use Lemma 3.4 in the first
inequality below,

1, s (-1)" fe"‘p.(x + L)dG..(e) snark p0,,(x + L)
S”SI-k po,(x)eBL =ll3l—k h'pGn (’0

Similarly, we have |B|"‘ h'pGn(x) as the common bound for each iterated integral

after the second one. For the first one, denoted as 1,, applying Lemma 3.2 in the first

inequality and Lemma 3.4 in the second inequality below,

43

+L |H+L z+L I+1. 2 ._
It=f (isle-ll:l dSk-2”'[2 dsnfl [ii—:16, W'(Pa(5)+PB(S))

 

 

 

 

 

 

+D-(n)]ds
k CW 'oW
<|l3|' [2r1 1(Pa(X)+Pp(X))+Tr(e-a -1)Pa(X)]
,_ ,+L [+Ld +L Tp (s)
+deSk1l, ds" 2 ds'f. \/(n—1)hu(s, h) d8
k 26‘”. "'W
<IBI- [n 1[(130.004’I);3(X))+T (6“! -1)P. (X)]
T +L k +L
+\/(n—ln)hu1(x,kL+h)[ dsk“£-.d ”WI ds'ffi 1" (3) )ds

1
Because by the concavity of x2 and Jensen inequality,

n+L l n+L l. 1 _1 1
[ Jp.(s>dssuf [ p.(s)ds>* sum! was.)

 

 

 

Therefore,
2e‘""‘"- LkTp-(x)
I 5 .k + + T ‘a -1 hr - + '
. IBI [ n_1 (p.00 p.00) ,(e ) p.(x)] \/(n_1)hu(x,kL+h)lmk
Combined with the first part of this proof, we have our lemma. 0

Lemma 3.6 Let {pe(-):9 66)} be the one dimensional exponential family (1.1)
mentioned in the beginning of Section 3.1. Assume that for numbers Cl,C2 and I?

with c,,c2 20 and E s 0,

 

pv(x) % i m
(3.26) ML” J( W, (h I)“ h)> du(X) s h (C. + Czlloghl )

holds for O < h < l and some real number m. Let Vk(x) be the RHS (3.25). Here V0(x)

equals to U0(x) in Lemma 3.3. Then

44
(3.27)

qu(X)du(x)

q

 

T _ -
—(n—1)‘ h‘ '*’(C. +C2|loghlm )

e“"“"- + T,(e'°‘ —1)h']+ 2C 'qu

.<.|B|“‘ [W + _1

forOSqSk+landO<h<l.

Proof Through (3.25),

 

LHS(3.27) 3| or“ [qh' +

n
+’L 6-“; _W) Pa(x)+Pp(X)d
(n— DhIBIq I u (x qL+ h) dMX)

(3.27) is obtained by the fact that u(x,qL+h) 2 u(x,(k+l)L+h) for O .<_ q S k+1 and our

4 .
_1 e‘" ‘W° + T,(e‘°‘ —1)h']

 

 

assumption. 0

”110x —m.

Example 3. 4 Let u be a measure that has density e 'llop) (ml <1 and m0 2 0)
with respect to the Lebesgue measure. For the exponential family (1.1) with respect to u,

N=(-oo, mo).ForO<h<l,

 

v i pf,(x) du(x)_ < v ozfopj (x)e 1 1%“me + kL + h)”7 du(x)
vetam UI(X,1(L+ h) vetam
m;,k:

s h 2" ((:l + Czllog bl?)

r and m=fl.
2

 

Condition (3.26) is satisfied with E = 20k

Theorem 3.2 Let {pe(-):G e (9} be the exponential family with (3.9) as the cumulant

generating function. Assume that the condition (3.26) holds. Let

45

’t‘(X)= w l{W(a)v[—Zk:qa 2119‘) b13____E”(X.)

q .13.(X.) p..(X ) ———-—]A w(l3)}

Then {i = (M29, - 22, tn (3») is an asymptotically optimal set compound estimator and
k+l

(3.28) supelD (e, i)|= O{n'7[ZO(M (logn)’ +M (logn)m+’)]},

q=0

l'

where =—————,
Y 2(r—£’)+1

{Mq} and { M q } are nonnegative constants.

Proof Using the fact that |a v f A b - a v g A bl SI f — glAl a — b| for any real numbers a, b

and any functions f, g and the Mean Value Theorem to the function \iI"(-) for the first

inequality, Singh-Datta Lemma for the second inequality and Lemma 3.6 for the third

 

inequality below,
RHS(3.13)$

.. iqa.f>l“‘“"‘<X.)—bfiE“‘(X.) iqa.p‘..‘f““"(X.—) bp‘5“(X.)
Bocon-'§E,{|q=‘ “.(Xn —“" p5,(X.) IA(2iI(B)-2il(a))}

sweat-‘2 Egpz.‘ (X ){Zmna na‘ “‘*"‘(X) p‘5 "f.‘*”‘(X>I)
i=1

+(-b)|13l'”(X.)- 9511(X.)l+[lv(a)IVIw(B)l+\iI(B)- v(a)llf).(X.) — po,(X.)|}

4
< Boco {ZMquGBF‘M [(q +1)hr+ewl —2v + T (e—a— 1)hr]
q+l

IBIrWHY—hw‘q+C2I‘Oghl'“ >>1+(— beBr (1.2.. 4 —-e“’ -~2 +T(e‘°‘ 1m

—1

+2C

 

+ zc,/|3‘m—T(n -1)-%1.-%+2(c, + Czlloghl‘“ )) + [I «anvwwwm — woo]

 

( 4 152“"- + r,(e'°‘ - l)h' + 2cfie2‘W""~‘(n —1)'2h‘2"+‘(cl + c2| log hl'“ ))) ,
n _—

46
for0<h<l.

___L_ _ _
At the choice of h = n """" , (3.28) 18 obtained. 0

k
Remark 3.2 Let (1(9) = Zetqe‘q + b ln(—9), with o< a s e s (3 < -oo , b s 0, {aq} being

q=l
all nonnegative. Then it is a cumulant generating function with support{u} being a subset

of (-00, 0] and N = (v, +00), open or closed for some v 2 -oo.

47

BIBLIOGRAPHY

Boas, RP. and Widder, D.V. (1940). An Inversion Formula for the Laplace Integral.
Duke Mathematical Journal, 6, l - 26.

Brown, L.D. (1986). Fundamentals of Statistical Exponential Families with Applications
in Statistical Decision Theory. IMS Lecture Notes -- Monograph Series 9.

Datta, S. (1991a). Asymptotic Optimality of Bayes Compound Estimators in Compact
Exponential Families. Ann. Statist. 19, 354 - 365.

Datta, S. (1991b). On the Consistency of Posterior Mixtures and its Applications. Ann.
Statist. 19, 338 - 353.

D. K. Dey, M. Ghosh and C. Srinivasan (1987). Simultaneous Estimation of Parameter
under Entropy Loss. Journal of Statistical Planning and Inference 15 (1987) 347-
363.

M. Ghosh and M. C. Yang (1988). Simultaneous Estimation of Poisson Means under
Entropy Loss. Ann. Statist. Vol. 16, No.1, 278-291.

Gilliland, Dennis C. (1968). Sequential Compound Estimation. Ann. Math. Statist. 39,
No.6, 1890 - 1904.

Gilliland, Dennis C., Hannan, James F. and Huang, J.S. (1976). Asymptotic Solutions
to the Two State Component Decision Problem, Bayes Versus Diffuse Priors on
Pr0portions. Ann. Statist. 4, 1101 - 1112.

Gilliland, Dennis C. and Harman, James F. (1986). The Finite State Compound
Decision Problem, Equivariance and Restricted Risk Components. Adaptive
Statistical Procedures and Related Topics, IMS Lecture Notes - Monograph Series
8, 129 - 145.

Hannan, James F. (1956). The Dynamical Statistical Decision Problem When the
Component Problem Involves a Finite Number, m of Distributions. (Abstract)
Ann. Math. Statist. 27, 212.

48

Hannan, James F. (1957). Approximation to Bayes Risk in Repeated Play, Contribution
to the Theory of Games 3. Ann. Math. Studies, No.39, Princeton University Press,
97 - 139.

Hannan, James and Huang, J.S. (1972a). Equivariant Procedures in the Compound
Decision Problem. Ann. Math. Statist. 43, 102-112.

Hannan, James and Huang, J.S. (1972b). A Stability of Symmetrization of Product
Measures with Few Distinct Factors. Ann. Math. Statist. 43, No.1, 308 - 319.

Majumdar, S. (1993). Asymptotically Optimal and Admissible Estimators in Compound
Compact Gaussian Shifi Experiments. Ph.D. Thesis, Department of Statistics and
Probability, Michigan State University.

Mashayekhi, M. (1990). Stability of Symmetrized Probabilities and Compact
Equivariant Compound Decisions. Ph.D. Thesis, Department of Statistics and
Probability, Michigan State University.

Mashayekhi, M. (1995). On Symmetrization of Product Measures. Mathematical
Methods of Statistics 4, No.3, 334-343.

Rainville, E. D. (1967). Infinite Series, Macmillan, New York.

Robbins, H. (1951). Asymptotically Subminimax Solutions of Compound Statistical
Decision Problems. Proc. Second Berkeley Symp. Math. Statist. Prob., 131 - 148,
University of California Press.

Rudin, W. (1973). Functional Analysis, McGraw-Hill.
Samuel, E. (1965). Sequential Compound estimators. Ann. Math. Statist. 36, 879-889

Singh, RS. (1974). Estimation of Derivatives of Average of u-Densities and Sequence -
Compound Estimation in Exponential Families. Ph.D. Thesis, Department of
Statistics and Probability, Michigan State University.

Swain, D. D. (1965). Bounds and Rates of Convergence for the Extended Compound
Estimation Problem in the Sequence Case. TR-8l, Department of Statistics,
Stanford.

Vardeman, SB. (1982). Approximation to Minimum K - Extended Bayes Risk in
Sequences of Finite State Decision Problems and Games. Bulletin of the Institute
of Mathematics Academica Sinica 10, No. 1, 35 - 52.

49

Yu, B. (1971). Rates of Convergence in Empirical Bayes Two-action and Estimation
Problems and in Extended Sequence - Compound Estimation Problems. RM-279,
Statistics and Probability MSU.

Zhu, J. (1992). Asymptotic Behavior of Compound Rules in Compact Regular and
Nonregular Families. Ph.D. Thesis, Department of Statistics and Probability,
Michigan State University.

0001.

 

 

1

Minimum
ems uefiiuom

AHVHGI'I

 

 

MICHIGAN stars UNIV. LIBRARIES
lllllllllllllllllllllIlllullll”Illllllllllllllll
31293015706199