Experimental and theoretical study of nuclear reaction rates in the rp-process