No more than three times

倒推。

第 3 次不可再继续。

第 2 次看到 $x$：继续的期望为 $3.5$，所以 $x\ge 4$ 则停，否则继续。

因此从第 1 次选择继续时的期望为

$$\frac{1}{6}(3\cdot 3.5+4+5+6)=4.25.$$

第 1 次看到 $y$：若 $y\ge 5$ 则停，否则继续。

该策略下总体期望为

$$\frac{1}{6}(4\cdot 4.25+5+6)=\boxed{\frac{14}{3}}\approx 4.667.$$

---

Original Explanation

Work backwards.

On the third roll you must stop.

On the second roll, after seeing value $x$, continuing has expected value $3.5$, so stop if $x\ge 4$ and continue otherwise.

Therefore, if you continue after the first roll, the expected value is

$$\frac{1}{6}(3\cdot 3.5+4+5+6)=4.25.$$

On the first roll, after seeing value $y$, stop if $y\ge 5$ and continue otherwise.

Under this optimal strategy, the total expected value is

$$\frac{1}{6}(4\cdot 4.25+5+6)=\boxed{\frac{14}{3}}\approx 4.667.$$