I think that this question is best approached through careful modelling, rather than pure mathematics. Here's the approach I took. I don't claim that this is the perfect approach by any means, but it's a start.
Spoiler: My simulations give a rate of approximately once every 0.66 years, for a population of 7 billion people who share US mortality statistics.
First, I took the US mortality tables from the Center for Disease Control. They only go up to age 100, so I need to extrapolate beyond that. I fitted a power law to the hazard rate $h(A)$ which gives the probability of dying between age $A$ and $A+1$, getting
$$h(A) = 3.54 \times 10^{-15} \times A^{6.933}$$
I assume $h(A)=1$ in the case that my power law gives me a number above 1. This occurs at $A=122$, which seems realistic (the oldest person to ever live died at age 122).
I then simulated an evolving population until it converged on a stable distribution. I assume $n(A)$ people at age $A$, and a constant birth rate of $9\times 10^7$ people every year (chosen to give a stable popluation of 7 billion). The result is a reasonable-looking population pyramid:
Now that I have a stable population, I simulate again. For each age $A$, the number of people of age $A$ in year $t$ is the fraction of the population aged $A-1$ at time $t-1$ who don't die, i.e.
$$n(t,A) = (1-h(A-1)) \times n(t-1, A-1)$$
When appropriate I approximate the number of deaths with the normal distribution, but for small populations I use the binomial distribution. In the case that there are some deaths in the highest age bracket, I calculate the probability that the person who died was the oldest person in the world at that time, and record this as an event.
Taking the total number of events, and dividing by the number of years that I run the simulation for, gives an approximate rate. The punchline is that in my simulation, I see 15,234 events in 10,000 years, for an approximate rate of once in every 0.66 years.
Assuming a population of one billion people (the population of the developed world, to which the US mortality statistics are most likely to apply) we can see the following histogram, which gives the age of the oldest person in the world at the time they die. Comparing to the wikipedia page for oldest people it looks as though the numbers are too high by 1-2 years, but otherwise I'm surprised at how accurate this crude model is!
One final chart. This is how the number of deaths of the oldest living person each year varies as a function of the total population. Roughly, it seems to be linear in the logarithm of the population. I'd be interested to see a more rigorous mathematical treatment that can get this result out
Edit: I corrected a bug which was causing me to estimate the rate as too high. I was approximating the binomial with a normal distribution with standard deviation of $np(1-p)$ rather than variance $np(1-p)$.
Edit no. 2: It was pointed out in the comments that I had another bug, and I also realized that I wasn't ever checking for the possibility that more than 1 'oldest person' dies in a given year.