Crthuky

Heaps' Law basically is an empirical function that says the number of distinct words you'll find in a document grows as a function to the length of the document. The equation given in the Wikipedia link is

where $V_R$ is the number of distinct words in a document of size $n$, and $K$ and $beta$ are free parameters that are chosen empirically (usually $0 le K le 100$ and $0.4 le beta le 0.6$).

I'm currently following a course on Youtube called Deep Learning for NLP by Oxford University and DeepMind. There is a slide in a lecture that demonstrates Heaps' Law in a rather different way:

The equation given with the logarithms apparently is also Heaps' Law. The fastest growing curve is a corpus for Twitter data and the slowest is for the Wall Street Journal. Tweets usually have less structure and more spelling errors, etc. compared to the WSJ which would explain the faster-growing curve.

The main question that I had is how Heaps' Law seems to have taken on the form that the author has given? It's a bit of a reach but the author didn't specify what any of these parameters are and I was wondering if anybody might be familiar with Heaps' Law to give me some advise on how to solve my question.

A straightforward rewriting of the Wikipedia formula gives

log V_R(n) = log K*n^beta
 = log K + log n^beta
 = log K + beta*log n

This allows us to identify K=C and beta=-alpha (probably the WSJ uses a different formulation of Heaps' law

V_R (n) = fracKn^alpha

). The remaining b is a strange additional parameter not present in the original formulation of the law (and irrelevant, too, because the law is about large numbers where n-b is approximately equal to n).

I am going to guess and I hope someone has a clearer idea.

The question is interesting from a (my) novice math perspective, the wording suggests it was moved from mathoverflow.se?

From a basic linguistic perspective, there is little to no difference, all you need is a slowly decreasing slope. Both describe standard distributions, a concepts that's naturally observed in nature, from the distribution of raindrops to the dispersion of a laser beam. The specific choice depends on an accurate model. If it's instead just chosen to fit the data somehow, it doesn't hold much explanatory power, but it's a heuristic. For the specifics you should check out datascience.se, or whatever it's called where statistics are treated (compression of text is also rather important in signal processing).

The first one, V = k * n ^ b, is akin to the area of a circle, A = pi * r ^ 2, but inverted (taking the square root) and with a random factor, instead of pi, which can be pictured various ways, e.g. as a circle projected onto a wavy area or through a lense (doesn't really matter unless there's a specific need). If b is not exactly 0.5 the picture is a little different, but not really. The point is, this appears as the inverse square law, e.g. if a light cone hits a wall further away, the radius will increase linearly, but the power per square are will diminish proportionally with the inverse square of the distance. A^1/2 ~ r. The length of a text, n, increases likewise proportionally with the number of new words, n^0.5 ~ V. In other words, the text grows squarely with each new new word. That's also proportional to the circumference.

The second one seems more elaborate. I too have no idea what the extra variables are. Removing the logarithm we have *f(w) = C * (r(w)-b)^(-alpha)*. And transposed 1/C * (r(w)-b)^a = 1 / f(w). This is in principle the same polynomial form as V=K*n^b with several new parameters, if it were that V = 1 / f(w), k = 1/C, n = (r(w) - b), beta = -alpha.

There are a few notable differences. What's with those parameters? I'd assume the following:

• b is likely a threshold under which the distribution is useless, because if r(w)Basic vocabulary.




• If C is a constant as usual, then writing log(C), which would be constant as well, might just be a courtesy to ease solving for (w). It's inversely proportional to k, but that shouldn't trouble us now. One way or another it will be normalizing the results. I'm keen to assume that it means Corpus, but that gives me troubles. [todo]




• That leaves alpha to be explained, which seems to be a variable nudge factor determined per corpus by a specific statistical procedure for error correction.

The last one is crucial. Raising to a negative power of alpha (=reciproke of the power of alpha) is not quite the same as taking the square root (power of 0.5). But it is similar in effect because the ranges of the exponents are also different, b < 1 < alpha. The very important difference is that the number of new words will tend to zero as the number of typed words tends to infinity--which, if I may say so, is a rather real possibility with twitter ;-) While the old formula would require ever new words to grow the text.

Somehow I'm trying to see 1/f as a derivative, compared to mechanical accelleration. But I'll rather leave the rest of the exercise to the reader. Please add a link to the video to your question. thx bye