r/ChineseLanguage • u/3eeeeeeeeeeeh • 4h ago
Discussion How many chinese characters do you need to learn to know 99 out of 100 in an average text?
For quite a long time I have wondered how many characters one needs to know to know 90%, 95% or even 99% of Chinese characters in an average text.
On the internet numbers of about 2000 get thrown around frequently, but rarely more concrete evidence is provided. In this post I would like to showcase the following short python script and graph:
Using the data from https://huggingface.co/datasets/zetavg/tw-sinica-corpus-word-freequency/raw/main/words.json and the script
```python import numpy as np import plotly.graph_objects as go import json
def gen_normalization_table(): return str.maketrans({"麽": "麼", "爲": "為", "爲": "為", "説": "說", "裏": "裡"})
def normalize_chinese(txt, table=gen_normalization_table()): return txt.translate(table)
def getFreqDict(): with open("words.json") as f: freqList = json.load(f)
freqDict = {}
total = 0
for x in freqList:
freq = x["frequency"]
word = normalize_chinese(x["word"])
total += freq * len(word)
for c in word:
freqDict[c] = freqDict.get(c, 0) + freq
return {word: freq / total for word, freq in freqDict.items()}
d = getFreqDict()
X = np.arange(1, len(d) + 1)
vals = [val for _, val in sorted(d.items(), key=lambda item: -item[1])] Y = np.cumsum(np.fromiter(vals, float))
fig = go.Figure()
fig.add_trace(go.Scatter(x=X, y=Y, mode="lines"))
fig.update_layout( xaxis=dict(dtick=500, showgrid=True), yaxis=dict(dtick=0.10, showgrid=True), template="plotly_white", )
fig.show()
Export as interactive HTML
fig.write_html("plot.html", full_html=True, include_plotlyjs=True) ```
I generated the figure you can see above. On [https://valentin-herrmann.de/p/how-many-chinese-characters-do-you-need-to-learn-to-know-99-out-of-100-in-an-average-text/](my blog) you can also play around with an interactive version of the graph.
From this graph we can read off the following table:
| % | n |
|---|---|
| 50 | 170 |
| 60 | 260 |
| 70 | 395 |
| 80 | 610 |
| 90 | 1030 |
| 95 | 1510 |
| 99 | 2700 |
| 99.9 | 4350 |
So for knowledge of 99% of characters in an average text one needs to know at least 2700 characters.
