HiveSQL
Search…
Technical informations
Tutorials
Language detection
Each time a post is created, the body content is analyzed to determine which language(s) it contains.
The result is stored in JSON format in the field
body_language in the table Comments
As a post can contain multiple languages, the result is an array.Each item in the array contains the following values:
- Language code
- Confidence score
- Is reliable - true/false
If the language cannot be determined (ex: post containing pictures only), the array will be left empty
The result is something like this:
A post with several languages will have his
body_language field set to something like1
[{"language":"es","isReliable":true,"confidence":8.22},
2
{"language":"en","isReliable":false,"confidence":5.12}]
Copied!
The
confidence value is related to how much text the post contains. The more text analyzed, the better the language analysis, the higher the confidence value. Confidence is not a ratio and can be higher than 100.If the post contains words in different languages,
isReliable will be set to true to identify the most probable language, even if its confidence value is lower. If there is only one language and
isReliable is set to false, this indicates confidence is too low.Be aware that language detector works using probabilities and sometimes it is not accurate with very short texts. The same happens when different languages used in the post have similar words.
The language detector can also be tricked when the content of a post contains lots of "technical noise" like pictures, source code, edit tags, ...
code
language
code
language
code
language
code
language
aa
Afar
ab
Abkhazian
af
Afrikaans
ak
Akan
am
Amharic
ar
Arabic
as
Assamese
ay
Aymara
az
Azerbaijani
ba
Bashkir
be
Belarusian
bg
Bulgarian
bh
Bihari
bi
Bislama
bn
Bengali
bo
Tibetan
br
Breton
bs
Bosnian
bug
Buginese
ca
Catalan
ceb
Cebuano
chr
Cherokee
co
Corsican
crs
Seselwa
cs
Czech
cy
Welsh
da
Danish
de
German
dv
Dhivehi
dz
Dzongkha
egy
Egyptian
el
Greek
en
English
eo
Esperanto
es
Spanish
et
Estonian
eu
Basque
fa
Persian
fi
Finnish
fj
Fijian
fo
Faroese
fr
French
fy
Frisian
ga
Irish
gd
Scots_Gaelic
gl
Galician
gn
Guarani
got
Gothic
gu
Gujarati
gv
Manx
ha
Hausa
haw
Hawaiian
hi
Hindi
hmn
Hmong
hr
Croatian
ht
Haitian Creole
hu
Hungarian
hy
Armenian
ia
Interlingua
id
Indonesian
ie
Interlingue
ig
Igbo
ik
Inupiak
is
Icelandic
it
Italian
iu
Inuktitut
iw
Hebrew
ja
Japanese
jw
Javanese
ka
Georgian
kha
Khasi
kk
Kazakh
kl
Greenlandic
km
Khmer
kn
Kannada
ko
Korean
Copy link