掲示板 Forums - incorrect kana type frequency (hiragana vs katakana)

I have noticed that some words have the incorrect kana as in hiragana vs katakana usage when being quizzed.

this is a separate from whether or not kanji is more popular via JMDict than the kana form.

e.g. in master schedule for beginner japanese, katakana shows as かたかな, but カタカナ frequency is 70%, 片仮名 is remaining percent and then very small percent is かたかな.

i just come across another word like this, カニ for crab. on renshuu card, it defaults to the hiragana again. kanji is under 40% frequency with 蟹, but when written using kana it is almost exclusively written as カニ rather than かに, but it shows on the card in hiragana.

i am not sure if there is something on backend with frequency calculations that defaults every time to hiragana for these types of words, but that might need to be look into. for exclusively katakana foreign words that dont have kanji, they appear correctly in katakana. i am reporting cards like this, but I am seeing it quite a bit so there might be something that needs to be looked at as in general way in which these cards are being processed that have kanji components.

hi Michael! im pulling the data from jpdb and jmdictdb (on edrdg dot org)

here is jpdb assessment https://jpdb.io/search?q=%E3%8...

here is jmdictdb assessment https://www.edrdg.org/jmwsgi/e...

jpdb I think uses hybrid of jmdict comments and some proprietary frequency corpus list/ngram.

jpdb:

カニ 61%

蟹 38%

蠏 minor

for jmdict:

eij katakana
n-grams
蟹 32938
蠏 13
カニ 58605

which is very close to those same percentages

so here we can see that there are no uses of かに　in hiragana from major or known corpora

for your lookup on jisho, there is no plain hiragana usage, it defaults to the kanji (the 30s%) n-gram frequency, and the katakana is shown as alternate, but there is no single hiragana usage listed. かに should not be used in hiragana form for a card here imho because it just isnt used that way in japanese (and ive never seen it that way either, restaurant or story uses one of the kanji or katakana)

sorry if im not explaining it well, but i do think for some things like this it is important that the common kana form (if kanji is not being used) is displayed when learning, because that is what the learner will see (u will probably never see かに, so why learn it as the first exposure to the word)

i understand that this might not be a frequent occurance of frequency lolz, but for certain cards and in the dictionary, think they should be manually edited such as this one to display katakana, not hiragana. i can just submit when i see ones like this. when user types "crab" into dictionary on renshuu, i think it should return either katakana or the major kanji form. currently it returns hiragana (virtually no real world usage), before then listing both kanji forms and the katakana.

this is just how i feel, its ofc fine however u choose to make it and i dont know how complicated this is to change, but for some words like this i think even if just manually edited on backend that would be good

hi Michael!! that is astute noticing it is just a comment, and not from Jim Breen himself there. i just did little searching just now, this seems like it is open source frequency https://clrd.ninjal.ac.jp/bccw...

you can skip to later where I type skip to here!!! it is also included on a list of open source and free for educational usages on this list of yomitan dictionaries that also includes some other processed n-grams and frequency lists https://github.com/Kuuuube/yom...

some of those like jpdb I am assuming are non-free, but that BCCWJ one is "It is free for use for research or educational purposes." on their website. that one looks like it was processed from this https://github.com/toasted-nut...

also found this paper, which i just skim now http://faculty-sgs.tama.ac.jp/...

after thinking for a bit and doing some more searching, i think that toasted script just processes surface/lexical freuqency alone, and not agglomerated into lemma frequency...

u can SKIP TO HERE!!!!!!

did some more searching, and now find this http://faculty-sgs.tama.ac.jp/...

SUW and LUW data for lemmae! "The corpus word lists are grouped according to the two word unit definitions used within the BCCWJ project (NINJAL, 2011) and UniDic (Den, et al, 2007); namely, short unit words (SUW) and long unit words (LUW)." it includes OrthBFreq for lemmae, and you dont even need to perform the calculations because there's also OrthCover which is "Ratio of total lemma frequency covered by a particular orthographic base form"

so I think that's the what to do, should this be something that you end up being interested in doing. because renshuu is a learning resource and not some technical dictionary, I think that starting even with just the SUW data (not even needing to merge it with the LUW) would be good. could even add a checkmark flag/option in renshuu's dictionary to display percentages next to surface realizations/forms. then under some experimental feature, there can be a button that says something like "reprocess schedule to orthographic ratio" which will modify cards in that user's schedule (if they are not touching the dictionary?) or are the cards still not duplicated somehow? maybe shim layer or can create a shadow/duplicate copy of cards in the schedule and then rearrange forms according to ratio. that's more suggestion at the end, but for now i think i will do that with my exported test schedule (run it through SUW ratio processor i will program to rearrange the terms according to freq ratio).

in summary i think that something like former part about adding percentages to dictionary is at least simple, only thing that terry joyce asks is "Respecting standard curtesies in these matters, individuals who utilize the corpus word lists within published works are politely asked to cite the Joyce, et al (2012) paper." so could say something like when u check the experimental box to "add orthographic frequency/ratio to dictionary entries" there can be a small subtext that says "utilizes Japanese lexical properties database (JLPD) processed by terry joyce, as described in Joyce, Hodošček & Nishina (2012), based upon BCCWJ corpus (licensed as free use for educational purposes)" that is the paper that i listed before (this one: http://faculty-sgs.tama.ac.jp/... )

sorry i type a lot stream of consciousness, this was fun for me too and that paper interesting that i will read in more detail later. maybe you also find this interesting, maybe not though hehe

interesting! i wonder why those are so much larger? i cannot download to chek file of that size now, but the joyce ones have those computed ratios already included, and the files are only several MB, rather than GB... strange!

maybe because BCCWJ is textbook corpus, but google n-gram is... everything that people type online?

ok reading through that s-yata site more and it seem to me like those a large because most of the n-grams are of large N, and also web. i think to start at least use the small several MB ones from joyce because that's more vocabulary type listing. the s-yata ones are probably for training language models or machine translator programs? but 頻度 1000 以上のファイルリスト 圧縮時 107MB，展開時 541MB I guess not too large... though u will have to process 500 MB file and perhaps calculate ratios. not sure how large the dictionary here is, but for expression morpheme processing, maybe if u have the backend compute?

the SUW data is only 13 MB, and even the LUW data is 115 MB... but just see that 115 MB compressed... close to the 107 MB from s-yata. maybe comparing the datasets and seeing how structured. having to compute orthographic frequences within lemmae might difficult, and I doubt that the s-yata file even has lemmae in it, as in no normalized dict headwords like joyce which is computed from unidic. morphological analysis segmentation on the s-yata set is going to probably require u to normalize the data somehow and will probably take some good compute time. joyce is already precomputed.

The google-based ones are not just 1-gram, and they are not tied to just dictionary entries. I've been running them through a script I wrote, and I think it's maybe using 1-2% of the actual entries in those files, so most of them are not what we'd consider to be dictionary entries (at least, not a dictionary like jmdict).

i assume the numbers for the s-yata set will have different frequencies because of how the internet is structured (the internet is more imbalanced than textbook corpora, as certain types of data are magnified (or suppressed) in frequency because of the nature of what types of content gets put online and how often it is "repeated" compared with a selected array of textbooks or other such written documents, and what parts of that are accessible to google. google does have a ngram viewer for books, but that dataset u linked on s-yata uses google internet searches, not books. Nグラムは一般に公開されている日本語のWebページでGoogleがクロールしたものから抽出されている。

the high number of かに might b becuz it is from a different lemma? is there lemma information in that dataset? there might be morphological forms of かに that imply crab (or otherwise) that are being parsed differently. かに座 versus カニ。

re: how you can exclude

I think this is done by whatever method is used to normalize the dataset into lemmae. in the joyce paper, it briefly describes and links how the unidic source handles it under § 5, orthographic representation, issues and data. it describes it well there i think, but also there are links to the actual unidic methodology.

are the lemmae within the google dataset normalized into semantic groupings? if not, that's probably why there is this descrepancy. i think maybe to compare the structure of both datasets maybe can see why there is difference. even though the upper set you are using is morpheme based, thats still different from semantically normalized.

ok i think of another example besides かに座 to help make it more clear maybe, なにかかにか surface form, as in 何かかにか、this should morpheme parse into 何 + か＋か＋に＋か。　　2--gram from these morphemes will give you かに。 but this is not crab. it's a 2-gram even after morphological parsing (not just a 2-gram by single kana parsing), but with no semantic normalization or disambiguation to crab (after all here it is not a crab).

that's why those ngram datasets, even with morphological parsing/segementation arent going to really work here i dont think (unless there is normalization in it but reading the site it doesnt seem it).

are you sure that page there is returning a 1-gram and not a 2-gram for かに? even if it's a morphologically parsed wordlevel or subword level 1-gram, it still isn't semantically normalized though, it's just for things like collocations and other types of language modelling. those aren't semantically linked as crab, the surface realization of かに there can have any other 1-gram forms dpeending on how that MeCab/IPADIC tokenizes it and how aggressively it splits particles. look at the example かれは難病の持ち主でした where you get a 1-gram for かれ bad example sry because かれ is a proper parsing of the morpheme. particle sequences get agglomerated without really aggressive stripping/tokenization, so かに will still be tokenized into aas a 1-gram in 何かに、どうかに、するかに、things like this. but its not meaning that it's a crab, because no semantic normalization. particle usage in japanese is really common, so particle chains like かに are going to appear a ton unless the parser is set to really aggressively strip them down.

this lines up with the data you posted. have you ever seen かに used for crab? in a restaurant, in a book, even in a google search? i dont think i have, or it has been just a few times, and certainly less than カニ. but with such a large hiragana count in that google ngram viewer of 13565484, you would think that it would be far and away the most common, right? well that's because that 1-gram of かに with such a large count is almost all instances of か＋に that are not being further aggressively stripped into か and に。

this is just for fun and i am sorry if it took a bit of time away from more important things, but mbe u stil find it interesting hehe, just want to say though that u need to be really careful if u are using non-normalized ngrams for something like a dictionary/frequency list, because that type of data isnt meant to be used for a human/semantic dictionary.

I believe it was 1-gram, but I am not sure if it specifies on that webpage.

As to the "real usage" of かに, that is unfortunately irrelevant. I do agree with you - I never see かに used. This is just one example that we've been using, though, but a one-by-one investigation of terms is simply not something I can do at scale, hence the hope for some kind of external data to use. If the external data, though, ultimately still needs a person to say "well, the 1-gram for かに is extremely high, but I judge that to not be referring to "crab", then we're back to square one - human intervention makes this project a non-starter.

This is all good info, and I feel like it's a good start to something we can use in the future!