■■　短単位自動解析性能評価結果　■■ 評価には以下のスクリプトを使用 https://github.com/teru-oka-1933/unidic_ma_factory https://github.com/teru-oka-1933/unidic_ma_factory/blob/master/ph7_evaluation.py スクリプト中では形態素解析器性能評価システムMevALを使用している https://teru-oka-1933.github.io/meval/ 評価用データはtrain_list.tsv(9):test_list.tsv（1）のサイズで実施。 結果の見方は、https://teru-oka-1933.github.io/meval/#scorer　を参照。 フィールド番号： 1 品詞大分類（pos1） 2 品詞中分類（pos2） 3 品詞小分類（pos3） 4 品詞細分類（pos4） 5 活用型（cType） 6 活用形（cForm） 7 語彙素読み（lForm） 8 語彙素（lemma） 10 発音形出現形（pron） 性能評価レベル LEVEL 0 : 表層形認定性能 LEVEL 1 : 品詞認定性能 LEVEL 2 : 語彙素認定性能 LEVEL 3 : 発音形認定性能 ■■　フルサイズの語彙を使った評価結果　■■ ======================================== MEVAL SCORER ======================================== Gold: train_test_mecab/all.test.mecab Pred: eval_work_3.0.1/all_lex/pred.mecab ---------------------------------------- Field Num LEVEL 0 : 0 LEVEL 1 : +1+2+3+4+5+6 LEVEL 2 : +7+8 LEVEL 3 : +10 ---------------------------------------- Sentence Num: 24146 Gold Word Num (GLD): 246268 Pred Word Num (PRD): 245996 Character Num: 400759 ======================================== LEVEL 0 : 0 ======================================== Correctly Analysed Sentences: 96.68% (23345/24146 = 0.9668268) ---------------------------------------- COR : 244135 ---------------------------------------- Prec. : 99.24% (244135/245996 = 0.99243486) Rec. : 99.13% (244135/246268 = 0.99133873) F : 99.19 (0.9918865) ======================================== LEVEL 1 : +1+2+3+4+5+6 ======================================== Correctly Analysed Sentences: 87.35% (21091/24146 = 0.873478) ---------------------------------------- COR : 240565 ---------------------------------------- Prec. : 97.79% (240565/245996 = 0.9779224) Rec. : 97.68% (240565/246268 = 0.9768423) F : 97.74 (0.97738206) ======================================== LEVEL 2 : +7+8 ======================================== Correctly Analysed Sentences: 85.23% (20580/24146 = 0.85231507) ---------------------------------------- COR : 239715 ---------------------------------------- Prec. : 97.45% (239715/245996 = 0.97446704) Rec. : 97.34% (239715/246268 = 0.97339076) F : 97.39 (0.9739286) ======================================== LEVEL 3 : +10 ======================================== Correctly Analysed Sentences: 82.39% (19894/24146 = 0.8239046) ---------------------------------------- COR : 238460 ---------------------------------------- Prec. : 96.94% (238460/245996 = 0.96936536) Rec. : 96.83% (238460/246268 = 0.9682947) F : 96.88 (0.96882975) ■■　辞書未登録の表記が現れた時の性能評価　■■ 学習用データには出現せず、評価用データにのみ出現した書字形出現形のエントリ（lex.csvファイルの行）をlex.csvファイルから除外して学習・評価した結果 ======================================== MEVAL SCORER ======================================== Gold: train_test_mecab/all.test.mecab Pred: eval_work_3.0.1/shallow_unk/pred.mecab ---------------------------------------- Field Num LEVEL 0 : 0 LEVEL 1 : +1+2+3+4+5+6 LEVEL 2 : +7+8 LEVEL 3 : +10 ---------------------------------------- Sentence Num: 24146 Gold Word Num (GLD): 246268 Pred Word Num (PRD): 248711 Character Num: 400759 ======================================== LEVEL 0 : 0 ======================================== Correctly Analysed Sentences: 88.72% (21422/24146 = 0.8871863) ---------------------------------------- COR : 240822 ---------------------------------------- Prec. : 96.83% (240822/248711 = 0.96828043) Rec. : 97.79% (240822/246268 = 0.9778859) F : 97.31 (0.9730595) ======================================== LEVEL 1 : +1+2+3+4+5+6 ======================================== Correctly Analysed Sentences: 79.54% (19206/24146 = 0.7954112) ---------------------------------------- COR : 236161 ---------------------------------------- Prec. : 94.95% (236161/248711 = 0.94953984) Rec. : 95.9% (236161/246268 = 0.95895934) F : 95.42 (0.9542263) ======================================== LEVEL 2 : +7+8 ======================================== Correctly Analysed Sentences: 77.69% (18758/24146 = 0.77685744) ---------------------------------------- COR : 235025 ---------------------------------------- Prec. : 94.5% (235025/248711 = 0.9449723) Rec. : 95.43% (235025/246268 = 0.9543465) F : 94.96 (0.9496363) ======================================== LEVEL 3 : +10 ======================================== Correctly Analysed Sentences: 75.47% (18224/24146 = 0.75474197) ---------------------------------------- COR : 233722 ---------------------------------------- Prec. : 93.97% (233722/248711 = 0.93973327) Rec. : 94.91% (233722/246268 = 0.9490555) F : 94.44 (0.9443714) ■■　辞書未登録の語形が現れた時の性能評価　■■ 学習用データには出現せず、評価用データにのみ出現した語形基本形（語彙素読み（lForm）+語彙素（lemma）+語彙素類（lType）+語形基本形（formBase））に対し、階層的な見出し構造の木構造において当該語形基本形以下のすべてのエントリを除外して学習・評価 ======================================== MEVAL SCORER ======================================== Gold: train_test_mecab/all.test.mecab Pred: eval_work_3.0.1/middle_unk/pred.mecab ---------------------------------------- Field Num LEVEL 0 : 0 LEVEL 1 : +1+2+3+4+5+6 LEVEL 2 : +7+8 LEVEL 3 : +10 ---------------------------------------- Sentence Num: 24146 Gold Word Num (GLD): 246268 Pred Word Num (PRD): 248170 Character Num: 400759 ======================================== LEVEL 0 : 0 ======================================== Correctly Analysed Sentences: 90.24% (21789/24146 = 0.9023855) ---------------------------------------- COR : 241572 ---------------------------------------- Prec. : 97.34% (241572/248170 = 0.9734134) Rec. : 98.09% (241572/246268 = 0.98093134) F : 97.72 (0.97715795) ======================================== LEVEL 1 : +1+2+3+4+5+6 ======================================== Correctly Analysed Sentences: 81.62% (19707/24146 = 0.81616) ---------------------------------------- COR : 237401 ---------------------------------------- Prec. : 95.66% (237401/248170 = 0.9566064) Rec. : 96.4% (237401/246268 = 0.9639945) F : 96.03 (0.96028626) ======================================== LEVEL 2 : +7+8 ======================================== Correctly Analysed Sentences: 79.65% (19232/24146 = 0.79648805) ---------------------------------------- COR : 236322 ---------------------------------------- Prec. : 95.23% (236322/248170 = 0.9522585) Rec. : 95.96% (236322/246268 = 0.9596131) F : 95.59 (0.95592165) ======================================== LEVEL 3 : +10 ======================================== Correctly Analysed Sentences: 77.36% (18679/24146 = 0.7735857) ---------------------------------------- COR : 235059 ---------------------------------------- Prec. : 94.72% (235059/248170 = 0.9471693) Rec. : 95.45% (235059/246268 = 0.9544845) F : 95.08 (0.9508129) ■■　完全な未知語（=新語）が現れた時の性能評価　■■ 学習用データには出現せず、評価用データにのみ出現した語彙素（語彙素読み（lForm）+語彙素（lemma）+語彙素類（lType））について、当該語彙素が階層的な見出し構造においてrootとなるすべてのエントリを除外して学習・評価 ======================================== MEVAL SCORER ======================================== Gold: train_test_mecab/all.test.mecab Pred: eval_work_3.0.1/deep_unk/pred.mecab ---------------------------------------- Field Num LEVEL 0 : 0 LEVEL 1 : +1+2+3+4+5+6 LEVEL 2 : +7+8 LEVEL 3 : +10 ---------------------------------------- Sentence Num: 24146 Gold Word Num (GLD): 246268 Pred Word Num (PRD): 248145 Character Num: 400759 ======================================== LEVEL 0 : 0 ======================================== Correctly Analysed Sentences: 90.32% (21809/24146 = 0.9032138) ---------------------------------------- COR : 241622 ---------------------------------------- Prec. : 97.37% (241622/248145 = 0.9737129) Rec. : 98.11% (241622/246268 = 0.98113436) F : 97.74 (0.9774095) ======================================== LEVEL 1 : +1+2+3+4+5+6 ======================================== Correctly Analysed Sentences: 81.69% (19725/24146 = 0.8169055) ---------------------------------------- COR : 237450 ---------------------------------------- Prec. : 95.69% (237450/248145 = 0.9569002) Rec. : 96.42% (237450/246268 = 0.96419346) F : 96.05 (0.96053296) ======================================== LEVEL 2 : +7+8 ======================================== Correctly Analysed Sentences: 79.76% (19259/24146 = 0.7976062) ---------------------------------------- COR : 236396 ---------------------------------------- Prec. : 95.27% (236396/248145 = 0.9526527) Rec. : 95.99% (236396/246268 = 0.9599136) F : 95.63 (0.9562694) ======================================== LEVEL 3 : +10 ======================================== Correctly Analysed Sentences: 77.51% (18716/24146 = 0.77511805) ---------------------------------------- COR : 235173 ---------------------------------------- Prec. : 94.77% (235173/248145 = 0.9477241) Rec. : 95.49% (235173/246268 = 0.9549475) F : 95.13 (0.9513221)