• Outline

• How to Apply

  • Usage Fees
  • Q & A

• BCCWJ Design

  • Data Sampling
  • XML Document Structure
  • Morphological Information
  • Integration of Morphological Information in XML (M-XML)

• Paid Edition Contents

  • C-XML Details
  • M-XML Details
  • TSV Details
  • Bibliographical Information
  • Bug Fixes
• Documents

• Research Results

  • Typological Report on the BCCWJ Lexicon
  • Summary of Research Results Using the BCCWJ

• Home
• Balanced Corpus of Contemporary Written Japanese
　　
• Paid Edition Contents

Paid Edition Contents

The Paid edition is a system of access for users who wish to be able to personally access and analyse the entire database of written language, such as information processing researchers. We currently offer access to all data, contained on four DVDs. Please note that software environments for accessing all the data are not provided.

• Provides access to all corpus data.
• Data cannot be redistributed.
• Before accessing the data, in order to protect copyright you must enter a usage contract with NINJAL.

An image of a file included in the Paid version.

The following data is compiled on the Paid edition.

Core Data (Roughly 1% of the total)  Approx. 1 million short unit words.

• The core data comprising Short Unit Words (SUW) and Long Unit Words (LUW), integrated XML (morphological analyses included), with core data for six different registers offered. Total number of samples: 1,980.

Character-based XML (C-XML)  Approx. 100 million short unit words*.

• XML data containing tags related to document structure. Data is divided into Fixed- and Variable-length samples. 172,675 total samples.
• Fixed-length samples are taken from the following 5 registers: OW, PB, PN, PM, LB, while variable-length samples are taken from all 13 registers.

Integrated-format XML (M-XML)  Approx. 100 million short unit words*.

• Integrated-format XML documents (morphological information included) is provided (there is no distinction between the data for fixed- and variable-length samples). Data is provided for all 13 registers.

Documentation

• Manuals and bibiliographical information are included (5 types). Copyright data is also provided.

Tabulated Data

Tabulated data is offered for both Short and Long Unit Words. Data from all 13 registers is provided.

Total Word Count

The included morphological analyses cover the short unit words and long unit words from each category of each subcorpus as listed in the table below (both TSV and M-XML). Here, the totals for each sub-corpora is presented, with white spaces and symbols excluded.

Word Counts of Small and Long Unit Words in Each Sub-Corpus

Important Points Regarding the Text

The provided XML data corresponds closely to the original text ofthe BCCWJ, but before morphological analysis is conducted, an XML process (numTrans) is applied to ease reading. This process produces output as in the following example:

In this way, the text "２０１１年" is parsed as 「二千」「十」「一」「年」 (two thousand eleven) rather than as 「２」「０」「１」「１」「年」 (two zero one one). The actual meaning of the original character string will be retained through the parsing process. Also, at points where fractions occur, the process will also shift characters as necessary to maintain readability.

Because these processes occur, the XML containing morphological analyses (M-XML) will not always correspond exactly to the text extracted and used in the character-based XML (C-XML). However, the TSV and M-XML data is retrieved from the same source as the C-XML data.

In portions of the data including morphological analyses, original character strings are marked with the tag "originalText", while character strings modified for morphological analysis are marked with the tag "orthToken".

In addition, because this number conversion process occurs automatically on extracted text, there is a chance of errors on data outside of the core data set which is checked manually.

*Reference：C-XML Details

Important Points Regarding Morphological Analysis

The morphological information regarding short and long unit words is the same in both TSV and M-XML formats, and so there is no need to parse identical SUWs and LUWs as separate categories.

The Short Unit Words were all manually checked based on analyses of UniDic. Because the analysis was not done using only a single version of UniDic, the SUWs of the BCCWJ and UniDic may not always be the same when compared. This is also true of Long Unit Words.

All of the morphological analysis includes any information that was thought to be important, regardless of the danger of redundancy. Although it would be enough to include an ID with each SUW to indicate its correspondence with data from UniDic, it was decided not to pursue this method.

In addition, the M-XML and TSV formats do not include bibliographical information, so it would be necessary to retrieve the sample IDs separately in some cases.