From f5b5065025116d3786ad5cc6d7a2ae3ae4d87015 Mon Sep 17 00:00:00 2001 From: Alec Date: Fri, 3 May 2019 15:53:48 -0700 Subject: [PATCH] initial analysis --- README.md | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 83cb562..120d082 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,14 @@ # gpt-2-output-dataset This dataset contains: -- 250K samples from the WebText test set -- For each GPT-2 model (trained on the WebText training set), 250K plain samples (temperature 1, no truncation) and 250K samples generated with top-k 40 truncation +- 250K documents from the WebText test set +- For each GPT-2 model (trained on the WebText training set), 250K random samples (temperature 1, no truncation) and 250K samples generated with Top-K 40 truncation We look forward to the research produced using this data! ### Download -For each, we have a training split of 500K total examples, as well as validation and test splits of 10K examples. +For each model, we have a training split of 250K generated examples, as well as validation and test splits of 5K examples. All data is located in Google Cloud Storage, at under the directory `gs://gpt-2/output-dataset/v1`. @@ -30,7 +30,7 @@ We've provided a script to download all of them, in `download_dataset.py`. ### Detectability baselines -We're interested in seeing research in detectability of our model generations. +We're interested in seeing research in detectability of GPT-2 model family generations. We've provided a starter baseline which trains a logistic regression detector on TF-IDF unigram and bigram features, in `baseline.py`. @@ -41,6 +41,16 @@ We've provided a starter baseline which trains a logistic regression detector on | 762M | 77.16% | 94.43% | | 1542M | 74.31% | 92.69% | +### Initial Analysis + +![Impact of Document Length](https://i.imgur.com/PZ3GOeS.png) + +Shorter documents are harder to detect. Accuracy of detection of a short documents of 500 characters (a long paragraph) is about 15% lower. + +![Part of Speech Analysis](https://i.imgur.com/eH9Ogqo.png) + +Truncated sampling, which is commonly used for high-quality generations from the GPT-2 model family, results in a shift in the part of speech distribution of the generated text compared to real text. A clear example is the underuse of proper nouns and overuse of pronouns which are more generic. This shift contributes to the 8% to 18% higher detection rate of Top-K samples compared to random samples across models. + ### Data removal requests -If you believe your work is included in our dataset and would like us to remove it, please let us know at webtextdata@openai.com. +If you believe your work is included in WebText and would like us to remove it, please let us know at webtextdata@openai.com.