Data Science for Humanities 1¶

Session: Data Science: Preliminary steps¶

Part 1: Data Acquisition and Preparation¶

Winter term 22/23¶

Prof. Goran Glavaš, Lennart Keller¶

Goals¶

After this part you'll know about:

  • Different types of data
  • The fundamentals of Pandas
  • Data preprocessing
  • Data cleaning
  • How to look for publicly available datasets
  • How to use web scraping to create your text collections/ datasets yourself

Types of data¶

Structured data¶

Any type of data that is available in a fixed shape and that follows a rigorous data model, (often in form of tables).

The shape is determined by the nature of the data, and provides useful metadata.

Entries have a well-defined and fixed datatype

Example: Sales Table¶

Transaction_ID: int Product: str Price: float Quantity: int
0 Beer 0.89 6
0 Chips 1.99 1
1 Milk 1.20 3
2 Bread 2.55 1
... ... ... ...

Unstructured data¶

Data that does not have a well-defined shape (or underlying data model).

Metadata or other characteristics are not explicitly marked.

The content might mix up different modalities (text, audio, images, videos, ...).

Data might be messy and contain errors, or erroneous artifacts.

Example: Web-crawl¶

<html>
    <head>
        ...
    </head>
    <body>
        <h1>Isn't HTML a well-defined data model?</h1>
        <p>I mean it has: </p>
        <ul>
            <li>A <b>fixed</b> set of elements</li>
            <li>Additional attributes to mark metadata</li>
        </ul>
    </body>
</html> 

----------------

<html>
    <head>
        <script type="javascript">
            $(document).ready(function() {loadContentForUser();});
        </script>
    </head>
    <body>
        <h1>Why HTML isn't a well defined data model:</h1>
        <div class="bulletpoints">
            <div class="bp align-left">HTML defines what which elements to use, but no how to use them</div>
            <div>A strict and coherent form is never enforced or even guaranteed.</div>
            <div class="bp-last align-center italic">HTML is quickly evolving, and nowadays often used to define a template to render (multimedia) content dynamically</div>
        </div>
        <img src="../like-button" on-click="updateLikeButton()">
        <embed class="ContentForUser videos "></embed>
    </body>
</html>

Semi-structured data¶

We call everything between unstructured and structured data semi-structured data.

Does not follow a well-defined data model, but contains markers indicating semantic structures.

But those markers do not explicitly enforce a certain datatype.

Structures might change.

Example: Message-Log¶

messages = [
    ...
    {   
        "type": "text-message",
        "id": 1001,
        "timestamp": "Mo 28 Nov 2022 10:56:35 CET",
        "from": "Bob",
        "to": "Alice",
        "content": "Do you have any plans for dinner? 🧐 I found this cool recipe, wanna try it out?",
        "attachment": "image_00001.png"
    },
    {
        "type": "text-message",
        "id": 1002,
        "timestamp": "Mo 28 Nov 2022 10:59:46 CET",
        "from": "Alice",
        "to": "Bob",
        "content": "No, not yet. But that looks delicious! I'm in!"
    },
    {
        "type": "reaction",
        "id": 1003,
        "timestamp": "Mo 28 Nov 2022 11:01:00 CET",
        "from": "Bob",
        "to": 1002,
        "reaction-type": "👍🏻" 
    }
    ... 
]

Pandas: Wrangling with data in Python¶

Pandas is the most popular library to load and process structured (and to some extend semi structured data) in Python.

Installation¶

pip|conda install pandas

Import¶

Like Numpy there is one canonical way to do it.

import pandas as pd

DataFrame¶

Like Numpy's ndarray, there is one essential container class which builds the core of the library, the DataFrame.

Conceptually, you can think of a DataFrame as the Python version of a Spreadsheet.

Consider, the sales table from above:

Transaction_ID: int Product: str Price: float Quantity: int
0 Beer 0.89 6
0 Chips 1.99 1
1 Milk 1.20 3
2 Bread 2.55 1
... ... ... ...

It can directly be represented as a DataFrame:

In [1]:
import pandas as pd

columns = ["Transaction", "Product", "Price", "Quantity"]

data = [
    [0, "Beer", 0.89, 6],
    [0, "Chips", 1.99, 1],
    [1, "Milk", 1.20, 3],
    [2, "Bread", 2.55, 1],
]

df = pd.DataFrame(data=data, columns=columns)
df
Out[1]:
Transaction Product Price Quantity
0 0 Beer 0.89 6
1 0 Chips 1.99 1
2 1 Milk 1.20 3
3 2 Bread 2.55 1

DataFrame consists of columns and rows.

Typically each column represents a feature, or field of your data. And each row represents a represents a record/ instance of your dataset.

In [2]:
df.shape, df.columns, df.index
Out[2]:
((4, 4),
 Index(['Transaction', 'Product', 'Price', 'Quantity'], dtype='object'),
 RangeIndex(start=0, stop=4, step=1))

Pandas offers a wide variety of different way to access your data:

In [3]:
# Access columns
df["Price"]
Out[3]:
0    0.89
1    1.99
2    1.20
3    2.55
Name: Price, dtype: float64
In [4]:
# Access rows
df.loc[0]
Out[4]:
Transaction       0
Product        Beer
Price          0.89
Quantity          6
Name: 0, dtype: object

Also there is a wide variety of additional functions to manipulate your data, or create new aggregated views on it:

In [5]:
# Sorting based on specific columns
df.sort_values(by="Price", ascending=False)
Out[5]:
Transaction Product Price Quantity
3 2 Bread 2.55 1
1 0 Chips 1.99 1
2 1 Milk 1.20 3
0 0 Beer 0.89 6
In [6]:
# Aggregating all rows with a specific value
df.groupby("Transaction")["Price"].sum()
Out[6]:
Transaction
0    2.88
1    1.20
2    2.55
Name: Price, dtype: float64
In [7]:
df.groupby("Product")["Quantity"].mean()
Out[7]:
Product
Beer     6.0
Bread    1.0
Chips    1.0
Milk     3.0
Name: Quantity, dtype: float64
In [8]:
# Counting unique values within columns
df.Transaction.value_counts()
Out[8]:
0    2
1    1
2    1
Name: Transaction, dtype: int64
In [9]:
# Saving data in a variety of commonly used formats
df.to_csv("sales.csv", index=False)
df.to_excel("sales.xlsx", index=False)
In [10]:
# Loading data in various formats
df = pd.read_excel("sales.xlsx")
df
Out[10]:
Transaction Product Price Quantity
0 0 Beer 0.89 6
1 0 Chips 1.99 1
2 1 Milk 1.20 3
3 2 Bread 2.55 1

And so much more!

How to learn Pandas¶

Learning Pandas - even more like Numpy - is best done, while you work on your specific projects.

Going through all its features in the course would be too extensive. So we'll introduce additional features as we go on.

You are strongly encouraged to stay curious and always keep on googling "How to do xy in Pandas?" 😊

But there are some great resources to start digging deeper:

  • Pandas Docs
  • Pandas Cheatsheet
  • Chapter on Pandas in the Data Science Handbook

Data Preprocessing¶

The vast amount of real word data has to be preprocessed before you starting to analyze it.

Common preprocessing steps may include but are not limited to operations like:

  • Data cleaning
  • Deduplication
  • Handling missing values

Let's go through them in detail.

We'll use a dataset containing the participation of countries in the Winter- and Summer Olympics.

Data Cleaning¶

In [11]:
df = pd.read_csv("olympics.csv")
print(df.shape)
df

(150, 1)
Out[11]:
0;1;2;3;4;5;6;7;8;9;10;11;12;13;14;15
0 ;? Summer;01 !;02 !;03 !;Total;? Winter;01 !;0...
1 Afghanistan (AFG);13;0;0;2;;0;0;0;0;0;13;0;0;2;2
2 Algeria (ALG);12;5;2;8;15;3;0;0;0;0;15;5;2;8;15
3 Argentina (ARG);23;18;24;28;70;18;0;0;0;0;41;1...
4 Armenia (ARM);5;1;2;9;12;6;0;0;0;0;11;1;2;9;12
... ...
145 Independent Olympic Participants (IOP) ;1;0;1;...
146 Zambia (ZAM) ;12;0;1;1;2;0;0;0;0;0;12;0;1;1;2
147 Zimbabwe (ZIM) ;12;3;4;1;8;1;0;;0;0;13;3;4;1;8
148 Mixed team (ZZX) ;3;8;5;4;17;0;0;0;0;0;3;8;5;4;17
149 Totals;27;4809;4775;5130;14714;22;959;958;948;...

150 rows × 1 columns

As you can see, even though the file is a CSV-table (comma-separated-values), the file is not correctly loaded.

The reason is that instead of commas, the file's authors used semicolons as separator symbols.

Luckily, we can specify custom seperators in Pandas .read_csv function:

In [12]:
df = pd.read_csv("olympics.csv", sep=";")
df
Out[12]:
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
0 NaN ? Summer 01 ! 02 ! 03 ! Total ? Winter 01 ! 02 ! 03 ! Total ? Games 01 ! 02 ! 03 ! Combined total
1 Afghanistan (AFG) 13 0 0 2 NaN 0 0 0 0 0 13 0 0 2 2
2 Algeria (ALG) 12 5 2 8 15 3 0 0 0 0 15 5 2 8 15
3 Argentina (ARG) 23 18 24 28 70 18 0 0 0 0 41 18 24 28 70
4 Armenia (ARM) 5 1 2 9 12 6 0 0 0 0 11 1 2 9 12
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
145 Independent Olympic Participants (IOP) 1 0 1 2 3 0 0 0 0 0 1 0 1 2 3
146 Zambia (ZAM) 12 0 1 1 2 0 0 0 0 0 12 0 1 1 2
147 Zimbabwe (ZIM) 12 3 4 1 8 1 0 NaN 0 0 13 3 4 1 8
148 Mixed team (ZZX) 3 8 5 4 17 0 0 0 0 0 3 8 5 4 17
149 Totals 27 4809 4775 5130 14714 22 959 958 948 2865 49 5768 5733 6078 17579

150 rows × 16 columns

Now it looks much better, but is also apparent that the table's header (i.e., first row) wasn't correctly identified.

Let's fix this:

In [13]:
df = pd.read_csv("olympics.csv", sep=";", header=1)
df
Out[13]:
Unnamed: 0 ? Summer 01 ! 02 ! 03 ! Total ? Winter 01 !.1 02 !.1 03 !.1 Total.1 ? Games 01 !.2 02 !.2 03 !.2 Combined total
0 Afghanistan (AFG) 13 0 0 2 NaN 0 0.0 0.0 0.0 0 13 0 0.0 2 2
1 Algeria (ALG) 12 5 2 8 15.0 3 0.0 0.0 0.0 0 15 5 2.0 8 15
2 Argentina (ARG) 23 18 24 28 70.0 18 0.0 0.0 0.0 0 41 18 24.0 28 70
3 Armenia (ARM) 5 1 2 9 12.0 6 0.0 0.0 0.0 0 11 1 2.0 9 12
4 Australasia (ANZ) 2 3 4 5 12.0 0 0.0 0.0 0.0 0 2 3 4.0 5 12
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
144 Independent Olympic Participants (IOP) 1 0 1 2 3.0 0 0.0 0.0 0.0 0 1 0 1.0 2 3
145 Zambia (ZAM) 12 0 1 1 2.0 0 0.0 0.0 0.0 0 12 0 1.0 1 2
146 Zimbabwe (ZIM) 12 3 4 1 8.0 1 0.0 NaN 0.0 0 13 3 4.0 1 8
147 Mixed team (ZZX) 3 8 5 4 17.0 0 0.0 0.0 0.0 0 3 8 5.0 4 17
148 Totals 27 4809 4775 5130 14714.0 22 959.0 958.0 948.0 2865 49 5768 5733.0 6078 17579

149 rows × 16 columns

In [14]:
df.columns
Out[14]:
Index(['Unnamed: 0', '? Summer', '01 !', '02 !', '03 !', 'Total', '? Winter',
       '01 !.1', '02 !.1', '03 !.1', 'Total.1', '? Games', '01 !.2', '02 !.2',
       '03 !.2', 'Combined total'],
      dtype='object')

As you can see Pandas automatically added prefixes to some column names, because it requires them to be unique.

This further adds further blurriness to the already confusing names, so let's rename them:

In [15]:
column_rename_map = {
    '01 !': 'Gold-Summer',
    '01 !.1': 'Gold-Winter',
    '01 !.2': 'Gold-Total',
    '02 !': 'Silver-Summer',
    '02 !.1': 'Silver-Winter',
    '02 !.2': 'Silver-Total',
    '03 !': 'Bronze-Summer',
    '03 !.1': 'Bronze-Winter',
    '03 !.2': 'Bronze-Total',
    '? Games': 'Participation-Total',
    '? Summer': 'Participation-Summer',
    '? Winter': 'Participation-Winter',
    'Combined total': 'Combined-Medals',
    'Total': 'Combined-Medals-Summer',
    'Total.1': 'Combined-Medals-Winter',
    'Unnamed: 0': 'Country'
 }

df = df.rename(column_rename_map, axis=1)
df
Out[15]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals
0 Afghanistan (AFG) 13 0 0 2 NaN 0 0.0 0.0 0.0 0 13 0 0.0 2 2
1 Algeria (ALG) 12 5 2 8 15.0 3 0.0 0.0 0.0 0 15 5 2.0 8 15
2 Argentina (ARG) 23 18 24 28 70.0 18 0.0 0.0 0.0 0 41 18 24.0 28 70
3 Armenia (ARM) 5 1 2 9 12.0 6 0.0 0.0 0.0 0 11 1 2.0 9 12
4 Australasia (ANZ) 2 3 4 5 12.0 0 0.0 0.0 0.0 0 2 3 4.0 5 12
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
144 Independent Olympic Participants (IOP) 1 0 1 2 3.0 0 0.0 0.0 0.0 0 1 0 1.0 2 3
145 Zambia (ZAM) 12 0 1 1 2.0 0 0.0 0.0 0.0 0 12 0 1.0 1 2
146 Zimbabwe (ZIM) 12 3 4 1 8.0 1 0.0 NaN 0.0 0 13 3 4.0 1 8
147 Mixed team (ZZX) 3 8 5 4 17.0 0 0.0 0.0 0.0 0 3 8 5.0 4 17
148 Totals 27 4809 4775 5130 14714.0 22 959.0 958.0 948.0 2865 49 5768 5733.0 6078 17579

149 rows × 16 columns

Now, the columns content is much less opaque.

Let's look at the details:

The .rename-method receives a mapping from the old name to the new names as input, and applies this map along the given dimension.

axis=0 would rename the index (not useful for us because for now we just work with simple numerical indices).

axis=1 applies the renaming along the column dimension.

Also, not that the function returns a renamed copy of the DataFrame it was called on. You have to explicitly save it (e.g. save it in an old or new variable), to retain the changes to your data.

Atomizing values¶

Another important step while cleaning your data is to atomize your features, meaning that each column should only contain one single value.

For the numerical parts of our dataset this is already the case, but if we look at the Country columns we see, that it contains the country's fullname and its country code.

Let's disentangle them:

In [16]:
df.Country
Out[16]:
0                            Afghanistan (AFG)
1                                Algeria (ALG)
2                              Argentina (ARG)
3                                Armenia (ARM)
4                           Australasia (ANZ) 
                        ...                   
144    Independent Olympic Participants (IOP) 
145                              Zambia (ZAM) 
146                            Zimbabwe (ZIM) 
147                          Mixed team (ZZX) 
148                                     Totals
Name: Country, Length: 149, dtype: object

Because the Country column only contains strings, Pandas offers us a convenient set of string operations (e.g., the string-methods of Python's string you are already familiar with) that we can directly apply the values.

In [17]:
df["Code"] = df["Country"].str.findall(r"\(.+?\)").apply(lambda results: results[0].strip("()") if results else "")
df["Country"] = df["Country"].str.split(" ").apply(lambda parts: parts[0])
df
Out[17]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
0 Afghanistan 13 0 0 2 NaN 0 0.0 0.0 0.0 0 13 0 0.0 2 2 AFG
1 Algeria 12 5 2 8 15.0 3 0.0 0.0 0.0 0 15 5 2.0 8 15 ALG
2 Argentina 23 18 24 28 70.0 18 0.0 0.0 0.0 0 41 18 24.0 28 70 ARG
3 Armenia 5 1 2 9 12.0 6 0.0 0.0 0.0 0 11 1 2.0 9 12 ARM
4 Australasia 2 3 4 5 12.0 0 0.0 0.0 0.0 0 2 3 4.0 5 12 ANZ
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
144 Independent 1 0 1 2 3.0 0 0.0 0.0 0.0 0 1 0 1.0 2 3 IOP
145 Zambia 12 0 1 1 2.0 0 0.0 0.0 0.0 0 12 0 1.0 1 2 ZAM
146 Zimbabwe 12 3 4 1 8.0 1 0.0 NaN 0.0 0 13 3 4.0 1 8 ZIM
147 Mixed 3 8 5 4 17.0 0 0.0 0.0 0.0 0 3 8 5.0 4 17 ZZX
148 Totals 27 4809 4775 5130 14714.0 22 959.0 958.0 948.0 2865 49 5768 5733.0 6078 17579

149 rows × 17 columns

Let's breakdown what we just did here:

First, we created a new column called Country-Code.

To fill this column we used a regular expression to find all brackets and their content in the Country column.

Because the findall method returns a list of what is found, we had to extract those codes from these lists using the apply-method of the column.

apply receives an function as an argument, applies it to each entry in the series and returns a new series containing all the results.

We defined an anonymous lambda function here, which either returns the first entry of a list (and removes the round brackets) or just an empty string if the list is empty.

We needed this special case empty string case because the last entry in our dataset is a row with the total values and thus does not have a country code.


Secondly, we updated the original Country column by splitting it (based on spaces) and only taking the first value of these splits (= the full country names) as new values, thereby effectively removing the country codes from this column.

Deduplication¶

Now, that our dataset is correctly loaded and suitable named, let's check if the content itself is clean.

First, we check if there any duplications and if yes, drop them.

The identifying column of our dataset, is Country, because the number of participations, or medals might obviously contain duplicate values.

Note¶

Whenever we remove entries from our data, it's always as good advice to manually check what we are removing!

In [18]:
df.shape, df.drop_duplicates(subset=["Code"]).shape
Out[18]:
((149, 17), (148, 17))

We see that there is one country appearing twice in our dataset, let's search for it.

In [19]:
df["Code"].value_counts()
Out[19]:
UKR    2
AFG    1
POR    1
PAK    1
PAN    1
      ..
GBR    1
GRE    1
GRN    1
GUA    1
       1
Name: Code, Length: 148, dtype: int64

Ukraine appears twice so, now let's see if the values are the same too!

In [20]:
df.query("Code == 'UKR'")
Out[20]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
134 Ukraine 5 33 27 55 115.0 6 2.0 1.0 4.0 7 11 35 28.0 59 122 UKR
135 Ukraine 5 33 27 55 115.0 6 2.0 1.0 4.0 7 11 35 28.0 59 122 UKR

Luckily this is the case, so we can just keep the first entry and remove the latter. (Which is exactly what the .remove_duplicates-method does.)

In [21]:
df = df.drop_duplicates("Code")
df.query("Code == 'UKR'")
Out[21]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
134 Ukraine 5 33 27 55 115.0 6 2.0 1.0 4.0 7 11 35 28.0 59 122 UKR

Handling missing values¶

Often, your data will be incomplete.

Those cases of missing values have to be handled carefully to both ensure the integrity of the data and also prevent just throwing away swaths of the dataset.

In general and only with few valid exceptions, only values that can be correctly deduced from other data can be reconstructed.

Entries that exhibit other values missing, should be removed, or manually corrected.

In our case, we can relatively safely repair missing values in the combined-* field, by using the other fields to recompute them.

Also, we can potentially fill up the other fields under certain conditions.

But first, let's check how many entries lack entries:

Pandas uses a special value np.NaN (i.e., Not a Number) to indicate missing values like Python uses None. In contrast toNone` values can be used in computations without provoking an exception.

But the result of any computation of the form values <operator> np.nan will always result in another nan value.

So whenever you receive a nan values as result, go back to your data preprocessing stage and check if you handled all missing values.

In [22]:
import numpy as np

values = np.arange(5)
values = values + np.nan
values
Out[22]:
array([nan, nan, nan, nan, nan])

Now let's check how many incomplete rows we have

In [23]:
df.shape[0] - df.dropna().shape[0]
Out[23]:
4

If drop_na is called without any additinal arguments it just removes any row in the DataFrame that contains one or more nan-values.

As we see now we have four rows with incomplete entries.

Let's dig a little bit deeper and look at those cases:

In [24]:
df[df.isna().any(axis=1)]
Out[24]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
0 Afghanistan 13 0 0 2 NaN 0 0.0 0.0 0.0 0 13 0 0.0 2 2 AFG
110 Senegal 13 0 1 0 1.0 5 0.0 0.0 NaN 0 18 0 1.0 0 1 SEN
128 Togo 9 0 0 1 1.0 1 NaN 0.0 0.0 0 10 0 NaN 1 1 TOG
146 Zimbabwe 12 3 4 1 8.0 1 0.0 NaN 0.0 0 13 3 4.0 1 8 ZIM

To find those rows we used two techniques that underline Panda's Numpy-backend.

The isna-method returns a mask of the dataset where each cell that contains nan is marked with True and all other cells with False.

In [25]:
df.isna()
Out[25]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
0 False False False False False True False False False False False False False False False False False
1 False False False False False False False False False False False False False False False False False
2 False False False False False False False False False False False False False False False False False
3 False False False False False False False False False False False False False False False False False
4 False False False False False False False False False False False False False False False False False
... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ... ...
144 False False False False False False False False False False False False False False False False False
145 False False False False False False False False False False False False False False False False False
146 False False False False False False False False True False False False False False False False False
147 False False False False False False False False False False False False False False False False False
148 False False False False False False False False False False False False False False False False False

148 rows × 17 columns

To be able to find those rows who contain at least one nan value we can use the any method to reduce those values along the row dimension (axis=1).

The any-method simply checks if any of the values along this dimension is True

In [26]:
df.isna().any(axis=1), df.isna().any(axis=1).shape
Out[26]:
(0       True
 1      False
 2      False
 3      False
 4      False
        ...  
 144    False
 145    False
 146     True
 147    False
 148    False
 Length: 148, dtype: bool,
 (148,))

Finally, we can use this boolean array as row mask over our original dataset to only select those row, which contain at leat one nan value

In [27]:
df[df.isna().any(axis=1)]
Out[27]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
0 Afghanistan 13 0 0 2 NaN 0 0.0 0.0 0.0 0 13 0 0.0 2 2 AFG
110 Senegal 13 0 1 0 1.0 5 0.0 0.0 NaN 0 18 0 1.0 0 1 SEN
128 Togo 9 0 0 1 1.0 1 NaN 0.0 0.0 0 10 0 NaN 1 1 TOG
146 Zimbabwe 12 3 4 1 8.0 1 0.0 NaN 0.0 0 13 3 4.0 1 8 ZIM

If we look carefully at the data, we see that we can luckily infer all the missing values using our domain knowledge and contextual data.

To do so, we have to directly index the nan cell to fill in the repaired values.

We can do this, by using a row- and column wise index

In [28]:
df.at[0, "Combined-Medals-Summer"] = 2
df.at[110, "Bronze-Winter"] = 0
df.at[128, "Gold-Winter"] = 0
df.at[128, "Silver-Total"] = 0
df.at[146, "Silver-Winter"] = 0
df[df.isna().any(axis=1)]
Out[28]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code

Last step: Eliminating implausible entries¶

Sometimes we can leverage our domain knowledge and also search for implausible entries.

In our example, there are at least two possible ways to look for implausible records:

  • Looking for countries that do not exist
    • We might eliminate countries that do not exists.
  • Checking the total number of participations
    • Countries that never participated in any games, can also be evicted safely

Let's start first by examining the countries:

Of course, we know that country code ought are always made up of three characters, so let's see if there any outliers:

In [29]:
df.query("Code.str.len() != 3")
Out[29]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code
109 Atlantis 1 1 1 1 1.0 1 1.0 1.0 1.0 1 1 1 1.0 1 1 ATLA
148 Totals 27 4809 4775 5130 14714.0 22 959.0 958.0 948.0 2865 49 5768 5733.0 6078 17579

The mythic city-state of Atlantis is a product of imagination, and should be removed!

Once again, we employ the drop-method to do this:

In [30]:
df = df.drop(109, axis="index")
df.query("Code == 'ATLA'")
Out[30]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code

Now let's look if there are any countries that did not participate in the games:

In [31]:
df[df.sum(axis=1, numeric_only=True) == 0]
Out[31]:
Country Participation-Summer Gold-Summer Silver-Summer Bronze-Summer Combined-Medals-Summer Participation-Winter Gold-Winter Silver-Winter Bronze-Winter Combined-Medals-Winter Participation-Total Gold-Total Silver-Total Bronze-Total Combined-Medals Code

Luckily there is none, so our dataset should now be somewhat clean.

Are we ever finished?¶

The data cleaning preparation process in an iterative one, often while analyzing your data, you'll find additional issues or errors introduced by your operations and have to go back to your cleaning stage.

So make sure you to not overwrite the original data and safe your cleaning code!

Normalization¶

One thing that we left out in this example is data normalization.

This is because this term can have multiple meanings:

  • Numerical normalization: Ensure that your data has beneficial numerical properties (e.g., same value range, normed statistical properties)
  • Normalized data model: Your data follows the same data model, even if the data itself stems from multiple sources
  • Text normalization
    • Normalized encoding: If you operate on text data it ideally follows the same encoding.
    • "Linguistic" normalization: Certain word forms might be normalized to one single common root, like [(walking, walked, walk) => walk] (e.g., stemming, or preferably lemmatisation)

We'll cover numerical normalization and text normalization in later sessions.

Data acquisition¶

Often and for various reasons, we can't rely on prebuilt datasets and have to create them ourselves.

To check if there is no dataset, that fits our needs, there is a specialized search engine by google, tailored to find publicly available datasets:

Dataset Search

Over the past few years the Huggingface Dataset Hub also has become a decent source for various text datasets:

Huggingface Dataset Hub

Depending on your topic of interest there are also other options at your disposal:

  • Scraping the web, or certain platforms (Twitter, Wikipedia, ...) for collecting data
  • Specialized data repositories
  • Creating paid surveys to make people create data for you

Example: Web scraping - Downloading articles from wikipedia¶

For prominent platforms, there is a high chance that you can already rely on pre-built software to gather some data.

In this example, we'll scrape some articles from wikipedia using the wikipedia Python package

Installation¶

pip install wikipedia

Also, we need two additional libraries:

  • tqdm: Gives us a fancy progress bar
  • requests: Allows us to sent any type of HTTP-request anywhere
pip install tqdm requests
In [32]:
import wikipedia
import requests

page = wikipedia.page("Python (programming language)")

page.revision_id
print(page.url)
print(page.title)
print(page.content[:1000], "...\n\n")
print(page.categories)

https://en.wikipedia.org/wiki/Python_(programming_language)
Python (programming language)
Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation.Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including structured (particularly procedural), object-oriented and functional programming. It is often described as a "batteries included" language due to its comprehensive standard library.Guido van Rossum began working on Python in the late 1980s as a successor to the ABC programming language and first released it in 1991 as Python 0.9.0. Python 2.0 was released in 2000 and introduced new features such as list comprehensions, cycle-detecting garbage collection, reference counting, and Unicode support. Python 3.0, released in 2008, was a major revision that is not completely backward-compatible with earlier versions. Python 2 was discontinued with version 2.7.18 in 2020.Python consistently ranks as one of the most popular programming languag ...


['All articles containing potentially dated statements', 'Articles containing potentially dated statements from November 2022', 'Articles containing potentially dated statements from October 2021', 'Articles with BNF identifiers', 'Articles with FAST identifiers', 'Articles with GND identifiers', 'Articles with J9U identifiers', 'Articles with LCCN identifiers', 'Articles with NKC identifiers', 'Articles with SUDOC identifiers', 'Articles with example Python (programming language) code', 'Articles with short description', 'CS1 maint: archived copy as title', 'Class-based programming languages', 'Computer science in the Netherlands', 'Concurrent programming languages', 'Cross-platform free software', 'Cross-platform software', 'Dutch inventions', 'Dynamically typed programming languages', 'Educational programming languages', 'Good articles', 'High-level programming languages', 'Information technology in the Netherlands', 'Multi-paradigm programming languages', 'Notebook interface', 'Object-oriented programming languages', 'Pages using Sister project links with hidden wikidata', 'Pages using Sister project links with wikidata namespace mismatch', 'Pattern matching programming languages', 'Programming languages', 'Programming languages created in 1991', 'Python (programming language)', 'Scripting languages', 'Short description matches Wikidata', 'Text-oriented programming languages', 'Use dmy dates from November 2021', 'Webarchive template wayback links']
In [33]:
import wikipedia
import requests
import pandas as pd
from tqdm.auto import tqdm

HEADERS = {'User-Agent': 'DataScienceBot/0.0 (https://example.org/DataScienceBot/; DataScienceBot@example.org)'}

def build_wiki_dataset(query: str, max_results: int = 100):
    
    data = []
    progress_bar = tqdm(wikipedia.search(query, results=max_results))
    for result in progress_bar:
        progress_bar.set_description(f"Downloading page {result}")
        try:
            page = wikipedia.page(result)
            page_name = page.url.split("/")[-1]
            edit_request = requests.get(
                f"https://en.wikipedia.org/w/rest.php/v1/page/{page_name}/history/counts/edits?",
                headers=HEADERS
            )
            num_edits = edit_request.json()["count"]
            num_views_request = requests.get(
                f"https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/all-agents/{page_name}/daily/2000010100/2022112700",
                headers=HEADERS
            )
            num_views = sum([entry["views"] for entry in num_views_request.json()["items"]])
        
        except Exception:
            # We discard the page and move on
            continue

        page_data = {
            "url": page.url,
            "title": page.title,
            "summary": page.summary,
            "text": page.content,
            "num_views": num_views,
            "num_edits": num_edits,
            "categories": ", ".join(page.categories),
            "revision_id": page.revision_id
        }
        
        data.append(page_data)
    
    df = pd.DataFrame.from_records(data)
    return df
In [34]:
df = build_wiki_dataset("History of ", max_results=50)

  0%|          | 0/50 [00:00<?, ?it/s]
/opt/homebrew/Caskroom/miniforge/base/envs/python_intro/lib/python3.9/site-packages/wikipedia/wikipedia.py:389: GuessedAtParserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 389 of the file /opt/homebrew/Caskroom/miniforge/base/envs/python_intro/lib/python3.9/site-packages/wikipedia/wikipedia.py. To get rid of this warning, pass the additional argument 'features="lxml"' to the BeautifulSoup constructor.

  lis = BeautifulSoup(html).find_all('li')
In [35]:
df
Out[35]:
url title summary text num_views num_edits categories revision_id
0 https://en.wikipedia.org/wiki/History History History (from Ancient Greek ἱστορία (historía... History (from Ancient Greek ἱστορία (historía... 8628272 6617 All Wikipedia articles written in American Eng... 1124009190
1 https://en.wikipedia.org/wiki/History_of_Europe History of Europe The history of Europe is traditionally divided... The history of Europe is traditionally divided... 2764829 4533 All articles needing additional references, Al... 1123723906
2 https://en.wikipedia.org/wiki/History_of_China History of China The earliest known written records of the hist... The earliest known written records of the hist... 6608376 7269 All articles containing potentially dated stat... 1124025086
3 https://en.wikipedia.org/wiki/History_of_Islam History of Islam The history of Islam concerns the political, s... The history of Islam concerns the political, s... 5703155 6107 All articles with unsourced statements, Articl... 1123323843
4 https://en.wikipedia.org/wiki/History_of_Japan History of Japan The first human inhabitants of the Japanese ar... The first human inhabitants of the Japanese ar... 5031470 7603 All Wikipedia articles written in American Eng... 1123804351
5 https://en.wikipedia.org/wiki/History_of_telev... History of television The concept of television was the work of many... The concept of television was the work of many... 5074930 3906 All articles to be expanded, All articles with... 1123620788
6 https://en.wikipedia.org/wiki/History_of_Chris... History of Christianity The history of Christianity concerns the Chris... The history of Christianity concerns the Chris... 3762801 5801 All articles with bare URLs for citations, All... 1123072552
7 https://en.wikipedia.org/wiki/History_of_religion History of religion The history of religion refers to the written ... The history of religion refers to the written ... 1075565 654 All articles with specifically marked weasel-w... 1123342929
8 https://en.wikipedia.org/wiki/History_of_Poland History of Poland The history of Poland spans over a thousand ye... The history of Poland spans over a thousand ye... 1542336 8199 All articles lacking reliable references, Arti... 1115808344
9 https://en.wikipedia.org/wiki/End_of_history End of history The end of history is a political and philosop... The end of history is a political and philosop... 499923 105 1860s neologisms, All articles to be expanded,... 1123918495
10 https://en.wikipedia.org/wiki/History_of_Russia History of Russia The history of Russia begins with the historie... The history of Russia begins with the historie... 3289180 4944 All articles with minor POV problems, All arti... 1124023255
11 https://en.wikipedia.org/wiki/History_of_banking History of banking The history of banking began with the first pr... The history of banking began with the first pr... 2333050 1619 All articles with unsourced statements, All ar... 1116409705
12 https://en.wikipedia.org/wiki/History_of_Rome History of Rome The history of Rome includes the history of th... The history of Rome includes the history of th... 2079609 2796 All articles containing potentially dated stat... 1123947372
13 https://en.wikipedia.org/wiki/History_of_Germany History of Germany The concept of Germany as a distinct region in... The concept of Germany as a distinct region in... 2741092 5078 All articles lacking reliable references, Arti... 1124396390
14 https://en.wikipedia.org/wiki/History_of_Spain History of Spain The history of Spain dates to the pre-Roman pe... The history of Spain dates to the pre-Roman pe... 2355382 4119 All accuracy disputes, All articles with dead ... 1124373367
15 https://en.wikipedia.org/wiki/History_of_Austr... History of Australia The history of Australia is the story of the l... The history of Australia is the story of the l... 3782171 5856 AC with 0 elements, All Wikipedia articles wri... 1123124583
16 https://en.wikipedia.org/wiki/History_of_Arda History of Arda In J. R. R. Tolkien's legendarium, the history... In J. R. R. Tolkien's legendarium, the history... 605422 610 Articles with short description, Fictional his... 1123256525
17 https://en.wikipedia.org/wiki/History_of_mathe... History of mathematics The history of mathematics deals with the orig... The history of mathematics deals with the orig... 3315598 4317 All articles needing additional references, Al... 1122788585
18 https://en.wikipedia.org/wiki/History_of_Bangl... History of Bangladesh Civilisational history of Bangladesh previousl... Civilisational history of Bangladesh previousl... 1835490 2666 All articles lacking reliable references, All ... 1123069979
19 https://en.wikipedia.org/wiki/History_of_Hinduism History of Hinduism The history of Hinduism covers a wide variety ... The history of Hinduism covers a wide variety ... 1797592 2935 All articles lacking reliable references, All ... 1124512840
20 https://en.wikipedia.org/wiki/History_of_Ireland History of Ireland The first evidence of human presence in Irelan... The first evidence of human presence in Irelan... 2701679 3338 All accuracy disputes, All articles with style... 1123697361
21 https://en.wikipedia.org/wiki/History_of_agric... History of agriculture Agriculture began independently in different p... Agriculture began independently in different p... 2516191 1977 All articles with dead external links, Article... 1123034506
22 https://en.wikipedia.org/wiki/History_of_pharmacy History of pharmacy The history of pharmacy as an independent scie... The history of pharmacy as an independent scie... 450258 174 All articles with unsourced statements, Articl... 1113261032
23 https://en.wikipedia.org/wiki/History_of_democ... History of democracy A democracy is a political system, or a system... A democracy is a political system, or a system... 1972822 2383 All Wikipedia articles needing clarification, ... 1122481158
24 https://en.wikipedia.org/wiki/History_of_pizza History of pizza The history of pizza begins in antiquity, as v... The history of pizza begins in antiquity, as v... 4086441 4146 All articles with failed verification, All art... 1124192347
25 https://en.wikipedia.org/wiki/History_of_syphilis History of syphilis The first recorded outbreak of syphilis in Eur... The first recorded outbreak of syphilis in Eur... 1548048 479 All articles with dead external links, All art... 1123246612
26 https://en.wikipedia.org/wiki/History_of_Italy History of Italy The history of Italy covers the ancient period... The history of Italy covers the ancient period... 1824793 3518 All articles with incomplete citations, Articl... 1124364753
27 https://en.wikipedia.org/wiki/History_of_medicine History of medicine The history of medicine is both a study of med... The history of medicine is both a study of med... 2008806 2819 All articles containing potentially dated stat... 1123053788
28 https://en.wikipedia.org/wiki/History_of_calen... History of calendars The history of calendars, that is, of people c... The history of calendars, that is, of people c... 1411516 680 All articles with incomplete citations, All ar... 1123446945
29 https://en.wikipedia.org/wiki/History_of_Facebook History of Facebook Facebook is a social networking service origin... Facebook is a social networking service origin... 4916523 1735 All Wikipedia articles in need of updating, Al... 1124469003
30 https://en.wikipedia.org/wiki/History_of_cricket History of cricket The sport of cricket has a known history begin... The sport of cricket has a known history begin... 3234448 2040 All articles needing additional references, Al... 1123571727
31 https://en.wikipedia.org/wiki/History_of_Earth History of Earth The history of Earth concerns the development ... The history of Earth concerns the development ... 4602476 4317 AC with 0 elements, All articles with unsource... 1124371678
32 https://en.wikipedia.org/wiki/History_of_the_U... History of the United States The history of the lands that became the Unite... The history of the lands that became the Unite... 6916785 11165 All Wikipedia articles needing clarification, ... 1124535526
33 https://en.wikipedia.org/wiki/History_of_Afgha... History of Afghanistan The history of Afghanistan as a state began in... The history of Afghanistan as a state began in... 2155957 2971 All articles with dead external links, Article... 1121424232
34 https://en.wikipedia.org/wiki/History_of_Korea History of Korea The Lower Paleolithic era in the Korean Penins... The Lower Paleolithic era in the Korean Penins... 2743246 3470 Accuracy disputes from December 2017, All accu... 1124522331
35 https://en.wikipedia.org/wiki/History_of_baske... History of basketball Basketball began with its invention in 1891 in... Basketball began with its invention in 1891 in... 4434485 3010 All articles to be expanded, All articles with... 1117994066
36 https://en.wikipedia.org/wiki/History_of_commu... History of communication The history of communication technologies (med... The history of communication technologies (med... 1620064 957 All articles needing additional references, Al... 1123372214
37 https://en.wikipedia.org/wiki/History_of_marke... History of marketing The study of the history of marketing, as a di... The study of the history of marketing, as a di... 931211 785 Articles with short description, EngvarB from ... 1117474637
38 https://en.wikipedia.org/wiki/History_of_Lambo... History of Lamborghini Automobili Lamborghini S.p.A. is an Italian br... Automobili Lamborghini S.p.A. is an Italian br... 960020 271 All articles with unsourced statements, Articl... 1108125867
39 https://en.wikipedia.org/wiki/History_of_Pakistan History of Pakistan The history of the lands of Pakistan for the p... The history of the lands of Pakistan for the p... 1806158 5054 All Wikipedia articles written in Pakistani En... 1123802420
40 https://en.wikipedia.org/wiki/History_of_cotton History of cotton The history of cotton can be traced to domesti... The history of cotton can be traced to domesti... 716500 343 All articles with unsourced statements, Articl... 1118655355
41 https://en.wikipedia.org/wiki/History_of_cheese History of cheese The production of cheese predates recorded his... The production of cheese predates recorded his... 653140 632 All articles with vague or ambiguous time, Art... 1123553575
42 https://en.wikipedia.org/wiki/History_of_Belgium History of Belgium The history of Belgium extends before the foun... The history of Belgium extends before the foun... 1119414 2301 All articles covered by WikiProject Wikify, Al... 1124232962
43 https://en.wikipedia.org/wiki/History_of_corsets History of corsets The corset has been an indispensable supportiv... The corset has been an indispensable supportiv... 706954 553 16th-century fashion, 17th-century fashion, 18... 1083739897
44 https://en.wikipedia.org/wiki/History_of_Ghana History of Ghana The Republic of Ghana is named after the medie... The Republic of Ghana is named after the medie... 1305934 1479 All Wikipedia articles written in Ghanaian Eng... 1123589131
In [36]:
df.sort_values("num_views")
Out[36]:
url title summary text num_views num_edits categories revision_id
22 https://en.wikipedia.org/wiki/History_of_pharmacy History of pharmacy The history of pharmacy as an independent scie... The history of pharmacy as an independent scie... 450258 174 All articles with unsourced statements, Articl... 1113261032
9 https://en.wikipedia.org/wiki/End_of_history End of history The end of history is a political and philosop... The end of history is a political and philosop... 499923 105 1860s neologisms, All articles to be expanded,... 1123918495
16 https://en.wikipedia.org/wiki/History_of_Arda History of Arda In J. R. R. Tolkien's legendarium, the history... In J. R. R. Tolkien's legendarium, the history... 605422 610 Articles with short description, Fictional his... 1123256525
41 https://en.wikipedia.org/wiki/History_of_cheese History of cheese The production of cheese predates recorded his... The production of cheese predates recorded his... 653140 632 All articles with vague or ambiguous time, Art... 1123553575
43 https://en.wikipedia.org/wiki/History_of_corsets History of corsets The corset has been an indispensable supportiv... The corset has been an indispensable supportiv... 706954 553 16th-century fashion, 17th-century fashion, 18... 1083739897
40 https://en.wikipedia.org/wiki/History_of_cotton History of cotton The history of cotton can be traced to domesti... The history of cotton can be traced to domesti... 716500 343 All articles with unsourced statements, Articl... 1118655355
37 https://en.wikipedia.org/wiki/History_of_marke... History of marketing The study of the history of marketing, as a di... The study of the history of marketing, as a di... 931211 785 Articles with short description, EngvarB from ... 1117474637
38 https://en.wikipedia.org/wiki/History_of_Lambo... History of Lamborghini Automobili Lamborghini S.p.A. is an Italian br... Automobili Lamborghini S.p.A. is an Italian br... 960020 271 All articles with unsourced statements, Articl... 1108125867
7 https://en.wikipedia.org/wiki/History_of_religion History of religion The history of religion refers to the written ... The history of religion refers to the written ... 1075565 654 All articles with specifically marked weasel-w... 1123342929
42 https://en.wikipedia.org/wiki/History_of_Belgium History of Belgium The history of Belgium extends before the foun... The history of Belgium extends before the foun... 1119414 2301 All articles covered by WikiProject Wikify, Al... 1124232962
44 https://en.wikipedia.org/wiki/History_of_Ghana History of Ghana The Republic of Ghana is named after the medie... The Republic of Ghana is named after the medie... 1305934 1479 All Wikipedia articles written in Ghanaian Eng... 1123589131
28 https://en.wikipedia.org/wiki/History_of_calen... History of calendars The history of calendars, that is, of people c... The history of calendars, that is, of people c... 1411516 680 All articles with incomplete citations, All ar... 1123446945
8 https://en.wikipedia.org/wiki/History_of_Poland History of Poland The history of Poland spans over a thousand ye... The history of Poland spans over a thousand ye... 1542336 8199 All articles lacking reliable references, Arti... 1115808344
25 https://en.wikipedia.org/wiki/History_of_syphilis History of syphilis The first recorded outbreak of syphilis in Eur... The first recorded outbreak of syphilis in Eur... 1548048 479 All articles with dead external links, All art... 1123246612
36 https://en.wikipedia.org/wiki/History_of_commu... History of communication The history of communication technologies (med... The history of communication technologies (med... 1620064 957 All articles needing additional references, Al... 1123372214
19 https://en.wikipedia.org/wiki/History_of_Hinduism History of Hinduism The history of Hinduism covers a wide variety ... The history of Hinduism covers a wide variety ... 1797592 2935 All articles lacking reliable references, All ... 1124512840
39 https://en.wikipedia.org/wiki/History_of_Pakistan History of Pakistan The history of the lands of Pakistan for the p... The history of the lands of Pakistan for the p... 1806158 5054 All Wikipedia articles written in Pakistani En... 1123802420
26 https://en.wikipedia.org/wiki/History_of_Italy History of Italy The history of Italy covers the ancient period... The history of Italy covers the ancient period... 1824793 3518 All articles with incomplete citations, Articl... 1124364753
18 https://en.wikipedia.org/wiki/History_of_Bangl... History of Bangladesh Civilisational history of Bangladesh previousl... Civilisational history of Bangladesh previousl... 1835490 2666 All articles lacking reliable references, All ... 1123069979
23 https://en.wikipedia.org/wiki/History_of_democ... History of democracy A democracy is a political system, or a system... A democracy is a political system, or a system... 1972822 2383 All Wikipedia articles needing clarification, ... 1122481158
27 https://en.wikipedia.org/wiki/History_of_medicine History of medicine The history of medicine is both a study of med... The history of medicine is both a study of med... 2008806 2819 All articles containing potentially dated stat... 1123053788
12 https://en.wikipedia.org/wiki/History_of_Rome History of Rome The history of Rome includes the history of th... The history of Rome includes the history of th... 2079609 2796 All articles containing potentially dated stat... 1123947372
33 https://en.wikipedia.org/wiki/History_of_Afgha... History of Afghanistan The history of Afghanistan as a state began in... The history of Afghanistan as a state began in... 2155957 2971 All articles with dead external links, Article... 1121424232
11 https://en.wikipedia.org/wiki/History_of_banking History of banking The history of banking began with the first pr... The history of banking began with the first pr... 2333050 1619 All articles with unsourced statements, All ar... 1116409705
14 https://en.wikipedia.org/wiki/History_of_Spain History of Spain The history of Spain dates to the pre-Roman pe... The history of Spain dates to the pre-Roman pe... 2355382 4119 All accuracy disputes, All articles with dead ... 1124373367
21 https://en.wikipedia.org/wiki/History_of_agric... History of agriculture Agriculture began independently in different p... Agriculture began independently in different p... 2516191 1977 All articles with dead external links, Article... 1123034506
20 https://en.wikipedia.org/wiki/History_of_Ireland History of Ireland The first evidence of human presence in Irelan... The first evidence of human presence in Irelan... 2701679 3338 All accuracy disputes, All articles with style... 1123697361
13 https://en.wikipedia.org/wiki/History_of_Germany History of Germany The concept of Germany as a distinct region in... The concept of Germany as a distinct region in... 2741092 5078 All articles lacking reliable references, Arti... 1124396390
34 https://en.wikipedia.org/wiki/History_of_Korea History of Korea The Lower Paleolithic era in the Korean Penins... The Lower Paleolithic era in the Korean Penins... 2743246 3470 Accuracy disputes from December 2017, All accu... 1124522331
1 https://en.wikipedia.org/wiki/History_of_Europe History of Europe The history of Europe is traditionally divided... The history of Europe is traditionally divided... 2764829 4533 All articles needing additional references, Al... 1123723906
30 https://en.wikipedia.org/wiki/History_of_cricket History of cricket The sport of cricket has a known history begin... The sport of cricket has a known history begin... 3234448 2040 All articles needing additional references, Al... 1123571727
10 https://en.wikipedia.org/wiki/History_of_Russia History of Russia The history of Russia begins with the historie... The history of Russia begins with the historie... 3289180 4944 All articles with minor POV problems, All arti... 1124023255
17 https://en.wikipedia.org/wiki/History_of_mathe... History of mathematics The history of mathematics deals with the orig... The history of mathematics deals with the orig... 3315598 4317 All articles needing additional references, Al... 1122788585
6 https://en.wikipedia.org/wiki/History_of_Chris... History of Christianity The history of Christianity concerns the Chris... The history of Christianity concerns the Chris... 3762801 5801 All articles with bare URLs for citations, All... 1123072552
15 https://en.wikipedia.org/wiki/History_of_Austr... History of Australia The history of Australia is the story of the l... The history of Australia is the story of the l... 3782171 5856 AC with 0 elements, All Wikipedia articles wri... 1123124583
24 https://en.wikipedia.org/wiki/History_of_pizza History of pizza The history of pizza begins in antiquity, as v... The history of pizza begins in antiquity, as v... 4086441 4146 All articles with failed verification, All art... 1124192347
35 https://en.wikipedia.org/wiki/History_of_baske... History of basketball Basketball began with its invention in 1891 in... Basketball began with its invention in 1891 in... 4434485 3010 All articles to be expanded, All articles with... 1117994066
31 https://en.wikipedia.org/wiki/History_of_Earth History of Earth The history of Earth concerns the development ... The history of Earth concerns the development ... 4602476 4317 AC with 0 elements, All articles with unsource... 1124371678
29 https://en.wikipedia.org/wiki/History_of_Facebook History of Facebook Facebook is a social networking service origin... Facebook is a social networking service origin... 4916523 1735 All Wikipedia articles in need of updating, Al... 1124469003
4 https://en.wikipedia.org/wiki/History_of_Japan History of Japan The first human inhabitants of the Japanese ar... The first human inhabitants of the Japanese ar... 5031470 7603 All Wikipedia articles written in American Eng... 1123804351
5 https://en.wikipedia.org/wiki/History_of_telev... History of television The concept of television was the work of many... The concept of television was the work of many... 5074930 3906 All articles to be expanded, All articles with... 1123620788
3 https://en.wikipedia.org/wiki/History_of_Islam History of Islam The history of Islam concerns the political, s... The history of Islam concerns the political, s... 5703155 6107 All articles with unsourced statements, Articl... 1123323843
2 https://en.wikipedia.org/wiki/History_of_China History of China The earliest known written records of the hist... The earliest known written records of the hist... 6608376 7269 All articles containing potentially dated stat... 1124025086
32 https://en.wikipedia.org/wiki/History_of_the_U... History of the United States The history of the lands that became the Unite... The history of the lands that became the Unite... 6916785 11165 All Wikipedia articles needing clarification, ... 1124535526
0 https://en.wikipedia.org/wiki/History History History (from Ancient Greek ἱστορία (historía... History (from Ancient Greek ἱστορία (historía... 8628272 6617 All Wikipedia articles written in American Eng... 1124009190
In [37]:
df.sort_values("num_edits")
Out[37]:
url title summary text num_views num_edits categories revision_id
9 https://en.wikipedia.org/wiki/End_of_history End of history The end of history is a political and philosop... The end of history is a political and philosop... 499923 105 1860s neologisms, All articles to be expanded,... 1123918495
22 https://en.wikipedia.org/wiki/History_of_pharmacy History of pharmacy The history of pharmacy as an independent scie... The history of pharmacy as an independent scie... 450258 174 All articles with unsourced statements, Articl... 1113261032
38 https://en.wikipedia.org/wiki/History_of_Lambo... History of Lamborghini Automobili Lamborghini S.p.A. is an Italian br... Automobili Lamborghini S.p.A. is an Italian br... 960020 271 All articles with unsourced statements, Articl... 1108125867
40 https://en.wikipedia.org/wiki/History_of_cotton History of cotton The history of cotton can be traced to domesti... The history of cotton can be traced to domesti... 716500 343 All articles with unsourced statements, Articl... 1118655355
25 https://en.wikipedia.org/wiki/History_of_syphilis History of syphilis The first recorded outbreak of syphilis in Eur... The first recorded outbreak of syphilis in Eur... 1548048 479 All articles with dead external links, All art... 1123246612
43 https://en.wikipedia.org/wiki/History_of_corsets History of corsets The corset has been an indispensable supportiv... The corset has been an indispensable supportiv... 706954 553 16th-century fashion, 17th-century fashion, 18... 1083739897
16 https://en.wikipedia.org/wiki/History_of_Arda History of Arda In J. R. R. Tolkien's legendarium, the history... In J. R. R. Tolkien's legendarium, the history... 605422 610 Articles with short description, Fictional his... 1123256525
41 https://en.wikipedia.org/wiki/History_of_cheese History of cheese The production of cheese predates recorded his... The production of cheese predates recorded his... 653140 632 All articles with vague or ambiguous time, Art... 1123553575
7 https://en.wikipedia.org/wiki/History_of_religion History of religion The history of religion refers to the written ... The history of religion refers to the written ... 1075565 654 All articles with specifically marked weasel-w... 1123342929
28 https://en.wikipedia.org/wiki/History_of_calen... History of calendars The history of calendars, that is, of people c... The history of calendars, that is, of people c... 1411516 680 All articles with incomplete citations, All ar... 1123446945
37 https://en.wikipedia.org/wiki/History_of_marke... History of marketing The study of the history of marketing, as a di... The study of the history of marketing, as a di... 931211 785 Articles with short description, EngvarB from ... 1117474637
36 https://en.wikipedia.org/wiki/History_of_commu... History of communication The history of communication technologies (med... The history of communication technologies (med... 1620064 957 All articles needing additional references, Al... 1123372214
44 https://en.wikipedia.org/wiki/History_of_Ghana History of Ghana The Republic of Ghana is named after the medie... The Republic of Ghana is named after the medie... 1305934 1479 All Wikipedia articles written in Ghanaian Eng... 1123589131
11 https://en.wikipedia.org/wiki/History_of_banking History of banking The history of banking began with the first pr... The history of banking began with the first pr... 2333050 1619 All articles with unsourced statements, All ar... 1116409705
29 https://en.wikipedia.org/wiki/History_of_Facebook History of Facebook Facebook is a social networking service origin... Facebook is a social networking service origin... 4916523 1735 All Wikipedia articles in need of updating, Al... 1124469003
21 https://en.wikipedia.org/wiki/History_of_agric... History of agriculture Agriculture began independently in different p... Agriculture began independently in different p... 2516191 1977 All articles with dead external links, Article... 1123034506
30 https://en.wikipedia.org/wiki/History_of_cricket History of cricket The sport of cricket has a known history begin... The sport of cricket has a known history begin... 3234448 2040 All articles needing additional references, Al... 1123571727
42 https://en.wikipedia.org/wiki/History_of_Belgium History of Belgium The history of Belgium extends before the foun... The history of Belgium extends before the foun... 1119414 2301 All articles covered by WikiProject Wikify, Al... 1124232962
23 https://en.wikipedia.org/wiki/History_of_democ... History of democracy A democracy is a political system, or a system... A democracy is a political system, or a system... 1972822 2383 All Wikipedia articles needing clarification, ... 1122481158
18 https://en.wikipedia.org/wiki/History_of_Bangl... History of Bangladesh Civilisational history of Bangladesh previousl... Civilisational history of Bangladesh previousl... 1835490 2666 All articles lacking reliable references, All ... 1123069979
12 https://en.wikipedia.org/wiki/History_of_Rome History of Rome The history of Rome includes the history of th... The history of Rome includes the history of th... 2079609 2796 All articles containing potentially dated stat... 1123947372
27 https://en.wikipedia.org/wiki/History_of_medicine History of medicine The history of medicine is both a study of med... The history of medicine is both a study of med... 2008806 2819 All articles containing potentially dated stat... 1123053788
19 https://en.wikipedia.org/wiki/History_of_Hinduism History of Hinduism The history of Hinduism covers a wide variety ... The history of Hinduism covers a wide variety ... 1797592 2935 All articles lacking reliable references, All ... 1124512840
33 https://en.wikipedia.org/wiki/History_of_Afgha... History of Afghanistan The history of Afghanistan as a state began in... The history of Afghanistan as a state began in... 2155957 2971 All articles with dead external links, Article... 1121424232
35 https://en.wikipedia.org/wiki/History_of_baske... History of basketball Basketball began with its invention in 1891 in... Basketball began with its invention in 1891 in... 4434485 3010 All articles to be expanded, All articles with... 1117994066
20 https://en.wikipedia.org/wiki/History_of_Ireland History of Ireland The first evidence of human presence in Irelan... The first evidence of human presence in Irelan... 2701679 3338 All accuracy disputes, All articles with style... 1123697361
34 https://en.wikipedia.org/wiki/History_of_Korea History of Korea The Lower Paleolithic era in the Korean Penins... The Lower Paleolithic era in the Korean Penins... 2743246 3470 Accuracy disputes from December 2017, All accu... 1124522331
26 https://en.wikipedia.org/wiki/History_of_Italy History of Italy The history of Italy covers the ancient period... The history of Italy covers the ancient period... 1824793 3518 All articles with incomplete citations, Articl... 1124364753
5 https://en.wikipedia.org/wiki/History_of_telev... History of television The concept of television was the work of many... The concept of television was the work of many... 5074930 3906 All articles to be expanded, All articles with... 1123620788
14 https://en.wikipedia.org/wiki/History_of_Spain History of Spain The history of Spain dates to the pre-Roman pe... The history of Spain dates to the pre-Roman pe... 2355382 4119 All accuracy disputes, All articles with dead ... 1124373367
24 https://en.wikipedia.org/wiki/History_of_pizza History of pizza The history of pizza begins in antiquity, as v... The history of pizza begins in antiquity, as v... 4086441 4146 All articles with failed verification, All art... 1124192347
31 https://en.wikipedia.org/wiki/History_of_Earth History of Earth The history of Earth concerns the development ... The history of Earth concerns the development ... 4602476 4317 AC with 0 elements, All articles with unsource... 1124371678
17 https://en.wikipedia.org/wiki/History_of_mathe... History of mathematics The history of mathematics deals with the orig... The history of mathematics deals with the orig... 3315598 4317 All articles needing additional references, Al... 1122788585
1 https://en.wikipedia.org/wiki/History_of_Europe History of Europe The history of Europe is traditionally divided... The history of Europe is traditionally divided... 2764829 4533 All articles needing additional references, Al... 1123723906
10 https://en.wikipedia.org/wiki/History_of_Russia History of Russia The history of Russia begins with the historie... The history of Russia begins with the historie... 3289180 4944 All articles with minor POV problems, All arti... 1124023255
39 https://en.wikipedia.org/wiki/History_of_Pakistan History of Pakistan The history of the lands of Pakistan for the p... The history of the lands of Pakistan for the p... 1806158 5054 All Wikipedia articles written in Pakistani En... 1123802420
13 https://en.wikipedia.org/wiki/History_of_Germany History of Germany The concept of Germany as a distinct region in... The concept of Germany as a distinct region in... 2741092 5078 All articles lacking reliable references, Arti... 1124396390
6 https://en.wikipedia.org/wiki/History_of_Chris... History of Christianity The history of Christianity concerns the Chris... The history of Christianity concerns the Chris... 3762801 5801 All articles with bare URLs for citations, All... 1123072552
15 https://en.wikipedia.org/wiki/History_of_Austr... History of Australia The history of Australia is the story of the l... The history of Australia is the story of the l... 3782171 5856 AC with 0 elements, All Wikipedia articles wri... 1123124583
3 https://en.wikipedia.org/wiki/History_of_Islam History of Islam The history of Islam concerns the political, s... The history of Islam concerns the political, s... 5703155 6107 All articles with unsourced statements, Articl... 1123323843
0 https://en.wikipedia.org/wiki/History History History (from Ancient Greek ἱστορία (historía... History (from Ancient Greek ἱστορία (historía... 8628272 6617 All Wikipedia articles written in American Eng... 1124009190
2 https://en.wikipedia.org/wiki/History_of_China History of China The earliest known written records of the hist... The earliest known written records of the hist... 6608376 7269 All articles containing potentially dated stat... 1124025086
4 https://en.wikipedia.org/wiki/History_of_Japan History of Japan The first human inhabitants of the Japanese ar... The first human inhabitants of the Japanese ar... 5031470 7603 All Wikipedia articles written in American Eng... 1123804351
8 https://en.wikipedia.org/wiki/History_of_Poland History of Poland The history of Poland spans over a thousand ye... The history of Poland spans over a thousand ye... 1542336 8199 All articles lacking reliable references, Arti... 1115808344
32 https://en.wikipedia.org/wiki/History_of_the_U... History of the United States The history of the lands that became the Unite... The history of the lands that became the Unite... 6916785 11165 All Wikipedia articles needing clarification, ... 1124535526
In [38]:
df.shape
Out[38]:
(45, 8)
In [39]:
df["text_length"] = df.text.str.split().apply(len)
In [40]:
df.sort_values("text_length")
Out[40]:
url title summary text num_views num_edits categories revision_id text_length
9 https://en.wikipedia.org/wiki/End_of_history End of history The end of history is a political and philosop... The end of history is a political and philosop... 499923 105 1860s neologisms, All articles to be expanded,... 1123918495 814
41 https://en.wikipedia.org/wiki/History_of_cheese History of cheese The production of cheese predates recorded his... The production of cheese predates recorded his... 653140 632 All articles with vague or ambiguous time, Art... 1123553575 1453
7 https://en.wikipedia.org/wiki/History_of_religion History of religion The history of religion refers to the written ... The history of religion refers to the written ... 1075565 654 All articles with specifically marked weasel-w... 1123342929 1652
43 https://en.wikipedia.org/wiki/History_of_corsets History of corsets The corset has been an indispensable supportiv... The corset has been an indispensable supportiv... 706954 553 16th-century fashion, 17th-century fashion, 18... 1083739897 1671
22 https://en.wikipedia.org/wiki/History_of_pharmacy History of pharmacy The history of pharmacy as an independent scie... The history of pharmacy as an independent scie... 450258 174 All articles with unsourced statements, Articl... 1113261032 1700
24 https://en.wikipedia.org/wiki/History_of_pizza History of pizza The history of pizza begins in antiquity, as v... The history of pizza begins in antiquity, as v... 4086441 4146 All articles with failed verification, All art... 1124192347 2640
36 https://en.wikipedia.org/wiki/History_of_commu... History of communication The history of communication technologies (med... The history of communication technologies (med... 1620064 957 All articles needing additional references, Al... 1123372214 2983
29 https://en.wikipedia.org/wiki/History_of_Facebook History of Facebook Facebook is a social networking service origin... Facebook is a social networking service origin... 4916523 1735 All Wikipedia articles in need of updating, Al... 1124469003 3215
40 https://en.wikipedia.org/wiki/History_of_cotton History of cotton The history of cotton can be traced to domesti... The history of cotton can be traced to domesti... 716500 343 All articles with unsourced statements, Articl... 1118655355 4067
30 https://en.wikipedia.org/wiki/History_of_cricket History of cricket The sport of cricket has a known history begin... The sport of cricket has a known history begin... 3234448 2040 All articles needing additional references, Al... 1123571727 4203
35 https://en.wikipedia.org/wiki/History_of_baske... History of basketball Basketball began with its invention in 1891 in... Basketball began with its invention in 1891 in... 4434485 3010 All articles to be expanded, All articles with... 1117994066 4218
25 https://en.wikipedia.org/wiki/History_of_syphilis History of syphilis The first recorded outbreak of syphilis in Eur... The first recorded outbreak of syphilis in Eur... 1548048 479 All articles with dead external links, All art... 1123246612 4314
28 https://en.wikipedia.org/wiki/History_of_calen... History of calendars The history of calendars, that is, of people c... The history of calendars, that is, of people c... 1411516 680 All articles with incomplete citations, All ar... 1123446945 4801
16 https://en.wikipedia.org/wiki/History_of_Arda History of Arda In J. R. R. Tolkien's legendarium, the history... In J. R. R. Tolkien's legendarium, the history... 605422 610 Articles with short description, Fictional his... 1123256525 5398
0 https://en.wikipedia.org/wiki/History History History (from Ancient Greek ἱστορία (historía... History (from Ancient Greek ἱστορία (historía... 8628272 6617 All Wikipedia articles written in American Eng... 1124009190 6912
38 https://en.wikipedia.org/wiki/History_of_Lambo... History of Lamborghini Automobili Lamborghini S.p.A. is an Italian br... Automobili Lamborghini S.p.A. is an Italian br... 960020 271 All articles with unsourced statements, Articl... 1108125867 6995
37 https://en.wikipedia.org/wiki/History_of_marke... History of marketing The study of the history of marketing, as a di... The study of the history of marketing, as a di... 931211 785 Articles with short description, EngvarB from ... 1117474637 7300
21 https://en.wikipedia.org/wiki/History_of_agric... History of agriculture Agriculture began independently in different p... Agriculture began independently in different p... 2516191 1977 All articles with dead external links, Article... 1123034506 7546
34 https://en.wikipedia.org/wiki/History_of_Korea History of Korea The Lower Paleolithic era in the Korean Penins... The Lower Paleolithic era in the Korean Penins... 2743246 3470 Accuracy disputes from December 2017, All accu... 1124522331 9137
31 https://en.wikipedia.org/wiki/History_of_Earth History of Earth The history of Earth concerns the development ... The history of Earth concerns the development ... 4602476 4317 AC with 0 elements, All articles with unsource... 1124371678 9435
23 https://en.wikipedia.org/wiki/History_of_democ... History of democracy A democracy is a political system, or a system... A democracy is a political system, or a system... 1972822 2383 All Wikipedia articles needing clarification, ... 1122481158 10521
11 https://en.wikipedia.org/wiki/History_of_banking History of banking The history of banking began with the first pr... The history of banking began with the first pr... 2333050 1619 All articles with unsourced statements, All ar... 1116409705 10707
17 https://en.wikipedia.org/wiki/History_of_mathe... History of mathematics The history of mathematics deals with the orig... The history of mathematics deals with the orig... 3315598 4317 All articles needing additional references, Al... 1122788585 10765
19 https://en.wikipedia.org/wiki/History_of_Hinduism History of Hinduism The history of Hinduism covers a wide variety ... The history of Hinduism covers a wide variety ... 1797592 2935 All articles lacking reliable references, All ... 1124512840 11124
2 https://en.wikipedia.org/wiki/History_of_China History of China The earliest known written records of the hist... The earliest known written records of the hist... 6608376 7269 All articles containing potentially dated stat... 1124025086 11454
4 https://en.wikipedia.org/wiki/History_of_Japan History of Japan The first human inhabitants of the Japanese ar... The first human inhabitants of the Japanese ar... 5031470 7603 All Wikipedia articles written in American Eng... 1123804351 11506
6 https://en.wikipedia.org/wiki/History_of_Chris... History of Christianity The history of Christianity concerns the Chris... The history of Christianity concerns the Chris... 3762801 5801 All articles with bare URLs for citations, All... 1123072552 12372
20 https://en.wikipedia.org/wiki/History_of_Ireland History of Ireland The first evidence of human presence in Irelan... The first evidence of human presence in Irelan... 2701679 3338 All accuracy disputes, All articles with style... 1123697361 12433
33 https://en.wikipedia.org/wiki/History_of_Afgha... History of Afghanistan The history of Afghanistan as a state began in... The history of Afghanistan as a state began in... 2155957 2971 All articles with dead external links, Article... 1121424232 12743
39 https://en.wikipedia.org/wiki/History_of_Pakistan History of Pakistan The history of the lands of Pakistan for the p... The history of the lands of Pakistan for the p... 1806158 5054 All Wikipedia articles written in Pakistani En... 1123802420 12999
12 https://en.wikipedia.org/wiki/History_of_Rome History of Rome The history of Rome includes the history of th... The history of Rome includes the history of th... 2079609 2796 All articles containing potentially dated stat... 1123947372 14297
27 https://en.wikipedia.org/wiki/History_of_medicine History of medicine The history of medicine is both a study of med... The history of medicine is both a study of med... 2008806 2819 All articles containing potentially dated stat... 1123053788 14577
5 https://en.wikipedia.org/wiki/History_of_telev... History of television The concept of television was the work of many... The concept of television was the work of many... 5074930 3906 All articles to be expanded, All articles with... 1123620788 15321
18 https://en.wikipedia.org/wiki/History_of_Bangl... History of Bangladesh Civilisational history of Bangladesh previousl... Civilisational history of Bangladesh previousl... 1835490 2666 All articles lacking reliable references, All ... 1123069979 16505
10 https://en.wikipedia.org/wiki/History_of_Russia History of Russia The history of Russia begins with the historie... The history of Russia begins with the historie... 3289180 4944 All articles with minor POV problems, All arti... 1124023255 16810
3 https://en.wikipedia.org/wiki/History_of_Islam History of Islam The history of Islam concerns the political, s... The history of Islam concerns the political, s... 5703155 6107 All articles with unsourced statements, Articl... 1123323843 17921
14 https://en.wikipedia.org/wiki/History_of_Spain History of Spain The history of Spain dates to the pre-Roman pe... The history of Spain dates to the pre-Roman pe... 2355382 4119 All accuracy disputes, All articles with dead ... 1124373367 17955
44 https://en.wikipedia.org/wiki/History_of_Ghana History of Ghana The Republic of Ghana is named after the medie... The Republic of Ghana is named after the medie... 1305934 1479 All Wikipedia articles written in Ghanaian Eng... 1123589131 18356
42 https://en.wikipedia.org/wiki/History_of_Belgium History of Belgium The history of Belgium extends before the foun... The history of Belgium extends before the foun... 1119414 2301 All articles covered by WikiProject Wikify, Al... 1124232962 19192
8 https://en.wikipedia.org/wiki/History_of_Poland History of Poland The history of Poland spans over a thousand ye... The history of Poland spans over a thousand ye... 1542336 8199 All articles lacking reliable references, Arti... 1115808344 19658
26 https://en.wikipedia.org/wiki/History_of_Italy History of Italy The history of Italy covers the ancient period... The history of Italy covers the ancient period... 1824793 3518 All articles with incomplete citations, Articl... 1124364753 20851
32 https://en.wikipedia.org/wiki/History_of_the_U... History of the United States The history of the lands that became the Unite... The history of the lands that became the Unite... 6916785 11165 All Wikipedia articles needing clarification, ... 1124535526 21447
1 https://en.wikipedia.org/wiki/History_of_Europe History of Europe The history of Europe is traditionally divided... The history of Europe is traditionally divided... 2764829 4533 All articles needing additional references, Al... 1123723906 25151
13 https://en.wikipedia.org/wiki/History_of_Germany History of Germany The concept of Germany as a distinct region in... The concept of Germany as a distinct region in... 2741092 5078 All articles lacking reliable references, Arti... 1124396390 32621
15 https://en.wikipedia.org/wiki/History_of_Austr... History of Australia The history of Australia is the story of the l... The history of Australia is the story of the l... 3782171 5856 AC with 0 elements, All Wikipedia articles wri... 1123124583 35046
In [41]:
import seaborn as sns

sns.pairplot(data=df[["text_length", "num_views", "num_edits"]])
Out[41]:
<seaborn.axisgrid.PairGrid at 0x116a8da60>