Wikipedia Pull Data from table sample

I was researching some data sets and had the need to pull population data from Wikipedia. I looked at several articles on doing this and they all seemed either very basic or not current. I had already done an article on the Wikipedia Python package so I decided to try to write one on pulling table data out of a Wikipedia page. The packages I used were urllib and Beautiful Soup. I could have done this with other packages (requests comes to mind) but this is very simple and it works. Many Wikipedia pages offer tables to display data and most of them happen to be the only tables on the page or have unique IDs. Unfortunately, on the "List_of_states_and_territories_of_the_United_States_by_population" page on Wikipedia, There is a table for state rankings and another for rankings by region as well. So I would need to only pull table data for the specific table I needed, which was not too hard.

So to start, how do you pull data from a Wikipedia page? urllib works for me so that's what I used:

So we import urlopen from urllib.request and beautiful soup from bs4. Setup the url and then open and read in the content. The next step is to setup your data with BeautifulSoup. Quick note: I have hardcoded some things and removed the code from the class it was in for simplicity sake. The next code snippet reads through the Beautiful soup(BS) dataset and parses the data for me.

That first line is where most of the magic happens. Most tables on the Wikipedia site use the table classes, wikitable and sortable. BS has a great interface called select where you can use HTML tags (table in this case) and then further find the exact table you want using css tags, so that explains the .wikitable.sortable . Then we tell BS that we want to further filter our page table with the tbody and tr tags that are found within the table we want. The rest is just simplified for loop to loop through the table rows and pull out the State name and population statistics I needed. The result is the data shown below.

Here are the results for U.S Population by State:

State2019 Population Estimate2010 Census PopulationPercent of US Population
 California 39,538,223 37,253,956 702,905
 Texas 29,145,505 25,145,561 698,488
 Florida 21,538,187 18,801,310 696,345
 New York 20,201,249 19,378,102 717,707
 Pennsylvania 13,002,700 12,702,379 705,688
 Illinois 12,812,508 12,830,632 712,813
 Ohio 11,799,448 11,536,504 721,032
 Georgia 10,711,908 9,687,653 691,975
 North Carolina 10,439,388 9,535,483 733,499
 Michigan 10,077,331 9,883,640 705,974
 New Jersey 9,288,994 8,791,894 732,658
 Virginia 8,631,393 8,001,024 727,366
 Washington 7,705,281 6,724,540 672,454
 Arizona 7,151,502 6,392,017 710,224
 Massachusetts 7,029,917 6,547,629 727,514
 Tennessee 6,910,840 6,346,105 705,123
 Indiana 6,785,528 6,483,802 720,422
 Maryland 6,177,224 5,773,552 748,616
 Missouri 6,154,913 5,988,927 721,694
 Wisconsin 5,893,718 5,686,986 710,873
 Colorado 5,773,714 5,029,196 718,457
 Minnesota 5,706,494 5,303,925 662,991
 South Carolina 5,118,425 4,625,364 660,766
 Alabama 5,024,279 4,779,736 682,819
 Louisiana 4,657,757 4,533,372 755,562
 Kentucky 4,505,836 4,339,367 723,228
 Oregon 4,237,256 3,831,074 766,215
 Oklahoma 3,959,353 3,751,351 750,270
 Connecticut 3,605,944 3,574,097 714,819
 Puerto Rico 3,285,874 3,725,789
 Utah 3,271,616 2,763,885 690,971
 Iowa 3,190,369 3,046,355 761,589
 Nevada 3,104,614 2,700,551 675,138
 Arkansas 3,011,524 2,915,918 728,980
 Mississippi 2,961,279 2,967,297 741,824
 Kansas 2,937,880 2,853,118 713,280
 New Mexico 2,117,522 2,059,179 686,393
 Nebraska 1,961,504 1,826,341 608,780
 Idaho 1,839,106 1,567,582 783,791
 West Virginia 1,793,716 1,852,994 617,665
 Hawaii 1,455,271 1,360,301 680,151
 New Hampshire 1,377,529 1,316,470 658,235
 Maine 1,362,359 1,328,361 664,181
 Rhode Island 1,097,379 1,052,567 526,284
 Montana 1,084,225 989,415 989,415
 Delaware 989,948 897,934 897,934
 South Dakota 886,667 814,180 814,180
 North Dakota 779,094 672,591 672,591
 Alaska 733,391 710,231 710,231
 District of Columbia 689,545 601,723
 Vermont 643,077 625,741 625,741
 Wyoming 576,851 563,626 563,626
 Guam[8] 153,836 159,358
 U.S. Virgin Islands[9] 87,146 106,405
 American Samoa[10] 49,710 55,519
 Northern Mariana Islands[11] 47,329 53,883
Back to top Total States and Territories found: 56