Working PANDAS and American Community Survey Summary File

Want to be able to work with American Community Survey data offline using your own local copy of the ACS 5-year Summary File? It’s pretty easy to do with PANDAS. If you are planning a lot of Census queries, this can be a very fast way to extract data.

Before you can use this script, you will need to download some data:

import pandas as pd

path = '/home/andy/Desktop/acs-summary-file/'

# list of geography
geo = pd.read_excel(path+'5_year_Mini_Geo.xlsx', sheet_name='ny',index_col='Logical Record Number')

# load headers
header = pd.read_excel(path+'ACS_5yr_Seq_Table_Number_Lookup.xlsx')

# create a column with census variable headers
header['COL_NAME'] = header['Table ID'] + '_' + header['Line Number'].apply(lambda a: "{0:.0f}".format(a).zfill(3))

# segment id, along with ACS year and state
segId = 135
year = 2019
state = 'ny'

# create a list of headers for segment file
segHead = ['FILEID','FILETYPE','STUSAB','CHARITER','SEQUENCE','LOGRECNO'] \
    + header.query('`Sequence Number` == '+str(segId)).dropna(subset=['Line Number'])['COL_NAME'].to_list()

# read the segment file, including column names above    
seg = pd.read_csv(path+'e'+str(year)+'5'+state+(str(segId).zfill(4))+'000.txt',header=None, names=segHead, index_col=5)

# join the segment file to geography using Logical Record number
seg = geo.join(seg)

# calculate percentage of households with internet subscriptions -- codes from ACS_5yr_Seq_Table_Number_Lookup.xlsx
seg['Internet Subscription']=seg['B28011_002']/seg['B28011_001']*100

# output the percentage of households by county with internet subscriptions
seg[seg['Geography ID'].str.startswith('050')][['Geography Name','Internet Subscription']]

Geography NameInternet Subscription
Logical Record Number
13Albany County, New York83.888889
14Allegany County, New York76.248050
15Bronx County, New York75.917821
16Broome County, New York82.222562
17Cattaraugus County, New York72.431480
70Washington County, New York80.224036
71Wayne County, New York81.508715
72Westchester County, New York86.371288
73Wyoming County, New York78.387887
74Yates County, New York75.916583
# alternatively you can display human readable columns automatically
seg.rename(dict(zip(header['COL_NAME'],header['Table Title'])),axis=1)
StateGeography IDGeography NameFILEIDFILETYPESTUSABCHARITERSEQUENCETotal:Has one or more types of computing devices:
Logical Record Number
1NY04000US36New YorkACSSF201900000.0ny0.0135.07343234.06581493.0
2NY04001US36New York — UrbanACSSF201900000.0ny0.0135.06433524.05771681.0
3NY04043US36New York — RuralACSSF201900000.0ny0.0135.0909710.0809812.0
4NY040A0US36New York — In metropolitan or micropolitan st…ACSSF201900000.0ny0.0135.07189902.06449723.0
5NY040C0US36New York — In metropolitan statistical areaACSSF201900000.0ny0.0135.06796057.06109882.0
28400NY97000US3631920Yonkers City School District, New YorkACSSF201900000.0ny0.0135.074897.065767.0
28401NY97000US3631950York Central School District, New YorkACSSF201900000.0ny0.0135.02116.01964.0
28402NY97000US3631980Yorktown Central School District, New YorkACSSF201900000.0ny0.0135.07068.06751.0
28403NY97000US3632010Cuba-Rushford Central School District, New YorkACSSF201900000.0ny0.0135.02629.02186.0
28404NY97000US3699999Remainder of New York, New YorkACSSF201900000.0ny0.0135.079779.075425.0

Too much work or don’t want to download the summary file yourself? You can query the Census API directly using PyPI’s censusdata library from PIP. For infrequent queries where you are online, for those with Internet at home, you would be much better off just querying the API directly.

import pandas as pd
import censusdata as cd

# attributes to load
cdcol=['B28011_001','B28011_002']

cdf = cd.download('acs5', 2019,
           cd.censusgeo([('state', '36'),
                         ('county','*')]),
          cdcol)


# seperate out the geoid and geography name
geoid=[]
geoname=[]

for index in cdf.index.tolist():
    geopart=''
    for part in index.geo:
        geopart = geopart + part[1]
    geoid.append(geopart)
    geoname.append(index.name)

cdf['geoid']=geoid
cdf['geoname']=geoname

# calculate percentage with internet subscriptions
cdf['Internet Subscription']=cdf['B28011_002']/cdf['B28011_001']*100

# output a similar table as above
cdf

Learn how to load into PANDAS the PL 94-171 2020 Redistricting Data, a process that is similar but different then ACS data.

Also, calculate the population of an area and it’s average demographics, including areas that don’t have Census demographics such as Election Districts or County Legislative districts.

Leave a Reply

Your email address will not be published. Required fields are marked *