Bear Rocks View
My camera was almost dead when I reached Bear Rocks at the end of the day, but it still has some beautiful views of the valley below.
Friday October 11, 2019 — Notes
Afternoon coffee and plans to stay up late
☕️While I am not planning on pulling an all-nighter tonight, I think I want to stay up late and get up early tomorrow before my LASIK procedure mid-day. I’ll set my alarm for 5 AM. Seems silly, but I want to be pretty darn tired so that I pretty much can sleep it off that afternoon through evening into morning. Skipping coffee in the morning. I am thinking it will be like a bad hang-over, but then come Saturday things will be better.
xargs is awesome tool for running parallel processes
xargs is a powerful but simple command for parallel processing. You should use it more.
#!/bin/bash
# Call dmv command on each district
# in Manhattan, processing 3 districts
# at a time
seq 65 76 | xargs -nl -P3 -I{} dmv -a {}
# Call the dmv command on selected districts
# in the Bronx, running 3 at a time
echo "77 83 85 86 87" | xargs -nl -P3 -I{} -d' ' dmv -a {}
I used to do such things with loops and job control, which is fine, but xargs is much more compact and less error prone. I used to think of xargs as a program that was used primarily for taking a multi-line text file and appending it to a command but it’s actually very useful with the -P parallel flag. Moreover, modern computers are very good at handling multiple threads at once — and actually quite slow when they don’t have threads to work with because processor clock speeds haven’t increased in decades now due the clock speed barrier.
I also use xargs now a lot with wget2 when downloading LIDAR with the parrallel flag, as you can usually download multiple files quicker then one at a time. I used to do that with multiple loops and job control, but that’s a bad way to do it as it causes race conditions and other problems with the complexity.
echo "https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN580400.img https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN595400.img https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN610400.img https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN580385.img https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN595385.img https://gisdata.ny.gov/elevation/DEM/USGS_Schoharie2014/18TWN610385.img" | xargs -I {} -d ' ' -P 3 wget2 -c {}
As a bonus, here’s the R script I use to create that output to download those LIDAR tiles.
library(tidyverse)
library(sf)
library(terra)
library(arcpullr)
library(mapedit)
rm(list = ls())
shape_to_download <-
mapedit::drawFeatures() %>%
.[1] %>% st_transform(4326)
lidar.url <- tibble()
for (i in seq(2, 14)) {
if (nrow(lidar.url) == 0)
lidar.url <- get_layer_by_poly(
str_c(
'https://elevation.its.ny.gov/arcgis/rest/services/Dem_Indexes/MapServer/',
i
),
shape_to_download,
sp_rel = 'intersects'
)
}
lidar.url <- lidar.url %>% st_drop_geometry()
str_c('echo "',
paste(lidar.url$DIRECT_DL, collapse = ' '),
'" | xargs -I {} -d \' \' -P 3 wget2 -c {}') -> out
cat(out)
clipr::write_clip(out)
Threads and Tidyverse
While it is true that the Census API and tidycensus only supports pulling one state at a time for anything smaller then a county, when you want to pull all Census Tracts or Block Groups in America you can run it through purrr::map.
library(tidyverse)
library(tidycensus)
library(sf)
library(purrr)
map(
state.abb,
\(x) get_decennial(
geography = 'block group',
variables = 'P1_001N',
state = x,
geometry = TRUE,
cache_table = TRUE
)
) %>%
bind_rows() %>%
write_sf('/tmp/blockgroup-population-of-usa.gpkg')
A better way to do this is to use furr::future_map and pull multiple states at once. You do have to be careful how many threads you pull at once to not exceed Census API limits but if you limit it to a few threads and sign up for a free Census API Key you should be okay. On Linux it’s possible to use ‘multicore’ for even slightly better performance, as uses a fork of the existing session rather then opening a separate R session as is necessary on Windows as forking isn’t supported in that operating system.
library(tidyverse)
library(tidycensus)
library(sf)
library(furrr)
plan('multisession', workers = 3)
future_map(state.abb,
\(x) get_acs(
'tract',
'B19013_001',
cache_table = TRUE,
year = 2023,
state = x,
geometry = TRUE
)) %>%
bind_rows() %>%
write_sf('/tmp/median-hh-income-tract-usa.gpkg')
Cheater Glasses 👓
I was actually looking at my pictures of me wearing my glasses at that public hearing on Tuesday and thinking I kind of like the way I look with glasses when wearing the Salvation Army stripped-dress shirt. Makes me look like a real data scientist, whatever that term means. Or as they say where I work, King of the IOI (Interest-Occupation-Issue) Codes. I do dream of that day when we have a half billion data points on every New Yorker, though I’m sure database administrator will have to increase my quota before then and buy a bigger hard drive. I’m good at spending money and consuming resources like a liberal.
Going to rain today, so I think I will have to drive my big jacked up pickup truck to work, 🛻 which really isn’t the worse thing in the world, I mean the office is only like 8 3/4 miles on the mountain bike and probably somewhat less on the motorway. Don’t you know, normal people not only pay for high speed internet and garbage service, they also drive their 20-year old Honda Civics to the suburban office park every day from their vinyl and asphalt covered homes in suburbs. After all, driving a 20-year old Honda Civic and owning a vinyl-covered plywood house is considered to be a responsible thing to do unlike a mid-40 year old smoking pot and spending your weekends up in wilderness dreaming of having an off-grid hog farm somewhere in wilderness where nobody wants you to wash out the plastic before tossing it into the burn barrel. 🛢️ 🐽 Be proud of me, I did the urban recycling ♻️ thing today and washed out my milk bottle for recycling.
I keep telling myself that maybe the rain will stop and I can ride to work on my mountain bike, 🚲 but I know the forecast and that’s fucking stupid, because even if it isn’t pouring on the way in it, things will be very wet by afternoon. Plus I want to stock up on groceries before my eye surgery tomorrow. 🛒 I wonder how painful chopping onions will be in days directly after the big snip of my eyes? 🧅I guess I could take the yokel-bus downtown and spend half my day waiting for the transfer to the shuttle at Empire Plaza, walking laps in Concourse as I get yelled at by roving gangs of the homeless schizophrenic people that seem to be only ones crazy enough to be there after hours, 🚶👹 I mean Trumpster isn’t wrong on there being too many crazies in the cities. 🤪 I’ll probably do that enough once winter comes and snow is too deep to ride the rail trail to work. ☃️ I’m not going to trash my bike so badly riding on the ice this year, I tell myself. Or at least I’ll take Delmar Bypass and Corning’s Hill to work on my mountain bike in January when it’s negative 30 degrees out.
It just was such a wet bike ride on home. 🌧️🚲 My jeans and boots were absolutely soaked, as the sky opened up 5 minutes away from the office, and Erie Boulevard became a canal once again as I headed through the old Lumber District. The rain basically stopped by the time I was home, but I was pretty damn soaked by then. But was it worth it? Well I saved on bus fare 🚍 and gas ⛽ and the slowness of the yokel bus that stops at every block. But I got through more of Hunter S. Thompson’s Rum 🥃 Diaries, and got some exercise in. I went to the Farmers Market but it’s already transitioning to winter squash, which are fine, but I like the convinence of eating a shit ton of zucchuni that I can just fry up. 🍆 It’s unfair as all hell that we are already talking about putting a fork into summer.
Last night I got reading about choosing cheater glasses 👓 after I get LASIK tomorrow. I am not sure if I will need reading glasses right away, the surgeon’s assistant doesn’t think it’s necessary for me right away for me but I now kind of like the way glasses look and I’ve started to get used to wearing them after 3 days without contact lens. 👀 Cheater glasses are so damn cheap, you can get 5 packs for $20, and it’s not like once my distant vision is fixed they are essential except maybe eventually for reading. I doubt I’ll ever need cheater glasses in the wilderness or when I’m not wearing my Salvation Army work clothes. LASIK can make your reading vision worse by pushing the focus point out, though most people when they get to my age start to need bifocals. Barry Goldwater just wore cheater glasses with a very weak magnification because he liked the look. So maybe I’ll do that too. I never liked glasses after wearing contacts for decades much until I started to wearing them before the big snip by Optometrist Edward Sissorhands. ✂️ Truth is you want to appreciate something, or really want it, they have to threaten to take it away. 🐶 🔫 They never did find out who shot Cheeseface up in Vermont, did they? Edward Sissorhands has my big credit card purchase 💳 and I transferred in the money to reimburse the bank, and now I just got to pick up narcotics from pharmacy ⚕️ and then catch the 12:45 PM bus. I probably need to ask for somebody to cover for me for work tomorrow, as I probably shouldn’t be reviewing shit when I’m blind and high as fuck after surgery.
