Husker Gym Scraper

Scraping the live gym counts at https://connect2concepts.com/connect2/?type=circle&key=2A2BE0D8-DF10-4A48-BEDD-B3BC0CD628E7

Check out husker-nu/husker-gym for the website.

This project scrapes the Connect2Concepts live counts website every 15 minutes using GitHub Actions, and saves the counts to a database. The data is used for https://husker-gym.vercel.app/

Scrape

Run scrape with

pdm run python -m gymtime -f

Database

Create database

pdm run db-create

Seed

pdm run db-drop-all
pdm run db-seed-gyms
pdm run db-seed-test-data

Warning Running pdm run db-drop-all will delete all rows in the database. Only run in locally.

Planetscale

Connect to the organization:

pscale org switch husker-gym

List all branches:

pscale branch list gymtime

Creating a dev branch for local development:

pscale branch create gymtime dev

Access the dev branch of the database with

pscale shell gymtime dev

This opens a mysql shell of the branch dev.

Connect to the branch with

pscale connect gymtime dev

This makes the dev branch accessible as if it were on localhost.

Adding gyms to the main DB branch

This needs to be run only once:

pscale connect gymtime main
pdm run db-seed-gyms

Once this is done, confirm the data has been added with

pscale shell gymtime main

show tables;
select * from gym;
select * from section;

GitHub Actions

Add DATABASE_URL to Settings > Secrets and Variables > Actions > Secrets > Repository secrets. Remove the ?sslaccept=strict from the Planetscale URL.

Tests

To run all tests, run

pdm run test

Tests are located in the test/ folder.

Other

Local TZ to UTC

(Credits)

from datetime import datetime

# We're using 3.8, but zoneinfo was added in 3.9
from backports.zoneinfo import ZoneInfo

local_tz = ZoneInfo("America/New_York")
utc_tz = ZoneInfo("UTC")

# Boston time
est = datetime(year=2023, month=1, day=15, hour=20, minute=30, second=0)
est = est.replace(tzinfo=local_tz)
# UTC time
utc = est.astimezone(utc_tz)

print(est)
print(utc)

Contributors

• Parth Kabra
• ... (more from Northeastern Electric Racing)