How Almanac works Where the data comes from, what happens to it every night and how mistakes are kept out.
What Almanac is
Almanac is a linked reference of sport. Every club, competition, season, player, stadium and match has its own page, and the pages link to each other, so you can follow a club from its current table back through every season it has played. Football comes first. Other sports will follow once football is solid.
Where the data comes from
Season tables, structure and history come from Wikipedia. Fixtures, match events, line-ups, players and transfers come from API-Football. Shot data comes from StatsBomb open data. Each source, and the licence it comes with, is listed on the data sources page.
When two sources disagree about the structure of a season, Wikipedia wins. When they disagree about what happened in a match, API-Football wins.
What happens every night
Nothing is typed in by hand. Scheduled jobs keep the data up to date and fill in history a little more each night, in this order.
- 00:00 UTC
Refresh this season
Current standings and fixtures are pulled again from API-Football for every league in progress. Seasons that have finished are rolled over.
- 01:00 UTC
Run the data checks
Automated checks look for things that cannot be right: a table that does not add up, a match with the same team on both sides, a club with two entries. New problems are recorded.
- 03:00 UTC
Update player leaderboards
Top scorers, assists and cards for current seasons.
- 03:30 UTC
Fill in history
A second set of jobs works through the gaps, one after another: past fixtures that are missing, shot data from StatsBomb open data, match events and line-ups, player transfers, then new historical seasons from Wikipedia. After those come missing player names and photos, the safe automatic fixes for problems the data checks found, and once a week a search for new kinds of problem to check for.
- After the history jobs
Walk the site
A last job opens the live site, follows links from page to page and raises an alert for any page that keeps failing.
- 04:00 UTC
Count duplicate clubs
The number of clubs that have been created automatically and not yet matched to a real club is recorded, along with any that look like duplicates.
How bad data is kept out
- Each historical season is compared with Wikipedia and with a second source before it is imported.
- If the sources disagree, or the importer cannot tell which real club a name refers to, the season is held back and not shown.
- Imported seasons are checked again after the import has finished.
- The data checks run every night. Problems they find are either fixed automatically when the fix is safe, or listed on the data checks page for a person to look at.
- Every fix is logged on the Data tab of the changelog.
What the last run did
These are the recorded results of the most recent nightly run. The coverage page shows the full picture, including anything that got stuck.
| Job | Started | Result |
|---|---|---|
| Refresh this season | 00:00 UTC | 111 competitions refreshed |
| Data checks | 01:00 UTC | 796 new findings, 2,943 open in total |
| Past fixtures | 03:30 UTC | 11432 matches added across 39 seasons |
| Match events and line-ups | 03:33 UTC | 11470 matches filled in |
| Transfers | 04:07 UTC | 9799 transfers stored for 1500 players |
| Historical seasons | 04:15 UTC | 24 seasons imported, 3 held for review |
| Player profiles | 04:16 UTC | 1407 players filled in |
Follow along
The changelog lists every change to the site and every data fix. The coverage page shows which leagues and seasons are complete and what the nightly jobs are working on next.