Evaluating trendiness in open source software
- What are we trying to measure?
- Are they a good measure of success for open source software?
- Can we simply rely on projects with lots of stars?
Trendiness of OSS should be assessed by contribution rate, not by stars
Enters the Github Pulse page
We can't sort projects by contribution rate...
|| * authors
|| * merged and proposed pull requests
|| * new and closed issues
|| * commits
Poll the Github API to get all of the repositories on Github, in a
loop, 100 at a time.
Scrape the Pulse page for each repo, extract last week contribution
- Ignores repositories with only one author.
- Ignores repositories with Krihelimeter < 30.
Calculate Krihelimeter and persist to DB.
Kicks in every 6 hours.
Scrapes the Github trending page, pass to the scrapers.
Keeps the 500 repositories with the highest Krihelimeter, deletes the rest.
Rescrapes and updates all of the repositories in the DB.
Simpler concurrency model
Higher level language
No job queue, no workers, no celery...
- ~20MB inbound per second
- Scraping ~140 pages per second
- ~40 milliseconds per request