← All products02 / 07
eLearnScanner
42,000 online courses, one search box.
Role
Co-founder, engineerJun 2018 – Jan 2020
Status
Shut down 2020
In numbers
42,000 courses · 22 platforms · hourly crawls
Hosted on
Google App Engine
Stack
PythonScrapyDjangoPostgreSQLElasticsearchCelery
The problem
Online courses were spread over dozens of platforms with no way to search or compare them in one place.
What I built
A search engine over 42,000 courses from 22 platforms: one crawler per platform, a shared taxonomy, a search site with comparison. Monetised through affiliate links.
Outcome
It made money, far less than hoped. We knew how to build, not how to market a platform.
Build the minimum that tests whether anyone wants it. I built the whole crawler and its integration before the first visitor arrived.what I took from it
How it was built
Decisions
- One Scrapy spider per platform at a polite hourly interval, so a layout change on one platform never broke the others.
- Every platform's field mapping lived in Django as configuration. Adding a provider was an admin task, not a code change.
- Search on Elasticsearch through Haystack, so category, price and language were facets rather than SQL.
Timeline
- 2018Crawler framework, first ten providers, taxonomy
- 2019Search site, Elasticsearch, affiliate feeds
- Jan 2020Shut down