Share this project

Done

Share this project

Done
Fix, open source & host OSM Stats
45 backers pledged $1,097 to help bring this project to life.

Funded!

Posted by Steve Coast (Creator)
2 likes

Thanks for funding!

I've spent a large amount of the last few days debugging the code that runs the stats site. We have two major bugs. First, some countries are missing and second, the stats end in 2016.

I think I've fixed both, but it takes a lot of time and double-checking. Running the full stats for a week (e.g. the last week) for the whole world takes about 3 days on a 64Gb VM with 8 CPUs. A lot of that work is reusable if you wanted to do, say, the last 10 weeks. So it doesn't take 30 days to do 10 weeks. But it does take hours to run a test which makes debugging slow.

So what were the bugs? For the data ending in 2016 it looks like a combination of running out of disk space and the machine hanging. For the missing countries it's a little more complicated.

There's a script which spits out osmosis-like polygon files from a world polygon shapefile via GDAL. For some reason I don't quite understand, it spits out some polygons one way, and others using a different loop. One of the loops had whitespace before the coordinates, and the other didn't. Inserting the whitespace works, but I don't yet understand why the multiple loops exist.

Therefore, I think I have an idea why the major bugs exist but it will be days before I can confirm it, and then possibly weeks before all the data is fully parsed and propagated.

Once this is done and working then the hosting needs to be shifted (more on this later). The code needs to be open sourced (there are database credentials baked in to the code that need to be ripped out). We need to figure out some t-shirts and put your names on the website.

Happy Tuesday!

Steve

Tieg Zaharia and kirbsmilieu like this update.

Comments

Only backers can post comments. Log In