OsmExtract output

This run was on MMU (may 2017, 16 Gb, only SSDs) with various optimizing choices.

To be compared with this.

It took nearly 96 hours (nearly 4 days):

>> osmExtract
This script converts OSM data into postgres DB
Supposed to run on MMU (execution time ~1 hour)
interactive because asking password
see documentation in wiki page GeoJava

-rw-r--r-- 1 pge pge 35G Mar 17 11:55 /media/Shared/server.geo/planet-170306.osm.pbf
Tue May 23 09:25:36 CEST 2017
OSM -> postgres gis database : start

osm2pgsql SVN <span style="color:#008000;">version 0.88.1</span> (64bit id space)

Password:qwert
Using built-in tag processing pipeline
Using projection SRS 4326 (Latlong)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads
Allocating memory for dense node cache
Allocating dense node cache in one big chunk
Allocating memory for sparse node cache
Sharing dense sparse
Node-cache: cache=8000MB, maxblocks=1024000*8192, allocation method=11
Mid: loading persistent node cache from /media/Shared/server.geo/FlatNodes
Allocated space for persistent node cache file
Maximum node in persistent node cache: 0
Mid: pgsql, scale=10000000 cache=8000
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels

Reading in file: /media/Shared/server.geo/planet-170306.osm.pbf
Processing: Node(3780985k 1694.7k/s) Way(398714k 15.50k/s) Relation(4165320 17.96/s)
Standard exception processing relation id=6244046: TopologyException: side location conflict at -75.702342200000004 41.350785100000003
Processing: <span style="color:#008000;">Node(3780985k 1694.7k/s) Way(398714k 15.50k/s) Relation(4839260 18.54/s)</span>  parse time: 288955s
Node stats: total(3780985254), max(4719536321) in 2231s
Way stats: total(398714532), max(478741177) in 25727s
Relation stats: total(4839264), max(7037743) in 260997s
Maximum node in persistent node cache: 4719640575
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Mid: loading persistent node cache from /media/Shared/server.geo/FlatNodes
Maximum node in persistent node cache: 4719640575
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /media/Shared/server.geo/FlatNodes
Maximum node in persistent node cache: 4719640575
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /media/Shared/server.geo/FlatNodes
Maximum node in persistent node cache: 4719640575
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Mid: loading persistent node cache from /media/Shared/server.geo/FlatNodes
Maximum node in persistent node cache: 4719640575
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline

Going over pending ways...
    15229821 ways are pending

Using 4 helper-processes
Finished processing 15229821 ways in 17608 sec

15229821 Pending ways took 17608s at a rate of 864.94/s
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads
Committing transaction for planet_osm_point
Committing transaction for planet_osm_line
Committing transaction for planet_osm_polygon
Committing transaction for planet_osm_roads

Going over pending relations...
    0 relations are pending

Using 4 helper-processes
Finished processing 0 relations in 0 sec

Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_point
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_line
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_polygon
WARNING:  there is no transaction in progress
Committing transaction for planet_osm_roads
WARNING:  there is no transaction in progress
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Sorting data and creating indexes for planet_osm_line
Stopping table: planet_osm_ways
Building index on table: planet_osm_ways
Stopping table: planet_osm_rels
Building index on table: planet_osm_rels
Sorting data and creating indexes for planet_osm_polygon
Sorting data and creating indexes for planet_osm_roads
Stopped table: planet_osm_nodes in 0s
Stopped table: planet_osm_rels in 4352s
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on  planet_osm_point
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on  planet_osm_roads
Creating osm_id index on  planet_osm_point
Creating osm_id index on  planet_osm_roads
Creating indexes on  planet_osm_roads finished
Creating indexes on  planet_osm_point finished
All indexes on  planet_osm_roads created  in 21368s
Completed planet_osm_roads
All indexes on  planet_osm_point created  in 21513s
Completed planet_osm_point
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on  planet_osm_polygon
/home/pge/Scripts/osmExtract: line 26: 22466 Killed                  osm2pgsql --hstore --create -l -d gis --slim -C $CACHESIZE -U pge -W --flat-nodes $FLATNODES --number-processes 4 --style $STYLEFILE $OSMFILE

<span style="color:#008000;">real    5735m45.130s
user    634m55.836s
sys    63m50.672s</span>

OSM -> postgres gis database : done
Sat May 27 09:01:21 CEST 2017