SlideShare a Scribd company logo
1 of 82
Using Python to Analyze
Spatial Data
Juan Carlos Méndez
CTO @gkudos
juan@gkudos.com
Topics
● Spatial Data?
● Python?
● Visualizing Geodata
● Analyzing Geodata (Basics)
About me
● Juan Carlos Méndez
○ CTO at @gkudos
○ GIS Consultant
○ Software Architect / Programmer / “Data Engineer”
○ https://github.com/dersteppenwolf
● Education
○ Systems Engineer- Universidad Nacional de Colombia, Bogotá
○ Telematics / eBusiness specialist - Escuela Colombiana de ingeniería
○ Information Engineering (Student) - Universidad de los Andes
https://github.com/dersteppenwolf/pycon
Requirements
● Qgis ( http://www.qgis.org/es/site/ )
pip install -r requirements.txt
brew install mapnik
brew install spatialindex
Spatial Data?
Location, location, location!
“You can buy the right
home in the wrong
location. You can change
the structure, remodel it or
alter the home's layout
but, ordinarily, you cannot
move it. It's attached to the
land”
http://bit.ly/2fz2ySD
Spatio-temporal data
“...in Google, about 25 PB of data is
being generated per day, and a
significant portion of the data falls into
the realm of spatio-temporal data...”
Lee, J.-G., & Kang, M. (2015). Geospatial Big Data:
Challenges and Opportunities. Big Data Research, 2(2),
74–81. http://doi.org/10.1016/j.bdr.2015.01.003
What for?
Using python to analyze spatial data
Using python to analyze spatial data
15 Global challenges
facing humanity
http://www.millennium-project.org/
millennium/challeng.html
How?
Using python to analyze spatial data
Using python to analyze spatial data
Using python to analyze spatial data
Using python to analyze spatial data
But...
What is Spatial Data?
Spatial Data
● Located on the surface of the
earth
● Coordinate Systems
GIS (Geographic Information
Systems)
● Body of Knowledge
● Tools
● Science
Spatial is special?
Yes:
● Multidimensional
● Voluminous
● Special methods for analysis
● Updating: Slow, complex and expensive
“Everything is related to everything else, but near
things are more related than distant things”
Tobler, W. 1970. A
Spatial is special?
No:
● “spatial is not special, it’s just another column in the
database...” Michael Terner
http://bit.ly/2jZAWav
Spatial Data: Vector vs Raster
Vector
http://gisgeography.com/spatial-data-types-vector-raster/
Spatial Data: Vector vs Raster
Map projection
“...formal process which
converts features between a
spherical or ellipsoidal surface
and a projection surface,
which is often flat…”
http://bit.ly/2bA9Szk
Using python to analyze spatial data
Using python to analyze spatial data
Spatial Data - File Formats
...lots… (open and proprietary)...
● Shapefile
● Geojson
● KML
● Filegeodatabase
● Geotiff
● Geopackage
● Spatialite
● … ...
Spatial is special?
...It can be complex...
… using python it will be more fun and easy…
Python?
Python for Geospatial Data
Lots of…
● Tools / Libraries
● Propietary / Open
● Desktop / Server
● Analysis / Visualization / ETL
Python for Geospatial Data
ESRI
● Arcpy
○ Desktop: Automation / Customization. E.g. CartoDB Toolbox : Import Data From
to Carto
○ Server: Geoprocessing as a “Web Service”
● Arcgis Web : ArcGIS API for Python (2017)
QGIS
● Open source desktop GIS Tool written in C++, Python, Qt
● QGIS API E.g. CartoDB Plugin for QGis
● PyQGIS: Scripting using Python
● Server: Qgis Server python plugins
Python for Geospatial Data
● CKAN - web-based open source management system for the storage and
distribution of open data (including geospatial data).
● GeoDjango - storing and manipulating geographic data using the Django ORM
● Geonode - web-based application and platform for developing geospatial
information systems (GIS) and for deploying spatial data infrastructures (SDI)
Python for Geospatial Data
● pyshp - For reading and writing shapefiles (in pure Python)
● pyproj - For conversions between projections
● shapely - For geometry handling
● fiona - For making it easy to read/write geospatial data formats
● ogr/gdal - For reading, writing, and transforming geospatial data formats *
● Rasterio - reads and writes geospatial raster datasets
Show me the code
Using python to analyze spatial data
Describe Data
data/world_borders
Using python to analyze spatial data
Using python to analyze spatial data
Fiona
● https://github.com/Toblerity/Fiona
● Fiona is OGR's neat and nimble API for Python programmers.
● Fiona does reading and writing data formats. For this it uses OGR, the most
popular open-source conversion system
● The OGR Simple Features Library is a C++ open source library providing read
(and sometimes write) access to a variety of vector file formats including ESRI
Shapefiles, S-57, SDTS, PostGIS, Oracle Spatial, and Mapinfo mid/mif and TAB
formats.
Using python to analyze spatial data
Data Format Conversion
From CSV to SHP
Shapely
● https://github.com/Toblerity/Shapely
● Manipulation and analysis of geometric objects in the Cartesian plane
● With Shapely, you can do things like buffers, unions, intersections, centroids,
convex hulls, and lots more.
Using python to analyze spatial data
Using python to analyze spatial data
Geocoding
What is geocoding?
● Geocoding is the process of transforming a description of a location—such as a
pair of coordinates, an address, or a name of a place—to a location on the
earth's surface. http://arcg.is/2kUedk7
GeoPy
● https://github.com/geopy/geopy
● geopy makes it easy for Python developers to locate the coordinates of
addresses, cities, countries, and landmarks across the globe using third-party
geocoders and other data sources
● geopy includes geocoder classes for the OpenStreetMap Nominatim, ESRI
ArcGIS, Google Geocoding API (V3), Baidu Maps, Bing Maps API, Mapzen
Search, Yandex, IGN France, GeoNames, NaviData, OpenMapQuest,
What3Words, OpenCage, SmartyStreets, geocoder.us, and GeocodeFarm
geocoder services.
Using python to analyze spatial data
Reverse Geocoding
Using python to analyze spatial data
Visualize?
Mapnik
● http://mapnik.org/
● the core of geospatial visualization & processing
● mapnik combines pixel-perfect image output with lightning-fast cartographic
algorithms, and exposes interfaces in C++, Python, and Node.
Mapnik
● Installing Mapnik on OS X with Homebrew
https://github.com/mapnik/mapnik/wiki/MacInstallation_Homebrew
○ brew install mapnik
● Python bindings for mapnik https://github.com/mapnik/python-mapnik
● Stacks built with mapnik: OpenStreetMap , Mapbox , CartoDB , Stamen ,
MapQuest , Kosmtik
Mapnik
Basics
Using python to analyze spatial data
Mapnik - XML
Using python to analyze spatial data
Mapnik
Complex Styles
Using python to analyze spatial data
Mapnik
Markers
Using python to analyze spatial data
Mapnik
Composite
Compositing operations affect the way
colors and textures of different elements
and styles interact with each other.
Two main categories: color and alpha
E.g. Multiply literally multiplies the color of
the top layer by the color of each layer
beneath, which usually means overlapping
areas become darker.
Using python to analyze spatial data
What is Spatial Analysis?
Spatial Analysis
Geospatial data is more than maps!
What is Geoprocessing:?
● Geoprocessing is any GIS operation used to manipulate data.
● A typical geoprocessing operation takes an input dataset, performs an
operation on that dataset, and returns the result of the operation as an output
dataset, also referred to as derived data.
● Common geoprocessing operations: geographic feature overlay, feature
selection and analysis, topology processing, and data conversion.
● Geoprocessing allows you to define, manage, and analyze geographic
information used to make decisions.
● http://bit.ly/2k8l3P8
Spatial Analysis
● Spatial analysis includes
any of the formal
techniques which study
entities using their
topological, geometric,
or geographic
properties.
Spatial Analysis
Using python to analyze spatial data
Using python to analyze spatial data
Jupyter
● http://jupyter.org/
● web application that allows you to create
and share documents that contain live
code, equations, visualizations and
explanatory text
● Uses include: data cleaning and
transformation, numerical simulation,
statistical modeling, machine learning
and much more
Geopandas
● https://github.com/geopandas/geopandas
● pandas is an open source, BSD-licensed library providing high-performance,
easy-to-use data structures and data analysis tools for the Python
programming language.
● GeoPandas is a project to add support for geographic data to pandas objects.
PySAL
● https://github.com/pysal/pysal
● PySAL: Python Spatial Analysis Library
● Sub-packages
pysal.cg – Computational Geometry
pysal.core — Core Data Structures and IO
pysal.esda — Exploratory Spatial Data Analysis
pysal.inequality — Spatial Inequality Analysis
pysal.region — Spatially Constrained Clustering
pysal.spatial_dynamics — Spatial Dynamics
pysal.spreg — Regression and Diagnostics
pysal.weights — Spatial Weights
pysal.network — Network Constrained Analysis
pysal.contrib – Contributed Modules
Using python to analyze spatial data
Using python to analyze spatial data
Using python to analyze spatial data
Geoprocessing
Geoprocessing
● Spatial Joins
● Data Clipping
● Data Enrichment
● Data filtering
● Spatial Analysis
data/ideca/Loca.shp
Localidades, Bogotá
data/ideca/sitios_interes.shp
Sitios de Interés, Bogotá
data/ideca/clasificacion_sitios_interes.csv
Using python to analyze spatial data
juan@gkudos.com http://gkudos.com/
info@gkudos.com
Thanks!!

More Related Content

What's hot

Python - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesPython - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesAndrew Ferlitsch
 
WEB GIS AND WEB MAP.pptx
WEB GIS AND WEB MAP.pptxWEB GIS AND WEB MAP.pptx
WEB GIS AND WEB MAP.pptxAsim Pt
 
Map reduce presentation
Map reduce presentationMap reduce presentation
Map reduce presentationateeq ateeq
 
Map reduce in BIG DATA
Map reduce in BIG DATAMap reduce in BIG DATA
Map reduce in BIG DATAGauravBiswas9
 
6. Shapefiles in gis
6. Shapefiles in gis6. Shapefiles in gis
6. Shapefiles in gisKU Leuven
 
Network analysis for shortest optimum path
Network analysis for shortest optimum pathNetwork analysis for shortest optimum path
Network analysis for shortest optimum pathSourabh Jain
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big DataVipin Batra
 
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...Simplilearn
 
R programming language in spatial analysis
R programming language in spatial analysisR programming language in spatial analysis
R programming language in spatial analysisAbhiram Kanigolla
 

What's hot (20)

Python - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning LibrariesPython - Numpy/Pandas/Matplot Machine Learning Libraries
Python - Numpy/Pandas/Matplot Machine Learning Libraries
 
WEB GIS AND WEB MAP.pptx
WEB GIS AND WEB MAP.pptxWEB GIS AND WEB MAP.pptx
WEB GIS AND WEB MAP.pptx
 
Map Reduce
Map ReduceMap Reduce
Map Reduce
 
Map reduce presentation
Map reduce presentationMap reduce presentation
Map reduce presentation
 
Web GIS using Google Map and QGIS
Web GIS using Google Map and QGISWeb GIS using Google Map and QGIS
Web GIS using Google Map and QGIS
 
Spatial Data Model
Spatial Data ModelSpatial Data Model
Spatial Data Model
 
Hive(ppt)
Hive(ppt)Hive(ppt)
Hive(ppt)
 
Map reduce in BIG DATA
Map reduce in BIG DATAMap reduce in BIG DATA
Map reduce in BIG DATA
 
6. Shapefiles in gis
6. Shapefiles in gis6. Shapefiles in gis
6. Shapefiles in gis
 
Web Mapping
Web MappingWeb Mapping
Web Mapping
 
QGIS Tutorial 2
QGIS Tutorial 2QGIS Tutorial 2
QGIS Tutorial 2
 
GIS
GISGIS
GIS
 
Network analysis for shortest optimum path
Network analysis for shortest optimum pathNetwork analysis for shortest optimum path
Network analysis for shortest optimum path
 
Spark graphx
Spark graphxSpark graphx
Spark graphx
 
QGIS Tutorial 1
QGIS Tutorial 1QGIS Tutorial 1
QGIS Tutorial 1
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big Data
 
Mobile gis
Mobile gisMobile gis
Mobile gis
 
Principal Component Analysis
Principal Component AnalysisPrincipal Component Analysis
Principal Component Analysis
 
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
 
R programming language in spatial analysis
R programming language in spatial analysisR programming language in spatial analysis
R programming language in spatial analysis
 

Similar to Using python to analyze spatial data

Large Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopLarge Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopChristoph Körner
 
Powerful geographic web framework GeoDjango
Powerful geographic web framework GeoDjangoPowerful geographic web framework GeoDjango
Powerful geographic web framework GeoDjangoOMEGA (@equal_001)
 
APPLICATION OF PYTHON IN GEOSCIENCE
APPLICATION OF  PYTHON IN GEOSCIENCEAPPLICATION OF  PYTHON IN GEOSCIENCE
APPLICATION OF PYTHON IN GEOSCIENCEAhasanHabibSajeeb
 
Open source Geospatial Business Intelligence in action with GeoMondrian and S...
Open source Geospatial Business Intelligence in action with GeoMondrian and S...Open source Geospatial Business Intelligence in action with GeoMondrian and S...
Open source Geospatial Business Intelligence in action with GeoMondrian and S...Thierry Badard
 
Saving Money with Open Source GIS
Saving Money with Open Source GISSaving Money with Open Source GIS
Saving Money with Open Source GISbryanluman
 
Software & programming languages for undergraduate students (Ag. Engg)
Software & programming languages for undergraduate students (Ag. Engg)Software & programming languages for undergraduate students (Ag. Engg)
Software & programming languages for undergraduate students (Ag. Engg)Moudud Hasan
 
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...Thierry Badard
 
GeoNode intro and demo
GeoNode intro and demoGeoNode intro and demo
GeoNode intro and demoPaolo Corti
 
c,c++,java and python in gis development
c,c++,java and python in gis developmentc,c++,java and python in gis development
c,c++,java and python in gis developmentSakthivel R
 
LocationTech Projects
LocationTech ProjectsLocationTech Projects
LocationTech ProjectsJody Garnett
 
Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding HadoopAhmed Ossama
 
Mapping, GIS and geolocating data in Java
Mapping, GIS and geolocating data in JavaMapping, GIS and geolocating data in Java
Mapping, GIS and geolocating data in JavaJoachim Van der Auwera
 
Introduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGIntroduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGAdam Kawa
 
My experience and suggestions as a web-GIS developer
My experience and suggestions as a web-GIS developerMy experience and suggestions as a web-GIS developer
My experience and suggestions as a web-GIS developerTek Kshetri
 
Pycon 2012 Taiwan
Pycon 2012 TaiwanPycon 2012 Taiwan
Pycon 2012 TaiwanDongpo Deng
 

Similar to Using python to analyze spatial data (20)

Large Scale Geo Processing on Hadoop
Large Scale Geo Processing on HadoopLarge Scale Geo Processing on Hadoop
Large Scale Geo Processing on Hadoop
 
Powerful geographic web framework GeoDjango
Powerful geographic web framework GeoDjangoPowerful geographic web framework GeoDjango
Powerful geographic web framework GeoDjango
 
APPLICATION OF PYTHON IN GEOSCIENCE
APPLICATION OF  PYTHON IN GEOSCIENCEAPPLICATION OF  PYTHON IN GEOSCIENCE
APPLICATION OF PYTHON IN GEOSCIENCE
 
Open source Geospatial Business Intelligence in action with GeoMondrian and S...
Open source Geospatial Business Intelligence in action with GeoMondrian and S...Open source Geospatial Business Intelligence in action with GeoMondrian and S...
Open source Geospatial Business Intelligence in action with GeoMondrian and S...
 
Saving Money with Open Source GIS
Saving Money with Open Source GISSaving Money with Open Source GIS
Saving Money with Open Source GIS
 
Software & programming languages for undergraduate students (Ag. Engg)
Software & programming languages for undergraduate students (Ag. Engg)Software & programming languages for undergraduate students (Ag. Engg)
Software & programming languages for undergraduate students (Ag. Engg)
 
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...
Open Source Geospatial Business Intelligence (GeoBI): Definition, architectur...
 
GeoNode intro and demo
GeoNode intro and demoGeoNode intro and demo
GeoNode intro and demo
 
Open geo data - technical issue
Open geo data  - technical issueOpen geo data  - technical issue
Open geo data - technical issue
 
Geonode 2.0
Geonode 2.0Geonode 2.0
Geonode 2.0
 
c,c++,java and python in gis development
c,c++,java and python in gis developmentc,c++,java and python in gis development
c,c++,java and python in gis development
 
LocationTech Projects
LocationTech ProjectsLocationTech Projects
LocationTech Projects
 
Understanding Hadoop
Understanding HadoopUnderstanding Hadoop
Understanding Hadoop
 
Introduction to GIS
Introduction to GISIntroduction to GIS
Introduction to GIS
 
Gdal introduction
Gdal introductionGdal introduction
Gdal introduction
 
Intro to Spatial data
Intro to Spatial data Intro to Spatial data
Intro to Spatial data
 
Mapping, GIS and geolocating data in Java
Mapping, GIS and geolocating data in JavaMapping, GIS and geolocating data in Java
Mapping, GIS and geolocating data in Java
 
Introduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUGIntroduction To Apache Pig at WHUG
Introduction To Apache Pig at WHUG
 
My experience and suggestions as a web-GIS developer
My experience and suggestions as a web-GIS developerMy experience and suggestions as a web-GIS developer
My experience and suggestions as a web-GIS developer
 
Pycon 2012 Taiwan
Pycon 2012 TaiwanPycon 2012 Taiwan
Pycon 2012 Taiwan
 

More from Kudos S.A.S

AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...
AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...
AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...Kudos S.A.S
 
Miradas al exterior.
Miradas al exterior. Miradas al exterior.
Miradas al exterior. Kudos S.A.S
 
Bringing GEOSS services into Practice for Beginners: GeoNode Tutorial
Bringing GEOSS services into Practice for Beginners: GeoNode TutorialBringing GEOSS services into Practice for Beginners: GeoNode Tutorial
Bringing GEOSS services into Practice for Beginners: GeoNode TutorialKudos S.A.S
 
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...Kudos S.A.S
 
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en Masa
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en MasaProyecto Mapa Nacional de Amenaza Relativa por Movimientos en Masa
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en MasaKudos S.A.S
 
Presentación software libre v2
Presentación software libre v2Presentación software libre v2
Presentación software libre v2Kudos S.A.S
 
A Web Application Designed to Publish Information of Surface Manifestations o...
A Web Application Designed to Publish Information of Surface Manifestations o...A Web Application Designed to Publish Information of Surface Manifestations o...
A Web Application Designed to Publish Information of Surface Manifestations o...Kudos S.A.S
 
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013Kudos S.A.S
 
Conferencias ACIS: Análisis y visualización de información georreferenciada u...
Conferencias ACIS: Análisis y visualización de información georreferenciada u...Conferencias ACIS: Análisis y visualización de información georreferenciada u...
Conferencias ACIS: Análisis y visualización de información georreferenciada u...Kudos S.A.S
 
Implementación de Fábricas de Software en el Sector Público Colombiano
Implementación de Fábricas de Software en el Sector Público ColombianoImplementación de Fábricas de Software en el Sector Público Colombiano
Implementación de Fábricas de Software en el Sector Público ColombianoKudos S.A.S
 
Presentacion SNC : Sistema Nacional Catastral de Colombia
Presentacion SNC : Sistema Nacional Catastral de ColombiaPresentacion SNC : Sistema Nacional Catastral de Colombia
Presentacion SNC : Sistema Nacional Catastral de ColombiaKudos S.A.S
 
Analizando y visualizando información geoespacial utilizando cloud computing
Analizando y visualizando información geoespacial utilizando cloud computingAnalizando y visualizando información geoespacial utilizando cloud computing
Analizando y visualizando información geoespacial utilizando cloud computingKudos S.A.S
 
Manejo de información raster_en_post_gis-2.0-
Manejo de información raster_en_post_gis-2.0-Manejo de información raster_en_post_gis-2.0-
Manejo de información raster_en_post_gis-2.0-Kudos S.A.S
 
Crowd-sourcing free and open geographic data: information poverty solutions f...
Crowd-sourcing free and open geographic data: information poverty solutions f...Crowd-sourcing free and open geographic data: information poverty solutions f...
Crowd-sourcing free and open geographic data: information poverty solutions f...Kudos S.A.S
 
Software libre y Sistemas de Información Geográfica
Software libre y Sistemas de Información GeográficaSoftware libre y Sistemas de Información Geográfica
Software libre y Sistemas de Información GeográficaKudos S.A.S
 
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...Kudos S.A.S
 
Kudos Flashcamp Bogota 2009
Kudos Flashcamp Bogota 2009Kudos Flashcamp Bogota 2009
Kudos Flashcamp Bogota 2009Kudos S.A.S
 
Geo Marketing, ¿Herramienta o Gadget?:
Geo Marketing, ¿Herramienta o Gadget?: Geo Marketing, ¿Herramienta o Gadget?:
Geo Marketing, ¿Herramienta o Gadget?: Kudos S.A.S
 
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...Kudos S.A.S
 
Fundamentos VB - VBA
Fundamentos VB - VBAFundamentos VB - VBA
Fundamentos VB - VBAKudos S.A.S
 

More from Kudos S.A.S (20)

AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...
AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...
AWS - Ingrese a la era de las bases de datos personalizadas: Encuentre la bas...
 
Miradas al exterior.
Miradas al exterior. Miradas al exterior.
Miradas al exterior.
 
Bringing GEOSS services into Practice for Beginners: GeoNode Tutorial
Bringing GEOSS services into Practice for Beginners: GeoNode TutorialBringing GEOSS services into Practice for Beginners: GeoNode Tutorial
Bringing GEOSS services into Practice for Beginners: GeoNode Tutorial
 
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...
Uso de Tecnologías Geoespaciales y Big Data para mejorar los procesos de actu...
 
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en Masa
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en MasaProyecto Mapa Nacional de Amenaza Relativa por Movimientos en Masa
Proyecto Mapa Nacional de Amenaza Relativa por Movimientos en Masa
 
Presentación software libre v2
Presentación software libre v2Presentación software libre v2
Presentación software libre v2
 
A Web Application Designed to Publish Information of Surface Manifestations o...
A Web Application Designed to Publish Information of Surface Manifestations o...A Web Application Designed to Publish Information of Surface Manifestations o...
A Web Application Designed to Publish Information of Surface Manifestations o...
 
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013
Portafolio Empresarial Egresados Universidad Nacional de Colombia 2013
 
Conferencias ACIS: Análisis y visualización de información georreferenciada u...
Conferencias ACIS: Análisis y visualización de información georreferenciada u...Conferencias ACIS: Análisis y visualización de información georreferenciada u...
Conferencias ACIS: Análisis y visualización de información georreferenciada u...
 
Implementación de Fábricas de Software en el Sector Público Colombiano
Implementación de Fábricas de Software en el Sector Público ColombianoImplementación de Fábricas de Software en el Sector Público Colombiano
Implementación de Fábricas de Software en el Sector Público Colombiano
 
Presentacion SNC : Sistema Nacional Catastral de Colombia
Presentacion SNC : Sistema Nacional Catastral de ColombiaPresentacion SNC : Sistema Nacional Catastral de Colombia
Presentacion SNC : Sistema Nacional Catastral de Colombia
 
Analizando y visualizando información geoespacial utilizando cloud computing
Analizando y visualizando información geoespacial utilizando cloud computingAnalizando y visualizando información geoespacial utilizando cloud computing
Analizando y visualizando información geoespacial utilizando cloud computing
 
Manejo de información raster_en_post_gis-2.0-
Manejo de información raster_en_post_gis-2.0-Manejo de información raster_en_post_gis-2.0-
Manejo de información raster_en_post_gis-2.0-
 
Crowd-sourcing free and open geographic data: information poverty solutions f...
Crowd-sourcing free and open geographic data: information poverty solutions f...Crowd-sourcing free and open geographic data: information poverty solutions f...
Crowd-sourcing free and open geographic data: information poverty solutions f...
 
Software libre y Sistemas de Información Geográfica
Software libre y Sistemas de Información GeográficaSoftware libre y Sistemas de Información Geográfica
Software libre y Sistemas de Información Geográfica
 
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...
Modelamiento en Web de Geoinformación de la Amazonía Colombiana con el uso de...
 
Kudos Flashcamp Bogota 2009
Kudos Flashcamp Bogota 2009Kudos Flashcamp Bogota 2009
Kudos Flashcamp Bogota 2009
 
Geo Marketing, ¿Herramienta o Gadget?:
Geo Marketing, ¿Herramienta o Gadget?: Geo Marketing, ¿Herramienta o Gadget?:
Geo Marketing, ¿Herramienta o Gadget?:
 
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...
Integración de Adobe Flex y Google Maps: Aplicaciones Geográficas Enriquecida...
 
Fundamentos VB - VBA
Fundamentos VB - VBAFundamentos VB - VBA
Fundamentos VB - VBA
 

Recently uploaded

Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageDista
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadIvo Andreev
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdfMeon Technology
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsJaydeep Chhasatia
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Jaydeep Chhasatia
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024Mind IT Systems
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorShane Coughlan
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyRaymond Okyere-Forson
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesSoftwareMill
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLAlluxio, Inc.
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampVICTOR MAESTRE RAMIREZ
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...OnePlan Solutions
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesShyamsundar Das
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmonyelliciumsolutionspun
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfTobias Schneck
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntelliSource Technologies
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIIvo Andreev
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxAutus Cyber Tech
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Incrobinwilliams8624
 

Recently uploaded (20)

Sales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales CoverageSales Territory Management: A Definitive Guide to Expand Sales Coverage
Sales Territory Management: A Definitive Guide to Expand Sales Coverage
 
Cybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and BadCybersecurity Challenges with Generative AI - for Good and Bad
Cybersecurity Challenges with Generative AI - for Good and Bad
 
online pdf editor software solutions.pdf
online pdf editor software solutions.pdfonline pdf editor software solutions.pdf
online pdf editor software solutions.pdf
 
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software TeamsYour Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
Your Vision, Our Expertise: TECUNIQUE's Tailored Software Teams
 
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
Optimizing Business Potential: A Guide to Outsourcing Engineering Services in...
 
Top Software Development Trends in 2024
Top Software Development Trends in  2024Top Software Development Trends in  2024
Top Software Development Trends in 2024
 
OpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS CalculatorOpenChain Webinar: Universal CVSS Calculator
OpenChain Webinar: Universal CVSS Calculator
 
AI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human BeautyAI Embracing Every Shade of Human Beauty
AI Embracing Every Shade of Human Beauty
 
Growing Oxen: channel operators and retries
Growing Oxen: channel operators and retriesGrowing Oxen: channel operators and retries
Growing Oxen: channel operators and retries
 
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/MLBig Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
Big Data Bellevue Meetup | Enhancing Python Data Loading in the Cloud for AI/ML
 
Kawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in TrivandrumKawika Technologies pvt ltd Software Development Company in Trivandrum
Kawika Technologies pvt ltd Software Development Company in Trivandrum
 
Deep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - DatacampDeep Learning for Images with PyTorch - Datacamp
Deep Learning for Images with PyTorch - Datacamp
 
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
Transforming PMO Success with AI - Discover OnePlan Strategic Portfolio Work ...
 
Watermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security ChallengesWatermarking in Source Code: Applications and Security Challenges
Watermarking in Source Code: Applications and Security Challenges
 
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine HarmonyLeveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
Leveraging DxSherpa's Generative AI Services to Unlock Human-Machine Harmony
 
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdfARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
ARM Talk @ Rejekts - Will ARM be the new Mainstream in our Data Centers_.pdf
 
Introduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptxIntroduction-to-Software-Development-Outsourcing.pptx
Introduction-to-Software-Development-Outsourcing.pptx
 
JS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AIJS-Experts - Cybersecurity for Generative AI
JS-Experts - Cybersecurity for Generative AI
 
ERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptxERP For Electrical and Electronics manufecturing.pptx
ERP For Electrical and Electronics manufecturing.pptx
 
Enterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze IncEnterprise Document Management System - Qualityze Inc
Enterprise Document Management System - Qualityze Inc
 

Using python to analyze spatial data

  • 1. Using Python to Analyze Spatial Data Juan Carlos Méndez CTO @gkudos juan@gkudos.com
  • 2. Topics ● Spatial Data? ● Python? ● Visualizing Geodata ● Analyzing Geodata (Basics)
  • 3. About me ● Juan Carlos Méndez ○ CTO at @gkudos ○ GIS Consultant ○ Software Architect / Programmer / “Data Engineer” ○ https://github.com/dersteppenwolf ● Education ○ Systems Engineer- Universidad Nacional de Colombia, Bogotá ○ Telematics / eBusiness specialist - Escuela Colombiana de ingeniería ○ Information Engineering (Student) - Universidad de los Andes
  • 4. https://github.com/dersteppenwolf/pycon Requirements ● Qgis ( http://www.qgis.org/es/site/ ) pip install -r requirements.txt brew install mapnik brew install spatialindex
  • 6. Location, location, location! “You can buy the right home in the wrong location. You can change the structure, remodel it or alter the home's layout but, ordinarily, you cannot move it. It's attached to the land” http://bit.ly/2fz2ySD
  • 7. Spatio-temporal data “...in Google, about 25 PB of data is being generated per day, and a significant portion of the data falls into the realm of spatio-temporal data...” Lee, J.-G., & Kang, M. (2015). Geospatial Big Data: Challenges and Opportunities. Big Data Research, 2(2), 74–81. http://doi.org/10.1016/j.bdr.2015.01.003
  • 11. 15 Global challenges facing humanity http://www.millennium-project.org/ millennium/challeng.html
  • 12. How?
  • 18. Spatial Data ● Located on the surface of the earth ● Coordinate Systems GIS (Geographic Information Systems) ● Body of Knowledge ● Tools ● Science
  • 19. Spatial is special? Yes: ● Multidimensional ● Voluminous ● Special methods for analysis ● Updating: Slow, complex and expensive “Everything is related to everything else, but near things are more related than distant things” Tobler, W. 1970. A
  • 20. Spatial is special? No: ● “spatial is not special, it’s just another column in the database...” Michael Terner http://bit.ly/2jZAWav
  • 21. Spatial Data: Vector vs Raster
  • 23. Spatial Data: Vector vs Raster
  • 24. Map projection “...formal process which converts features between a spherical or ellipsoidal surface and a projection surface, which is often flat…” http://bit.ly/2bA9Szk
  • 27. Spatial Data - File Formats ...lots… (open and proprietary)... ● Shapefile ● Geojson ● KML ● Filegeodatabase ● Geotiff ● Geopackage ● Spatialite ● … ...
  • 28. Spatial is special? ...It can be complex... … using python it will be more fun and easy…
  • 30. Python for Geospatial Data Lots of… ● Tools / Libraries ● Propietary / Open ● Desktop / Server ● Analysis / Visualization / ETL
  • 31. Python for Geospatial Data ESRI ● Arcpy ○ Desktop: Automation / Customization. E.g. CartoDB Toolbox : Import Data From to Carto ○ Server: Geoprocessing as a “Web Service” ● Arcgis Web : ArcGIS API for Python (2017) QGIS ● Open source desktop GIS Tool written in C++, Python, Qt ● QGIS API E.g. CartoDB Plugin for QGis ● PyQGIS: Scripting using Python ● Server: Qgis Server python plugins
  • 32. Python for Geospatial Data ● CKAN - web-based open source management system for the storage and distribution of open data (including geospatial data). ● GeoDjango - storing and manipulating geographic data using the Django ORM ● Geonode - web-based application and platform for developing geospatial information systems (GIS) and for deploying spatial data infrastructures (SDI)
  • 33. Python for Geospatial Data ● pyshp - For reading and writing shapefiles (in pure Python) ● pyproj - For conversions between projections ● shapely - For geometry handling ● fiona - For making it easy to read/write geospatial data formats ● ogr/gdal - For reading, writing, and transforming geospatial data formats * ● Rasterio - reads and writes geospatial raster datasets
  • 34. Show me the code
  • 40. Fiona ● https://github.com/Toblerity/Fiona ● Fiona is OGR's neat and nimble API for Python programmers. ● Fiona does reading and writing data formats. For this it uses OGR, the most popular open-source conversion system ● The OGR Simple Features Library is a C++ open source library providing read (and sometimes write) access to a variety of vector file formats including ESRI Shapefiles, S-57, SDTS, PostGIS, Oracle Spatial, and Mapinfo mid/mif and TAB formats.
  • 43. From CSV to SHP
  • 44. Shapely ● https://github.com/Toblerity/Shapely ● Manipulation and analysis of geometric objects in the Cartesian plane ● With Shapely, you can do things like buffers, unions, intersections, centroids, convex hulls, and lots more.
  • 48. What is geocoding? ● Geocoding is the process of transforming a description of a location—such as a pair of coordinates, an address, or a name of a place—to a location on the earth's surface. http://arcg.is/2kUedk7
  • 49. GeoPy ● https://github.com/geopy/geopy ● geopy makes it easy for Python developers to locate the coordinates of addresses, cities, countries, and landmarks across the globe using third-party geocoders and other data sources ● geopy includes geocoder classes for the OpenStreetMap Nominatim, ESRI ArcGIS, Google Geocoding API (V3), Baidu Maps, Bing Maps API, Mapzen Search, Yandex, IGN France, GeoNames, NaviData, OpenMapQuest, What3Words, OpenCage, SmartyStreets, geocoder.us, and GeocodeFarm geocoder services.
  • 54. Mapnik ● http://mapnik.org/ ● the core of geospatial visualization & processing ● mapnik combines pixel-perfect image output with lightning-fast cartographic algorithms, and exposes interfaces in C++, Python, and Node.
  • 55. Mapnik ● Installing Mapnik on OS X with Homebrew https://github.com/mapnik/mapnik/wiki/MacInstallation_Homebrew ○ brew install mapnik ● Python bindings for mapnik https://github.com/mapnik/python-mapnik ● Stacks built with mapnik: OpenStreetMap , Mapbox , CartoDB , Stamen , MapQuest , Kosmtik
  • 64. Mapnik Composite Compositing operations affect the way colors and textures of different elements and styles interact with each other. Two main categories: color and alpha E.g. Multiply literally multiplies the color of the top layer by the color of each layer beneath, which usually means overlapping areas become darker.
  • 66. What is Spatial Analysis?
  • 67. Spatial Analysis Geospatial data is more than maps! What is Geoprocessing:? ● Geoprocessing is any GIS operation used to manipulate data. ● A typical geoprocessing operation takes an input dataset, performs an operation on that dataset, and returns the result of the operation as an output dataset, also referred to as derived data. ● Common geoprocessing operations: geographic feature overlay, feature selection and analysis, topology processing, and data conversion. ● Geoprocessing allows you to define, manage, and analyze geographic information used to make decisions. ● http://bit.ly/2k8l3P8
  • 68. Spatial Analysis ● Spatial analysis includes any of the formal techniques which study entities using their topological, geometric, or geographic properties.
  • 72. Jupyter ● http://jupyter.org/ ● web application that allows you to create and share documents that contain live code, equations, visualizations and explanatory text ● Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more
  • 73. Geopandas ● https://github.com/geopandas/geopandas ● pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language. ● GeoPandas is a project to add support for geographic data to pandas objects.
  • 74. PySAL ● https://github.com/pysal/pysal ● PySAL: Python Spatial Analysis Library ● Sub-packages pysal.cg – Computational Geometry pysal.core — Core Data Structures and IO pysal.esda — Exploratory Spatial Data Analysis pysal.inequality — Spatial Inequality Analysis pysal.region — Spatially Constrained Clustering pysal.spatial_dynamics — Spatial Dynamics pysal.spreg — Regression and Diagnostics pysal.weights — Spatial Weights pysal.network — Network Constrained Analysis pysal.contrib – Contributed Modules
  • 79. Geoprocessing ● Spatial Joins ● Data Clipping ● Data Enrichment ● Data filtering ● Spatial Analysis
  • 80. data/ideca/Loca.shp Localidades, Bogotá data/ideca/sitios_interes.shp Sitios de Interés, Bogotá data/ideca/clasificacion_sitios_interes.csv