The document introduces Weather Underground's personal weather station (PWS) network and making your own weather station. It discusses how Weather Underground ingests and quality checks data from over 200,000 PWSs around the world before presenting it on their website and apps. It also provides guidance on building a basic weather station using inexpensive sensors and a microcontroller, and outlines some of the technical challenges involved in collecting and transmitting reliable weather data.
11. What is a Weather Station
Traditional stations
Qualitative reporting (crowd reports)
Image recognition
Phone Sensors
Car sensors
Maker Station
11
12. Weather hungry data monsters
To serve globally we need more data
-Engage with local met offices (if they exist)
-Engage with education/maker community
More data, better data = better forecasts.
12
13. Roll your own
Open source weather stations make IoT and
weather more available/flexible for local
needs
Can be part of an education program
13
14. What does it take
1.Sensor (Temp, precip, humidity, uv, etc)
2.Controller (arduino, particle, etc)
3.Memory and/or
Transmitter (flash,wifi, cellular)
4.Power (solar, battery, mains)
14
16. Station challenges
Biggest contributor to data variation:
Enclosure design
The Effectiveness of the ASOS, MMTS, Gill, and CRS Air Temperature Radiation Shields: K. G. Hubbard, X. Lin, and E. A. Walter-Shea 16
17. Tiny wifi
Tiny wifi connected station
limited battery life
Used to monitor terrarium
17
18. Ol faithful
Good reliability, online over a year
Solar and battery powered
Enclosure made from
~$6 garden supplies
Particle Photon (WiFi)
Spark Fun Weather Shield
-HTU21D humidity sensor
-MPL3115A2 pressure sensor
18
19. Cell-o there
Particle Electron: cell radio + microcontroller
BMP280: temp, humidity, pressure sensor
Enclosure made from a painted soda cup
Data is good if kept in shade however:
no venting = heat buildup
ok proof of concept, needs refinement
19
21. Ingestion
Rapidfire
● Ingests and stores data reported at rates as fast as one observation
every 2 seconds
● Stores data in current condition file, records history data at as high
resolution as once every 5 seconds
21
25. The QC Checks
● Range Check
● Stuck Sensor Check
● Neighbor Check
25
26. Range Check
Have these readings ever happened on Earth?
Temperature < -130º F or > 135º F.
Dew Point < -90º F or > 90º F.
Wind Speed < 0 mph or > 279 mph.
Wind Direction < 0º or > 360º.
Pressure < 846 inHg or > 1100 inHg.
26
27. Stuck Sensor Check
Has the temperature changed in the past 6 hours?
● by at least 0.1°F
● lack of change is often an indication of
other stuck sensors as well
27
28. Neighbor Check
Is the temperature of this station similar to the majority of stations nearby?
● collect sensors in 15 km of current sensor
● find clusters divided by 3° F
● determine majority cluster(s)
● throw out statistical outliers
Most essential customer-facing check
28
30. The Next Step - QC on Ingest
● Current QC
○ cycle is 15 minutes, allowing bad observations to linger on the site
and apps during that time
○ written in difficult to maintain and extend multi-threaded C++ code
● IBM Streams + QC
○ clean obs all the time
○ written in single threaded Python with better performance, stability,
extensibility, third-party libraries like Spark, and support for modern
technologies like JSON and REST
30
34. Uptime: 99.95 %
Latency ~25 ms
Autoscale to 20B requests per day
Scalability
Average 10s of Billions requests per day
Global Coverage
(US East, US West, EU, Asia)
Partial DeploymentsVersioned artifacts
and rollbacks
Faster code to prod:
Less dependency b/w teams
Your favorite tech /
language here
34
35. Architecture: Storage Polyglot
Real time data
and caching
Historical weather data
Data Migration
Gateway Data
Analytics
Archives
Images
Videos
Analytics
Informatica
Drupal
35