5. What is LOFAR
● Low-Frequency Array
● Distributed sensor network
● No dishes
● Simple and cheap antenna's
● Omni-directional
● Concurrent observations
● Software telescope
– Flexible
– Requires supercomputer
6. What is LOFAR
Biggest Radio telescope
array in the world
● 7000 antennas
● 10 to 80 MHz (LBA)
● 120 to 240 MHz (HBA)
● Multiple small stations
● NL, DE, UK, FR, SE
● New (unique) design
7. New design
● Many unsolved problems
● Multi-terabit/s data scale
● How to
– Process
– Store
– Distribute
– Make results useful
● Small scale example for SKA
14. Transient Key Project
● Static sources are boring
●
Transient – something that changes
● Detect sources
●
Store extracted information
●
Associate with existing observations
●
Did something change?
●
This is where the Python fun starts!
15. ● The glue
● Distributing computation
● Image processing
● Source extraction
● Source association
● Database interactions
16. Why Python
● A lot of existing code
● Good (scientific) libraries
● Good astronomical libraries
● Easy to use
● Get it working first, then fast
● NumPy is quite fast when used correctly
17. What do we use?
● Python
● NumPy / SciPy
● Django
● MonetDB
● Pyrap (casacore wrapper)
– Does all astro shizzle you can imagine
18. Distributed Computation
● Home made (ASTRON)
●
SSH based
●
Calling remote scripts
● Shared code base (NFS)
● Protocol for passing arguments and results
●
Uses logging module a lot
●
Difficult to debug
● Difficult to profile
●
Hadoop probably better
19. The Lofar Cluster
● 100 nodes
● 24 cores each
● 20 TB raid5 storage
● 40 Gbps InfiniBand
● But for AARTFAAC we build our own
● Eventually
20. Main loop
● Extract source properties
● Store properties
● Do association
● Detect transients
21. The data
● 1 datacube per second
● 10 frequency bands
● In the future 10 images per second
● In the future 4 different polarization
● Non stop
25. Database
● Move calculation to the data
● Independent data
● Naturally separable by sky coordinates
● ~100 TB/year
● 10.000 insert/second
26. MonetDB
● Don't believe the hype
● NoNoSQL
● Relational
● Column store DB
● Fast
● Auto tuning!
● Developers next door (CWI)
● Cross research funding
27. INSERT INTO tempbasesources
(xtrsrc_id
,datapoints
,I_peak_sum
,I_peak_sq_sum
,weight_peak_sum
,weight_I_peak_sum
,weight_I_peak_sq_sum
)
SELECT b0.xtrsrc_id
,b0.datapoints
+ 1 AS datapoints
,b0.I_peak_sum
+ x0.I_peak AS i_peak_sum
,b0.I_peak_sq_sum
+ x0.I_peak * x0.I_peak AS i_peak_sq_sum
,b0.weight_peak_sum
+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum
,b0.weight_I_peak_sum
+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sum
,b0.weight_I_peak_sq_sum
+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sq_sum
FROM basesources b0
,extractedsources x0
WHERE x0.image_id = @imageid
AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight
) AS INTEGER)
AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight
) AS INTEGER)
AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)
+(x0.y - b0.y)*(x0.y - b0.y)
+(x0.z - b0.z)*(x0.z - b0.z)
)/2
)
/
SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err
+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)
< @assoc_r;
32. VO events
● Standardized language
● Report observations of astronomical events
● Hey world, check this supernova out over
there
● http://comet.transientskp.org
