Andrew Fogg from Import.io shares the design of an approach to get data about Christmas jumpers. Video available here https://www.youtube.com/watch?v=upQkfsrrFE4

1. Andrew Fogg
Import.io
the ultimate
christmas jumper?

2. andrew fogg

3. co-founder of import.io

4. import.io allows you to turn other people’s websites
into apis without writing any code

5. ever used beautiful soup or scrapy?

6. use import.io and never do that again

7. yesterday i wrote to emlyn responding to the
request for xmas content

8. i set myself a challenge

9. i promised to stand here and tell you the ultimate
christmas jumper to buy for christmas jumper day

10. wtf

11. i started working on the code this morning

12. the analysis is running as i speak

13. the results are not in yet

14. but i want to share the design with you
of how i did this

15. hopefully it will inspire you
to do some crazy data work over the holidays

16. the first thing to do is to get some jumpers

17. we have a new version of import.io
that requires no training

18. we call it magic

19. i did some googling for “christmas jumpers”

20. found lots of retailers linking to “christmas jumper”
sections on their websites

21. copying these urls and pasting them into magic i
can see that they parse very well

22. magic is also available over an api

23. here is the simplest possible integration in python
(no authentication required, no limit on requests)

24. i passed 10 retailer urls into the magic api

25. magic works with pagination

26. iterating through the pages i was able to literally get
1,000s of christmas jumpers

27. the next thing to do is to analyse the jumpers

28. lots of great image libraries and apis for doing
computer vision things

29. but there is no api or image library for christmas
jumpers (and i was not about to write one myself)

30. i turned instead to a human api

31. amazon’s mechanical turk

32. mechanical turk is a marketplace
for human micro-tasking

33. there are workers and requesters

34. as a requester, you programatically
specify your micro-task

35. you create a simple templated
user interface in HTML

36. upload your dataset
(christmas jumpers in my case)

37. specify a price per micro-task

38. fill your account with money

39. and release your micro tasks to the market

40. there are 500,000 registered workers
programatically available 24 hours a day

41. and let me repeat,
they are available over an api

42. as soon as you release your micro-tasks to the
market, you start getting data back

43. it is most commonly used for things like detecting
adult content in images

44. $9 per hour
a worker can sit there all day and crank through
micro tasks and make a decent wage

45. the questions that i am
asking my workers

46. are there snowflakes?

47. how many reindeer?

48. how many snowmen?

49. is father christmas present?

50. transcribe the message

51. the workers are on this at the moment

52. when they are finished

53. imagine a dataviz!

54. check the import.io blog on 12th december to find
out what the ultimate christmas jumper looks like

55. we are hiring for python developers
– interviews on friday!

56. learn more at http://www.import.io