Testing Ember Apps: Full-stack integration testing with Capybara (Selenium & friends); client-side integration testing with Konacha (Mocha & friends).

1. Testing Ember Apps
Jo Liss
https://twitter.com/jo_liss
Hi, I’m Jo, I’m on the
http://solitr.com/blog
Capybara and Konacha
core teams, and I’m an
Ember contributor.
This talk will be more an
architectural over view
than a tutorial. I’ll
assume Rails in some
places, but the concepts
apply to other backends
as well.
Slides licensed under CC BY 3.0.

2. This presentation has 2 parts:
Part 1: Full-Stack Integration Tests
Capybara similar: PhantomJS

3. Part 1: Full-Stack Integration Tests
Capybara
Part 2: Client-Side Integration Tests
Konacha similar: QUnit, Mocha, Jasmine
I like to test apps with a
combination of Capybara
and Konacha tests.
I noticed that having a
good test suite makes me
*much* more productive.

4. Part 1: Full-Stack Integration Tests
Capybara
It’s not very hard to get
Capybara working right.
So I want to focus on
high-level architecture
instead. I think that’s
sometimes under-
appreciated.

5. Capybara
Quick refresher:
“Selenium for Rails”
You can plug in different Q: Who has used Capybara to
drivers, like WebKit. All test a JavaScript app?
of the following applies ------->
to any driver. Q: Who ran into problems with
either performance, or
flickering tests (brittleness)?

6. Capybara
These pains are architectural.
These pains are because of the
architecture.

7. Capybara
These pains are architectural.
Performance :-(
Brittleness :-(
There are probably limits So let’s talk about why
to how fast you can this happens.
make Capybara tests.
And you can hunt down
flickering tests (and you
should), but they will still
pop up occasionally.

8. So you have a test suite
written in Ruby, and it
talks to a DB backend.
Test suite
DB

9. Firefox + So when you have a
Capybara test, Capybara
will automatically spin up a
Selenium Firefox for you, and your
test suite will send requests
to that Firefox.
Test suite
But Selenium commands are mostly
non-blocking, i.e. they return before
they finish. So when you have a
DB sequence of commands, even with
Ajax, Capybara does some intelligent
polling to make them appear
synchronous.
This works transparent 99% of the
time, and it makes for very readable
test code, but very occasionally, you’ll
end up with a race condition, and then
you have to understand what’s going
on underneath.
So let me double that line to indicate
that it’s async.

10. So who does the Firefox make
Firefox + HTTP requests to? Clearly, Ruby is
busy with the test suite.
Selenium Capybara actually spins up
another ser ver thread.
Test suite
DB

11. Firefox +
Selenium
Server Test suite
So now Firefox has a
server to load the pages DB
from.
If there’s Ajax requests,
this whole thing becomes
asynchronous. So that’s
another source of race
conditions. Let me double
that line...

12. Firefox +
Selenium
Server Test suite
And of course the server
talks to the database -- DB
another line...

13. Firefox +
Selenium
Server Test suite
Which means you also This is clearly fundamentally quite complex
have to be careful not to
have the t wo Ruby DB and error-prone.
threads interfere with I’m not trying to convince you that Capybara
each other while they is a bad tool. But I think it’s actually useful to
access the database. understand this architecture.
By the way, just to be clear, that’s not
Selenium’s fault. If you drop in WebKit or
PhantomJS into Capybara, it suffers from the
same issues.

14. Firefox +
Selenium
At the moment, this kind of architecture is
the only solution to give us a full-stack
integration test, database to DOM. So it’s
actually very useful.
Server If you don’t use Rails, I’d definitely Test suite
recommend finding a comparable solution,
or hand-rolling one.
DB

15. Capybara
Powerful but clunky
So one solid strategy ...

16. ... is limiting yourself to
exercising one happy path for
Capybara
each model, to make sure that
stuff in the DB ends up in the
DOM, and vice versa.
Powerful but clunky
Strategy: Only test every model once
DB-to-DOM, read and write

17. And then we can use pure client-
side integration testing to get
Capybara
more fine-grained testing.
Powerful but clunky
Strategy: Only test every model once
DB-to-DOM, read and write
Move finer-grained tests to client side

18. Part 2: Client-Side Integration Tests
with Konacha
The idea is to limit the
architectural complexity.
No backend server, no DB,
test runs directly in the
browser, not in a separate
process.
This makes it faster, and
extremely reliable.

19. Konacha
Rails gem packaging
Konacha is pretty
Mocha.js testing framework + simple, so if you’re not on
Rails, you can easily
hand-roll an equivalent
test setup.
Chai.js assertion lib
Let me show you what
Mocha and Chai do...

20. Mocha + Chai
describe 'todo list', ->
it 'can add items', ->
Mocha is a testing
framework, like QUnit or
Jasmine, or RSpec on
Ruby.
But Mocha doesn’t do
assertions, so you
typically combine it with
the Chai assertion
library.

21. Mocha + Chai
describe 'todo list' ->
it 'can add items', ->
...
$('.todo').should
.contain 'Buy milk'

22. Interlude: Konacha Video
Before I go on, let me show you
what we’re trying to achieve.
http://www.youtube.com/watch?v=heK78M6Ql9Q
So we basically just hit Cmd+R on
Konacha’s development ser ver.
Konacha automatically runs your tests in
an iframe, and loads your application.css.
That’s also really easy to do yourself if
you don’t use Rails with Konacha.
This is also really useful to ramp up people
to Ember, because you can visually see
what your test does. And if a test fails,
you can just click into the iframe and
interact with the application.

23. Konacha
In the browser (Cmd+R) for dev
rake task for CI CI task through Capybara

24. Why is Konacha fast?
No page loads You don’t have to serve and parse the entire
stack of assets for every test case.

25. Why is Konacha fast?
No page loads
100% synchronous No polling, no waiting.

26. Why is Konacha fast?
No page loads
100% synchronous
No stack Most expensive thing is the DOM.

27. Unit vs. Integration
What you just saw I call “client-side
integration tests”. They don’t involve the
backend, but on the frontend, they exercise
the entire Ember app.

28. Unit vs. Integration
Lots of simple layers ==> integration
tests win.
On Ember, I avoid unit tests. The reason is, the
individual layers of an Ember app are very
simple. What I care about is whether they play
together. Unit tests tend to be very “weak”, i.e.
they’ll just keep passing even when your app
Okay, bread’n’butter
breaks.
time.
As a rule of thumb, I tend to do zero unit
testing for Ember apps.

29. Ember setup
No “official” support for testing.
So we wing it. A bunch of pieces are
missing to make this
easy like with Rails.
(Notably, good fixture
handling, package
manager, ...)
Here’s some practical
tips:

30. There’s no repeated app
instantiation yet, so I
like to start up the
Starting the app
application at the
beginning of the entire
test suite.

31. Starting the app
# Parse time: We call
App.deferReadiness, and
advance readiness it
App.deferReadiness() once you want the app to
run.
# Global before all:
before -> UPDATE Feb 26 2013:
App.advanceReadiness() Instead you can now Enable
Ember.testing *before*
parsing your app code, and
kick it off with “before ->
App.initialize()”; no more
advanceReadiness.

32. Router
We tell the router not to
mess with the URL.
App.Router.reopen
location: 'none'

33. Router
App.Router.reopen
location: 'none'
beforeEach: ->
App.router.transitionTo('...')
Before each test, UPDATE Feb 26 2013:
transition back to the
root state. You can now use
“beforeEach ->
This is a pretty hackish App.reset()”.
way to reset app state,
but for now it’s
surprisingly reliable.

34. Reset store
beforeEach ->
App.store.destroy() if App.store
App.store = DS.Store.create()

35. Konacha: Keep body
Konacha.reset = function() { }
Konacha by default
cleans the <body>
element before each test.
But our app keeps
running bet ween tests,
so we disable this by
nuking Konacha.reset.

36. Runloop & setTimeout
Ember automatically
schedules runloops with
setTimeout. But setTimeout is
Ember.testing = true
the enemy of deterministic
tests. So we disable
automatic runloop creation
by enabling the Ember.testing
flag.
You want to put this before
loading (parsing) your app
modules.

37. Runloop & setTimeout
Ember.testing = true
It’s OK to have Em.run everywhere:
Em.run => foo.set ...
Em.run => $(...).click() Most actions have their
effects deferred to the
end of the runloop. In test
code you need the effects
immediately, so you wrap
things in Ember.run.
Looks funny, but it’s
nothing to worry
about. :-)

38. Animations
jQuery.fx.off = true
Nothing like this for D3. :-( This is not Ember-related,
but you don’t want
animations, because they
are asynchronous. For
D3,, this may require
support in your
production code. :-(
Probably the trickiest
thing is data fixtures:

39. Model Fixtures
1. Client-side fixtures. You have a choice
whether you write them
in JavaScript or
2. Server-side fixtures. generate them from the
server side, and it’s kind
of a trade off.

40. (1) Client-Side Fixtures
FixtureAdapter (immature)
App.TodoList.FIXTURES = [
{ ... }, { ... }
For client-side fixtures,
] there is a FixtureAdapter
in Ember. It still needs
some love, but we can
probably get it there
pretty soon.
And basically you just
define an array of
fixtures for every model.

41. (1) Client-Side Fixtures
:-) Easy to set up

42. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
But you don’t know if
your fixtures actually
represent the reality of
your backend.

43. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
:-( Fragile You can even have bugs
in your fixtures, where
you don’t set up
bidirectional belongsTo
and hasMany
relationships properly.

44. (1) Client-Side Fixtures
:-) Easy
:-( Goes out of sync with backend
:-( Fragile
:-( Server-side computed attributes
JavaScript is still not very powerful. Oftentimes it’s So the alternative to all
easier in practice to implement computed properties on the this is...
backend side and ser ve them out as read-only attributes.
In one backend I worked on, half of the properties were DB
columns, and half were just methods on the Rails models.
Trying to keep these properties manually updated in your
fixture data is obviously painful.

45. (2) Server-Side Fixtures
rake test:fixtures
1. Write fixtures to DB
2. Generate JSON to fixtures.js

46. (2) Server-Side Fixtures
rake test:fixtures
1. Write fixtures to DB
2. Generate JSON to fixtures.js
Load through RESTAdapter
Load that data in fixtures.js before every test
case, perhaps using your RESTAdapter so you
translate the JSON correctly.

47. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
with no speed penalty

48. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain Compact definitions, esp. w/ FactoryGirl; stuff doesn’t
go out of sync.

49. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain
:-( Usability Generated fixtures file can go stale and you have to regenerate. It’s not bad,
just bothersome.

50. (2) Server-Side Fixtures
:-) Covers models, serializers, adapter
:-) Easy to maintain
:-( Usability
:-( Complex to set up Work to set up. You end up with some
custom code, and it ties tightly into
backend.
I personally think it’s generally worth it,
but it also depends on your specific app.

51. Fixture Woes
Global fixtures :-( Both of these techniques
mean that you have the same
fixture set for the entire test
suite.
want FactoryGirl Ideally we’d build something
like FactoryGirl, but I don’t
think we’re quite there yet.

52. Bonus: JS-driven?
Capybara but in JavaScript?
One thing that
sometimes comes up is,
can we have a full-stack
integration test written
in JavaScript?

53. Firefox +
So instead of having the
Selenium Test suite written in
Ruby...
Server Test suite
DB

54. Firefox w/
JS tests
... we push them into the
browser, and so we avoid
the t wo-threaded problem.
Server
I think that would be
really awesome -- and the
truth is, we just don’t have
the tooling yet to make
that work easily.
Perhaps the biggest hurdle
DB is that we don’t have a way
to reset the database, add
fixtures, and perhaps
query database records
from JavaScript.
But it is probably a good
direction to move towards

55. Q &A
Notes from the talk, thanks to @cgcardona:
Q: With Konacha do you see opportunity for TTD?
A: Not quite TDD (hard with visual stuff), but continuous.
When working with Konacha I wrote tests as I went.
Q: Can you test views in isolation?
A: It’s really tricky to instantiate a view in isolation, Ember
wants a whole app… It’s too ‘unit testy’. It might be
possible.

56. Q &A
Comment: We use ‘Rosie’ (https://github.com/bkeepers/
rosie) for Javascript factories.
Comment: Have you tried VCR for server side things? It’s a
Ruby library that will record request responses so you can
play them back later. We run VCR against a live production
server and generate response data that the tests use.

57. Q &A
Q: A lot of bugs come from asynchronicity. Have you tried
to test that specifically?
A: No I haven’t. My hope is that a lot of these bugs
disappear with ember-data. Mocha allows for asynchronous
tests where you set a callback for when your test is
complete.

58. Thanks for listening!
Jo
https://twitter.com/jo_liss
http://solitr.com/blog