51. A Year in Review
• Data::Verifier
• 15+ releases and now version 0.47
• Data::Manager
• 2 releases, could use more. 0.07
• Message::Stack
• Stable API, simple workhorse. 0.19.
52. Where we are now.
• Elegant handling of multiple scopes.
• We know how to use this stuff better.
• The code hasn’t changed that much.
• API is stable.
• Supports more things as needed. Fork it!
70. Scopes
<fieldset>
<legend>[% c.loc('About You') %]</legend>
[%
context.scope = 'person';
text_field({ label => 'Your Name',
name => 'name', hint => 'Your Name',
required => 1 });
text_field({ label => 'Your Email', name => 'email',
hint => 'you@company.com', type=>'email',
required => 1,
more => 'This will be your username.' });
context.scope = 'identity';
text_field({ label => 'Password', type => 'password',
name => 'password', required => 1 });
text_field({ label => 'Confirm Password',
type => 'password',
name => 'confirm_password', required => 1 });
%]
</fieldset>
71. Just markup.
• Easily define the layout form.
• Macros make it sensible.
• Markup for forms can be provided by designers
and easily adapted.
• Multiple macro “styles” can be included.
72. We have the biggest Macros
of them all.
• Too big to put on a slide, so they’re on github.
• Well defined.
• Easy to use.
• Not that hard to update.
73. The gist of the macro
• Too big to put on a slide (288L).
• Ask me to see them, happy to share.
• Looks at context.scope, determines:
• Message::Stack scope.
• Data::Verifier::Results
76. Priorities:
• Good Ideal user experience.
• Maximize reliability.
• Optimize for use, hide the complexity.
77. CRUD ≠ User Experience
• CRUD principles are awesome.
78. CRUD ≠ User Experience
• CRUD principles are awesome.
• Behind the scene.
79. CRUD ≠ User Experience
• CRUD principles are awesome.
• Behind the scene.
• Many forms encompass multiple objects.
80. CRUD ≠ User Experience
• CRUD principles are awesome.
• Behind the scene.
• Many forms encompass multiple objects.
• Making a user view 3 pages to update 3 objects
is bad form.
89. Optimize for use
• Write code behind a clean API.
• Make methods that do smaller work.
• That gives us easy code.
• This is a different topic.
90. Optimized for use:
# In your controller: (see Catalyst::Action::REST for *_POST)
sub root_POST {
my ( $self, $c ) = @_;
my $data = $c->req->params;
my $results = $c->model(‘DataManager’)->verify($scope,
$data);
unless ( $results->success ) {
$c->message({
type => ‘error’, message => ‘You Fail‘
});
$c->res->redirect(
$c->uri_for_action('/object/create_form') );
$c->detach;
}
# Success, store from $results, verify and redirect.
}
91. Simple Model
package MyApp::Model::DataManager;
use Moose;
with 'Catalyst::Component::InstancePerContext';
has 'profiles' => ( ... );
sub build_per_context_instance {
my ( $self, $c ) = @_;
my $dm = Data::Manager->new;
foreach my $scope ( $self->scopes ) {
$dm->set_verifier(
$scope => Data::Verifier->new( $self-
>get_profile($scope) )
);
}
$dm;
}
93. Going Further
(with type constraints)
username => {
type => MyEmailType,
required => 1,
coerce => 1,
},
94. Type Checking
• Handles coercion (or not)
• Verifies if data is of type
• Emits “invalid” error in results if invalid.
95. extend Data::Manager
• Add your own methods.
• Make it better.
• Some ideas:
• all_valids, hashref of all valid fields.
• data_for_scope, hashref of fields in scope.
• error_scopes, list of scopes needing
attention.
96. extend Data::Manager
• Setup defaults for verifiers by scope.
• By DBIC result sources.
• By Catalyst models.
• Lots of options.
98. Issues:
• Must filter out ‘$self ’ from the result set.
• Must then load $self.
• Must figure out create vs. update.
99. A simple unique check
my $unique_check = sub {
my $r = shift;
my $user = $r->get_value('username');
my $id = $r->get_value('id');
my $rec = $c->model('DB::User')
->search({ username => $user })->first;
return 1 if not defined $rec;
return 1 if $rec->id == $id;
return 0;
};
101. Difficulty in testing.
• Verifier doesn’t know the database.
• Verifier shouldn’t know the database.
• Modify verification profiles where it makes
sense.
102. Inserting
sub build_per_context_instance {
my ( $self, $c ) = @_;
my $unique_check = sub { ... };
my $dm = Data::Manager->new;
foreach my $scope ( $self->scopes ) {
$dm->set_verifier(
$scope => Data::Verifier->new( $self-
>get_profile($scope) )
);
}
$dm->get_verifier('create_account')
->profile->{username}->{post_check} = $unique_check;
$dm;
}
103. On a user object…
sub build_per_context_instance {
my ( $self, $c ) = @_;
my $unique_check = $c->model(‘DB::User’)->unique_check;
my $dm = Data::Manager->new;
foreach my $scope ( $self->scopes ) {
$dm->set_verifier(
$scope => Data::Verifier->new( $self-
>get_profile($scope) )
);
}
$dm->get_verifier('create_account')
->profile->{username}->{post_check} = $unique_check;
$dm;
}
104. Not ideal
• Can add same check for backend tools
• Access in similar fashion
• Verification profiles are modifiable
105. Still optimized for use:
# Same controller!
sub create_account_POST {
my ( $self, $c ) = @_;
my $data = ‘create_account’;
my $data = $c->req->params;
my $results = $c->model(‘DataManager’)->verify($scope,
$data);
unless ( $results->success ) {
$c->message({
type => ‘error’, message => ‘You Fail‘
});
$c->res->redirect(
$c->uri_for_action('/create_account') );
$c->detach;
}
# Success, store from $results, verify and redirect.
}
109. Thank you.
Please think about your applications.
Think about the problems.
Fix the problems.
Editor's Notes
\n
\n
This is more a story. Partially an adventure story, for those who find building applications adventurous. In the beginning, we built applications. Applications accepted user input via silly things like strings with regular expressions applied to them. Perhaps just a simple, &#x201C;is it there&#x201D; check.\n
This was all really stupid. It was really bad, too. It was stupid and bad.\n
It always bothered me so I&#x2019;m going to talk about what we set out to fix it. When I say we I mostly mean me and Cory Watson. I talked a lot and he wrote good code. I talked more and he told me patches welcome. I then talked and provided patches.\n
And we built something. At first it wasn&#x2019;t very good. It was a tool.\n
And we create tools because they make us better. They help us do our job better, rather. That is their job.\n
The best tools are simple tools. They do one thing and do it very well.\n
The power of tools is not to make a swiss army knife that does everything, but to have everything that is needed to do something. You use them together.\n
There is a word for this. Usually only tools use it.\n
And I have an example of something comprised of many tools that works really well. Catalyst! It&#x2019;s a great framework that holds things together, but the power is that you can use any tool you like with it.\n
It even grew some new tools. Taking things it did that would be better suited by something doing it exclusively. HTTP::Body came about this way and is very useful. Plack uses HTTP::Body. It benefits the ecosystem to have tools do one thing. Even if that one thing is using many tools together.\n
Forms are a tool.\n
Or maybe better said, they were a tool. Forms are changing.\n
So is Perl\n
So is the world.\n
The new world doesn&#x2019;t have this notion of forms. It doesn&#x2019;t have this idea that you load a page to edit a record in a database. It&#x2019;s seamless and connected, the interface may have many faces.\n
The old way of doing things is quickly changing. By changing I mean dying and going away. html5 changes a lot in the way forms are handled. Just local storage changes a lot of what is possible. The file browsing in html5. The very shape of building web applications is changing.\n
And I bring this up because of this very bad assertion.\n
The two have nothing to do with each other. Attempting to bind them together is bad.\n
The world is quickly advancing. Far faster than I thought when I was a child. Now I have children. It&#x2019;s spooky. But faster than that is how quickly technology advances.\n
We&#x2019;re not building apps that talk over old copper wires anymore.\n
We&#x2019;re building apps that are apps that run on devices with no wires.\n
We&#x2019;re building apps that are apps that run on devices with no wires.\n
We&#x2019;re building apps that are apps that run on devices with no wires.\n
We&#x2019;re building apps that are apps that run on devices with no wires.\n
We&#x2019;re building apps that are apps that run on devices with no wires.\n
I don&#x2019;t think our tools have grown with us.\n
It&#x2019;s because we were too committed. There are lots of things a person should be married to.\n
\n
\n
\n
Tools don&#x2019;t make sense. You can be passionate about a tool without marrying it.\n
So just say no, and move on, and improve and grow.\n
How do you tell if you&#x2019;re successful? Well, I think it&#x2019;s pretty easy. If you need to hire someone, like a lawyer, to get away from it you are probably married. If you can just stop talking to it, you&#x2019;re good.\n
Keeping your tools single purpose helps keep this line uncrossed.\n
And now everybody close your eyes and think about the application you are building now. Whatever it is, and imagine in 5 years you are called in to work on it.\n
How many people here cringed? How many people here think back to what they did 5 years ago and laugh? It&#x2019;s because tool-marriage. Applications were architected in such a way they could not grow.\n
It&#x2019;s because programming was itself quite young. We didn&#x2019;t have the knowledge we have today. Now we know better, and we know how to pick tools.\n
First is syntax. Ignore it. We aren&#x2019;t paid to type, we&#x2019;re paid to produce awesome applications. Arguing about syntax and style is foolish. I&#x2019;m pretty sure lawyers don&#x2019;t bitch about the font in their case law books. Rise above such silliness and pick a tool for what it does, what it can do and more importantly what is has been proven to do.\n
Second is to seek out opinions. Opinions are very valid. Sometimes it&#x2019;s just feel, but talk to people. I don&#x2019;t like jquery but I love YUI3. It&#x2019;s my opinion. I can back it with reasoned discourse. Talking it out helps clear things up, sometimes just clarifying a nagging worry that is unable to be articulated.\n
Third is how it uses other tools. Does it include everything itself even if something better exists? Does it tell you how to do everything, even things unrelated to its purpose?\n
If it does, it imposes. Pick tools that don&#x2019;t impose on you. Imposing is one step away from marriage.\n
We went through all that and didn&#x2019;t find anything. So we built our own. We didn&#x2019;t want to, we wanted someone to do something for us.\n
But nobody did, so we first wrote Data::Verifier and then Message::Stack. After that we wrapped it up in Data::Manager. There was much rejoicing. Mostly from our users.\n
Here&#x2019;s a quick recap on the usage for those who missed the talk at YAPC last year or from OPW in January.\n
This is bad messaging. It makes users guess. Don&#x2019;t make your users guess.\n
This is good messaging. It tells people where to look.\n
This is the usage, taken from Cory&#x2019;s original talk on it. It&#x2019;s very simple. You create your verifier, call verify with a hashref, and it gives you a results object. If it&#x2019;s successful, you party.\n
Data::Manager introduces multiple verifiers and a consistent serializable object. This means you can say, &#x201C;Verify this address!&#x201D; and against the same object verify something else.\n
These are called scopes. It&#x2019;s important when you are binding this with messaging.\n
Messages are easy to get. Just ask for them. What are my messages related to address verification?\n
The results are also serializable and just Data::Verifier::Results. It&#x2019;s a decorated hash ref, really. Nothing fancy.\n
Since YAPC last year, we&#x2019;ve still been adding more to it. Data::Verifier has had the most. Message::Stack only had one release which was to make it work better with internationalization.\n
As far as our usage, we&#x2019;ve also done more. The more applications we build the more it does. The better we do. Through the years we&#x2019;ve still never changed the API and haven&#x2019;t needed to. It&#x2019;s on github and we accept patches.\n
Right now we&#x2019;re about half way through. I told you why we built Data::Manager and the current status, but I haven&#x2019;t spoken about how we use it. That&#x2019;s really important because it shows how we build better applications.\n
Right, so back to the story.\n
We used Catalyst and had controllers and we wrote markup by hand like a lot of people do.\n
We had the same problems other people do.\n
Sometimes our users were confused. Sometimes annoyed.\n
Sometimes our developers became confused or annoyed. Sometimes both.\n
Then we would get a designer suggesting some change. We really hated that.\n
We were in a bad relationship.\n
We thought and looked at a lot. We looked at form generators and played with them. Reflectors and all that.\n
We never seemed to find one that worked the way we did. I&#x2019;ll explain more later, but they all were frustrating. I deleted many, many branches after porting to form generators.\n
So we got married (slightly) to our templating engine. Most templating engines support macros to some extent, so I think this is safe. We just use macros.\n
The usage isn&#x2019;t bad. It&#x2019;s not good but there are worse things out there. Like form generators.\n
It renders consistently.\n
We have multiple scopes, which is something we always had trouble with when using form generators. Here we store our person information, stored in a person table.\n
But here we have our user information, stored in an identities table.\n
But we get messaging on both, on the same screen.\n
And it does a good job checking it&#x2019;s not garbage in.\n
But there is no messaging on the macros. There isn&#x2019;t anything more than what we need. We only need to define what scope to look for results and messages in. That&#x2019;s really the only &#x201C;convention&#x201D; thing here. No magic, though.\n
This gives us the ability to create complex and beautiful forms, exactly what the user would expect. Designed by a human, passed off and generated with simple macros. It&#x2019;s easy to change, since you just edit the macros in one place, and it is consistent.\n
I&#x2019;d show you the macros, but they&#x2019;re really big. It&#x2019;s basically broken up in the main structure, the label field, input, a messaging section. I use YUI3 grids for layout and it works really well. I can create all sorts of complex forms that look great.\n
The context.scope just tells the macros where to look. The information is in the stash, just have to get at it. It&#x2019;s what I call &#x201C;sensible defaults&#x201D;. You can override or add messaging explicitly in the macros, or it can fetch it from the results or message::stack.\n
It gives us the three things I think are key to building good applications. I don&#x2019;t know if we&#x2019;ll ever build a Good user experience. Not without a true to form user experience engineer. Those guys are picky *and* expensive. So let&#x2019;s settle for ideal. No surprises, consistent and works. We maximize reliability, we want it to always work the same way. Emails are validated the same way, passwords, too. We optimize it for the developer, developers should want to use these tools. Hide the complexity but it should always be accessible.\n
It gives us the three things I think are key to building good applications. I don&#x2019;t know if we&#x2019;ll ever build a Good user experience. Not without a true to form user experience engineer. Those guys are picky *and* expensive. So let&#x2019;s settle for ideal. No surprises, consistent and works. We maximize reliability, we want it to always work the same way. Emails are validated the same way, passwords, too. We optimize it for the developer, developers should want to use these tools. Hide the complexity but it should always be accessible.\n
It gives us the three things I think are key to building good applications. I don&#x2019;t know if we&#x2019;ll ever build a Good user experience. Not without a true to form user experience engineer. Those guys are picky *and* expensive. So let&#x2019;s settle for ideal. No surprises, consistent and works. We maximize reliability, we want it to always work the same way. Emails are validated the same way, passwords, too. We optimize it for the developer, developers should want to use these tools. Hide the complexity but it should always be accessible.\n
When I&#x2019;m talking about complexity, it&#x2019;s usually for things not complex for geeks. Like CRUD. Very easy. You get a page, you edit a record. That&#x2019;s a really bad experience for the user. The way a user will logically join and separate objects is not how it makes sense in a database. Don&#x2019;t impose this on your users because of inadequate tool usage. This system works for that.\n
When I&#x2019;m talking about complexity, it&#x2019;s usually for things not complex for geeks. Like CRUD. Very easy. You get a page, you edit a record. That&#x2019;s a really bad experience for the user. The way a user will logically join and separate objects is not how it makes sense in a database. Don&#x2019;t impose this on your users because of inadequate tool usage. This system works for that.\n
When I&#x2019;m talking about complexity, it&#x2019;s usually for things not complex for geeks. Like CRUD. Very easy. You get a page, you edit a record. That&#x2019;s a really bad experience for the user. The way a user will logically join and separate objects is not how it makes sense in a database. Don&#x2019;t impose this on your users because of inadequate tool usage. This system works for that.\n
When I&#x2019;m talking about complexity, it&#x2019;s usually for things not complex for geeks. Like CRUD. Very easy. You get a page, you edit a record. That&#x2019;s a really bad experience for the user. The way a user will logically join and separate objects is not how it makes sense in a database. Don&#x2019;t impose this on your users because of inadequate tool usage. This system works for that.\n
I don&#x2019;t like coding the data model first. I go for a UI.\n
I see how the application is going to work and then I pick the tools. It&#x2019;s like test driven development, but more like User Acceptance Test Driven Development. The users of my applications often do so for their jobs, at least in some capacity, and I should respect that. Just like if someone asks me for technical help I expect them to respect my job.\n
Once we do pick these tools the methods fall into place. If you use form generators, usually it&#x2019;s a simple 1 to 1 CRUD system. That&#x2019;s not good for the users.\n
I&#x2019;m going to talk more about reliability now\n
Reliability is best dealt with by not dealing with it. Use tools that do it that you can tap into when you need to. Convention works. Rails does this well. If you don&#x2019;t have to think about making sure your users see the errors, you can focus on making sure the errors happen. And that they happen correctly. You build simple APIs and a plan for how it happens.\n
And this makes your users happy. Or it should, but it may not immediately. Sometimes it takes a bit longer. The key is to learn from your mistakes!\n
The final point is about developers. Optimizing for development speed is just as important as optimizing for operational speed. Imagine in 5 years...\n
If your APIs, including your data validation, are all well tested they will be easy to change. Easy to update. You have the tests to prove it.\n
Think of everything in your application as single purpose. Each method should do one thing. Each method should be behind a simple API. Each simple API should have an easy test. This is really outside the scope so I&#x2019;m going to move on.\n
To a method that does only one thing. It&#x2019;s a POST action. This is how we verify data. Nearly all of our controller actions that handle a POST follow this pattern or a similar pattern. Because it is so similar, we use either base classes or roles.\n
In the simplest form, you just create a simple class that builds Data::Manager objects. You have a big list of profiles (just hashes) and you create Data::Verifier objects from it.\nI prefer creating this per request (lazily) because I never have to worry about data leakage. The Data::Manager belongs to that request and that request alone.\n
This is the profile hash. It is keyed by scope and the body is all of what gets passed into Data::Verifier.\n
When you start to impose more rigid business rules, it&#x2019;s much easier to adapt using this. You can define Moose types (which are easily testable), coercions and run from there. It gives you a great deal of increasing flexibility on data verification. If you decide usernames should be email addresses, just define an email type. All done.\n
The type checking is one of the coolest features of Data::Verifier. You can specify to use the default coercion, your own coercion or none at all. It just verifies if the incoming data passes the type constraint. You get the reason why from Data::Verifier, but the automatic parsing that Data::Manager uses just throws &#x201C;invalid&#x201D; since it&#x2019;s the lowest common denominator.\n
Data::Manager is also easily extended. In a lot of applications we end up adding our own methods for whatever we need. I have one that generates a list of traits based on what scopes. It&#x2019;s application specific, but works well. Some ideas maybe should go into Data::Manager itself but we&#x2019;re not sure. We don&#x2019;t want to clutter it with a confusing set of methods.\n
The other way is that you can automatically setup profiles. I have code that crawls the loaded DBIC result sources, looks at the columns and generates a verification profile for them. Another similar one for any Catalyst model loaded, crawling all models if they have a Verification trait applied and then loading the profile accordingly. There&#x2019;s a lot of ways to handle this. I still don&#x2019;t think I have one that unlocked the Awesome Badge.\n
This section covers a more advanced situation. Something often done in the controller that shouldn&#x2019;t be done in the controller. Depending upon the application, this would either be in the model layer or beneath in the business/domain layer.\n
Why is this hard? Well, it has its hands in many pies. The case of create or update must be looked at.\n
Here is a simple closure that does the work. The $r is the Data::Verifier::Results object *after* filter, coerce and validation but before post-checking. Data::Verifier fails on the first failure, so a post_check won&#x2019;t be called if a coerce or type fail.\n
Immediately this shows some problems. Not functional or logical problems, but the problems of a more difficult variety. Problems 6 months later. Those are the worst problems.\n
It&#x2019;s hard to test, and this is the biggest problem. When things are hard to test we don&#x2019;t test them. We have more important things to do! The solution is to write code that is easy to test, obviously.\n
So here&#x2019;s how we use this. It&#x2019;s specific to the model in the application. This is Catalyst syntax, but any other framework could do the same thing. When the Data::Manager object is instantiated for the request, you just shove the closure in and be done with it.\nSince the closure can reside anywhere you can get a scalar out of, it&#x2019;s easy to put it where it makes sense. Maybe an attribute on your user object.\n
So here&#x2019;s how we use this. It&#x2019;s specific to the model in the application. This is Catalyst syntax, but any other framework could do the same thing. When the Data::Manager object is instantiated for the request, you just shove the closure in and be done with it.\nSince the closure can reside anywhere you can get a scalar out of, it&#x2019;s easy to put it where it makes sense. Maybe an attribute on your user object.\n
Now you can put your unique check where it is easily accessible, some method that just returns the code ref. If you follow this, you can access and modify your verification profiles and not worry. More importantly, as long as you unit test your unique_check method appropriately (which you can do outside the confines of Data::Verifier) you can succeed in testing everything. Data::Verifier::Results->new is the secret here, stuff it with what you want.\n
The great thing here is that our controller is the same as it was. No modification to the controller for what is a business or model level decision. Using these techniques, you still get the same exact errors, split up by scope and subject but they&#x2019;re more informative and you have more coverage. No template change, no markup change, no controller change. Just inserting a post_check into your profile.\n
And that&#x2019;s the point of this story.\n
We&#x2019;ve done something that is consistent for the user, reliable in use, easily testable and optimized for the developer. If you look at this and go, &#x201C;No, no, I don&#x2019;t want this solid messaging in my application&#x201D; then I&#x2019;ve failed to tell this story. I hope everybody here says, &#x201C;YES! This is what I want!&#x201D; Maybe not with these tools, after all, we&#x2019;re not married to them.\n
But this is what we use and it works for us. I hope you find something that just works for you and your users. Building better applications is hard. You have just witnessed 100 slides of me telling the story of how we got to where we are. It&#x2019;s been a long, hard road. Arduous even, but I think we&#x2019;ve succeeded. We took one of the most painful parts *for users* of building applications and creating a system we, as developers, can easily cope with.\n
