Joint presentation by the DC Promise Neighborhood Initiative (DCPNI) and Urban Institute staff at the Eastern Evaluation Research Society's Annual Conference in 2014. Presentation focuses on DCPNI's neighborhood survey - a community wide data collection project. The slides offer tips and suggestions on how to make the process as smooth as possible without compromising data collection rigor.

1. A Neighborhood Survey in the Nation’s Capital:
Balancing Rigor, Resources, and Respect
www.dcpni.org
@dcpni
1
www.urban.org
@urbaninstitute
Eastern Evaluation Research Society
37th Annual Conference
April 28, 2014

2. Today’s Panelists
2
Isaac Castillo
Director of Data and Evaluation
DC Promise Neighborhood
Initiative
@isaac_outcomes
Samantha Greenberg
Data and Evaluation Specialist
DC Promise Neighborhood
Initiative
@eval_revolution
Maia Woluchem
Research Assistant
Metropolitan Housing and
Communities Policy Center
Urban Institute
Megan Gallagher
Senior Research Associate
Metropolitan Housing and
Communities Policy Center
Urban Institute
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

3. A Neighborhood Survey
in the Nation’s Capital
• The DC Promise Neighborhood Initiative’s
Neighborhood Survey
• It Takes a Neighborhood to Design a Survey
• Getting Community Residents Excited About a
Neighborhood Survey.
• Data Collection in the 21st Century
3April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

4. The DC Promise Neighborhood Initiative’s
Neighborhood Survey:
When Community, Research, Evaluation,
and Rigor All Come Together
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
4
Isaac Castillo
Director of Data and Evaluation
DC Promise Neighborhood
Initiative
Twitter: @Isaac_outcomes

5. As part of the White House Neighborhood
Revitalization Initiative, Promise Neighborhoods
align federal funding streams that invest in
transforming neighborhoods of concentrated
poverty into neighborhoods of opportunity.
What is the Promise Neighborhood Approach?
The INSPIRATION….
Harlem Children’s Zone (HCZ) inspired the U.S.
Department of Education’s Promise
Neighborhoods program, which launched in 2010.
5April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

6. What is DCPNI?
• DCPNI is one of 12 communities awarded
funding from the U.S. Department of
Education.
• DCPNI’s focus is on the Kenilworth-Parkside
neighborhood of Washington, DC.
• Founded as an independent nonprofit in 2012
shortly after receiving federal funding.
6April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

7. Kenilworth-Parkside by the Numbers
• 5,725 people – 1,800 children
• 98% African American, 55% are female
• 50% of adults live in poverty
• 49% of K-P residents experience food insecurity
• 70% of K-P residents have a high school
education or higher (compared to 88% in DC)
• 85% of households with children are headed by
single females
7April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

8. DCPNI’s Vision
To end intergenerational
poverty in the
Kenilworth-Parkside
community.
8April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

9. THE FIVE PROMISES
9April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

10. What does DCPNI do?
10
Fund
Community-based organizations
and schools to provide effective
and research based programming
Coordinate
All services provided in the
Promise Neighborhood to
maximize effectiveness and avoid
duplication
Connect
Community residents and
students to available services
provided by partners
Train
Everyone in the Promise
Neighborhood to utilize
information and advocate for
themselves and their neighbors
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

11. DCPNI’s Work at a Glance
11
Fund
Coordinate
Provide
Train
Schools
Partners
Parents
Children
=+
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

12. How Does DCPNI Measure Progress
on Each Promise?
• DCPNI has 15 federal indicators that we are
required to track by the Department of
Education.
• Developing additional indicators on issues that
are unique to Kenilworth-Parkside.
– Teen birth rate
– Mother’s educational attainment
– Food insecurity
• These indicators help us determine if we are
progressing towards the Promises.
12April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

13. Sample of DCPNI Indicators
• Parents read to their 0-5 year old
children.
• Increased feelings of safety among
community residents.
• Children ages 0-5 that have a medical
home.
13April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
These data points need to be collected
directly from community residents.

14. The Need for a Neighborhood Survey
• Collect information directly from community
residents.
• Limited other publically (or privately) available data
sets for the Kenilworth-Parkside population.
• Desire to measure community level change over
time.
• What little data that was available was old, limited
to parts of the community, or collected using low
rigor methods.
14April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

15. The Most Significant Challenges
• Very short timeframe – needed to complete entire
process (from survey design to final analysis of
data) in 9 months.
• Low levels of literacy in the community (and low
comfort level with technology).
• High levels of suspicion of outsiders (and even
suspicion of those from another part of the
community).
• High levels of distrust of government and
government-like entities.
15April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

16. It Takes a Neighborhood to Design a Survey:
The Methodology Behind the Kenilworth-
Parkside Neighborhood Survey
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
16
Megan Gallagher
Senior Research Associate
Metropolitan Housing and
Communities Policy Center
Urban Institute

17. Methodology Overview
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
17
• Instrument Design
• IRB Review
• Sampling
• Sampling frame
• Strata, or key groupings
• Random selection
• Response Rates
• Population Weights

18. Instrument Design
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
18
• Create a baseline assessment of the neighborhood
from which to assess change
• Many considerations, including
• Department of Education Government Performance and
Results Act (GPRA) indicators
• DCPNI programmatic priorities
• DCHA Choice Neighborhoods data needs
• Urban Institute suggestions
• Resident interests and concerns
• Instrument length, nature of questions

19. Instrument Design
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
19
• Topics include:
• Housing
• Neighborhood amenities
• Access to Food
• Neighborhood Supports
• Neighborhood Conditions and Safety
• Household Composition
• Travel, Education, Employment and Public Assistance
• Child Health, Education, and Well-being

20. IRB Review
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
20
• Institutional Review Board (IRB) makes sure that
research minimizes risks to human subjects
• Reviewed and approved the plan for:
• Requiring confidentiality pledges from staff
• Obtaining consent from research subject
• Roles for community resident and City Year corps
member
• Providing incentives for completed surveys
• Keeping data confidential

21. Sampling
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
21
• Used a list of addresses for 2 census tracts that
make up Kenilworth Parkside
• Separated them into 3 sub-neighborhoods, or strata
• Group 1: Kenilworth Courts
• Group 2: KPRMC, Mayfair, Paradise, Lotus Square and
Victory Square
• Group 3: Eastland Gardens and Paradise
• Randomly select addresses within each sub-
neighborhood (SAS Proc Surveyselect)
• Oversampled Group 1

22. Response Rate
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
22
• Eligible households
• 872 addresses randomly selected
• 115 unoccupied
• 872 – 115 = 757 eligible households
• Completed surveys
• 444 completes
• Response rate
• 444 / 757 = 59% response rate

23. Population Weights
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
23
Sub-
neighborhood Population
% of
Population
Completed
Surveys
% of
Surveys Weight
Group 1:
Kenilworth
Courts
283 12% 133 30%
=12%/30%
=.3857
Group 2: KPRMC,
Mayfair,
Paradise, Lotus
Square and
Victory Square
1620 68% 217 49%
=68%/49%
=1.4012
Group 3:
Eastland Gardens
and Paradise
476 20% 94 21%
=20%/21%
=.9731
Total 2379 100% 444 100% 1.0000

24. Doing the Impossible:
Getting Community Residents Excited
About a Neighborhood Survey
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
24
Samantha Greenberg
Data and Evaluation Specialist
DC Promise Neighborhood
Initiative
Twitter: @eval_revolution

25. April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
25
Be inclusive
Gain community
support
Build
relationships
Be culturally
sensitive
Add unique
value
Goals for Kenilworth-Parkside Survey:

26. April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
26
Survey Administration Teams:
Kenilworth-
Parkside
Community
Resident
• Navigated the neighborhood
• Greeted person at door
• Introduced survey
• Documented households visited
• Handled incentives
• Consent process
• Asked survey questions
• Documented answers on tablet
• Handled technology
Step 1: Build an Inclusive Process

27. 27
Community residents:
• Familiar with neighborhood
• Have relationships with survey takers
• Knowledgeable about community history and assets
Step 1: Build an Inclusive Process
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

28. 28
City Year corps team members :
• Versed in technology
• Able to offer confidentiality (not from K-P)
• Experienced serving D.C. communities
Step 1: Build an Inclusive Process
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

29. 29
Community residents:
• Recruitment and screening
• Background checks
• Mandatory training session
City Year Corps Team:
• Partnership with City Year
• Recruitment at full team event
• Mandatory training session
Hiring Process
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

30. 30
NEIGHBORHOOD SURVEY TRAINING
30
• Structure of each
work day
• Safety precautions
• How to introduce
survey
• How to document
responses from
potential survey
respondents
Training Process
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

31. 31
Kenilworth-
Parkside
Community
Resident
Advantages of Two-person Teams
• Increased likelihood of “open doors”
• Decreased # of survey days needed
• Easy navigation of community
• Division of labor
• Two people available to answer questions
• Balance between confidentiality and trust
• Increased safety
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

32. 32
Challenges of Two-person Teams
• Massive recruiting and hiring efforts
• DCPNI staff capacity
• Difficult to ensure fidelity
• Downtime during surveys
• Answering questions about survey when
approached on street
Kenilworth-
Parkside
Community
Resident
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

33. 33
Step 2: Prioritize cultural sensitivity
• Survey questions tested with community
• Community feedback encouraged throughout
• Survey administrators
participated in focus
group
• Concerns from
community members
noted for next survey
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

34. 34
Step 3: Add Unique Value
• I heart data booklet
provides data
snapshot
• Data now exists on
topics for which
there are no other
sources
• Ongoing
conversations with
the K-P community
about data
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute

35. 35April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
Step 4: Build Relationships

36. 36April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
• Impacts of Kenilworth-Parkside neighborhood survey:
Inclusive data
collection
Gained community
support
Built
relationships
Culturally
sensitive survey
Unique data
collected
Results: Community Support for Survey:

37. Data Collection in the 21st Century:
The Use of Tablet Computers for a
Neighborhood Survey
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
37
Maia Woluchem
Research Assistant
Metropolitan Housing and
Communities Policy Center
Urban Institute
mwoluchem@urban.org

38. Challenges for Our Survey
• Several teams approaching hundreds of
doorsteps over several points in time
• Each respondent was different from the last
• Each interviewer is different from the last
• Presenting several challenges
– Monetary costs
– Confidentiality
– Standardization
– Flexibility
– Ease
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
38

39. Paper vs Plastic
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
39
Feature Paper Surveys Tablet Surveys
Money
Printing supplies,
associated materials, post
survey dataentry
$200 per tablet,KeySurvey
subscription
Confidentiality
Maintaining security of
documentsin field, post
survey
Tablet’s secure connection
and privacy capabilities
Standardization Interview cues
Flexibility May involve skipping pages Skip logic, conditionality
Ease

40. Program
• Used KeySurvey as
our survey platform
– Allowed ability to build conditional sections and
complicated skip patterns
– Allowed personalization based on household
traits
• Cost-reducing in terms of time, money and flexibility
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
40

41. Back-End v. Front-End
• Level one: 159 questions and hundreds of
answer choices
• Level two: Variable names
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
41

42. Back-End v. Front-End
• Level Three: Skip patterns
– Sometimes a simple skip:
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
42

43. Back-End v. Front-End
• Sometimes it’s something else entirely:
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
43

44. Back-End v. Front-End
• Everything was automated, making things
easy for everyone—
• Setting:
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
44

45. Back-End v. Front-End (Consent)
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
45

46. Back-End v. Front-End
(Personalization)
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
46

47. Back-End v. Front-End
• Once programmed, survey looked like this—
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
47

48. Back-End v. Front-End
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
48

49. Administration:
Who conducted the survey?
• AmeriCorps members and college volunteers
– Read aloud the questions to the householder
– Householder responds verbally
• Why?
– Literacy rates among the community
– Unfamiliarity with the technology
– Standardization of questions and answers
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
49

50. Training
• Back-end was sophisticated but front-end was
very user friendly.
• Training session has some ground rules:
– Trust the programming
– Don’t be pushy
• And some golden rules:
– Read the exact wording of the question as presented
– Obtain consent
– Work with your partner
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
50

51. Partnerships in Action
• Morning Briefing (DCPNI Offices)
– Community Resident:
• Receive ten addresses, gift cards (4), survey receipts,
door hangers, and a walkie talkie
– Interviewer:
• Receive same ten addresses and a tablet computer
• Walk
– Resident: Fill out tracking form to document
address condition
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
51

52. Partnerships in Action
• Doorstep
– Community Resident:
• Explain survey and present opportunity for giftcard
• Answer questions householder may have
• Document what happens
– Interviewer:
• Obtain consent
• Administer survey
• Administration (Interviewer & Community Resident)
– At a comfortable place
– Resident Role
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
52

53. Partnerships in Action
• Post-Survey
– Community Resident:
• Offer respondent choice of incentives
• Everyone signs receipt
• What if no one is home?
– Resident:
• Leave door hanger
• Document address so that another
team could make attempt
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
53

54. Tablets are not Without Challenges
• What isn’t done by hand must be done by
computer
– Careful time-intensive programming necessary
• Technical difficulties
– Potential for tablet failure (need for backup tablets)
• Post-launch errors are data errors
– Examples of missing answers or misplaced variables
– Typing answers in leads to some data quality errors
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
54

55. But the Benefits?
• Benefits greater than challenges (in our case)
• Data/Tablet Security
– End of day wipe and upload to UI SFTP
– Confidential drives at UI
– Cerberus for tablet security
• Substantial results
– Over four weekends in October and November,
got 444 responses, a response rate of 59% overall
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
55

56. Audience Q&A
April 28, 2014
DC Promise Neighborhood Initiative and
The Urban Institute
56
Isaac Castillo
Director of Data and Evaluation
DC Promise Neighborhood
Initiative
@isaac_outcomes
Samantha Greenberg
Data and Evaluation Specialist
DC Promise Neighborhood
Initiative
@eval_revolution
Maia Woluchem
Research Assistant
Metropolitan Housing and
Communities Policy Center
Urban Institute
Megan Gallagher
Research Associate
Metropolitan Housing and
Communities Policy Center
Urban Institute