Evaluation Process
About the Evaluation Rubric
The Scaling Pathways to Homeownership Open call Evaluation Rubric, below, will be used to assess applications during Participatory Review and during Evaluation Panel Review. During Participatory Review, each valid application will be reviewed by at least four (4) peer Applicants who will assess each application on the four criteria that comprise the evaluation rubric. Once Participatory Review is complete, top Applicants will advance to the Evaluation Panel, where they will be reviewed by at least four (4) Evaluation Panel reviewers who will use the same evaluation rubric to assess each application.
The rubric is built upon four distinct criteria, guiding reviewers to consider each trait in isolation when evaluating applications. In both stages of review, reviewers will assign a score for each of the four criteria and provide a comment to justify each decision. Those scores and comments will be compiled into one composite assessment of each application, and each Applicant will receive its “package” of scores/comments from the assigned reviewers (who will be anonymous to the Applicants).
This process is designed to be transparent, fair, and open while providing a value-added experience for Applicants. Even Applicants who do not receive an award will receive feedback from peers and many will receive feedback from Evaluation Panel reviewers about their applications.
The Evaluation Rubric includes four criteria that are equally weighted across each five-point scale. Each Applicant can receive between 1 and 5 points on each scale, representing a total possible scoring range between 4-20 from each Reviewer. When reviewing the rubric, there are different descriptors within each scale. They are:
- Trait: The name attributed to each Review Criterion (e.g., “Durable”).
- Prompt: One or more questions, intended to clarify areas of consideration when defining a Trait.
- Evaluation Range Definitions: Clarifying statements to define sub-scoring ranges (e.g., 2 vs. a 4).
With four (4) reviews per application, the total maximum score from the four assigned reviewers for each trait is 20, and the maximum aggregate total score is 80 points.
#1 INNOVATIVE
CONVENTIONAL
1: Proposal is not novel and does not challenge the status quo. It replicates or repurposes existing frameworks, tools, or models and does not present a disruptive, future-oriented vision for the housing attainability field.
2: Proposal shows limited novelty with minor adaptations to existing frameworks, tools, or models and offers an uninventive, status quo vision for the housing attainability field.
3: Proposal demonstrates some original thinking by combining or modifying existing frameworks, tools, or models and offers a disruptive, future-oriented vision that could potentially inspire or shift the housing attainability field.
4: Proposal presents significant novel elements with creative frameworks, tools, or models that challenge the status quo and demonstrates a disruptive, future oriented vision that could likely inspire or shift the housing attainability field.
5: Proposal offers groundbreaking thinking, tools, or models in the field in ways that challenge the status quo and demonstrate a disruptive, future-oriented vision with exceptional potential to inspire or shift the housing attainability field.
GROUNDBREAKING
#2 IMPACTFUL
INCONSEQUENTIAL
1: Proposal lacks potential to expand pathways to home ownership and housing attainability through knowledge and action, provides no measurable progress indicators, and does not demonstrate any meaningful benefit for low- and moderate-income households.
2: Proposal shows limited potential to expand pathways to homeownership and housing attainability through knowledge and action with weak measurement strategies and minimal demonstrated benefits for low- and moderate-income households.
3: Proposal demonstrates some potential to expand pathways to homeownership and housing attainability through knowledge and action with basic measurement approaches and shows reasonable benefit for low- and moderate-income households.
4: Proposal presents strong potential to expand homeownership or housing attainability through knowledge and action with clear measurement strategies and substantially demonstrated capacity to benefit low- and moderate-income households.
5: Proposal offers exceptional potential to expand homeownership or housing attainability with robust measurement frameworks and compelling demonstrated capacity to transform outcomes for low- and moderate-income households
TRANSFORMATIVE
#3 BRIDGE-BUILDING
SILOED
1: Proposal lacks evidence of connecting a wide range of stakeholders, does not foster any meaningful cross-sector collaboration, and does not demonstrate trust, credibility, or any track record in the community it aims to serve.
2: Proposal shows limited evidence of stakeholder engagement with weak collaboration across sectors or perspectives and demonstrates minimal trust, credibility, or a proven track record in the community it aims to serve.
3: Proposal shows satisfactory evidence of bringing together a wide range of stakeholders with basic collaboration across sectors or perspectives and demonstrates some trust, credibility, or a proven track record in the community it aims to serve.
4: Proposal presents solid evidence of effectively engaging a wide range of stakeholders with meaningful collaborations across sectors or perspectives and demonstrates trust, credibility, or a proven track record in the community it aims to serve.
5: Proposal offers compelling evidence of unifying a wide range of stakeholders with robust collaboration across sectors and perspectives and clearly demonstrates trust, credibility, or a proven track record in the community it aims to serve.
UNIFYING
#4 DURABLE
TRANSIENT
1: Proposal lacks a sustainable pathway to address housing attainability barriers over time, does not address the underlying causes of the specific problem, and shows no evidence of potential support from private, philanthropic, or public sources.
2: Proposal shows limited sustainability in addressing housing attainability barriers, only lightly addresses the underlying causes of the specific problem, with weak evidence of potential support from private, philanthropic, or public sources.
3: Proposal demonstrates some sustainable pathway to address housing attainability barriers, provides a satisfactory approach to address the underlying causes of the specific problem, and offers basic evidence of potential support from private, philanthropic, or public sources.
4: Proposal presents a strong sustainable pathway to address housing attainability barriers, addresses the underlying causes of the specific problem, and offers substantial evidence of potential support from funding sources.
5: Proposal offers an exceptional sustainable pathway to address housing attainability barriers, comprehensively addresses the underlying causes of the specific problem, and offers compelling evidence of robust support from broad private, philanthropic, and public sources.
ENDURING
Lever for Change addresses possible biases in the evaluation process both on the front end, before the scoring begins, and on the back end, after the reviewers have submitted their scores.
On the front end, Lever for Change’s Participatory Review and Evaluation Panel Review webinars review the open call criteria and the evaluation rubric and includes training on recognizing and addressing the biases that we all bring as we read and assess proposals.
Lever for Change also works to address unconscious bias inherent in the evaluation process.
During both Participatory Review and Evaluation Panel Review, each reviewer only reviews a very small subsample of the entire set of applications. Therefore, each reviewer has a very different reference point by which to adjudicate each application. For example, a reviewer may only see very good applications. Because these applications are of similarly good quality, the reviewer may decide to score these applications lower than they deserve. By contrast, a reviewer that only sees poor applications may give better scores on average. Furthermore, reviewers may have intrinsic cognitive biases. Even under ideal conditions, reviewers are prone to making non-objective judgements about the applications they review, and so Lever for Change normalizes the raw scores provided by our evaluators to reduce the risk that bias in the assessment process eliminates a strong application from consideration.
Normalization relies on finding the mean score given for all applications and the mean score given by each reviewer. Reviewers whose mean score is higher than the global average will have their scores devalued to account for whatever biases led to inflated scores. Similarly, reviewers whose scores are lower than the global average will have their scores boosted to compensate for their tougher-than-average assessments.
The evaluation process will be conducted at two points during the open call process: first, after the conclusion of Participatory Review and, later, after the conclusion of Evaluation Panel Review. A submission that ranks among the top applications, as determined by Participatory Review scores, will be eligible to advance to Evaluation Panel Review. A submission that ranks among the top applications, as determined by Evaluation Panel Review scores, may be considered for selection as a finalist.
As a further risk mitigation step, Lever for Change staff review reviewers’ scores and comments for every submission.
If at any point in the process, it is determined that an applicant does not meet the eligibility criteria of the Challenge as laid out in the Open Call Rules, the applicant will be notified and will not be eligible to progress to further stages of the open call.
Find more technical information in our Help Center.
