Evaluating Innovation with Scorecards
In this article we will take a look at some scorecards used to evaluate innovations, innovators, ideas and startups. If you’d like to use a scorecard shown here feel free to create an account at Innoscout – the tool for innovation scouting – or Innovote – the mobile award system for startup competitions.
When deciding on an evaluation scorecard you have two major things to think about: the criteria and the scales.
A criteria represents a property or attribute that is important for you or in general. For example the innovator (person or team) is generally important for successful execution on any innovation. What represents a large opportunity on the other hand can differ if you’re evaluating as an angel investors, government sponsored fund or venture capitalist.
For each criteria the evaluator chooses a value on a scale as their evaluation. Often times you provide guidance on how many points to award based on qualifying questions or attributes. Let’s take the example above. You might give one point each for a team that includes an CEO, CTO and CMO, and additional points if they are serial entrepreneurs. For the size of opportunity criteria you may define a minimum or maximum value representing 0 or 5 points and the evaluator can choose values in between.
Here we present a number of scorecards sourced from various use cases that were adapted to be used as early stage evaluation questionnaires for innovations.
Cabbage and Zhang Scorecard
This six part scorecard is used to evaluate an innovative company (startup) along six criteria, with each consisting of 3 possible question that each award a point and results in 0-3 quantitative scale (0 meaning that none of the factors are met) for each of the six equally weighted criteria.
- Customer
- Is there unmet need or desire?
- Is the market large enough? Either a niche market that they are the only player (big fish strategy) or a much larger market that they can get market share in (big market strategy).
- Do they have reliable access to that market? No point if the sole channel is a single point of failure or market regulation/manipulation is in place or to be expected.
- Product
- Is the solution customer focused? No point if value is unclear or multiple goals are targeted.
- Does the solution have a low barrier to adoption? Cost of the solution also includes migration or adoption cost, so a point if the solution has mostly a low financial and time investment to adopt, low learning curve and easily integrate-able into other systems and processes.
- Is the value proposition clear? A point if the value calculation (ROI) for the solution can be made easily understood and perceived.
- Competition
- Is a clear market inefficiency being met? One point for new markets (demand exceeds supply), fragmented market with no clear market leader or stagnant market that is ready for disription.
- Is there a barrier to entry? Consider no point if there are existing economies of scale, existing mature products, well-established brands or price competition.
- Does the solution have a defend-able USP (e.g. patent, technology, experience, unique approach)?
- Timing
- Is this a new innovation?
- Does the demand exist?
- Is the solution already commoditized? No point if low-cost players exist or many players with similar products that are substitutable.
- Financial
- Is large capital risk involved? 1 point for minimal sunk costs.
- Is a large amount of working capital required? 1 point for less working capital required.
- Are economies of scale expected? 1 point if it can be proven that margins increase with volume.
- Team
- Do they team have the experience? Are they subject matter experts.
- Do they have the skills to deliver? Technical, engineering, professional or network.
- Do they have the network to deliver? Connections to partners and suppliers.
Adapted Angel Scorecard Method
Popularized by Bill Payne in 2011 this scorecard is often used by angel investors for evaluating their financial investment opportunities. It can certainly be used as a formal method for a detailed evaluation but this minimally adapted version can also used in earlier and faster evaluation rounds.
1. Strength of the Entrepreneur and the Management Team
A large part of the succcess of any innovation is it’s execution team starting with the founder (or founding team) and the team they can assembly around them.
| Subcriteria points weighted at 30% | -2 | -1 | 0 | 1 | 2 | 3 |
| Experience Founder | no business experience | in sales or technology | as a product manager or many years of any business sector | as COO, CFO, CTO or in this business sector | as CEO | |
| Team Strength | Only the entrepreneur | One competent player in place | Team identified on the sidelines | Competent team in place |
2. Size of the Opportunity
The most difficult criteria to provide a reasonable scale for as it will vary between countries, investment types and the evaluator. What one person sees as a small market is a large one for another. I have decided to keep the figures are originally provided by Payne but caution to adapt these to your own needs if used in a differetn context.
Note though that the future revenue scale is not linear but parabolic. Both a too small potential and a too large potential are deemed negative for early stage investments. The latter usually implies further required capital down the line and introducing more risk.
| Subcriteria points weighted at 25% | -2 | -1 | 0 | 1 | 2 | 3 |
| Size of the target market (total sales) | < $50 million | $100 million | >$100 million | |||
| Potential for revenues of target company in 5 years | < $20 million | > $100 million | $20 – $50 million |
3. Strength of the Product and Intellectual Property
Here the questions ask how far along and mature their product development is and if can be protected from competing innovators.
| Subcriteria points weighted at 15% | -2 | -1 | 0 | 1 | 2 | 3 |
| Is the product defined and developed? | Not well define, still looking a prototypes | Well defined, prototype looks interesting | Good feedback from potential customers | Orders or early sales from customers | ||
| Is the product compelling to customers? | This product is a vitamin pill | This product is a pain killer | This product is a pain killer with no side effects | |||
| Can this product be duplicated by the others? | Easily copied, no intellectual property | Duplication difficult | Product unique and protected by trade secrets | Solid patent protections |
4. Competitive Environment
Will the innovation have difficulties entering the market and acquiring market share?
| Subcriteria points weighted at 15% | -2 | -1 | 0 | 1 | 2 | 3 |
| Strength of competitors in this marketplace | Dominated by a single large player | Dominated by several players | Fractured, many small players | |||
| Strength of competitive products | Competitive products are excellent | Competitive products are weak |
5. Marketing/Sales/Partners
Can they product the innovation and deliver it to the market?
| Subcriteria points weighted at 10% | -2 | -1 | 0 | 1 | 2 | 3 |
| Sales channels, sales and marketing partners | Haven’t even discussed sales channels | Key beta testers identified and contacted | Channels secure, customers placed trial orders | |||
| Channels secure, customers placed trial orders | No partners identified | Key partners in place |
6. Funding needs
Similar to the second factor in the criteria on opportunity size this scale should be adapted to the audience or organisation performing the evaluation as a later stage or corporate investor will have a different scale to the angel as depicted here.
| Subcriteria points weighted at 5% | -2 | -1 | 0 | 1 | 2 | 3 |
| Need for additional rounds of funding | Need venture capital | Another angel round | None |
7. Other factors
It is not efficient to add more elements to the above criteria, especially when used for fast evaluations. But there may be obvious factors not mentioned in the previous criteria that have a substantial impact on the innovation’s success. These are represented by this last criterium.
| Subcriteria points weighted at 5% | -2 | -1 | 0 | 1 | 2 | 3 |
| Need for additional rounds of funding | Negative other factors | Positive other factors |
Adapted from Source and Source
Ulu Rubric
Developed by Ulu ventures to determine the early parts of their funnel this scorecard approach evaluates innovators in 7 categories with a 5 part scale each.
| Points | Verdict in that category |
| -2 | Showstopper |
| -1 | Fares poorly |
| 0 | Neutral |
| 1 | Stands out |
| 2 | Outstanding |
Fit
In this category the fit between innovator and adopting/interested organisation is measured. This will usually be the score for how well it matches on of your innovation scouting search scopes.
Factors to consider are:
- Is the innovation in the correct stage for your purpose/interest?
- Does it fit the industry/area of expertise?
- Does it fit your (desired) organisational values?
- Does it fit your location requirements?
Market / Opportunity
- Is the problem adressing a top 3 problem for the customer with budget?
- Does the solution have traction?
- Is the total addressable market large enough for you to invest?
- Does the team have a focused go-to-market strategy?
Team to Market Fit
- Does the team have the required domain expertise?
- Is the team committed?
Team in General
- Does the team come across as authentic?
- Does the team have good ethics?
- Does the team have character?
Product Development
- Is learning baked into the product development?
- Is the solution complexity handled well?
- How well does the demo communicate the value?
- How is the overall product experience?
Financial Viability
- Is the current valuation good for us to invest?
- Is the business model sound?
- How diluted is the cap table now and will it be in future?
- Are there Exit possibilities?
Super Powers
- Do they have a substantial competitive advantage?
- Are there network effects built-in to the innovation?
Adapted from Source
Anchored Risk Comparison Scorecard
If you have existing innovators, innovations, projects or experiences to anchor your evaluation against then you the risk comparison scorecard is a useful tool. Although when evaluating in a team the anchor needs to be either identical or at least similar in most characteristics. Alternatively you can define an anchor value for each of the 7 risk categories and evaluate against that:
- Management Risk
- Stage of Business Risk
- Legislation / Political Risk
- Manufacturing Risk
- Sales and Marketing Risk
- Funding / Capital Raising Risk
- Competition Risk
For each of the six risk categories you compare the risk for that category against the anchor and give points as follows:
| Points | Compare to the anchor |
| -2 | A lot worse |
| -1 | Slightly worse |
| 0 | Normal / About the same |
| 1 | Slightly better |
| 2 | A lot better |
Depending on your innovation activity (e.g. investments or procurement) you may weight each risk category differently to calculate the overall score.
Conclusion
In the scorecards we’ve looked at their are many similarities. Depending on where they are used there is a different focus (e.g. on the financials) but the repeating factors are:
- Team / Execution: Is the innovator the right person and/or assembled the right team to execute on the innovation?
- Product / Innovation: Is the innovation solving a market need successfully?
- Business Model / Delivery: Can the innovator ultimately deliver the innovation to market at a cost below what someone is willing to pay for it?
- Market / Opportunity: Does the result on investing in the innovation justify the cost (time, money and any other resources including opportunity costs)?
Most evaluations (especially early in your innovation funnel) will boil down to these factors. You may choose to weigh one factor higher than the other (i.e. commonly team is more important than the others as a good team can change course on a bad product, but a bad team cannot execute on a good product).