When one person is staffed across several client projects, no single lead saw the whole cycle — each watched a different slice, under a different scope, difficulty and client. A fair review isn't the average of their scores; it's a deliberately assembled picture. You judge every engagement against one shared bar, weight each input by how much that lead actually saw, treat disagreement between projects as signal to investigate — not noise to average away — normalize for how hard each engagement was, and calibrate across the leads so one generous or harsh reviewer, or the single loudest project, doesn't decide the outcome. The result reflects the whole period, not just the engagement that left the longest paper trail.
Why can't you just average the scores? Because most of any single score is the scorer, not the person. When researchers decomposed managers' ratings, about 62% of the variance came from the rater's idiosyncrasies and only ~21% from actual performance (Scullen, Mount & Goff, Journal of Applied Psychology, 2000) — and two competent leads rating the same person agree only moderately, with interrater reliability around .52 (Viswesvaran, Ones & Schmidt, Journal of Applied Psychology, 1996). Averaging N partial, rater-flavoured views doesn't cancel that out; assembling them deliberately does.
Why is reviewing someone across several client projects so hard?
Because the thing you're rating was never seen whole by anyone. In a project-based firm a consultant or creative is spread across several engagements a cycle, and each lead saw only their slice — a different scope, a different duration, a different client, a different level of difficulty. Four problems follow from that:
- No one has the full picture. Each lead reviews a fraction and, quite reasonably, rates what they saw. Stitch those fractions together naively and you get a blur, not a person.
- The loudest project dominates. The longest engagement, the most visible client win, or the project with the most documentation crowds out a quiet but harder piece of work. Volume of evidence isn't the same as weight of evidence.
- Each lead is a noisy instrument. A single reviewer's score is mostly their own perspective and leniency (Scullen, Mount & Goff, 2000); one vantage point is only moderately reliable (Viswesvaran, Ones & Schmidt, 1996).
- The bar drifts between leads. "Strong" on a demanding partner's engagement and "strong" on an easy retainer are not the same standard — unless someone made them the same on purpose.
What does "fair" mean when every lead saw a different slice?
Fair means every engagement is judged against one shared bar, not each lead's private standard. Before you reconcile anything, you need a single definition — in observable behaviours — of what "good" looks like at this person's level: what a consultant, a senior, a principal is expected to do, written down in advance (CIPD, Competence and competency frameworks). When the bar is explicit and common, a rating from a demanding partner and a rating from a gentle account lead finally mean the same thing, because both were measured against the same yardstick rather than against the reviewer's mood.
This is also why you don't simply average. If each score is partly the rater (Scullen, Mount & Goff, 2000) and single-rater agreement is only about .52 (Viswesvaran, Ones & Schmidt, 1996), the honest move is to treat the scores as evidence to be weighed against a common standard — not as measurements to be summed. Kahneman, Sibony and Sunstein call the discipline that contains this kind of variability "decision hygiene": a shared scale, structure, and calibration (Kahneman, Sibony & Sunstein, Noise, 2021).
How do you map who saw what before you judge?
Build a coverage map first — you can't weight inputs you haven't laid out. Before anyone proposes a rating, capture, for each engagement the person worked on, five things:
- The lead. Who is giving the input, and in what role did they see the person.
- Exposure. How much of the person's time and the cycle that engagement actually covered — a six-month staffing or a two-week overflow task.
- Scope. What the person was responsible for, and what this lead could genuinely observe (a lead who never saw them run a client meeting can't rate client presence).
- Difficulty and context. How hard and how ambiguous the engagement was — the client, the stakes, the constraints.
- Evidence. The concrete examples tied to the shared bar, gathered before any score is named.
The coverage map turns "here are five scores" into "here is who saw what, how much, and under what conditions" — which is the only basis on which the scores can be fairly combined.
How should you weight feedback across different leads and projects?
Weight by exposure and proximity to the work — not by how much each lead wrote. The lead who staffed the person for most of the cycle, on work central to their role, should count more than the lead who supervised a two-week overflow task, however detailed that lead's write-up. Proximity beats volume: a reviewer who directly watched the person handle the client is a better witness on client skill than a partner who only saw the final deck.
This is the opposite of averaging. Averaging treats a two-week task and a six-month engagement as equal votes; exposure-weighting treats them as what they are — a glimpse and a long look. And collecting more inputs is not enough on its own: multisource feedback only improves things when it's structured and followed up, not when scores are merely gathered (Smither, London & Reilly, Personnel Psychology, 2005). The weighting, the synthesis, and the follow-up are the work.
What do you do when the projects disagree — and how do you handle difficulty and bench time?
Treat divergence as signal to investigate, not noise to split down the middle. When one lead rates the person a clear "exceeds" and another a flat "meets," the average — "slightly above" — is usually the least true answer available. The useful question is why they diverge:
- Stretch versus steady state. One engagement pushed the person past their level; the other kept them comfortable. Both are real; the rating should reflect what they did at the top of their range and how consistently.
- Difficulty, not performance. A strong showing on a hard, ambiguous, high-stakes engagement is not the same as the same score on an easy retainer. Normalize for difficulty so the person who did well under pressure isn't rated below someone who coasted on something simple.
- Fit, not ability. A bad client-personality fit can depress an otherwise strong performer's reviews on one project. Name it as context; don't bake it into the number as if it were a skill gap.
Handle thin coverage and bench time honestly, too. If no lead saw enough to judge a dimension, say so — don't manufacture a rating from a fragment. Time on the bench between staffings is not evidence of poor performance and shouldn't be scored as such; note it as a coverage gap and rate on the engagements you actually have. A century of appraisal research is blunt about this: the value isn't the score itself but the reliable, useful judgment around it (DeNisi & Murphy, Journal of Applied Psychology, 2017).
How do you calibrate across leads so one project (or one reviewer) doesn't skew it?
Calibrate the per-engagement inputs against the shared bar before anything rolls up into a decision. Calibration is where the weighted, difficulty-adjusted inputs from different leads are compared side by side so the outliers show up: the lead who rates everyone high to avoid a hard conversation, the partner who marks everyone down, the single dominant project trying to stand in for the whole cycle. You correct against the bar — not to hit a quota, but to stop one reviewer's leniency or severity, or one loud engagement, from deciding the outcome. This is exactly the "decision hygiene" that reduces both bias and noise in professional judgment (Kahneman, Sibony & Sunstein, Noise, 2021), and it's the practitioner standard for fair, defensible appraisal (CIPD, Performance management).
Then end where it counts: one synthesized growth picture and a forward plan, kept separate from the calibrated rating. The rating answers "where did this person land against the bar this cycle"; the growth plan answers "what happens next." Keep them apart, because feedback that's fused to a verdict can backfire — across 607 studies feedback raised performance on average, but over a third of interventions actually made it worse (Kluger & DeNisi, Psychological Bulletin, 1996). The synthesis has to be forward-looking to be worth doing.
Why does this matter more for agencies and consulting boutiques?
Because in a client-services firm this isn't an edge case — it's the default shape of every review. A consultant or creative is staffed on several engagements a year under different leads and partners, rated per engagement, and those per-engagement ratings roll up into calibration, promotion, and up-or-out on a partner track. Billable pressure guarantees the coverage is uneven: no one lead saw the whole person, and the engagements that happened to generate the most visible output aren't necessarily the ones that show the most growth. The 62% rater-idiosyncrasy finding (Scullen, Mount & Goff, 2000) isn't abstract here — it's the gap between a defensible promotion case and a political one.
So the fair-review method is structural, not a nicety. A coverage map stops the loudest project from standing in for the cycle. Exposure-weighted inputs keep a two-week overflow task from outvoting a six-month engagement. Difficulty normalization protects the person who did hard things well. Calibration across leads and partners catches the reviewer whose ratings say more about them than about the people they rated — which is what makes the result defensible when it decides who makes partner. And because a senior consultant reads an unfair verdict as exactly that and leaves over it, defensible reviews are also retention. Build the whole thing light enough to run between billable hours, or it won't run at all.
A quick self-check: is your multi-project review actually fair?
Score one point per "yes". Six or seven and your process genuinely assembles a fair picture; four or fewer and the loudest project is still deciding the rating.
- There's one shared, written bar (observable behaviours at each level) that every engagement is judged against.
- You build a coverage map — lead, exposure, scope, difficulty, evidence — for each engagement before anyone names a score.
- Inputs are weighted by how much each lead actually saw, not by how much they wrote.
- Divergence between projects is investigated (stretch, difficulty, fit) rather than averaged away.
- You normalize for engagement difficulty so a hard project done well isn't undersold.
- Thin coverage and bench time are flagged honestly, not scored as poor performance.
- Ratings are calibrated across leads and partners before they feed promotion or partner-track decisions.
Scored four or fewer? Book a call and we'll show you where the loudest project is skewing your ratings and what to fix first.
FAQ
How do you review someone who worked on several client projects with different leads?
Don't average the scores — assemble a picture. Set one shared bar for the person's level, build a coverage map of who saw what (lead, exposure, scope, difficulty, evidence), weight each input by how much that lead actually observed, investigate where projects disagree instead of splitting the difference, normalize for how hard each engagement was, and calibrate across leads before anything rolls up into a decision. Averaging fails because most of a single score is the rater, not the person (Scullen, Mount & Goff, 2000).
Why not just take the average of each engagement rating?
Because the scores aren't clean measurements — each is heavily flavoured by the individual rater (about 62% rater idiosyncrasy vs ~21% performance; Scullen, Mount & Goff, 2000), and single-rater agreement is only about .52 (Viswesvaran, Ones & Schmidt, 1996). Averaging noisy, partial, unequal-exposure views produces a confident-looking number that hides all of that. Weighting by exposure and calibrating against a shared bar is what actually makes the inputs comparable.
What do you do when two projects give very different ratings?
Treat the gap as information, not an error to average out. Ask why: was one engagement a stretch beyond the person's level and the other steady-state? Was one far harder or more ambiguous? Was there a client-fit problem that depressed otherwise strong work? Resolve the divergence against the shared bar and the difficulty of each engagement, and record the reasoning — a "slightly above" average is usually the least true answer.
How should bench time or thin coverage affect a review?
Neither should be scored as poor performance. Time on the bench between staffings is a coverage gap, not evidence — note it and rate on the engagements you actually have. If no lead saw enough to judge a dimension, say so rather than inventing a rating from a fragment; honest gaps beat manufactured precision.
Why does this problem hit agencies and consulting boutiques hardest?
Because their people are staffed across multiple engagements under different leads every cycle, rated per engagement, and those ratings feed calibration, promotion, and partner-track decisions. Coverage is always uneven under billable pressure, so without a coverage map, exposure-weighting, difficulty normalization, and cross-lead calibration, the most visible project — not the fullest picture — decides someone's career.
