Hackathon judging criteria: how to prepare the judges
How to connect judging criteria to the hackathon's goal, describe a scoring scale, assign projects to judges and check the final results.
·6 min read·
Hackathon judging criteria should tell participants what work to show and judges what evidence to rely on. Words such as "quality," "innovation" and "impact" leave too much room for personal interpretation unless the scale is described. Give each criterion an observable basis for assessment.
Start with the event's goal. If the hackathon is about testing technical approaches, the criteria need to allow assessment of the implementation and experiment. If the challenge requires an understanding of a user process, include a relevant check. Do not add business metrics merely because another competition used them.
Separate eligibility from quality
Formal requirements determine whether a project can be judged: the submission deadline, team roster, required materials and compliance with the rules. Judging criteria determine the quality of the eligible work. Mixing these two levels makes the result harder to explain.
For example, a missing required link to code calls for the checking procedure defined in the rules. You cannot simply deduct points for "technical complexity" if that criterion evaluates something else. Teams need to know the consequences of a violation before submission.
Assign people to check whether submissions are complete and handle disputed cases. A judge given a few minutes for a demo should not also have to look for missing files and invent a penalty. That is a separate task for the organizer.
Describe each criterion through evidence
To assess whether the solution addresses the challenge, ask the team to show how its scenario relates to the published brief. For implementation, define what can be checked in the prototype or materials. For user testing, specify which observations teams should describe and how to distinguish them from assumptions.
Try not to assess the same property several times. If the persuasiveness of the pitch, presentation quality and the impression made by the speaker each carry a large weight, confidence on stage can overshadow the purpose of a technical competition. Keep only the distinctions you need.
Originality also needs an explanation. A new combination of familiar tools, an independent implementation and novelty in the challenge itself are different properties. Choose the definition that fits and explain what counts as sufficient evidence.
You do not have to start the matrix from a blank page. Adapt the Stavleak judging criteria template and check the weights and examples using practice projects.
An example scale for a working scenario
For the criterion "the main scenario works," you can use this example for practice:
Lower level: the action is described, but its execution is not shown.
Middle level: the main path works with the stated data, and limitations are disclosed.
Higher level: the main path can be reproduced, and important errors are handled and explained.
These are example descriptions, not a complete scoring system. The organizer sets the number of levels, weights and thresholds. Make sure a judge can apply the scale in the time available and a participant can understand it before development starts.
If the final result uses a weighted score, publish how it is calculated. Define in advance what happens when results are tied, a score is missing or a judge recuses themselves. Do not leave these cases to improvisation after looking at the leaderboard.
Calibrate the judging panel
Before judging begins, give the judges one practice project or a recording from a past event that you are allowed to use. Ask them to assess it independently, then discuss the differences. The meeting aims to build a shared understanding of the criteria, not to demand identical tastes.
Clarify what to do when information is insufficient. A judge can ask a question, note the lack of evidence or contact the organizer through the defined procedure. They should not infer test results from a speaker's confidence.
Check the judging form using the same example. Judges need to be able to save a result, correct an error before judging closes and report a conflict of interest. A technical walkthrough is more useful in a calm setting before the first real project.
Plan the workload and handle conflicts
Calculate the available time, including questions, transitions and recording comments. Assign projects so that each receives the required number of independent reviews. For specialist tracks, check that suitable experts are available.
Collect information about judges' connections to teams and projects through the organizer's working process. Mentoring, working together or a personal interest may require a judge to recuse themselves. Define the rule in advance and apply it consistently.
The MLH judging plan covers project allocation and a trial run of the process separately. Choose a ranking method that fits your event and tell participants before the final.
Collect useful comments
Ask each judge to record an observation, the basis for the assessment and one question or next step. "Weak implementation" does not help explain a result. It is more useful to state which claimed scenario could not be reproduced and what information was missing.
Separate comments about the presentation from assessment of the product. If the demonstration was unclear, say so directly. Do not assume a team lacks a feature merely because it was absent from a slide when the conditions allow the materials to be checked separately.
Clarify which comments will be shared with participants. Judges need to know who will read their notes. Remove information about other teams and personal judgments about people; feedback should concern the work presented.
Check the result before announcing it
Check that all required scores are present, the formula is correct, recusals have been handled and formal requirements have been met. Save the final version and identify who is responsible for confirming it. If you find a data entry error, correct it through a transparent procedure that records the reason.
Do not change the weights after seeing the leaders. If a situation arises that the rules do not cover, record the authorized organizers' decision and explain how it applies. Then use the lesson when preparing the next set of rules.
Announce the results after the checks are complete. The procedure for participant requests and the issues that may be reconsidered need to be described in the hackathon rules. This helps distinguish correcting a factual error from disagreeing with a professional assessment.
