HackMyPaper
Sign inSign up free

Let's hack the reviewers who use AI.
(Use GPT-6-Sol)

What prompt would a reviewer use? If my paper were run 10 times with different prompts, roughly what score would it get?

And if I revise one part, will the score go up?

Sign up free, get 5 creditsHave an account? Sign in
  • Quick review from 3 credits
  • Automatic refund on failure
  • Papers never used for training

Sample report

Sparse retrieval-augmented question answering over long documents (sample)

NeurIPS 2026

6
Simulated score / 10
This review's simulated scoreOther scoresExpected score 5.65

Chance of scoring 6 or above: 65%.

Review (excerpt)

The problem is clearly defined, the division of labor between retrieval and generation is well explained, and the gains are consistent across three long-document benchmarks. The main weaknesses are the lack of a direct comparison with recent strong baselines and an ablation that does not separate the contributions of the retriever and the reranker.

The problem

AI reviewers roll dice. The dice are loaded by your paper.

We took a set of 200 papers covering several sub-areas and ran more than 3,800 reviews on them.

53%

of papers got the same integer score all five times

40%

landed on two adjacent scores, e.g. 5 5 5 6 6

7%

spread over three scores

If we assume a large model can tell good papers from bad ones, that at least means it can produce a stable ordering of papers. So assume there is a mapping from paper quality to score. But scores are discrete: what happens when a paper's quality falls between two levels? A paper that sits at “5.5” in the model's head is a coin flip at every review. Three lucky reviewers give you 6 and you are in; three unlucky ones give you 5 and you are out. Worse: you fix a figure, the score goes from 6 to 5, and you cannot tell whether your edit hurt or the dice did.

The instability is mostly a property of the paper, not only of the model's randomness: papers that score 5 5 5 5 5 keep scoring 5 on the tenth run too. What you actually need is each paper's own split between its two candidate scores. That is what the tiers below estimate.

Four tiers

Pick by what you need to know

Quick
3 credits · ~30 s

1 call, reasoning off. We read the model's probability over every score 1–10.

Returns a probability list and a decimal score (its expectation), e.g. 5.4 — 4: 20%, 5: 50%, 6: 30%.

Use for: comparing drafts, ranking several papers. The most stable tier we have. Not for reading an absolute score.

Single
3 credits · ~2 min

1 full reasoning review, like pasting your paper into the chat app: written feedback and one integer score.

Returns a review and one score.

Use for: reading a real review. Treat the score as one draw: two runs differ by a point 24% of the time.

Standard Recommended
5 credits · ~2 min

2 reasoning reviews + 1 Quick, each counting equally. The reviews fix the position, Quick fixes the shape.

Returns 2 reviews, a probability list and a decimal score.

Use for: “what will I get, and what are my odds above 6?” Matches 3 reasoning reviews on distance and 5 on likelihood, at half the price of Full.

Full
12 credits · ~3 min

10 independent reasoning reviews. No fitting: the raw histogram.

Returns 10 reviews and the observed distribution.

Use for: deciding whether an edit really moved the score, or when you sit on the acceptance line. The only reasoning tier that resolves half-point changes.

Results

How close each tier gets to “ask the reasoning model forever”

Target: the distribution of scores the reasoning model gives a paper across many runs. Grey rows are plain repeated sampling, for reference. Lower is better unless marked ↑.

TierCallsTop score hit ↑Scores covered ↑Distance to truth (L2²)Centre error (pts)≈ samples (L2)Acceptance AUC ↑Run-to-run sd (pts)MDC₉₅ (pts)
Quick160%100%0.2410.441.00.7140.100.28
Single172%68%0.2450.391.00.6860.391.07
Standard369%96%0.0840.242.90.7200.200.55
3 reasoning reviews377%86%0.0820.223.00.7060.220.62
5 reasoning reviews580%90%0.0490.175.00.7100.170.48
Full (10 reviews)1083%97%0.0250.12100.7130.120.34
∞ reviews (ceiling)∞88%100%00∞0.71600

Top score hit: our most likely score equals the mode of 10 fresh reviews. Scores covered: share of scores that actually occurred to which we gave ≥ 5%. Distance to truth and centre error: unbiased estimates against the model's limiting distribution, 200 papers. MDC₉₅: the smallest before/after change that is not noise. Calibration parameters fitted on half the papers, scored on the other half.

Full table (16 metrics)
TierCallsTop score hit ↑Scores covered ↑W1 (pts)Log-likelihood ↑BrierTVMean errorP(≥6) errorDistance to truth (L2²)Centre error (pts)≈ samples (L2)≈ samples (centre)Acceptance AUC ↑Run-to-run sd (pts)ICC ↑MDC₉₅ (pts)
Quick160%100%0.36−0.760.4970.330.3114.1%0.2410.441.00.80.7140.100.9870.28
Single172%68%0.28−1.270.5520.280.2813.2%0.2450.391.01.00.6860.390.8961.07
Standard369%96%0.19−0.550.3530.190.1611.3%0.0840.242.92.60.7200.200.9640.55
3 reasoning reviews377%86%0.18−0.710.3790.180.1810.4%0.0820.223.03.00.7060.220.9630.62
5 reasoning reviews580%90%0.14−0.640.3440.140.1410.0%0.0490.175.05.00.7100.170.9770.48
Full (10 reviews)1083%97%0.10−0.420.2790.100.104.2%0.0250.1210100.7130.120.9880.34
∞ reviews (ceiling)∞88%100%0.08−0.380.2550.080.083.2%00∞∞0.716010

Observed-target metrics (W1 through P(≥6) error): 10 papers × 10 fresh reasoning reviews; the 10 and ∞ rows are simulated from 15 reviews per paper. Log-likelihood floors zero probabilities at 0.01.

No tier predicts ICLR acceptance better than the others: all sit at AUC 0.69–0.72, and the ceiling for infinite sampling is 0.72. What the pricier tiers buy is a more faithful reproduction of what a careful AI reviewer would give you, and how much it would vary.

Method

Why one cheap call can describe a distribution

  1. 1

    Read the probabilities, not the number

    With reasoning off, the API exposes token probabilities. At the score position the model holds a probability over 1–10. The number it writes is one draw from that; the vector is the whole thing.

  2. 2

    Calibrate to the reasoning model

    Those probabilities are over-confident (about 5×) and sit higher than a reasoning review. We fit a temperature and a shift on 200 ICLR papers against 1,000 reasoning reviews, half for fitting, half for testing.

  3. 3

    Add reasoning reviews for position

    A no-reasoning model is a different judge: its per-paper bias (≈0.5 pt) does not average out. Standard mixes two real reasoning reviews (position) with the calibrated vector (shape, stability).

  4. 4

    Or just sample

    Full runs 10 reasoning reviews and reports the histogram. Error falls as 1/√n: ten reviews capture most of what infinite sampling would.

Other

  • Our technical report will be out before 15 October.

How it works

  1. 1

    Upload your paper PDF

    Up to 50 MB and 15 pages, or 45 pages with Long paper review. You can note the target conference or journal to tell versions apart.

  2. 2

    Wait one to three minutes

    The model reads the full paper and judges it point by point against the review criteria of top computer science conferences. The page refreshes automatically.

  3. 3

    Check the score, revise, review again

    Get a simulated score from 1 to 10, the score distribution and the full review. Standard and Full reviews run several independent reviews so you can see how stable the score is.

Pricing

Pay per review. Quick review from 3 credits, and credits stay valid long-term. Sign up and get 5 credits, enough for a free first review.

Sign up to buy credits
TierRunsCreditsDetails
Quick131 call with reasoning off: the model's probability over every score, as a probability list and a decimal score (its expectation)
Single131 full review with reasoning on: written feedback and one integer score
Standard352 reviews with reasoning on plus 1 quick run, each counting equally, combined into an estimate of the scores many reviews would give
Full101210 independent reviews with reasoning on, reporting the observed score distribution
10 credits
NZ$2.59
≈ US$1.47 · ≈ ¥9.8
10 credits · about 3 Quick reviews
About NZ$1.30 per Standard review
30 creditsSave 10%
NZ$6.99
≈ US$3.97 · ≈ ¥26.5
30 credits · about 10 Quick reviews
About NZ$1.17 per Standard review
50 creditsSave 15%
NZ$10.99
≈ US$6.24 · ≈ ¥41.6
50 credits · about 16 Quick reviews
About NZ$1.10 per Standard review
100 creditsRecommendedSave 31%
NZ$17.99
≈ US$10.22 · ≈ ¥68.2
100 credits · about 33 Quick reviews
About NZ$0.90 per Standard review

FAQ

Is the score accurate? Does it reflect the real review outcome?

No. It is a simulated score, meant to catch obvious problems before you submit. The model reads the full text and figures, but it does not know individual reviewers' preferences or unpublished progress in your field. Treat it as a careful peer who is not familiar with your area: focus on the issues it raises and take the score itself as a reference.

Will my paper be used to train models?

No. The paper text is used only to generate this one review. It is sent to a third-party model provider for processing, but it is not used for training and is never shown to other users. You can delete the paper text and the review at any time from the review page; deletion cannot be undone.

Can I get credits back if a review fails or I am not satisfied?

If a review fails because of a system error, the credits are refunded automatically, with no request needed. Completed reviews are not refunded because you are unhappy with the score. For questions about purchases or charges, email support@hackmypaper.com.

Which conferences and journals does it work for?

Scoring follows the general standards of top computer science conferences, so it best suits papers in machine learning, natural language processing, computer vision, systems and similar areas. The target venue you enter is used to label and tell versions apart; it does not currently change the scoring criteria. Papers from other disciplines work too, but the result will be less informative.

How long does a review take?

Usually one to three minutes. Queued reviews generally start within a minute, and the model needs 30–90 seconds. You can leave the page after submitting; the result is saved in your history on the Dashboard.