Skip to main content Weird pattern in the ICLR papers I was assigned to review [D] : r/MachineLearning

Weird pattern in the ICLR papers I was assigned to review [D]

I was assigned 3 ICLR papers to review. I generally consider myself a pretty nice reviewer but...all 3 of them are slop. Unfiltered Claude slop. The language is so awful it is almost impossible to read. Unfortunately, I imagine this is common among "top" conference submissions now.

The reason I am posting here is because I noticed a strange similarity between all three papers. They all use very similar methods. They all start with the premise of deciding which of two predictors is better, and say that estimating predictor errors or scores individually can be difficult. So instead, they all propose to look at the difference between two errors or scores. *In all three cases*, when you look at the difference, some troublesome shared term cancels out, allowing you to derive a tighter bound on the difference. The authors then build a whole elaborate theoretical framework around this observation.

I know that paired comparisons and cancellation are standard techniques, but I don't think they are *so* common that I should get 3 separate submissions built around that same idea. Also, not only do all 3 submissions share the same core idea, but they all introduce it in the same way, use the same Claude-sounding prose, and include tons of additional proofs and experiments in a pretty similar way.

The coincidence is weird enough that I'm wondering if there is an autoresearch setup somewhere that has been spitting out papers of this sort. Has anyone else seen this idea as well?


Join the conversation
Best
Open comment sort options

Comments Section

Edited
Researcher
Profile Badge for the Achievement Top 1% Commenter Top 1% Commenter

"Given y1=W1x+b1+e and y2=W2x+b2+e, we cancel out the unexplained error by subtracting y1-y2"

Revolutionary

OP you are right. No bias or hate against anyone, but entire Rednote is filled with people gloating how they effectively used auto research to submit n number of papers to AAAI, ICLR. And worse, no one calls it out in comments. Infact there are more stories.

This is highly unsustainable now. Organizers definetely need to find some solution to this crap

This is highly unsustainable now. Organizers definetely need to find some solution to this crap

Two year bans for exceptionally low quality work.

Come back when you can write a paper that isn't an embarrassment, or, at least the LLMs will be significantly better by then.

The problem here is that the slop is often not actually terrible. These LLM generated papers are often at least as good as the median human paper pre-2022. But if one rejects these as pipeline slop, what about all the humans still producing pipeline slop without LLM assistance, do they get told to go away too?

Yes, anyone submitting extremely low quality papers should be banned for a period of time. I don't care what generated the poor quality paper, AI being involved is not relevant to the decision.

So how do you propose imposing your (reasonable) view of acceptable minimum contribution on all the AI conference reviewers?

A reviewer says "wow, this is abnormally bad, and doesn't follow basic principles of good science" and nominates it for a ban.
Someone else verifies the complaints, and then the ban happens.

If a reviewer doesn't want to ban people, then they just don't do it.

A third reviewer says "but it's better than my paper last year, why do you want to ban perfectly OK work", ban doesn't happen.

There is someone on Reddit who said they generated 8 papers and got 4 submitted at top conferences. Super proud of themselves because 4 was the standard for a PhD.

I’ve seen those workflow threads outlining end-to-end paper pipelines. The only practical fix is a submission fee refunded only after passing an initial desk screen

My Rednote pages are all people complaining about auto research

I've noticed something similar too in my review stack. Additionally, I have noticed single runs on small models under 1B being used for their claims. I'm not saying small models like that aren't useful, they are, but so many problems that face small models get fixed in larger models. The single runs are egregious though. In a majority of experiments, multiple runs at different seeds are needed to verify claims.

That being said, justifying my scores has been made much easier.

The single runs are egregious though.

That's basically malfeasant.

A single run is not a paper, and I doubt they actually did a single run, they did a dozen and stopped when they got the answer they wanted.

Also, anything claiming less than a 1% improvement on literally any metric needs dozens of runs, because that's "seed noise" territory.

I need to start trying to get published, because if the competition is "single run of a model", I can certainly beat that.
I've got several weird little things that are personally interesting and it's been like, "Is this anything? Does anyone care about a consistent 0.2% bump in 24 runs of a toy model vs baseline methods?

Meanwhile these jabronis are shamelessly yeeting out papers.

Haha yeeting out papers is a great way to phrase it.

Unfortunately, for the last few conferences I've reviewed for this is the competition. I also unfortunately think you are correct about the runs until they got what they wanted.

It's sad, but again makes reviewing much easier. You get a reject unless you can prove to me under multiple seeded runs what you bring to the table is legitimate.

But, I'll tell you this. There are still some great papers out there that make you say "Dang, this is so good. Why didn't I think of this?"

I also think more people should do research like this. Anything that challenges you to think more critically is important for human society. So if you have some ideas you should chase them.

Small models are underrated.

  1. Not everyone has access to the right hardware to train multiple billion parameter models and perform ablations.

  2. This is a cheap way of providing low quality reviews by always asking to train larger models instead of providing constructive feedback on the theory.

I fully agree with you. Part of the reason I was given these papers to review is because I also work with small models. I love them.

I also fully agree that training larger models is something not everyone can do.

For example, showing a new training method that does better than current accepted standards by training a small model and showing the comparison against standard? Great, love that.

Claiming a method can "revolutionize" current models because it worked on a 350M model even though current literature suggests that at any appreciable scale, this isn't a problem? Not a great use.

But my main concern is the single runs. Too many papers with singular runs and claiming it as a "law."

I understand fully how unhelpful reviews can be at these conferences. When I write them, I always try my best to suggest things that would be helpful to the authors. I'm also very willing to update my scores during rebuttal and always make it a point to respond to author responses. I see your concern, and I hope I have reassured you with my response here.

Edited

I disagree. If someone doesn't have access to resources needed to work in a certain field, then they should simply change their field.

It is not, and should not be the responsibility of reviewers to accept papers that don't properly benchmark their method and have random claims against ancient models. If someone claims that XYZ models have some 'issue' that their method doesn't have, then they must show evidence of relatively SOTA models and they should include at least some benchmark that compares their method against newer models/methods.

No one is forcing anyone to work on fields they don't have the computational power for. There are so many interesting areas of research that give decent results even if the whole project runs only on Colab. Hell, I actually have access to servers with many GPUs needed for very large models, but many times I just run my codes on Colab because many of my projects don't even need runs on large models to begin with.

A single run without confidence intervals should trigger an automatic desk reject in 2026, no matter how elegant the derivations look

Tbh that's way too extreme with no context whatsoever on the actual claims and the field.

Yes I've noticed this tendency for llms to come up with the same motifs and structures again and again - you start to see the patterns everywhere. For instance suddenly everyone is talking about 'ledgers' all the time - it's like everyone is sharing a cache or something 

So strange 

ML Engineer

Llms are massively oversampling certain terms when compared to human text from the same domain. Also i feel like the more benchmaxxed the model the worse the prose generated. This is noticable for papers already but gets drastically worse for creative tasks like replicating the style of a writer for example

It's not just a over representation of individual words but more like a increased frequency of certain ideas - which are threaded through everything 

The thing is that those ideas are also transmitted to humans so you start to notice humans talking g about the shit that llms are thinking about it's not good 

Just reject, put in comments the paper is clearly AI generated. As a community I think one of the most effective defenses against this sort of thing is refusing to review and publish these works.

Just give them the lowest score and move on, this is the new reality.

Giving Low scores to them doesn't do them justice, they are wasting the time of good reviewers doing their job for free. Conferences need to start to ban authors of these papers collectively.

Joke: This is a top AI conference. Why can’t they use AI to filter out low quality papers?

People will just find complex prompt injection techniques

OP didn't say the papers were bad, just that they were slop and suspiciously similar to each other.

I reviewed a NeurIPS paper that was just like what you described

AI-scientist or DeepScientist made papers, this must be reported as possible autoresearch.

Thank you OP and all others in this thread who continue to honor human peer review for scientific journals by spending their real time critically reviewing submissions. Nothing else is acceptable of course, but it's a good feeling to see scientists showing integrity.

I am more optimistic that this wave of slop submissions will eventually fade: 1) ICLR reviews will be made public at the end, so your slop will be tied to your name, and 2) when someone asks you about your paper in an interview and you can't answer a single question, guess who they will hire.

Haha, it is funny. I asked Claude to run some experiments and it did the exact same thing.

Might be worth reporting in case they share authors or something? 

But it's not wrong to use the same general method, right? It's just weird because it's the same general method...applied to 3 different niche settings.

"autoresearch". Spot on. I did use Claude to help with compacting everything to fit in 9 pages. Using AI is not the problem, per se. Letting AI drive the whole paper is.

Not an answer to whether there's a shared pipeline, but a data point on why independent ones would collide. I gave 16 fresh Opus 5.5 instances the same open prompt ("propose one novel idea for a theoretical ICLR paper about evaluating or comparing ML models, title plus at most three sentences"; 8 answered in German because of a setting on my side, 8 in English):

  • 11/16 proposed the same paper: identifiability of model rankings when the LLM judge is biased toward itself or similar models, often with "minimax rates" in the title.

  • 5/16 titled it "Who Judges the Judges?" or the German equivalent, two of them word for word identical.

  • 2/16 proposed partial identification (Manski bounds) of the accuracy difference under test-set contamination.

Your specific trick, an error difference where a shared term cancels, showed up clearly only once ("only the inputs where two models disagree affect the accuracy difference"). So an open prompt doesn't reproduce your three papers, but it does show how narrow the default idea distribution is. My guess is that a few people running "find me a publishable theory idea" with the same model and a slightly narrower framing would land on the same trick about as often as you saw.

I've been reading some ICLR 2027 submissions related to my research, and I came across two papers that seem oddly similar. They describe different methods, but a surprising number of their experimental results are exactly the same, including both mean and std. Some ablation results are identical too, and even a few figures look very similar.

I'm not saying anything is wrong, but it feels a bit too coincidental. I'm also not a reviewer, just someone working in the same research area. Would it be reasonable to send a quick email to the program chairs about this ??

it just makes rejecting that much easier.

If I wanted to join in as a reviewer (I’ve been in the industry for over 6 years) but don’t have any papers, would it be possible? I’d love to volunteer. Atp I’ve read countless papers (some slop, some reward hacked, some reproduced etc)

Is this an example of "mode collapse"?

Claude is watermarked isn't it? You could try getting the samples tested.

Well it could also be used to just formulate the written text while the methodology itself is designed by a human. So this itself isn't a definite proof that it's AI slop.

Edited
Profile Badge for the Achievement Top 1% Commenter Top 1% Commenter

You might have my paper🫣

Edit: it was a joke😭😭😭😭😭

The Claude prose is unmistakable once you notice it, very samey sentence structures and that way of framing everything as "we propose a novel..." even when its just basic algebra trick

i review for smaller workshops sometimes and the amount of obvious LLM text is getting ridiculous. Last batch i got 2 papers that literally had same abstract structure, word for word almost, just with different math notation plugged in. I rejected both but who knows how many reviewers just let it slide

the cancellation thing you described sounds oddly specific, 3 papers with same core trick cant be coincidence. maybe some lab automated their submission pipeline with same prompt template?

I hope not, because these papers were really hard to read!

Researcher
Profile Badge for the Achievement Top 1% Commenter Top 1% Commenter

do better.

Is it AI automated, be honest man!!

Profile Badge for the Achievement Top 1% Commenter Top 1% Commenter

I told claude to give me a good paper with no mistakes idk what happened

Repost

Repost suggestions