14 minute read

🇰🇷 이 글 한국어로 보기

Hi, I’m Kakyung Kim, a bioinformatics researcher.

On October 29 I presented four weeks of measurements on AI paper-review tools at PseudoLab’s Villain Night. Precision on the same set of detections ranged from 21.1% to 100% depending on what we counted as a hit, and the grounding check a cloud provider ships by default scored the same true sentence 0.97 and 0.07.

Three other project talks ran that evening: quality screening for 3D assets, memory and procedure in agents, and an experimentation platform for product changes. All four were stuck at the same point, which is how you know what a change actually did when you have nothing to compare it against. This post walks through those 99 minutes, from the seat of someone who was both presenting and listening, and includes the part of my own talk I ran out of time for.

Title slide from the Villain Night 2026 talk: villains who plant defects in papers and the hunters who chase those errors down, under the Falsify project name

The Question Left Over When AI Reviews Your Paper

The talk opened on something most people now do. Handing a draft to an AI for review has become routine, and what stays unanswered is how you would know whether that reviewer is actually seeing anything. A missed error is invisible to you, and so is an invented one.

Measuring it needs a problem whose answer you already know. So we plant defects in open-access papers (CC BY) and have a separate, blinded session hunt for them. The planting side is Red, the hunting side is Blue, and the roles swap the following round.

Slide showing the Red and Blue handoff: Red plants defects in the original paper, Blue hunts for them blinded, and scoring compares the findings against the answer key, with roles swapping next round. Below are the four planted defect types: number mismatches, invented citations, conclusions outrunning the evidence, and misreported result status

Figure 1. The Red and Blue handoff and the four defect types we plant. A value in the body that disagrees with the figure legend; a citation pointing at a panel or reference that does not exist; a conclusion covering all lineages when only two were examined; samples excluded late but written up as excluded from the start.

One design choice mattered more than the rest. The hunting side never sees the original. The first time we demoed this, we handed over both the original and the tampered copy, and the tool simply diffed the two files and reported what changed. The score looked excellent, but it was answer-matching, not detection. Now only the tampered copy goes out, the number of plants is withheld, and the scoring session is kept separate from the hunting session.

Precision That Shifts Fivefold With the Definition

In round one we planted eight defects and the hunting side found all eight. It also reported 38 findings in total. Sorting them out: eight were our plants, 19 were inconsistencies already present in the original paper, 11 were judgments that a conclusion outran its evidence, and zero were wrong on the evidence.

Left panel: a bar chart showing precision on the same detections splitting into 21.1%, 42.1% and 100% in round 1 and 23.8%, 83.3% and 100% in round 2, depending on what counts as a hit. Right panel: a bar chart showing the grounding check scoring a true sentence from the paper 0.97 when it sits inside the supplied source and 0.07 when it does not, while a planted false sentence inside the source scores 0.99

Figure 2. The numbers move with the definition and with what the source contains. (a) In round 1 we planted 8 defects, all 8 were found, and the hunting side reported 38 findings, so precision takes three different values depending on what counts as a hit. (b) The same true sentence scored 0.97 when it appeared in the supplied source and 0.07 when it did not. A false sentence we planted scored 0.99 when it appeared in the source.

Depending on what you count as a true positive, precision came out at 21.1% (8/38), 42.1% (8/19), or 100% (8/8), a spread of nearly five times. Round two repeated the pattern at 23.8%, 83.3%, and 100%. The conclusion matters more to us than any single number: if the source paper already contains inconsistencies, a better detector scores worse on precision, because everything outside the answer key counts against it. Without measuring the quality of the source paper first, precision stops meaning anything.

Four Detectors on One Paper

We ran four detectors over the same defect-planted paper in turn: the grounding check a cloud provider ships by default, a model called from that cloud, a general-purpose agent given instructions and a rubric, and the tool our own team built.

Slide comparing four detectors. Tier 1, the cloud's built-in grounding check, catches only invented citations. Tier 2, a cloud model, does not give the same answer twice. Tier 3, a general-purpose agent with instructions and a rubric, detected all 8 planted defects. Tier 4, the team's own tool, has detection still in development. The first two ran on AWS, the last two on our own laptops

Figure 3. The four detectors run over the same paper. The two general-purpose detectors ran in the cloud; the two we built ourselves ran on our own laptops.

Start with the built-in grounding check, which takes a source text and a sentence to test and returns a score between 0 and 1. On the first run, all twenty-four planted sentences scored 0.00, and I nearly wrote that down as perfect detection. A device that scores everything at zero produces exactly that result. So I fed it what should not have been flagged, namely true sentences from the same paper, and those came back at 0.00 as well.

Re-running with the conditions varied, the same true sentence scored 0.97 when it sat inside the supplied text and 0.07 when it did not, while a planted false sentence scored 0.99 when it sat inside. The score does not look at true or false. It looks at present or absent. That is not to say it is useless: it separated all 45 real-versus-invented citations correctly, and caught only 6 of 20 sentences where I had changed nothing but a number. It works for invented citations and not for contradictions or numbers.

I spent twice as many calls on controls as on measurement, and every conclusion came from the controls.

The Same Input Does Not Give the Same Answer

The models doing the reviewing wobbled too. Running the same input twice under identical settings, only one of three targets produced matching verdicts; swapping the model tier for light, mid, and large gave zero matches at every tier; repeating one model ten times split ten ways.

People varied the same way. When participants each ran the same tampered paper themselves, the number of findings ranged from 11 to 38, with 11 coming from a short prompt that blocked search and relied on background knowledge alone, and 32 from a run with the reasoning effort turned up. Same paper, same defects, and a threefold spread depending on who ran it and how. We use model-as-reviewer designs ourselves, and the lesson here is that whether that reviewer answers consistently has to be measured separately.

Paper Workbench, Manual Work Pulled Into One Screen

My talk handed off to a demo of Paper Workbench, built by Shinick Han, a participant in the study. It proposes candidate sentences to tamper with, produces a revised PDF from the edits you select, and keeps the model’s original answer alongside the human reviewer’s judgment. Planting a defect, pushing it into the PDF, and logging what changed used to be manual work attached to every single experiment, and the tool pulls all of it into one screen.

The screen that landed hardest lined up the numbers in the abstract against the same numbers in the body. Pick which sentence to change and how, and the choice flows straight into the revised PDF while the what and the why stay on the record. Four weeks of doing this by hand showed in the design, which is built by someone who knew exactly which steps were tedious. You also keep whatever AI tool you already use and bring its answer into Workbench.

He carved the time out of a full schedule to prepare and present this, and I am grateful for it. The tool is the clearest sign that measurement was not the only thing to come out of the study.

My own section ran 13 minutes 26 seconds, in line with the other talks, and the demo brought the slot to 27 minutes. The four slides on open problems, next steps, and the wrap-up are the part I ran out of time for, and they are written up separately on the PseudoLab blog (in Korean): the answer key that turned out to be wrong, the fact that the planting and hunting sides use the same family of tools and are therefore not independent, and what four weeks taught us.

Three Things That Move a Lazy Person, and a 3D Asset Catalogue

Ahead of the project itself, Jooyoung put up a laziness checklist: studying you keep postponing, articles saved to read later and never read, recommended books left unfinished, this year’s plan you cannot recall. Anyone matching four or more was asked to raise a hand, and most of the room did. Her definition of lazy was someone with plenty they want to do and no follow-through.

What moves that person anyway was the heart of the talk. Running a startup team taught her that motivation differs person to person, and that pressure never produced her own best work, whereas fun, people, and responsibility did: the fun of solving and learning, the members she works alongside, and a leader’s responsibility. PseudoLab sat at the intersection of the three. She found it in July this year and applied as a builder with no prior involvement.

The project catalogues 3D assets that fail in simulation: meshes that jitter at rest, drift when dropped, or sink into other objects on contact, with failures more frequent on jointed or soft bodies. The goal is a harness that judges, before you run anything, whether a mesh is likely to behave, at the level a human can see.

The motivation was convincing. In physical AI work, a 3D designer builds an asset and a simulation engineer tests it and sends feedback, and that loop is long because the person who finds the problem is not the person who fixes it, so issues surface late. A pre-check cuts the loop at the front. Asset selection for the evaluation set, 50 of them, starts next week, followed by the harness itself.

The team operations were worth borrowing as well: an in-person kickoff, pairs interviewing each other and then introducing their partner to the room, a working practitioner invited to speak in week two, and roles handed out early so everyone carried a small piece of responsibility.

Memory, Procedure, and a Report From an NVIDIA Seminar

Seoin’s talk came in two parts: the project, then a report from an outside event. Agent Experience Lab writes experience into memory and know-how into skills. Memory holds task background, rules, and what has been confirmed; skills hold the order of operations and the checks. The driving question is whether a change in working conditions calls for editing the memory alone or the procedure as well.

The design was clean. Plan A supplies memory only, Plan B supplies memory plus skills, and the two run under identical conditions. Weeks one through four cover study and practice while each member writes their own memory and skill files; weeks five through twelve put the two conditions side by side. They are in week four now, files written, comparison next.

The second half was a report from an NVIDIA RTX PC seminar where Seoin had spoken, an on-device AI session in a university hall titled “Today’s experience into tomorrow’s execution.” The audience was students, so it started from what separates cloud, local, and on-device. She spoke third and the room kept filling, which she admitted got to her. The proposed extension runs the same two conditions against local models on RTX GPUs and compares output quality, runtime, and reproducibility. The talk also argued that agents need an explicit permission boundary covering what to remember, how far to reach, and when to stop and ask a human. The same reproducibility problem I had reported, approached from the other side.

Apply First, Think Later, and a Hong Kong Workshop

Soohyun left the project content to a repository link and talked instead about her own path through the community. One piece of advice ran through the whole talk: apply as a builder first, think about it afterwards.

First contact came during season three as a job seeker, on Gather Town, when the pandemic had cut off most other routes. Being a developer rather than a researcher made applying as a builder feel out of reach, until it became clear there were many ways to contribute besides running a study. So the start was community work: writing and distributing a Discord guide, then running networking sessions, including one for roughly 250 people in a large hall at SKT T Tower where she built a bingo-style service to break the ice. She shipped a certificate-issuing system, and has led the experimentation platform project since the twelfth cohort. Agent work is what eventually put her on stage at events outside the community.

She applied for the Hong Kong program and was selected. There she sat on a seminar panel discussing AI in hospitality with university faculty, ran the multi-agent workshop, and joined an industry seminar. The workshop demonstrated a small system that reads hotel reviews and drafts both improvements and customer replies, to about 50 participants. What stayed with me was her saying that her English was not up to it, and that answering questions is what finally loosened her up. She closed by telling the room they were next, and to go apply.

One more thing came up in the discussion afterwards. In Hong Kong, ChatGPT, Gemini, and Anthropic are blocked and DeepSeek is what people use, so the workshop participants could not reach the tools they normally work with. Which tools you can open depends on where you are, and teaching plans are tied to that.

A Figure Card From Ten Questions

After the talks, the organizers shared a self-reflection prompt. You paste it into whichever AI chat service you use, trade about ten questions and answers, and end up matched with one of ten figures from computing.

Mine came back as a Demis Hassabis type, reasoning from a path that ran computer science to life science to molecular medicine, and from the fact that I now bundle all three into verifying research itself. The three other participants who read theirs aloud each drew someone different.

One observation landed hard: turning on conversation memory changes the result substantially. Same prompt, different accumulated input, different output. Exactly the problem I had spent four weeks measuring, reproduced in an evening’s party trick.

Meeting the Study Group Off-Screen

Falsify meets on Discord every Monday night. This was the first time I sat down in the same room as people I had only seen on a screen for four weeks. Sitting beside Shinick Han and watching his tool come up on the screen is not something that happens over video. Neither is watching someone’s face as they say, out loud, that they built the thing on the slide.

It was also the first time our measurements left the Discord server. The group had talked through the precision problem among ourselves many times, and explaining it to a room that does not work in the field showed me exactly where it fails to land. Most of the mistakes I later found in the recording were made for the first time in that room.

Networking worked by having everyone in the odd-numbered rows move to the back and sit four to a table. Each of us was holding a figure card, so there was no question of what to open with. Having sat through the same talks, the questions were specific.

Falsify started from a project in PseudoLab’s twelfth cohort, where I took a paper from idea to submission. What I brought to the table was one suggestion, that we try measuring this. Four weeks later there are two scoring sheets and a working tool. Ten participants carried it much further than I did from their own desks, and there has to be a place where that can happen.

The Control Group That Connected All Four

Writing this up, the shared thread surfaced. Every talk either built a comparison into its design or was blocked precisely because it had not.

Talk Compared against
Falsify Planted false sentences versus true sentences from the paper
3D asset catalogue Human pre-judgment versus actual simulation outcome
Agent Experience Lab Memory only versus memory plus skills
Experimentation platform Product before versus after a change

In my case, “we caught all twenty-four” was the result right up until I added a control, and only then did I learn the result carried no information. The asset work needs to measure how often human pre-judgment matches real simulation before the harness means anything, and the agent work needs both conditions on the same task before it can attribute a performance change to memory or to skills.

Watching four teams on four unrelated topics stall at the same spot, in one room, was what I took away from the evening.

Next

Falsify swaps roles at the next meeting and runs another round on a new paper, because planting teaches you how to hunt. We will also rerun the same material with conversation memory switched off, to see whether the results move.

It was good to meet the AWSKRUG user group organisers at the event as well. On Sunday November 22 I continue this work at AWS Community Day 2026, held at SpaceShare in Samseong-dong, Seoul: fifteen sessions across three halls, hands-on labs on Strands Agents and Lightsail, community booths, and an evening networking party. My session opens Track 3 in Delpino Hall at 14:00, immediately after the keynote, under the title “Automating Biomedical Research with Multi-Agent Workflows.” It asks who verifies the output when you hand research to agents, and extends the measurements in this post across several cloud models. Registration is open on the AWSKRUG event page.

Before then I will work through the recording and write out the sentences for every slide that carries a number. The recording is how I learned that I never once said the three precision figures out loud.

Updated: