LLM-as-a-Judge & reliable automated evaluation
How LLMs are used to judge software artifacts — and what it would take for their verdicts to be trustworthy.
LLM-as-a-Judge for SE 2026Software engineering AI systems
Junda He
Ph.D. Candidate Singapore Management University
I am a Ph.D. candidate in the SOAR group at Singapore Management University, advised by Prof. David Lo. Previously, I received my M.Sc. in Software Engineering and B.Sc. in Computer Science from University College London.
Background
Education
- 2022.08 – PresentPh.D., Computer ScienceSingapore Management University, SingaporeSchool of Computing and Information Systems · Advisor: Prof. David Lo
- 2019.09 – 2020.12M.Sc., Software Engineering (Distinction)University College London, United KingdomDepartment of Computer Science
- 2016.09 – 2019.06B.Sc., Computer Science (First Class Honours)University College London, United KingdomDepartment of Computer Science
Honours & Awards
- 2025PhD Research Excellence AwardSingapore Management University
- 2025SCIS Dean's ListSingapore Management University
- 2024, 2025SMU Presidential Doctoral FellowshipSingapore Management University
- 2024ACM SIGSOFT CAPS Travel FundACM SIGSOFT
Experience, funding, talks & service
Experience
- Research EngineerSingapore Management UniversityAI for software engineering: LLM-based agents, benchmark construction, and evaluation methodology.
- Research AssistantChina Academy of Industrial Internet
- Software EngineeriCarbonX
Grants & Funding
- 2023 – 2025TitanCA: Vulnerability Detection using Large Code ModelsGovernment Technology Agency (GovTech), Singapore — total award over SGD $3.8M
Invited Talks
- 2025.11From Code to Courtroom: LLMs as the New Software JudgesWorkshop on the Next Generation of CodeLLMs and Their Applications, Korea University, SeoulInvited by Prof. Dongsun Kim
- 2025.10When LLMs Judge Software: Toward Trustworthy Automated EvaluationLab seminar, Osaka University, JapanInvited by Prof. Raula Gaikovina Kula
Teaching
- 2026.01Course DevelopmentSingapore Management University, SingaporeCS706: Software Mining and Analysis — contributed to course design discussions with Prof. David Lo and prepared lecture slides
- 2023.10Course DevelopmentNorth Carolina State University, USACSC791 (graduate special topics): designed laboratory exercises and instructional materials for the course
Professional Service
- CommitteesJunior Program Committee, MSR 2026, 2024 (Technical Track)Shadow Program Committee, ICSE 2025 (Research Track)
- Conference reviewerAAAI 2027ACL 2025ICLR 2027UIST 2025
- Journal reviewerTSE (Distinguished Reviewer)TOSEMEMSETISTMachine LearningNeurocomputingCACMJSSAutomated Software Engineering
- Sub-reviewerICSE 2022–2024ASE 2022–2025FSE 2023–2024SANER 2023
Selected Software
- Code generation benchmark with diverse function calls and complex instructions (co-author, ICLR 2025). Public leaderboard of 160+ models; built into AI2 OLMES, NVIDIA NeMo-Skills and OpenCompass.
- Pre-trained-model tag recommendation for Stack Overflow posts.
Research
I develop methods to evaluate and test LLMs and AI agents, with a focus on their reliability in software engineering.
Reliable evaluation
Understanding when we can trust LLMs to judge software artifacts.
LLM-as-a-JudgeBetter benchmarks
Measuring code-generation capabilities with realistic, complex tasks.
BigCodeBenchAgent testing
Finding failures in learning agents through systematic testing.
Deep RL testingAll research areas & related work
Testing & securing reinforcement-learning agents
Curiosity-driven and self-referential differential testing for deep-RL and sequential decision-making systems; runtime safety shields, safety-violation search, and backdoors in offline RL data.
CureFuzz 2024 Self-referential differential testing 2026 Runtime shields 2026 Safety violations via proxy programs 2025 Baffle 2024LLM agents for software engineering
A literature review and vision for LLM-based multi-agent systems in SE, and chaos engineering for agent systems.
LLM multi-agent systems for SE 2025 AgentChaos 2026Benchmarks & evaluation for code LLMs
BigCodeBench, whose public leaderboard covers 160+ models; a critical assessment of SE benchmarks; reference-free evaluation; and data leakage across 83 SE benchmarks.
BigCodeBench 2025 SE benchmark assessment 2025 Reference-free BRE evaluation 2026 LessLeak-Bench 2025Representation learning for developer content
Pre-trained models for Stack Overflow posts — tag recommendation and a systematic study of post representations.
PTM4Tag 2022 PTM4Tag+ 2025 SO post representations 2024News
- 2026
AgentChaos: Chaos Engineering for Agent Systems was accepted at ASE 2026.
- 2026
Beyond Text Matching, our work on reference-free evaluation for binary reverse engineering, was accepted at ASE 2026.
- 2026
Learning from the Test, our work on self-referential differential testing for deep RL agents, was accepted at ISSTA 2026.
- 2026
LLM-as-a-Judge for Software Engineering, our literature review and research roadmap, is published in TOSEM.
- 2026
Our community guidelines for empirical SE studies involving LLMs were accepted by EMSE.
- 2026
Our systematic review of AI support for software architecture practice is published in TOSEM.
Show 11 more updatesShow recent updates only
- 2026
Identifying and Mitigating API Misuse in Large Language Models is published in TSE.
- 2026
Our work on efficient and permissive runtime shields for neural policies is published in TOSEM.
- 2026
Compiling Code LLMs into Lightweight Executables was accepted at FSE 2026.
- 2025.12
Our survey Assessing and Advancing Benchmarks for Evaluating LLMs in SE Tasks is published in TOSEM.
- 2025.11
Invited talk at Korea University, Seoul: From Code to Courtroom: LLMs as the New Software Judges.
- 2025.10
Invited talk at Osaka University: When LLMs Judge Software: Toward Trustworthy Automated Evaluation.
- 2025.05
LLM-Based Multi-Agent Systems for Software Engineering is published in TOSEM, as part of its 2030 roadmap special issue.
- 2025
BigCodeBench was accepted at ICLR 2025 as an oral presentation.
- 2025
Our work on finding safety violations through synthesized proxy programs was accepted by TOSEM.
- 2025
Received the SMU PhD Research Excellence Award and was named to the SCIS Dean’s List.
- 2024.11
PTM4Tag+, our work on Stack Overflow tag recommendation with pre-trained models, is published online in EMSE (Volume 30, 2025).
Get in touch
Let’s connect.
For research conversations and collaborations.
jundahe.2022@phdcs.smu.edu.sg
School of Computing and Information Systems
Singapore Management University · Singapore
Selected Publications
Google Scholar
Show 23 more papersShow selected papers only
Figure 1 Source 
Figure 5 Source 
Figure 9 Source 
Figure 2 Source 
Table 2 Source 
Figure 1 Source 
Figure 3 Source 
Figure 1 Source 
Figure 3 Source 
Figure 1 Source 
Figure 2 Source 
Figure 3 Source Leveraging Large Language Model for Automatic Patch Correctness Assessment
TSE 2024
Figure 4 Source 
Figure 3 Source 
Figure 1 Source 
Table 2 Source 
Figure 3 Source 
Figure 2 Source 
Figure 2 Source 
Figure 2 Source 
Figure 2 Source 
Figure 1 Source
No papers found. Try another title, author, venue or year.