Aleksei Khvostov

AI safety & evaluation engineering.

I build evaluation harnesses, audit evaluator reliability, and test agent oversight. I’m interested in AI safety, evaluation engineering, and AI assurance roles, and collaborations on reproducible evaluations.

ResearchGitHubLinkedIn

Research

Evidence blocks

My public work focuses on evaluation reliability: comparing agent reports with execution evidence, auditing the evaluators, and documenting what small adversarial pilots can establish. Each project separates implemented tooling, observations, and unresolved questions.

Evaluation & Oversight

How do we know whether reports, metrics, and oversight claims stay consistent with the evidence?

Report integrity & measurement audit

Multi-Agent Safety Sim

A Python research harness for comparing agent reports with independently recorded execution state. Offline simulations and deterministic regression checks are reproducible. Two documented 75-request model pilots exposed classifier artifacts; their saved outputs are not public. Classifier-v3 was implemented for diagnostic evaluation but not adopted. Phase 3.8 closed after failed conformance reviews. Evaluator validity remains unresolved.

Python · Deterministic simulations · Evaluation harnesses · Measurement audit

Current research statusRepository & offline checksHistorical Phase 3.7 case study

Exploratory Russian-language evaluation

RuSafetyBench

An exploratory collection of Russian-language adversarial prompts, expected behaviors, and manually reviewed pilot reports covering authority framing, narrative manipulation, and crisis-resource localization. Findings are qualitative; consumer-interface outputs cannot be regenerated from the repository alone. The collection has no executable evaluator or validated model rankings.

Adversarial cases · Manual evaluation · Qualitative pilot reports

Cases, findings & limitations

Methodology

Working rules

How I keep evaluation results interpretable.

Ground truth first

Ground truth lives in the environment, not in agent reports.

Reproducible checks

Use deterministic tests to protect a measurement contract. Validate its meaning separately; passing fixtures alone is insufficient.

Measurement before claims

No finding without a measurement behind it.

Preserve originals under revision

When a measurement layer needs revision, keep the original results, preregister the change, and treat rescored outputs as diagnostic rather than replacement.

Observable over inferred

Separate what a system does from what we think it intends.

Explicit limitations

State the ceiling. Avoid overclaiming.

Bounded experiments

Start with a small, explicit question. Pause when the measurement cannot support the intended interpretation.

Criteria-Based Verification

A practical method for writing falsifiable, atomic, output-focused evaluation criteria. Observable results over process; compound conditions split; core correctness weighted over polish.

Notes

Measurement lessons

Lessons from building and auditing evaluation methods.

#001

A changed score can reflect a changed evaluator.

Compare the scoring rules before interpreting a metric shift as a model-behavior change. Preserve original results and label diagnostic rescoring separately.

#002

A report is evidence, not ground truth.

Record execution state independently, then compare the report and evaluator interpretation against it. Keep these three layers separate.

#003

A useful fixture result still needs a scope boundary.

In the Phase 2 study, partial-progress signals had the largest leave-one-out impact on detection across 24 authored cases with a rule-based planner and watchdog. This was deterministic fixture evidence, with no LLM inference or general oversight-effectiveness validation.

Phase 2 study and limitations

#004

Passing calibration is not semantic validation.

The Phase 3.8 structured-report extractors passed their authored calibration suites but failed fresh conformance reviews. The development line closed without adoption; passing checks did not justify a broader validity claim.

Current research state and closure

Questions

Current research questions

  • How can evaluator failures be distinguished from the model behavior being measured?
  • What independent evidence is needed to validate a report-integrity measure?
  • How should small adversarial pilots communicate observations without implying general model rankings?

About

Background

I’m an independent AI safety and evaluation researcher based in North Carolina’s Research Triangle. My professional background includes AI evaluation and technical editing with Invisible Technologies, Outlier.ai, and Perle.ai, and editorial leadership at PRIME Business News Agency, Rambler & Co, and Lenta.ru (2012–2023). That experience in verification, rubric design, and Russian–English localization QA informs my work on evaluator reliability and agent oversight.

Contact

Links

Contact on LinkedInGitHub