TLDR: Writing professional reports with LLMs requires proactive mitigation of source bias and LLM narrative license behaviours. Reliable results are achievable by producing an internal editorial rulebook and implementing independent proofreading agents that verify compliance at specific writing stages.

Problem & Context

Preliminary was commissioned by a regulatory affairs consultancy to evaluate whether LLMs could be used to automate the production of a regulatory monitoring report. The current process involved three consultants reviewing 50–60 source articles across 15 languages, then compiling a 5,000-word report consisting of country regulatory updates and regional highlights covering broader trends in the region. The team collectively spent 20–25 hours each week on the project; the drive to automate was to free up consultants’ time to focus on higher-value work.

Objective

Our primary goal with this project was to reduce the amount of time the team spent each week on the task. To achieve this, the solution had to take the weekly inputs of 50+ articles and output a draft report in the house style for the team to work from. This consisted of article fetching, summarisation, thematic classification, writing in the house style, and assessing the significance of each update. Success was measured by the number of hours spent by each team member on the task to produce the deliverable.

Approach

As a pilot project, we decided early on with the client to build this as a skill that could be deployed via existing ChatGPT / Claude apps. This served as a technical constraint, but the initial trade-off for distribution was deemed worthwhile since it enabled faster feedback cycles.

Human intervention was set at three key points throughout the workflow. This aligned with our goal to streamline an existing workflow rather than automate it end to end. The intervention steps were:

• Initial article curation: team members decided weekly what was deemed of interest to the client and remained the starting point
• Outline review: our workflow produces an outline first with recommendations on which articles to include, exclude, or merge based on preset editorial constraints. The team can amend or approve the outline before proceeding further
• Report finalisation: the workflow produces a draft in the house style and applies editorial rules. The team then does a final round of edits and review prior to publication

For automation steps, we split the work into what would best be handled by deterministic scripts and what required LLMs:

Deterministic elements of the workflow were relatively simple to codify, so we will keep the discussion focused on LLM implementation for this piece.

The high-level workflow we implemented looks like this:

Challenges & Solutions

Lacking historical context: regulatory monitoring seeks to understand the trajectory of regulation and the drivers of change. Our client’s team has extensive experience in the field and can understand the nuance and implications of specific developments. Recreating this experience is objectively hard, as it is mostly captured in individual memory and lived experience. This is a substantial context and memory engineering challenge if the goal is to build an entirely automated workflow. However, for the purposes of our project, it is deliberately out of scope.

Mitigating secondary source bias: the team covers 15 markets with a total of ~300 regulators, and leverages local news media sources to surface salient developments. Their domain expertise and historical knowledge enable them to effectively substantiate a secondary source’s claims. No aggregator of regional primary sources exists, and building our own was out of scope for a two-week pilot.

LLM narrative license: sensationalisation and exaggeration are well-documented issues with modern LLMs and often manifested as reviewer feedback that the AI was “jumping to conclusions”. The report format had an “analyst perspective” section for each entry, where this behaviour was especially visible.

We mitigated these challenges in three ways:

1. Modified our country update prompts to retain source verbs, adjectives, and caveats when writing individual article summaries. While the sources themselves were not perfect, we would at least aim to match their specific language.
2. Added an independent proofreader agent to the individual entry writing loop to verify compliance, reverting to source language if drift was observed.
3. Built an editorial rule book within the skill to codify team practices specific to each section of the report. Guardrails with specific examples of what constituted a regulatory development further improved output consistency. Independent proofreader agents specific to each section were responsible for enforcement.

Our workflow retains a human review step, and all source articles are initially selected by the team, meaning that familiarity with the sources is expected. The mitigation steps above helped reduce instances of team members having to review sources again to verify specific details.

Results

The pilot programme has reduced team time spent on report writing from 20–25 hours to 10–12 hours each week, representing a 40–60% reduction.

The current implementation as a skill limits it to being an individual tool, rather than the ideal setup: a collaborative workspace akin to Google Docs.

While the concept can be pushed further towards automation, the current iteration strikes a strong balance between ROI and implementation effort.

What we would do differently

The Claude skill approach traded speed for transparency and control. We built intermediate state output, but it proved impractical due to length. In retrospect, building the workflow with a visual editor such as n8n or Gumloop would have provided more granular control for this project while still providing the ability to leverage ChatGPT / Claude as interaction surfaces.

Lastly, we also underestimated the eval burden the client team was willing to shoulder. With our initial run, we used a human-written report and produced a paragraph-by-paragraph comparison table with the AI version. The team could then pass or fail each paragraph and provide nuanced feedback as well. The review was never completed because it proved too time-intensive. Subsequently, we captured feedback by extracting team edits and codifying them into a living editorial rule book. For each run, we reviewed edits with the team to identify gaps, refinements, and contradictions. This proved to be the repeatable approach, as it surfaced existing practices while minimising the cost of the team’s time. How we integrate our feedback loop into the workflow is a practical consideration that should have been aligned up front.

Tech stack: Claude Skills Firecrawl Python