No. 21 · Technical

The Shape of Data

Making information into structured, searchable, auditable knowledge that an AI can reason over.

Most demonstrations of artificial intelligence tools look impressive for about five minutes. Someone uploads a folder of documents to a chat tool, asks a question in plain language, and a fluent answer comes back in seconds. The room is impressed, and it should be, because the tools and models that do this are genuinely powerful. The trouble starts the moment the work has to be repeatable, correct rather than merely fluent, comparable across thousands of items rather than answered one at a time, and defensible months later when someone asks how the system reached its conclusion. That is the work most of government actually needs, and it is not what a chatbot with a pile of files can do.

Over the past year, Technology and Innovation has built systems that do this harder kind of work for ministries across Alberta. They read entire libraries of legislation, the full provincial curriculum, and large bodies of policy and technical documentation, and they turn that raw material into knowledge an AI can reason over reliably, cite precisely, and be held to account for. Behind every one of these systems is the same machinery: a pipeline that takes ordinary documents and makes them, in a word, AI-ready. This paper explains that pipeline in plain language, so that a leader who will never write a line of code can understand what it actually takes to build an AI system worth trusting, and can tell the difference between a demonstration and a reliable system.


## §01 Why the easy tools are not enough

The most commonly used AI tools today are conversational assistants. Microsoft Copilot sits inside the documents people already work in and drafts, summarizes, and answers questions about them. Google's NotebookLM takes a set of sources a person hands it and lets them interview that set, and it will even narrate a tidy audio overview. These tools are excellent, and Alberta uses them. For reading a long report, drafting a memo, or getting oriented in a handful of files, they save real time, and we recommend them for exactly that.

They were built for an individual working with a small, temporary set of documents, and that is the limit of their design. They hold your files only for the length of a session, with a limited amount of text in view at once, so they cannot keep an entire ministry's corpus in mind. They answer in free-flowing prose, which reads well and cannot be sorted, counted, or compared across ten thousand items. They will summarize a clause, but they will not reliably tell you which clause, on which page, of which document an answer came from, and they keep no durable record you could audit later. When the session ends the answer evaporates and nothing is built, and no shared, structured body of knowledge is left behind for the next person, or the next tool, to stand on. Recreating an exact answer, or saving the thinking which went into the answer, is often impossible.

The use of AI for complex government tasks fails when we cannot audit the work. A cabinet decision has to trace to its evidence. A benchmark across fourteen jurisdictions has to be defendable and rooted in rational and measurable principles, or the comparison means nothing. And the value has to outlast just one question or interaction, because the next program, and the one after it, will need the same body of knowledge. For that, the documents themselves have to be rebuilt into something an AI can work with at scale. This rebuilding requires a systematic approach for transforming information from incompatible formats to AI-readable insights and is a critical step in building any intelligence-driven product.


## §02 What AI-ready means

To make information AI-ready is to carry it through a fixed sequence of steps, each of which transforms or adds properties or structures the raw document did not have. Luckily, we can use a common series of sequences for nearly every serious system we build regardless of the subject. A document, like a PDF, a Word file, or a web page, is first transformed into static images of its pages, converted into clean structured text, and broken into a labelled hierarchy that mirrors how the document is actually organized. Each piece of that hierarchy is given a numerical fingerprint that captures its meaning. The fingerprints make the whole corpus searchable by meaning rather than by keyword. Across collections of documents, sections, or even sentences, a search retrieves the most relevant pieces. Those structured answers become the inputs to agentic systems which can interpret this data to build dashboards, custom tools, and the explanatory text, images, and audio for the human audience. And every step in this transformation is recorded, so the whole chain can be audited and reproduced.

Said more plainly, a real AI system needs seven things that a chatbot does not provide on its own: a durable **memory** of what it has seen, a way to **extract** clean structure from messy sources, a way to **retrieve** the right material on demand, the numerical fingerprints, called **embeddings**, that make retrieval work by meaning, a **search** built on them, a disciplined way to **generate** structured output, and an **audit** trail for all of it. The system works with structured data, not loose text, which helps make each answer comparable, countable, and traceable.


## §03 Getting the words out

The pipeline begins by gathering the source documents and tracking where each one came from. Every document captures its provenance, the exact address it was downloaded from, the moment it was downloaded, its size, and a unique fingerprint (the MD5 hash) of its contents that proves the file was not altered in transit. Anyone can rebuild the whole library from that record alone and arrive at the identical set of files. This step requires significant human judgement in setting the criteria for selection and validating that the dataset is complete. This is what people mean when they say "You need to get the data right" to effectively use AI. There are no real shortcuts here because human experts must curate this content, and it forms the foundation of everything that follows.

Part of this process is converting the hundreds of different file formats we use into a reliable structure. PowerPoint presentations, Word documents, and PDFs are difficult for machines to read reliably. The text inside a PDF is often locked in a layout that is lost or scrambled when ordinary tools try to pull it out, and multi-column pages, tables, footnotes, and stamped amendments come apart. Graphs and charts are also frequently lost when extracting out the content for the AI, losing the context and the intent of the source knowledge. So we treat the data the same way a human would, by viewing and interpreting the literal, hierarchical, and symbolic content as visual images. Each page of the source material is broken apart into separate bite-sized pieces (known as chunks), ready for AI analysis. The source documents, and their derived images, remain available as the ground truth for future reference or challenge.

Each page image is then handed to an advanced AI model, in our case Claude running inside Google's Enterprise Agent Platform, which is asked to read the page and return back clean structured text that preserves the structure a reviewer cares about: the headings, the section numbers, the footnotes, the page markers, and any emphasis. This format is called "markdown" and is one of two important document formats we will introduce. Documents carry meaning in their structures and relationships which go beyond the words on the page. This careful reading and conversion to markdown costs more per page than simpler and cheaper automatic text extraction (known as OCR), but for dense legal or curricular text the accuracy is worth it, because every error introduced here becomes costly rework downstream. Quality here needs to be audited, and the pipeline evaluated closely for quality.

Markdown format preserves the page formatting, but the next step interprets the shape of the document itself. Legislation, curricula, and policies all have their own hierarchies. An act divides into parts, parts into divisions, divisions into sections, sections into subsections and paragraphs and finally single sentences. To compare a piece of Alberta legislation against another province's equivalent, the system has to know which part of the document to compare. And because each document is different, and each government uses different formats and standards, it is impossible to define a rigid template. We ask AI to understand and define a structure for each document dynamically. The model reads the whole document and describes the logical drafting conventions it sees, then marks up the document level by level adding tags which define each part. Documents are scoped from high to low, and each section is further reviewed and analyzed deeper until even every sentence is extracted individually and tagged within this hierarchy. Comparison is possible across documents, their sections, divisions, and clauses, even when they look nothing alike.

At each juncture we add audits, reassembling the broken up document back to its whole, ensuring that not a single character is out of place from its original format. Strip every label back out, and the result has to be identical to the original text. It is a small but important test with a large consequence, because it means the structure the AI imposed never silently changed a word of the document.

Structure added, never a word changed 100% aligned. Every document in the corpus passes a mechanical check: strip the AI's structural labels back out, and the text is identical, character for character, to the source the page reader produced. The structure is merely annotation which gives meaning in the future steps.


## §04 Giving the words meaning

Clean, structured text is readable in its whole or in part. Words are ready to be converted into meanings. The next step makes it possible to search and compare documents and their chunks based on their meaning and the relationship between their concepts. Pieces that discuss similar things end up with similar lists, even when they share no vocabulary, so a clause about "protecting aquatic habitat" and a clause about "fish-bearing waters" land near each other although the words differ. We use a special kind of AI model called an 'embedding model' to convert meaning to numeric values which are stored in a database. The system is configured to search and find the nearest matches across millions of items almost instantly. Think of each number as a fingerprint of meaning for that document or part thereof. Once computed, these values are static unless the text itself changes, making retrieval and analysis fast.

These so-called "vector embeddings", numeric and searchable relationships between concepts, enable us to move past keyword search into semantic search. A person can type a phrase like "discretionary approval timelines", and the system returns the pieces from across the entire library whose fingerprints sit closest to it, regardless of the exact words they use. This makes recall of information much more human, since our own semantic memory systems work similarly. This retrieval step is the heart of what the industry calls retrieval-augmented generation. Instead of asking a model to answer from its general memory, which typically may produce confident errors, we first retrieve the specific, real passages that relate to a subject and then hand them to the model as the starting point to work from. AI models are highly capable when the ground truth information is provided, and hallucination and guesswork largely disappear.

With the correct data being added to the prompt as context, an AI agent can be asked to undertake a complex, structured task, and to convert that raw material into a highly standardized form. The system is forced to respond using a structured data format called JavaScript Object Notation, or JSON, an industry standard for managing hierarchical and reliable data structures. A layperson might think of it as a form, where different fields, known as keys or attributes, can only be filled in with specific values. Every AI-generated answer comes back with the same fields. Large quantities of tasks can be given to the AI, and the format of the answer is always enforced. Because the shape is fixed, ten thousand answers can be sorted, counted, filtered, and compared. This drives real insight from the analysis because answers to questions can be compared and deep analysis performed against a large corpus of knowledge.

Structured answers are the raw material for everything a human finally reviews. These structured AI outputs can flow to dashboards and heat maps, and can rank information in a way which enables visualization and review. They also provide predictable formats to generate other sorts of media, such as explanatory text, generated images, and narrated audio, a compressed analysis a busy executive can absorb. The machine assembles the information into a clear structure and makes recommendations. People still decide what it means.


## §05 Memory and audit

Once our knowledge is made searchable and retrievable, every request we send to a model can record an auditable set of inputs. Repeated questions return highly similar outputs because the inputs are defined by the embeddings and the outputs are defined by the JSON schemas. The deterministic inputs into the AI are stored, so we understand what we asked the system and we know how it started its task. And its outputs are stored in a structured format, meaning that these can be tracked and evaluated over time to monitor the performance of the system. Instead of creative, open text outputs, the AI is filling out a structured report, and providing the evidence of why it made a decision, recommendation, or analysis.

This structure enables a deep audit of the AI's performance. Every analysis run records the AI's reasoning: the documents it saw, the model it used, the version of the instruction it followed, and the date it ran. Analysis can be re-run numerous times against the same subject and audited for drift or difference. And it enables comparison between documents and the AI's output as the corpus itself changes. A reviewer can compare two versions of an act and observe how the AI's answers change. If someone asks, six months later, why the system recommended a particular action, we can show the precise clauses the AI considered, how they were compared, the wording that drove the conclusion, and the broader body of knowledge any answer was drawn from. That is what makes an AI analysis defensible enough to put in front of a decision-maker.


## §06 A regulator reducing red tape

Consider a provincial regulator with a legal duty to reduce red tape while preserving the intent and integrity of its acts and regulations. In Alberta, every ministry carries the obligation to find and remove unnecessary processes, delays, and duplication without weakening the protections the rules exist to provide. For a regulator that approves projects under a large environmental statute, the practical question is sharp: where does our own legislation make an applicant do more, or wait longer, than the equivalent rules in other parts of the country require, for no environmental benefit? Answering that by hand would mean reading an entire act against the acts of a dozen other jurisdictions, clause by clause, a process which could take a team months or years to complete manually.

This challenge was an excellent opportunity for such a data transformation pipeline. The regulator's own act and its regulations became the baseline, and the equivalent statutes from every other province, the territories, and the federal Crown were carried through the same steps described above, with dozens of acts from fourteen jurisdictions structured, fingerprinted, and searchable side by side. For each of the two hundred sixty-five sections of the baseline act, the system found the closest matching section in each of the other jurisdictions, which produced just under eight thousand precomputed comparisons. It then read each Alberta section against its peers through four defined lenses of administrative burden: whether the rule dictates how to do something rather than the result to achieve, whether an approval rests on open-ended discretion with no published criteria or deadline, whether a decision has any time limit at all, and whether Alberta requires its own permit for something already authorized elsewhere.

The result was a little over one thousand structured observations, findings, and recommendations, each marking where Alberta is more demanding than a peer and, where it is, naming the jurisdiction whose lighter-touch wording is the best model, with a concrete suggestion for how Alberta could achieve the same protection with less friction. Because the AI produces structured outputs, the findings roll up into a heat map a reviewer can read at a glance, by subject area, by lens, and by which peer jurisdiction keeps appearing as the cleaner example. Every finding traces back to the page of the source it came from, so a recommendation can be evaluated and defended down to the clause. The lenses look only for burden without benefit, and the analysis is told explicitly to set aside any case where Alberta is simply more protective.

To benchmark an act against the country about $200. Carrying thirty-one acts from fourteen jurisdictions through the full pipeline cost around two hundred dollars in model usage. The same pipeline works for other topics, like interprovincial regulation, occupational health, privacy, or tax law. Only the documents and the considerations change.

Following this analysis, the regulator now has an evidence base, built in hours, that turns a political question into benchmarked insights. The outputs of the agents present specific clauses where Alberta may ask more from regulated industries than our peers and presents a range of options in language and process for the regulator to consider. The tool is not specific to the environment. Swap the corpus and the objectives for a different ministry's documents and definitions of success, and the identical pipeline benchmarks any body of law. We have run this pattern of deep analysis and benchmarking dozens of times over the past year, across widely different subjects, to inform strategy and to give leaders a defensible basis for a decision.

"The metadata changes from one ministry to the next. The pipeline stays the same." · Paper 21 · The Shape of Data


## §07 Good for people, not for AI

Alberta's K-12 curriculum is published, openly and well, at LearnAlberta.ca. For a parent reading on a Tuesday night, a teacher planning a unit, or a student looking something up, it is a genuinely good resource, organized into clear pages a person can browse. It is exactly the kind of open public information a government should provide. But in its published form, it is almost unusable to an AI. The content is spread across thousands of separate web pages and hundreds of PDFs, it is carried in two different curriculum frameworks that organize the same subjects in different ways, much of it is locked inside documents, and none of it carries the structure or the fingerprints a machine needs to reason over it. It was built for a person to read, and an AI agent cannot work with it until it is rebuilt.

We developed an agent to browse this curriculum and pass it through this data processing pipeline. A crawler agent systematically gathered every curriculum page and every linked document, more than two thousand four hundred files in all, the page reader turned forty-two thousand pages of PDF into clean text. The material was tagged with more than fifty thousand curriculum 'tags', each one a learning outcome or a piece of one, a set of subjects, and a grade. The public learning management system became an AI-readable database of learning materials, objectives, and outcomes, with structured, indexed knowledge sets an AI tool can query by subject, by grade, and across its full text.

To make a province's curriculum AI-ready about $2,500. Turning the entire published K-12 curriculum into an AI-ready substrate cost roughly twenty-five hundred dollars in model usage. A government can do this once, and then any Albertan, a parent, a teacher, a student, can build on it.

With the curriculum already extracted, structured, and tagged to grade and subject, it becomes possible to build new tools, resources, lesson plans, and even new applications which are completely curriculum aligned in a matter of minutes. Once the data was processed, it took less than an hour to begin to build new learning solutions on top of this machine-readable curriculum. Technology and Innovation created a small grade-appropriate learning game, called Age of Alberta, which provided an interactive gamified experience in the style of Microsoft's Age of Empires, where the gameplay was curriculum-aligned math, reading, and homework exercises. The game took only an hour to build as a prototype because the data was ready for AI interpretation. A government can make a vast public corpus AI-ready one time, for the price of a modest laptop, and in doing so hand every parent, teacher, and student in the province the raw material to build their own tools on top, in an afternoon. Learning is grounded in fact, outcome driven in the curriculum, and ready for the AI tools parents, teachers, and students are already using.

"Because the data was ready for AI, building a curriculum-aligned, age-appropriate, custom learning experience only took an hour and was completed for less than $4." · Paper 21 · The Shape of Data


## §08 Building reusable data processing layers

These two systems look unrelated, one a legal benchmark for a regulator and one a curriculum for a classroom, but underneath they are the same machine. Documents in, structure and fingerprints in the middle, structured and audited answers out, and a durable body of knowledge left behind. We have built this same pipeline for legislation, for curriculum, for policy, for large bodies of technical documentation, and for market and competitive research across a range of sectors, dozens of times in the past year. Each time the subject changes, and the definitions of what matters change, and the pipeline does not.

Once the data is curated, and for little cost, it becomes fast to build meaningful tools and insights rapidly. Best of all, they are built for almost no cost and grounded in real, reliable data. The solutions carry the evidence needed to be challenged, audited, and to start serious conversations about how to introduce AI into a particular ministry or profession. A regulatory analysis prototype can be completed in hours for work which would be procured and completed manually over a year. And the work is auditable, because the data processing pipeline shows exactly what the AI worked with, and can be recreated and benchmarked. Rigorous data management practices are needed, but the payoff is profound.

For a leader deciding how to spend on AI, the lesson is to stop buying chat conversations and start building data pipelines and memory systems. A chat tool answers today's question quickly, but forgets just as fast. A pipeline turns a body of information into a durable, structured, auditable asset that answers this question and the next hundred, that other tools can be built on, and that, when the information is public, becomes a public good. The chatbot is the part everyone can see. The pipeline is the part that matters, and it costs far less to build than most expect. That is what it takes to make information AI-ready, and it is within reach of any government, and any Albertan, willing to build it.

Tags: data-pipeline, rag, embeddings, benchmarking, open-data, ai-systems

Open the interactive version