Agentic AI for data professionals
Data platforms now include assistants that can generate a SQL query, document a table, identify an anomaly or suggest a transformation. These capabilities are already changing how data practitioners design, develop and operate their pipelines.
But one question remains open: how do you move from assistants that can carry out one-off tasks to a reliable, reusable automation capability that can be industrialized across the entire data value chain?
That is the ambition of "MAD", the agentic harness framework developed by Niji for data professionals. MAD goes further than handing an agent to a data engineer: it provides a structured framework to encapsulate methods, orchestrate skills and automate complex data workflows, in a controlled environment that adapts to each organization's context.
The goal: to make agentic AI a true data delivery capability, serving quality, productivity and team upskilling.
Our conviction: agentic AI creates the most value when it is framed by well-mastered methods, standards and processes.
Divide and conquer
A data project never comes down to a single isolated task. It usually draws on several areas of expertise: ingestion, transformation, modeling, quality, governance, operations, visualization and machine learning.
In a traditional approach, these activities are carried out by different profiles, with methods, documentation levels and practices that are sometimes inconsistent. AI assistants can speed up certain steps, but they do not necessarily guarantee overall consistency. A relevant answer at one step is not enough to produce a reliable end-to-end deliverable.
MAD starts from this operational reality and breaks the data value chain down into roles matching the categories of skills expected from a data practitioner:
- Data Ingestion Engineer
- Data Transformation & Modeling Engineer
- Data Quality & Reliability Engineer
- Analytics & BI Engineer
- Data Platform Operations Engineer
- Data Migration Engineer
- Data Privacy & Compliance Agent
- Data Governance Steward
- Data Solution & Product Architect
- ML & AI Engineer
Each role has its own area of expertise and working principles. This expertise is documented in a skills catalog, made up of elementary capabilities that can be invoked directly or by an orchestrator agent.
The key shift happens when these skills are called in coherent sequences (workflows). MAD can then automate not just a single action but a complete processing chain, with identified steps, conditions, checks and deliverables.
Note: the number of roles is deliberately high, to divide skills evenly across the different roles. This fine-grained breakdown came out of several iterations: it makes it smoother to activate the right skills for each need, improving results and mechanically lightening the LLMs' context.
A 3-level model
Roles give a business view of the range of MAD skills. They bring the agentic organization closer to how data teams are actually organized, associating each role with capabilities, a scope of intervention and a level of expertise.
Skills carry the unit actions: assessing a data source, designing an ingestion, implementing a SQL or Spark transformation, detecting anomalies, defining business metrics, managing quality rules or preparing a migration. They turn the expertise built up by teams into reusable capabilities.
Workflows combine several skills and organize their execution in a deterministic way. They define the macro-steps, the conditions for moving on, the checkpoints and the expected outputs.
It is the combination of specialized skills through structured, deterministic orchestration that allows MAD to go beyond the isolated use of the assistants built into data platforms.
What MAD brings in practice
- Speeding up development and upskilling. MAD is a toolbox for data practitioners. Teams can rely on preconfigured capabilities to complete certain tasks faster, while benefiting from explicit, documented methods. Beyond speed, MAD makes certain practices accessible to a wider range of profiles, with a framework that makes learning, reuse and knowledge transfer easier.
- Systematizing best practices. Skills and workflows make it possible to build standards into the design of data deliverables. These standards can cover object structure, naming conventions, documentation, tests, quality, security and deployment methods. They can be configured to reflect the organization's context: technical stack, regulatory constraints, maturity level, security rules or architecture choices.
- Automating complex production. One demonstration case starts from a raw dataset and produces an end-to-end medallion architecture. The workflow successively delivers data analysis, ingestion design and implementation, quality checks, cleaning, semantic modeling, transformation into the Silver and Gold layers, pipeline orchestration and the generation of business insights.
- Preserving portability. MAD is built on a platform-agnostic foundation. The framework is structured as a Git repository to import, and can adapt to the tools and constraints of the target environment. This portability avoids rebuilding the same capabilities for each platform and makes it easier to capitalize on methods.
Use cases across the entire data chain
Configuring ingestion flows, generating transformations, setting up tests and preparing orchestration, within a consistent design and documentation framework.
Building the Bronze, Silver and Gold layers from raw data, with source qualification, cleaning, transformation, modeling and checks.
Managing quality rules, running checks, detecting anomalies, preparing a remediation plan and tracking indicators.
Classification, protection, access reviews and retention management, with a more systematic and traceable application of governance requirements.
Readiness assessment, resource mapping, wave planning, workload conversion and cutover validation.
Observed results
A first order of magnitude, measured on the initial tests.
In the initial tests, building an end-to-end medallion architecture for a business model made up of 4 tables and 50 fields was completed in 20 minutes, following Niji standards and with no intervention from the Data Engineer.
Lessons learned from using MAD
Workflows add an extra level of control
The native assistants in data platforms can often handle a large share of the tasks MAD covers. Their execution remains variable, however: technical choices, file locations, methods used or documentation levels can differ from one run to the next. Workflows add a layer of deterministic orchestration and checkpoints.
Skills must be designed to be orchestrated
A skill designed for one specific use is not necessarily ready to be plugged into an automated chain. The MAD experiment showed how important it is to standardize input and output contracts, clearly isolate unit tasks and precisely define the expected outputs.
The harness matters as much as the model
MAD's value lies less in finding a model that can do everything than in building a controlled working environment. The framework governs the methods used, where outputs are produced, the conventions and the technical choices allowed.
The cost is low, but must be managed
Initial tests show a cost of a few dollars to run a complex workflow (excluding compute). This LLM inference cost is low compared with the work done and the significant time savings over equivalent manual operations, but it is not negligible at industrial scale. FinOps tracking must accompany the framework as it scales, with regular assessment of the value produced.
How to activate MAD in your organization
MAD is an augmented capability that extends the expertise of your teams. Making it available therefore comes with an adoption program and a continuous improvement loop:
Identify one or more eligible use cases: configuring an ETL flow, implementing a medallion architecture, analyzing and remediating data quality. The test is run against the usual methods, with criteria for measuring the quality and efficiency of the work defined upfront.
Define the situations in which MAD should be used, adapt the standards to the organization's practices, add the necessary skills and workflows, set up FinOps tracking, train teams and gradually fine-tune the configuration.
This approach makes it possible to measure the framework's real value in the organization's context, beyond a technology demo. Once the value is validated, MAD can become a toolbox shared by data practitioners.
Take action. Want to assess MAD's potential on a concrete data use case? Niji supports you with framing, experimentation, value measurement and scaling.
