Agent Science and Engineering: The Future of CS and Data Science

Earlier this Spring I taught my revamped grad course on Data-Centric AI and AI Engineering. We covered evals, retrieval and context engineering, agent engineering, and LLM customization. I saw a curious fault line in the class this time. One student’s contemptuous take captured one side perfectly: “This isn’t a course, it’s like a bunch of GeeksforGeeks articles.” :) No beautiful math, no smart algorithms, no elegant system designs. But many others had a very different take: “Gives the best understanding of the ongoing AI revolution.”; “Directly relevant for my new AI job this Summer — glad to learn this systematically.” Same syllabus, same instructor, polar opposite reactions!

I realized the fault line is not noise but what a Kuhnian paradigm shift looks like from the inside. It is not just software/data jobs that are being disrupted by AI — the very intellectual foundations of Computer Science (CS) and Data Science (DS) are. My headline take: CS and DS are converging into a new field — I call it Agent Science and Engineering (ASE).

Read on if you are curious about the shape of this emerging paradigm, how it relates to the old paradigm, what is still missing, and what I think academia must do to not become intellectually irrelevant in this “AGI” era

My vantage point

Let me start by explaining my dual vantage point spanning academia and industry.

My research is on data management + ML/AI + software systems. One line of work I’ve followed is query languages/APIs—think SQL, (Py)Spark, other DSLs, natural-language-to-SQL translation, data agents, etc. For 50 years we have asked computers to turn “declared human intent” into generated low-level code. In fact, SQL was originally called “Structured English Query Language”! We state high-level intent and trust the generated code (more on this later). LLMs/agents have widened this aperture massively from just computing queries over data to virtually any computing task.

Over the last two years as CTO of RapidFire AI, I have seen this disruption by LLMs (especially coding agents) in industry too. Entire nation states are investing in AI at historic levels. This is not just another hype cycle; we are at a civilizational inflection point. And yet, I also see where Anthropic, OpenAI, et al. bury their bodies. And I see a trichotomy in the impact of AI on practice so far: prosumers/tech firms vs. consumers vs. enterprises.

The nature of the paradigm shift

The figure below captures my mental model of the shift. We used to move rightward from precise programming languages (PLs) with their context-free grammars (CFGs). Now we move leftward from free-form human languages that are naturally unruly while still expecting precision-like benefits. When people declare “English is the new programming language,” it makes me cringe — because they misapply an old-paradigm term for the new! The new-paradigm term I suggest is “Computer-use Languages” (CLs) to cover the whole spectrum from PLs to human languages to hybrids.

A key distinction in this emerging paradigm of CLs moving leftward vs. prior paradigm of PLs moving rightward is that the former is often application-specific, often even query-specific. The need for specificity did not disappear—even human-human interactions need that — we just start from free form and add specificity on demand, not always entirely upfront.

One obvious caveat here is that unlike many past Kuhnian paradigm shifts, e.g., geocentrism to heliocentrism, the old paradigm here will NOT be replaced wholesale. Rather we will see a great schism: the new paradigm will get bigger, more important, and eventually become the default.

How can I tell, you ask? Because we have seen a similar movie before in the DB world. SQL came into existence precisely because writing low-level imperative code was too much of a chore for data retrieval. Entire new professions grew up around it: data analysts and DBAs. Later on, as dataflow DSLs and higher-level APIs arrived, more such professions emerged: data engineers, data scientists, and analytics engineers.

What is ASE? How are CS and DS converging?

In DS we have long used ML/AI to produce outputs: predictions (e.g., your phone’s face ID) and generations (e.g., machine translation). Agentic AI has now expanded that to virtually any computing task — Q&A, data analysis, code generation, and more. Queries, code, and proofs are “just” additional data structures on a rich rigidity-reliability landscape for AI, as the conceptual figure below shows. Basically any “digital content” is fair game in ASE.

To put it more bluntly, software engineering is now a major “domain” for AI/DS, including, recursively, the software engineering that is part of AI/DS itself! Said another way, hand-crafted queries, code, and proofs are going the way of hand-crafted feature/knowledge engineering in NLP and computer vision of yesteryear. I recognize this might sound jarring to some “classical” CS or math folks, but it should be déjà vu for most AI folks. :)

In fact, I posit that the majority of today’s software engineers, data engineers, and data scientists will become ASE professionals. A minority will still hand-write low-level code in C++, Python, SQL, or other PLs — skilled, well-paid, but a smaller cohort, akin to Assembly or Verilog experts of the old paradigm.

Essentially, CS is being pulled up the abstraction ladder, away from hand-crafted code, to become more DS-like. DS is being pulled the other way, toward whole-application/system awareness rather than studying a model or dataset in isolation, to become more CS-like. This brings us to my working definition of ASE:

ASE is the science of measuring, reasoning about, and interacting with AI agent applications/systems AND the engineering of building, improving, operating, and maintaining them AND the bidirectional flow of ideas between the two.

Of course, not all data structures are made equal, as the figure itself shows. The ones on the upper-right — code, queries, configs, agent plans — have much stricter requirements. A wrong pointer can crash a system. A wrong query can mislead business decisions. A clumsy sentence in a document is just that: clumsy. This brings us to my next point about ASE.

The emerging science of ASE

The astute reader would have noted something I flagged earlier: trust. This is the biggest wildcard in AI practice right now. I do not mean some buzzwordy “trustworthy AI” methods — I mean semantic correctness for a given task. SQL optimizers are correct by construction: the generated codepaths for a given query can vary in speed/resource needs, but they must all be correct. Otherwise, that whole edifice crumbles. LLMs and coding agents today are far from such guarantees.

The above said, IMO such a level of correctness guarantees is often too high a bar — because we do not always expect that from humans either. In most use cases — setting aside very high-stakes ones (e.g., surgical robots, nuclear reactors, etc.) — imbuing more trust in agents/LLMs doing a given task done correctly will likely require mixed-initiative mechanisms combining AI, humans, and deterministic verifiers to various degrees. This is for both verifying its work/outputs and for reinforcing that over time.

This is one of the intellectual cores of the emerging science of ASE: enabling agents to do useful work while keeping the trust of their users.

IMO to achieve the above, CS/DS need to embrace their “social sciences analog” stage. What do I mean by that? In science, we delineate three major levels of abstraction for understanding reality: physical sciences, life sciences, and social sciences. Computing has its own analogs:

• Physical Sciences analog: Working with low-level PLs and mathematically precise reasoning about algorithms and data structures.
• Life Sciences analog: Working with higher level PLs and APIs that still offer some precision, not necessarily in math terms, but as causal reasoning chains/workflows.
• Social Sciences analog: Working with fickle and finicky AI agents that are not necessarily deterministic, akin to the non-determinism and complexity of human free will that still confounds the precision of understanding in many social sciences today.

For too long, CS and DS — including AI — have suffered from the intellectual disease of math-supremacism (“mathiness” is a symptom). That is, unless one has a mathematically precise understanding of something, it is not worth its salt. But math is just one of many complementary ways of grasping the truths of reality in science — conceptual, mechanistic, causal, computational, phenomenological, historical, etc. — none always necessary nor always sufficient. Now that math itself is in a “crisis,” it is high time for these other epistemic modes to step up too.

Amusingly, just as many so-called “hard sciences” folks often sneer at so-called “soft sciences” I see similar things in CS/DS now, including jokingly calling ASE professionals “barbarians”! :)

Anyway, I don’t think there will be a one-size-fits-all form for what the science of ASE will look like — it will vary by the domain of AI use because the data/processes/stakes vary widely. Analyzing particle physics data is very different from analyzing brain scan images, which is very different from building a Q&A chatbot, which is very different from creating a social media app, which is very different from implementing a scalable data system. Yet, they will all share some intellectual foundations of ASE.

I see at least three major groups of scientific questions in ASE to help imbue more trust in agents doing useful work: Evals, Optimizations, and Tradeoffs.

First, creating effective evals. Basically, evals are mechanisms to verify and refine AI’s work—a key part of imbuing trust in agents. We already practice evals pervasively for humans: managers offer “evals” aka feedback to workers; faculty to their advisees; teachers to their students; even parents to their kids! :) For AI this can take many forms: deterministic verifiers for code/proofs, statistics+code-based, model-based (e.g., LLM-as-judge), human-in-the-loop, and hybrids. New principles, protocols, and tools are needed for various domain and application contexts. Statistics, HCI, visualization, psychometrics, data management, formal methods and verification, software engineering, etc. have major roles to play here.

Second, optimizations. These are more akin to the old paradigm systems-y and algorithmic-y work in CS/DS, or even operations research before. The desire to reduce costs / token spend, work completion time, and human grunt work is universal and eternal. AI-created software is not exempt from the laws/limits of physics, economics, and human biology. In ASE, all that must be balanced with trust requirements. We need better algorithms, software, and hardware to, say, reduce agent latency, improve data preprocessing and retrieval efficacy, and secure agent execution. Microeconomics and HCI will also have key roles here.

Third, tradeoffs. These are inherent in any engineering discipline — and often they lead to new science/math, a virtuous loop. While optimizations address the how of improving something, the why and what of decision-making is just as universal and eternal a need — and getting those right is crucial to making decisions trustworthy. This requires not just classical CS/DS-style science but also its intersection with decision theory, game theory, economics, (social) psychology, cognitive science, philosophy/ethics, law/policy, management, and more.

The above three are not exhaustive, of course. Other big topics exist too — how to architect (multi-)agent systems in the first place (which I cover more under the engineering interplay below); how to secure them against misuse or misbehavior; and how all of these evolve as conditions drift. I highlight evals, optimizations, and tradeoffs because they bear most directly on trust, but the science of ASE will have more branches as the field matures.

The interplay of engineering and science in ASE

Unless you have been living under a rock, you must know that the software industry is already awash with agent engineering. Practitioners wield a whole gamut of methods/techniques: prompt/context engineering, tool use, multi-agent orchestration, retrieval engineering, harness engineering, memory engineering, guardrails, deterministic verifiers, RLVR, and much more. All of these are at the heart of the practice of ASE. New R&D to improve (upon) such methods/techniques will be a central part of ASE.

Note that understanding eval metrics, LLM-as-judge rigs, when to distrust outputs, how to debug failures, the rigorous experimental design that separates “this new prompt is better” vs. “we got lucky on twelve examples” beyond just vibes — these are the mental muscle that statistics and DS have been building for a century. ASE will need a lot more of such cross-pollination between science and engineering — something I also wrote about in an earlier post analogizing multi-agent workflows with ensembles in classical statistical ML; podcast with Ben Lorica here.

Let me give two concrete examples of this interplay that I myself work on. Retrieval engineering — preprocessing raw data to help fetch relevant, non-redundant context — is fundamentally an optimization problem, reminiscent of feature selection/ranking in classical ML. New science of knowledge indexing and approximate search can improve engineering; in turn, deeper understanding of retrieval failure modes can inform new optimization science. Agent orchestration — say, how many sub-agents to spawn and when to stop — can draw on the science of decision-making under uncertainty; in turn, as agents become more autonomous, new scientific questions arise about human-agent teaming. Our imagination is the limit of this rich bidirectional flow in ASE!

What does academia need now?

Perhaps surprisingly, I think what is needed is not some major overhaul but a more controlled evolution. We don’t need a decade to figure this out — IMO faculty expertise and industry relationships exist today to make this happen now. The binding constraint is will and governance, not knowledge. And the cost of inaction will compound quickly.

1. Launch an ASE degree or distinct pathways in CS/DS.
   A standalone degree or parallel tracks within existing programs, whichever clears governance faster. Crucially, they must NOT be buried under outdated prerequisites.
   We must train ASE professionals for what the world needs now and for the future, not what was needed for CS/DS c. 2021. Many students now arrive having built non-trivial software or analyzed complex datasets with agents — some as early as middle school! Empower them to build on that foundation, not unlearn it or start over.
   This applies not just to programming courses but to all courses with hands-on work. For example, instead of building toy components of a DBMS or OS, students can team up with agent swarms to build a more solid system; instead of replicating one ML experiment from a domain-science paper, they can use agents to reproduce and verify all its results and also extrapolate to new questions.
   Faculty must shed romantic notions of how we were trained “back in the day,” upskill on AI tools, and adapt our teaching and research practices — without throwing the baby out with the bathwater.
2. Real industry capstones.
   Strong DS programs (including us at HDSI) already have these. Their importance only grows in ASE. It is impossible (almost surely) to emulate the messiness of real-world agents, evals, decision tradeoffs, etc. with just toy/benchmark workloads in academia. Students must get to work with real workloads and stakeholders.
   Industry must step up to partner with academia to bridge this gap across all domains. Otherwise, they will be forced to carry the full burden of re-skilling new hires, effectively acting as surrogate academia.
3. Make both frontier APIs and open LLMs universally available.
   All students, faculty, and staff must be given access to both frontier closed-model APIs (e.g., Claude) and cutting-edge open LLMs (e.g., GLM) by university IT. To me this is now akin to WiFi — basic utilities to help knowledge work. Of course, usage can be metered with gradations of token spend limits, akin to cluster allocations. That also has the positive side effect of teaching everyone the prudence of balancing quality, cost, and latency — a key skill in ASE.

The market has spoken — academia fails to listen at its own peril

The decade-long gold rush in CS majors has ended: CS undergrad enrollments fell 8.1% in 2025, after more than doubling in the prior decade. LLMs/agents are also causing a bloodbath in entry-level software/data jobs. The programs that did the best had already re-pointed at AI. IMO this student flight is rational, but it does NOT point to the death of CS or DS. Rather it is a massive evolutionary pressure to move up the levels of abstraction. CS and DS will NOT collapse into ASE so much as be reshaped around it, the low-level core enduring as a smaller skilled cohort while ASE becomes the mainstream of both.

CS/DS departments that treat this historic disruption as just another wrinkle — a course on coding agents here, a “GenAI policy” there, all LLM-assisted learning treated as cheating — are mistaking the forest for the trees. This is not a moment to cling to past paradigms in fear. In partnership with industry and governments, we must guide the leftward move of computer-use languages in this era of AGI — and boldly forge the new intellectual foundations of ASE.

Additional Relevant Readings and Links:

1. CCC/CRA report on how software engineering must shift toward specifying, verifying, and maintaining AI-generated code.
2. MIT report on AI Use in Teaching, Learning, and Research Training.
3. Karpathy interview on Software 3.0 and Agentic Engineering.
4. CACM Opinion piece on Redefining the Software Engineering Profession for AI.
5. CACM Opinion piece on When Science Goes Agentic.
6. ACM Conference on AI and Agentic Systems (CAIS).
7. IEEE International Conference on Agentic AI (ICA).

About the Author: I am a Professor in the Department of Computer Science and Engineering and the Halıcıoğlu Data Science Institute at the University of California, San Diego. I am also CTO and Cofounder of RapidFire AI. All views expressed in this article are my own and do not necessarily reflect the views of UC San Diego or RapidFire AI. Not unironically, this post was written in part via an extended collaboration with Claude Opus. :)

Acknowledgments: Thank you to the following people for their feedback on this article: Eddy Chen, Justin Eldridge, Earlence Fernandes, Sam Lau, Sam Madden, Aditya Parameswaran, Nadia Polikarpova, Sajjadur Rahman, Balaraman Ravindran, Krishna Sivalingam, Frank Wuerthwein, and a few more who preferred to stay anonymous. Their presence on this list implies neither endorsement nor rejection of any or all of the views in this post, which are the author’s alone.

Contributed Comments:

The following comments were kindly contributed by some readers. If you have any comments / critiques / questions, please do post below. In particular, if your departments are upgrading curricula in this direction, I am curious to hear more. I look forward to the discussion!

I’ve spent the past ten years working on programming and data science pedagogy, and among all my experiences, nothing has had as much impact on our discipline as generative AI. Educators are hastily retrofitting existing curricula in response to modern AI tools. But AI has changed some of the foundational assumptions underlying our pedagogy — for example, that the ability to write large amounts of code correctly is a marker of skill.
How should we rethink the way we teach programming and data science in response? Agent science and engineering offers a promising path. By leveraging agents to write code and by using agents as a core primitive within programs, we can begin teaching students the skills they need to succeed in this new era of software.
- Sam Lau (UC San Diego)

I’ve spent my professional career moving across three roles: teaching software engineering principles to undergrads, researching at the intersection of data management/AI/HCI, and building enterprise agentic systems as an applied scientist. The concept of ASE rings true to me! Software development has always had a “social science” aspect to its overall process: requirements are ambiguous and stakeholders undergo periodic alignment to define correct or expected behavior. The “specificity on demand” framing matches what I see in production: enterprise users rarely state intent precisely upfront. They refine it over time through feedback, policies, and domain data. Agents make that the default experience for all of computing.
From my enterprise vantage I can relate the post’s framing of evals as the one of the intellectual core for ASE. In enterprise settings, “correct” is defined by business context, governance, and trust, not only by a test suite. Evaluation can’t be a one-off benchmark. It has to be continuous and treated as a first-class engineering artifact, drawing from various time-tested principles already informed by various disciplines of computer science. That is exactly the science-engineering interplay ASE calls for, and it’s where I’d most like to see academia and industry build together.
- Sajjadur Rahman (Adobe)