From Copilot to a team of AI software engineers

Loïc Gouarin

• Research engineer in scientific computing at CNRS
• Co-leader of the HPC@Maths team
• Member of the groupe Calcul board
• Developer of open-source software

Some comments before we start

• I’m not an expert of LLM and their construction.
• The world of LLM is vast and the field of possibilities almost infinite.
• Some points may be wrong and lack precision.
• The learning curve is steep.
• I’m a baby of a one month and a half in this world.

What can you do with GitHub Copilot ?

• Code completion as you type (2,000 intelligent code completions a month)
• Chat discussion to explain code, to write tests or documentation, … (50 Copilot Chat messages a month)
• Use the open files and your GitHub repositories as a knowledge base
• Web search and other agents on the marketplace
• Two models available: Anthropic’s Claude 3.5 Sonnet or OpenAI’s GPT-4o

https://github.blog/ai-and-ml/github-copilot/what-can-github-copilot-do-examples/

free since December 2024 for everyone. You can also modify multiple files at once. You have more models if you pay.

You can also add various extension in VSCode to search on the web for example. (see https://github.com/marketplace?page=2&type=apps&copilot_app=true)

The problem here is that the LLM models are not free and that you have limited resources to use them, even if their number may seem sufficient for personal use.

Can we have the same functionality for free ?

The Copilot alternatives

https://www.augmentcode.com/blog/rethinking-llm-inference-why-developer-ai-needs-a-different-approach In their article, they said that context is everything for coding and I think it’s completely true. The context is your files, your dependencies, the call-sites, … If we put all this context in the prompt, the token numbers will be so huge that it would be impossible to use it or would greatly slow down the inference of LLM. But what we expect is low latency.They said that they are expert to retrieve the most relevant context. It’s not a general use of LLM when we code. We have a lot of input context and not necessarily a long output. They said that they serve requests with 10k input tokens to Llama3 70B with a TTFT of less than 300ms (3x faster). Their infrastructure is composed by NVIDIA H100. You can easily add software or library documentation and the code completion is smart enough to suggest the next steps. But you can’t add your LLM model.

https://blog.continue.dev/continue-1-0/ continue offers the same functionality as Copilot but you can specify which LLM models you want to use for the different tasks: chat, completion, embeddings, … You can also add your own documentation and build your RAG.

What models do I need to choose to help me develop my software ?

Do I have enough resources to use them ?

What are the limitations of these models ?

There is a lot of available models and it’s not easy to choose the right one.

The augment code blog tells us that the inference is made on 4 H100 for a small model of 7B.

The models are for generic questions and know nothing about your software.

Where to find LLM models ?

Ollama website

Hugging Face website

Learderboards

LiveCodeBench: https://arxiv.org/pdf/2403.07974

Learderboards

Other leaderboards

• Code evaluation
  • BigCode’s Models Leaderboard
  • BigCode’s BigCodeBench
  • Meta’s CyberSecEval
• Mathematics abilities
  • NPHardEval

Look at the article to explain the results. They seem really bad.

Can I use LLM models locally ?

1 GPU A100 - Juliet

MacBook Pro M1

Msty

• Offline-first, online-ready

  Works seamlessly offline while supporting online models.

• Parallel multiverse chats

  Compare responses from different AI models in real-time.

• Unified access to models

  Supports models from Hugging Face, Ollama, and Open Router.

• Prompt management

  Offers a library of prompts and allows custom additions.

• Ultimate privacy

  No personal data leaves the user’s machine

But finally what do I need ?

• To have an AI assistant that knows
  • the programming language I use
  • the documentation of my third-party libraries
  • my software and the mathematics behind it
• Specialized models for each step of development
• Not to iterate over LLM models again and again
• To have a memory of what was done

What we call an agent ?

Use case: depixelizing

Use case: Hilbert curve

def rot(n, x, y, rx, ry):
    if ry == 0:
        if rx == 1:
            x = n - 1 - x
            y = n - 1 - y
        return y, x
    return x, y

def d2xy(n: int, d: int):
    t = d
    x = y = 0
    s = 1

    while (s < n):
        rx = 1 & (t//2)
        ry = 1 & (t ^ rx)
        x, y = rot(s, x, y, rx, ry)
        x += s * rx
        y += s * ry
        t = t//4
        s *= 2
    return x, y

if __name__ == "__main__":
  x = y = 0
  n = 8
  coords = []
  for i in range(1<<n):
      coords.append(d2xy(1<<n, i))

How can I enrich a generic LLM model ?

The problem with RAG is that you only have a chunk of the relevant data. You can take the neighbors if you have stored the data using an ID but it can still be not enough. CAG seems to eliminate retrieval latency or mitigate retrieval errors.

• Web search (Tavily, DuckDuckGo, Brave, …)
• Python script execution
• GitHub interactions
• …

From scratch

The multi agent tools

• CrewAI is a lean, lightning-fast Python framework built entirely from scratch—completely independent of LangChain or other agent frameworks.
• Langflow is a new, visual framework for building multi-agent and RAG applications. It is open-source, Python-powered, fully customizable, and LLM and vector store agnostic.
• AutoAgent is a Fully-Automated and highly Self-Developing framework that enables users to create and deploy LLM agents through Natural Language Alone.
• SmolAgents is the simplest framework out there to build powerful agents!
• …

Some demos

So, what did I learn?

• The context is probably the most important.
• XML can help you structure your output messages.
• It’s not that simple to get the tools and the LLM to talk to each other.
• There are so many software packages out there that it’s hard to choose the right one.
• I won’t be getting a team of AI assistants working on my software developments any time soon.

But I’m not done yet !!

Model Context Protocol

References

• More about RAG

  Advanced RAG Techniques: Elevating Your Retrieval-Augmented Generation Systems

• More about CAG

  Don’t Do RAG: When Cache-Augmented Generation is All You Need for Knowledge Tasks

• If there is only one article that you need to read

  A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges

• Prompt examples on various topics

  fabric is an open-source framework for augmenting humans using AI

Thank you for your attention

And thanks to MesoNET

especially the ROMEO team to let me use their computing resources !

https://github.com/gouarin/2025-04-ia4dev

https://github.com/gouarin/llm4code