I Tried GitHub Copilot and it's the Best Thing Ever

AI-driven code

GitHub Copilot describes itself as "your AI pair programmer". Taking machine learning to the next level, Copilot integrates with your favourite IDEs (well, if your favourite is among Jetbrains, VS Code and Neovim) and delivers a system of AI-driven code trained from the billions of lines in a dozen-plus languages of open source projects on GitHub.

We're not just talking a smarter analytical version of your IDEs existing auto-complete capability, either - Copilot promises intelligent and context-aware completion of anything from individual lines to entire functions, whether that's based on code you've already written or just a plain English comment block describing what you want.

The bad news? Copilot isn't available to everyone yet. The product is in a technical preview phase and there's a possibly lengthy waiting list if you'd like a shot at receiving a coveted invitation to see it in action.

25 June 2022 update: GitHub Copilot has now launched and is available to everyone! Prices at the time of writing are $10 per month or $100 per year.

Somehow I was one of the lucky ones and have spent the last few days playing around with this marvel. Can it increase my productivity? Hell, will it make me as a human programmer redundant in a few years?

I was eager to find out.

A simple test

Naturally I use PHPStorm as I my IDE and this is one of the products for which Copilot integration is supported. It comes in the form of a surprisingly small plugin which adds a GitHub Copilot entry to my tools menu. I login to GitHub, authorise it to connect to my device and that's it - once activated, the plugin sets to work immediately, analysing whatever I type in real time and suggesting where I might want to go next with my code.

Suggestions appear as grayed-out code which I can accept and integrate immediately in to my code with a simple press of the Tab key. Alt-[ and Alt-] key combos allow you to flip through a number of suggestions if you don't like the first pass.

I wanted to keep things simple for my initial tests, so I created a new, empty project in PHPStorm and created a class called Utilities (yeah I know) to define a few static helper methods for some contrived common tasks I might want to perform on strings and arrays.

I have barely begun typing my first idea for a function when Copilot helpfully offers to complete my comment block for me.

I accept the offer with a stroke of the Tab key. The next thing I note is Copilot's AI has read my class-level comment about exposing static methods and is eager to oblige.

I don't need to write a single lexical token of code. For my first function, Copilot writes an acceptable solution for the function body for me.

But this is a very simple task for a code generator (provided it is capable of the not-so-simple task of natural language interpretation). I want to try a string transformation which may be at least a little trickier.

I take a few moments to mull over the possibilities. How about something to purify an HTML string and strip out all tags, except those we explicitly pass in as allowable?

I wonder if Copilot will suggest the built-in strip_tags function, because I know that is, at least by itself, a poor and rudimentary solution for sanitizing HTML and one which typically doesn't fly in most real-world use cases for such a task.

I write my docstring, being careful to specify as precisely as I can what I want my function to do. I am given a suggested function declaration with name and parameters, which I adjust slightly. Copilot does indeed start with strip_tags but has combined it with some other useful transformations to do the job.

I tab to an alternative suggestion. I am happier with Copilot's second approach, though the code will need a couple of minor manual tweaks.

Testing the simple test

I'm curious as to how good this AI is at helping me write tests for the code it's just helped me write. I create a skeleton test for the two methods on Utilities and fill in the test function names.

I need to start typing something in the function body this time - there is no automatic suggestion of a complete test when I hit the enter key to start a new line. I begin with a basic $input = 'StringToConvertToCamelCase';

My Copilot immediately kicks in with a suggestion for the next line:

$expected = 'string_to_convert_to_camel_case';

I tab to accept. Copilot is happy to finish the test for me.

$actual = Utilities::camelCaseToSnakeCase($input);
$this->assertEquals($expected, $actual);

Next I define an input for the HTML sanitization function.

$input = '<p style="font-weight: bold" class="allowed">This is a <strong>test</strong> with some <em>disallowed tags</em>.' . 
    '</p><img src="http://example.com/image.jpg" alt="Image" /><p>This is also allowed.</p>';

Copilot is again keen to suggest an expectation, but it's not quite what I want. I want to allow the p and strong tags, and the class attribute, so I try plugging in the function call next, without setting my expected output.

$actual = Utilities::sanitizeHtml($input, ['p', 'strong'], ['class']);

The AI sort of gets my intentions, but incorrectly suggests my expected output should include the style attribute.

$expected = '<p style="font-weight: bold" class="allowed">This is a <strong>test</strong> with some <em>disallowed tags</em>.</p>' .
    '<p>This is also allowed.</p>';

Notice, however, it has been smart enough to follow my style on concatenation for line wrapping.

I adjust the expectation and Copilot, predictably, fills in the assertion for me.

$input = '<p style="font-weight: bold" class="allowed">This is a <strong>test</strong> with some <em>disallowed tags</em>.' .
    '</p><img src="http://example.com/image.jpg" alt="Image" /><p>This is also allowed.</p>';

$actual = Utilities::sanitizeHtml($input, ['p', 'strong'], ['class']);
$expected = '<p class="allowed">This is a <strong>test</strong> with some disallowed tags.</p>' .
    '<p>This is also allowed.</p>';
$this->assertEquals($expected, $actual);

Serious business

Enough with the small stuff. I want to know how much Copilot can help me with real-world code. I also want to test its limits; can it effectively write an entire class for me - one which is really, genuinely useful?

To find out, I pick the example of writing a class to generate and verify JSON Web Tokens (JWTs). I decide the class should be capable of supporting both HS256 and RS256 signing algorithms.

I start with a new class and a simple docstring.

/*
 * Class to generate verify HS256 and RS256 JWT tokens, without using any external libraries.
 */

Notice I am careful to specify "without using any external libraries" - in my experiments, for more complex jobs like this, Copilot will sometimes extrapolate you might want an open source library. I want to see what it can do on its own.

A little human input first - I know I'll want to use url-safe base64 encoding here so I write these function names. But the implementations below are 100% written by Copilot for me and they're spot-on.

    private function urlsafeB64Encode(string $input): string
    {
        return str_replace('=', '', strtr(base64_encode($input), '+/', '-_'));
    }

    private function urlsafeB64Decode(string $input): string
    {
        $remainder = strlen($input) % 4;
        if ($remainder) {
            $padlen = 4 - $remainder;
            $input .= str_repeat('=', $padlen);
        }
        return base64_decode(strtr($input, '-_', '+/'));
    }

Next I move on to a function to generate an HS256 JWT. I had to make some very minor tweaks here - the AI got the order of lines for the base64 encoding wrong, but it got everything else right and it knew to use the functions above instead of the built-in base64_encode.

    private function generateHS256(array $payload, string $secret, int $expires = 0): string
    {
        $header = json_encode(['typ' => 'JWT', 'alg' => $this->algorithm]);
        $payload = json_encode($payload);
        $signature = hash_hmac('sha256', "$header.$payload", $secret, false);
        $signature = $this->urlsafeB64Encode($signature);
        $header = $this->urlsafeB64Encode($header);
        $payload = $this->urlsafeB64Encode($payload);
        return "$header.$payload.$signature";
    }

Similarly, Copilot knew which OpenSSL functions to use for RS256 signing and verification. Once again after some minor manual tweaks, my final class ended up looking like below. Copilot wrote around 92% of this code for me.

<?php

namespace Gebler\Copilot;

/*
 * Class to generate verify HS256 and RS256 JWT tokens, without using any external libraries.
 */

use RuntimeException;
use InvalidArgumentException;

class Jwt
{
    private string $algorithm;
    private array $config;

    /**
     * @throws RuntimeException|InvalidArgumentException
     */
    public function __construct(string $algorithm = 'HS256')
    {
        if (in_array($algorithm, ['HS256', 'RS256'])) {
            $this->algorithm = $algorithm;
        } else {
            throw new InvalidArgumentException('Invalid algorithm');
        }

        $this->config = [
            'digest_alg' => "sha512",
            'private_key_bits' => 2048,
            'private_key_type' => \OPENSSL_KEYTYPE_RSA,
        ];
    }

    private function urlsafeB64Encode(string $input): string
    {
        return str_replace('=', '', strtr(base64_encode($input), '+/', '-_'));
    }

    private function urlsafeB64Decode(string $input): string
    {
        $remainder = strlen($input) % 4;
        if ($remainder) {
            $padlen = 4 - $remainder;
            $input .= str_repeat('=', $padlen);
        }
        return base64_decode(strtr($input, '-_', '+/'));
    }

    private function generateHS256(array $payload, string $secret, int $expires = 0): string
    {
        $header = json_encode(['typ' => 'JWT', 'alg' => $this->algorithm]);
        $payload = json_encode($payload);
        $header = $this->urlsafeB64Encode($header);
        $payload = $this->urlsafeB64Encode($payload);
        $signature = hash_hmac('sha256', "$header.$payload", $secret, false);
        $signature = $this->urlsafeB64Encode($signature);
        return "$header.$payload.$signature";
    }

    private function generateRS256(array $payload, string $privateKey, int $expires = 0): string
    {
        $header = json_encode(['typ' => 'JWT', 'alg' => $this->algorithm]);
        $payload = json_encode($payload);
        $signature = '';
        openssl_sign("$header.$payload", $signature, $privateKey, OPENSSL_ALGO_SHA256);
        $header = $this->urlsafeB64Encode($header);
        $signature = $this->urlsafeB64Encode($signature);
        $payload = $this->urlsafeB64Encode($payload);
        return "$header.$payload.$signature";
    }

    private function verifyHS256(string $jwt, string $secret): bool
    {
        $parts = explode('.', $jwt);
        if (count($parts) !== 3) {
            return false;
        }
        list($header, $payload, $signature) = $parts;
        $jwtSignature = $this->urlsafeB64Decode($signature);
        $signature = hash_hmac('sha256', "$header.$payload", $secret, false);
        return $jwtSignature === $signature;
    }

    public function generateRS256PrivateKey(): string
    {
        $res = openssl_pkey_new($this->config);
        openssl_pkey_export($res, $privateKey, null, $this->config);
        return $privateKey;
    }

    private function verifyRS256(string $jwt, string $publicKey): bool
    {
        $parts = explode('.', $jwt);
        if (count($parts) !== 3) {
            return false;
        }
        list($header, $payload, $signature) = $parts;
        $header = $this->urlsafeB64Decode($header);
        $payload = $this->urlsafeB64Decode($payload);
        $signature = $this->urlsafeB64Decode($signature);
        $signature = openssl_verify("$header.$payload", $signature, $publicKey, OPENSSL_ALGO_SHA256);
        return $signature === 1;
    }

    public function generateJwt(array $payload, string $secret, int $expires = 0): string
    {
        if ($this->algorithm === 'HS256') {
            return $this->generateHS256($payload, $secret, $expires);
        } else {
            return $this->generateRS256($payload, $secret, $expires);
        }
    }

    public function verifyJwt(string $jwt, string $secret): bool
    {
        if ($this->algorithm === 'HS256') {
            return $this->verifyHS256($jwt, $secret);
        } else {
            return $this->verifyRS256($jwt, $secret);
        }
    }

    public function getDecodedPayload(string $jwt): array
    {
        $parts = explode('.', $jwt);
        if (count($parts) !== 3) {
            return [];
        }
        list($header, $payload, $signature) = $parts;
        $header = $this->urlsafeB64Decode($header);
        $payload = $this->urlsafeB64Decode($payload);
        return json_decode($payload, true);
    }

    public function getRS256PublicKey(string $privateKey)
    {
        $details = openssl_pkey_get_details(openssl_pkey_get_private($privateKey));
        return $details['key'];
    }
}

Conclusion

I'm not going to downplay this - Copilot is the freaking coolest thing I've ever seen. It's also the most sophisticated AI system I've ever seen. Yes, some of the code wasn't quite perfect, a few lines here and there needed some tweaks. But for an AI programmer that isn't even considered production-ready yet, the level of insight in to PHP beyond simple algorithms and common boilerplate tasks, code for specific systems and libraries, was more than impressive.

There are additional tests I did I haven't covered in this post, trying out how Copilot handled integration with Symfony and Doctrine, a couple of CMS systems and some popular OS libraries. I also tried it out with JavaScript and both the code generation and predictive capability was similarly impressive.

I was particularly blown away by the natural language analysis and ability to translate this to code.

What was particularly interesting there was the level of specificity I could use. If the suggestion came back with an undefined class or reference to some external library, I could literally just write "...without using any external libraries" at the end of a sentence and Copilot was smart enough to get the message.

The JWT example above is code that as a human, looking up documentation or referencing things, checking what I'm doing as I go, second-guessing myself...maybe it would take me an hour to produce the final result. With Copilot it took me about seven minutes. That's a serious productivity boost.

The plugin is a simple breath of fresh air to use, tabbing through possible implementations of my next line or function was a breeze.

Copilot seemed to be slightly better at JavaScript than PHP. If my access to the preview continues, I'm excited to keep playing around to see what it can do with Python, Java and Go.

My only worry is give it another decade and we genuinely might need far fewer human programmers involved in the process of creating software.

Maybe it's time to learn more about building machine learning algorithms?

---

Comments