I’ve seen a lot of IT trends come and go over my lifetime, so I’m quite skeptical about any new development; for example, I didn’t even bother getting out of bed for NFTs. However, I believe the recent advances in AI will have huge and jarring impacts on society. As an aside, I promise that this is really me writing this article, not an AI!
I’ve literally been waiting for these sorts of machine learning advances for 20 years. I left the field of robotics around 2005, because I saw that little meaningful progress was going to be made for decades. But this really is the sci-fi future/dystopia we’ve been waiting for.
Note that this information will probably go out of date very quickly, since the capabilities of AI systems are exploding, and I am also concentrating on building up my expertise in this area. I am excited by new technology, but I wish these earth-shaking advances were coming one per year, not several per week!
Everyone. If you've been discouraged by the hype and haven't got an account on chat.openai.com yet; I’d strongly urge you to get one, so you can see how it can magically write your website content or other text, or accelerate other business tasks.
I think soon these AI tools will become ubiquitous, and impossible to ignore in any aspect of life.
I won't list the everyday examples of how AI is already transforming whole industries; that has enough exposure in the media, and everyone has a friend with who is booasting about how they have already automated an annoying part of their job away.
Personally, I am using ChatGPT and other similar systems to take over part of my programming workload, and using AI to bootstrap even better open-source AI systems. I am using AI to rapidly port code from one platform to another.
This is complete speculation, but based on the previous PC and Internet revolutions, here my optimistic take on how AI will impact most industries:
Usable content generated by AI requires manual prompt writing and reviewing by someone with expertise at the level of the content.
For example: if I write some non-trivial code using ChatGPT, I need to have enough prior experience in that language to make sure there are no subtle bugs, and to coax the correct content out via nuanced prompts.
AI may replace "junior" level work, but existing professionals will have their capabilities enhanced, as lower-value work is automated.
If AI means that it costs almost nothing to produce throwaway media, ie "content", then its value will be reduced to zero. "Content" will become so easy to spam out, that search engines will stop indexing it, and companies will want something bespoke, with a "human touch".
If most personal or business correspondence could be written by LLMs, then it will be more polite and personal to just send the original prompting content to people, without any time-wasting lily-gilding added by AI.
I think that etiquette, legislation, and norms will change to accomodate AI, and that there will still be work for humans. We have already seen industrial automation displace heavy manual labour, and create new realms of human endeavour.
Perhaps, if the changes are too disruptive, then AI could be a major impetus for the wide adoption of Universal Basic Incomes.
Skip this section if you don’t care about the technological details.
Please note that I have been out of academia for a long time, and have only been studying Machine Learning informally, and through practical applications.
The basic technology of neural networks was nailed down in the 80s - it's mostly based on high-school-level calculus and linear algebra. What we are seeing now is that these models have been optimised to run on modern graphics cards, scaling up their computing power by orders of magnitude. Modern AI uses similar “graphics card” hardware to that used by cryptocurrency mining and video games, so there is plenty of this hardware already lying around.
The Rolls-Royce of AI hardware, which trains and runs the models in data centres, is the A100 "video card” (GPU), which costs 20000e+. These are usually deployed in groups of several per computer. However, a normal high-end Nvidia graphics card with CUDA can already do a lot, and the costs are already coming down hugely as more efficient algorithms are developed. These machines can be rented in the cloud.
ChatGPT and other chatbots are based on Large Language Model (LLM) neural networks, which are trained by essentially feeding their inputs with the whole internet. They are text prediction programs: it is quite amusing that all their human-like behaviour comes from their sole ability to complete sentences.
For example, if you feed this sentence into a LLM it will mindlessly complete it: “To be or not to be, that is the …”. But once these models grew so large that they could “read the internet”, their power became something else.
You can feed surprisingly nuanced prompts into LLMs, to be “completed”, not just Shakespeare quotes. Here is an actual prompt which turns the LLaMa model (A GPT competitor from Facebook) into a chatbot. This is the “beginning” of a paragraph, which will be “completed” by the LLM:
A dialog, where User interacts with AI. AI is helpful, kind, obedient, honest, and knows its own limits.
User: Hello, AI.
AI: Hello! How can I assist you today?
User:
And then when the user finishes this text, the LLM tries to complete the answer, playing the role of a chatbot. However it would be just as easy to write: “A dialog, where User interacts with AI. AI thinks it is a magical turkey, talks like a pirate, tells lies, and wants the destruction of mankind.”
Let’s start with the most fundamental, powerful AI technologies.
”Foundation” models like GPT-4 can be created with some basic undergrad knowledge of neural networks and calculus. However it takes a massive number of graphics cards, running for many months, to create the models from the internet. Only huge tech companies can afford to build these. Some of these models are open source, but not the most recent or powerful ones. You can get paid, metered access to these more powerful models, but they are heavily commercially restricted, for anti-competitive and safety reasons.
But that is not all. To turn these models into good chatbots, it takes an army of real humans to give feedback on the responses, until the model starts giving helpful answers to conversational questions.
The ability to create foundation and chat models on the level of ChatGPT isn’t really within our budgets or resources (yet), so we need to build on top of them.
These LLMs can be fine-tuned, their knowledge “transferred” to other models, or built upon, much more cheaply than the original foundation models. For example, the Alpaca model, a competing model to ChatGPT which has been “leaked” to the public, was made by abusing OpenAI’s paid text-davinci-003 model to improve its abilities. In this way the model made itself so efficient that it could fit within the resources of a mobile phone.
If you aren’t ready to try to compete with Google, Microsoft, and Facebook, and you don’t want to develop your own “transfer” models, you can pay for access to their models via APIs. You can write your own prompts, to create whatever sort of chatbot or intelligent system that you want. An app or service can be made “intelligent” in a few minutes - by using the AI to write its own code. It is quite cheap, costing fractions of cents for each user interaction.
These models can also be further “fine tuned” to work better for a specific use case.
I am currently experimenting with linking multiple models together. For example, an online free Google text-to-speech AI can feed text transcripts into an LLM model. This creates a custom voice assistant like Siri, but completely free, and able to be heavily personally customised.
Another opportunity for making money out of AI is to develop your own smaller models. These might not be as generally smart as something like ChatGPT, but they can be specific to your particular business domain, using the data that you gather. These can then be complemented with foundation models, or sold to other companies.
Generally, machine learning models need to ingest huge amounts of data, and many need a large amount of human work to prepare the data for being fed into a model.
It’s currently unclear where this will all end, and what legislation will need to be created to keep a lid on this. And also all the ways in which miscreants will use this to cause harm - the scamming possibilities are endless.
The impact will likely be similar to that of personal computers: those who adapt to AI quickly will see their productivity skyrocket, and those who do not will be left behind.
Please note that this technology is being hyped as sci-fi magic, but it has hard limitations.
Every model has a context window: it will begin to forget what it has said, the longer it runs. This means that although initially impressive on small examples, current AI can generally not grasp the “big picture”. If it was writing a book, by the time it finished a few pages or chapters, depending on the model, it would forget how it had begun.
LLMs have also learnt how to lie convincingly. They are trained to make human users satisfied with their answers, not to always be correct. And after reading millions of internet ads and marketing copy, they have also learnt to be excellent liars. I would argue that it takes as much expertise to review an AI's responses as it would have taken to do the original work.
I expect this document will be out of date incredibly soon, and it will seem a naive take on something which ended up having huge societal implications. I hope that legislation and politics has time to catch up.
For the moment, we have amazing new tools to grow whole new fields of hman endeavour, and remove a lot of drudge work.
James Hudson
If you have any thoughts on this, please email me. Or at least "like" or "share" this post if you think others would appreciate it.
Needless to say, this blog isn't financial or legal advice, an excuse for getting fired, or promising that any of these ideas will work for you. The companies or people I mention may not agree with my opinions here. Don't do anything reckless, damaging or hurtful to anyone! In the future you might need your bridges unburnt. Images used under fair use, and are copyright their respective owners. © 2014-2022 James Hudson