This post is a guide to building your own software services, to alleviate some of the drudgery of your adult existence. I assume you know, or are willing to learn: entry-level coding, plus how to publish a simple web app. I use the Python language in my examples, but you could substitute your own stack of choice.
This post is based on my own experiences with personal services; I'm sure there are devops masters out there who have done way more effective automations than me.
Humans did not evolve to do repetitive, mechanical work; this is a spirit-crushing life for an intelligent hunting and gathering ape. Although the modern world is far from the mechanised hellscape of the early industrial revolution, life still requires some minimal level of drudgery: accounting, paperwork, emailing, shopping lists, reminders, and so on. I also run a business in Germany, which adds an extra shovel-load onto the piles of mindless paperwork. Three months of a freelancer's paperwork burden in Germany is larger than an entire year's worth for a company director in Australia.
I am a programmer, which by definition is somebody who automates things. So why not take that into my personal life, and try to automate some of my more repetitive chores?
Over the last 12 months, I’ve been running an experiment to automate away a lot of the bureaucratic burdens of being a freelancer. I no longer dread tax time, but almost look forward to watching my own creations churn through it for me. I’m spending more time doing valuable work and living life, not wasting time for no benefit.
It's addictive to see how far personal software services can be pushed - the more you systems you build for yourself, the more they amplify each other. My business basically runs itself now; as soon as some paperwork becomes annoying enough to start giving me deja vu, it gets automated.
I sometimes perform with the Austrian musician Lulu Schmidt, and live-animate the light costumes I built for the band. My "Mr Happy" robot LED mask started as something of a joke, but it stuck - literally.
Usually, I am hunched over my laptop controlling the lights; the most free I could make myself was by roving around with a controller app on my phone. But the last time we performed, we tried driving the light animations via MIDI automation messages from Lulu's audio workstation. The timing of the effects was flawless, and the light show literally ran itself most of the time. However, that left me with barely anything to do on stage except dance and grind: much more natural, interesting, and enjoyable activities for a human. Next time we will be free to do more theatrical stage direction, and not worry about the tedious work of remembering cues and fiddling with laptops.
Automation made it a much better show, and freed me up to do more interesting things on stage.
People have asked me: 'What about BlahBooks? It does bookkeeping already'. I had a look at some of these existing systems, but there is only so far that they can be personalised. For example, I don't know of any system that could open my browser on the local German tax authority webpage and prefill the numbers, deal with Australian invoices, and also batch generate a bunch of signed documents that my tax advisor suddenly needed.
If you are a good IT person, you already know that when you find yourself repeating the same actions over and over, you should start thinking about writing a script to do them automatically. Why shouldn’t this logic carry over into everyday life?
Whenever you feel the dread of having to do something yet again, it is time to look for elements of the task which could be automated. One handy tip: if you already need a spreadsheet to keep track of something, it’s probably a good target for supercharging that spreadsheet with more automation.
All of these tasks could potentially be automated: tax declarations, scanning lists looking for overdue invoices, copy/pasting data into emails, printing out invoices, and so on.
Once you have a few pain points identified, it is worthwhile to look for where the tasks cross over. The power of automation is multiplied when the results of one automation step can feed into another. For example, writing invoices is directly related to tax, so your system for making tax declarations could also generate your invoices.
Now that you have some ideas of what you could automate, I’ll take you through the steps of designing and building your personal service.
Firstly, you should decide how to divide up the problem into separate modules. “Microservice” is a trendy buzzword, which basically means splitting a service up into several smaller services which communicate between each other via standard web APIs. Quite often, this is a bad idea in practice, but there are some circumstances where it works.
Can you break apart the problem into a few separate pieces which share data in a very minimal, clearly defined way? Then this problem could be a good candidate for a microservice architecture.
For example, I have my own graph-drawing microservice. If I ever need to graph some data online, with a few lines of code it’s done. I can forget maintaining my code for rendering graphs, which “just works”, and share it between multiple larger services on very different platforms. I can also host it on a free server and not pay for its resources, and experiment with some different technology stacks that I wouldn’t normally use for a bigger service.
My graph drawing microservice has a very simple API: I pass in simple JSON with graph points and labels, and get back a nice graph which I can embed in the page. It is written in Python using the Flask framework, and uses Pygal for the graph rendering.
The “cloud computing” revolution is old news, so there are plenty of options for getting your services online within minutes. Most “Platform-as-a-Service” plans have a free tier, which you can upgrade for a small monthly cost, usually under $10. They come already set up with SQL databases.
Here are a few I know:
How will your data be structured? Take some time to sketch out the database schema.
Some examples of how real-world things could translate to database objects:
Your system should be functional, in the sense of functional programming: store as little state as possible. This reduces storage requirements, the chances of having unsynchronised data, and the possibility for bugs. Do not store pdfs, for example, but generate them as needed from the database. When you email or download pdfs, that email or file can serve as the "definitive" copy for your records.
Now, you are ready to start coding! Once you have the minimal system up and running, there are a few extra considerations which could amplify the power of the system even more:
It is really useful to be able to shoot off emails from your system, and it is really easy to add to your code. You can run a task every day, or every hour, with a cron script, to check if some deadline has been met to send an email. You will need an existing email account: a new gmail account with all the security turned off will work in a pinch (just be prepared for Google to shut it down arbitrarily if they decide you are using it for spam), or a mass mailer like sendinblue.com. I had problems with Mailjet, so I cannot recommend it.
In practice, however, most emails you will want to personalise to some degree, instead of just spamming everyone. I often have an “open email” button in the right place in my web interface, which is just a “mailto:” weblink with prefilled subject and content.
To generate PDF documents in Python, I use jinja HTML templates to first get it looking right in HTML. Then I feed that HTML into the “weasyprint” Python libary, which generates a PDF from the HTML. It can be helpful for debugging to have separate endpoints on your application for HTML and PDF download.
Lots of progressive web companies are as passionate about inter-operation as you are. If they have a nice REST API, you can connect your service to theirs. For example, Google Docs has an API which lets you feed data directly into your spreadsheets.
If these services don't have an API, the next best thing is an export function. Your bank website probably has a CSV export option. Failing that, you can always scrape a website using curl and a shell script.
Your automation can help others - ask around. Maybe your tax consultant would like your data in a certain format to feed into their own software? Maybe your customers have their own accounting or timesheet APIs that you could connect to directly?
However, in practice I've found that many traditional businesses do not see the value of automation, even when it doesn't cost them anything. In fact, the idea of automation may actually be hostile to the parts of a large company which are "self-sucking lollipops" (entities which only exist to make more work for itself, in order to justify its own existence).
Also, it costs nothing to create extra user accounts, so you could perhaps offer free accounts to your customers, friends, and family.
Log everything! Databases and disk space are cheap - do not throw away data. Use it for debugging when things go wrong.
Even though it’s your own data, it would be hugely painful if your automated life vanished into smoke. Regularly pull backups using a cron script. Store them off site. Check regularly that they are being collected. And do not forget to check you can still open and restore your backups, and that you are not saving empty or corrupted files.
If it works for you, other users might want it - it could be the start of a business. I have quite a few of my family and friends on my own system now.
The humble spreadsheet launched the personal computer revolution, and is the backbone of modern business. Billions of dollars of research and development have been invested into it. People have built first-person shooters in spreadsheets. Chances are, some smart spreadsheet magic, coupled with forms, can solve 90% of your admin problems, without you needing to reinvent the wheel. Spreadsheets stop being optimal when you need complex interactions with users and the outside world, in a time-critical manner, such as sending emails or checking then submitting details through an API.
You might begin seeing opportunities for automating things everywhere.
I went a bit wild this year - I even built my own hardware to integrate with my automation: an open source smartwatch.
A Raspberry Pi can also make a reliable, cheap hardware platform to bring your automation into the physical world.
You could try making an automatic cryptocurrency trading bot (be aware of the tax implications of this before making it trade real crypto). There's nothing like the feeling of going on holiday, then forgetting that you left an experimental trading bot running at home with your real credit card details.
You are a programmer living in a world which is being consumed by the digital; you can use what you already know about software to make your personal life better. Experiment with how these systems can link together and support each other. And enjoy your extra free time!
Have you built your own personal services? I'd be curious to hear your own ideas and experiences. Please email me.
Needless to say, this blog isn't financial or legal advice, an excuse for getting fired, or promising that any of these ideas will work for you. I really, really, hope that you don't think it is investment advice. The companies or people I mention may not agree with my opinions here. Don't do anything reckless, damaging or hurtful to anyone! In the future you might need your bridges unburnt. (c)2019 James Hudson