Things I mostly did for money

Skills:

Devops: Terraform for GCloud & AWS. Docker/Kubernetes/Helm config and management. GoCD/Concourse/Gitlab/Jenkins CI config and management. Ansible/Puppet design, build and scale-out. MCollective installation, management and agent creation. Rabbit/ActiveMQ installation and management. XenServer install and management. LAMP stack troubleshooting and deployment automation. Connecting all the above with Ruby. Explaining to dev teams and/or management why this is a good thing. Code on Github, opinions/presentations on http://ops.failcake.net

Devops is a culture, not a team. Was a culture. Like SAFe was Agile, once.

Tradops: High-availability systems design, specification and installation using the appropriate technologies for the required service-level. Hard and soft RAID subsystems, NetApp, ZFS, Veritas VXFS, VXVM or native LVM, HP MC/ServiceGuard. Linux, Solaris, HP-UX, BSD & OSX system administration, ranging from installing the OS on a ‘bare’ system, through day-to-day user admin and troubleshooting, to resolving complex configuration problems and identifying & fixing potential security breaches. Documenting these things, usually in a wiki, but Markdown at a pinch. Advocating for keyword tagging over fixed hierarchies. Advocating for documents that tell the stories of how the systems got to where they are, rather than a state of permanent now, so repetition of past mistakes may be avoided. People cannot learn without history. Advocating for a culture of experimentation, the testing of assumptions, and the recording of the results, good and bad.

Presentations/Lectures:

  • ‘How to do feminist devops’ – Queer Tech Bristol, February 2024
  • ‘Out daemons, out! Exploring depression through sweary automata.’ - EMFcamp 2018
  • ‘Having daft ideas’ – Workshop – EMFCamp 2016, SHA2017
  • ‘The BOfH must die – terrible stereotypes breed terrible environments’ - OHM2013
  • ‘Beards vs. process – two-fisted tales of devops’ – EMFcamp 2012

Experience:

March 2022 - April 2025 Sage / Brightpearl, Bristol

  • Senior cloud ops engineer. Helped migrate undocumented/legacy VMs from AWS EC2-classic infra on an accelerated timescale. Implemented determinedly standard Ansible config so the thing in the git repo matched the examples on the website, thus lowering the possibility for confusion. Used that Ansible config to instrument otherwise unmonitored VMs in the run-up to Black Friday.
  • Worked with the rest of the Ops/SRE teams to migrate production kubernetes clusters from a largely undocumented and non-repeatable mixture of terraform and hand- rolling to a repeatable and documented pure-terraform config so that dev-sandbox replicas of the production clusters are available. Migrated several microservices into that new environment with minimal downtime.
  • Ignored the received wisdom about the mail system being complicated and made Postfix and the mail-related DNS work right. Ignored the received wisdom about Mongodb being fragile and made the clusters survive reboots so the VMs could be patched regularly.

February 2018 – February 2022 Flexys Systems, Bristol

  • Ops lead. Designed & created the test and customer GKE clusters/config using Terraform, designed and installed the initial monitoring/graphing system, migrated/ rebuilt the CI system from Hetzner VMs to a GKE cluster so we were testing on a representative environment, prototyped/wrote a deployment automation tool that used Helm charts as first-class objects and stored only the config diffs in SCM, wrote procedures and processes to support the ISO certification effort, automated workstation commissioning ditto.

January 2008 – January 2018 Future Publishing, Bath

  • Senior Unix administrator in the IT Engineering dept, which supports the core publishing systems, VM automation servers and web infrastructure for an organisation that serves 50 million unique users per month.
  • Worked as part of a small team to move all the internet-facing systems to a new network stack, new IPv4 netblock and additional IPv6 netblock. Rebuilt/redesigned code and processes to cope with IPv6 and a new security architecture. Renumbered/ rebuilt all internet-facing services with zero downtime.
  • Rolled out Puppet-based XenServer VM automation system. With this we can manage our entire Linux infrastructure rapidly and repeatably, which ensures no surprises when moving from dev to stage to live environments. This is quietly being containerised so that everyone can have a dev stack and as many stage environments as git branches.
  • Designed and built a lightweight ‘one-click deploy’ system using an event-driven messaging architecture. This enables the website development teams to run their own deploys without requiring logins on the live servers, thus freeing up the Operations team from the requirement to manage/oversee anything up to 20 site deploys per day. The complete toolkit is available on Github.
  • Worked with the corporate Java team to integrate common build tools - Jenkins, Sonar, Nexus - into a similar deployment system. This again allows the developers to manage their own deploys and remove the requirement for cross-team scheduling.
  • Designed, built, tested and installed multiply-redundant 48Tb storage arrays using off-the-shelf hardware and open-source software to serve as ‘second tier’ storage. Headed the project from identification of business need through board-level signoff, hardware and software qualification and test to installation and handover to the backup team.
  • Managed Netapp cluster. Performed hardware and software upgrades, divined future storage requirements and managed customer expectation during capacity planning, storage allocation and data lifecycle management.
  • Installed/managed Kerberos-based SSO infrastructure across all internet-facing servers - OS X, Linux & Solaris.

March 2001 – November 2007 Hewlett-Packard Labs, Bristol.

  • Unix administrator for the RESS (Research & Engineering Support Services) organisation, which supports both the core computing infrastructure and the requirements of the research teams across the sites in Bristol and Palo-Alto.
  • Primarily responsible for the unix mailsystems, DNS, DHCP, Kerberos servers, Usenet system, security and patching of all ‘bastion’ (dual-homed, internet-facing) equipment.
  • Project lead during the renumbering of all Bristol equipment from 15/8 to 16/8 on an aggressive schedule. Moved the majority of the equipment from static IP allocation to DHCP, liased with the customers to ensure no interruption of service and wrote code to manage the new and old IP ranges.
  • Investigated, tested, and rolled out a mailserver-based open-source virus-scanning system. Later upgraded that system to offer anti-spam functionality and assisted in the installation on the other two labs sites.
  • Carried out a rolling process of improvements and upgrades to the mailsystems to increase reliability, availability and functionality. Provided extensive documentation for the other members of the team so that the mailsystems ceased to be a ‘black box’. Worked with my counterpart in PA to ensure compatibility in the shared parts of the system.
  • Maintained the disaster-recovery plan and had cause to successfully use that plan ‘in anger’.
  • Worked with the networks team to roll out and test the infrastructure to support an experimental IPv6 installation ‘to the desktop’.

April 2000 – February 2001: Another.com, London.

  • Unix administrator for one of the UK’s fastest-growing web destinations, providing free webmail for over 1,000,000 users. Part of a team responsible for keeping the site live 24x7.
  • Specified, commissioned, installed and tested servers for many parts of the system, including backup, developer support, monitoring and SMTP engines.
  • Worked as part of a team which managed the rapid migration of the mail-stores from SCSI-attached JBOD units to multiply-redundant fibre-attached MTI storage arrays, and from those to an EMC unit when the MTI kit proved less than capable. Responsible for ongoing management of those arrays using Veritas products.
  • Managed all backup systems within the organisation.

June 1998 – March 2000: Orchestream Ltd, London.

  • Unix administrator for start-up concerned with QoS over IPv4. Directly responsible for specification, evaluation, installation and day-to-day running of all Unix servers and workstations.
  • Responsible for tracking and applying upgrades and security patches to Solaris, Linux and FreeBSD systems, also for evaluation, test and installation of new and upgraded system, utility and development software.
  • Wrote and implemented strategy to ensure all Unix equipment was Y2K compliant.
  • Liased with external connectivity providers to ensure 99% uptime for IP connection.
  • Co-wrote, tested and implemented disaster-recovery plans for all mission-critical equipment.

May 1997 – June 1998: VBCnet GB Ltd, London.

  • Unix administrator for a major internet backbone provider. Responsible for ensuring 24x7 availability of internet provision to ISP and corporate customers countrywide.
  • Wrote & maintained client-server system for allocation, management and billing of local rate dial-up users, utilising a secure web-based front end communicating with a SQL database using custom Perl-5 CGI scripts.
  • Worked with two other programmers to create a robust, portable, C-based domain- name registration toolkit for top-level registries.
  • Built, configured & evaluated a variety of Apache modules to enable web/database connectivity using SSL to provide strong crypto and transparent end-to-end authentication.

Feb. 1992 – May 1997: Epinet Data Networks, Cheltenham.

  • Worked as part of a small team to create an Internet Service Provider for the Gloucestershire area. Involved in all aspects of the project, from equipment evaluation through to installation and configuration of routers, terminal servers, modem rack, Linux-based FTP server and Sun web server.

More skills:

  • Custom kernel configuration, installation, test, debugging and tuning on HP-UX 10.20 – 11.11, Linux 2.x, and FreeBSD 2.x - 14.x.
  • IPv4 and IPv6 network design, installation and troubleshooting. Firewall design, installation and maintenance. ‘Hardening’ and testing of Internet-facing equipment.
  • C, Perl & Ruby under Solaris, HP-UX, BSD & Linux – Development of web-based front ends for database engines & high-reliability IPC code. C under Windows and DOS - Concerned with robust comms over anything from three-wire serial links to thin-wire Ethernet.
  • Installation, set-up and maintenance of named, unbound, NSD, sendmail V5 & V8, Postfix, Amavisd, SpamAssassin, UWash & Courier IMAP, Jenkins, OpenSSH/ OpenSSL, Apache, Nginx, samba, gated, FreeRadius and ActiveMQ installations.
  • Hacking on Arduino/ESP32/RPi microcontrollers and camp badges in C to make them do cool things with RGB LEDs at 2am in fields in the UK, The Netherlands, and Germany.
  • Component-level fault diagnosis and repair on a variety of computer equipment ranging from DEC LSI-11 downwards, both on the bench and in the field.

Education:

September 1985 - July 1987 Cheltenham & Gloucester College of Arts & Technology. TEC II Electronic Engineering.

Interests:

  • Writing queer SF & romance stories and getting published.
  • DJing industrial techno, prog house & EBM.
  • Creating automata from recycled computer hardware.
  • Making filthy industrial techno.
  • Pointing a camera at things.
  • Cold War and Modernist architecture.
  • Volunteering at EMFCamp 2012-2024, CCCamp23, Hxx events (HEU93, WTH05, HAR09, OHM13, SHA17).
  • BFS Karl Edward Wagner award winner 2016.
  • Nominated for BSFA and BFS awards - well, the anthology I had a story in was - 2025.