Senior Software Engineer

About the role

We are looking for an experienced software engineer to join Outcold Solutions. We are a small, independent company, and you will work across the whole product with the founder and the rest of the team: the Collectord agent, the Splunk apps, the services behind them, and everything our customers read and watch.

This is a systems engineering role first. Collectord runs on every node of a customer’s cluster, on Linux and Windows, in environments we do not control. It has to be correct under rotation, restarts, and back pressure, cheap on CPU and memory, and easy to reason about when something goes wrong at three in the morning in someone else’s data center. If that kind of problem appeals to you, read on.

• Design, build, and maintain Collectord in Go: log tailing, metrics collection from cgroups and /proc, Kubernetes API watchers, and the output pipelines to Splunk, Elasticsearch, and Open Search.
• Work close to the operating system on both Linux and Windows: container runtimes, file systems, process and cgroup accounting, and the behaviors that differ between the two.
• Build and extend our Splunk apps: dashboards, alerts, custom search commands, and React setup pages. Experience with a similar platform such as Elasticsearch, Open Search, Grafana, or Datadog is equally welcome.
• Work with data at rest: embedded SQLite in the agent, Athena over telemetry, and the Splunk indexes our dashboards search.
• Write JavaScript and Type Script where the product needs a user interface, and keep it small.
• Write the documentation, release notes, and blog posts for what you ship. Record screencasts and build demos.
• Help customers: reproduce their environments, answer support requests, and fix root causes. Take part in an occasional on-call rotation.
• Represent the company at conferences a few times a year.

You do not need to check every box. If you are strong in most of these and curious about the rest, we want to hear from you.

• A track record of shipping and operating production software, with the scars to show for it.
• Fluency in Go, or deep experience in another systems language such as Rust, C, or C++ and the willingness to work in Go every day.
• A working understanding of Linux internals: processes, cgroups, name spaces, file systems, signals, and networking. Familiarity with the Windows equivalents is a plus.
• Hands-on experience running Kubernetes, Open Shift, or Docker in production, and an understanding of how logs and metrics actually flow through them.
• Comfort with SQL and with the trade-offs of embedded and analytical databases.
• Experience with Splunk or a comparable log and metrics platform: how data is indexed, searched, and visualized.
• Enough JavaScript to build and maintain a modest web interface without a large framework.
• Clear written communication. Much of our work reaches customers as documentation, and you will write a lot of it.
• The habit of measuring before deciding, and of proving a change with a test before calling it done.

• Contributions to open source projects in the Kubernetes, container, or observability ecosystems.
• Experience with AWS Lambda, Cloud Formation, or metered billing systems such as Stripe.
• Experience giving talks, recording videos, or writing for a technical audience.
• Familiarity with FIPS or other compliance requirements for shipped binaries.

• Small team, no hand-offs. You own a change from the first design note to the support ticket it eventually generates.
• Every change is tested. Unit tests run in containers, and integration suites bring up real Kubernetes clusters and real Splunk, Elasticsearch, and Open Search instances.
• We write things down. Design notes, README files, and release notes are part of the work, not an…