Starship’s Site Reliability Engineering | Author: Martin Pihlak | Starship Technology

Running autonomous robots on city streets is largely a software engineering challenge. Some of this software runs on the robot itself, but many of it actually runs on the back end. Contents such as remote control, wayfinding, matching robots with customers, fleet health management, and interaction with customers and businesses. All of these need to run 24×7 uninterrupted and scale dynamically to match the workload.

Starship’s SRE is responsible for providing cloud infrastructure and platform services to run these back-end services.We have standardized Governor Used for and runs on our microservices vending machine. database Is the main database for most back-end services, but we also like PostgreSQL, Especially when strong typing and transaction guarantees are required.For asynchronous messages Kafka It is the preferred messaging platform. We use it for almost all other purposes except for delivering video streams from robots.For observability, we rely on Prometheus with Grafana, Rocky, remain with Jaeger. CICD by Jenkins.

Most of SRE’s time is spent maintaining and improving the Kubernetes infrastructure. Kubernetes is our main deployment platform, and there are always some areas for improvement, whether it’s fine-tuning the auto-scaling settings, adding a Pod interruption strategy, or optimizing the use of Spot instances. Sometimes it’s like laying bricks-just install the Helm chart to provide a specific function. But usually you must carefully select and evaluate the “bricks” (whether Loki is suitable for log management, what is Service Mesh, and which). Sometimes functions don’t exist in the world and must be written from scratch. When this happens, we usually turn to Python and Golang, but we also turn to Rust and C when we need it.

Another major infrastructure that SRE is responsible for is data and databases. Starship started as a single MongoDb-a strategy that has worked well so far. However, as the business grows, we need to re-examine this architecture and begin to consider supporting thousands of robots. Apache Kafka is part of the expansion story, but we also need to figure out sharding, regional clusters, and microservice database architecture. Most importantly, we continue to develop tools and automation to manage the current database infrastructure. example: Use a custom sidecar agent to add MongoDb observability to analyze database traffic, enable PITR support for the database, automatic periodic failover and recovery tests, collect Kafka re-sharding metrics, and enable data retention.

Finally, one of the most important goals of site reliability engineering is to minimize the downtime of Starship production. Although SRE is occasionally called to handle infrastructure outages, it has done more influential work in preventing outages and ensuring that we can recover quickly. This can be a very broad topic, from having a rock-solid K8s infrastructure all the way to engineering practices and business processes. There is a great opportunity to make an impact!

A day in the life of SRE

Working hours are between 9 am and 10 am (sometimes working remotely). Have a cup of coffee and check Slack messages and emails. Check out the alarms at night to see if there is anything interesting with us.

It was found that MongoDb connection latency soared at night. Using Grafana to dig deeper into Prometheus metrics, I found that this happened during the backup run. Why has this suddenly become a problem, we have been running these backups for years? It turns out that we are very aggressively compressing backups to save network and storage costs, which is consuming all available CPU. It seems that the load on the database has increased a bit, making this obvious. This happens on the standby node and will not affect production, but if the primary node fails, it is still a problem. Add a Jira project to solve this problem.

By the way, change the MongoDb detector code (Golang) to add more histogram buckets to better understand the delay distribution. Run the Jenkins pipeline to put the new probe into production.

There is a Standup meeting at 10 am to share your updates with the team and learn about the latest developments of others-set up VPN server monitoring, use Prometheus to detect Python applications, set up ServiceMonitors for external services, debug MongoDb connection issues, use Flager to try it out Canary deployment.

After the meeting, resume the planning work for the day. One of the plans I plan to do today is to set up an additional Kafka cluster in a test environment. We run Kafka on Kubernetes, so it should be easy to get the existing cluster YAML files and adjust them for the new cluster. Or, think again, should we use Helm instead, or is there a good Kafka operator now available? No, don’t go there-it’s amazing, I want more explicit control over my statefulset. The original YAML is. After an hour and a half, a new cluster started running. The setup is quite simple; just registering the initial container of the Kafka proxy in DNS requires a configuration change. Generating credentials for the application requires a small bash script to set up an account on Zookeeper. An unresolved issue is setting up Kafka Connect to capture database change log events-the result is that the test database is not running in ReplicaSet mode, and Debezium cannot obtain oplogs from it. Backlog this and move on.

It’s time to prepare a scene for the Wheel of Misfortune practice. At Starship, we run these to improve our understanding of the system and share troubleshooting techniques. It works by destroying certain parts of the system (usually in testing) and letting some unfortunate people try to troubleshoot and alleviate the problem.In this case, I will set up a load test Hey Reload microservices for routing calculations. Deploy it as a Kubernetes job named “haymaker” and hide it well enough so that it does not appear in the Linkerd service mesh immediately (yes, evil ????). Run the “wheel” exercise later and pay attention to any gaps in our scripts, indicators, alerts, etc.

In the last few hours of the day, stop all interruptions and try to finish some coding. I have reimplemented the Mongoproxy BSON parser as streaming asynchronous (Rust+Tokio), and want to figure out how it works with real data. It turns out that there is an error somewhere inside the parser, and I need to add a deep log to solve this problem. Find a great tracking library for Tokio and get fascinated by it…

Disclaimer: The events described here are based on real stories. Not everything happens on the same day. Some meetings and interactions with colleagues have been deleted. We are hiring.