matthiasr

Q: We are redoing our centralized on-call, ideal would be to have each 4-5 person pod on call for their own services but there's concern about the productivity hit.

A: For now, let's set the human element aside and only consider work hours.

Something I think about a lot in these situations is the feedback loop and responsibility. Take the point of view of whoever decides what's a priority sprint over sprint: if the systems are shoddy and it takes one of four engineers to keep them alive, they take the hit to their feature delivery. If the keeping-alive is done by some central ops team, or a diffuse on-call rotation that only hits your team every 3 months, congratulations you've externalized the consequences of your decisions while getting to claim credit for all the stuff you've delivered. The SWE productivity hit your managers are lamenting is the cost of internalizing these costs. The benefit is autonomy in making prioritization decisions locally based on what makes sense for this area. Structurally, you

matthiasr

Abstractly, a vector is a direction and a length in some space. For example, in the space around you, a vector might be "from your face to your cup". There's multiple ways you can represent that vector with numbers. For example, it might be 1 foot forward, 1ft to the left, 1 ft down in the way you're facing, or it could be 1.4 ft north, 0ft east, 1ft down. Or even 45°left, 45°down, 1.7ft away. These are all equivalent descriptions of the same vector assuming you're facing northeast. In the space around you, you always need exactly three numbers to describe a vector, it's 3-dimensional. The numbers describe the vector, but the vector itself is just direction+length, no matter how you decide to put that into numbers.

Now there's other spaces that you can also use vectors in, that need a different number of numbers to describe. For example, the "what's the sportsball doing right this instant" is a 6-dimensional space: you need 3 numbers to describe where it is, and another 3 to describe in which direction and h

matthiasr

I'm generally in favor of picking up new versions automatically IFF you have enough guard rails in place to deal with the breakage whenever it occurs. You'll inevitably encounter that breakage at some point, you kind of have a three way choice when:

You can update explicitly and regularly, doing the appropriate checking at that point, but that takes a lot of discipline to do across the board You can pin and leave it until you need to upgrade, but then you're dealing with all the problems at once, less well exercised upgrade paths, and sometimes having to figure out multi step upgrades to get around all the mutual version constraints. Generally this happens at a time when you really need to get something else done and you did not plan to spend a week sorting out dependencies. You can leave versions floating and deal with any breakage as it comes up. This means only dealing with one or two issues at a time. You don't really control when, but it's less disruptive each time, and you can temporarily pin if yo

matthiasr

In my experience, there are three major sources of work for an opsy team:

• Keeping the lights on – updates, maintenance, keeping entropy at bay. You can barely control how much of this there is, however you can control how much work it is to deal with through
• Work to improve your own situation: automating stuff, architecting better systems, evolving processes, getting dev teams to do something in more ops friendly ways. Both of these compete for focus with
• Support work, that is helping product teams in all the things they need to build product. This is almost always interrupt-driven, you can try to get wind of these needs ahead of time but if you insist on exact requests in time for your own sprint planning, you're instantly at least doubling the lead time on everything.

Things I have seen work to deal with this:

• At an appropriate granularity of organization (by default, per team), designate a rotating first responder. This is the person who takes in short term requests on whatever communications ch

matthiasr

With the caveat that this is based on a sample size of 2, I would lean towards one repository per deliverable thing – one app, one service, or service with worker components.

Tooling support for monorepos is not great. You can make do but you end up either running bazel or wishing you were running bazel. The usual hosted CI feels geared towards repos per use case, GitHub gets creaky with large amounts of code and change, IDEs start to struggle.

In a monorepo, it's easy to share code, but hard to keep a handle on ownership of that shared code. With separate repositories it's much more legible to management when a core piece of code is unowned and unmaintained, so it's easier to get clear funding for this.

Versioned releases of shared frameworks, and generally smaller repositories, make it a lot easier to see what's in a deploy. Consequently they make deployment and rollback a lot safer. You pay a price for that though: either you live with a huge amount of drift in those core frameworks, making it nearly im

matthiasr

If VS Code does not format Terraform code on save, but does format correctly when ask explicitly (Command palette -> Format File), add this to the configuration JSON:

    "[terraform]": {
        "editor.defaultFormatter": "hashicorp.terraform",
        "editor.formatOnSave": true
    }

source

matthiasr

Problem statement

Signing with GPG in Windows System for Linux (WSL2) does not work smoothly out of the box. Notably, when using a TTY-based pinentry, signing in Visual Studio Code does not work at all.

Solution

1. Install Gpg4Win: winget install -e GnuPG.Gpg4win or download and install manually
2. Start Kleopatra and generate or import keys
3. Insert links to gpg.exe inside of WSL:

matthiasr

First of all, read about NALSD. Not every company leans that hard into the scaling aspect but even if they don't, having it in the back of your head helps.

If they give you any numbers on scale at all, write them down. You will need them shortly. For online interviews, have something you can draw into. If something is provided in the invite, play around with it ahead of time to make sure you can draw boxes of different shapes and connect them with arrows. If not, have an empty canvas ready in your online drawing tool of choice.

Figure out the general shape of the problem. Is it a request-response type system (we want a website that does X), a stream type system (we get events and process them as they come), a batch type system (we have a pile of data and want to get through it)?

Clarify anything you are unsure about – this is about communication as much as system design. Listen for what they want you to do – ask them for every underspecified detail, or us

matthiasr

20% time is a buffer against many kinds of process failure. If your planning is perfect, you can always exactly predict what the costs and benefit of future work will be, and how long everything takes, you don't need it. If your process is perfect, you can perfectly line up every tooling improvement, every product experimentation, every tech debt repayment, every personal development investment, at 100% utilization.

But you can't. Sometimes you can't really formulate how something will improve productivity in the future, what feature your product has been missing, which blog post is the one that each engineer must read. 20% time acknowledges that you're only going to be right about 80% on all this – and that there is value in having slack for the other 20%.

This time becomes a buffer for planning emergencies (because let's be real, sometimes planned work spills over into it), for exploration that doesn't make it into a roadmap until the exploration has already happened, for productivity improvements that yo

matthiasr

That's a whole system design interview question right there 😉

Fundamentally, if you need that level of detail and fidelity, you compromise on timeliness. You (somehow) record an event for every message sent, then tally those up on a daily or monthly basis.

It depends on the system and the details of your requirements. I would start by looking at how messages are being sent to begin with. For example, if there's a job in a database, you might be able to count the jobs marked as completed right there.

If the message system is fed by a Kafka topic, and you can get away with billing for messages attempted to send, you can count those from the source topic.

You can also explicitly record events, say into another Kafka topic or a database table. You will need to make some choices about CAP un that case: if recording the event fails, do you want to send anyway? What if you're not sure recording it worked?