A large part of a manager’s role is to make decisions and be responsible for their outcomes. While there is ample advice on how to be successful in many other managerial core areas, such as growing your people, the domain of high-quality decision-making seems less crowded. In this post, I summarize what I have found during my 20+ years as a manager to be a simple and effective way to approach decision-making.

The first step in making a decision is to identify that there is a decision to be made to begin with. Ask yourself: Does something have to be done in a particular way, or are there options? If there is only one option, is the decision about timing when to execute it? To be able to drive influence as a manager, one must be able to recognize the opportunities to make decisions.

A manager might also be explicitly asked to make a decision. People might be at crossroads and waiting for a manager to take the responsibility of choosing which way to proceed. In those cases, a manager should also start by finding out what the possible options are beyond what was presented initially.

The ability to grasp a process, break it down into smaller steps, and exhaustively find the options is a skill that can be honed. You can, however, accelerate acquiring this skill by constantly asking yourself, “Are there more options?”

Once you have at least two options you can embark on the exploratory phase of collecting data on the various options. During this phase you can still discover more options, and then also explore them.

The reason you need at least two options to explore is that you need to be able to rank them on some metric. Hence, the exploratory phase should also lead you to uncover what are the metrics you actually care about. In addition to discovering options you thus also discover what is the optimal outcome of the decision. The need to make a decision often arises from some sort of problem that was faced, but quite often it is unclear what would actually be the best possible outcome. Effective decision-making hinges both on discovering the options, as well as on discovering what outcome is most valued. The alternative options as external factors dictated by the situation, while the discovery of the desired outcome comes from within and from one’s values.

Once you understand the options and the optimal outcome, it is fairly easy to rank them or use techniques such as a SWOT matrix to compare the options. With complete information, most decisions become clear and most managers would end up making the identical decision. The differentiating factor is thus not the actual decision making given the facts, but mostly at what point are different people content with the data points collected. As a general rule, if you have the time and resources, continue to push for slightly more data points even after you reach a level where you initially thought you had enough information.

When prioritizing data collection, try to think ahead what kind of information would definitely confirm something, or what discovery would disprove an assumption. If you start to strongly lean towards some decision already during the exploratory phase, put effort in specifically seeking for views and information that would disprove it.

Obviously, when the stakes are high, a manager needs to spend a lot more energy on making the decision. To ensure the correct amount of effort is spent on each decision, be explicit about assessing the importance and urgency of a decision. Small decisions should be made frequently and without a delay. If a decision has significant impact, but is not urgent, use that to your advantage and postpone it to allow more information about options to be collected.

Additionally, consider the finality of a decision. If the decision is easy to revert, there should be less of a reason to delay it. The famous concept of “one-way and two-way doors” helps to understand this well. No decision is ever totally free from consequences if reverted, but understanding where the decision sits on a scale of “picking a hat, haircut or tattoo” significantly helps in making a better decision.

This is probably the biggest challenge, even for very experienced leaders. There is no source of absolute truth and everything we hold in our heads is a result of ingesting decades worth of information with varying degrees of trustworthiness. Even if there were some imaginary filter that ensures we only learn about things that are true, many snippets of information will eventually become outdated and false over time. Our brain is constantly in a flux of multiple levels of different thought processes, emotions and moods. We can make good decisions only by using our own brain, yet at the same time we need to be aware that we can’t fully trust our brain.

Setting aside the epistemological thoughts that there is no absolute knowledge, even when we could have access to the truth we might fail to recognize it if our brain is trapped by unconscious bias. To have a better chance at breaking free, everyone should familiarize themselves with the common cognitive biases in order to recognize when running at risk of any of them. Listening attentively to people with opposing views is also a good way to break bias.

You should be particularly careful in your decision if you feel you knew the decision before collecting enough data points to support it. Don’t let your quick but stupid lower limbic brain system in the driver seat, but instead make a continuous effort to allow your brain cortex to process things and only then make the decision.

The previous paragraph neatly leads us to the last, but not least, important advice on how to make good decisions: sleep on it. Ask anyone who worked with me and they will remember at least one case where I said this and postponed making the final decision by one extra day.

You should continue working on a decision until you think you have everything you need to make the decision, no further research or consultations are needed, and you could just announce it. However, if the decision is not urgent to the day, rather write down your decision as a draft, keep it for yourself and sleep on it. The next morning look at the draft, ask yourself if you still agree to it, and only then commit. Having that extra night of sleep not only ensures our energy levels are recharged and we are more likely to think clearly, but also allows our brain and unconscious to process the information and thoughts from the previous day. Sometimes you might wake up in the morning having realized that you missed something or that you actually value a certain outcome more than another.

If the decision is urgent and it can’t wait until the next day, you can still drastically increase the quality or at least the confidence of a decision by taking a break, going for a walk or at least taking a couple of deep breaths before committing.

The saying “sleep on it” specifically means just one night of sleep, or perhaps a weekend, but not postponing a decision for too long. If a decision is postponed for weeks or months, the circumstances are likely to change, and it is no longer the same decision. Something can of course be postponed, but in those cases the decision should be explicit that the decision is to postpone. Avoiding making explicit decisions is just bad management.

It is said that the Finnish army taught its leaders during WWII that if they don’t know what decision to make, then always just “hook from the right’’. It was considered both detrimental to troop morale and tactically inferior to stay put in the same location on the battlefield. Keeping the troops moving and executing a maneuver, even with incomplete information about the enemy positions on their right flank and taking a huge risk, was considered a superior option compared to not making any decision at all.

Luckily, very few of us are forced to make decisions about matters of life and death, but all managers need to remember that making decisions is a core duty of theirs, and that taking some action that leads somewhere is better than not making a decision at all. If the decision was wrong, one should own it and learn from it, so that the next decision will be better. Indecisiveness is worse, and will eventually lead to a figurative slow death.

If you have principles you follow or anecdotes about making decisions, please share them in the comments!

In software engineering, most ideas can be implemented without writing any design document at all. This is particularly prominent in open source communities. For example, the Linux kernel has 35 million lines of code that have been written and rewritten many times over alongside 30 years of mailing list discussions. Linux wasn’t created as a result of a grandiose design document by Linus Torvalds, but it evolved organically in small increments of actual running code.

In open source and in software engineering in general, ideas are presented most of the time directly as patches that add, delete, or change specific lines of code. Those patches are not only read but also directly built, run, tested – and contested. Design and implementation are intertwined, and decisions are relatively small, quick, and done in writing.

However, this is not always the best way to evolve software. I would argue that even in open source, design documents are needed, and it would be good to see more of them being written than what is the practice today. Let’s dive into why, when, and how to write design documents.

Design documents have three clear benefits. First of all, a design document helps to manage technical risk and organizational cost risk. If it takes several months or years to develop something, starting the process with a high-quality design document helps to map out unknown dimensions and decreases the technical risk of the idea being impossible to implement. If there is no design and something is developed right away, there is also a risk that it might be rejected after implementation by downstream users or collaborators, and thus all work put in would be wasted. This is the first benefit of design documents.

Secondly, design documents are an excellent medium to communicate the intent of the change to others. An engineering team might have stakeholders, executives, customers, maintainers of other dependent software packages, legal requirements, etc. While it’s perfectly possible to develop something without writing the design first, designing forces a clear articulation of what the idea is about and why it is needed.

Thirdly, writing documents grows the author’s ability to think. When writing, it becomes clear very quickly what parts of the idea are still vague, ambiguous, incomplete, or even misunderstood. It helps in revealing blind spots, giving ideas shape and detail, and thus increasing their quality. Dumping a part of your brain in a document, and then revisiting and restructuring those thoughts many times over will always lead to a better thought process and higher quality of outcome.

In open source, you rarely see people writing design documents or conducting formal review and approval processes. They do, however, exist, and open source developers should also practice their skills in writing design docs. A well-written design document not only conveys the merits of a great idea clearly but also shows that the author is a great thinker and fully understands what they are doing.

The above benefits are universal and benefit any type of software development. In the enterprise software setting, these additional aspects are often true, making design documents more common:

• Making the change happen requires a significant investment, potentially multiple people working full-time for an extended time. A written description of how that time will be used and what it will produce is needed to get buy-in from the authority that funds the development.

• Rolling out the change affects many other people and the software they develop and maintain. The whole change might be moot if there isn’t prior approval and commitment from the stakeholders to adapt to the change.

• The change might involve technical risks or have security considerations, and a technical plan needs to be vetted and approved by risk bearers before implementation starts to avoid major technical catastrophes.

Ad-hoc implementation may take place both in the enterprise setting and in open source, but ad-hoc is much more common in open source, as the person carrying the cost of the implementation work and the consequences is often the very same person. In contrast, in an enterprise setting, the costs and risks are carried by a larger group that needs to agree before any work starts.

In open source communities, ideas that are too large to be put into code directly typically surface first as a mailing list discussion or in an issue tracker thread. Full-fledged design documents are rare, but they do exist, particularly in large projects. Examples include:

• Request for Comments (RFC) documents from The Internet Engineering Task Force

• Python Enhancement Proposals (PEP)

• Debian Enhancement Proposals (DEP)

• Ethereum Improvement Proposals (EIP)

If in doubt, just start writing a design document. It is cheap, and you can always stop in the middle and never publish it. The mere fact that you are contemplating it is a sign that you probably have some unstructured thoughts floating around in your brain, and starting the writing process will benefit you a lot.

To decide if you should go all the way and actually finalize and publish a design document in contrast to just implementing the idea directly, consider the three benefits listed above: risk management, communicating intent, and building trust in the author’s ability. Are these benefits relevant for your idea? If so, publish a design document.

If you decide to start writing a design document, don’t use a template; simply start with a blank sheet. Writing down any idea — large or small — should start with a paper where the author notes down the core idea first. Don’t start from a template, and in particular, avoid the pitfall of grandiose thoughts that lead to convoluted designs and bloated documents.

I’ve seen many times over the past 15 years the pattern of design document templates repeat, and it has never resulted in high-quality outcomes. Various design document templates, and fancy-sounding software development methodologies in general, are surely a good business for large consulting companies, but I have never seen them actually increase the quality of software development. In the best case, templates result in good-looking documents that are thin on content, but in the worst case, they massively dumb down the authors and put them in a mode where they are exonerated from all responsibility about the contents.

Just start from scratch and focus on the core idea. If you can’t express it briefly and clearly in a one-pager, you need to spend more time thinking about the core idea. Polish the idea before you even consider polishing the document.

Only once the core idea is crisp and clear, and there is agreement to invest in it, should the work to polish the design document start. If there is a template, this is the point in time when it makes sense to start applying the template.

Having a crisp one-pager first also helps write the design document in a way that first presents the solution, and only then the motivations for the solution. If people jump directly to write the final design document, it often starts with a lengthy analysis of the problem and motivation of the solution, and only then the solution. Such a structure reflects the thought process of the author but is not a good way to structure a design document. The design doc should always start with the solution, and after that, the rest of the document exists to motivate and support the solution.

Writing out the full design document is all about constantly expanding and iterating it. To flesh out all aspects, it is good to have a list of questions and ensure that they are all addressed while working through the document:

• What is the title? If there are similar competing or earlier designs, what is the unique identifier of this specific design doc?

• Who is the author? Who is responsible for the design being good and correct?

• Who is going to implement it? Who is sponsoring or funding the design or implementation work?

• What is the status of the document? Is it, for example, just a draft, or is it pending comments or review, or has it already been approved?

• If the document is approved, who approved it and when? How is the approval tracked? Is it easy to prove what specific version of the design each approver read when giving their approval to it?

• What is the proposed solution exactly? Are there diagrams, pictures, prototypes, or others that cover the key parts of the proposed solution?

• What is the scope of the solution? Was something intentionally left out and why?

• Why should this solution be done now? What happens if it is not done at all, or if it is postponed? Are there workarounds?

• What assumptions is the design based on? What happens to those assumptions over time? Will they still hold?

• What alternative designs were considered? Why is the proposed solution the best?

• What are the known trade-offs and downsides of the proposed solution? How are those being mitigated?

• What is the work estimate and cost of the solution? What is the long-term cost and total cost of ownership?

• What is the development and testing plan? How will it be rolled out? Can the rollout be phased? Can it be rolled back if needed?

• What is the quality assurance process? How is security being reviewed and assured?

• What is the impact on performance? How does the system scale? Where are the limits of scalability? What is the maximum load to be used in load testing and benchmarking?

• How will the solution be operated, monitored, and measured?

• How is the success of the solution measured and validated? How does one know if the solution actually worked?

• When the solution is ready, how will it be documented and communicated? Are there different audiences that need different communications (e.g., internal vs external, developers vs users)?

Designing is not fast. Authors should not expect to be able to sit down one day and write a design document. A good idea takes time to mature, several revisions of writing down to become clear, and a lot of time spent on waiting and getting feedback. For this reason, designing can’t be the main task for anybody but should be done on the side of other work. The design of the next idea should typically be in progress already while implementing the previous one.

Jeff Bezos famously wrote in the 2017 letter to shareholders:

The great memos are written and re-written, shared with colleagues who are asked to improve the work, set aside for a couple of days, and then edited again with a fresh mind. They simply can’t be done in a day or two.

― Jeff Bezos, founder of Amazon

A complete and high-quality design document takes a lot of calendar time. A good design matures like a bottle of wine. It can’t be forced to take shape quickly. Designing is like practicing wisdom – give it time.

The XZ Utils backdoor, discovered last week, and the Heartbleed security vulnerability ten years ago, share the same ultimate root cause. Both of them, and in fact all critical infrastructure open source projects, should be fixed with the same solution: ensure baseline funding for proper open source maintenance.

Open source software is the foundation of much of our digital infrastructure. From web servers to encryption libraries to operating systems, open source code powers systems that millions rely on daily. The open source model has proven tremendously successful at producing innovative, reliable, and widely used software.

However, the Heartbleed vulnerability in OpenSSL and the recent backdoor discovered in the XZ Utils compression library have highlighted potential weaknesses in how open source software is funded, developed, and maintained. These incidents showed that even very widely used open source projects can have serious, undiscovered bugs due to lack of resources.

Today, April 7th, 2024, marks the 10-year anniversary since CVE-2014-0160 was published. This security vulnerability known as “Heartbleed” was a flaw in the OpenSSL cryptography software, the most popular option to implement Transport Layer Security (TLS). In more layman’s terms, if you type https:// in your browser address bar, chances are high that you are interacting with OpenSSL.

The fallout from Heartbleed was immense, prompting widespread panic among developers, businesses, and users alike. About one-fifth of all web servers in the world at the time were believed to be vulnerable to the attack, allowing theft of the servers’ private keys and users’ session cookies and passwords.

The software bug existed in OpenSSL’s codebase for over two years before being discovered. While code reviews were in place, the bug wasn’t spotted and went into OpenSSL’s source code repository on New Year’s Eve, December 31st, 2011. At the time, the OpenSSL project was maintained by a small 4-person team with limited funding and basically working as volunteers, driven by just the importance of their mission.

This was the ultimate root cause – a piece of software that had started as a hobby project (just like Linux) grew over time and became part of the Internet infrastructure, but there was no mechanism to ensure resources would grow to be able to maintain it well long-term.

In April 2014, the Linux Foundation Executive Director Jim Zemlin seized the opportunity to get visibility and managed to get Amazon Web Services, Cisco, Dell, Facebook, Fujitsu, Google, IBM, Intel, Microsoft, NetApp, Qualcomm, Rackspace, and VMware to all pledge to commit at least $100,000 a year for at least three years to the Core Infrastructure Initiative. The initiative continued for many years and eventually transformed into the Open Source Security Foundation. Also due to Heartbleed, the European Commission launched the EU-Free and Open Source Software Auditing project and spent at least a million euros on auditing OpenSSL, the Apache Server, KeePass, and other security-critical open source software.

This relatively modest funding, along with code audits and process improvements, allowed OpenSSL to become more secure and sustainable. Today the OpenSSL project is thriving: it is FIPS 140-2 certified and has a healthy base of both financial and code contributors.

While there are surely still more details to uncover in the coming weeks, when the news broke about the XZ compression software backdoor (CVE-2024-3094), it was immediately clear that it happened because XZ had become hugely popular and widely used but was still maintained by one single overworked person as a spare time project. A well-resourced malicious actor was able to manipulate and pressure the maintainer to give them commit access, and thus the software supply chain was compromised. We should not blame the original maintainer, but rather everyone else for not realizing how widely used XZ was, yet going by with very little support and resources.

A huge number of applications depend on XZ. Right now the priority should be to offer help to maintain it properly, both upstream and at various downstreams, such as in Linux distributions, so the whole software supply chain is secured. It does not require a massive effort – just having a couple more maintainers to share the maintenance and review work should go a long way.

In both cases, the vulnerabilities were fixed quickly because the world had access to the source code of the affected software. This is a major advantage of open source software: it allows anyone to inspect the code and find potential vulnerabilities, and submit fixes to them.

In the case of Heartbleed, Google’s security team reported it to OpenSSL first, but the Finnish national NCSC-FI has records of local cybersecurity company Codenomicon reporting it independently. In the case of XZ, a Microsoft employee and PostgreSQL developer Andres Freund found the backdoor while doing performance regression testing in a Debian Linux development version. It was a huge fluke of luck that the XZ backdoor didn’t go in any actual Linux distribution releases. Next time we might not be as lucky, so more reviews, testing, and validation are needed. It will need resources, but at least public review is possible – thanks to this infrastructure-level software being open source.

Public scrutiny, testing, and validation are not possible for closed-source software. In fact, if closed-source code gets backdoored, it will go unnoticed for a much longer time. For example, the 2020 U.S. government data breach was possible due to multiple backdoors and flaws that went undetected for a long time in closed-source software from SolarWinds, Microsoft, VMware, and Zerologon. In theory, companies always have money (unless they are bankrupt), but in practice, the pressure to channel that money into software review and testing varies wildly, and working without exposure to public scrutiny often incentivizes companies to skimp on security to maximize profits.

Thus, I firmly believe in open source software having a better overall security posture as long as there are reasonable resources. And if the source code is public, anybody can audit how active the maintenance is and thus also the fact if maintenance is funded itself is a public and auditable property of open source.

Both Heartbleed and the XZ backdoor incident underscore the critical role that open source software plays in powering the digital infrastructure of today’s world. Such important and widely used projects shouldn’t be struggling to get by. It’s time for companies to step up and provide reasonable funding to the projects they depend on.

You don’t need billions to meaningfully improve open source security – the OpenSSL example shows that even modest funding increases can have an outsized impact. A tiny slice of the corporate IT budget pie could go a long way. Additionally, some of the government defense spending should be funneled into key open source software projects that our society relies on.

The incidents of Heartbleed and the XZ backdoor serve as sobering reminders of the vulnerabilities that may exist within our open source infrastructure today. However, they also present an opportunity for positive change. By investing in the security and maintenance of open source projects through moderate funding and support, we can enhance the resilience of our digital infrastructure and ensure a safer and more secure internet for all.

For a software engineering organization to be efficient, it is key that everyone is an efficient communicator. Everybody needs to be calibrated in what to communicate, to whom and how to ensure information spreads properly in the organization. Having smart people with a lot of knowledge results in progress only if information flows well in the veins of the organization.

This does not mean that everyone needs to communicate everything – on the contrary, it is also the responsibility of every individual to make sure there is the right amount of communication, not too much and not too little. From an individual point of view, it is also important to be a good communicator, both verbally and in writing, as that defines to a large degree how professionally others will perceive you.

Reflecting on the principles below may help both your organization and you personally to become a more efficient communicator.

Foster a culture where people share small updates early. When you introduce a change, describe it immediately. Don’t accept “I will document it later” from yourself or others. People are interested in the change when they learn about it, and should be able to immediately read up on git commit messages, ticket communications, etc. When you make a change that affects the workflow of others, announce it immediately. Don’t wait until other people run into problems and start asking questions.

When you announce a change, be exact. If the change has a commit id or a URL, or if there is a document that describes it in detail, reference it. Avoid abbreviations and spell out the names of things to avoid misunderstandings. Use the same name consistently when referencing the same thing. Don’t be vague if being exact requires just a couple seconds more of effort. If you know something, spell it out and don’t force other people to guess. In a larger organization, it might even make sense to have a written vocabulary to ensure that people understand the daily jargon and assign the same meaning to the words used.

Keep in mind that you, the announcer, are one person, but your audience consists of many people: if you take additional time and effort to be precise, you may save a great deal of repeated effort by many others to determine precisely what you were referring to. When you see other people putting effort into making easy-to-understand, brief, and crisp communication, thank them for it.

Always keep communication in context. If a piece of code does something that needs an explanation, do it in the inline comments rather than in an external document. For example, if the user of a software application needs guidance, don’t offer it on a completely separate system that is hard to discover, but instead offer it via the user interface of the software itself, or in a man page or a README that a user is likely to come across when trying to use the software. If there is a bug that needs to be debugged and fixed, discuss it in the issue tracker about the bug itself, not in a separate communication channel elsewhere. If there is code review open on GitHub or GitLab, don’t discuss it on Slack or equivalent, but do it in the code review comments – as it was designed for. Always communicate about something as closely as possible to the subject of the message.

Prefer asynchronous channels over synchronous channels. Chat systems like Slack or Zulip are better than a phone call. A well-written email is better than scheduling a 30-minute meeting. In an async channel, people can process incoming communication and respond in their own time. Sometimes a response requires some research and is not possible in real-time anyway. Having to juggle schedules can also be a wasteful use of time compared to working through a queue of tasks. Interrupting software engineers is very costly, as it can take tens of minutes before one gets back into “flow” and back to cracking a hard engineering problem. Also, as many teams today work across many time zones, you might need to wait until the next day for the reply anyway.

When using Slack and similar chat software, try to pick the most appropriate channel. Avoid private one-on-one discussions unless the matter is personal or confidential. The discussion in a public channel is more inclusive and allows others to join spontaneously or to be pinged in the discussion to join it. In Slack and similar chat systems that have threads, use them correctly to make it easier for participants to follow up on the topic while at the same time keeping the main channel cleaner. Avoid using @here on channels with more than 20 participants. Get into the habit of using Shift+Enter to write multiple paragraphs in one message instead of multiple short ones in succession which might cause unnecessary notifications.

In chat systems, do not send people messages that only say “Hello” . Get straight to the point.

Have a chat when it is appropriate. If you feel there is miscommunication and you can’t resolve it async with well-written and thoughtful messages, a short chat 1:1 or with a small group of people can bring a lot of clarity. An in-person meeting or video chat usually works best, as both parties can read each other’s cues to see that they understand and can follow the topic.

Team members interact mostly with others inside their team. It is not the responsibility of individual team members to know what people in other teams are doing. If something noteworthy is happening or is planned to happen, it is the responsibility of the team lead to communicate that upwards and laterally along the organizational lines. The team lead is also responsible for the inward flow of information and making team members aware of things that are relevant to the team.

The reason teams exist is to limit the flow of information. Most organizations are divided into 5–15 person teams simply because if teams were very large with 20 or more people, the overhead of everybody communicating with everybody would eat up too much time.

With this in mind, please be considerate and try to avoid approaching individual engineers in other teams too often. Channel communication through managers and architects, who are responsible for gatekeeping and prioritizing things. In a large organization, if you notice that people are reaching out to you personally all the time, just politely refer those requests to your manager.

In particular, when doing cross-org communication for large groups of people, think about the signal vs noise ratio. Free flow of information may sound like a noble principle, but a lot of information does not necessarily convert into real knowledge sharing. Write and share summaries instead of raw information. Be deliberate in selecting who should know what and when.

Principles for good meetings:

• If you organize a meeting, make sure it has an agenda in the meeting invite so attendees know what the meeting is about, and have a chance to prepare for the meeting in advance. The agenda also allows people to make a better-informed decision if they can skip the meeting or not

• If the meeting makes decisions, those should be written down somewhere (e.g. design doc, issue, ticket, meeting minutes). People tend to forget, so there must be some way to recall what the meeting decided.

• Don’t invite too many people. If there are more than 5 attendees in a 30-minute meeting, there will be no genuine discussions as there would be less than 5 minutes of speaking time per participant.

• Don’t attend all possible meetings. If dozens of people are invited to the meeting and it seems like an announcement event rather than a discussion, maybe just skip it and read the announcement documents instead.

The purpose of progress information is to allow others to learn the state of an issue and allow them to adapt their own work in relation to that issue. Issue status information also helps the author themself to remember where and in what state they left something, essentially being communication to their future self.

Good principles to follow:

• Avoid duplication. If an issue tracking system is in use in an organization, and an issue tracker entry has been filed, maintain it and do not disperse the information out in multiple places. Focus your energy on making sure the issue tracker is up-to-date so followers of that issue don’t need to ask about status or search for separate updates in email or old chat messages.

• Keep status information current. There is no point in a status-tracking system if the statuses are out-of-date. On the other hand, there is no need to update the status daily. A good rule of thumb is to update important status information immediately when it happens and less important statuses perhaps bi-weekly or monthly, depending on what the normal cadence of reviewing and prioritizing work is.

• Annotate status changes. If an issue was closed but there is no comment whatsoever on why it was closed, it will raise more questions than it answers. When updating the status of issues and in particular when closing them, add a comment on what changed and why the status changed. Use to your convenience the feature present in most issue tracking software (e.g. GitLab and GitHub) that they automatically close issues when a commit with a closing note lands on the mainline git branch, and those status updates are automatically annotated with a link to the change, including date and author.

• No news is bad news. In the context of status information and progress communication, people tend to view a lack of communication as a sign of a lack of progress. If something is blocked and there is no progress, a quick message noting “no progress” is better than silence and letting people stare at issues with no updates. Eventually people will start to worry and will reach out for updates, so skipping status updates to save effort might not actually save any effort.

• Remember the purpose. At the end of the day, progress is more important than communication. If a task is small and you work on it alone, issues/status reporting may be omitted completely. If you find yourself spending more effort on communication about an issue than working on the issue itself, something is wrong with the overall process and you should review it.

Be honest. Engineering is about building stuff that works, and a lot of effort goes into making sure stuff actually works. That requires a constant loop of testing and feedback. If you spot something that is off, report it. Don’t waste time on sugar-coating when communicating from a professional engineer to another, but just report the problem you’ve spotted and do it exactly.

Be grateful for all the feedback you get. Thank the person for taking the time to give feedback. The more you get, the better. Never scold another engineer for giving you feedback you don’t like. Stay professional and just read/listen and try to understand it.

Not all feedback is valid, and as a professional you choose what feedback you act on. If you don’t understand the feedback, ask for clarification. Engineering is, and should be, full of debate about what is the wrong or right solution, so that the chances of landing on the actually right solution are maximized. Be professional and don’t take those discussions emotionally. Engage in them, and make sure all data points are analyzed. Intentionally challenge your own bias and preconceptions, and try to reach the best conclusion you can.

Delete stuff that is outdated or obsolete. Remove weasel words and duplicate texts. If you come across some documentation that is clearly outdated but might be needed for archival purposes, then add a banner to it stating that it is no longer up-to-date and kept only for archival purposes (in particular in a wiki where anybody can contribute to maintaining the contents). False information may cause more harm than no information.

Avoid link rot. If a document is moved from one place to another, delete the old version and replace it with a link to the new one.

Always shorten and simplify code when you can do so without sacrificing other desirable qualities: correctness, readability, maintainability, and consistency with the rest of the codebase. For example, if you are writing or maintaining 10 unit tests which are 20 lines of code each, but differ only in a couple of inputs and outputs, then combine them into a single parameterized test.

Layering and abstractions are valuable techniques for writing reusable and correct code. However, too much abstraction can make code difficult to understand and reason about. An integrated development environment (IDE) can be a useful tool for quickly navigating a code base. However, if you have to use an IDE to follow the code’s logic, it is a telltale sign that the code is way too abstracted.

The primary goal for code is to be easy to understand and follow. Optimize for readability and maintainability. Do not optimize for speed or build layers of abstractions upfront. Instead, do such things only later on if and when you really need to.

Principles to make code easy to understand:

• Start with good naming: Files, functions, variables should all be named in such a way that one can guess from the name what they do or contain. Don’t be afraid of changing a name if you realize that something else would describe it much better. Functionality and contents evolve – so should the naming.

• Use the project’s coding conventions. A suboptimal but consistent convention is better than mixing multiple conventions in one code base. Use correct indentation, line length, white space, etc. to enhance the readability of your code. Keep the flow easy to follow.

• Add inline comments in places that require some additional explanation of what the code does or why the code was written in a particular way. Good inline comments prevent the code from being deleted or refactored by somebody else, or by yourself a year later when you can no longer recall the details.

• Longer or higher-level documentation should go into README files. The convention of using README files in code repositories is a great application of coupling code and documentation. A README is easy to discover in a code repository, and very likely to get updated by the same commits that update the code itself. If a code repository completely lacks a README file, the code is most likely not intended to be long-lived and should be avoided.

Last but not least - remember that good software engineers write both great code as well as brilliant git commit messages. The more senior the engineer, the more they value both the code and the description of it, and the whole feedback cycle that eventually leads to making the world a better place – or at least one piece of software better. Practice this skill even if it requires a bit of extra effort initially.

Most of the world’s population are not native English speakers – including myself. Yet, as English is the lingua franca of the software profession world, we all need to put effort into becoming more fluent in English. The best way to become fluent is to simply force yourself to read, write, listen, and speak English, and to do it in a way where you intentionally try to improve your English. Personally, I, for example, watch YouTube videos in well-articulated English, such as the British comedy panel show QI, or listen to podcasts attentively, trying to pick up new expressions that help articulate ideas and opinions more accurately, and in general to expand my vocabulary.

From an organizational point of view, it doesn’t matter how many amazingly smart engineers you hire if there are not proper mechanisms in place to ensure that the right amount of relevant information flows between the experts. Efficient communication is vital also for growing junior engineers quickly. You don’t want to have any engineers wasting time trying to solve problems that have already been solved. The whole organization will be vastly more productive if everyone is able to find information, easily and quickly at the time they need it.

Achieving it is not hard nor expensive – it just requires setting a couple of ground rules, reflecting on their meaning, and executing them consistently. Managers play a vital role in achieving a strong communication culture by leading with their example and showing that good communication is valued across the organization.

People usually associate advanced software engineering with gray-bearded experts with vast knowledge of how computers and things like compiler internals work. However, having technical knowledge is just the base requirement to work in the field. In my experience, the greatest minds in the field are not just experts in knowledge, but also extremely efficient communicators, particularly in writing.

Following these 8 principles can help you maximize your efficiency in written communication:

In a workplace setting, the ability to summarize something in three sentences is far more valuable than the ability to write fancy-looking research papers. Forget school assignments with minimum lengths – in reality, you need to put in effort to specifically keep it short.

Unless you are a professional novel writer building up an arc of drama, your readers are most likely not captivated enough to read all of your text fully. Therefore, you need to put forward your main suggestion or request as early in the text as possible. In ideal cases, the main message you want to convey is already in the title.

If you are an expert, people will value your opinions. But it is always much more compelling if they are delivered with supporting facts, numbers, timelines, and references. Ideally, there is a reliable source to refer to or an indicator or statistic to look at, but a couple of anecdotal case examples also work well as both evidence and as a concrete story to showcase cause and effect.

A number is always more expressive than an adjective. Instead of a vague “expensive”, just write “500 USD/h” if the price is known. Don’t state that something is “significantly faster” as it does not actually mean anything. Saying, for example, “travel time decreased to 5 hours (down 30% from 7 hours)” paints a much more accurate picture.

Instead of a verbal reference like “read the report for more,” make a service to readers and include a direct URL they can simply click. When describing a system or a problem, include the documentation link or issue tracker identifier.

After stating facts, ask yourself “so what?”. Cater to readers who are not fully familiar with the domain by being explicit on why something matters and what it means, in as concrete terms as possible.

Before sending out a text to a large group of recipients, ask one person to read it first. If your main message does not get across, iterate on your text until at least one person understands it in the way you intended. If the text has great significance, you might continue to ask for feedback from two or three more people, but remember that everyone has an opinion, and there is no guarantee that getting more opinions will converge on one opinion. Asking multiple people for opinions is not directly bad, but perhaps wasteful, as it quickly leads to diminishing returns.

When it comes to your own text, the most important opinion is your own. A good way to figure out what you really want and value is to write a text, put it away, and then return to it one or more days later and ask yourself if you still really agree with it.

Last but not least, remember it is the responsibility of the broadcaster to make sure the message was received. Don’t assume people received and saw your message, or that they read it, or that they understood what they read. You need to put in the effort into preparing your message and following up on how it was received.

Writing well is also a way to show respect for the reader’s intellect and time. Think about it this way: If you send a message to a hundred people and expect them to spend 6 minutes each reading it, you are spending 600 minutes (10 hours) of organizational time. If you spend 15 minutes extra to polish your message so it can be read and understood in just 2 minutes, you save the organization almost a full workday (600 minutes vs. 15 + 200 minutes equals 385 minutes or 6 ½ hours less).

It does not matter how good your idea is if the text describing it is bad. If you practice writing well, people near and far will become more receptive to your ideas.

Are you a seasoned professional who masters written communication? What are your tips? Please comment below!

What is the single most common action you repeat over and over when using your computer? Let me guess – opening a new tab in the browser. Here are my tips for opening, switching and closing tabs everyone should know.

This one most people know: press Ctrl+T to open a new tab. But did you know that you don’t always need to type an URL or start a web search? You can also jump directly to the content you wanted to view by using custom address bar shortcuts.

All popular browsers support defining customer keywords so that what you type in the address bar can take you where you are going even faster. In Chrome (and Chromium) you can customize what shortcuts can be used in the address bar by opening Settings > Search engine > Manage search engines and site search. Below are my favorite custom searches.

Want to quickly ask an AI for something? Just configure @p in your shortcuts to query https://www.perplexity.ai/search?q=%s&focus=internet and you are never further than a couple key strokes away from asking Perplexity AI.

I used to always google everything I wanted to know, but nowadays I find myself doing it less and less. Instead, I type @p <question> in the address bar, press enter and immediately get the answer from Perplexity along with links to the information sources. No more wasting time on skimming through unrelevant search result pages!

Yes, any man page can be accessed easily by running on the command-line man followed by the command name. But reading man pages in a browser window with nice fonts and in a separate window next to the command-line window is much more ergonomic and an easier way to craft commands. For this use case, I have configured the shortcut @man that jumps to the latest version of the man page in Debian using URL https://dyn.manpages.debian.org/jump?suite=unstable&language=en&q=%s.

Oddly enough, Chrome does not have any built-in shortcut to Google Drive. Adding a shortcut with this url will achieve it https://drive.google.com/drive/u/0/search?q=%s.

If you are like me and have dozens of tabs open simultaneously, learn to use keyboard shortcut Ctrl+Tab. This will jump to the next tab. Pressing Ctrl+Shift+Tab will do the same in reverse direction. By pressing Ctrl+1 you can instantly jump to the first tab, and with Ctrl+2 to the second tab and so forth. This is handy in particular if your first tabs are pinned and always have your e-mail or calendar and you need to open them frequently.

Too many tabs to cycle through them? No worries, you can always press Ctrl+Shift+A to open a dialog where you can search the tab based on the website title.

In Chrome you can also type @tabs in the address bar to search your open tabs, or @history to search tabs and pages you recently closed.

To close a tab, press Ctrl+W. Oops – if you accidentally close a tab, re-open it quickly with Ctrl+Shift+T. You can even press it multiple times to re-open several old tabs in the reverse order of closing them, basically undo for tab closing.

What if you have too many tabs open and you need to close the browser window? In Chrome, there is a handy shortcut Ctrl+Shift+D that will bookmark all open tabs in a folder name you choose. Then you can safely close the window knowing that you will always find them in that specific folder in your bookmarks.

Action Shortcut Open a new tab Ctrl+T Close a tab Ctrl+W Undo closing a tab Ctrl+Shift+T Jump one tab to the right Ctrl+Tab Jump one tab to the left Ctrl+Shift+Tab Open first tab, open nth tab Ctrl+1, Ctrl+2, … Search tab by website title Ctrl+Shift+A Bookmark all open tabs (e.g. before closing window) Ctrl+Shift+D Open link in a new tab without leaving current web page Ctrl+click

Knowing how to use a web browser efficiently should be considered a basic life skill in modern society. The above keyboard shortcuts work in all browsers and are as universal as Ctrl+C and Ctrl+V.

What is your additional browser productivity tip? Share it in a comment below.

Git is by far the most popular software version control system today, and every software developer surely knows the basics of how to make a git commit. Given the popularity, it is surprising how many people don’t actually know the advanced commands. Mastering them might help you unlock a new level of productivity. Let’s dive in!

When working with large git repositories, it is not always desirable to clone the full repository as it would take too long to download. Instead you can execute the clone for example like this:

Copy

git clone --branch 11.5 --shallow-since=3m https://github.com/MariaDB/server.git mariadb-server

git clone --branch 11.5 --shallow-since=3m https://github.com/MariaDB/server.git mariadb-server

This will make a clone that only tracks branch 11.5 and no other branches. Additionally this uses the shallow clone feature to fetch commit history only for the past 3 months instead of the entire history (which in this example otherwise would be 20+ years). You could also specify 3w or 1y to fetch three weeks or one year. After the initial clone, you can use git remote set-branches --add origin 10.11 to start tracking an additional branch, which will be downloaded on git fetch.

If you already have a git repository, and all you want to do is fetch one single branch from a remote repository one-off, without adding it as a new remote, you can run:

Copy

$ git fetch https://github.com/robinnewhouse/mariadb-server.git ninja-build-cracklib From https://github.com/robinnewhouse/mariadb-server * branch ninja-build-cracklib -> FETCH_HEAD $ git merge FETCH_HEAD Updating 112eb14f..c649d78a Fast-forward plugin/cracklib_password_check/CMakeLists.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) $ git show commit c649d78a8163413598b83f5717d3ef3ad9938960 (HEAD -> 11.5) Author: Robin

$ git fetch https://github.com/robinnewhouse/mariadb-server.git ninja-build-cracklib From https://github.com/robinnewhouse/mariadb-server * branch ninja-build-cracklib -> FETCH_HEAD $ git merge FETCH_HEAD Updating 112eb14f..c649d78a Fast-forward plugin/cracklib_password_check/CMakeLists.txt | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) $ git show commit c649d78a8163413598b83f5717d3ef3ad9938960 (HEAD -> 11.5) Author: Robin

This is a very fast and small download, which will not persist as a remote. It creates a temporary git reference called FETCH_HEAD, which you can then use to inspect the branch history by running git show FETCH_HEAD, or you can merge it or cherry-pick or whatever.

If you want to download the bare minimum, you can even operate on individual commits as raw patch files. A typical example would be to download a GitHub Pull Request as a patch file and apply it locally:

Copy

$ curl -LO https://patch-diff.githubusercontent.com/raw/MariaDB/server/pull/3026.patch $ git am 3026.patch Applying: Fix ninja build for cracklib_password_check $ git show commit a9c44bc204735574f2724020842373b53864e131 (HEAD -> 11.5) Author: Robin

$ curl -LO https://patch-diff.githubusercontent.com/raw/MariaDB/server/pull/3026.patch $ git am 3026.patch Applying: Fix ninja build for cracklib_password_check $ git show commit a9c44bc204735574f2724020842373b53864e131 (HEAD -> 11.5) Author: Robin

The same works for GitLab Merge Requests as well – just add .patch at the end of the MR url. This will apply both the code change inside the patch, as well as honor the author field, and use the patch description as the commit subject line and message body. However, when running git am, the committer name, email and date will be that of the user applying the patch, and thus the SHA-sum of the commit ID will not be identical.

The latest git has a new experimental command sparse-checkout that allows one to checkout only a subset of files, but I won’t recommend it as this post is purely about best practices and tips I myself find frequently useful to know.

The best command to view the history of a single file is:

Copy

git log --oneline --follow path/to/filename.ext

git log --oneline --follow path/to/filename.ext

The extra --follow makes git traverse the history longer to find if the same contents existed previously with a different file name, thus showing file contents across file renames. Using --oneline provides a nice short list of just the git subject lines. To view the full git commit messages as well as the actual changes, use this:

Copy

git log --patch --follow path/to/filename.ext

git log --patch --follow path/to/filename.ext

If there is a specific change you are looking for, search it with git log --patch -S <keyword>.

To view the project history in general, having this alias is handy:

Copy

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'"

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'"

The output shows all references, multiple branches in parallel and it is nicely colorized. If the project has a lot of messy merges, sticking to one branch main be more readable:

Copy

git log --oneline --no-merges --first-parent

git log --oneline --no-merges --first-parent

However, an even better option is to use gitk --all &. This standard git graphical user interface allows you to browse the history, search for changes with a specific string, jump to a specific commit to quickly inspect it and what preceded it, open a graphical git blame in a new window, etc. The --all instructs gitk to show all branches and references, and the ampersand backgrounds the process so that your command-line prompt is freed to run other commands. If your workflow is based on working over SSH on a remote server, simply connect with ssh -X remote.server.example to have X11 forwarding enabled (only works on Linux). Then on the SSH command-line just run gitk --all & and a window should pop up.

Copy

laptop$ ssh -X remote-server.example.com server$ echo $DISPLAY :0 (X11 forwarding is enabled and xauth running) server$ cd path/to/git/repo server$ gitk --all &

laptop$ ssh -X remote-server.example.com server$ echo $DISPLAY :0 (X11 forwarding is enabled and xauth running) server$ cd path/to/git/repo server$ gitk --all &

A typical need is also to compare the files and changes across multiple commits or branches using git diff. The nicer graphical option to it is to run git difftool --dir-diff branch1..branch2 which will open the diff program of your choice. Personally I have opted to always use Meld with git config diff.tool meld.

When making a git commit, doing it graphically with git citool helps to clearly see what the changes have been done, and to select the files and even the exact lines to be committed with the click of a mouse. The tool also offers built-in spell-checking, and the text box is sized just right to visually enforce keeping line lengths within limits. Since development involves committing and amending commits all the time, I recommend having these aliases:

Copy

alias g-commit='git citool &' alias g-amend='git citool --amend &'

alias g-commit='git citool &' alias g-amend='git citool --amend &'

Personally, I practically never commit by simply running git commit. If I commit from the command line at all, it is usually due to the need to do something special, such as change the author with:

Copy

git commit --amend --no-edit --author "Otto Kekäläinen <otto@debian.org>"

git commit --amend --no-edit --author "Otto Kekäläinen <otto@debian.org>"

Another case where a command-line commit fits my workflow well is during final testing before a code submission when I find a flaw on the branch I am working on. In these cases, I fix the code, and quickly issue:

Copy

git commit -a --fixup a1b2c3 git rebase -i --autosquash main

git commit -a --fixup a1b2c3 git rebase -i --autosquash main

This will commit the change, mark it as a fix for commit a1b2c3, and then open the interactive rebase view with the fixup commit automatically placed at the right location, resulting in a quick turnaround to make the branch flawless and ready for submission.

Occasionally a git commit needs to be applied to multiple branches. For example, after making a bugfix with the id a1b2c3 on the main branch, you might want to backport it to release branches 11.4 and 11.3 with:

Copy

git cherry-pick -x a1b2c3

git cherry-pick -x a1b2c3

The extra -x will make git amend the commit message with a reference to the commit id it originated from. In this case, it would state: (cherry picked from commit a1b2c3). This helps people reading the commit messages later to track down when and where the commit was first made.

When doing merges, the most effective way to handle conflicts is by using Meld to graphically compare and resolve merges:

Copy

$ git merge branch2 Auto-merging VERSION CONFLICT (content): Merge conflict in SOMEFILE Automatic merge failed; fix conflicts and then commit the result. $ git mergetool Merging: SOMEFILE Normal merge conflict for 'SOMEFILE': {local}: modified file {remote}: modified file $ git commit -a [branch1 e4952e06] Merge branch 'branch2' into branch1

$ git merge branch2 Auto-merging VERSION CONFLICT (content): Merge conflict in SOMEFILE Automatic merge failed; fix conflicts and then commit the result. $ git mergetool Merging: SOMEFILE Normal merge conflict for 'SOMEFILE': {local}: modified file {remote}: modified file $ git commit -a [branch1 e4952e06] Merge branch 'branch2' into branch1

One more thing to remember is that if a merge or rebase fails, remember to run git merge --abort or git rebase --abort to stop it and get back to the normal state. Another typical need is to discard all temporary changes and get back to a clean state ready to do new commits. For that I recommend this alias:

Copy

alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

This will reset all modified files to their pristine state from the last commit, as well as delete all files that are not in version control but may be present in the project directory.

The most important tip for working with git repositories is to remember at the start of every coding session to always run git remote update. This will fetch all remotes and make sure you have all the latest git commits made since the last time you worked with the repository.

Copy

$ git remote update Fetching origin Fetching upstream remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 3 (delta 2), pack-reused 0 Unpacking objects: 100% (3/3), 445 bytes | 55.00 KiB/s, done. From https://github.com/eradman/entr e2a6ab7..6fa963e master -> upstream/master

$ git remote update Fetching origin Fetching upstream remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 2), reused 3 (delta 2), pack-reused 0 Unpacking objects: 100% (3/3), 445 bytes | 55.00 KiB/s, done. From https://github.com/eradman/entr e2a6ab7..6fa963e master -> upstream/master

In the example above, you can see that there isn’t just the origin, but also a second remote called upstream. Most people use git in a centralized model, meaning that there is one central main repository on e.g. GitHub or GitLab, and each developer in the project pushes and pulls that central repository. However, git was designed from the start to be a distributed system that can sync with multiple remotes. To understand how to control this one needs to learn the concept of tracking branches and learn the options of the git remote command.

Consider this example that has two remotes, origin and upstream, and the origin remote has 3 push urls:

Copy

$ git remote -v origin git@salsa.debian.org:debian/entr.git (fetch) origin git@salsa.debian.org:debian/entr.git (push) origin git@gitlab.com:ottok/entr.git (push) origin git@github.com:ottok/entr.git (push) upstream https://github.com/eradman/entr (fetch) upstream https://github.com/eradman/entr (push) $ cat .git/config [remote "origin"] url = git@salsa.debian.org:debian/entr.git fetch = +refs/heads/*:refs/remotes/origin/* pushurl = git@salsa.debian.org:debian/entr.git pushurl = git@gitlab.com:ottok/entr.git pushurl = git@github.com:ottok/entr.git [remote "upstream"] url = https://github.com/eradman/entr fetch = +refs/heads/*:refs/remotes/upstream/* [branch "debian/latest"] remote = origin merge = refs/heads/debian/latest [branch "master"] remote = upstream merge = refs/heads/master

$ git remote -v origin git@salsa.debian.org:debian/entr.git (fetch) origin git@salsa.debian.org:debian/entr.git (push) origin git@gitlab.com:ottok/entr.git (push) origin git@github.com:ottok/entr.git (push) upstream https://github.com/eradman/entr (fetch) upstream https://github.com/eradman/entr (push) $ cat .git/config [remote "origin"] url = git@salsa.debian.org:debian/entr.git fetch = +refs/heads/*:refs/remotes/origin/* pushurl = git@salsa.debian.org:debian/entr.git pushurl = git@gitlab.com:ottok/entr.git pushurl = git@github.com:ottok/entr.git [remote "upstream"] url = https://github.com/eradman/entr fetch = +refs/heads/*:refs/remotes/upstream/* [branch "debian/latest"] remote = origin merge = refs/heads/debian/latest [branch "master"] remote = upstream merge = refs/heads/master

In this repository, the branch master is configured to track the remote upstream. Thus, if I am in the branch master and run git pull it will fetch master from the upstream repository. I can then checkout the debian/latest branch, merge on upstream and do other changes. Eventually, when I am done and issue git push, the changes on branch debian/latest will go to remote origin automatically. The origin has 3 pushurls, which means that the updated debian/latest will end up on both the Debian server as well as GitHub and GitLab.

The commands to set this up was:

Copy

git clone git@salsa.debian.org:debian/entr.git cd entr git remote set-url --add --push origin git@salsa.debian.org:otto/entr.git git remote set-url --add --push origin git@gitlab.com:ottok/entr.git git remote set-url --add --push origin git@github.com:ottok/entr.git git remote add upstream https://github.com/eradman/entr

git clone git@salsa.debian.org:debian/entr.git cd entr git remote set-url --add --push origin git@salsa.debian.org:otto/entr.git git remote set-url --add --push origin git@gitlab.com:ottok/entr.git git remote set-url --add --push origin git@github.com:ottok/entr.git git remote add upstream https://github.com/eradman/entr

As most developers use feature and bug branches to make changes and submit them for review, a lot of old and unnecessary branches will start to pollute the git history over time. Therefore it is good to check from time to time what branches have been merged with git branch --merged and delete them.

If a branch is deleted remotely as a result of somebody else doing cleanup, you can make git automatically delete those branches for you locally as well with git config --local fetch.prune true. You can run this one-off as well with git fetch --prune --verbose --dry-run.

When working with multiple remotes, it might at times be hard to reason what will happen on a git pull or git push command. To see what tags and branches are updated and how without actually updating them run:

Copy

git fetch --verbose --dry-run git push --verbose --dry-run git push --tags --verbose --dry-run

git fetch --verbose --dry-run git push --verbose --dry-run git push --tags --verbose --dry-run

Using the --dry-run option is particularly important when running push or pull with --prune or --prune-tags to see which branches or tags would be deleted locally or on the remote.

Another maintenance task to occasionally spend time on is to run this command to make git delete all unreachable objects and to pack the ones that should be kept forever:

Copy

git prune --verbose --progress; git repack -ad; git gc --aggressive; git prune-packed

git prune --verbose --progress; git repack -ad; git gc --aggressive; git prune-packed

To do this for every git repository on your computer, you can run:

Copy

find ~ -name .git -type d | while read D do echo "===> $D: " (cd "$D"; git prune --verbose --progress; nice -n 15 git repack -ad; nice -n 15 git gc --aggressive; git prune-packed) done

find ~ -name .git -type d | while read D do echo "===> $D: " (cd "$D"; git prune --verbose --progress; nice -n 15 git repack -ad; nice -n 15 git gc --aggressive; git prune-packed) done

It is not practical to constantly run git status (or git status --ignored) or to press F5 in a gitk window to be aware of the git repository status. A much handier solution is to have the git status integrated in the command-line prompt. My favorite is Liquid Prompt, which shows the branch name, and a nice green if everything is committed and clean, and red if there are uncommitted changes, and yellow if changes are not pushed.

Another additional tool I recommend is the Fuzzy Finder fzf. It has many uses in the command-line environment, and for git this alias is handy for changing branches:

Copy

alias g-checkout="git checkout "$(git branch --sort=-committerdate --no-merged | fzf)""

alias g-checkout="git checkout "$(git branch --sort=-committerdate --no-merged | fzf)""

This will list all local branches with the recent ones topmost, and present the list in an interactive form using fzf so you can select the branch either using arrow keys, or typing a part of the branch name.

While git has its own alias system, I prefer to have everything in plain Bash aliases defined in my .bashrc. Many of these are explained in this post, but there are a couple extra as well. I leave it up to the reader to study the git man page to learn for example what git push --force-with-lease does.

Copy

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'" alias g-history='gitk --all &' alias g-checkout='git checkout $(git branch --sort=-committerdate --no-merged | fzf)' alias g-commit='git citool &' alias g-amend='git citool --amend &' alias g-rebase='git rebase --interactive --autosquash' alias g-pull='git pull --verbose --rebase' alias g-pushf='git push --verbose --force-with-lease' alias g-status='git status --ignored' alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

alias g-log="git log --graph --format='format:%C(yellow)%h%C(reset) %s %C(magenta)%cr%C(reset)%C(auto)%d%C(reset)'" alias g-history='gitk --all &' alias g-checkout='git checkout $(git branch --sort=-committerdate --no-merged | fzf)' alias g-commit='git citool &' alias g-amend='git citool --amend &' alias g-rebase='git rebase --interactive --autosquash' alias g-pull='git pull --verbose --rebase' alias g-pushf='git push --verbose --force-with-lease' alias g-status='git status --ignored' alias g-clean='git clean -fdx && git reset --hard && git submodule foreach --recursive git clean -fdx && git submodule foreach --recursive git reset --hard'

As a programmer, it is not enough to know programming languages and how to write code well. You also need to understand the software lifecycle and change management. Understanding git deeply helps you better prepare for situations where potentially hundreds of people collaborate on the same code base for years and years.

To learn more about git concepts, I recommend reading the entire Pro Git book. The original version is over a decade old, but the online version keeps getting regular updates by people contributing to it in the open source spirit. As an example, I wrote a new section last year about automatically signing git commits. Skimming through the git reference documentation (online version of man pages) is also a great way to become aware of what capabilities git offers.

What is your favorite command-line git trick or favorite tool? Comment below.

When people learn programming they – for completely obvious and natural reasons – initially focus on learning the syntax of programming languages and libraries. However, these are just tools. The essence of software engineering is about automating thought, applying algorithmic thinking and anticipation of the known and unknown. The code might be succinct, but the reasoning behind it can be extensive, and it needs to show in the communication around the code. The more senior a programmer is, the more their success depends on their communication skills.

One could even claim that software development teams thrive or fall based on how quick and efficient the feedback cycle about the code is and how well the team shares information while researching and solving problems.

At the core of code-related communication is git commit messages. When a team member shares a new code change for others to review, the speed and accuracy of the reviewers depends heavily on how well the intent of the change was described and motivated.

In addition to reviews, a great git commit also has permanent utility as part of the code base. If it later turns out the commit had a bug, whoever is trying to fix it will have a much easier time reading in the commit what the change was supposed to do, and consequently understanding where it fell short, and will thus be able to rewrite the same change in the correct way. This leads to bugs being fixed much more quickly and with less effort – and most often the person doing the fix is a future you who no longer remembers what the present you were thinking while making that commit, and the future you just have to stare at the commit message and contents until it makes sense.

If you haven’t already, first read How to make a good git commit. In addition to knowing what a good end result looks like, it might be useful to learn the typical mistakes developers make and to know explicitly what to not do.

Repeating and extending the recommendations from the Git Pro book (authored by, among others, GitHub co-founder Scott Chacon):

• Never exceed 70 characters in the git title, and preferably keep it under 50

• Use imperative format, not past tense: instead of “Fixed” or “Added”, write “Fix” or “Add”

• Write at least one sentence in the git message body

• Separate the message body by one empty line from the subject line

• A title is like an e-mail subject line – no dot at the end

• The body should use full sentences that end with a dot

• Wrap the message body around 72 characters, lines should not overly long

• Don’t use diary-like language to explain what you did, but rather what the change does: if the description starts with “In this commit I..” or “I checked..”, there surely is a simpler way to express it clearly and universally

• If you have multiple commit messages that have exactly the same title, you are surely also doing something wrong, as the change itself for sure isn’t identical

• Writing “Update ” is never a good description of the change, as it just states the obvious – instead, describe the intent of what the change tries to achieve

• Don’t use AI to write your commit messages – AI can only see what changed in the files, it cannot possibly know why you made the change, which is exactly the essence of the git commit message

In this example the improved version has a more descriptive title that captures both what the change was, as well why it was made. The git commit message is restructured to explain the same thing it less repetition.

Initial Copy

Fix security vulnerabilities found by FlawFinder Fixing security issues found by FlawFinder. Project code base contains a number of old-style unsafe C function usage. In this commit we are replacing string functions: `strcpy()` `strcat()` and `sprint()` with the safe new and/or custom functions such as `snprintf()` `safe_strcpy()` and `safe_strcat()` The FlawFinder log before changes: $ cat flawfinder-all-vulnerabilities.html | grep "Hits =" Hits = 14955 After the change: $ cat flawfinder-all-vulnerabilities.html | grep "Hits =" Hits = 14668 The number of fixes - 287

Fix security vulnerabilities found by FlawFinder Fixing security issues found by FlawFinder. Project code base contains a number of old-style unsafe C function usage. In this commit we are replacing string functions: `strcpy()` `strcat()` and `sprint()` with the safe new and/or custom functions such as `snprintf()` `safe_strcpy()` and `safe_strcat()` The FlawFinder log before changes: $ cat flawfinder-all-vulnerabilities.html | grep "Hits =" Hits = 14955 After the change: $ cat flawfinder-all-vulnerabilities.html | grep "Hits =" Hits = 14668 The number of fixes - 287

Improved Copy

Fix insecure use of strcpy, strcat and sprintf in Connect Old style C functions `strcpy()`, `strcat()` and `sprintf()` are vulnerable to security issues due to lacking memory boundary checks. Replace these in the Connect storage engine with safe new and/or custom functions such as `snprintf()` `safe_strcpy()` and `safe_strcat()`. With this change, FlawFinder static security analyzer reports 287 fewer findings.

Fix insecure use of strcpy, strcat and sprintf in Connect Old style C functions `strcpy()`, `strcat()` and `sprintf()` are vulnerable to security issues due to lacking memory boundary checks. Replace these in the Connect storage engine with safe new and/or custom functions such as `snprintf()` `safe_strcpy()` and `safe_strcat()`. With this change, FlawFinder static security analyzer reports 287 fewer findings.

In this example the title was changed to used imperative format, and to more precisely tell what was changed in order to distinguish the commit from other similar ones that fix cppcheck failures. The message body explains what the change does instead of what “we” did, and shows the error message verbatim so anybody seraching for the error message will find this text.

Initial Copy

Fixing cppcheck failure We have an error while running CI in gitlab "There is an unknown macro here somewhere. Configuration is required. If DBUG_EXECUTE_IF is a macro then please configure it." Add a workaround - change problematic string with false alarm before cppcheck run then revert it back.

Fixing cppcheck failure We have an error while running CI in gitlab "There is an unknown macro here somewhere. Configuration is required. If DBUG_EXECUTE_IF is a macro then please configure it." Add a workaround - change problematic string with false alarm before cppcheck run then revert it back.

Improved Copy

Add certain DBUG_EXECUTE_IF cases to cppcheck allowlist Cppcheck failed on error: There is an unknown macro here somewhere. Configuration is required. If DBUG_EXECUTE_IF is a macro then please configure it. This is a false positive and safe to ignore. Extend filtering to exclude it from cppcheck results.

Add certain DBUG_EXECUTE_IF cases to cppcheck allowlist Cppcheck failed on error: There is an unknown macro here somewhere. Configuration is required. If DBUG_EXECUTE_IF is a macro then please configure it. This is a false positive and safe to ignore. Extend filtering to exclude it from cppcheck results.

Here again the title was made more specific about which fix this is about exactly to distinguish it from other similar fixes, and since both the error message was known and the previous commit that caused it was identified, they are included in the git message to clearly justify the change, as well as make debugging similar things much easier in the future.

Initial Copy

Releaser fix releaser failed due to missing manifest path. Adding return statement to the function

Releaser fix releaser failed due to missing manifest path. Adding return statement to the function

Improved Copy

Add missing return to find_manifest_file() The refactor in f9f6d299 split load_manifest() into two functions by simply copy-pasting the lines. This omitted that the new function needs to have a `return` added, otherwise it might return `None`. This fixes the releaser failure about: line 214, in get_engine_name_from_manifest_file with open(manifest_filename, "r") as manifest: TypeError: expected str, bytes or os.PathLike object, not NoneType

Add missing return to find_manifest_file() The refactor in f9f6d299 split load_manifest() into two functions by simply copy-pasting the lines. This omitted that the new function needs to have a `return` added, otherwise it might return `None`. This fixes the releaser failure about: line 214, in get_engine_name_from_manifest_file with open(manifest_filename, "r") as manifest: TypeError: expected str, bytes or os.PathLike object, not NoneType

This example shows make the title more specific by spelling out what component exactly is extended and with which variables. In the descriptiong use imperative ‘Add’ instead of ‘Adding’, and restructure the text to clearly say what is being done, followed by why it is useful, and include explanation about backwards compatibility to further champion that the change is safe to do. Also fix line breaks and add space between paragraphs.

Initial Copy

Add TLS version to auth plugin available variables The authentication audit plugins currently do not have access to the TLS version used. Adding this variable to list of available variables for audit plugin. Logging the TLS version can be useful for traceability and to help identify suspicious or malformed connections attempting to use unsupported TLS versions. This can be used to detect and block malicious connection attempts.

Add TLS version to auth plugin available variables The authentication audit plugins currently do not have access to the TLS version used. Adding this variable to list of available variables for audit plugin. Logging the TLS version can be useful for traceability and to help identify suspicious or malformed connections attempting to use unsupported TLS versions. This can be used to detect and block malicious connection attempts.

Improved Copy

Extend audit plugin to include tls_version and tls_version_length variables Add tls_version and tls_version_length variables to the audit plugin so they can be logged. This is useful to help identify suspicious or malformed connections attempting to use unsupported TLS versions. A log with this information will allow to detect and block more malicious connection attempts. Users with 'server_audit_events' empty will have these two new variables automatically visible in their logs, but if users don't want them, they can always configure what fields to include by listing the fields in 'server_audit_events'.

Extend audit plugin to include tls_version and tls_version_length variables Add tls_version and tls_version_length variables to the audit plugin so they can be logged. This is useful to help identify suspicious or malformed connections attempting to use unsupported TLS versions. A log with this information will allow to detect and block more malicious connection attempts. Users with 'server_audit_events' empty will have these two new variables automatically visible in their logs, but if users don't want them, they can always configure what fields to include by listing the fields in 'server_audit_events'.

In this example the title can be simplified to summarize the change. The change was initially tested, but later came permanent. There was no change in the contents of the change however, and thus in this case it was better to not describe the lifecycle of the commit in the git commit messge title, but instead keep that information elsewhere among the developers, or perhaps inferred from the fact that the commit was initially on a development branch and only later applied on mainline.

Also avoid writing in “I have ..”, and instead use imperative format that describes what the change is and most importantly extend it to explain why the change was made. Who renamed what file or changed what line of code is always visible in the git commit anyway. The focus of the git commit should be on communicating the intent of the change, as why something was changed isn’t always obvious, yet incredibly important in order to assess if the change is correct or not.

Initial Copy

Switch to `new-archives` branch to test the new archives layout I have renamed `archives.html` to `.archives.html` to disable overriding. I have modified archives.md to output JSON.

Switch to `new-archives` branch to test the new archives layout I have renamed `archives.html` to `.archives.html` to disable overriding. I have modified archives.md to output JSON.

Improved Copy

Use new layout for archives page Update theme to version with new archives page and disable the old archives page. Ensure new archive page is also published as JSON so that the interactive search can use the JSON file as backend.

Use new layout for archives page Update theme to version with new archives page and disable the old archives page. Ensure new archive page is also published as JSON so that the interactive search can use the JSON file as backend.

Writing a good git commit message while preparing a code submission is much easier if you follow this process:

1. Start writing code changes

2. Save intermediate changes with git commit -am WIP

3. Test, polish, iterate

4. Run git citool --amend to polish the git commit message when the code change is final and you can see the whole change and thus are able to easily explain what you did and why

5. Rebase on latest main branch, e.g. git fetch origin main; git rebase -i origin/main

6. Push to code review

If you find yourself doing frequent rebases and amends, congratulations! It means that you have mastered the craft of preparing great code submissions.

One extra benefit of having a great commit message is that you don’t have to rewrite anything when submitting the code for review. Every single code review system I have ever used will automatically use the git commit title and message as the review title and message (at least if the review is a single commit review).

If you are proud of your work and like doing things well, you will follow these guidelines by nature. However, some lazy ass might say that while they agree with the principles, they don’t have time to follow them. To that I always respond that humans tend to get really good at the things we practice. If you do the wrong thing over and over, you become an expert at doing things incorrectly. Is that really what you want?

Doing things correctly from the onset will steer you away from situations where you are knowingly doing lousy quality just to “save time”. Practicing doing something well will ultimately lead you to become a person who does that thing well, effortlessly.

Have you run into situations where you find it challenging to write a good git commit title and message? Share your example in the comments below and I will try to help you formulate the text and capture the essence of the change in a concise title and description.

The stock market is a powerful globally distributed forecasting system. Last Friday it was forecasting a rosy future as the MSCI World index got to a new all-time high. But it does not make sense.

There is no single entity controlling stock prices. They represent the collective best guess of the whole economic world on what the value of each company should be. Historically the market has been pretty accurate. For example, when the news broke about COVID-19 spreading, stocks plummeted in anticipation of a global crisis. And indeed, a pandemic followed that forced a large part of the economy to a standstill, and major government intervention was necessary to avoid total mayhem. As another example, in the first month of the Russian invasion of Ukraine in February-March 2022, the stock price of Cheniere Energy went up almost 50%, as the market predicted that this American liquified natural gas producer would hugely benefit from Russian competitors being blockaded. Again, the market was right – the by the end of 2022, Cheniere’s profits had doubled.

The MSCI World index has grown about 30% since the low point in the fall of 2022. The U.S. focused S&P index is even more extreme, having grown almost 40% since the low point in fall of 2022, ending last Friday at 4840, slightly above the previous closing high record 4797 set on Jan 3, 2022. Investors participating in the market are collectively forecast that the U.S. economy has recovered and things will be fine. There is just one problem. Everyone is wrong.

The S&P will soon probably surpass 5000 points, but this party can’t last for long. Reuters has a great summary of economic events overlaid on the S&P 500 in past two years:

The situation seems very contradictory. Increased inflation should affect the purchasing power of consumers and companies negatively. While salary increases help consumers survive, and higher sales prices help companies offset increased costs, undeniably the overall effect is still negative. The U.S Consumer Price Index growth has slowed down a bit since the peak in 2022, but prices are still growing faster than in all of 2010’s. Due to inflation, the U.S. Federal Reserve increased in July 2023 its interest rate to 5.5%, and still keeps it there. High interest rates discourages companies from taking loans and making investments, so for the time being the economy should slow down, not accelerate.

Following the insolvency of the Silicon Valley Bank on March 10th, 2023, the U.S. Federal Reserve announced it will guarantee the 200+ billion dollars the bank had lost. Soon it repeated it to a few more banks. Essentially the Fed printed 300+ billion dollars in March 2023. When you overlay the S&P 500 and the U.S Federal Reserve balance sheet the correlation seems pretty strong:

The upward trend in November and December might be related to U.S. lawmakers passing so-called “appropriation bills” to allow the U.S. government to continue overspending. Similar bills and resolutions are likely to repeat on March 1st and 8th, 2024. All of this makes me think the stock prices are out of touch with the underlying companies growth, and mostly just a function of how much money is being pumped into the U.S. economy by the government and the Fed.

This is not exactly anything new. Taking on more national debt seems to be the U.S. policy, no matter which president or party is on power:

What is new is the scale it has reached now. The sum is rapidly approaching 34 trillion U.S. dollars. That is 34 000 000 000 000 (12 zeros!). The sum is insanely large. Think about the most expensive single item one could buy: a USS Gerald R. Ford class nuclear powered aircraft carrier has a price tag of 12 000 000 000 (billion) dollars. The U.S. has 11 aircraft carriers and all other countries combined have 6–9, so in total under 20 in the world. With 34 trillion one could buy 2833 aircraft carriers á 12 billion.

The U.S. budget projections for 2024-2034 is a grim read. The deficit in 2024 is 1.6 trillion USD, and is forecasted to fluctuate between 1.6 and 2.6 trillion for the next 10 years. The interest payment for the national debt is 870 billion in 2024, and 951 billion in 2025 – the first year when U.S. debt interest payments exceeds the U.S. national defence budget!

In 2024 the U.S. government is going to issue 1.6 trillion USD worth of completely new bonds. In addition, the U.S. government also needs to issue several trillion worth of new bonds in order to pay old lenders for bonds that mature in 2024. Somebody needs to buy this 5–10 trillion of U.S. bonds, and it is very unlikely the normal money markets would be able to absorb this supply. Thus, the Federal reserve bank system will need to step in and “print money” to buy up the bonds that don’t sell on the free market.

The USA is not alone in having a lot of debt. The national debt in Greece is around 150% of their GDP, and Japan has been at over 200% for a decade. However, the U.S. has a huge GDP, so having 120% debt-to-GDP ratio is exceptional in absolute numbers. Consider that when comparing national debt per capita, Japan and the U.S. both hover around 100k USD (depending on source), so actually the U.S. situation is already very worrying.

What is extra concerning, is that for the past decades the GDP growth in the U.S. seem to correlate with their government overspending and taking on debt:

This begs the question: how much of the economic growth in the U.S. has actually been based on printing money to begin with? How much has there been improvement in real productivity? Due to how GDP is accounted, public debt automatically increases it, so some correlation is expected, but a very strong correlation as seen in the graph above could indicate that the real economy is not growing, only debt is. The stock market is supposed to grow over time as a function of the GDP and overall productivity growing, but is that really happening now?

One interesting observation when inspecting the S&P 500 and the MSCI world index constituents, is that they are both dominated by just seven large U.S. tech firms. However, their effect on U.S. GDP is limited, as their corporate profits are accounted in Ireland due to tax planning. This has led to the GDP of Ireland to skyrocket in the past 10 years from 50k to 100k USD per capita. If the U.S. government runs out of money, it will be much more likely to crack down on tax evasion practices and force the large tech companies to pay more corporate tax in the U.S. Seems like a major disruption factor to me. Yet their stock prices keep growing.

The extremely high valuation is justified only if these seven companies are about to make a huge productivity leap that lifts the entire U.S. economy, including creating 1.7 trillion in additional tax revenue to close the deficit gap. Seems the stock market is predicting exactly that. Sure, artificial intelligence will help increase productivity everywhere, but I find it very hard to believe that AI productivity gains would accumulate on these tech companies to such a degree that their astronomical valuations would be justified. To me it seems that the market is now just plain wrong.

According to the World Federation of Exchanges (WFE) exchanges there are over 58 000 stock listed companies in the world. There should be plenty of options for the stock market to bet on. Yet the market predicts that the U.S. stocks in particular would be future winners.

My prediction for 2024 is a large correction across the U.S. stock market. If it does not happen already during the spring, surely latest after the U.S. presidential elections in November when keeping appearances ends and policy makers are ready to face the reality.

What is your read of the situation? Post in the comments below.

First we make our habits, and then they make us

― John Dryden, poet and literary critic

Are you perhaps planning to make a new year’s resolution to run a marathon? Or are you committing to a new sales quota at work for the year 2024? If you haven’t achieved the goal by July 2024, will you be unhappy? Will you give yourself some slack and simply postpone the goal? And if you exceed it, will you immediately make a new goal? Would you feel happy about it?

Most people are really bad at setting and reaching goals. First of all, people seldom reach their goals. In environments where people consistently achieve their goals, it is usually because the goal was set too low to begin with. In rare cases where goals are set high and then met, or even far exceeded, the fact that the goal existed or the level of the goal doesn’t seem to make any difference.

As I see it, thinking about goals isn’t healthy. I find it a much better approach to think about habits. Habits focus on the immediate action instead of a long-term outcome. People tend to be much more efficient and also happier after adopting habits. For example, if you want to run a marathon, start by adopting a habit of running 10 km twice a week instead of the sporadic running you would do otherwise. If you work in sales – or if you manage a team of sellers – don’t think about the annual sales quota but instead focus on adopting a habit of reaching out to, for example, 10 new customers daily.

Habits are better than goals because:

• Habits are adopted immediately. You know in a matter of days if you are keeping the habit or not, while goals make it too easy to postpone doing the work.

• Habits are easier to plan. Selecting a goal that is concrete and measurable is hard. Even if a good metric exists, it is hard to choose a level low enough to be reachable and thus motivating to work towards, yet high enough to be admirable and to fuel people to make extra effort. For a habit, you pick something concrete, and just do it consistently.

• Habits are easy to track and measure. It is immediately evident if somebody is not sticking to a habit. There is no denial, just effort to get back into the habit. Goals are vague, and not meeting a goal is evident only when it is too late to do anything about it.

• Habits can be sustained for years and years. Goals often compel acts of heroism, which are not sustainable in the long run. As Bruce Lee once said, “long-term consistency trumps short-term intensity.”

• Habits make people happier. If you forget or are unable to do something, just get back into the habit the following day. If you fail to meet a goal, you just feel miserable and have no immediate way to rectify the failure.

The smaller the step is, the easier it is to take. Microhabits are a concept of improving something, one tiny step at a time.

An example of a microhabit could be the act of drinking a glass of water every morning. This is something I have been doing myself for almost 3 years now: Every morning, I go straight out of bed to the kitchen and drink a glass of water. Only after that do I allow myself to brush my teeth and do other things that are part of the usual morning routine.

This is a great microhabit, as after a full night’s sleep, one is bound to have a dry mouth and a little bit of dehydration, for which water is the best cure.

Starting the day with a glass of water gives a nice head start into a larger habit of drinking water frequently. People typically don’t drink enough plain water, even though it is cheap (practically free), and ensuring good water balance takes away all symptoms of dehydration, such as headache and tiredness. For some reason, modern humans tend to prefer other drinks, such as beer or coffee, which can actually make dehydration worse. Water is essential for all life on earth, and we need to stress the importance of it. Humans can go without food and fast for 2–3 weeks, but without water, we perish in only 3–5 days. Water has zero calories, and the sensation of fullness from drinking benefits weight control.

If you lapse from a habit one day, don’t worry, just get back into the habit the next day. Think of ways to remind yourself of and strengthen the habit. The above example of drinking a glass of water every morning can be strengthened by keeping a water purifier filled with water on the kitchen countertop. Every time you walk past the kitchen, it will remind you of the habit. This setup also minimizes the effort required to perform the microhabit, as you always have the water easily at hand. Personally, I also think water at room temperature feels healthier than drinking ice cold water directly from the tap.

The fact that you decided to do this, and you keep doing it after weeks and months will help strengthen your willpower. Studies on neuroplasticity show that it will physically help to rewire the circuits in your brain into making a decision, and sticking to it. Once you master this one microhabit, you’ll find it easier to adopt other habits that take more effort to get into.

As explained in a review article in Frontiers in Neuroscience of 63 meta-analyses, this occurs because of an increased ability to exert effortful control in our brains. Simply put, as with all biological organisms, our brain has evolved to save energy and only think in situations where that extra energy consumption is necessary. Most of the time, our brains run on autopilot, which means not only that the existing pathways keep getting reinforced, but also that the pathways of the pathway control system itself stay weak, as new pathways don’t need to be formed. Hence, when we take on a new habit and exert the mental effort to repeat the routine with conscious intent over and over, it leads to creation and reinforcement of pathways related to the habit. The day a habit has grown strong enough to become part of our brain’s autopilot program, the system that decides what goes into the autopilot and what goes out is also at its strongest.

After successfully adopting your first intentional microhabit, the next step could be to either expand the first microhabit to make it more complicated, or to adopt a second microhabit.

1. An example of expanding the first one could be taking a multivitamin pill with the water, or adding salt to the water to follow the Huberman routine.

2. An example of an easy second microhabit would be to drink a cup of warm water every evening before going to bed. Since it is just water, you can drink it even after brushing your teeth. It feels somewhat filling, and helps me avoid drinking or snacking on anything else, which is likely unhealthy. The warmness of the water feels good, and I think it is plausible that it has similar health benefits to drinking tea, just without any flavor or sweeteners.

Get into good habits. Keep them for the rest of your life. Change them only if you find a new, better habit to replace the old. If you slip from your habit one day, forgive yourself, and get back into the habit the next day. By keeping the good habits, you will eventually reach your goal – and eventually also far exceed the goal. Make habits, not goals.

In software development, the code review process stands as a crucial checkpoint for ensuring code quality, fostering collaboration, and promoting knowledge sharing among team members. Despite the importance, many engineers lack a clear mental map of how effective reviews work. This is my attempt to help code reviews and reviewers improve.

Focus first on the git commit title and message. Do you understand the description? Does the proposed change make sense based on only the description? Is the idea clear? Ask yourself if you can come up with reasons why the change should not be made, or if there are obvious better alternatives.

If the description does not make sense to you, immediately share that as your initial feedback. If the description about the intent is incomprehensible, don’t waste time reviewing code implementation details. It could be that the submission was still just a draft, and the only (and immediate) feedback should be a request for the submitter clarify their intent.

The next thing to check is if the code matches the description. For example, if the change proposes to fix a bug, it should not include extra code that adds a new feature. Sometimes when you review the code, 90% of the changes make sense and do exactly what the git commit message stated, but there might be some code lines changed that you don’t understand. Ask the submitter about these. Maybe they were included by mistake, or for an unobvious reason. This should be addressed by some comments inline next to the code, or by amending the change description.

Assuming the description and changes align and make sense, shift your focus to the implementation details and assessing how the problem was solved. As a reviewer, you should ask yourself: Are there alternative ways to achieve the same outcome? Would these alternatives bring clear advantages, or is the current proposal the most sensible one?

If the code changes look mostly good and you don’t anticipate a complete overhaul in the next revision, proceed to giving feedback about the details in the code. Look for potential logical errors, deviations from coding conventions, anti-patterns, spelling mistakes, and even hint at any security concerns or performance issues you may anticipate. Scrutinize every aspect that catches your eye and suggest improvements wherever possible.

If you find yourself commenting on nearly every code line changed, consider aborting the review and asking the submitter to address the first set of comments before completing a comprehensive review.

Beware that some people might take every comment literally. When pointing out things that can be improved, remember to include phrases like “in general it is better to..” or “if you have time do..” to soften the general feedback, and clarify what actually needs to be addressed before you can give your approval for the change, and which things you merely suggest as quality improvements without insisting on them. Also beware of submitters who address comments mechanically. Instead of giving your own proposed better implementation as a code snippet, simply ask the submitter to “please rethink X so it always does Y without doing Z” which forces the submitter to write the better version themselves, and in the process train their brain muscle. Who knows, after thinking about the thing you pointed out, the submitter might even write a better fix than you initially thought of.

Some may be hesitant to nitpick in code reviews, as nitpicking is not something civilized people tend to do in normal daily interactions. However, nitpicking is an integral part of code reviews and software engineering in general. Computers read every dot and comma literally, and these need to be exactly correct. The whole idea of a code review is to maximize quality and minimize the room for error, in both the short term and the long term. Thus, in the context of a code review, please nitpick as much as you can.

The amount of feedback should be maximal, but the bar for approval does not need to be maximally high. The requirements for minimum quality should be suitable for the skill level of the developer team. Yes, over time the quality bar can be raised as the development team collectively learns about best practices, but if the quality bar is too high from the onset, it will discourage developers from making new submissions. And the iterative cycle that allows for learning to happen won’t take place.

Managing quality is hard, but ensuring acceptable quality is central to the code review process. As a general rule, however, if you are in doubt about what is reasonable to require, err on the side of demanding higher quality rather than settling for something you feel is substandard. In the long term, you are more likely to be happy about having made such a decision.

Keep in mind that the main reason people settle for low quality is that doing things at the highest quality level requires more effort, and some people are lazy. However, if people are constantly pushed to do things at high quality, doing things correctly will soon become effortless.

To support the review process by humans, all software projects should have a CI in place that runs automatic tests on every code change and helps detect at least all easily detectable regressions. Code submissions with failing CI typically don’t attract many reviewers, so submitters should review and fix all CI issues themselves as soon as possible.

On a related note, all high quality code submissions that change the program’s behavior should also update and extend the associated test code. If, for example, the software project has unit tests, and a submitter sends new code without new unit tests, the reviewer should request the submitter to write the missing tests. This feedback can even be automatic – a good CI system could detect a drop in test coverage.

In the review process, it is important that both the submitter and the reviewer communicate clearly and precisely.

From the reviewer’s side, this means that the reviewer should be clear on what things are required from the submitter and what things are just nice to have. If a reviewer thinks the whole approach is bad, they should be frank and reject the submission from the get go.

From the submitter’s side, the initial submission should be as complete as possible. If code is submitted for review (e.g. Pull Request or Merge Request) but it does not have the final contents, the submitter should be explicit about this and prefix their git commits or the review title with WIP: to signify that it is work-in-progress.

In my experience, the typical reason for code submissions and reviews getting stalled is simply unclear communication. The code submitted might be fully correct, but the plain English part is lacking, leading to miscommunication. Typically, reviewers also postpone diving into submissions that they don’t understand at first glance, as having to do detective work to figure out an unclear code submission can feel overwhelming. Therefore, the best way to ensure submissions are reviewed timeously is to communicate clearly.

Needless to say, both submitter and reviewer should know how to properly use the review tool. For example, both GitHub and GitLab have “Start review” and “Finish review” features that allow the reviewer to write multiple comments without spamming the submitter with multiple emails. Pressing “Finish review” will trigger the submission of one single email with all comments. Most systems also have buttons for requesting review and re-requesting review that the submitter should use to communicate clearly when the review feedback is addressed.

When a submitter re-submits code for review, they should always use git push --force. The reviewer is always looking at the submission with the question “Is this ready to be merged?” in their mind, and the submission they look at should be the final polished code. There should be no WIP commits or multiple intermediate git commits – the reviewer is not interested in how the submitter ended up with the correct end result. Only the final version matters.

Reviews require time. The submitter should be extra diligent not to waste the reviewer’s time. Likewise, the reviewer should try to review as quickly as possible, or at least share their initial impression as a quick short comment.

If reviewers are short on time, they should prioritize re-reviewing submissions they’ve already once given feedback on. People who already have context about something should continue on it, as it is most efficient for them. Other people on the team are more likely to review code submissions that have no reviews at all.

Likewise, submitters should prioritize responding to review feedback and updating and polishing their existing submissions as soon as possible. A reviewer is much more likely to re-review and approve a submission they have already looked at a couple days earlier than submissions that have been lingering for weeks or months.

A code submission and the review process serves as a process for the new submitter to become initiated with the software project. Reviewers should keep this in mind, and not rush to grab the code, but put in a little extra effort to guide the submitter on code base and on the quality bar. Reviewers might feel frustrated at times, but that is not an excuse for bad behaviour.

Sometimes, the reviewer might be tempted to fix the issues in the submission themselves instead of giving the original submitter a chance to do the final polish. A reviewer must resist this, as this will kill the feeling of ownership of the original submitter. In the context of open source projects, grabbing somebody else’s submission and committing it yourself might also constitute a copyright violation, and it certainly does not encourage the submitter to continue making further submissions. The reviewer should at most just rebase the commit, nothing more.

If the review process takes too many rounds of back-and-forth, then as a compromise the reviewer could merge the submission as-is, and immediately follow up with their own changes on top of the commit.

In both open source and company-internal work, the achievement that one’s code got merged is very valuable, in particular if it was the first accepted contribution to a particular project from that person. Don’t rob this from code submitters; let them earn it and have their git credits and purple GitHub merge badges in their profile.

A fundamental principle in code submissions is maintaining a single code submitter. While there may be multiple people reviewing and posting comments, there should never be more than one person submitting the code (and subsequently improved revisions). Having one author ensures a clear owner, who iterates the submission and decides how to address all feedback. If using a git branch, that person is the only one who amends commits and force pushes the branch until the final version has been completed.

If the submission is a patch on a mailing list, having multiple submitters of a single email is impossible and authorship and ownership not an issue. On GitHub or GitLab, this problem might arise, as the submission is a git branch, and the branch permissions may allow multiple people to push to it. Having multiple people pushing to the same pull request, however, is plain wrong.

In some projects, git pull requests are intentionally misused by having multiple authors push git commits on them and discussing the collaboration in the “review comments” section. The correct way for a group of people to write code together in git before the code goes on the mainline branch is to have a feature branch. Both GitHub Pull Requests and GitLab Merge Requests allow the submitter to select the target branch of the submission. With a feature branch model, each collaborator submits their own individual pull requests towards the feature branch, and once the feature is complete, the feature owner carries the responsibility of getting the feature branch accepted into the mainline branch of the git project.

Sometimes it may happen that there is disagreement or laziness, and the submitter refuses to properly address all feedback. In these cases, the recipient/reviewer either dismisses the submission completely or accepts it as-is (merges on target branch), where work can continue in the form of follow-up submissions from other people to further improve it.

It may also happen that a submission has a great idea, but the implementation is bad and unacceptable. The submitter is asked to improve the implementation, but the submitter might fail to do so. In this situation, nobody else should rewrite that same submission, as it blurs the lines of authorship and ownership. Instead, the other person with a better implementation should simply open a new pull request or submit an email with their own version and in their own name. After that, it will be up to the reviewers to decide which submission gets accepted and which one declined.

Keep in mind that the reviewer does not need to be a superior developer. Anybody can be a reviewer, no matter how senior or junior a software developer they are. Reading code written by somebody else is a good way to get exposure to various coding styles and various ways that different developers solve coding problems. Reviewing code is a learning opportunity for the reviewer, and thus the review process ultimately leads to both the submitter and the reviewer writing better code in the future.

If you have an opportunity to do code reviews, do it, even if you don’t feel like an authority on a specific domain. You can still make a valuable contribution by giving feedback on some aspects, while leaving the final approval decision to a domain expert.

Remember to communicate clearly – the better the code is documented, and the better the feedback is explained, the more both submitter and reviewer learn from the experience.

If you have additional tips on best pratices for code reviews, please add a comment below!

While large organisations scale best by emphasizing asynchronous communications, in-person or video meetings also have their place. As a manager who is involved in a lot of planning and coordination work I’ve noticed that I’ve spent the majority of my working time in meetings in past years. These are my 5 tips to make meetings as efficient as possible.

While a simple ‘hey we should chat’ works for small informal meetings, having an agenda always makes the meeting much more efficient – even if there are just two participants. Put an agenda in every meeting invite you send. Explain what is the purpose of the meeting, who is attending and what should be the outcome. It does not have to be formal. Even a tiny agenda summary is enough to show you’ve put some thought into why the meeting is taking place.

When the meeting starts, show the agenda to remind everyone what the purpose of the meeting is so that attendees can help make the meeting productive. When possible, I prefer to organize meetings with Google Calendar because Meet will automatically show the invite text (the agenda) in the video call as a small popup in the left lower corner.

If you find yourself chairing a meeting unprepared, remember the acronym WHOT to improvise an agenda and meeting structure that works in most situations:

• Why are we here - Explain why the thing that is the topic of the meeting matters, and briefly the context or perhaps latest developments around it.

• How attendees relate to the topic - introduce participants if they have not met before, and even if they know each other, explain how they ended up being invited to the meeting or how they are expected to participate in the topic.

• Opinions (or orders) - Ask attendees for opinions on the topic. If the purpose of the meeting was to get input from everyone, your duty as the chair of the meeting is to make sure everyone has a chance to talk. Most meetings about non-urgent topics are about collecting and aligning views. If the meeting is about an urgent topic such as an operational issue, the O in the acronym stands for orders and the main part of the meeting is to make sure all participants know what they should execute immediately after the meeting. Even when giving out orders you should still verify that the participants understood and agree with the ask and not assume too much.

• Timeframe - Conclude the meeting with summarizing what is the timeframe of the agreed actions, or when the next event in the topic is expected to happen, or if a follow-up meeting is expected state the timeframe for it.

Don’t be afraid of moments of silence when running a meeting. In fact, you should ensure that there are some pauses so that people who need more time to formulate their thoughts have an opportunity to speak up. This is important in particular if many participants are not native speakers of the meeting language. The best ideas might not be the ones that are voiced first, but the ones that come after some pondering on the topic.

If some participants talk too much, ask them politely to give space to others. If everyone seems silent, ask open-ended questions. Maintain a welcoming environment for discussion where speakers or their opinions are not directly criticised as that might silence some participants from sharing their honest opinions. Make sure all discussion have a respectful tone and opinions are voiced in a constructive manner with focus on solutions, or when discussing problems encourage participants to put forward concrete data points instead of pure opinions. As a chair for the meeting strive to be kind, but firm.

As the organizer you are responsible for allocating enough time. If the meeting is going overtime, end it, and schedule a follow-up meeting. If you think some participants talk too much or off-topic, steer the discussion back on-topic and remind about the time remaining. Even if there is allocated time left, don’t allow room for random mumblings. Respectful use of people’s time includes cutting the meeting short if the goal was met.

To keep meetings efficient, they should not take longer than one hour. When planning the meeting and thinking about the agenda, make sure there are not too many items for one meeting. Also make sure that there are not too many participants. If the intent is to discuss something for an hour, there should be max 12 participants (which means on average 5 minutes of speak time per person). If the meeting has 20 or more participants, it is not a meeting but mostly a one-way announcement to the audience.

One large meeting with 20+ people could alternatively also be a series of meetings where the same chair talks to groups of 5 at a time, and after all the meetings sends out summary of all of them with a final conclusion.

People have a tendency to forget what was discussed or agreed after 2-3 weeks. Meetings are at risk of being wasted time unless the outcome is recorded at least in a small written summary. Writing formal meeting minutes is not an efficient use of time for everyday business meetings, but a 5 minute investment in a short summary is always worth it.

Formal meeting minutes are justified if the meeting is making decisions with ramifications in the tens or hundreds of thousands of dollars. Formal meeting minutes should record who was present, what the decision was, and why. For formal meetings it is important to agree to have some kind of approval mechanism for the minutes, and to not execute the decisions until the minutes have been approved with e.g. signatures.

If the meeting topic is very contentious, the meeting minutes should be written during the meeting and shared on-screen so participants can already during the meeting object in real-time if they see the decisions or their justifications not written down in the correctly.

The last tip is to take some time and think if a meeting really is needed? If you just want to have the opinion on one specific thing, perhaps the meeting could be replaced with a chat thread where your first message is the question followed with a bit of context. This might even work better, as people can take as much time as they want to formulate their best possible reply in writing and with exact data points and references. To ensure everybody participates, you can request everyone acknowledges the question with an 👀 emoji or a short reply. Such ‘meetings’ have the additional benefit that they don’t need a separate agenda or summary, as the chat was in recorded writing automatically.

As a replacement for a large 20+ participant meeting one could have a well written announcement email, or a video recording, since most of the participants in a large meeting will not have time to speak anyway, and they will mostly just be receiving information without participation.

If you feel that a meeting is needed for social cohesion, perhaps get the core decision done in an asynchronous chat or email discussion, and organize a breakfast or lunch separately to get the best social effect in a setting that is free from debating and just purely positive social time together.

If the meeting is a one-on-one between two people, it could be replaced with a walk in the park. That will make sure you both get some fresh air, and who knows, maybe also some fresh ideas?

The key to being productive as a programmer is to have a great code editor. I have been an avid user of Atom since 2014, and its successor Pulsar since now in 2023.

Atom the code editor was created by Nathan Sobo, who joined GitHub in 2011 specifically to build the best possible code editor in the world. He also co-led the development of Teletype for Atom, which pioneered collaborative code editing, i.e. multiple developers writing the same code files at the same time. The team also created the Electron Framework, which lives on as the user interface framework for e.g. Microsoft VS Code (created a couple years after Atom).

VS Code is actually the reason why Atom eventually died. In 2018, Microsoft acquired GitHub, the world’s largest open source hosting platform, to get premier mind share of developers, and as part of that plan, Microsoft also wanted as many developers as possible to do all their coding using an editor controlled by Microsoft. To achieve this goal, Microsoft invested hugely in VS Code, and naturally discontinued Atom after gaining control of it through the GitHub acquisition.

However, being open source software, Atom can’t fully be killed. Yes, Microsoft controls the name, trademark and website – which are all now shutdown – but the source code is MIT licensed and has resurrected as Pulsar the editor.

The key innovation in Atom (and now Pulsar) was to build the whole editor as a web application that runs locally (offline) inside a dedicated browser window (Electron). This radically lowers the barrier of entry for all the millions of web developers out there to participate in the development of the editor. This is reflected today in Pulsar’s tagline “hyper-hackable text editor”, with reference to the original meaning of the word “hackable” as users’ ability to use the tool in ways the original designer had not anticipated.

To make Pulsar even more flexible, it also has a plugin system (called packages) with over 10 000 community developed easily installable packages one can use to extend the features of the editor.

Pulsar is feature rich and very powerful already out-of-the-box, but thanks to the vast number of additional packages, a developer can easily optimize one’s workflow to be as productive as possible. If a package does not already exist for your use case, creating one yourself is moderately easy to most developers, as you only need to know web development tech (JavaScript/CSS/HTML) to do it.

Consider the screenshot below showing a simple C source code file being edited.

In this picture, you can see that there is:

• a linter that automatically highlights with yellow a specific code word the user should improve

• a yellow vertical line shows what lines in the file have been changed and are not yet committed in git

• a file browser showing the files in the project, with modified files being yellow, new files being green and files ignored by git being greyed out

• a rich status bar showing the git status, cursor location, file encoding and type, git branch name and even the GitLab CI status for the latest git commit

This is perfect for my coding workflow – it does not feel bloated but has all the features I need without excess. Check out the post about How to code 10x faster than an average programmer to see the Pulsar autosave feature being used to create the optimal code-test-repeat workflow.

For maximum productivity, most developers prefer to be able to do everything using the keyboard, without the need to lift your hand to fiddle with the mouse. Pulsar has a keyboard shortcut for almost everything. If something is missing, you can also configure more yourself, and you can change the default keyboard bindings if you have existing preferences. However, the best part is that you don’t really need to spend any time learning keyboard shortcuts - you just need to remember Ctrl+Shift+P. This opens the command palette, where you can write in a keyword, select the action and press enter, or see on screen the keyboard shortcut and use it until it has gone effortlessly into your muscle memory.

The two other keyboard shortcuts worth remembering are (same but without the shift) Ctrl+P that allows you to type a partial filename and press enter to quickly open any file in the project, and Ctrl+Shift+F to search a string anywhere in the project. The search feature is amazing, and also comes with a powerful search-and-replace feature with interactive previews on what string it would replace in which file.

I know several world class programmers, and interestingly, the commonality among them is that they all seem to use Vim as their code editor. Many people I know who think of themselves as world class programmers use Emacs.

While both of these can probably compete with Pulsar in number of extensions available and the amount of customizations and keyboard shortcuts, neither of them have a graphical user interface, and they are completely incapable of doing things like showing images or, for example, a Markdown preview (which by the way you can open in Pulsar by pressing Ctrl+Shift+M).

Also, Vim and Emacs use modes to allow users to either write the text in the file itself or to enter commands for the editor. Modes are terrible to use, and thanks to people like Larry Tesler, we haven’t had modes in any new software since the 1980’s, and nearly all of humanity is blissfully unaware of what modes even are and will never be exposed to them. Instead of using keyboard shortcuts via modes, we today have keyboard shortcuts like Ctrl+C and Ctrl+V which are easy to use.

Admittedly, Pulsar is a bit slow to start up. It takes maybe 6–8 seconds for it to fully load from scratch, but once it is open, using it feels pretty snappy. The slowness is due to being an Electron app, which is basically a web browser. Indeed, the original author of Atom/Pulsar, Nathan Sobo, started a new code editor project after Atom was shut down. The new editor Zed is extremely fast, as it is written in Rust and also because it utilizes the GPU for rendering. Zed, however, is not open source, and it is not available for Linux, so I have not even tried it. Instead, I tested another promising Rust-based blazingly fast editor, Lapce. It is truly fast and pleasant to use. However, it lacks a lot of features, so I end up having to do extra manual work.

Thus, Pulsar holds up as my favorite editor, and for me Pulsar is the fastest one when measuring how quickly I am able to deliver code changes and write complete and correct text.

If you want to replicate the Pulsar setup I have going, this is my .pulsar/config.cson:

yaml Copy

"*": autocomplete-plus: confirmCompletion: "tab always, enter when suggestion explicitly selected" autosave: enabled: true welcome: showChangeLog: false showOnStartup: false

"*": autocomplete-plus: confirmCompletion: "tab always, enter when suggestion explicitly selected" autosave: enabled: true welcome: showChangeLog: false showOnStartup: false

These are some of my favorite packages:

• Sort Lines

• Linter Spell (human languages)

• Linter Flake8 (Python)

• Linter Clang

• Linter ShellCheck

• Linter MarkDown

• Markdown language grammar

• Debian control and rules file linter

• ANSI color styles

• Nim language support

• GitLab Manager

• Better Git Blame

• Minimap

• Minimap autohider

In popularizing Linux containers, Docker brought about a new era of systems design based on these lightweight platforms, rather than heavy virtual machines. However, now that Docker is slowly declining, it’s time to learn about the next generation of Linux container tools.

When Docker officially launched in 2013, it was not the first containerization solution for Linux. For example, Linux already had LXC back in 2008 (early versions of Docker ran on top of it), and FreeBSD jails had been around since 1999. Nevertheless, Docker was the first developer-friendly and complete end-to-end solution that let us easily create, distribute, and run Linux containers.

Not only was it technically sound and convenient to use, but Docker was also a great example of a successful and well-run open source project. I experienced this personally during a couple of contributions where two people did the initial review within 24h of my Pull Request and a third person merged it in less than two weeks from the submission date. Docker developers also contributed back to Linux plenty of containerization-related improvements, started drove standardization efforts, and spun off many subcomponents (e.g., containerd, OCI, BuildKit).

Today, container-based system architectures and development workflows are extremely popular, as seen with, for instance, the rise of Kubernetes. While we are still waiting for the ‘year of the Linux desktop’ to happen, Docker did certainly make more Windows and Mac users run a virtual Linux machine on their laptops than ever before.

The company Docker Inc was, from the start, a venture-funded endeavor centered around an open core model and launched many closed-source products that drove revenue over the years. What used to be the core Docker software was renamed Moby in 2017, and that is where the open-source contributions (e.g., mine from 2015) can be found. The founder Solomon Hykes no longer works for Docker Inc, and in recent years public sentiment around Docker has suffered due to various controversies. Yet at the same time, many similar (and some perhaps better) solutions have entered the space.

To build a container, a software developer first writes a Dockerfile, which defines what Linux distribution the container is based on along with what software and configuration files and data it has. Much of the Dockerfile contents are basically shell script.

The build is done with command docker build, which executes the contents of the Dockerfile line-by-line and creates a Linux-compatible root filesystem (files under /). This is done utilizing a clever overlay filesystem, where each line in the Dockerfile amounts to one new layer. Thus, rebuilds of the container do not need to rebuild the whole filesystem, but can just execute the Dockerfile lines that changed from the previous build.

On a typical Linux system, the filesystem layers after a docker build execution can be found at /var/lib/docker/. If the container was based on Debian, one could find, for example, the apt-get binary of the image at a path like /var/lib/docker/overlay2/c1ead1[...]d04e06/diff/usr/bin/apt-get.

Additionally, some metadata is created in the process, which designates among other things the entrypoint of the container — i.e. what binary on the root filesystem to run when starting the container.

To inspect what the root filesystem of the Docker image debian:sid looks like, one could create a container and inspect the mounted merged filesystem:

Copy

$ docker container create -i -t --name demo debian:sid 2734eb[...]d18852 $ cat /var/lib/docker/image/overlay2/layerdb/mounts/2734eb[...]d18852/mount-id 2854c7[...]9dfe25 $ find /var/lib/docker/overlay2/2854c7[...]9dfe25 | grep apt-get /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/share/man/man8/apt-get.8.gz /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/share/man/pt/man8/apt-get.8.gz /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/bin/apt-get

$ docker container create -i -t --name demo debian:sid 2734eb[...]d18852 $ cat /var/lib/docker/image/overlay2/layerdb/mounts/2734eb[...]d18852/mount-id 2854c7[...]9dfe25 $ find /var/lib/docker/overlay2/2854c7[...]9dfe25 | grep apt-get /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/share/man/man8/apt-get.8.gz /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/share/man/pt/man8/apt-get.8.gz /var/lib/docker/overlay2/2854c7[...]9dfe25/merged/usr/bin/apt-get

The command docker export makes it easy to get the root filesystem into, for example, a tar package.

Copy

$ docker export demo > debian-sid.tar $ tar xvf debian-sid.tar .dockerenv bin boot/ dev/ dev/console dev/pts/ dev/shm/ etc/ etc/.pwd.lock etc/alternatives/ etc/alternatives/README etc/alternatives/awk ... var/spool/mail var/tmp/ $ find . | grep apt-get ./usr/share/man/man8/apt-get.8.gz ./usr/bin/apt-get

$ docker export demo > debian-sid.tar $ tar xvf debian-sid.tar .dockerenv bin boot/ dev/ dev/console dev/pts/ dev/shm/ etc/ etc/.pwd.lock etc/alternatives/ etc/alternatives/README etc/alternatives/awk ... var/spool/mail var/tmp/ $ find . | grep apt-get ./usr/share/man/man8/apt-get.8.gz ./usr/bin/apt-get

In theory, anything could create this root filesystem, and likewise anything starting could run a binary inside it — even the classic chroot. If you edit the files and want to get them back into Docker to run as a container, docker import makes it easy.

To export a full container image with both the root filesystem and the metadata, the docker buildx command offers some output format options, such as the Open Container Initiative standard format or the Docker native image format. To import a full container image with metadata, refer to the docker load command.

In the above example, a container was created, but not started. To start a container, one can try running:

Copy

$ docker run -it debian:sid bash root@c9a8e6c222ae:/#

$ docker run -it debian:sid bash root@c9a8e6c222ae:/#

From a user experience point of view, you are basically dropped into a Bash shell in a Debian Sid container. Under the hood, the docker command-line tool sends an HTTP request to the dockerd daemon running on the local system, which in turn asks containerd to run the container, which in turn starts runc directly or (due to backwards compatibility reasons) a containerd-runc-shim. Inside this one, you can find the actual running Bash binary:

Copy

$ ps fax | grep -C container 1122 /usr/bin/containerd 1660 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 55409 /usr/bin/containerd-shim-runc-v2 -namespace moby -id c9a8e[..]0847e -address /run/containerd/containerd.sock 55428 \_ bash

$ ps fax | grep -C container 1122 /usr/bin/containerd 1660 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock 55409 /usr/bin/containerd-shim-runc-v2 -namespace moby -id c9a8e[..]0847e -address /run/containerd/containerd.sock 55428 \_ bash

Anyway, if you’re fine with slightly less automation and having more of a “hands-on” experience, read the man page for runc and try running the container directly with it.

The Linux Foundation has a nice architecture schema to illustrate the various components and alternatives in the stack that originally evolved from Docker:

The runc is the OCI reference implementation of their runtime specification. Popular alternatives to runc include: crun (implemented in C to be faster and use less memory than runc, which is in Go) and CRI-O (smaller and faster, with just enough features to be perfect for Kubernetes).

There are also container runtimes such as Kata and ZeroVM based on the idea of running each container inside a minimal virtual machine, which achieve better isolation between the containers compared to running them directly on the same host. This design aims to hit a “sweet spot” between the optimized performance of lightweight containers and the security of traditional full virtual machines.

Missing from that diagram above is the current major competitor, the Red Hat-sponsored Podman, which offers a complete replacement to the whole Docker stack.

The command-line tool podman is designed to be a drop-in-replacement for docker, so one can run the earlier command examples by just changing the first word: podman build .., podman container create ..., podman export .. and so forth. Even podman volume prune --force && podman system prune --force does exactly the same as the Docker equivalent — which is nice, as I tend to run that frequently to clean away containers and free disk space when I’m not actively using them.

To start a container one can run (for example):

Copy

$ podman run -it debian:sid bash root@312cbccb5938:/#

$ podman run -it debian:sid bash root@312cbccb5938:/#

When a container started like this is running, you would see in the process list something along the lines of:

Copy

87524 \_ podman 99902 \_ /usr/libexec/podman/conmon --api-version 1 -c 312cbc[...]93a0e1 -u 312cbc[...]93a0e1 -r /usr/bin/crun -b /home/otto/.local/share/containers/storage/overlay-containers/312cbc[...]93a0e1/userdata -p /run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/pidfile -n naughty_dewdney --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/oci-log -t --conmon-pidfile /run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/otto/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 312cbc[...]93a0e1 99905 \_ bash

87524 \_ podman 99902 \_ /usr/libexec/podman/conmon --api-version 1 -c 312cbc[...]93a0e1 -u 312cbc[...]93a0e1 -r /usr/bin/crun -b /home/otto/.local/share/containers/storage/overlay-containers/312cbc[...]93a0e1/userdata -p /run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/pidfile -n naughty_dewdney --exit-dir /run/user/1001/libpod/tmp/exits --full-attach -l journald --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/oci-log -t --conmon-pidfile /run/user/1001/containers/overlay-containers/312cbc[...]93a0e1/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/otto/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1001/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1001/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 312cbc[...]93a0e1 99905 \_ bash

Unlike Docker, there is no containerd or runc at play, but instead conmon runs crun, which is the actual container runtime. Note also that the container runs with regular user permissions (no need for root) and that the default location for storing container images and other data is in ~/.local/share/containers/ in the user home directory.

While I personally prefer to work on the command-line, I need to give a shoutout to Podman for also having a nifty desktop application for those who prefer to use graphical tools:

The basic utility of Linux containers is to give system administrators a building block which behaves a bit like a virtual machine in terms of being an encapsulated unit — but without being so slow and resource hungry as actual virtual machines! Although containers typically boast a full root filesystem, the Docker philosophy was that each container should run just one process — and run it well — and crucially, not have any process managers or init systems inside the container. Many system administrators, however, do in practice run Docker containers that use, as an example, runit to ‘boot’ the container and manage server daemon processes inside them.

The Canonical-backed LXD however tailors itself specifically for this type of use case, building upon LXC. After installing LXD and running lxd init to configure it, you can run full containerized operating systems with:

Copy

$ lxc launch images:debian/sid demo Creating demo Starting dem1 $ lxc exec demo -- bash root@demo:~#

$ lxc launch images:debian/sid demo Creating demo Starting dem1 $ lxc exec demo -- bash root@demo:~#

The host process list will show something along the lines of:

Copy

root 105632 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid root 105466 /bin/sh /snap/lxd/24061/commands/daemon.start root 105645 \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd lxd 105975 \_ dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --dhcp-rapid-commit --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.199.145.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.199.145.2,10.199.145.254,1h --listen-address=fd42:3147:bafe:37e5::1 --enable-ra --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd --interface-name _gateway.lxd,lxdbr0 -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd ... root 107867 \_ /snap/lxd/current/bin/lxd forkexec demo /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/demo/lxc.conf 0 0 0 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root LANG=C.UTF-8 TERM=xterm-256color -- cmd bash 1000000 107870 \_ bash root 106209 [lxc monitor] /var/snap/lxd/common/lxd/containers demo 1000000 106221 \_ /sbin/init 1000000 106372 \_ /lib/systemd/systemd-journald 1000000 106401 \_ /lib/systemd/systemd-udevd 1000997 106420 \_ /lib/systemd/systemd-resolved 1000100 106431 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 1000000 106433 \_ /lib/systemd/systemd-logind 1000000 106436 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 linux 1000998 106446 \_ /lib/systemd/systemd-networkd

root 105632 lxcfs /var/snap/lxd/common/var/lib/lxcfs -p /var/snap/lxd/common/lxcfs.pid root 105466 /bin/sh /snap/lxd/24061/commands/daemon.start root 105645 \_ lxd --logfile /var/snap/lxd/common/lxd/logs/lxd.log --group lxd lxd 105975 \_ dnsmasq --keep-in-foreground --strict-order --bind-interfaces --except-interface=lo --pid-file= --no-ping --interface=lxdbr0 --dhcp-rapid-commit --quiet-dhcp --quiet-dhcp6 --quiet-ra --listen-address=10.199.145.1 --dhcp-no-override --dhcp-authoritative --dhcp-leasefile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.leases --dhcp-hostsfile=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.hosts --dhcp-range 10.199.145.2,10.199.145.254,1h --listen-address=fd42:3147:bafe:37e5::1 --enable-ra --dhcp-range ::,constructor:lxdbr0,ra-stateless,ra-names -s lxd --interface-name _gateway.lxd,lxdbr0 -S /lxd/ --conf-file=/var/snap/lxd/common/lxd/networks/lxdbr0/dnsmasq.raw -u lxd -g lxd ... root 107867 \_ /snap/lxd/current/bin/lxd forkexec demo /var/snap/lxd/common/lxd/containers /var/snap/lxd/common/lxd/logs/demo/lxc.conf 0 0 0 -- env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin HOME=/root USER=root LANG=C.UTF-8 TERM=xterm-256color -- cmd bash 1000000 107870 \_ bash root 106209 [lxc monitor] /var/snap/lxd/common/lxd/containers demo 1000000 106221 \_ /sbin/init 1000000 106372 \_ /lib/systemd/systemd-journald 1000000 106401 \_ /lib/systemd/systemd-udevd 1000997 106420 \_ /lib/systemd/systemd-resolved 1000100 106431 \_ /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog-only 1000000 106433 \_ /lib/systemd/systemd-logind 1000000 106436 \_ /sbin/agetty -o -p -- \u --noclear --keep-baud - 115200,38400,9600 linux 1000998 106446 \_ /lib/systemd/systemd-networkd

Notice how the daemon runs as root (and interacting with lxd/lxc requires root permissions). However, thanks to UID mapping, the root user inside the container is not a root user as found on the host system. This is one of the key design differences — and why LXD is considered more secure than Docker.

The downloaded root filesystems are stored at /var/snap/lxd/common/lxd/images/ while the filesystems of running containers can be found at /var/snap/lxd/common/lxd/storage-pools/default/containers/ as long as the LXD storage is directory-based (as opposed to a LVM or OpenZFS pool).

The examples above all have snap in their path, as there is no native Ubuntu package for LXD…but it forces users to install a Snap even when running apt install lxd.

As lxd controls the whole system, the command for managing individual containers is lxc:

Copy

$ lxc list +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ | demo | RUNNING | 10.199.145.6 (eth0) | fd42:3147:bafe:37e5:216:3eff:fe01:8da8 (eth0) | CONTAINER | 0 | +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ $ lxc delete demo --force

$ lxc list +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ | demo | RUNNING | 10.199.145.6 (eth0) | fd42:3147:bafe:37e5:216:3eff:fe01:8da8 (eth0) | CONTAINER | 0 | +------+---------+---------------------+-----------------------------------------------+-----------+-----------+ $ lxc delete demo --force

Ergo, the process of creating LXC compatible container images is fairly simple. One can use any container builder to create the root filesystem (the LXC docs recommend using deboostrap directly), and the basic metadata yaml file is so brief, it can be written manually. These are then imported to LXC with lxc image import metadata.tar.gz rootfs.tar.gz --alias demo.

The whole LXD stack ships with integrated tooling—even offering metal-as-a-service capabilities (MAAS) — so it goes way beyond what the Docker stack has.

To fully grasp how containers actually work, you should read the Linux kernel documentation on namespaces and permission control via capabilities. Keeping an eye on the progress of the Open Container Initiative will keep you right on top of the latest developments, and considering OCI compatibility in your infrastructure will enable you to migrate between Docker, Podman, and LXD easily.

Choosing the right container technology to use depends on where you intend to ship your containers. For developers targeting Kubernetes compatible production environments, Podman probably makes the most sense at the moment. Or, if your infrastructure consists of a lot of virtualized Ubuntu hosts and you want to have more flexibility, LXD is probably a good choice.

Podman is certainly gaining a lot of popularity according to Google Trends. Docker will, however, continue to have the largest mindshare among average developers for years to come. For now, my recommendation is for all systems administrators and software architects to try and understand how these tools you rely on actually work — by getting your hands dirty with them. Choose the solutions you understand best, and keep an eye on the horizon for what’s coming next!

The perfect home office setup achieves two things: it helps you stay focused for extended periods, allowing you to be in “the flow” and it prioritizes ergonomic design to ensure that long hours at the computer don’t compromise your health.

I’ve spent a lot of time and thought creating my personal setup so that it that optimizes home office efficiency and inspiration. Here’s a glimpse into my workspace:

• At the center of my workspace is an extra wide, curved monitor. It sits on an elevated shelf, at eye level. This positioning keeps my neck straight, reducing tension in my shoulders.

• The heart of my setup is an adjustable electric desk. I can set it to the perfect height, so arms maintain a comfortable 90-degree angle. This eliminates slouching, saving my lower back from unnecessary stress.

• With my desk is a comfortable chair, adjustable for the most optimal positions. The design ensures my feet and waist maintain the right angles. Plus, its wheels offer the flexibility to slide it away when I prefer standing.

• At my fingertips is a wireless mouse and an ergonomic split keyboard. This design allows my wrists to rest, and my elbows to angle outward at a comfy 45 degrees, reducing wrist pain or carpal tunnel syndrome.

• I keep my workspace feeling fresh and at a cool, consistent temp with an air conditioner. Additionally, a higher ceiling ensures ample airflow, while my laptop sits on a cooling fan.

• Positioned above the keyboard and behind the monitor is a strategically placed light source. This minimizes eye strain and can also be used to cast a flattering light on me during video calls. Speaking of which…

• My external high-resolution camera equipped with a microphone and supplemental lighting, so I’m ready to take video calls. It’s easy to adjust, so I can look my best on screen.

As you can see in the pictures, my office doesn’t have windows. A distant view can provide a restful break for the eyes, so having windows would be good, but natural light isn’t always optimal. A room illuminated solely by engineered light ensures I always optimal conditions even into the wee hours, as I tend to be a night owl. To avoid disrupting your circadian rhythm remember to use display settings that avoid decrease emitted blue light in the evenings automatically.

Ensuring cables are arranged efficiently, and accessibly, guarantees not just an aesthetically pleasing office but a functional one, too. Hidden from view (not pictured) is a metallic shelf beneath the desk with an extension cord boasting multiple electrical sockets and USB ports. This solitary extension powers the entire desk setup, with just a singular cord discreetly extending to the wall. Its design is flexible, moving in harmony as I adjust the desk’s height. A one-button power-off feature on the extension cord is particularly handy when I’m about to leave for a trip and want to quickly shut everything off.

My monitor isn’t just for visuals—it’s also a hub. Equipped with USB ports, it connects my keyboard, mouse, and monitor light. Docking my laptop is a breeze. A single USB-C cable handles both charging and connectivity to all peripheral devices (monitor, keyboard and mouse).

Integrated within my desk are two USB ports. One port for a USB-A to multi-device converter which charges essentials like my phone and headset. Despite having numerous USB-powered gadgets like my laptop cooler, watch winder, and LED light strip on the table, I always have ports at the ready for anything else that needs a charge.

Harnessing the elusive state of ‘flow’ is the holy grail of productivity for those engaged in creative endeavors. It’s that state of immersive focus where hours can feel like mere minutes and often where our most significant accomplishments emerge. Central to this deep concentration is the environment in which I work.

The foundation of my productivity temple is its isolation — a dedicated room free from distractions. This tranquility, essential for anyone looking to dive deep into their work, forms the cornerstone of my home office. I’ve chosen background lighting that sets the right mood to help you get ideas. Every item on my desk has been chosen intentionally and there is nothing extra. A clutter-free workspace translates to a clutter-free mind. With integrated USB outlets, a comprehensive cable management system, and a single power cord, I ensure my physical space remains pristine. A concealed drawer further houses miscellaneous items, keeping them out of sight but within reach.

The acoustics of my setup is the only thing that needs more work. The rug behind my computer and the carpet on the floor help cut down on echoes, but I want to add acoustic panels to further minimize the echo. Next time I can choose the room layout, I will try to avoid a wall behind my screen and narrow walls as it creates echo too easily.

Over time, I tried integrating scheduled breaks into my routine, using apps like Workrave designed to prompt hourly pauses. However, these frequent interruptions shattered my flow state more than they helped. So, I’ve shifted my strategy. Instead of disrupting the rhythm, I ensure the ergonomics of my space support extended hours of comfortable, strain-free work.

Thanks to having my laptop docker with a single USB-C cable it is convenient to unplug and transition from a stationary work setting to a mobile one, be it another spot in my home or stepping out for a change of scenery. 

The audio experience shouldn’t be tethered either. Investing in a high-quality Bluetooth headset equipped with noise-cancelling capabilities is invaluable. While it’s an obvious tool during travel or cafe-work sessions, its utility extends to the home office too. With it, I can freely roam my living space during remote meetings, providing a much-needed break from prolonged sitting. 

The importance of physical activity during break times can’t be overstated. Interspersing work hours with moments of physical exertion, whether through pull-ups or exercises with parallettes, combats desk-bound fatigue and invigorates the mind.

One of the healthiest things we can do is keep active and go for a walk. On occasion, I’ve joined meetings with my headset on, walking outside, absorbing the content of the conversation and the rejuvenating effects of nature. Even though this may be limited to meetings that don’t require my visual attention, I find myself reenergized and coming to projects with fresh perspectives. 

I can envision a not-so-distant future where technology may also play a role in this. Imagine augmented reality goggles paired with headsets that allow you to both view and participate in presentations as you walk a woodland path. Visualize crafting emails or designing projects using intuitive hand gestures while you’re perched on a park bench. This immersive, tech-driven experience could redefine our notion of a ‘workspace,’ transforming the outdoors into a potential office.

As technology continues to evolve and offers us new ways to work, the value of a dedicated workspace—especially a home office setup—remains undeniable. While the boundaries of ‘workspaces’ may expand and become more fluid in the future, the significance of a well-optimized home office is an investment that will stand the test of time. Here’s to many more productive days and better work habits!

As a software developer, your core skill is how to improve an existing code base to make the software better iteratively, patch by patch.

To be a good software developer you need to:

• understand the concept of a code patch,

• know how do code improvements in well sized and properly documented patches, and

• skillfully use git version control software to manage patches.

A patch defines the changes to be made to the code base. It is basically a list of code lines to be added, removed or modified in a code base. Each patch always also has an author, a timestamp when it was written, a title that describes it and a longer text body that explains why this particular patch is good and applying it on the code base is beneficial.

Example:

patch Copy

Author: Otto Kekäläinen Date: June 22nd, 2022 08:08:08 Make output friendlier for users Add line break so text is readable and add a 2 second delay between messages so it does not scroll too fast. --- a/demo.c +++ b/demo.c @@ -8,7 +8,8 @@ int main() { for(;;) { - printf("Hello world!"); + printf("Hello world!\n"); + sleep(2); } return 0; }

Author: Otto Kekäläinen Date: June 22nd, 2022 08:08:08 Make output friendlier for users Add line break so text is readable and add a 2 second delay between messages so it does not scroll too fast. --- a/demo.c +++ b/demo.c @@ -8,7 +8,8 @@ int main() { for(;;) { - printf("Hello world!"); + printf("Hello world!\n"); + sleep(2); } return 0; }

You can make a patch by simply copying a file, changing something in it, and then comparing the copy to the original file using the command diff and saving the output.

Copy

$ cp demo.c demo.c.orig $ nano demo.c $ diff -u demo.c.orig demo.c > demo.patch $ cat demo.patch

$ cp demo.c demo.c.orig $ nano demo.c $ diff -u demo.c.orig demo.c > demo.patch $ cat demo.patch

patch Copy

--- demo.c.orig +++ demo.c @@ -8,7 +8,8 @@ int main() { for(;;) { - printf("Hello world!"); + printf("Hello world!\n"); + sleep(2); } return 0; }

--- demo.c.orig +++ demo.c @@ -8,7 +8,8 @@ int main() { for(;;) { - printf("Hello world!"); + printf("Hello world!\n"); + sleep(2); } return 0; }

The patch can be sent by email or uploaded somewhere. After that, anybody can download the patch, read it, and apply it to their copy of the code base using the command patch.

shell Copy

$ grep Hello demo.c printf("Hello world!"); $ curl -O https://…/demo.patch $ patch -p0 < demo.patch patching file demo.c $ grep Hello demo.c printf("Hello world!\n");

$ grep Hello demo.c printf("Hello world!"); $ curl -O https://…/demo.patch $ patch -p0 < demo.patch patching file demo.c $ grep Hello demo.c printf("Hello world!\n");

As this is not very fast nor convenient, software developers like to use git, a version control software that automates all of this. In git, we tend to talk about git commits, which basically just means a patch that has been applied on a code base.

A good git commit message typically has these characteristics (adapted from the Git Pro book):

Copy

Capitalized, short summary of what the change is More detailed explanatory text that focuses on the 'why' to motivate the change. Use present tense and imperative format (write "Fix bug", not "Fixed bug"). Wrap it to about 72 characters or so. The blank line separating the summary from the body is critical. Further paragraphs come after blank lines. - Bullet points are okay, too - Typically a hyphen or asterisk is used for the bullet, followed by a single space, with blank lines in between, but conventions vary here - Use a hanging indent

Capitalized, short summary of what the change is More detailed explanatory text that focuses on the 'why' to motivate the change. Use present tense and imperative format (write "Fix bug", not "Fixed bug"). Wrap it to about 72 characters or so. The blank line separating the summary from the body is critical. Further paragraphs come after blank lines. - Bullet points are okay, too - Typically a hyphen or asterisk is used for the bullet, followed by a single space, with blank lines in between, but conventions vary here - Use a hanging indent

Here are a couple of real-world examples in pure text form:

From MariaDB@ff1d8fa7:

Copy

Deb: Clean away Buster to Bookworm upgrade tests in Salsa-CI Upgrades from Debian 10 "Buster" directly to Debian 12 "Bookworm", skipping Debian 11 "Bullseye", fail with apt erroring on: libcrypt.so.1: cannot open shared object file This is an intentional OpenSSL transition as described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755

Deb: Clean away Buster to Bookworm upgrade tests in Salsa-CI Upgrades from Debian 10 "Buster" directly to Debian 12 "Bookworm", skipping Debian 11 "Bullseye", fail with apt erroring on: libcrypt.so.1: cannot open shared object file This is an intentional OpenSSL transition as described in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=993755

From MariaDB@2c529441:

Copy

Deb: Run wrap-and-sort -av Sort and organize the Debian packaging files. Also revert 4d03269 that was done in vain.

Deb: Run wrap-and-sort -av Sort and organize the Debian packaging files. Also revert 4d03269 that was done in vain.

In order of importance:

The first and most important thing about a good patch or a commit is that it should be a self-standing change. If a commit fixes a bug, it should not at the same time add a new feature or fix some other completely unrelated bug, otherwise it is not atomic. If you add a new feature, the same commit should ideally also add automatic tests for the feature to ensure it won’t regress, and the same commit should update the documentation to mention the feature, as it is all related and should either go or not go into the code base along with the feature itself.

If your changes are not properly scoped and self-standing, you might end up in a situation later on where somebody decides to revert or reject the commit that introduced a new feature, but miss removing the tests or documentation about it, which would not have happened if they were added in separate commits.

There is no clear rule on what is the optimal scope for a commit; it is something you will learn by experience. Sometimes it makes sense to have several separate changes in one single commit simply because of each one of them being so small. In other cases, one single logical change might span multiple commits, because it was perhaps clearer to move or rename files in one commit and then update their commits in another. This is something you will just have to learn over time as you become a more experienced software developer.

A title starts with a capital letter and has no trailing dot. Just like the subject line in an email. The title should make sense when read in a list of commits. If the title is too long, it will be cut off. A limit of 72 characters is safe to all typical places where people will be reading it, such as in a terminal window or when browsing GitHub or GitLab, but striving for under 50 characters is even better.

The text should be verbose enough for anybody reviewing the commit to understand why it was made, and to be convinced that the change is good. Every commit must have a text body, even if it is very short. This forces the author to spend a few seconds thinking about the change before committing.

Note that the commit message is about the change itself, so it should answer the question ‘why’. If you want to explain how a certain line of code works, simply use an inline comment next to the code itself. That way, the documentation is in the correct context. The git commit description should have just a tiny bit of ‘what’ and ‘how’, and mostly focus on the ‘why’.

The commit body should be wrapped at about 72 characters. Proper use of empty lines and lists that are indented with a dash or star makes the body more readable.

Remember to use imperative format. Don’t write Fixed bug or Added feature. Instead write Fix bug or Add feature. The patch hasn’t added or fixed anything at the time you wrote it. Think about it like an order you give to the code base to start following. Also, try to keep your text in the current tense and passive format. Don’t write This commit makes X but simply Make X. Don’t write I changed Y but just simply Change Y.

If your code change is related to a previous commit, mention the commit ID. In most software commits, IDs will automatically become links. If the code change is related to something that was discussed or tracked elsewhere, please include the bug tracker ID or a URL to the discussion. However, the reference alone does not remove the need to write a git commit message. You cannot expect that somebody reading your commit has time or even access to open and read all references - use them only as pointers for more information.

Viewing one of the earlier git commit message examples in gitk, GitLab and GitHub illustrates how a git commit automatically becomes a link:

The author name and timestamp are automatic if you configure your git correctly, so this should be a non-issue. If you neglect to configure git with your real name and email, you will be muddling the waters for anybody who later wants to verify something about authorship. In the worst case scenario, all your commits might be purged from the git repository due to unclear copyright.

Also keep in mind that if you commit code on behalf of somebody else, you must tell git that the author for a particular commit was somebody else and you only committed it. Read up on git commit –author for details.

Using a good tool to craft your git commits goes a long way in making the commit flawless.

My personal choice is git-citool, which is distributed together with git itself, so anybody can use it on any operating system. It does not use the native graphics of each operating system, but a cross-platform graphics library, which may look a bit ugly. It is, however, very easy and convenient to use, so I love it.

To make a new commit, simply run git citool. It starts off empty and then you can select which files you want to stage, and write the git commit message in this box, and press commit. Super easy, and it is very clear what changes you are committing.

If you are not happy with your commit and want to edit it, or to use git terminology you want to “amend”, this is possible only for the topmost git commit that does not have any child commits yet. Run git-citool --amend.

Here you can see a git commit that is really bad, so it really needs to be fixed. However, with git-citool it is easy and fast.

Remember that you don’t have to make a perfect git commit right off the bat. Do it only once you know what you actually want to write in it. While still working on the code and saving intermediate versions of it, I recommend using WIP commits where the title is simply WIP, or if you already have some commit text draft, prefix the title with WIP:.

When you are done with WIP commits, you can run git rebase -i to squash them together and write the final git commit message.

For a visual explanation see the presentation A Branch in Time (a story about revision histories) by Tekin Süleyman on how to use interactive rebase in git and why rebasing and amending commits will end up making the code quality better in the long run.

Someone who is lazy might say that while they agree with the principles, they don’t have time to follow them. To that I respond that doing things correctly from the onset actually saves time down the road.

• If your git commits are good, the job of the reviewer will be much easier. They won’t waste time on just trying to understand your change, but they will get it directly and will be able to focus their energy on actually reviewing and spotting flaws in your code. If you avoid shipping a bug, you save a lot of work not having to debug, write a fix and ship a new release.

• A great git commit is also useful even if it later turns out the commit had a bug, because whoever fixes that bug will have a much easier time reading in the commit what the change was supposed to do, and understanding where it fell short, and then making the same change in the correct way. This leads to bugs being fixed much more quickly and with less effort - and most often the person doing the fix is a future you who no longer remembers what the present you was thinking while making that commit, with the future you just having to stare at the commit until it makes sense.

• You don’t have to rewrite anything when it comes time to submit the commit for review. Every single code review system I have ever used will automatically use the commit title and message as the review title and message if the review is a single commit review.

Now you know how to make a good git commit message. If you are proud of your work and like doing things well, you will follow these guidelines. To further learn how to polish your git commit messages, see also the post on git commit messages by example.

The Linux kernel developer’s guide has also an excellent description of how to seprate changes into self-standing logical changes, and how to describe the changes.

The MariaDB server has over 2 million lines of code. Downloading, compiling (and re-compiling) and running the test suite can potentially consume a lot of time away from actually making the code changes and being productive. Knowing a few simple shortcuts can help avoid wasting time.

While the official build instructions on mariadb.org and mariadb.com/kb are useful to read, there are ways to make the build (and rebuild) significantly faster and more efficient.

TL;DR for Debian/Ubuntu users

Get the latest MariaDB 11.0 source code, install build dependencies, configure, build and run test suite to validate binaries work:

shell Copy

mkdir quick-rebuilds cd quick-rebuilds git clone --branch 11.0 --shallow-since=3m \ --recurse-submodules --shallow-submodules \ https://github.com/MariaDB/server.git mariadb-server mkdir -p ccache build data docker run --interactive --tty --rm -v ${PWD}:/quick-rebuilds \ -w /quick-rebuilds debian:sid bash echo 'deb-src http://deb.debian.org/debian sid main' \ > /etc/apt/sources.list.d/deb-src-sid.list apt update apt install -y --no-install-recommends \ devscripts equivs ccache eatmydata ninja-build clang entr moreutils mk-build-deps -r -i mariadb-server/debian/control \ -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends' export CCACHE_DIR=$PWD/ccache export CXX=${CXX:-clang++} export CC=${CC:-clang} export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++} export CC_FOR_BUILD=${CC_FOR_BUILD:-clang} export CFLAGS='-Wno-unused-command-line-argument' export CXXFLAGS='-Wno-unused-command-line-argument' cmake -S mariadb-server/ -B build/ -G Ninja --fresh \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \ -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \ -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF eatmydata cmake --build build/ ./build/mysql-test/mysql-test-run.pl --force --parallel=auto

mkdir quick-rebuilds cd quick-rebuilds git clone --branch 11.0 --shallow-since=3m \ --recurse-submodules --shallow-submodules \ https://github.com/MariaDB/server.git mariadb-server mkdir -p ccache build data docker run --interactive --tty --rm -v ${PWD}:/quick-rebuilds \ -w /quick-rebuilds debian:sid bash echo 'deb-src http://deb.debian.org/debian sid main' \ > /etc/apt/sources.list.d/deb-src-sid.list apt update apt install -y --no-install-recommends \ devscripts equivs ccache eatmydata ninja-build clang entr moreutils mk-build-deps -r -i mariadb-server/debian/control \ -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends' export CCACHE_DIR=$PWD/ccache export CXX=${CXX:-clang++} export CC=${CC:-clang} export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++} export CC_FOR_BUILD=${CC_FOR_BUILD:-clang} export CFLAGS='-Wno-unused-command-line-argument' export CXXFLAGS='-Wno-unused-command-line-argument' cmake -S mariadb-server/ -B build/ -G Ninja --fresh \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \ -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \ -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF eatmydata cmake --build build/ ./build/mysql-test/mysql-test-run.pl --force --parallel=auto

To rebuild after code change simply run:

shell Copy

eatmydata cmake --build build/

eatmydata cmake --build build/

For full details, read the whole article.

First step is to create the working directory and some directories inside it to:

shell Copy

mkdir quick-rebuilds cd quick-rebuilds mkdir -p ccache build data

mkdir quick-rebuilds cd quick-rebuilds mkdir -p ccache build data

The directory ccache will be used by the tool with the same name to store build cache permanently. Build artifacts will be output in the directory build to avoid polluting the source code directory so that git in the source tree will not accidentally commit any machine-generated files. The data directory is useful for temporary test installs.

The next step is to get the source code into this working directory.

The oldest git commit in the project is from July, 2000. Since then, MariaDB has had nearly 200 000 commits. To build the latest version and perhaps submit a Pull Request to commit your improvement to the project, you don’t necessarily need to have all those 200 000 commits available in your git clone. You can use shallow git clone to, for example, fetch only the history of past 3 months:

Copy

$ git clone --branch 11.0 --shallow-since=3m \ --recurse-submodules --shallow-submodules \ https://github.com/MariaDB/server.git mariadb-server Cloning into 'mariadb-server'... remote: Enumerating objects: 41075, done. remote: Counting objects: 100% (41075/41075), done. remote: Compressing objects: 100% (29333/29333), done. remote: Total 41075 (delta 19706), reused 20092 (delta 10708), pack-reused 0 Receiving objects: 100% (41075/41075), 75.85 MiB | 8.48 MiB/s, done. Resolving deltas: 100% (19706/19706), done. Checking out files: 100% (24070/24070), done. Submodule 'extra/wolfssl/wolfssl' (https://github.com/wolfSSL/wolfssl.git) registered for path 'extra/wolfssl/wolfssl' Submodule 'libmariadb' (https://github.com/MariaDB/mariadb-connector-c.git) registered for path 'libmariadb' Submodule 'storage/columnstore/columnstore' (https://github.com/mariadb-corporation/mariadb-columnstore-engine.git) registered for path 'storage/columnstore/columnstore' Submodule 'storage/maria/libmarias3' (https://github.com/mariadb-corporation/libmarias3.git) registered for path 'storage/maria/libmarias3' Submodule 'storage/rocksdb/rocksdb' (https://github.com/facebook/rocksdb.git) registered for path 'storage/rocksdb/rocksdb' Submodule 'wsrep-lib' (https://github.com/codership/wsrep-lib.git) registered for path 'wsrep-lib' Cloning into '/srv/sources/mariadb/quick-rebuilds/mariadb-server/extra/wolfssl/wolfssl'... remote: Enumerating objects: 2851, done. remote: Counting objects: 100% (2851/2851), done. remote: Compressing objects: 100% (2124/2124), done. remote: Total 2851 (delta 800), reused 1576 (delta 589), pack-reused 0 Receiving objects: 100% (2851/2851), 20.91 MiB | 10.43 MiB/s, done. Resolving deltas: 100% (800/800), done. ... Unpacking objects: 100% (3/3), done. From https://github.com/codership/wsrep-API * branch 694d6ca47f5eec7873be99b7d6babccf633d1231 -> FETCH_HEAD Submodule path 'wsrep-lib/wsrep-API/v26': checked out '694d6ca47f5eec7873be99b7d6babccf633d1231' $ git -C mariadb-server/ show --oneline --summary f2dc4d4c (HEAD -> 11.0, origin/HEAD, origin/11.0) MDEV-30673 InnoDB recovery hangs when buf_LRU_get_free_block $ git -C mariadb-server submodule 4fbd4fd36a21efd9d1a7e17aba390e91c78693b1 extra/wolfssl/wolfssl (4fbd4fd) 12bd1d5511fc2ff766ff6256c71b79a95739533f libmariadb (12bd1d5) 8b032853b7a200d9af4d468ac58bb9f4b6ac7040 storage/columnstore/columnstore (8b03285) 3846890513df0653b8919bc45a7600f9b55cab31 storage/maria/libmarias3 (3846890) bba5e7bc21093d7cfa765e1280a7c4fdcd284288 storage/rocksdb/rocksdb (bba5e7b) 275a0af8c5b92f0ee33cfe9e23f3db5f59b56e9d wsrep-lib (275a0af) $ du -shc mariadb-server/.git/modules/{storage/*,extra/wolfssl,libmariadb,wsrep-lib} \ mariadb-server/.git mariadb-server/ 30M mariadb-server/.git/modules/storage/columnstore 1M mariadb-server/.git/modules/storage/maria 20M mariadb-server/.git/modules/storage/rocksdb 40M mariadb-server/.git/modules/extra/wolfssl 2M mariadb-server/.git/modules/libmariadb 1M mariadb-server/.git/modules/wsrep-lib 80M mariadb-server/.git 548M mariadb-server/ =720M total

$ git clone --branch 11.0 --shallow-since=3m \ --recurse-submodules --shallow-submodules \ https://github.com/MariaDB/server.git mariadb-server Cloning into 'mariadb-server'... remote: Enumerating objects: 41075, done. remote: Counting objects: 100% (41075/41075), done. remote: Compressing objects: 100% (29333/29333), done. remote: Total 41075 (delta 19706), reused 20092 (delta 10708), pack-reused 0 Receiving objects: 100% (41075/41075), 75.85 MiB | 8.48 MiB/s, done. Resolving deltas: 100% (19706/19706), done. Checking out files: 100% (24070/24070), done. Submodule 'extra/wolfssl/wolfssl' (https://github.com/wolfSSL/wolfssl.git) registered for path 'extra/wolfssl/wolfssl' Submodule 'libmariadb' (https://github.com/MariaDB/mariadb-connector-c.git) registered for path 'libmariadb' Submodule 'storage/columnstore/columnstore' (https://github.com/mariadb-corporation/mariadb-columnstore-engine.git) registered for path 'storage/columnstore/columnstore' Submodule 'storage/maria/libmarias3' (https://github.com/mariadb-corporation/libmarias3.git) registered for path 'storage/maria/libmarias3' Submodule 'storage/rocksdb/rocksdb' (https://github.com/facebook/rocksdb.git) registered for path 'storage/rocksdb/rocksdb' Submodule 'wsrep-lib' (https://github.com/codership/wsrep-lib.git) registered for path 'wsrep-lib' Cloning into '/srv/sources/mariadb/quick-rebuilds/mariadb-server/extra/wolfssl/wolfssl'... remote: Enumerating objects: 2851, done. remote: Counting objects: 100% (2851/2851), done. remote: Compressing objects: 100% (2124/2124), done. remote: Total 2851 (delta 800), reused 1576 (delta 589), pack-reused 0 Receiving objects: 100% (2851/2851), 20.91 MiB | 10.43 MiB/s, done. Resolving deltas: 100% (800/800), done. ... Unpacking objects: 100% (3/3), done. From https://github.com/codership/wsrep-API * branch 694d6ca47f5eec7873be99b7d6babccf633d1231 -> FETCH_HEAD Submodule path 'wsrep-lib/wsrep-API/v26': checked out '694d6ca47f5eec7873be99b7d6babccf633d1231' $ git -C mariadb-server/ show --oneline --summary f2dc4d4c (HEAD -> 11.0, origin/HEAD, origin/11.0) MDEV-30673 InnoDB recovery hangs when buf_LRU_get_free_block $ git -C mariadb-server submodule 4fbd4fd36a21efd9d1a7e17aba390e91c78693b1 extra/wolfssl/wolfssl (4fbd4fd) 12bd1d5511fc2ff766ff6256c71b79a95739533f libmariadb (12bd1d5) 8b032853b7a200d9af4d468ac58bb9f4b6ac7040 storage/columnstore/columnstore (8b03285) 3846890513df0653b8919bc45a7600f9b55cab31 storage/maria/libmarias3 (3846890) bba5e7bc21093d7cfa765e1280a7c4fdcd284288 storage/rocksdb/rocksdb (bba5e7b) 275a0af8c5b92f0ee33cfe9e23f3db5f59b56e9d wsrep-lib (275a0af) $ du -shc mariadb-server/.git/modules/{storage/*,extra/wolfssl,libmariadb,wsrep-lib} \ mariadb-server/.git mariadb-server/ 30M mariadb-server/.git/modules/storage/columnstore 1M mariadb-server/.git/modules/storage/maria 20M mariadb-server/.git/modules/storage/rocksdb 40M mariadb-server/.git/modules/extra/wolfssl 2M mariadb-server/.git/modules/libmariadb 1M mariadb-server/.git/modules/wsrep-lib 80M mariadb-server/.git 548M mariadb-server/ =720M total

With a 3-month history, the main git data for MariaDB is about 50 MB, and the submodules as shallow clones add 30 MB more. If not using shallow cloning, the whole MariaDB repository and submodules would amount to over 1 GB of data, so using shallow clones cuts the amount of data to be downloaded by over 80%.

The checked out data is almost 550 MB, but that is unpacked from the git data, so actual network transfer was at max 80 MB of git data.

In addition to the source code, one also needs a long list of build dependencies installed. Instead of polluting your laptop/workstation with tens of new libraries, install all the dependencies inside a container that has a working directory mounted inside it. This way your system will stay clean, but files written in the working directory will be accessible both inside and outside the container, and persist after the container is gone.

Next, start the container:

shell Copy

docker run --interactive --tty --rm \ -v ${PWD}:/quick-rebuilds -w /quick-rebuilds debian:sid bash

docker run --interactive --tty --rm \ -v ${PWD}:/quick-rebuilds -w /quick-rebuilds debian:sid bash

This example uses Docker, but the principle is the same with any Linux container tool, such as Podman.

Inside the Debian container, use apt to automatically install all dependencies (about 160 MB download, over 660 MB when unpacked to disk) as defined in MariaDB sources file debian/control:

shell Copy

echo 'deb-src http://deb.debian.org/debian sid main' \ > /etc/apt/sources.list.d/deb-src-sid.list apt update apt install -y --no-install-recommends \ devscripts equivs ccache eatmydata ninja-build clang entr moreutils mk-build-deps -r -i mariadb-server/debian/control \ -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends'

echo 'deb-src http://deb.debian.org/debian sid main' \ > /etc/apt/sources.list.d/deb-src-sid.list apt update apt install -y --no-install-recommends \ devscripts equivs ccache eatmydata ninja-build clang entr moreutils mk-build-deps -r -i mariadb-server/debian/control \ -t 'apt-get -y -o Debug::pkgProblemResolver=yes --no-install-recommends'

The single biggest boost to the (re-)compilation speed is gained with Ccache:

shell Copy

export CCACHE_DIR=$PWD/ccache ccache --show-stats --verbose

export CCACHE_DIR=$PWD/ccache ccache --show-stats --verbose

We also want to prime the environment to use Clang:

shell Copy

export CXX=${CXX:-clang++} export CC=${CC:-clang} export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++} export CC_FOR_BUILD=${CC_FOR_BUILD:-clang} export CFLAGS='-Wno-unused-command-line-argument' export CXXFLAGS='-Wno-unused-command-line-argument'

export CXX=${CXX:-clang++} export CC=${CC:-clang} export CXX_FOR_BUILD=${CXX_FOR_BUILD:-clang++} export CC_FOR_BUILD=${CC_FOR_BUILD:-clang} export CFLAGS='-Wno-unused-command-line-argument' export CXXFLAGS='-Wno-unused-command-line-argument'

The first step in actual compilation is to run CMake, instructing it to look at the source in directory mariadb-server/, output build artifacts in directory build/ and use Ninja as the build system. This line also always forces a fresh configuration, discarding any previous CMakeCache.txt files, to use ccache instead of calling gcc/c++ directly, and also skip a bunch of rarely used large plugins to save a lot of compilation time.

shell Copy

cmake -S mariadb-server/ -B build/ -G Ninja --fresh \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \ -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \ -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF

cmake -S mariadb-server/ -B build/ -G Ninja --fresh \ -DCMAKE_CXX_COMPILER_LAUNCHER=ccache -DCMAKE_C_COMPILER_LAUNCHER=ccache \ -DPLUGIN_COLUMNSTORE=NO -DPLUGIN_ROCKSDB=NO -DPLUGIN_S3=NO \ -DPLUGIN_MROONGA=NO -DPLUGIN_CONNECT=NO -DPLUGIN_TOKUDB=NO \ -DPLUGIN_PERFSCHEMA=NO -DWITH_WSREP=OFF

If you are interested in knowing all possible build flags available, simply query them from CMake with:

shell Copy

cmake build/ -LH

cmake build/ -LH

Note that after the configure stage has run, there are no traditional Makefiles in ‘build/’, only a ninja.build since we are using Ninja. Thus, running make build will build. With Ninja it will be ninja -C build. However, we don’t need to call Ninja directly either but just let CMake orchestrate everything with:

Copy

$ eatmydata cmake --build build/ [173/1462] Building C object plugin/auth_ed25519/CMakeFiles/ref10.dir/ref10/ge_add.c.o

$ eatmydata cmake --build build/ [173/1462] Building C object plugin/auth_ed25519/CMakeFiles/ref10.dir/ref10/ge_add.c.o

In interactive mode, Ninja will have just one line of output at the time showing progress. The numbers inside the brackets show how many files have been compiled of the total number of files to compile, and the filename after it shows which file is currently being compiled. Ninja runs by default on all available CPU cores, so there is no need to define parallelism manually. If Ninja encounters warnings or errors, it will spit them out but continue to show the one-liner status at the bottom of the terminal. To abort Ninja, feel free to press Ctrl+C at any time.

Re-starting the compilation will continue where it left off – Ninja is very smart and fast in figuring out what files need to compiled.

While the MariaDB server does have a small amount of CTest unit tests, the main test system is the mariadb-test-run script (inherited from mysql-test-run). Each test file (suffix .test) consists mainly of SQL code which is executed by mariadb-test-run (MTR) and output compared to the corresponding file with the expected output in text format (suffix .result).

To start the MTR with CMake run:

shell Copy

cmake --build build/ -t test-force

cmake --build build/ -t test-force

Alternatively, one can simply invoke the script directly after the binaries have been compiled:

shell Copy

./build/mysql-test/mysql-test-run.pl --force

./build/mysql-test/mysql-test-run.pl --force

This offers more flexibility, as you can easily add parameters such as --parallel=auto (as the default is to run just one test worker on one CPU) or limit the scope to just one suite or just one individual test:

shell Copy

./build/mysql-test/mysql-test-run.pl --force --parallel=auto --skip-rpl --suite=main

./build/mysql-test/mysql-test-run.pl --force --parallel=auto --skip-rpl --suite=main

Note that all commands in this example run as root, as it necessary to start the whole container with a root user inside it to have permissions to apt install build dependencies. However, the mariadb-test-run is actually not designed to be run as root and will end up skipping some tests when run as root. Also, when run like this, a lot of the debugging information isn’t fully shown. To make most out of the mysql-test-run/mariadb-test-run script, read more in the post Grokking the MariaDB test runner (MTR).

As concluded above, the target test-force was for MTR, and the plainly named target test is for CUnit tests. The equivalent direct Ninja command for running target test would be ninja -C build/ test. To list all targets, run cmake --build build/ --target help or ninja -C build/ -t targets all.

MariaDB 11.0 has currently over 1300 targets. There does not seem to be a very consistent pattern in how build targets are named or how they are intended to be used. One way to find CMake targets that might be more important than others is to simply grep them from the main level CMake configuration file:

Copy

$ grep ADD_CUSTOM_TARGET mariadb-server/CMakeLists.txt ADD_CUSTOM_TARGET(import_executables ADD_CUSTOM_TARGET(INFO_SRC ALL ADD_CUSTOM_TARGET(INFO_BIN ALL ADD_CUSTOM_TARGET(minbuild) ADD_CUSTOM_TARGET(smoketest

$ grep ADD_CUSTOM_TARGET mariadb-server/CMakeLists.txt ADD_CUSTOM_TARGET(import_executables ADD_CUSTOM_TARGET(INFO_SRC ALL ADD_CUSTOM_TARGET(INFO_BIN ALL ADD_CUSTOM_TARGET(minbuild) ADD_CUSTOM_TARGET(smoketest

One of the standard targets is install, which can be run ninja -C build install or CMake:

Copy

$ cmake --install build/ -- Install configuration: "RelWithDebInfo" -- Up-to-date: /usr/local/mysql/./README.md -- Up-to-date: /usr/local/mysql/./CREDITS -- Up-to-date: /usr/local/mysql/./COPYING -- Up-to-date: /usr/local/mysql/./THIRDPARTY -- Up-to-date: /usr/local/mysql/./INSTALL-BINARY -- Up-to-date: /usr/local/mysql/lib/plugin/dialog.so -- Up-to-date: /usr/local/mysql/lib/plugin/client_ed25519.so -- Up-to-date: /usr/local/mysql/lib/plugin/caching_sha2_password.so -- Up-to-date: /usr/local/mysql/lib/plugin/sha256_password.so ... -- Installing: /usr/local/mysql/support-files/systemd/mysql.service -- Installing: /usr/local/mysql/support-files/systemd/mysqld.service -- Installing: /usr/local/mysql/support-files/systemd/mariadb@.service -- Installing: /usr/local/mysql/support-files/systemd/mariadb@.socket -- Installing: /usr/local/mysql/support-files/systemd/mariadb-extra@.socket -- Up-to-date: /usr/local/mysql/support-files/systemd/mysql.service -- Up-to-date: /usr/local/mysql/support-files/systemd/mysqld.service

$ cmake --install build/ -- Install configuration: "RelWithDebInfo" -- Up-to-date: /usr/local/mysql/./README.md -- Up-to-date: /usr/local/mysql/./CREDITS -- Up-to-date: /usr/local/mysql/./COPYING -- Up-to-date: /usr/local/mysql/./THIRDPARTY -- Up-to-date: /usr/local/mysql/./INSTALL-BINARY -- Up-to-date: /usr/local/mysql/lib/plugin/dialog.so -- Up-to-date: /usr/local/mysql/lib/plugin/client_ed25519.so -- Up-to-date: /usr/local/mysql/lib/plugin/caching_sha2_password.so -- Up-to-date: /usr/local/mysql/lib/plugin/sha256_password.so ... -- Installing: /usr/local/mysql/support-files/systemd/mysql.service -- Installing: /usr/local/mysql/support-files/systemd/mysqld.service -- Installing: /usr/local/mysql/support-files/systemd/mariadb@.service -- Installing: /usr/local/mysql/support-files/systemd/mariadb@.socket -- Installing: /usr/local/mysql/support-files/systemd/mariadb-extra@.socket -- Up-to-date: /usr/local/mysql/support-files/systemd/mysql.service -- Up-to-date: /usr/local/mysql/support-files/systemd/mysqld.service

To better understand the full capabilities of the build tools, it is recommended to skim through the cmake man page and the ninja man page.

Instead of wasting time on running the install target, one can simply invoke the build binaries directly:

Copy

$ ./build/client/mariadb --version ./build/client/mariadb from 11.0.1-MariaDB, client 15.2 for Linux (x86_64) using EditLine wrapper $ ./build/sql/mariadbd --version ./build/sql/mariadbd Ver 11.0.1-MariaDB for Linux on x86_64 (Source distribution)

$ ./build/client/mariadb --version ./build/client/mariadb from 11.0.1-MariaDB, client 15.2 for Linux (x86_64) using EditLine wrapper $ ./build/sql/mariadbd --version ./build/sql/mariadbd Ver 11.0.1-MariaDB for Linux on x86_64 (Source distribution)

To actually run the server, it needs a data directory and a user, which can be created with:

Copy

$ ./build/scripts/mariadb-install-db --srcdir=mariadb-server $ adduser --disabled-password mariadb $ chown -R mariadb:mariadb ./data $ ./build/sql/mariadbd --datadir=./data --user=mariadb & [Note] Starting MariaDB 11.0.1-MariaDB source revision as process 5428 [Note] InnoDB: Compressed tables use zlib 1.2.13 [Note] InnoDB: Using transactional memory [Note] InnoDB: Number of transaction pools: 1 [Note] InnoDB: Using crc32 + pclmulqdq instructions [Warning] mariadbd: io_uring_queue_init() failed with errno 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) [Note] InnoDB: Opened 3 undo tablespaces [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. [Note] InnoDB: log sequence number 47391; transaction id 14 [Note] InnoDB: Loading buffer pool(s) from /quick-rebuilds/data/ib_buffer_pool [Note] InnoDB: Buffer pool(s) load completed at 230220 20:28:45 [Note] Plugin 'FEEDBACK' is disabled. [Note] Server socket created on IP: '0.0.0.0'. [Note] Server socket created on IP: '::'. [Note] ./build/sql/mariadbd: ready for connections. Version: '11.0.1-MariaDB' socket: '/tmp/mysql.sock' port: 3306 Source distribution

$ ./build/scripts/mariadb-install-db --srcdir=mariadb-server $ adduser --disabled-password mariadb $ chown -R mariadb:mariadb ./data $ ./build/sql/mariadbd --datadir=./data --user=mariadb & [Note] Starting MariaDB 11.0.1-MariaDB source revision as process 5428 [Note] InnoDB: Compressed tables use zlib 1.2.13 [Note] InnoDB: Using transactional memory [Note] InnoDB: Number of transaction pools: 1 [Note] InnoDB: Using crc32 + pclmulqdq instructions [Warning] mariadbd: io_uring_queue_init() failed with errno 0 [Warning] InnoDB: liburing disabled: falling back to innodb_use_native_aio=OFF [Note] InnoDB: Initializing buffer pool, total size = 128.000MiB, chunk size = 2.000MiB [Note] InnoDB: Completed initialization of buffer pool [Note] InnoDB: File system buffers for log disabled (block size=512 bytes) [Note] InnoDB: Opened 3 undo tablespaces [Note] InnoDB: 128 rollback segments in 3 undo tablespaces are active. [Note] InnoDB: Setting file './ibtmp1' size to 12.000MiB. Physically writing the file full; Please wait ... [Note] InnoDB: File './ibtmp1' size is now 12.000MiB. [Note] InnoDB: log sequence number 47391; transaction id 14 [Note] InnoDB: Loading buffer pool(s) from /quick-rebuilds/data/ib_buffer_pool [Note] InnoDB: Buffer pool(s) load completed at 230220 20:28:45 [Note] Plugin 'FEEDBACK' is disabled. [Note] Server socket created on IP: '0.0.0.0'. [Note] Server socket created on IP: '::'. [Note] ./build/sql/mariadbd: ready for connections. Version: '11.0.1-MariaDB' socket: '/tmp/mysql.sock' port: 3306 Source distribution

It is necessary to define the custom data directory path and custom user, otherwise mariadbd will fail to start:

Copy

[Warning] Can't create test file /usr/local/mysql/data/03727bdc8fe2.lower-test ./build/sql/mariadbd: Can't change dir to '/usr/local/mysql/data/' (Errcode: 2 "No such file or directory") [ERROR] Aborting ./build/sql/mariadbd: Please consult the Knowledge Base to find out how to run mysqld as root! [ERROR] Aborting

[Warning] Can't create test file /usr/local/mysql/data/03727bdc8fe2.lower-test ./build/sql/mariadbd: Can't change dir to '/usr/local/mysql/data/' (Errcode: 2 "No such file or directory") [ERROR] Aborting ./build/sql/mariadbd: Please consult the Knowledge Base to find out how to run mysqld as root! [ERROR] Aborting

To gracefully stop the server, send it the SIGTERM signal:

Copy

$ pkill -ef mariadbd [Note] ./build/sql/mariadbd (initiated by: unknown): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /quick-rebuilds/data/ib_buffer_pool [Note] InnoDB: Buffer pool(s) dump completed at 230220 20:29:05 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 47391; transaction id 15 [Note] ./build/sql/mariadbd: Shutdown complete mariadbd killed (pid 5428)

$ pkill -ef mariadbd [Note] ./build/sql/mariadbd (initiated by: unknown): Normal shutdown [Note] InnoDB: FTS optimize thread exiting. [Note] InnoDB: Starting shutdown... [Note] InnoDB: Dumping buffer pool(s) to /quick-rebuilds/data/ib_buffer_pool [Note] InnoDB: Buffer pool(s) dump completed at 230220 20:29:05 [Note] InnoDB: Removed temporary tablespace data file: "./ibtmp1" [Note] InnoDB: Shutdown completed; log sequence number 47391; transaction id 15 [Note] ./build/sql/mariadbd: Shutdown complete mariadbd killed (pid 5428)

With this setup, you can invoke eatmydata cmake --build build/ to have the source code re-compiled as quickly as possible.

The ‘screenshot’ below showcases how Ninja/CMake will only rebuild the file with changes and its dependencies. In the case of a simple MariaDB client version string change, only 5 files needed to be re-built, and it took less than a second:

Copy

$ sed 's/*VER= "15.1"/*VER= "15.2"/' -i mariadb-server/client/mysql.cc $ time eatmydata cmake --build build/ [5/5] Linking CXX executable client/mariadb real 0m0.992s user 0m0.374s sys 0m0.353s

$ sed 's/*VER= "15.1"/*VER= "15.2"/' -i mariadb-server/client/mysql.cc $ time eatmydata cmake --build build/ [5/5] Linking CXX executable client/mariadb real 0m0.992s user 0m0.374s sys 0m0.353s

A similar version string change in the server leads to having to rebuild over a thousand files:

Copy

$ sed 's/MYSQL_VERSION_PATCH=1/MYSQL_VERSION_PATCH=2/' -i mariadb-server/VERSION $ time eatmydata cmake --build build/ [0/1] Re-running CMake... -- Running cmake version 3.25.1 -- MariaDB 11.0.2 -- Packaging as: mariadb-11.0.2-Linux-x86_64 -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) == Configuring MariaDB Connector/C -- SYSTEM_LIBS: dl;m;dl;m;/usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libz.so -- Configuring OQGraph -- Configuring done -- Generating done -- Build files have been written to: /quick-rebuilds/build [377/1257] Generating user.t troff: fatal error: can't find macro file m [378/1257] Generating user.ps troff: fatal error: can't find macro file m [433/1257] Building CXX object storage/archive/CMakeFiles/archive.dir/ha_archive.cc.o In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29: /quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:91:15: warning: 'index_type' overrides a member function but is not marked 'override' [-Winconsistent-missing-override] const char *index_type(uint inx) { return "NONE"; } ^ /quick-rebuilds/mariadb-server/sql/handler.h:3915:23: note: overridden virtual function is here virtual const char *index_type(uint key_number) { DBUG_ASSERT(0); return "";} ^ [...] In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29: /quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:163:7: warning: 'external_lock' overrides a member function but is not marked 'override' [-Winconsistent-missing-override] int external_lock(THD *thd, int lock_type); ^ /quick-rebuilds/mariadb-server/sql/handler.h:5153:15: note: overridden virtual function is here virtual int external_lock(THD *thd __attribute__((unused)), ^ 36 warnings generated. [1257/1257] Linking CXX executable extra/mariabackup/mariadb-backup real 2m7.786s user 12m56.232s sys 1m57.842s

$ sed 's/MYSQL_VERSION_PATCH=1/MYSQL_VERSION_PATCH=2/' -i mariadb-server/VERSION $ time eatmydata cmake --build build/ [0/1] Re-running CMake... -- Running cmake version 3.25.1 -- MariaDB 11.0.2 -- Packaging as: mariadb-11.0.2-Linux-x86_64 -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE) == Configuring MariaDB Connector/C -- SYSTEM_LIBS: dl;m;dl;m;/usr/lib/x86_64-linux-gnu/libssl.so;/usr/lib/x86_64-linux-gnu/libcrypto.so;/usr/lib/x86_64-linux-gnu/libz.so -- Configuring OQGraph -- Configuring done -- Generating done -- Build files have been written to: /quick-rebuilds/build [377/1257] Generating user.t troff: fatal error: can't find macro file m [378/1257] Generating user.ps troff: fatal error: can't find macro file m [433/1257] Building CXX object storage/archive/CMakeFiles/archive.dir/ha_archive.cc.o In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29: /quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:91:15: warning: 'index_type' overrides a member function but is not marked 'override' [-Winconsistent-missing-override] const char *index_type(uint inx) { return "NONE"; } ^ /quick-rebuilds/mariadb-server/sql/handler.h:3915:23: note: overridden virtual function is here virtual const char *index_type(uint key_number) { DBUG_ASSERT(0); return "";} ^ [...] In file included from /quick-rebuilds/mariadb-server/storage/archive/ha_archive.cc:29: /quick-rebuilds/mariadb-server/storage/archive/ha_archive.h:163:7: warning: 'external_lock' overrides a member function but is not marked 'override' [-Winconsistent-missing-override] int external_lock(THD *thd, int lock_type); ^ /quick-rebuilds/mariadb-server/sql/handler.h:5153:15: note: overridden virtual function is here virtual int external_lock(THD *thd __attribute__((unused)), ^ 36 warnings generated. [1257/1257] Linking CXX executable extra/mariabackup/mariadb-backup real 2m7.786s user 12m56.232s sys 1m57.842s

The above example also shows how Ninja spits out warnings.

Despite the majority of the project files being re-built, it still took only two minutes, mainly thanks to ccache having a high hit-rate.

Copy

$ ccache --show-stats Cacheable calls: 3235 / 3235 (100.0%) Hits: 1932 / 3235 (59.72%) Direct: 49 / 1932 ( 2.54%) Preprocessed: 1883 / 1932 (97.46%) Misses: 1303 / 3235 (40.28%) Local storage: Cache size (GB): 0.11 / 5.00 ( 2.18%)

$ ccache --show-stats Cacheable calls: 3235 / 3235 (100.0%) Hits: 1932 / 3235 (59.72%) Direct: 49 / 1932 ( 2.54%) Preprocessed: 1883 / 1932 (97.46%) Misses: 1303 / 3235 (40.28%) Local storage: Cache size (GB): 0.11 / 5.00 ( 2.18%)

Without ccache, the build time in the same scenario is 6–8 minutes. There are some extra flags in ccache (such as CCACHE_SLOPPINESS) which can be used to further tune the ccache speed, but when I did some experimenting, I didn’t discover any that made a visible impact.

Without eatmydata, the build takes 10-20 seconds longer, as the system calls to disk will wait for fsync and the like to complete, but which we are fine skipping since we don’t care about data durability and crash recovery as this is a throwaway environment anyway. Using regular GNU GCC instead of Clang adds another 20–40 seconds to the rebuild time.

The current two minutes for the build time on my laptop with 8-core Intel i7-8650U CPU @ 1.90GHz is not exactly instant, but it is fast enough that I can sit and wait it out without feeling the need to context switch and loose my focus.

As showcased in the post How to code 10x faster than an average programmer, as a high-performing software developer, you don’t want to waste time on manually running a lot of commands to build and test your code, but instead you want to have a setup where you write code in your editor and have the code automatically re-compile and run when the source code file is saved.

For MariaDB, the automatic rebuild part can easily be achieved with:

shell Copy

find mariadb-server/* | entr eatmydata cmake --build build/

find mariadb-server/* | entr eatmydata cmake --build build/

To automatically rebuild and also run a binary (in this case the mariadb client), define multiple commands in quotes to the -s parameter:

shell Copy

find mariadb-server/* | \ entr -s 'eatmydata cmake --build build/; ./build/client/mariadb --version'

find mariadb-server/* | \ entr -s 'eatmydata cmake --build build/; ./build/client/mariadb --version'

When running the server use the -r parameter to have Entr automatically restart it:

shell Copy

find mariadb-server/* | \ entr -r ./build/sql/mariadbd --datadir=./data --user=mariadb

find mariadb-server/* | \ entr -r ./build/sql/mariadbd --datadir=./data --user=mariadb

If the you are developing an MTR test by editing *.test files, there is no need to recompile anything, and you can simply have Entr re-run the test every time a file is changed:

shell Copy

find mariadb-server/* | entr -r ./build/mysql-test/mysql-test-run.pl main.connect

find mariadb-server/* | entr -r ./build/mysql-test/mysql-test-run.pl main.connect

The examples above are specific for MariaDB and illustrate in detail how to be efficient and avoid wasting time compiling, but the principles of utilizing ccache/clang/ninja apply for any software project in C/C++, and entr comes in handy in a myriad of situations.

Hopefully this inspires you to raise the bar on what to expect of speed and efficiency in the future!

The main test system in the MariaDB open source database project is the mariadb-test-run script (inherited from mysql-test-run). It is easy to run to and does not require you to compile any source code.

While writing MTR tests is relevant only for MariaDB developers, knowing how to run MTR is useful for any database administrator running MariaDB, as it is a quick way to validate that MariaDB can run correctly on your hardware and operating system version.

TL;DR for Debian/Ubuntu users

Run the full 6000+ test suite as current user in temporary directory with multiple workers in parallel and with detailed logging on failures, typically taking over 30 minutes on modern laptop:

Copy

apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb cd /usr/share/mysql/mysql-test export MTR_PRINT_CORE=detailed ./mtr --force --parallel=auto --vardir=$(mktemp -d) \ --skip-test-list=unstable-tests.amd64 --big-test

apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb cd /usr/share/mysql/mysql-test export MTR_PRINT_CORE=detailed ./mtr --force --parallel=auto --vardir=$(mktemp -d) \ --skip-test-list=unstable-tests.amd64 --big-test

For full details, read the whole article.

To avoid polluting your actual system with new packages, it is convenient to run MTR in a throwaway container. Start one with some RAM-memory-backed disk allocated with --shm-size=1G so running MTR with --mem later on is possible:

shell Copy

docker run --interactive --tty --rm --shm-size=1G debian:sid bash

docker run --interactive --tty --rm --shm-size=1G debian:sid bash

This example uses Docker, but the principle is the same with any Linux container tool, such as Podman.

Next, install the MariaDB test suite package. This will also pull in the MariaDB server and all the necessary dependencies. Additionally, also install the GNU Debugger (gdb) for automatic stack traces and patch.

shell Copy

apt update apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb

apt update apt install -y mariadb-test mariadb-backup mariadb-plugin-* patch gdb

The mariadb-test-run is not intended to be run as root and will skip some tests if run as root. Therefore, create a new user inside the container and grant it permissions to the test directory:

shell Copy

adduser --disabled-password mariadb-test-runner chown -R mariadb-test-runner /usr/share/mysql/mysql-test

adduser --disabled-password mariadb-test-runner chown -R mariadb-test-runner /usr/share/mysql/mysql-test

At minimum, the test runner user needs to be able to write to the path /usr/share/mysql/mysql-test/var (unless some other path is defined with --vardir and --tmpdir), but some tests also run patch to modify the test files on-the-fly, so just grant permissions to the whole test directory. As this is a throwaway container anyway, there is no need to be prudent with the permissions.

Next, switch to the test user and test directory and start the run with default settings:

Copy

$ su - mariadb-test-runner $ cd /usr/share/mysql/mysql-test $ ./mariadb-test-run Logging: ./mariadb-test-run VS config: vardir: /usr/share/mysql/mysql-test/var Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Using suites: main-,archive-,atomic-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_uuid,user_variables Collecting tests... ... main.subselect_innodb 'innodb' w4 [ pass ] 3359 main.subselect_sj2 'innodb' w1 [ pass ] 2737 main.subselect_sj2_jcl6 'innodb' w3 [ pass ] 2933 main.parser_bug21114_innodb 'innodb' w8 [ pass ] 15364 innodb_gis.rtree_search 'innodb' w5 [ pass ] 52379 -------------------------------------------------------------------------- The servers were restarted 1736 times Spent 8527.863 of 1659 seconds executing testcases Completed: All 5131 tests were successful. 995 tests were skipped, 280 by the test itself.

$ su - mariadb-test-runner $ cd /usr/share/mysql/mysql-test $ ./mariadb-test-run Logging: ./mariadb-test-run VS config: vardir: /usr/share/mysql/mysql-test/var Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Using suites: main-,archive-,atomic-,binlog-,binlog_encryption-,client-,csv-,compat/oracle-,compat/mssql-,compat/maxdb-,encryption-,federated-,funcs_1-,funcs_2-,gcol-,handler-,heap-,innodb-,innodb_fts-,innodb_gis-,innodb_i_s-,innodb_zip-,json-,maria-,mariabackup-,multi_source-,optimizer_unfixed_bugs-,parts-,perfschema-,plugins-,roles-,rpl-,stress-,sys_vars-,sql_sequence-,unit-,vcol-,versioning-,period-,sysschema-,disks,func_test,metadata_lock_info,query_response_time,sequence,sql_discovery,type_inet,type_uuid,user_variables Collecting tests... ... main.subselect_innodb 'innodb' w4 [ pass ] 3359 main.subselect_sj2 'innodb' w1 [ pass ] 2737 main.subselect_sj2_jcl6 'innodb' w3 [ pass ] 2933 main.parser_bug21114_innodb 'innodb' w8 [ pass ] 15364 innodb_gis.rtree_search 'innodb' w5 [ pass ] 52379 -------------------------------------------------------------------------- The servers were restarted 1736 times Spent 8527.863 of 1659 seconds executing testcases Completed: All 5131 tests were successful. 995 tests were skipped, 280 by the test itself.

If no test suite is selected, MTR will run about 6000 tests, which on my laptop takes about 30 minutes. If MTR is started with additional --big-test parameter, it will run additional tests that are resource intensive and consume, for example, a lot of RAM memory and take a long time to run, totalling to a test run that has 6100 tests and takes 43 minutes to complete (on my laptop). To only run big tests, use --big --big.

If there is a need to limit the scope, such as in build systems that want to validate that the built binary works without running all tests, typically --suite=main --skip-rpl is used. This results in about 1000 tests being run, which on my laptop takes about 3½ minutes.

Even when running without any limitations on what tests are run, many tests have code that makes them opt out automatically based on some condition being missing, and on my laptop about 1000 tests end up being skipped. Some examples:

Copy

main.connect2 [ skipped ] Requires debug build main.mysql_client_test_comp [ skipped ] No IPv6 main.connect-abstract [ skipped ] Need Linux main.grant_cache_ps_prot [ skipped ] Need ps-protocol main.fix_priv_tables [ skipped ] Test need MYSQL_FIX_PRIVILEGE_TABLES main.no-threads [ skipped ] Test requires: 'one_thread_per_connection' main.udf_skip_grants [ skipped ] Need udf example main.lowercase_mixed_tmpdir_innodb [ skipped ] Test requires: 'lowercase2' main.innodb_load_xa [ skipped ] Need InnoDB plugin mariabackup.alter_copy_excluded [ skipped ] No mariabackup binlog.binlog_expire_warnings [ skipped ] Test needs --big-test encryption.innodb-spatial-index [ skipped ] requires patch executable plugins.pam_cleartext [ skipped ] Not run as user owning auth_pam_tool_dir rpl.rpl_gtid_mdev4474 'innodb,row' [ skipped ] Neither MIXED nor STATEMENT binlog format

main.connect2 [ skipped ] Requires debug build main.mysql_client_test_comp [ skipped ] No IPv6 main.connect-abstract [ skipped ] Need Linux main.grant_cache_ps_prot [ skipped ] Need ps-protocol main.fix_priv_tables [ skipped ] Test need MYSQL_FIX_PRIVILEGE_TABLES main.no-threads [ skipped ] Test requires: 'one_thread_per_connection' main.udf_skip_grants [ skipped ] Need udf example main.lowercase_mixed_tmpdir_innodb [ skipped ] Test requires: 'lowercase2' main.innodb_load_xa [ skipped ] Need InnoDB plugin mariabackup.alter_copy_excluded [ skipped ] No mariabackup binlog.binlog_expire_warnings [ skipped ] Test needs --big-test encryption.innodb-spatial-index [ skipped ] requires patch executable plugins.pam_cleartext [ skipped ] Not run as user owning auth_pam_tool_dir rpl.rpl_gtid_mdev4474 'innodb,row' [ skipped ] Neither MIXED nor STATEMENT binlog format

If you are interested in one particular test, just give it as the last argument:

Copy

$ ./mariadb-test-run main.connect Logging: ./mariadb-test-run main.connect Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Collecting tests... Installing system database... ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- main.connect [ pass ] 14206 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 14.206 of 20 seconds executing testcases Completed: All 1 tests were successful.

$ ./mariadb-test-run main.connect Logging: ./mariadb-test-run main.connect Creating var directory '/usr/share/mysql/mysql-test/var'... Checking supported features... MariaDB Version 10.11.2-MariaDB-1 - SSL connections supported - binaries built with wsrep patch Collecting tests... Installing system database... ============================================================================== TEST RESULT TIME (ms) or COMMENT -------------------------------------------------------------------------- main.connect [ pass ] 14206 -------------------------------------------------------------------------- The servers were restarted 0 times Spent 14.206 of 20 seconds executing testcases Completed: All 1 tests were successful.

There are three parameters can greatly improve how quickly MTR runs. The primary one is --parallel=auto, which will run MTR in parallel with as many workers as there are CPUs (by default MTR runs with just one worker). On my laptop, going from one MTR worker to 8 in parallel reduced the total run time for the main suite from 17 to about 3 minutes.

Another parameter is --fast, which will make the MTR kill all server processes violently without waiting for them to gracefully shutdown. The test run restarts the MariaDB server hundreds of times, so saving half a second on every shutdown results in the main suite completing 20 seconds faster.

The parameter --mem instructs MTR to make the directory var/ a symbolic link to a subdirectory on the shared memory device (/dev/shm). This works if the container is started with --shm-size and has at least 350MB of space on the ramdisk. On my laptop, this further reduced the main test suite duration down to 2½ minutes.

When running a single test, one might use --verbose as an additional argument to see the commands that run in the test. It is also possible to define --verbose --verbose twice, but that makes the test run so verbose that is it unusable.

When running a suite of tests, you only want to have extra output visible if the test fails. If the server fails to start and a particular test doesn’t run at all, using --verbose-restart might be beneficial. If running the test in an automated system, one might want to save results in a JUnit-compatible XML file that can, for example, be rendered by GitLab CI.

This is the command that several CI systems run MTR with:

shell Copy

export MTR_PRINT_CORE=detailed eatmydata perl -I. ./mariadb-test-run \ --force --testcase-timeout=120 --suite-timeout=540 --retry=3 \ --verbose-restart --max-save-core=1 --max-save-datadir=1 \ --parallel=auto --skip-rpl --suite=main \ --xml-report=mariadb-test-run-junit.xml

export MTR_PRINT_CORE=detailed eatmydata perl -I. ./mariadb-test-run \ --force --testcase-timeout=120 --suite-timeout=540 --retry=3 \ --verbose-restart --max-save-core=1 --max-save-datadir=1 \ --parallel=auto --skip-rpl --suite=main \ --xml-report=mariadb-test-run-junit.xml

The command above also showcases use of eatmydata, which makes fsync and similar system calls to skip memory-to-disk guarantees, but in my testing with MTR, it didn’t affect speed.

The above summarizes everything one typically needs to know for running the mariadb-test-run.

For a DBA, there are also several other tools that ship with MariaDB, which can help with testing and validating ones own environment, such as mariadb-stress-test and mariadb-slap.

If you want to write a new test or fix an existing test, there are many more additional parameters to learn, such as --record and --gcov. The official contribution docs at mariadb.org list some of the most useful MTR parameters. The mariadb.com knowledge base article lists them all. As with all commands in Linux, there is also the main page for mariadb-test-run.

The structure of the tests is actually quite easy, and requires no C/C++ skills to write. Each test file (suffix .test) consists mainly of SQL code which is executed by mariadb-test-run (MTR), with the output compared (with diff) to the corresponding file with the expected output in text format (suffix .result).

In my development workflow, writing a test and running MTR might look something like this:

If you want to contribute to the MariaDB open source project, extending the test coverage is a great place to start. To scratch your own itch, think about a MariaDB bug you encountered while using it, and think if that can be reproduced as a test and submitted upstream so that it becomes part of the body of tests and thus easy to catch if it ever regresses again.

To learn more about writing mariadb-test-run tests, read the test case authoring guide at mariadb.org.

What is the key to being an efficient programmer? Well, the answer is surprisingly simple. Having a setup where you can write and test your code over and over in an uninterrupted flow will dramatically increase your productivity.

The cost of doing just one more tweak to make the code perfect should be as close to zero as possible. The developer should not feel any drain when doing just one more test to ensure everything is absolutely correct. The experience should be fast and frictionless.

I always try to set up my development environment in a way that I can write code in one window, and immediately see the result in another. It does not matter if I am doing front-end or back-end development – I insist on having the code in one window and the result update in another window as soon as I press Ctrl+S or switch focus between windows.

I achieve this with a combination of two great developer tools:

• The Atom Pulsar code editor with autosave to automatically save code files.

• The Entr command-line tool to restart programs automatically when files change.

The basic usage of Entr is to list all files in your coding project with find and pipe the list to entr telling it what command to run when any of the files are updated:

shell Copy

find * | entr python3 demo.py

find * | entr python3 demo.py

If you are working on a long-running process which does not exit on every run, such as a server app, you might want to use the -r and -z parameters to make Entr restart the program:

shell Copy

find * | entr -rz node server.js

find * | entr -rz node server.js

Occasionally the workflow might need two commands, such as in this example compiling and running a demo program in C. That can be achieved with the -s parameter along with the commands as one quoted string:

shell Copy

find * | entr -s "gcc demo.c -o demo; ./demo"

find * | entr -s "gcc demo.c -o demo; ./demo"

But what if the full development cycle includes uploading the files to a remote server? No problem, Entr and Rsync can handle that as well, and with the addition of ts you will also see the timestamps of when Rsync last ran.

shell Copy

find * | entr rsync -avz --delete-after * example.com:/path-to-target-dir/ | ts

find * | entr rsync -avz --delete-after * example.com:/path-to-target-dir/ | ts

The same principle applies to all development workflows, not just command-line stuff. For example this very blog is written in Markdown and is converted to static HTML pages with Hugo, which has a built-in command hugo server that serves the pages locally and automatically reloads pages thanks to livereload.js.

While optimizing the code-change-compile-run cycle is the key to productivity, additional gains can also be achieved when the code editor gives real-time visual clues of what is going on and what might be an issue.

The screenshot below shows how the GCC linter in Atom Pulsar adds a red dot to the line with an issue, and underlines the exact section on the line. A popup shows in-context information when the cursor is in the problematic function call.

In the same screenshot, note also how nicely Atom Pulsar shows (with a yellow hint) the filename that has uncommitted changes, and inside the file we can see the lines that have changed. The dark grey communicates that a file is excluded from git tracking with .gitignore (in this case the demo binary, as we only want to have source code in git).

In Atom Pulsar I have linters for all the programming languages I use, and also Shellcheck for bash scripts and yamllint for configuration files.

During my career in software engineering, I’ve noticed that some coders simply have an instinct for what is too slow. While most people keep on grinding on the path they set on when trying to solve a problem, the more passionate programmers feel frustrated if their progress is too slow. An effective programmer will switch to optimize the speed of their progress – and only once they are happy with their velocity they will switch back to solve the original problem. This means they are eventually much faster at it and all future similar problems. Smart programmers have a great sense of when it is appropriate to stop the process, improve tooling, and then have the process run much faster.

Next time you catch yourself wasting time on doing something repetitive and slow, stop and ask yourself why you are doing it. A good programmer will always raise the bar for what they consider acceptable development velocity.

This is the biggest and most common mistake I see that prevents people from thinking clearly, and it is also the most difficult one to unlearn.

Too often people rush to the first solution they see. It is just too easy. But the risk of the first solution not being the best one is way too high. If you want to think clearly, innovate and solve problems in engineering or just in life in general, learn this.

Always resist the temptation to define the problem so that it fits the solution. It must always be the other way around: first learn as much as possible about the problem. Define it well. You must first know what problem you are solving before you can even start to think about potential solutions. If you rush to the first solution you come across, you will most likely focus too much on that particular solution, with the result that you become blind to the real problem and lose the ability to see a good solution that will actually solve the problem.

Focus your effort on the problem. Once you fully understand the problem, the solution will follow almost naturally and without too much effort, since the insight you will have gained of the problem will automatically direct you towards the right solution.

Resisting the urge is easier said than done. There are three things I practice to break away from my bias:

1. Write it down. Putting it in writing forces you to produce coherent sentences and thus think it through. Seeing it in writing is a quick way to distance yourself from the issue and be your own first critic.

2. Go for a walk. Take a timeout. Grab some fresh air to elevate your mood and attentiveness. Try to forget the issue for a while. This will allow you to approach the issue from a new angle when you return to it. Sleep on it, and maybe your unconscious self will process something while you sleep. Let time pass to increase the odds that you come up with fresh revelation about the issue. Even if no new thoughts come, the fact that time passed without any new aspects surfacing will increase the odds that you have understood the issue well.

3. Present to or teach somebody else. Find somebody who is smart and honest. Then either in writing or in person, explain the issue and all the relevant data points. When you try to convince somebody else of your point of view, your own brain will try to anticipate counterpoints, which forces you to do your research, and when you actually present to the other person, their feedback will help you solidify your solution. The feedback from the other person does not need to be correct. If it is, that is great, but even the mere fact that you defended an idea will let you know how you felt about it.

If you still feel inclined after this, go for it.