I’ve been calling it a “toxic blaming culture“, until I connected with one of my peers who experienced and witnessed same/similar cases, he raised it is “gaslighting“.

You can search what gaslighting is online. Essentially it’s a manipulation of mindset, in the case of working, it’s the bosses abuse authorities to blame and torture direct reports.

It’s the first time I experienced and witnessed gaslighting end to end in my professional career (I witnessed for a short term in prior company but didn’t have full awareness or end to end experience). It’s upsetting and disturbing to both myself and others that experienced it that I can see. It’s the moment that I learned the term “gaslighting“ that I decided I must leave the org (by either transferring inside the comp or leave the comp).

It makes you mentally unhealthy - you feel your work is not appreciated, you feel everyone around you is aggressive and attacking, you try to attack others to divert the accusation, you become a machine and lose your motivation, you feel so unsafe that would do everything to try prevent from being blamed, and you just can’t work the normal way any more.

Besides the negative impact of gaslighting on a person’s working hours, I can’t help but notice it’s toxically impacted my personal private life after work. E.g. I become so sensitive to people’s questions and their tones that I start to fight back when I sensed even a slight doubt in their messaging and I felt blamed - because I’m being blamed at work and I don’t want to get it at home or personal life! Worse thing is many trivial things can bring up fights with my wife, my parents and others in life. This is where I realized I need to get out of such environment as quickly as possible, because it starts to expand into

• BTC/ETH是加杠杆的美股, 山寨是加大号杠杆的美股, 都紧随美股的走势

• 如果完全以美股走势计算, 纳指重回6月中价格, btc也是回到6月中价格 ($21k-$22k)， 所以周末会横盘等待美股选择方向

• 中期走势要看美股, 如果美股计价衰退创新低, btc有可能重回$15k

• 市场已经彻底消化ftx事件的负面影响

1. 币圈与美股的关系

在过去这轮周期机构深度介入后

• BTC/ETH变成 "加杠杆的美股” - 紧随美股的走势, 涨时比美股更多, 跌时比美股更深!

• 山寨变成加 ”大号杠杆的美股”

• 整个币圈都紧随美股的走势

来看图

1.1 牛市期间

• 2/2020 covid期间

  • 大饼于2/12/2020形成高点并开始跌, 美股指数于2/20/2020形成高点

  • 大饼于3/12/2020触底, 美股指数于3/23/2020触底

• 4/2021 阶段性高点

  • 大饼于4/13/2021触顶, 纳指于4/13-4/27期间走出双顶

• 11/2021 本轮周期顶部

  • 大饼于11/8/2021触顶, 美股指数于年底12/28期间触顶

• 初步结论: 牛市期间, BTC走势先于美股指数触顶和触底

  • 可能原因: 币圈流动性好, 且24/7运作, 不受美股交易时间的限制, 导致资本更容易察觉异动, 比提前行动

1.2 熊市期间

• 5/2022 反弹

  • 大饼与美股指数基本同时于5/31见顶

  • 顶部调整期, 大饼的震荡幅度更剧烈

• 8/2022 反弹见顶

  • 大饼于8/12/2022触顶, 美指于8/16触顶

• 10/2022 新低

  • 美指于10/13触底创造新低, 大饼重新试探$20k

• 12/2022 反弹见顶, 并创造higher low

  • 美指于12/1反弹见顶, 大饼于12/14上探$18k, 大概是收到ftx事件影响, 反弹滞后

• 1/2023 反弹

  • 大饼与美股指数基本同时从1/6开始反弹

  • 大饼反弹幅度更大, 可能是回补之前ftx深跌的影响

  • 
    

  • 在此之前, 大饼这轮熊市都是紧随美股的走势, 涨时比美股更多, 跌时比美股更深!

  • 但此次反弹, 大饼第一次比美股涨的更多! 这表明, 市场已经彻底消化ftx事件的负面影响, 暴力弥补缺口

2. 中期走势

中期走势还要看美股, 看美股1月底的财报季, 以及是否计价衰退.

如果美股计价衰退, 并二次探底或者创新低, 则BTC和币圈整体会跟随, 并且下跌幅度更大. BTC可能触及$12k区间

首先他是一个云基础设施；再进一步说，他其实是一个多云的基础设施。那怎么理解呢？

第一，云基础设施就是它是一个cloud native的observability提供商，我们在上面可以观察到各种云原生的这个应用的指标。在这一块儿，他做的比较牛逼的地方是，他首先跟aws进行了深度的整合，AWS有自己的原生observability服务，叫做这个CloudWatch，但是很令人惊讶的是，CloudWatch自己的这个服务竟然没有Datadog全面和深入，Datadog跟AWS的整合更深入和更优秀。GCP和Azure我不清楚，但是根据AWS的这个情况来看，Datadog在Business Development和technical方面有着极深的公关和开发能力。

第二点是Datadog是多云设施的基础，也就是说它不仅是为AWS服务，他还支持GCP和Azure。现在多云是一个趋势，特别是大企业啊，他们不希望在一个云厂商lock-in，希望他的平台可能跨多个云厂商。而跨多个云厂商的问题呢，就是开发人员同时也希望他们用的observability也是能够跨越云平台的，对吧？比如说你在GCP上就没法用Azure上面的observability tool，但是你可以在AWS,GCP,Azure都用Datadog，是占据了极有力的多云厂商的趋势。

第三点，产品领先。在竞争对手方面，还没有看到一个特别有力的竞争对手。observability这种tool，说白了，除非它产品有严重的问题，否则其实是不会轻易的去换的。Datadog的产品体验和用户体验非常顺畅，这个非常令人惊讶，就是他的使用体验比CloudWatch要好很多，还没有看听到有人抱怨说对他不好用。这是一个非常强的护城河，就是这个工具一定要让大家觉得它好用，不给大家添乱，是不是啊。就是怎么说呢，就是最好的情况就是让大家都不知道你有这个工具，他自然而然融为你engineering的一部分。同时，他的产品支持也很好，我之前用的时候体验不错，support可以跟用户及时进行沟通以解决问题。

第四，不断扩展的边界。Datadog在他本职工作的护城河的基础上，又开发了很多新的业务。比如说自然的就延伸到log（日志）领域，和security（安全）领域。所以Datadog在慢慢的向一站式的这个云observability，log，security提供商的这个方向在稳步发展。

所以Datadog是会吃到云和多云的这个红利。

It hit me hard and uncomfortable because, during the whole conversation, the focus was always on “the project“, “the progress“, etc, and not me as an individual. Frankly, I’d appreciate a lot more if my manager can check in to see how I am doing as a person, e.g. am I exhausted and need a break?

I totally respect my manager. In many ways he is a good, experienced manager and have taught me a lot of things. This reflection is not about him as a person, but how to approach and treat people in general.

Many people keep the word “people first“ in their mouths but never exercise it practically. In these daily conversations, they may just forget these principles, and it can hurt people a lot. These casual things just make people feel they are a screw and only matter as a screw.

Ever since then, I’ve changed format of my 1:1 with team members. Previously, the agenda was:

1. anything you wanna talk about? anything in your mind? (let my team member speak first on whatever topics they have in mind)

2. how’s your project going? what help do you need?

3. how are you doing? how’s your family?

I’ve permanently changed it by switching points 2 and 3, to:

1. anything you wanna talk about? anything in your mind?

2. how are you doing? how’s your family?

3. how’s your project going? what help do you need?

By nature, I care deeply of each person in team. With my own experience, at least I will never let my team members feel that way

Writing customer success stories isn’t as easy as you imagine without trying to do one yourself, especially if you are an engineer. Writing a story is not easier than writing a big feature.

It first requires extraordinary skills of story telling. Listening and reading stories is one of humans’ instincts. A use case or customer success story is first a story, then a marketing piece. Readers usually make a decision whether to continue reading or not, by just browsing the first few paragraphs and making an intuitive decision on how good, smooth, and rational the story would be — and pass it if it’s not.

Note that I’m not emphasizing on skills of writing, but story telling. To me, the words and phrases in a writing do not need to be gorgeous, especially for non-English-native writers. As long as the terminologies are correct and the structure of the blog is well planned, other words usually don’t matter that much. By the way, you can always hire some native English speakers on contracting platform like Upwork to help polish the wordage, but they cannot help with structure of your story.

Second, collaboration and communication. The story is usually first drafted by customers, and reviewed by the writer, it’s a loop with many iterations that needs collaboration and guidance. Many times I found that customers don’t really write about what we really care in their first few iterations — they may get too narrow-focused on a specific feature; or they don’t explain well what their previous pain points were; or they don’t demonstrate what the amazing outcomes are after adopting our technology.

Third, writing an attractive customer story requires deep understanding of the vertical field your product is in, and extensive knowledge of your product. You can’t just hire someone as advocate and ask for a customer success story within a week or a month.

Of course there are still many best practices, and I’m just writing some simple but crucial thoughts now.

Note that this is actually generally applicable to any IC, but the very senior ones may be worth more of your time.

0. Who are they? What characteristic do they have?

Before talking about hiring and retaining, let’s discuss your targets first. This is critical to help you analyze actions in following steps.

So what characteristics do they very senior people have? I try to list a few major points: they are very smart, are deep expert in certain areas— they can be hands-on, but they also need to more junior people to help on execution so they can move on something more interesting

• they are ambitious — they want their own scope, they want to navigate directions and ideas, control their own destinations rather than following someone else

• they usually are very humble and friendly. But occasionally exceptional people have bad temper and not collaborative — you should avoid these people

• they like to be challenged, rather than staying within comfort zone — throw them new problems, the bigger the better; they get bored at keeping status quo

• they may or may not like leading people or managing projects — some prefer working alone, some prefer only developing POC

• they are usually well paid in their existing companies — more money alone is usually not attractive to them

1. Don’t rush into it! Evaluate hiring first

First of all, don’t rush into it, especially when you are told to do so by your leadership, and it’s not your idea originally!

Evaluate and decide what level you really need to hire, and whether you really need a very senior (distinguish/principle/staff) level IC v.s. just a senior engineer can satisfy the requirement.

Analyze the requirement and considerations, e.g. including but not limited to:

• required domain expertise and technical experience

• required leadership and communication skills

• required credibility (e.g. open source project titles?)

• required scope and trajectory of the person, aka, growth and retention effort from you (e.g. do you have hire a person to just do one project, or have a ton of room for them in the future)

2. Keep balance

There’re many balances a manager need to take before and during hiring very senior ICs. These questions are

• structure and percentage of the team— how many very senior ICs you already have? are there enough middle or junior level ICs for very senior ICs to lead? It’s usually good to not have more than 30% within team

• project scope and career path — do you or your team have enough scope for them? Do these people have scope overlap (good to have no-to-minimal overlap) or are mostly independent? what about your existing team members who are in short term (e.g.< 1y) to be promoted to very senior title?

3. Sourcing

I source and screen all my candidates, especially the very senior ones. Though recruiters and sourcers are great, I usually find there are gaps in their understanding and skills which make it harder for them to source and identify super senior ICs.

The most critical thing about sourcing is to figure out the right channel:

• LinkedIn: certainly one of the most useful ones. You can either search key words and technology terms on it, however it’s not very intelligent and accurate from time to time; or use it to connect to strangers or friends’s friends and build your network as talent pool

• Word of mouth: tell people you are hiring! tell more people so they can actually help! I find certain managers really underestimate the power of spreading the words. My believe is that you can only get help if you tell people

• Domain specific channel: this is the secret source to validate if a manager has deep roots and accumulated experience in the field they are hiring or not. E.g. friends made in some meetups or conferences; people you know thru open source projects; open source projects committer/committee list; people in email group or slack channel of open source project; etc. I see these secret sauces as bars to validate if a hiring manager is qualified or not

4. Negotiation and Hiring

The key of negotiation and hiring is to figure out what these candidates value the most, and get them excited.

The key here is to find out what excites them:

• compensation — just note higher comp is not always the answer alone, so touch on it but don’t spend all your time talking about it, they can do the math themselves!

• personal situation — e.g. are they looking for relocation or a different timezone? do they need help to deal with working visa? do they need to deal with a family situation?

• environment — e.g. are they not well supported or not respected by existing manager/team? Is the current company going south and not doing well?

• interests and challenges — e.g. are they looking for different, harder, more cutting edge tech problems to solve? do they want to exercise their leadership and management skills?

• flexibility and scope — e.g. how they can gain more flexibility and bigger scope? how autonomous they can be?

• open source — e.g. some people really wants to work on open source

Take a personal approach — identify what excites them on case by case, and sell your team/opportunity by emphasizing on those points.

All in all, this is a general framework to think of how to hire super senior ICs. Managing and retaining them is another big topic that’s worth a separate blog.

Such engineering team is unique in the way that, despite all the normal, common engineering culture and practices, they have special requirements in roles and missions at the OSS companies. Here are the 4 unique areas are critical to the success of the team and the company:

1.Set the Foundation for Core Open Source Software and Project

The OSS team needs to set the foundations for both technical and non-technical areas:

For technical areas:

• design and software quality (reliability, HA, performance, scalability, forward-looking extensibility, etc)

• code quality (code readability, comment, etc)

• test coverage and quality for core software (unit, integration, E2E, etc)

• CI of core software. Usually no CD since this team doesn’t run the service themselves

• software releases (quality, cadence/frequency, maintenance, backward compatibility, etc)

For non-technical areas:

• Software development process and best practices

• PR review process and standard in open source

• Completeness of documentation

Overall, the OSS team in a OSS company is at center of the stage to drive these areas.

2. Foster and Cultivate Open Source Engineering Culture and Bridge OSS and non-OSS Culture within Company

Only people who have deep experience involved in open source understand that there are differences working in OSS v.s. within a wall-guarded company internal codebase.

“Why can’t we make a big decision and pivot in a single day in open source project?! That’s inefficiency!” “Why does someone outside our company have a saying in our project?!” “Why can’t we just change the public API and force users to migrate?” Such questions are good indicator that the asker haven’t had hands-on experience in a large OSS project before, and you need to be cautious of having them run an OSS engineering team.

OSS engineering best practices have very different perspective than those of working in a private codebase, as your audience is critically different - OSS engineers work with a community that do not belong to your company. The community members may have different priorities, use cases, concerns, and timeline than yours. E.g. :

• top-down decisions do not work well in OSS as there’s no management chain or leveling authority; credibility is earned in OSS community thru individual’s past involvement

• decisions needs consensus and reaching consensus can take days, if not weeks, as external stakeholders have different time schedules and concerns you have no control of; participants have to be patient, plan ahead for the timeline, and follow the community process; community does not accept arguments like “this is urgent and we have to decide today” except rare cases like security breach (e.g. recent log4j issue)

• standards and bars are held high across the board, in every details, e.g. design and code review, or as small as coding styles. Things that can sneak thru in company’s private codebase can no longer make it. Newbies in OSS may feel it’s overdoing and get frustrated easily, believing they are being specifically targeted by someone which is not the case at all

• project stability comes first. Minima breaking API changes, any major changes requires careful planning, proactive communication and warning, releases and upgrade are taken seriously and in right cadence (majority people won’t be able to upgrade more than a couple times a year), back compatibility is always desired, define well short term maintenance and long term maintenance support scope, etc

• community health and success is the top consideration, rather than individual companies etc

Depending on project, foundations, and licenses, the specific details can vary. E.g. one of the most strict and counter-intuitive rules is the famous “The Apache Way” , which is the standards and best practices to manage Apache Foundation projects. Apache projects have boards and OSS hierarchy to monitor the daily operations and ensure their success. MongoDB and Elastic on the other hand are relatively loose in rules as they are controlled by one company.

The key here is bridging the OSS engineering culture with a company’s general engineering culture, e.g. how to balance the decision making in OSS which can take days/weeks v.s. that within company which needs to be quick? how to communicate and collaborate with engineers working on private code with different expectation of “responsiveness”? how to blend the OSS standards with company internal ones? Not be able to address this will create conflicts and frictions across the board.

3. Own The Advocacy and Channel of User Acquisition, Marketing, and Branding

If your company is built around an OSS project, that project is “the” single most important channel for customer acquisition/engagement, product marketing and branding. Being the engineering face of the company and product to external audience, OSS engineering team must own the OSS project, the advocacy and the channel.

This include, e.g.:

Customer Acquisition & User Engagement:

• listen to and collect customer requirements/requests

• be responsive in answering questions and helping users debug issues

• be respectful and welcoming in communication

• be transparent in defining processes and making decisions, enforce same process and standards across the board

• grow OSS project adoptions

• encourage contribution and grow OSS contributors

• define and drive OSS governance, e.g. define titles and responsibilities (e.g. “committer”/“PMC” in Apache projects, “maintainer” in some projects)

Product Marketing & Branding:

• lead OSS roadmap and priority with community

• organize and speak at meetups, conferences

• etc

4. Drive Open Source Business Model and Monetization by Collaborating Closely within Internal Teams

The last but not the least point is the open source engineering team must collaborate closely with internals teams. Being named as “open source engineering” can create the wrong illusion that you only need to focus on open source, which is not true. Growing business (revenue, profit, etc) is the ultimate goal of a company, and having a OSS project alone doesn’t make you there. To achieve the goals, the OSS team has to partner with comprehensive, cross-functional internal teams, like service engineering, product, marketing, sales, legal, support, on strategy and execution.

Potential Collaborations:

• with service engineering team on paid products (e.g. SaaS)and areas like reliability/HA, multi-tenancy, deployment, operation easiness, monitoring/alerting, scalability, etc

• with product team on product roadmap and prioritization

• with product and design team on UI and UX

• with product, marketing, sales team to decide which feature should be in OSS version v.s. enterprise/SaaS version. There are usually 3 areas of such features, 1) security, 2) collaboration, 3) DevOps (tools of observability, monitoring/alerting, CI/CD, operations, etc)

• with pre-sales or sales team on lead generation thru OSS channel

• with marketing team on new feature/product announcement

• with support and service engineering teams on customer support, SLA, oncall, etc

The last thing I want to highlight on is that, being unique doesn’t mean such a team should adopt a different engineering culture that is completely different from rest of the engineering team. On contrary, they should share, on higher level, the same values and same mission/visions, and on lower level details, the same performance review criteria, etc. Taking it as a divergence is a total misunderstanding of this blog and the intentions here.

Here’s the context: the very senior engineer has joined for over 6 months. Before joining, he open sourced a project at the previous company, he takes pride in it and thus ever since joining my team, he has been trying to bring in that project and grow its adoption. My team has a rule of 20% work-on-anything-you-are-interested-in time so he has been using it to promote his project. However the project turned out to be of relatively small scope, and not of critical value. there’re progress made but not a lot, not many user demand and thus we cannot commit more resources on this project. He is a bit frustrated and says he believes this is due to a different company culture.

This is the moment I realized I have to step in immediately because it has nothing to do with company or organizational culture, but that he did not fully realize where the problem is.

To tackle that, I helped him analyze the situation and status quo:

First, I clarified to him priority of the project — fact is this is not a top priority, and actually it’s not even in our priority or even backup list, it will remain a personal project and not become a priority for the team/organization until he can acquire critical users and prove there are strong business need, value, and customer demand for it. This is the most common and standard process in companies on how to bring up new initiatives and not specific to him or his project.

Second, as his manager, I’ve been and will be continuously supportive for him to use the 20% time to work on this project and grow its adoption internally, as 20% time is exactly for engineers to work on things they are interested in but not necessarily a priority.

Third, as one of the most senior people in the team, he needs to level up and think of an overall picture rather than over focus on his own little thing, in two folds: One, his official project scope is wide enough, and as a leader in the team, he needs to focus more on real priorities; The other one, he can be more artfully to push for his project rather than keeping on bringing up the project name — e.g. one of the org’s top priority is to break service silos within the organization, and his project can actually integrate and fill a gap between a couple services, so what he can strategically do is to emphasize on the integration and totally hide the project away, rather than keeping on pushing for the project using its name.

Last but not the least, let your engineering pride and ego go. The fundamental root cause of this struggling and entanglement here is that he felt the project was created by him, he is so proud of it and he gives himself a mission to drive the org to adopt the tool. Thus, upon frictions, he become frustrated. To solve the situation, he eventually has to let go the pride, and be brave enough to admit that, maybe the project is not popular for a reason, maybe it’s not bringing so much value as he originally imagined, maybe it’s ok to not try to drive the whole organization adopt it. Instead of keep on denying this fact, he needs to face it directly, as users and markets do not lie — they only pay for and ask for the most valuable things, not something that can bring “a little value” or “a little improvement”.

One example I gave him is my personal experience. Back in 2018, 2019, I was working on a state-of-art stream processing engine which had an ambitious goal and vision of unifying batch processing and stream processing architecture and use case. I’m so proud of it and so convinced that it’s going to succeed and beat the existing popular batch engines on market, like Spark, to the hell. I became blindly confident about it because the project was kind of my baby, thus blindly ignored all the other factors. At some extreme time, I had the opportunity to join Databricks but I was so proud of my project and believe in the judgement that Databricks will die as Spark is inferior to what I’m building.

Time proved I’m wrong. Databricks keeps on growing and expanding, while the adoption of my project is staggering and the OSS company behind the stream processing engine was acquired in a cheap price and eventually almost went dissembled. I paid a costly price for the unforgettable lesson that, users and markets don’t lie, and we have to bravely face the facts and let go our engineering pride and ego.

E.g. - monitoring/metrics/log vertical like DataDog, Dynatrace, Chronosphere, Splunk, SumoLogic gaming vertical like Game Analytics location vertical like SafeGraph, MapBox, Esri crypto vertical like Nansen and Dune Analytics

They collect data (e.g. metrics, logs, point of interest, blockchain) as asset, and then sell access (dashboards, APIs, etc) to you

Here are a few critical aspects to take into consider (The following may apply to all kinds of companies, but I’m being specific to data companies here):

1.Cost Scalability

For data assets company, processing and analyzing data is a must-have base cost. To successfully scale, you have to keep data infra cost growth linear while business and profits growth exponential.

The initial one is always cost of people v.s. time/speed. E.g. you don’t want to build your own infra if it’s only a 10 people team. Since it’s a well known tradeoff and topic, I’ll skip it here.

What not so obvious is the tech cost.Despite necessary passing-on cost to customers, core cost cannot grow at the same rate of core business, otherwise, it may not be a good business, e.g. in tech consulting, # of clients you can serve is linear to # of consultants you hire — this is not a good business. How to scale cost and value disproportionally at data asset company? It’s case-by-case. We should understand how vendors charge you first. Vendors can charge by:

a) constants, e.g. $/license. It’s like a SaaS model, usually applicable to low data volume areas like metadata and security management, etc. In general should be ok to just swallow them.

b) usage, e.g. $/sec. This is the most common pricing model in infra areas, e.g. compute service like AWS EC2 charges by $/core*sec (or /instance type), storage service like AWS S3 or GCP Cloud Storage charges by $/ MB or GB stored. It’s likely the cost you pass-on to users.

c) hidden! This is the worst. Unlike above two, you don’t understand how it’s calculated and have no control. E.g. GCP BigQuery can charge by amount of data scanned. It’s bad in every single way — you can do nothing about it (e.g. barely no optimization can be done), and vendor are not motivated to optimize and reduce the amount of data scanned. If your core business highly depend on BigQuery, give it a second thoughts.

Once understanding your dependency cost model, then you can analyze your business and do a simple projection see if your business can outgrow the cost. Eg, it doesn’t make sense for Datadog to use a 3rd party time series database for metrics, they have to build and down this core piece themselves from cost and other perspectives.

2. Control your own destiny: Vendor Lock-ins

Vendor lock-in first comes with pricing and technical barriers. How much negotiation power do you have over vendors? If vendors said they are gonna raise the price due to inflation, are you able to counter that ask and move off quickly if necessary? I won’t expand here as there’re plenty of content online of this.

I want to highlight on non-monetary/technical areas where people usually do not have correct assessment if they’ve never been there. E.g.

• How soon do vendors deliver on feature and bug-fix requests?

• How well do they provide support? e.g. docs, SLAs, response time, oncalls

• How aligned are your roadmap with their actual roadmap? How much can you impact their roadmap and management chains?

Note that these are also closely related to your contract $ amount and thus company size, as well as vendor’s size and stage. E.g. a big customer may be better treated by a small vendor, or by a large vendor if you yourself are a renowned brand. But to be frank and yet crude, I’ve seen from time to time even big companies I worked at are ignored by vendors as big as AWS and as small as startups, e.g. slow response, slow execution, never deliver on features and bug-fixes which resulting in us building internal solutions to replace them, let alone small companies and users. So be prepared and ensure you can tolerate all these hurdles if you decide to use a vendor.

3. System Composability and Extensibility for Future Growth

Vendors usually put in barriers to defend themselves from competitors, but your data strategy only works when it’s composable.

Be aware of what you can do and cannot do down the road with a vendor. E.g. vendor A may never connect to vendor B, vendor C will never allow you to replace or plug in component X,Y,Z inside their systems. Such things will can be a show stopper — if the query engine you purchased cannot read data from one of your data store, that’s very bad. So, avoid short sighted engineering decisions, and take these factors into your early decisions.

4. Team Expertise and Hiring

If your early team only have data engineering experience, use a vendor solution, and design it to keep retreat routes open.

A big misunderstanding of using data infra vendors is that people think they don’t need a platform team anymore since they buy vendors services and imagine that vendor can take care everything. That’s a truly mistake and false assumption to make. https://www.safegraph.com/blog/scaling-data-as-a-service-daas-with-platform-engineering this is a good example of why you still need platform team even with a vendor solution.

Coming to build your own platform, even if you want to, how skilled is your team? Building a platform or infra is not sth someone can learn overnight, you’d better have someone did it before to lead the effort.

What’s more, what’s your hiring plan look like? Start to hire talent to prepare in advance. If you only start to hire when an immediate need is there already, that’s too late, as hiring a qualified lead and a starting team can take you more than a year.

5. Hidden Cost, of Even Free Software

Be aware of the hidden cost when trying to leverage OSS software. They may appear to be open and free, but some projects may be a trap. E.g. Delta Lake only open sources the very basic functions, and Databricks is keeping all advanced features private in commercial version. So if you decide to use Delta Lake mainly because it’s free, think twice.

6. Taking Proper Tradeoffs According to Company Maturity, Timing, Resources

Ultimately, the decision makers should take proper tradeoffs based on the situation, e.g. maturity of the business and company, business demand, etc.

Understand the tradeoffs you are taking and make the right decisions accordingly. E.g. if you are a small startup that is still looking for product market fit, your backend infra is certainly not top priority. You thoroughly understand the pros & cons and tradeoffs you are taking at that moment (e.g. velocity, cost, flexibility, performance, scalability, etc), and must have a long term plan in of what is the ultimately right way, and how to gradually shift to that direction.

Well, the Rome of data platform isn’t built in a day. The architecture is super nice but no one can get there single-handedly. It takes strategy and time to plan and make it play out. I want to share my experience and thoughts on how to lead and build a strategy of data platform that can actually make all the goods in that article to happen.

1. Business Oriented (Business Driven) and Create Alignment

The first thing I have to call out is that, though that architecture is great and attractive, do not take it as your whole world by aiming for it in day one, and forget what’s right in front of eyes — the business. The platform can be internal or external facing, can be fully vendor-based and self-built, but the most critical thing is always align with your companies strategy and keep in mind that data platform exists to serve business.

It should emphasize on business priorities, and pride itself to empower and enable business. It should not be built in a silo that floats outside of core business — otherwise, no matter how good your data platform vision is, the strategy fails.

One example I experienced is when in a startup, I witnessed how its infra strategy failed due to infra takes precedence over business. It’s a general infra not data infra failure but still relevant. It’s a marketplace and business priority at the time is to bump sales/profits to be self-sustainable. The tragedy started with platform engineer team who were responsible for building micro-service foundation for product engineering somehow became obsessed with K8S. K8S was new and exciting at 2017, but unstable and no one actually knew how to run them. The platform engineering somehow shut their ears, isolated themselves and started to build K8S, while ignoring the whole company’s business priority. As you can imagine, the K8S effort delivered no value and failed to empower product engineers, the infra team left, the business goal was impacted, all because manager of that team failed to align their team’s priority and goals with the business.

So, put your business first. If your core business cannot grow or survive, forget about data platform.

2. Talent, Team and Hiring Plan

A technology platform can be as good only as the talents who build, manage, and operate it.

Once the business goal is determined, next is making sure we as a leader put a solid team in place. Hire leads first in key areas, and let them build the team.

3 pitfalls to watch out for:

a) It can be a big mistake to think there’s no need for a platform team if you are using vendor solution. https://www.safegraph.com/blog/scaling-data-as-a-service-daas-with-platform-engineering this is a good example of why.

b) Hiring junior engineer to balance the team usually should happen at later stage with key seniors are already in place. Try avoid doing it reversely, otherwise you’ll burn yourself out to hand hold them. I’ve heard excuses from startup founders, like “oh, our startup is just series A, it’s so hard to hire senior people so we can just attract and hire junior folks”. That is BS. That attitude can lead to the same result wherever they are, small or big companies. The right way to think of it, is that you haven’t hired the key leader who can hire more senior engineers. Once such a key leader come in, hiring should become much easier. BTW, if you had a hard time finding such a leader, something may be wrong — e.g. uncompetitive comp, or maybe the startup’s business is just not good enough to attract people (99% startups fail anyway so be brave and face it :)

c) Act early. Do not wait till you really need such someone then start to hire them. Start early as it can take months to source and hire the proper people.

3. BI, AI/ML, etc- Deliver the Last Mile of Data Value End-to-end

Building a platform is not end of the story, data is only valuable when turning into certain form of product, e.g. BI and dashboards for visualization, AI/ML algorithms for recommendations and forecasting, events (alerts, messages) in the case of streaming.

More often than not, leaders of the platform should take the lead in filling the last miles, rather than waiting for product engineering, because:

a) Your data users may not have the motivation to do so. What data users want is straightforward value, not the plumbings, e.g. they usually do not want to figure out what BI tools to use, how to connect their dashboard to data sources/warehouses, how to set caching or refreshing strategy. They’d expect data platform has already done so for them, thus your solution either works or not work for them at all. In this case, the data platform team can build a few best practices, e.g. 1) set clear responsibility boundaries and expectations upfront 2) continuously educating users 3) have good documentations and guidances 4) pre-config params or other things to the best general values or situations for the users

b) Your data users may not even know what they can do with the data you provided. Thus data platform has to step forward and tell the users what magic they can create, e.g. “you can use dataset A in warehouse B to analyze X”, “use dataset B and infra C to build ML model for use case Y”, “join data D and E to get Z”, etc.

4. Looking Ahead, not Following Behind

A solid strategy should account for potential growth in 3–5 years internally, and keep close eye on industry trend externally.

• If there’s only data warehouse and structured data now, what about data lake and un/semi-structured data?

• If there’s only offline data processing now, what about streaming and stream processing to reduce data latency?

• If there’s only small traffic, what about the volume in next 1–3 years and 3–5 years

• Does it make sense to bring vendor solutions in-house for flexibility, cost, or other reasons?

• If entering new markets, what about the data privacy and compliance laws there?

• How to make all data easily discoverable and leveraged to generate value?

• Where is the industry going, and what are the new technologies? ……

Looking ahead is hard, and any preparation certainly cannot be comprehensive enough to cover all details. But the key is to have this mentality and habits of thinking of these aspects, so we are better prepared when things come.

5. Technicals— Scalability, Composability, etc

They are discussed in this post.

In short, we can summarize the aspects to be — figure out

• what you should do it for? and why?

• who can help you do it?

• who you should empower and collaborate with? and how?

• what’s potentially next?

• what you need to do now? and potentially next?

The tech trend in the past few decades has always been that innovations from US are exporting to the rest of the world, including China, with big impacts.

It’s the same in infrastructure software field in China. Open source software originated from US and EU flourish in adoptions at all sized Chinese software companies. Papers from US have inspired many Chinese companies, large or small, to follow the ideas and build their own versions. Chinese markets are pretty much dominated by original ideas from US in the past 20 years.

As internet and tech industries has exploded and been maturing in China, I’ve seen a new trend in the recent 5 years where Chinese software companies start to innovate with original ideas. Top-notch infrastructure, in both open source and commercial space, emerges from such market to solve its unique challenges. Such innovation and contribution also start to influence the US and EU markets.

One of the best example in commercial area is OceanBase. Developed for AliPay in Ant Financial, it’s designed from ground up to handle extremely large amount of online transactions for the global e-commerce giant Alibaba, at an unprecedented scale. It completely replaced Oracle throughout Alibaba and Ant Financial in 2016. In 11.11 global shopping festival in 2019, Alibaba made $38.3 billion GMV in a single day sales, and OceanBase handles 610,000,000 transactions per second at its peak traffic as the core transaction infrastructure. It also ranked №1 at TPC-C benchmark in 2019 and 2020.

There’s also open source examples. Apache Flink is an open source stream processing framework that was developed by a Germany crew. It was not really taking off until Alibaba adopted it in 2015 and made many fundamental production-critical changes which reshaped it more and more towards a SQL-based scalable streaming database engine.

TiDB from PingCAP is another good example. Inspired by Google Spanner paper, TiDB is built from scratch to tackle distributed transaction challenges and are widely adopted in many of the biggest Chinese tech companies, where it’s battle-hardened at a much larger scale than its US peers CockroachDB and YugaByte.

Apache Kylin and the commercial company behind it, Kyligence, are among the best BI cube solutions on the market.

Going to US and EU!

As these tech companies, from tech giants like Alibaba to independent ones like PingCAP and Kyligence, grow, they are all starting to look at overseas markets in US and EU. As US enterprise customers are more used to pay for services enterprise software and whereas Chinese enterprise customers tend not, they realize that be constrained within China market won’t meet their growth and revenue expectations.

Investors have been pushing PingCAP hard to come to US. PingCAP has recently expanded quite a bit of its silicon valley office, hired a global head of solution engineering, and launched TiDB cloud on AWS and GCP. Square is said to be considering adopting open source TiDB as its current Vitess solution has many hidden bugs and the company cannot bear with it any more.

Kyligence is one of the first Chinese tech companies that set up an office in US. And Alibaba also has AliCloud in US.

We believe one tech trend in the next decade is how innovative technology from China are coming to US/EU and influencing the world.

This phenomenon is already happening in consumer tech field very successfully. Musically and TikTok are taking over the short video space globally like virus. CleanMaster from Cheetah Mobile is among the most popular cleaning app on Android.

However, the reality is that the going-oversea strategy hasn’t played out very well yet in the enterprise domain. Until this point, PingCAP and Kyligence only have a few scattered paid customers in US and EU. Alibaba is not expanding its cloud services and data centers in US and EU any more.

Where are the problems?

I’ll just list 5 problems I observed here.

Politics is definitely among the biggest ones. Infrastructure software is fundamental to companies. Companies, especially those in US, don’t want to have a Chinese vendor hosting their data for security and compliance purposes influenced by the political conflicts between US and China. So having a hosting service under a Chinese entity’s name or its subsidiary’s most likely won’t work. This week US just announced to discourage users using Chinese cloud providers.

Second, these Chinese companies don’t have full footprint in the US and EU market. Selling enterprise software requires a full functional team including not only developers, but also marketing, pre-sales, sales, support, legal, etc. The two factors of 1) the companies aren’t willing to or cannot afford to invest heavily in US/EU and 2) they cannot acquire paying customers, seem to form a negative reinforcement loop, and becomes a chicken-n-egg problem.

Third, Chinese companies are lack of the influence that US companies usually have in their home court. The influence covers a pretty wide range, including developer community, relationship with the tech influencers, ecosystem (especially cloud ecosystem to integrate with US cloud providers’ services), partnership with ecosystem companies, help of introductions from VCs or networks to Fortune 500, etc.

Fourth, enterprise software requires a broader ecosystem integration to function well. Software never works alone, they need to be able to work with all kinds of other software in order to deliver a larger value. Such ecosystem integrations come down both open source software as well as cloud services. The open source software ecosystem is easier to integrate, as it’s basically standard across borders. In big data, it’s basically Hive/HDFS for big data storage, Spark/Flink for big data processing, Kafka for pub-sub, HBase/Cassandra for k-v services. These tools are widely used in Chinese companies, thus the infrastructure software vendor actually can and will integrate with such software well if there’re customer requirements, regarding data connectivity, sharing, ingestion, or migration. On the other hand, cloud services are pretty segmented between markets and thus are totally different. Kinesis is the de facto pub-sub service on AWS, no one uses AWS/Kinesis in China, and a data software won’t work in US if it can’t ingest data from Kinesis. Such cloud service integration are crucial to localizing any enterprise software to US markets.

Last, the requirements of none-core components of the software, like user experience, UI, security, etc, are very different between China and US enterprises. Chinese companies focus more on speed and growth, they are willing to try new technologies, and generally pay less attention to details like UX, UI, security. They can tolerate enterprise software that doesn’t have nice, intuitive UX and UI. US companies, however, demand a lot more ease-of-use. The labor is expensive, they would much rather have fewer people finish more tasks. Thus US enterprise vendors put lots of emphasis on the design, and they are really proud of that. If a software cannot be tried out with a 5-min quickstart, it pretty much claims the end of the deal. Such difference in standards and requirements makes Chinese software, without much rebuilding, almost impossible to fit US market .

Thus, even though these companies have best-in-breed technology that’s better and much cheaper than their peers, they haven’t be able to take a share in the market.

Advantages of Such Software in US/EU market

Cheaper.

Better Performance and Scale.

Battle Tested.

Possible Solutions?

People might argue there are possible solutions.

For example, if US companies aren’t willing to use hosted service from Chinese companies, what about just offering an on-premise version where the users can have full control over the software?

I’m not sure that’s going to play out as it’s comes down to many other factors mentioned above when US companies make decisions. E.g. how well it integrates with the other software to fit into their stacks? what about support services and well-written documentations? Let alone that they have to be willing to and have resources to host the software themselves.

This situation actually reminds me of the challenges that US companies have ran into when trying to enter the China market and their methodology for solving it. It’s well known that US companies have been having a hard time to get into China market since early 2000s, like the failure of Ebay and the retreat of Google. Lessons from lots of failures boil down to not being able to fully localize the business in many areas like policies, user experience, different ways of doing business and keeping relationships. The best way out turned out to be partnering with local companies and set up a joint venture.

There are also some clues in the consumer tech field in US that may help. $20 Wyze Camera, co-founded by Chinese entrepreneurs, is taking over smart home by using out-of-shelf cameras launched by Chinese smart-home startups and localizing its app, cloud services, and user experience to the US market. Wish the online shopping retailer, co-founded by Chinese entrepreneur, thrives by localizing the app and sales channels in US and then helping Chinese manufacturers, who used to only sell on Alibaba or Pinduoduo in China, to sell products in US.

If you think from manufacturing cases, such a re-branding and localization is not a surprise at all. It’s actually so common that it’s happening on a daily basis. Brands from US/EU place manufacturing orders to China factories, have products shipped overseas, and distribute in their own markets.

The recent case of what happened between Tiktok V.S. Zoom in the US also reveals some clues. Tiktok actually has been fully following US laws but is still ordered to shut down or for sale. Whereas Zoom is having almost all its engineering team in China, and even had servers in China a couple years back, and Zoom still flourishes in the US, creating record high stock price everyday. Technology is important, but business aspects are just of the same importance, if not more.

I believe, for such a Chinese software infrastructure company to succeed, the route should be like this:

1. Set up a US company with founders being Chinese Americans, even better if there are Americans co-founders, have US VCs fund the company, and thus Americans hold majority of the company. The Chinese company may have a small share in this US company to keep the relationship.

2. The US and Chinese companies clearly separate their markets and do not get into the other one’s. E.g. the Chinese company takes care of all Asia, and the US company takes care of US, EU, Australia.

3. The US company licenses the technology from Chinese company with a low fee (almost free), and sell the product in the US in its brand.

4. The US company is a full functional company, including pre-sales, sales, marketing, R&D, support, etc, to fully address the US/EU market. The R&D may initially include localize the product with user experiences regarding UIs and management flow, integrate it with US/EU technology ecosystem (e.g. cloud services), etc. As the US company grows, it will has more power to ask for source code of the core product.

5. The Chinese company can reuse the customer success stories which the US company as its own customer success story, and market the product in China, East Asia, and South East Asia markets, to gain more profits in its home court. Any success story from US Fortune 500 can be very attractive and convincing to customers in Asia, and such success stories are actually strategic assets which offer huge competitive advantage for these Chinese companies competing with each other in Asia market. When you sell a product to Chinese banks, how convincing it will be if you say Chase and CitiBank already use your product? It applied to all industries. I bet the deals will come faster than you can handle.

Ok, ok, I still will not join an “AI” company. Here’s why.

First, “AI” has become a buzz word now for marketing to win users/customers and attracting VC to raise money. The hype it created in both tech and VC industry is unhealthy and unsustainable.

What does “AI” even mean in different context? Are you just talking about some statistics model and machine learning algorithms with supervised/unsupervised learning? Or a long-term, end-state vision in computer? Or some big data processing and analytics that can help bring data insights? Or a marketing term you use to attract attention for whatever reason? Or a term that you just unintentionally use because you don’t know what it’s exactly either but feel it’s shinning and cool to say?

Avoid a crowded place — when everyone, even people don’t know computer science/programming start to talk about it as if they are experts, you know this thing is at a local top on the market.

Second, “AI” is overexaggerated in its impact and magic. I have friends work at cutting edge tech companies like Facebook, Amazon, Alibaba, Pinterest, etc, and they are the data scientists and AI engineers. What do they say? They say only the simplest ML algorithms works the best, e.g. k means, k nearest neighbors, clustering. These algorithms are so dead simple that you probably cannot even call them “AI”, but just “statistics”. I have close friends who work at social network company and told me their “AI” production system is usually updated once a year, if ever, as it’s so fragile that any attempt to improve these simple statistics model will actually make the result (e.g. user engagement rate, click-thru rate) worse according to real A/B testing! Hilarious, right?! If these top tech companies cannot even harness or extract the power of “AI”, how can rest of the world?

Last, “AI” is only applicable when there’s a real business use case, and “AI” itself is not scalable enough to be the main business of a company. “AI” without a concrete business use case is useless, you have to apply “AI” to scenarios like online shopping, content recommendations (video, text, photos, etc), industry automation, etc, in order for it to be useful. Also note that, a successful AI strategy will be super unique from company to company, and most likely not transferable, even for companies in the same industry. Eg, in e-commerce, what works for Amazon likely will be vastly different from what actually works for Shopify, Walmart, and Etsy; Same for Facebook, Snap, and Tiktok. I think of this as “context-driven” “AI” because there’re a huge set of factors that can impact the final deliverables, e.g. business models, leadership background, company culture, business priorities, hiring budget, existing team’s skills, specific use cases, etc. So in practice, due to this complexity, there is never and will not be generic, scalable “AI” strategy or business. Any attempt to do so will likely going into the business of consulting firms which has been proven to be human-heavy, unavailable business like Accenture.

Do note that I’m talking about “AI” itself, not “AI” infrastructure. The infrastructure can indeed be generic, e.g. notebooks like Jupyter, framework service like tensorflow, TPU/GPU/CPU, etc, as they are actually not “AI”,. Such plays are by nature SaaS, cloud service, or hardware, so selling “AI” infra can be profitable, like what GCP/AWS/Nvidia/AMD have achieved. Selling “AI” itself is not scalable nor generic nor profitable.

An example is C3.ai whose stock tick is actually “AI” — it dropped 85% since IPO while US stock index on average only dropped like 10–20%. Unlike growth stock or crypto, I don’t expect its price can come back.

一个明显的case是，我团队里有一个员工，能力很强，产出很高，但是前提是维护成本也很高。

作为一个小团队的老板，我把员工分成两个坐标四个象限 （即四种）

• 坐标是output(产出) v.s. maintenance (维护成本）

• 四类四象限是：

  • 低产出，低维护

  • 低产出，高维护

  • 高产出，低维护

  • 高产出，高维护

（低产出，高维护）明显是要开除的，这类员工不开就是累赘，有信号以后要早早跟老板反映，尽快开始PIP。一般开掉这类员工都问题不大，因为他们的缺点如此明显，以至于你老板肯定都知道。开除更多的是时间问题而已，有些人识趣自己就走了，不识趣的需要3-6个月走PIP。

（高产出，低维护）明显是人才，这类员工是团队的顶梁柱，不太耗费你的时间，但是能分担你很多工作量。作为manager, 要帮助这类员工规划好职业路径，想方设法留住人才并帮助他们成长。

剩下的两类就有意思了。

（低产出，低维护）这类员工一眼看上去是要干掉的，特别是对新manager来说。但是，作为一个有自身经验的manger，就需要仔细琢磨了。首先，什么叫“低产出”，这个指标你需要仔细衡量下。大部分“低产出”都是跟同组里强人的“高产出”相比的，但这不一定是最合理的，比如，这些低产出是否是因为员工没有相关背景因而需要时间ramp up？是否是因为家里情况不一样？（一般年轻没小孩的员工会比有小孩的员工产出高很多）是否是因为整个组太强，但这些所谓低产出在业界也是中游偏上的？当然如果是真的超低产出，那开了没话说。但如果不是？

其次，低维护代表了这个员工没有找事惹事，有自知之明（self-aware），知道自己几斤几两，明白自己想要什么，那其实对manager来说到是一件轻松的事。

（高产出，高维护）这类员工一眼开上去是要保留的，特别是对新manager来说。但是，作为一个有自身经验的manger，也需要仔细琢磨了。这类员工其实是最难搞的，我团队里就有一个。这类员工的特征是，高产出并容易得到各种认可，经常是团队的光环，也许你老板的老板都觉得他很厉害。但同时这种高产出是建立在你的痛苦之上的 - 比如他们无法和其他团队成员融洽相处，需要你作为manager经常介入调停；比如他们在细枝末节上经常较真，固执己见，需要你作为manager花比别人多几倍的时间去跟他们解释，让他们信服；比如他们经常喜欢高成就的工作，独享荣誉，不留给团队其他人；比如他们经常不能静下心来在一个领域做深入，经常喜欢东瞧瞧西看看，尝试各种新东西，浅尝辄止；比如他们经常有套自己的优先级，而不顾团队优先级；等等。你碰到的情况经常可能是上面问题的一个子集，而非全集，要不你早把他们开了，还留着他们是因为高维护的问题不是每时每刻都有的。但是这种时不时的高维护会让你搞到心力交瘁，浪费时间和生命。这种时候，说明你要对这个人下手了。

处理这类员工最棘手的如何告知你的老板和老板的老板，因为告诉上边一个高产出高光环的人不适合团队是件很难的事。我的教训就是这件事说晚了，等我手下人要走的时候才告诉老板，我老板就很疑惑，说”你怎么不早说“？对这类员工，要我们不停的跟老板说，一遍又一遍，才能建立起基本认知，改变他们在整条管理链（management chain）上的形象，从而为对付他们留出空间和余地。

一种普遍的处理手法是，不对这种人进行晋升，一般他们都会心高气傲，导致自己就走了或者转组。

这两类员工之所以值得额外关注，是因为打眼看上去，直觉好像前者要开除，后者要保留。但事实并非如此。除了产出外，员工的维护成本是个很容易被忽视，但相当重要的维度。manager在产出和维护上要做tradeoff，特别是在维护层面上，进而这两类员工的去留就不那么明显了。

比如低产出低维护的员工，对团队来说，从分布上来说，不可能每个人都是高产出，manager要心知肚明。如果你团队里的人每个都是高产出，那反而可能造成团队结构的问题，比如晋升机会就那么几个，谁该晋升？非要将军里面挑矬子，那么总有人会离开的。如果这些人都不需要你花太多精力，而且产出也都说的过去，其实完全没问题。团队里必须有层次，有分档，即有强人，也要有新人老人，才能组建一个平衡、持久、融洽的团队。