These are the “market-goers” at Alibaba who clamored to do open source.
In the circle, open source was once called a “bazaar.” Compared to the closed and strict traditional enterprise IT culture, open source emphasizes free and open community culture. InfoQ has tracked Alibaba for many years and found several key “market-goers” in Alibaba’s entire open source development process, attempting to understand the open source history of this representative Chinese tech company over the past decade.
In the past ten years, Alibaba has experienced the growth of Taobao, the establishment of Alibaba Cloud Computing, IPO, and the ever-climbing traffic of Double 11. Each stage had its significance and goals, and open source showed different states in different periods with these changes. What remains unchanged is that this group of “market-goers” at Alibaba explored a path of independent open source and walked it for ten years.
The Pioneers of Open Source
2008–2010: Dubbo emerged, Fastjson was brewing, and a group of developers with open source spirit joined
2008 was a complex year for Chinese people, with ice disasters, major earthquakes, and the Olympics coming one after another. This year, the Dubbo project was born, though it wasn’t called Dubbo yet—it was still an internal system at Alibaba. Everyone probably couldn’t imagine that this project would become the best “witness” of Alibaba’s entire open source history and be welcomed by so many developers ten years later.
Dubbo founding team
This year, Yubo, the core promoter of the Ant Design project, left the Institute of Software at the Chinese Academy of Sciences in Beijing before the Olympics’ temporary population screening. With his years of experience in program development and optimism about the emerging frontend industry, he came to Hangzhou to join Taobao UED—the best frontend team in China at that time.
This year, Gao Tie, the core promoter of Fastjson and Druid projects, was still developing some basic components in Java at Kingdee. Mo Wen, now the core promoter of the Flink project, had already been fired up by Jack Ma’s passionate speech at a campus recruitment presentation and subsequently chose to join this company that didn’t have many people in Beijing yet. Gu Jin, the main author of the “Java Development Handbook,” didn’t know yet that he would rely on “forcing an interview” to enter Alibaba upon graduation.
In 2009, the Dubbo team’s main work was fixing bugs. In 2010, the entire team felt this architecture was overwhelmed, so they decided to rewrite it.
The arrival of industry technology leader Dr. Zhang Wensong brought his years of open source ideals to Alibaba. He believed that open source not only benefits the technology itself but also helps technology people grow. With Dr. Zhang Wensong’s strong advocacy and support, open source culture gradually sprouted on Alibaba’s soil and had robust vitality. At that time, both Taobao and Alibaba B2B had many teams promoting open source. Alibaba B2B decided to open source the Dubbo project first, so in 2011, the entire team systematically organized the documentation but didn’t conduct large-scale strong promotion.
Overall, during this period, Alibaba’s entire open source was still in a hazy state, but there was already preliminary exploration, and a group of engineers with open source spirit joined one after another. Jack Ma also poached Wang Jian from his position as Executive Vice President of Microsoft Research Asia and began planning how to solve the computing power problem, which was a major crisis facing the entire company at the time, and Alibaba Cloud was born at this opportunity.
On September 10, 2009, Alibaba Group celebrated its 10th anniversary and simultaneously established Alibaba Cloud Computing. This year, not many people thought cloud computing could succeed, let alone promote open source development.
A Host of Open Source Projects Emerged
2011–2012: Dubbo had more and more users, Fastjson was officially open sourced, and began building a big data system based on Hadoop and HBase
“After Alibaba Cloud was established, what I felt most deeply was the establishment of the self-developed technology system (Apsara). This was the first time I strongly felt that Alibaba attached great importance to technology. This stage also cultivated many talents with strong technical capabilities and made me feel that Alibaba’s technology system was becoming more and more open,” said Mo Wen, the leader of the big data real-time computing engine Flink.
Yubo, Gao Tie, Gu Jin, Mo Wen… These developers with technical strength and strong open source sentiments joined and spontaneously set off a wave of open source within Alibaba. During this time, a host of good open source projects appeared within Alibaba: Fastjson, Druid, Sea.js, Arale, etc.
“When I first started software development, I didn’t participate in much open source work, but I always had contact with Linux and was influenced by the open source spirit. At that time, I felt open source was a very great thing. Alibaba’s internal open source projects at that time mainly came from the Taobao series and B2B series. The Taobao series mostly chose GPL license, while the B2B series chose Apache license more. I also chose Apache license for Fastjson. When doing Druid, I didn’t actually think about open source, but it happened that the B2B platform had an open source plan, so it was open sourced together,” Gao Tie said.
Alibaba’s first generation of open source people, FastJson & Druid founder Gao Tie
During this period, Alipay on the other side was also trying to escape the cycle of reinventing wheels. The fragmentation of the technical environment created a trend of frontend basic technology and many positions. Like most frontend teams in China at that time, Alipay was also reinventing its own frontend wheels, heavily borrowing from YUI’s component architecture system, using MVN in the Java ecosystem for dependency management and building at the bottom layer. All wheels and nails had to be made internally, getting further and further from the frontend community, and the solution was open source.
On April 29, 2012, Yubo posted on Weibo, announcing to build Alipay’s next-generation frontend framework in an open source way, which was very radical at the time.
Later, the Sea.js/Arale/spm suite began to develop within Alipay and in the domestic open source community.
At the same time, Dubbo also welcomed a small peak in development, with more and more users, including those doing cars, securities, cement, electrical appliances, and even companies willing to pay for Dubbo, hoping to ask the team’s developers for help when problems occurred.
During this period, not only did many excellent open source projects appear, but Alibaba developers also began actively contributing to open source. Mo Wen recalled: “At the beginning, our entire team didn’t start doing open source from Flink. As early as 2010, we started using open source Hadoop and HBase to build a big data system for processing data in search scenarios and contributed our practical results to the community.”
Compared to the open source business progressing like wildfire, Alibaba Cloud’s progress was not smooth. From 2010 to 2012, Alibaba Cloud went through the most difficult three years. Due to no progress in results, Alibaba Cloud’s entire department received the lowest scores in the group for consecutive years. Resignation letters and transfer requests appeared in Wang Jian’s mailbox one after another. Almost everyone was convinced that Alibaba Cloud couldn’t build a cloud computing system.
At this time, the whole country didn’t have much feeling about cloud computing, but was quite concerned about mobile internet development.
On Double 11 in 2012, Tmall and Taobao’s total transaction volume rose from 5.2 billion RMB in 2011 to 19.1 billion, breaking the global online shopping single-day transaction record. Alibaba sat firmly in the dominant position of China’s e-commerce era.
At this time, the wave of mobilization came.
Before many people could react, WeChat achieved over 100 million users in March 2013. At that time, Taobao also made a mobile version, but the positioning was more “multi-terminal,” meaning mainly on computers, with mobile terminals only able to achieve some simple functions. Everyone felt the huge changes that mobilization would bring, and Alibaba also began seeking change.
In 2013, all of Alipay and even all of Alibaba’s business platforms came to a crossroads. The group announced: ALL IN Wireless.
A transformation thus began…
Huge Changes Coming
2013–2016: Alipay frontend disbanded, the group strengthened One Company and began architecture adjustments, many projects merged, open source progressed slowly.
Before the “ALL IN Wireless” strategy was announced, frontend and business were very closely combined. UED and design had high say in the company, with a frontend performance optimization campaign every half year. In the “Those Years in the Experience Design Department” series of articles, this period was described as:
This was an era where one line of JS code error could cause a decline in site-wide transactions, and it was also the last golden age for PC frontend developers.
After the “ALL IN Wireless” strategy was announced, many people were moved from their original departments to the front lines of the battlefield. Some people didn’t adapt and left, with others filling in later. Alipay’s frontend development department of about 60 people faced disbandment, with more than half of the people transferred to support Alipay’s wireless business. Facing huge changes, the entire team was demoralized. Everyone faced very difficult choices: stay or leave?
Ant Design, AntV, Egg and other frontend open source project leader Yubo
In the end, Yubo decided to stay in this team with the remaining 17 people, supporting PC business whose main iteration was adding QR codes to various pages and the self-developed frontend technology system that had entered a dead end. At the first weekly meeting after the adjustment, everyone was silent. Compared to those who summoned the courage to make the choice to leave, those who stayed had more of an attitude of the unknown and embracing change.
Besides personnel adjustments, the Alibaba Group at that time wanted to strengthen One Company and began architecture adjustments. At the technical level, the entire company was unified, hoping not to have duplicate construction. Any identical projects had to be merged. Taking Dubbo as an example, Taobao at that time had a project called HSF, which was also a middleware service framework, highly overlapping with what Dubbo did.
The author of the HSF project, Lin Hao (Bi Xuan), was also a well-known technology leader in the Java field in China at that time. When OSGi was very popular, Bi Xuan might have been one of the people in China who could explain OSGi most clearly. HSF and Dubbo, although doing highly overlapping things, had different design philosophies. Although there were some collisions, the ultimate goal was still “strong-strong combination.”
Not long after, the Dubbo team adjusted and went to various places. From the outside, the Dubbo project never updated after 2014.
The entire company’s changes caused many open source projects’ maintainers to change. Some projects progressed slowly during this process, while others stagnated.
After these adjustments, Alibaba welcomed a milestone year.
On September 19, 2014, Alibaba Group officially listed on the New York Stock Exchange with stock code “BABA.”
In the few years after this, Alibaba’s business entered a high-speed development stage: completing restructuring with Ant Financial Group, with Ant Financial becoming Alipay’s parent company; Alibaba Health became a subsidiary of Alibaba Group; Alibaba Group and Ant Financial Services Group jointly announced the establishment of “Koubei”; announced the establishment of Alibaba Music Group; acquired Youku Tudou Group; officially established Alibaba Digital Media and Entertainment… A large number of developers began putting their energy into business development, with hardly any time and energy to do open source, especially when this project had nothing to do with personal KPIs.
At the same time, Mo Wen, who had been working in Beijing, also encountered problems. The business wasn’t doing very well, so he decided to pull another partner to study Flink together. At that time, Mo Wen was in Alibaba’s core frontline battlefield—the search team. Because he felt “fast” was a very important trend for the future, he firmly invested in real-time computing research. As mentioned above, when the Flink project first started, there were only two people.
Mo Wen said: “A project definitely needs resources, that is, people, but at that time, it was too hard to find a few capable people with time in Alibaba. Alibaba’s internal competition was also very fierce, business developed very fast. From writing the first line of Blink (Flink’s internal version at Alibaba) code, there was only about half a year to prepare, and then it needed to support the next year’s Double 11.”
While the entire Flink team gradually iterated and slowly developed, Gu Jin also began the long promotion road of the “Java Development Handbook.”
“Java Development Handbook” main author Gu Jin
“In March 2016, I published the first version of ‘Alibaba Group Java Technical Specification’ on ATA (Alibaba’s internal technical forum). Some people thought it was good, many thought it wasn’t. Some people thought I was too bold and posted on Zhihu asking who Gu Jin was and how dare I write technical specifications for the entire Alibaba Group?
Every time I was ridiculed, I’d ask the cafeteria auntie to give me two extra chicken legs for dinner to comfort myself. Criticism proves there’s attention, and attention means hope.
At that time, I went from department to department to lobby. Failed once, tried a second time. Failed 99 times, then try once more. Nothing else, just to make it a round number.”
During this process, Alibaba gradually developed open source from the earliest individual behavior to organizational behavior, but hadn’t yet launched unified planning at the group level. Many open source projects also gradually got lost in the collision with commercial value, unable to find the motivation to continue.
Contrary to the low ebb of open source, Alibaba Cloud was boiling. In 2015, at the Olympics of computing Sort Benchmark, Alibaba Cloud Computing sorted 100TB of data in less than 7 minutes, cutting the previous world record of 23 minutes created by Apache Spark by more than half. The result was obvious: Alibaba Cloud became the first company in China with complete cloud computing capabilities.
Full-Scale Attack
2017 to present: Open source fortunately, cloud explosion
In 2017, Alibaba’s Double 11 for the first time enabled a transaction dashboard and live-streamed globally. Watching the numbers constantly scrolling on the screen, the engineers in Mo Wen’s entire team were extremely nervous because this was the first time Flink supported almost all of Alibaba’s Double 11 core business.
“Before Double 11, we didn’t widely publicize this because we didn’t know what the result would be. As soon as Double 11 started, various battle reports came out—how long it took to reach 10 billion, how global transaction volume was distributed, which category ranked first, etc. These rankings were all based on Flink, which was equivalent to our first big test,” Mo Wen recalled.
During the National Day holiday before Double 11, the engineers in the entire team didn’t take seven days off. They spontaneously came to the company, even at home doing various debugging and modifications. This state continued until the countdown to Double 11 began, and the entire team felt they had reached a stable state and didn’t need to adjust anymore.
When smoothly passing Double 11, at that moment, everything felt worth it.—Mo Wen
The story after that, everyone knows: Alibaba acquired Flink’s parent company Data Artisans and decided to push all modifications back to the community one by one.
While the entire Flink team was fighting for Double 11, Bei Wei, born in Jiangnan but with the temperament of a Northeastern man, led his team to decide to restart the Dubbo project. Dubbo’s turning point was the explosion of Alibaba Cloud.
Bei Wei who awakened the sleeping lion “Dubbo”
In 2017, Alibaba Cloud found that after a batch of customers went to the cloud, they wanted to use Dubbo. Because they were already very familiar with Dubbo, they didn’t want to be forced to change their usage habits because of going to the cloud. Real customers raised requirements, and improving customer confidence in Dubbo became something valuable at the company level.
Learning from previous lessons: on the open source path, one person can walk very well, a group of people can walk very far. Must rely on community power for joint maintenance. Bei Wei’s entire team made a major decision: donate Dubbo to the Apache Foundation.
Ideals are丰满, reality is骨感. During the incubation period, the entire team encountered many problems: previously, team members were used to communicating through DingTalk, but according to regulations in the foundation, they needed to communicate through email; for the addition of new functions and features, team members needed to consider and ask for opinions from broad community participants instead of deciding on their own. Behind this, essentially, was shifting from pursuing efficiency first to prioritizing consensus, transparency, and openness.
Fortunately, this wasn’t Alibaba’s first contact with the Apache Foundation. In 2016, distributed message middleware RocketMQ successfully graduated from Apache for the first time, accumulating rich experience. Following the path explored by “predecessors,” Dubbo successfully graduated in just 15 months.
After a long period of personnel adjustments, Yubo’s team also saw the dawn in Ant Design’s promotion. Ant Design’s positioning and timing as a mid-to-back-end design language were very suitable. It began to receive a lot of attention outside the company, with GitHub star count rising in a straight line, breaking 10,000 stars in March 2017, and reaching 20,000 stars in March 2018.
Yubo recalled that once, the Ant Design team held a developer salon by West Lake. A tall, handsome German guy came, speaking not-so-fluent Mandarin. It turned out that after soaking in the community for a long time, he not only chose Ant Design as the theme for his graduation thesis but also learned Chinese with great effort. After graduation, he ran all the way from Germany to Hangzhou to work and fell in love with this thousand-year-old ancient city.
On the other hand, in the spring of 2017, the “Java Development Handbook” was officially released. Over 2.6 million engineers have downloaded and consulted the handbook. In thousands of enterprise applications, the handbook has become a development specification recognized by the industry. The supporting P3C plugin’s star count on GitHub is close to 20,000.
At the beginning of this year, InfoQ counted more than 2,800 projects from 7 first-tier internet companies that are relatively active on GitHub in China. Among the top ten projects with the most stars, six projects were contributed by Alibaba. All projects’ star counts exceeded 660,000. From the numbers alone, Alibaba is worthy of being one of the Chinese companies contributing most to open source. Even though quite good open source results have been achieved, it still relies on developers’ self-drive. Fortunately, these developers found a balance between open source and commercial value, just like Dubbo and Ant Design.
On September 27, 2019, Jia Yangqing officially became the head of Alibaba’s Open Source Technology Committee. In an interview, Jia Yangqing said that in the past, the growth of these open source projects relied more on communication and exchange between Alibaba’s internal employees and the open source community. Now, Alibaba realizes that this can’t just rely on developers’ enthusiasm but needs a mature methodology to help developers participate more in open source, contribute to open source, and provide a series of support and guarantees from the organization. This is the original intention of Alibaba establishing the Open Source Committee.
Father of Caffe, one of TensorFlow and PyTorch authors Jia Yangqing
Now, Alibaba has clearly upgraded open source to one of the group’s technology strategies, from embracing open source, contributing to open source, and independent open source to continuously empowering open source. Open source fortunately, cloud explosion—this wave of open source climax also benefits from the comprehensive development of cloud computing. Just like not many people firmly believed Alibaba Cloud could succeed at the beginning, not many people thought about what impact cloud computing would have on open source.
The explosion of cloud computing made tech companies realize the importance of ecosystem and developers, and perhaps the best way to deal with developers is open source. An excellent open source project inevitably has an excellent community, and a community can easily gather excellent developers together; in addition, open source will also increase developers’ trust in projects… Various reasons make open source something major cloud computing companies want to do at this stage.
The addition of these tech companies also gives open source projects the best landing scenarios. Transaction volumes and traffic peaks like Double 11 are hard to encounter. Flink, which has become more mature and reliable after being tempered by the Double 11 scenario, fully verified the important role of tech companies in promoting open source.
However, gathering a group of developers who truly have open source sentiments and are willing to persist is not easy. At the end of the interview, Gao Tie said: “I often spend time at night and on weekends solving community issues. Fastjson and Druid are like my two children. I’ve raised them for almost 10 years. Fortunately, I’ve received many people’s support and use. This is everyone’s trust in me, and I must maintain them well.”
Until now, open source has accompanied this group of Alibaba engineers for ten years. Thinking of forcing his way into Alibaba back then, Gu Jin still remembers saying: “You missed me, you missed an era.” Gu Jin is particularly grateful to that good-looking HR brother who didn’t get angry because of his ignorance but instead made an exception to give an interview opportunity. “Although until now, I still haven’t changed an era, I’m very happy that I brought a little change to the world.”
Yes, one “market-goer” brought a little change to the world. How much energy will a group of “market-goers” bring to the world? This is an important microcosm of Alibaba’s open source history, and this is also an indispensable stroke in China’s open source history. Fortunately, everything is in progress, and the ending is still worth looking forward to.
Author: Zhao Yuying
Original: Alibaba’s Ten Years of Open Source: A Turbulent History
Reposted from: Developer Relations »
