Pinterest and coronavirus vaccinations both originated from Amazon’s original web service. Insiders tell Protocol how it expanded to hold more over 100 trillion objects 15 years later. Don Alvarez, like many other software entrepreneurs in late 2005, was trying to get a new company off the ground when a friend working at Amazon offered to show him a top-secret project that would alter the course of human history.
Adaptability-focused construction
During an interview in 2014, Vermeulen drew a line in the air over his head and to the right and said, “When people think bigger and quicker in computers, they think of this.” He compared the development of storage technology to the difference between driving his Tesla and piloting an aeroplane, saying, “It’s the difference between driving my Tesla and flying my aeroplane.” S3 was a radical departure from the norm. That was a boon to software engineers like Alvarez, who previously had to spend a fortune on storage hardware just to get their work done. Nothing “we had access to” could do “even remotely” what S3 could do, Alvarez said. “It was like being handed the keys to a candy store.” S3, like most of AWS, is the result of Amazon learning the ins and outs of distributed computing while establishing and scaling Amazon.com.
“The requirement that all Amazon internal and external developers use the same Amazon S3 distributed system was a driving factor in the design. This implies it needs to be robust enough to power Amazon.com’s websites, yet versatile enough to be used by any programmer for any data storage purpose “With the 2006 announcement of S3’s availability, AWS stated the following. Cloud computing has come a long way from its early days, when performance and dependability were major concerns. Such worries were amplified when it came to data, which, even a decade and a half ago, was recognised as one of the most valuable assets a business could possess.
There were only eight microservices available on S3 when it was first released 15 years ago; today, that number is closer to 300. Tomsen Bukovec made this remark on a novel approach to software development in which huge, dependent chunks of code are broken up into smaller, self-contained services.With the help of microservices architecture, Amazon was able to spread out potential S3 failure sites and create a system that takes into account the fact that failures can and will occur in distributed cloud services without bringing down the entire system. Also, the foundation was laid for the company to add new features without touching the original components: Unlike self-managed storage devices, AWS boasts that their S3 service has an astounding “11 9s” of reliability, or 99.999999999% uptime. (Competitors in the cloud storage market have caught up to this level.) S3 was originally designed as a storage space for static online assets, such as photographs and videos, which webmasters would fetch from AWS and send to your browser when you loaded a page. As time went on and businesses gained trust in cloud storage, they began to deposit all sorts of information in S3.
The situation then became a bit chaotic
Alvarez’s company, FilmmakerLive, was one of many at the time trying to solve the all-too-common problem of storage space by developing online collaboration apps for creative professions. The tech industry was just beginning to recover from the excesses of the dot-com boom, and spending heavily on hardware was a gamble for a new company. If you don’t buy enough, your site will go down. When you spend too much money. That was a bold move in the often unpredictable world of a new company. He accepted the friend’s offer despite his doubts that an online retailer would have anything useful to teach him about teamwork in the film industry.
Alvarez said to Protocol, “Rudy Valdez blew my mind.” Back when AWS offered only a few rudimentary services, Valdez oversaw business development for the company. He showed Mural’s current director of engineering, Alvarez, around Amazon’s S3 object storage service on the cloud. Fifteen years ago this weekend, the Simple Storage Service (S3) was released. Years would pass before “the cloud” would emerge as a game-changer in business IT. While announcing S3 on March 14, 2006, Amazon didn’t even use the word. Yet, the introduction of the storage service immediately resolved several difficult issues for entrepreneurs like Alvarez, and would eventually cause a paradigm shift in how all organisations viewed the acquisition of information technology.
In the years that followed, AWS saw a surge in interest from startups like Pinterest, Airbnb, and Stripe, as well as adoption by established businesses like Netflix, which had previously operated as a DVD mail order service. There was nothing like Amazon, as “Amazon was putting endless disc space in the hands of every startup at incredibly low and pay-for-what-you-need pricing point,” as Alvarez described it. Two, their application programming interface (API) was so intuitive that I was able to create anything of value in it within the first day of using a secret product. As a result of S3, AWS was able to bring in over $45 billion in revenue last year. A set of design principles developed by a team led by Allan Vermeulen, Amazon’s chief technical officer in the early days of AWS, have remained central to the company’s strategy despite the company’s multiple pivots over the past 15 years.
“We knew what [customers] wanted to do then,” said Mai-Lan Tomsen Bukovec, vice president for AWS Storage and current head of S3, in an interview with Protocol. “But we also understood that applications would evolve because our customers are tremendously innovative, and what they’re doing out there in all the different industries is going to change annually,” the company said.
Fixing dripping pails
Looking back over the previous few years, many security incidents may be traced back to “leaky buckets,” the fundamental unit of S3 storage. Although other cloud providers have also experienced similar occurrences, Amazon has had to deal with this issue frequently due to its dominant market position. Customers are responsible for protecting their accounts to a reasonable degree, while AWS takes reasonable precautions to ensure that no unauthorised parties get access to its servers or network. To rephrase, if your laptop is stolen from an open rental car, you have only yourself to blame. Nevertheless, cloud users repeatedly expose their own customers’ private information by storing it in unencrypted buckets that may be easily accessed by anyone. This is just one example of how AWS has had to adapt some of its staple offerings to the needs of its client base, especially those who came later and are used to getting all their information over internal networks.
According to Tomsen Bukovec, “in the realm of business applications, you don’t need to have access beyond the firm, or really outside a group of people within the business.” Tools like Block Public Access, which can secure all storage buckets connected to a company account, were developed because it became apparent that AWS needed to do more to help its clients help themselves. Alvarez remarked that during Amazon Web Services’ explosive early expansion, the company’s infamous “two-pizza teams” were “both a strength and a problem.”
“That allowed for the rapid development of all those services at a rate at which their rivals could not keep up. In addition, this meant that there was initially much less consistency, which was challenging to decipher and oversee “He remarked that things have gotten better over time.
After that, more security solutions emerged to help clients monitor their accounts for intrusion from the outside world and provide various employees varying degrees of access. According to Tomsen Bukovec, “where we’re seeing customers go with their migrations is that they typically have hundreds of buckets and lots and lots of [various] roles.” Customers’ desires to audit and restrict access to their S3 storage resources inform “what we consider to develop to assist them secure the perimeter of their AWS resource.”
Reaching a hundred trillion
In the years following its first release, S3 had numerous refinements and price drops: When Amazon Web Services (AWS) finally held its first major re:Invent developer conference in 2012, one of the biggest announcements from that week was a 24th consecutive price cut for S3 storage. Alyssa Henry, then-vice president of AWS Storage Services, noted in a 2012 keynote talk that the pricing reductions were made possible by AWS’s ability to improve the underlying S3 service on the fly.
The number of objects stored in S3 reached 9 billion in its first year of operation, much above its original intended capacity of 20 billion. Without causing any problems for the original S3 users, the business improved the underlying storage service in 2012, allowing it to store 1 trillion objects; by 2020, that number is expected to increase to 100 trillion
You didn’t have to go out and buy the next upgrade—v2 of Amazon S3; you didn’t have to do the migration yourself; you just got it all for free, it just worked, things just got better,” said Henry, who is now executive vice president and head of Square’s Seller unit, at the 2012 event. That’s one of the ways in which cloud computing differs from conventional approaches to information technology. Last year, AWS provided high consistency throughout S3, which was a comparable enhancement.
Older storage systems, like the original S3, were built around “eventual consistency,” which means that a storage service might not always be able to tell you immediately if a new piece of data has settled into its allotted storage bucket, but it would eventually catch up. However, with the increased speed of modern applications, it is essential that any component making a query to a storage service have access to the most up-to-date list of data that is currently accessible. As other cloud providers offer strong consistency principles, but can only roll them out against a much smaller user base, AWS has spent the last couple of years rebuilding S3 around these ideas.
Tomsen Bukovec’s “That is a very complex engineering problem” was one of the most talked-about announcements at re:Invent 2020, especially among the more technically minded AWS customer base. Moving into the next decade, Tomsen Bukovec and her team are investigating methods to enhance the functionality and performance of data lakes, which enable Amazon customers to do granular analysis of internal and customer data. Indeed, an S3 data lake was important in the creation of the Moderna COVID-19 vaccine, as stated by Tomsen Bukovec. Having “this unique insight that we built up over 15 years of usage,” she explained, “where we can determine what our customers are attempting to achieve, and how we can create [S3] in such a manner that it maintains true to that simple, cost-effective, secure, durable, dependable, and highly-performing storage.”
Meet Khurram Raheel Akbar our senior content writer. With over 10 years of experience in the field of content writing, Raheel, has established himself as an expert in creating engaging and informative content. His exceptional writing skills have enabled him to craft compelling stories that resonate with audiences across a variety of industries. Raheel’s writing style is concise, clear, and impactful, making him a go-to writer for any business looking to enhance its brand’s online presence. His dedication to staying up-to-date with the latest trends and strategies in content marketing allows him to create relevant and informative content that drives traffic and increases conversion rates. Raheel’s passion for writing is matched only by his commitment to delivering exceptional results to his clients. Whether you’re looking to revamp your website, launch a new product, or establish your brand’s voice, Raheel is the senior content writer you need to bring your vision to life.
