Ten years ago, a database was needed to scale out for both reads and writes to meet the long-term needs of a growing business. The Amazon Dynamo database was developed and WERNER VOGELS, CTO of Amazon.com explains how its has grown from then.
It all started in 2004 when Amazon was running Oracle’s enterprise edition with clustering and replication. We had an advanced team of database administrators and access to top experts within Oracle. We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded.
Our straining database infrastructure on Oracle led us to evaluate if we could develop a purpose-built database that would support our business needs for the long term. We prioritized focusing on requirements that would support high-scale, mission-critical services like Amazon’s shopping cart, and questioned assumptions traditionally held by relational databases such as the requirement for strong consistency. Our goal was to build a database that would have the unbounded scalability, consistent performance and the high availability to support the needs of our rapidly growing business.
A deep dive on how we were using our existing databases revealed that they were frequently not used for their relational capabilities. About 70 percent of operations were of the key-value kind, where only a primary key was used and a single row would be returned. About 20 percent would return a set of rows, but still operate on only a single table.
With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.
The success of our early results with the Dynamo database encouraged us to write Amazon’s Dynamo whitepaper and share it at the 2007 ACM Symposium on Operating Systems Principles (SOSP conference), so that others in the industry could benefit. The Dynamo paper was well-received and served as a catalyst to create the category of distributed database technologies commonly known today as “NoSQL.”
Of course, no technology change happens in isolation, and at the same time NoSQL was evolving, so was cloud computing. As we began growing the AWS business, we realized that external customers might find our Dynamo database just as useful as we found it within Amazon.com. So, we set out to build a fully hosted AWS database service based upon the original Dynamo design.
The requirements for a fully hosted cloud database service needed to be at an even higher bar than what we had set for our Amazon internal system. The cloud-hosted version would need to be:
- Scalable – The service would need to support hundreds of thousands, or even millions of AWS customers, each supporting their own internet-scale applications.
- Secure – The service would have to store critical data for external AWS customers which would require an even higher bar for access control and security.
- Durable and Highly-Available – The service would have to be extremely resilient to failure so that all AWS customers could trust it for their mission-critical workloads as well.
- Performant – The service would need to be able to maintain consistent performance in the face of diverse customer workloads.
- Manageable – The service would need to be easy to manage and operate. This was perhaps the most important requirement if we wanted a broad set of users to adopt the service.
With these goals in mind, In January, 2012 we launched Amazon DynamoDB, our cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance and manageability needed to run mission-critical workloads.
Today, DynamoDB powers the next wave of high-performance, internet-scale applications that would overburden traditional relational databases. Many of the world’s largest internet-scale businesses such as Lyft, Tinder and Redfin as well as enterprises such as Comcast, Under Armour, BMW, Nordstrom and Toyota depend on DynamoDB’s scale and performance to support their mission-critical workloads.
DynamoDB is used by Lyft to store GPS locations for all their rides, Tinder to store millions of user profiles and make billions of matches, Redfin to scale to millions of users and manage data for hundreds of millions of properties, Comcast to power their XFINITY X1 video service running on more than 20 million devices, BMW to run its car-as-a-sensor service that can scale up and down by two orders of magnitude within 24 hours, Nordstrom for their recommendations engine reducing processing time from 20 minutes to a few seconds, Under Armour to support its connected fitness community of 200 million users, Toyota Racing to make real time decisions on pit-stops, tire changes, and race strategy, and another 100,000+ AWS customers for a wide variety of high-scale, high-performance use cases.
With all the real-world customer use, DynamoDB has proven itself on those original design dimensions:
- Scalable – DynamoDB supports customers with single tables that serve millions of requests per second, store hundreds of terabytes, or contain over 1 trillion items of data. In support of Amazon Prime Day 2017, the biggest day in Amazon retail history, DynamoDB served over 12.9 million requests per second. DynamoDB operates in all AWS regions (16 geographic regions now with announced plans for six more Regions in Bahrain, China, France, Hong Kong, Sweden), so you can have a scalable database in the geographic region you need.
- Secure – DynamoDB provides fine-grained access control at the table, item, and attribute level, integrated with AWS Identity and Access Management. VPC Endpoints give you the ability to control whether network traffic between your application and DynamoDB traverses the public Internet or stays within your virtual private cloud. Integration with AWS CloudWatch, AWS CloudTrail, and AWS Config enables support for monitoring, audit, and configuration management. SOC, PCI, ISO, FedRAMP, HIPAA BAA, and DoD Impact Level 4 certifications allows customers to meet a wide range of compliance standards.
- Durable and Highly-Available – DynamoDB maintains data durability and 99.99 percent availability in the event of a server, a rack of servers, or an Availability Zone failure. DynamoDB automatically re-distributes your data to healthy servers to ensure there are always multiple replicas of your data without you needing to intervene.
- Performant – DynamoDB consistently delivers single-digit millisecond latencies even as your traffic volume increases. In addition, DynamoDB Accelerator (DAX) a fully managed, highly available, in-memory cache further speeds up DynamoDB response times from milliseconds to microseconds and can continue to do so at millions of requests per second.
- Manageable – DynamoDB eliminates the need for manual capacity planning, provisioning, monitoring of servers, software upgrades, applying security patches, scaling infrastructure, monitoring, performance tuning, replication across distributed datacenters for high availability, and replication across new nodes for data durability. All of this is done for you automatically and with zero downtime so that you can focus on your customers, your applications, and your business.
- Adaptive Capacity –DynamoDB intelligently adapts to your table’s unique storage needs, by scaling your table storage up by horizontally partitioning them across many servers, or down with Time To Live (TTL) that deletes items that you marked to expire. DynamoDB provides Auto Scaling, which automatically adapts your table throughput up or down in response to actual traffic to your tables and indexes. Auto Scaling is on by default for all new tables and indexes.
Ten years ago, we never would have imagined the lasting impact our efforts on Dynamo would have. What started out as an exercise in solving our own needs in a customer obsessed way, turned into a catalyst for a broader industry movement towards non-relational databases, and ultimately, an enabler for a new class of internet-scale applications.
As we say at AWS, It is still Day One for DynamoDB. We believe we are in the midst of a transformative period for databases, and the adoption of purpose-built databases like DynamoDB is only getting started. We expect that the next ten years will see even more innovation in databases than the last ten. I know the team is working on some exciting new things for DynamoDB – I can’t wait to share them with you over the upcoming months.
Online retail gets real
After decades of experience in selling online, retailers still seek out the secret of reaching the digital consumer, writes ARTHUR GOLDSTUCK.
It’s been 23 years since the first pizza and the first bunch of flowers was sold online. One would think, after all this time, that retailers would know exactly what works, and exactly how the digital consumer thinks.
Yet, in shopping-mad South Africa, only 4% of adults regularly shop online. One could blame high data costs, low levels of tech-savviness, or lack of trust. However, that doesn’t explain why a population where more than a quarter of people have a debit or credit card and almost 40% of people use the Internet is staying away.
The new Online Retail in South Africa 2019 study, conducted by World Wide Worx with the support of Visa and Platinum Seed, reveals that growth is in fact healthy, but is still coming off a low base. This year, the total sale of retail products online is expected to pass the R14-billion mark, making up 1.4% of total retail.
This figure represents 25% growth over 2017, and comes after the same rate of growth was seen in 2017. At this rate, it is clear that online retail is going mainstream, driven by aggressive marketing, and new shopping channels like mobile shopping.
But it is equally clear that not all retailers are getting it right. According to the study, the unwillingness of business to reinvest revenue in developing their online presence is one of the main barriers to long-term success. Only one in five companies surveyed invested more than 20% of their online turnover back into their online store. Over half invested less than 10% back.
On the surface, the industry looks healthy, as a surprisingly high 71% of online retailers surveyed say they are profitable. But this brings to mind the early days of Amazon.com, in 1996, when founder Jeff Bezos was asked when it would become profitable.
He declared that it would not be profitable for at least another five years. And if it did, he said, it would be in big trouble. He meant that it was so important for long-term sustainability that Amazon reinvest all its revenues in customer systems, that it could not afford to look for short-term profits.
According to the South African study, the single most critical factor in the success of online retail activities is customer service. A vast majority, 98% of respondents, regarded it as important. This positions customer service as the very heart of online retail. For Amazon, investment back into systems that would streamline customer service became the key to the world’s digital wallets.
In South Africa online still make up a small proportion of overall retail, but for the first time we see the promise of a broader range of businesses in terms of category, size, turnover and employee numbers. This is a sign that our local market is beginning to mature.
Clothing and apparel is the fastest growing sector, but is also the sector with the highest turnover of businesses. It illustrates the dangers of a low barrier to entry: the survival rate of online stores in this sector is probably directly opposite to the ease of setting up an online apparel store.
A fast-growing category that was fairly low on the agenda in the past, alcohol, tobacco and vaping, has benefited from the increased online supply of vapes, juices and accessories. It also suggests that smoking bans, and the change in the legal status of marijuana during the survey, may have boosted demand.
In the coming weeks, we can expect online retail to fall under the spotlight as never before. Black Friday, a shopping tradition imported “wholesale” from the United States, is expected to become the biggest online shopping day of the year in South Africa, as it is in the USA.
Initially, it was just a gimmick in South Africa, attempting to cash in on what was a purely American tradition of insane sales on the Friday after Thanksgiving Day, which occurs on the third Thursday of November every year. It is followed by Cyber Monday, making the entire weekend one of major promotions and great bargains.
It has grown every year in South Africa since its first introduction about six years ago, and last year it broke into the mainstream, with numerous high profile retailers embracing it, and many consumers experiencing it for the first time.
It is now positioned as the prime bargain day of the year for consumers, and many wait in anticipation for it, as they do in the USA. Along with Cyber Monday, it provides an excuse for retailers to go all out in their marketing, and for consumers to storm the display shelves or web pages. South African shoppers, clearly, are easily enticed by bargains.
Word of mouth around Black Friday has also grown massively in the past two years, driven by both media and shoppers who have found ridiculous bargains. As news spreads that the most ridiculous of the bargains are to be had online, even those who were reticent of digital shopping will be tempted to convert.
The Online Retail in SA 2019 report has shown over the years that, as people become more experienced in using the Internet, their propensity to shop online increases. This is part of the World Wide Worx model known as the Digital Participation Curve. The key missing factor in the Curve is that most retailers do not know how to convert that propensity into actual online shopping behaviour. Black Friday will be one of the keys to conversion.
Carry on reading to find out about the online retailers of the year.
Reliable satellite Internet?
MzansiSat, a satellite-Internet business, aims to beam Internet connections to places in South Africa which don’t have access to cabled and mobile network infrastructure, writes BRYAN TURNER.
Stellenbosch-based MzansiSat promises to provide cheap wholesale Internet to Internet Service Providers for as little as R25 per Gigabyte. Providers who offer more expensive Internet services could benefit greatly from partnering with MzansiSat, says the company.
“Using MzansiSat, we hope that we can carry over cost-savings benefits to the consumer,” says Victor Stephanopoli, MzansiSat chief operating officer.
The company, which has been spun off from StellSat, has been looking to increase its investor portfolio while it waits for spectrum approval. The additional investment will allow MzansiSat’s satellite to operate in more regions across Africa.
The MzansiSat satellite is being built by Thales Alenia Space, a French company which is also acting as technical partner to MzansiSat. In addition to building the satellite, Thales Alenia Space will also be assisting MzansiSat in coordinating the launch. The company intends to launch the satellite into the 56°E orbital slot in a geostationary orbit, which enables communication almost anywhere in Africa. The launch is expected to happen in 2022.
The satellite will have 76 transponders, 48 of which will be Ku-band and 28 C-band. Ku-band is all about high-speed performance, while C-band deals with weather-resistance. The design intention is for customers of MzansiSat to choose between very cheap, reliable data and very fast, power-efficient data.
C-band is an older technology, which makes bandwidth cheaper and almost never affected by rain but requires bigger dishes and slower bandwidth compared to Ku-band connections. On the other hand, Ku-band is faster, experiences less microwave interference, and requires less power to run – but is less reliable with bad weather conditions.
MzansiSat’s potential military applications are significant, due to the nature of the military being mobile and possibly in remote areas without connectivity. Connectivity everywhere would be potentially be life-saving.
Consumers in remote areas will benefit, even though satellite is higher in latency than fibre and LTE connections. While this level of latency is high (a fifth of a second in theory), satellite connections are still adequate for browsing the Internet and watching online content.
The Internet of Things (IoT) may see the benefits of satellite Internet before consumers do. The applications of IoT in agriculture are vast, from hydration sensors to soil nutrient testers, and can be realised with an Internet connection which is available in a remote area.
Stephanopoli says that e-learning in remote areas can also benefit from MzansiSat’s presence, as many school resources are becoming readily available online.
“Through our network, the learning experience can be beamed into classrooms across the country to substitute or complement local resources within the South African schooling system.”