Ten years ago, a database was needed to scale out for both reads and writes to meet the long-term needs of a growing business. The Amazon Dynamo database was developed and WERNER VOGELS, CTO of Amazon.com explains how its has grown from then.
It all started in 2004 when Amazon was running Oracle’s enterprise edition with clustering and replication. We had an advanced team of database administrators and access to top experts within Oracle. We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded.
Our straining database infrastructure on Oracle led us to evaluate if we could develop a purpose-built database that would support our business needs for the long term. We prioritized focusing on requirements that would support high-scale, mission-critical services like Amazon’s shopping cart, and questioned assumptions traditionally held by relational databases such as the requirement for strong consistency. Our goal was to build a database that would have the unbounded scalability, consistent performance and the high availability to support the needs of our rapidly growing business.
A deep dive on how we were using our existing databases revealed that they were frequently not used for their relational capabilities. About 70 percent of operations were of the key-value kind, where only a primary key was used and a single row would be returned. About 20 percent would return a set of rows, but still operate on only a single table.
With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.
The success of our early results with the Dynamo database encouraged us to write Amazon’s Dynamo whitepaper and share it at the 2007 ACM Symposium on Operating Systems Principles (SOSP conference), so that others in the industry could benefit. The Dynamo paper was well-received and served as a catalyst to create the category of distributed database technologies commonly known today as “NoSQL.”
Of course, no technology change happens in isolation, and at the same time NoSQL was evolving, so was cloud computing. As we began growing the AWS business, we realized that external customers might find our Dynamo database just as useful as we found it within Amazon.com. So, we set out to build a fully hosted AWS database service based upon the original Dynamo design.
The requirements for a fully hosted cloud database service needed to be at an even higher bar than what we had set for our Amazon internal system. The cloud-hosted version would need to be:
- Scalable – The service would need to support hundreds of thousands, or even millions of AWS customers, each supporting their own internet-scale applications.
- Secure – The service would have to store critical data for external AWS customers which would require an even higher bar for access control and security.
- Durable and Highly-Available – The service would have to be extremely resilient to failure so that all AWS customers could trust it for their mission-critical workloads as well.
- Performant – The service would need to be able to maintain consistent performance in the face of diverse customer workloads.
- Manageable – The service would need to be easy to manage and operate. This was perhaps the most important requirement if we wanted a broad set of users to adopt the service.
With these goals in mind, In January, 2012 we launched Amazon DynamoDB, our cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance and manageability needed to run mission-critical workloads.
Today, DynamoDB powers the next wave of high-performance, internet-scale applications that would overburden traditional relational databases. Many of the world’s largest internet-scale businesses such as Lyft, Tinder and Redfin as well as enterprises such as Comcast, Under Armour, BMW, Nordstrom and Toyota depend on DynamoDB’s scale and performance to support their mission-critical workloads.
DynamoDB is used by Lyft to store GPS locations for all their rides, Tinder to store millions of user profiles and make billions of matches, Redfin to scale to millions of users and manage data for hundreds of millions of properties, Comcast to power their XFINITY X1 video service running on more than 20 million devices, BMW to run its car-as-a-sensor service that can scale up and down by two orders of magnitude within 24 hours, Nordstrom for their recommendations engine reducing processing time from 20 minutes to a few seconds, Under Armour to support its connected fitness community of 200 million users, Toyota Racing to make real time decisions on pit-stops, tire changes, and race strategy, and another 100,000+ AWS customers for a wide variety of high-scale, high-performance use cases.
With all the real-world customer use, DynamoDB has proven itself on those original design dimensions:
- Scalable – DynamoDB supports customers with single tables that serve millions of requests per second, store hundreds of terabytes, or contain over 1 trillion items of data. In support of Amazon Prime Day 2017, the biggest day in Amazon retail history, DynamoDB served over 12.9 million requests per second. DynamoDB operates in all AWS regions (16 geographic regions now with announced plans for six more Regions in Bahrain, China, France, Hong Kong, Sweden), so you can have a scalable database in the geographic region you need.
- Secure – DynamoDB provides fine-grained access control at the table, item, and attribute level, integrated with AWS Identity and Access Management. VPC Endpoints give you the ability to control whether network traffic between your application and DynamoDB traverses the public Internet or stays within your virtual private cloud. Integration with AWS CloudWatch, AWS CloudTrail, and AWS Config enables support for monitoring, audit, and configuration management. SOC, PCI, ISO, FedRAMP, HIPAA BAA, and DoD Impact Level 4 certifications allows customers to meet a wide range of compliance standards.
- Durable and Highly-Available – DynamoDB maintains data durability and 99.99 percent availability in the event of a server, a rack of servers, or an Availability Zone failure. DynamoDB automatically re-distributes your data to healthy servers to ensure there are always multiple replicas of your data without you needing to intervene.
- Performant – DynamoDB consistently delivers single-digit millisecond latencies even as your traffic volume increases. In addition, DynamoDB Accelerator (DAX) a fully managed, highly available, in-memory cache further speeds up DynamoDB response times from milliseconds to microseconds and can continue to do so at millions of requests per second.
- Manageable – DynamoDB eliminates the need for manual capacity planning, provisioning, monitoring of servers, software upgrades, applying security patches, scaling infrastructure, monitoring, performance tuning, replication across distributed datacenters for high availability, and replication across new nodes for data durability. All of this is done for you automatically and with zero downtime so that you can focus on your customers, your applications, and your business.
- Adaptive Capacity –DynamoDB intelligently adapts to your table’s unique storage needs, by scaling your table storage up by horizontally partitioning them across many servers, or down with Time To Live (TTL) that deletes items that you marked to expire. DynamoDB provides Auto Scaling, which automatically adapts your table throughput up or down in response to actual traffic to your tables and indexes. Auto Scaling is on by default for all new tables and indexes.
Ten years ago, we never would have imagined the lasting impact our efforts on Dynamo would have. What started out as an exercise in solving our own needs in a customer obsessed way, turned into a catalyst for a broader industry movement towards non-relational databases, and ultimately, an enabler for a new class of internet-scale applications.
As we say at AWS, It is still Day One for DynamoDB. We believe we are in the midst of a transformative period for databases, and the adoption of purpose-built databases like DynamoDB is only getting started. We expect that the next ten years will see even more innovation in databases than the last ten. I know the team is working on some exciting new things for DynamoDB – I can’t wait to share them with you over the upcoming months.
Homemation creates comfort through smart homes
Home automation is more than just turning the lights on and off, Homemation’s Gedaliah Tobias tells BRYAN TURNER
The world is taking interior design notes from the Danish, in a style of living called hygge (pronounced hoo-gah). Its meaning varies from person to person: some see hygge as a warm fire on a cold winter’s night, others see it as a cup of hot coffee in the morning. The amount of “good feelings” one gets from these relaxing activities depends on what one values as indulgent.
But how does technology fit into this “art of feeling good”?
We asked Homemation marketing manager Gedaliah Tobias to take us through a fully automated home of the future and show us how automation creates comfort and good feelings.
“The house is powered by Control4, which you can think of as the brain of the smart home,” says Tobias. “It controls everything from the aircon to smart vacuum cleaners.”
The home of the future is secured by a connected lock. It acts like other locks with keypads and includes a key in the event of a power interruption. The keypad is especially useful to those who want to provide temporary access to visitors, staff, or simply kids who might lose their parents’ house keys.
“The keypad is especially useful for temporary access,” says Tobias. “For example, if you have a garden service that needs to use the home for the day, they can be given a code that only turns off the perimeter alarm beams in the garden for the day and time. If that code is used outside of the day and time range, users can set up alerts for their armed response to be alerted. This type of smart access boosts security.”
Once inside, one is greeted with a “scene” – a type of recipe for electronic success. The scene starts by turning on the lights, then by alerting the user to disarm the alarm. After the alarm is disarmed, the user can start another more complicated scene.
“Users can request customised scene buttons,” says Tobias. “For example, if I press the ‘Dinner call’ scene, the lights start to flash in the bedroom, there’s an announcement from the smart speakers, the blinds start to come down, the lighting is shifted to the dinner table. Shifting focus with lighting creates a mood to bring the house together for dinner.”
Homemation creates these customised scene buttons to enable users to control their homes without having to use another device. In addition to scene buttons, there are several ways to control the smart home.
“Everything in the smart home is controllable from your phone, the touchscreens around the house, the TV, and the dedicated remote control. Everyone is different, so having multiple ways to control the house is a huge value add.”
We ask Tobias where Homemation recommends non-smart home users should start on their smart home journey.
“Before anything, the Control4 infrastructure needs to be set up. This involves a lot of communications and electrical cabling to be run to different areas of the home to enable connectivity throughout the home. After the infrastructure is set up, the system is ready for smart home devices, like lighting and sound.”
“For new smart home users, the best bang for their buck would be to start with lighting once the infrastructure is set up. Taking it one step at a time is wise.”
• For more information, visit https://www.homemation.co.za/
Face App grabs SA attention
South Africans generated more than 100 000 search queries for “Face App” on Wednesday, while only generating 50 000 for “Mandela Day”. The Internet wentcrazy over the two-year-old app, which uses artificial intelligence to create a rendering of what users might look like in a few decades. Face App went viral as users posted their aged likenesses on social media in the #faceappchallenge. Privacy experts, however, warned that the app (made in Russia) may pose a threat to users’ privacy as it stores photos on its servers, with US Senate minority leader, Chuck Schumer, appealing to the FBI to investigate the app.
In other top searches on Google this week, “Johnny Clegg” garnered more than 500 000 search queries on Tuesday as the news of his passing broke. The ‘White Zulu’ of Juluka and Savuka fame was an internationally acclaimed musician who was also an important figure in the fight against apartheid. Tributes to Clegg have been flooding media and social media over the past couple of days. Clegg succumbed to pancreatic cancer at the age of 66.
More than 200 000 search queries were generated for “Mark Batchelor” on Monday after the former soccer star was brutally gunned down outside his Olivedale home in Gauteng. Investigations into the shooting are still ongoing. Batchelor played for Orlando Pirates, Wits University, Kaizer Chiefs, Mamelodi Sundowns, Moroka Swallows and Bafana Bafana.
“Jacob Zuma” also garnered more than 100 000 search queries on Monday as he made his first, much-anticipated appearance in front of the Zondo Commission on state capture.
On Sunday “Macdonald Ndou” picked up more than 10 000 search queries after reports of theMuvhango actor’s arrest made the rounds. Ndou was held on various charges including extortion and kidnapping. The Hawks have reportedly provisionally withdrawn charges against the TV star, but a spokesperson said the decision to withdraw does not mean the charges will not be reinstated.
“Serena Williams” garnered more than 50 000 searches on Saturday as the tennis superstar suffered a 6-2, 6-2 defeat against Simona Halep in a Wimbledon final that lasted just 56 minutes. Williams later told Agence France Presse, “She [Halep] played out of her mind” and “I was like a deer in headlights”.
Last Friday, South Africans produced more than 20 000 search queries for “Duduzane Zuma” as the Randburg Magistrates Court found the former first son not guilty of a charge of culpable homicide. In February 2014, Zuma was involved in a car crash that took the life of Phumzile Dube when his vehicle crashed into the taxi she was travelling in.
Search trends information is gleaned from data collated by Google based on what South Africans have been searching for and asking Google. Google processes more than 40 000 search queries every second. This translates to more than a billion searches per day and 1.2 trillion searches per year, worldwide. Live Google search trends data is available at https://www.google.co.za/trends/hottrends#pn=p40