Ten years ago, a database was needed to scale out for both reads and writes to meet the long-term needs of a growing business. The Amazon Dynamo database was developed and WERNER VOGELS, CTO of Amazon.com explains how its has grown from then.
It all started in 2004 when Amazon was running Oracle’s enterprise edition with clustering and replication. We had an advanced team of database administrators and access to top experts within Oracle. We were pushing the limits of what was a leading commercial database at the time and were unable to sustain the availability, scalability and performance needs that our growing Amazon business demanded.
Our straining database infrastructure on Oracle led us to evaluate if we could develop a purpose-built database that would support our business needs for the long term. We prioritized focusing on requirements that would support high-scale, mission-critical services like Amazon’s shopping cart, and questioned assumptions traditionally held by relational databases such as the requirement for strong consistency. Our goal was to build a database that would have the unbounded scalability, consistent performance and the high availability to support the needs of our rapidly growing business.
A deep dive on how we were using our existing databases revealed that they were frequently not used for their relational capabilities. About 70 percent of operations were of the key-value kind, where only a primary key was used and a single row would be returned. About 20 percent would return a set of rows, but still operate on only a single table.
With these requirements in mind, and a willingness to question the status quo, a small group of distributed systems experts came together and designed a horizontally scalable distributed database that would scale out for both reads and writes to meet the long-term needs of our business. This was the genesis of the Amazon Dynamo database.
The success of our early results with the Dynamo database encouraged us to write Amazon’s Dynamo whitepaper and share it at the 2007 ACM Symposium on Operating Systems Principles (SOSP conference), so that others in the industry could benefit. The Dynamo paper was well-received and served as a catalyst to create the category of distributed database technologies commonly known today as “NoSQL.”
Of course, no technology change happens in isolation, and at the same time NoSQL was evolving, so was cloud computing. As we began growing the AWS business, we realized that external customers might find our Dynamo database just as useful as we found it within Amazon.com. So, we set out to build a fully hosted AWS database service based upon the original Dynamo design.
The requirements for a fully hosted cloud database service needed to be at an even higher bar than what we had set for our Amazon internal system. The cloud-hosted version would need to be:
- Scalable – The service would need to support hundreds of thousands, or even millions of AWS customers, each supporting their own internet-scale applications.
- Secure – The service would have to store critical data for external AWS customers which would require an even higher bar for access control and security.
- Durable and Highly-Available – The service would have to be extremely resilient to failure so that all AWS customers could trust it for their mission-critical workloads as well.
- Performant – The service would need to be able to maintain consistent performance in the face of diverse customer workloads.
- Manageable – The service would need to be easy to manage and operate. This was perhaps the most important requirement if we wanted a broad set of users to adopt the service.
With these goals in mind, In January, 2012 we launched Amazon DynamoDB, our cloud-based NoSQL database service designed from the ground up to support extreme scale, with the security, availability, performance and manageability needed to run mission-critical workloads.
Today, DynamoDB powers the next wave of high-performance, internet-scale applications that would overburden traditional relational databases. Many of the world’s largest internet-scale businesses such as Lyft, Tinder and Redfin as well as enterprises such as Comcast, Under Armour, BMW, Nordstrom and Toyota depend on DynamoDB’s scale and performance to support their mission-critical workloads.
DynamoDB is used by Lyft to store GPS locations for all their rides, Tinder to store millions of user profiles and make billions of matches, Redfin to scale to millions of users and manage data for hundreds of millions of properties, Comcast to power their XFINITY X1 video service running on more than 20 million devices, BMW to run its car-as-a-sensor service that can scale up and down by two orders of magnitude within 24 hours, Nordstrom for their recommendations engine reducing processing time from 20 minutes to a few seconds, Under Armour to support its connected fitness community of 200 million users, Toyota Racing to make real time decisions on pit-stops, tire changes, and race strategy, and another 100,000+ AWS customers for a wide variety of high-scale, high-performance use cases.
With all the real-world customer use, DynamoDB has proven itself on those original design dimensions:
- Scalable – DynamoDB supports customers with single tables that serve millions of requests per second, store hundreds of terabytes, or contain over 1 trillion items of data. In support of Amazon Prime Day 2017, the biggest day in Amazon retail history, DynamoDB served over 12.9 million requests per second. DynamoDB operates in all AWS regions (16 geographic regions now with announced plans for six more Regions in Bahrain, China, France, Hong Kong, Sweden), so you can have a scalable database in the geographic region you need.
- Secure – DynamoDB provides fine-grained access control at the table, item, and attribute level, integrated with AWS Identity and Access Management. VPC Endpoints give you the ability to control whether network traffic between your application and DynamoDB traverses the public Internet or stays within your virtual private cloud. Integration with AWS CloudWatch, AWS CloudTrail, and AWS Config enables support for monitoring, audit, and configuration management. SOC, PCI, ISO, FedRAMP, HIPAA BAA, and DoD Impact Level 4 certifications allows customers to meet a wide range of compliance standards.
- Durable and Highly-Available – DynamoDB maintains data durability and 99.99 percent availability in the event of a server, a rack of servers, or an Availability Zone failure. DynamoDB automatically re-distributes your data to healthy servers to ensure there are always multiple replicas of your data without you needing to intervene.
- Performant – DynamoDB consistently delivers single-digit millisecond latencies even as your traffic volume increases. In addition, DynamoDB Accelerator (DAX) a fully managed, highly available, in-memory cache further speeds up DynamoDB response times from milliseconds to microseconds and can continue to do so at millions of requests per second.
- Manageable – DynamoDB eliminates the need for manual capacity planning, provisioning, monitoring of servers, software upgrades, applying security patches, scaling infrastructure, monitoring, performance tuning, replication across distributed datacenters for high availability, and replication across new nodes for data durability. All of this is done for you automatically and with zero downtime so that you can focus on your customers, your applications, and your business.
- Adaptive Capacity –DynamoDB intelligently adapts to your table’s unique storage needs, by scaling your table storage up by horizontally partitioning them across many servers, or down with Time To Live (TTL) that deletes items that you marked to expire. DynamoDB provides Auto Scaling, which automatically adapts your table throughput up or down in response to actual traffic to your tables and indexes. Auto Scaling is on by default for all new tables and indexes.
Ten years ago, we never would have imagined the lasting impact our efforts on Dynamo would have. What started out as an exercise in solving our own needs in a customer obsessed way, turned into a catalyst for a broader industry movement towards non-relational databases, and ultimately, an enabler for a new class of internet-scale applications.
As we say at AWS, It is still Day One for DynamoDB. We believe we are in the midst of a transformative period for databases, and the adoption of purpose-built databases like DynamoDB is only getting started. We expect that the next ten years will see even more innovation in databases than the last ten. I know the team is working on some exciting new things for DynamoDB – I can’t wait to share them with you over the upcoming months.
Samsung unleashes the beast
Most new smartphone releases of the past few years have been like cat-and-mouse games with consumers and each other. It has been as if morsels of cheese are thrown into the box to make it more interesting: a little extra camera here, a little more battery there, and incremental changes to size, speed (more) and weight (less). Each change moves the needle of innovation ever-so-slightly. Until we find ourselves, a few years later, with a handset that is revolutionary compared to six years ago, but an anti-climax relative to six months before.
And then came Samsung. Probably stung by the “incremental improvement” phrase that has become almost a cliché about new Galaxy devices, the Korean giant chose to unleash a beast last week.
The new Galaxy Note 9 is not only the biggest smartphone Samsung has ever released, but one of the biggest flagship handsets that can still be called a phone. With a 6.4” display, it suddenly competes with mini-tablets and gaming consoles, among other devices that had previously faced little contest from handsets.
It offers almost ever cutting edge introduced to the Galaxy S9 and S9+ smartphones earlier this year, including the market-leading f1.5 aperture lens, and an f2.4. telephoto lens, each weighing in at 12 Megapixels. The front lens is equally impressive, with an f1.7 aperture – first introduced on the Note 8 as the widest yet on a selfie camera.
So far, so S9. However, the Note range has always been set apart by its S Pen stylus, and each edition has added new features. Born as a mere pen that writes on screens, it evolved through the likes of pressure sensitivity, allowing for artistic expression, and cut-and-paste text with translation-on-the-fly.
(Click here or below to read more about the Samsung Galaxy S Pen stylus) Samsung Galaxy S9 Features)
SA ride permit system ‘broken’
Despite the amendments to the National Land Transport Act, ALON LITS, General Manager, Uber in Sub Saharan Africa, believes that many premature given that the necessary, well-functioning systems and processes are not yet in place to make these regulatory changes viable.
The spirit and intention of the amendments to the National Land Transport Act No 5 (NLTA), 2009 put forward by the Ministry of Transport are to be commended. It is especially pleasing that these amendments include ridesharing and e-hailing operators and drivers as legitimate participants in the country’s public transport system, which point to government’s willingness to embrace the changes and innovation taking place in the country’s transport industry.
However, there are aspects of the proposed amendments that are, at best, premature given that the necessary, well-functioning systems and processes are not yet in place to make these regulatory changes viable.
Of particular concern are the significant financial penalties that will need to be paid by ridesharing and e-hailing companies whose independent operators are found to be transporting passengers without a legal permit issued by the relevant local authority. These fines can be as high as R100 000 per driver operating without a permit. Apart from being an excessive penalty it is grossly unfair given that a large number of local authorities don’t yet have functioning permit issuing systems and processes in place.
The truth is that the operating permit issuance system in South Africa is effectively broken. The application and issuance processes for operating licenses are fundamentally flawed and subject to extensive delays, sometimes over a year in length. This situation is exacerbated by the fact that it is very difficult for applicants whose permit applications haven’t yet been approved to get reasons for the extensive delays on the issuing of those permits.
Uber has had extensive first-hand experience with the frustratingly slow process of applying for these permits, with drivers often having to wait months and, in some cases more than a year, for their permits.
Sadly, there appears to be no sense of urgency amongst local authorities to prioritise fixing the flawed permit issuing systems and processes or address the large, and growing, backlogs of permit applications. As such, in order for the proposed stringent permit enforcement rules to be effective and fair to all role players, the long-standing issues around permit issuance first need to be addressed. At the very least, before the proposed legislation amendments are implemented, the National Transport Ministry needs to address the following issues:
- Efficient processes and systems must be put in place in all local authorities to allow drivers to easily apply for the operating permits they require
- Service level agreements need to be put in place with local authorities whereby they are required to assess applications and issue permits within the prescribed 60-day period.
- Local authorities need to be given deadlines by which their current permit application backlogs must be addressed to allow for faster processing of new applications once the amendments are promulgated.
If the Transport Ministry implements the proposed legislation amendments before ensuring that these permit issuance challenges are addressed, many drivers will be faced with the difficult choice of either having to operate illegally whilst awaiting their approved permits and risking significant fines and/or arrest, or stopping operations until they receive their permits, thereby losing what is, for many of them, their only source of income.
As such, if the Ministry of Transport is not able to address these particular challenges, it is only reasonable to ask it to reconsider this amendment and delay its implementation until the necessary infrastructure is in place to ensure it does not impact negatively on the country’s transport industry. The legislators must have been aware of the challenges of passing such a significant law, as the Amendment Bill allows for the Minister to use his discretion to delay implementation of provisions for up to 5 years.
Fair trade and healthy competition are the cornerstones of any effective and growing economy. However, these clauses (Section 66 (7) and Section 66A) of the NLTA amendment, as well as the proposal that regulators be given authority to define the geographic locations or zones in which vehicles may operate, are contrary to the spirit of both. As a good corporate citizen, Uber is committed to supplementing and enhancing South Africa’s national transport system and contributing positively to the industry. If passed into law without the revisions suggested above, these new amendments will limit our business and many others from playing the supportive roles we all can, and should, in growing the SA transport and tourism industries as well as many other key economic sectors.
What’s more, if passed as they currently stand, the amendments will effectively limit South African consumers from having full access to the range of convenient transport options they deserve; which has the potential to harm the reputation and credibility of the entire transport industry.