An ambitious star-mapping project highlights the growing importance of big data and the cloud, writes ARTHUR GOLDSTUCK.
At an event in Berlin today, the European Space Agency (ESA) is unveiling the biggest set of data about the stars ever gathered. The positions and magnitudes of no less than 1.7 billion stars of our Milky Way galaxy have been gathered by the Gaia spacecraft, which took off in 2013 and began collecting data a year later.
The ship is also transmitting a vast range of additional data, with distances, motions and colours of more than 1.3 billion stars collected so far. And that is without counting temperature measures, solar system analysis and radiation sources from outside the galaxy.
“The extraordinary data collected by Gaia throughout its mission will be used to eventually build the most accurate three-dimensional map of the positions, motions, and chemical composition of stars in our Galaxy,” according to a project document. “By reconstructing the properties and past trajectories of all the stars probed by Gaia, astronomers will be able to delve deep into the history of our Galaxy’s formation and evolution.”
The entire project would be impossible were it not for advances in cloud computing storage, big data analysis and artificial intelligence systems during this decade. The storage demands alone are mind-boggling. The ESA roped in cloud data services company NetApp, which focuses on management of applications and data across cloud and on-premise environments.
NetApp was previously involved with the Rosetta space mission, which landed a spacecraft on a comet in 2016. Lauched as far back as 2004, ten years later it became the first spacecraft to go into orbit around a comet, and its lander made the first successful landing on a comet.
“For the next two years Rosetta was following the comet and streaming data,” says Morne Bekker, NetApp South African country manager. “But with the comet speeding away from the sun at 120 000kph, Rosetta would soon lose solar power. Scientists seized the opportunity to attempt what no one had ever tried before — to gather unique observations through a controlled impact with the comet. Despite blistering speeds and countless unknowns, the spacecraft landed just 33m from its target point.
“It’s quite phenomenal when you think of the data and analytics harvested, and the information it can send back. Now we’re helping with the Gaia project. You can imagine how much data is being collected daily. The catalogue will probably end up at 2 Petabytes in size – that’s 2-million gigabytes. If you think of the minute points of data being extracted, obviously you have to be using AI and machine learning to analyse all of this.”
Ruben Alvarez, IT manager at the ESA, sums it up simply: “Data is everything. Our biggest challenge is processing of the data.”
Naturally, ESA required absolute reliability from data storage. It also demanded almost infinite scalability to support the massive data requirements of past, present, and future missions.
“We have a commitment to deliver data to different institutes in Europe on a daily basis,” says Alvarez. “Adding to the challenge, data from every mission must be accessible indefinitely. In the coming years, we will be launching new missions that will demand huge amounts of data. NetApp provided us with solutions that were scalable, even if we didn’t know in advance how much disk storage we were going to need.”
ESA says it expects to publish the full Gaia catalogue in 2020, making it available online to professional astronomers and the general public, with interactive, graphical interfaces.
The catalogue, says Alvarez, will unlock many mysteries of the stars.
“We call our site the Library of the Universe because we keep the science archive of
all of our scientific missions. This is how we allow people to really investigate the universe. t’s all about the data.”
The mission has tremendous scientific implications, but also makes a powerful business case for big data and cloud computing.
“The capabilities for AI and machine learning in the processing of mass amounts of data are far-reaching,” says Bekker. “Not only does it equate to extreme performance, but also to massive non-disruptive scalability where scientists can scale to 20 PB and beyond, to support the largest of learning data sets. Importantly it also allows scientists to expand their data where needed.”
Across Africa, the power of the cloud and big data is only slowly being harnessed. A new research project, Cloud Africa 2018, conducted by World Wide Worx for global networking application company F5 Networks, shows that cloud uptake is now pervasive across Kenya, Nigeria and South Africa.
However, the research reveals that each country experiences the benefits of the cloud differently. Respondents in Nigeria and Kenya named Business efficiency and Scalability by far the most important benefit, with 80% and 75% respectively selecting it as an advantage. Only 61% of South African respondents cited it.
The opposite happened with the most important benefit among South Africans: Time-to-market or speed of deployment came in as the most prominent, at 68% of respondents. In contrast, only 48% of companies in Kenya and 28% in Nigeria named it as a key benefit.
This appears to be a function of the infrastructure challenges in developing information technology markets like Nigeria and Kenya, where the cloud is used to overcome the obstacles that get in the way of efficiency.
In South Africa, where construction of the giant Square Kilometre Array multi radio telescope is due to begin next year, the learnings of Rosetta and Gaia will ensure that data collection, storage and analysis will no longer be a challenge.
- For the latest on project Gaia, visit http://sci.esa.int/gaia/
Smart home arrives in SA
The smart home is no longer a distant vision confined to advanced economies, writes ARTHUR GOLDSTUCK.
The smart home is a wonderful vision for controlling every aspect of one’s living environment via remote control, apps and sensors. But, because it is both complex and expensive, there has been little appetite for it in South Africa.
The two main routes for smart home installation are both fraught with peril – financial and technical.
The first is to call on a specialist installation company. Surprisingly, there are many in South Africa. Google “smart home” +”South Africa”, and thousands of results appear. The problem is that, because the industry is so new, few have built up solid track records and reputations. Costs vary wildly, few standards exist, and the cost of after-sales service will turn out to be more important than the upfront price.
The second route is to assemble the components of a smart home, and attempt self-installation. For the non-technical, this is often a non-starter. Not only does one need a fairly good knowledge of Wi-Fi configuration, but also a broad understanding of the Internet of Things (IoT) – the ability for devices to sense their environment, connect to each other, and share information.
The good news, though, is that it is getting easier and more cost effective all the time.
My first efforts in this direction started a few years ago with finding smart plugs on Amazon.com. These are power adaptors that turn regular sockets into “smart sockets” by adding Wi-Fi and an on-off switch, among other. A smart lightbulb was sourced from Gearbest in China. At the time, these were the cheapest and most basic elements for a starter smart home environment.
Via a smartphone app, the light could be switched on from the other side of the world. It sounds trivial and silly, but on such basic functions the future is slowly built.
Fast forward a year or two, and these components are available from hundreds of outlets, they have plummeted in cost, and the range of options is bewildering. That, of course, makes the quest even more bewildering. Who can be trusted for quality, fulfilment and after-sales support? Which products will be obsolete in the next year or two as technology advances even more rapidly?
These are some of the challenges that a leading South African technology distributor, Syntech, decided to address in adding smart home products to its portfolio. It selected LifeSmart, a global brand with proven expertise in both IoT and smart home products.
Equally significantly, LifeSmart combines IoT with artificial intelligence and machine learning, meaning that the devices “learn” the best ways of connecting, sharing and integrating new elements. Because they all fall under the same brand, they are designed to integrate with the LifeSmart app, which is available for Android and iOS phones, as well as Android TV.
Click here to read about how LifeSmart makes installing smart home devices easier.
Matrics must prepare for AI
By Vian Chinner, CEO and founder of Xineoh.
Many in the matric class of 2018 are currently weighing up their options for the future. With the country’s high unemployment rate casting a shadow on their opportunities, these future jobseekers have been encouraged to look into which skills are required by the market, tailoring their occupational training to align with demand and thereby improving their chances of finding a job, writes Vian Chinner – a South African innovator, data scientist and CEO of the machine learning company specialising in consumer behaviour prediction, Xineoh.
With rapid innovation and development in the field of artificial intelligence (AI), all careers – including high-demand professions like engineers, teachers and electricians – will look significantly different in the years to come.
Notably, the third wave of internet connectivity, whereby our physical world begins to merge with that of the internet, is upon us. This is evident in how widespread AI is being implemented across industries as well as in our homes with the use of automation solutions and bots like Siri, Google Assistant, Alexa and Microsoft’s Cortana. So much data is collected from the physical world every day and AI makes sense of it all.
Not only do new industries related to technology like AI open new career paths, such as those specialising in data science, but it will also modify those which already exist.
So, what should matriculants be considering when deciding what route to take?
For highly academic individuals, who are exceptionally strong in mathematics, data science is definitely the way to go. There is, and will continue to be, massive demand internationally as well as locally, with Element-AI noting that there are only between 0 and 100 data scientists in South Africa, with the true number being closer to 0.
In terms of getting a foot in the door to become a successful data scientist, practical experience, working with an AI-focused business, is essential. Students should consider getting an internship while they are studying or going straight into an internship, learning on the job and taking specialist online courses from institutions like Stanford University and MIT as they go.
This career path is, however, limited to the highly academic and mathematically gifted, but the technology is inevitably going to overlap with all other professions and so, those who are looking to begin their careers should take note of which skills will be in demand in future, versus which will be made redundant by AI.
In the next few years, technicians who are able to install and maintain new technology will be highly sought after. On the other hand, many entry level jobs will likely be taken care of by AI – from the slicing and dicing currently done by assistant chefs, to the laying of bricks by labourers in the building sector.
As a rule, students should be looking at the skills required for the job one step up from an entry level position and working towards developing these. Those training to be journalists, for instance, should work towards the skill level of an editor and a bookkeeping trainee, the role of financial consultant.
This also means that new workforce entrants should be prepared to walk into a more demanding role, with more responsibility, than perhaps previously anticipated and that the country’s education and training system should adapt to the shift in required skills.
The matric classes of 2018 have completed their schooling in the information age and we should be equipping them, and future generations, for the future market – AI is central to this.