What to do with the data? The evolution of data platforms in a post big data world

big data world

big data world

Big data world -Note: I’ve had the eminent thought leader Esteban Kolsky, founder and managing principal of ThinkJar, doing guest posts before on this blog. Time and again, the guy simply nails what the core of contemporary thinking is and how to approach it.

This time, he goes to the heart of how the business world is evolving and what it takes to have a transformative success – and that means ecosystems and platforms.

This post is the first of two that he will have here. (Part two comes next week.) The idea for these posts grew out of research that Esteban just finished for Radius, a company that characterizes itself as providing Customer Data Platforms (CDP) for B2B revenue teams. This research inspired more than simply a post with market data; this is significant thinking on where data platforms are going in a world that has solved (more or less) big data.

So, Esteban, start the ball rolling…

Thanks, Paul, for letting me use your blog to spout on data and data platforms. I want to split the research I did in two posts (for easier consumption). First one (this one) on the evolution of data, and the second one (next one) on the evolution of data platforms.

There has been a lot of discussion recently on the “thought leadership interwebz” about what is the best way to aggregate data. We talk about data lakes, swamps, BI, MDM, CDP, and much, much more — but none of this provides a simple solution to the problem of how to optimize data use in a digitally transformed organization.

The problem has recently risen to the executive level, where I am having conversations about the differences between all of them. Where did all this problem start? Glad you asked.

EVOLUTION OF DATA: WHERE IT ALL STARTED

Mind-blowing volumes of data started the problem.

By 2025, the volume of all data created will top 163ZB (zettabytes). Enterprises will experience a 50-fold increase in data they must manage. This is what we started using the last five to six years under the name of big data. As with all technology-only solutions, they quickly became “solutions” looking for problems to solve — not the solution to existing problems.

What is available today is focused on the sheer amount of data available (big data), and how to store it, rather than finding value from it. If we only wanted to process data, the big data movement would’ve been fine, but since we want more (actionable insights became the holy grail of data processing shortly after big data started, and the origin of digital transformation), we need to find different value propositions for that tidal wave of data.

Read More Here

Article Credit: ZDNet

Go to Source

What to do with the data? The evolution of data platforms in a post big data world

big data world

big data world

Big data world -Note: I’ve had the eminent thought leader Esteban Kolsky, founder and managing principal of ThinkJar, doing guest posts before on this blog. Time and again, the guy simply nails what the core of contemporary thinking is and how to approach it.

This time, he goes to the heart of how the business world is evolving and what it takes to have a transformative success – and that means ecosystems and platforms.

This post is the first of two that he will have here. (Part two comes next week.) The idea for these posts grew out of research that Esteban just finished for Radius, a company that characterizes itself as providing Customer Data Platforms (CDP) for B2B revenue teams. This research inspired more than simply a post with market data; this is significant thinking on where data platforms are going in a world that has solved (more or less) big data.

So, Esteban, start the ball rolling…

Thanks, Paul, for letting me use your blog to spout on data and data platforms. I want to split the research I did in two posts (for easier consumption). First one (this one) on the evolution of data, and the second one (next one) on the evolution of data platforms.

There has been a lot of discussion recently on the “thought leadership interwebz” about what is the best way to aggregate data. We talk about data lakes, swamps, BI, MDM, CDP, and much, much more — but none of this provides a simple solution to the problem of how to optimize data use in a digitally transformed organization.

The problem has recently risen to the executive level, where I am having conversations about the differences between all of them. Where did all this problem start? Glad you asked.

EVOLUTION OF DATA: WHERE IT ALL STARTED

Mind-blowing volumes of data started the problem.

By 2025, the volume of all data created will top 163ZB (zettabytes). Enterprises will experience a 50-fold increase in data they must manage. This is what we started using the last five to six years under the name of big data. As with all technology-only solutions, they quickly became “solutions” looking for problems to solve — not the solution to existing problems.

What is available today is focused on the sheer amount of data available (big data), and how to store it, rather than finding value from it. If we only wanted to process data, the big data movement would’ve been fine, but since we want more (actionable insights became the holy grail of data processing shortly after big data started, and the origin of digital transformation), we need to find different value propositions for that tidal wave of data.

Read More Here

Article Credit: ZDNet

Go to Source

7 key features of big data analytics tools to take into account

big data analytics features

big data analytics features

Big data analytics features – Gone are the days when big data analytics was considered as a mere buzzword, now it has successfully ended in becoming a new fact of business life. The trend is being warmly welcomed by organisations across the globe. For those who have no idea regarding big data analytics, it is the process of examining large data sets containing a variety of data types, starting from uncovering hidden patterns to unknown correlations, market trends, customer preferences and so forth.

The situation before Big data

Like I said before, big data describes large volumes, both in the structured and unstructured format generated day in day out. With such technology, one cannot just make better business decisions but will also be able to make some strategic moves. Earlier people used different kinds of system or should I say business intelligence solutions to extract, transform and load data to obtain important reports. The only problem was that the database technology was unable to handle multiple continuous streams of data at a time. As a result, it couldn’t modify the input info in real-time, and the reporting tools couldn’t handle anything other than a relational query on the backend.

With the introduction of big data solutions, reporting interfaces, extraction capabilities, automatic file, optimised and highly indexed data structures, and Cloud hosting also came into existence. Companies can now make better decisions to increase the effectiveness of sales and marketing.

Its benefits include:

#1 All questions are answered-  Managing business or business procedures involves much of answering questions in terms of what do customer want, who are your best and loyal customers, why do people think of choosing different brand or solution, etc. Let’s do some activity, try and figure out who were the 10 worst customers? Before big data, it could take up to 60 days to figure out the answer but after the evolution of such technology, answering these questions become a relatively straight-forward process. In fact, the whole process of answering complex questions can be shortened from months and weeks to days and even hours or minutes.

#2 Gain confidence with accurate data- Incorporating big data into your question/answer process can offer you a complete view of answers but more kind of an accurate view. Previously, taking decisions on the basis of manual data always carried an inherent risk that false or incomplete data could lead to uninformed or even misinformed decisions. With the emergence of big data technology, businesses can gather data from a huge number of sources, reducing the risk of siloed, valuable information. Using inaccurate data isn’t just an inconvenience but even can mislead you to a great extent

Read More Here

Article Credit: ITProPortal

Go to Source

5 ways how Big data analytics help understand cryptocurrency

Big data cryptocurrency

Big data cryptocurrency

Big data cryptocurrency-Big data market is predicted to reach a total value of $275 bn somewhere around the year 2023. More and more companies understand the importance of big data analytics and its influence on market comprehension, consumer trends analytics, detection of consumer patterns, etc. With cryptocurrency reaching its peak last year and establishing a stable business front it was only a matter of time when people are going to introduce big data into cryptocurrency mining process. The benefits of combining big data with cryptocurrency are numerous; however, we’re going to check out those which seem to be on top of the list.

Identifying fake and dangerous users

Safety and anonymity are some of the perks of cryptocurrency trade. However, doing business in the dark carries an amount of risk that you could end up trading with criminals or dangerous organizations, which could easily get your business blacklisted. With more than 200.000 transactions per day, it’s safe to assume that this risk is more than real, and such a vast trade volume makes it even harder to track down the flow of currency.

This is where big data steps in with pattern recognition across transactions and in-depth blockchain data analytics. Just by using these two simple tools it’s easy to identify dangerous users and avoid doing business with them.

Theft prevention

No matter how secure a system may be, it can still leak information or suffer a hack attack. Cryptocurrencies, especially Bitcoin, are very secure and provide a limited amount of public data; however, with the rising tide of data-based hacking and quantum computer technology, the risk of losing all your hard-earned cryptocurrency is very real. You could think that you’re paying for that essay you ordered on aussiewritings.com and find out that it was just a ruse to break into your private information. In order to identify the potential leaks and security hazards, security analysts use big data analysis so they could improve the overall safety and prevent theft.

Predicting trends

Since cryptocurrency value depends heavily on trade volume, it’s important to have some sort of advantage over the market if a person wants to make a profit. According to an article published by IBM’s Ralph Jacobson, there is more than 2.4 Exabyte of data being created on a daily level. Such an ocean of information is a valuable source of data which, when carefully analyzed, can show us in which direction the value and trade volume of Bitcoin or any other cryptocurrency for that matter is going to move.

Read More Here

Article Credit: TG

Go to Source

IoT At The Edge: How High-Performance Processing And Internet Productivity Are Changing Businesses Worldwide

IoT  is now all around us. From consumer products to industrial, agricultural, medical and commercial, it’s changing the way things are done.

IoT

Most IoT devices, at the most basic level, are input and/or output devices that are connected to the internet. The input devices are designed to generate data—cameras produce images in the form of data. Microphones digitize audio as data files. A wide variety of specialized sensors generate data. At the output end, IoT devices can perform an immeasurable range of tasks: A robot can perform complex actions (even surgery when guided by a human surgeon); a speaker can respond to a question or generate verbal instructions; a drone can be controlled by an expert on the ground or in a control room; a door may remotely shut and lock if an intruder is detected.

For IoT devices to provide value beyond passively streaming data to the internet though, intelligence must be built into the device. A camera that just sends a constant stream of video isn’t adding much value. In fact, if you multiply a single camera by the thousands that may also be streaming continuous data, you can end up overwhelming the internet backbone with video images that are of little value. Yet when you add intelligence to the camera—making it a smart camera (enabling computer vision)—configuring it to monitor an area of interest, recognizing the things that it’s programmed to identify, and sending aggregated data at specific intervals (rather than continuously streaming video), you’ve made the camera more valuable.

Similarly, when you enable IoT devices to analyze and operate on the incoming data, the value increases exponentially. Industrial IoT (IIoT) has been around many years. It’s being used on shop floors, controlling robotic manufacturing equipment. It’s being used for visual inspection of parts. One of the key features of IoT and IIoT devices is that they don’t just collect data, they do something with the data. Another key feature is that they can be controlled over the internet.

IoT and IIoT devices are now able to produce usable data and perform tasks in hours or days that used to take weeks to accomplish. For example, Nature Fresh Farms, a company that grows a variety of produce in a 130-acre greenhouse facility, has embraced IoT in its operations. Picked produce is placed on a sorting line. As it passes through the line, an IoT camera that is connected to the internet takes pictures of everything passing by. The pictures are then run through a computer, powered by Intel, and automatically sorted. With intelligent pre-sorting of the produce, the company reports, packaging times have dropped from 35 to 45 seconds down to eight seconds.

A greenhouse automation software system is fed data by IoT sensors that are designed to monitor internal and external temperatures and track weather conditions. Additionally, the software controls the delivery of water and fertilizer, using internet-controlled IoT watering systems. Real-time monitoring, using IoT devices, and a high-performance computer have provided more information about the produce and the greenhouse operation, and have increased product yield while also boosting productivity per employee.

In another example, a product called Robovator is designed to detect and eliminate weeds. The device passes through rows of produce, as a camera scans each row. The image is processed, in real time, and if a weed is detected, the weed is removed or sprayed with a chemical that kills it. Being able to detect and remove weeds this way has reportedly saved a third of the time that was previously required to weed an area.

Field sensors are IoT devices that detect such factors as air temperature, soil temperature, wind and rain. New software rapidly processes this information and can advise a farmer when to fumigate or how much to irrigate in real time. The rapid data acquisition that is enabled by IoT thus enables farmers to increase their produce yields.

IoT Transforming Industries

There are dozens of examples of how IoT and IIoT have improved agriculture and other industries. From automated inventory management systems that use IoT to scan items on shelves more rapidly than any human employee can, to drones with smart cameras that can scan and detect errors in fields of solar panels, to robots that can be used to pick and sort produce, organizations have benefited from the ability to get real-time data from IoT devices. The availability of IoT data can reduce cost, increase accuracy and increase efficiency.

When paired with high-performance computing components, like field programmable gate array (FPGA) chips that are extremely fast and easily reprogrammed to perform new tasks, IoT will change agriculture, medicine, manufacturing, sales and many other businesses for the better.

Go to Source

The Day When the Industrial IoT Gets Hacked

iot hacked

iot hacked

Iot hacked -The more devices that get connected to the industrial internet of things (IIoT) networks, the more that those networks get hacked and attacked. Cyberattacks of all kinds used to be directed mostly at IT networks but not anymore. Many of today’s attackers are going after the industrial control system (ICS) and operational technology (OT) side of the IIoT.

Here, the threats are potentially larger and much more damaging, from ransomware demands to industrial espionage to altering production process code that can change industrial robot safety levels, affect product contents and manufacturing yields, or even cause massive damage.

We break down those attacks and attackers in “Real-life Industrial IoT Cyberattack Scenarios.” To tackle this complex topic, we turned to Ann R. Thryft, the industrial control & automation designline editor at EE Times, who has pioneered our coverage of cybersecurity on the OT side.

Why are these attacks happening more often? Because of a perfect storm of major differences between IT and OT environments. We examine the elements of that storm in “What Makes the IIoT So Vulnerable to Cyberattacks?.”

From the design engineer’s point of view, effective cybersecurity for ICS and everything else in a firm’s IIoT comprises two different but related efforts:

  • On one hand, designing security into an embedded device that forms all, or part of, an IIoT endpoint
  • On the other hand, acquiring and managing cybersecurity technology that protects those devices as they are manufactured in the engineer’s company and as they, and other IIoT devices, are deployed on the company’s factory floor and throughout the plant

In “Designers’ Guide to IIoT Security,” Nitin Dahad, an EE Times European correspondent, addresses the first effort by explaining basic security concepts and design considerations for embedded devices.

Read More Here

Article Credit: EET

Go to Source

What Every Executive Needs To Understand About IoT Security Architectures

About IoT

About IoT

About IoT- Businesses are increasingly part of a highly connected world. The PCs and mobiles used by employees, suppliers and customers to communicate with your enterprise are just the tip of the iceberg. Now, industrial machines, power generators, medical equipment, vehicles and buildings are hooking up with IT systems online, remotely sending and receiving data and commands.

As part of a team focused on online security, I can say that internet of things (IoT) security can be a much bigger deal than PC or mobile security. If a hacker breaks into a mobile phone and compromises a bank account, it may wreck the phone owner’s day. But when a hacker gets into relays for a power grid or the controls of a hospital dialysis machine, entire populations can be put at risk and lives can be threatened.

Start With An Application And A Threat Model

Devices and machines on the IoT are driven by software applications. Each application can be described in terms of its top-level functionality. For instance, pumps at an oil well might be remotely monitored, activated at different levels and shut down. IoT security threats to the pumps might then include attackers stealing data, preventing the pumps from working or forcing them to work at the wrong speed.

There are different models for assessing security threats, several of which apply naturally to the IoT context. These models have their own strengths and weaknesses. Often, the best results are obtained by working with multiple models. These correspond to multiple different ways of thinking about security and possible attacks, giving you a better chance of seeing things as a hacker would and therefore of putting more effective security in place.

Use STRIDE for assessing IoT security threats.

STRIDE is an acronym for the following threat categories:

• Spoofing. Attackers pretend to be someone or something they are not. To continue our example of oil well pumps, an attacker might mimic a command and authorization from a central system to dangerously accelerate pump speeds.

• Tampering. The attacker changes the data that is being transmitted, such as changing a pump status code to read “broken” instead of operational, to force the pump owner to send out a repair person.

• Repudiation. An action happens but the perpetrator then claims not to have done it. Perhaps a third-party maintenance company sends a command to stop an operational pump — deliberately or not — then denies having sent it.

Read More Here

Article Credit: Forbes

Go to Source

The Current State of Public Cloud Enterprise IoT Platforms

current state of iot

current state of iot

Current state of iot – Internet of Things has become one of the critical workloads to run in the public cloud. Especially, enterprises from the manufacturing, healthcare and automobile industries are betting big on IoT PaaS offerings from the public vendors.

Here are ten trends that reflect the current state of enterprise IoT PaaS:

1. Data is the biggest driver for IoT deployments

Customers are adopting cloud-based IoT because of the rich data platform services that complement the core IoT services. Data ingestion, transformation, data storage, processing, analysis and integration with 3rd party databases are the key capabilities of these data platforms. Mature IoT platforms offer real-time stream analytics powered by Apache Spark and batch processing based on Apache Hadoop. This seamless integration between device management and data platforms makes public cloud IoT services attractive for enterprises.

2. IoT solution development is getting complex

IoT adoption is experiencing tremendous growth, and so is the complexity involved in designing and deploying the solutions. What started as a simple device management and connectivity service has transformed into a collection of services increasing the complexity. Like other cloud services, IoT is experiencing fragmentation and service sprawl, which has increased the time and cost of deploying a connected solution in the cloud. Majority of the enterprise IoT solutions involve a system integrator or a solutions partner. The in-house development teams are unable to keep up with the pace and complexity involved in rolling out an IoT solution.

3. There is a steep learning curve for developers involved in IoT

The focus on enterprise and industrial IoT increased the time it takes to deploy a solution. Developers find it challenging to apply existing skills to IoT projects. They are unable to leverage their existing cloud skills to build connected devices and applications. Cloud providers should focus on simplifying the developer experience to drive increased adoption of IoT services.

Read More Here

Article Credit: Forbes

Go to Source

How IoT Is Changing The Science Of Medicine

Iot science- Less than a century ago, medicine transformed from an art to a science. The methods used in medicine became standardized. The scientific approach made it easier to tell what worked—and what didn’t. A standard method for researching, testing and reporting on results developed. Fortunate accidents occurred (think penicillin)—but weren’t implemented until their effectiveness was demonstrated multiple times.

Technology has proven to be a critical driver for effective modern medicine. Where would medicine be without, say, accurate diagnostic equipment? The drive towards improving diagnoses, improving patient care and improving patient outcomes continues to push forward, and the Internet of Things (IoT) is now accelerating things even further. Many modern medical miracles have been accomplished by placing IoT devices at, near or even inside of patients. Here’s a look at how IoT is advancing healthcare today:

  • Keeping The Records: Medical data has moved from hard copy—paper records, X-rays on film, EKGs on paper and other forms of hard copy that are stored for seven years for inactive patients and may never be viewed again—to digital. X-rays, MRIs, CT scans, EKGs, patient records, etc., are either created digitally or digitized for rapid retrieval and to potentially reduce physical storage requirements.

IoT is also used for such things as kiosks where patients can enter data, research medical issues and even sign in for appointments, which ultimately eliminates the mountains of paper that records systems previously used.

  • Seeing What’s There—Better Than Humans Do: Enabling the move towards digital medicine are the many IoT devices involved. X-ray devices are now IoT devices: The images are usually not recorded onto film—instead, sensors inside the X-ray plate detect the X-rays and form a digital image. That image is then moved over the internet where it can be viewed by technicians or medical personnel. In many cases, the images are reviewed before the doctor even sees them—artificial intelligence (AI) is now being used to detect problems earlier than is usually possible for a human expert.

Film X-rays can also be read by scanners that convert the image into a digital file. These X-ray scanners are IoT devices that put the digital X-ray images into patients’ records.

  • From Vision To The Operating Room: Medical IoT goes far beyond X-rays and other diagnostic procedures. A project is currently in the works to help detect skin cancers. This project combines computer vision, powered by Intel Movidius technology that has been trained to process the image, and a library of normal and abnormal skin images. Paired with AI, images can be quickly evaluated for potential diseases.

Read More Here

Article Credit: Forbes

Go to Source

How to Build Trust With A.I.

Trust AI

Trust AI

Trust AI -As recently as a couple of decades ago, artificial intelligence was little more than science fiction. When it started to become a reality, the excitement of it all was overshadowed by fear that smart machines would make humans obsolete.

Today, that fear is tempered by the nearly endless possibilities of AI. Companies like Amazon and Google have embraced innovative AI algorithms, and educators are using it to overhaul an increasingly burdened educational system. In Beijing, researchers have developed a system called BioMind that uses AI to diagnose cancer with unprecedented accuracy.

The possibilities may be endless, but the fear of AI still persists in numerous ways. You might love the convenience of telling Alexa what to do, but what if it’s also listening to your children when it shouldn’t be? If a Tesla vehicle is in an accident, how can you trust that the AI system isn’t at fault? These fears are compounded by the fact that, unlike most other technologies, very few people can easily explain exactly how AI works.

AI systems are artificial neural networks, meaning they are computing systems that are designed to analyze vast amounts of information and learn to perform tasks in the same way your brain does. The algorithms grow through machine learning and adaptation, and sometimes even their initial designers don’t always understand the specific ways in which they evolve.

The implications of letting artificially designed brains make critical decisions are profound. In addition to diagnosing cancer, AI algorithms could potentially be used to guide more comprehensive applications such as municipal project planning, provision of public services, and predictive crime models in urban areas. But, if you don’t understand a machine’s thought process, how can you trust its decisions in those domains?

AI still has a long way to go toward being fully trustworthy and free of bias. But the good news is that any trust you do invest in it isn’t unfounded – because this tech is truly capable of making our lives better. As your relationship with AI grows, keep the following in mind as you determine just how much you should trust the robots:

1. Rather than taking jobs, AI has improved them.

Mistrust was a natural response to the prospect of AI taking over jobs; after all, people’s livelihoods were at stake. But we’ve since learned that automating jobs often leads to different, more advanced opportunities for human employees. Rather than making humans obsolete, implementing AI is paving the way for them to broaden their skills. In fact, a recent Gartner report predicts that AI may eliminate up to 1.8 million jobs by 2020, but it will create 2.3 million additional positions.

So, remember: No matter how much potential AI holds, humans are the ones tapping into it. That means the more AI takes over, the more roles will open up for humans to optimize and maintain it. Or as Gartner’s research director, Manjunath Bhat, puts it, “Robots are not here to take away our jobs, they’re here to give us a promotion.” He predicts that positive impact will continue to be the norm for AI’s future.

2. Humans are still accountable for AI systems.

The key to building trust is time, transparency, and accountability, especially for technology that’s designed to think like humans. Nevertheless, trust can evolve over time. K.R. Sanjiv, CTO at Wipro Limited, emphasizes that until AI is fully explainable, humans will remain accountable. For instance, doctors interpret their patients’ AI-derived pathology reports, and airplane autopilot systems alert human pilots to take over in emergencies. “In each of these cases,” Sanjiv explains, “we allow humans to resolve the uncertainty.”

In addition, keep in mind that AI algorithms are designed to think like humans. Much like your own brain, the appropriate algorithm will cultivate vast stores of data and identify patterns to predict the future. Many attempts are unsuccessful or unsatisfactory because the humans building the system input biased or inaccurate information. AI is smart, but like all technology, it’s a tool — one that you can trust to do what people tell it to do.

Read More Here

Article Credit: INC.

Go to Source