Category Archives: Machine Learning

By:

Dr. Basheer Hawwash, Principal Data Scientist

Amanda Coogan, Risk-Based Monitoring Senior Product Manager

Rhonda Roberts, Senior Data Scientist

Remarque Systems Inc.

Everyone knows the terms machine learning and artificial intelligence. Few can define them, much less explain their inestimable value to clinical trials. So, its not surprising that, despite their ability to minimize risk, improve safety, condense timelines, and save costs, these technology tools are not widely used by the clinical trial industry.

Basheer Hawwash

There are lots of reasons for resistance: It seems complicated. Those who are not statistically savvy may find the thought of algorithms overwhelming. Adopting new technology requires a change in the status quo.

Yet, there are more compelling reasons for adoption especially as the global pandemic has accelerated a trend toward patient-centricity and decentralized trials, and an accompanying need for remote monitoring.

Machine learning vs. artificial intelligence. Whats the difference?

Lets start by understanding what the two terms mean. While many people seem to use them interchangeably, they are distinct: machine learning can be used independently or to inform artificial intelligence; artificial intelligence cannot happen without machine learning.

Machine learning is a series of algorithms that analyze data in various ways. These algorithms search for patterns and trends, which can then be used to make more informed decisions. Supervised machine learning starts with a specific type of data for instance, a particular adverse event. By analyzing the records of all the patients who have had that specific adverse event, the algorithm can predict whether a new patient is also likely to suffer from it. Conversely, unsupervised machine learning applies analysis such as clustering to a group of data; the algorithm sorts the data into groups which researchers can then examine more closely to discern similarities they may not have considered previously.

In either case, artificial intelligence applies those data insights to mimic human problem-solving behavior. Speech recognition, self-driving cars, even forms that auto-populate all exist because of artificial intelligence. In each case, it is the vast amounts of data that have been ingested and analyzed by machine learning that make the artificial intelligence application possible.

Physicians, for instance, can use a combination of machine learning and artificial intelligence to enhance diagnostic abilities. In this way, given a set of data, machine learning tools can analyze images to find patterns of chronic obstructive pulmonary disease (COPD); artificial intelligence may be able to further identify that some patients have idiopathic pulmonary fibrosis (IPF) as well as COPD, something their physicians may neither have thought to look for, nor found unaided.

Amanda Coogan

Now, researchers are harnessing both machine learning and artificial intelligence in their clinical trial work, introducing new efficiencies while enhancing patient safety and trial outcomes.

The case of the missing data

Data is at the core of every clinical trial. If those data are not complete, then researchers are proceeding on false assumptions, which can jeopardize patient safety and even the entire trial.

Traditionally, researchers have guarded against this possibility by doing painstaking manual verification, examining every data point in the electronic data capture system to ensure that it is both accurate and complete. More automated systems may provide reports that researchers can look through but that still requires a lot of human involvement. The reports are static and must be reviewed on an ongoing basis and every review has the potential for human error.

Using machine learning, this process happens continually in the background throughout the trial, automatically notifying researchers when data are missing. This can make a material difference in a trials management and outcomes.

Consider, if you will, a study in which patients are tested for a specific metric every two weeks. Six weeks into the study, 95 percent of the patients show a value for that metric; 5 percent dont. Those values are missing. The system will alert researchers, enabling them to act promptly to remedy the situation. They may be able to contact the patients in the 5 percent and get their values, or they may need to adjust those patients out of the study. The choice is left to the research team but because they have the information in near-real time, they have a choice.

As clinical trials move to new models, with greater decentralization and greater reliance on patient-reported data, missing data may become a larger issue. To counteract that possibility, researchers will need to move away from manual methods and embrace both the ease and accuracy of machine-learning-based systems.

The importance of the outlier

In research studies, not every patient nor even every site reacts the same way. There are patients whose vital signs are off the charts. Sites with results that are too perfect. Outliers.

Rhonda Roberts

Often researchers discover these anomalies deep into the trial, during the process of cleaning the data in preparation for regulatory submission. That may be too late for a patient who is having a serious reaction to a study drug. It also may mean that the patients data are not valid and cannot be included in the end analysis. Caught earlier, there would be the possibility of a course correction. The patient might have been able to stay in the study, to continue to provide data; alternatively, they could be removed promptly along with their associated data.

Again, machine learning simplifies the process. By running an algorithm that continually searches for outliers, those irregularities are instantly identified. Researchers can then quickly drill down to ascertain whether there is an issue and, if so, determine an appropriate response.

Of course, an anomaly doesnt necessarily flag a safety issue. In a recent case, one of the primary endpoints involved a six-minute walk test. One site showed strikingly different results; as it happened, they were using a different measurement gauge, something that would have skewed the study results, but, having been flagged, was easily modified.

In another case, all the patients at a site were rated with maximum quality of life scores and all their blood pressure readings were whole numbers. Machine learning algorithms flagged these results because they varied dramatically from the readings at the other sites. On examination, researchers found that the site was submitting fraudulent reports. While that was disturbing to learn, the knowledge gave the trial team power to act, before the entire study was rendered invalid.

A changing landscape demands a changing approach

As quality management is increasingly focusing on risk-based strategies, harnessing machine learning algorithms simplifies and strengthens the process. Setting parameters based on study endpoints and study-specific risks, machine learning systems can run in the background throughout a study, providing alerts and triggers to help researchers avoid risks.

The need for such risk-based monitoring has accelerated in response to the COVID-19 pandemic. With both researchers and patients unable or unwilling to visit sites, studies have rapidly become decentralized. This has coincided with the emergence and growing importance of patient-centricity and further propelled the rise of remote monitoring. Processes are being forced online. Manual methods are increasingly insufficient and automated methods that incorporate machine learning and artificial intelligence are gaining primacy.

Marrying in-depth statistical thinking with critical analysis

The trend towards electronic systems does not replace either the need for or the value of clinical trial monitors and other research personnel; they are simply able to do their jobs more effectively. A machine-learning-based system runs unique algorithms, each analyzing data in a different way to produce visualizations, alerts, or workflows, which CROs and sponsors can use to improve patient safety and trial efficiency. Each algorithm is tailored to the specific trial, keyed to endpoints, known risks, or other relevant factors. While the algorithms offer guidance, the platform does not make any changes to the data or the trial process; it merely alerts researchers to examine the data and determine whether a flagged value is clinically significant. Trial personnel are relieved of much tedious, reproducible, manual work, and are able to use their qualifications to advance the trial in other meaningful ways.

The imperative to embrace change

Machine learning and artificial intelligence have long been buzzwords in the clinical trial industry yet these technologies have only haltingly been put to use. Its time for that pendulum to swing. We can move more quickly and more precisely than manual data verification, and data cleaning allow. We can work more efficiently if we harness data to drive trial performance rather than simply to prove that the study endpoints were achieved. We can operate more safely if we are programmed for risk management from the outset. All this can be achieved easily, with the application of machine learning and artificial intelligence. Now is the time to move forward.

Here is the original post:
How machine learning and artificial intelligence can drive clinical innovation - PharmaLive

It is unarguably true that the advent of machine learning and artificial intelligence has brought a revolutionary change in various industries globally. Both these technologies have made applications and machines way smarter than our imaginations. But, have you ever wondered how AI and ML work or how they make machines act, think, and behave like human beings.

To understand this, you have to dig deeper into the technical things. It is actually the trained data sets that do the magic to create automated machines and applications. These data sets are further needed to be created and trained through a process named Data annotation.

Data annotation is the technique of labeling the data, which is present in different formats such as images, texts, and videos. Labeling the data makes objects recognizable to computer vision, which further trains the machine. In short, the process helps the machine to understand and memorize the input patterns.

To create a data set required for machine learning, different types of data annotation methods are available. The prime aim of all these types of annotations is to help a machine to recognize text, images, and videos (objects) via computer vision.

Bounding boxesLines and splinesSemantic segmentation3D cuboidsPolygonal segmentationLandmark and key-pointImages and video annotationsEntity annotationContent and text categorization

Lets read them in detail:

The most common kind of data annotation is bounding boxes. These are the rectangular boxes used to identify the location of the object. It uses x and y-axis coordinates in both the upper-left and lower-right corners of the rectangle. The prime purpose of this type of data annotation is to detect the objects and locations.

This type of data annotation is created by lines and splines to detect and recognize lanes, which is required to run an autonomous vehicle.

This type of annotation finds its role in situations where environmental context is a crucial factor. It is a pixel-wise annotation that assigns every pixel of the image to a class (car, truck, road, park, pedestrian, etc.). Each pixel holds a semantic sense. Semantic segmentation is most commonly used to train models for self-driving cars.

This type of data annotation is almost like bounding boxes but it provides extra information about the depth of the object. Using 3D cuboids, a machine learning algorithm can be trained to provide a 3D representation of the image.

The image can further help in distinguishing the vital features (such as volume and position) in a 3D environment. For instance- 3D cuboids help driverless cars to utilize the depth information to find out the distance of objects from the vehicle.

Polygonal segmentation is used to identify complex polygons to determine the shape and location of the object with the utmost accuracy. This is also one of the common types of data annotations.

These two annotations are used to create dots across the image to identify the object and its shape. Landmark and key-point annotations play their role in facial recognitions, identifying body parts, postures, and facial expressions.

Entity annotation is used for labeling unstructured sentences with the relevant information understandable by a machine. It can be further categorized into named entity recognition and intent extraction.

Data annotation offers innumerable advantages to machine learning algorithms that are responsible for training predicting data. Here are some of the advantages of this process:

Enhanced user experience

Applications powered by ML-based trained models help in delivering a better experience to end-users. AI-based chatbots and virtual assistants are a perfect example of it. The technique makes these chatbots to provide the most relevant information in response to a users query.

Improved precision

Image annotations increase the accuracy of output by training the algorithm with huge data sets. Leveraging these data sets, the algo will learn various kinds of factors that will further assist the model to look for the suitable information in the database.

The most common annotation formats include:

COCOYOLOPascal VOC

By now, you must be aware of the different types of data annotations. Lets check out the applications of the same in machine learning:

Sequencing- It includes text and time series and a label.

Classification- Categorizing the data into multiple classes, one label, multiple labels, binary classes, and more.

Segmentation- It is used to search the position where a paragraph splits, search transitions between different topics, and for various other purposes.

Mapping- It can be done for language to language translation, to convert a complete text into the summary, and to accomplish other tasks.

Check out below some of the common tools used for annotating images:

RectlabelLabelMeLabelImgMakeSense.AIVGG image annotator

In this article, we have mentioned what data annotation or labeling is, and what are its types and benefits. Besides this, we have also listed the top tools used for labeling images. The process of labeling texts, images, and other objects help ML-based algorithms to improve the accuracy of the output and offer an ultimate user experience.

A reliable and experienced machine learning company holds expertise on how to utilize these data annotations for serving the purpose an ML algorithm is being designed for. You can contact such a company or hire ML developers to develop an ML-based application for your startup or enterprise.

Read More: How does Machine Learning Revolutionizing the Mobile Applications?

Read the original post:
Data Annotation- Types, Tools, Benefits, and Applications in Machine Learning - Customer Think

Apples artificial intelligence (AI) chief says that Apple is using machine learning in almost every aspect of how we interact with our devices, but there is much more to come.

John Giannandrea says he moved from Google to Apple because the potential of machine learning (ML) to impact peoples lives is so much greater at the Cupertino company

Giannandrea spoke with ArsTechnicas Samuel Axon, outlining how Apple uses ML now.

Theres a whole bunch of new experiences that are powered by machine learning. And these are things like language translation, or on-device dictation, or our new features around health, like sleep and hand washing, and stuff weve released in the past around heart health and things like this. I think there are increasingly fewer and fewer places in iOS where were not using machine learning.

Its hard to find a part of the experience where youre not doing some predicative [work]. Like, app predictions, or keyboard predictions, or modern smartphone cameras do a ton of machine learning behind the scenes to figure out what they call saliency, which is like, whats the most important part of the picture? Or, if you imagine doing blurring of the background, youre doing portrait mode []

Savvy iPhone owners might also notice that machine learning is behind the Photos apps ability to automatically sort pictures into pre-made galleries, or to accurately give you photos of a friend named Jane when her name is entered into the apps search field []

Most [augmented reality] features are made possible thanks to machine learning []

Borchers also pointed out accessibility features as important examples. They are fundamentally made available and possible because of this, he said. Things like the sound detection capability, which is game-changing for that particular community, is possible because of the investments over time and the capabilities that are built in []

All of these things benefit from the core machine learning features that are built into the core Apple platform. So, its almost like, Find me something where were not using machine learning.

He was, though, surprised at areas where Apple had not been using ML before he joined the company.

When I joined Apple, I was already an iPad user, and I loved the Pencil, Giannandrea (who goes by J.G. to colleagues) told me. So, I would track down the software teams and I would say, Okay, wheres the machine learning team thats working on handwriting? And I couldnt find it.It turned out the team he was looking for didnt exista surprise, he said, given that machine learning is one of the best tools available for the feature today.

I knew that there was so much machine learning that Apple should do that it was surprising that not everything was actually being done.

That has changed, and will continue to change, however.

That has changed dramatically in the last two to three years, he said. I really honestly think theres not a corner of iOS or Apple experiences that will not be transformed by machine learning over the coming few years.

Its long been thought that Apples privacy focus wanting to do everything on the device, and not analyzing huge volumes of personal data means that it cant compete with Google, because it cant benefit from masses of data pulled from millions of users. Giannandrea says this is absolutely not the case.

I understand this perception of bigger models in data centers somehow are more accurate, but its actually wrong. Its actually technically wrong. Its better to run the model close to the data, rather than moving the data around.

In other words, you get better results when an ML model learns from your usage of your device than when it relies on aggregated data from millions of users. Local processing can also be used in situations where it simply wouldnt be realistic to send data to a server, like choosing the exact moment to act on you pressing the Camera app shutter release button for the best frame.

Understandably, Giannandrea wouldnt be drawn on what Apple is working on now, but did give one example of what might be possible when you combine the power of Apple Silicon Macs with machine learning.

Imagine a video editor where you had a search box and you could say, Find me the pizza on the table. And it would just scrub to that frame.

The whole piece is very much worth reading.

Photo: WFMJ

FTC: We use income earning auto affiliate links. More.

Check out 9to5Mac on YouTube for more Apple news:

Read more:
Apple using machine learning for almost everything, and privacy-first approach actually better - 9to5Mac

Key Takeaways

In recent years, machine learning has been very successful in solving a wide range of problems.

In particular, neural networks have reached human, and sometimes super-human, levels of ability in tasks such as language translation, object recognition, game playing, and even driving cars.

With this growth in capability has come a growth in complexity. Data scientists and machine learning engineers must perform feature engineering, design model architectures, and optimize hyperparameters.

Since the purpose of the machine learning is to automate a task normally done by humans, naturally the next step is to automate the tasks of data scientists and engineers.

This area of research is called automated machine learning, or AutoML.

There have been many exciting developments in AutoML recently, and it's important to take a look at the current state of the art and learn about what's happening now and what's coming up in the future.

InfoQ reached out to the following subject matter experts in the industry to discuss the current state and future trends in AutoML space.

InfoQ:What is AutoML and why is it important?

Francesca Lazzeri:AutoML is the process of automating the time consuming, iterative tasks of machine learning model development, including model selection and hyperparameter tuning. When automated systems are used, the high costs of running a single experiment (e.g. training a deep neural network) and the high sample complexity (i.e. large number of experiments required) can be decreased. Auto ML is important because data scientists, analysts, and developers across industries can leverage it to:

Matthew Tovbin:Similarly to how we use software to automate repetitive or complex processes, automated machine learning is a set of techniques we apply to efficiently build predictive models without manual effort. Such techniques include methods for data processing, feature engineering, model evaluation, and model serving. With AutoML, we can focus on higher-level objectives such as answering questions and delivering business values faster while avoiding mundane tasks, e.g., data wrangling, by standardizing the methods we apply.

Adrian de Wynter:AutoML is the idea that the machine learning process, from data selection to modeling, can be automated by a series of algorithms and heuristics. In its most extreme version, AutoML is a fully automated system: you give it data, and it returns a model (or models) that generalizes to unseen data. The common hurdles that modelers face, such as tuning hyperparameters, feature selection--even architecture selection--are handled by a series of algorithms and heuristics.

I think its importance stems from the fact that a computer does precisely what you want it to do, and it is fantastic at repetition. The large majority of the hurdles I mentioned above are precisely that: repetition. Finding a hyperparameter set that works for a problem is arduous. Finding a hyperparameter set and an architecture that works for a problem is even harder. Add to the mix data preprocessing, the time spent on debugging code, and trying to get the right environment to work, and you start wondering whether computers are actually helping you solve said problem, or just getting in the way. Then, you have a new problem, and you have to start all over again.

The key insight of AutoML is that you might be able to get away by using some things you tried out before (i.e., your prior knowledge) to speed up your modeling process. It turns out that said process is effectively an algorithm, and thus it can be written into a computer program for automation.

Leah McGuire:AutoML is machine learning experts automating themselves. Creating quality models is a complex, time-consuming process. It requires understanding the dataset and question to be answered. This understanding is then used to collect and join the needed data, select features to use, clean the data and features, transform the features into values that can be used by a model, select an appropriate model type for the question, and tune feature-engineering and model parameters. AutoML uses algorithms based on machine learning best practices to build high-quality models without time-intensive work from an expert.

AutoML is important because it makes it possible to create high quality models with less time and expertise. Companies, non-profits, and government agencies all collect vast amounts of data; in order for this data to be utilized, it needs to be synthesized to answer pertinent questions. Machine learning is an effective way of synthesizing data to answer relevant questions, particularly if you do not have the resources to employ analysts to spend huge amounts of time looking at the data. However, machine learning requires both expertise and time to implement. AutoML seeks to decrease these barriers. This means that more data can be analyzed and used to make decisions.

Marios Michailidis:Broadly speaking, I would call it the process of automatically deriving or extracting useful information from data via harnessing the power of machines. Digital data is being produced at an incredible pace. Now that companies have found ways to harness it to extract value, it has become imperative to invest in data science and machine learning. However, the supply of data science (in human resource) is not enough to meet the current needs, hence making existing data scientists more productive is of the essence. This is where the notion of automated machine learning can provide the most value, via equipping the existing data scientists with tools and processes that can make their work easier, quicker, and generally more efficient.

InfoQ:What parts of the ML process can be automated and what are some parts unlikely to be automated?

Lazzeri:With Automated ML, the following tasks can be automated:

However, there are a few important tasks that cannot be automated during the model development cycle, such us developing industry-specific knowledge and data acumen, which are hard to automate and it is impossible to not keep humans in the loop. Another important aspect to consider is about operationalizing machine learning models: AutoML is very useful for the machine learning model development cycle; however, for the automation of the deployment step, there are other tools that need to be used, such as MLOps, which enables data science and IT teams to collaborate and increase the pace of model development and deployment via monitoring, validation, and governance of machine learning models.

Tovbin:Through the years of development of the machine learning domain, we have seen that a large number of tasks around data manipulation, feature engineering, feature selection, model evaluation, hyperparameter tuning can be defined as an optimization problem and, with enough computing power, efficiently automated. We can see numerous proofs for that not only in research but also in the software industry as platform offerings or open-source libraries. All these tools use predefined methods for data processing, model training, and evaluation.

The creative approach to framing problems and applying new techniques to existing problems is the one that is not likely to be replicated by machine automation, due to a large number of possible permutations, complex context, and expertise the machine lacks. As an example, look at the design of neural net architectures and their applications, a problem where the search space is so ample, where the progress is still mostly human-driven.

de Wynter:In theory, the entire ML process is computationally hard. From fitting data to, say, a neural network, to hyperparameter selection, to neural architecture search (NAS), these are all hard problems in the general case. However, all of these components have been automated with varying degrees of success for specific problems thanks to a combination of algorithmic advances, computational power, and patience.

I would like to think that the data preprocessing step and feature selection processes are the hardest to automate, given that a machine learning model will only learn what it has seen, and its performance (and hence the solution provided by the system) is dependent on its input. That said, there is a growing body of research on that aspect, too, and I hope that it will not remain hard for many natural problems.

McGuire:I would break the process of creating a machine learning model into four main components: data ETL and cleaning, feature engineering, model selection and tuning, and model explanation and evaluation.

Data cleaning can be relatively straight forward or incredibly challenging, depending on your data set. One of the most important factors is history; if you have information about your data at every point in time, data cleaning can be automated quite well. If you have only a static representation of current state, cleaning becomes much more challenging. Older data systems designed before relatively cheap storage tend to keep only the current state of information. This means that many important datasets do not have a history of actions taken on the data. Cleaning this type of history-less data has been a challenge for AutoML to provide good quality models for our customers.

Feature engineering is - again - a combination of easy and extremely difficult to automate steps. Some types of feature engineering are easy to automate given sufficient metadata about particular features. For example, parsing a phone number to validate and extract the location from the area code is straightforward as long as you know that a particular string is a phone number. However, feature engineering that requires intimate, domain-specific knowledge of how a business works are unlikely to be automated. For example, if profits from a sale need to account for local taxes before being analyzed for cost-to-serve, some human input is likely required to establish this relationship (unless you have a massive amount of data to learn from). One reason deep learning has overtaken feature engineering in fields like vision and speech is the massive amounts of high quality training data. Tabular data is often quite source specific making it difficult to generalize and feature engineering remains a challenge. In addition, defining the correct way to combine sources of data is often incredibly complex and labor intensive. Once you have the relationship defined, the combination can be automated, but establishing this relationship takes a fair amount of manual work and is unlikely to be automated any time soon.

Model selection and tuning is the easiest component to automate and many libraries already do this; there are even AutoML algorithms to find entirely new deep learning architectures. However, model selection and tuning libraries assume that the data you are using for modeling is clean and that you have a good way of evaluating the efficacy of your model. Massive data sets also help. Establishing clean datasets and evaluation frameworks still remain the biggest challenges.

Model explanations have been an important area of research for machine learning in general. While it is not strictly speaking part of AutoML, the growth of AutoML makes it even more important. It is also the case that the way in which you implement automation has implications for explainability. Specifically tracking metadata about what was tried and selected determines how deep explanations can go. Building explanations into AutoML requires a conscious effort and is very important. At some point the automation has to stop and someone will look at and use the result. The more information the model provides about how it works the more useful it is to the end consumer.

Michailidis:I would divide the areas where automation can be applied to the following main areas:

Regarding problems which are hard to automate, the first thing that pops into my mind is anything related to translating the business problem into a machine learning problem. For AutoML to succeed, it would require mapping the business problem into a type of solvable machine learning problem. It will also need to be supported by the right data quality/relevancy. The testing of the model and the success criteria need to be defined carefully by the data scientist.

Another area that will be hard for AutoML to succeed is whenethical dilemmasmay arise from the use of machine learning. For example, if there is an accident involved due to an algorithmic error, who will be responsible? I feel this kind of situation can be a challenge for AutoML.

InfoQ: What type of problems or use cases are better candidates to use AutoML?

Lazzeri:Classification, regression, and time series forecasting are the best candidates for AutoML. Azure Machine Learning offers featurizations specifically for these tasks, such as deep neural network text featurizers for classification.

Common classification examples include fraud detection, handwriting recognition, and object detection. Different from classification where predicted output values are categorical, regression models predict numerical output values based on independent predictors. For example automobile price based on features like, gas mileage, safety rating, etc.

Finally, building forecasts is an integral part of any business, whether its revenue, inventory, sales, or customer demand. Data Scientists can use automated ML to combine techniques and approaches and get a recommended, high-quality time series forecast.

Tovbin:Classification or regression problems relying on structured or semi-structured data, where one can define an evaluation metric, can usually be automated. For example, predicting user churn, real estate price prediction, autocomplete.

de Wynter:It depends. Let us assume that you want the standard goal of machine learning: you need to learn an unseen probability distribution from samples. You also know that there is some AutoML system that does an excellent job for various, somewhat related tasks. Theres absolutely no reason why you shouldnt automate it, especially if you dont have the time to be trying out possible solutions by yourself.

I do need to point out, however, that in theory a model that performs well for a specific problem does not have any guarantees around other problemsin fact, it is well-known that there exists at least one task where it will fail. Still, this statement is quite general and can be worked around in practice.

On the other hand, from an efficiency point of view, a problem that has been studied for years by many researchers might not be a great candidate, unless you are particularly interested in marginal improvements. This follows immediately from the fact that most AutoML results, and more concretely, NAS results, for well-known problems usually are equivalent within a small delta to the human-designed solutions. However, making the problem "interesting" (e.g., by including newer constraints such as parameter size) makes it effectively a new problem, and again perfect for AutoML.

McGuire:If you have a clean dataset that has a very well defined evaluation method it is a good candidate for AutoML. Early advances in AutoML have focused on areas such as hyper parameter tuning. This is a well defined but time consuming problem. These AutoML solutions are essentially taking advantage of increases in computational power combined with models of the problem space to arrive at solutions that are often better than an expert could achieve with less human time input. The key here is the clean dataset with a direct and easily measurable effect on the well defined evaluation set. AutoML will maximize your evaluation criteria very well. However, if there is any mismatch between that criteria and what you are trying to do or any confounding factors in the data AutoML will not see that in the way a human expert (hopefully) would.

Michailidis:Well-defined problemsare good use cases for AutoML. In these problems, the preparatory work has already been done. There are clear inputs and outputs and well-defined success criteria. Under these constraints, AutoML can produce the best results.

InfoQ: What are some important research problems in AutoML?

Lazzeri:An interesting research open question in AutoML is the problem of feature selection in supervised learning tasks. This is also called the differentiable feature selection problem, a gradient-based search algorithm for feature selection. Feature selection remains a crucial step in machine learning pipelines and continues to see active research: a few researchers from Microsoft Research are developing a feature selection method that is statistically efficient and computationally efficient.

Tovbin:The two significant ones that come to my mind are the transparency and bias of trained models.

Both experts and users often disagree or do not understand why ML systems, especially automated ones, make specific predictions. It is crucial to provide deeper insights into model predictions to allow users to gain confidence in such predictive systems. For example, when providing recommendations of products to consumers, a system can additionally highlight the contributing factors that influenced particular recommendations. In order to provide such functionality, in addition to the trained model, one would need to maintain additional metadata and expose it together with provided recommendations, which often cannot be easily achieved due to the size of the data or privacy concerns.

The same concerns apply to model bias, but the problem has different roots, e.g., incorrect data collection resulting in skewed datasets. This problem is more challenging to address because we often need to modify business processes and costly software. With applied automation, one can detect invalid datasets and sometimes even data collection practices early and allow removing bias from model predictions.

de Wynter:I think first and foremost, provably efficient and correct algorithms for hyperparameter optimization (HPO) and NAS. The issue with AutoML is that you are solving the problem of, well, problem solving (or rather, approximation), which is notoriously hard in the computational sense. We as researchers often focus on testing a few open benchmarks and call it a day, but, more often than not, such algorithms fail to generalize, and, as it was pointed out last year, they tend to not outperform a simple random search.

There is also the issue that from a computational point of view, a fully automated AutoML system will face problems that are not necessarily similar to the ones that it has seen before; or worse, they might have a similar input but completely different solutions. Normally, this is related to the field of "learning to learn", which often involves some type of reinforcement learning (or neural network) to learn how previous ML systems solved a problem, and approximately solve a new one.

McGuire:I think there is a lot of interesting work to do on automating feature engineering and data cleaning. This is where most of the time is spent in machine learning and domain expertise can be hugely important. Add to that the fact that most real world data is extremely messy and complex and you see that the biggest gains from automation are from automating as much data processing and transformation as possible.

Automating the data preparation work that currently takes a huge amount of human expertise and time is not a simple task. Techniques that have removed the need for custom feature engineering in fields like vision and language do not currently generalize to small messy datasets. You can use deep learning to identify pictures of cats because a cat is a cat and all you need to do is get enough labeled data to let a complex model fill in the features for you. A table tracking customer information for a bank is very different from a table tracking customer information for a clothing store. Using these datasets to build models for your business is a small data problem. Such problems cannot be solved simply by throwing enough data at a model that can capture the complexities on its own. Hand cleaning and feature engineering can use many different approaches and determining the best is currently something of an art form. Turning these steps into algorithms that can be applied across a wide range of data is a challenging but important area of research.

Being able to automatically create and more importantly explain models of such real world data is invaluable. Storage is cheap but experts are not. There is a huge amount of data being collected in the world today. Automating the cleaning and featurization of such data provides the opportunity to use it to answer important real world questions.

Michailidis:I personally find the area of (automation-aided)explainable AIand machine learning interpretability very interesting and very important for bridging the gap between Blackbox modelling and a model that stakeholders can comfortably trust.

Another area I am interested in is "model compression". I think it can be a huge game changer if we can automatically go from a powerful, complicated solution down to a much simpler one that canbasically produce the same/similar performance, but much faster, utilizing less resources.

InfoQ What are some AutoML techniques and open-source tool practitioners can use now?

Lazzeri:AutoML democratizes the machine learning model development process, and empowers its users, no matter their data science expertise, to identify an end-to-end machine learning pipeline for any problem. There are several AutoML techniques that practitioners can use now, my favorite ones are:

Tovbin:In recent years we have seen an explosion of tooling for machine learning practitioners starting from cloud platforms (Google Cloud AutoML, Salesforce Einstein, AWS SageMaker Autopilot, H2O AutoML) to open-source software (TPOT, AutoSklearn, TransmogrifAI). Here one can find more information on these and other solutions:

de Wynter:Disclaimer: I work for Amazon. This is an active area of research, and theres quite a few well-known algorithms (with more appearing every day) focusing on different parts of the pipeline, and with well-known successes on various problems. Its hard to name them all, but some of the best-known examples are grid search, Bayesian, and gradient-based methods for HPO; and search strategies (e.g., hill climbing), population/RL-based methods (e.g., ENAS, DARTS for one-shot NAS, and the algorithm used for AmoebaNet) for NAS. On the other hand, full end-to-end systems have achieved good results for a variety of problems.

McGuire:Well of course I need to mention our own open source AutoML library TransmogrifAI. We focus mainly on automating data cleaning and feature engineering with some model selection and are built on top of Spark.

There are also a large number of interesting AutoML libraries coming out in python including Hyperopt, scikit-optimize, and TPOT.

Michailidis:In the open source space, H2O.ai for has a tool called AutoML, that incorporates many of the elements I mentioned in the previous questions. It is also very scalable and can be used in any OS.Other tools are the autosklearnor autoweka.

InfoQ: What are the limitations of AutoML?

Lazzeri:Auto ML is raising a few challenges such as model parallelization, result collection, resource optimization, and iteration. Searching for the best model and hyperparameters is an iterative process constrained by many limitations, such as compute, money and time. Machine learning pipelines provide a solution to answer those AutoML challenges with a clear definition of the process and automation features. Azure Machine Learning pipeline is an independently executable workflow of a complete machine learning task. Pipelines should focus on machine learning tasks such as:

Tovbin:One problem that AutoML does not handle well is complex data types. The majority of automated methods expect certain data types, e.g., numerical, categorical, text, geo coordinates, and, therefore, specific distributions. Such methods are a poor fit to handle more complicated scenarios, such as behavioral data, e.g., online store visit sessions.

Another problem is feature engineering that needs to consider domain-specific properties of the data. For example, if we would like to build a system to automate email classification for an insurance sales team. The input from the sales team members that define which parts of the email are and are not necessary would usually be more valuable than a metric. When building such systems, it is essential to reinforce the system with domain expert feedback to achieve more reliable results.

de Wynter:There is the practical limitation of the sheer amount of computational resources you have to throw at a problem to get it solved. It is not a true obstacle insofar as you can always use more machines, but--environmentally speakingthere are consequences associated with such a brute-force approach. Now, not all of AutoML is brute-force (as I mentioned earlier, this is a computationally hard problem, so brute-forcing a problem will only get you so far), and relies heavily on heuristics, but you still need sizable compute to solve a given AutoML problem, since you have to try out multiple solutions end-to-end. Theres a push in the science community to obtain better, "greener" algorithms, and I think its fantastic and the way to go.

From a theoretical point of view, the hardness of AutoML is quite interestingultimately, it is a statement on how intrinsically difficult the problem is, regardless of what type or number of computers you use. Add to that what I mentioned earlier that there is no such thing as "one model to rule them all," (theoretically) and AutoML becomes a very complex computational problem.

Lastly, current AutoML systems have a well-defined model search space (e.g., neural network layers, or a mix of classifiers), which is expected to work for every input problem. This is not the case. However, the search spaces that provably generalize well for all possible problems are somewhat hard to implement in practice, so there is still an open question on how to bridge such a gap.

McGuire:I dont think AutoML is ready to replace having a human in the loop. AutoML can build a model, but as we automate more and more of modeling, developing tools to provide transparency into what the model is doing becomes more and more important. Models are only as good as the data used to build them. As we move away from having a human spending time to clean and deeply understand relationships in the data we need to provide new tools to allow users of the model to understand what the models are doing. You need a human to take a critical look at the models and the elements of the data they use and ask: is this the right thing to predict, and is this data OK to use? Without tools to answer these questions for AutoML models we run the risk unintentionally shooting ourselves in the foot. We need the ability to ensure we are not using inappropriate models or perpetuating and reinforcing issues and biases in society without realizing it.

Michailidis:This was covered mostly in previous sections. Another thing I would like to mention is that performance is greatly affected by theresources allocated. More powerful machines will be to cover a search space of potential algorithms, features and techniques much faster.

These tools (unless they are built to support very specific applications)do not have domain knowledgebut are made to solve generic problems. For example, they would not know out of the box that if a field in the data is called "distance travelled" and another one is called "duration in time" , they can be used to compute "speed" which may be an important feature for a given task. They may have a chance to generate that feature via stochastically trying different transformations in the data but a domain expert would figure this out much quicker, hence these tools will produce better results under the hands of an experienced data practitioner. Hence, these tools will be more successful if they have the option to incorporate domain knowledge coming from the expert.

The panelists agreed that AutoML is important because it saves time and resources, removing much of the manual work and allowing data scientist to deliver business value faster and more efficiently. The panelists predict, however, that AutoML will not likely remove the need for a "human in the loop," particularly for industry-specific knowledge and the ability to translate business problems into machine-learning problems. Important research areas in AutoML include feature engineering and model explanation.

The panelists highlighted several existing commercial and open-source AutoML tools and described the different parts of the machine-learning process that can be automated. Several panelists noted that one limitation of AutoML is the amount of computational resources required, while others pointed out the need for domain knowledge and model transparency.

Francesca Lazzeri, PhD is an experienced scientist and machine learning practitioner with over 12 years of both academic and industry experience. She is the author of a number of publications, including technology journals, conferences, and books. She currently leads an international team of cloud advocates and AI developers at Microsoft. Before joining Microsoft, she was a research fellow at Harvard University in the Technology and Operations Management Unit. Find her on Twitter:@frlazzeriand Medium:@francescalazzeri

Matthew Tovbinis a Co-Founder of Faros AI, a software automation platform for DevOps. Before founding Faros AI, he acted as Software Engineering Architect at Salesforce, developing the Salesforce Einstein AI platform, which powers the worlds smartest CRM. In addition, Matthew is a creator of TransmogrifAI, co-organizer of Scala Bay meetup, presenter and an active member in numerous functional programming groups. Matthew lives in the San Francisco Bay area with his wife and kid, enjoys photography, hiking, good whisky and computer gaming.

Adrian de Wynteris an Applied Scientist in Alexa AIs Secure AI Foundations organization. His work can be categorized in three broad, sometimes overlapping, areas: language modeling, neural architecture search, and privacy-preserving machine learning. His research interests involve meta-learning and natural language understanding, with a special emphasis on the computational foundations of these topics.

Leah McGuireis a Machine Learning Architect at Salesforce, working on automating as many of the steps involved in machine learning as possible. This automation has been instrumental in developing and shipping a number of customer facing machine learning offerings at Salesforce. Our goal is to bring intelligence to each customers unique data and business goals. Before focusing on developing machine learning products, she completed a PhD and a Postdoctoral Fellowship in Computational Neuroscience at the University of California, San Francisco, and at University of California, Berkeley, where she studied the neural encoding and integration of sensory signals.

MariosMichailidisis a Competitive data scientist at H2O.ai, developing the next generation of machine learning products in the AutoML space. He holds a Bsc in accounting Finance from the University of Macedonia in Greece, an Msc in Risk Management from the University of Southampton and a PhD in machine learning from the University College London (UCL) with focus on ensemble modelling. He is the creator ofKazAnova, a freeware GUI for credit scoring and data mining 100% made in Java as well as is the creator ofStackNet Meta-Modelling Framework. In his spare time he loves competing on data science challenges where he was ranked1st out of 500,000 members in the popular Kaggle.comdata science platform.

Read more from the original source:
State of the Art in Automated Machine Learning - InfoQ.com

I remember the first time I ever tried to learn to code. I was in middle school, and my dad, a programmer himself, pulled open a text editor and typed this on the screen:

Excuse me? I said.

It prints Hello World, he replied.

Whats public? Whats class? Whats static? Whats

Ignore that for now. Its just boilerplate.

But I was pretty freaked out by all that so-called boilerplate I didnt understand, and so I set out to learn what each one of those keywords meant. That turned out to be complicated and boring, and pretty much put the kibosh on my young coder aspirations.

Its immensely easier to learn software development today than it was when I was in high school, thanks to sites likecodecademy.com, the ease of setting up basic development environments, and a generalsway towards teaching high-level, interpreted languageslike Python and Javascript. You can go from knowing nothing about coding to writing your first conditional statements in a browser in just a few minutes. No messy environmental setup, installations, compilers, or boilerplate to deal with you can head straight to the juicy bits.

This is exactly how humans learn best. First, were taught core concepts at a high level, and onlythencan we appreciate and understand under-the-hood details and why they matter. We learn Python,thenC,thenassembly, not the other way around.

Unfortunately, lots of folks who set out to learn Machine Learning today have the same experience I had when I was first introduced to Java. Theyre given all the low-level details up front layer architecture, back-propagation, dropout, etc and come to think ML is really complicated and that maybe they should take a linear algebra class first, and give up.

Thats a shame, because in the very near future, most software developers effectively using Machine Learning arent going to have to think or know about any of that low-level stuff. Just as we (usually) dont write assembly or implement our own TCP stacks or encryption libraries, well come to use ML as a tool and leave the implementation details to a small set of experts. At that point after Machine Learning is democratized developers will need to understand not implementation details but instead best practices in deploying these smart algorithms in the world.

Today, if you want to build a neural network that recognizes your cats face in photos or predicts whether your next Tweet will go viral, youd probably set off to learn eitherTensorFloworPyTorch. These Python-based deep learning libraries are the most popular tools for designing neural networks today, and theyre both under 5 years old.

In its short lifespan, TensorFlow has already become way,waymore user-friendly than it was five years ago. In its early days, you had to understand not only Machine Learning but also distributed computing and deferred graph architectures to be an effective TensorFlow programmer. Even writing a simple print statement was a challenge.

Just earlier this fall, TensorFlow 2.0 officially launched, making the framework significantly more developer-friendly. Heres what a Hello-World-style model looks like in TensorFlow 2.0:

If youve designed neural networks before, the code above is straight-forward and readable. But if you havent or youre just learning, youve probably got some questions. Like, what is Dropout? What are these dense layers, how many do you need and where do you put them? Whatssparse_categorical_crossentropy? TensorFlow 2.0 removes some friction from building models, but it doesnt abstract away designing the actual architecture of those models.

So what will the future of easy-to-use ML tools look like? Its a question that everyone from Google to Amazon to Microsoft and Apple are spending clock cycles trying to answer. Also disclaimer it is whatIspend all my time thinking about as an engineer at Google.

For one, well start to see many more developers using pre-trained models for common tasks, i.e. rather than collecting our own data and training our own neural networks, well just use Googles/Amazons/Microsofts models. Many cloud providers already do something like this. For example, by hitting a Google Cloud REST endpoint, you can use a pre-trained neural networks to:

You can also run pre-trained models on-device, in mobile apps, using tools like GooglesML Kitor ApplesCore ML.

The advantage to using pre-trained models over a model you build yourself in TensorFlow (besides ease-of-use) is that, frankly, you probably cannot personally build a model more accurate than one that Google researchers, training neural networks on a whole Internet of data and tons GPUs andTPUs, could build.

The disadvantage to using pre-trained models is that they solve generic problems, like identifying cats and dogs in images, rather than domain-specific problems, like identifying a defect in a part on an assembly line.

But even when it comes to training custom models for domain-specific tasks, our tools are becoming much more user-friendly.

Screenshot of Teachable Machine, a tool for building vision, gesture, and speech models in the browser.

Googles freeTeachable Machinesite lets users collect data and train models in the browser using a drag-and-drop interface. Earlier this year, MIT released a similarcode-free interfacefor building custom models that runs on touchscreen devices, designed for non-coders like doctors.Microsoftand startups likelobe.aioffer similar solutions. Meanwhile,Google Cloud AutoMLis an automated model-training framework for enterprise-scale workloads.

As ML tools become easier to use, the skills that developers hoping to use this technology (but not become specialists) will change. So if youre trying to plan for where, Wayne-Gretsky-style, the puck is going, what should you study now?

What makes Machine Learning algorithms distinct from standard software is that theyre probabilistic. Even a highly accurate model will be wrong some of the time, which means its not the right solution for lots of problems, especially on its own. Take ML-powered speech-to-text algorithms: it might be okay if occasionally, when you ask Alexa to Turn off the music, she instead sets your alarm for 4 AM. Its not ok if a medical version of Alexa thinks your doctor prescribed you Enulose instead of Adderall.

Understanding when and how models should be used in production is and will always be a nuanced problem. Its especially tricky in cases where:

Take medical imaging. Were globally short on doctors and ML models are oftenmore accuratethan trained physicians at diagnosing disease. But would you want an algorithm to have the last say on whether or not you have cancer? Same thing with models that help judges decide jail sentences.Models can be biased, but so are people.

Understanding when ML makes sense to use as well as how to deploy it properly isnt an easy problem to solve, but its one thats not going away anytime soon.

Machine Learning models are notoriously opaque. Thats why theyre sometimes called black boxes. Its unlikely youll be able to convince your VP to make a major business decision with my neural network told me so as your only proof. Plus, if you dont understand why your model is making the predictions it is, you might not realize its making biased decisions (i.e. denying loans to people from a specific age group or zip code).

Its for this reason that so many players in the ML space are focusing on building Explainable AI features tools that let users more closely examine what features models are using to make predictions. We still havent entirely cracked this problem as an industry, but were making progress. In November, for example, Google launched a suite of explainability tools as well as something calledModel Cards a sort of visual guide for helping users understand the limitations of ML models.

Googles Facial Recognition Model Card shows the limitations of this particular model.

There are a handful of developers good at Machine Learning, a handful of researchers good at neuroscience, and very few folks who fall in that intersection. This is true of almost any sufficiently complex field. The biggest advances well see from ML in the coming years likely wont be from improved mathematical methods but from people with different areas of expertise learning at least enough Machine Learning to apply it to their domains. This is mostly the case in medical imaging, for example, where themost exciting breakthroughs being able to spot pernicious diseases in scans are powered not by new neural network architectures but instead by fairly standard models applied to a novel problem. So if youre a software developer lucky enough to possess additional expertise, youre already ahead of the curve.

This, at least, is whatIwould focus on today if I were starting my AI education from scratch. Meanwhile, I find myself spending less and less time building custom models from scratch in TensorFlow and more and more time using high-level tools like AutoML and AI APIs and focusing on application development.

This article was written by Dale Markowitz, an Applied AI Engineer at Google based in Austin, Texas, where she works on applying machine learning to new fields and industries. She also likes solving her own life problems with AI, and talks about it on YouTube.

Read more from the original source:
Hey software developers, youre approaching machine learning the wrong way - The Next Web

BMW, Red Hat, and Malong Share Insights on AI and Machine Learning During Transform 2020Denrie Caila Perez posted on August 07, 2020 | Executives from BMW, Red Hat and Malong discuss how AI is transforming manufacturing and retail.

(From left to right) Maribel Lopez of Lopez Research, Jered Floyd of Red Hat, Jimmy Nassif of BMW Group, and Matt Scott of Malong Technologies.

The VentureBeat 2020 conference welcomed the likes of BMW Groups Jimmy Nassif, Red Hats Jered Floyd, and Malong CEO Matt Scott, who shared their insights on challenges with AI in their respective industries. Nassif, who deals primarily with robotics, and Floyd, who works in retail, both agreed that edge computing and the Internet of Things (IoT) has become powerful in accelerating production while introducing new capabilities in operations. According to Nassif, BMWs car sales have already doubled over the past decade, with 2.5 million in 2019. With over 4,500 suppliers dealing 203,000 unique parts, logistics problems are bound to occur. In addition to that, approximately 99 percent of orders are unique, which means there are over 100 end-customer options.

Thanks to platforms such as NVIDIAs Isaac, Jetson AXG Xavier, and DGX, BMW was able to come up with five navigation and manipulation robots that transport and manage parts around its warehouses. Two of the robots have already been deployed to four facilities in Germany. Using computer vision techniques, the robots are able to successfully identify parts, as well as people and potential obstacles. According to BMW, the algorithms are also constantly being optimized using NVIDIAs Omniverse simulator, which BMW engineers can access anytime from any of their global facilities.

In contrast, Malong uses machine learning in a totally different playing fieldself-checkout stations in retail locations. Overhead cameras are able to feed images of products as they pass the scanning bed to algorithms capable of detecting mis-scans. This includes mishaps such as occluded barcodes, products left in shopping carts, dissimilar products, and even ticket switching, which is when a products barcode is literally switched with that of a cheaper product.

These algorithms also run on NVIDIA hardware and are trained with minimal supervision, allowing them to learn and identify products using various video feeds on their own. According to Scott, edge computing is particularly significant in this area due to the necessity of storing closed-circuit footage via the cloud. Not only that, but it enables easier scalability to thousands of stores in the long term.

Making an AI system scalable is very different from making it run, he explained. Thats sometimes a mirage that happens when people are starting to play with these technologies.

Floyd also stressed how significant open platforms are when playing with AI and edge computing technology. With open source, everyone can bring their best technologies forward. Everyone can come with the technologies they want to integrate and be able to immediately plug them into this enormous ecosystem of AI components and rapidly connect them to applications, he said.

Malong has been working with Open Data Hub, a platform that allows for end-to-end AI and is designed for engineers to conceptualize AI solutions without needing complicated and costly machine learning workflows. In fact, its the very foundation of Red Hats data science software development stack.

All three companies are looking forward to more innovation in applications and new technologies.

Visit VentureBeats website for more information on Transform 2020. You can also watch the Transform 2020 sessions on demand here.

For more news and stories, check out how a machine learning system detects manufacturing defects using photos here.

See the original post here:
BMW, Red Hat, and Malong Share Insights on AI and Machine Learning During Transform 2020 - ENGINEERING.com