Automating Visual Inspections in Energy and Manufacturing with AI (Cloud Next '19)

my name is Mandi 4-h and I lead the industrial AI initiative for Google cloud thank you so much for joining really delighted to have you here at Google we believe that the goal of every technology should be to enrich our lives to take our societies our collective humanity forward and do so in a responsible manner so we're constantly thinking of ways in which technology and particularly AI can help us realize this bright and promising future so we've been thinking how can we apply our advanced computer vision technology for solving some of the very hard incumbent problems in the industrial sectors and how can we make these sectors more efficient and more sustainable so in the next 50 minutes we'll be talking about how with industrial inspection AI that is powered by the auto ml vision technology can help make industrial inspections more easier faster accurate and more importantly more safer and we'll also look at how to leading companies are applying this technology to the energy and manufacturing sector so let's get started so AI hold great promise for solving some real world problems from detecting glaucoma with retinal images to processing millions or even billions of documents to understand their content to automatically moderating unsafe and inappropriate content we are applying this technology across all of these use case but we also recognize that developing this technology building these custom vision models is laborious and it's hard so we wanted to enable even the non programmers to be able to tap into the power of AI and that is precisely why we created Auto ml vision so while our standard ap eyes are a great powerhouse for pre-trained models on the massive google image datasets all ML allows you to train custom models that are specific to your industry needs to your use case needs how do we do that so in a very simple clean UI you are able to upload the images labeled images if you're looking to classify a problem or you can draw bounding boxes at we as we take a look to detect specific objects within those images once you've done that with a click of a button you've got a model trained and that model can be used to detect shark species in this case or you can use that to detect defects anomalies breakage in your specific industrial products we already seeing use cases with wind turbine degradation inspection with outages on solar panel forms or failures on electric poles and we'll be looking into some of these examples in more detail shortly at this point I want to take a moment to talk about data protection and privacy so your data sets your images are your images all of these custom trained models are used only on your use cases by you Google does not pull these images into any common deposit trees or use this across customers so your data sets your images we'll take a look at how this technology can be applied for aerial inspection in wind turbines and then an application of that on the production line in a manufacturing company but before I begin there we want to share Google's stance on the use of this technology so Google cares deeply that it's technology is used for creating a positive impact in the world and in that win Google created air principles in last June they set the standard of the application of these air technologies and we abide by these principles for any work that involves AI and similarly for the use of this technology and for this product we expect that this technology be applied in accordance to the air principle which prohibit explicitly the use of this technology for any nefarious purposes so we'll now take a look at how one of the leading energy companies in the world is applying this technology to create a brighter and greener future for us all let's take a look at global yes wind turbine inspections not the biggest you know several times now we really do have the technology to address the issue of carbon footprint greenhouse gases from the electric sector dey's corporation is one of the leaders in new technologies for renewables and energy storage it's a fortune 500 company our mission is accelerating a safer and greener energy future right now we have eight wind farms each farm has different capacity starting from 50 turbines up to 300 turrets they cover large spans of geography and land they're spread across hilltops and mountain sides all these turbines needs annual inspections originally it could take up to two weeks to do one inspection we partnered with leading drone service company measure right now with drones we can do it in two days and this is safe and quick for a wind turbine inspection we go out with our pilots and what we're looking for is cracks or defects things that may need to be prepared on a typical inspection we're coming back with 30,000 images spending four weeks reviewing images I don't think anyone's gonna argue that that the best use of a highly trained engineers time how do we speed that up and how they make it 10x more efficient that's where machine learning and AI comes in we've built a great and an solution using Google class tools and platform with the auto amount vision tool we've trained it to detect damage we're able to eliminate approximately half of the images from needing human review remaining 50% of their time can now be very focused on identifying that damage and really determining the right course of action to immediate it moving from reviewing images to training machine learning models it's a much higher order employment opportunity for people and one where we're trying to develop on our team Google cloud has been a great partner there technology's consistently among the world leaders and I'm just a great partner to work with person-to-person at the end of the day we won't reach the cleaner energy future without advanced tools like machine learning technology will allow the renewable energy to be cheaper than conventional ownership artificial intelligence robotics this is really where the future is all about please join me in welcoming Nico's born from it yes Thank You Mandy and thank you to the team that put that great video together the power industry is enormous it touches all of our lives and the impacts are felt around the world the industry investments are often quoted in the trillions of dollars the opportunities for improvement are often in the billions if not tens or even hundreds of billions of dollars the industry is also going through significant and profound change renewable energy is continuing to fall dramatically in price solar wind and battery energy storage are not just possible or practical the consumer is also driving change they are much more aware of both the opportunities and the costs associated with their energy use and the third megatrend are the new digital tools cloud AI and many others that are changing the economics of insight I'm here today to share one story where we've partnered with Google to improve lives by accelerating a safer cleaner energy future we call this our vision or aerial intelligence platform first a little bit about myself and the company I work for I'm Nick Osborne I'm the business leader focused on understanding and applying advanced analytic tools like artificial intelligence and machine learning to applied business cases jobs really quite simple I accelerate coordinate and facilitate the adoption of these new tools across the organization AES is a global power company were headquartered in India in the United States but operate in 15 countries around the world we've made a very significant commitment to reduce our carbon intensity by 70% by the year 2030 to help us achieve this we've made some very significant investments in new technologies we're the world leader in battery energy storage using lithium-ion batteries and we're also the largest owner of solar assessing and in the it states on a personal note it feels good to come home at the end of the day and know I'm working with a company that's putting its money where its mouth is to drive that change that is core to our mission applying new technologies is core to how we operate our business our drone program is considered world leading in the end of in the energy industry we developed this program by partnering with measure measure is a professional drone services organization and the measure ground control software is an enterprise caliber drone operations platform through this partnership we've improved the cost safety and performance of our inspections another consideration is that what we often hear about the threat of technology taking jobs or eliminating jobs that's clearly not the case with what we're seeing in our drone program and many other technologies that we're exploring we now have over a hundred and seventy pilots trained in our organization performing operations in over a hundred locations around the world these are employees with tremendous value for our company for their personal advancement and their broader career growth prior to drones these inspections were typically done manually so it was either someone climbing up the turbine and then rappelling down to inspect the blade or hiking around the turbine with a large telephoto lens trying to capture an angle and trying to see if they could detect damage neither of these were as effective or as efficient or as safe as what we're able to do with drones so using drones we're now able to take that partial inspection that was taking two weeks of time and do a full inspection in two days a much lower cost much higher quality and a much safer manner tremendous improvement in efficiency and velocity in our organization but there was one new workflow we're now when we do a single turbine inspection so single turbine has around 300 images when we do an entire field this means we're coming back with 30,000 or even 60,000 images this takes a lot of meticulous and detail review to complete the inspection work so we saw this as a great opportunity for artificial intelligence and this is really where our partnership with Google started to grow to understand our journey towards AI you need to understand with where where we started we started with an investment in talent we sent two classes of six people to Google's advanced to solution lab for intense training and supervised machine learning this cohort became the foundation for our work in AI internally we refer to this decision as a no regrets decision meaning that we were able to quickly move forward make this investment with little or no hesitation on our part a few keys for ROI is one is don't just send IT people to this training a lot of the value from data science in general and this program comes from the mixture of expertise and ideas that you get when you send multiple multiple types of people through the program the second piece of advice and this is maybe a bit selfish on my part is make sure you have a good commitment to work on your projects after this training we only sent high-performing individuals to the training and the risk with sending high-performing individuals is that they're going to get quickly pulled back into their day job and that's definitely something we had to work through as an organization so this investment set the groundwork to accelerate our progress that we were making as a company and is another example of where new technologies are increasing opportunities for our employees so from this foundation we got to work we went through a proof pilot production process with each step being a stage gate for further investment so starting with our proof we built a custom tensor flow model leveraging the openly available inception v3 vision model and it worked we were able to detect damage but it also showed us where our shortcomings were our data needed work and setting up the end-to-end platform was going to be difficult and we were going to need some help so in speaking with Google about our progress and our learnings we discussed the possibility of partnering on a pilot phase so in the pilot phase we are we are were labeling sorry we were using Google's data labeling service and Google's Auto mail vision tool to really accelerate our efforts and boost our efficiency and again it worked false negatives were seen as a key business risk for our organization so not detecting damage is something that we weren't willing to accept in our inspection process so using our most restrictive precision recall metrics during this pilot phase we were able to show that we could eliminate 30% of the images from needing any human review so that four-week review process was now down to three works three weeks really accelerating our velocity and our time to action time to action has really become one of those key metrics that we look at with this project so this gave us the commitment our yeah commitment and ability to move forward with our production environment so our production environment is a scalable platform for us to label images train new models and manage those models in production we're still iterating and refining on this model but we're again showing some very promising results we're now showing that we can eliminate 50% of the images from needing any human review and the remaining 50% of the images are now categorized and classified by type of damage further improving our time to action and focusing our engineers on the most important and most critical types of damage so going back to data one of the things that we had learned about early on was that our data all we had a lot of data that was not at the quality or level of consistency we needed for machine learning so working with measure we developed in nine category classification of damage this includes things like cracks gel coat damage different types of delamination and splitting as well as some non damage categories like serial numbers lightning protection points stickers and whatnot so we also worked with Google's data labeling team to iterate and walk through many many edge cases of different types of damage that are out there we started with a series of batches small in size doing a full and complete review of all the labels that were coming back but as the quality of labeling improved and our batch sizes improved we've moved towards a sample basis we also needed to develop a platform to manage the labeling effort model training prediction process working with Google we identified clear object to be a local GCP partner to help us architect and develop our platform using the latest thinking and cloud and serverless tools available from Google clear object has been a great partner and work to quickly develop this platform for us the platform leverages Auto ml for our core modeling engine cloud storage and cloud SQL for our image repository and metadata as well as cloud functions and app engine for to manage our interactions and orchestrations so now that we have this platform we're continuing to improve on the model or we're also looking to expand its use we're looking at new business cases solar transmission infrastructure and even safety as well as looking at new inspection modalities for example infrared and even lidar we're also looking at pushing the model to the edge or in this case the drone so I'm really excited to hear about what LG is going to be sharing next energy is a trillion dollar business it impacts lives in every day in every country around the world the challenge and the real-world impact are huge if you're interested in working with or for company that is improving lives by accelerating a safer cleaner energy future please come talk to me mandeep [Applause] thank you very much make for that great presentation so we saw how Auto ml version can be used for visual inspections to make them more easy faster accurate and safer in speaking with lot of experts from the industry we learned that there are some specific requirements for manufacturing use case a lot of a time this data sits on premise there's latency requirements and most of the image and data sets are in a format that requires it to be processed on the edge devices this be a mobile phone this be an edge TPU a CPU or a GPU so with our Auto ml version on edge solution you're able to take your custom trained model and then download them in an edge device and you can run those inferences from your edge devices I think you'd much rather see that in action and hear directly from a manufacturing company which has deployed these models on the production line so it's a great pleasure for me to invite mr. soon book leave from LG and share more about this initiative mr. Lee a good afternoon everyone I'm very thank you for your attention to our previous presentation my name is Tom Oakley and the vice-president of AI and picked a business unit at LG's Janice it seems there are many Isis fascists in our audience today I think if you are like me I expect we share many great hopes to apply AI tulear word assertions I also hope this short overview our collaboration with Google Auto ml will help you all in your AI work today we'll be looking at how additional send Coogler has successfully collaborated on hey I immediately commission technologies and how we have been apply our leisure to pigeon inspection systems and several manufacturing solutions let's begin with a little background over jeez Janice I think you may know the name of LG group but you don't know about it what kinds of companies in the energy group so I want to introduce some companies we have LG Electronics which produces the television and refrigerator and we have LG Display a produces world reading or LED panel and the LG Innotech produces a camera model so I think a half of you the have already the LG no text camera in your cell phone oh sorry smartphone and LG Chemical produces electric battery is another and world reading company under age group so you may know that almost all the LG group company is working in the manufacturing industry as LG sentences supplies the IT solutions for the LG group affiliates and other companies the working in the manufacturing industry we are constantly working on how to best apply a high technology to improve the manufacturing processes and we all know that it can be really challenged to use the big data and AI technology to ensure product quality on a largest scale production this is where our discussion of a Google or ml comes in today edition has started working with the Google team l in the summer of last year we started our collaboration after seeing the Google was achieving in their immediate recognition technology because we thought Google or ml could help to improve vision inspection for LG production processes and to our great satisfaction our collaboration has been a success okay before we work with Google ultramel actually we had already developed our own in-house AI system a photos of you familiar with the manufacturing process you will like clear recognize that the picture the left of the screen is the typical visual inspection system that reliance relies on the human operators while many production lines can't have a camera and IOT sensors and other detection technologies but non while the many production lines can use camera but it is still hard to find the rear defect efficiently sometimes non defective product open misjudged as defective because of minor factors like a small dust particles or low resolution images and it is still more effective rely on people to complete visual inspections and while people get better the Ridgid that monotony of a visual inspection made by workers also lead to many errors as well to solve this problem indigenous made a tradition we moved from the traditional visual inspection the left image II you can see to the AI inspection system shown on the right I'm sure many of you also working on the inspection technologies so you will be familiar the Trier and era Mossad we need to improve our system with artificial intelligence anyway within our with our in-house system we increase the accuracy and performance and even improved our process speed and efficiency it means that we could reduce our the operation costs or zone with our in-house AI system we were able to apply to over the three production lines only in age group some of these include the first picture as you can see we could improve the defect detection in LCD and LED panels and under the middle of the picture we could remove in pretties from the optical Trillium and even improving the quality control for the production so about automotive efflux can be made with our in-house AI system but even with this improvement our system wasn't working optimally because it still requires a lot of time and effort to perform well and now I will talk about a little about the downside of this system as it's often the case with the success we also ran into some obstacles as we expanded the application of our AI pigeon inspection into other area we have experienced a shortage of skilled AI developers it is very hard to hire the good AI developers for the company they're located in South Korea so it is very hard times when the one a I developers leave our company the parry impetus is so big to our company so and while we designed the AI model they need to spend a lot of time and effort to achieve high performance additionally as we develop the model using service located at the production site the compressed T of architecture has been increased so it is hard to be served so now we require the process to sentry design and this treat the model to the edgy and to centrally control the the performance of the deployed model in one integrity system collaboration with Google has been a critical to find the solution to these problems the performance of Google ultramel has been truly exciting even though our the AIS person doesn't like it one of the key areas we need to improve in our system was our productivity in terms of the moral development time as you can see in the diagram on the left our top arrow bar shows it took roughly seven days to complete our model before using ultramen but afterwards we brought that down to a mere two two hours with Google Reutimann the other area we need to improve was the accuracy of our system in addition to being faster from the diagram on the right pictured Google automates performance exceeded that of the AI experts in many times our test lizard showed the average is six percent improvement in terms of performance we can expect when using a Google chairman I think while we have made advances using Google or ml and integrating that with our visual inspection we are still facing several challenges in many cases we could not meet our clients requirement and you found that the many of them comes from the low image quality not from the model that we made with the Google so to solve this problem we listen to launch it immediate pre-processing lizards team the members of this team spend more time on exploratory data analysis and pre-processing data and try how to try hard to augment data for getting better machine learning models so they became to spend a lot of time on thinking how to the changing the inspection process itself I estimate now our members could use their time and effort for more strategy work now we are planning to expand our business into consulting services so we will provide expertise to enhanced in to enhance overall inspection processes as a one-stop solution we are hopeful that we will see the first manufacturing visual inspection area where humans and a I share areas the least panzerotti very optimally do you agree ok I would like to announce that we have built integrated AI a vision inspection architecture so our system and Google ultramel is connected seamlessly with this architecture we will be able to maximize humans capability and utilization of Google or ml this architecture starts from the data scientist past the bottom they will ensure a major quality so they will produce a clear image and will send to the Google or travel and Google Tom a text a clear image and produced a a model with efficiency and with effectiveness awesome the Morris will be completely managed with all the history data and performance status and automated learning processes with this architecture the elegiggle is now developed now can developed and managing thousands of a aia morris simultaneously in addition to vision inspection our goal is to expand the architecture to the other the manufacturing use cases to manage the whole factory equipment facilities and the safe things and so on i think you may think however many use cases we can expand this instrument architecture in the manufacturing industry to this point we have gone over how collaboration with google Tramel has improved our visual inspection systems now let's look at a to the future based on our AI integration success with within the edge group we will keep going to be positioned as leading AI visual inspection total service provider so we recover from the pre-processing area and we will cover learning the model and then we will manage all the Morris melt with Google attainment whether the cause of poor inspection quality is motion running attainment over the image quality or data labeling over the operators themselves working with Google ml we will strive to achieve our goal of 99.9 percent accuracy and the leak Lake of 0.001 percent under all conditions if you were experiencing the similar issues in your industry I hope that this session could be helpful I really appreciate your attention and thank you for listening thank you [Applause] Thank You mr. Lee so the goal that mr. Lee shared about LG is very much what we share for our product and for our roadmap as well which is to make our inferences faster our interfaces more intuitive and easier and our results more accurate within manufacturing we seeing many more use cases beyond automotive beyond electronics into the food into retail and many more categories and we are very excited to work on these new use cases with you we saw how a eye and visual inspection can be applied to the manufacturing use cases and we looked at how this can be applied for the aerial inspection use cases beyond the three use cases that we talked about on the aerial inspection side we are also exploring more work on the agriculture monitoring and construction site monitoring as of today this technology is available to use in Bera please visit slash vision to register your interest you can use the technology right away but by registering at this site we are able to partner with you and work with you on our upcoming releases and our early access program so we look forward to hearing from you thank you so much for joining us in this shared vision and we really look forward to working with you in creating a more brighter more greener and more positive future thank you very much all [Applause]