This lesson is from AI For Everyone

Smart Speaker

• Smart speakers and voice activated devices are taking the world by storm and if you don’t already have one in your home, maybe you’ll buy one someday.
  • How you would write AI software to get a smart speaker to respond to a verbal command such as “Hey device, tell me a joke.”
  • For this example, rather than using Alexa, or Okay Google, or Hey Siri, or Hello Baidu as the wake word or trigger word, to be more device agnostic, I’m just going to use “Hey device” as the trigger word or wake word to wake up, say, a smart speaker and let’s say you wanted to tell you a joke.
  • So how do you build a piece of AI software to understand a command like this and executes on it? These are the steps needed to process the command.
  • There can be four steps.
  • Step one is the trigger word or the wake word detection.
  • The speaker uses a machine learning algorithm to input the audio clip and output.
  • Did they just hear the wake word or the trigger word “Hey Device,” so outputs 0 or one and once it hears the trigger word or wake word, once it hears “Hey device,” it then has to perform step two,
  • which is speech recognition. So what the software has to do is take the audio of what came after “Hey device” and map that to “Tell me a joke.”
  • This is also done with machine learning.
  • Whereas the first step here used an A to B mapping to just tell it that it heard the trigger word, this uses a different A to B mapping to map the audio to a text transcript of what you just said, which in this case is the four words: tell me a joke.
  • Now, the algorithm has to figure out what you actually want by saying these four words.
  • And so the third step is intent recognition.
  • That means to take what have you said and to figure out what you actually wanted to do.
  • So today’s smart speakers have a limited set of commands such as they can tell a joke or they can tell the time.
  • So you can say “Hey device what time is it.”
  • They can play music.
  • They can sometimes help you make a phone call.
  • They can tell you what’s the weather, “Hey device, what’s the weather tomorrow?”
  • So, what intent recognition does is take the four words, the speech recognition’s output, and use another piece of AI software, another A to B mapping, that inputs those four words and outputs which of these five or other intents do you have.
  • So in this implementation of a machine learning algorithm, the input A is the text transcript “tell me a joke” and the output B is which of these five types of commands did the user just utter.
  • Of course, your smart speaker may be able to understand even more commands than just these five, in which case B would be whichever of the five or 20 or 100 commands your smart speaker knows how to execute.
  • No matter how you ask the smart speaker to tell you a joke, hopefully, the intent recognition components will recognize your intent correctly.
  • So that you can also say not just “Hey device, tell me a joke,” but also “Hey device, do you know any good jokes” or “Hey device, tell me something funny.”
  • Turns out, there are lot of ways for you to ask a smart speaker for a joke and a well-designed intent recognition system should recognize most of them.
  • Finally, now that your smart speaker has figured out that you really, really want to hear a joke, the last step is that there will be a software engineer that has written a piece of code to randomly select a joke and to play the joke back through the speaker.
  • In other words, they’ll execute a joke.
  • Four steps of the algorithm
    • first step is trigger word detection,
    • second step, speech recognition, then
    • intent recognition, and then
    • finally, execution of what are the command the user asked the smart speaker to execute.
  • So the process of having four steps in an AI system like this or multiple steps, this is sometimes called an AI pipeline, where you have multiple AI components.
  • Yes, it’s possible to have machine learning components which process data one step after another and it would not be unusual to have, say, four different teams in the company where each team focuses on one of the components of this AI pipeline.
• Let’s now look at a more complex example. One of you issue a more complex commands like “Hey device, set timer for 10 minutes.”
  • First step, same as before, is trigger word detection.
  • So input an audio and just let me know when someone said the trigger word hey device.
  • Then speech recognition, where you input the rest of the audio and transcribe the rest of the sound, the rest of the audio, “set timer for 10 minutes.”
  • And now, intent recognition has to input that text and output that your intent is that you want to set a timer.
  • One difference between “set timer for 10 minutes” compared to the earlier example of “tell me a joke” is that you need to know how long to actually set the timer for.
  • So in the execution step, you actually need to do two things.
  • One is extract the duration. That means, look at the text, set timer for 10 minutes and pull out the phrase that tells you how long to actually set the timer for.
  • And so if the user were to say “Hey device, let me know when 10 minutes is up,” then this extract duration step would have to pull out, again, the 10 minute phrase right there.
  • And of course, there are lots of ways to ask for a 10-minute timer.
  • You can also say, “let me know when 10 minutes are up” or “set an alarm for 10 minutes from now” and hopefully, the intent recognition and extract duration components will both be robust enough to recognize that all of these are different ways of asking for a 10-minute timer.
  • Finally, to execute the command, there should be a specialized software component in the smart speaker that can start a timer with a set duration.
  • And after it has extracted your intent and the duration, it would just start the timer with that duration, so that the alarm goes off at the end of 10 minutes.
  • Today’s smart speakers have many functions.

Self-Driving Car

• One of the most exciting products of the AI era is the self-driving car.
• These are the key steps for deciding how to drive your self-driving car.
• The car will take as input various sensors such as pictures of what’s in front of the car or to the sides or behind, as well as maybe radar or Lidar meaning laser sensor readings.
• Given these inputs of a picture and maybe other things, it then has to detect other cars.
• So given that, hopefully, you’ll figure out there’s a car there, as well as where are the pedestrians, because we want to avoid other cars as well as avoid pedestrians.
• Both car detection and pedestrian detection can be done with machine learning using input/ output or A to B mappings that takes as input the picture and maybe radar and Lidar, sends the inputs and tells us where are the other cars and pedestrians.
• Finally, now that you know where are the other cars and where are the pedestrians, you can then feed this information into another specialized piece of software, it’s called a motion planning piece of software that plans the motion or plans the path that you want your car to take, so that you can make progress to your destination while avoiding any collisions.
• Once you’ve planned out the motion for your car, you can then translate that into specific steering angle for your steering wheel and acceleration and brake commands, so how much to step on the gas pedal, and how much to brake in order to get your car to move at the desired angle as well as speed.
• Let’s look at the three key steps of
  • car detection,
  • pedestrian detection, and
  • motion planning
• Car detection
  • uses supervised learning.
  • So, you’ve already seen how a learning algorithm can take as input pictures like these and output the detected cars.
  • For most self-driving cars rather than using only a front-facing camera, so a camera that looks forward, also often use cameras that look to the left, to the right as well as to the back so it can detect cars not just to the front but all around it.
  • This is usually done using not just cameras but other sensors as well such as radar and Lidar.
• Pedestrian detection
  • and using a pretty similar type of sensors as well as techniques, self-driving cars can detect pedestrians.
• Motion planning step
  • Let’s say you’re driving your car and there’s this light blue car in front of you.
  • The motion planning software’s job is to tell you what is the path, you should drive in order to follow the road and not have an accident.
  • So the motion planning software’s job is to output the path as well as speed at which you should drive your car in order to follow the road, and the speed should be set so that you don’t run into the other car, but you also drive at a reasonable speed on this road.
  • Here’s another example. If there’s this gray car parked on the right side of the road, so you want to overtake this stopped car, then the motion planning software’s job is to plot a path like that to let you veer a little bit to the left/right and safely overtake a stopped car.
• Actual self-driving car might work
  • This is a picture you’ve seen so far. Input image, radar, or Lidar, sensor readings into car detection and pedestrian detection, and that is then fed to motion planning to help you select your path and speed.
  • Now in a real self-driving car, you would usually use more than just cameras, radar, and Lidar.
  • Most self-driving cars today will also use GPS to sense its position as well as accelerometers, sometimes called an IMU, this means accelerometers, and gyroscopes as well as a map because we know that cars are more likely to be found on a road, pedestrians are more likely to be found on sidewalks, although they are sometimes found on the road as well.
  • All this is usually additional information that’s fed in to detect cars and pedestrians as well as other objects
  • Rather than just detecting cars and pedestrians, in order to drive safely you also need to know where these cars and pedestrians are going in the future.
  • So, another common component of self-driving cars is trajectory prediction, where there’s another AI component that tells you not just the cars and pedestrians you found, but also where they’re likely to go in the next few seconds, so you can avoid them even as they’re moving.
  • To drive safely requires more than just navigating other cars and pedestrians.
  • You also need to know where are the lanes so you might detect lane markings.
  • If there’s a traffic light you also need to figure out where’s the traffic light, and is it showing a red, yellow, or green signal.
  • Sometimes there are other obstacles, such as unexpected traffic cones or maybe there’s a flock of geese walking in front of your car.
  • That needs to be detected as well so that your car can avoid even other obstacles than cars and pedestrians.
  • On a large self-driving car team, it would not be that unexpected to have a team or maybe a few people working on each of the componenets, and it’s by building all of these components and putting them together that you can build a self-driving car.
  • As you can tell both from this rather complex example of an AI pipeline, as well as the early example of the four-step AI pipeline for the smart speaker, sometimes it takes a team to build all of these different components of a complex AI product.

AI Team

• AI products may require a large AI team, maybe you have a 100 engineers or sometimes many more than a 100 to build.
• One caveat, because AI is evolving so fast, the job titles and various responsibilities are not yet a 100 percent defined, and they are a little bit different across different companies.
• So, your company may use their job titles differently.
• Software Engineers
  • So, for example, for the smart speaker we needed to write specialized software to execute on the joke or to set a timer or to answer questions about today’s weather.
  • So, those are traditional software engineering tasks. Or
  • you’re building a self-driving car to make sure that your self-driving car software is reliable and doesn’t crash.
  • These are software engineering tasks.
  • So, it’s not uncommon for AI teams to have enlarged fractions sometimes 50 percent, sometimes much much much more than 50 percent of Software Engineers in them.
• Machine Learning Engineer
  • Machine Learning Engineer might write the software responsible for generating the A to B mapping or for building other machine learning algorithms needed for your product.
  • They might gather the data of pictures of cars and positions of cars, train a neural network or train a deep learning algorithm and work iteratively to make sure that the learning algorithm is giving accurate outputs.
• Machine Learning Researcher
  • The typical row of the Machine Learning Researcher is to extend the state of the art in machine learning.
  • Machine learning and AI more broadly are still advancing rapidly.
  • So, many companies for profit and non-profit, more have Machine Learning Researchers responsible for extending the state-of-the-art.
  • Some Machine Learning Researchers will publish papers, but many companies also have Machine Learning Researchers that do research, but are less focused on publishing.
• Applied Machine Learning Scientists
  • Somewhere between Machine Learning Engineer and Machine Learning Researcher.
  • The Machine Learning Scientists kind of does a bit of both.
  • They are often responsible for going to the academic literature or the research literature and finding the techniques and finding ways to adapt them to the problem they are facing.
• Data Scientists
  • One of the primary responsibilities of Data Scientists is to examine data and provide insights, as well as to make presentations to teams or the executives in order to help drive business decision-making.
  • There are also Data Scientists today whose work looks more like the Machine Learning Engineers.
• Data Engineers
  • With the rise of big data, there are also more and more Data Engineers whose main role is to help you organize your data, meaning, to make sure that your data is saved and is easily accessible, secure and cost-effective way.
  • A self-driving car may collect multiple gigabytes of information every single minute of operations.
  • Self-driving cars actually generate lot of data and saving the data for many days or weeks or months or years of operation starts to require serious data engineering.
  • A terabyte is 1,000 times bigger than that and a petabyte is yet another 1,000 times bigger than that of that teams that were responsible for saving several petabytes of information per day, but other than pretty large internet companies is not that common for a company to generate multiple petabytes of information per day.
  • As you move down this scale to bigger and bigger datasets, it becomes harder and harder to make sure that data is stored in a easy accessible, secure and cost-effective way, which is why Data Engineers become more and more important.
• AI Product Managers
  • whose job is to help decide what to build.
  • In other words, they help to figure out what’s feasible and valuable.
  • Traditional Product Manager’s job was already to decide what to build as well as sometimes some other roles, but the AI Product Manager now has to do this in the AI era and they’re needing new skill sets to figure out what’s feasible and valuable in light of what AI can and cannot do today.

AI Transformation

• How can you help your company become good at AI?
• Five Steps
  • Execute pilot projects to gain momentum.
    • Start to know what it feels like to work on AI projects.
  • Build an in-house AI team
  • Provide broad AI training
    • not just the engineers but to many levels within a company including executives.
  • Develop your AI strategy
  • Develop internal and external communications about your company and AI.
• The way your company will execute the steps may not be totally sequential and so the different steps may overlap.

Executing pilot projects to gain momentum.

• If you want your company to gain momentum with AI, the most important consideration for the initial project or projects
  • is for them to be successful
  • rather than necessarily be the most valuable.
• So, when selecting your initial project, try to pick something that you think has a good chance of success.
• They can start to fly wheel turning even if it may not be the most valuable project that you could eventually do for the company.
• Because the goal of the first few projects is just to gain momentum is also now you see if you can pick something that can show traction within six to 12 months.
  • So you can start to fly wheel turning quickly.
• Finally, your first one or two pilot projects can be either in-house or outsource.
• If you do not yet have a large in-house AI team, might be possible, might even be advisable to outsource some or all of your first couple of AI projects in order to get more expertise in house and to let you start building that momentum faster.
• Now, beyond a certain point, you will need your own in-house AI team to execute a long-term sequence of maybe many dozens of AI projects.

Build an in-house AI team

• A lot of companies are organized like this, where, there’s a CEO and multiple business units (BU) that reports up to the CEO.
• Build a centralized AI team
  • Why essentialize the AI? Let’s take an example.
  • Have essentialized AI team to give the team the community to talk to each other about how AI applies to your business radical.
  • AI business unit can be under the CTO, the CIO, the chief data officer or the Chief Digital Officer or it could also be under a new chief AI officer.
  • CEO to provide funding to build at the AI unit rather than required AI unit to get funding from the business units.
  • Eventually, after the initial investment and after the initial ramp up, the AI unit will have to show its value that is creating for the business units but having CEO inject funding at the outset so they can get going, will often help you get that initial momentum much faster.

Broad AI training.

• Now, for accompany that become good at AI, is not just that you need engineers to know AI, you need multiple people at multiple levels of the company to understand how AI interacts with their roles.
• Executives and senior business leaders learn what AI can do for your enterprise, that they learned the basics of these of AI strategy and they learned enough about AI to make resource allocation decisions.

AI strategy

• leverage AI to create an advantage specific to your industry sector.
• One unusual part of this playbook is that developing the AI strategy is step four not step one.
  • If first, companies may end up with sometimes very academic strategies that are sometimes not true to life.
    • My strategy is to focus on collecting a lot of data
      • but for your company, that data may or may not be valuable, and may or may not be a good strategy for your company.
  • One of the reasons that web search is a very defensible business, meaning is very difficult for new entrants to compete with the incumbents with the existing large web search engines, is this:
    • If a company has a better product, maybe a slightly better product, then that web search engine can acquire more users.
    • Having more users means that you collect more data because you get to observe what different users click on when they search for different terms, and that data can be fed into an AI engine to produce an even better product.
    • So, this means that the company with somewhat better product, ends up with even more users, ends up with even more data, and does an even better product with this link being created by modern AI technology.
    • It makes it very difficult for a new entrant to break into this self-reinforcing positive feedback loop, called the virtuous cycle of AI.
    • Fortunately though, this virtuous cycle of AI can be used by smaller teams entering new verticals as well.
    • So, I think today is very difficult to build a new web search engine to compete with Google, or Baidu, or Bing, or Yandex.
  • But if you are entering a new vertical, a new application area where there isn’t a entrenched incumbent, then you might really develop a strategy that lets you be the one to take advantage.
    • There is a company called Blue River that was acquired by John Deere for over US$300 million, and Blue River makes agricultural technology using AI.
    • So, what they did was build these machines that would be towed behind a tractor, in a big agricultural fields.
    • This machine would take pictures of crops and figure out which is a crop and which is a weed, and use precision AI to kill off just the weeds, but not the crop.
      • They were Stanford students are taking my class. So, to get the project started, they actually just use strap in his and sweat, they use their personal cameras and went out to a bunch of farms, and took a lot of pictures of crops in these agricultural fields.
      • So, they started to collect pictures of heads of cabbage and weeds around the cabbage. Once they had enough data, starts off with a small data set, they could train a basic product.
      • The first product, frankly wasn’t that great. It was trained on a small data set, but it worked well enough to start to convince some farmers, some users to start to use their product, to tow this machine behind the tractor, in order to start killing weeds for the farmers.
      • Once this thing was running around the farms through the process of taking pictures of heads of cabbage and killing off weeds, they naturally acquired more and more data.
      • Over the next few years, what they did was they were able to enter this positive feedback loop, where having more data allows you to have a better product.
      • Having a better product allows you to convince more farmers to use it.
      • Having farmers use it allows you to collect more data.
      • Over several years, entering a virtuous cycle , can allow you to collect a huge data asset that then makes your business quite defensible.
    • Other elements of an AI strategy.
      • You should also consider creating a data strategy.
        • Leading AI companies are very good at strategic data acquisition.
        • For example, some of the large consumer facing AI companies will launch services, like a free email service, or a free photo-sharing service, or many other free services that do not monetize, but allows them to collect data in all sorts of ways that lets them learn more about you, so they can serve you more rather than adds, and thereby monetize their data in a way that is quite different than direct monetization about that product.
        • You might also consider building a unified data warehouse.
        • If you have 50 different data warehouses under the control of 50 different vice presidents, then is almost impossible for an AI engineer or for a piece of AI software to pull together all of this data in order to connect the dots.
        • For example, if the data warehouse for manufacturing is in a totally different place than the data warehouse for customer complaints, then how can an AI engineer pull together this data to figure out, whether the things that might happen in manufacturing, that causes you to ship a faulty cell phone, that causes a customer to complain two months later.
        • So, a lot of leading AI companies have put a lot of upfront effort into pulling the data into a single data warehouse because this increases the odds that a engineer or a piece of software, can connect the dots and spot the patterns between how a elevated temperature in manufacturing today may result in a faulty device that leads to a customer complaint two months in the future, thus letting you go back to improve your manufacturing processes.
      • You can also use AI to create network effects and platform advantages.
        • In industries with winner take all dynamics, AI can be a huge accelerator.
        • For example, take the ride-sharing or the ride-hailing business.
        • Today, companies like Uber, and Lyfts, and Ola, and DiDi, and Grab seemed like they have relatively defensible businesses because they are platforms that connect drivers with passengers, and is quite difficult for a new entrant to accumulate both a large rider audience and a large passenger audience at the same time.
        • Social media platforms like Twitter and Facebook are also very defensible because they are very strong network effects where having a lot of people on one platform makes that platform more attractive to other people.
        • So, it’s very difficult for a new entrant to break in. If you are working in a business with these types of winner take all dynamics or winner take most dynamics, then if AI can be used to help you we’re growing faster.

Develop internal and external communications.

• AI can change a company and its products, and its important to communicate appropriately with the relevant stakeholders about this.
  • For example, this may include investor relations to make sure that your investors can value your company appropriately as an AI company.
• Investor relations may also includes government relations.
  • For example, AI is entering health care, which is a highly regulated industry because government has a legitimate need to protect patients, and so for AI to affect these highly regulated industries, it is important for companies to communicate with government, and to work collaboratively with them in public-private partnerships to make sure that AI solutions bring people the benefits it can, while also making sure that governments can protect consumers and protect patients.
  • So, this would be true for health care or be true for self-driving cars, it would be true for finance and many other AI industry verticals.
• Finally, internal communications is also important if you’re making a shift in your company, then many people internally may have worries, some legitimate and some less rational about AI and internal communications, so reassure people where appropriate can only be helpful.

AI Pitfalls to Avoid

• Don’t expect AI to solve everything
  • You already know that AI can do a lot but there’s also lots AI cannot do.
  • Instead, you should be realistic about what AI can or cannot do, given the limitations of technology, data, and engineering resources.
  • That’s why I think technical diligence in addition to business diligence is important for selecting feasible and valuable AI projects.
• Don’t just hire two or three machine learning engineers and count solely on them to come up with use cases for your company.
  • Machine learning engineers are a scarce resource but you should instead air the engineer talents with business talent and work cross-functionally to find feasible and valuable projects.
  • Often the combination of the machine-learning talents worked to business talent that can select the most valuable and feasible projects.
• Don’t expect AI project to work the first time
  • As you’ve already seen, AI development is often an iterative process so should plan for it through an iterative process with multiple attempts needed to succeed.
• Don’t expect traditional planning processes to apply without changes.
  • Instead, you should work with the AI team to establish timeline estimates, milestones, KPIs or metrics that do make sense.
  • The types of timeline estimates, milestones, and KPIs or metrics associated with AI projects are a bit different than the same things associated with non AI projects.
  • So, hopefully working with some individuals knowledge about AI can help you come up with better ways of planning AI projects.
• Don’t think you need superstar AI engineers before you can do anything.
  • Instead, keep building the team and get going with a team you have realizing that there are many AI engineers in the world today including many that have learned primarily from online courses.
  • They can do a great job building valuable and feasible projects.