Cognitive Timing for AI Self-Driving Cars

By Lance Eliot, the AI Trends Insider

How fast can you think?

If I give you a jigsaw puzzle and ask you to assemble it, you would likely take some amount of time to look at the puzzle pieces and mull over in your mind which piece might go where. You might create a kind of mental picture of how the pieces could potentially fit together. Even before you pick-up a single piece, you might have done a lot of mental calculations and cranial contortions about the pieces and the overall jigsaw puzzle.

Suppose there are just ten pieces to the jigsaw puzzle. Imagine how long it would take for you to mentally envision how those ten pieces will fit together. In contrast, suppose I give you a jigsaw puzzle of 100 pieces, or 1,000 pieces? I’m betting that the amount of mental calculus would go up for those larger-sized jigsaw puzzles.

Suppose further that I give you a jigsaw puzzle that is a picture of farm and has distinctive features such as the farmhouse, cows, horses, and the like. This would tend to make it easier to spot which jigsaw pieces go toward what part of the puzzle. On the other hand, if I gave you a jigsaw puzzle that is all blue in color, and thus there’s nothing distinctive about what it portrays, you would find yourself likely struggling to figure out where the jigsaw pieces go.

All of these variable factors about the jigsaw puzzle will tend toward having you use more mental calculations or less mental calculations, and it will correspondingly cause you to spend more time or less time while trying to put together the jigsaw puzzle. Your mental calculations require the consumption of time. If I force you into a situation wherein you need to do more and more mental calculations, it would to some proportional degree increase the time required for you to solve the mental aspects of the puzzle.

I might sit you down in a room and ask you to do a jigsaw puzzle. In so doing, I might tell you that you have as much time as you need. Take your time, I tell you. In that case, assuming you don’t have to rush to the dentist or have some other upcoming time-based obligation, you could spend minutes or maybe even hours on the puzzle.

Let’s for the moment subtract from the time consumed the “non-mental” time that involves moving the physical pieces. I realize this is somewhat argumentative to suggest that there aren’t mental calculations going on while you are moving the pieces, and indeed the reality would be that you likely would be still thinking about the pieces as you move them, but let’s just focus for now on the mental time that involves none of the physical manipulation of the puzzle.

Solving the Puzzle with Time Constraints

OK, so there you are, solving this jigsaw puzzle. But, instead of having told you that you can take as much time as you like, suppose I said that you have ten minutes to solve it. Not one second more than ten minutes. If this puzzle seems like it might take more than ten minutes to solve, you would begin to more urgently try to solve the puzzle. You might take less mental breaks. You might take greater chances and your level of concentration might go up (versus a more leisurely mental concentration when you knew that you had unlimited time).

Let’s add more pressure. I tell you that you have just ten minutes, and that if you can complete the entire puzzle in the ten minutes that I will give you a million dollars. Whoa! You are suddenly really motivated to solve that puzzle. Your concentration might go up even more so. This is a chance of a lifetime.

The opportunity to win the money was a positive kind of reward. Let’s take a different tack. If you don’t solve the puzzle in ten minutes, your chair falls through a trapdoor into a pit below and you are a goner. Don’t want to get into something untoward here, and I am just trying to suggest that your perspective on the situation might change if you knew it was a life-and-death matter. Would you still be using your same mind and doing the same overall mental calculations? It would seem so. Would you be rushing those mental calculations? It would seem so.

The life-and-death pressure might actually harm your ability to think and solve the puzzle. Often, people discover that in dire situations their mental clarity drops and their mind becomes befuddled. You might also start using your mind to try and find a means to escape from the situation, and as such you are now splitting the use of your mental capacity. Maybe one-third of your mental capacity is on the puzzle, another third on how to escape, and perhaps another third is just in pure panic mode and racing wildly. All in all, this is probably not going to aid you in solving the puzzle. In fact, you might have done better with no pressure or you might have done better with the pressure for the million dollars. Ironically, though solving the puzzle is much more crucial in the life-and-death case, you might do worse mentally.

This discussion about solving the puzzle is an illustration of the cognitive capabilities and effort that we humans have and can make use of. Via your mind, you are able to solve a jigsaw puzzle. Your cognitive efforts take time. Time can be vital to making a mental calculation or decision. We used the time limit of ten minutes to solve the puzzle. In everyday life, you might have to decide which flavor of ice cream to get when at your local creamery and have a minute to figure this out, or you might be a police officer confronting a gunman and you need to mentally decide in a split second whether to fire your gun or not at the assailant.

How do we think?

That’s a big question and one that nobody can fully answer as yet. No one has yet cracked open the inner workings of the mind. Step by step, we are all gradually reverse engineering the brain to see how biologically and chemically it works. We can do all kinds of testing and measurements while you are in the process of thinking, but this has not yet fully revealed how cognition really happens. Efforts to reverse engineer mental algorithms are ongoing and it is hoped that someday we’ll know exactly how thinking works.

One aspect of cognition is that we seem to be able to undertake cognitive economy when we are aware of the need to do so. Allow me to elaborate on this.

You are told to solve the jigsaw puzzle in ten minutes, and then I cut the time to five minutes. You might be able to adjust your mental processes to be more economical in your thinking, doing so in an effort to cut the needed thinking time to five minutes from ten minutes. Perhaps you opt to examine each jigsaw piece only once, while your normal approach would be to look at each piece twice. Maybe you decide that rather than going deeper in your thinking, you’ll do a shallow approach and just move around the pieces based on a hunch or a quicker sense of what to do.

Cognitive Cost Minimization

We seem to be able to do some kind of cognitive cost minimization. We take into account the mental problem to be solved, we consider the amount of time we have, we include the motivation underlying the task, and we then try to shape the nature and amount of cognitive effort accordingly. Now, not everyone seems to do this, or there are many that seem to do a poor job of it. In some cases, it takes specialized training and awareness to realize that you should be doing this kind of cognitive cost minimization.

Little children tend to learn about this aspect without necessarily being explicitly told per se. In kindergarten, they might be given simple tests of guessing at letters of the alphabet or colors on flashcards. They are likely given a limited amount of time to guess. They are motivated to want to do well and appease the teacher and please their parents. Without explicitly realizing how they are doing it, they are bound to be controlling their minds to become able to do cognitive cost minimization.

What does this have to do with AI self-driving cars?

At the Cybernetic Self-Driving Car Institute, we are developing AI systems for self-driving cars and of which include various means to leverage cognitive economy into these systems.

Cognitive economy for self-driving cars is a life-and-death matter. I’ll bet it’s something you might not have given much consideration to. A self-driving car is a real-time system, meaning that in real-time things are happening. The AI cannot just leisurely figure out what needs to be done. Akin to my example earlier about solving a jigsaw puzzle in ten minutes, the AI is confronted with the harsh reality of needing to come to decisions and take actions that are at times a split second in nature.

Suppose a self-driving car is driving along on a highway at 40 miles per hour. Up ahead, a pedestrian steps into the street. The AI of the self-driving car has to ascertain what to do. Should the AI instruct the self-driving car to come to a halt? But, if so, how long will it take for the car to come to a halt and will it stop soon enough to keep from hitting the pedestrian? Maybe the AI should tell the self-driving car to swerve and try to avoid hitting the pedestrian.

Setting aside for the moment the physical movements of the car itself, consider the kind of “mental calculations” that the AI needs to do in this situation. It needs to detect where the pedestrian is. It needs to detect how fast the self-driving car is going. It needs to try and predict what will happen in the future, which in this case is the potential collision with the pedestrian. It needs to identify alternative future scenarios such as trying to come to a halt, and figure out whether halting will solve the problem. It needs to identify whether swerving would solve the problem.

All of this is a kind of cognitive effort (not of a human kind, and so please understand that I am using the word cognition for ease of discussion and not to suggest that the AI is thinking in a manner that humans think).

This is a cognitive effort and one that is time bounded. The AI cannot take twenty minutes to derive a viable solution to the problem. It must use whatever limited time is available. As such, the question arises as to how the AI should employ cognitive economy. This could be seen as like trying to solve a chess problem. For chess, you can look at just one move ahead, or two moves ahead, or fifty moves ahead. The more moves ahead you consider (which are called ply), the more time it takes. If you are playing chess and have a three minute per move timer involved, you likely cannot afford to do a fifty moves ahead exploration.

There’s the self-driving car, zooming at 40 miles per hour, heading directly toward the pedestrian. The AI has to quickly assess the situation and decide what to do. Pretend that it takes 1 second of time to collect data about the aspect that there’s a pedestrian up ahead and in the way, and it takes 1 second of time to consider what actions to take, and another 1 second to issue commands to the car to let’s say swerve, and it takes the car 1 second to receive the command and begin the swerving action. That’s a total of 4 seconds, just involved in initiating the car to swerve.

A car moving at 40 miles per hour is moving at about 60 feet per second. We have just determined that it will take about 4 seconds to make a decision, and so that means that the car will have moved forward a total of about 240 feet (that’s 4 x 60). From an outsider’s perspective, the car has seemingly not done anything as yet, in that the AI was “thinking” internally for the four seconds of time. Meanwhile, the car was continuing to barrel forward for the 240 feet.

Suppose I told you that the pedestrian was about 120 feet in front of the self-driving car at the moment when the AI first began its deliberations. Well, sadly, the self-driving car would have plowed into the pedestrian. This is because the AI was still in the process of deciding what to do. The AI had needed the four seconds to figure out what to do, and during that time the self-driving car was like a guided missile continuing on its course and waiting for any new command as to what else to do.

Similar to my story earlier about telling you that you had ten minutes to solve the jigsaw puzzle and then I said that you only had five minutes, the AI of the self-driving car cannot just blindly take whatever amount of time it wants to make a decision, and instead must try to ascertain how much time it does have. And, once it estimates the amount of allowable time, it needs to make use of cognitive economy to try and find a means to make a decision in the shorter allowed time.

Take a look at Figure 1 to see an indication of the stages involved in the AI core processes for a self-driving car.

The first stage involves getting sensor data and doing some rudimentary analysis of it, which we’ll say takes an amount of time t1. The second stage is the sensor fusion that coalesces the sensor data and tries to resolve conflicts and bad data, and does so in an amount time t2. The third stage is the updating of the virtual world model, which is a representation of the self-driving car’s surroundings and existing situation, and we’ll say that this updating takes an amount of time t3. The fourth stage is the AI action plan that needs to be formulated, such as the different scenarios about how to avoid the pedestrian, and we’ll say that this stage takes an amount of time t4. The fifth stage consists of issuing commands to the driving controls of the car, and we’ll say this takes an amount of time t5. Note that t5 does not include the time for the car to react to the commands and take actions, it is only the amount of time needed to issue the commands to the driving controls and for those controls to receive the commands.

Thus, overall, we need an amount of total time consisting of Total Time = t1 + t2 + t3 + t4 + t5 in order to have the AI figure out what to do.

As shown in Figure 2, these times of t1, t2, t3, t4, t5, will vary from moment to moment, and each will take its own amount of time. I say this to clarify that it is not as though we have predetermined that they are all the same amount of time, such as if we said that each would be a half second in length. Nor are these each the same amount of time, in the sense that t1 could be longer than or short than say t2, in any given circumstance. Also, keep in mind that these five stages are repeatedly looping, over and over, while the self-driving car is underway.

We might have an instance wherein the t1 is the longest of the times of the t2, t3, t4, t5 (see the first example shown in Figure 2). This would be an occasion of a long sensor time aspect. Generally, getting the sensor data is going to be one of the longest of the timings, since it involves collecting physical data and having to get it through the sensory devices and then into the processing of the AI system. Or, we might have a long time needed to issue commands (second example in Figure 2).

Overall, the amount of time will be mixed and varies (see the third example in Figure 2).

You might be wondering whether there is anything doing time tracking. The answer is yes, there is an element of the AI system that needs to be the Time Master.

Take a look at Figure 3.

I’ve shown that we’ll have for each of the t1, t2, t3, t4, t5 stages there is a system element that is keeping track of time. The t1 is shown as an example.

Before entering into any of the stages, a Time Estimator module is used to estimate how much time that the stage will take. This is crucial for purposes of being able to then figure out the total estimated time for all five of the stages, as will be further discussed in a moment.

There is a Time Tracker module that ticks time during the stage and is reporting how much time is being consumed. During the stage, a Time Check might be undertaken to gauge how well the stage is proceeding and ascertain whether it is going to finish on-time of the estimate, or go long or maybe go shorter. There is a Time Interrupter module that can choose to intercede during the stage, depending upon whether something else is occurring that needs to either momentarily disrupt the stage or divert the stage to achieve some other task. The Time Stopper module will aim to prevent a stage from going excessively long, which could happen for a variety of reasons, including that the stage might have gotten itself into some kind of loop that it can’t get out of on its own.

Take a look at Figure 4 to see a situation whereby the estimation total time has been calculated as Q, and the allowable time is considered time R. In this case Q > R, which is bad because it means that the cognitive effort is going to take longer than the system wants it to take. This could be the circumstance of having 2 seconds to decide what to do about a pedestrian, but the initial estimate of the time required to decide about the pedestrian is say 4 seconds. In that instance, Q = 4, while R = 2, and so we have a problem that Q > R, or 4 > 2.

How would the system deal with such a time crunch?

Let’s consider some scenarios.

Suppose a self-driving car is driving in downtown Los Angeles. The day of the week and the time of the day makes a big difference as to what is happening around the car. If it’s a Sunday morning at 6 a.m., the odds are that the streets are empty, there’s very little traffic, there are almost no pedestrians, there aren’t likely bike riders, etc. In this case, the amount of sensory data that needs to be closely examined is relatively small, since there’s few objects being detected and little movement surrounding the car. The virtual world model is relatively simple and barren. Any AI action plan is rather straight forward. The AI can quickly do the t1, t2, t3, t4, t5, over and over, and be moving the self-driving car along its way in the streets of downtown.

Let’s instead consider a Monday early evening of around 6 p.m. in downtown Los Angeles. Cars have flooded into the streets and they are jockeying for position. Bike riders are everywhere, weaving into and out of traffic. Pedestrians are everywhere, including jaywalking and cutting into traffic. It’s chaos. It’s a mess. For this circumstance, the sensors are finding tons of objects that need to be assessed. The virtual world model is jammed with all sorts of indications. The AI action plan is complicated by all of this surrounding activity and potential threats. The t1, t2, t3, t4, t5 is going to stretch out.

What can be done to deal with the downtown driving scenario to allow for cognitive economy, if the AI and the self-driving car is having troubles trying to process all of the hectic activity in real-time and ensure that the self-driving car remains safely on the road?

One aspect is the bubble effect. You could have the AI opt to ignore anything that is beyond a close-in bubble of the self-driving car. Anything beyond five feet the AI decides that for the moment it will pretty much only consider if it somehow seems to be moving fast and toward the self-driving car. Otherwise, it just ignores it for the moment. This would be like a novice teenage driver that is overwhelmed when driving in downtown Los Angeles, and so they simply consider that their focus will be to continue driving forward, and they hope that everyone else will get out of their way.

This has risks associated with it, for sure. If you were an occupant in the AI self-driving car, would you be aware that it has opted to reduce the cognitive load and therefore is now possibly taking on greater risks of hitting something or getting hit? Most of us wouldn’t know, and the AI self-driving car isn’t being programmed to tell us. Some say that the AI self-driving cars have a duty to inform the occupants. Others say that when you do a ridesharing and have a human driver, the human driver is not under any obligation to tell you that they are taking mental shortcuts while driving the car.

Another cognitive shortcut for the AI could be to lump together objects.

For example, suppose there are eight pedestrians standing at the street corner up ahead. You could either try to track each one separately, or you might consider them as a clump of pedestrians. Cognitively dealing with the clump will save on cognitive cycles and you can trim the time it takes to figure out what’s going on with those pesky pedestrians. This might work out OK, though if a specific pedestrian, let’s say a small child, suddenly darts into the street, the question is whether the AI could have predicted this, and whether it therefore might have ensured that the self-driving car would avoid the child. If instead the pedestrians are being treated as a clump, it could be that the AI was not able to predict the chances of the behavior of a member of the clump.

Another cognitive shortcut involves templating a scene or situation. If the self-driving car has driven along this busy street many times before, it could already have prepared a model of the street and all of the likely aspects involved. For me, since I’ve driven in downtown Los Angeles for many years, I can predict when cars will pull out of a parking garage “unexpectedly” and be ready for it. I can anticipate that those cars in the right turn lane will realize they didn’t intend to be in that lane, and they will at the last moment dart into the next lane to the left. These are all predictable based on having been in the driving scene before. I am able to reduce my own cognitive cycles since these aspects are all known beforehand and I am ready to have them happen.

Another aspect of the cognitive load involves the dynamics of motion. When the self-driving car is in motion, it is having to re-calibrate in real-time where it is, and where things around it are. It also should be predicting where other nearby in-motion objects will soon be. Experienced drivers anticipate that a car coming along on a cross-street will in say 2 seconds be potentially intersecting with them, and so must constantly be predicting the paths of other moving objects.

The AI is not only updating the virtual world model, but also using it for prediction purposes. It’s almost like being an air traffic controller that is looking at planes and figuring out when they might intersect with each other. A particular plane is moving north to south at such-and-such speed, and in four seconds will be over here. Another plane is moving west to east, and will possibly intersect with the first plane at four seconds from now. These kinds of considerations need to be undertaken for the moving objects that are surrounding the AI self-driving car, involving moving cars, moving pedestrians, moving bicyclists, and so on, and of course the moving of the self-driving car itself.

Machine learning is involved in these activities too. Often using an artificial neural network, the AI is able to spot patterns in the nature of the traffic situations, and has a prior set of already identified patterns to make use of. One timing aspect that needs to be considered is how fast the neural network can generate a result for the AI to make use of. Suppose there’s a neural network being used in the sensor fusion portion t2, and the neural network is going to take N amount of time to figure out the pattern. Imagine that we need to crunch down on the N amount of time because the self-driving car is in a tight predicament.

What we do is exercise beforehand our neural networks to get a gauge of how long they will take and thus we generally know what their likely timing will be in various circumstances. If you are sufficiently prepared beforehand, you can be ready to have the neural network take shortcuts, such as not make use of the entire network or otherwise take a faster approach to the pattern matching. Once again, this does likely raise risks, and so the short cutting of the neural network must be done in a prudent manner. As an aside, if the neural network is doing learning on-the-fly, you can gain some time by temporarily disabling that aspect until the time crunch efforts have subsided.

As they say, time flies.

They also say, don’t waste time.

The AI of a self-driving car has to do a continual battle with time. The processors on-board the self-driving car can only go so fast. You can tweak the speeds by putting in faster processors and faster memory. That though does not obviate the need for the AI system to be aware of time and be watching time. This is a real-time system that involves life-and-death. Knowing how much time you have, and knowing how to get things done in time, are essential for achieving a successful AI self-driving car.

This content is originally posted on AI Trends.