Environments and agent structures
Environment types and agent structures: how agents map percepts to actions, models of environments, and standard agent architectures (reflex, model-based, goal-based, utility-based, learning) used to design rational agents.
Drafted with Aria, reviewed by the AiCanCode.org team · Updated . How we write and check our notes. Spotted an error? Use Give Feedback at the bottom of the page.
Why it matters
Engineers design autonomous systems (robots, software agents, self-driving cars, recommendation engines). Knowing how to classify an environment and choose an agent structure ensures the agent can sense, reason, and act to achieve goals reliably in that environment.
Key ideas
Actuator — device or software output that performs an action in the environment. Examples: motors, speakers, network request.
Percept — the data the agent receives at one instant from its sensors.
Percept history — the sequence of percepts the agent has received so far.
Agent function — an abstract mathematical mapping from percept histories to actions. Exact statement: Agent function is a function f: P* → A, where P* is the set of all possible percept sequences and A is the set of possible actions. (Assumes discrete-time steps so percepts arrive in ordered sequence.)
Agent program — a concrete implementation (software/hardware) that computes the agent function given inputs from sensors. (Program runs on an architecture; may approximate the ideal agent function.)
Performance measure — a quantitative score that evaluates how well the agent accomplishes the designer's objectives over an environment run. Example: distance travelled, items collected, total reward. Task environment specification (PEAS) is commonly examined: Performance measure, Environment, Actuators, Sensors (in that order). When asked to specify a task environment, provide a PEAS description.
Rational agent — an agent that acts to maximize expected value of the performance measure, given its percept sequence and built-in knowledge. Exact statement: For each percept sequence, a rational agent selects an action that is expected to maximize the performance measure, relative to the information provided by that percept sequence and any prior knowledge. (Assumes a well-defined performance measure and available information to compute expectations.)
Environment — everything external to the agent that provides percepts and receives actions. To design agents, we classify environments along these dimensions (define before using):
- Observable vs Partially Observable: Observable means sensors provide enough information to determine the true state of the environment at each time step. Partially observable means sensors do not give complete information about the environment's true state.
- Deterministic vs Stochastic: Deterministic means the next state is fully determined by current state and agent action. Stochastic means transitions have randomness or unknowns.
- Episodic vs Sequential: Episodic environments break interaction into independent episodes where current decision does not depend on earlier ones. Sequential environments have decisions that affect future decisions.
- Static vs Dynamic: Static means the environment does not change while the agent deliberates. Dynamic means the environment changes over time (including due to other agents or exogenous time). Also note: Semidynamic means the environment itself is static but the performance measure changes over time.
- Discrete vs Continuous: Discrete means a finite or countable set of distinct percepts/actions/states; Continuous means values vary over a continuum.
- Single-agent vs Multi-agent: Whether other agents exist whose actions affect the environment.
- Known vs Unknown: In a known environment, the agent knows the rules/models governing outcomes (e.g., transition probabilities and sensor models); in an unknown environment, these must be learned or estimated.
State — a complete description of the environment at an instant. True state is the real but possibly unknown state of the environment. Belief state is the agent's internal estimate of the true state (used in partially observable domains).
Model of the environment (transition model) — a specification that describes how environment states change when actions occur (may be deterministic or stochastic). When designing a model-based agent, we assume a model M that relates (s, a) → s' or P(s' | s, a).
Agent architectures / structures — standard templates for agent programs. Define each before use and describe key properties and assumptions:
- Simple reflex agent: selects actions only by current percept (or its immediate interpretation), using condition-action rules. Assumes fully observable environment or that the needed information is present in current percept. Draw diagram: box labelled "Agent (Rule set)" receiving arrow "percept" from box "Environment" and returning arrow "action".
- Model-based reflex agent: maintains an internal state (model) that summarizes unseen aspects of the current world; actions chosen from condition-action rules that access the internal state. Derivation universities ask for: show how internal state s' = update(s, a, percept) lets reflex rules act correctly in partially observable settings.
- Goal-based agent: has explicit goals (descriptions of desirable states) and uses search/planning to choose actions that achieve goals. Assumes a model that can predict results of actions (transition model). Requires defining a goal test and search algorithm.
- Utility-based agent: attaches a numeric utility to states and chooses actions to maximize expected utility (handles trade-offs and uncertainty). Relies on ability to compute expected utilities using probabilities and utilities.
- Learning agent: has components to improve its performance over time: performance element, learning element, critic, and problem generator. Assumes possibility to evaluate past performance and modify internal components.
Structure vs behavior: Architecture (hardware/software platform) constrains the class of agent programs possible; agent program determines behavior in a given environment.
Detailed derivation commonly required in exams: Why fully observable permits simple reflex, and partially observable requires internal state. Step-by-step:
- Fully observable means percept at time t uniquely determines the true state s_t.
- Thus a mapping from current percept to action can be equivalent to mapping from state to action: f: P → A.
- Simple reflex agents implement f by condition-action rules.
- If partially observable, percept p_t does not determine s_t; we must estimate s_t using prior belief s_{t-1} and the percept p_t. Hence maintain internal state: s_t = update(s_{t-1}, a_{t-1}, p_t).
- Therefore decision requires history or internal state, not only current percept.
Formulas
f : P* → A — Agent function: maps a sequence of percepts (P*) to an action A. P*: set of all finite percept sequences. Applies in discrete-time agent models.
s_t = update(s_{t-1}, a_{t-1}, p_t) — Internal state update function: s_t is the agent's belief of current state after taking action a_{t-1} and receiving percept p_t. Symbols: s_t (agent belief state at time t), a_{t-1} (action at time t−1), p_t (percept at time t). No SI units.
argmax_a E[Perf | percept history, a] — Rational action selection: choose action a that maximizes expected performance Perf conditioned on current percept history. Symbols: E[·] is expectation; Perf is the performance measure (numeric). Applies when probability distribution over outcomes is known or estimable.
P(s' | s, a) — Transition model in stochastic environments: probability that next true state is s' given current state s and action a. Applies when environment dynamics are stochastic.
EU(a) = Σ_{s'} ( Σ_s b(s) · P(s' | s, a) ) · U(s') — Expected utility of action a using current belief b(s) over current states s and the transition model P(s' | s, a). Symbols: b(s) belief (probability) over s, EU(a) expected utility, P(...) probability, U(s') utility of state s'. Applies for utility-based agents.
Worked examples
Example 1 — Model answer: Compare simple reflex and model-based reflex agents using the vacuum world.
Question: Explain why a simple reflex agent may fail in a two-location vacuum world with a non-observable bumper, and show how a model-based reflex agent solves it. Draw any diagrams needed.
Model answer (step-by-step):
- Define vacuum world: Two locations A and B. Each location can be Clean or Dirty. Agent can sense only whether current location is Dirty or Clean. Location indicator is unavailable (so the agent cannot directly observe whether it is in A or B).
- Simple reflex agent rules: If current location is Dirty then Suck; else if at A then MoveRight; else MoveLeft. This agent assumes it knows its location (needs location info) — if it does not, it will not know whether MoveRight moves to the other square or hits a wall.
- Failure case: If percepts do not include a location indicator, and the agent starts in A, it will alternate decisions incorrectly: After moving Right, it may receive the same percept as before and not know which location it is in; it can loop forever or miss cleaning B.
- Model-based reflex agent: Maintain internal state variable location ∈ {A,B}. Update rule using the known two-cell map: if previous location = A and action = MoveRight then set location ← B; if previous location = B and action = MoveLeft then set location ← A; otherwise location remains unchanged. With this internal state, decision rules can use (location, percept) to choose the correct action. This avoids loops and ensures both squares are cleaned.
- Diagram to draw: One box "Environment" with two cells labelled A and B. Arrow labeled "percept (dirty/clean)" pointing to box "Agent". Inside agent box show small memory cell labelled "location" and rule set that uses location + percept → action. Arrow from Agent back to Environment labeled "action (Suck/MoveLeft/MoveRight)".
Conclusion: The model-based agent uses internal state to represent unobserved world aspects; simple reflex fails when current percepts are insufficient.
Example 2 — Model answer: Design a rational agent for a taxi in a grid city.
Question: For a taxi operating in a grid city with passengers requesting pickup/dropoff, list environment type, sensors, actuators, performance measure, and suggest a suitable agent architecture.
Model answer (step-by-step):
- Environment classification: Partially observable (other cars, pedestrians may be occluded), stochastic (traffic, travel time uncertain), sequential (current actions affect future), dynamic (other agents moving), continuous in time but can be discretized into grid/time steps, multi-agent (other vehicles).
- Sensors: GPS (current position), map data (static map), camera/LiDAR (nearby obstacles), passenger request API (pickup/dropoff requests), speedometer.
- Actuators: Throttle/brake control, steering control, horn, indicator lights, radio/communication to dispatch.
- Performance measure: weighted combination such as - passenger wait time, - travel time, + safety score (no collisions), - fuel/energy consumption, + customer satisfaction. Make numeric form for engineering: Perf = −α·mean_wait − β·mean_trip_time − γ·collision_penalty − δ·fuel_used + ε·on_time_rate (choose positive weights α...ε).
- Agent architecture: A layered architecture with perception layer (process sensors → world model), planning layer (goal-based planner that uses predicted travel times and search to pick route and schedule pickups), local control layer (reactive safety reflexes to avoid immediate collisions), and a learning module (improves travel time predictions and routing policy using historical data). This is a hybrid: combines model-based planning (for scheduling) with reflex safety actions and learning for stochastic uncertainty.
- Rationality application: Agent chooses actions that maximize expected Perf given current beliefs about traffic and passenger requests. Use predicted distributions P(traffic | time, location) to compute expected utility of routes.
Diagrams to draw: layered boxes: Sensors → Perception → World Model/Belief → Planner (goal-based + utility evaluation) → Local Controller → Actuators. Arrows for data flow and feedback.
Common mistakes
- Claiming simple reflex is always unacceptable; it is fine for truly fully observable, static tasks.
- Forgetting to define performance measure before calling an agent rational.
- Omitting internal state update when designing model-based agents.
- Mixing up agent function f: P* → A (mathematical mapping) with agent program (implementation).
For your exam
Typical question types: definitions (agent, rational, environment types), short notes (compare agent architectures), derivations (show why internal state needed in partially observable setting), design question (specify sensors, actuators, architecture for a given task), diagrams (agent–environment interaction, layered architecture), and long questions asking to design a rational agent or analyze behavior in a toy world.
What full-marks answers include:
- Clear definitions of key terms (bolded in this note), with assumptions stated.
- Correct classification of the environment using the six dimensions and justification.
- A clear agent function or program sketch and a diagram showing percept → agent → action flow.
- If asked for rationality, state the performance measure, what information is available, and show how the agent maximizes expected performance (mention probabilities if stochastic).
- For design questions, list sensors, actuators, internal state, update rules, and choose an architecture (and justify why hybrid if used).
Quick check
- What is the agent function?
- Give two environment dimensions that are independent from each other.
- Why does partial observability require internal state?
Answers: f : P* → A; Observable/Partially observable and Deterministic/Stochastic (they are independent); Because current percept does not uniquely identify the true state, so the agent must use past percepts (internal state) to estimate the current state.
Finished this topic? Mark it so your progress, study plan and readiness keep up.
Stuck on something here?