Future Work and Research Directions
This project successfully demonstrates a complete pipeline for creating and training a custom Q-Learning agent in a game environment. It serves as a strong foundation with many exciting avenues for future development, research, and enhancement.
1. Advanced Reinforcement Learning Models
The current agent uses a Q-Table, which is perfect for a discrete and limited state space. The next logical step would be to explore algorithms that can handle more complex and continuous environments.
Deep Q-Networks (DQN)
Instead of a table, a DQN uses a deep neural network to approximate the Q-value function ($Q(s, a)$). * Benefits: This approach can handle vastly larger, or even continuous, state spaces. For example, instead of discretizing HP into 3 levels, a DQN could take the exact HP value as a direct input. * Application: A DQN could be trained on the turn-based game with an expanded state space (including status effects, turn count, etc.) or could be the first step in tackling a more complex, real-time environment.
Re-exploring Unity ML-Agents
The initial challenges with the ML-Agents toolkit were a critical part of this project's journey. However, with the foundational knowledge gained from building a Q-Learning agent from scratch, our team is now in a much stronger position to revisit this powerful toolkit. * Opportunity: Newer versions of ML-Agents are more stable and feature-rich. The toolkit provides built-in support for modern algorithms like Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), which are often more stable and sample-efficient than DQN. * Application: ML-Agents would be the ideal tool to bring the project full circle by attempting to create an AI for the original hack-and-slash game concept. The toolkit is specifically designed to handle the complex, continuous state and action spaces of real-time physics-based games.
2. Gameplay and Feature Enhancements
The game itself can be expanded to create a more dynamic training environment and a richer player experience.
Expanded State and Action Spaces
The turn-based battle system could be made more complex, providing a greater challenge for a more advanced AI. * Additions: New mechanics like buffs (e.g., "Attack Up"), debuffs (e.g., "Poison"), items (e.g., "Greater Heal Potion"), and more varied special attacks could be introduced. * Impact: This would exponentially increase the state space, making it a perfect testbed for a DQN model.
Procedural Content Generation (PCG)
Machine learning is not just for character AI. An exciting research direction would be to use RL or other generative models to create new platforming levels. An agent could be trained to design levels that are challenging but fair, providing endless replayability.
Player Modeling
A truly advanced AI could learn to adapt not just to the game state, but to the specific playstyle of a human player. By analyzing a player's tendencies over multiple battles (e.g., are they aggressive? defensive? do they overuse a certain move?), the AI could dynamically adjust its own strategy to be a more effective and personalized opponent.