Entities vs. Models in
Modern Web Apps
Introduction
Data modeling is crucial in the realm of web application development, significantly impacting the scalability and reliability of a system. It often guides subsequent implementations, such as domain-level constraints or how user interfaces are constructed. A key strategy that has gained traction among developers is the semantic separation of entities and models. This approach involves distinguishing between business data structures ("entities") and technical persistence data structures ("models"), allowing for a cleaner and more flexible architecture.
This separation of models and entities is a well-known concept within the principles of Clean Architecture. Clean Architecture, popularized by Robert C. Martin (Uncle Bob), emphasizes the separation of concerns and decoupling business logic from technical implementation details. By adhering to these principles, we can manage the complexities of evolving data relationships and business rules while maintaining a robust and flexible system.
Despite its widespread adoption in the software industry, modern software tooling does not treat both entities and models as first-class citizens, leaving their architectural integration up to the application developers. In contrast, handling this on the tooling level can significantly improve the developer experience.
In this article, we will explore the concept of semantic separation in detail, examining the historical development of this approach, its benefits, and my own experiences while building full-stack web applications. Additionally, we will discuss a practical attempt to promote this concept through the declarative Morphe YAML specification as an IR ("Intermediate Representation") and related open-source tooling, aiming to streamline and enhance the development process.
Historical Development of Clean Architecture
> Origins of Clean Architecture
Clean Architecture, as conceptualized by Robert C. Martin (Uncle Bob), stems from a culmination of software architecture principles that evolved over several decades. It finds its roots in the early principles of modular programming and the SOLID principles, which were initially proposed by Robert C. Martin and other software engineering pioneers. These principles aimed to enhance code maintainability, scalability, and testability by advocating for well-structured and loosely coupled systems.
The concept of Clean Architecture represents a significant evolution from these foundational principles. It introduces a holistic approach to software design that emphasizes separation of concerns and independence of implementation details. At its core, Clean Architecture advocates for organizing software systems into distinct layers or concentric circles, each with specific responsibilities and dependencies arranged in a hierarchical manner.
> Key Concepts and Principles
Clean Architecture introduces several key concepts that have become fundamental to modern software design:
- Dependency Rule: Outer layers (such as frameworks and delivery mechanisms) should depend on inner layers (such as business rules and entities), but not vice versa. This ensures that changes in the external layers do not affect the core business logic.
- Separation of Concerns: By separating concerns into distinct layers (e.g., presentation, application, domain, and infrastructure layers), Clean Architecture promotes clarity, maintainability, and flexibility in software systems.
- Testability and Maintainability: The architecture facilitates automated testing and reduces the impact of changes, making systems more adaptable to evolving business requirements.
In Clean Architecture, we have a clean separation of concerns into layers. Outer layers should depend on inner layers, but not vice-versa. This decoupling improves system robustness and maintainability.The adoption of Clean Architecture has reshaped software design practices across industries. It has provided a framework for developers to structure applications that are not only technically sound but also aligned with business goals and requirements. Clean Architecture enables teams to build robust, scalable, and maintainable software solutions by focusing on domain-centric design and minimizing dependencies on external frameworks and technologies.
> Relevance to Semantic Separation
In the context of semantic separation of entities and models, Clean Architecture plays a crucial role by providing guidelines on how to structure and organize these components. By adhering to Clean Architecture principles, developers can effectively manage the complexities of data relationships and business rules while maintaining a clear separation between domain-specific entities and technical persistence models.
Clean Architecture emphasizes the importance of separating concerns and minimizing dependencies between different layers of the application. This aligns closely with the goals of semantic separation, which aims to distinguish between business entities and technical models to achieve a cleaner and more flexible architecture.
More specifically, by mapping domain entities to technical models and operating on entities within the domain, we can successfully insulate the domain layer from structural changes in the underlying framework and application layers.
No Pain, No Gain - A Software Story
> The Pain
In April of 2020, I started a new job at a construction cooperative, tasked with building a web-based ERP and CRM system from the ground up for internal use. Up until then, the cooperative had been operating with a team of around ten employees on a single Microsoft Access database file on a shared network drive (yikes!). They needed to scale quickly and couldn't keep up with their workload without automation and efficiency improvements.
At first glance, building the first version didn't seem to complicated. To my dismay, as I delved deeper into their processes and data, I realized the inherent complexity of their daily business. It didn't map easily to existing solutions, motivating us to roll our own custom system.
Under immense time pressure to deliver a more efficient solution, I set to work with my most productive stack (Postgres, Go, and Vue.js). It took several days to hammer out the initial data structures, encompassing almost 30 models with complex relationships, to replace their existing Microsoft Access database.
After a stressful month, we launched the first version of the system to track registered cooperative members, their monthly membership payments, brokers for commission payouts, and more. Employees were onboarded, and it seemed like all was well that ends well, right?
Wrong! After almost a year of using the system, we realized that some of our core data models were not nuanced enough. We had to rework them without breaking anything. This was particularly challenging, as we had already built out additional automated accounting processes, added multi-channel communication, member ingestion interfaces, and much more.
> What I learned <
We were confronted with significant regression testing and manual data migration challenges, as our data models were tightly coupled to our business logic.> The Gain
After my experience with the construction cooperative, I was determined to apply the lessons learned to future projects. Together with a friend who had a business background, we founded a company aiming to disrupt the German real estate market with innovative, web-based software solutions. Our vision extended beyond just conquering the real estate market; we wanted to build a robust infrastructure that would allow us to quickly spin up multiple SaaS products across various industries using different declarative data models.
Our first major project was a real estate listing synchronization platform coupled with master data management functionality so that the customer could track which units they had with all the bells and whistles.
However, as we were building this project incrementally, I was wary of repeating my former mistakes. We had limited knowledge about the nitty-gritty of the real estate domain, and so I decided to incorporate the semantic model and entity splitting in a novel generative data modeling framework that would enable the rapid deployment and customization of SaaS products across industries.
A visual example of how entities as business objects (here: "Person") map to models as technical objects (here: "Person" and "Address") with semantic separation.In theory, we would be able to generate model / entity definitions in code, SQL scripts, front-end forms, and more from the same base declarative intermediate representation.
Although we ultimately only fully realized the ORM component of this system, it proved to be a game-changer. I first defined the models (technical persistence structures) while building the application, and then composed them into entities (business data structures) when I started working with domain logic.
In this setup, entities were implemented as an analogue version of SQL views, but realized in application code, with entity fields mapped to the relevant underlying model fields.
The benefits were immediate and significant. Spinning up new robust data models became straightforward and efficient. Refactoring technical model structures was simplified because our ORM handled the internal mapping. This separation also made it easy to implement optimizations directly within the ORM, such as entity caching. This architecture allowed us to later implement CRUD hook logic at different layers of abstraction—separating persistence concerns from business domain logic.
By decoupling our business logic from the underlying technical models, we achieved a level of flexibility and maintainability that was previously elusive.
The success of this approach not only validated our architectural decisions but also inspired the creation of the Morphe specification, which aims to streamline and enhance the development process for future projects.
Through this journey, I experienced firsthand the transformative power of semantic model and entity splitting. It enabled us to rapidly adapt to changing requirements, ensure scalability, and deliver high-quality software solutions efficiently. This approach has since become a cornerstone of my personal development philosophy.
- The semantic separation of business entities and persistence models is a powerful strategy that can significantly improve the scalability, maintainability, and flexibility of web applications.
- Clean Architecture provides a solid foundation for implementing this separation, emphasizing the importance of separating concerns and minimizing dependencies.
- By integrating the semantic separation directly into the development process, developers can streamline the creation of robust data models and entities, leading to more efficient and maintainable software solutions.
- The development of the Morphe specification and related tooling aims to further enhance the developer experience by providing a declarative approach to data modeling and architecture.
Conclusion
Reflecting on my journey, the importance of proper data modeling in web application development cannot be overstated. The distinction between entities and models has been instrumental in addressing the complexities of business logic and technical persistence. By separating these concerns, we achieved a cleaner and more flexible architecture, one that is capable of evolving with our needs.
Applying the principles of Clean Architecture provided a solid foundation for these efforts. Clean Architecture emphasizes the separation of concerns and independence from implementation details, aligning perfectly with the goals of semantic separation. It allowed us to manage the complexities of evolving data relationships and business rules while maintaining a robust and flexible system.
By integrating it directly into our tooling, we didn't have to worry as much about enforcing the separation on an application level, which further reduced boilerplate code. It ended up guiding our development process and integrated cleanly into each developmental stage.
Despite the widespread adoption of these principles in the software industry, modern tooling still lags in treating entities and models as first-class citizens. This gap leaves much of the architectural burden on developers. Through our work and the development of the Morphe specification, we aim to address this gap, providing tools and frameworks that streamline the development process and improve the overall developer experience.
In conclusion, the semantic separation of entities and models is not just a technical strategy but a paradigm shift that fosters better design, more manageable codebases, and ultimately, more successful software products. It's a testament to the power of thoughtful architecture and the relentless pursuit of improvement in the ever-evolving landscape of web application development.
Thanks for reading, and stay tuned for more to come!
