readme.md

![alt text](https://github.com/Bonkt/speculo/blob/master/speculo_demo_image_new.png)



# SPECULO
#### Experimental Real-Time Application Framework / Game Engine in C++
This project strives to design a framework for developing real-time interactive applications like games, vr and simulation software through a foundation that enables modular development of tools and features.

The goal is to use real-time type information, serialization, deserialization and DLL hot-reloading to achive both type and code redefinitions at runtime, without recompiling the core engine. 

This principle drives the design of both the runtime simulation database (the ecs) as well as the editor database (the hub). Both database designs aim to decouple data from behavior as much as possible, in order to better facilitate reusable, complex and emergent behavior from small pieces of atomic data and code functions. 

#### Note: Project Status
This project is highly experimental and in active development. Due to its explorative nature, the code contains sections of legacy and commented-out code, as well as small experiments throughout. Many features are untested or only tested through a single integration test. 
The focus so far has been on testing various architectural ideas rather than delivering a polished or production-ready codebase.

## Core Features
* Loading and rendering gltf models
* Runtime reflection, Rtti and serialization.
* Custom archetypal entity component system with automatic serialization, vectorizable iteration, and opt-in sparse components and tags.
* Modern API-agnostic Gpu/Rhi interface with Vulkan a backend implementation
* Bindless GpuResources (buffers, images and samplers)


## Entity Component System
The entity component system is an in-memory relational database consisting of handles (entities) to find data (components). It is optimized for cache-friendly, array-based processing of the common case: processing the same instructions over many sets of data packets. 

This is done through archtypes, all entities that have a certain set of component can be said to "have an archetype". The framework stores all entities that have the same archetype together in a dense table. This means that for finding all entities with component A, the framework just has to find all archtables with component A, and not search and find all individual entities, which potentially lowers the complexity. 

For multi-component queries, say [A, B], we similarly perform a join on all archetypes containing at least A and B, but potentially also other components. 

One issue with this model is that archetypes, and thus setting and removing components, forces a move of all entities. which means that frequently setting components becomes costly. 

The solution to this problem is introducing the option to mark components as "sparse", these are stored separately from the dense table, meaning we can add and remove it without having to fragment the rest of the components in the archetype. 

To make this solution also work for "empty" tag-components, we need to introduce an indirection table before the dense table that we call an "arch table". Lets say we have archetypes [D1, D2, S3] and [D1, D2, S4] where D is dense and S is sparse. This means they will have separate arch tables, this is needed because they are after all of different archetypes. But because they have the same dense subset of components they share the same dense table. So both of these arch tables will store indirection pointers to the dense table for every entity that belongs to the archetype.

So in total the Ecs supports four types of components we can add to entities:

|            | Component | Tag       |
|------------|-----------|-----------|
| **Sparse** | Sparse Component | Sparse Tag |
| **Dense**  | Dense Component  | Dense Tag  |

These are specified upon registration of a component, or upon creation of an entity:
```cpp
ecs.dense<MyType>();
ecs.sparse<MyType>();

```
Where templates check if MyType std::is_empty to determine if it is a tag or a component.

  
### Entity Component System Example:
```cpp
// EcsTransform.h
struct EcsTransform {
    vec3f pos = {0, 0, 0};
    vec3f s = {1, 1, 1};
    Quat rot = {1, 0, 0, 0};
    static void reflect(Refl& refl);
}

// EcsTransform.cpp
// This specifies "default" serialization by recursing through members until 
// fundamental or other directly serializable types are encountered.
REFL_TYPE_BEGIN(EcsTransform)
    REFL_MEMBER(pos)
    REFL_MEMBER(s)
    REFL_MEMBER(rot)
REFL_TYPE_END()

// One can also specify custom attributes on each member using: 
// REFL_TYPE_ATTRIBUTE(NetworkReplicatedAttribute, ReplicationPriority::High)
// In future will be replaced by Clang AST parsing and c++ custom attributes

// main.cpp
int main() {
	// register EcsTransform as a dense (fragmenting) component
    world->dense<EcsTransform>();
    world->dense<EcsCamera>();

	
	// Create an entity with name "camera" with the transform component
	Entity my_entity = world->entity("camera");
	my_entity.emplace<EcsTransform>({
        .pos = vec3f(0, 50, 0),
        .s = vec3f(1, 1, 1),
        .rot = {1, 0, 0, 0}
      });
      
	// Similar with other registered and reflected components: 	
	my_entity.emplace<EcsCamera>({
        .fov_y = 60,
        .near_z = 0.1f,
        .far_z = 1000.0f,
    });
	
	auto* transform = my_entity.get<EcsTransform>();
	assert(transform->pos.y  == 50); // true
	
	// Build a query, and iterate over it. 
	// Outer loop itererates archetypes, and the inner loop through the entities and components within it.
    auto my_query = world->queryBuilder<EcsCamera, EcsTransform>().build();
        my_query.runWithIterator([](Iterator* it) {
			while (it->nextTable()) {
				for (auto& [camera, transform] : it->fields<EcsCamera, EcsTransform>()) {
					//  read/write to the camera and transform references.
				}
			}
    });
	
	
	// Finally we can serialize a query to json and then deserialize into the Ecs world again:
	my_query.serialize("my_ecs_scene.json");
	my_entity.destruct();
	
	// Reload the entity from the file:
	world->deserializer("my_ecs_scene.json").loadWithTag();
	
	// While my_entity is no longer a valid entity handle, an otherwise identical entity has been spawned in the world.
	// If we iterate my_query again we will find an the entity with identical EcsTransform and EcsCamera values
}
```

## Development

### Wip Features + Roadmap:
* Implement authorative transactional data model for the persistent non-runtime state of the engine such as assets and config files. This will utilize the Rtti system to enable runtime definition and redefinition of asset and resource types. 
* Connect the data model database with a runtime resource storage with dependency tracking, used for things that components needs shared ownership of such as meshes or textures.
* Implement a DLL-plugin interface with automatic Ecs and datamodel database version bumps on hot-reloads.
* Refactor existing renderer as a DLL-plugin that defines its renderer assets and its gltf importer. 

### Known Issues
Due to a major refactor of the engine's underlying data model, core systems including the asset pipeline and pbr renderer are currently non-functional.

### CPM-downloaded dependencies 
* VulkanSDK
* Vulkan Memory Allocator   
* SDL2                      
* tinygltf          
* KTX
* fmt        
* nlohmann/json        
### Vendored dependencies:
* stb_image.h
* mikktspace.h


### Build
Config & Build & Run steps: (requires CMake and GCC or MSVC)
* cd ./build
* ./build.ps1 [windows] or ./build.sh [linux]

### Engine Structure:
|  Directory | Usage  |  
| ---      | ---      |
| Containers  | Custom data structures  |
| Ecs   |   Archetypal chunked Ecs   |
| Gpu   | Gpu/Rhi interface and implementations (vulkan) |
| Rtti   | Realtime type system and reflection utilities |
| Serialization   | ser/deser using Rtti system |
| Math    | Basic math functionality |
| Platform | Platform/os layer |


### Directories subject to be refactored into plugins
| loaders | files for loading specific files into engine native format |
| ---      | ---      |
| Loaders | files for loading specific files into engine native format |
| Renderer | Renderer that uses GpuBackendInterface |
| Resources | Definition of different kinds of realtime resources in the engine |


### Resources and inspiration used:
* Vulkan-Samples
* API without Secrets: Introduction to Vulkan
* vkGuide.dev
* Sander Mertens blogs and flecs library
* OurMachinery archived blog and podcast

## Copyright and License
© [John Larsson] [2024]. All rights reserved.

All files inside the directory ./Engine/ are, unless otherwise specified, proprietary software. No permission is granted to use, copy, modify, or distribute any part of those files without explicit permission.