Introduction

The Noodle Language is a visual programming language based on the findings of the paper "Investigation into the criteria of embeddability of visual scripting languages within the domain of game development.". The main goal for this visual scripting implementation specification is it to provide a prototype for a visual scripting language that can embedded without much effort into an application.

This collection of documents will discuss the core concepts of Noodle by following the result structure of the above mentioned paper. The paper found five major aspects that are potentially important when designing a language framework with embeddability in mind. The thesis defines them as follows:

1. performance and the identified subcategories
2. mechanical aspects of embedding software and their respective subcategories
3. license
4. documentation,
5. tooling and workflow

While we look into each of these aspects, we define what consequences it has for Noodle.

This Chapter is basically a breakdown version of the mentioned paper "Investigation into the criteria of embeddability of visual scripting languages within the domain of game development." as well as the accompanying industry article published on my blog.

General design overview

Before we dive deep into these findings, the main characteristics of Noodle will be briefly described. Noodle will be diagram-based [1], [4] which means that the language is graph-based and the execution can be followed by traversing nodes. As my research shows, this is the most commonly used VSL form in game engines, e.g., in Unreal Engine, Unity and Godot [5]–[8]. A popular alternative is the block-based design. A block-based design is used for Scratch and Google’s Blockly [9]–[11] but it is not often used in games. Moreover, Noodle will offer pure nodes, nodes without side effects and nodes that may have side effects [12]. They are differently identified by either having an “execution” wire/connection or just a data flow connection. This allows the dialect designer to decide if they need support for one of these concepts or both, depending on the use case requirements. On its own, the language is not event-based, so a graph can be triggered at any node at any point in time by the runtime. However, the dialect designer can specify nodes that can be triggered. These nodes can be used as entry points which would allow an event-based structure if needed. All these decisions are based on what is common in tools used in the game industry, be it Godot’s visual scripting or Unreal Engine’s Material Editor [8], [13]–[17].

Noodle’s design idea of being a composable language that one can shape depending on the users’ needs, or how literature would call it “extending” by being an extension language, is greatly inspired by the works of Hisham Muhammad and Roberto Ierusalimschy in their paper about the used API design of the Lua programming language [18]–[21]. Since we are talking about a visual scripting language which logic is expressed in composing visual elements [22], we need some form of specification. In Noodle the dialect is described via a protocol: what nodes it supports, what kind of types are supported, etc. Moreover, Noodle supports internal modules, i.e., subgraphs that can help organizing your graph, as well as external modules, if a module resolve function is provided. If not, only internal modules will work. However, the defined protocol might differ from dialect to dialect, depending on the use of the language. We will talk about the protocol an its uses later.

Performance

Development performance

One of the most important factors for the game industry is the ability to perform quick iteration cycles and to be able to quickly prototype a feature [23, pp. 956–957] [20], [24]. Many interviewees have expressed the opinion that script hot reloading is a crucial feature they would expect from a VSL. It has been shown also in literature [23] that script hot reloading is a great feature, that supports quick iteration and rapid prototyping. Based on these observations, Noodle will support script hot reloading to empower users with quick prototyping.

Runtime performance

When talking to any game programmer, they will most likely make a similar statement as many of the study participants, that memory is the number one bottleneck in games. Although, it greatly depends on the requirements of the game and how dramatic the runtime performance is influenced by the right memory access patterns or the right allocation strategies. Therefore, it is not surprising that most of the participants expressed their opinions on how memory management should work. The major conclusion is that in a VSL, we know from early on what kind of data we will handle since all inputs and outputs of all nodes are known at translation time, therefore the graph could either allocate a large chunk of memory and manage that like in WebAssembly where “...opcodes accessing memory are not given addresses, but offsets relative to the beginning of a linear memory segment whose size is always known.” The memory model of Noodle will follow the same mentality and will at the beginning allocate the needed memory through a memory allocator interface that the user can modify, if needed. If memory needs to be accessed in a form of a pointer, Noodle will not be able to do anything with these pointers, so it will just pass them down to native functions that are able to understand them.

When it comes to a scripting language and a VSL, which is nothing else than a subcategory [1], [25], the execution method is important. Robert Nystrom states in both of his books “Crafting interpreters” and “Game programming patterns” [26, p. 17], [27, pp. 155–179] that a tree-walking interpreter, a form of an interpreter that traverses the graph by recursively calling the nodes, is slower than compiling the graph down to bytecode that is executed in a virtual machine (VM) or transpiled to a different source code [26, pp. 16–20], [28], [27, pp. 155–179] ,[23, pp. 52,954-958]. Based on these findings, Noodle will compile to bytecode with the intention that the user can compile a Noodle graph representation to C and compile to native code depending on their platform as a last shipping step (if providing a backend). This transpiling or compiling to native code has been mentioned in the interviews as an important feature. It can also be considered to allow the language to enable hot patching of the native generated code. This means that if something is wrong with the compiled C code, a content update of a bytecode compiled graph can be used to hot patch this part of the code.

Mechanical aspects of embedding software

API design

As described in the above-mentioned papers from H.Muhammad and R.Ierusalimschy about the API design of Lua and other scripting languages such as Perl or Python, it can be concluded that an API should be flexible to provide the ability to extend the underlaying language not in a verbose (like Perl) manner, but more in a concise declarative manner. This view is supported by the interviewees who describe that an API should be able to bind, for example, an external editor but should also be small enough to be easily manageable. Moreover, academia as well as the interviewed industry professionals argue that the API should be written in the C programming language or at least provide C foreign function interface, since C is considered as the lingua franca of programming languages [18]–[21]. To provide maximal portability from an API point of view, the header files of Noodle will be written in C99 and the implementation will be done in C11.

Dependencies

The game industry is notorious for reinventing the wheel [29], [30] and this problem might stem from the platforms we are catering to and the software we are working with. Therefore, having many dependencies that we need to maintain and maybe port to different platforms is not desired. This is what the research suggests and what an industry professional states in an interview: “When you develop a commercial product, you also need to consider two things. Since dependencies might be taken offline at any time, it is very important to have your own copy of them. Also, for various certification on platforms you need to keep in mind that they perform security audits on those dependencies.”. These insights led to the decision for Noodle to have no external dependencies besides the OS dependencies on the platforms it supports.

License & Documentation

The game industry caters for many different platforms with different requirements, some are open source, some are not. Therefore, the license needs to be permissive since it is not always possible to open-source certain aspects of the entire codebase due to NDA regulations. This is the reason why Noodle will use the MIT license or the Apache License, Version 2.0, depending on the user’s need.

Besides the license, a good documentation on how to embed the language into your game framework is needed. The research for the thesis "Investigation into the criteria of embeddability of visual scripting languages within the domain of game development." has shown that it is expected by users for some form of online documentation and samples to exist. Therefore, Noodle has this page for documentation and help.

Workflow & Tools

So far, none of the categories of embeddability are really different from what one would expect from a regular scripting language. In fact, what literature says is that they are nearly matching, and Software Engineering Stack Exchange confirms it. The major difference explained in literature by B.Myers already in 1989 in “Taxonomies of Visual Programming” [1] and in the works of Nystrom and J.Gregory [23], [27], the biggest challenge for a VSL is the UI/UX aspect and mainly the visual scripting environment (VSE). A VSE can be seen as the integrated development environment (IDE) for VSLs. The thesis concluded that among all aspects, workflow and tooling are the most important.

Unfortunately, there is not much existing research on what qualifies a good visual scripting environment within the domain of game development. If we look outside of the game industry, there are a few papers on this topic, but they mostly describe the design of visual scripting environments for block-based languages, since they are mainly used for educational purposes. Although the industry professionals participating in the study shared some important insights, proper academic research would be needed to make academic claims. However, for this prototype the statements of industry professionals and the scarce UI / UX research on visual scripting environments will base the foundation for Noodle’s UI / UX.

The industry professionals stated that one cannot just use Notepad or Visual Studio to edit the “source code” of a VSL, therefore they would expect either a fully-fletched editor to come with the VSL or a flexible API to allow them to bind an editor themselves. One of the participants stated that they would use the provided editor to get acquainted with the language and then use the API to build their own that matches the paradigms and needs of their own ecosystem. Hence, Noodle will come with a prototype of an editor and a flexible API that allows the user to bind a custom editor.

For the first prototype, the language will come with a Visual Studio Code extension that enables the editor to understand Noodle files, while the editor will communicate with the Noodle runtime via the WebSocket API by using the protocol described later. Whenever a change occurs in the editor, the updated graph will be sent to the runtime and the runtime user can decide if it is needed to recompile the current graph and swap it with the new changes. The goal is to provide an embeddable editor view written in C or C++ as an external tool on top of the Noodle APIs.

The mentioned WebSocket-based API allows network communication to other software that might be the editor. The VS Code extension will be part of the first prototype to demonstrate this communication feature. A network-based communication might be more suitable for games that separate game runtime and editor runtime, while the API communication via the C interface might be better suited for custom game engines that have a built-in editor. Important to mention is that the WebSocket API is purely optional.

Since debugging and visualizing what is happening within the VSL has been classified as very important, Noodle should come with an ability to attach a debugger to it. In the first prototype we can try to make use of the VS Code Debugger framework and provide the ability to connect to it.

A Noodle document will be in JSON format that can be parsed with any JSON parser and can be checked in source control like a normal text file. The main reason why JSON has been chosen is that it provides a better source control management support. Also, it can be diffed without big issues, while for example a binary file format would cause source control issues (google ‘Blueprints and source control’).

Protocol

As mentioned above, Noodle runs on a protocol that gives an overview of what nodes are available and what types can the runtime understand. This indicates that Noodle can be either statically or more dynamically typed, depending on the needs of the dialect designer. The Noodle protocol can be generated via the API and will return either a protocol struct that can be used via the C interface or a JSON representation that can be used otherwise in tools or in the WebSocket API.

Besides the Noodle protocol that describes the language, there is the noodle file or noodle document – the script file. One can see the protocol like a header file in C or C++ and a noodle file as the source file. The noodle document describes the current file the VM is processing. As mentioned above, each noodle file may contain subgraphs (internal modules) or if enabled, external modules. In principle, every noodle document contains three regions:

Protocol – this region describes the meta data of the current language dialect. The VM will check this region and verify that the name of the dialect, as well as the version, are matching. If not, the file cannot be passed. Migration can be implemented with this approach.

Graph – this region describes the actual data of the graph, which includes the modules, nodes, connections, and the data.

Editor – implementation defined region that can be used to define editor specific data such as position of nodes, etc. This region is not used by the VM in any way.

The graph region is the only region where the VM interpreter needs to understand the data of the file, while the protocol region is there to make sure that the noodle file matches this dialect.

References

[1] B. A. Myers, “Taxonomies of Visual Programming,” 1989.

[2] M. F. Msiska and L. Van Zijl, “From visual scripting to Lua,” ACM Int. Conf. Proceeding Ser., pp. 94–99, 2012, doi: 10.1145/2389836.2389848.

[3] M. M. Burnett, “Visual object-oriented programming,” Proc. Conf. Object-Oriented Program. Syst. Lang. Appl. OOPSLA, vol. Part F1296, no. April 1994, pp. 127–129, 1993, doi: 10.1145/260303.261240.

[4] M. M. Burnett and M. J. Baker, “A Classification System for Visual Programming Languages,” J. Vis. Lang. Comput., vol. 5, no. 3, pp. 287–300, 1994, doi: https://doi.org/10.1006/jvlc.1994.1015.

[5] E. Games, “Blueprints,” 2021. https://docs.unrealengine.com/4.27/en-US/ProgrammingAndScripting/Blueprints/

[6] E. Games, “Kismet Visual Scripting.” https://docs.unrealengine.com/udk/Three/KismetHome.html

[7] Unity Technology, “Bolt Documentation.” https://docs.unity3d.com/bolt/1.4/manual/index.html

[8] G. Team, “Godot Visual Scripting.” https://docs.godotengine.org/en/stable/getting_started/scripting/visual_script/getting_started.html

[9] Google, “Blocky.” https://developers.google.com/blockly (accessed Apr. 14, 2022).

[10] J. Maloney, M. Resnick, N. Rusk, B. Silverman, and E. Eastmond, “The scratch programming language and environment,” ACM Trans. Comput. Educ., vol. 10, no. 4, pp. 1–15, 2010, doi: 10.1145/1868358.1868363.

[11] MIT, “Scratch.” https://scratch.mit.edu/

[12] P. Wadler, “The essence of functional ( Invited programming talk ) recursive a compiler language,” Proc. 19th ACM SIGPLANSIGACT Symp. Princ. Program. Lang., pp. 1–14, 1992.

[13] G. Team, “VisualShaders.” https://docs.godotengine.org/en/stable/tutorials/shading/visual_shaders.html

[14] “Unreal Engine 4 Material Editor.” https://docs.unrealengine.com/4.27/en-US/RenderingAndGraphics/Materials/Editor/

[15] Unity Technology, “Shader Graph”, [Online]. Available: https://unity.com/shader-graph

[16] M. Autodesk, “Maya Node Editor.” [Online]. Available: https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2019/ENU/Maya-Basics/files/GUID-23277302-6665-465F-8579-9BC734228F69-htm.html

[17] B. F. Blender 3D, “Shader Editor.” https://docs.blender.org/manual/en/latest/editors/shader_editor.html

[18] H. Muhammad and R. Ierusalimschy, “C APIs in extension and extensible languages,” J. Univers. Comput. Sci., vol. 13, no. 6, pp. 839–853, 2007.

[19] H. H. Muhammad, “A study on scripting language APIs,” 2006.

[20] R. Ierusalimschy, L. H. De Figueiredo, and W. C. Filho, “SPE paper Lua – an extensible extension language,” vol. 6, no. 1996, pp. 635–652, 2015.

[21] R. Ierusalimschy, L. De Figueiredo, and W. Celes, “The evolution of an extension language: A history of Lua,” Proc. V Brazilian Symp. Program. Lang., vol. 1, no. 1, pp. 1–16, 2001, [Online]. Available: http://www.lua.org/history.html%0Ahttp://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.15.9210&rep=rep1&type=pdf

[22] M. Idrees, F. Aslam, K. Shahzad, and S. M. Sarwar, “Towards a Universal Framework for Visual Programming Languages,” Pak. J. Engg. Appl. Sci., vol. 23, no. July, pp. 55–65, 2018, [Online]. Available: https://www.researchgate.net/publication/328191862_Towards_a_Universal_Framework_for_Visual_Programming_Languages

[23] J. Gregory, Game Engine Architecture, Second Edition, 2nd ed. USA: A. K. Peters, Ltd., 2014.

[24] R. Ierusalimschy, L. H. de Figueiredo, and W. Celes, “Passing a Language through the Eye of a Needle,” Queue, vol. 9, no. 5, pp. 20–29, 2011, doi: 10.1145/1978862.1983083.

[25] J. K. Ousterhout, “Scripting: Higher-level programming for the 21st century,” Computer (Long. Beach. Calif)., vol. 31, no. 3, pp. 23–30, 1998, doi: 10.1109/2.660187.

[26] R. Nystrom, Crafting Interpreters, 1st ed. ‎ Genever Benning. [Online]. Available: https://www.craftinginterpreters.com/

[27] R. Nystrom, “Game Programming Patterns,” in Game Programming Patterns, Genever Benning, 2014. [Online]. Available: https://books.google.nl/books?id=AnvVrQEACAAJ

[28] R. Nystrom, “A Virtual Machine,” 2021. https://craftinginterpreters.com/a-virtual-machine.html

[29] J. G. Guerrero, “Reinventing the wheel,” 2014.

[30] C. O’Toole-Bateman, “The History of the Game Engine: Part 5 – Reinventing the Wheel.” https://ultimategamingparadise.com/features/series/history-of-the-game-engine/part-5-reinventing-the-wheel/

Language Specification

The Noodle Language Specification defines the language dialect. A VM can usually only handle one dialect at the same time. The specification be retierved from the VM's API. The VM will provide a JSON document containing the language specification or a language specification structure in C.

Language Specification

The JSON representation of the language dialect (noodle) can be used to communicate with the VM either via the WebSocekt API or via the general API. The difference lays that the general API structures are C structs modeled in a similar manner than the JSON structures. This allows that any extern editor that can parse a json file can understand the language dialect and parse a .noodle file. This is demonstarted in the VS Code Extension.

How will the JSON be generated?

The JSON is generated via the API as mentioned above and reflects all registered nodes and data types to the VM. Also it will contain extra information about the current dialect. The C API can also read the protocol and configure itself based on this. Note that in this case native functions still need to be bound!

General JSON structure

{
    "protocol": {
    },
    "document": {
    },
    "types": [
    ],
    "nodes": [
    ]
}

Protocol

Contains the description of the protocol being use. Can be used to validate if the current file matches the currently used VM or Language file. Every VM can typicaly only handle one protocol at the same time.

{
    "protocol": {
        "version": "0.0.1",
        "lang": "noodle"
    },
    // ...
}

Document

Can be used a "untitled" document in a VSE/VPE such as in our example VS Code Extension Editor.

    "document": {
        "protocol": {
            "version": "0.0.1",
            "lang": "noodle"
        },
        "graphs":[],
        "editors": []
    },

For more information on the Document and how the graphs are structured please check later Document Specification.

Nodes

Bare bone description of the nodes. This can be used by a VSE/VEP to parse the .graph fields. All graphs are composed out of these nodes.

 {
 "name": "Number",
 "uuid": "123e4567-e89b-12d3-a456-426614174000",
 "input": [],
 "output": []
},

Details: input/output

{
"name": "num",
"pretty_name": "Number",
"type": "number",
"control": {},
"read_only": false,
"default_value": null,
"ref": false
}

Reference type is only supported if the VM settings allow this. Also read_only might be implcit and one needs to specifically state that a input/output can be written to. This will be subject to change and only avalilbe for reference types since all wires are in general const and not mutable.

Control Specifications

Can be ignored by the editor since the editor is implementation defined. This is just a method for the VM to provide suggestions or extra information.

General Overview

{
    "type": "select",
    "items": [
        "a","b"
    ],
    "options": [
        1,2
    ]
}

Types

These are the identifications of the supported data types. In its default configuration the VM understands:

• number - a float 64 value (double)
• boolean - true / false
• string - Its basically a const char* & uint32_t size and uint32_t end of line flag and owned by the VM

    "types": [
        {
            "name": "number",
            "compatible": []
        }
    ],

Document Specification

A Noodle file is a file with the file ending: .noodle and follows the same structure as defined in the Noodle Language Specifications.

General Overview

{
    "protocol": {},
    "graphs": [],
    "editors": []
}

Protocol

Contains the description of the protocol being use. Can be used to validate if the current file matches the currently used VM or Language file. Every VM can typicaly only handle one protocol at the same time.

{
    "protocol": {
        "version": "0.0.1",
        "lang": "noodle"
    },
    // ...
}

Graphs

{
    "protocol": {
        "version": "0.0.1",
        "lang": "noodle"
    },
    "graphs": [ ],
    "editors": []
}

Details graphs object

{
    "protocol": {
        "version": "0.0.1",
        "lang": "noodle"
    },
        "graphs": [
            {
                "name": "Main",
                "graph": {
                    "nodes": [],
                    "connections": [],
                    "data": [],                    
                }
            }
        ],
    "editors": []
}

Details graph

Details: node

            {
                "name": "Number",
                "node":"123e4567-e89b-12d3-a456-426614174000",
                "id": 1
            }

Details: connections

"connections": [
            [],
            [],
            [
                {
                    "type": "output",
                    "name": "res",                    
                    "extern": "b",
                    "node": 5                
                }
            ],
    ]

Details: data

"data": [
    {},
    {	"num":{
        	"type": "number",
	        "value": "1"
    	}
    }
    ]

The data object itself is structured in a way that the Connector name is the name of the object key.

Editor

The editor field is implementation defined for the specific Graph Editor. This can be anything the Visual Scripting/Programming Environment (VSE/VPE) needs to display and allow editing the graph.

Language Server API

The Language Server API provides a WebSocket API the end user can connect to. The WebSocket API allows a bidirectional communication which can be seen as a favourable model over a REST API.

Client Requests

The Language Server (LS) will operate on a specific port (specified in the configuration of the server) and allows N clients to connect. When a client connected the client has the possibility to request certain information from the Language Server:

Client Events

The Client can send the following events to the LS:

Server Events

The LS can update the client with:

Example Communication

The user changes a graph and sends /document_changed to the server the server will apply the changes to the VM. The server will return the validation results via /error if there has been a error otherwiese nothing will be send to the user.

Noodle API Specifications

Noodles API is written in the C programming language since C is considered as the lingua franca of programming languages [18]–[21]. To provide maximal portability from an API point of view, the header files of Noodle will be written in C99 and the implementation will be done in C11.

Including

There are two ways to get the Noodle embedded into your program:

1. Link the statically build library or the dyanmically library.
2. You can include the noodle source code as part of your application.

However you choose to embedd noodle, you also want to add src/include to your include path to be able to include the noodle.h file.

Noodle depends only on the C standard library, so you don’t usually need to link to anything else. Be aware that this is a subject to change and maybe we offer a customization point to get ride of these dependencies as well.

If your program is in C++ but its linking to the noodle library compiled as C you need to include the following header file:

#include "noodle.hpp"

Initilization

This will be added soon

Example

The Visual Studio Code Example