Embedding KCL in Rust

What is KCL?

KCL is a new & interesting configuration language.
It looks a bit like a mixture of YAML and HCL.

# This is a KCL document

title = "KCL Example"

owner = {
    name = "The KCL Authors"
    date = "2020-01-02T03:04:05"
}

database = {
    enabled = True
    ports = [8000, 8001, 8002]
    data = [["delta", "phi"], [3.14]]
    temp_targets = {cpu = 79.5, case = 72.0}
}

servers = [
    {ip = "10.0.0.1", role = "frontend"}
    {ip = "10.0.0.2", role = "backend"}
]

But what it brings to the table is KCL adds typing and data validation to your configuration.

Validation and typing can be applied to your data via schema.

schema User:
    name: str
    age: int
    message?: str
    data: Data
    labels: {str:}
    hc: [int]

    check:
        age > 10, "age must > 10"

schema Data:
    id: int
    value: int = 9000

With a schema we can define which fields a certain kind of data should have.
Furthermore we can add types, set default values and even apply constraints to the fields via check.

After the definition, a schema can be instantiated.

User {
  name = "Foo"
  age = 42
  data = Data {
    id = 0
  }
  labels = {
    bar = 12
    baz = "some label"
  }
  hc = [1,2,3]
}

KCL code can then be 'compiled' into standard JSON or YAML.
The user code snippet from above would 'compile' to:

{
  "name": "Foo",
  "age": 42,
  "data": {
    "id": 0,
    "value": 9000
  },
  "labels": {
    "bar": 12,
    "baz": "some label"
  },
  "hc": [
    1,
    2,
    3
  ]
}

or

name: Foo
age: 42
data:
  id: 0
  value: 9000
labels:
  bar: 12
  baz: some label
hc:
- 1
- 2
- 3

KCL has even more interesting features, like control flow, modules, and even a registry for these modules, so that schemas can be easily shared and reused.
But that should be enough as a KCL intro, if you want to know more about KCL just visit kcl-lang.io.

Embedding KCL in Rust

After realising that KCL itself is written in Rust (in most examples you see KCL as a CLI tool), I decided it would be an interesting idea to embed KCL in my Rust application to validate the configuration file of the application.

Luckily, KCL has a lib that offers multiple language bindings and can easily be added via:

cargo add --git https://github.com/kcl-lang/lib kcl-lang

Afterwards, you can compile KCL in your Rust code like this:

let api = API::default();
let args = &ExecProgramArgs {
    k_filename_list: vec!["main.k".to_string()],
    k_code_list: vec!["a = 1".to_string()],
    ..Default::default()
};
let exec_result = api.exec_program(args)?;
println!("{}", exec_result.yaml_result);

I assume that is the CLI API, because we have to add a list of files (they have no relevancy for that use case, but you can't skip them) and receive the Stdout & Stderr output the CLI would produce with the exec_result.

The idea was to add the configuration schema to the source code and include it during compile time via include_str!() macro.

The schema looks like this:

# schema.k
schema Configuration:
  api: API = API {}
  gateway: Gateway = Gateway {}

schema API:
  port: int = 8150
  check:
    1 <= port <= 65535, "API Port not in range 1 to 65535"

schema Gateway:
  port: int = 80
  check:
    1 <= port <= 65535, "Gatewayport Port not in range 1 to 65535"

The corresponding Rust struct would be:

#[derive(Debug, Serialize, Deserialize)]
pub struct Configuration {
    pub api: APIConfiguration,
    pub gateway: GatewayConfiguration,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct APIConfiguration {
    pub port: u16,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct GatewayConfiguration {
    pub port: u16,
}

I intended to enable the users to write their own configuration file in KCl like:

# Long version
api = {
  port = 8150
}
# Short version
gateway.port = 8080

Then compile that configuration file against my configuration schema that includes all necessary default values.

So I want to combine the user configuration file content with my schema definition to the following:

# schema.k content
schema Configuration:
  api: API = API {}
  gateway: Gateway = Gateway {}

schema API:
  port: int = 8150
  check:
    1 <= port <= 65535, "API Port not in range 1 to 65535"

schema Gateway:
  port: int = 80
  check:
    1 <= port <= 65535, "Gatewayport Port not in range 1 to 65535"

Configuration {
  # User configuration file content
  api = {
    port = 8150
  }
  gateway.port = 8080
}

And finally, the necessary Rust code to deserialize the configuration struct from the compiled KCL code:

impl Configuration {
    pub fn default() -> Result<Self> {
        Self::process_kcl(None)
    }

    pub fn from_configuration_file(path: &Path) -> Result<Self> {
        Self::process_kcl(Some(path))
    }

    fn process_kcl(path: Option<&Path>) -> Result<Self> {
        let mut file_name = "default-configuration.k".to_string();
        let mut code = String::new();
        if let Some(path) = path {
            file_name = "configuration.k".to_string();
            code = std::fs::read_to_string(path)?;
        }
        let code = format!("{}\nConfiguration{{\n{}}}", SCHEMA, code);
        let args = &ExecProgramArgs {
            k_filename_list: vec![file_name],
            k_code_list: vec![code],
            ..Default::default()
        };
        let api = API::default();
        let exec_result = api.exec_program(args)?;
        if !exec_result.err_message.is_empty() {
            return Err(anyhow!(
                "Configuration error:\n\n{}",
                exec_result.err_message
            ));
        }
        serde_json::from_str(&exec_result.json_result)
          .map_err(|err| anyhow!(err))
    }
}

Without providing a configuration file we compile against a Configuration{} instance, which then gets all the default values from the schema.

A bit strange seems that we don't get an error result if we compile against some user configuration that is not valid for the provided schema.

The only way to check for a compilation error is to check if the exec_result.err_message is "". This error message is the stderr output that the CLI tool would produce if the compilation fails.
Cumbersome, but at least we get a good error message with specific information about what is wrong with the configuration KCL code.

Last step is then to deserialize the Configuration struct from the json representation of the compilation result -> we can configure our Rust application via KCL configuration file.

Last step is then to deserialize the Configuration struct from the JSON representation of the compilation result -> we can configure our Rust application via KCL configuration file.

Conclusion

First thought: It works and that is a good thing!

Second thought: Working with the CLI API in the Rust code is somehow awkward and afterwards deserializing the struct again from JSON or YAML representation seems unnecessary.
I'm sure there is a smarter way to directly parse the KCL and create the struct with the serde values that are produced by the parser.
Sadly the documentation of the kcl-lang lib is kept very short and I couldn't find an approach like that.

What about a Procedural Macro?

All that boilerplate code could also be generated by a procedural macro that is applied to the Configuration struct, similar to what e.g. serde, clap etc. do.

Nevertheless, KCL is interesting

KCL configuration is in my opinion a great supplement to Rust.
Similar to Rust, KCL uses types & constraints to minimize the possibility that our configuration is incorrect.