MetaFFI is a Multilingual Indirect Interoperability System. It’s a clever solution to a common problem: how do we make different programming languages play nicely together?

Each language has its strengths, and sometimes we need to harness those strengths in harmony. But here comes the challenge, how do we get these languages to talk to each other without all the low-level hassle?

MetaFFI uses the concept of loading and using dynamic libraries, but expands it to any programming language module. Not just binary. It can be a Python package, a Go module, or even a Java class.

To be a bit more technical, MetaFFI uses Foreign Function Interface (FFI) and embedding mechanisms to link the runtimes together using a C-based Hub. It’s like a bridge that connects different programming languages, allowing them to communicate and share data.

There is no virtual machine, interpreters or re-compilation envolved - each programming language runs in its own original runtime.

Installation

Installer is available at:

v0.3.0 - Windows v0.3.0 - Ubuntu

-s: To install in silent mode with default installation directory.

To uninstall, execute uninstall executable in the installation directory.

Installers tested on: Windows 11, Windows Server 2022, Ubuntu 22.04

Windows CMD: curl -LO https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-installer-0.3.0.exe && metaffi-installer-0.3.0.exe -s

Windows PowerShell: Invoke-WebRequest -Uri https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-installer-0.3.0.exe -OutFile metaffi-installer.exe; Start-Process .\metaffi-installer.exe -ArgumentList '-s' -Wait

Ubuntu bash: wget https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-installer-0.3.0 && chmod +x metaffi-installer-0.3.0 && ./metaffi-installer-0.3.0 -s

Plugin installers:

• Python 3 - Windows, Ubuntu
  • PIP package is available at pip3 install metaffi-api
  • Tested with Python 3.11
  • Should work with Python >=3.11, but not tested with installer
  • Should work with Python <3.10, but not tested
• Go - Windows, Ubuntu
  • Currently, supports Go 1.23.1 - will be support any go version in near future
• OpenJDK - Windows, Ubuntu
  • JAR ships with the installer
  • Tested with OpenJDK 21
  • Tested with OpenJDK 11 without testing installer

Windows CMD: curl -LO https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-python311.exe && metaffi-plugin-installer-0.3.0-python311.exe && curl -LO https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-go.exe && metaffi-plugin-installer-0.3.0-go.exe && curl -LO https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-openjdk.exe && metaffi-plugin-installer-0.3.0-openjdk.exe

Windows PowerShell: Invoke-WebRequest https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-python311.exe -OutFile python311.exe; Start-Process .\python311.exe -Wait; Invoke-WebRequest https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-go.exe -OutFile go.exe; Start-Process .\go.exe -Wait; Invoke-WebRequest https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-openjdk.exe -OutFile openjdk.exe; Start-Process .\openjdk.exe -Wait

Ubuntu bash: wget https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-python311 -O python311 && chmod +x python311 && ./python311 && wget https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-go -O go && chmod +x go && ./go && wget https://github.com/MetaFFI/metaffi-root/releases/download/v0.3.0/metaffi-plugin-installer-0.3.0-openjdk -O openjdk && chmod +x openjdk && ./openjdk

What does the MetaFFI installer do?

Windows

• Checks for prerequisites for current programming languages support
  • Installer offers to install missing dependencies if somehthing is missing (automatic in silent mode)
• Copies the MetaFFI files to installation directory (default is %USERPROFILE%\metaffi)
• Add installation directory to PATH environment variables
• Adds METAFFI_HOME environment variable pointing at the installation directory

Linux

• Checks for prerequisites for current programming languages support
  • Installer offers to install missing dependencies if somehthing is missing (automatic in silent mode)
• Copies the MetaFFI files to installation directory (default is /usr/local/metaffi)
• Adds METAFFI_HOME environment variable to ~/.profile pointing at the installation directory

Pre-Installed Docker Container

Ubuntu 22.04: docker pull tscs/metaffi-u2204:0.3.0 Windows server core 2022: docker pull tscs/metaffi-win-core2022:0.3.0

Distribute Binaries

You can distribute MetaFFI binaries with your application (under the license terms). To use MetaFFI on the target machine, you can either install MetaFFI using the installer, or place the MetaFFI directory (without the executable) within your application directory.

Make sure to set METAFFI_HOME environment variable. You can set it in your application before your code uses MetaFFI. On windows, also add to PATH environment variable.

(Currently) Supported Programming Langauges

Language	Supported	Fully Tested With Installer
Go	From v1.22.7	v1.18 → v1.23
JVM Languages	JNI supported JVM	OpenJDK11/21 x64 Microsoft OpenJDK11/21 Hotspot JVM
Python3	v3.x	v3.11

• Note: Due to a bug in Go, using Go → OpenJDK in Windows, causes the process to crash. Fix is expected in Go1.23. In the meantime, MetaFFI install provides a temporary patch to fix the issue.

• Lack of support is not due to system limitations, but time. If you like the project, consider to contribute add a new language support 😊

(Currently) Supported Operating Systems

Operating System	Supported Versions	Tested with installer
Windows	>7	11
Ubuntu	>20.04	22.04

• Lack of support is not due to system limitations, but time. If you like the project, consider to contribute and add a new language support 😊

Build From Source

MetaFFI uses CMake build system and vcpkg for dependency management. The build system is cross-platform and should work on any platform supported by CMake.

Author uses CLion IDE for the core development, but you can use any IDE that supports CMake.

1. Install CMake and VCPKG
2. Pull https://github.com/MetaFFI/metaffi-root
3. Run the CMake configuration and build
4. CMake script will pull the required dependencies

The GitHub project contains:

• MetaFFI Core
• Python3 Plugin
• JVM Plugin
• Go Plugin

MetaFFI Paper

The paper (link) discusses the research and internals of MetaFFI. Sections for academic audiance or technical audiance are explicitly marked, as explained in the end of the introduction section.

Documentation

The following provide documentation and explaination about the following MetaFFI APIs:

• GO API documentation and Entity Path documentation
• JVM API documentation and Entity Path documentation
• Python3 API documentation and Entity Path documentation

Two Usage Examples

log4j from Python3 (link):

# load JVM
runtime = metaffi.metaffi_runtime.MetaFFIRuntime('openjdk')

# load log4j
log4j_api_module = runtime.load_module('log4j-api-2.21.1.jar;log4j-core-2.21.1.jar')

# load getLogger() method to get a new logger
getLogger = log4j_api_module.load_entity('class=org.apache.logging.log4j.LogManager,callable=getLogger', 
		[new_metaffi_type_info(metaffi_string8_type)], 
		[new_metaffi_type_info(metaffi_handle_type, 'org.apache.logging.log4j.Logger')])

# load error() method in logger
perror = log4j_api_module.load_entity('class=org.apache.logging.log4j.Logger,callable=error,instance_required',
	[new_metaffi_type_info(MetaFFITypes.metaffi_handle_type),
	new_metaffi_type_info(MetaFFITypes.metaffi_string8_type)],
	None)

# create logger with getLogger()
logger = getLogger('pylogger')
perror(logger, 'Logging error from python!')

runtime.release_runtime_plugin()

More examples from Python3, Java and Go.

Example using a compiler

Some programming languages, like Go, need some help from MetaFFI to build a module available to other programming languages. For that, MetaFFI also provides a compiler to build a MetaFFI enabled module.

For example, assume TestMap.go:

func NewTestMap() *TestMap{
	return &TestMap{ 
		m: make(map[string]interface{}),
		Name: "TestMap Name",
	}
}

func (this *TestMap) Set(k string, v interface{}){
	this.m[k] = v
}

func (this *TestMap) Get(k string) interface{}{
	v := this.m[k]
	return v
}

func (this *TestMap) Contains(k string) bool{
	_, found := this.m[k]
	return found
}

To use it from other languages using MetaFFI, execute the MetaFFI compiler:

metaffi -c --idl TestMap.go -g

This creates a dynamic library for TestMap (i.e. .so or .dll). To use it, simply load the dynamic library using MetaFFI. Here’s an example in Java using JVM MetaFFI API:

// Load Go runtime
MetaFFIRuntime runtime = new MetaFFIRuntime("go");
runtime.loadRuntimePlugin();

// Load the compiled module (in this case, .dll in windows)
MetaFFIModule module = runtime.loadModule("TestMap_MetaFFIGuest.dll");

// Load a function that creates an instance of TestMap
metaffi.Caller newTestMap = module.load("callable=NewTestMap",
		null,
		new MetaFFITypeInfo[]{
			// returns MetaFFI Handle (i.e. handle to the object)
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIHandle)
		});

// Load Set() method
metaffi.Caller testMapSet = module.load("callable=TestMap.Set,instance_required",
		new MetaFFITypeInfo[]{
			// 1st parameter is an instance of the object
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIHandle),
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIString8), // key
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIAny) // // value
		},
		null); // no return values

// Load Contains() method
metaffi.Caller testMapContains = module.load("callable=TestMap.Contains,instance_required",
		new MetaFFITypeInfo[]{ 
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIHandle),
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIString8) // key
		},
		new MetaFFITypeInfo[]{ 
			// boolean return value
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIBool)
		});

// Load Get() method
metaffi.Caller testMapGet = module.load("callable=TestMap.Get,instance_required",
		new MetaFFITypeInfo[]{
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIHandle),
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIString8) // key
		},
		new MetaFFITypeInfo[]{
			new MetaFFITypeInfo(MetaFFITypes.MetaFFIAny) // returned value
		});

// Create new TestMap
// call() method in the API returns Object[], as other languages
// can return multiple return values. We will update the API for a more convenient usage.
var testMap = ((Object[])newTestMap.call())[0];

// set into the map an array of strings:
testMapSet.call(testMap, "key", new ArrayList<>(Arrays.asList("one", "two", "three")));

// get the array list from the map:
var arr = testMapGet.call(testMap, "key");
ArrayList<String> list = (ArrayList<String>)arr[0];

Finding a foreign entity - Entity Path

Entity path refers to a string that represents the foreign entity within the loaded module.

For instance, in C, a module corresponds to a .so/.dll file, and the function path corresponds to the name of the exported function.

However, in other languages or for other entities besides functions, a single name is insufficient. Hence, the function path consists of a list of key-value pairs or tags separated by commas: key1=val1,tag1,...,...,tagN,keyN=valN.

Each plugin requires different keys and tags. Although the keys and tags are similar across plugins, they are not identical.

The following links provide the list of each runtime plugin:

Python3, Java Virtual Machine, Go

Bridging Language Boundaries - MetaFFI XCall and Capabilities-based calling convension

The XCall is the mechanism MetaFFI uses to facilitate cross-language calls. MetaFFI’s agnostic approach ensures that each language remains unaware of the others, allowing for independent plugin development.

XCall is a runtime-independent calling convention that uses Common Data Types (inspired by Microsoft’s Variant and GTK gObject) to enable languages to call and use entities in other languages, even if they lack certain features. While the generic calling convention supports a wide range of cross-language calls, it can affect the performance. </br>Therefore, XCall determines the calling convention used at runtime, based on the required capabilities. This allows MetaFFI to use the full-featured calling convention when necessary, a subset of the capabilities to improve performance, or revert completely to a direct function call when possible (like x64 calling convension), resulting in efficient cross-language interactions.

For more details, please refer to the MetaFFI paper.

Note: The current version of MetaFFI always chooses the generic calling convention due to the differences between the initial languages implemented.

Report a bug

Found a bug? You can report it here.

GitHub Projects

The MetaFFI Project on GitHub.com contains several repositories:

• MetaFFI Root contains the SCons build system root (SConstruct), VSCode workspace file and the dev containers devcontainer.json files. To build from source, clone this.
• MetaFFI Core contains MetaFFI CLI tool, XLLR (Cross-Language Link Runtime), implementation of XCall (cross-call) and CDTs (Common Data Types)
• Python3 Plugin implements MetaFFI support for Python3 using CTypes and CPython API
• JVM Plugin implements support for OpenJDK using JNI
• Go Plugin implements support for Go using CGo
• Installer implements the MetaFFI Python installer
• Containers contain the Dockerfiles for the pre-installed containers
• metaffi.github.io contains this GitHub pages website

Environment variable

METAFFI_HOME environment variable is set to MetaFFI installation directory. On Windows, also set METAFFI_HOME to the PATH environment variable.