Sandbox against malicious code in a Java application

In a simulation server environment where users are allowed to submit their own code to be run by the server, it would clearly be advantageous for any user-submitted code to be run in side a sandbox, not unlike Applets are within a browser. I wanted to be able to leverage the JVM itself, rather than adding another VM layer to isolate these submitted components.

This kind of limitation appears to be possible using the existing Java sandbox model, but is there a dynamic way to enable that for just the user-submitted parts of a running application?

1. Run the untrusted code in its own thread. This for example prevents problems with infinite loops and such, and makes the future steps easier. Have the main thread wait for the thread to finish, and if takes too long, kill it with Thread.stop. Thread.stop is deprecated, but since the untrusted code shouldn't have access to any resources, it would be safe to kill it.

2. Set a SecurityManager on that Thread. Create a subclass of SecurityManager which overrides checkPermission(Permission perm) to simply throw a SecurityException for all permissions except a select few. There's a list of methods and the permissions they require here: Permissions in the Java^TM 6 SDK.

3. Use a custom ClassLoader to load the untrusted code. Your class loader would get called for all classes which the untrusted code uses, so you can do things like disable access to individual JDK classes. The thing to do is have a white-list of allowed JDK classes.

4. You might want to run the untrusted code in a separate JVM. While the previous steps would make the code safe, there's one annoying thing the isolated code can still do: allocate as much memory as it can, which causes the visible footprint of the main application to grow.

JSR 121: Application Isolation API Specification was designed to solve this, but unfortunately it doesn't have an implementation yet.

This is a pretty detailed topic, and I'm mostly writing this all off the top of my head.

But anyway, some imperfect, use-at-your-own-risk, probably buggy (pseudo) code:

ClassLoader

class MyClassLoader extends ClassLoader {
  @Override
  public Class<?> loadClass(String name) throws ClassNotFoundException {
    if (name is white-listed JDK class) return super.loadClass(name);
    return findClass(name);
  }
  @Override
  public Class findClass(String name) {
    byte[] b = loadClassData(name);
    return defineClass(name, b, 0, b.length);
  }
  private byte[] loadClassData(String name) {
    // load the untrusted class data here
  }
}

SecurityManager

class MySecurityManager extends SecurityManager {
  private Object secret;
  public MySecurityManager(Object pass) { secret = pass; }
  private void disable(Object pass) {
    if (pass == secret) secret = null;
  }
  // ... override checkXXX method(s) here.
  // Always allow them to succeed when secret==null
}

Thread

class MyIsolatedThread extends Thread {
  private Object pass = new Object();
  private MyClassLoader loader = new MyClassLoader();
  private MySecurityManager sm = new MySecurityManager(pass);
  public void run() {
    SecurityManager old = System.getSecurityManager();
    System.setSecurityManager(sm);
    runUntrustedCode();
    sm.disable(pass);
    System.setSecurityManager(old);
  }
  private void runUntrustedCode() {
    try {
      // run the custom class's main method for example:
      loader.loadClass("customclassname")
        .getMethod("main", String[].class)
        .invoke(null, new Object[]{...});
    } catch (Throwable t) {}
  }
}

Obviously such a scheme raises all sorts of security concerns. Java has a rigorous security framework, but it isn't trivial. The possibility of screwing it up and letting an unprivileged user access vital system components shouldn't be overlooked.

That warning aside, if you're taking user input in the form of source code, the first thing you need to do is compile it to Java bytecode. AFIAK, this cannot be done natively, so you'll need to make a system call to javac, and compile the source code to bytecode on disk. Here's a tutorial that can be used as a starting point for this. Edit: as I learned in the comments, you actually can compile Java code from source natively using javax.tools.JavaCompiler

Once you have JVM bytecode, you can load it into the JVM using a ClassLoader's defineClass function. To set a security context for this loaded class you will need to specify a ProtectionDomain. The minimal constructor for a ProtectionDomain requires both a CodeSource and a PermissionCollection. The PermissionCollection is the object of primary use to you here- you can use it to specify the exact permissions the loaded class has. These permissions should be ultimately enforced by the JVM's AccessController.

There's a lot of possible points of error here, and you should be extremely careful to completely understand everything before you implement anything.

Java-Sandbox is a library for executing Java code with a limited set of permissions.

It can be used to allow access to only a set of white-listed classes and resources. It doesn't seem to be able to restrict access to individual methods. It uses a system with a custom class loader and security manager to achieve this.

I have not used it but it looks well designed and reasonably well documented.

@waqas has given a very interesting answer explaining how this is possible to implement yourself. But it is much safer to leave such security critical and complex code to experts.

Note: The project has not been updated since 2013 and the the creators describe it as "experimental". Its home page has disappeared but the Source Forge entry remains.

Example code adapted from the project web site:

SandboxService sandboxService = SandboxServiceImpl.getInstance();

// Configure context 
SandboxContext context = new SandboxContext();
context.addClassForApplicationLoader(getClass().getName());
context.addClassPermission(AccessType.PERMIT, "java.lang.System");

// Whithout this line we get a SandboxException when touching System.out
context.addClassPermission(AccessType.PERMIT, "java.io.PrintStream");

String someValue = "Input value";

class TestEnvironment implements SandboxedEnvironment<String> {
    @Override
    public String execute() throws Exception {
        // This is untrusted code
        System.out.println(someValue);
        return "Output value";
    }
};

// Run code in sandbox. Pass arguments to generated constructor in TestEnvironment.
SandboxedCallResult<String> result = sandboxService.runSandboxed(TestEnvironment.class, 
    context, this, someValue);

System.out.println(result.get());

To address the problem in the accepted answer whereby the custom SecurityManager will apply to all threads in the JVM, rather than on a per-thread basis, you can create a custom SecurityManager that can be enabled/disabled for specific threads as follows:

import java.security.Permission;

public class SelectiveSecurityManager extends SecurityManager {

  private static final ToggleSecurityManagerPermission TOGGLE_PERMISSION = new ToggleSecurityManagerPermission();

  ThreadLocal<Boolean> enabledFlag = null;

  public SelectiveSecurityManager(final boolean enabledByDefault) {

    enabledFlag = new ThreadLocal<Boolean>() {

      @Override
      protected Boolean initialValue() {
        return enabledByDefault;
      }

      @Override
      public void set(Boolean value) {
        SecurityManager securityManager = System.getSecurityManager();
        if (securityManager != null) {
          securityManager.checkPermission(TOGGLE_PERMISSION);
        }
        super.set(value);
      }
    };
  }

  @Override
  public void checkPermission(Permission permission) {
    if (shouldCheck(permission)) {
      super.checkPermission(permission);
    }
  }

  @Override
  public void checkPermission(Permission permission, Object context) {
    if (shouldCheck(permission)) {
      super.checkPermission(permission, context);
    }
  }

  private boolean shouldCheck(Permission permission) {
    return isEnabled() || permission instanceof ToggleSecurityManagerPermission;
  }

  public void enable() {
    enabledFlag.set(true);
  }

  public void disable() {
    enabledFlag.set(false);
  }

  public boolean isEnabled() {
    return enabledFlag.get();
  }
    
}

ToggleSecurirtyManagerPermission is just a simple implementation of java.security.Permission to ensure that only authorised code can enable/disable the security manager. It looks like this:

import java.security.Permission;

public class ToggleSecurityManagerPermission extends Permission {

  private static final long serialVersionUID = 4812713037565136922L;
  private static final String NAME = "ToggleSecurityManagerPermission";

  public ToggleSecurityManagerPermission() {
    super(NAME);
  }

  @Override
  public boolean implies(Permission permission) {
    return this.equals(permission);
  }

  @Override
  public boolean equals(Object obj) {
    if (obj instanceof ToggleSecurityManagerPermission) {
      return true;
    }
    return false;
  }

  @Override
  public int hashCode() {
    return NAME.hashCode();
  }

  @Override
  public String getActions() {
    return "";
  }

}

Here's a thread-safe solution for the problem:

https://svn.code.sf.net/p/loggifier/code/trunk/de.unkrig.commons.lang/src/de/unkrig/commons/lang/security/Sandbox.java

package de.unkrig.commons.lang.security;

import java.security.AccessControlContext;
import java.security.Permission;
import java.security.Permissions;
import java.security.ProtectionDomain;
import java.util.Collections;
import java.util.HashMap;
import java.util.Map;
import java.util.WeakHashMap;

import de.unkrig.commons.nullanalysis.Nullable;

/**
 * This class establishes a security manager that confines the permissions for code executed through specific classes,
 * which may be specified by class, class name and/or class loader.
 * <p>
 * To 'execute through a class' means that the execution stack includes the class. E.g., if a method of class {@code A}
 * invokes a method of class {@code B}, which then invokes a method of class {@code C}, and all three classes were
 * previously {@link #confine(Class, Permissions) confined}, then for all actions that are executed by class {@code C}
 * the <i>intersection</i> of the three {@link Permissions} apply.
 * <p>
 * Once the permissions for a class, class name or class loader are confined, they cannot be changed; this prevents any
 * attempts (e.g. of the confined class itself) to release the confinement.
 * <p>
 * Code example:
 * <pre>
 *  Runnable unprivileged = new Runnable() {
 *      public void run() {
 *          System.getProperty("user.dir");
 *      }
 *  };
 *
 *  // Run without confinement.
 *  unprivileged.run(); // Works fine.
 *
 *  // Set the most strict permissions.
 *  Sandbox.confine(unprivileged.getClass(), new Permissions());
 *  unprivileged.run(); // Throws a SecurityException.
 *
 *  // Attempt to change the permissions.
 *  {
 *      Permissions permissions = new Permissions();
 *      permissions.add(new AllPermission());
 *      Sandbox.confine(unprivileged.getClass(), permissions); // Throws a SecurityException.
 *  }
 *  unprivileged.run();
 * </pre>
 */
public final
class Sandbox {

    private Sandbox() {}

    private static final Map<Class<?>, AccessControlContext>
    CHECKED_CLASSES = Collections.synchronizedMap(new WeakHashMap<Class<?>, AccessControlContext>());

    private static final Map<String, AccessControlContext>
    CHECKED_CLASS_NAMES = Collections.synchronizedMap(new HashMap<String, AccessControlContext>());

    private static final Map<ClassLoader, AccessControlContext>
    CHECKED_CLASS_LOADERS = Collections.synchronizedMap(new WeakHashMap<ClassLoader, AccessControlContext>());

    static {

        // Install our custom security manager.
        if (System.getSecurityManager() != null) {
            throw new ExceptionInInitializerError("There's already a security manager set");
        }
        System.setSecurityManager(new SecurityManager() {

            @Override public void
            checkPermission(@Nullable Permission perm) {
                assert perm != null;

                for (Class<?> clasS : this.getClassContext()) {

                    // Check if an ACC was set for the class.
                    {
                        AccessControlContext acc = Sandbox.CHECKED_CLASSES.get(clasS);
                        if (acc != null) acc.checkPermission(perm);
                    }

                    // Check if an ACC was set for the class name.
                    {
                        AccessControlContext acc = Sandbox.CHECKED_CLASS_NAMES.get(clasS.getName());
                        if (acc != null) acc.checkPermission(perm);
                    }

                    // Check if an ACC was set for the class loader.
                    {
                        AccessControlContext acc = Sandbox.CHECKED_CLASS_LOADERS.get(clasS.getClassLoader());
                        if (acc != null) acc.checkPermission(perm);
                    }
                }
            }
        });
    }

    // --------------------------

    /**
     * All future actions that are executed through the given {@code clasS} will be checked against the given {@code
     * accessControlContext}.
     *
     * @throws SecurityException Permissions are already confined for the {@code clasS}
     */
    public static void
    confine(Class<?> clasS, AccessControlContext accessControlContext) {

        if (Sandbox.CHECKED_CLASSES.containsKey(clasS)) {
            throw new SecurityException("Attempt to change the access control context for '" + clasS + "'");
        }

        Sandbox.CHECKED_CLASSES.put(clasS, accessControlContext);
    }

    /**
     * All future actions that are executed through the given {@code clasS} will be checked against the given {@code
     * protectionDomain}.
     *
     * @throws SecurityException Permissions are already confined for the {@code clasS}
     */
    public static void
    confine(Class<?> clasS, ProtectionDomain protectionDomain) {
        Sandbox.confine(
            clasS,
            new AccessControlContext(new ProtectionDomain[] { protectionDomain })
        );
    }

    /**
     * All future actions that are executed through the given {@code clasS} will be checked against the given {@code
     * permissions}.
     *
     * @throws SecurityException Permissions are already confined for the {@code clasS}
     */
    public static void
    confine(Class<?> clasS, Permissions permissions) {
        Sandbox.confine(clasS, new ProtectionDomain(null, permissions));
    }

    // Code for 'CHECKED_CLASS_NAMES' and 'CHECKED_CLASS_LOADERS' omitted here.

}

Please comment!

CU

Arno

Well it's very late to give any suggestions or solutions, but still I was facing similar kind of issue, kind of more research oriented. Basically I was trying to provide a provision and automatic evaluations for programming assignments for Java course in e-learning platforms.

1. one way could be, Create a separate virtual machines (not JVM) but actual virtual machines with minimum configuration possible OS for each of student.
2. Install JRE for Java or libraries according to your programming languages, whichever you want students to compile and execute on these machines.

I know this sounds a quite complex and lot of tasks, but Oracle Virtual Box already provides Java API to create or clone virtual machines dynamically. https://www.virtualbox.org/sdkref/index.html (Note, even VMware also provides API for doing same)

And for the minimum size and configuration Linux distribution you can refer to this one here http://www.slitaz.org/en/,

So now if students messes up or tries to do it, may be with memory or file system or networking, socket, maximum he can damage his own VM.

Also internally into these VM's you can provide additional security like Sandbox (security manager ) for Java or creating user specific accounts on Linux and thus restricting access.

Hope this helps !!

You will probably need to use a custom SecurityManger and/or AccessController. For lots of detail, see Java Security Architecture and other security documentation from Sun.