A new form of WebXR

By Kuldeep Singh

May 27, 2020 - 6 minutes read - 1110 words

Accessing native XR features from the device using a web page is another form of WebXR.

In this article, I want to share an experiment where I tried to control the android device’s native features such as controlling volume and camera torch from a web page. This use case can be really useful in the XR world, where I can control the feature access in the device from cloud services, even the whole look and feel for showing the device controls can be managed from a hosted web page.

I treat this use case and as a new form of Web XR, where I try to use native features of devices from the web. It opens the web page in Android WebView in the device, and then intercept the commands from the web page, and handle them natively. This way, we can also create a 3D app in Unity using 3D webview and show any 2D webpage in 3D. There could be many use-cases of this trial, I will leave up to you, and think more and let me know where you could use it more.

Let me start, and show you what I have done.

The WebView Activity

Let’s create an activity which opens webview, I have implemented it in one of my earlier articles. It intercepts the parameter “command” whenever there is a change in the URL. On the successful interception, it loads the URL with a new parameter called “result”. We will discuss the use or result later.

protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    final String baseURL = getIntent().getStringExtra(INTENT_INPUT_URL);
    webView = new WebView(this);
    webView.setWebViewClient(new WebViewClient() {
        public boolean shouldOverrideUrlLoading(WebView view, String url) {
            Uri uri = Uri.parse(url);
            Log.d(TAG, "shouldOverrideUrlLoading" + url);
            String command =  uri.getQueryParameter("command");
            if (command !=null) {
                ResultReceiver resultReceiver = getIntent().getParcelableExtra(RESPONSE_CALLBACK);
                if(resultReceiver!=null) {
                    Bundle bundle = new Bundle();
                    bundle.putString("command", command);
                    resultReceiver.send(RESULT_CODE, bundle);
                    Log.d(TAG, "sent response: -> " + command);
                }
                view.loadUrl(baseURL + "?result=" +  URLEncoder.encode("Command " + command + " executed"));
                return true;
            }
            view.loadUrl(url);
            return true; // then it is not handled by default action
        }
    });
    WebSettings settings = webView.getSettings();
    settings.setJavaScriptEnabled(true);
    setContentView(webView);
   //clear cache
    webView.loadUrl(baseURL);
}

Pass the command to the originator in the, via callbacks.

public static void launchActivity(Activity activity, String url, IResponseCallBack callBack){
    Intent intent = new Intent(activity, WebViewActivity.class);
    intent.putExtra(WebViewActivity.INPUT_URL, url);
    intent.putExtra(WebViewActivity.RESPONSE_CALLBACK, new CustomReceiver(callBack));
    activity.startActivity(intent);
  }
private static class CustomReceiver extends ResultReceiver{
  private  IResponseCallBack callBack;
  CustomReceiver(IResponseCallBack responseCallBack){
    super(null);
    callBack = responseCallBack;
  }
  @Override
  protected void onReceiveResult(int resultCode, Bundle resultData) {
    if(WebViewActivity.RESULT_CODE == resultCode){
      String command = resultData.getString(WebViewActivity.RESULT_PARAM_NAME);
      callBack.OnSuccess(command);
    } else {
      callBack.OnFailure("Not a valid code");
    }
  }
}

The originator needs to pass the callback while launching the webview.

public interface IResponseCallBack  {
    void OnSuccess(String result);
    void OnFailure(String result);
}

I have created the above activity in a separate library and defined it in the manifest of the library. however, it is not necessary you may keep it in one project as well. I kept it in the library so, in the future, I may include it in Unity applications.

The Main Activity

In the main project, lets now create the main activity which launches the WebView activity with a URL where our web page is hosted for controlling the android device, along with the Command Handler callbacks.

The Command Handler

It can handle volume up, down, and torch on and off.

class CommandHandler implements IResponseCallBack {
    @Override
    public void OnSuccess(String command) {
        Log.d("CommandHandler", command);
        if(COMMAND_VOLUME_UP.equals(command)){
            volumeControl(AudioManager.ADJUST_RAISE);
        } else if(COMMAND_VOLUME_DOWN.equals(command)){
            volumeControl(AudioManager.ADJUST_LOWER);
        } else if(COMMAND_TORCH_ON.equals(command)){
            switchFlashLight(true);
        } else if(COMMAND_TORCH_OFF.equals(command)){
            switchFlashLight(false);
        } else {
            Log.w("CommandHandler", "No matching handler found");
        }
    }

    @Override
    public void OnFailure(String result) {
        Log.e("CommandHandler", result);
    }

    void volumeControl (int type){
        AudioManager audioManager = (AudioManager) getApplicationContext().getSystemService(Context.AUDIO_SERVICE);
        audioManager.adjustVolume(type, AudioManager.FLAG_PLAY_SOUND);
    }

     void switchFlashLight(boolean status) {
        try {
            boolean isFlashAvailable = getApplicationContext().getPackageManager()
                    .hasSystemFeature(PackageManager.FEATURE_CAMERA_FLASH);
            if (!isFlashAvailable) {
                OnFailure("No flash available in the device");
            }
            CameraManager mCameraManager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);
            String mCameraId = mCameraManager.getCameraIdList()[0];
            mCameraManager.setTorchMode(mCameraId, status);
        } catch (CameraAccessException e) {
            OnFailure(e.getMessage());
        }
    }
}

Load Web Page in WebView

It will just launch the webview activity, with a URL.

protected void onCreate(@Nullable Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    WebViewActivity.launchActivity(this, "https://thinkuldeep.github.io/webxr-native/", new CommandHandler());
}

Define the Main Activity in Android Manifest and that’s it, the android side work is done.

The Web XR Page

Now the main part, how the Web XR page can pass the commands and reads the result back from android. It is quite simple, Here is the Index.html, which has 4 buttons on it to control the android native functionalities.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>New form of Web XR</title>
    <link rel="stylesheet" href="style.css">
</head>
<body>
<div  id="myDIV" class="header">
    <h2>New form of Web XR</h2>
    <h6>Control a device from a web page without a native app</h6>
    <div>
    <span onclick="javascript: doAction('volume-up')" class="btn">Volume Up</span>
    <span onclick="javascript: doAction('volume-down')" class="btn">Volume Down</span>
    </div>
    <div style="margin: 25px">
    <span onclick="javascript: doAction('torch-on')" class="btn">Torch On</span>
    <span onclick="javascript: doAction('torch-off')" class="btn">Torch Off</span>
    </div>
</div>
<div id="result" class="result">
</div>
<script>
    function onLoad(){
        const urlParams = new URLSearchParams(window.location.search);
        const result = urlParams.get('result');
        var resultElement = document.getElementById('result');
        resultElement.innerHTML = result;
    }
    onLoad();
    function doAction(command){
       var baseURL = window.location.protocol + "//" + window.location.host + window.location.pathname;
       window.location = baseURL + "?command=" + command;
    }
</script>
</body>
</html>

Look at the onClick action listener, it is just appending the command parameter and reload the page. That’s where webview will intercept the command pass it to the Command Handler in the android native layer.

Also, see the OnLoad method, it looks for the “result” parameter, and it found then it shows the result on load. The webview append a result parameter after intercepting a command.

The webpage is hosted in the same source code repo — https://thinkuldeep.github.io/webxr-native/

Try all the buttons and see if they should correctly add the command in the URL.

If you don’t want to get into the publishing webpage, then you can package the HTML pages in the android assets as well as follows

put the index.html and css in /src/main/assets
and change URL in Main Activity

@Override
protected void onCreate(@Nullable Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    WebViewActivity.launchActivity(this, "file:///android_asset/index.html", new CommandHandler());
}

That’s it.

Demo

So now you understand the whole flow, its time for a demo, right. Let’s go with it.

Try it on your phone, and see it working.

Conclusion

In this article, I have tried a few basics commands passing from a web app and android then can handle those commands natively. Now get the idea,

Here is the source code, Now I leave to you to extend it further. https://github.com/thinkuldeep/webxr-native

Try to address the following things —

Security implications in this approach?
How to handle the offline scenario
Reload/Sleep/Cache scenarios.
Build 3D app on top of webview activity, and build the app for say Oculus device.

It is originally published at XR Practices Medium Publication

＃xr ＃android ＃webxr ＃native app ＃tutorial ＃technology