Sensors found in smartphones define the mobile experience. GPS and the magnetometer enable the fluid experience of maps; motion sensing enables activity recognition and games, and of course the camera and microphone allow whole categories of rich media applications. Beyond these now obvious examples, sensors can also enable clever inventions, such as Cycloramic, which used the vibrator motor in iPhones (4 and 5) to rotate the phone and take a panorama, pushup counters which use the proximity sensor to count repetitions, and Send Me To Heaven, which uses the accelerometer to determine flight time of a phone thrown vertically as high as possible. I've had some experience using and abusing sensors too, most recently for the Cardboard magnet button.

However, over the last couple of years, I've had to step away from the web as a development platform, in part because of the poor state of sensor APIs. In this post, I will describe some of the problems, take a look at sensor APIs on iOS and Android, and suggest a solution in the spirit of the extensible web manifesto.

Existing sensor APIs are underspecified

One of the most popular sensor APIs on the web is the DeviceMotion event API, which is basically always just an opaque abstraction around the accelerometer. The web, as always, tries to solve the problem in the most general way possible:

This specification provides several new DOM events for obtaining information about the physical orientation and movement of the hosting device. The information provided by the events is not raw sensor data, but rather high-level data which is agnostic to the underlying source of information. Common sources of information include gyroscopes, compasses and accelerometers.

This could be fine in theory, except the specs end up being so vague in their attempt to please everybody, that they under-specify the behavior of events such as DeviceOrientation. Throw in some rogue implementers, and you end up with huge discrepancies in browsers, as Pete found back in 2011:

For most browsers, alpha returns the compass heading, so when the device is pointed north, alpha is zero. With Mobile Safari, alpha is based on the direction the device was pointing when device orientation was first requested. The compass heading is available in the webkitCompassHeading parameter.

A useful sensor abstraction would be to build a compass on top of the magnetometer (and maybe gyro) sensors, and then expose that as a high level Compass API. Unfortunately many web sensor APIs give us a mid-level of abstraction. They don't map reliably to particular hardware sensors, nor do they provide much use. Sensors allow many applications that were not originally envisioned by the spec writers. By choosing poorly specified ivory-tower abstractions, the web limits what can be done on the platform.

Low level sensor APIs don't exist

While you can work around the insanity of Device* style events on the web with platform-specific shims, you cannot work around missing sensor APIs. Magnetometers, pressure sensors, proximity, light, temperature, battery, etc. These are mostly missing, and the ones that are specified are specified in a very narrow way that does not generalize across to other types of sensors (eg. DeviceLightEvent).

Unfortunately it seems that previous attempts to push for a general low level sensor API haven't really gotten much traction. In fact, it's a bit unclear whether or not the Device API working group, is even the right place for sensor APIs, since their mandate is supposedly more about services than sensors:

[To] enable the development of Web Applications and Web Widgets that interact with devices services such as Calendar, Contacts, Camera, etc.

Except here's a sensor API from the same group, which seems to be abandoned... I don't even

There are more recent voices (circa 2014) that seem to be pushing in a generic sensor API direction, from folks like Rick Waldron and Tim Volodine. Many of these ideas are still working within the confines of a sensor API for each type of sensor. This does not scale well for the web, which tends to take a long time for any new web standard, but this renewed interest is very exciting and promising!

Sensors on other platforms

The web is woefully behind native platforms in almost every regard (with possibly the exception of audio). Sensors on iOS and Android have a rich history, and ended up in a pretty similar place as the two platforms have scrambled to converge. Let's take a look.

iOS started off with a UIAccelerometer API, which was replaced by CoreMotion in iOS 5. Rather than providing a series of specific APIs for each type of sensor API as it had before, CoreMotion provides a unified framework for sensor events. Each data type inherits from a common base class CMLogItem, and most of the API is encapsulated in CMMotionManager, which explicitly lists accelerometer, gyroscope and magnetometer-related APIs. iOS went from specific to generic, which makes it super easy to add new types of sensor data. That said, the API is generic only for motion sensors, which excludes a bunch of sensors not directly related to motion like temperature, humidity, etc.

Android started off right, and hasn't had to change much, providing a generic API for sensors since API level 3. Android's API is accessed through a SensorManager, which provides a somewhat overly abstract API, because of its support for multiple sensors of one type (eg. two accelerometers) in the same device. Still, the idea is good, and all of the low level sensor data are well specified (per sensor type)so the hardware/firmware vendor knows what data format their sensor should stream. Of course there are still rogue implementations that don't follow the spec, but that is a perennial problem for any open-ish ecosystem.

Android also has a distinction between software-based sensors and hardware-based ones. The idea is that the same framework can provide both the low level data coming directly from the hardware, as well as useful higher level data obtained through sensor fusion. As of API level 19, Android also provides batch mode for sensor data, which is very useful for conserving battery and CPU for applications where some delay is acceptable.

One nice advantage of an iOS style API is that each sensor type has its own structure (rather than just an amorphous array of floats, as in Android), which is quite a bit easier to parse. The downside is that adding new sensor types introduces more overhead, since each one requires a new structure to be defined and agreed upon. Since we are talking about web standards, which evolve at a glacial pace, we should err on a simple API that works well without spec modifications.

Great artists steal

There is no need for the web to reinvent the wheel. The wheel has already been invented by iOS and Android. All we need to do is take the good parts from these successful sensor platforms, and integrate them into the web in a way that makes sense. The web is not the place for innovation, but for standardization.

Conceptually, a sensor provides a stream of data. The developer should be able to configure the rate at which new data comes in, as well as batching the data in windows of sensor data (as is customarily done with audio data, for example). In Android, because of a plurality of devices, it's important to be able to check if a particular sensor is available. The same concept maps well to the web.

Toward A Web Sensor API

In general, here are the requirements for a Web Sensor API that works:

• A specification defining the format of the data, similar to Android.
• A way to feature detect for the existence of a particular sensor.
• A way to request (and revoke) a stream of sensor data.
• A way to specify how often to poll the sensor.
• Bonus: A way to request sensor data in batch form.

While bringing an API like this to the web is a huge undertaking, there is a silver lining. The sensors we're talking about are all considered (at least for now) low-security, in the sense that on native platforms, there is no extra permission required to access them. This makes it possible to simply propose an API, convince everybody of it's worth, and then have it implemented across the web!

I don't have a strong opinion about how the API itself looks like as long as it fulfils the above requirements. Here's a simple strawman which should satisfy them:

// Check for magnetometer support.
if (sensors.Magnetometer === undefined) {
  console.error('No magnetometer found');
}

// Start listening for changes to the sensor.
var magnetometer = sensors.Magnetometer;
magnetometer.addEventListener('changed', onMagnetometer, {
  sample_rate: sensors.POLL_FAST, // In hertz, eg. POLL_FAST == 100
  batch: 1 // Number of data points to provide in a single poll.
});

// Handle sensor events.
function onMagnetometer(event) {
  var data = event.data[0];
  // Get the timestamp (in millis).
  var t = data.timestamp;
  // Get the data (in this case µT, as per spec).
  var x = data.values[0];
  var y = data.values[1];
  var z = data.values[2];
  // Process the data.
  superAdvancedSensorFusionThing.addData(t, x, y, z);
}

// Stop listening.
magnetometer.removeEventListener('changed', onMagnetometer);

Conclusion

If you aren't yet convinced that we need access to low level sensors on the web, recall web developers scoffing at device pixel ratio (DPR), really questioning the need for to ever go above 2x. Now that some screens are ending up 5cm from your face, the current generation of 4x displays isn't enough. The same exact thing applies to sensors. The need is there, but it is not seen as enough of a priority by the web community.

By enabling low level sensor access, we can allow new experiences never before possible on the web. Pushup rep counters, the magnet button in Cardboard, and myriads more applications yet to be concieved could all be built on the web platform, eliminating a big reason why the web is increasingly losing its relevance on mobile devices. Providing low level sensor access is critical and aligns perfectly with the extensible web vision.

Update (Nov 14, 2014): There was a W3C call about this very topic yesterday! Kicking off efforts in this github repo. Join us!