JavaScript bridge API

QR and barcode scanner

Google’s scanner screen from one promise, AppMint.scanCode() - no camera permission, and the parsed URL, Wi-Fi, phone or contact for common QR codes.

`AppMint.scanCode` helper#

AppMint.scanCode(options)

Example

Opens Google's QR code and barcode scanner screen and gives you what was scanned. One call - no camera permission, no <video>, no UI of your own.

Returns: a Promise that always resolves: {ok: true, value, displayValue, format, type, …} or {ok: false, code, message}. Needs: nothing to switch on. The phone needs Google Play services.

Any code:

async function scan() {
  if (!(window.AppMint && AppMint.scanCode)) { alert('Open this page in the app to scan'); return; }
  var r = await AppMint.scanCode();
  if (!r.ok) {
    if (r.code === 'cancelled') return;                          // the user closed the scanner
    if (r.code === 'unavailable') alert('The scanner is not available on this phone');
    else if (r.code === 'busy') alert('A scan is already open');
    else alert('Scan failed: ' + r.message);                     // 'failed'
    return;
  }
  console.log(r.value, r.format, r.type);   // "TICKET-4821", "qr_code", "text"
}

Only some formats, and a "type it in" option for damaged labels:

var r = await AppMint.scanCode({
  formats: ['ean_13', 'ean_8', 'upc_a', 'upc_e'],   // shop barcodes
  manualInput: true,                                // lets the user type the code
  autoZoom: true                                    // default true
});
if (r.ok) lookUpProduct(r.value);

Ready-made parts - only the part that matches r.type is present:

var r = await AppMint.scanCode({ formats: ['qr_code'] });
if (r.ok) {
  if (r.type === 'url')     openLink(r.url.url);                      // url: { url, title }
  if (r.type === 'wifi')    showWifi(r.wifi.ssid, r.wifi.password, r.wifi.encryption); // 'wpa'|'wep'|'open'
  if (r.type === 'phone')   callNumber(r.phone);                      // a string
  if (r.type === 'email')   writeMail(r.email.address, r.email.subject, r.email.body);
  if (r.type === 'sms')     writeSms(r.sms.phoneNumber, r.sms.message);
  if (r.type === 'geo')     showMap(r.geo.lat, r.geo.lng);
  if (r.type === 'contact') addContact(r.contact.name, r.contact.organization, r.contact.phones, r.contact.emails);
}

Notes: Format names: qr_code, aztec, codabar, code_39, code_93, code_128, data_matrix, ean_8, ean_13, itf, pdf417, upc_a, upc_e (unknown names are ignored; none = all formats). type is one of url, wifi, text, phone, email, sms, geo, contact, calendar, isbn, product, driver_license, unknown. Error codes: cancelled (closed, including with Back), unavailable (no Play services, or the scanner could not be downloaded), busy (a scan is already open), failed. The first scan on a phone can take a few seconds while Play services downloads the scanner. AI-built apps use scanCode() from @/lib/appmintNative, which also scans on the web.

`AppMint.textRecognition` helper#

AppMint.textRecognition.recognize(image, {script}) / prepare(script, onProgress)

Example

Reads text in scripts beyond Latin: Devanagari (Hindi, Marathi, Nepali …), Chinese, Japanese and Korean - or 'auto', which tries each script and keeps the reading with the most characters. The same option works on the standard detector: new TextDetector({ script: 'devanagari' }). Without the option everything reads Latin exactly as before.

Returns: recognize() → a Promise of {script, text, blocks: [{text, box, corners, lines}]} (script = the one used); it rejects with a DOMException - NotSupportedError for an unknown script or no Google Play services, OperationError otherwise. prepare() → a Promise of {script, status: 'ready'}. Needs: Google Play services. Each script's recognizer is a Play services download: scripts your code names are installed with the app, any other on first use.

Hindi text from a photo:

document.getElementById('photo').addEventListener('change', async function (e) {
  var file = e.target.files[0];
  if (!file || !(window.AppMint && AppMint.textRecognition)) return;
  try {
    var r = await AppMint.textRecognition.recognize(file, { script: 'devanagari' });
    document.getElementById('out').textContent = r.text;
  } catch (err) {
    alert('Could not read the photo: ' + err.name);
  }
});

Download a script's model first, with progress - the first use on a phone otherwise waits for the download:

async function getReady() {
  var bar = document.getElementById('model-progress');
  await AppMint.textRecognition.prepare('japanese', function (s) {
    if (s.progress != null) bar.value = s.progress;      // 0..1 while 'downloading'
  });
  bar.hidden = true;
}

// The same states arrive as events, also when a recognize() call starts a download:
window.addEventListener('appmint:ocr-model', function (e) {
  console.log(e.detail.script, e.detail.status, e.detail.progress);   // 'downloading' | 'ready' | 'failed'
});

Mixed documents - let the app pick among the scripts you expect:

var r = await AppMint.textRecognition.recognize(img, { script: 'auto', scripts: ['latin', 'chinese', 'korean'] });
console.log('read as ' + r.script + ': ' + r.text);

var lines = await new TextDetector({ script: 'korean' }).detect(img);   // standard shape, one entry per line

Notes: scripts is AppMint's list: latin, devanagari, chinese, japanese, korean. Arabic, Cyrillic, Thai and Hebrew have no on-device recognizer. WebToApk.__ocrRecognizeWith(requestId, base64, optionsJson) is the internal transport; use the helper.

`__ocrAvailable` bridge#

window.WebToApk.__ocrAvailable(): Boolean

TextDetector / recognizeText(): Google Play services is here (the OCR module's home).

Example

Internal transport behind TextDetector: whether Google Play services (where the recognizer comes from) is on this phone. Pages use the standard API; never call this directly - the __ methods may change without notice.

Returns: true or false. When false, TextDetector is not installed at all. Needs: nothing.

Use the public API:

if ('TextDetector' in window) scanButton.hidden = false;

Choose between the scan flow and a typed form once, at start-up. When the phone has no Google Play services the detector is simply absent, so the page must offer another way in:

function setUpReceiptEntry() {
  var canScan = typeof window.TextDetector === 'function';
  document.getElementById('scan-receipt').hidden = !canScan;
  document.getElementById('type-receipt').hidden = canScan;
  if (!canScan) document.getElementById('entry-hint').textContent =
    'Text scanning is not available on this phone. Type the total below.';
}
document.addEventListener('DOMContentLoaded', setUpReceiptEntry);

A detector that exists can still answer that recognition is unavailable at the moment of the call. Tell the two failures apart by the error name:

async function readLabel(img) {
  if (typeof window.TextDetector !== 'function') return null;
  try {
    var lines = await new TextDetector().detect(img);
    return lines.map(function (l) { return l.rawValue; }).join('\n');
  } catch (e) {
    if (e.name === 'NotSupportedError') showMessage('Text recognition is not available right now.');
    else showMessage('Could not read the label: ' + e.message);
    return null;
  }
}

Notes: Implemented in the app shell (TextRecognitionBridge.kt).

`__ocrRecognize` bridge#

window.WebToApk.__ocrRecognize(requestId: String, base64: String)

Recognises the text in [base64] (PNG/JPEG); the answer arrives as `appmint:ocr {requestId, blocks|error}`.

Example

Internal transport behind TextDetector.detect(): recognises the text in a PNG/JPEG. Pages use the standard API; never call this directly - the __ methods may change without notice.

Returns: nothing; the answer arrives as the appmint:ocr event {requestId, blocks:[{text, box, corners, lines}]} or {requestId, error} (unavailable, bad-image, failed: …). Needs: nothing.

Use the public API:

const lines = await new TextDetector().detect(document.getElementById('receipt'));
console.log(lines.map((l) => l.rawValue));

Read the text of a photo the user picks. detect() accepts a Blob/File as well as an <img>, <canvas> or ImageData:

<input type="file" id="doc" accept="image/*" capture="environment">
<pre id="out"></pre>
<script>
document.getElementById('doc').addEventListener('change', async function (e) {
  var file = e.target.files[0];
  if (!file || typeof window.TextDetector !== 'function') return;
  try {
    var lines = await new TextDetector().detect(file);
    document.getElementById('out').textContent = lines.map(function (l) { return l.rawValue; }).join('\n');
  } catch (err) {
    document.getElementById('out').textContent = 'Could not read the photo (' + err.name + ')';
  }
});
</script>

Outline each recognised line on top of the image. boundingBox is in the image's own pixels, so scale it to the displayed size. The image must have finished loading, or detect() rejects with InvalidStateError:

async function highlightLines(img, overlay) {
  if (!img.complete) await new Promise(function (r) { img.onload = r; });
  var lines = await new TextDetector().detect(img);
  var sx = img.clientWidth / img.naturalWidth, sy = img.clientHeight / img.naturalHeight;
  overlay.innerHTML = '';
  lines.forEach(function (l) {
    var b = l.boundingBox, box = document.createElement('div');
    box.className = 'ocr-box';
    box.title = l.rawValue;
    box.style.cssText = 'position:absolute;left:' + b.x * sx + 'px;top:' + b.y * sy +
      'px;width:' + b.width * sx + 'px;height:' + b.height * sy + 'px';
    overlay.appendChild(box);
  });
}

Pull an amount out of a receipt. A cross-origin image that taints the canvas rejects with SecurityError, so serve the picture from the app's own origin:

async function findTotal(img) {
  try {
    var lines = await new TextDetector().detect(img);
    var hit = lines.find(function (l) { return /total/i.test(l.rawValue); });
    var m = hit && hit.rawValue.match(/(\d+[.,]\d{2})/);
    return m ? parseFloat(m[1].replace(',', '.')) : null;
  } catch (e) {
    if (e.name === 'SecurityError') console.warn('Image is from another origin; load it from the app instead.');
    else console.warn('OCR failed:', e.name, e.message);
    return null;
  }
}

Notes: Implemented in the app shell (TextRecognitionBridge.kt).

`__ocrRecognizeWith` bridge#

window.WebToApk.__ocrRecognizeWith(requestId: String, base64: String, optionsJson: String)

[__ocrRecognize] with options [optionsJson] `{script: 'latin'|'devanagari'|'chinese'|'japanese'|'korean'|'auto', scripts?}`; the `appmint:ocr` answer also names the `script` used.

Example

Documented together with AppMint.textRecognition - the example there shows this one too.

Hindi text from a photo:

document.getElementById('photo').addEventListener('change', async function (e) {
  var file = e.target.files[0];
  if (!file || !(window.AppMint && AppMint.textRecognition)) return;
  try {
    var r = await AppMint.textRecognition.recognize(file, { script: 'devanagari' });
    document.getElementById('out').textContent = r.text;
  } catch (err) {
    alert('Could not read the photo: ' + err.name);
  }
});

Download a script's model first, with progress - the first use on a phone otherwise waits for the download:

async function getReady() {
  var bar = document.getElementById('model-progress');
  await AppMint.textRecognition.prepare('japanese', function (s) {
    if (s.progress != null) bar.value = s.progress;      // 0..1 while 'downloading'
  });
  bar.hidden = true;
}

// The same states arrive as events, also when a recognize() call starts a download:
window.addEventListener('appmint:ocr-model', function (e) {
  console.log(e.detail.script, e.detail.status, e.detail.progress);   // 'downloading' | 'ready' | 'failed'
});

Mixed documents - let the app pick among the scripts you expect:

var r = await AppMint.textRecognition.recognize(img, { script: 'auto', scripts: ['latin', 'chinese', 'korean'] });
console.log('read as ' + r.script + ': ' + r.text);

var lines = await new TextDetector({ script: 'korean' }).detect(img);   // standard shape, one entry per line

`prepareTextRecognition` bridge#

window.WebToApk.prepareTextRecognition(script: String): String

Downloads the text recognizer for [script] ahead of use. Returns "checking" (then `appmint:ocr-model {script, status: 'downloading'|'ready'|'failed', progress?}`), "unsupported-script" or "unavailable" (no Google Play services).

Example

Downloads the on-device text recognizer for one script ahead of use, so the first recognize() does not wait. Pages normally call AppMint.textRecognition.prepare(script), which wraps this.

Returns: a string at once - "checking" (the outcome follows as appmint:ocr-model events {script, status: 'downloading' | 'ready' | 'failed', progress?, error?}), "unsupported-script" or "unavailable" (no Google Play services). 'auto' prepares every script. Needs: Google Play services.

if (window.WebToApk && typeof window.WebToApk.prepareTextRecognition === 'function') {
  window.addEventListener('appmint:ocr-model', function (e) {
    if (e.detail.script !== 'chinese') return;
    if (e.detail.status === 'downloading') console.log('downloading', Math.round((e.detail.progress || 0) * 100) + '%');
    if (e.detail.status === 'ready') document.getElementById('scan').disabled = false;
    if (e.detail.status === 'failed') console.warn(e.detail.error);
  });
  var state = window.WebToApk.prepareTextRecognition('chinese');
  if (state !== 'checking') console.warn('Chinese text recognition: ' + state);
}

Prepare, then scan through the wrapper:

async function readSign(canvas) {
  if (!(window.AppMint && AppMint.textRecognition)) return null;
  try {
    await AppMint.textRecognition.prepare('japanese');
    return await AppMint.textRecognition.recognize(canvas, { script: 'japanese' });
  } catch (err) {
    console.warn('Text recognition:', err.name, err.message);   // e.g. NotSupportedError without Play services
    return null;
  }
}

Unsupported scripts and missing Play services are reported in the return value:

const W = window.WebToApk;
if (W && typeof W.prepareTextRecognition === 'function') {
  const state = W.prepareTextRecognition('arabic');
  if (state === 'unsupported-script') showMessage('Arabic text cannot be read on the phone.');
  else if (state === 'unavailable') showMessage('Text recognition needs Google Play services.');
}

Notes: Scripts: latin, devanagari, chinese, japanese, korean, auto. A script already on the phone answers ready straight away.

`scanCode` bridge#

window.WebToApk.scanCode(optionsJson: String, callbackId: String)

Opens Google's code scanner; [optionsJson] `{formats?, autoZoom?, manualInput?}`. Resolves the AppMint.scanCode promise [callbackId].

Example

Internal plumbing behind AppMint.scanCode() - pages use that Promise helper, which opens Google's QR/barcode scanner and resolves with the result.

Returns: nothing directly; the result resolves the AppMint.scanCode() promise. Needs: nothing to switch on (no camera permission). The phone needs Google Play services.

The public API:

async function scanTicket() {
  if (!(window.AppMint && AppMint.scanCode)) { alert('Open this page in the app to scan'); return; }
  var r = await AppMint.scanCode({ formats: ['qr_code'] });
  if (r.ok) checkTicket(r.value);                  // r.format, r.type, r.displayValue…
  else if (r.code !== 'cancelled') alert('Scan failed (' + r.code + ')');
}

Join a Wi-Fi network from a printed QR code. A Wi-Fi code comes back with type: 'wifi' and a wifi object:

async function scanWifi() {
  if (!(window.AppMint && AppMint.scanCode)) return;
  var r = await AppMint.scanCode({ formats: ['qr_code'] });
  if (!r.ok) { if (r.code !== 'cancelled') alert('Scan failed (' + r.code + ')'); return; }
  if (r.type !== 'wifi') { alert('That is not a Wi-Fi code.'); return; }
  showWifiCard(r.wifi.ssid, r.wifi.password, r.wifi.encryption);   // encryption: 'wpa' | 'wep' | 'open'
}

Look up a product by its shop barcode, letting the user type the digits when the label is damaged (manualInput). The first scan on a phone may download the scanner, which fails with unavailable when offline:

async function scanProduct() {
  if (!(window.AppMint && AppMint.scanCode)) { showManualEntry(); return; }
  var r = await AppMint.scanCode({ formats: ['ean_13', 'ean_8', 'upc_a', 'upc_e'], manualInput: true });
  if (r.ok) return lookUpProduct(r.value, r.format);            // e.g. '4006381333931', 'ean_13'
  if (r.code === 'busy') return;                                  // a scan is already open
  if (r.code === 'unavailable') alert('The scanner is not available: ' + r.message);
  else if (r.code !== 'cancelled') alert('Scan failed (' + r.code + ')');
}

Open whatever a QR code points at, by its decoded type:

async function scanAndOpen() {
  var r = await AppMint.scanCode({ formats: ['qr_code'], autoZoom: true });
  if (!r.ok) return;
  if (r.type === 'url') location.href = r.url.url;
  else if (r.type === 'phone') location.href = 'tel:' + r.phone;
  else if (r.type === 'email') location.href = 'mailto:' + r.email.address;
  else if (r.type === 'geo') showOnMap(r.geo.lat, r.geo.lng);
  else if (r.type === 'contact') saveContact(r.contact.name, r.contact.phones, r.contact.emails);
  else showText(r.displayValue);
}

Notes: The raw method takes optionsJson ({formats?, autoZoom?, manualInput?} as a JSON string) and a callbackId; the answer is delivered to the helper's internal reply table, so a page calling it directly gets no answer. Only one scan runs at a time - a second call answers code: 'busy'. See AppMint.scanCode for every result field and error code.

AppMint.scanCode helper#

AppMint.textRecognition helper#

__ocrAvailable bridge#

__ocrRecognize bridge#

__ocrRecognizeWith bridge#

prepareTextRecognition bridge#

scanCode bridge#

`AppMint.scanCode` helper#

`AppMint.textRecognition` helper#

`__ocrAvailable` bridge#

`__ocrRecognize` bridge#

`__ocrRecognizeWith` bridge#

`prepareTextRecognition` bridge#

`scanCode` bridge#