Redirecting to the About Me page…

Translating Kibana with the Globalization Pipeline

Introduction

kibana0.png

This post (and video) will explain how to translate Kibana using the Globalization Pipeline service on Bluemix. Note that some of the steps shown here depend on kibana:8766 which was not merged as this article went to press. (Portions are based on the development-internationalization.asciidoc document from that PR.)

Prerequisites

Setting up Globalization Pipeline

  • Follow the GP Quick Start Guide to create a service instance. Copy down the "credentials" into a new file, gp-credentials.json which should look something like the following:
1
2
3
4
5
6
{
"url": "https://gp-rest.bluemix.example.com/translate/rest",
"userId": "c2lnbiB1cCBAIGJsdWVtaXgubmV0ISEK",
"password": "aHVudGVyNDIK",
"instanceId": "aHR0cHM6Ly9ibHVlbWl4Lm5ldCA8PDwK"
}
  • Create the bundle on the GP instance. The example below uses English (en) as the source langage and requests Spanish, Japanese, and French targets (es,ja,fr).
1
2
$ java -jar {wherever}/gp-cli.jar create-bundle -j {wherever}/gp-credentials.json -b 'kibana_core' -l en,es,ja,fr
A new bundle 'kibana_core' was successfully created.
  • The bundle will show up in the Bluemix dashboard under the service’s console, but as empty.

  • We are going to translate the src/core_plugins/kibana/translations/en.json file in Kibana. Upload that file to the Globalization Pipeline service using the command line:

1
2
3
$ cd ~/src/kibana
$ java -jar {wherever}/gp-cli.jar import -j {wherever}/gp-credentials.json -b 'kibana_core' -l en -f src/core_plugins/kibana/translations/en.json -t json
Resource data extracted from src/core_plugins/kibana/translations/en.json was successfully imported to bundle:kibana_core, language:en
  • If you head back over to the Bluemix dashboard, you can now see the populated bundle with translated content:
gp-dash.png

What you see was done with machine translation, hence the red “U” (Unreviewed). The content here can be corrected manually by clicking the Pencil icon, or marked as manually reviewed by clicking the Checkmark. It’s also possible to download the translated content for offline review or use, or to upload a corrected version of one of the translations.

Head back over to the command line, though, because it is time to create our plugin.

Creating the plugin

Something like this:

1
2
3
4
5
6
7
8
9
10
$ npm install -g yo generator-kibana-plugin
Everything looks all right!
$ yo kibana-plugin
? Your Plugin Name gp srl kibana plugin
? Short Description An awesome Kibana translation plugin
? Target Kibana Version master
I'm all done. Running npm install for you to install the required dependencies. If this fails, try running the command yourself.
  • You will notice that the generator has created a translations/es.json file. We will replace this with our translated content.
1
$ rm translations/es.json
  • Now, download the translated content into the correct files:
1
2
3
4
5
6
7
8
$ java -jar {wherever}/gp-cli.jar export -j {wherever}/gp-credentials.json -b 'kibana_core' -t json -l es -f translations/es.json
Resource data exported from bundle:kibana_core, language: es was successfully saved to file translations/es.json
$ java -jar {wherever}/gp-cli.jar export -j {wherever}/gp-credentials.json -b 'kibana_core' -t json -l fr -f translations/fr.json
Resource data exported from bundle:kibana_core, language: fr was successfully saved to file translations/fr.json
$ java -jar {wherever}/gp-cli.jar export -j {wherever}/gp-credentials.json -b 'kibana_core' -t json -l ja -f translations/ja.json
Resource data exported from bundle:kibana_core, language: ja was successfully saved to file translations/ja.json
  • Update the index.js file in the plugin to mention the updated translations files.

You will see a section like this:

1
2
3
translations: [
resolve(__dirname, './translations/es.json')
],

Change it to mention all of the language files we have just downloaded:

1
2
3
4
5
translations: [
resolve(__dirname, './translations/es.json'),
resolve(__dirname, './translations/ja.json'),
resolve(__dirname, './translations/fr.json')
],
  • That's all the coding we'll need for today…

  • Copy your entire translations plugin directory to the Kibana plugins (<kibana_root>/plugins/) directory

Trying it out

Fire up Kibana and you should see the translated content!

kibana1.png

More steps

  • By the way, French isn’t included in the video or images becuase I ran into kibana#10580 during the production of this video. When this is fixed I will come back and edit this video, but until then, beware single quotes (') in your translated strings.

  • Note that if you repeat the import and export steps of the gp-cli tool, the Globalization Pipeline will automatically manage translation changes if, for example, translated keys are added or removed, or translated content changes.

  • Follow the progress of Kibana Globalization on Github: (kibana#6515).

  • Read more about Globalization Pipeline

  • Connect with the Globalization Pipeline Open Source Community

Acknowledgements

  • Thanks to fellow IBMers Martin Hickey, Shikha Srivastava, and Jonathan Lo for the Kibana G11n work (kibana#6515), also the elastic/kibana team for being a great OSS community, and last but not least the entire Globalization Pipeline team.

Literate Programmers

Besides the Globalization Pipeline mug, one of my favorite coffee mugs says:

03: MAKE IT POSSIBLE FOR PROGRAMMERS TO WRITE IN ENGLISH AND YOU WILL FIND OUT THAT PROGRAMMERS CANNOT WRITE IN ENGLISH.

On the serious side, we need to emphasize communication skills in the technology industry. Even if I have a great idea, if I can’t communicate it, it will go nowhere. And neither will I.

Just to be clear, by “communication” I mean “talking with other humans”. Which brings me to today’s topic on the lighter side, and that is the overloading of English. Words such as function, overload, network, build all have specific meanings that weren’t originally found in Webster’s. The 1828 definition of computer, for example, is:

One who computes; a reckoner; a calculator.

In i18n, there are other words that have very specific meanings: global, globalization, collation, contraction, and of course locale, just to name a few.

To that end, I have started to add some tongue-in-cheek “redefinitions” to the bottom of the blog just to remind us all that these words have non-software meanings.

If you want to see them all without hitting reload an infinite number of times, you can see the original source here.

Speaking of i18n, this overloading doesn’t apply to English only. Most of my devices are set to es-US as their locale, so I see a lot of translated error message. gcc for example has a thriving translation project where dedicated persons cause “English” to be translated into, for example, “Spanish” such as:

#~ msgid "function ‘%D’ declared overloaded, but no definitions appear with which to resolve it?!?"

#~ msgstr "¿!¿se declaró la función ‘%D’ sobrecargada, pero no aparece ninguna definición con la cual resolverlo?!?"

Not sure why that’s ¿!¿ where I might expect ¿¡¿ — perhaps the initial ! just shows the compiler’s incredulity. In any event, sobrecargada seems to be a great cognate for overloaded. And with that, I will let you goto whatever you were doing before you started reading.

PR’s are welcome on my little list, or leave comments below. What are your favorite examples of overloaded terms, in any language?

Fallbacks in ICU4C Converters

Unicode’s ICU version 59 is well underway at this point. While ideally everything would use Unicode, there still remains many systems — and much content — that is in non-Unicode encodings. For this reason, ICU, in both the C/C++ and the Java flavors, has rich support for codepage conversion.

One of many great features in ICU is the callback support. A lot can go wrong during codepage conversion, but in ICU, you can control what happens during exceptional situations.

Let’s try a simple sample. By the way, see the end of this post for hints on compiling the samples.

Substitute, Always

Our task is to convert black-bird (but with a U+00AD, “Soft Hyphen” in between the two words) to ASCII.

substituteTest-0.cppview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
#include <unicode/utypes.h>
#include <unicode/ustdio.h>
#include <unicode/ucnv.h>
int main(int /*argc*/, const char * /*argv*/ []) {
UErrorCode status=U_ZERO_ERROR;
LocalUConverterPointer cnv(ucnv_open("us-ascii", &status));
if(U_FAILURE(status)) {
u_printf("Error opening: %s\n", u_errorName(status));
return 1;
}
UnicodeString str("black-bird");
str.setCharAt(5, 0x00AD); // soft hyphen
const UChar *uch = str.getTerminatedBuffer();
u_printf("Input String: %S length %d\n", uch, str.length());
char bytes[1024];
int32_t bytesWritten =
ucnv_fromUChars(cnv.getAlias(), bytes, 1024, uch, -1, &status);
if(U_FAILURE(status)) {
u_printf("Error converting: %s\n", u_errorName(status));
return 1;
}
u_printf("Converted %d bytes\n", bytesWritten);
for(int32_t i=0; i<bytesWritten; i++) {
u_printf("\\x%02X ", bytes[i]&0xFF);
}
u_printf("\n");
// try to print it out on the console
bytes[bytesWritten]=0; // terminate it first
puts(bytes);
return 0; // LocalUConverterPointer will cleanup cnv
}

Output:

1
2
3
4
Input String: black­bird length 10
Converted 9 bytes
\x62 \x6C \x61 \x63 \x6B \x62 \x69 \x72 \x64
blackbird

Hm. Ten characters in, nine out. What happened? Well, U+00AD is not a part of ASCII. ASCII is a seven bit encoding, thus only maps code points \x00 through \x7F inclusively. Furthermore, U+00AD is Default Ignorable, and as of ICU 54.1 (2014) in #10551, the soft hyphen can just be dropped.

But what if, for some reason, you don’t want the soft hyphen dropped? The pre ICU 54.1 behavior can be brought back easily with a custom call back. So, roll up your collective sleeves, and:

alwaysSubstitute.hview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
// © 2016 and later: Unicode, Inc. and others.
// License & terms of use: http://www.unicode.org/copyright.html
#include <unicode/ucnv.h>
#include <unicode/ucnv_err.h>
#include <unicode/ucnv_cb.h>
/**
* This is a modified version of ICU’s UCNV_FROM_U_CALLBACK_SUBSTITUTE
* it unconditionally substitutes on irregular codepoints.
*
* Usage:
* ucnv_setFromUCallBack(c, UCNV_FROM_U_CALLBACK_SUBSTITUTE_ALWAYS, NULL, NULL, NULL, &status);
*/
U_CAPI void U_EXPORT2
UCNV_FROM_U_CALLBACK_SUBSTITUTE_ALWAYS (
const void *context,
UConverterFromUnicodeArgs *fromArgs,
const UChar* codeUnits,
int32_t length,
UChar32 codePoint,
UConverterCallbackReason reason,
UErrorCode * err)
{
(void)codeUnits;
(void)length;
if (reason <= UCNV_IRREGULAR) {
*err = U_ZERO_ERROR;
ucnv_cbFromUWriteSub(fromArgs, 0, err);
/* else the caller must have set the error code accordingly. */
}
/* else ignore the reset, close and clone calls. */
}

If we #include this little header, and set it on the converter before we convert…

1
2
3
LocalUConverterPointer cnv(ucnv_open("us-ascii", &status));
ucnv_setFromUCallBack(cnv.getAlias(), UCNV_FROM_U_CALLBACK_SUBSTITUTE_ALWAYS, NULL, NULL, NULL, &status);

… we get the following result:

1
2
3
4
Input String: black­bird length 10
Converted 10 bytes
\x62 \x6C \x61 \x63 \x6B \x1A \x62 \x69 \x72 \x64
black?bird

Great! Now, we are getting \x1A (ASCII SUB). It works.

When missing goes missing

A related question to the above has to do with converting from codepage to Unicode. That’s a better direction anyway. Convert to Unicode and stay there! One can hope. In any event…

For this task, we will convert 0x61, 0x80, 0x94, 0x4c, 0xea, 0xe5 from Shift-JIS to Unicode.

substituteTest-2.cppview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include <unicode/utypes.h>
#include <unicode/ustdio.h>
#include <unicode/ucnv.h>
int main(int /*argc*/, const char * /*argv*/ []) {
UErrorCode status=U_ZERO_ERROR;
LocalUConverterPointer cnv(ucnv_open("shift-jis", &status));
if(U_FAILURE(status)) {
u_printf("Error opening: %s\n", u_errorName(status));
return 1;
}
#define NRBYTES 6
const uint8_t bytes[NRBYTES] = { 0x61, 0x80, 0x94, 0x4c, 0xea, 0xe5 };
u_printf("Input Bytes: length %d\n", NRBYTES);
#define NRUCHARS 50
UChar uchars[NRUCHARS];
int32_t ucharsRead =
ucnv_toUChars(cnv.getAlias(), uchars, NRUCHARS, (const char*)bytes, NRBYTES, &status);
if(U_FAILURE(status)) {
u_printf("Error converting: %s\n", u_errorName(status));
return 1;
}
u_printf("Converted %d uchars\n", ucharsRead);
for(int32_t i=0; i<ucharsRead; i++) {
u_printf("U+%04X ", uchars[i]);
}
u_printf("\n");
// try to print it out on the console
u_printf("Or string: '%S'\n", uchars);
return 0; // LocalUConverterPointer will cleanup cnv
}

Output:

1
2
3
4
Input Bytes: length 6
Converted 4 uchars
U+0061 U+001A U+732B U+FFFD
Or string: 'a猫�'

So, the letter "a" byte \x61 turned into U+0061, and then we have an illegal byte \x80 which turned into U+001A. Next, the valid sequence \x94 \x4c turns into U+732B which is 猫 (“cat”). Finally, the unmapped sequence \xea \xe5 turns into U+FFFD. Notice that the single byte illegal sequence turned into (SUB, U+001A), but the two byte sequence turned into U+FFFD. This is discussed somewhat here.

So far so good?

But what if you actually want U+FFFD as the substitution character for both sequences? This would be unexpected, but perhaps you have code that is particularly looking for U+FFFDs. We can write a similar callback:

alwaysFFFD.hview raw
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
// © 2016 and later: Unicode, Inc. and others.
// License & terms of use: http://www.unicode.org/copyright.html
#include <unicode/ucnv.h>
#include <unicode/ucnv_err.h>
#include <unicode/ucnv_cb.h>
static const UChar kFFFD[] = { 0xFFFD };
/**
* This is a modified version of ICU’s UCNV_TO_U_CALLBACK_SUBSTITUTE
* it unconditionally substitutes U+FFFD.
*
* Usage:
* ucnv_setToUCallBack(c, UCNV_TO_U_CALLBACK_SUBSTITUTE_FFFD, NULL, NULL, NULL, &status);
*/
U_CAPI void U_EXPORT2
UCNV_TO_U_CALLBACK_SUBSTITUTE_FFFD (
const void *context,
UConverterToUnicodeArgs *toArgs,
const char* codeUnits,
int32_t length,
UConverterCallbackReason reason,
UErrorCode * err)
{
(void)codeUnits;
(void)length;
if (reason <= UCNV_IRREGULAR)
{
*err = U_ZERO_ERROR;
ucnv_cbToUWriteUChars(toArgs, kFFFD, 1, NULL, err);
// see ucnv_cbToUWriteSub()
}
/* else ignore the reset, close and clone calls. */
}

Let’s hook it up, as before:

1
2
3
LocalUConverterPointer cnv(ucnv_open("shift-jis", &status));
ucnv_setToUCallBack(cnv.getAlias(), UCNV_TO_U_CALLBACK_SUBSTITUTE_FFFD, NULL, NULL, NULL, &status);

And drumroll please…

1
2
3
4
Input Bytes: length 6
Converted 4 uchars
U+0061 U+FFFD U+732B U+FFFD
Or string: 'a�猫�'

Garbage out never looked so good…


Building (or, nothing-up-my-sleeve)

To build these little snippets, I recommend the shell script icurun

If ICU is already installed in your appropriate paths, (visible to pkg-config or at least icu-config), you can simply run:

1
icurun some-great-app.cpp

… and icurun will compile and run a one-off.

If, however, you’ve built ICU yourself in some directory, you can instead use:

1
icurun -i path/to/your/icu some-great-app.cpp

… where path/to/you/icu is the full path to an ICU build or install directory.

If you are on windows… well, there isn’t a powershell version yet. Contributions welcome!

Globalization Pipeline for iOS

Yesterday we just tagged v1.0 of the Globalization Pipeline SDK for iOS. What can an iOS client do? Well, let’s build a simple app and find out.

Starting Out

First, I’ll launch XCode 8 and create a new workspace.

While that is launching, I’ll warn you that your author is only a recent graduate of the Swift playground, who once deployed some toy apps to a then-new iPhone 3GS. So, it’s been a while. Any suggestions for improvement are welcome. The actual SDK, however, was a team effort.

Today’s app will be a color mixer, to help artists mix their colors. You know, red and blue makes purple, and so on.

00_title.png

I will name the workspace gp-ios-color-mixer, and create a new single view app called GP Color Mixer. To simplify things, for now, I disable the checkbox “automatically manage signing.”

01_singleview.png

I want to include the new SDK. I’ll use Carthage to install it. Since I already have Homebrew installed, I only need to do

$ brew install carthage

Now I need a Cartfile that mentions the SDK. So I create one at the same level as my XCode project, containing:

github "IBM-Bluemix/gp-ios-client"

Following the Carthage instructions, I next run

$ carthage update

which results in

*** Fetching gp-ios-client
*** Checking out gp-ios-client at "v1.0"
*** xcodebuild output can be found in /var/folders/j9/yn_32djn36x4d4c2mvcr1kgm0000gn/T/carthage-xcodebuild.p2nKN2.log
*** Building scheme "GPSDK" in TestFramework.xcworkspace

So far so good. Looking in the Finder, I now have GPSDK.framework right where I expect.

02_framework.png

I’ll add it under “Linked frameworks and Libraries”.

03_linked.png

We also need to make sure the framework is available at runtime. To do that, we add a build phase with a one-line script: /usr/local/bin/carthage copy-frameworks with a single input file - $(SRCROOT)/Carthage/Build/iOS/GPSDK.framework

04_buildphase.png

Will it build? I add this to the top of my generated ViewController.swift:

1
import GPSDK

I mentioned turning off code signing, but I still ran into some odd warnings:

1
2
3
4
5
A shell task (/usr/bin/xcrun codesign --force --sign - --preserve-metadata=identifier,entitlements "/Users/srl/Library/Developer/Xcode/DerivedData/gp-ios-color-mixer-evyxcmilwuakdmdvxqqpmmnzisnn/Build/Products/Debug-iphonesimulator/GP Color Mixer.app/Frameworks/GPSDK.framework") failed with exit code 1:
/Users/srl/Library/Developer/Xcode/DerivedData/gp-ios-color-mixer-evyxcmilwuakdmdvxqqpmmnzisnn/Build/Products/Debug-iphonesimulator/GP Color Mixer.app/Frameworks/GPSDK.framework: replacing existing signature
/Users/srl/Library/Developer/Xcode/DerivedData/gp-ios-color-mixer-evyxcmilwuakdmdvxqqpmmnzisnn/Build/Products/Debug-iphonesimulator/GP Color Mixer.app/Frameworks/GPSDK.framework: resource fork, Finder information, or similar detritus not allowed
Command /bin/sh failed with exit code 1

Following QA1940 I was able to make some progress by running xattr -cr './Carthage/Build/iOS/GPSDK.framework'. Now, ⌘R Run rewards me with a blank app window and no errors. Let’s write some code!

Applying myself to the App

By code, of course, I mean a trip to the storyboard. Let's add a launch icon, because we can.

Now, I add some static fields, two picker views (for the input colors), and a button for action.

Starting to look like an app…

I wrote Color.swift to handle the color mixing. It will only support mixing from three of the primary colors - Red, Yellow, Blue. Any other mixing turns into muddy brown. Playground tested, ready to go.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
enum Color : Int {
case red = 0, orange, yellow, green, blue, purple, muddy;
// r+y = o, y+b = g, b+r = p
func simpleDescription() -> String {
switch self {
case .red: return "red"
case .orange: return "orange"
case .yellow: return "yellow"
case .green: return "green"
case .blue: return "blue"
case .purple: return "purple"
case .muddy: return "muddy brown" // use this if we don't know how to mix a color
// should be exhaustive
}
}
/**
* Mix the colors, return the result
*/
func mix( with: Color ) -> Color {
if( self == .muddy || with == .muddy ) {
return .muddy // anything + mud = mud
}
if( with == self ) {
return self // identity!
}
switch self {
case .red:
switch with {
case .yellow: return .orange
case .blue: return .purple
default: return .muddy
}
case .yellow:
switch with {
case .red: return .orange
case .blue: return .green
default: return .muddy
}
case .blue:
switch with {
case .red: return .purple
case .yellow: return .green
default: return .muddy
}
default: return .muddy
}
}
}

Time to wire it up. We create IBOutlets for each of the items. And, I’ll clear the result label just to verify that things are wired up. It runs OK, good.

Wired for sound

Now, let’s set up the delegate stuff so that we can get the list of colors showing.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
class ViewController: UIViewController, UIPickerViewDelegate, UIPickerViewDataSource {
// …
// pickerview stuff
func numberOfComponents(in pickerView: UIPickerView) -> Int {
return 1;
}
func pickerView(_ pickerView: UIPickerView, numberOfRowsInComponent component: Int) -> Int {
return 3;
}
let primaryColors = [ Color.red, Color.blue, Color.yellow ]
func pickerView(_ pickerView: UIPickerView, titleForRow row: Int, forComponent component: Int) -> String? {
return primaryColors[row].simpleDescription()
}

Hey, just a little more code and we’re feature complete!

1
2
3
4
5
6
7
8
@IBAction func doMix(_ sender: Any) {
let color1 = primaryColors[mixOne.selectedRow(inComponent: 0)]
let color2 = primaryColors[mixTwo.selectedRow(inComponent: 0)]
let newColor = color1.mix(with: color2)
resultLabel.text = newColor.simpleDescription()
}
Works in English… Ship it (just kidding)

At least, feature complete in English.

I’ll next take stock of the resource strings we need to have translated, so that we can run them through the Globalization Pipeline. I’ll call this gp-color-mixer.json

1
2
3
4
5
6
7
8
9
10
11
{
"red": "red",
"orange": "orange",
"yellow": "yellow",
"green": "green",
"blue": "blue",
"purple": "purple",
"muddy brown": "muddy brown",
"title": "Color Mixer",
"mix": "Mix"
}

Mixing the Blue

Time to fire up Bluemix. We are going to basically follow the Globalization Pipeline Quick Start Guide for the this part, which I will refer to.

10_service.png

First, I create an instance of the Globalization Pipeline. The name you give the instance doesn’t matter here.

11_instance.png

Now I create a bundle named gp-color-mixer. This name does matter, as our iOS app will use it to access the content.

12_bundle.png

I’ll Upload the gp-color-mixer.json file above as the source English content, choosing JSON format for the upload. I pick a few languages for the target.

If I view the bundle, I can see our strings there, as well as translated versions.

The Globalization Pipeline offers this web UI to manage content, as well as powerful REST APIs for managing the translation workflow. I need to grant access to the iOS app so that it can read but not modify the translations. So, switching over to the API Users tab…

14_api.png

The result of creating the API user is that some access information is shown, something like the following:

API User ID: 5726d656c6f6e7761746572
Password: aHVudGVyNDIK
Instance ID: 77617465726d656c6f6e77617465726d
URL: https://something.something.bluemix.net/something/something

I take these and plug them into a new swift file named ReaderCredentials.swift like so: (this is a variant of ReaderCredentials-SAMPLE.swift in the SDK’s repo)

1
2
3
4
5
6
7
struct ReaderCredentials {
static let userId = "5726d656c6f6e7761746572";
static let password = "aHVudGVyNDIK";
static let instanceId = "77617465726d656c6f6e77617465726d";
static let url = "https://something.something.bluemix.net/something/something";
static let bundleId = "gp-color-mixer";
}

(Now, after putting my actual credentials in, and a brief offscreen struggle with .gitignore, I move on…)

Putting it all together

I’m almost done.

First, in the ViewController.swift, we initialize the GP service and start setting up a few UI items:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
let gp = GPService()
func get(key: String) -> String {
return gp.localizedString(key, nil)
}
func get(color: Color) -> String {
return get(key: color.simpleDescription())
}
override func viewDidLoad() {
super.viewDidLoad()
resultLabel.text = "Loading…"
do {
try gp.initService(url: ReaderCredentials.url,
instanceId: ReaderCredentials.instanceId,
bundleId: ReaderCredentials.bundleId,
userId: ReaderCredentials.userId,
password: ReaderCredentials.password,
languageId:nil,
alwaysLoadFromServer: false,
expireAfter: 0)
// set up strings
titleLabel.text = get(key: "title")
mixButton.setTitle(get(key: "mix"), for: UIControlState.normal)
mixButton.titleLabel?.text = get(key: "mix")
resultLabel.text = "" // clear this
} catch GPService.GPError.languageNotSupported {
resultLabel.text = ("This language is not supported...")
} catch GPService.GPError.requestServerError(let errorDescription) {
resultLabel.text = ("Request server error: " + errorDescription)
} catch GPService.GPError.HTTPError(let statusCode) {
resultLabel.text = ("Request server error: HTTP \(statusCode)")
} catch {
resultLabel.text = ("Other error")
}
}

Here we set up the service with our credentials. Then, we use our new get(key: ) function to set the title and mix button’s label.

There is also a get(color: ) variant that will translate one of our Color objects. So we use that for the actual mixing function:

1
2
3
@IBAction func doMix(_ sender: Any) {
resultLabel.text = get(color: newColor)

Similarly, we can get the UIPickerView to use localized color names by using this same function:

1
2
3
func pickerView(_ pickerView: UIPickerView, titleForRow row: Int, forComponent component: Int) -> String? {
return get(color: primaryColors[row])
}

Looks good!

Now we can ship it… to the world!

Conclusion

The iOS app will pick up changes if the translated content changes on the server. We could experiment with adding or removing languages, or updating translated keys.

You can find the source code at https://github.com/srl295/gp-ios-color-mixer.

Let me know if this works for you. This is my first post, and as I mentioned first app, in Swift so that’s a milestone. And, do let me know if^H^H what can be done to improve the sample app.

Thanks! Now go and make it global.

GP Client for JavaScript v1.3.0 released

It’s time for a refresh on the Globalization Pipeline Node.js client. I’ve just released v1.3.0 of this SDK. You can update your package.json the usual way, with npm install --save g11n-pipeline

npm version

I managed to close about 13 issues since v1.2.x

Quality

I was able to increase function coverage to 100% thanks to the VSCode coverage plugin, and increase line coverage to 91%. Of course, when you test, you find bugs. Bugs such as realizing that updateResourceStrings() was unusable because there was no way to pass the languageId parameter.

Features

First of all, I synchronized the client with the latest current REST API. So take a peek at the docs again and see if there are any new features or fields.

I also tried to add some convenience functions. For example, getting the full list of language IDs supported used to require concatenating the source and target lists. Now, with #40 you can call .languages() on the Bundle object and it will build this list for you. There is also a bundle.entries() accessor as of #14 which returns ResourceEntry objects.

Speaking of convenience, most places where you used to call .someFunction({}, function callback(…){}); the {} are optional. If it worked with {} before, it's now optional.

The sample PR where I updated the sample code shows some of the code improvements.

There are more features to add here, but I hope you like the changes in v1.3.0!

40th Internationalization and Unicode Conference

I'll start, and could almost end, my post with this tweet:

Yes, that exactly. November 1-3, 2016 was the 40th Unicode conference. They used to be twice a year, in multiple locations, as in, outside of Santa Clara, California, USA.

Now that the conference is over, I’ll have to take some time to view slides from all of the other great presentations I missed while giving a personal record number of talks (long story), apart from the lightning talks which were apparently not recorded.

The conference, and Unicode in general, is about people. It is always great to see so many folks I've kept up with over the years… including of course my fellow IBMers from many time zones away.

Off the top of my head, the important technical (besides personal) conversations I've had include:

Next week : IBM is hosting UTC 149 !

gp-angular-client v1.2.0

I just pushed out version 1.2.0 of our Angular Client for Globalization Pipeline to the usual places. gp-angular-client on bower, angular-g11n-pipeline on npm.

Bower version
npm version

Thanks to IBMer @ckoberlein (GitHub) this SDK now supports variable substitution. So you can have a string such as Hello and translate and substitute this same string, so that for example in Spanish it will be Bienvenidos . So, output would be Hello Steven or Bienvenidos Steven depending on language.

More details on our README and be sure to connect with us over on developerWorks Open!

Translating ICU4C with Globalization Pipeline

Disclaimer

This is a work in progress. If you read to the end, you’ll see we almost reached our goal here.

Background

I work on ICU4C (the premier C/C++ library for Unicode support). And I work on Globalization Pipeline. These two haven’t really crossed paths… until now.

What we’ll do

This blog will cover how to use the Globalization Pipeline to translate uconv, one of ICU’s sample command line apps. We'll be translating the resource files you can see in source/extra/uconv/resources

Gathering the tools

  • First, Download ICU4C source code (as a tarball or from the SVN repository) and compile it. See its readme for more details.

  • Now, set up Globalization Pipeline. See our Quick Start Guide for getting your Globalization Pipeline instance created and set up.

    • In the GP dashboard, create a bundle named uconv. Select which languages you want to translate into, but don’t upload any strings. Click Save.

    • Also from the Bluemix dashboard, get the service credentials for your service. Save these in a file called mycreds.json that should look like the example in this document.

  • We’ll also need the gp-cli java tool, so download the latest jar from gp-java-tools.

Into the Pipeline

Now, let's get some translated content

Hm. These files are in icu4c resource format, which isn't (yet?) supported by Globalization Pipeline… directly. Let's try an interim step.

  • genrb -x root root.txt
  • genrb -x fr fr.txt

Now we have root.xlf and fr.xlf (for good measure).

Here's a snippet of root.xlf:

1
2
3
4
5
<group id = "root" restype = "x-icu-table">
<trans-unit id = "U_BUFFER_OVERFLOW_ERROR" resname = "U_BUFFER_OVERFLOW_ERROR">
<source>Buffer overflow</source>
</trans-unit>

OK. The gp-cli tool says it handles XLIFF as a file format. Let's get that set up.

  • java -jar gp-cli-1.1.0.jar import -b uconv -f root.xlf -l en -t xliff -j mycreds.json

Note that we use the language tag en for English here, while the file was originally entitled root. This is because Globalization Pipeline works with the explicit source language, whereas for ICU, root is what will be consulted as a fallback if no other languages are available.

It says it uploaded… but let’s check in the Globalization Pipeline dashboard:

English content uploaded

OK! That’s great. Browsing over to the other language translations, we can see that the MT engines are hard at work. However, we happen to already have some French translations in the ICU source base. We'll upload this, to overwrite some of the Machine-translated entries for French:

  • java -jar gp-cli-1.1.0.jar import -b uconv -f fr.xlf -l fr -t xliff -j mycreds.json

Great. Now we have some human translated content as well. We can now correct, upload/download content in the dashboard until we are happy with the translations there.

Out of the Pipeline

OK, now for the next step- getting those translations back into ICU4C.

We can list the bundle status from the command line:

  • java -jar gp-cli-1.1.0.jar show-bundle -b uconv -j mycreds.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"sourceLanguage": "en",
"targetLanguages": [
"de",
"es",
"fr",
"it",
"ja",
"ko",
"pt-BR",
"zh-Hans",
"zh-Hant"
],
"readOnly": false,
"updatedBy": "…srloomis@us.ibm.com",
"updatedAt": "2016-07-14T15:22:40.233-07"
}

Now, we’ll download the files in XLIFF format again:

  • java -jar gp-cli-1.1.0.jar export -b uconv -f fr.xlf -l fr -t xliff -j mycreds.json
  • java -jar gp-cli-1.1.0.jar export -b uconv -f es.xlf -l es -t xliff -j mycreds.json
  • java -jar gp-cli-1.1.0.jar export -b uconv -f de.xlf -l de -t xliff -j mycreds.json
  • java -jar gp-cli-1.1.0.jar export -b uconv -f zh.xlf -l zh-Hans -t xliff -j mycreds.json

… and so on. Repeat for each language you wish to download. Note that we’ve used zh for Chinese instead of zh-Hans.

OK, we have XLIFF format. How to convert it to ICU format? genrb only writes XLIFF, it can’t read it.

And back again… almost.

We need the XLIFF2ICU Converter as is noted here.

To build it, at present, this worked for me:

  • download ICU4J source (yes, J)
  • run ant xliff from the top level
  • you will end up with an out/xliff/lib/xliff.jar

Still with me? Head back to the uconv/resources directory, and now run:

  • java -jar xliff.jar -s . -d . fr.xlf

And that brings us to…

1
2
3
4
Processing file: ./fr.xlf
The XLIFF document is invalid, please check it first:
Line 3, Column 81
Error: cvc-elt.1: No se ha encontrado la declaración del elemento 'xliff'.

Hrm. Seems like the XLIFF output isn't quite ready to be consumed. I filed a bug on this, of course.

Plan B

We're so close… let's see what we can do. What if we fetch the data in JSON format, and then hack up something to convert it to ICU format? It might suffice for this blog post.

Let's run the fetches again, but get JSON this time:

  • java -jar gp-cli-1.1.0.jar export -b uconv -f fr.json -l fr -t json -j mycreds.json

Now, run the following Node.js script over the JSON files:

  • node js2icu.js fr.json es.json …
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// js2icu.js
const fs=require('fs');
const args = process.argv.slice(2);
for (var i in args) {
const f = args[i];
console.log('# read ' + f);
const loc = f.split('.')[0];
const json = JSON.parse(fs.readFileSync(f));
var s = '\ufeff// -*- Coding: utf-8; -*-\n//auto converted\n' + loc + '\n{\n';
for (var k in json) {
s = s + ' ' + k + ' { "' + json[k].replace(/"/g,'\\"') + '" }\n';
}
s = s + '}\n';
console.log('# wrote ' + loc + '.txt');
fs.writeFileSync(loc+'.txt', s);
}

You should be the proud owner of .txt files matching all of the languages you are using.

We're almost there. Let's go up and build uconv:

  • cd ..

Now edit resfiles.mk and change the RESSRC line to reference the new translations:

1
RESSRC = $(RESOURCESDIR)$(FILESEPCHAR)root.txt $(RESOURCESDIR)$(FILESEPCHAR)fr.txt $(RESOURCESDIR)$(FILESEPCHAR)es.txt $(RESOURCESDIR)$(FILESEPCHAR)zh.txt

Build uconv

  • make

Done

Let’s test it. I know uwmsg.o isn't really utf-8, that's why this is a test.

  • env LC_ALL=es ./../../bin/uconv -f utf-8 < uwmsg.o
1
La conversión a Unicode de página de códigos falló en posición de byte de entrada de 0. Bytes: Error de cf: El carácter ilegal encontró La conversión a Unicode de página de códigos falló en posición de byte de entrada de 1. ……

Looks like we have a (more) translated uconv now. Some of the messages don’t quite work correctly due to ICU4C message conventions. Perhaps we will investigate this in the future.

Perl on Bluemix

Quick Start

  1. Marcus DelGreco at #FluentConf said something about perl support on platforms. I mentioned Bluemix allowed bring your own buildpack

  2. Looking through the buildpack lists didn't turn up Perl per se but…

  3. … enter sourcey-buildpack. It's a generic buildpack! From its README I knew I was in the right spot:

    Isn't it simply amazing to see these demos, where they throw a bunch of php, ruby, Java or python code at a Cloud Foundry site and it gets magically turned into a running web applications. Alas for me, life is often a wee bit more complicated than that. My projects always seem to required a few extra libraries or they are even written in an dead scripting language like Perl.

  4. And now for that tl;dr-inspiring moment:

Let's see if the canned sample works. Hint: yes.

First, cf login into Bluemix, and then:

$ git clone https://github.com/oetiker/sourcey-buildpack
$ cd sourcey-buildpack/example
$ cf push MYAPPLICATION$$ -m 128M -b https://github.com/oetiker/sourcey-buildpack

The above builds perl (takes a while the first time) and deploys a little app that just dumps the deserialized JSON out.

Improving

But wait! It could be even simpler. So, I opened PR oetiker/sourcey-buildpack#2 which adds a manifest file to the example. Then, only cf push is needed, the -b … option is now unnecessary.