Four months have passed since our first article about Moon — a Selenium-compatible browser automation solution created to work in Kubernetes or Openshift cluster. In this article I mostly described our motivation to build such solution and how to quickly try it. Today I would like to dive into the implementation details and explain how Moon works under the hood and why it is so efficient.
Moon is a first-class Kubernetes citizen and is always working inside it. Like any other application Moon is deployed as one or more pods and a service delivering network connectivity. Every pod includes:
- Moon — the main application providing Selenium API and by default listening on standard Selenium port
4444. Your tests should be launched against this port.
- Moon API — a supplementary component providing
/statusHTTP API and by default available on port
8080. Moon API returns additional cluster status information such as total cluster capacity, list of running browser sessions and so on.
- Moon UI — an optional graphical user interface visualizing cluster state retrieved from Moon API
/statusAPI. Currently we are using Selenoid UI as Moon user interface but more advanced solution is going to be delivered soon.
All these components are web-applications and in truly fault-tolerant cluster should have 2 or more replicas behind load-balancer. You can organize load-balancing using two Kubernetes primitives:
- LoadBalancer — a generic L4 network load-balancer distributing IP-traffic across configured nodes.
- Ingress — an L7 HTTP load-balancer with an ability to proxy requests to concrete HTTP URLs to specified services. For example you can give access to Selenium API in Moon service at
/statuswill lead to Moon API.
Internally Moon is using Kubernetes API to launch pods (and corresponding services) with browsers and then retrieve the entire cluster state. If you have already worked with Kubernetes — you should know that every service has a name which is also a network host name available in Kubernetes DNS. For browser pods Moon uses generated unique names containing information about launched browser, e.g.
firefox-61-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840 which corresponds to
Firefox 61.0. When Moon user requests a new browser session it receives created service name as a Selenium session ID instead of random UUID. This session ID is then sent in subsequent Selenium session requests and Moon is able to proxy the traffic to correct Kubernetes service with browser. Such simple idea makes Moon completely stateless and allows to start an unlimited number of replicas behind chosen load balancer.
What’s Inside Browser Pod
aandryashin:/$ kubectl describe po \
chrome-68-0-a607028a-3f25-4752-b463-94438cdd8aaf -n moon
Ports: 4444/TCP, 6099/TCP
Output above is a list of containers running in every browser pod when video recording and S3 upload are enabled. As you can see it consists of 4 containers:
- browser — an image with browser that actually executes Selenium tests. Is is exposing standard Selenium port
4444and its X-server port
6099for video recording purposes.
- defender — an utility proxying container guaranteeing that only one Selenium session is created in the pod. This container actively prevents (or defends) browser container from accepting the second new session request and also handles session timeout cases when pod is automatically deleted. Because of defender Moon does not need to store timeout state for every running session and thus remains completely stateless.
- video-recorder — a supplementary container with FFMpeg video processing software. It is used to capture the video from X-server when browser is actually running. When S3 uploading is enabled — this container additionally uploads recorded video file to S3.
- logger — a supplementary container used to upload browser session logs to S3 when it finishes.
So depending on Moon configuration and capabilities set in test code browser pod contains from two (browser and defender) to four containers.
Controlling Resources Consumption
The next important question I would like to talk about is browser resources consumption. Moon allows you to easily adjust guaranteed and maximum CPU and memory quantities for every running browser pod. This can be done globally for every browser version by providing a flag to Moon container:
Running Moon in Isolated Environment
A frequent situation in big companies is Kubernetes running in isolated environment without access to Internet. This is mostly being done because of security reasons and Moon fully supports such mode of operation. To work in isolated environment you have to use your own internal Docker images registry to store Moon images. Having a working registry and network access to it from Kubernetes you have to reconfigure Moon to use this internal registry. This is done by specifying desired Docker images in
browsers.json file to override browser image and in
service.json file to override system containers such as
defender. For example to use custom browser images your
browsers.json file can look like the following:
Overriding system images is done as shown below:
These configuration files are then mounted to Moon pod as config maps and Moon is able to read and apply them.
Running Android Emulators in Kubernetes
The last question I would like to discuss in this article is how to run tests on “difficult” platforms such as Android. If you take a look at my Android-related article — you will understand that the most efficient and the least expensive way to run tests on Android platform is using Android emulators. Fast Android emulators compared to desktop browsers have an important particularity. Using virtualization technologies they require a certain number of processor instructions to be supported on the host where they are started. Such virtualization support usually exists either or hardware servers or virtual machines with nested virtualization support. And such machines are usually slightly more expensive than standard ones.
Fortunately Kubernetes can easily run on any heterogeneous set of hardware and virtual machines. The only thing we need to stay efficient is to be able to run Android pods on expensive hardware and the rest of browsers — on the cheap VMs. Such feature already exists and is called —
node selectors. Every machine hosting Kubernetes is called a
node and can be marked with a set of custom labels. When launching a pod you can set a requirement to start only on a node having some labels set. For example for Android you can mark every hardware node with
expensive-hardware label and then configure Moon
browsers.json file to follow such limitation:
Now Android emulators will be running on such expensive hardware and the rest of browsers will be cheaper for you.
Still Having Questions?
We have several free support channels:
- Moon documentation: https://aerokube.com/moon/latest
- Our support email: firstname.lastname@example.org
- Our Telegram support channel: https://t.me/aerokube_moon
- Our Github issues page: https://github.com/aerokube/moon/issues
Bonus: Free License Keys
We want to deliver a first-class polished Selenium solution and thus need more feedback from you. So here is how to get a free license key for desired number of parallel sessions:
- Open https://moon.aerokube.com/ and find Moon Evaluation License form.
- Fill your company name, an email address to send the license key and desired number of parallel sessions.
- Click on “Send me a key”.
You will then immediately receive a free license key valid until March, 1 2019 to the email specified.
Any More Articles?
Definitely. Take a look at these ones:
- Selenium: back to the Moon
- Selenoid: storing data efficiently
- Selenoid: more Android sweets
- Selenium: a new hope (part I)
- Selenium: a new hope (part II)
- Selenium: done in 60 seconds
- Selenium on Windows: revisited
- Selenium: easy as a pie
- Selenium: an Apple story
- Selenium: Growing Muscles
- Selenium: Clear as a Bell